今日论文合集：cs.SD语音12篇，eess.AS音频处理15篇。
cs.SD语音
【1】 Developing Speech Processing Pipelines for Police Accountability标题：开发警察问责语音处理流水线链接：https://arxiv.org/abs/2306.06086
作者：Anjalie Field,Prateek Verma,Nay San,Jennifer L. Eberhardt,Dan Jurafsky备注：Accepted to INTERSPEECH 2023摘要：警察随身携带的摄像机有可能改善警务的问责制和透明度。然而，在实践中，他们导致数百万小时的镜头，从来没有审查。我们研究了大型预训练语音模型的潜力，以促进审查，重点是ASR和官员的语音检测从交通站的镜头。我们提出的管道包括训练数据对齐和过滤，资源约束的微调，以及将官员语音检测与ASR相结合，以实现完全自动化的方法。我们发现，（1）微调强烈地提高了对官员语音的ASR性能（WER=12-13%），（2）对官员语音的ASR比对社区成员语音的ASR准确得多（WER=43.55-49.07%），（3）诸如官员语音检测和日志化的特定领域任务仍然具有挑战性。我们的工作提供了审查身体摄像机镜头的实际应用，以及将预训练的语音模型适应嘈杂的多说话者域的一般指导。摘要：Police body-worn cameras have the potential to improve accountability and transparency in policing. Yet in practice, they result in millions of hours of footage that is never reviewed. We investigate the potential of large pre-trained speech models for facilitating reviews, focusing on ASR and officer speech detection in footage from traffic stops. Our proposed pipeline includes training data alignment and filtering, fine-tuning with resource constraints, and combining officer speech detection with ASR for a fully automated approach. We find that (1) fine-tuning strongly improves ASR performance on officer speech (WER=12-13%), (2) ASR on officer speech is much more accurate than on community member speech (WER=43.55-49.07%), (3) domain-specific tasks like officer speech detection and diarization remain challenging. Our work offers practical applications for reviewing body camera footage and general guidance for adapting pre-trained speech models to noisy multi-speaker domains.
【2】 Improving Fairness and Robustness in End-to-End Speech Recognition  through unsupervised clustering标题：无监督聚类提高端到端语音识别的公平性和稳健性链接：https://arxiv.org/abs/2306.06083
作者：Irina-Elena Veliche,Pascale Fung备注：None摘要：当自动语音识别（ASR）系统对人口的所有子组的表现不相同时，公平性的挑战出现。在过去的几年里，语音识别的整体质量有了很大的提高，但对于那些系统性能不佳的用户群体来说，并没有特别关注促进平等和公平。ASR公平性因此也是鲁棒性问题。同时，数据隐私在生产系统中也具有优先级。在本文中，我们提出了一种隐私保护方法，以提高端到端ASR的公平性和鲁棒性，而不使用元数据，邮政编码，甚至扬声器或话语嵌入直接在训练中。我们使用在公共数据集上训练的扬声器ID模型提取话语级嵌入，然后以无监督的方式使用该模型来创建声学集群。在模型训练过程中，我们使用聚类ID而不是说话者话语嵌入作为额外的特征，这表明所有人口统计组，特别是不同口音的改进。摘要：The challenge of fairness arises when Automatic Speech Recognition (ASR) systems do not perform equally well for all sub-groups of the population. In the past few years there have been many improvements in overall speech recognition quality, but without any particular focus on advancing Equality and Equity for all user groups for whom systems do not perform well. ASR fairness is therefore also a robustness issue. Meanwhile, data privacy also takes priority in production systems. In this paper, we present a privacy preserving approach to improve fairness and robustness of end-to-end ASR without using metadata, zip codes, or even speaker or utterance embeddings directly in training. We extract utterance level embeddings using a speaker ID model trained on a public dataset, which we then use in an unsupervised fashion to create acoustic clusters. We use cluster IDs instead of speaker utterance embeddings as extra features during model training, which shows improvements for all demographic groups and in particular for different accents.
【3】 Reconstructing Human Expressiveness in Piano Performances with a  Transformer Network标题：用变形网络重建钢琴表演中的人类表现力链接：https://arxiv.org/abs/2306.06040
作者：Jingjing Tang,Geraint Wiggins,George Fazekas备注：12 pages, 5 figures, submitted to CMMR 2023摘要：使用计算方法捕捉音乐表演中人类表现力的复杂和微妙变化是具有挑战性的。在本文中，我们提出了一种新的方法来重建人类的表现力在钢琴演奏与多层双向Transformer编码器。为了解决在训练神经网络时对大量准确捕获和分数对齐的性能数据的需求，我们使用从现有转录模型获得的转录分数来训练我们的模型。我们整合钢琴家的身份来控制采样过程，并探索我们的系统的能力，为不同的钢琴家的表现力的变化模型。该系统是通过统计分析所产生的表现力和听力测试进行评估。总的来说，结果表明，我们的方法实现了最先进的从转录的分数生成类人的钢琴表演，而完全和一致的重建人类的表现力带来了进一步的挑战。摘要：Capturing intricate and subtle variations in human expressiveness in music performance using computational approaches is challenging. In this paper, we propose a novel approach for reconstructing human expressiveness in piano performance with a multi-layer bi-directional Transformer encoder. To address the needs for large amounts of accurately captured and score-aligned performance data in training neural networks, we use transcribed scores obtained from an existing transcription model to train our model. We integrate pianist identities to control the sampling process and explore the ability of our system to model variations in expressiveness for different pianists. The system is evaluated through statistical analysis of generated expressive performances and a listening test. Overall, the results suggest that our method achieves state-of-the-art in generating human-like piano performances from transcribed scores, while fully and consistently reconstructing human expressiveness poses further challenges.
【4】 Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion标题：利用线性扩散技术促进快速高质量语音合成链接：https://arxiv.org/abs/2306.05708
作者：Haogeng Liu,Tao Wang,Jie Cao,Ran He,Jianhua Tao摘要：去噪扩散概率模型在各种生成任务上表现出非凡的能力。然而，他们的推理速度慢，使他们不切实际的语音合成。本文提出了一种基于常微分方程的线性扩散模型（LinDiff），同时达到快速推理和高样本质量。首先，我们采用目标和噪声之间的线性插值来设计用于训练的扩散序列，而先前连接噪声和目标的扩散路径是曲线段。当减少采样步骤的数量（即，用于拟合路径的线段的数量），与曲线相比拟合直线的容易性允许我们以更少的迭代从随机噪声生成更高质量的样本。其次，为了降低计算复杂度，实现有效的噪声语音的全局建模，LinDiff采用了基于补丁的处理方法，将输入信号划分成小补丁。分片令牌利用Transformer架构对全局信息进行有效建模。对抗训练用于进一步提高样本质量，减少采样步骤。我们测试所提出的方法与语音合成条件的声学特征（梅尔频谱图）。实验结果表明，该模型可以合成高质量的语音，即使只有一个扩散步骤。主观和客观的评价表明，我们的模型可以合成语音的质量相媲美的自回归模型具有更快的合成速度（3扩散步骤）。摘要：Denoising Diffusion Probabilistic Models have shown extraordinary ability on various generative tasks. However, their slow inference speed renders them impractical in speech synthesis. This paper proposes a linear diffusion model (LinDiff) based on an ordinary differential equation to simultaneously reach fast inference and high sample quality. Firstly, we employ linear interpolation between the target and noise to design a diffusion sequence for training, while previously the diffusion path that links the noise and target is a curved segment. When decreasing the number of sampling steps (i.e., the number of line segments used to fit the path), the ease of fitting straight lines compared to curves allows us to generate higher quality samples from a random noise with fewer iterations. Secondly, to reduce computational complexity and achieve effective global modeling of noisy speech, LinDiff employs a patch-based processing approach that partitions the input signal into small patches. The patch-wise token leverages Transformer architecture for effective modeling of global information. Adversarial training is used to further improve the sample quality with decreased sampling steps. We test proposed method with speech synthesis conditioned on acoustic feature (Mel-spectrograms). Experimental results verify that our model can synthesize high-quality speech even with only one diffusion step. Both subjective and objective evaluations demonstrate that our model can synthesize speech of a quality comparable to that of autoregressive models with faster synthesis speed (3 diffusion steps).
【5】 Low-rank Adaptation Method for Wav2vec2-based Fake Audio Detection标题：基于Wav2ve2的低阶自适应伪音频检测方法链接：https://arxiv.org/abs/2306.05617
作者：Chenglong Wang,Jiangyan Yi,Xiaohui Zhang,Jianhua Tao,Le Xu,Ruibo Fu备注：None摘要：自监督语音模型是虚假音频检测中一个发展迅速的研究课题。许多预先训练的模型可以作为特征提取器，学习更丰富和更高级别的语音特征。然而，当对预训练模型进行微调时，通常会面临训练时间过长和内存消耗过高的挑战，并且完全微调也非常昂贵。为了缓解这个问题，我们将低秩自适应（LoRA）应用于wav2vec2模型，冻结预训练的模型权重，并将可训练的秩分解矩阵注入到Transformer架构的每一层中，大大减少了下游任务的可训练参数的数量。与Adam在包含317M训练参数的wav2vec2模型上进行微调相比，LoRA通过将可训练参数的数量减少198倍来实现类似的性能。摘要：Self-supervised speech models are a rapidly developing research topic in fake audio detection. Many pre-trained models can serve as feature extractors, learning richer and higher-level speech features. However,when fine-tuning pre-trained models, there is often a challenge of excessively long training times and high memory consumption, and complete fine-tuning is also very expensive. To alleviate this problem, we apply low-rank adaptation(LoRA) to the wav2vec2 model, freezing the pre-trained model weights and injecting a trainable rank-decomposition matrix into each layer of the transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared with fine-tuning with Adam on the wav2vec2 model containing 317M training parameters, LoRA achieved similar performance by reducing the number of trainable parameters by 198 times.
【6】 Detecting Check-Worthy Claims in Political Debates, Speeches, and  Interviews Using Audio Data标题：使用音频数据在政治辩论、演讲和采访中检测值得检查的声明链接：https://arxiv.org/abs/2306.05535
作者：Petar Ivanov,Ivan Koychev,Momchil Hardalov,Preslav Nakov备注：check-worthy claims, fake news, political debates, audio摘要：社会上有很大一部分人团结在同一个愿景和想法周围，蕴藏着巨大的能量。这正是政治人物希望为他们的事业积累的东西。考虑到这一目标，他们有时会无意或有意地歪曲或隐藏真相，这为错误信息和虚假信息打开了大门。自动检测值得检查的声明的工具将对辩论主持人、记者和事实核查组织有很大帮助。虽然以前的工作检测检查值得索赔的重点是文本，在这里，我们探讨的实用程序的音频信号作为一个额外的信息源。我们创建了一个新的多模态数据集（英语文本和音频），其中包含48小时的语音。我们的评估结果表明，在多个扬声器的情况下，音频模态与文本一起产生的改进超过单独的文本。此外，对于单个说话者，仅音频模型的性能可能优于仅文本模型。摘要：A large portion of society united around the same vision and ideas carries enormous energy. That is precisely what political figures would like to accumulate for their cause. With this goal in mind, they can sometimes resort to distorting or hiding the truth, unintentionally or on purpose, which opens the door for misinformation and disinformation. Tools for automatic detection of check-worthy claims would be of great help to moderators of debates, journalists, and fact-checking organizations. While previous work on detecting check-worthy claims has focused on text, here we explore the utility of the audio signal as an additional information source. We create a new multimodal dataset (text and audio in English) containing 48 hours of speech. Our evaluation results show that the audio modality together with text yields improvements over text alone in the case of multiple speakers. Moreover, an audio-only model could outperform a text-only one for a single speaker.
【7】 An Efficient Speech Separation Network Based on Recurrent Fusion Dilated  Convolution and Channel Attention标题：基于递归融合膨胀卷积和通道注意的高效语音分离网络链接：https://arxiv.org/abs/2306.05887
作者：Junyu Wang备注：Accepted by Interspeech 2023摘要：我们提出了一个有效的语音分离神经网络，ARFDCN，它结合了扩张卷积，多尺度融合（MSF），和信道的注意力，以克服有限的接收域的卷积为基础的网络和高计算成本的变压器为基础的网络。所建议的网络架构是基于编码器-解码器的。通过使用具有逐渐增加的膨胀值的膨胀卷积来学习局部和全局特征并在相邻阶段融合它们，模型可以学习丰富的特征内容。同时，通过在网络中加入信道关注度模块，该模型可以提取信道权值，学习更重要的特征，从而提高其表达能力和鲁棒性。实验结果表明，该模型实现了性能和计算效率之间的一个体面的平衡，使其成为一个有前途的替代目前的主流模型的实际应用。摘要：We present an efficient speech separation neural network, ARFDCN, which combines dilated convolutions, multi-scale fusion (MSF), and channel attention to overcome the limited receptive field of convolution-based networks and the high computational cost of transformer-based networks. The suggested network architecture is encoder-decoder based. By using dilated convolutions with gradually increasing dilation value to learn local and global features and fusing them at adjacent stages, the model can learn rich feature content. Meanwhile, by adding channel attention modules to the network, the model can extract channel weights, learn more important features, and thus improve its expressive power and robustness. Experimental results indicate that the model achieves a decent balance between performance and computational efficiency, making it a promising alternative to current mainstream models for practical applications.
【8】 Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive  Feature Learning in Speech Enhancement标题：语音增强中用于综合特征学习的高效编解码器和双径整形器链接：https://arxiv.org/abs/2306.05861
作者：Junyu Wang备注：Accepted at Interspeech2023摘要：目前的语音增强（SE）的研究在很大程度上忽略了信道的注意力和空间的注意力，和编码器-解码器架构为基础的网络没有充分考虑如何提供有效的输入到中间增强层。为了解决这些问题，本文提出了一个时间-频率（T-F）域SE网络（DPCFCS-Net），它采用了改进的密集连接块，双路径模块，卷积增强Transformers（构象），信道注意力和空间注意力。与以前的模型相比，我们提出的模型具有更有效的编码器-解码器，可以学习全面的功能。在VCTK+DEMAND数据集上的实验结果表明，该方法在SE性能上优于现有技术。此外，在这项工作中开发的改进的密集连接块和二维注意力模块具有高度的适应性，易于集成到现有的网络。摘要：Current speech enhancement (SE) research has largely neglected channel attention and spatial attention, and encoder-decoder architecture-based networks have not adequately considered how to provide efficient inputs to the intermediate enhancement layer. To address these issues, this paper proposes a time-frequency (T-F) domain SE network (DPCFCS-Net) that incorporates improved densely connected blocks, dual-path modules, convolution-augmented transformers (conformers), channel attention, and spatial attention. Compared with previous models, our proposed model has a more efficient encoder-decoder and can learn comprehensive features. Experimental results on the VCTK+DEMAND dataset demonstrate that our method outperforms existing techniques in SE performance. Furthermore, the improved densely connected block and two dimensions attention module developed in this work are highly adaptable and easily integrated into existing networks.
【9】 HRTF upsampling with a generative adversarial network using a gnomonic  equiangular projection标题：基于等角投影的生成性对抗性网络HRTF上采样链接：https://arxiv.org/abs/2306.05812
作者：Aidan O. T. Hogg,Mads Jenkins,He Liu,Isaac Squires,Samuel J. Cooper,Lorenzo Picinali备注：13 pages, 9 figures, Preprint (Submitted to Transactions on Audio, Speech and Language Processing on the 24 Feb 2023)摘要：个性化的头部相关传递函数（HRTF）对于创建逼真的虚拟现实（VR）和增强现实（AR）环境至关重要。然而，声学测量高质量的HRTF需要昂贵的设备和声学实验室设置。为了克服这些限制并使该测量更有效，过去已经利用了HRTF上采样，其中从低分辨率HRTF创建高分辨率HRTF。本文演示了如何将生成对抗网络（GANs）应用于HRTF上采样。我们提出了一种新的方法，将HRTF数据转换为方便使用的卷积超分辨率生成对抗网络（SRGAN）。这一新方法以两个基线为基准：重心上采样和HRTF选择方法。实验结果表明，所提出的方法优于两个基线的对数谱失真（LSD）和使用感知模型时，输入HRTF是稀疏的定位性能。摘要：An individualised head-related transfer function (HRTF) is essential for creating realistic virtual reality (VR) and augmented reality (AR) environments. However, acoustically measuring high-quality HRTFs requires expensive equipment and an acoustic lab setting. To overcome these limitations and to make this measurement more efficient HRTF upsampling has been exploited in the past where a high-resolution HRTF is created from a low-resolution one. This paper demonstrates how generative adversarial networks (GANs) can be applied to HRTF upsampling. We propose a novel approach that transforms the HRTF data for convenient use with a convolutional super-resolution generative adversarial network (SRGAN). This new approach is benchmarked against two baselines: barycentric upsampling and a HRTF selection approach. Experimental results show that the proposed method outperforms both baselines in terms of log-spectral distortion (LSD) and localisation performance using perceptual models when the input HRTF is sparse.
【10】 Learning Emotional Representations from Imbalanced Speech Data for  Speech Emotion Recognition and Emotional Text-to-Speech标题：用于语音情感识别和情感文语转换的不平衡语音数据的情感表示学习链接：https://arxiv.org/abs/2306.05709
作者：Shijun Wang,Jón Guðnason,Damian Borth备注：Accepted by INTERSPEECH2023摘要：有效的言语情感表征在言语情感识别（SER）和情感文语转换（TTS）任务中起着关键作用。然而，与中性风格的语音相比，情感语音样本更难获得，也更昂贵，这导致了大多数相关作品不幸忽视的一个问题：不平衡的数据集模型可能会过度拟合大多数中性类，无法产生强大而有效的情感表示。在本文中，我们提出了一个情感提取器来解决这个问题。我们使用增强方法来训练模型，使其能够从不平衡的数据集中提取有效和可推广的情感表示。我们的实验结果表明：（1）对于SER任务，所提出的情感提取器在三个不平衡数据集上超过了最先进的基线;（2）情感提取器产生的情感表征有利于TTS模型，使其能够合成更具表达力的语音。摘要：Effective speech emotional representations play a key role in Speech Emotion Recognition (SER) and Emotional Text-To-Speech (TTS) tasks. However, emotional speech samples are more difficult and expensive to acquire compared with Neutral style speech, which causes one issue that most related works unfortunately neglect: imbalanced datasets. Models might overfit to the majority Neutral class and fail to produce robust and effective emotional representations. In this paper, we propose an Emotion Extractor to address this issue. We use augmentation approaches to train the model and enable it to extract effective and generalizable emotional representations from imbalanced datasets. Our empirical results show that (1) for the SER task, the proposed Emotion Extractor surpasses the state-of-the-art baseline on three imbalanced datasets; (2) the produced representations from our Emotion Extractor benefit the TTS model, and enable it to synthesize more expressive speech.
【11】 Domestic Activities Classification from Audio Recordings Using  Multi-scale Dilated Depthwise Separable Convolutional Network标题：基于多尺度扩展纵向可分离卷积网络的家庭活动分类链接：https://arxiv.org/abs/2306.05624
作者：Yufei Zeng,Yanxiong Li,Zhenfeng Zhou,Ruiqi Wang,Difeng Lu备注：5 pages, 2 figures, 4 tables. Accepted for publication in IEEE MMSP2021摘要：音频记录的家庭活动分类（DAC）旨在将音频记录分类到预定义的家庭活动类别中，这是用于估计在家庭环境中进行的日常活动的有效方式。在本文中，我们提出了一种使用多尺度扩张深度可分离卷积网络（DSCN）从音频记录中进行DAC的方法。DSCN是具有小尺寸参数的轻量级神经网络，因此适合于部署在具有有限计算资源的便携式终端中。为了在DSCN参数相同的情况下扩大感受野，在DSCN中使用扩张卷积代替常规卷积，以进一步提高DSCN的性能。此外，由扩张的DSCN学习的各种尺度的嵌入被连接为多尺度嵌入，用于表示各种类别的国内活动之间的属性差异。在2018年声学场景和事件的检测和分类挑战（DCASE-2018）的任务5的公共数据集上进行评估，结果显示：扩张卷积和多尺度嵌入都有助于改进算法的性能;并且所提出的方法在分类精度方面优于基于最新技术的轻量级网络的方法。摘要：Domestic activities classification (DAC) from audio recordings aims at classifying audio recordings into pre-defined categories of domestic activities, which is an effective way for estimation of daily activities performed in home environment. In this paper, we propose a method for DAC from audio recordings using a multi-scale dilated depthwise separable convolutional network (DSCN). The DSCN is a lightweight neural network with small size of parameters and thus suitable to be deployed in portable terminals with limited computing resources. To expand the receptive field with the same size of DSCN's parameters, dilated convolution, instead of normal convolution, is used in the DSCN for further improving the DSCN's performance. In addition, the embeddings of various scales learned by the dilated DSCN are concatenated as a multi-scale embedding for representing property differences among various classes of domestic activities. Evaluated on a public dataset of the Task 5 of the 2018 challenge on Detection and Classification of Acoustic Scenes and Events (DCASE-2018), the results show that: both dilated convolution and multi-scale embedding contribute to the performance improvement of the proposed method; and the proposed method outperforms the methods based on state-of-the-art lightweight network in terms of classification accuracy.
【12】 Acoustic Scene Clustering Using Joint Optimization of Deep Embedding  Learning and Clustering Iteration标题：深度嵌入学习和聚类迭代联合优化的声场景聚类链接：https://arxiv.org/abs/2306.05621
作者：Yanxiong Li,Mingle Liu,Wucheng Wang,Yuhan Zhang,Qianhua He备注：9 pages, 6 figures, 11 tables. Accepted for publication in IEEE TMM摘要：最近的努力已经在音频信号处理社区中的声学场景分类。相比之下，很少有研究已经进行了声学场景聚类，这是一个新出现的问题。声学场景聚类的目的是在不利用先验信息和训练分类器的情况下，将同一类声学场景的音频记录合并到一个单独的簇中。在这项研究中，我们提出了一种声学场景聚类的方法，共同优化的程序的特征学习和聚类迭代。在所提出的方法中，学习的特征是从深度卷积神经网络（CNN）中提取的深度嵌入，而聚类算法是凝聚层次聚类（AHC）。我们制定了一个统一的损失函数整合和优化这两个程序。各种功能和方法进行了比较。实验结果表明，该方法优于其他无监督方法的归一化互信息和聚类精度。此外，深度嵌入优于许多最先进的功能。摘要：Recent efforts have been made on acoustic scene classification in the audio signal processing community. In contrast, few studies have been conducted on acoustic scene clustering, which is a newly emerging problem. Acoustic scene clustering aims at merging the audio recordings of the same class of acoustic scene into a single cluster without using prior information and training classifiers. In this study, we propose a method for acoustic scene clustering that jointly optimizes the procedures of feature learning and clustering iteration. In the proposed method, the learned feature is a deep embedding that is extracted from a deep convolutional neural network (CNN), while the clustering algorithm is the agglomerative hierarchical clustering (AHC). We formulate a unified loss function for integrating and optimizing these two procedures. Various features and methods are compared. The experimental results demonstrate that the proposed method outperforms other unsupervised methods in terms of the normalized mutual information and the clustering accuracy. In addition, the deep embedding outperforms many state-of-the-art features.
eess.AS音频处理
【1】 Speaker Embeddings as Individuality Proxy for Voice Stress Detection标题：说话人嵌入作为语音重音检测的个性代理链接：https://arxiv.org/abs/2306.05915
作者：Zihan Wu,Neil Scheidwasser-Clow,Karl El Hajal,Milos Cernak备注：5 pages, 2 figures. Accepted at Interspeech 2023摘要：由于说话者的精神状态调节语音，因此可以在语音中检测到由认知或物理负荷引入的压力。现有的语音重音检测基准测试表明，从混合BYOL-S自监督模型中提取的音频嵌入表现良好。然而，基准测试仅在每个数据集上单独评估性能，而不评估不同类型的压力和不同语言的性能。此外，先前的研究发现，压力易感性存在很强的个体差异。本文介绍了语音压力检测的设计和开发，来自9个语言组和五种不同类型的压力超过100扬声器的培训。我们通过在混合BYOL-S特征中添加扬声器嵌入来解决语音应力分析中的个体差异。所提出的方法显着提高语音重音检测性能，输入音频长度仅为3-5秒。摘要：Since the mental states of the speaker modulate speech, stress introduced by cognitive or physical loads could be detected in the voice. The existing voice stress detection benchmark has shown that the audio embeddings extracted from the Hybrid BYOL-S self-supervised model perform well. However, the benchmark only evaluates performance separately on each dataset, but does not evaluate performance across the different types of stress and different languages. Moreover, previous studies found strong individual differences in stress susceptibility. This paper presents the design and development of voice stress detection, trained on more than 100 speakers from 9 language groups and five different types of stress. We address individual variabilities in voice stress analysis by adding speaker embeddings to the hybrid BYOL-S features. The proposed method significantly improves voice stress detection performance with an input audio length of only 3-5 seconds.
【2】 An Efficient Speech Separation Network Based on Recurrent Fusion Dilated  Convolution and Channel Attention标题：基于递归融合膨胀卷积和通道注意的高效语音分离网络链接：https://arxiv.org/abs/2306.05887
*与cs.SD语音【7】为同一篇
作者：Junyu Wang备注：Accepted by Interspeech 2023摘要：我们提出了一个有效的语音分离神经网络，ARFDCN，它结合了扩张卷积，多尺度融合（MSF），和信道的注意力，以克服有限的接收域的卷积为基础的网络和高计算成本的变压器为基础的网络。所建议的网络架构是基于编码器-解码器的。通过使用具有逐渐增加的膨胀值的膨胀卷积来学习局部和全局特征并在相邻阶段融合它们，模型可以学习丰富的特征内容。同时，通过在网络中加入信道关注度模块，该模型可以提取信道权值，学习更重要的特征，从而提高其表达能力和鲁棒性。实验结果表明，该模型实现了性能和计算效率之间的一个体面的平衡，使其成为一个有前途的替代目前的主流模型的实际应用。摘要：We present an efficient speech separation neural network, ARFDCN, which combines dilated convolutions, multi-scale fusion (MSF), and channel attention to overcome the limited receptive field of convolution-based networks and the high computational cost of transformer-based networks. The suggested network architecture is encoder-decoder based. By using dilated convolutions with gradually increasing dilation value to learn local and global features and fusing them at adjacent stages, the model can learn rich feature content. Meanwhile, by adding channel attention modules to the network, the model can extract channel weights, learn more important features, and thus improve its expressive power and robustness. Experimental results indicate that the model achieves a decent balance between performance and computational efficiency, making it a promising alternative to current mainstream models for practical applications.
【3】 Efficient Encoder-Decoder and Dual-Path Conformer for Comprehensive  Feature Learning in Speech Enhancement标题：语音增强中用于综合特征学习的高效编解码器和双径整形器链接：https://arxiv.org/abs/2306.05861
*与cs.SD语音【8】为同一篇
作者：Junyu Wang备注：Accepted at Interspeech2023摘要：目前的语音增强（SE）的研究在很大程度上忽略了信道的注意力和空间的注意力，和编码器-解码器架构为基础的网络没有充分考虑如何提供有效的输入到中间增强层。为了解决这些问题，本文提出了一个时间-频率（T-F）域SE网络（DPCFCS-Net），它采用了改进的密集连接块，双路径模块，卷积增强Transformers（构象），信道注意力和空间注意力。与以前的模型相比，我们提出的模型具有更有效的编码器-解码器，可以学习全面的功能。在VCTK+DEMAND数据集上的实验结果表明，该方法在SE性能上优于现有技术。此外，在这项工作中开发的改进的密集连接块和二维注意力模块具有高度的适应性，易于集成到现有的网络。摘要：Current speech enhancement (SE) research has largely neglected channel attention and spatial attention, and encoder-decoder architecture-based networks have not adequately considered how to provide efficient inputs to the intermediate enhancement layer. To address these issues, this paper proposes a time-frequency (T-F) domain SE network (DPCFCS-Net) that incorporates improved densely connected blocks, dual-path modules, convolution-augmented transformers (conformers), channel attention, and spatial attention. Compared with previous models, our proposed model has a more efficient encoder-decoder and can learn comprehensive features. Experimental results on the VCTK+DEMAND dataset demonstrate that our method outperforms existing techniques in SE performance. Furthermore, the improved densely connected block and two dimensions attention module developed in this work are highly adaptable and easily integrated into existing networks.
【4】 HRTF upsampling with a generative adversarial network using a gnomonic  equiangular projection标题：基于等角投影的生成性对抗性网络HRTF上采样链接：https://arxiv.org/abs/2306.05812
*与cs.SD语音【9】为同一篇
作者：Aidan O. T. Hogg,Mads Jenkins,He Liu,Isaac Squires,Samuel J. Cooper,Lorenzo Picinali备注：13 pages, 9 figures, Preprint (Submitted to Transactions on Audio, Speech and Language Processing on the 24 Feb 2023)摘要：个性化的头部相关传递函数（HRTF）对于创建逼真的虚拟现实（VR）和增强现实（AR）环境至关重要。然而，声学测量高质量的HRTF需要昂贵的设备和声学实验室设置。为了克服这些限制并使该测量更有效，过去已经利用了HRTF上采样，其中从低分辨率HRTF创建高分辨率HRTF。本文演示了如何将生成对抗网络（GANs）应用于HRTF上采样。我们提出了一种新的方法，将HRTF数据转换为方便使用的卷积超分辨率生成对抗网络（SRGAN）。这一新方法以两个基线为基准：重心上采样和HRTF选择方法。实验结果表明，所提出的方法优于两个基线的对数谱失真（LSD）和使用感知模型时，输入HRTF是稀疏的定位性能。摘要：An individualised head-related transfer function (HRTF) is essential for creating realistic virtual reality (VR) and augmented reality (AR) environments. However, acoustically measuring high-quality HRTFs requires expensive equipment and an acoustic lab setting. To overcome these limitations and to make this measurement more efficient HRTF upsampling has been exploited in the past where a high-resolution HRTF is created from a low-resolution one. This paper demonstrates how generative adversarial networks (GANs) can be applied to HRTF upsampling. We propose a novel approach that transforms the HRTF data for convenient use with a convolutional super-resolution generative adversarial network (SRGAN). This new approach is benchmarked against two baselines: barycentric upsampling and a HRTF selection approach. Experimental results show that the proposed method outperforms both baselines in terms of log-spectral distortion (LSD) and localisation performance using perceptual models when the input HRTF is sparse.
【5】 Learning Emotional Representations from Imbalanced Speech Data for  Speech Emotion Recognition and Emotional Text-to-Speech标题：用于语音情感识别和情感文语转换的不平衡语音数据的情感表示学习链接：https://arxiv.org/abs/2306.05709
*与cs.SD语音【10】为同一篇
作者：Shijun Wang,Jón Guðnason,Damian Borth备注：Accepted by INTERSPEECH2023摘要：有效的言语情感表征在言语情感识别（SER）和情感文语转换（TTS）任务中起着关键作用。然而，与中性风格的语音相比，情感语音样本更难获得，也更昂贵，这导致了大多数相关作品不幸忽视的一个问题：不平衡的数据集模型可能会过度拟合大多数中性类，无法产生强大而有效的情感表示。在本文中，我们提出了一个情感提取器来解决这个问题。我们使用增强方法来训练模型，使其能够从不平衡的数据集中提取有效和可推广的情感表示。我们的实验结果表明：（1）对于SER任务，所提出的情感提取器在三个不平衡数据集上超过了最先进的基线;（2）情感提取器产生的情感表征有利于TTS模型，使其能够合成更具表达力的语音。摘要：Effective speech emotional representations play a key role in Speech Emotion Recognition (SER) and Emotional Text-To-Speech (TTS) tasks. However, emotional speech samples are more difficult and expensive to acquire compared with Neutral style speech, which causes one issue that most related works unfortunately neglect: imbalanced datasets. Models might overfit to the majority Neutral class and fail to produce robust and effective emotional representations. In this paper, we propose an Emotion Extractor to address this issue. We use augmentation approaches to train the model and enable it to extract effective and generalizable emotional representations from imbalanced datasets. Our empirical results show that (1) for the SER task, the proposed Emotion Extractor surpasses the state-of-the-art baseline on three imbalanced datasets; (2) the produced representations from our Emotion Extractor benefit the TTS model, and enable it to synthesize more expressive speech.
【6】 Domestic Activities Classification from Audio Recordings Using  Multi-scale Dilated Depthwise Separable Convolutional Network标题：基于多尺度扩展纵向可分离卷积网络的家庭活动分类链接：https://arxiv.org/abs/2306.05624
*与cs.SD语音【11】为同一篇
作者：Yufei Zeng,Yanxiong Li,Zhenfeng Zhou,Ruiqi Wang,Difeng Lu备注：5 pages, 2 figures, 4 tables. Accepted for publication in IEEE MMSP2021摘要：音频记录的家庭活动分类（DAC）旨在将音频记录分类到预定义的家庭活动类别中，这是用于估计在家庭环境中进行的日常活动的有效方式。在本文中，我们提出了一种使用多尺度扩张深度可分离卷积网络（DSCN）从音频记录中进行DAC的方法。DSCN是具有小尺寸参数的轻量级神经网络，因此适合于部署在具有有限计算资源的便携式终端中。为了在DSCN参数相同的情况下扩大感受野，在DSCN中使用扩张卷积代替常规卷积，以进一步提高DSCN的性能。此外，由扩张的DSCN学习的各种尺度的嵌入被连接为多尺度嵌入，用于表示各种类别的国内活动之间的属性差异。在2018年声学场景和事件的检测和分类挑战（DCASE-2018）的任务5的公共数据集上进行评估，结果显示：扩张卷积和多尺度嵌入都有助于改进算法的性能;并且所提出的方法在分类精度方面优于基于最新技术的轻量级网络的方法。摘要：Domestic activities classification (DAC) from audio recordings aims at classifying audio recordings into pre-defined categories of domestic activities, which is an effective way for estimation of daily activities performed in home environment. In this paper, we propose a method for DAC from audio recordings using a multi-scale dilated depthwise separable convolutional network (DSCN). The DSCN is a lightweight neural network with small size of parameters and thus suitable to be deployed in portable terminals with limited computing resources. To expand the receptive field with the same size of DSCN's parameters, dilated convolution, instead of normal convolution, is used in the DSCN for further improving the DSCN's performance. In addition, the embeddings of various scales learned by the dilated DSCN are concatenated as a multi-scale embedding for representing property differences among various classes of domestic activities. Evaluated on a public dataset of the Task 5 of the 2018 challenge on Detection and Classification of Acoustic Scenes and Events (DCASE-2018), the results show that: both dilated convolution and multi-scale embedding contribute to the performance improvement of the proposed method; and the proposed method outperforms the methods based on state-of-the-art lightweight network in terms of classification accuracy.
【7】 Acoustic Scene Clustering Using Joint Optimization of Deep Embedding  Learning and Clustering Iteration标题：深度嵌入学习和聚类迭代联合优化的声场景聚类链接：https://arxiv.org/abs/2306.05621
*与cs.SD语音【12】为同一篇
作者：Yanxiong Li,Mingle Liu,Wucheng Wang,Yuhan Zhang,Qianhua He备注：9 pages, 6 figures, 11 tables. Accepted for publication in IEEE TMM摘要：最近的努力已经在音频信号处理社区中的声学场景分类。相比之下，很少有研究已经进行了声学场景聚类，这是一个新出现的问题。声学场景聚类的目的是在不利用先验信息和训练分类器的情况下，将同一类声学场景的音频记录合并到一个单独的簇中。在这项研究中，我们提出了一种声学场景聚类的方法，共同优化的程序的特征学习和聚类迭代。在所提出的方法中，学习的特征是从深度卷积神经网络（CNN）中提取的深度嵌入，而聚类算法是凝聚层次聚类（AHC）。我们制定了一个统一的损失函数整合和优化这两个程序。各种功能和方法进行了比较。实验结果表明，该方法优于其他无监督方法的归一化互信息和聚类精度。此外，深度嵌入优于许多最先进的功能。摘要：Recent efforts have been made on acoustic scene classification in the audio signal processing community. In contrast, few studies have been conducted on acoustic scene clustering, which is a newly emerging problem. Acoustic scene clustering aims at merging the audio recordings of the same class of acoustic scene into a single cluster without using prior information and training classifiers. In this study, we propose a method for acoustic scene clustering that jointly optimizes the procedures of feature learning and clustering iteration. In the proposed method, the learned feature is a deep embedding that is extracted from a deep convolutional neural network (CNN), while the clustering algorithm is the agglomerative hierarchical clustering (AHC). We formulate a unified loss function for integrating and optimizing these two procedures. Various features and methods are compared. The experimental results demonstrate that the proposed method outperforms other unsupervised methods in terms of the normalized mutual information and the clustering accuracy. In addition, the deep embedding outperforms many state-of-the-art features.
【8】 Latent Phrase Matching for Dysarthric Speech标题：韵律障碍语音中的潜在短语匹配链接：https://arxiv.org/abs/2306.05446
作者：Colin Lea,Dianna Yee,Jaya Narain,Zifang Huang,Lauren Tooley,Jeffrey P. Bigham,Leah Findlater摘要：许多消费者语音识别系统没有针对有语音障碍的人进行调整，导致识别和用户体验不佳，特别是对于严重的语音差异。最近的研究强调了人们对非典型语音模式的个性化语音模型的兴趣。我们提出了一个查询的基于示例的个性化短语识别系统，使用少量的语音训练，是语言不可知的，不承担传统的发音词典，并概括以及跨语音差异的严重程度。在从32名构音障碍患者收集的内部数据集上，这种方法无论严重程度如何都有效，并且相对于商业语音识别系统，其召回率提高了60%。在公开的EasyCall构音障碍语音数据集上，我们的方法将准确率提高了30.5%。性能随着短语数量的增加而下降，但在使用50个唯一短语进行训练时，性能始终优于ASR系统。摘要：Many consumer speech recognition systems are not tuned for people with speech disabilities, resulting in poor recognition and user experience, especially for severe speech differences. Recent studies have emphasized interest in personalized speech models from people with atypical speech patterns. We propose a query-by-example-based personalized phrase recognition system that is trained using small amounts of speech, is language agnostic, does not assume a traditional pronunciation lexicon, and generalizes well across speech difference severities. On an internal dataset collected from 32 people with dysarthria, this approach works regardless of severity and shows a 60% improvement in recall relative to a commercial speech recognition system. On the public EasyCall dataset of dysarthric speech, our approach improves accuracy by 30.5%. Performance degrades as the number of phrases increases, but consistently outperforms ASR systems when trained with 50 unique phrases.
【9】 Developing Speech Processing Pipelines for Police Accountability标题：开发警察问责语音处理流水线链接：https://arxiv.org/abs/2306.06086
*与cs.SD语音【1】为同一篇
作者：Anjalie Field,Prateek Verma,Nay San,Jennifer L. Eberhardt,Dan Jurafsky备注：Accepted to INTERSPEECH 2023摘要：警察随身携带的摄像机有可能改善警务的问责制和透明度。然而，在实践中，他们导致数百万小时的镜头，从来没有审查。我们研究了大型预训练语音模型的潜力，以促进审查，重点是ASR和官员的语音检测从交通站的镜头。我们提出的管道包括训练数据对齐和过滤，资源约束的微调，以及将官员语音检测与ASR相结合，以实现完全自动化的方法。我们发现，（1）微调强烈地提高了对官员语音的ASR性能（WER=12-13%），（2）对官员语音的ASR比对社区成员语音的ASR准确得多（WER=43.55-49.07%），（3）诸如官员语音检测和日志化的特定领域任务仍然具有挑战性。我们的工作提供了审查身体摄像机镜头的实际应用，以及将预训练的语音模型适应嘈杂的多说话者域的一般指导。摘要：Police body-worn cameras have the potential to improve accountability and transparency in policing. Yet in practice, they result in millions of hours of footage that is never reviewed. We investigate the potential of large pre-trained speech models for facilitating reviews, focusing on ASR and officer speech detection in footage from traffic stops. Our proposed pipeline includes training data alignment and filtering, fine-tuning with resource constraints, and combining officer speech detection with ASR for a fully automated approach. We find that (1) fine-tuning strongly improves ASR performance on officer speech (WER=12-13%), (2) ASR on officer speech is much more accurate than on community member speech (WER=43.55-49.07%), (3) domain-specific tasks like officer speech detection and diarization remain challenging. Our work offers practical applications for reviewing body camera footage and general guidance for adapting pre-trained speech models to noisy multi-speaker domains.
【10】 Improving Fairness and Robustness in End-to-End Speech Recognition  through unsupervised clustering标题：无监督聚类提高端到端语音识别的公平性和稳健性链接：https://arxiv.org/abs/2306.06083
*与cs.SD语音【2】为同一篇
作者：Irina-Elena Veliche,Pascale Fung备注：None摘要：当自动语音识别（ASR）系统对人口的所有子组的表现不相同时，公平性的挑战出现。在过去的几年里，语音识别的整体质量有了很大的提高，但对于那些系统性能不佳的用户群体来说，并没有特别关注促进平等和公平。ASR公平性因此也是鲁棒性问题。同时，数据隐私在生产系统中也具有优先级。在本文中，我们提出了一种隐私保护方法，以提高端到端ASR的公平性和鲁棒性，而不使用元数据，邮政编码，甚至扬声器或话语嵌入直接在训练中。我们使用在公共数据集上训练的扬声器ID模型提取话语级嵌入，然后以无监督的方式使用该模型来创建声学集群。在模型训练过程中，我们使用聚类ID而不是说话者话语嵌入作为额外的特征，这表明所有人口统计组，特别是不同口音的改进。摘要：The challenge of fairness arises when Automatic Speech Recognition (ASR) systems do not perform equally well for all sub-groups of the population. In the past few years there have been many improvements in overall speech recognition quality, but without any particular focus on advancing Equality and Equity for all user groups for whom systems do not perform well. ASR fairness is therefore also a robustness issue. Meanwhile, data privacy also takes priority in production systems. In this paper, we present a privacy preserving approach to improve fairness and robustness of end-to-end ASR without using metadata, zip codes, or even speaker or utterance embeddings directly in training. We extract utterance level embeddings using a speaker ID model trained on a public dataset, which we then use in an unsupervised fashion to create acoustic clusters. We use cluster IDs instead of speaker utterance embeddings as extra features during model training, which shows improvements for all demographic groups and in particular for different accents.
【11】 Reconstructing Human Expressiveness in Piano Performances with a  Transformer Network标题：用变形网络重建钢琴表演中的人类表现力链接：https://arxiv.org/abs/2306.06040
*与cs.SD语音【3】为同一篇
作者：Jingjing Tang,Geraint Wiggins,George Fazekas备注：12 pages, 5 figures, submitted to CMMR 2023摘要：使用计算方法捕捉音乐表演中人类表现力的复杂和微妙变化是具有挑战性的。在本文中，我们提出了一种新的方法来重建人类的表现力在钢琴演奏与多层双向Transformer编码器。为了解决在训练神经网络时对大量准确捕获和分数对齐的性能数据的需求，我们使用从现有转录模型获得的转录分数来训练我们的模型。我们整合钢琴家的身份来控制采样过程，并探索我们的系统的能力，为不同的钢琴家的表现力的变化模型。该系统是通过统计分析所产生的表现力和听力测试进行评估。总的来说，结果表明，我们的方法实现了最先进的从转录的分数生成类人的钢琴表演，而完全和一致的重建人类的表现力带来了进一步的挑战。摘要：Capturing intricate and subtle variations in human expressiveness in music performance using computational approaches is challenging. In this paper, we propose a novel approach for reconstructing human expressiveness in piano performance with a multi-layer bi-directional Transformer encoder. To address the needs for large amounts of accurately captured and score-aligned performance data in training neural networks, we use transcribed scores obtained from an existing transcription model to train our model. We integrate pianist identities to control the sampling process and explore the ability of our system to model variations in expressiveness for different pianists. The system is evaluated through statistical analysis of generated expressive performances and a listening test. Overall, the results suggest that our method achieves state-of-the-art in generating human-like piano performances from transcribed scores, while fully and consistently reconstructing human expressiveness poses further challenges.
【12】 Boosting Fast and High-Quality Speech Synthesis with Linear Diffusion标题：利用线性扩散技术促进快速高质量语音合成链接：https://arxiv.org/abs/2306.05708
*与cs.SD语音【4】为同一篇
作者：Haogeng Liu,Tao Wang,Jie Cao,Ran He,Jianhua Tao摘要：去噪扩散概率模型在各种生成任务上表现出非凡的能力。然而，他们的推理速度慢，使他们不切实际的语音合成。本文提出了一种基于常微分方程的线性扩散模型（LinDiff），同时达到快速推理和高样本质量。首先，我们采用目标和噪声之间的线性插值来设计用于训练的扩散序列，而先前连接噪声和目标的扩散路径是曲线段。当减少采样步骤的数量（即，用于拟合路径的线段的数量），与曲线相比拟合直线的容易性允许我们以更少的迭代从随机噪声生成更高质量的样本。其次，为了降低计算复杂度，实现有效的噪声语音的全局建模，LinDiff采用了基于补丁的处理方法，将输入信号划分成小补丁。分片令牌利用Transformer架构对全局信息进行有效建模。对抗训练用于进一步提高样本质量，减少采样步骤。我们测试所提出的方法与语音合成条件的声学特征（梅尔频谱图）。实验结果表明，该模型可以合成高质量的语音，即使只有一个扩散步骤。主观和客观的评价表明，我们的模型可以合成语音的质量相媲美的自回归模型具有更快的合成速度（3扩散步骤）。摘要：Denoising Diffusion Probabilistic Models have shown extraordinary ability on various generative tasks. However, their slow inference speed renders them impractical in speech synthesis. This paper proposes a linear diffusion model (LinDiff) based on an ordinary differential equation to simultaneously reach fast inference and high sample quality. Firstly, we employ linear interpolation between the target and noise to design a diffusion sequence for training, while previously the diffusion path that links the noise and target is a curved segment. When decreasing the number of sampling steps (i.e., the number of line segments used to fit the path), the ease of fitting straight lines compared to curves allows us to generate higher quality samples from a random noise with fewer iterations. Secondly, to reduce computational complexity and achieve effective global modeling of noisy speech, LinDiff employs a patch-based processing approach that partitions the input signal into small patches. The patch-wise token leverages Transformer architecture for effective modeling of global information. Adversarial training is used to further improve the sample quality with decreased sampling steps. We test proposed method with speech synthesis conditioned on acoustic feature (Mel-spectrograms). Experimental results verify that our model can synthesize high-quality speech even with only one diffusion step. Both subjective and objective evaluations demonstrate that our model can synthesize speech of a quality comparable to that of autoregressive models with faster synthesis speed (3 diffusion steps).
【13】 Low-rank Adaptation Method for Wav2vec2-based Fake Audio Detection标题：基于Wav2ve2的低阶自适应伪音频检测方法链接：https://arxiv.org/abs/2306.05617
*与cs.SD语音【5】为同一篇
作者：Chenglong Wang,Jiangyan Yi,Xiaohui Zhang,Jianhua Tao,Le Xu,Ruibo Fu备注：None摘要：自监督语音模型是虚假音频检测中一个发展迅速的研究课题。许多预先训练的模型可以作为特征提取器，学习更丰富和更高级别的语音特征。然而，当对预训练模型进行微调时，通常会面临训练时间过长和内存消耗过高的挑战，并且完全微调也非常昂贵。为了缓解这个问题，我们将低秩自适应（LoRA）应用于wav2vec2模型，冻结预训练的模型权重，并将可训练的秩分解矩阵注入到Transformer架构的每一层中，大大减少了下游任务的可训练参数的数量。与Adam在包含317M训练参数的wav2vec2模型上进行微调相比，LoRA通过将可训练参数的数量减少198倍来实现类似的性能。摘要：Self-supervised speech models are a rapidly developing research topic in fake audio detection. Many pre-trained models can serve as feature extractors, learning richer and higher-level speech features. However,when fine-tuning pre-trained models, there is often a challenge of excessively long training times and high memory consumption, and complete fine-tuning is also very expensive. To alleviate this problem, we apply low-rank adaptation(LoRA) to the wav2vec2 model, freezing the pre-trained model weights and injecting a trainable rank-decomposition matrix into each layer of the transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared with fine-tuning with Adam on the wav2vec2 model containing 317M training parameters, LoRA achieved similar performance by reducing the number of trainable parameters by 198 times.
【14】 Detecting Check-Worthy Claims in Political Debates, Speeches, and  Interviews Using Audio Data标题：使用音频数据在政治辩论、演讲和采访中检测值得检查的声明链接：https://arxiv.org/abs/2306.05535
*与cs.SD语音【6】为同一篇
作者：Petar Ivanov,Ivan Koychev,Momchil Hardalov,Preslav Nakov备注：check-worthy claims, fake news, political debates, audio摘要：社会上有很大一部分人团结在同一个愿景和想法周围，蕴藏着巨大的能量。这正是政治人物希望为他们的事业积累的东西。考虑到这一目标，他们有时会无意或有意地歪曲或隐藏真相，这为错误信息和虚假信息打开了大门。自动检测值得检查的声明的工具将对辩论主持人、记者和事实核查组织有很大帮助。虽然以前的工作检测检查值得索赔的重点是文本，在这里，我们探讨的实用程序的音频信号作为一个额外的信息源。我们创建了一个新的多模态数据集（英语文本和音频），其中包含48小时的语音。我们的评估结果表明，在多个扬声器的情况下，音频模态与文本一起产生的改进超过单独的文本。此外，对于单个说话者，仅音频模型的性能可能优于仅文本模型。摘要：A large portion of society united around the same vision and ideas carries enormous energy. That is precisely what political figures would like to accumulate for their cause. With this goal in mind, they can sometimes resort to distorting or hiding the truth, unintentionally or on purpose, which opens the door for misinformation and disinformation. Tools for automatic detection of check-worthy claims would be of great help to moderators of debates, journalists, and fact-checking organizations. While previous work on detecting check-worthy claims has focused on text, here we explore the utility of the audio signal as an additional information source. We create a new multimodal dataset (text and audio in English) containing 48 hours of speech. Our evaluation results show that the audio modality together with text yields improvements over text alone in the case of multiple speakers. Moreover, an audio-only model could outperform a text-only one for a single speaker.
【15】 Towards End-to-end Speech-to-text Summarization标题：走向端到端的语音到文本摘要链接：https://arxiv.org/abs/2306.05432
作者：Raul Monteiro,Diogo Pernes备注：Accepted to the 26th International Conference of Text, Speech and Dialogue (TSD2023)摘要：语音到文本（S2T）摘要是一种节省时间的技术，用于过滤和跟上每天在线上传的广播新闻。深度学习中的大型语言模型具有令人印象深刻的文本生成功能，其兴起使研究重点放在了摘要系统上，该系统可以生成文档内容的释义紧凑版本，也称为抽象摘要。端到端（E2E）建模的S2T抽象摘要是一种很有前途的方法，提供了产生丰富的潜在表示，利用非语言和声学信息的可能性，而不是使用的语言信息，从自动生成的成绩单在级联系统。然而，E2E建模的这项任务的少数文献未能探索不同的领域，即广播新闻，这是具有挑战性的领域，每天向用户呈现大量和多样化的数据。我们的模型S2T总结与级联和E2E系统的语料库广播新闻在法国。我们的新型E2E模型通过从预先训练的T2T摘要器中进行迁移学习来利用外部数据。实验表明，我们的级联和E2E抽象摘要比提取基线更强。然而，E2E模型的性能仍然落后于级联模型，这是广泛分析的对象，包括未来缩小这一差距的方向。摘要：Speech-to-text (S2T) summarization is a time-saving technique for filtering and keeping up with the broadcast news uploaded online on a daily basis. The rise of large language models from deep learning with impressive text generation capabilities has placed the research focus on summarization systems that produce paraphrased compact versions of the document content, also known as abstractive summaries. End-to-end (E2E) modelling of S2T abstractive summarization is a promising approach that offers the possibility of generating rich latent representations that leverage non-verbal and acoustic information, as opposed to the use of only linguistic information from automatically generated transcripts in cascade systems. However, the few literature on E2E modelling of this task fails on exploring different domains, namely broadcast news, which is challenging domain where large and diversified volumes of data are presented to the user every day. We model S2T summarization both with a cascade and an E2E system for a corpus of broadcast news in French. Our novel E2E model leverages external data by resorting to transfer learning from a pre-trained T2T summarizer. Experiments show that both our cascade and E2E abstractive summarizers are stronger than an extractive baseline. However, the performance of the E2E model still lies behind the cascade one, which is object of an extensive analysis that includes future directions to close that gap.
机器翻译，仅供参考