语音/音频处理学术速递[12.19]

今日论文合集：cs.SD语音23篇，eess.AS音频处理24篇。

cs.SD语音

【1】Evaluation of Barlow Twins and VICReg self-supervised learning for sound patterns of bird and anuran species

标题：Barlow Twins和VICReg自监督学习对鸟类和无尾类物种声音模式的评估链接：https://arxiv.org/abs/2312.11240

作者：Fábio Felix Dias,Moacir Antonelli Ponti,Mílton Cezar Ribeiro,Rosane Minghim备注：10 pages, 2 figures, 3 tables摘要：利用大型数据集的结构来预训练深度学习模型是一种很有前途的策略，可以减少对监督数据的需求。自监督学习方法，如对比及其变体，是在许多深度学习应用中获得更好表示的一种很有前途的方法。声景生态学是一种注释昂贵且稀缺的应用，因此值得研究不需要注释的近似方法。我们的研究涉及使用Barlow Twins和VICReg方法来预训练不同的模型，这些模型具有相同的小型数据集，其中包含鸟类和无尾类物种的声音模式。在对这些动物物种进行分类的下游任务中，模型获得的结果接近监督结果，在大型通用数据集中进行了预训练，并在相同的任务中进行了微调。摘要：Taking advantage of the structure of large datasets to pre-train Deep Learning models is a promising strategy to decrease the need for supervised data. Self-supervised learning methods, such as contrastive and its variation are a promising way towards obtaining better representations in many Deep Learning applications. Soundscape ecology is one application in which annotations are expensive and scarce, therefore deserving investigation to approximate methods that do not require annotations to those that rely on supervision. Our study involves the use of the methods Barlow Twins and VICReg to pre-train different models with the same small dataset with sound patterns of bird and anuran species. In a downstream task to classify those animal species, the models obtained results close to supervised ones, pre-trained in large generic datasets, and fine-tuned with the same task.

【2】 Perceptual Musical Features for Interpretable Audio Tagging标题：用于可解释音频标注的感知音乐特征链接：https://arxiv.org/abs/2312.11234

作者：Vassilis Lyberatos,Spyridon Kantarelis,Edmund Dervakos,Giorgos Stamo备注：Github Repository: this https URL摘要：在音乐流媒体平台时代，自动标记音乐音频的任务已经引起了极大的关注，促使研究人员设计旨在增强标准数据集性能指标的方法。大多数最近的方法依赖于深度神经网络，尽管它们的性能令人印象深刻，但具有不透明性，因此很难阐明它们对给定输入的输出。虽然可解释性的问题在医学等其他领域得到了强调，但在音乐相关的任务中却没有得到关注。在这项研究中，我们探讨了自动音乐标注的背景下，可解释性的相关性。我们构建了一个工作流程，其中包含三种不同的信息提取技术：a）利用符号知识，b）利用辅助深度神经网络，c）采用信号处理从音频文件中提取感知特征。这些特征随后被用于训练用于标签预测的可解释机器学习模型。我们在两个数据集上进行了实验，即MTG-Jamendo数据集和GTZAN数据集。我们的方法在这两项任务中都超过了基线模型的性能，并且在某些情况下，表现出与当前最先进技术的竞争力。我们得出结论，在某些用例中，性能的恶化被可解释性的价值所抵消。摘要：In the age of music streaming platforms, the task of automatically tagging music audio has garnered significant attention, driving researchers to devise methods aimed at enhancing performance metrics on standard datasets. Most recent approaches rely on deep neural networks, which, despite their impressive performance, possess opacity, making it challenging to elucidate their output for a given input. While the issue of interpretability has been emphasized in other fields like medicine, it has not received attention in music-related tasks. In this study, we explored the relevance of interpretability in the context of automatic music tagging. We constructed a workflow that incorporates three different information extraction techniques: a) leveraging symbolic knowledge, b) utilizing auxiliary deep neural networks, and c) employing signal processing to extract perceptual features from audio files. These features were subsequently used to train an interpretable machine-learning model for tag prediction. We conducted experiments on two datasets, namely the MTG-Jamendo dataset and the GTZAN dataset. Our method surpassed the performance of baseline models in both tasks and, in certain instances, demonstrated competitiveness with the current state-of-the-art. We conclude that there are use cases where the deterioration in performance is outweighed by the value of interpretability.

【3】 Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakers标题：通过对主说话人和非主说话人进行联合建模来改进长格式语音识别链接：https://arxiv.org/abs/2312.11123

作者：Guru Prakash Arumugam,Shuo-yiin Chang,Tara N. Sainath,Rohit Prabhavalkar,Quan Wang,Shaan Bijwadia备注：8 pages, ASRU 2023摘要：ASR模型通常会遇到长格式删除问题，其中在转录冗长的音频（以分钟或小时的顺序）时，模型预测连续的空白而不是单词。从消费ASR结果的用户或下游系统的角度来看，这种行为可以被感知为模型“被卡住”，并且可能使产品难以使用。长格式删除的罪魁祸首之一是训练-测试数据不匹配，即使模型是在从多个应用程序域收集的各种大规模数据上训练的，也会发生这种情况。在这项工作中，我们介绍了一种新的技术，同时模拟音频中的不同扬声器组以及标准的成绩单令牌。扬声器被分组为主要和非主要扬声器，这连接了应用程序域，并大大简化了长格式删除问题。这种改进的模型既不需要任何额外的训练数据，也不产生额外的训练或推理成本。摘要：ASR models often suffer from a long-form deletion problem where the model predicts sequential blanks instead of words when transcribing a lengthy audio (in the order of minutes or hours). From the perspective of a user or downstream system consuming the ASR results, this behavior can be perceived as the model "being stuck", and potentially make the product hard to use. One of the culprits for long-form deletion is training-test data mismatch, which can happen even when the model is trained on diverse and large-scale data collected from multiple application domains. In this work, we introduce a novel technique to simultaneously model different groups of speakers in the audio along with the standard transcript tokens. Speakers are grouped as primary and non-primary, which connects the application domains and significantly alleviates the long-form deletion problem. This improved model neither needs any additional training data nor incurs additional training or inference cost.

【4】 3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications标题：3S-TSE：面向实时和低资源应用的高效三阶段目标说话人提取链接：https://arxiv.org/abs/2312.10979

作者：Shulin He,Jinjiang liu,Hao Li,Yang Yang,Fei Chen,Xueliang Zhang备注：Submitted to ICASSP 2024摘要：目标说话人提取（TSE）的目的是依靠一个注册的样本从多个混合说话人中分离出一个特定的语音。由于声纹特征通常变化很大，当前的端到端神经网络需要大的模型参数，这是计算密集型的，对于实时应用是不切实际的，特别是在资源受限的平台上。在本文中，我们使用麦克风阵列来解决TSE任务，并介绍了一种新的三阶段解决方案，该解决方案系统地描述了该过程：首先，训练神经网络来估计目标说话者的方向。其次，在确定了目标语音的方向后，利用广义旁瓣对消器（GSC）提取目标语音。第三，原位卷积递归神经网络（ICRN）作为去噪后处理器，细化GSC输出以产生最终的分离语音。我们的方法提供了卓越的性能，同时大大降低了计算负载，为高效的实时目标说话人提取设定了新的标准。摘要：Target speaker extraction (TSE) aims to isolate a specific voice from multiple mixed speakers relying on a registerd sample. Since voiceprint features usually vary greatly, current end-to-end neural networks require large model parameters which are computational intensive and impractical for real-time applications, espetially on resource-constrained platforms. In this paper, we address the TSE task using microphone array and introduce a novel three-stage solution that systematically decouples the process: First, a neural network is trained to estimate the direction of the target speaker. Second, with the direction determined, the Generalized Sidelobe Canceller (GSC) is used to extract the target speech. Third, an Inplace Convolutional Recurrent Neural Network (ICRN) acts as a denoising post-processor, refining the GSC output to yield the final separated speech. Our approach delivers superior performance while drastically reducing computational load, setting a new standard for efficient real-time target speaker extraction.

【5】 Generative linguistic representation for spoken language identification标题：用于口语识别的生成语言表示链接：https://arxiv.org/abs/2312.10964

作者：Peng Shen,Xuguang Lu,Hisashi Kawai备注：Accepted by IEEE ASRU2023摘要：语言特征的有效提取和应用是提高口语识别性能的关键。随着GPT和Whisper等最近大型模型的成功，利用这些预训练模型提取LID任务的语言特征的潜力已成为一个很有前途的研究领域。在本文中，我们探讨了利用基于解码器的网络从耳语模型提取语言特征，通过其生成机制，以提高分类精度LID任务。我们设计了两种策略-一种基于语言嵌入方法，另一种专注于直接优化LID输出，同时增强语音识别任务。我们在大规模多语言数据集MLS，VoxLingua 107和CommonVoice上进行了实验，以测试我们的方法。实验结果表明，该方法在域内和域外数据集上的LID任务的有效性。摘要：Effective extraction and application of linguistic features are central to the enhancement of spoken Language IDentification (LID) performance. With the success of recent large models, such as GPT and Whisper, the potential to leverage such pre-trained models for extracting linguistic features for LID tasks has become a promising area of research. In this paper, we explore the utilization of the decoder-based network from the Whisper model to extract linguistic features through its generative mechanism for improving the classification accuracy in LID tasks. We devised two strategies - one based on the language embedding method and the other focusing on direct optimization of LID outputs while simultaneously enhancing the speech recognition tasks. We conducted experiments on the large-scale multilingual datasets MLS, VoxLingua107, and CommonVoice to test our approach. The experimental results demonstrated the effectiveness of the proposed method on both in-domain and out-of-domain datasets for LID tasks.

【6】 Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition标题：用于多说话人重叠语音识别的说话人面具转换器链接：https://arxiv.org/abs/2312.10959

作者：Peng Shen,Xugang Lu,Hisashi Kawai摘要：多说话人重叠语音识别仍然是一个重大的挑战，不仅需要语音识别，但说话人日记的任务要解决。在本文中，为了更好地解决这些任务，我们首先将说话人标签引入到基于自回归变换的语音识别模型中，以支持多说话人重叠语音识别。然后，为了改进说话人日志化，我们提出了一种新的说话人掩码分支来检测单个说话人的语音段。使用该模型，我们可以使用一个模型同时执行语音识别和说话人日志化任务。基于LibriSpeech的重叠数据集上的实验结果表明，该方法在语音识别和说话人日志化任务中均有效，特别是在相对复杂的多说话人场景中提高了说话人日志化的准确性。摘要：Multi-talker overlapped speech recognition remains a significant challenge, requiring not only speech recognition but also speaker diarization tasks to be addressed. In this paper, to better address these tasks, we first introduce speaker labels into an autoregressive transformer-based speech recognition model to support multi-speaker overlapped speech recognition. Then, to improve speaker diarization, we propose a novel speaker mask branch to detection the speech segments of individual speakers. With the proposed model, we can perform both speech recognition and speaker diarization tasks simultaneously using a single model. Experimental results on the LibriSpeech-based overlapped dataset demonstrate the effectiveness of the proposed method in both speech recognition and speaker diarization tasks, particularly enhancing the accuracy of speaker diarization in relatively complex multi-talker scenarios.

【7】 Soft Alignment of Modality Space for End-to-end Speech Translation标题：端到端语音翻译中的通道空间软对齐链接：https://arxiv.org/abs/2312.10952

作者：Yuhao Zhang,Kaiqi Kou,Bei Li,Chen Xu,Chunliang Zhang,Tong Xiao,Jingbo Zhu备注：Accepted to ICASSP2024摘要：端到端语音翻译（ST）的目的是在一个统一的模型内将语音转换为目标文本。言语和语篇模态之间的固有差异往往阻碍有效的跨模态和跨语言迁移。现有方法通常采用单个语音和文本片段的硬对齐（H-Align），这会降低文本表示。为了解决这个问题，我们引入了软对齐（S-Align），使用对抗训练来对齐两种模式的表示空间。S-Align创建模态不变空间，同时保留单个模态质量。在MuST-C数据集的三种语言上进行的实验表明，S-Align在多个任务中的表现优于H-Align，并提供了与专业翻译模型相当的翻译功能。摘要：End-to-end Speech Translation (ST) aims to convert speech into target text within a unified model. The inherent differences between speech and text modalities often impede effective cross-modal and cross-lingual transfer. Existing methods typically employ hard alignment (H-Align) of individual speech and text segments, which can degrade textual representations. To address this, we introduce Soft Alignment (S-Align), using adversarial training to align the representation spaces of both modalities. S-Align creates a modality-invariant space while preserving individual modality quality. Experiments on three languages from the MuST-C dataset show S-Align outperforms H-Align across multiple tasks and offers translation capabilities on par with specialized translation models.

【8】 Leveraged Mel spectrograms using Harmonic and Percussive Components in Speech Emotion Recognition标题：在语音情感识别中使用调和和敲击分量的杠杆Mel谱图链接：https://arxiv.org/abs/2312.10949

作者：David Hason Rudd,Huan Huo,Guandong Xu备注：None摘要：语音情感识别（SER）情感技术使智能嵌入式设备能够灵敏地进行交互。同样，呼叫中心的员工从客户的音调、能量和语调中识别出客户的情绪，从而修改他们的讲话，以便与客户进行高质量的互动。这项工作探讨，第一次，在SER的谐波和谐波分量的梅尔频谱图的影响。我们试图利用梅尔频谱图分解可区分的声学特征，利用我们提出的架构，其中包括一种新的特征图生成器算法，基于CNN的网络特征提取器和多层感知器（MLP）分类。本研究特别关注有效的数据增强技术，用于构建丰富的基于混合的特征图。该过程产生输出2D图像的函数，使得其可以用作预训练的CNN-VGG 16特征提取器的输入数据。此外，我们还研究了其他声学特征，如MFCC，色谱图，光谱对比度，和tonnetz来评估我们提出的框架。在柏林EMO-DB数据库上的测试准确率达到92.79%。我们的结果是高于以前的作品使用CNN-VGG 16。摘要：Speech Emotion Recognition (SER) affective technology enables the intelligent embedded devices to interact with sensitivity. Similarly, call centre employees recognise customers' emotions from their pitch, energy, and tone of voice so as to modify their speech for a high-quality interaction with customers. This work explores, for the first time, the effects of the harmonic and percussive components of Mel spectrograms in SER. We attempt to leverage the Mel spectrogram by decomposing distinguishable acoustic features for exploitation in our proposed architecture, which includes a novel feature map generator algorithm, a CNN-based network feature extractor and a multi-layer perceptron (MLP) classifier. This study specifically focuses on effective data augmentation techniques for building an enriched hybrid-based feature map. This process results in a function that outputs a 2D image so that it can be used as input data for a pre-trained CNN-VGG16 feature extractor. Furthermore, we also investigate other acoustic features such as MFCCs, chromagram, spectral contrast, and the tonnetz to assess our proposed framework. A test accuracy of 92.79% on the Berlin EMO-DB database is achieved. Our result is higher than previous works using CNN-VGG16.

【9】 An Extended Variational Mode Decomposition Algorithm Developed Speech Emotion Recognition Performance标题：一种提高语音情感识别性能的扩展变分模式分解算法链接：https://arxiv.org/abs/2312.10937

作者：David Hason Rudd,Huan Huo,Guandong Xu备注：None摘要：从语音信号中进行情感识别（ER）是一种鲁棒的方法，因为它不能像面部表情或基于文本的情感分析那样被模仿。情感背后的有价值的信息对于人机交互非常重要，使智能机器能够在现实世界中与敏感性进行交互。以前的ER研究，通过语音信号处理只集中在不同的信号模式分解方法和隐藏的信息功能之间的关联。然而，不适当的分解参数的选择导致信息的信号分量的损失，由于模式复制和混合。相比之下，目前的研究提出了VGG-OptiVMD，授权变分模式分解算法，区分有意义的语音特征，并自动选择分解模式的数量和最佳平衡参数的数据保真度约束，通过评估其对VGG 16扁平化输出层的影响。采用不同的特征向量在不同的数据库上训练VGG 16网络，并评估VGG-optiVMD的再现性和可靠性。一个，两个，和三维特征向量的构造，通过级联梅尔频率倒谱系数，色度图，梅尔频谱图，Tonnetz图，和频谱质心。结果证实了信号采样率和分解参数的微调与分类精度之间的协同关系，在柏林EMO-DB数据库上预测七种情绪的准确率达到了最先进的96.09%。摘要：Emotion recognition (ER) from speech signals is a robust approach since it cannot be imitated like facial expression or text based sentiment analysis. Valuable information underlying the emotions are significant for human-computer interactions enabling intelligent machines to interact with sensitivity in the real world. Previous ER studies through speech signal processing have focused exclusively on associations between different signal mode decomposition methods and hidden informative features. However, improper decomposition parameter selections lead to informative signal component losses due to mode duplicating and mixing. In contrast, the current study proposes VGG-optiVMD, an empowered variational mode decomposition algorithm, to distinguish meaningful speech features and automatically select the number of decomposed modes and optimum balancing parameter for the data fidelity constraint by assessing their effects on the VGG16 flattening output layer. Various feature vectors were employed to train the VGG16 network on different databases and assess VGG-optiVMD reproducibility and reliability. One, two, and three-dimensional feature vectors were constructed by concatenating Mel-frequency cepstral coefficients, Chromagram, Mel spectrograms, Tonnetz diagrams, and spectral centroids. Results confirmed a synergistic relationship between the fine-tuning of the signal sample rate and decomposition parameters with classification accuracy, achieving state-of-the-art 96.09% accuracy in predicting seven emotions on the Berlin EMO-DB database.

【10】 AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis标题：AE-NERF：音频增强神经辐射场在少发声头部合成中的应用链接：https://arxiv.org/abs/2312.10921

作者：Dongze Li,Kang Zhao,Wei Wang,Bo Peng,Yingya Zhang,Jing Dong,Tieniu Tan备注：Accepted by AAAI 2024摘要：音频驱动的说话人头像合成在数字人、电影制作、虚拟现实等领域有着广泛的应用前景。最近的NeRF为基础的方法已经显示出优越性的质量和保真度相比，以前的研究。然而，当涉及到Few-Shot说话头部生成时，一个身份只有几秒钟的说话视频可用的实际场景出现了两个限制：1）它们要么没有基本模型，其用作快速收敛的面部先验，要么在建立先验时忽略音频的重要性; 2）它们中的大多数忽略了不同面部区域与音频之间的相关程度，例如，嘴是音频相关的，而耳朵是音频独立的。在本文中，我们提出了音频增强神经辐射场（AE-NeRF）来解决上述问题，它可以用少量的数据集生成新说话者的逼真肖像。具体来说，我们引入一个音频感知聚合模块的参考方案的特征融合阶段，其中的权重是由参考和目标图像之间的音频的相似性。然后，提出了一种音频对齐的人脸生成策略，分别对音频相关区域和音频无关区域进行建模，采用双NeRF框架。大量的实验表明，AE-NeRF在图像保真度，音频-唇同步和泛化能力方面超越了最先进的水平，即使在有限的训练集或训练迭代中也是如此。摘要：Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality. Recent NeRF-based approaches have shown superiority in quality and fidelity compared to previous studies. However, when it comes to few-shot talking head generation, a practical scenario where only few seconds of talking video is available for one identity, two limitations emerge: 1) they either have no base model, which serves as a facial prior for fast convergence, or ignore the importance of audio when building the prior; 2) most of them overlook the degree of correlation between different face regions and audio, e.g., mouth is audio related, while ear is audio independent. In this paper, we present Audio Enhanced Neural Radiance Field (AE-NeRF) to tackle the above issues, which can generate realistic portraits of a new speaker with fewshot dataset. Specifically, we introduce an Audio Aware Aggregation module into the feature fusion stage of the reference scheme, where the weight is determined by the similarity of audio between reference and target image. Then, an Audio-Aligned Face Generation strategy is proposed to model the audio related and audio independent regions respectively, with a dual-NeRF framework. Extensive experiments have shown AE-NeRF surpasses the state-of-the-art on image fidelity, audio-lip synchronization, and generalization ability, even in limited training set or training iterations.

【11】 Exploring Sound vs Vibration for Robust Fault Detection on Rotating Machinery标题：声振比在旋转机械故障稳健检测中的应用链接：https://arxiv.org/abs/2312.10742

作者：Serkan Kiranyaz,Ozer Can Devecioglu,Amir Alhams,Sadok Sassi,Turker Ince,Onur Avci,Moncef Gabbouj备注：8 pages摘要：旋转机械故障的鲁棒和实时检测已成为各个行业预测性维护的最终目标。基于振动的深度学习（DL）方法已经成为轴承故障检测的事实标准，因为它们可以在某些条件下产生最先进的检测性能。尽管这种特别关注的振动信号，声音的利用，另一方面，已被忽视，而只有少数研究已提出在过去的二十年中，所有这些都是基于传统的ML方法。一个主要原因是缺乏一个基准数据集，为不同的机器和传感器位置提供大量的振动和声音数据。在这项研究中，我们通过提出新的基准卡塔尔大学双机轴承故障基准数据集（QU-DMBF）来解决这一需求，该数据集封装了在1080个工作条件下运行的两个不同电机的声音和振动数据。然后，我们画的主要局限性和缺点的振动为基础的故障检测，由于众多的安装和操作条件的重点。最后，我们提出了第一个基于声音的故障检测的DL方法，并在QU-DMBF数据集上进行声音和振动之间的比较评估。广泛的实验结果表明，基于声音的故障检测方法比基于振动的方法具有更强的鲁棒性，因为它完全独立于传感器的位置，具有成本效益（无需传感器和传感器维护），并且可以实现与基于振动的方法相同水平的最佳检测性能。通过这项研究，QU-DMBF数据集，PyTorch中优化的源代码和比较评估现在公开共享。摘要：Robust and real-time detection of faults on rotating machinery has become an ultimate objective for predictive maintenance in various industries. Vibration-based Deep Learning (DL) methodologies have become the de facto standard for bearing fault detection as they can produce state-of-the-art detection performances under certain conditions. Despite such particular focus on the vibration signal, the utilization of sound, on the other hand, has been neglected whilst only a few studies have been proposed during the last two decades, all of which were based on a conventional ML approach. One major reason is the lack of a benchmark dataset providing a large volume of both vibration and sound data over several working conditions for different machines and sensor locations. In this study, we address this need by presenting the new benchmark Qatar University Dual-Machine Bearing Fault Benchmark dataset (QU-DMBF), which encapsulates sound and vibration data from two different motors operating under 1080 working conditions overall. Then we draw the focus on the major limitations and drawbacks of vibration-based fault detection due to numerous installation and operational conditions. Finally, we propose the first DL approach for sound-based fault detection and perform comparative evaluations between the sound and vibration over the QU-DMBF dataset. A wide range of experimental results shows that the sound-based fault detection method is significantly more robust than its vibration-based counterpart, as it is entirely independent of the sensor location, cost-effective (requiring no sensor and sensor maintenance), and can achieve the same level of the best detection performance by its vibration-based counterpart. With this study, the QU-DMBF dataset, the optimized source codes in PyTorch, and comparative evaluations are now publicly shared.

【12】 Meta-AF Echo Cancellation for Improved Keyword Spotting标题：Meta-AF回声消除用于改进关键字定位链接：https://arxiv.org/abs/2312.10605

作者：Jonah Casebeer,Junkai Wu,Paris Smaragdis备注：5 pages, 2 figures, ICASSP 2024摘要：自适应滤波器（AF）对于增强下游任务的性能至关重要，例如语音识别，声音事件检测和关键字定位。然而，传统的AF设计优先考虑孤立的信号级目标，往往忽视下游任务的性能。这可能导致性能不佳。最近的研究利用元学习从数据中自动学习AF更新规则，从而在使用简单的信号级目标时减轻了手动调整的需要。本文通过扩展Meta-AF框架以支持任意下游任务的端到端训练来改进该框架。我们专注于分类任务，在那里我们引入了一种新的训练方法，利用自我监督和分类器反馈。我们评估我们的方法的声学回声消除和关键字定位的组合任务。我们的研究结果表明，在合成和真实回放中，预训练和联合训练的关键字定位模型都有一致的性能改进。值得注意的是，这些改进不需要额外的调优，增加推理时间复杂性，或依赖于Oracle信号级训练数据。摘要：Adaptive filters (AFs) are vital for enhancing the performance of downstream tasks, such as speech recognition, sound event detection, and keyword spotting. However, traditional AF design prioritizes isolated signal-level objectives, often overlooking downstream task performance. This can lead to suboptimal performance. Recent research has leveraged meta-learning to automatically learn AF update rules from data, alleviating the need for manual tuning when using simple signal-level objectives. This paper improves the Meta-AF framework by expanding it to support end-to-end training for arbitrary downstream tasks. We focus on classification tasks, where we introduce a novel training methodology that harnesses self-supervision and classifier feedback. We evaluate our approach on the combined task of acoustic echo cancellation and keyword spotting. Our findings demonstrate consistent performance improvements with both pre-trained and joint-trained keyword spotting models across synthetic and real playback. Notably, these improvements come without requiring additional tuning, increased inference-time complexity, or reliance on oracle signal-level training data.

【13】 Seq2seq for Automatic Paraphasia Detection in Aphasic Speech标题：Seq2seq用于失语症语音中的自动失语症检测链接：https://arxiv.org/abs/2312.10518

作者：Matthew Perez,Duc Le,Amrit Romana,Elise Jones,Keli Licata,Emily Mower Provost摘要：失语症是失语症的特征性言语错误，是评估疾病严重程度和亚型的重要信号。传统上，临床医生通过转录和分析语音语言样本来手动识别失语症，这可能是一个耗时且繁重的过程。自动识别失语症可以极大地帮助临床医生的转录过程，并最终促进更有效和一致的失语症评估。先前的研究已经证明了自动错语检测的可行性，通过训练自动语音识别（ASR）模型来提取成绩单，然后在一组手工设计的特征上训练单独的错语检测模型。在本文中，我们提出了一种新的序列到序列（seq2seq）模型，该模型经过端到端（E2E）训练，可以执行ASR和失语症检测任务。我们表明，该模型优于以前的国家的最先进的方法，无论是单词级和话语级的失语症检测任务，并提供额外的后续评估，以进一步了解拟议的模型行为。摘要：Paraphasias are speech errors that are often characteristic of aphasia and they represent an important signal in assessing disease severity and subtype. Traditionally, clinicians manually identify paraphasias by transcribing and analyzing speech-language samples, which can be a time-consuming and burdensome process. Identifying paraphasias automatically can greatly help clinicians with the transcription process and ultimately facilitate more efficient and consistent aphasia assessment. Previous research has demonstrated the feasibility of automatic paraphasia detection by training an automatic speech recognition (ASR) model to extract transcripts and then training a separate paraphasia detection model on a set of hand-engineered features. In this paper, we propose a novel, sequence-to-sequence (seq2seq) model that is trained end-to-end (E2E) to perform both ASR and paraphasia detection tasks. We show that the proposed model outperforms the previous state-of-the-art approach for both word-level and utterance-level paraphasia detection tasks and provide additional follow-up evaluations to further understand the proposed model behavior.

【14】 Annotation-Free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion标题：具有可扩展合成数据和对抗性领域混淆的无注释自动音乐转录链接：https://arxiv.org/abs/2312.10402

作者：Gakusei Sato,Taketo Akama备注：6 pages, 1 figure摘要：自动音乐转录是音乐信息处理中的一项关键技术。尽管最近通过机器学习方法在性能上有所改进，但现有方法通常在具有丰富注释数据的领域中实现高准确性，这主要是由于难以创建注释数据。实际的转录模型需要不需要注释数据的架构。在本文中，我们提出了一个无注释的转录模型，通过利用可伸缩的合成音频进行预训练和对抗域混淆，使用未注释的真实音频。通过评估实验，我们证实，我们提出的方法可以实现更高的准确性，在无注释的条件下相比，当学习与注释的真实音频数据的混合。此外，通过消融研究，我们深入了解了这种方法的可扩展性以及AMT研究领域面临的挑战。摘要：Automatic Music Transcription (AMT) is a crucial technology in music information processing. Despite recent improvements in performance through machine learning approaches, existing methods often achieve high accuracy in domains with abundant annotation data, primarily due to the difficulty of creating annotation data. A practical transcription model requires an architecture that does not require an annotation data. In this paper, we propose an annotation-free transcription model achieved through the utilization of scalable synthetic audio for pre-training and adversarial domain confusion using unannotated real audio. Through evaluation experiments, we confirm that our proposed method can achieve higher accuracy under annotation-free conditions compared to when learning with mixture of annotated real audio data. Additionally, through ablation studies, we gain insights into the scalability of this approach and the challenges that lie ahead in the field of AMT research.

【15】 SECap: Speech Emotion Captioning with Large Language Model标题：SECap：基于大语言模型的语音情感字幕链接：https://arxiv.org/abs/2312.10381

作者：Yaoxun Xu,Hangting Chen,Jianwei Yu,Qiaochu Huang,Zhiyong Wu,Shixiong Zhang,Guangzhi Li,Yi Luo,Rongzhi Gu备注：Accepted by AAAI 2024摘要：语音情感在人类交流中至关重要，广泛应用于语音合成和自然语言理解等领域。大多数先前的研究，如语音情感识别，都将语音情感归类为一组固定的类别。然而，在人类语音中表达的情感通常是复杂的，并且将它们分类到预定义的组中可能不足以充分地表示语音情感。相反，通过自然语言直接描述言语情感可能是一种更有效的方法。令人遗憾的是，没有太多的研究集中在这个方向。因此，本文提出了一个语音情感字幕框架SECap，旨在有效地描述语音情感使用自然语言。由于大型语言模型在语言理解和文本生成方面的强大功能，SECap采用LLAMA作为文本解码器，以生成连贯的语音情感字幕。此外，SECap利用HuBERT作为音频编码器来提取一般语音特征，利用Q-Former作为桥接网络来为LLAMA提供与情感相关的语音特征。为了实现这一点，Q-Former利用互信息学习来理清与情感相关的语音特征和语音内容，同时实施对比学习来提取更多与情感相关的语音特征。客观和主观评价的结果表明：1）SECap框架在所有客观评价中优于HTSAT-BART基线; 2）SECap可以生成高质量的语音情感字幕，在主观平均意见评分测试中达到与人类注释者相当的性能。摘要：Speech emotions are crucial in human communication and are extensively used in fields like speech synthesis and natural language understanding. Most prior studies, such as speech emotion recognition, have categorized speech emotions into a fixed set of classes. Yet, emotions expressed in human speech are often complex, and categorizing them into predefined groups can be insufficient to adequately represent speech emotions. On the contrary, describing speech emotions directly by means of natural language may be a more effective approach. Regrettably, there are not many studies available that have focused on this direction. Therefore, this paper proposes a speech emotion captioning framework named SECap, aimed at effectively describing speech emotions using natural language. Owing to the impressive capabilities of large language models in language comprehension and text generation, SECap employs LLAMA as the text decoder to allow the production of coherent speech emotion captions. In addition, SECap leverages HuBERT as the audio encoder to extract general speech features and Q-Former as the Bridge-Net to provide LLAMA with emotion-related speech features. To accomplish this, Q-Former utilizes mutual information learning to disentangle emotion-related speech features and speech contents, while implementing contrastive learning to extract more emotion-related speech features. The results of objective and subjective evaluations demonstrate that: 1) the SECap framework outperforms the HTSAT-BART baseline in all objective evaluations; 2) SECap can generate high-quality speech emotion captions that attain performance on par with human annotators in subjective mean opinion score tests.

【16】 MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion标题：MUSER：基于音乐元素的正规化生成带情感的符号音乐链接：https://arxiv.org/abs/2312.10307

作者：Shulei Ji,Xinyu Yang备注：Accepted by AAAI 2024摘要：在自动音乐生成中，情感是通过各种随时间变化并相互协作的音乐元素（如音高和持续时间）来唤起的。然而，此前关于基于深度学习的情感音乐生成的研究很少探讨不同音乐元素对情感的贡献，更不用说刻意操纵这些元素来改变音乐的情感，这不利于对情感进行细粒度的元素级控制。为了解决这一问题，我们提出了一种新的方法，采用基于音乐元素的正则化的潜在空间来解开不同的元素，研究它们在区分情绪的作用，并进一步操纵元素来改变音乐情绪。具体来说，我们提出了一种新的VQ-VAE为基础的模型命名为MusER。MusER引入了正则化损失来加强音乐元素序列与潜在变量序列的特定维度之间的对应关系，为解开离散序列提供了一种新的解决方案。为了更好地预测元素，设计了一种两级解码策略，该策略包括多个解码器，用于处理具有不同语义的潜在向量。通过可视化潜在空间，我们得出结论，MusER产生了一个解开和可解释的潜在空间，并深入了解不同元素对情感维度的贡献（即，唤醒和效价）。实验结果表明，MusER优于国家的最先进的模型产生的情感音乐在客观和主观评价。此外，我们还通过元素转移来重新编排音乐，并试图通过转移情感可识别的元素来改变音乐的情感。摘要：Generating music with emotion is an important task in automatic music generation, in which emotion is evoked through a variety of musical elements (such as pitch and duration) that change over time and collaborate with each other. However, prior research on deep learning-based emotional music generation has rarely explored the contribution of different musical elements to emotions, let alone the deliberate manipulation of these elements to alter the emotion of music, which is not conducive to fine-grained element-level control over emotions. To address this gap, we present a novel approach employing musical element-based regularization in the latent space to disentangle distinct elements, investigate their roles in distinguishing emotions, and further manipulate elements to alter musical emotions. Specifically, we propose a novel VQ-VAE-based model named MusER. MusER incorporates a regularization loss to enforce the correspondence between the musical element sequences and the specific dimensions of latent variable sequences, providing a new solution for disentangling discrete sequences. Taking advantage of the disentangled latent vectors, a two-level decoding strategy that includes multiple decoders attending to latent vectors with different semantics is devised to better predict the elements. By visualizing latent space, we conclude that MusER yields a disentangled and interpretable latent space and gain insights into the contribution of distinct elements to the emotional dimensions (i.e., arousal and valence). Experimental results demonstrate that MusER outperforms the state-of-the-art models for generating emotional music in both objective and subjective evaluation. Besides, we rearrange music through element transfer and attempt to alter the emotion of music by transferring emotion-distinguishable elements.

【17】 Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction标题：自监督解缠表征学习用于鲁棒目标语音提取链接：https://arxiv.org/abs/2312.10305

作者：Zhaoxi Mu,Xinyu Yang,Sining Sun,Qing Yang备注：Accepted by AAAI2024摘要：语音信号本身是复杂的，因为它们包含全局声学特性和局部语义信息。然而，在目标语音提取任务中，参考语音中与说话人身份无关的全局和局部语义信息的某些元素会导致语音提取网络内的说话人混淆。为了克服这一挑战，我们提出了一种自监督的解纠缠表示学习方法。我们的方法通过两个阶段的过程来解决这个问题，利用参考语音编码网络和全球信息解纠缠网络逐渐解开其他无关因素的扬声器身份信息。我们专门采用解纠缠的说话人身份信息来指导语音提取网络。此外，我们引入了自适应调制Transformer，以确保混合信号的声学表示不受扬声器嵌入的干扰。该组件将说话人嵌入作为条件信息，促进语音提取网络的自然和有效的指导。实验结果证实了我们精心制作的方法的有效性，展示了说话人混淆的可能性大幅减少。摘要：Speech signals are inherently complex as they encompass both global acoustic characteristics and local semantic information. However, in the task of target speech extraction, certain elements of global and local semantic information in the reference speech, which are irrelevant to speaker identity, can lead to speaker confusion within the speech extraction network. To overcome this challenge, we propose a self-supervised disentangled representation learning method. Our approach tackles this issue through a two-phase process, utilizing a reference speech encoding network and a global information disentanglement network to gradually disentangle the speaker identity information from other irrelevant factors. We exclusively employ the disentangled speaker identity information to guide the speech extraction network. Moreover, we introduce the adaptive modulation Transformer to ensure that the acoustic representation of the mixed signal remains undisturbed by the speaker embeddings. This component incorporates speaker embeddings as conditional information, facilitating natural and efficient guidance for the speech extraction network. Experimental results substantiate the effectiveness of our meticulously crafted approach, showcasing a substantial reduction in the likelihood of speaker confusion.

【18】 VoCopilot: Voice-Activated Tracking of Everyday Interactions标题：VoCopilot：声控跟踪日常互动链接：https://arxiv.org/abs/2312.10265

作者：Sheen An Goh,Manoj Gulati,Ambuj Varshney摘要：声音在我们的生活中起着重要的作用，通过促进沟通，传达情感和指示健康。因此，跟踪语音交互可以为我们生活的许多方面提供有价值的见解。本文介绍了我们正在努力设计一个新的语音跟踪系统，我们称之为VoCopilot。VoCopilot是一个端到端系统，以节能声学硬件和固件为中心，结合先进的机器学习模型。因此，VoCopilot能够持续跟踪对话，记录它们，转录它们，然后从中提取有用的见解。通过利用大型语言模型，VoCopilot确保用户可以从记录的交互中提取有用的见解，而无需学习复杂的机器学习技术。为了保护终端用户的隐私，VoCopilot使用了一种新颖的唤醒机制，只记录终端用户的对话。此外，管道的所有其余部分都可以在商品计算机（Mac Mini M2）上运行。在这项工作中，我们展示了VoCopilot在现实环境中两个用例的有效性。摘要：Voice plays an important role in our lives by facilitating communication, conveying emotions, and indicating health. Therefore, tracking vocal interactions can provide valuable insight into many aspects of our lives. This paper presents our ongoing efforts to design a new vocal tracking system we call VoCopilot. VoCopilot is an end-to-end system centered around an energy-efficient acoustic hardware and firmware combined with advanced machine learning models. As a result, VoCopilot is able to continuously track conversations, record them, transcribe them, and then extract useful insights from them. By utilizing large language models, VoCopilot ensures the user can extract useful insights from recorded interactions without having to learn complex machine learning techniques. In order to protect the privacy of end users, VoCopilot uses a novel wake-up mechanism that only records conversations of end users. Additionally, all the rest of pipeline can be run on a commodity computer (Mac Mini M2). In this work, we show the effectiveness of VoCopilot in real-world environment for two use cases.

【19】 A Refining Underlying Information Framework for Speech Enhancement标题：一种用于语音增强的精化底层信息框架链接：https://arxiv.org/abs/2312.11201

作者：Rui Cao,Tianrui Wang,Meng Ge,Longbiao Wang,Jianwu Dang备注：5 pages摘要：深度神经网络（DNN）的出现显著提高了单声道语音增强（SE）的性能。他们大多试图通过分布近似来隐式地捕捉语音的结构特征。然而，现有的方法容易受到退化的语音和残留噪声的问题。这封信以信息瓶颈为基础，作为重新思考SE的锚点。通过定义语音特征间互信息的增量收敛，阐明语音的声学特征对于缓解上述问题至关重要，因为它的明确引入有助于进一步逼近优化的最优信息论上界.参考熵的链式规则，我们还提出了一个框架来重构优化目标的信息组成，旨在整合和细化这一基本特征而不失一般性。可视化反映了与使用信息论的分析的一致性。实验结果表明，只有1.18 M的额外参数，改进的CRN取得了实质性的进展，一些先进的方法。源代码可在https://github.com/caoruitju/RUI_SE上获得。摘要：The advent of deep neural networks (DNN) has significantly improved the performance of monaural speech enhancement (SE). Most of them attempt to implicitly capture the structural features of speech through distribution approximation. However, existing methods are susceptible to the issues of degraded speech and residual noise. This letter is grounded in the Information Bottleneck as an anchor to rethink the SE. By defining the incremental convergence of mutual information between speech characteristics, we elucidate that the acoustic characteristic of speech is crucial in alleviating the above issues, for its explicit introduction contributes to further approximating the optimal information-theoretic upper bound of the optimization. Referring to the chain rule of entropy, we also propose a framework to reconstruct the information composition of the optimization objective, aiming to integrate and refine this underlying characteristic without loss of generality. The visualization reflects consistency with analysis using information theory. Experimental results show that with only 1.18 M additional parameters, the refined CRN has yielded substantial progress over a number of advanced methods. The source code is available at https://github.com/caoruitju/RUI_SE.

【20】 StyleSinger: Style Transfer for Out-Of-Domain Singing Voice Synthesis标题：StyleSinger：域外歌唱声音合成的风格转换链接：https://arxiv.org/abs/2312.10741

作者：Yu Zhang,Rongjie Huang,Ruiqi Li,JinZheng He,Yan Xia,Feiyang Chen,Xinyu Duan,Baoxing Huai,Zhou Zhao备注：Accepted by AAAI 2024摘要：域外（OOD）歌唱声音合成（SVS）的风格转移的重点是生成高质量的歌唱声音与看不见的风格（如音色，情感，发音和发音技巧）来自参考歌唱声音样本。然而，努力模拟复杂的细微差别的歌声风格是一项艰巨的任务，因为歌声具有显着程度的表现力。此外，现有的SVS方法在OOD场景中遇到合成歌声质量下降的问题，因为它们依赖于目标声音属性在训练阶段是可辨别的假设。为了克服这些挑战，我们提出了StyleSinger，第一个歌唱声音合成模型的zero-shot风格转移域外的参考歌唱声音样本。StyleSinger采用了两种关键方法来提高效率：1）残余风格适配器（RSA），它采用残余量化模块来捕获歌唱声音中的不同风格特征，以及2）不确定性建模层归一化（UMLN），以在训练阶段扰动内容表示中的风格属性，从而提高模型泛化能力。我们在zero-shot风格转移中的广泛评估合理地建立了StyleSinger在音频质量和与参考歌唱声音样本的相似性方面都优于基线模型。访问唱歌的声音样本可以在https://stylesinger.github.io/找到。摘要：Style transfer for out-of-domain (OOD) singing voice synthesis (SVS) focuses on generating high-quality singing voices with unseen styles (such as timbre, emotion, pronunciation, and articulation skills) derived from reference singing voice samples. However, the endeavor to model the intricate nuances of singing voice styles is an arduous task, as singing voices possess a remarkable degree of expressiveness. Moreover, existing SVS methods encounter a decline in the quality of synthesized singing voices in OOD scenarios, as they rest upon the assumption that the target vocal attributes are discernible during the training phase. To overcome these challenges, we propose StyleSinger, the first singing voice synthesis model for zero-shot style transfer of out-of-domain reference singing voice samples. StyleSinger incorporates two critical approaches for enhanced effectiveness: 1) the Residual Style Adaptor (RSA) which employs a residual quantization module to capture diverse style characteristics in singing voices, and 2) the Uncertainty Modeling Layer Normalization (UMLN) to perturb the style attributes within the content representation during the training phase and thus improve the model generalization. Our extensive evaluations in zero-shot style transfer undeniably establish that StyleSinger outperforms baseline models in both audio quality and similarity to the reference singing voice samples. Access to singing voice samples can be found at https://stylesinger.github.io/.

【21】 MM-TTS: Multi-modal Prompt based Style Transfer for Expressive Text-to-Speech Synthesis标题：MM-TTS：基于多模态提示的表达性文本到语音合成风格转换链接：https://arxiv.org/abs/2312.10687

作者：Wenhao Guan,Yishuang Li,Tao Li,Hukai Huang,Feng Wang,Jiayan Lin,Lingyan Huang,Lin Li,Qingyang Hong摘要：文本到语音转换中的风格转换任务是指将风格信息转换成文本内容，生成具有特定风格的语音的过程。然而，现有的风格迁移方法大多是基于固定的情感标签或参考语音片段，不能实现灵活的风格迁移。近年来，一些方法采用文本描述来引导风格迁移。在本文中，我们提出了一个更灵活的多模态和风格可控的TTS框架命名为MM-TTS。它可以在统一的多模态提示空间中使用任何模态作为提示，包括参考语音，情感面部图像和文本描述，以控制系统中生成的语音的风格。建模这种多模态风格可控的TTS的挑战主要在于两个方面：1）将多模态信息对齐到统一的风格空间中，以使得能够在单个系统中输入任意模态作为风格提示，以及2）有效地将统一的风格表示转换为给定的文本内容，从而赋予生成提示风格相关语音的能力。为了解决这些问题，我们提出了一个对齐的多模态提示编码器，它将不同的模态嵌入到一个统一的风格空间中，支持不同模态的风格转换。此外，我们提出了一种新的自适应风格转换方法命名为风格自适应卷积，以实现更好的风格表示。此外，我们设计了一个基于整流的细化器，以解决过度平滑梅尔频谱图的问题，并产生更高保真的音频。由于没有公开的多模态TTS数据集，我们构建了一个名为MEAD-TTS的数据集，它与表达性说话头部领域有关。我们在MEAD-TTS数据集和域外数据集上的实验表明，MM-TTS可以获得令人满意的结果，基于多模态提示。摘要：The style transfer task in Text-to-Speech refers to the process of transferring style information into text content to generate corresponding speech with a specific style. However, most existing style transfer approaches are either based on fixed emotional labels or reference speech clips, which cannot achieve flexible style transfer. Recently, some methods have adopted text descriptions to guide style transfer. In this paper, we propose a more flexible multi-modal and style controllable TTS framework named MM-TTS. It can utilize any modality as the prompt in unified multi-modal prompt space, including reference speech, emotional facial images, and text descriptions, to control the style of the generated speech in a system. The challenges of modeling such a multi-modal style controllable TTS mainly lie in two aspects:1)aligning the multi-modal information into a unified style space to enable the input of arbitrary modality as the style prompt in a single system, and 2)efficiently transferring the unified style representation into the given text content, thereby empowering the ability to generate prompt style-related voice. To address these problems, we propose an aligned multi-modal prompt encoder that embeds different modalities into a unified style space, supporting style transfer for different modalities. Additionally, we present a new adaptive style transfer method named Style Adaptive Convolutions to achieve a better style representation. Furthermore, we design a Rectified Flow based Refiner to solve the problem of over-smoothing Mel-spectrogram and generate audio of higher fidelity. Since there is no public dataset for multi-modal TTS, we construct a dataset named MEAD-TTS, which is related to the field of expressive talking head. Our experiments on the MEAD-TTS dataset and out-of-domain datasets demonstrate that MM-TTS can achieve satisfactory results based on multi-modal prompts.

【22】 On Robustness to Missing Video for Audiovisual Speech Recognition标题：视听语音识别对丢失视频的鲁棒性研究链接：https://arxiv.org/abs/2312.10088

作者：Oscar Chang,Otavio Braga,Hank Liao,Dmitriy Serdyuk,Olivier Siohan摘要：已经表明，学习视听特征可以导致改进的语音识别性能超过仅音频特征，特别是对于有噪声的语音。然而，在许多常见的应用中，视觉特征部分或全部缺失，例如~扬声器可能会移出屏幕。多模态模型需要具有鲁棒性：丢失的视频帧不应使视听模型的性能劣于单模态仅音频模型的性能。虽然已经有许多尝试建立稳健的模型，但在如何评估稳健性方面几乎没有共识。为了解决这个问题，我们引入了一个框架，允许以精确和可测试的方式评估关于鲁棒性的声明。我们还进行了一系列的声学噪声条件和测试套件的常见的视听语音识别架构的鲁棒性的系统的实证研究。最后，我们表明，基于级联的架构不可知的解决方案可以始终实现对丢失视频的鲁棒性，即使在现有的鲁棒性技术（如dropout）不足的情况下也是如此。摘要：It has been shown that learning audiovisual features can lead to improved speech recognition performance over audio-only features, especially for noisy speech. However, in many common applications, the visual features are partially or entirely missing, e.g.~the speaker might move off screen. Multi-modal models need to be robust: missing video frames should not degrade the performance of an audiovisual model to be worse than that of a single-modality audio-only model. While there have been many attempts at building robust models, there is little consensus on how robustness should be evaluated. To address this, we introduce a framework that allows claims about robustness to be evaluated in a precise and testable way. We also conduct a systematic empirical study of the robustness of common audiovisual speech recognition architectures on a range of acoustic noise conditions and test suites. Finally, we show that an architecture-agnostic solution based on cascades can consistently achieve robustness to missing video, even in settings where existing techniques for robustness like dropout fall short.

【23】 Revisiting the Entropy Semiring for Neural Speech Recognition标题：神经语音识别中的熵半环再认识链接：https://arxiv.org/abs/2312.10087

作者：Oscar Chang,Dongseong Hwang,Olivier Siohan摘要：在流媒体设置中，语音识别模型必须在完整的音频流可用之前将语音的子序列映射到文本。然而，由于语音和文本之间的对齐信息在训练过程中很少可用，因此模型需要以完全自我监督的方式学习它。在实践中，可能的对齐数量呈指数级，这使得这一点极具挑战性，模型通常会学习峰值或次优对齐。初步看来，对齐空间的指数性质使得甚至难以量化模型的对齐分布的不确定性。幸运的是，几十年来人们已经知道，概率有限状态换能器的熵可以通过基于半环的动态规划约简在时间上线性地计算到换能器的大小。在这项工作中，我们重新审视了神经语音识别模型的熵半环，并展示了对齐熵如何通过正则化或蒸馏来监督模型。我们还在半环框架中贡献了CTC和RNN-T的开源实现，其中包括熵半环的数值稳定和高度并行的变体。从经验上讲，我们观察到，添加对齐蒸馏提高了已经优化的师生蒸馏模型的准确性和延迟，在流媒体场景中实现了Librispeech数据集的最新性能。摘要：In streaming settings, speech recognition models have to map sub-sequences of speech to text before the full audio stream becomes available. However, since alignment information between speech and text is rarely available during training, models need to learn it in a completely self-supervised way. In practice, the exponential number of possible alignments makes this extremely challenging, with models often learning peaky or sub-optimal alignments. Prima facie, the exponential nature of the alignment space makes it difficult to even quantify the uncertainty of a model's alignment distribution. Fortunately, it has been known for decades that the entropy of a probabilistic finite state transducer can be computed in time linear to the size of the transducer via a dynamic programming reduction based on semirings. In this work, we revisit the entropy semiring for neural speech recognition models, and show how alignment entropy can be used to supervise models through regularization or distillation. We also contribute an open-source implementation of CTC and RNN-T in the semiring framework that includes numerically stable and highly parallel variants of the entropy semiring. Empirically, we observe that the addition of alignment distillation improves the accuracy and latency of an already well-optimized teacher-student distillation model, achieving state-of-the-art performance on the Librispeech dataset in the streaming scenario.

eess.AS音频处理

【1】 A Refining Underlying Information Framework for Speech Enhancement标题：一种用于语音增强的精化底层信息框架链接：https://arxiv.org/abs/2312.11201

【2】 Attention-Driven Multichannel Speech Enhancement in Moving Sound Source Scenarios标题：运动声源场景下基于注意力驱动的多通道语音增强链接：https://arxiv.org/abs/2312.10756

作者：Yuzhu Wang,Archontis Politis,Tuomas Virtanen摘要：目前的多通道语音增强算法通常假设一个固定的声源，一个常见的不匹配的现实，限制了他们在现实世界的情况下的性能。本文重点介绍了为动态设置而设计的注意力驱动的空间滤波技术。具体来说，我们研究了应用线性和非线性的注意力为基础的方法来估计时变空间协方差矩阵用于设计的过滤器。我们还研究了直接估计的空间滤波器的注意力为基础的方法，而不明确估计空间统计。利用WSJ 0中的干净语音片段模拟混响环境中运动说话人的语音信号。实验数据集是通过将模拟语音信号与CHiME-3的多通道真实噪声混合来构建的。评估结果表明，注意力驱动的方法是强大的，始终优于传统的空间滤波方法在静态和动态的声音环境。摘要：Current multichannel speech enhancement algorithms typically assume a stationary sound source, a common mismatch with reality that limits their performance in real-world scenarios. This paper focuses on attention-driven spatial filtering techniques designed for dynamic settings. Specifically, we study the application of linear and nonlinear attention-based methods for estimating time-varying spatial covariance matrices used to design the filters. We also investigate the direct estimation of spatial filters by attention-based methods without explicitly estimating spatial statistics. The clean speech clips from WSJ0 are employed for simulating speech signals of moving speakers in a reverberant environment. The experimental dataset is built by mixing the simulated speech signals with multichannel real noise from CHiME-3. Evaluation results show that the attention-driven approaches are robust and consistently outperform conventional spatial filtering approaches in both static and dynamic sound environments.

【3】 StyleSinger: Style Transfer for Out-Of-Domain Singing Voice Synthesis标题：StyleSinger：域外歌声合成的风格转换链接：https://arxiv.org/abs/2312.10741

【4】 MM-TTS: Multi-modal Prompt based Style Transfer for Expressive Text-to-Speech Synthesis标题：MM-TTS：基于多模态提示的表达性文本到语音合成风格转换链接：https://arxiv.org/abs/2312.10687

【5】 On Robustness to Missing Video for Audiovisual Speech Recognition标题：视听语音识别对丢失视频的鲁棒性研究链接：https://arxiv.org/abs/2312.10088

【6】 Revisiting the Entropy Semiring for Neural Speech Recognition标题：神经语音识别中的熵半环再认识链接：https://arxiv.org/abs/2312.10087

【7】 Evaluation of Barlow Twins and VICReg self-supervised learning for sound patterns of bird and anuran species标题：Barlow双胞胎和VICReg自监督学习对鸟类和无尾两栖动物声音模式的评估链接：https://arxiv.org/abs/2312.11240

【8】 Perceptual Musical Features for Interpretable Audio Tagging标题：用于可解释音频标注的感知音乐特征链接：https://arxiv.org/abs/2312.11234

作者：Vassilis Lyberatos,Spyridon Kantarelis,Edmund Dervakos,Giorgos Stamo备注：Github Repository: this https URL摘要：None摘要：In the age of music streaming platforms, the task of automatically tagging music audio has garnered significant attention, driving researchers to devise methods aimed at enhancing performance metrics on standard datasets. Most recent approaches rely on deep neural networks, which, despite their impressive performance, possess opacity, making it challenging to elucidate their output for a given input. While the issue of interpretability has been emphasized in other fields like medicine, it has not received attention in music-related tasks. In this study, we explored the relevance of interpretability in the context of automatic music tagging. We constructed a workflow that incorporates three different information extraction techniques: a) leveraging symbolic knowledge, b) utilizing auxiliary deep neural networks, and c) employing signal processing to extract perceptual features from audio files. These features were subsequently used to train an interpretable machine-learning model for tag prediction. We conducted experiments on two datasets, namely the MTG-Jamendo dataset and the GTZAN dataset. Our method surpassed the performance of baseline models in both tasks and, in certain instances, demonstrated competitiveness with the current state-of-the-art. We conclude that there are use cases where the deterioration in performance is outweighed by the value of interpretability.

【9】 Improved Long-Form Speech Recognition by Jointly Modeling the Primary and Non-primary Speakers标题：通过对主说话人和非主说话人进行联合建模来改进长格式语音识别链接：https://arxiv.org/abs/2312.11123

【10】 3S-TSE: Efficient Three-Stage Target Speaker Extraction for Real-Time and Low-Resource Applications标题：3S-TSE：面向实时和低资源应用的高效三阶段目标说话人提取链接：https://arxiv.org/abs/2312.10979

【11】 Generative linguistic representation for spoken language identification标题：口语识别的生成性语言表征链接：https://arxiv.org/abs/2312.10964

【12】 Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition标题：用于多说话人重叠语音识别的说话人面具转换器链接：https://arxiv.org/abs/2312.10959

作者：Peng Shen,Xugang Lu,Hisashi Kawai摘要：None摘要：Multi-talker overlapped speech recognition remains a significant challenge, requiring not only speech recognition but also speaker diarization tasks to be addressed. In this paper, to better address these tasks, we first introduce speaker labels into an autoregressive transformer-based speech recognition model to support multi-speaker overlapped speech recognition. Then, to improve speaker diarization, we propose a novel speaker mask branch to detection the speech segments of individual speakers. With the proposed model, we can perform both speech recognition and speaker diarization tasks simultaneously using a single model. Experimental results on the LibriSpeech-based overlapped dataset demonstrate the effectiveness of the proposed method in both speech recognition and speaker diarization tasks, particularly enhancing the accuracy of speaker diarization in relatively complex multi-talker scenarios.

【13】 Soft Alignment of Modality Space for End-to-end Speech Translation标题：端到端语音翻译中的通道空间软对齐链接：https://arxiv.org/abs/2312.10952

【14】 Leveraged Mel spectrograms using Harmonic and Percussive Components in Speech Emotion Recognition标题：基于谐波和敲击分量的Mel谱图在语音情感识别中的应用链接：https://arxiv.org/abs/2312.10949

作者：David Hason Rudd,Huan Huo,Guandong Xu备注：None摘要：None摘要：Speech Emotion Recognition (SER) affective technology enables the intelligent embedded devices to interact with sensitivity. Similarly, call centre employees recognise customers' emotions from their pitch, energy, and tone of voice so as to modify their speech for a high-quality interaction with customers. This work explores, for the first time, the effects of the harmonic and percussive components of Mel spectrograms in SER. We attempt to leverage the Mel spectrogram by decomposing distinguishable acoustic features for exploitation in our proposed architecture, which includes a novel feature map generator algorithm, a CNN-based network feature extractor and a multi-layer perceptron (MLP) classifier. This study specifically focuses on effective data augmentation techniques for building an enriched hybrid-based feature map. This process results in a function that outputs a 2D image so that it can be used as input data for a pre-trained CNN-VGG16 feature extractor. Furthermore, we also investigate other acoustic features such as MFCCs, chromagram, spectral contrast, and the tonnetz to assess our proposed framework. A test accuracy of 92.79% on the Berlin EMO-DB database is achieved. Our result is higher than previous works using CNN-VGG16.

【15】 An Extended Variational Mode Decomposition Algorithm Developed Speech Emotion Recognition Performance标题：一种提高语音情感识别性能的扩展变分模式分解算法链接：https://arxiv.org/abs/2312.10937

作者：David Hason Rudd,Huan Huo,Guandong Xu备注：None摘要：None摘要：Emotion recognition (ER) from speech signals is a robust approach since it cannot be imitated like facial expression or text based sentiment analysis. Valuable information underlying the emotions are significant for human-computer interactions enabling intelligent machines to interact with sensitivity in the real world. Previous ER studies through speech signal processing have focused exclusively on associations between different signal mode decomposition methods and hidden informative features. However, improper decomposition parameter selections lead to informative signal component losses due to mode duplicating and mixing. In contrast, the current study proposes VGG-optiVMD, an empowered variational mode decomposition algorithm, to distinguish meaningful speech features and automatically select the number of decomposed modes and optimum balancing parameter for the data fidelity constraint by assessing their effects on the VGG16 flattening output layer. Various feature vectors were employed to train the VGG16 network on different databases and assess VGG-optiVMD reproducibility and reliability. One, two, and three-dimensional feature vectors were constructed by concatenating Mel-frequency cepstral coefficients, Chromagram, Mel spectrograms, Tonnetz diagrams, and spectral centroids. Results confirmed a synergistic relationship between the fine-tuning of the signal sample rate and decomposition parameters with classification accuracy, achieving state-of-the-art 96.09% accuracy in predicting seven emotions on the Berlin EMO-DB database.

【16】 AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis标题：AE-NeRF：用于少数镜头说话头部合成的音频增强神经辐射场链接：https://arxiv.org/abs/2312.10921

作者：Dongze Li,Kang Zhao,Wei Wang,Bo Peng,Yingya Zhang,Jing Dong,Tieniu Tan备注：Accepted by AAAI 2024摘要：None摘要：Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality. Recent NeRF-based approaches have shown superiority in quality and fidelity compared to previous studies. However, when it comes to few-shot talking head generation, a practical scenario where only few seconds of talking video is available for one identity, two limitations emerge: 1) they either have no base model, which serves as a facial prior for fast convergence, or ignore the importance of audio when building the prior; 2) most of them overlook the degree of correlation between different face regions and audio, e.g., mouth is audio related, while ear is audio independent. In this paper, we present Audio Enhanced Neural Radiance Field (AE-NeRF) to tackle the above issues, which can generate realistic portraits of a new speaker with fewshot dataset. Specifically, we introduce an Audio Aware Aggregation module into the feature fusion stage of the reference scheme, where the weight is determined by the similarity of audio between reference and target image. Then, an Audio-Aligned Face Generation strategy is proposed to model the audio related and audio independent regions respectively, with a dual-NeRF framework. Extensive experiments have shown AE-NeRF surpasses the state-of-the-art on image fidelity, audio-lip synchronization, and generalization ability, even in limited training set or training iterations.

【17】 Exploring Sound vs Vibration for Robust Fault Detection on Rotating Machinery标题：旋转机械鲁棒故障检测的声振对比研究链接：https://arxiv.org/abs/2312.10742

作者：Serkan Kiranyaz,Ozer Can Devecioglu,Amir Alhams,Sadok Sassi,Turker Ince,Onur Avci,Moncef Gabbouj备注：8 pages摘要：None摘要：Robust and real-time detection of faults on rotating machinery has become an ultimate objective for predictive maintenance in various industries. Vibration-based Deep Learning (DL) methodologies have become the de facto standard for bearing fault detection as they can produce state-of-the-art detection performances under certain conditions. Despite such particular focus on the vibration signal, the utilization of sound, on the other hand, has been neglected whilst only a few studies have been proposed during the last two decades, all of which were based on a conventional ML approach. One major reason is the lack of a benchmark dataset providing a large volume of both vibration and sound data over several working conditions for different machines and sensor locations. In this study, we address this need by presenting the new benchmark Qatar University Dual-Machine Bearing Fault Benchmark dataset (QU-DMBF), which encapsulates sound and vibration data from two different motors operating under 1080 working conditions overall. Then we draw the focus on the major limitations and drawbacks of vibration-based fault detection due to numerous installation and operational conditions. Finally, we propose the first DL approach for sound-based fault detection and perform comparative evaluations between the sound and vibration over the QU-DMBF dataset. A wide range of experimental results shows that the sound-based fault detection method is significantly more robust than its vibration-based counterpart, as it is entirely independent of the sensor location, cost-effective (requiring no sensor and sensor maintenance), and can achieve the same level of the best detection performance by its vibration-based counterpart. With this study, the QU-DMBF dataset, the optimized source codes in PyTorch, and comparative evaluations are now publicly shared.

【18】 Meta-AF Echo Cancellation for Improved Keyword Spotting标题：用于改进关键字检测的Meta-AF回波消除链接：https://arxiv.org/abs/2312.10605

【19】 Seq2seq for Automatic Paraphasia Detection in Aphasic Speech标题：用于失语语音中失语自动检测的序列2seq链接：https://arxiv.org/abs/2312.10518

【20】 Annotation-Free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion标题：具有可扩展合成数据和对抗性领域混淆的无注释自动音乐转录链接：https://arxiv.org/abs/2312.10402

【21】 SECap: Speech Emotion Captioning with Large Language Model标题：SECap：基于大语言模型的语音情感字幕链接：https://arxiv.org/abs/2312.10381

【22】 MusER: Musical Element-Based Regularization for Generating Symbolic Music with Emotion标题：MUSER：基于音乐元素的正规化生成带情感的符号音乐链接：https://arxiv.org/abs/2312.10307

作者：Shulei Ji,Xinyu Yang备注：Accepted by AAAI 2024摘要：None摘要：Generating music with emotion is an important task in automatic music generation, in which emotion is evoked through a variety of musical elements (such as pitch and duration) that change over time and collaborate with each other. However, prior research on deep learning-based emotional music generation has rarely explored the contribution of different musical elements to emotions, let alone the deliberate manipulation of these elements to alter the emotion of music, which is not conducive to fine-grained element-level control over emotions. To address this gap, we present a novel approach employing musical element-based regularization in the latent space to disentangle distinct elements, investigate their roles in distinguishing emotions, and further manipulate elements to alter musical emotions. Specifically, we propose a novel VQ-VAE-based model named MusER. MusER incorporates a regularization loss to enforce the correspondence between the musical element sequences and the specific dimensions of latent variable sequences, providing a new solution for disentangling discrete sequences. Taking advantage of the disentangled latent vectors, a two-level decoding strategy that includes multiple decoders attending to latent vectors with different semantics is devised to better predict the elements. By visualizing latent space, we conclude that MusER yields a disentangled and interpretable latent space and gain insights into the contribution of distinct elements to the emotional dimensions (i.e., arousal and valence). Experimental results demonstrate that MusER outperforms the state-of-the-art models for generating emotional music in both objective and subjective evaluation. Besides, we rearrange music through element transfer and attempt to alter the emotion of music by transferring emotion-distinguishable elements.

【23】 Self-Supervised Disentangled Representation Learning for Robust Target Speech Extraction标题：自监督解缠表征学习用于鲁棒目标语音提取链接：https://arxiv.org/abs/2312.10305

【24】 VoCopilot: Voice-Activated Tracking of Everyday Interactions标题：VoCopilot：声控跟踪日常互动链接：https://arxiv.org/abs/2312.10265

机器翻译由腾讯交互翻译提供，仅供参考

语音/音频处理学术速递[12.19]

评论 0