今日论文合集：cs.SD语音10篇，eess.AS音频处理12篇。
cs.SD语音
【1】 A Review of Differentiable Digital Signal Processing for Music & Speech  Synthesis标题：用于音乐和语音合成的可区分数字信号处理综述链接：https://arxiv.org/abs/2308.15422
作者：Ben Hayes,Jordie Shier,György Fazekas,Andrew McPherson,Charalampos Saitis机构：Centre for Digital Music, School of Electronic Engineering and Computer Science, Queen Mary University of London, United Kingdom, Dyson School of Design Engineering, Imperial College London, United Kingdom, Correspondence:, Bethnal Green, London E,NS, UK备注：Under review for Frontiers in Signal Processing摘要：术语“可微分数字信号处理”描述了一系列技术，其中损失函数梯度通过数字信号处理器反向传播，便于将其集成到神经网络中。本文综述了可微分音频信号处理的文献，重点介绍了它在音乐和语音合成中的应用。我们的目录应用程序的任务，包括音乐表演渲染，声音匹配，和语音转换，讨论的动机和使用这种方法的影响。这是伴随着一个数字信号处理操作的概述，已经实现了差异。最后，我们强调开放的挑战，包括优化病理学，鲁棒性，现实世界的条件下，设计权衡，并讨论未来的研究方向。摘要：The term "differentiable digital signal processing" describes a family of techniques in which loss function gradients are backpropagated through digital signal processors, facilitating their integration into neural networks. This article surveys the literature on differentiable audio signal processing, focusing on its use in music & speech synthesis. We catalogue applications to tasks including music performance rendering, sound matching, and voice transformation, discussing the motivations for and implications of the use of this methodology. This is accompanied by an overview of digital signal processing operations that have been implemented differentiably. Finally, we highlight open challenges, including optimisation pathologies, robustness to real-world conditions, and design trade-offs, and discuss directions for future research.
【2】 Killing two birds with one stone: Can an audio captioning system also be  used for audio-text retrieval?标题：一举两得：音频字幕系统也可以用于音频文本检索吗？链接：https://arxiv.org/abs/2308.15090
作者：Etienne Labbé,Thomas Pellegrini,Julien Pinquier机构：IRIT, Universit´e Paul Sabatier, CNRS, Toulouse, France, Artificial and Natural Intelligence Toulouse Institute (ANITI)备注：None摘要：自动音频字幕（AAC）旨在开发能够使用文本语句来描述音频记录的系统。相比之下，音频-文本检索（ATR）系统寻求找到针对给定文本查询（文本到音频）或反之亦然（音频到文本）的最佳匹配音频记录。这些任务需要不同类型的系统：AAC采用序列到序列模型，而ATR利用在共享投影子空间内比较音频和文本表示的排名模型。然而，这项工作调查AAC和ATR之间的关系，通过探索ATR的能力的未经修改的AAC系统，没有微调的新任务。我们的AAC系统由一个音频编码器（ConvNeXt-Tiny）和一个Transformer解码器组成，前者在AudioSet上训练，用于音频标记，后者负责生成句子。对于AAC，它在Clotho上达到了0.298的高SPIDER-FL分数，在AudioCaps上平均达到了0.472。对于ATR，我们建议使用针对任何音频/字幕对获得的标准交叉熵损失值。Clotho和AudioCaps数据集上的实验结果表明，使用这种简单的方法可以获得不错的召回值。例如，我们为AudioCaps获得了0.382的Text-to-Audio R@1值，这高于没有外部数据的当前最先进的方法。有趣的是，我们观察到，标准化的损失值是必要的音频到文本检索。摘要：Automated Audio Captioning (AAC) aims to develop systems capable of describing an audio recording using a textual sentence. In contrast, Audio-Text Retrieval (ATR) systems seek to find the best matching audio recording(s) for a given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks require different types of systems: AAC employs a sequence-to-sequence model, while ATR utilizes a ranking model that compares audio and text representations within a shared projection subspace. However, this work investigates the relationship between AAC and ATR by exploring the ATR capabilities of an unmodified AAC system, without fine-tuning for the new task. Our AAC system consists of an audio encoder (ConvNeXt-Tiny) trained on AudioSet for audio tagging, and a transformer decoder responsible for generating sentences. For AAC, it achieves a high SPIDEr-FL score of 0.298 on Clotho and 0.472 on AudioCaps on average. For ATR, we propose using the standard Cross-Entropy loss values obtained for any audio/caption pair. Experimental results on the Clotho and AudioCaps datasets demonstrate decent recall values using this simple approach. For instance, we obtained a Text-to-Audio R@1 value of 0.382 for Au-dioCaps, which is above the current state-of-the-art method without external data. Interestingly, we observe that normalizing the loss values was necessary for Audio-to-Text retrieval.
【3】 Audio Deepfake Detection: A Survey标题：音频深伪检测研究综述链接：https://arxiv.org/abs/2308.14970
作者：Jiangyan Yi,Chenglong Wang,Jianhua Tao,Xiaohui Zhang,Chu Yuan Zhang,Yan Zhao机构：)•Jianhua Tao is with the Department of Automation,  Tsinghua University, •Chenglong Wang is with the University of Science and Technology ofChina, •Xiaohui Zhang is with the Beijing Jiaotong University, •Yan Zhao is with the Hebei University of Technology摘要：音频deepfake检测是一个新兴的活跃话题。越来越多的文献致力于研究deepfake检测算法并取得了有效的性能，但这一问题远未得到解决。虽然有一些综述文献，但还没有全面的调查，为研究者提供一个系统的概述这些发展和统一的评价。因此，在这篇调查论文中，我们首先强调了各种类型的deepfake音频的关键差异，然后概述和分析竞争，数据集，特征，分类和最先进方法的评估。对于每一个方面，基本技术，先进的发展和主要的挑战进行了讨论。此外，我们分别对ASVspoof 2021，ADD 2023和In-the-Wild数据集进行了代表性特征和分类器的统一比较，用于音频deepfake检测。调查表明，未来的研究应该解决缺乏大规模数据集在野外，现有的检测方法的未知虚假攻击的泛化能力差，以及检测结果的可解释性。摘要：Audio deepfake detection is an emerging active topic. A growing number of literatures have aimed to study deepfake detection algorithms and achieved effective performance, the problem of which is far from being solved. Although there are some review literatures, there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences across various types of deepfake audio, then outline and analyse competitions, datasets, features, classifications, and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are discussed. In addition, we perform a unified comparison of representative features and classifiers on ASVspoof 2021, ADD 2023 and In-the-Wild datasets for audio deepfake detection, respectively. The survey shows that future research should address the lack of large scale datasets in the wild, poor generalization of existing detection methods to unknown fake attacks, as well as interpretability of detection results.
【4】 Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech标题：Zero-Shot多说话人文语转换中的自我注意剪枝链接：https://arxiv.org/abs/2308.14909
作者：Hyungchan Yoon,Changhwan Kim,Eunwoo Song,Hyun-Wook Yoon,Hong-Goo Kang机构：Department of Electrical and Electronic Engineering, Yonsei University, Seoul, Korea, NAVER Cloud, South Korea备注：None摘要：对于个性化语音生成，神经文本到语音（TTS）模型必须成功地实现与有限的数据从目标说话人。为此，需要将基线TTS模型充分推广到域外数据（即，目标说话人的语音）。然而，方法来解决这个域外的泛化问题，TTS尚未得到彻底的研究。在这项工作中，我们提出了一种有效的修剪方法，称为稀疏注意，以提高TTS模型的泛化能力。特别是，我们修剪掉冗余连接的自我注意力层的注意力权重低于阈值。为了灵活地确定剪枝强度搜索最佳泛化度，我们还提出了一种新的可微剪枝方法，允许模型自动学习阈值。对zero-shot多说话人TTS的评价验证了我们的方法在语音质量和说话人相似性方面的有效性。摘要：For personalized speech generation, a neural text-to-speech (TTS) model must be successfully implemented with limited data from a target speaker. To this end, the baseline TTS model needs to be amply generalized to out-of-domain data (i.e., target speaker's speech). However, approaches to address this out-of-domain generalization problem in TTS have yet to be thoroughly studied. In this work, we propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. In particular, we prune off redundant connections from self-attention layers whose attention weights are below the threshold. To flexibly determine the pruning strength for searching optimal degree of generalization, we also propose a new differentiable pruning method that allows the model to automatically learn the thresholds. Evaluations on zero-shot multi-speaker TTS verify the effectiveness of our method in terms of voice quality and speaker similarity.
【5】 Neural approaches to spoken content embedding标题：语音内容嵌入的神经方法链接：https://arxiv.org/abs/2308.14905
作者：Shane Settle机构：A thesis submitted, in partial fulfillment of the requirements for, the degree of, Doctor of Philosophy in Computer Science, at the, TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO, Chicago, Illinois, Thesis Committee:, Karen Livescu (Thesis Advisor), Kevin Gimpel备注：PhD thesis摘要：语音片段的比较是语音处理的核心操作。在该领域的传统方法已经青睐帧级动态编程算法，如动态时间扭曲，因为它们不需要监督，但它们的性能和效率是有限的。作为替代方案，声学词嵌入-可变长度口语单词段的固定维矢量表示-已开始被考虑用于此类任务。然而，这种区分性嵌入模型、训练方法及其在现实世界下游任务中的应用的当前空间是有限的。我们首先考虑“单视图”训练损失，其目标是学习一个声学单词嵌入模型，该模型将相同单词和不同单词的口语片段对分开。然后，我们考虑“多视图”对比损失。在这种设置中，声学单词嵌入与字符序列的嵌入联合学习，以生成书面单词的声学接地嵌入或声学接地单词嵌入。  在这篇论文中，我们提出了新的判别声学词嵌入（AWE）和声学接地词嵌入（AGWE）的递归神经网络（RNNs）的基础上的方法。我们在效率和性能方面改进模型训练。我们将这些发展从英语扩展到几种低资源语言，并表明当标记数据有限时，多语言训练可以提高性能。我们应用我们的嵌入模型，单语和多语种，查询的下游任务，例如语音搜索和自动语音识别。最后，我们展示了我们的嵌入方法如何与最近的自监督语音模型进行比较和补充。摘要：Comparing spoken segments is a central operation to speech processing. Traditional approaches in this area have favored frame-level dynamic programming algorithms, such as dynamic time warping, because they require no supervision, but they are limited in performance and efficiency. As an alternative, acoustic word embeddings -- fixed-dimensional vector representations of variable-length spoken word segments -- have begun to be considered for such tasks as well. However, the current space of such discriminative embedding models, training approaches, and their application to real-world downstream tasks is limited. We start by considering ``single-view" training losses where the goal is to learn an acoustic word embedding model that separates same-word and different-word spoken segment pairs. Then, we consider ``multi-view" contrastive losses. In this setting, acoustic word embeddings are learned jointly with embeddings of character sequences to generate acoustically grounded embeddings of written words, or acoustically grounded word embeddings.  In this thesis, we contribute new discriminative acoustic word embedding (AWE) and acoustically grounded word embedding (AGWE) approaches based on recurrent neural networks (RNNs). We improve model training in terms of both efficiency and performance. We take these developments beyond English to several low-resource languages and show that multilingual training improves performance when labeled data is limited. We apply our embedding models, both monolingual and multilingual, to the downstream tasks of query-by-example speech search and automatic speech recognition. Finally, we show how our embedding approaches compare with and complement more recent self-supervised speech models.
【6】 Multiscale Contextual Learning for Speech Emotion Recognition in  Emergency Call Center Conversations标题：多尺度上下文学习用于紧急呼叫中心会话中的语音情感识别链接：https://arxiv.org/abs/2308.14894
作者：Théo Deschamps-Berger,Lori Lamel,Laurence Devillers机构：LISN-CNRS, Paris-Saclay University, France, LISN-CNRS, Sorbonne-University摘要：会话中的情感识别对于确保高级人机交互至关重要。然而，在现实生活中创建鲁棒和准确的情感识别系统是具有挑战性的，主要是由于在野外收集的情感数据集的稀缺性和无法考虑对话上下文。CEMO数据集由法国呼叫中心紧急呼叫期间代理和患者之间的对话组成，填补了这一空白。这些互动的性质突出了谈话的情绪流在预测患者情绪方面的作用，因为上下文通常可以在理解实际感受方面产生差异。本文提出了一种用于语音情感识别的多尺度会话上下文学习方法。我们研究了这种方法的语音转录和声学段。实验上，我们的方法使用目标段的前一个或下一个信息。在文本域中，我们测试了上下文窗口使用广泛的令牌（从10到100）和在语音轮水平，考虑来自相同和相反的发言人的输入。根据我们的测试，从之前的令牌派生的上下文比以下令牌对准确预测的影响更大。此外，在对话中采取同一发言者的最后一轮发言似乎是有用的。在声学领域，我们进行了深入的分析，周围的情绪对预测的影响。虽然使用Transformers的多尺度会话上下文学习可以增强紧急呼叫记录的文本模态的性能，但结合声学上下文更具挑战性。摘要：Emotion recognition in conversations is essential for ensuring advanced human-machine interactions. However, creating robust and accurate emotion recognition systems in real life is challenging, mainly due to the scarcity of emotion datasets collected in the wild and the inability to take into account the dialogue context. The CEMO dataset, composed of conversations between agents and patients during emergency calls to a French call center, fills this gap. The nature of these interactions highlights the role of the emotional flow of the conversation in predicting patient emotions, as context can often make a difference in understanding actual feelings. This paper presents a multi-scale conversational context learning approach for speech emotion recognition, which takes advantage of this hypothesis. We investigated this approach on both speech transcriptions and acoustic segments. Experimentally, our method uses the previous or next information of the targeted segment. In the text domain, we tested the context window using a wide range of tokens (from 10 to 100) and at the speech turns level, considering inputs from both the same and opposing speakers. According to our tests, the context derived from previous tokens has a more significant influence on accurate prediction than the following tokens. Furthermore, taking the last speech turn of the same speaker in the conversation seems useful. In the acoustic domain, we conducted an in-depth analysis of the impact of the surrounding emotions on the prediction. While multi-scale conversational context learning using Transformers can enhance performance in the textual modality for emergency call recordings, incorporating acoustic context is more challenging.
【7】 Let There Be Sound: Reconstructing High Quality Speech from Silent  Videos标题：让声音存在：从无声视频中重建高质量的语音链接：https://arxiv.org/abs/2308.15256
作者：Ji-Hoon Kim,Jaehun Kim,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea备注：10 pages, 2 figures, 5 tables摘要：这项工作的目标是重建高品质的语音从嘴唇运动单独，一项任务也被称为唇到语音。唇到语音系统的一个关键挑战是由（1）同音异义词的存在和（2）多个语音变化引起的一对多映射，从而导致错误发音和过度平滑的语音。在本文中，我们提出了一种新的唇到语音系统，显着提高了生成质量，从多个角度减轻一对多的映射问题。具体来说，我们将（1）自我监督的语音表示消除歧义同音异义，和（2）声学方差信息建模不同的语音风格。此外，为了更好地解决上述问题，我们采用了一个基于流的后网络，它捕获和细化所生成的语音的细节。我们进行了大量的实验，并证明我们的方法实现了生成质量接近真实的人类话语，优于现有的方法在语音自然度和可懂度方面的一个很大的利润。合成样本可在匿名演示页面获得：https://mm.kaist.ac.kr/projects/LTBS。摘要：The goal of this work is to reconstruct high quality speech from lip motions alone, a task also known as lip-to-speech. A key challenge of lip-to-speech systems is the one-to-many mapping caused by (1) the existence of homophenes and (2) multiple speech variations, resulting in a mispronounced and over-smoothed speech. In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives. Specifically, we incorporate (1) self-supervised speech representations to disambiguate homophenes, and (2) acoustic variance information to model diverse speech styles. Additionally, to better solve the aforementioned problem, we employ a flow based post-net which captures and refines the details of the generated speech. We perform extensive experiments and demonstrate that our method achieves the generation quality close to that of real human utterance, outperforming existing methods in terms of speech naturalness and intelligibility by a large margin. Synthesised samples are available at the anonymous demo page: https://mm.kaist.ac.kr/projects/LTBS.
【8】 Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for  Automatic Speech Recognition标题：无监督主动学习：优化自动语音识别的标注成本效益链接：https://arxiv.org/abs/2308.14814
作者：Zhisheng Zheng,Ziyang Ma,Yu Wang,Xie Chen机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai AI Laboratory备注：5 pages, 3 figures. Accepted to Interspeech 2023摘要：近年来，基于语音的自我监督学习（SSL）在包括自动语音识别（ASR）在内的各种任务中取得了重大进展。具有良好性能的ASR模型可以通过微调具有一小部分标记数据的SSL模型来实现。减少对标记数据的需求总是具有很大的实用价值。在本文中，我们进一步扩展SSL的使用，以减少标签成本与主动学习。三种类型的单位在不同粒度的语音信号在一个无监督的方式，并通过应用对比数据选择方法比较它们的效果。实验结果表明，我们提出的数据选择框架可以有效地提高字错误率（WER）超过11%，相同数量的标记数据，或减半的标记成本，同时保持相同的WER，相比随机选择。摘要：In recent years, speech-based self-supervised learning (SSL) has made significant progress in various tasks, including automatic speech recognition (ASR). An ASR model with decent performance can be realized by fine-tuning an SSL model with a small fraction of labeled data. Reducing the demand for labeled data is always of great practical value. In this paper, we further extend the use of SSL to cut down labeling costs with active learning. Three types of units on different granularities are derived from speech signals in an unsupervised way, and their effects are compared by applying a contrastive data selection method. The experimental results show that our proposed data selection framework can effectively improve the word error rate (WER) by more than 11% with the same amount of labeled data, or halve the labeling cost while maintaining the same WER, compared to random selection.
【9】 EEG-Derived Voice Signature for Attended Speaker Detection标题：一种基于脑电信号的语音特征提取方法链接：https://arxiv.org/abs/2308.14774
作者：Hongxu Zhu,Siqi Cai,Yidi Jiang,Qiquan Zhang,Haizhou Li机构：and Haizhou Li arewith the Department of Electrical and Computer Engineering,  NationalUniversity of Singapore, Qiquan Zhang is also with the School of Electrical Engineering andTelecommunications,  University of New South Wales备注：8 pages, 2 figures摘要：传统的基于EEG的听觉注意检测（AAD）是通过比较时变语音刺激和诱发的EEG信号来实现的。然而，为了获得可靠的相关值，这些方法需要长的决策窗口，从而导致长的检测延迟。人类有一种非凡的能力来识别和跟随一个已知的说话者，而不管说话的内容。在本文中，我们试图检测出席的发言人中的预先登记的发言人从诱发的脑电信号。以这种方式，我们避免在运行时依赖于用于AAD的语音刺激。在这样做时，我们提出了一种新的基于脑电图的出席说话人检测（E-ASD）的任务。\textit{Methods：}我们用固定维向量编码说话人的声音，称为说话人嵌入，并将其投影到音频衍生的声音签名，该签名表征说话人的独特声音，而不管说话的内容如何。我们假设这样的语音签名也存在于听者的大脑中，其可以从引出的EEG信号中解码，称为EEG衍生的语音签名。通过比较音频衍生的语音签名和EEG衍生的语音签名，我们能够有效地检测在听脑中的出席说话人。结果：实验表明，E-ASD能有效地从0.5s的脑电决策窗中检测出说话人，AAD准确率为99.78%，AUC为99.94%，EER为0.27%。\texit {结论：}我们得出结论，从EEG信号中导出被关注说话人的语音特征，以便在听的大脑中检测被关注说话人。\texit {意义：}我们提出了第一个概念证明，用于从鸡尾酒会环境中诱发的EEG信号中检测出席的说话者。E-ASD的成功实施标志着迈向智能助听器的重要一步。摘要：\textit{Objective:} Conventional EEG-based auditory attention detection (AAD) is achieved by comparing the time-varying speech stimuli and the elicited EEG signals. However, in order to obtain reliable correlation values, these methods necessitate a long decision window, resulting in a long detection latency. Humans have a remarkable ability to recognize and follow a known speaker, regardless of the spoken content. In this paper, we seek to detect the attended speaker among the pre-enrolled speakers from the elicited EEG signals. In this manner, we avoid relying on the speech stimuli for AAD at run-time. In doing so, we propose a novel EEG-based attended speaker detection (E-ASD) task. \textit{Methods:} We encode a speaker's voice with a fixed dimensional vector, known as speaker embedding, and project it to an audio-derived voice signature, which characterizes the speaker's unique voice regardless of the spoken content. We hypothesize that such a voice signature also exists in the listener's brain that can be decoded from the elicited EEG signals, referred to as EEG-derived voice signature. By comparing the audio-derived voice signature and the EEG-derived voice signature, we are able to effectively detect the attended speaker in the listening brain. \textit{Results:} Experiments show that E-ASD can effectively detect the attended speaker from the 0.5s EEG decision windows, achieving 99.78\% AAD accuracy, 99.94\% AUC, and 0.27\% EER. \textit{Conclusion:} We conclude that it is possible to derive the attended speaker's voice signature from the EEG signals so as to detect the attended speaker in a listening brain. \textit{Significance:} We present the first proof of concept for detecting the attended speaker from the elicited EEG signals in a cocktail party environment. The successful implementation of E-ASD marks a non-trivial, but crucial step towards smart hearing aids.
【10】 VoiceBank-2023: A Multi-Speaker Mandarin Speech Corpus for Constructing  Personalized TTS Systems for the Speech Impaired标题：VoiceBank-2023：用于构建个性化语音合成系统的多人普通话语料库链接：https://arxiv.org/abs/2308.14763
作者：Jia-Jyu Su,Pang-Chen Liao,Yen-Ting Lin,Wu-Hao Li,Guan-Ting Liou,Cheng-Che Kao,Wei-Cheng Chen,Jen-Chieh Chiang,Wen-Yang Chang,Pin-Han Lin,Chen-Yu Chiang机构：National Taipei University, Taiwan,National Yang Ming Chiao Tung University, Taiwan,AcoustInTek Co., Ltd., Taiwan备注：submitted to 26th International Conference of the ORIENTAL-COCOSDA摘要：针对普通话言语障碍者的个性化TTS系统服务很少被提及。台湾于2020年启动了语音银行项目，旨在建立一整套服务，为肌萎缩侧索硬化症患者提供个性化的普通话TTS系统。本文报道了语料库的设计，语料库记录，数据清洗和校正语料库，并评价开发的个性化TTS系统，语音银行项目。开发的语料库以VoiceBank-2023语音语料库命名，因为它的发布年份。语料库包含了111位母语为普通话的人29.78小时的短段落和常用短语的话语。语料库标记有关于性别、言语障碍程度、用户类型、转录、SNR和说话速率的信息。VoiceBank-2023可应要求提供，用于非商业用途，并欢迎各方加入VoiceBanking项目，以改善为语言障碍者提供的服务。摘要：Services of personalized TTS systems for the Mandarin-speaking speech impaired are rarely mentioned. Taiwan started the VoiceBanking project in 2020, aiming to build a complete set of services to deliver personalized Mandarin TTS systems to amyotrophic lateral sclerosis patients. This paper reports the corpus design, corpus recording, data purging and correction for the corpus, and evaluations of the developed personalized TTS systems, for the VoiceBanking project. The developed corpus is named after the VoiceBank-2023 speech corpus because of its release year. The corpus contains 29.78 hours of utterances with prompts of short paragraphs and common phrases spoken by 111 native Mandarin speakers. The corpus is labeled with information about gender, degree of speech impairment, types of users, transcription, SNRs, and speaking rates. The VoiceBank-2023 is available by request for non-commercial use and welcomes all parties to join the VoiceBanking project to improve the services for the speech impaired.
eess.AS音频处理
【1】 Let There Be Sound: Reconstructing High Quality Speech from Silent  Videos标题：让声音存在：从无声视频中重建高质量的语音链接：https://arxiv.org/abs/2308.15256
作者：Ji-Hoon Kim,Jaehun Kim,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea备注：10 pages, 2 figures, 5 tables摘要：这项工作的目标是重建高品质的语音从嘴唇运动单独，一项任务也被称为唇到语音。唇到语音系统的一个关键挑战是由（1）同音异义词的存在和（2）多个语音变化引起的一对多映射，从而导致错误发音和过度平滑的语音。在本文中，我们提出了一种新的唇到语音系统，显着提高了生成质量，从多个角度减轻一对多的映射问题。具体来说，我们将（1）自我监督的语音表示消除歧义同音异义，和（2）声学方差信息建模不同的语音风格。此外，为了更好地解决上述问题，我们采用了一个基于流的后网络，它捕获和细化所生成的语音的细节。我们进行了大量的实验，并证明我们的方法实现了生成质量接近真实的人类话语，优于现有的方法在语音自然度和可懂度方面的一个很大的利润。合成样本可在匿名演示页面获得：https://mm.kaist.ac.kr/projects/LTBS。摘要：The goal of this work is to reconstruct high quality speech from lip motions alone, a task also known as lip-to-speech. A key challenge of lip-to-speech systems is the one-to-many mapping caused by (1) the existence of homophenes and (2) multiple speech variations, resulting in a mispronounced and over-smoothed speech. In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives. Specifically, we incorporate (1) self-supervised speech representations to disambiguate homophenes, and (2) acoustic variance information to model diverse speech styles. Additionally, to better solve the aforementioned problem, we employ a flow based post-net which captures and refines the details of the generated speech. We perform extensive experiments and demonstrate that our method achieves the generation quality close to that of real human utterance, outperforming existing methods in terms of speech naturalness and intelligibility by a large margin. Synthesised samples are available at the anonymous demo page: https://mm.kaist.ac.kr/projects/LTBS.
【2】 Preference-based training framework for automatic speech quality  assessment using deep neural network标题：基于偏好的深度神经网络语音质量自动评估训练框架链接：https://arxiv.org/abs/2308.15203
作者：Cheng-Hung Hu,Yusuke Yasuda,Tomoki Toda机构：Nagoya University备注：Accepted by Interspeech 2023, oral摘要：语音质量评估（SQA）的一个目的是估计合成语音系统的等级。然而，最近的SQA模型通常使用低精度的直接分数（如平均意见分数（MOS））作为训练目标进行训练，这并不直接估计排名。虽然它对预测单个句子的质量分数是有效的，但是当对多个系统进行排名时，这种方法不考虑语音和系统偏好。我们提出了一个训练框架的SQA模型，可以训练只有偏好得分从对MOS，以提高排名预测。我们的实验揭示了条件，我们的框架工作最好的对生成，聚合函数，以获得系统评分从话语的偏好，和阈值函数，以确定从一对MOS的偏好。我们的研究结果表明，我们提出的方法显着优于基线模型的斯皮尔曼的秩相关系数。摘要：One objective of Speech Quality Assessment (SQA) is to estimate the ranks of synthetic speech systems. However, recent SQA models are typically trained using low-precision direct scores such as mean opinion scores (MOS) as the training objective, which is not straightforward to estimate ranking. Although it is effective for predicting quality scores of individual sentences, this approach does not account for speech and system preferences when ranking multiple systems. We propose a training framework of SQA models that can be trained with only preference scores derived from pairs of MOS to improve ranking prediction. Our experiment reveals conditions where our framework works the best in terms of pair generation, aggregation functions to derive system score from utterance preferences, and threshold functions to determine preference from a pair of MOS. Our results demonstrate that our proposed method significantly outperforms the baseline model in Spearman's Rank Correlation Coefficient.
【3】 Unsupervised Active Learning: Optimizing Labeling Cost-Effectiveness for  Automatic Speech Recognition标题：无监督主动学习：优化自动语音识别的标注成本效益链接：https://arxiv.org/abs/2308.14814
作者：Zhisheng Zheng,Ziyang Ma,Yu Wang,Xie Chen机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai AI Laboratory备注：5 pages, 3 figures. Accepted to Interspeech 2023摘要：近年来，基于语音的自我监督学习（SSL）在包括自动语音识别（ASR）在内的各种任务中取得了重大进展。具有良好性能的ASR模型可以通过微调具有一小部分标记数据的SSL模型来实现。减少对标记数据的需求总是具有很大的实用价值。在本文中，我们进一步扩展SSL的使用，以减少标签成本与主动学习。三种类型的单位在不同粒度的语音信号在一个无监督的方式，并通过应用对比数据选择方法比较它们的效果。实验结果表明，我们提出的数据选择框架可以有效地提高字错误率（WER）超过11%，相同数量的标记数据，或减半的标记成本，同时保持相同的WER，相比随机选择。摘要：In recent years, speech-based self-supervised learning (SSL) has made significant progress in various tasks, including automatic speech recognition (ASR). An ASR model with decent performance can be realized by fine-tuning an SSL model with a small fraction of labeled data. Reducing the demand for labeled data is always of great practical value. In this paper, we further extend the use of SSL to cut down labeling costs with active learning. Three types of units on different granularities are derived from speech signals in an unsupervised way, and their effects are compared by applying a contrastive data selection method. The experimental results show that our proposed data selection framework can effectively improve the word error rate (WER) by more than 11% with the same amount of labeled data, or halve the labeling cost while maintaining the same WER, compared to random selection.
【4】 EEG-Derived Voice Signature for Attended Speaker Detection标题：一种基于脑电信号的语音特征提取方法链接：https://arxiv.org/abs/2308.14774
作者：Hongxu Zhu,Siqi Cai,Yidi Jiang,Qiquan Zhang,Haizhou Li机构：and Haizhou Li arewith the Department of Electrical and Computer Engineering,  NationalUniversity of Singapore, Qiquan Zhang is also with the School of Electrical Engineering andTelecommunications,  University of New South Wales备注：8 pages, 2 figures摘要：传统的基于EEG的听觉注意检测（AAD）是通过比较时变语音刺激和诱发的EEG信号来实现的。然而，为了获得可靠的相关值，这些方法需要长的决策窗口，从而导致长的检测延迟。人类有一种非凡的能力来识别和跟随一个已知的说话者，而不管说话的内容。在本文中，我们试图检测出席的发言人中的预先登记的发言人从诱发的脑电信号。以这种方式，我们避免在运行时依赖于用于AAD的语音刺激。在这样做时，我们提出了一种新的基于脑电图的出席说话人检测（E-ASD）的任务。\textit{Methods：}我们用固定维向量编码说话人的声音，称为说话人嵌入，并将其投影到音频衍生的声音签名，该签名表征说话人的独特声音，而不管说话的内容如何。我们假设这样的语音签名也存在于听者的大脑中，其可以从引出的EEG信号中解码，称为EEG衍生的语音签名。通过比较音频衍生的语音签名和EEG衍生的语音签名，我们能够有效地检测在听脑中的出席说话人。结果：实验表明，E-ASD能有效地从0.5s的脑电决策窗中检测出说话人，AAD准确率为99.78%，AUC为99.94%，EER为0.27%。\texit {结论：}我们得出结论，从EEG信号中导出被关注说话人的语音特征，以便在听的大脑中检测被关注说话人。\texit {意义：}我们提出了第一个概念证明，用于从鸡尾酒会环境中诱发的EEG信号中检测出席的说话者。E-ASD的成功实施标志着迈向智能助听器的重要一步。摘要：\textit{Objective:} Conventional EEG-based auditory attention detection (AAD) is achieved by comparing the time-varying speech stimuli and the elicited EEG signals. However, in order to obtain reliable correlation values, these methods necessitate a long decision window, resulting in a long detection latency. Humans have a remarkable ability to recognize and follow a known speaker, regardless of the spoken content. In this paper, we seek to detect the attended speaker among the pre-enrolled speakers from the elicited EEG signals. In this manner, we avoid relying on the speech stimuli for AAD at run-time. In doing so, we propose a novel EEG-based attended speaker detection (E-ASD) task. \textit{Methods:} We encode a speaker's voice with a fixed dimensional vector, known as speaker embedding, and project it to an audio-derived voice signature, which characterizes the speaker's unique voice regardless of the spoken content. We hypothesize that such a voice signature also exists in the listener's brain that can be decoded from the elicited EEG signals, referred to as EEG-derived voice signature. By comparing the audio-derived voice signature and the EEG-derived voice signature, we are able to effectively detect the attended speaker in the listening brain. \textit{Results:} Experiments show that E-ASD can effectively detect the attended speaker from the 0.5s EEG decision windows, achieving 99.78\% AAD accuracy, 99.94\% AUC, and 0.27\% EER. \textit{Conclusion:} We conclude that it is possible to derive the attended speaker's voice signature from the EEG signals so as to detect the attended speaker in a listening brain. \textit{Significance:} We present the first proof of concept for detecting the attended speaker from the elicited EEG signals in a cocktail party environment. The successful implementation of E-ASD marks a non-trivial, but crucial step towards smart hearing aids.
【5】 VoiceBank-2023: A Multi-Speaker Mandarin Speech Corpus for Constructing  Personalized TTS Systems for the Speech Impaired标题：VoiceBank-2023：用于构建个性化语音合成系统的多人普通话语料库链接：https://arxiv.org/abs/2308.14763
作者：Jia-Jyu Su,Pang-Chen Liao,Yen-Ting Lin,Wu-Hao Li,Guan-Ting Liou,Cheng-Che Kao,Wei-Cheng Chen,Jen-Chieh Chiang,Wen-Yang Chang,Pin-Han Lin,Chen-Yu Chiang机构：National Taipei University, Taiwan,National Yang Ming Chiao Tung University, Taiwan,AcoustInTek Co., Ltd., Taiwan备注：submitted to 26th International Conference of the ORIENTAL-COCOSDA摘要：针对普通话言语障碍者的个性化TTS系统服务很少被提及。台湾于2020年启动了语音银行项目，旨在建立一整套服务，为肌萎缩侧索硬化症患者提供个性化的普通话TTS系统。本文报道了语料库的设计，语料库记录，数据清洗和校正语料库，并评价开发的个性化TTS系统，语音银行项目。开发的语料库以VoiceBank-2023语音语料库命名，因为它的发布年份。语料库包含了111位母语为普通话的人29.78小时的短段落和常用短语的话语。语料库标记有关于性别、言语障碍程度、用户类型、转录、SNR和说话速率的信息。VoiceBank-2023可应要求提供，用于非商业用途，并欢迎各方加入VoiceBanking项目，以改善为语言障碍者提供的服务。摘要：Services of personalized TTS systems for the Mandarin-speaking speech impaired are rarely mentioned. Taiwan started the VoiceBanking project in 2020, aiming to build a complete set of services to deliver personalized Mandarin TTS systems to amyotrophic lateral sclerosis patients. This paper reports the corpus design, corpus recording, data purging and correction for the corpus, and evaluations of the developed personalized TTS systems, for the VoiceBanking project. The developed corpus is named after the VoiceBank-2023 speech corpus because of its release year. The corpus contains 29.78 hours of utterances with prompts of short paragraphs and common phrases spoken by 111 native Mandarin speakers. The corpus is labeled with information about gender, degree of speech impairment, types of users, transcription, SNRs, and speaking rates. The VoiceBank-2023 is available by request for non-commercial use and welcomes all parties to join the VoiceBanking project to improve the services for the speech impaired.
【6】 A Review of Differentiable Digital Signal Processing for Music & Speech  Synthesis标题：用于音乐和语音合成的可区分数字信号处理综述链接：https://arxiv.org/abs/2308.15422
作者：Ben Hayes,Jordie Shier,György Fazekas,Andrew McPherson,Charalampos Saitis机构：Centre for Digital Music, School of Electronic Engineering and Computer Science, Queen Mary University of London, United Kingdom, Dyson School of Design Engineering, Imperial College London, United Kingdom, Correspondence:, Bethnal Green, London E,NS, UK备注：Under review for Frontiers in Signal Processing摘要：术语“可微分数字信号处理”描述了一系列技术，其中损失函数梯度通过数字信号处理器反向传播，便于将其集成到神经网络中。本文综述了可微分音频信号处理的文献，重点介绍了它在音乐和语音合成中的应用。我们的目录应用程序的任务，包括音乐表演渲染，声音匹配，和语音转换，讨论的动机和使用这种方法的影响。这是伴随着一个数字信号处理操作的概述，已经实现了差异。最后，我们强调开放的挑战，包括优化病理学，鲁棒性，现实世界的条件下，设计权衡，并讨论未来的研究方向。摘要：The term "differentiable digital signal processing" describes a family of techniques in which loss function gradients are backpropagated through digital signal processors, facilitating their integration into neural networks. This article surveys the literature on differentiable audio signal processing, focusing on its use in music & speech synthesis. We catalogue applications to tasks including music performance rendering, sound matching, and voice transformation, discussing the motivations for and implications of the use of this methodology. This is accompanied by an overview of digital signal processing operations that have been implemented differentiably. Finally, we highlight open challenges, including optimisation pathologies, robustness to real-world conditions, and design trade-offs, and discuss directions for future research.
【7】 Killing two birds with one stone: Can an audio captioning system also be  used for audio-text retrieval?标题：一举两得：音频字幕系统也可以用于音频文本检索吗？链接：https://arxiv.org/abs/2308.15090
作者：Etienne Labbé,Thomas Pellegrini,Julien Pinquier机构：IRIT, Universit´e Paul Sabatier, CNRS, Toulouse, France, Artificial and Natural Intelligence Toulouse Institute (ANITI)备注：None摘要：自动音频字幕（AAC）旨在开发能够使用文本语句来描述音频记录的系统。相比之下，音频-文本检索（ATR）系统寻求找到针对给定文本查询（文本到音频）或反之亦然（音频到文本）的最佳匹配音频记录。这些任务需要不同类型的系统：AAC采用序列到序列模型，而ATR利用在共享投影子空间内比较音频和文本表示的排名模型。然而，这项工作调查AAC和ATR之间的关系，通过探索ATR的能力的未经修改的AAC系统，没有微调的新任务。我们的AAC系统由一个音频编码器（ConvNeXt-Tiny）和一个Transformer解码器组成，前者在AudioSet上训练，用于音频标记，后者负责生成句子。对于AAC，它在Clotho上达到了0.298的高SPIDER-FL分数，在AudioCaps上平均达到了0.472。对于ATR，我们建议使用针对任何音频/字幕对获得的标准交叉熵损失值。Clotho和AudioCaps数据集上的实验结果表明，使用这种简单的方法可以获得不错的召回值。例如，我们为AudioCaps获得了0.382的Text-to-Audio R@1值，这高于没有外部数据的当前最先进的方法。有趣的是，我们观察到，标准化的损失值是必要的音频到文本检索。摘要：Automated Audio Captioning (AAC) aims to develop systems capable of describing an audio recording using a textual sentence. In contrast, Audio-Text Retrieval (ATR) systems seek to find the best matching audio recording(s) for a given textual query (Text-to-Audio) or vice versa (Audio-to-Text). These tasks require different types of systems: AAC employs a sequence-to-sequence model, while ATR utilizes a ranking model that compares audio and text representations within a shared projection subspace. However, this work investigates the relationship between AAC and ATR by exploring the ATR capabilities of an unmodified AAC system, without fine-tuning for the new task. Our AAC system consists of an audio encoder (ConvNeXt-Tiny) trained on AudioSet for audio tagging, and a transformer decoder responsible for generating sentences. For AAC, it achieves a high SPIDEr-FL score of 0.298 on Clotho and 0.472 on AudioCaps on average. For ATR, we propose using the standard Cross-Entropy loss values obtained for any audio/caption pair. Experimental results on the Clotho and AudioCaps datasets demonstrate decent recall values using this simple approach. For instance, we obtained a Text-to-Audio R@1 value of 0.382 for Au-dioCaps, which is above the current state-of-the-art method without external data. Interestingly, we observe that normalizing the loss values was necessary for Audio-to-Text retrieval.
【8】 Audio Deepfake Detection: A Survey标题：音频深伪检测研究综述链接：https://arxiv.org/abs/2308.14970
作者：Jiangyan Yi,Chenglong Wang,Jianhua Tao,Xiaohui Zhang,Chu Yuan Zhang,Yan Zhao机构：)•Jianhua Tao is with the Department of Automation,  Tsinghua University, •Chenglong Wang is with the University of Science and Technology ofChina, •Xiaohui Zhang is with the Beijing Jiaotong University, •Yan Zhao is with the Hebei University of Technology摘要：音频deepfake检测是一个新兴的活跃话题。越来越多的文献致力于研究deepfake检测算法并取得了有效的性能，但这一问题远未得到解决。虽然有一些综述文献，但还没有全面的调查，为研究者提供一个系统的概述这些发展和统一的评价。因此，在这篇调查论文中，我们首先强调了各种类型的deepfake音频的关键差异，然后概述和分析竞争，数据集，特征，分类和最先进方法的评估。对于每一个方面，基本技术，先进的发展和主要的挑战进行了讨论。此外，我们分别对ASVspoof 2021，ADD 2023和In-the-Wild数据集进行了代表性特征和分类器的统一比较，用于音频deepfake检测。调查表明，未来的研究应该解决缺乏大规模数据集在野外，现有的检测方法的未知虚假攻击的泛化能力差，以及检测结果的可解释性。摘要：Audio deepfake detection is an emerging active topic. A growing number of literatures have aimed to study deepfake detection algorithms and achieved effective performance, the problem of which is far from being solved. Although there are some review literatures, there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences across various types of deepfake audio, then outline and analyse competitions, datasets, features, classifications, and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are discussed. In addition, we perform a unified comparison of representative features and classifiers on ASVspoof 2021, ADD 2023 and In-the-Wild datasets for audio deepfake detection, respectively. The survey shows that future research should address the lack of large scale datasets in the wild, poor generalization of existing detection methods to unknown fake attacks, as well as interpretability of detection results.
【9】 Robust Open-Set Spoken Language Identification and the CU MultiLang  Dataset标题：稳健开放集口语识别与CU多语言数据集链接：https://arxiv.org/abs/2308.14951
作者：Mustafa Eyceoz,Justin Lee,Siddharth Pittie,Homayoon Beigi机构：Columbia University, New York, USA, Recognition Technologies, Inc., New York, USA备注：None摘要：大多数最先进的口语识别模型都是闭集的;换句话说，他们只能从他们训练的类别集合中输出语言标签。然而，开放式口语识别系统获得了检测输入时没有表现出原始语言的能力。在本文中，我们实现了一种新的方法来开集口语识别，使用MFCC和音高特征，TDNN模型来提取有意义的特征嵌入，softmax输出的置信度阈值，以及LDA和pLDA用于学习分类新的未知语言。我们提出了一个口语识别系统，达到91.76%的准确率训练语言，并有能力适应未知的语言飞行。为此，我们还建立了CU MultiLang Dataset，一个大型和多样化的多语言语音语料库，用于训练和评估我们的系统。摘要：Most state-of-the-art spoken language identification models are closed-set; in other words, they can only output a language label from the set of classes they were trained on. Open-set spoken language identification systems, however, gain the ability to detect when an input exhibits none of the original languages. In this paper, we implement a novel approach to open-set spoken language identification that uses MFCC and pitch features, a TDNN model to extract meaningful feature embeddings, confidence thresholding on softmax outputs, and LDA and pLDA for learning to classify new unknown languages. We present a spoken language identification system that achieves 91.76% accuracy on trained languages and has the capability to adapt to unknown languages on the fly. To that end, we also built the CU MultiLang Dataset, a large and diverse multilingual speech corpus which was used to train and evaluate our system.
【10】 Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech标题：Zero-Shot多说话人文语转换中的自我注意剪枝链接：https://arxiv.org/abs/2308.14909
作者：Hyungchan Yoon,Changhwan Kim,Eunwoo Song,Hyun-Wook Yoon,Hong-Goo Kang机构：Department of Electrical and Electronic Engineering, Yonsei University, Seoul, Korea, NAVER Cloud, South Korea备注：None摘要：对于个性化语音生成，神经文本到语音（TTS）模型必须成功地实现与有限的数据从目标说话人。为此，需要将基线TTS模型充分推广到域外数据（即，目标说话人的语音）。然而，方法来解决这个域外的泛化问题，TTS尚未得到彻底的研究。在这项工作中，我们提出了一种有效的修剪方法，称为稀疏注意，以提高TTS模型的泛化能力。特别是，我们修剪掉冗余连接的自我注意力层的注意力权重低于阈值。为了灵活地确定剪枝强度搜索最佳泛化度，我们还提出了一种新的可微剪枝方法，允许模型自动学习阈值。对zero-shot多说话人TTS的评价验证了我们的方法在语音质量和说话人相似性方面的有效性。摘要：For personalized speech generation, a neural text-to-speech (TTS) model must be successfully implemented with limited data from a target speaker. To this end, the baseline TTS model needs to be amply generalized to out-of-domain data (i.e., target speaker's speech). However, approaches to address this out-of-domain generalization problem in TTS have yet to be thoroughly studied. In this work, we propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. In particular, we prune off redundant connections from self-attention layers whose attention weights are below the threshold. To flexibly determine the pruning strength for searching optimal degree of generalization, we also propose a new differentiable pruning method that allows the model to automatically learn the thresholds. Evaluations on zero-shot multi-speaker TTS verify the effectiveness of our method in terms of voice quality and speaker similarity.
【11】 Neural approaches to spoken content embedding标题：语音内容嵌入的神经方法链接：https://arxiv.org/abs/2308.14905
作者：Shane Settle机构：A thesis submitted, in partial fulfillment of the requirements for, the degree of, Doctor of Philosophy in Computer Science, at the, TOYOTA TECHNOLOGICAL INSTITUTE AT CHICAGO, Chicago, Illinois, Thesis Committee:, Karen Livescu (Thesis Advisor), Kevin Gimpel备注：PhD thesis摘要：语音片段的比较是语音处理的核心操作。在该领域的传统方法已经青睐帧级动态编程算法，如动态时间扭曲，因为它们不需要监督，但它们的性能和效率是有限的。作为替代方案，声学词嵌入-可变长度口语单词段的固定维矢量表示-已开始被考虑用于此类任务。然而，这种区分性嵌入模型、训练方法及其在现实世界下游任务中的应用的当前空间是有限的。我们首先考虑“单视图”训练损失，其目标是学习一个声学单词嵌入模型，该模型将相同单词和不同单词的口语片段对分开。然后，我们考虑“多视图”对比损失。在这种设置中，声学单词嵌入与字符序列的嵌入联合学习，以生成书面单词的声学接地嵌入或声学接地单词嵌入。  在这篇论文中，我们提出了新的判别声学词嵌入（AWE）和声学接地词嵌入（AGWE）的递归神经网络（RNNs）的基础上的方法。我们在效率和性能方面改进模型训练。我们将这些发展从英语扩展到几种低资源语言，并表明当标记数据有限时，多语言训练可以提高性能。我们应用我们的嵌入模型，单语和多语种，查询的下游任务，例如语音搜索和自动语音识别。最后，我们展示了我们的嵌入方法如何与最近的自监督语音模型进行比较和补充。摘要：Comparing spoken segments is a central operation to speech processing. Traditional approaches in this area have favored frame-level dynamic programming algorithms, such as dynamic time warping, because they require no supervision, but they are limited in performance and efficiency. As an alternative, acoustic word embeddings -- fixed-dimensional vector representations of variable-length spoken word segments -- have begun to be considered for such tasks as well. However, the current space of such discriminative embedding models, training approaches, and their application to real-world downstream tasks is limited. We start by considering ``single-view" training losses where the goal is to learn an acoustic word embedding model that separates same-word and different-word spoken segment pairs. Then, we consider ``multi-view" contrastive losses. In this setting, acoustic word embeddings are learned jointly with embeddings of character sequences to generate acoustically grounded embeddings of written words, or acoustically grounded word embeddings.  In this thesis, we contribute new discriminative acoustic word embedding (AWE) and acoustically grounded word embedding (AGWE) approaches based on recurrent neural networks (RNNs). We improve model training in terms of both efficiency and performance. We take these developments beyond English to several low-resource languages and show that multilingual training improves performance when labeled data is limited. We apply our embedding models, both monolingual and multilingual, to the downstream tasks of query-by-example speech search and automatic speech recognition. Finally, we show how our embedding approaches compare with and complement more recent self-supervised speech models.
【12】 Multiscale Contextual Learning for Speech Emotion Recognition in  Emergency Call Center Conversations标题：多尺度上下文学习用于紧急呼叫中心会话中的语音情感识别链接：https://arxiv.org/abs/2308.14894
作者：Théo Deschamps-Berger,Lori Lamel,Laurence Devillers机构：LISN-CNRS, Paris-Saclay University, France, LISN-CNRS, Sorbonne-University摘要：会话中的情感识别对于确保高级人机交互至关重要。然而，在现实生活中创建鲁棒和准确的情感识别系统是具有挑战性的，主要是由于在野外收集的情感数据集的稀缺性和无法考虑对话上下文。CEMO数据集由法国呼叫中心紧急呼叫期间代理和患者之间的对话组成，填补了这一空白。这些互动的性质突出了谈话的情绪流在预测患者情绪方面的作用，因为上下文通常可以在理解实际感受方面产生差异。本文提出了一种用于语音情感识别的多尺度会话上下文学习方法。我们研究了这种方法的语音转录和声学段。实验上，我们的方法使用目标段的前一个或下一个信息。在文本域中，我们测试了上下文窗口使用广泛的令牌（从10到100）和在语音轮水平，考虑来自相同和相反的发言人的输入。根据我们的测试，从之前的令牌派生的上下文比以下令牌对准确预测的影响更大。此外，在对话中采取同一发言者的最后一轮发言似乎是有用的。在声学领域，我们进行了深入的分析，周围的情绪对预测的影响。虽然使用Transformers的多尺度会话上下文学习可以增强紧急呼叫记录的文本模态的性能，但结合声学上下文更具挑战性。摘要：Emotion recognition in conversations is essential for ensuring advanced human-machine interactions. However, creating robust and accurate emotion recognition systems in real life is challenging, mainly due to the scarcity of emotion datasets collected in the wild and the inability to take into account the dialogue context. The CEMO dataset, composed of conversations between agents and patients during emergency calls to a French call center, fills this gap. The nature of these interactions highlights the role of the emotional flow of the conversation in predicting patient emotions, as context can often make a difference in understanding actual feelings. This paper presents a multi-scale conversational context learning approach for speech emotion recognition, which takes advantage of this hypothesis. We investigated this approach on both speech transcriptions and acoustic segments. Experimentally, our method uses the previous or next information of the targeted segment. In the text domain, we tested the context window using a wide range of tokens (from 10 to 100) and at the speech turns level, considering inputs from both the same and opposing speakers. According to our tests, the context derived from previous tokens has a more significant influence on accurate prediction than the following tokens. Furthermore, taking the last speech turn of the same speaker in the conversation seems useful. In the acoustic domain, we conducted an in-depth analysis of the impact of the surrounding emotions on the prediction. While multi-scale conversational context learning using Transformers can enhance performance in the textual modality for emergency call recordings, incorporating acoustic context is more challenging.
机器翻译，仅供参考