今日论文合集：cs.SD语音15篇，eess.AS音频处理16篇。
cs.SD语音
【1】 Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard  Parameter Sharing标题：基于硬参数共享的跨通道多任务语音到文本翻译链接：https://arxiv.org/abs/2309.15826
作者：Brian Yan,Xuankai Chang,Antonios Anastasopoulos,Yuya Fujita,Shinji Watanabe摘要：最近在端到端语音到文本翻译（ST）中的工作已经提出了具有软参数共享的多任务方法，其经由将文本输入映射到最终的跨模态表示的次级编码器来利用机器翻译（MT）数据。在这项工作中，我们提出了一个ST/MT多任务框架，其中所有的模型参数共享硬参数跨模态。我们的方法通过预处理阶段将语音和文本输入转换为两个相似长度的离散令牌序列来减少语音-文本模态差距-这允许模型简单地使用联合词汇表不加区别地处理这两种模态。通过在MuST-C上的实验，我们证明了我们的多任务框架在没有任何外部MT数据的情况下，平均提高了+0.5 BLEU的注意力编码器-解码器、联结主义时间分类（CTC）、换能器和联合CTC/注意力模型。此外，我们还表明，该框架结合了外部MT数据，产生了+0.8 BLEU，并且还改进了预训练文本模型的迁移学习，产生了+1.8 BLEU。摘要：Recent works in end-to-end speech-to-text translation (ST) have proposed multi-tasking methods with soft parameter sharing which leverage machine translation (MT) data via secondary encoders that map text inputs to an eventual cross-modal representation. In this work, we instead propose a ST/MT multi-tasking framework with hard parameter sharing in which all model parameters are shared cross-modally. Our method reduces the speech-text modality gap via a pre-processing stage which converts speech and text inputs into two discrete token sequences of similar length -- this allows models to indiscriminately process both modalities simply using a joint vocabulary. With experiments on MuST-C, we demonstrate that our multi-tasking framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU without any external MT data. Further, we show that this framework incorporates external MT data, yielding +0.8 BLEU, and also improves transfer learning from pre-trained textual models, yielding +1.8 BLEU.
【2】 Exploring Speech Recognition, Translation, and Understanding with  Discrete Speech Units: A Comparative Study标题：用离散的语音单元探索语音识别、翻译和理解：一项比较研究链接：https://arxiv.org/abs/2309.15800
作者：Xuankai Chang,Brian Yan,Kwanghee Choi,Jeeweon Jung,Yichen Lu,Soumi Maiti,Roshan Sharma,Jiatong Shi,Jinchuan Tian,Shinji Watanabe,Yuya Fujita,Takashi Maekaku,Pengcheng Guo,Yao-Fei Cheng,Pavel Denisov,Kohei Saijo,Hsiu-Hsuan Wang备注：Submitted to IEEE ICASSP 2024摘要：通常以每秒数万的速率采样的语音信号包含冗余，引起序列建模中的低效率。高维语音特征，如频谱图，通常被用作后续模型的输入。然而，它们仍然可以是冗余的。最近的研究提出了使用离散的语音单元来自自监督学习表示，这显着压缩语音数据的大小。应用各种方法，例如去重和子字建模，可以进一步压缩语音序列长度。因此，训练时间显著减少，同时保持显著的性能。在这项研究中，我们进行了全面和系统的探索到应用程序的离散单元内的端到端的语音处理模型。在12个自动语音识别、3个语音翻译和1个口语理解语料库上的实验表明，离散单元在几乎所有的设置中都取得了相当好的结果。我们打算发布我们的配置和训练模型，以促进未来的研究工作。摘要：Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts.
【3】 HyPoradise: An Open Baseline for Generative Speech Recognition with  Large Language Models标题：HyPoradise：一种开放的大语言模型生成式语音识别基准链接：https://arxiv.org/abs/2309.15701
作者：Chen Chen,Yuchen Hu,Chao-Han Huck Yang,Sabato Macro Siniscalchi,Pin-Yu Chen,Eng Siong Chng备注：Accepted to NeurIPS 2023, 24 pages. Datasets and Benchmarks Track摘要：深度神经网络的进步已经允许自动语音识别（ASR）系统在几个公开可用的干净语音数据集上实现人类平等。然而，即使是最先进的ASR系统在遇到不利条件时也会经历性能下降，因为经过良好训练的声学模型对语音域中的变化敏感，例如，背景噪音直观地说，人类通过依赖他们的语言知识来解决这个问题：模糊的口语术语的含义通常从上下文线索推断，从而减少对听觉系统的依赖性。受此观察的启发，我们引入了第一个利用外部大型语言模型（LLM）进行ASR纠错的开源基准测试，其中N-最佳解码假设为真实的转录预测提供了信息元素。这种方法是从传统的语言模型重评分策略，只能选择一个候选假设作为输出转录的范式转变。建议的基准包含一个新的数据集，HyPorcycle（HP），包括超过334，000对的N-最好的假设和相应的准确转录在流行的语音域。鉴于这个数据集，我们研究了三种类型的纠错技术的基础上LLM与不同数量的标记的假设转录对，获得了显着的字错误率（WER）降低。实验证明，该技术实现了突破，超越了传统的重新排名的方法的上限。更令人惊讶的是，具有合理提示和生成能力的LLM甚至可以纠正N-best列表中缺失的那些令牌。我们通过发布的预训练模型将我们的结果公开提供给可重复的管道，从而为使用LLM进行ASR纠错提供了一种新的评估范式。摘要：Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-trained acoustic model is sensitive to variations in the speech domain, e.g., background noise. Intuitively, humans address this issue by relying on their linguistic knowledge: the meaning of ambiguous spoken terms is usually inferred from contextual cues thereby reducing the dependency on the auditory system. Inspired by this observation, we introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction, where N-best decoding hypotheses provide informative elements for true transcription prediction. This approach is a paradigm shift from the traditional language model rescoring strategy that can only select one candidate hypothesis as the output transcription. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses and corresponding accurate transcriptions across prevalent speech domains. Given this dataset, we examine three types of error correction techniques based on LLMs with varying amounts of labeled hypotheses-transcription pairs, which gains a significant word error rate (WER) reduction. Experimental evidence demonstrates the proposed technique achieves a breakthrough by surpassing the upper bound of traditional re-ranking based methods. More surprisingly, LLM with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list. We make our results publicly accessible for reproducible pipelines with released pre-trained models, thus providing a new evaluation paradigm for ASR error correction with LLMs.
【4】 Enhancing End-to-End Conversational Speech Translation Through Target  Language Context Utilization标题：利用目的语语境加强端到端会话语音翻译链接：https://arxiv.org/abs/2309.15686
作者：Amir Hussein,Brian Yan,Antonios Anastasopoulos,Shinji Watanabe,Sanjeev Khudanpur摘要：结合较长的上下文已被证明有利于机器翻译，但在端到端语音翻译（E2 E-ST）中包含上下文仍然研究不足。为了弥补这一差距，我们在E2 E-ST中引入目标语上下文，增强连贯性并克服扩展音频片段的记忆限制。此外，我们提出了上下文丢弃，以确保鲁棒性的情况下，上下文，并通过添加说话人信息进一步提高性能。我们提出的上下文E2 E-ST优于孤立的话语为基础的E2 E-ST的方法。最后，我们表明，在会话语音，上下文信息主要有助于捕捉上下文风格，以及解决照应和命名实体。摘要：Incorporating longer context has been shown to benefit machine translation, but the inclusion of context in end-to-end speech translation (E2E-ST) remains under-studied. To bridge this gap, we introduce target language context in E2E-ST, enhancing coherence and overcoming memory constraints of extended audio segments. Additionally, we propose context dropout to ensure robustness to the absence of context, and further improve performance by adding speaker information. Our proposed contextual E2E-ST outperforms the isolated utterance-based E2E-ST approach. Lastly, we demonstrate that in conversational speech, contextual information primarily contributes to capturing context style, as well as resolving anaphora and named entities.
【5】 Speech collage: code-switched audio generation by collaging monolingual  corpora标题：语音拼贴：通过拼贴单语语料库生成代码转换音频链接：https://arxiv.org/abs/2309.15674
作者：Amir Hussein,Dorsa Zeinali,Ondřej Klejch,Matthew Wiesner,Brian Yan,Shammur Chowdhury,Ahmed Ali,Shinji Watanabe,Sanjeev Khudanpur摘要：设计有效的自动语音识别（ASR）系统的代码转换（CS）往往取决于转录CS资源的可用性。为了解决数据短缺的问题，本文介绍了语音拼贴，一种方法，合成CS数据从单语语料库的拼接音频片段。我们使用重叠相加方法进一步提高音频生成的平滑质量。我们研究了两种情况下生成的数据对语音识别的影响：使用域内CS文本和具有合成CS文本的zero-shot方法。实证结果突出了高达34.4%和16.2%的相对减少混合错误率和字错误率的域和zero-shot的情况下，分别。最后，我们证明CS增强支持模型的代码转换倾向，并减少其单语偏见。摘要：Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zero-shot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model's code-switching inclination and reduces its monolingual bias.
【6】 Generative Speech Recognition Error Correction with Large Language  Models标题：基于大语言模型的产生式语音识别纠错链接：https://arxiv.org/abs/2309.15649
作者：Chao-Han Huck Yang,Yile Gu,Yi-Chieh Liu,Shalini Ghosh,Ivan Bulyko,Andreas Stolcke备注：Accepted to IEEE Automatic Speech Recognition and Understanding (ASRU) 2023摘要：我们探讨了大型语言模型（LLM）作为ASR后处理器执行重新评分和纠错的能力。我们的重点是指令提示，让LLM执行这些任务，而无需微调，我们评估不同的提示方案，零和Few-Shot的上下文学习，和一个新的任务激活提示（TAP）的方法，结合指令和示范。使用预训练的第一遍系统和两个域外任务（ATIS和WSJ）的重新评分输出，我们表明，重新评分仅通过在上下文中学习与冻结LLM实现的结果是有竞争力的重新评分域调整的LM。通过将提示技术与微调相结合，我们实现了低于N-最佳Oracle级别的错误率，展示了LLM的泛化能力。摘要：We explore the ability of large language models (LLMs) to act as ASR post-processors that perform rescoring and error correction. Our focus is on instruction prompting to let LLMs perform these task without fine-tuning, for which we evaluate different prompting schemes, both zero- and few-shot in-context learning, and a novel task-activating prompting (TAP) method that combines instruction and demonstration. Using a pre-trained first-pass system and rescoring output on two out-of-domain tasks (ATIS and WSJ), we show that rescoring only by in-context learning with frozen LLMs achieves results that are competitive with rescoring by domain-tuned LMs. By combining prompting techniques with fine-tuning we achieve error rates below the N-best oracle level, showcasing the generalization power of the LLMs.
【7】 Direct Models for Simultaneous Translation and Automatic Subtitling:  FBK@IWSLT2023标题：同声翻译和自动字幕的直接模型：FBK@IWSLT2023链接：https://arxiv.org/abs/2309.15554
作者：Sara Papi,Marco Gaido,Matteo Negri备注：None摘要：本文介绍了FBK参与IWSLT 2023评估活动的同声翻译和自动字幕跟踪。我们的提交集中在使用直接架构来执行这两项任务：对于同时推理，我们利用离线训练模型已经获得的知识，并直接应用策略来获得实时推理;对于字幕，我们采用直接ST模型来产生格式良好的字幕，并利用相同的架构来产生字幕与视听内容同步所需的时间戳。我们的英语-德语SimulST系统显示，与2021年和2022年任务中排名靠前的系统相比，计算感知延迟降低，增益高达3.5 BLEU。我们的自动字幕系统优于现有的解决方案的基础上的直接系统的3.7和1.7 SubER在英语-德语和英语-西班牙语分别。摘要：This paper describes the FBK's participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one achieved by the top-ranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively.
【8】 High-Fidelity Speech Synthesis with Minimal Supervision: All Using  Diffusion Models标题：最小监督下的高保真语音合成：全部使用扩散模型链接：https://arxiv.org/abs/2309.15512
作者：Chunyu Qiang,Hao Li,Yixin Tian,Yi Zhao,Ying Zhang,Longbiao Wang,Jianwu Dang备注：arXiv admin note: substantial text overlap with arXiv:2307.15484; text overlap with arXiv:2309.00424摘要：文语转换（TTS）方法在语音克隆方面已经显示出良好的效果，但是它们需要大量的标记的文本-语音对。最小监督语音合成通过组合两种类型的离散语音表示（语义\声学）并使用两个序列到序列任务来实现最小监督的训练来解耦TTS。然而，现有的方法遭受信息冗余和维度爆炸的语义表示，和高频波形失真的离散声学表示。自回归框架表现出典型的不稳定性和不可控性问题。非自回归框架遭受由时长预测模型引起的韵律平均。为了解决这些问题，我们提出了一种最小监督高保真语音合成方法，其中所有模块都是基于扩散模型构建的。非自回归框架增强了可控性，持续时间扩散模型使多样化的韵律表达。针对现有语义编码方法中存在的信息冗余和维数爆炸问题，采用对比令牌声学预训练（CTAP）作为中间语义表示。使用Mel谱图作为声学表示。语义和声学表示都通过连续变量回归任务来预测，以解决高频细粒度波形失真的问题。实验结果表明，我们提出的方法优于基线方法。我们在我们的网站上提供音频样本。摘要：Text-to-speech (TTS) methods have shown promising results in voice cloning, but they require a large number of labeled text-speech pairs. Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations(semantic \& acoustic) and using two sequence-to-sequence tasks to enable training with minimal supervision. However, existing methods suffer from information redundancy and dimension explosion in semantic representation, and high-frequency waveform distortion in discrete acoustic representation. Autoregressive frameworks exhibit typical instability and uncontrollability issues. And non-autoregressive frameworks suffer from prosodic averaging caused by duration prediction models. To address these issues, we propose a minimally-supervised high-fidelity speech synthesis method, where all modules are constructed based on the diffusion models. The non-autoregressive framework enhances controllability, and the duration diffusion model enables diversified prosodic expression. Contrastive Token-Acoustic Pretraining (CTAP) is used as an intermediate semantic representation to solve the problems of information redundancy and dimension explosion in existing semantic coding methods. Mel-spectrogram is used as the acoustic representation. Both semantic and acoustic representations are predicted by continuous variable regression tasks to solve the problem of high-frequency fine-grained waveform distortion. Experimental results show that our proposed method outperforms the baseline method. We provide audio samples on our website.
【9】 joint prediction and denoising for large-scale multilingual  self-supervised learning标题：大规模多语言自监督学习的联合预测与去噪链接：https://arxiv.org/abs/2309.15317
作者：William Chen,Jiatong Shi,Brian Yan,Dan Berrebbi,Wangyou Zhang,Yifan Peng,Xuankai Chang,Soumi Maiti,Shinji Watanabe备注：Accepted to ASRU 2023摘要：由于处理多种语言所需的费用和复杂性，多语言自监督学习（SSL）通常落后于最先进的（SOTA）方法。这进一步损害了SSL的可重复性，由于其资源使用，SSL已经仅限于少数研究小组。我们表明，更强大的技术实际上可以导致更有效的预训练，开放SSL更多的研究小组。我们提出了WavLabLM，它将WavLM的联合预测和去噪扩展到136种语言的4万小时数据。为了构建WavLabLM，我们设计了一种新的多阶段预训练方法，旨在解决多语言数据的语言不平衡问题。WavLabLM在ML-SUPERB上实现了与XLS-R相当的性能，训练数据不到10%，使得SSL可以通过学术计算实现。我们表明，进一步的效率可以实现与香草HuBERT基地模型，它可以保持94%的XLS-R的性能，只有3%的数据，4个GPU，和有限的试验。我们开源了ESPnet中的所有代码和模型。摘要：Multilingual self-supervised learning (SSL) has often lagged behind state-of-the-art (SOTA) methods due to the expenses and complexity required to handle many languages. This further harms the reproducibility of SSL, which is already limited to few research groups due to its resource usage. We show that more powerful techniques can actually lead to more efficient pre-training, opening SSL to more research groups. We propose WavLabLM, which extends WavLM's joint prediction and denoising to 40k hours of data across 136 languages. To build WavLabLM, we devise a novel multi-stage pre-training method, designed to address the language imbalance of multilingual data. WavLabLM achieves comparable performance to XLS-R on ML-SUPERB with less than 10% of the training data, making SSL realizable with academic compute. We show that further efficiency can be achieved with a vanilla HuBERT Base model, which can maintain 94% of XLS-R's performance with only 3% of the data, 4 GPUs, and limited trials. We open-source all code and models in ESPnet.
【10】 Low-rank Adaptation of Large Language Model Rescoring for  Parameter-Efficient Speech Recognition标题：参数高效语音识别中大语言模型研究的低阶自适应链接：https://arxiv.org/abs/2309.15223
作者：Yu Yu,Chao-Han Huck Yang,Jari Kolehmainen,Prashanth G. Shivakumar,Yile Gu,Sungho Ryu,Roger Ren,Qi Luo,Aditya Gourav,I-Fan Chen,Yi-Chieh Liu,Tuan Dinh,Ankur Gandhe,Denis Filimonov,Shalini Ghosh,Andreas Stolcke,Ariya Rastow,Ivan Bulyko备注：Accepted to IEEE ASRU 2023. Internal Review Approved摘要：我们提出了一个神经语言建模系统的基础上低秩自适应（LoRA）的语音识别输出rescoring。虽然像BERT这样的预训练语言模型（LM）在二次评分中表现出了优异的性能，但扩大预训练阶段并使预训练模型适应特定领域的高计算成本限制了它们在重新评分中的实际使用。在这里，我们提出了一种基于低秩分解的方法来训练重新评分BERT模型，并仅使用一小部分（0.08%）的预训练参数将其适应新的领域。这些插入的矩阵通过区分性训练目标以及基于相关性的正则化损失来优化。所提出的低秩自适应Rescore-BERT（LoRB）架构在LibriSpeech和内部数据集上进行评估，训练时间减少了5.4到3.6倍。摘要：We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.
【11】 Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music  Transcription标题：Timbre-Trap：一种乐器无关音乐改编的低资源框架链接：https://arxiv.org/abs/2309.15717
作者：Frank Cwitkowitz,Kin Wai Cheuk,Woosung Choi,Marco A. Martínez-Ramírez,Keisuke Toyama,Wei-Hsiang Liao,Yuki Mitsufuji备注：Submitted to ICASSP 2024摘要：近年来，对音乐转录的研究主要集中在架构设计和特定乐器的数据采集。由于缺乏多样的数据集，进展往往局限于独奏乐器任务，如钢琴转录。一些作品已经探索了多仪器转录作为一种手段，以加强低资源任务的模型的性能，但这些方法面临着相同的数据可用性问题。我们提出了Timbre-Trap，一个新的框架，它统一了音乐转录和音频重建，利用音高和音色之间的强分离性。我们训练一个单一的U-网络，同时估计音高显着性和重建复频谱系数，通过一个简单的开关机制在解码阶段的输出之间进行选择。通过这种方式，模型学习产生对应于无音色音频的系数，这可以被解释为音高显著性。我们证明，该框架导致的性能相当于国家的最先进的仪器不可知的转录方法，而只需要少量的注释数据。摘要：In recent years, research on music transcription has focused mainly on architecture design and instrument-specific data acquisition. With the lack of availability of diverse datasets, progress is often limited to solo-instrument tasks such as piano transcription. Several works have explored multi-instrument transcription as a means to bolster the performance of models on low-resource tasks, but these methods face the same data availability issues. We propose Timbre-Trap, a novel framework which unifies music transcription and audio reconstruction by exploiting the strong separability between pitch and timbre. We train a single U-Net to simultaneously estimate pitch salience and reconstruct complex spectral coefficients, selecting between either output during the decoding stage via a simple switch mechanism. In this way, the model learns to produce coefficients corresponding to timbre-less audio, which can be interpreted as pitch salience. We demonstrate that the framework leads to performance comparable to state-of-the-art instrument-agnostic transcription methods, while only requiring a small amount of annotated data.
【12】 Why do Angular Margin Losses work well for Semi-Supervised Anomalous  Sound Detection?标题：为什么角余量损失对半监督异常声音检测效果很好？链接：https://arxiv.org/abs/2309.15643
作者：Kevin Wilkinghoff,Frank Kurth摘要：现有技术的异常声音检测系统通常利用角度裕度损失来使用辅助任务学习声学数据的合适表示，该辅助任务通常是监督或自监督分类任务。其基本思想是，为了解决这个辅助任务，需要在学习的表示中捕获关于正常数据的特定信息，并且该信息也足以区分正常和异常样本。特别是在噪声条件下，基于角度裕度损失的判别模型往往显著优于基于生成或一类模型的系统。这项工作的目标是调查为什么使用辅助任务的角度裕度损失以及检测异常的声音。为此，它示出，无论是理论上和实验上，最大限度地减少角裕度损失也最大限度地减少紧凑性损失，同时固有地防止学习平凡的解决方案。此外，进行多个实验表明，使用相关的分类任务作为辅助任务教导模型学习表示适合于在嘈杂的条件下检测异常的声音。在这些实验中的性能评估，可视化的嵌入空间与t-SNE和可视化的输入表示相对于异常分数使用随机输入采样的解释。摘要：State-of-the-art anomalous sound detection systems often utilize angular margin losses to learn suitable representations of acoustic data using an auxiliary task, which usually is a supervised or self-supervised classification task. The underlying idea is that, in order to solve this auxiliary task, specific information about normal data needs to be captured in the learned representations and that this information is also sufficient to differentiate between normal and anomalous samples. Especially in noisy conditions, discriminative models based on angular margin losses tend to significantly outperform systems based on generative or one-class models. The goal of this work is to investigate why using angular margin losses with auxiliary tasks works well for detecting anomalous sounds. To this end, it is shown, both theoretically and experimentally, that minimizing angular margin losses also minimizes compactness loss while inherently preventing learning trivial solutions. Furthermore, multiple experiments are conducted to show that using a related classification task as an auxiliary task teaches the model to learn representations suitable for detecting anomalous sounds in noisy conditions. Among these experiments are performance evaluations, visualizing the embedding space with t-SNE and visualizing the input representations with respect to the anomaly score using randomized input sampling for explanation.
【13】 DualVC 2: Dynamic Masked Convolution for Unified Streaming and  Non-Streaming Voice Conversion标题：DualVC 2：用于统一流和非流语音转换的动态掩码卷积链接：https://arxiv.org/abs/2309.15496
作者：Ziqian Ning,Yuepeng Jiang,Pengcheng Zhu,Shuai Wang,Jixun Yao,Lei Xie,Mengxiao Bi摘要：语音转换越来越流行，越来越多的应用场景需要具有流式推理能力的模型。最近提出的DualVC试图通过流模型架构设计和模型内知识蒸馏以及混合预测编码来补偿未来信息的缺乏来实现这一目标。然而，DualVC遇到了几个限制其性能的问题。首先，自回归解码器本质上具有误差累积，并且也限制了推理速度。第二，因果卷积实现了流式传输能力，但不能充分使用块内的未来信息。第三，该模型无法有效地解决清音段中的噪声，降低了声音质量。在本文中，我们提出了DualVC 2来解决这些问题。具体来说，模型主干迁移到基于Conformer的架构，使并行推理。因果卷积由具有动态块掩码的非因果卷积代替，以更好地利用块内未来信息。此外，安静的注意力被引入，以提高模型的噪声鲁棒性。实验表明，DualVC 2在主观和客观指标上都优于DualVC和其他基线系统，延迟仅为186.4 ms。我们的音频样本是公开的。摘要：Voice conversion is becoming increasingly popular, and a growing number of application scenarios require models with streaming inference capabilities. The recently proposed DualVC attempts to achieve this objective through streaming model architecture design and intra-model knowledge distillation along with hybrid predictive coding to compensate for the lack of future information. However, DualVC encounters several problems that limit its performance. First, the autoregressive decoder has error accumulation in its nature and limits the inference speed as well. Second, the causal convolution enables streaming capability but cannot sufficiently use future information within chunks. Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality. In this paper, we propose DualVC 2 to address these issues. Specifically, the model backbone is migrated to a Conformer-based architecture, empowering parallel inference. Causal convolution is replaced by non-causal convolution with dynamic chunk mask to make better use of within-chunk future information. Also, quiet attention is introduced to enhance the model's noise robustness. Experiments show that DualVC 2 outperforms DualVC and other baseline systems in both subjective and objective metrics, with only 186.4 ms latency. Our audio samples are made publicly available.
【14】 Collaborative Watermarking for Adversarial Speech Synthesis标题：用于对抗性语音合成的协作水印链接：https://arxiv.org/abs/2309.15224
作者：Lauri Juvela,Xin Wang备注：Submitted to ICASSP 2024摘要：神经语音合成的进步为我们带来了不仅接近人类自然性的技术，而且能够用很少的数据进行即时语音克隆，并且可以通过预先训练的模型进行高度访问。自然地，所生成的内容的潜在泛滥提高了对合成语音检测和水印的需要。最近，合成语音检测中的相当大的研究工作已经涉及到自动说话人验证和欺骗对策挑战（ASVspoof），其重点是被动对策。本文对生成的语音检测采取补充观点：合成系统应当积极努力，以帮助另一机器检测但对人类收听者保持透明的方式，对所生成的语音加水印。我们提出了一种用于合成语音水印的协作训练方案，并表明与ASVspoof 2021基线对策模型合作的HiFi-GAN神经声码器始终提高了传统分类器训练的检测性能。此外，我们还展示了如何将协作训练与增强策略配对，以增加对噪声和时间拉伸的鲁棒性。最后，听力测试表明，协同训练声码语音的感知质量几乎没有不利影响。摘要：Advances in neural speech synthesis have brought us technology that is not only close to human naturalness, but is also capable of instant voice cloning with little data, and is highly accessible with pre-trained models available. Naturally, the potential flood of generated content raises the need for synthetic speech detection and watermarking. Recently, considerable research effort in synthetic speech detection has been related to the Automatic Speaker Verification and Spoofing Countermeasure Challenge (ASVspoof), which focuses on passive countermeasures. This paper takes a complementary view to generated speech detection: a synthesis system should make an active effort to watermark the generated speech in a way that aids detection by another machine, but remains transparent to a human listener. We propose a collaborative training scheme for synthetic speech watermarking and show that a HiFi-GAN neural vocoder collaborating with the ASVspoof 2021 baseline countermeasure models consistently improves detection performance over conventional classifier training. Furthermore, we demonstrate how collaborative training can be paired with augmentation strategies for added robustness against noise and time-stretching. Finally, listening tests demonstrate that collaborative training has little adverse effect on perceptual quality of vocoded speech.
【15】 A multi-modal approach for identifying schizophrenia using cross-modal  attention标题：利用跨通道注意识别精神分裂症的多通道方法链接：https://arxiv.org/abs/2309.15136
作者：Gowtham Premananth,Yashish M. Siriwardena,Philip Resnik,Carol Espy-Wilson摘要：本研究的重点是如何不同的人际交往方式可以用来区分健康对照组和精神分裂症患者谁表现出强烈的阳性症状。我们开发了一个多模态精神分裂症分类系统，使用音频，视频和文本。分别从视频和音频中提取面部动作单元和声道变量作为低级特征，然后用于计算高级协调特征，作为音频和视频模态的输入。从语音转录中提取的上下文无关的文本嵌入被用作文本模态的输入。多模态系统是通过融合视频和音频模态的段到会话级分类器与基于具有跨模态注意力的分层注意力网络（HAN）的文本模型来开发的。建议的多模态系统优于以前的国家的最先进的多模态系统的加权平均F1得分的8.53%。摘要：This study focuses on how different modalities of human communication can be used to distinguish between healthy controls and subjects with schizophrenia who exhibit strong positive symptoms. We developed a multi-modal schizophrenia classification system using audio, video, and text. Facial action units and vocal tract variables were extracted as low-level features from video and audio respectively, which were then used to compute high-level coordination features that served as the inputs to the audio and video modalities. Context-independent text embeddings extracted from transcriptions of speech were used as the input for the text modality. The multi-modal system is developed by fusing a segment-to-session-level classifier for video and audio modalities with a text model based on a Hierarchical Attention Network (HAN) with cross-modal attention. The proposed multi-modal system outperforms the previous state-of-the-art multi-modal system by 8.53% in the weighted average F1 score.
eess.AS音频处理
【1】 Learning from Flawed Data: Weakly Supervised Automatic Speech  Recognition标题：从有缺陷的数据中学习：弱监督自动语音识别链接：https://arxiv.org/abs/2309.15796
作者：Dongji Gao,Hainan Xu,Desh Raj,Leibny Paola Garcia Perera,Daniel Povey,Sanjeev Khudanpur摘要：训练自动语音识别（ASR）系统需要大量精心策划的配对数据。然而，人类注释者通常执行“非逐字”转录，这可能导致训练不佳的模型。在本文中，我们提出了全时分类（OTC），一种新的训练标准，明确纳入标签的不确定性，源于这样的弱监管。这使得模型能够有效地学习语音-文本对齐，同时适应训练成绩单中存在的错误。OTC通过利用加权有限状态换能器扩展了不完美转录物的常规CTC目标。通过在LibriSpeech和LibriVox数据集上进行的实验，我们证明了使用OTC训练ASR模型可以避免性能下降，即使转录本包含高达70%的错误，这是CTC模型完全失败的情况。我们的实施可在https://github.com/k2-fsa/icefall上获得。摘要：Training automatic speech recognition (ASR) systems requires large amounts of well-curated paired data. However, human annotators usually perform "non-verbatim" transcription, which can result in poorly trained models. In this paper, we propose Omni-temporal Classification (OTC), a novel training criterion that explicitly incorporates label uncertainties originating from such weak supervision. This allows the model to effectively learn speech-text alignments while accommodating errors present in the training transcripts. OTC extends the conventional CTC objective for imperfect transcripts by leveraging weighted finite state transducers. Through experiments conducted on the LibriSpeech and LibriVox datasets, we demonstrate that training ASR models with OTC avoids performance degradation even with transcripts containing up to 70% errors, a scenario where CTC models fail completely. Our implementation is available at https://github.com/k2-fsa/icefall.
【2】 Timbre-Trap: A Low-Resource Framework for Instrument-Agnostic Music  Transcription标题：Timbre-Trap：一种乐器无关音乐改编的低资源框架链接：https://arxiv.org/abs/2309.15717
作者：Frank Cwitkowitz,Kin Wai Cheuk,Woosung Choi,Marco A. Martínez-Ramírez,Keisuke Toyama,Wei-Hsiang Liao,Yuki Mitsufuji备注：Submitted to ICASSP 2024摘要：近年来，对音乐转录的研究主要集中在架构设计和特定乐器的数据采集。由于缺乏多样的数据集，进展往往局限于独奏乐器任务，如钢琴转录。一些作品已经探索了多仪器转录作为一种手段，以加强低资源任务的模型的性能，但这些方法面临着相同的数据可用性问题。我们提出了Timbre-Trap，一个新的框架，它统一了音乐转录和音频重建，利用音高和音色之间的强分离性。我们训练一个单一的U-网络，同时估计音高显着性和重建复频谱系数，通过一个简单的开关机制在解码阶段的输出之间进行选择。通过这种方式，模型学习产生对应于无音色音频的系数，这可以被解释为音高显著性。我们证明，该框架导致的性能相当于国家的最先进的仪器不可知的转录方法，而只需要少量的注释数据。摘要：In recent years, research on music transcription has focused mainly on architecture design and instrument-specific data acquisition. With the lack of availability of diverse datasets, progress is often limited to solo-instrument tasks such as piano transcription. Several works have explored multi-instrument transcription as a means to bolster the performance of models on low-resource tasks, but these methods face the same data availability issues. We propose Timbre-Trap, a novel framework which unifies music transcription and audio reconstruction by exploiting the strong separability between pitch and timbre. We train a single U-Net to simultaneously estimate pitch salience and reconstruct complex spectral coefficients, selecting between either output during the decoding stage via a simple switch mechanism. In this way, the model learns to produce coefficients corresponding to timbre-less audio, which can be interpreted as pitch salience. We demonstrate that the framework leads to performance comparable to state-of-the-art instrument-agnostic transcription methods, while only requiring a small amount of annotated data.
【3】 Why do Angular Margin Losses work well for Semi-Supervised Anomalous  Sound Detection?标题：为什么角余量损失对半监督异常声音检测效果很好？链接：https://arxiv.org/abs/2309.15643
作者：Kevin Wilkinghoff,Frank Kurth摘要：现有技术的异常声音检测系统通常利用角度裕度损失来使用辅助任务学习声学数据的合适表示，该辅助任务通常是监督或自监督分类任务。其基本思想是，为了解决这个辅助任务，需要在学习的表示中捕获关于正常数据的特定信息，并且该信息也足以区分正常和异常样本。特别是在噪声条件下，基于角度裕度损失的判别模型往往显著优于基于生成或一类模型的系统。这项工作的目标是调查为什么使用辅助任务的角度裕度损失以及检测异常的声音。为此，它示出，无论是理论上和实验上，最大限度地减少角裕度损失也最大限度地减少紧凑性损失，同时固有地防止学习平凡的解决方案。此外，进行多个实验表明，使用相关的分类任务作为辅助任务教导模型学习表示适合于在嘈杂的条件下检测异常的声音。在这些实验中的性能评估，可视化的嵌入空间与t-SNE和可视化的输入表示相对于异常分数使用随机输入采样的解释。摘要：State-of-the-art anomalous sound detection systems often utilize angular margin losses to learn suitable representations of acoustic data using an auxiliary task, which usually is a supervised or self-supervised classification task. The underlying idea is that, in order to solve this auxiliary task, specific information about normal data needs to be captured in the learned representations and that this information is also sufficient to differentiate between normal and anomalous samples. Especially in noisy conditions, discriminative models based on angular margin losses tend to significantly outperform systems based on generative or one-class models. The goal of this work is to investigate why using angular margin losses with auxiliary tasks works well for detecting anomalous sounds. To this end, it is shown, both theoretically and experimentally, that minimizing angular margin losses also minimizes compactness loss while inherently preventing learning trivial solutions. Furthermore, multiple experiments are conducted to show that using a related classification task as an auxiliary task teaches the model to learn representations suitable for detecting anomalous sounds in noisy conditions. Among these experiments are performance evaluations, visualizing the embedding space with t-SNE and visualizing the input representations with respect to the anomaly score using randomized input sampling for explanation.
【4】 DualVC 2: Dynamic Masked Convolution for Unified Streaming and  Non-Streaming Voice Conversion标题：DualVC 2：用于统一流和非流语音转换的动态掩码卷积链接：https://arxiv.org/abs/2309.15496
作者：Ziqian Ning,Yuepeng Jiang,Pengcheng Zhu,Shuai Wang,Jixun Yao,Lei Xie,Mengxiao Bi摘要：语音转换越来越流行，越来越多的应用场景需要具有流式推理能力的模型。最近提出的DualVC试图通过流模型架构设计和模型内知识蒸馏以及混合预测编码来补偿未来信息的缺乏来实现这一目标。然而，DualVC遇到了几个限制其性能的问题。首先，自回归解码器本质上具有误差累积，并且也限制了推理速度。第二，因果卷积实现了流式传输能力，但不能充分使用块内的未来信息。第三，该模型无法有效地解决清音段中的噪声，降低了声音质量。在本文中，我们提出了DualVC 2来解决这些问题。具体来说，模型主干迁移到基于Conformer的架构，使并行推理。因果卷积由具有动态块掩码的非因果卷积代替，以更好地利用块内未来信息。此外，安静的注意力被引入，以提高模型的噪声鲁棒性。实验表明，DualVC 2在主观和客观指标上都优于DualVC和其他基线系统，延迟仅为186.4 ms。我们的音频样本是公开的。摘要：Voice conversion is becoming increasingly popular, and a growing number of application scenarios require models with streaming inference capabilities. The recently proposed DualVC attempts to achieve this objective through streaming model architecture design and intra-model knowledge distillation along with hybrid predictive coding to compensate for the lack of future information. However, DualVC encounters several problems that limit its performance. First, the autoregressive decoder has error accumulation in its nature and limits the inference speed as well. Second, the causal convolution enables streaming capability but cannot sufficiently use future information within chunks. Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality. In this paper, we propose DualVC 2 to address these issues. Specifically, the model backbone is migrated to a Conformer-based architecture, empowering parallel inference. Causal convolution is replaced by non-causal convolution with dynamic chunk mask to make better use of within-chunk future information. Also, quiet attention is introduced to enhance the model's noise robustness. Experiments show that DualVC 2 outperforms DualVC and other baseline systems in both subjective and objective metrics, with only 186.4 ms latency. Our audio samples are made publicly available.
【5】 Collaborative Watermarking for Adversarial Speech Synthesis标题：用于对抗性语音合成的协作水印链接：https://arxiv.org/abs/2309.15224
作者：Lauri Juvela,Xin Wang备注：Submitted to ICASSP 2024摘要：神经语音合成的进步为我们带来了不仅接近人类自然性的技术，而且能够用很少的数据进行即时语音克隆，并且可以通过预先训练的模型进行高度访问。自然地，所生成的内容的潜在泛滥提高了对合成语音检测和水印的需要。最近，合成语音检测中的相当大的研究工作已经涉及到自动说话人验证和欺骗对策挑战（ASVspoof），其重点是被动对策。本文对生成的语音检测采取补充观点：合成系统应当积极努力，以帮助另一机器检测但对人类收听者保持透明的方式，对所生成的语音加水印。我们提出了一种用于合成语音水印的协作训练方案，并表明与ASVspoof 2021基线对策模型合作的HiFi-GAN神经声码器始终提高了传统分类器训练的检测性能。此外，我们还展示了如何将协作训练与增强策略配对，以增加对噪声和时间拉伸的鲁棒性。最后，听力测试表明，协同训练声码语音的感知质量几乎没有不利影响。摘要：Advances in neural speech synthesis have brought us technology that is not only close to human naturalness, but is also capable of instant voice cloning with little data, and is highly accessible with pre-trained models available. Naturally, the potential flood of generated content raises the need for synthetic speech detection and watermarking. Recently, considerable research effort in synthetic speech detection has been related to the Automatic Speaker Verification and Spoofing Countermeasure Challenge (ASVspoof), which focuses on passive countermeasures. This paper takes a complementary view to generated speech detection: a synthesis system should make an active effort to watermark the generated speech in a way that aids detection by another machine, but remains transparent to a human listener. We propose a collaborative training scheme for synthetic speech watermarking and show that a HiFi-GAN neural vocoder collaborating with the ASVspoof 2021 baseline countermeasure models consistently improves detection performance over conventional classifier training. Furthermore, we demonstrate how collaborative training can be paired with augmentation strategies for added robustness against noise and time-stretching. Finally, listening tests demonstrate that collaborative training has little adverse effect on perceptual quality of vocoded speech.
【6】 A multi-modal approach for identifying schizophrenia using cross-modal  attention标题：利用跨通道注意识别精神分裂症的多通道方法链接：https://arxiv.org/abs/2309.15136
作者：Gowtham Premananth,Yashish M. Siriwardena,Philip Resnik,Carol Espy-Wilson摘要：本研究的重点是如何不同的人际交往方式可以用来区分健康对照组和精神分裂症患者谁表现出强烈的阳性症状。我们开发了一个多模态精神分裂症分类系统，使用音频，视频和文本。分别从视频和音频中提取面部动作单元和声道变量作为低级特征，然后用于计算高级协调特征，作为音频和视频模态的输入。从语音转录中提取的上下文无关的文本嵌入被用作文本模态的输入。多模态系统是通过融合视频和音频模态的段到会话级分类器与基于具有跨模态注意力的分层注意力网络（HAN）的文本模型来开发的。建议的多模态系统优于以前的国家的最先进的多模态系统的加权平均F1得分的8.53%。摘要：This study focuses on how different modalities of human communication can be used to distinguish between healthy controls and subjects with schizophrenia who exhibit strong positive symptoms. We developed a multi-modal schizophrenia classification system using audio, video, and text. Facial action units and vocal tract variables were extracted as low-level features from video and audio respectively, which were then used to compute high-level coordination features that served as the inputs to the audio and video modalities. Context-independent text embeddings extracted from transcriptions of speech were used as the input for the text modality. The multi-modal system is developed by fusing a segment-to-session-level classifier for video and audio modalities with a text model based on a Hierarchical Attention Network (HAN) with cross-modal attention. The proposed multi-modal system outperforms the previous state-of-the-art multi-modal system by 8.53% in the weighted average F1 score.
【7】 Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard  Parameter Sharing标题：基于硬参数共享的跨通道多任务语音到文本翻译链接：https://arxiv.org/abs/2309.15826
作者：Brian Yan,Xuankai Chang,Antonios Anastasopoulos,Yuya Fujita,Shinji Watanabe摘要：最近在端到端语音到文本翻译（ST）中的工作已经提出了具有软参数共享的多任务方法，其经由将文本输入映射到最终的跨模态表示的次级编码器来利用机器翻译（MT）数据。在这项工作中，我们提出了一个ST/MT多任务框架，其中所有的模型参数共享硬参数跨模态。我们的方法通过预处理阶段将语音和文本输入转换为两个相似长度的离散令牌序列来减少语音-文本模态差距-这允许模型简单地使用联合词汇表不加区别地处理这两种模态。通过在MuST-C上的实验，我们证明了我们的多任务框架在没有任何外部MT数据的情况下，平均提高了+0.5 BLEU的注意力编码器-解码器、联结主义时间分类（CTC）、换能器和联合CTC/注意力模型。此外，我们还表明，该框架结合了外部MT数据，产生了+0.8 BLEU，并且还改进了预训练文本模型的迁移学习，产生了+1.8 BLEU。摘要：Recent works in end-to-end speech-to-text translation (ST) have proposed multi-tasking methods with soft parameter sharing which leverage machine translation (MT) data via secondary encoders that map text inputs to an eventual cross-modal representation. In this work, we instead propose a ST/MT multi-tasking framework with hard parameter sharing in which all model parameters are shared cross-modally. Our method reduces the speech-text modality gap via a pre-processing stage which converts speech and text inputs into two discrete token sequences of similar length -- this allows models to indiscriminately process both modalities simply using a joint vocabulary. With experiments on MuST-C, we demonstrate that our multi-tasking framework improves attentional encoder-decoder, Connectionist Temporal Classification (CTC), transducer, and joint CTC/attention models by an average of +0.5 BLEU without any external MT data. Further, we show that this framework incorporates external MT data, yielding +0.8 BLEU, and also improves transfer learning from pre-trained textual models, yielding +1.8 BLEU.
【8】 Exploring Speech Recognition, Translation, and Understanding with  Discrete Speech Units: A Comparative Study标题：用离散的语音单元探索语音识别、翻译和理解：一项比较研究链接：https://arxiv.org/abs/2309.15800
作者：Xuankai Chang,Brian Yan,Kwanghee Choi,Jeeweon Jung,Yichen Lu,Soumi Maiti,Roshan Sharma,Jiatong Shi,Jinchuan Tian,Shinji Watanabe,Yuya Fujita,Takashi Maekaku,Pengcheng Guo,Yao-Fei Cheng,Pavel Denisov,Kohei Saijo,Hsiu-Hsuan Wang备注：Submitted to IEEE ICASSP 2024摘要：通常以每秒数万的速率采样的语音信号包含冗余，引起序列建模中的低效率。高维语音特征，如频谱图，通常被用作后续模型的输入。然而，它们仍然可以是冗余的。最近的研究提出了使用离散的语音单元来自自监督学习表示，这显着压缩语音数据的大小。应用各种方法，例如去重和子字建模，可以进一步压缩语音序列长度。因此，训练时间显著减少，同时保持显著的性能。在这项研究中，我们进行了全面和系统的探索到应用程序的离散单元内的端到端的语音处理模型。在12个自动语音识别、3个语音翻译和1个口语理解语料库上的实验表明，离散单元在几乎所有的设置中都取得了相当好的结果。我们打算发布我们的配置和训练模型，以促进未来的研究工作。摘要：Speech signals, typically sampled at rates in the tens of thousands per second, contain redundancies, evoking inefficiencies in sequence modeling. High-dimensional speech features such as spectrograms are often used as the input for the subsequent model. However, they can still be redundant. Recent investigations proposed the use of discrete speech units derived from self-supervised learning representations, which significantly compresses the size of speech data. Applying various methods, such as de-duplication and subword modeling, can further compress the speech sequence length. Hence, training time is significantly reduced while retaining notable performance. In this study, we undertake a comprehensive and systematic exploration into the application of discrete units within end-to-end speech processing models. Experiments on 12 automatic speech recognition, 3 speech translation, and 1 spoken language understanding corpora demonstrate that discrete units achieve reasonably good results in almost all the settings. We intend to release our configurations and trained models to foster future research efforts.
【9】 HyPoradise: An Open Baseline for Generative Speech Recognition with  Large Language Models标题：HyPoradise：一种开放的大语言模型生成式语音识别基准链接：https://arxiv.org/abs/2309.15701
作者：Chen Chen,Yuchen Hu,Chao-Han Huck Yang,Sabato Macro Siniscalchi,Pin-Yu Chen,Eng Siong Chng备注：Accepted to NeurIPS 2023, 24 pages. Datasets and Benchmarks Track摘要：深度神经网络的进步已经允许自动语音识别（ASR）系统在几个公开可用的干净语音数据集上实现人类平等。然而，即使是最先进的ASR系统在遇到不利条件时也会经历性能下降，因为经过良好训练的声学模型对语音域中的变化敏感，例如，背景噪音直观地说，人类通过依赖他们的语言知识来解决这个问题：模糊的口语术语的含义通常从上下文线索推断，从而减少对听觉系统的依赖性。受此观察的启发，我们引入了第一个利用外部大型语言模型（LLM）进行ASR纠错的开源基准测试，其中N-最佳解码假设为真实的转录预测提供了信息元素。这种方法是从传统的语言模型重评分策略，只能选择一个候选假设作为输出转录的范式转变。建议的基准包含一个新的数据集，HyPorcycle（HP），包括超过334，000对的N-最好的假设和相应的准确转录在流行的语音域。鉴于这个数据集，我们研究了三种类型的纠错技术的基础上LLM与不同数量的标记的假设转录对，获得了显着的字错误率（WER）降低。实验证明，该技术实现了突破，超越了传统的重新排名的方法的上限。更令人惊讶的是，具有合理提示和生成能力的LLM甚至可以纠正N-best列表中缺失的那些令牌。我们通过发布的预训练模型将我们的结果公开提供给可重复的管道，从而为使用LLM进行ASR纠错提供了一种新的评估范式。摘要：Advancements in deep neural networks have allowed automatic speech recognition (ASR) systems to attain human parity on several publicly available clean speech datasets. However, even state-of-the-art ASR systems experience performance degradation when confronted with adverse conditions, as a well-trained acoustic model is sensitive to variations in the speech domain, e.g., background noise. Intuitively, humans address this issue by relying on their linguistic knowledge: the meaning of ambiguous spoken terms is usually inferred from contextual cues thereby reducing the dependency on the auditory system. Inspired by this observation, we introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction, where N-best decoding hypotheses provide informative elements for true transcription prediction. This approach is a paradigm shift from the traditional language model rescoring strategy that can only select one candidate hypothesis as the output transcription. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses and corresponding accurate transcriptions across prevalent speech domains. Given this dataset, we examine three types of error correction techniques based on LLMs with varying amounts of labeled hypotheses-transcription pairs, which gains a significant word error rate (WER) reduction. Experimental evidence demonstrates the proposed technique achieves a breakthrough by surpassing the upper bound of traditional re-ranking based methods. More surprisingly, LLM with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list. We make our results publicly accessible for reproducible pipelines with released pre-trained models, thus providing a new evaluation paradigm for ASR error correction with LLMs.
【10】 Enhancing End-to-End Conversational Speech Translation Through Target  Language Context Utilization标题：利用目的语语境加强端到端会话语音翻译链接：https://arxiv.org/abs/2309.15686
作者：Amir Hussein,Brian Yan,Antonios Anastasopoulos,Shinji Watanabe,Sanjeev Khudanpur摘要：结合较长的上下文已被证明有利于机器翻译，但在端到端语音翻译（E2 E-ST）中包含上下文仍然研究不足。为了弥补这一差距，我们在E2 E-ST中引入目标语上下文，增强连贯性并克服扩展音频片段的记忆限制。此外，我们提出了上下文丢弃，以确保鲁棒性的情况下，上下文，并通过添加说话人信息进一步提高性能。我们提出的上下文E2 E-ST优于孤立的话语为基础的E2 E-ST的方法。最后，我们表明，在会话语音，上下文信息主要有助于捕捉上下文风格，以及解决照应和命名实体。摘要：Incorporating longer context has been shown to benefit machine translation, but the inclusion of context in end-to-end speech translation (E2E-ST) remains under-studied. To bridge this gap, we introduce target language context in E2E-ST, enhancing coherence and overcoming memory constraints of extended audio segments. Additionally, we propose context dropout to ensure robustness to the absence of context, and further improve performance by adding speaker information. Our proposed contextual E2E-ST outperforms the isolated utterance-based E2E-ST approach. Lastly, we demonstrate that in conversational speech, contextual information primarily contributes to capturing context style, as well as resolving anaphora and named entities.
【11】 Speech collage: code-switched audio generation by collaging monolingual  corpora标题：语音拼贴：通过拼贴单语语料库生成代码转换音频链接：https://arxiv.org/abs/2309.15674
作者：Amir Hussein,Dorsa Zeinali,Ondřej Klejch,Matthew Wiesner,Brian Yan,Shammur Chowdhury,Ahmed Ali,Shinji Watanabe,Sanjeev Khudanpur摘要：设计有效的自动语音识别（ASR）系统的代码转换（CS）往往取决于转录CS资源的可用性。为了解决数据短缺的问题，本文介绍了语音拼贴，一种方法，合成CS数据从单语语料库的拼接音频片段。我们使用重叠相加方法进一步提高音频生成的平滑质量。我们研究了两种情况下生成的数据对语音识别的影响：使用域内CS文本和具有合成CS文本的zero-shot方法。实证结果突出了高达34.4%和16.2%的相对减少混合错误率和字错误率的域和zero-shot的情况下，分别。最后，我们证明CS增强支持模型的代码转换倾向，并减少其单语偏见。摘要：Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zero-shot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model's code-switching inclination and reduces its monolingual bias.
【12】 Generative Speech Recognition Error Correction with Large Language  Models标题：基于大语言模型的产生式语音识别纠错链接：https://arxiv.org/abs/2309.15649
作者：Chao-Han Huck Yang,Yile Gu,Yi-Chieh Liu,Shalini Ghosh,Ivan Bulyko,Andreas Stolcke备注：Accepted to IEEE Automatic Speech Recognition and Understanding (ASRU) 2023摘要：我们探讨了大型语言模型（LLM）作为ASR后处理器执行重新评分和纠错的能力。我们的重点是指令提示，让LLM执行这些任务，而无需微调，我们评估不同的提示方案，零和Few-Shot的上下文学习，和一个新的任务激活提示（TAP）的方法，结合指令和示范。使用预训练的第一遍系统和两个域外任务（ATIS和WSJ）的重新评分输出，我们表明，重新评分仅通过在上下文中学习与冻结LLM实现的结果是有竞争力的重新评分域调整的LM。通过将提示技术与微调相结合，我们实现了低于N-最佳Oracle级别的错误率，展示了LLM的泛化能力。摘要：We explore the ability of large language models (LLMs) to act as ASR post-processors that perform rescoring and error correction. Our focus is on instruction prompting to let LLMs perform these task without fine-tuning, for which we evaluate different prompting schemes, both zero- and few-shot in-context learning, and a novel task-activating prompting (TAP) method that combines instruction and demonstration. Using a pre-trained first-pass system and rescoring output on two out-of-domain tasks (ATIS and WSJ), we show that rescoring only by in-context learning with frozen LLMs achieves results that are competitive with rescoring by domain-tuned LMs. By combining prompting techniques with fine-tuning we achieve error rates below the N-best oracle level, showcasing the generalization power of the LLMs.
【13】 Direct Models for Simultaneous Translation and Automatic Subtitling:  FBK@IWSLT2023标题：同声翻译和自动字幕的直接模型：FBK@IWSLT2023链接：https://arxiv.org/abs/2309.15554
作者：Sara Papi,Marco Gaido,Matteo Negri备注：None摘要：本文介绍了FBK参与IWSLT 2023评估活动的同声翻译和自动字幕跟踪。我们的提交集中在使用直接架构来执行这两项任务：对于同时推理，我们利用离线训练模型已经获得的知识，并直接应用策略来获得实时推理;对于字幕，我们采用直接ST模型来产生格式良好的字幕，并利用相同的架构来产生字幕与视听内容同步所需的时间戳。我们的英语-德语SimulST系统显示，与2021年和2022年任务中排名靠前的系统相比，计算感知延迟降低，增益高达3.5 BLEU。我们的自动字幕系统优于现有的解决方案的基础上的直接系统的3.7和1.7 SubER在英语-德语和英语-西班牙语分别。摘要：This paper describes the FBK's participation in the Simultaneous Translation and Automatic Subtitling tracks of the IWSLT 2023 Evaluation Campaign. Our submission focused on the use of direct architectures to perform both tasks: for the simultaneous one, we leveraged the knowledge already acquired by offline-trained models and directly applied a policy to obtain the real-time inference; for the subtitling one, we adapted the direct ST model to produce well-formed subtitles and exploited the same architecture to produce timestamps needed for the subtitle synchronization with audiovisual content. Our English-German SimulST system shows a reduced computational-aware latency compared to the one achieved by the top-ranked systems in the 2021 and 2022 rounds of the task, with gains of up to 3.5 BLEU. Our automatic subtitling system outperforms the only existing solution based on a direct system by 3.7 and 1.7 SubER in English-German and English-Spanish respectively.
【14】 High-Fidelity Speech Synthesis with Minimal Supervision: All Using  Diffusion Models标题：最小监督下的高保真语音合成：全部使用扩散模型链接：https://arxiv.org/abs/2309.15512
作者：Chunyu Qiang,Hao Li,Yixin Tian,Yi Zhao,Ying Zhang,Longbiao Wang,Jianwu Dang备注：arXiv admin note: substantial text overlap with arXiv:2307.15484; text overlap with arXiv:2309.00424摘要：文语转换（TTS）方法在语音克隆方面已经显示出良好的效果，但是它们需要大量的标记的文本-语音对。最小监督语音合成通过组合两种类型的离散语音表示（语义\声学）并使用两个序列到序列任务来实现最小监督的训练来解耦TTS。然而，现有的方法遭受信息冗余和维度爆炸的语义表示，和高频波形失真的离散声学表示。自回归框架表现出典型的不稳定性和不可控性问题。非自回归框架遭受由时长预测模型引起的韵律平均。为了解决这些问题，我们提出了一种最小监督高保真语音合成方法，其中所有模块都是基于扩散模型构建的。非自回归框架增强了可控性，持续时间扩散模型使多样化的韵律表达。针对现有语义编码方法中存在的信息冗余和维数爆炸问题，采用对比令牌声学预训练（CTAP）作为中间语义表示。使用Mel谱图作为声学表示。语义和声学表示都通过连续变量回归任务来预测，以解决高频细粒度波形失真的问题。实验结果表明，我们提出的方法优于基线方法。我们在我们的网站上提供音频样本。摘要：Text-to-speech (TTS) methods have shown promising results in voice cloning, but they require a large number of labeled text-speech pairs. Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations(semantic \& acoustic) and using two sequence-to-sequence tasks to enable training with minimal supervision. However, existing methods suffer from information redundancy and dimension explosion in semantic representation, and high-frequency waveform distortion in discrete acoustic representation. Autoregressive frameworks exhibit typical instability and uncontrollability issues. And non-autoregressive frameworks suffer from prosodic averaging caused by duration prediction models. To address these issues, we propose a minimally-supervised high-fidelity speech synthesis method, where all modules are constructed based on the diffusion models. The non-autoregressive framework enhances controllability, and the duration diffusion model enables diversified prosodic expression. Contrastive Token-Acoustic Pretraining (CTAP) is used as an intermediate semantic representation to solve the problems of information redundancy and dimension explosion in existing semantic coding methods. Mel-spectrogram is used as the acoustic representation. Both semantic and acoustic representations are predicted by continuous variable regression tasks to solve the problem of high-frequency fine-grained waveform distortion. Experimental results show that our proposed method outperforms the baseline method. We provide audio samples on our website.
【15】 joint prediction and denoising for large-scale multilingual  self-supervised learning标题：大规模多语言自监督学习的联合预测与去噪链接：https://arxiv.org/abs/2309.15317
作者：William Chen,Jiatong Shi,Brian Yan,Dan Berrebbi,Wangyou Zhang,Yifan Peng,Xuankai Chang,Soumi Maiti,Shinji Watanabe备注：Accepted to ASRU 2023摘要：由于处理多种语言所需的费用和复杂性，多语言自监督学习（SSL）通常落后于最先进的（SOTA）方法。这进一步损害了SSL的可重复性，由于其资源使用，SSL已经仅限于少数研究小组。我们表明，更强大的技术实际上可以导致更有效的预训练，开放SSL更多的研究小组。我们提出了WavLabLM，它将WavLM的联合预测和去噪扩展到136种语言的4万小时数据。为了构建WavLabLM，我们设计了一种新的多阶段预训练方法，旨在解决多语言数据的语言不平衡问题。WavLabLM在ML-SUPERB上实现了与XLS-R相当的性能，训练数据不到10%，使得SSL可以通过学术计算实现。我们表明，进一步的效率可以实现与香草HuBERT基地模型，它可以保持94%的XLS-R的性能，只有3%的数据，4个GPU，和有限的试验。我们开源了ESPnet中的所有代码和模型。摘要：Multilingual self-supervised learning (SSL) has often lagged behind state-of-the-art (SOTA) methods due to the expenses and complexity required to handle many languages. This further harms the reproducibility of SSL, which is already limited to few research groups due to its resource usage. We show that more powerful techniques can actually lead to more efficient pre-training, opening SSL to more research groups. We propose WavLabLM, which extends WavLM's joint prediction and denoising to 40k hours of data across 136 languages. To build WavLabLM, we devise a novel multi-stage pre-training method, designed to address the language imbalance of multilingual data. WavLabLM achieves comparable performance to XLS-R on ML-SUPERB with less than 10% of the training data, making SSL realizable with academic compute. We show that further efficiency can be achieved with a vanilla HuBERT Base model, which can maintain 94% of XLS-R's performance with only 3% of the data, 4 GPUs, and limited trials. We open-source all code and models in ESPnet.
【16】 Low-rank Adaptation of Large Language Model Rescoring for  Parameter-Efficient Speech Recognition标题：参数高效语音识别中大语言模型研究的低阶自适应链接：https://arxiv.org/abs/2309.15223
作者：Yu Yu,Chao-Han Huck Yang,Jari Kolehmainen,Prashanth G. Shivakumar,Yile Gu,Sungho Ryu,Roger Ren,Qi Luo,Aditya Gourav,I-Fan Chen,Yi-Chieh Liu,Tuan Dinh,Ankur Gandhe,Denis Filimonov,Shalini Ghosh,Andreas Stolcke,Ariya Rastow,Ivan Bulyko备注：Accepted to IEEE ASRU 2023. Internal Review Approved摘要：我们提出了一个神经语言建模系统的基础上低秩自适应（LoRA）的语音识别输出rescoring。虽然像BERT这样的预训练语言模型（LM）在二次评分中表现出了优异的性能，但扩大预训练阶段并使预训练模型适应特定领域的高计算成本限制了它们在重新评分中的实际使用。在这里，我们提出了一种基于低秩分解的方法来训练重新评分BERT模型，并仅使用一小部分（0.08%）的预训练参数将其适应新的领域。这些插入的矩阵通过区分性训练目标以及基于相关性的正则化损失来优化。所提出的低秩自适应Rescore-BERT（LoRB）架构在LibriSpeech和内部数据集上进行评估，训练时间减少了5.4到3.6倍。摘要：We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.
机器翻译，仅供参考