今日论文合集：cs.SD语音12篇，eess.AS音频处理16篇。
cs.SD语音
【1】 Audio-Visual Neural Syntax Acquisition标题：视听神经句法习得链接：https://arxiv.org/abs/2310.07654
作者：Cheng-I Jeff Lai,Freda Shi,Puyuan Peng,Yoon Kim,Kevin Gimpel,Shiyu Chang,Yung-Sung Chuang,Saurabhchand Bhati,David Cox,David Harwath,Yang Zhang,Karen Livescu,James Glass机构：TTIC, UT Austin, UC Santa Barbara, MIT-IBM Watson AI Lab摘要：我们研究短语结构归纳从视觉接地的讲话。其核心思想是首先将语音波形分割成词段序列，然后使用推断的段级连续表示来诱导短语结构。我们提出了视听神经网络学习器（AV-NSL），它通过听音频和看图像来学习短语结构，而不需要接触文本。通过对配对图像和语音字幕进行训练，AV-NSL能够推断出有意义的短语结构，这些短语结构与英语和德语的自然监督文本解析器得出的短语结构相当。我们的研究结果扩展了先前在无监督语言习得中从语音和接地语法归纳的工作，并提出了一种弥合这两个主题之间差距的方法。摘要：We study phrase structure induction from visually-grounded speech. The core idea is to first segment the speech waveform into sequences of word segments, and subsequently induce phrase structure using the inferred segment-level continuous representations. We present the Audio-Visual Neural Syntax Learner (AV-NSL) that learns phrase structure by listening to audio and looking at images, without ever being exposed to text. By training on paired images and spoken captions, AV-NSL exhibits the capability to infer meaningful phrase structures that are comparable to those derived by naturally-supervised text parsers, for both English and German. Our findings extend prior work in unsupervised language acquisition from speech and grounded grammar induction, and present one approach to bridge the gap between the two topics.
【2】 Adapting the adapters for code-switching in multilingual ASR标题：多语种ASR中代码转换适配器的适配链接：https://arxiv.org/abs/2310.07423
作者：Atharva Kulkarni,Ajinkya Kulkarni,Miguel Couceiro,Hanan Aldarmaki机构：∗ MBZUAI, UAE, † Universit´e de Lorraine, CNRS, LORIA, France, ‡Erisha Labs, India备注：Submitted to ICASSP 2024摘要：最近，大型预训练的多语言语音模型已经显示出将自动语音识别（ASR）扩展到许多低资源语言的潜力。这些模型中的一些在其公式化中使用语言适配器，这有助于提高单语性能，并避免在资源丰富的语言上进行多语言建模的一些缺点。然而，这种提法限制了代码转换语音，其中两种语言混合在一起，在同一话语这些模型的可用性。在这项工作中，我们提出了一些方法来有效地微调这种模型的代码切换语音，通过同化信息，从两种语言适配器在网络中的每个语言适应点。我们还将代码转换建模为一系列潜在的二进制序列，这些序列可用于在框架级别引导来自每个语言适配器的信息流。所提出的方法进行了评估，包括阿拉伯语，普通话，印地语与英语配对的三个代码转换数据集，在所有测试集的CER至少有10%的绝对减少，在代码转换性能的一致改善。摘要：Recently, large pre-trained multilingual speech models have shown potential in scaling Automatic Speech Recognition (ASR) to many low-resource languages. Some of these models employ language adapters in their formulation, which helps to improve monolingual performance and avoids some of the drawbacks of multi-lingual modeling on resource-rich languages. However, this formulation restricts the usability of these models on code-switched speech, where two languages are mixed together in the same utterance. In this work, we propose ways to effectively fine-tune such models on code-switched speech, by assimilating information from both language adapters at each language adaptation point in the network. We also model code-switching as a sequence of latent binary sequences that can be used to guide the flow of information from each language adapter at the frame level. The proposed approaches are evaluated on three code-switched datasets encompassing Arabic, Mandarin, and Hindi languages paired with English, showing consistent improvements in code-switching performance with at least 10\% absolute reduction in CER across all test sets.
【3】 DASpeech: Directed Acyclic Transformer for Fast and High-quality  Speech-to-Speech Translation标题：DASpeech：用于快速高质量语音到语音翻译的有向无环转换器链接：https://arxiv.org/abs/2310.07403
作者：Qingkai Fang,Yan Zhou,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：NeurIPS 2023. Audio samples are available at this https URL摘要：直接语音到语音翻译（S2 ST）使用单个模型将语音从一种语言翻译成另一种语言。然而，由于语言和声学多样性的存在，目标语音遵循复杂的多模态分布，为S2 ST模型实现高质量翻译和快速解码速度带来了挑战。在本文中，我们提出了DASpeech，一个非自回归直接S2 ST模型，实现了快速和高质量的S2 ST。为了更好地捕捉目标语音的复杂分布，DASpeech采用两遍架构将生成过程分解为两个步骤，语言解码器首先生成目标文本，声学解码器然后基于语言解码器的隐藏状态生成目标语音。具体来说，我们使用DA-变压器的解码器作为语言解码器，并使用FastSpeech 2作为声学解码器。DA-Transformer使用有向无环图（DAG）对翻译进行建模。为了在训练过程中考虑DAG中的所有潜在路径，我们通过动态规划计算每个目标令牌的预期隐藏状态，并将其馈送到声学解码器中以预测目标梅尔频谱图。在推理过程中，我们选择最可能的路径，并将该路径上的隐藏状态作为声学解码器的输入。在CVSS Fr-En基准测试上的实验表明，DASpeech可以实现与最先进的S2 ST模型Translatotron 2相当甚至更好的性能，同时与自回归基线相比，保持高达18.53倍的加速比。与以往的非自回归S2 ST模型相比，DASpeech不依赖于知识提炼和迭代解码，在翻译质量和解码速度上都有了显著的提升。此外，DASpeech显示了在翻译过程中保留源语音的说话者的声音的能力。摘要：Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr-En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.
【4】 Investigating the Effect of Language Models in Sequence Discriminative  Training for Neural Transducers标题：语言模型在神经传感器顺序辨别训练中的作用研究链接：https://arxiv.org/abs/2310.07345
作者：Zijian Yang,Wei Zhou,Ralf Schlüter,Hermann Ney机构：Machine Learning and Human Language Technology, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：accepted at ASRU 2023摘要：在这项工作中，我们调查的效果语言模型（LM）与不同的上下文长度和标签单位（音素与。字）用于基于音素的神经换能器的序列判别训练。无格和N-最佳列表的方法进行检查。对于具有音素级LM的无格方法，我们提出了一种方法来近似上下文历史，以采用具有全上下文依赖性的LM。这种近似可以扩展到任意的上下文长度，并使字级LM的无格方法的使用。此外，系统地比较了无格和基于N-最佳列表的方法。实验结果表明，词级LM的训练效果优于音素级LM。此外，我们发现用于概率计算的LM的上下文大小对性能的影响有限。此外，我们的研究结果揭示了序列判别训练中假设空间质量的关键重要性。摘要：In this work, we investigate the effect of language models (LMs) with different context lengths and label units (phoneme vs. word) used in sequence discriminative training for phoneme-based neural transducers. Both lattice-free and N-best-list approaches are examined. For lattice-free methods with phoneme-level LMs, we propose a method to approximate the context history to employ LMs with full-context dependency. This approximation can be extended to arbitrary context length and enables the usage of word-level LMs in lattice-free methods. Moreover, a systematic comparison is conducted across lattice-free and N-best-list-based methods. Experimental results on Librispeech show that using the word-level LM in training outperforms the phoneme-level LM. Besides, we find that the context size of the LM used for probability computation has a limited effect on performance. Moreover, our results reveal the pivotal importance of the hypothesis space quality in sequence discriminative training.
【5】 Enhancing expressivity transfer in textless speech-to-speech translation标题：增强无文本语音翻译中的表现力传递链接：https://arxiv.org/abs/2310.07279
作者：Jarod Duret,Benjamin O'Brien,Yannick Estève,Titouan Parcollet机构：LIA - Avignon Universite, France, University of Cambridge, United-Kingdom备注：None摘要：无文本语音到语音翻译系统正在快速发展，这要归功于自监督学习技术的集成。然而，现有的最先进的系统在跨不同语言准确地捕获和传输表达能力方面存在不足。表达性在传达情感、细微差别和文化微妙之处方面起着至关重要的作用，从而加强了不同语言之间的沟通。为了解决这个问题，本研究提出了一种新的方法，操作在离散语音单元的水平，并利用多语言情感嵌入捕获语言无关的信息。具体来说，我们演示了如何使用这些嵌入来有效地预测目标语言中语音单元的音高和持续时间。通过对法语到英语翻译任务进行的客观和主观实验，我们的研究结果突出了我们的方法与当前最先进的系统相比所实现的卓越的表达力转移。摘要：Textless speech-to-speech translation systems are rapidly advancing, thanks to the integration of self-supervised learning techniques. However, existing state-of-the-art systems fall short when it comes to capturing and transferring expressivity accurately across different languages. Expressivity plays a vital role in conveying emotions, nuances, and cultural subtleties, thereby enhancing communication across diverse languages. To address this issue this study presents a novel method that operates at the discrete speech unit level and leverages multilingual emotion embeddings to capture language-agnostic information. Specifically, we demonstrate how these embeddings can be used to effectively predict the pitch and duration of speech units in the target language. Through objective and subjective experiments conducted on a French-to-English translation task, our findings highlight the superior expressivity transfer achieved by our approach compared to current state-of-the-art systems.
【6】 Vec-Tok Speech: speech vectorization and tokenization for neural speech  generation标题：VEC-TOK语音：用于神经语音生成的语音矢量化和标记化链接：https://arxiv.org/abs/2310.07246
作者：Xinfa Zhu,Yuanjun Lv,Yi Lei,Tao Li,Wendi He,Hongbin Zhou,Heng Lu,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Ximalaya Inc., China备注：15 pages, 2 figures摘要：语言模型（LM）最近在自然语言处理和计算机视觉中蓬勃发展，在各种任务中生成高保真文本或图像。相比之下，目前的语音生成模型仍然在语音质量和任务泛化方面苦苦挣扎。本文介绍了Vec-Tok语音，一个可扩展的框架，类似于多个语音生成任务，生成表达和高保真语音。具体来说，我们提出了一种新的语音编码器的语音矢量和语义令牌的基础上。语音向量包含有助于高保真语音重建的声学细节，而语义令牌则专注于语音的语言内容，便于语言建模。Vec-Tok Speech基于所提出的语音编解码器，利用LM来承担语音生成的核心。此外，字节对编码（BPE）的引入，以减少令牌的长度和比特率，以降低曝光偏差和更长的上下文覆盖，提高LM的性能。Vec-Tok Speech可用于语言内和跨语言zero-shot语音转换（VC）、零触发说话风格转换文本到语音（TTS）、语音到语音翻译（S2 ST）、语音去噪以及说话者去识别和匿名化。实验表明，Vec-Tok语音，建立在5万小时的语音，性能优于其他SOTA模型。代码将在https://github.com/BakerBunker/VecTok上提供。摘要：Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-fidelity texts or images in various tasks. In contrast, the current speech generative models are still struggling regarding speech quality and task generalization. This paper presents Vec-Tok Speech, an extensible framework that resembles multiple speech generation tasks, generating expressive and high-fidelity speech. Specifically, we propose a novel speech codec based on speech vectors and semantic tokens. Speech vectors contain acoustic details contributing to high-fidelity speech reconstruction, while semantic tokens focus on the linguistic content of speech, facilitating language modeling. Based on the proposed speech codec, Vec-Tok Speech leverages an LM to undertake the core of speech generation. Moreover, Byte-Pair Encoding (BPE) is introduced to reduce the token length and bit rate for lower exposure bias and longer context coverage, improving the performance of LMs. Vec-Tok Speech can be used for intra- and cross-lingual zero-shot voice conversion (VC), zero-shot speaking style transfer text-to-speech (TTS), speech-to-speech translation (S2ST), speech denoising, and speaker de-identification and anonymization. Experiments show that Vec-Tok Speech, built on 50k hours of speech, performs better than other SOTA models. Code will be available at https://github.com/BakerBunker/VecTok .
【7】 Psychoacoustic Challenges Of Speech Enhancement On VoIP Platforms标题：VoIP平台上语音增强的心理声学挑战链接：https://arxiv.org/abs/2310.07161
作者：Joseph Konan,Ojas Bhargave,Shikhar Agnihotri,Shuo Han,Yunyang Zeng,Ankit Shah,Bhiksha Raj机构：KonanAI ,Carnegie Mellon University摘要：在VoIP（互联网协议语音）电信的范围内，由声学变换引入的复杂性值得进行严格的分析。这项研究植根于专有的发送方去噪效果的探索，仔细评估了Google Meets和Zoom等平台。该研究借鉴了深度噪声抑制（DNS）2020数据集，确保针对各种去噪设置和接收器接口进行结构化检查。通过瓦哈卡分解，传统上是一种计量经济学工具，在此重新利用来分析VoIP系统内的声学语音扰动，引入了一种新的方法。为了进一步研究这些转换的影响，心理声学指标，特别是PESQ和STOI，被用来提供对语音改变的全面理解。累积起来，所获得的见解强调了VoIP影响声学动力学的复杂景观。除了主要的调查结果，报告了大量的指标，扩大了研究范围。此外，时域和时频域语音增强模型的域外基准测试包括在内，从而提高了这种调查的深度和适用性。摘要：Within the ambit of VoIP (Voice over Internet Protocol) telecommunications, the complexities introduced by acoustic transformations merit rigorous analysis. This research, rooted in the exploration of proprietary sender-side denoising effects, meticulously evaluates platforms such as Google Meets and Zoom. The study draws upon the Deep Noise Suppression (DNS) 2020 dataset, ensuring a structured examination tailored to various denoising settings and receiver interfaces. A methodological novelty is introduced via the Oaxaca decomposition, traditionally an econometric tool, repurposed herein to analyze acoustic-phonetic perturbations within VoIP systems. To further ground the implications of these transformations, psychoacoustic metrics, specifically PESQ and STOI, were harnessed to furnish a comprehensive understanding of speech alterations. Cumulatively, the insights garnered underscore the intricate landscape of VoIP-influenced acoustic dynamics. In addition to the primary findings, a multitude of metrics are reported, extending the research purview. Moreover, out-of-domain benchmarking for both time and time-frequency domain speech enhancement models is included, thereby enhancing the depth and applicability of this inquiry.
【8】 LLark: A Multimodal Foundation Model for Music标题：Lark：音乐的多通道基础模型链接：https://arxiv.org/abs/2310.07160
作者：Josh Gardner,Simon Durand,Daniel Stoller,Rachel M. Bittner机构：University of Washington, Spotify摘要：音乐具有独特而复杂的结构，这对专家和现有的人工智能系统来说都是一个挑战，相对于其他形式的音频来说，这是一个独特的挑战。我们提出了LLark，一个用于音乐理解的调整的多模态模型。我们详细介绍了我们的数据集创建过程，其中包括增强各种开源音乐数据集的注释，并将其转换为统一的注释调整格式。我们为Lark提出了一个多模态架构，将音乐的预训练生成模型与预训练语言模型相结合。在对三种类型的任务（音乐理解，字幕和推理）的评估中，我们发现我们的模型在音乐理解的zero-shot泛化中匹配或优于现有的基线，并且人类在字幕和推理任务中与模型的反应高度一致。Lark完全是从开源音乐数据和模型中训练出来的，我们在发布本文时提供了我们的训练代码。其他结果和音频示例在https：//bit.ly/llark，我们的源代码在https://github.com/spotify-research/llark。摘要：Music has a unique and complex structure which is challenging for both expert humans and existing AI systems to understand, and presents unique challenges relative to other forms of audio. We present LLark, an instruction-tuned multimodal model for music understanding. We detail our process for dataset creation, which involves augmenting the annotations of diverse open-source music datasets and converting them to a unified instruction-tuning format. We propose a multimodal architecture for LLark, integrating a pretrained generative model for music with a pretrained language model. In evaluations on three types of tasks (music understanding, captioning, and reasoning), we show that our model matches or outperforms existing baselines in zero-shot generalization for music understanding, and that humans show a high degree of agreement with the model's responses in captioning and reasoning tasks. LLark is trained entirely from open-source music data and models, and we make our training code available along with the release of this paper. Additional results and audio examples are at https://bit.ly/llark, and our source code is available at https://github.com/spotify-research/llark .
【9】 Acoustic Model Fusion for End-to-end Speech Recognition标题：端到端语音识别的声学模型融合链接：https://arxiv.org/abs/2310.07062
作者：Zhihong Lei,Mingbin Xu,Shiyi Han,Leo Liu,Zhen Huang,Tim Ng,Yuanyuan Zhang,Ernest Pusateri,Mirko Hannemann,Yaqiao Deng,Man-Hung Siu机构：Apple摘要：深度学习和自动语音识别（ASR）的最新进展使端到端（E2 E）ASR系统成为可能，并将准确性提升到一个新的水平。E2 E系统隐式地在基于音频-文本对训练的单个网络中对所有常规ASR组件（诸如声学模型（AM）和语言模型（LM））进行建模。尽管这种简单的系统架构，融合一个单独的LM，专门训练的文本语料库，到E2 E系统已被证明是有益的。然而，LM融合的应用存在某些缺点，例如其不能解决内部AM固有的域失配问题。从LM融合的概念中获得灵感，我们提出将外部AM集成到E2 E系统中，以更好地解决域不匹配问题。通过实施这种新颖的方法，我们已经实现了显着降低单词错误率，在不同的测试集上，令人印象深刻的下降高达14.3%。我们还发现，这种AM融合方法在增强命名实体识别方面特别有益。摘要：Recent advances in deep learning and automatic speech recognition (ASR) have enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new level. The E2E systems implicitly model all conventional ASR components, such as the acoustic model (AM) and the language model (LM), in a single network trained on audio-text pairs. Despite this simpler system architecture, fusing a separate LM, trained exclusively on text corpora, into the E2E system has proven to be beneficial. However, the application of LM fusion presents certain drawbacks, such as its inability to address the domain mismatch issue inherent to the internal AM. Drawing inspiration from the concept of LM fusion, we propose the integration of an external AM into the E2E system to better address the domain mismatch. By implementing this novel approach, we have achieved a significant reduction in the word error rate, with an impressive drop of up to 14.3% across varied test sets. We also discovered that this AM fusion approach is particularly beneficial in enhancing named entity recognition.
【10】 Neural Harmonium: An Interpretable Deep Structure for Nonlinear Dynamic  System Identification with Application to Audio Processing标题：神经谐和：一种可解释的非线性动态系统辨识深层结构及其在音频处理中的应用链接：https://arxiv.org/abs/2310.07032
作者：Karim Helwani,Erfan Soltanmohammadi,Michael M. Goodwin摘要：最近，提高深度神经网络的可解释性受到了越来越多的关注，特别是当深度学习的力量被用来解决物理问题时。可解释性帮助我们理解模型的泛化能力和揭示其局限性。在本文中，我们介绍了一个因果可解释的深层结构建模动态系统。我们提出的模型利用谐波分析建模的系统在时间-频率域，同时保持高的时间和频谱分辨率。此外，该模型是建立在一个顺序递归的方式，允许快速，鲁棒性和精确的二阶优化，而不需要一个明确的海森计算。为了避免我们的系统的构建块的高维度，神经网络的设计，以确定频率的相互依赖性。所提出的模型说明和验证非线性系统识别问题所需的音频信号处理任务。众包实验对比所提出的方法的性能，以其他国家的最先进的解决方案上的声学回声消除方案证实了我们的方法在现实生活中的应用的有效性。摘要：Improving the interpretability of deep neural networks has recently gained increased attention, especially when the power of deep learning is leveraged to solve problems in physics. Interpretability helps us understand a model's ability to generalize and reveal its limitations. In this paper, we introduce a causal interpretable deep structure for modeling dynamic systems. Our proposed model makes use of the harmonic analysis by modeling the system in a time-frequency domain while maintaining high temporal and spectral resolution. Moreover, the model is built in an order recursive manner which allows for fast, robust, and exact second order optimization without the need for an explicit Hessian calculation. To circumvent the resulting high dimensionality of the building blocks of our system, a neural network is designed to identify the frequency interdependencies. The proposed model is illustrated and validated on nonlinear system identification problems as required for audio signal processing tasks. Crowd-sourced experimentation contrasting the performance of the proposed approach to other state-of-the-art solutions on an acoustic echo cancellation scenario confirms the effectiveness of our method for real-life applications.
【11】 Prosody Analysis of Audiobooks标题：有声读物的韵律分析链接：https://arxiv.org/abs/2310.06930
作者：Charuta Pethe,Yunting Yin,Steven Skiena机构：Department of Computer Science, Stony Brook University摘要：文本到语音的最新进展使得从文本生成听起来自然的音频成为可能。然而，有声读物的叙述涉及读者戏剧性的发声和语调，更依赖于叙述中的情感，对话和描述。使用我们的数据集93对齐的书-有声读物对，我们提出了改进的模型韵律预测属性（音高，音量和语音速率）从叙事文本使用语言建模。我们预测的韵律属性与人类有声读物阅读的相关性要比最先进的商业TTS系统的结果好得多：对于24本书中的22本，我们预测的音高显示出与人类阅读的更高相关性，而对于24本书中的23本，我们预测的音量属性证明更类似于人类阅读。最后，我们提出了一个人的评估研究，量化的程度，人们更喜欢韵律增强有声读物的商业文本到语音系统。摘要：Recent advances in text-to-speech have made it possible to generate natural-sounding audio from text. However, audiobook narrations involve dramatic vocalizations and intonations by the reader, with greater reliance on emotions, dialogues, and descriptions in the narrative. Using our dataset of 93 aligned book-audiobook pairs, we present improved models for prosody prediction properties (pitch, volume, and rate of speech) from narrative text using language modeling. Our predicted prosody attributes correlate much better with human audiobook readings than results from a state-of-the-art commercial TTS system: our predicted pitch shows a higher correlation with human reading for 22 out of the 24 books, while our predicted volume attribute proves more similar to human reading for 23 out of the 24 books. Finally, we present a human evaluation study to quantify the extent that people prefer prosody-enhanced audiobook readings over commercial text-to-speech systems.
【12】 Deep Video Inpainting Guided by Audio-Visual Self-Supervision标题：视听自律指导下的深度视频修复链接：https://arxiv.org/abs/2310.07663
作者：Kyuyeon Kim,Junsik Jung,Woo Jae Kim,Sung-Eui Yoon机构：School of Computing, Korea Advanced Institute of Science and Technology (KAIST)备注：Accepted at ICASSP 2022摘要：人类可以很容易地想象一个场景，从听觉信息的基础上，他们的先验知识的视听事件。在本文中，我们在深度学习模型中模仿这种人类固有的能力，以提高视频修复的质量。为了实现先验知识，我们首先训练视听网络，它学习听觉和视觉信息之间的对应关系。然后，视听网络被用作将视听对应的先验知识传递到视频修复网络的引导器。这种先验知识通过我们提出的两种新的损失转移：视听注意力损失和视听伪类一致性损失。这两个损失通过鼓励修复结果与其同步音频具有高度对应性来进一步提高视频修复的性能。实验结果表明，我们提出的方法可以恢复一个更广泛的领域的视频场景，是特别有效的，当场景中的声音对象是部分失明。摘要：Humans can easily imagine a scene from auditory information based on their prior knowledge of audio-visual events. In this paper, we mimic this innate human ability in deep learning models to improve the quality of video inpainting. To implement the prior knowledge, we first train the audio-visual network, which learns the correspondence between auditory and visual information. Then, the audio-visual network is employed as a guider that conveys the prior knowledge of audio-visual correspondence to the video inpainting network. This prior knowledge is transferred through our proposed two novel losses: audio-visual attention loss and audio-visual pseudo-class consistency loss. These two losses further improve the performance of the video inpainting by encouraging the inpainting result to have a high correspondence to its synchronized audio. Experimental results demonstrate that our proposed method can restore a wider domain of video scenes and is particularly effective when the sounding object in the scene is partially blinded.
eess.AS音频处理
【1】 Deep Video Inpainting Guided by Audio-Visual Self-Supervision标题：视听自律指导下的深度视频修复链接：https://arxiv.org/abs/2310.07663
作者：Kyuyeon Kim,Junsik Jung,Woo Jae Kim,Sung-Eui Yoon机构：School of Computing, Korea Advanced Institute of Science and Technology (KAIST)备注：Accepted at ICASSP 2022摘要：人类可以很容易地想象一个场景，从听觉信息的基础上，他们的先验知识的视听事件。在本文中，我们在深度学习模型中模仿这种人类固有的能力，以提高视频修复的质量。为了实现先验知识，我们首先训练视听网络，它学习听觉和视觉信息之间的对应关系。然后，视听网络被用作将视听对应的先验知识传递到视频修复网络的引导器。这种先验知识通过我们提出的两种新的损失转移：视听注意力损失和视听伪类一致性损失。这两个损失通过鼓励修复结果与其同步音频具有高度对应性来进一步提高视频修复的性能。实验结果表明，我们提出的方法可以恢复一个更广泛的领域的视频场景，是特别有效的，当场景中的声音对象是部分失明。摘要：Humans can easily imagine a scene from auditory information based on their prior knowledge of audio-visual events. In this paper, we mimic this innate human ability in deep learning models to improve the quality of video inpainting. To implement the prior knowledge, we first train the audio-visual network, which learns the correspondence between auditory and visual information. Then, the audio-visual network is employed as a guider that conveys the prior knowledge of audio-visual correspondence to the video inpainting network. This prior knowledge is transferred through our proposed two novel losses: audio-visual attention loss and audio-visual pseudo-class consistency loss. These two losses further improve the performance of the video inpainting by encouraging the inpainting result to have a high correspondence to its synchronized audio. Experimental results demonstrate that our proposed method can restore a wider domain of video scenes and is particularly effective when the sounding object in the scene is partially blinded.
【2】 Damping Density of an Absorptive Shoebox Room Derived from the  Image-Source Method标题：用像源法推导吸波鞋盒房间的阻尼密度链接：https://arxiv.org/abs/2310.07363
作者：Sebastian J. Schlecht,Karolina Prawda,Rudolf Rabenstein,Maximilian Schäfer机构：)Acoustics Lab, Department of Information and Communications Engineering, Aalto University, P.O. Box , FI-, Aalto, Finland, )Multimedia Communications and Signal Processing, Digital Communications, Friedrich-Alexander-摘要：像源法被广泛应用于计算任意吸收鞋盒型房间的房间脉冲响应。然而，随着RIR长度的增加，图像源的数量迅速增长，导致计算缓慢。在本文中，我们推导出一个封闭形式的表达的阻尼密度，它的特点是整体的多斜率能量衰减。随时间的全向能量衰减直接来自阻尼密度。由此产生的能量衰减模型准确地匹配通过图像源方法模拟的后期混响。该模型允许通过用能量包络整形噪声来快速随机合成后期混响。对不同壁面阻尼系数的仿真验证了模型的准确性。与最先进的近似方法相比，所提出的模型始终优于能量衰减预测精度。本文阐述了建议的阻尼密度的适用性建模多斜率的声音能量衰减，预测混响时间在非扩散声场，和快速频率相关的RIR合成。摘要：The image-source method is widely applied to compute room impulse responses (RIRs) of shoebox rooms with arbitrary absorption. However, with increasing RIR lengths, the number of image sources grows rapidly, leading to slow computation. In this paper, we derive a closed-form expression for the damping density, which characterizes the overall multi-slope energy decay. The omnidirectional energy decay over time is directly derived from the damping density. The resulting energy decay model accurately matches the late reverberation simulated via the image-source method. The proposed model allows the fast stochastic synthesis of late reverberation by shaping noise with the energy envelope. Simulations of various wall damping coefficients demonstrate the model's accuracy. The proposed model consistently outperforms the energy decay prediction accuracy compared to a state-of-the-art approximation method. The paper elaborates on the proposed damping density's applicability to modeling multi-sloped sound energy decay, predicting reverberation time in non-diffuse sound fields, and fast frequency-dependent RIR synthesis.
【3】 Magnitude-and-phase-aware Speech Enhancement with Parallel Sequence  Modeling标题：基于并行序列建模的幅度和相位感知语音增强链接：https://arxiv.org/abs/2310.07316
作者：Yuewei Zhang,Huanbin Zou,Jie Zhu机构：Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China,  Tencent Video Cloud, Shanghai, China备注：Accepted by ASRU 2023摘要：在语音增强（SE）中，相位估计对提高语音的感知质量至关重要，因此许多方法都是以纯净语音的复短时傅里叶变换（STFT）谱或复理想比率掩模（CSTFT）作为学习目标。为了预测这些复杂的目标，通常的解决方案是设计一个复杂的神经网络，或者使用一个真实的网络来分别预测目标的实部和虚部。而本文提出了一种利用真实网络来估计幅度掩模和归一化频率的方法，不仅避免了复杂网络导致的模型复杂度的显著增加，而且比以往的相位估计方法表现出更好的性能。同时，我们设计了一个并行序列建模（PSM）模块，以改善基于卷积递归网络（CRN）的SE模型中的RNN模块。我们将我们的方法命名为幅度和相位感知和基于PSM的CRN（MPCRN）。实验结果表明，我们的MPCRN具有优越的SE性能。摘要：In speech enhancement (SE), phase estimation is important for perceptual quality, so many methods take clean speech's complex short-time Fourier transform (STFT) spectrum or the complex ideal ratio mask (cIRM) as the learning target. To predict these complex targets, the common solution is to design a complex neural network, or use a real network to separately predict the real and imaginary parts of the target. But in this paper, we propose to use a real network to estimate the magnitude mask and normalized cIRM, which not only avoids the significant increase of the model complexity caused by complex networks, but also shows better performance than previous phase estimation methods. Meanwhile, we devise a parallel sequence modeling (PSM) block to improve the RNN block in the convolutional recurrent network (CRN)-based SE model. We name our method as magnitude-and-phase-aware and PSM-based CRN (MPCRN). The experimental results illustrate that our MPCRN has superior SE performance.
【4】 VSANet: Real-time Speech Enhancement Based on Voice Activity Detection  and Causal Spatial Attention标题：VSANet：基于语音活动检测和因果空间注意的实时语音增强链接：https://arxiv.org/abs/2310.07295
作者：Yuewei Zhang,Huanbin Zou,Jie Zhu机构：Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai, China,  Tencent Video Cloud, Shanghai, China备注：Accepted by ASRU 2023摘要：基于深度学习的语音增强（SE）方法通常以纯净语音的波形或时频谱特征作为学习目标，通过减少深度神经网络（DNN）输出与目标之间的误差损失来训练DNN。这是一个传统的单任务学习范式，已被证明是有效的，但我们发现，多任务学习框架可以提高SE性能。具体来说，我们设计了一个包含SE模块和语音活动检测（VAD）模块的框架，这两个模块共享同一个编码器，整个网络通过两个模块的加权损耗进行优化。此外，我们设计了一个因果空间注意（CSA）块，以提高DNN的表征能力。结合VAD辅助多任务学习框架和CSA块，我们的SE网络被命名为VSANet。实验结果证明了多任务学习和CSA块的好处，这使VSANet具有出色的SE性能。摘要：The deep learning-based speech enhancement (SE) methods always take the clean speech's waveform or time-frequency spectrum feature as the learning target, and train the deep neural network (DNN) by reducing the error loss between the DNN's output and the target. This is a conventional single-task learning paradigm, which has been proven to be effective, but we find that the multi-task learning framework can improve SE performance. Specifically, we design a framework containing a SE module and a voice activity detection (VAD) module, both of which share the same encoder, and the whole network is optimized by the weighted loss of the two modules. Moreover, we design a causal spatial attention (CSA) block to promote the representation capability of DNN. Combining the VAD aided multi-task learning framework and CSA block, our SE network is named VSANet. The experimental results prove the benefits of multi-task learning and the CSA block, which give VSANet an excellent SE performance.
【5】 Typing to Listen at the Cocktail Party: Text-Guided Target Speaker  Extraction标题：在鸡尾酒会上打字听：文本引导的目标说话人提取链接：https://arxiv.org/abs/2310.07284
作者：Xiang Hao,Jibin Wu,Jianwei Yu,Chenglin Xu,Kay Chen Tan机构：Department of Computing, The Hong Kong Polytechnic University, Tencent AI Lab备注：Under review摘要：人类拥有在复杂的声学环境中选择性地关注感兴趣的声源的非凡能力，通常被称为鸡尾酒会场景。为了在机器中复制这种显著的听觉注意能力，已经开发了目标说话人提取（TSE）模型。这些模型利用目标说话者的预先注册的线索来提取感兴趣的声源。然而，这些模型的有效性是在现实世界中的情况下，由于潜在的变化，甚至没有预先注册的线索受到阻碍。为了解决这个问题，本研究探讨了自然语言的整合，以提高现有的TSE模型的灵活性和可控性。具体来说，我们提出了一个名为LLM-TSE的模型，其中一个大的语言模型（LLM），以提取有用的语义线索，从用户的键入文本输入，它可以补充预先注册的线索或独立工作，以控制TSE过程。我们的实验结果表明，只有基于文本的线索时，具有竞争力的性能，一个新的国家的最先进的设置时，结合预先注册的声学线索。据我们所知，这是第一个成功地结合了基于文本的线索来指导目标说话人提取的工作，这可以成为鸡尾酒会问题研究的基石。摘要：Humans possess an extraordinary ability to selectively focus on the sound source of interest amidst complex acoustic environments, commonly referred to as cocktail party scenarios. In an attempt to replicate this remarkable auditory attention capability in machines, target speaker extraction (TSE) models have been developed. These models leverage the pre-registered cues of the target speaker to extract the sound source of interest. However, the effectiveness of these models is hindered in real-world scenarios due to the potential variation or even absence of pre-registered cues. To address this limitation, this study investigates the integration of natural language to enhance the flexibility and controllability of existing TSE models. Specifically, we propose a model named LLM-TSE, wherein a large language model (LLM) to extract useful semantic cues from the user's typed text input, which can complement the pre-registered cues or work independently to control the TSE process. Our experimental results demonstrate competitive performance when only text-based cues are presented, and a new state-of-the-art is set when combined with pre-registered acoustic cues. To the best of our knowledge, this is the first work that has successfully incorporated text-based cues to guide target speaker extraction, which can be a cornerstone for cocktail party problem research.
【6】 Audio-Visual Neural Syntax Acquisition标题：视听神经句法习得链接：https://arxiv.org/abs/2310.07654
作者：Cheng-I Jeff Lai,Freda Shi,Puyuan Peng,Yoon Kim,Kevin Gimpel,Shiyu Chang,Yung-Sung Chuang,Saurabhchand Bhati,David Cox,David Harwath,Yang Zhang,Karen Livescu,James Glass机构：TTIC, UT Austin, UC Santa Barbara, MIT-IBM Watson AI Lab摘要：我们研究短语结构归纳从视觉接地的讲话。其核心思想是首先将语音波形分割成词段序列，然后使用推断的段级连续表示来诱导短语结构。我们提出了视听神经网络学习器（AV-NSL），它通过听音频和看图像来学习短语结构，而不需要接触文本。通过对配对图像和语音字幕进行训练，AV-NSL能够推断出有意义的短语结构，这些短语结构与英语和德语的自然监督文本解析器得出的短语结构相当。我们的研究结果扩展了先前在无监督语言习得中从语音和接地语法归纳的工作，并提出了一种弥合这两个主题之间差距的方法。摘要：We study phrase structure induction from visually-grounded speech. The core idea is to first segment the speech waveform into sequences of word segments, and subsequently induce phrase structure using the inferred segment-level continuous representations. We present the Audio-Visual Neural Syntax Learner (AV-NSL) that learns phrase structure by listening to audio and looking at images, without ever being exposed to text. By training on paired images and spoken captions, AV-NSL exhibits the capability to infer meaningful phrase structures that are comparable to those derived by naturally-supervised text parsers, for both English and German. Our findings extend prior work in unsupervised language acquisition from speech and grounded grammar induction, and present one approach to bridge the gap between the two topics.
【7】 Adapting the adapters for code-switching in multilingual ASR标题：多语种ASR中代码转换适配器的适配链接：https://arxiv.org/abs/2310.07423
作者：Atharva Kulkarni,Ajinkya Kulkarni,Miguel Couceiro,Hanan Aldarmaki机构：∗ MBZUAI, UAE, † Universit´e de Lorraine, CNRS, LORIA, France, ‡Erisha Labs, India备注：Submitted to ICASSP 2024摘要：最近，大型预训练的多语言语音模型已经显示出将自动语音识别（ASR）扩展到许多低资源语言的潜力。这些模型中的一些在其公式化中使用语言适配器，这有助于提高单语性能，并避免在资源丰富的语言上进行多语言建模的一些缺点。然而，这种提法限制了代码转换语音，其中两种语言混合在一起，在同一话语这些模型的可用性。在这项工作中，我们提出了一些方法来有效地微调这种模型的代码切换语音，通过同化信息，从两种语言适配器在网络中的每个语言适应点。我们还将代码转换建模为一系列潜在的二进制序列，这些序列可用于在框架级别引导来自每个语言适配器的信息流。所提出的方法进行了评估，包括阿拉伯语，普通话，印地语与英语配对的三个代码转换数据集，在所有测试集的CER至少有10%的绝对减少，在代码转换性能的一致改善。摘要：Recently, large pre-trained multilingual speech models have shown potential in scaling Automatic Speech Recognition (ASR) to many low-resource languages. Some of these models employ language adapters in their formulation, which helps to improve monolingual performance and avoids some of the drawbacks of multi-lingual modeling on resource-rich languages. However, this formulation restricts the usability of these models on code-switched speech, where two languages are mixed together in the same utterance. In this work, we propose ways to effectively fine-tune such models on code-switched speech, by assimilating information from both language adapters at each language adaptation point in the network. We also model code-switching as a sequence of latent binary sequences that can be used to guide the flow of information from each language adapter at the frame level. The proposed approaches are evaluated on three code-switched datasets encompassing Arabic, Mandarin, and Hindi languages paired with English, showing consistent improvements in code-switching performance with at least 10\% absolute reduction in CER across all test sets.
【8】 DASpeech: Directed Acyclic Transformer for Fast and High-quality  Speech-to-Speech Translation标题：DASpeech：用于快速高质量语音到语音翻译的有向无环转换器链接：https://arxiv.org/abs/2310.07403
作者：Qingkai Fang,Yan Zhou,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：NeurIPS 2023. Audio samples are available at this https URL摘要：None摘要：Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS Fr-En benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to 18.53x speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.
【9】 Investigating the Effect of Language Models in Sequence Discriminative  Training for Neural Transducers标题：语言模型在神经传感器顺序辨别训练中的作用研究链接：https://arxiv.org/abs/2310.07345
作者：Zijian Yang,Wei Zhou,Ralf Schlüter,Hermann Ney机构：Machine Learning and Human Language Technology, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：accepted at ASRU 2023摘要：在这项工作中，我们调查的效果语言模型（LM）与不同的上下文长度和标签单位（音素与。字）用于基于音素的神经换能器的序列判别训练。无格和N-最佳列表的方法进行检查。对于具有音素级LM的无格方法，我们提出了一种方法来近似上下文历史，以采用具有全上下文依赖性的LM。这种近似可以扩展到任意的上下文长度，并使字级LM的无格方法的使用。此外，系统地比较了无格和基于N-最佳列表的方法。实验结果表明，词级LM的训练效果优于音素级LM。此外，我们发现用于概率计算的LM的上下文大小对性能的影响有限。此外，我们的研究结果揭示了序列判别训练中假设空间质量的关键重要性。摘要：In this work, we investigate the effect of language models (LMs) with different context lengths and label units (phoneme vs. word) used in sequence discriminative training for phoneme-based neural transducers. Both lattice-free and N-best-list approaches are examined. For lattice-free methods with phoneme-level LMs, we propose a method to approximate the context history to employ LMs with full-context dependency. This approximation can be extended to arbitrary context length and enables the usage of word-level LMs in lattice-free methods. Moreover, a systematic comparison is conducted across lattice-free and N-best-list-based methods. Experimental results on Librispeech show that using the word-level LM in training outperforms the phoneme-level LM. Besides, we find that the context size of the LM used for probability computation has a limited effect on performance. Moreover, our results reveal the pivotal importance of the hypothesis space quality in sequence discriminative training.
【10】 Enhancing expressivity transfer in textless speech-to-speech translation标题：增强无文本语音翻译中的表现力传递链接：https://arxiv.org/abs/2310.07279
作者：Jarod Duret,Benjamin O'Brien,Yannick Estève,Titouan Parcollet机构：LIA - Avignon Universite, France, University of Cambridge, United-Kingdom备注：None摘要：None摘要：Textless speech-to-speech translation systems are rapidly advancing, thanks to the integration of self-supervised learning techniques. However, existing state-of-the-art systems fall short when it comes to capturing and transferring expressivity accurately across different languages. Expressivity plays a vital role in conveying emotions, nuances, and cultural subtleties, thereby enhancing communication across diverse languages. To address this issue this study presents a novel method that operates at the discrete speech unit level and leverages multilingual emotion embeddings to capture language-agnostic information. Specifically, we demonstrate how these embeddings can be used to effectively predict the pitch and duration of speech units in the target language. Through objective and subjective experiments conducted on a French-to-English translation task, our findings highlight the superior expressivity transfer achieved by our approach compared to current state-of-the-art systems.
【11】 Vec-Tok Speech: speech vectorization and tokenization for neural speech  generation标题：VEC-TOK语音：用于神经语音生成的语音矢量化和标记化链接：https://arxiv.org/abs/2310.07246
作者：Xinfa Zhu,Yuanjun Lv,Yi Lei,Tao Li,Wendi He,Hongbin Zhou,Heng Lu,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Ximalaya Inc., China备注：15 pages, 2 figures摘要：语言模型（LM）最近在自然语言处理和计算机视觉中蓬勃发展，在各种任务中生成高保真文本或图像。相比之下，目前的语音生成模型仍然在语音质量和任务泛化方面苦苦挣扎。本文介绍了Vec-Tok语音，一个可扩展的框架，类似于多个语音生成任务，生成表达和高保真语音。具体来说，我们提出了一种新的语音编码器的语音矢量和语义令牌的基础上。语音向量包含有助于高保真语音重建的声学细节，而语义令牌则专注于语音的语言内容，便于语言建模。Vec-Tok Speech基于所提出的语音编解码器，利用LM来承担语音生成的核心。此外，字节对编码（BPE）的引入，以减少令牌的长度和比特率，以降低曝光偏差和更长的上下文覆盖，提高LM的性能。Vec-Tok Speech可用于语言内和跨语言zero-shot语音转换（VC）、零触发说话风格转换文本到语音（TTS）、语音到语音翻译（S2 ST）、语音去噪以及说话者去识别和匿名化。实验表明，Vec-Tok语音，建立在5万小时的语音，性能优于其他SOTA模型。代码将在https://github.com/BakerBunker/VecTok上提供。摘要：Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-fidelity texts or images in various tasks. In contrast, the current speech generative models are still struggling regarding speech quality and task generalization. This paper presents Vec-Tok Speech, an extensible framework that resembles multiple speech generation tasks, generating expressive and high-fidelity speech. Specifically, we propose a novel speech codec based on speech vectors and semantic tokens. Speech vectors contain acoustic details contributing to high-fidelity speech reconstruction, while semantic tokens focus on the linguistic content of speech, facilitating language modeling. Based on the proposed speech codec, Vec-Tok Speech leverages an LM to undertake the core of speech generation. Moreover, Byte-Pair Encoding (BPE) is introduced to reduce the token length and bit rate for lower exposure bias and longer context coverage, improving the performance of LMs. Vec-Tok Speech can be used for intra- and cross-lingual zero-shot voice conversion (VC), zero-shot speaking style transfer text-to-speech (TTS), speech-to-speech translation (S2ST), speech denoising, and speaker de-identification and anonymization. Experiments show that Vec-Tok Speech, built on 50k hours of speech, performs better than other SOTA models. Code will be available at https://github.com/BakerBunker/VecTok .
【12】 Psychoacoustic Challenges Of Speech Enhancement On VoIP Platforms标题：VoIP平台上语音增强的心理声学挑战链接：https://arxiv.org/abs/2310.07161
作者：Joseph Konan,Ojas Bhargave,Shikhar Agnihotri,Shuo Han,Yunyang Zeng,Ankit Shah,Bhiksha Raj机构：KonanAI ,Carnegie Mellon University摘要：在VoIP（互联网协议语音）电信的范围内，由声学变换引入的复杂性值得进行严格的分析。这项研究植根于专有的发送方去噪效果的探索，仔细评估了Google Meets和Zoom等平台。该研究借鉴了深度噪声抑制（DNS）2020数据集，确保针对各种去噪设置和接收器接口进行结构化检查。通过瓦哈卡分解，传统上是一种计量经济学工具，在此重新利用来分析VoIP系统内的声学语音扰动，引入了一种新的方法。为了进一步研究这些转换的影响，心理声学指标，特别是PESQ和STOI，被用来提供对语音改变的全面理解。累积起来，所获得的见解强调了VoIP影响声学动力学的复杂景观。除了主要的调查结果，报告了大量的指标，扩大了研究范围。此外，时域和时频域语音增强模型的域外基准测试包括在内，从而提高了这种调查的深度和适用性。摘要：Within the ambit of VoIP (Voice over Internet Protocol) telecommunications, the complexities introduced by acoustic transformations merit rigorous analysis. This research, rooted in the exploration of proprietary sender-side denoising effects, meticulously evaluates platforms such as Google Meets and Zoom. The study draws upon the Deep Noise Suppression (DNS) 2020 dataset, ensuring a structured examination tailored to various denoising settings and receiver interfaces. A methodological novelty is introduced via the Oaxaca decomposition, traditionally an econometric tool, repurposed herein to analyze acoustic-phonetic perturbations within VoIP systems. To further ground the implications of these transformations, psychoacoustic metrics, specifically PESQ and STOI, were harnessed to furnish a comprehensive understanding of speech alterations. Cumulatively, the insights garnered underscore the intricate landscape of VoIP-influenced acoustic dynamics. In addition to the primary findings, a multitude of metrics are reported, extending the research purview. Moreover, out-of-domain benchmarking for both time and time-frequency domain speech enhancement models is included, thereby enhancing the depth and applicability of this inquiry.
【13】 LLark: A Multimodal Foundation Model for Music标题：Lark：音乐的多通道基础模型链接：https://arxiv.org/abs/2310.07160
作者：Josh Gardner,Simon Durand,Daniel Stoller,Rachel M. Bittner机构：University of Washington, Spotify摘要：音乐具有独特而复杂的结构，这对专家和现有的人工智能系统来说都是一个挑战，相对于其他形式的音频来说，这是一个独特的挑战。我们提出了LLark，一个用于音乐理解的调整的多模态模型。我们详细介绍了我们的数据集创建过程，其中包括增强各种开源音乐数据集的注释，并将其转换为统一的注释调整格式。我们为Lark提出了一个多模态架构，将音乐的预训练生成模型与预训练语言模型相结合。在对三种类型的任务（音乐理解，字幕和推理）的评估中，我们发现我们的模型在音乐理解的zero-shot泛化中匹配或优于现有的基线，并且人类在字幕和推理任务中与模型的反应高度一致。Lark完全是从开源音乐数据和模型中训练出来的，我们在发布本文时提供了我们的训练代码。其他结果和音频示例在https：//bit.ly/llark，我们的源代码在https://github.com/spotify-research/llark。摘要：Music has a unique and complex structure which is challenging for both expert humans and existing AI systems to understand, and presents unique challenges relative to other forms of audio. We present LLark, an instruction-tuned multimodal model for music understanding. We detail our process for dataset creation, which involves augmenting the annotations of diverse open-source music datasets and converting them to a unified instruction-tuning format. We propose a multimodal architecture for LLark, integrating a pretrained generative model for music with a pretrained language model. In evaluations on three types of tasks (music understanding, captioning, and reasoning), we show that our model matches or outperforms existing baselines in zero-shot generalization for music understanding, and that humans show a high degree of agreement with the model's responses in captioning and reasoning tasks. LLark is trained entirely from open-source music data and models, and we make our training code available along with the release of this paper. Additional results and audio examples are at https://bit.ly/llark, and our source code is available at https://github.com/spotify-research/llark .
【14】 Acoustic Model Fusion for End-to-end Speech Recognition标题：端到端语音识别的声学模型融合链接：https://arxiv.org/abs/2310.07062
作者：Zhihong Lei,Mingbin Xu,Shiyi Han,Leo Liu,Zhen Huang,Tim Ng,Yuanyuan Zhang,Ernest Pusateri,Mirko Hannemann,Yaqiao Deng,Man-Hung Siu机构：Apple摘要：深度学习和自动语音识别（ASR）的最新进展使端到端（E2 E）ASR系统成为可能，并将准确性提升到一个新的水平。E2 E系统隐式地在基于音频-文本对训练的单个网络中对所有常规ASR组件（诸如声学模型（AM）和语言模型（LM））进行建模。尽管这种简单的系统架构，融合一个单独的LM，专门训练的文本语料库，到E2 E系统已被证明是有益的。然而，LM融合的应用存在某些缺点，例如其不能解决内部AM固有的域失配问题。从LM融合的概念中获得灵感，我们提出将外部AM集成到E2 E系统中，以更好地解决域不匹配问题。通过实施这种新颖的方法，我们已经实现了显着降低单词错误率，在不同的测试集上，令人印象深刻的下降高达14.3%。我们还发现，这种AM融合方法在增强命名实体识别方面特别有益。摘要：Recent advances in deep learning and automatic speech recognition (ASR) have enabled the end-to-end (E2E) ASR system and boosted the accuracy to a new level. The E2E systems implicitly model all conventional ASR components, such as the acoustic model (AM) and the language model (LM), in a single network trained on audio-text pairs. Despite this simpler system architecture, fusing a separate LM, trained exclusively on text corpora, into the E2E system has proven to be beneficial. However, the application of LM fusion presents certain drawbacks, such as its inability to address the domain mismatch issue inherent to the internal AM. Drawing inspiration from the concept of LM fusion, we propose the integration of an external AM into the E2E system to better address the domain mismatch. By implementing this novel approach, we have achieved a significant reduction in the word error rate, with an impressive drop of up to 14.3% across varied test sets. We also discovered that this AM fusion approach is particularly beneficial in enhancing named entity recognition.
【15】 Neural Harmonium: An Interpretable Deep Structure for Nonlinear Dynamic  System Identification with Application to Audio Processing标题：神经谐和：一种可解释的非线性动态系统辨识深层结构及其在音频处理中的应用链接：https://arxiv.org/abs/2310.07032
作者：Karim Helwani,Erfan Soltanmohammadi,Michael M. Goodwin摘要：最近，提高深度神经网络的可解释性受到了越来越多的关注，特别是当深度学习的力量被用来解决物理问题时。可解释性帮助我们理解模型的泛化能力和揭示其局限性。在本文中，我们介绍了一个因果可解释的深层结构建模动态系统。我们提出的模型利用谐波分析建模的系统在时间-频率域，同时保持高的时间和频谱分辨率。此外，该模型是建立在一个顺序递归的方式，允许快速，鲁棒性和精确的二阶优化，而不需要一个明确的海森计算。为了避免我们的系统的构建块的高维度，神经网络的设计，以确定频率的相互依赖性。所提出的模型说明和验证非线性系统识别问题所需的音频信号处理任务。众包实验对比所提出的方法的性能，以其他国家的最先进的解决方案上的声学回声消除方案证实了我们的方法在现实生活中的应用的有效性。摘要：Improving the interpretability of deep neural networks has recently gained increased attention, especially when the power of deep learning is leveraged to solve problems in physics. Interpretability helps us understand a model's ability to generalize and reveal its limitations. In this paper, we introduce a causal interpretable deep structure for modeling dynamic systems. Our proposed model makes use of the harmonic analysis by modeling the system in a time-frequency domain while maintaining high temporal and spectral resolution. Moreover, the model is built in an order recursive manner which allows for fast, robust, and exact second order optimization without the need for an explicit Hessian calculation. To circumvent the resulting high dimensionality of the building blocks of our system, a neural network is designed to identify the frequency interdependencies. The proposed model is illustrated and validated on nonlinear system identification problems as required for audio signal processing tasks. Crowd-sourced experimentation contrasting the performance of the proposed approach to other state-of-the-art solutions on an acoustic echo cancellation scenario confirms the effectiveness of our method for real-life applications.
【16】 Prosody Analysis of Audiobooks标题：有声读物的韵律分析链接：https://arxiv.org/abs/2310.06930
作者：Charuta Pethe,Yunting Yin,Steven Skiena机构：Department of Computer Science, Stony Brook University摘要：文本到语音的最新进展使得从文本生成听起来自然的音频成为可能。然而，有声读物的叙述涉及读者戏剧性的发声和语调，更依赖于叙述中的情感，对话和描述。使用我们的数据集93对齐的书-有声读物对，我们提出了改进的模型韵律预测属性（音高，音量和语音速率）从叙事文本使用语言建模。我们预测的韵律属性与人类有声读物阅读的相关性要比最先进的商业TTS系统的结果好得多：对于24本书中的22本，我们预测的音高显示出与人类阅读的更高相关性，而对于24本书中的23本，我们预测的音量属性证明更类似于人类阅读。最后，我们提出了一个人的评估研究，量化的程度，人们更喜欢韵律增强有声读物的商业文本到语音系统。摘要：Recent advances in text-to-speech have made it possible to generate natural-sounding audio from text. However, audiobook narrations involve dramatic vocalizations and intonations by the reader, with greater reliance on emotions, dialogues, and descriptions in the narrative. Using our dataset of 93 aligned book-audiobook pairs, we present improved models for prosody prediction properties (pitch, volume, and rate of speech) from narrative text using language modeling. Our predicted prosody attributes correlate much better with human audiobook readings than results from a state-of-the-art commercial TTS system: our predicted pitch shows a higher correlation with human reading for 22 out of the 24 books, while our predicted volume attribute proves more similar to human reading for 23 out of the 24 books. Finally, we present a human evaluation study to quantify the extent that people prefer prosody-enhanced audiobook readings over commercial text-to-speech systems.
机器翻译，仅供参考