今日论文合集：cs.SD语音9篇，eess.AS音频处理10篇。
cs.SD语音
【1】 Server-side Rescoring of Spoken Entity-centric Knowledge Queries for  Virtual Assistants标题：针对虚拟助理的以实体为中心的口头知识查询的服务器端重新评分链接：https://arxiv.org/abs/2311.01398
作者：Youyuan Zhang,Sashank Gondala,Thiago Fraga-Silva,Christophe Van Gysel摘要：由自动语音识别（ASR）支持的设备上虚拟助理（VA）需要有效的知识集成，以实现具有挑战性的实体丰富的查询识别。在本文中，我们进行了一项实证研究的建模策略，服务器端rescoring口语信息域查询使用各种类别的语言模型（LM）（N元词LM，子词神经LM）。我们研究了设备上和服务器端信号的组合，并通过集成各种服务器端LM，与仅在设备上执行ASR相比，在各种以实体为中心的查询子群体上展示了23%-35%的显著WER改进。我们还比较了在领域数据上训练的LM和OpenAI提供的GPT-3变体作为基线。此外，我们还表明，从零开始训练的多个服务器端LM的模型融合最有效地结合了每个模型的互补优势，并将从特定领域数据中学到的知识集成到VA ASR系统中。摘要：On-device Virtual Assistants (VAs) powered by Automatic Speech Recognition (ASR) require effective knowledge integration for the challenging entity-rich query recognition. In this paper, we conduct an empirical study of modeling strategies for server-side rescoring of spoken information domain queries using various categories of Language Models (LMs) (N-gram word LMs, sub-word neural LMs). We investigate the combination of on-device and server-side signals, and demonstrate significant WER improvements of 23%-35% on various entity-centric query subpopulations by integrating various server-side LMs compared to performing ASR on-device only. We also perform a comparison between LMs trained on domain data and a GPT-3 variant offered by OpenAI as a baseline. Furthermore, we also show that model fusion of multiple server-side LMs trained from scratch most effectively combines complementary strengths of each model and integrates knowledge learned from domain-specific data to a VA ASR system.
【2】 DistilWhisper: Efficient Distillation of Multi-task Speech Models via  Language-Specific Experts标题：DistilWhisper：通过特定语言专家高效地提取多任务语音模型链接：https://arxiv.org/abs/2311.01070
作者：Thomas Palmeira Ferraz,Marcely Zanon Boito,Caroline Brun,Vassilina Nikoulina备注：Submitted to IEEE ICASSP 2024 in September 2023摘要：Whisper是一个多任务和多语言语音模型，涵盖99种语言。它在其覆盖的语言子集中产生了值得称赞的自动语音识别（ASR）结果，但该模型在不可忽略的代表性不足的语言数量上仍然表现不佳，这一问题在较小的模型版本中加剧。在这项工作中，我们提出了DistilWhisper，一种能够弥合这些语言的ASR性能差距的方法，同时保留多任务和多语言功能的优势。我们的方法涉及两个关键策略：使用特定语言专家对whisper-small进行轻量级模块化ASR微调，以及从whisper-large-v2中提取知识。这种双重方法使我们能够有效地提高ASR性能，同时保持从多任务和多语言预训练中继承的鲁棒性。结果表明，我们的方法比标准的微调或LoRA适配器更有效，提高了目标语言的域内和域外测试集的性能，同时在推理时只引入了可以忽略不计的参数开销。摘要：Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still under-performs on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we propose DistilWhisper, an approach able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.
【3】 Video2Music: Suitable Music Generation from Videos using an Affective  Multimodal Transformer model标题：视频2Music：使用情感多模式转换器模型从视频中生成合适的音乐链接：https://arxiv.org/abs/2311.00968
作者：Jaeyong Kang,Soujanya Poria,Dorien Herremans摘要：在音乐生成领域的许多研究已经证明了令人印象深刻的性能，但几乎没有模型能够直接生成音乐来匹配伴随的视频。在这项工作中，我们开发了一个生成音乐AI框架Video 2 Music，它可以匹配提供的视频。我们首先策划了一个独特的音乐视频收藏。然后，我们分析了音乐视频，以获得语义，场景偏移，运动和情感特征。这些独特的功能，然后作为指导输入到我们的音乐生成模型。我们将音频文件转录为音符和和弦，并提取音符密度和响度等特征。这导致了一个丰富的多模态数据集，称为MuVi-Sync，我们训练一个新的情感多模态Transformer（AMT）模型来生成给定视频的音乐。该模型包括一种新的机制，以加强视频和音乐之间的情感相似性。最后，基于基于biGRU的回归模型执行后处理，以基于视频特征来估计音符密度和响度。这确保了所生成的和弦具有不同的节奏和音量的动态渲染。在一个彻底的实验中，我们表明，我们提出的框架可以生成音乐，匹配的视频内容的情感。在用户研究中确认了音乐质量以及音乐视频匹配的质量。所提出的AMT模型以及新的MuVi-Sync数据集为视频音乐生成的新任务迈出了有希望的一步。摘要：Numerous studies in the field of music generation have demonstrated impressive performance, yet virtually no models are able to directly generate music to match accompanying videos. In this work, we develop a generative music AI framework, Video2Music, that can match a provided video. We first curated a unique collection of music videos. Then, we analysed the music videos to obtain semantic, scene offset, motion, and emotion features. These distinct features are then employed as guiding input to our music generation model. We transcribe the audio files into MIDI and chords, and extract features such as note density and loudness. This results in a rich multimodal dataset, called MuVi-Sync, on which we train a novel Affective Multimodal Transformer (AMT) model to generate music given a video. This model includes a novel mechanism to enforce affective similarity between video and music. Finally, post-processing is performed based on a biGRU-based regression model to estimate note density and loudness based on the video features. This ensures a dynamic rendering of the generated chords with varying rhythm and volume. In a thorough experiment, we show that our proposed framework can generate music that matches the video content in terms of emotion. The musical quality, along with the quality of music-video matching is confirmed in a user study. The proposed AMT model, along with the new MuVi-Sync dataset, presents a promising step for the new task of music generation for videos.
【4】 E3 TTS: Easy End-to-End Diffusion-based Text to Speech标题：E3 TTS：简单的基于端到端扩散的文本到语音链接：https://arxiv.org/abs/2311.00945
作者：Yuan Gao,Nobuyuki Morioka,Yu Zhang,Nanxin Chen备注：Accepted by ASRU 2023摘要：我们提出了一种简单有效的基于扩散的端到端文本到语音转换模型--Easy End-to-End Diffusion-based Text to Speech。E3 TTS直接将纯文本作为输入，并通过迭代细化过程生成音频波形。与许多先前的工作不同，E3 TTS不依赖于任何中间表示，如频谱图特征或对齐信息。相反，E3 TTS通过扩散过程对波形的时间结构进行建模。在不依赖于附加条件信息的情况下，E3 TTS可以支持给定音频内的灵活潜在结构。这使得E3 TTS能够轻松地适应zero-shot任务，例如编辑，而无需任何额外的培训。实验表明，E3语音合成系统能够产生高保真的语音，接近目前最先进的神经语音合成系统的性能。音频样本可在https://e3tts.github.io上获得。摘要：We propose Easy End-to-End Diffusion-based Text to Speech, a simple and efficient end-to-end text-to-speech model based on diffusion. E3 TTS directly takes plain text as input and generates an audio waveform through an iterative refinement process. Unlike many prior work, E3 TTS does not rely on any intermediate representations like spectrogram features or alignment information. Instead, E3 TTS models the temporal structure of the waveform through the diffusion process. Without relying on additional conditioning information, E3 TTS could support flexible latent structure within the given audio. This enables E3 TTS to be easily adapted for zero-shot tasks such as editing without any additional training. Experiments show that E3 TTS can generate high-fidelity audio, approaching the performance of a state-of-the-art neural TTS system. Audio samples are available at https://e3tts.github.io.
【5】 On The Open Prompt Challenge In Conditional Audio Generation标题：条件音频生成中的开放提示挑战链接：https://arxiv.org/abs/2311.00897
作者：Ernie Chang,Sidd Srinivasan,Mahi Luthra,Pin-Jie Lin,Varun Nagaraja,Forrest Iandola,Zechun Liu,Zhaoheng Ni,Changsheng Zhao,Yangyang Shi,Vikas Chandra备注：5 pages, 3 figures, 4 tables摘要：文本到音频生成（TTA）从文本描述中生成音频，从成对的音频样本和手工注释的文本中学习。然而，商业化的音频生成是具有挑战性的，因为与用于训练TTA模型的文本描述相比，用户输入提示通常是指定不足的。在这项工作中，我们将TTA模型视为“黑盒”，并通过两个关键见解解决用户提示的挑战：（1）用户提示通常未被指定，导致用户提示和培训提示之间存在很大的对齐差距。(2)有一个音频描述的分布，TTA模型更擅长生成更高质量的音频，我们称之为“audiones”。为此，我们重写提示与调整模型，并建议利用文本音频对齐作为反馈信号，通过利润排名学习音频改进。在客观和主观的人类评估中，我们观察到文本音频对齐和音乐音频质量都有显着改善。摘要：Text-to-audio generation (TTA) produces audio from a text description, learning from pairs of audio samples and hand-annotated text. However, commercializing audio generation is challenging as user-input prompts are often under-specified when compared to text descriptions used to train TTA models. In this work, we treat TTA models as a ``blackbox'' and address the user prompt challenge with two key insights: (1) User prompts are generally under-specified, leading to a large alignment gap between user prompts and training prompts. (2) There is a distribution of audio descriptions for which TTA models are better at generating higher quality audio, which we refer to as ``audionese''. To this end, we rewrite prompts with instruction-tuned models and propose utilizing text-audio alignment as feedback signals via margin ranking learning for audio improvements. On both objective and subjective human evaluations, we observed marked improvements in both text-audio alignment and music audio quality.
【6】 In-Context Prompt Editing For Conditional Audio Generation标题：用于条件音频生成的上下文提示编辑链接：https://arxiv.org/abs/2311.00895
作者：Ernie Chang,Pin-Jie Lin,Yang Li,Sidd Srinivasan,Gael Le Lan,David Kant,Yangyang Shi,Forrest Iandola,Vikas Chandra备注：5 pages, 3 figures, 2 tables摘要：分布转移是部署机器学习模型的一个核心挑战，因为它们可能无法适应真实世界的数据。这在文本到音频生成中特别明显，其中编码表示很容易被看不见的提示破坏，这导致生成的音频的降级-有限的文本-音频对集合仍然不足以用于野外的条件音频生成，因为用户提示未被指定。特别是，我们观察到一个一致的音频质量下降，在生成的音频样本与用户提示，而不是训练集提示。为此，我们提出了一个基于检索的上下文提示编辑框架，利用培训字幕作为示范范例，重新访问用户提示。我们表明，该框架提高了音频质量的一组收集的用户提示，这是编辑参考的训练字幕作为范例。摘要：Distributional shift is a central challenge in the deployment of machine learning models as they can be ill-equipped for real-world data. This is particularly evident in text-to-audio generation where the encoded representations are easily undermined by unseen prompts, which leads to the degradation of generated audio -- the limited set of the text-audio pairs remains inadequate for conditional audio generation in the wild as user prompts are under-specified. In particular, we observe a consistent audio quality degradation in generated audio samples with user prompts, as opposed to training set prompts. To this end, we present a retrieval-based in-context prompt editing framework that leverages the training captions as demonstrative exemplars to revisit the user prompts. We show that the framework enhanced the audio quality across the set of collected user prompts, which were edited with reference to the training captions as exemplars.
【7】 Low-latency Real-time Voice Conversion on CPU标题：在CPU上实现低延迟实时语音转换链接：https://arxiv.org/abs/2311.00873
作者：Konstantine Sadov,Matthew Hutter,Asara Near备注：8 pages, 2 figures摘要：我们将以前的音频操作和生成神经网络的架构适应于实时任意对一语音转换的任务。我们得到的模型LLVC（$\textbf{L}$ow-latency $\textbf{L}$ow-resource $\textbf{V}$oice $\textbf{C}$onversion）在16 kHz的比特率下具有低于20 ms的延迟，并且在消费者CPU上的运行速度比实时快近2.8倍。LLVC使用生成对抗架构和知识蒸馏来实现这种性能。据我们所知，LLVC实现了任何开源语音转换模型的最低资源使用率和最低延迟。我们在https://github.com/KoeAI/LLVC上提供开源样本、代码和预训练模型权重。摘要：We adapt the architectures of previous audio manipulation and generation neural networks to the task of real-time any-to-one voice conversion. Our resulting model, LLVC ($\textbf{L}$ow-latency $\textbf{L}$ow-resource $\textbf{V}$oice $\textbf{C}$onversion), has a latency of under 20ms at a bitrate of 16kHz and runs nearly 2.8x faster than real-time on a consumer CPU. LLVC uses both a generative adversarial architecture as well as knowledge distillation in order to attain this performance. To our knowledge LLVC achieves both the lowest resource usage as well as the lowest latency of any open-source voice conversion model. We provide open-source samples, code, and pretrained model weights at https://github.com/KoeAI/LLVC.
【8】 Investigating Self-Supervised Deep Representations for EEG-based  Auditory Attention Decoding标题：基于EEG的听觉注意解码的自监督深度表征研究链接：https://arxiv.org/abs/2311.00814
作者：Karan Thakkar,Jiarui Hai,Mounya Elhilali备注：Submitted to ICASSP 2024摘要：听觉注意解码（AAD）算法在将具有挑战性的声学环境中的所需声源直接与大脑活动隔离方面发挥着至关重要的作用。虽然最近的研究表明，在AAD中使用听觉包络和声谱图等浅层表征是有希望的，但在更大范围内对深层自我监督（SS）表征的探索有限。在这项研究中，我们对线性解码器在12种深度和2种浅层表示中的性能进行了全面的研究，这些表示应用于来自57个受试者和多种语言的多项研究的EEG数据。我们的实验结果一致地揭示了AAD在解码背景扬声器时深度特征的优越性，无论数据集和分析窗口如何。这一结果表明，使用深度非线性特征揭示的大脑中无人值守信号的可能非线性编码。此外，我们分析了不同层次的SS表示和窗口大小对AAD性能的影响。这些发现强调了通过整合深度特征表示来增强基于EEG的AAD系统的潜力。摘要：Auditory Attention Decoding (AAD) algorithms play a crucial role in isolating desired sound sources within challenging acoustic environments directly from brain activity. Although recent research has shown promise in AAD using shallow representations such as auditory envelope and spectrogram, there has been limited exploration of deep Self-Supervised (SS) representations on a larger scale. In this study, we undertake a comprehensive investigation into the performance of linear decoders across 12 deep and 2 shallow representations, applied to EEG data from multiple studies spanning 57 subjects and multiple languages. Our experimental results consistently reveal the superiority of deep features for AAD at decoding background speakers, regardless of the datasets and analysis windows. This result indicates possible nonlinear encoding of unattended signals in the brain that are revealed using deep nonlinear features. Additionally, we analyze the impact of different layers of SS representations and window sizes on AAD performance. These findings underscore the potential for enhancing EEG-based AAD systems through the integration of deep feature representations.
【9】 Expressive TTS Driven by Natural Language Prompts Using Few Human  Annotations标题：自然语言驱动的表达型TTS链接：https://arxiv.org/abs/2311.01260
作者：Hanglei Zhang,Yiwei Guo,Sen Liu,Xie Chen,Kai Yu备注：5 pages,3 figures, submitted to ICASSP 2024摘要：表达性文本到语音（TTS）的目标是合成语音与人类一样的音调，情绪，甚至艺术属性。表达性TTS的最新进展使用户能够通过自然语言提示直接控制合成风格。然而，这些方法通常需要使用大量的样式注释数据进行过度训练，这可能具有挑战性。此外，由于固定的样式注释，它们可能具有有限的适应性。在这项工作中，我们提出了FreeStyleTTS（FS-TTS），一个可控的表达TTS模型与最小的人类注释。我们的方法利用了一个大的语言模型（LLM）表达TTS转换成一个风格检索任务。LLM基于外部样式提示从注释的话语中选择最匹配的样式引用，所述外部样式提示可以是原始输入文本或自然语言样式描述。选定的参考指导TTS管道合成具有预期风格的语音。这种创新的方法提供了灵活、多功能和精确的风格控制，最大限度地减少了人工工作量。普通话讲故事语料库上的实验表明，FS-TTS的熟练程度，利用LLM的语义推理能力，检索所需的风格从输入文本或用户定义的描述。这导致合成的语音与指定的风格紧密结合。摘要：Expressive text-to-speech (TTS) aims to synthesize speeches with human-like tones, moods, or even artistic attributes. Recent advancements in expressive TTS empower users with the ability to directly control synthesis style through natural language prompts. However, these methods often require excessive training with a significant amount of style-annotated data, which can be challenging to acquire. Moreover, they may have limited adaptability due to fixed style annotations. In this work, we present FreeStyleTTS (FS-TTS), a controllable expressive TTS model with minimal human annotations. Our approach utilizes a large language model (LLM) to transform expressive TTS into a style retrieval task. The LLM selects the best-matching style references from annotated utterances based on external style prompts, which can be raw input text or natural language style descriptions. The selected reference guides the TTS pipeline to synthesize speeches with the intended style. This innovative approach provides flexible, versatile, and precise style control with minimal human workload. Experiments on a Mandarin storytelling corpus demonstrate FS-TTS's proficiency in leveraging LLM's semantic inference ability to retrieve desired styles from either input text or user-defined descriptions. This results in synthetic speeches that are closely aligned with the specified styles.
eess.AS音频处理
【1】 Expressive TTS Driven by Natural Language Prompts Using Few Human  Annotations标题：自然语言提示使用少量人工标注驱动的表现型TTS链接：https://arxiv.org/abs/2311.01260
作者：Hanglei Zhang,Yiwei Guo,Sen Liu,Xie Chen,Kai Yu备注：5 pages,3 figures, submitted to ICASSP 2024摘要：None摘要：Expressive text-to-speech (TTS) aims to synthesize speeches with human-like tones, moods, or even artistic attributes. Recent advancements in expressive TTS empower users with the ability to directly control synthesis style through natural language prompts. However, these methods often require excessive training with a significant amount of style-annotated data, which can be challenging to acquire. Moreover, they may have limited adaptability due to fixed style annotations. In this work, we present FreeStyleTTS (FS-TTS), a controllable expressive TTS model with minimal human annotations. Our approach utilizes a large language model (LLM) to transform expressive TTS into a style retrieval task. The LLM selects the best-matching style references from annotated utterances based on external style prompts, which can be raw input text or natural language style descriptions. The selected reference guides the TTS pipeline to synthesize speeches with the intended style. This innovative approach provides flexible, versatile, and precise style control with minimal human workload. Experiments on a Mandarin storytelling corpus demonstrate FS-TTS's proficiency in leveraging LLM's semantic inference ability to retrieve desired styles from either input text or user-defined descriptions. This results in synthetic speeches that are closely aligned with the specified styles.
【2】 Automatic Disfluency Detection from Untranscribed Speech标题：从未转录的语音中自动检测不流利性链接：https://arxiv.org/abs/2311.00867
作者：Amrit Romana,Kazuhito Koishida,Emily Mower Provost摘要：言语不流利，如充满停顿或重复，是在典型的语音流中断。口吃是一种言语障碍，其特征是不流利率高，但所有个体说话都有一些不流利，并且不流利率可能会因认知负荷等因素而增加。临床上，自动不流利检测可能有助于口吃患者的治疗计划。在诊所之外，自动不流利检测可以用作预处理步骤，以提高下游应用中的自然语言理解。考虑到这一广泛的应用，我们调查的语言，声学和多模态的方法帧级自动不流利检测和分类。这些方法中的每一种都依赖于音频作为输入。首先，我们评估几个自动语音识别（ASR）系统的能力，转录不流利，不流利的错误率。然后，我们使用这些ASR成绩单作为输入的语言为基础的不流利检测模型。我们发现，不流利的检测性能在很大程度上是有限的成绩单和比对的质量。我们发现，一个基于声学的方法，不需要转录作为中间步骤优于ASR语言的方法。最后，我们提出了多模态架构，我们发现提高不流利的检测性能的单峰方法。最后，这项工作介绍了新的方法，自动帧级不流利和分类。从长远来看，这将有助于研究人员将自动不流利检测纳入一系列应用中。摘要：Speech disfluencies, such as filled pauses or repetitions, are disruptions in the typical flow of speech. Stuttering is a speech disorder characterized by a high rate of disfluencies, but all individuals speak with some disfluencies and the rates of disfluencies may by increased by factors such as cognitive load. Clinically, automatic disfluency detection may help in treatment planning for individuals who stutter. Outside of the clinic, automatic disfluency detection may serve as a pre-processing step to improve natural language understanding in downstream applications. With this wide range of applications in mind, we investigate language, acoustic, and multimodal methods for frame-level automatic disfluency detection and categorization. Each of these methods relies on audio as an input. First, we evaluate several automatic speech recognition (ASR) systems in terms of their ability to transcribe disfluencies, measured using disfluency error rates. We then use these ASR transcripts as input to a language-based disfluency detection model. We find that disfluency detection performance is largely limited by the quality of transcripts and alignments. We find that an acoustic-based approach that does not require transcription as an intermediate step outperforms the ASR language approach. Finally, we present multimodal architectures which we find improve disfluency detection performance over the unimodal approaches. Ultimately, this work introduces novel approaches for automatic frame-level disfluency and categorization. In the long term, this will help researchers incorporate automatic disfluency detection into a range of applications.
【3】 Server-side Rescoring of Spoken Entity-centric Knowledge Queries for  Virtual Assistants标题：针对虚拟助理的以实体为中心的口头知识查询的服务器端重新评分链接：https://arxiv.org/abs/2311.01398
作者：Youyuan Zhang,Sashank Gondala,Thiago Fraga-Silva,Christophe Van Gysel摘要：由自动语音识别（ASR）支持的设备上虚拟助理（VA）需要有效的知识集成，以实现具有挑战性的实体丰富的查询识别。在本文中，我们进行了一项实证研究的建模策略，服务器端rescoring口语信息域查询使用各种类别的语言模型（LM）（N元词LM，子词神经LM）。我们研究了设备上和服务器端信号的组合，并通过集成各种服务器端LM，与仅在设备上执行ASR相比，在各种以实体为中心的查询子群体上展示了23%-35%的显著WER改进。我们还比较了在领域数据上训练的LM和OpenAI提供的GPT-3变体作为基线。此外，我们还表明，从零开始训练的多个服务器端LM的模型融合最有效地结合了每个模型的互补优势，并将从特定领域数据中学到的知识集成到VA ASR系统中。摘要：On-device Virtual Assistants (VAs) powered by Automatic Speech Recognition (ASR) require effective knowledge integration for the challenging entity-rich query recognition. In this paper, we conduct an empirical study of modeling strategies for server-side rescoring of spoken information domain queries using various categories of Language Models (LMs) (N-gram word LMs, sub-word neural LMs). We investigate the combination of on-device and server-side signals, and demonstrate significant WER improvements of 23%-35% on various entity-centric query subpopulations by integrating various server-side LMs compared to performing ASR on-device only. We also perform a comparison between LMs trained on domain data and a GPT-3 variant offered by OpenAI as a baseline. Furthermore, we also show that model fusion of multiple server-side LMs trained from scratch most effectively combines complementary strengths of each model and integrates knowledge learned from domain-specific data to a VA ASR system.
【4】 DistilWhisper: Efficient Distillation of Multi-task Speech Models via  Language-Specific Experts标题：DistilWhisper：通过特定语言专家高效地提取多任务语音模型链接：https://arxiv.org/abs/2311.01070
作者：Thomas Palmeira Ferraz,Marcely Zanon Boito,Caroline Brun,Vassilina Nikoulina备注：Submitted to IEEE ICASSP 2024 in September 2023摘要：Whisper是一个多任务和多语言语音模型，涵盖99种语言。它在其覆盖的语言子集中产生了值得称赞的自动语音识别（ASR）结果，但该模型在不可忽略的代表性不足的语言数量上仍然表现不佳，这一问题在较小的模型版本中加剧。在这项工作中，我们提出了DistilWhisper，一种能够弥合这些语言的ASR性能差距的方法，同时保留多任务和多语言功能的优势。我们的方法涉及两个关键策略：使用特定语言专家对whisper-small进行轻量级模块化ASR微调，以及从whisper-large-v2中提取知识。这种双重方法使我们能够有效地提高ASR性能，同时保持从多任务和多语言预训练中继承的鲁棒性。结果表明，我们的方法比标准的微调或LoRA适配器更有效，提高了目标语言的域内和域外测试集的性能，同时在推理时只引入了可以忽略不计的参数开销。摘要：Whisper is a multitask and multilingual speech model covering 99 languages. It yields commendable automatic speech recognition (ASR) results in a subset of its covered languages, but the model still under-performs on a non-negligible number of under-represented languages, a problem exacerbated in smaller model versions. In this work, we propose DistilWhisper, an approach able to bridge the performance gap in ASR for these languages while retaining the advantages of multitask and multilingual capabilities. Our approach involves two key strategies: lightweight modular ASR fine-tuning of whisper-small using language-specific experts, and knowledge distillation from whisper-large-v2. This dual approach allows us to effectively boost ASR performance while keeping the robustness inherited from the multitask and multilingual pre-training. Results demonstrate that our approach is more effective than standard fine-tuning or LoRA adapters, boosting performance in the targeted languages for both in- and out-of-domain test sets, while introducing only a negligible parameter overhead at inference.
【5】 Video2Music: Suitable Music Generation from Videos using an Affective  Multimodal Transformer model标题：视频2Music：使用情感多模式转换器模型从视频中生成合适的音乐链接：https://arxiv.org/abs/2311.00968
作者：Jaeyong Kang,Soujanya Poria,Dorien Herremans摘要：None摘要：Numerous studies in the field of music generation have demonstrated impressive performance, yet virtually no models are able to directly generate music to match accompanying videos. In this work, we develop a generative music AI framework, Video2Music, that can match a provided video. We first curated a unique collection of music videos. Then, we analysed the music videos to obtain semantic, scene offset, motion, and emotion features. These distinct features are then employed as guiding input to our music generation model. We transcribe the audio files into MIDI and chords, and extract features such as note density and loudness. This results in a rich multimodal dataset, called MuVi-Sync, on which we train a novel Affective Multimodal Transformer (AMT) model to generate music given a video. This model includes a novel mechanism to enforce affective similarity between video and music. Finally, post-processing is performed based on a biGRU-based regression model to estimate note density and loudness based on the video features. This ensures a dynamic rendering of the generated chords with varying rhythm and volume. In a thorough experiment, we show that our proposed framework can generate music that matches the video content in terms of emotion. The musical quality, along with the quality of music-video matching is confirmed in a user study. The proposed AMT model, along with the new MuVi-Sync dataset, presents a promising step for the new task of music generation for videos.
【6】 E3 TTS: Easy End-to-End Diffusion-based Text to Speech标题：E3 TTS：基于端到端扩散的简易文本到语音转换链接：https://arxiv.org/abs/2311.00945
作者：Yuan Gao,Nobuyuki Morioka,Yu Zhang,Nanxin Chen备注：Accepted by ASRU 2023摘要：None摘要：We propose Easy End-to-End Diffusion-based Text to Speech, a simple and efficient end-to-end text-to-speech model based on diffusion. E3 TTS directly takes plain text as input and generates an audio waveform through an iterative refinement process. Unlike many prior work, E3 TTS does not rely on any intermediate representations like spectrogram features or alignment information. Instead, E3 TTS models the temporal structure of the waveform through the diffusion process. Without relying on additional conditioning information, E3 TTS could support flexible latent structure within the given audio. This enables E3 TTS to be easily adapted for zero-shot tasks such as editing without any additional training. Experiments show that E3 TTS can generate high-fidelity audio, approaching the performance of a state-of-the-art neural TTS system. Audio samples are available at https://e3tts.github.io.
【7】 On The Open Prompt Challenge In Conditional Audio Generation标题：条件音频生成中的开放提示挑战链接：https://arxiv.org/abs/2311.00897
作者：Ernie Chang,Sidd Srinivasan,Mahi Luthra,Pin-Jie Lin,Varun Nagaraja,Forrest Iandola,Zechun Liu,Zhaoheng Ni,Changsheng Zhao,Yangyang Shi,Vikas Chandra备注：5 pages, 3 figures, 4 tables摘要：文本到音频生成（TTA）从文本描述中生成音频，从成对的音频样本和手工注释的文本中学习。然而，商业化的音频生成是具有挑战性的，因为与用于训练TTA模型的文本描述相比，用户输入提示通常是指定不足的。在这项工作中，我们将TTA模型视为“黑盒”，并通过两个关键见解解决用户提示的挑战：（1）用户提示通常未被指定，导致用户提示和培训提示之间存在很大的对齐差距。(2)有一个音频描述的分布，TTA模型更擅长生成更高质量的音频，我们称之为“audiones”。为此，我们重写提示与调整模型，并建议利用文本音频对齐作为反馈信号，通过利润排名学习音频改进。在客观和主观的人类评估中，我们观察到文本音频对齐和音乐音频质量都有显着改善。摘要：Text-to-audio generation (TTA) produces audio from a text description, learning from pairs of audio samples and hand-annotated text. However, commercializing audio generation is challenging as user-input prompts are often under-specified when compared to text descriptions used to train TTA models. In this work, we treat TTA models as a ``blackbox'' and address the user prompt challenge with two key insights: (1) User prompts are generally under-specified, leading to a large alignment gap between user prompts and training prompts. (2) There is a distribution of audio descriptions for which TTA models are better at generating higher quality audio, which we refer to as ``audionese''. To this end, we rewrite prompts with instruction-tuned models and propose utilizing text-audio alignment as feedback signals via margin ranking learning for audio improvements. On both objective and subjective human evaluations, we observed marked improvements in both text-audio alignment and music audio quality.
【8】 In-Context Prompt Editing For Conditional Audio Generation标题：用于条件音频生成的上下文提示编辑链接：https://arxiv.org/abs/2311.00895
作者：Ernie Chang,Pin-Jie Lin,Yang Li,Sidd Srinivasan,Gael Le Lan,David Kant,Yangyang Shi,Forrest Iandola,Vikas Chandra备注：5 pages, 3 figures, 2 tables摘要：分布转移是部署机器学习模型的一个核心挑战，因为它们可能无法适应真实世界的数据。这在文本到音频生成中特别明显，其中编码表示很容易被看不见的提示破坏，这导致生成的音频的降级-有限的文本-音频对集合仍然不足以用于野外的条件音频生成，因为用户提示未被指定。特别是，我们观察到一个一致的音频质量下降，在生成的音频样本与用户提示，而不是训练集提示。为此，我们提出了一个基于检索的上下文提示编辑框架，利用培训字幕作为示范范例，重新访问用户提示。我们表明，该框架提高了音频质量的一组收集的用户提示，这是编辑参考的训练字幕作为范例。摘要：Distributional shift is a central challenge in the deployment of machine learning models as they can be ill-equipped for real-world data. This is particularly evident in text-to-audio generation where the encoded representations are easily undermined by unseen prompts, which leads to the degradation of generated audio -- the limited set of the text-audio pairs remains inadequate for conditional audio generation in the wild as user prompts are under-specified. In particular, we observe a consistent audio quality degradation in generated audio samples with user prompts, as opposed to training set prompts. To this end, we present a retrieval-based in-context prompt editing framework that leverages the training captions as demonstrative exemplars to revisit the user prompts. We show that the framework enhanced the audio quality across the set of collected user prompts, which were edited with reference to the training captions as exemplars.
【9】 Low-latency Real-time Voice Conversion on CPU标题：在CPU上实现低延迟实时语音转换链接：https://arxiv.org/abs/2311.00873
作者：Konstantine Sadov,Matthew Hutter,Asara Near备注：8 pages, 2 figures摘要：None摘要：We adapt the architectures of previous audio manipulation and generation neural networks to the task of real-time any-to-one voice conversion. Our resulting model, LLVC ($\textbf{L}$ow-latency $\textbf{L}$ow-resource $\textbf{V}$oice $\textbf{C}$onversion), has a latency of under 20ms at a bitrate of 16kHz and runs nearly 2.8x faster than real-time on a consumer CPU. LLVC uses both a generative adversarial architecture as well as knowledge distillation in order to attain this performance. To our knowledge LLVC achieves both the lowest resource usage as well as the lowest latency of any open-source voice conversion model. We provide open-source samples, code, and pretrained model weights at https://github.com/KoeAI/LLVC.
【10】 Investigating Self-Supervised Deep Representations for EEG-based  Auditory Attention Decoding标题：研究用于基于EEG的听觉注意解码的自我监督深度表征链接：https://arxiv.org/abs/2311.00814
作者：Karan Thakkar,Jiarui Hai,Mounya Elhilali备注：Submitted to ICASSP 2024摘要：听觉注意解码（AAD）算法在将具有挑战性的声学环境中的所需声源直接与大脑活动隔离方面发挥着至关重要的作用。虽然最近的研究表明，在AAD中使用听觉包络和声谱图等浅层表征是有希望的，但在更大范围内对深层自我监督（SS）表征的探索有限。在这项研究中，我们对线性解码器在12种深度和2种浅层表示中的性能进行了全面的研究，这些表示应用于来自57个受试者和多种语言的多项研究的EEG数据。我们的实验结果一致地揭示了AAD在解码背景扬声器时深度特征的优越性，无论数据集和分析窗口如何。这一结果表明，使用深度非线性特征揭示的大脑中无人值守信号的可能非线性编码。此外，我们分析了不同层次的SS表示和窗口大小对AAD性能的影响。这些发现强调了通过整合深度特征表示来增强基于EEG的AAD系统的潜力。摘要：Auditory Attention Decoding (AAD) algorithms play a crucial role in isolating desired sound sources within challenging acoustic environments directly from brain activity. Although recent research has shown promise in AAD using shallow representations such as auditory envelope and spectrogram, there has been limited exploration of deep Self-Supervised (SS) representations on a larger scale. In this study, we undertake a comprehensive investigation into the performance of linear decoders across 12 deep and 2 shallow representations, applied to EEG data from multiple studies spanning 57 subjects and multiple languages. Our experimental results consistently reveal the superiority of deep features for AAD at decoding background speakers, regardless of the datasets and analysis windows. This result indicates possible nonlinear encoding of unattended signals in the brain that are revealed using deep nonlinear features. Additionally, we analyze the impact of different layers of SS representations and window sizes on AAD performance. These findings underscore the potential for enhancing EEG-based AAD systems through the integration of deep feature representations.
机器翻译，仅供参考