今日论文合集：cs.SD语音12篇，eess.AS音频处理15篇。【1】 Towards small and accurate convolutional neural networks for acoustic biodiversity monitoring
标题：用于声学生物多样性监测的小而准确的卷积神经网络
链接：http://arxiv.org/pdf/2312.03666v1
作者：Serge Zaugg,Mike van der Schaar,Florence Erbs,Antonio Sanchez,Joan V. Castell,Emiliano Ramallo,Michel André摘要：动物声音的自动分类是大规模监测生物多样性的先决条件。卷积神经网络（CNN）是最有前途的算法之一，但它们速度慢，通常在该领域的分类效果较差，并且通常需要大量的训练数据集。我们的目标是设计在推理时速度快的CNN，并在从中等规模的数据中学习时实现良好的分类性能。使用了来自雨林生态系统的记录。从20种鸟类的声音的开始和结束点手动注释。来自10秒片段的频谱图用作CNN输入。我们设计了具有频率解包层的简单CNN（SIMP-FU模型），使得任何输出单元都连接到所有频谱图频率，但仅连接到时间的子区域，即接收场（RF）。我们的模型允许使用不同的RF持续时间进行实验。模型要么使用编码声音的开始和结束点的时间索引标签，要么使用更简单的片段级标签。从时间索引标签学习的模型比它们的段级对应物表现得好得多。最好的分类性能，实现了与中间射频持续时间为1.5秒的模型。最好的SIMP-FU模型在测试集上的20个类别中的18个类别中实现了AUC超过0.95。在紧凑的低成本硬件上，最好的SIMP-FU模型的评估速度比实时数据采集快7倍。射频持续时间是分类性能的主要驱动因素。1.5s的最佳值与声音的持续时间相同。我们的模型在从中等规模的训练数据中学习的同时取得了良好的分类性能。这可以通过在训练期间使用时间索引标签和适当大小的RF来解释。结果证实了在紧凑型低成本设备上部署具有良好分类性能的小型CNN的可行性。摘要：Automated classification of animal sounds is a prerequisite for large-scale monitoring of biodiversity. Convolutional Neural Networks (CNNs) are among the most promising algorithms but they are slow, often achieve poor classification in the field and typically require large training data sets. Our objective was to design CNNs that are fast at inference time and achieve good classification performance while learning from moderate-sized data. Recordings from a rainforest ecosystem were used. Start and end-point of sounds from 20 bird species were manually annotated. Spectrograms from 10 second segments were used as CNN input. We designed simple CNNs with a frequency unwrapping layer (SIMP-FU models) such that any output unit was connected to all spectrogram frequencies but only to a sub-region of time, the Receptive Field (RF). Our models allowed experimentation with different RF durations. Models either used the time-indexed labels that encode start and end-point of sounds or simpler segment-level labels. Models learning from time-indexed labels performed considerably better than their segment-level counterparts. Best classification performances was achieved for models with intermediate RF duration of 1.5 seconds. The best SIMP-FU models achieved AUCs over 0.95 in 18 of 20 classes on the test set. On compact low-cost hardware the best SIMP-FU models evaluated up to seven times faster than real-time data acquisition. RF duration was a major driver of classification performance. The optimum of 1.5 s was in the same range as the duration of the sounds. Our models achieved good classification performance while learning from moderate-sized training data. This is explained by the usage of time-indexed labels during training and adequately sized RF. Results confirm the feasibility of deploying small CNNs with good classification performance on compact low-cost devices.
【2】 Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models标题：利用大型基础模型进行多模式数据和资源高效的设备定向语音检测链接：http://arxiv.org/pdf/2312.03632v1作者：Dominik Wagner,Alexander Churchill,Siddharth Sigtia,Panayiotis Georgiou,Matt Mirsamadi,Aarshee Mishra,Erik Marchi摘要：与虚拟助理的交互通常以触发短语开始，然后是命令。在这项工作中，我们探讨了通过消除对触发短语的需要来使这些交互更自然的可能性。我们的目标是根据从设备麦克风记录的流式音频中获得的信号来确定用户是否对虚拟助理进行了寻址。我们解决这个任务相结合的1-最好的假设和解码器信号从一个自动语音识别系统与声学表示从音频编码器作为输入功能的大型语言模型（LLM）。特别是，我们对数据和资源高效的系统感兴趣，这些系统只需要少量的训练数据，并且可以在设备上只有单个冻结LLM可用的情况下运行。出于这个原因，我们的模型使用低秩自适应和前缀调整的组合在80 k或更少的多模态数据示例上进行训练。我们将所提出的系统与单峰基线进行了比较，并表明多模态方法实现了较低的等误差率（EER），同时仅使用一小部分训练数据。我们还表明，低维专门的音频表示导致较低的EER比高维一般的音频表示。摘要：Interactions with virtual assistants typically start with a trigger phrase followed by a command. In this work, we explore the possibility of making these interactions more natural by eliminating the need for a trigger phrase. Our goal is to determine whether a user addressed the virtual assistant based on signals obtained from the streaming audio recorded by the device microphone. We address this task by combining 1-best hypotheses and decoder signals from an automatic speech recognition system with acoustic representations from an audio encoder as input features to a large language model (LLM). In particular, we are interested in data and resource efficient systems that require only a small amount of training data and can operate in scenarios with only a single frozen LLM available on a device. For this reason, our model is trained on 80k or less examples of multimodal data using a combination of low-rank adaptation and prefix tuning. We compare the proposed system to unimodal baselines and show that the multimodal approach achieves lower equal-error-rates (EERs), while using only a fraction of the training data. We also show that low-dimensional specialized audio representations lead to lower EERs than high-dimensional general audio representations.【3】 Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis标题：基于Schrodinger Bridges差频扩散模型的文语合成链接：http://arxiv.org/pdf/2312.03491v1作者：Zehua Chen,Guande He,Kaiwen Zheng,Xu Tan,Jun Zhu摘要：在文语转换（TTS）合成中，扩散模型已经取得了很好的生成质量。然而，由于预定义的数据-噪声扩散过程，它们的先验分布被限制为噪声表示，这提供了很少的生成目标的信息。在这项工作中，我们提出了一种新的TTS系统，Bridge-TTS，第一次尝试用一个干净的和确定性的，它提供了强大的结构信息的目标，以取代噪声高斯先验在已建立的基于扩散的TTS方法。具体来说，我们利用从文本输入中获得的潜在表示作为先验知识，并在它和地面真实梅尔频谱图之间建立一个完全易于处理的薛定谔桥，从而实现数据到数据的过程。此外，我们的配方的易处理性和灵活性，使我们能够凭经验研究的设计空间，如噪声时间表，以及开发随机和确定性采样器。LJ-Speech数据集上的实验结果表明，我们的方法在合成质量和采样效率方面都是有效的，在50步 1000步合成中显着优于我们的扩散对应物Grad-TTS，在几步场景中显着优于快速TTS模型。项目页面：https: bridge-tts.github.io摘要：In text-to-speech (TTS) synthesis, diffusion models have achieved promising generation quality. However, because of the pre-defined data-to-noise diffusion process, their prior distribution is restricted to a noisy representation, which provides little information of the generation target. In this work, we present a novel TTS system, Bridge-TTS, making the first attempt to substitute the noisy Gaussian prior in established diffusion-based TTS methods with a clean and deterministic one, which provides strong structural information of the target. Specifically, we leverage the latent representation obtained from text input as our prior, and build a fully tractable Schrodinger bridge between it and the ground-truth mel-spectrogram, leading to a data-to-data process. Moreover, the tractability and flexibility of our formulation allow us to empirically study the design spaces such as noise schedules, as well as to develop stochastic and deterministic samplers. Experimental results on the LJ-Speech dataset illustrate the effectiveness of our method in terms of both synthesis quality and sampling efficiency, significantly outperforming our diffusion counterpart Grad-TTS in 50-step 1000-step synthesis and strong fast TTS models in few-step scenarios. Project page: https: bridge-tts.github.io【4】 JAMMIN-GPT: Text-based Improvisation using LLMs in Ableton Live标题：Jammin-GPT：在Ableton Live中使用LLMS进行基于文本的即兴表演链接：http://arxiv.org/pdf/2312.03479v1作者：Sven Hollowell,Tashi Namgyal,Paul Marshall备注：Conference: 24th International Society for Music Information Retrieval. Late Breaking Demo. 2023摘要：我们介绍了一个系统，允许用户Ableton现场创建MIDI剪辑命名与音乐的描述。用户可以直接在Ableton的剪辑视图中键入所需的音乐内容，然后由我们的集成系统插入。这使得用户可以在快速产生音乐创意的同时保持创作流程。该系统的工作原理是提示ChatGPT使用几种基于文本的音乐格式之一进行回复，例如ABC符号，和弦符号或鼓指谱。这是将生成式人工智能工具集成到现有音乐工作流程中的重要一步，对于喜欢通过描述性语言表达创意愿景的内容制作者来说可能很有价值。代码可在https: github.com supersational JAMMIN-GPT上获得。摘要：We introduce a system that allows users of Ableton Live to create MIDI-clips by naming them with musical descriptions. Users can compose by typing the desired musical content directly in Ableton's clip view, which is then inserted by our integrated system. This allows users to stay in the flow of their creative process while quickly generating musical ideas. The system works by prompting ChatGPT to reply using one of several text-based musical formats, such as ABC notation, chord symbols, or drum tablature. This is an important step in integrating generative AI tools into pre-existing musical workflows, and could be valuable for content makers who prefer to express their creative vision through descriptive language. Code is available at https: github.com supersational JAMMIN-GPT.【5】 Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inference标题：随行子网络：具有动态训练和可定制推理的弹性神经网络链接：http://arxiv.org/pdf/2312.03464v1作者：Kai Li,Yi Luo备注：5 pages, 3 figures摘要：将神经网络部署到不同的设备或平台通常具有挑战性，特别是当模型大小很大或模型复杂度很高时。尽管存在用于模型修剪或提炼的方法，但是通常需要执行完整轮的模型训练或微调过程，以便获得满足模型大小或复杂性约束的较小模型。受最近动态神经网络工作的启发，我们提出了一种简单的方法来训练一个大的网络，并灵活地从它提取一个子网络的模型大小或复杂性约束在推理过程中。我们引入了一种新的方法，允许在训练阶段以动态深度和宽度训练大型模型，并且在训练大型模型之后，我们可以在推理阶段从其中选择具有任意深度和宽度的子网络，与从头开始独立训练子网络相比，性能相对更好。在音乐源分离模型上的实验结果表明，该方法可以有效地提高单个大模型在不同子网络大小和复杂度下的分离性能，并且训练大模型所需的时间比训练所有不同子网络所需的时间要短得多。摘要：Deploying neural networks to different devices or platforms is in general challenging, especially when the model size is large or model complexity is high. Although there exist ways for model pruning or distillation, it is typically required to perform a full round of model training or finetuning procedure in order to obtain a smaller model that satisfies the model size or complexity constraints. Motivated by recent works on dynamic neural networks, we propose a simple way to train a large network and flexibly extract a subnetwork from it given a model size or complexity constraint during inference. We introduce a new way to allow a large model to be trained with dynamic depth and width during the training phase, and after the large model is trained we can select a subnetwork from it with arbitrary depth and width during the inference phase with a relatively better performance compared to training the subnetwork independently from scratch. Experiment results on a music source separation model show that our proposed method can effectively improve the separation performance across different subnetwork sizes and complexities with a single large model, and training the large model takes significantly shorter time than training all the different subnetworks.【6】 Data is Overrated: Perceptual Metrics Can Lead Learning in the Absence of Training Data标题：数据被高估：在缺乏训练数据的情况下，感知度量可以引导学习链接：http://arxiv.org/pdf/2312.03455v1作者：Tashi Namgyal,Alexander Hepburn,Raul Santos-Rodriguez,Valero Laparra,Jesus Malo备注：Machine Learning for Audio Workshop, NeurIPS 2023摘要：感知度量传统上用于评估自然信号（例如图像和音频）的质量。它们旨在模仿人类观察者的感知行为，通常反映自然信号中的结构。这促使它们用作训练生成模型的损失函数，以便模型将学习捕获度量中的结构。我们在音频领域将这一想法发挥到了极致，通过训练压缩自动编码器来重建均匀的噪声，而不是自然数据。我们表明，与标准欧几里得损失训练的模型相比，感知损失训练提高了测试时频谱图和重新合成音频的重建。这证明了在使用感知度量时对看不见的自然信号的更好的概括。摘要：Perceptual metrics are traditionally used to evaluate the quality of natural signals, such as images and audio. They are designed to mimic the perceptual behaviour of human observers and usually reflect structures found in natural signals. This motivates their use as loss functions for training generative models such that models will learn to capture the structure held in the metric. We take this idea to the extreme in the audio domain by training a compressive autoencoder to reconstruct uniform noise, in lieu of natural data. We show that training with perceptual losses improves the reconstruction of spectrograms and re-synthesized audio at test time over models trained with a standard Euclidean loss. This demonstrates better generalisation to unseen natural signals when using perceptual metrics.【7】 Detecting Voice Cloning Attacks via Timbre Watermarking链接：http://arxiv.org/pdf/2312.03410v1作者：Chang Liu,Jie Zhang,Tianwei Zhang,Xi Yang,Weiming Zhang,Nenghai Yu备注：NDSS 2024摘要：如今，向公众发布音频内容是常见的。然而，随着语音克隆技术的兴起，攻击者有可能在未经任何许可的情况下利用公开发布的音频轻松冒充特定的人。因此，检测所发布的音频内容的任何潜在误用并保护其音色不被模仿变得重要。为此，我们引入了一个新的概念，“音色水印”，它嵌入到目标个人的语音水印信息，最终击败了语音克隆攻击。为了保证水印对语音克隆模型的学习过程具有鲁棒性，我们设计了一个端到端的抗语音克隆检测框架。我们的解决方案的核心思想是嵌入和提取水印在频域中的时间不变的方式。为了在不同的语音克隆攻击中获得泛化，我们调整了它们的共享过程，并将其作为失真层集成到我们的框架中。实验表明，该算法能有效地抵抗各种语音克隆攻击，对各种自适应攻击（如基于重构的移除攻击、水印移除攻击），并在现实世界的服务中实现实用性，例如PaddleSpeech、Voice-Cloning-App和so-vits-svc。此外，还进行了消融研究以验证我们设计的有效性。一些音频样本可在https: timbrewatermarking.github.io samples上获得。摘要：Nowadays, it is common to release audio content to the public. However, with the rise of voice cloning technology, attackers have the potential to easily impersonate a specific person by utilizing his publicly released audio without any permission. Therefore, it becomes significant to detect any potential misuse of the released audio content and protect its timbre from being impersonated. To this end, we introduce a novel concept, "Timbre Watermarking", which embeds watermark information into the target individual's speech, eventually defeating the voice cloning attacks. To ensure the watermark is robust to the voice cloning model's learning process, we design an end-to-end voice cloning-resistant detection framework. The core idea of our solution is to embed and extract the watermark in the frequency domain in a temporally invariant manner. To acquire generalization across different voice cloning attacks, we modulate their shared process and integrate it into our framework as a distortion layer. Experiments demonstrate that the proposed timbre watermarking can defend against different voice cloning attacks, exhibit strong resistance against various adaptive attacks (e.g., reconstruction-based removal attacks, watermark overwriting attacks), and achieve practicality in real-world services such as PaddleSpeech, Voice-Cloning-App, and so-vits-svc. In addition, ablation studies are also conducted to verify the effectiveness of our design. Some audio samples are available at https: timbrewatermarking.github.io samples.【8】 Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation标题：优化两遍跨语言迁移学习：音素识别和音素到字素的翻译链接：http://arxiv.org/pdf/2312.03312v1作者：Wonjun Lee,Gary Geunbae Lee,Yunsu Kim备注：8 pages, ASRU 2023 Accepted摘要：该研究通过增强音素识别和音素到字素翻译模型来优化低资源语言中的两遍跨语言迁移学习。我们的方法优化了这两个阶段，以提高跨语言的语音识别。我们优化音素词汇覆盖率合并音素的基础上共享的发音特征，从而提高识别精度。此外，我们引入了一个全局音素噪声发生器，用于在音素到字素训练期间产生真实的ASR噪声，以减少错误传播。在CommonVoice 12.0数据集上的实验表明，低资源语言的单词错误率（WER）显著降低，突出了我们方法的有效性。这项研究有助于在低资源语言中推进双通道ASR系统，为改进跨语言迁移学习提供了潜力。摘要：This research optimizes two-pass cross-lingual transfer learning in low-resource languages by enhancing phoneme recognition and phoneme-to-grapheme translation models. Our approach optimizes these two stages to improve speech recognition across languages. We optimize phoneme vocabulary coverage by merging phonemes based on shared articulatory characteristics, thus improving recognition accuracy. Additionally, we introduce a global phoneme noise generator for realistic ASR noise during phoneme-to-grapheme training to reduce error propagation. Experiments on the CommonVoice 12.0 dataset show significant reductions in Word Error Rate (WER) for low-resource languages, highlighting the effectiveness of our approach. This research contributes to the advancements of two-pass ASR systems in low-resource languages, offering the potential for improved cross-lingual transfer learning.【9】 Golden Gemini is All You Need: Finding the Sweet Spots for Speaker Verification标题：金双子座是你所需要的：为说话人验证寻找甜蜜点链接：http://arxiv.org/pdf/2312.03620v1作者：Tianchi Liu,Kong Aik Lee,Qiongqiong Wang,Haizhou Li备注：Submitted to IEEE TASLP in Oct., 2023. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：先前的研究表明，残差神经网络（ResNet）在说话人确认中表现出令人印象深刻的性能。ResNet模型同等对待时间和频率维度。它们遵循为图像识别设计的默认步幅配置，其中水平轴和垂直轴表现出相似性。这种方法忽略了语音表示中时间和频率不对称的事实。在本文中，我们解决这个问题，并寻找最佳的步幅配置专门为说话人验证。我们在网格图上表示步幅空间，并对时间和频率分辨率对性能的影响进行了系统的研究，并进一步确定了两个最佳点，即Golden Gemini，这是设计基于2D ResNet的说话人确认模型的指导原则。通过遵循这一原则，最先进的ResNet基线模型在VoxCeleb、SITW和CNCeleb数据集上获得了显着的性能提升，在不同的网络深度（ResNet 18、34、50和101）上，平均EER minDCF分别降低了7.70% 11.76%，同时参数数量减少了16.5%，FLOP减少了4.1%。我们称之为Gemini ResNet。进一步的调查揭示了在各种训练条件和架构下所提出的Golden Gemini操作点的有效性。此外，我们提出了一个新的基准，即双子座DF-ResNet，使用尖端的模型。摘要：Previous studies demonstrate the impressive performance of residual neural networks (ResNet) in speaker verification. The ResNet models treat the time and frequency dimensions equally. They follow the default stride configuration designed for image recognition, where the horizontal and vertical axes exhibit similarities. This approach ignores the fact that time and frequency are asymmetric in speech representation. In this paper, we address this issue and look for optimal stride configurations specifically tailored for speaker verification. We represent the stride space on a trellis diagram, and conduct a systematic study on the impact of temporal and frequency resolutions on the performance and further identify two optimal points, namely Golden Gemini, which serves as a guiding principle for designing 2D ResNet-based speaker verification models. By following the principle, a state-of-the-art ResNet baseline model gains a significant performance improvement on VoxCeleb, SITW, and CNCeleb datasets with 7.70% 11.76% average EER minDCF reductions, respectively, across different network depths (ResNet18, 34, 50, and 101), while reducing the number of parameters by 16.5% and FLOPs by 4.1%. We refer to it as Gemini ResNet. Further investigation reveals the efficacy of the proposed Golden Gemini operating points across various training conditions and architectures. Furthermore, we present a new benchmark, namely the Gemini DF-ResNet, using a cutting-edge model.【10】 Leveraging Laryngograph Data for Robust Voicing Detection in Speech标题：利用喉图数据进行语音中的健壮语音检测链接：http://arxiv.org/pdf/2312.03129v1作者：Yixuan Zhang,Heming Wang,DeLiang Wang摘要：准确地检测语音信号中的浊音间隔是基音跟踪中的关键步骤，并且具有许多应用。虽然传统的信号处理方法和深度学习算法已被提出用于此任务，但它们需要为不同的数据集微调阈值参数，并且泛化能力有限，这限制了它们在现实世界应用中的实用性。为了解决这些挑战，本研究提出了一个监督发声检测模型，利用记录的喉图数据。该模型是基于一个密集连接的卷积递归神经网络（DC-CRN），并训练的数据与参考发声决策提取的喉镜数据集。为了提高模型的泛化能力，还研究了预训练方法。该模型产生强大的浊音检测结果，优于其他强基线方法，并推广到看不见的数据集。所提出的模型与预训练的源代码一起提供了所使用的喉图数据集的列表，以方便在这一领域的进一步研究。摘要：Accurately detecting voiced intervals in speech signals is a critical step in pitch tracking and has numerous applications. While conventional signal processing methods and deep learning algorithms have been proposed for this task, their need to fine-tune threshold parameters for different datasets and limited generalization restrict their utility in real-world applications. To address these challenges, this study proposes a supervised voicing detection model that leverages recorded laryngograph data. The model is based on a densely-connected convolutional recurrent neural network (DC-CRN), and trained on data with reference voicing decisions extracted from laryngograph data sets. Pretraining is also investigated to improve the generalization ability of the model. The proposed model produces robust voicing detection results, outperforming other strong baseline methods, and generalizes well to unseen datasets. The source code of the proposed model with pretraining is provided along with the list of used laryngograph datasets to facilitate further research in this area.odels and significantly advance the usefulness of early fusion architectures.【11】 Distributed Speech Dereverberation Using Weighted Prediction Error链接：http://arxiv.org/pdf/2312.03034v1作者：Ziye Yang,Mengfei Zhang,Jie Chen摘要：语音去混响旨在减轻后期混响反射的负面影响。加权预测误差（WPE）方法是一种成熟的技术，以其在去混响方面的优越性能而闻名。然而，在麦克风节点分散的情况下，WPE方法的集中式方法需要聚合所有观测值以进行逆滤波，从而导致显著的计算负担。本文介绍了一种分布式的语音去混响方法，强调在每个节点的低计算复杂度。具体来说，我们利用分布式自适应节点特定的信号估计（DANSE）算法内的多通道线性预测（MCLP）过程。这种方法使每个节点执行本地操作，降低了复杂性，同时通过节点间的合作，实现全局性能。实验结果验证了该方法的有效性，展示了其在分散麦克风节点场景下实现高效语音去混响的能力。摘要：Speech dereverberation aims to alleviate the negative impact of late reverberant reflections. The weighted prediction error (WPE) method is a well-established technique known for its superior performance in dereverberation. However, in scenarios where microphone nodes are dispersed, the centralized approach of the WPE method requires aggregating all observations for inverse filtering, resulting in a significant computational burden. This paper introduces a distributed speech dereverberation method that emphasizes low computational complexity at each node. Specifically, we leverage the distributed adaptive node-specific signal estimation (DANSE) algorithm within the multichannel linear prediction (MCLP) process. This approach empowers each node to perform local operations with reduced complexity while achieving the global performance through inter-node cooperation. Experimental results validate the effectiveness of our proposed method, showcasing its ability to achieve efficient speech dereverberation in dispersed microphone node scenarios.【12】 Zipformer: A faster and better encoder for automatic speech recognition标题：Zipformer：一个更快更好的自动语音识别编码器链接：http://arxiv.org/pdf/2310.11230v2作者：Zengwei Yao,Liyong Guo,Xiaoyu Yang,Wei Kang,Fangjun Kuang,Yifan Yang,Zengrui Jin,Long Lin,Daniel Povey摘要：Conformer已成为自动语音识别（ASR）中最流行的编码器模型。它将卷积模块添加到Transformer中，以学习局部和全局依赖关系。在这项工作中，我们描述了一个更快，更高效的内存，性能更好的Transformer，称为Zipformer。建模更改包括：1）类似于U-Net的编码器结构，其中中间堆栈以较低的帧速率运行; 2）重新组织了具有更多模块的块结构，其中我们重新使用注意力权重以提高效率; 3）LayerNorm的修改形式称为BiasNorm，允许我们保留一些长度信息; 4）新的激活函数SwooshR和SwooshL比Swish更好地工作。我们还提出了一个新的优化器，称为ScaledAdam，它通过每个张量的当前尺度来缩放更新，以保持相对变化大致相同，并且还显式地学习参数尺度。它实现了比Adam更快的收敛和更好的性能。在LibriSpeech、Aishell-1和WenetSpeech数据集上进行的大量实验证明了我们提出的Zipformer比其他最先进的ASR模型更有效。我们的代码可在https: github.com k2-fsa icefall上公开获取。摘要：The Conformer has become the most popular encoder model for automatic speech recognition (ASR). It adds convolution modules to a transformer to learn both local and global dependencies. In this work we describe a faster, more memory-efficient, and better-performing transformer, called Zipformer. Modeling changes include: 1) a U-Net-like encoder structure where middle stacks operate at lower frame rates; 2) reorganized block structure with more modules, within which we re-use attention weights for efficiency; 3) a modified form of LayerNorm called BiasNorm allows us to retain some length information; 4) new activation functions SwooshR and SwooshL work better than Swish. We also propose a new optimizer, called ScaledAdam, which scales the update by each tensor's current scale to keep the relative change about the same, and also explictly learns the parameter scale. It achieves faster convergence and better performance than Adam. Extensive experiments on LibriSpeech, Aishell-1, and WenetSpeech datasets demonstrate the effectiveness of our proposed Zipformer over other state-of-the-art ASR models. Our code is publicly available at https: github.com k2-fsa icefall.【1】 Parameter-Efficient Transfer Learning of Audio Spectrogram Transformers链接：http://arxiv.org/pdf/2312.03694v1作者：Umberto Cappellazzo,Daniele Falavigna,Alessio Brutti,Mirco Ravanelli备注：The code is available at: this https URL摘要：微调大型预训练的Transformer模型的常见操作方式需要调整其所有参数（即，完全微调）。虽然在多个任务上取得了惊人的结果，但随着模型大小和下游任务数量的增加，这种方法变得不可行。在自然语言处理和计算机视觉中，参数高效的方法，如自动调优和适配器，已经成为可靠的替代方案，只需微调少量的额外参数，而不会牺牲性能准确性。具体来说，适配器，由于其灵活性，最近获得了显着的关注，导致几个变种。对于音频分类任务，音频频谱图Transformer模型显示了令人印象深刻的结果。然而，令人惊讶的是，如何有效地使其适应几个下游任务以前没有得到解决。在本文中，我们弥合这一差距，并提出了一个详细的调查常见的参数有效的方法，揭示适配器始终优于其他方法在四个基准。这种趋势也在Few-Shot学习设置中得到证实，并且当可训练参数的总数增加时，证明适配器具有卓越的可扩展性。最后，我们研究了最佳适配器配置，以及剩余连接在学习过程中的作用。我们的代码可在：https: github.com umbertocappellazzo PETL AST.摘要：The common modus operandi of fine-tuning large pre-trained Transformer models entails the adaptation of all their parameters (i.e., full fine-tuning). While achieving striking results on multiple tasks, this approach becomes unfeasible as the model size and the number of downstream tasks increase. In natural language processing and computer vision, parameter-efficient approaches like prompt-tuning and adapters have emerged as solid alternatives by fine-tuning only a small number of extra parameters, without sacrificing performance accuracy. Specifically, adapters, due to their flexibility, have recently garnered significant attention, leading to several variants. For audio classification tasks, the Audio Spectrogram Transformer model shows impressive results. However, surprisingly, how to efficiently adapt it to several downstream tasks has not been tackled before. In this paper, we bridge this gap and present a detailed investigation of common parameter-efficient methods, revealing that adapters consistently outperform the other methods across four benchmarks. This trend is also confirmed in few-shot learning settings and when the total number of trainable parameters increases, demonstrating adapters superior scalability. We finally study the best adapter configuration, as well as the role of residual connections in the learning process. Our code is available at: https: github.com umbertocappellazzo PETL AST.【2】An Integration of Pre-Trained Speech and Language Models for End-to-End Speech Recognition标题：用于端到端语音识别的预先训练的语音和语言模型的集成链接：http://arxiv.org/pdf/2312.03668v1作者：Yukiya Hono,Koh Mitsuda,Tianyu Zhao,Kentaro Mitsui,Toshiaki Wakatsuki,Kei Sawada备注：6 pages, 2 figures, 3 tables, The model is available at this https URL摘要：机器学习的进步使得以端到端（E2E）方式执行各种文本和语音处理任务（包括自动语音识别（ASR））成为可能。由于典型的E2E方法需要大量的训练数据和资源，因此利用预先训练的基础模型而不是从头开始训练越来越受到关注。虽然已经尝试在ASR中使用预先训练的语音和语言模型，但大多数都仅限于使用其中任何一种。本文探讨了将预训练的语音表示模型与E2E ASR的大型语言模型（LLM）集成的潜力。所提出的模型通过语音表示作为语音提示以自回归方式生成文本令牌，利用LLM提供的大量知识来实现E2E ASR。此外，该模型可以结合显着的发展，LLM利用，如推理优化和参数有效的域适应。实验结果表明，该模型实现了现代E2E ASR模型的性能相媲美。摘要：Advances in machine learning have made it possible to perform various text and speech processing tasks, including automatic speech recognition (ASR), in an end-to-end (E2E) manner. Since typical E2E approaches require large amounts of training data and resources, leveraging pre-trained foundation models instead of training from scratch is gaining attention. Although there have been attempts to use pre-trained speech and language models in ASR, most of them are limited to using either. This paper explores the potential of integrating a pre-trained speech representation model with a large language model (LLM) for E2E ASR. The proposed model enables E2E ASR by generating text tokens in an autoregressive manner via speech representations as speech prompts, taking advantage of the vast knowledge provided by the LLM. Furthermore, the proposed model can incorporate remarkable developments for LLM utilization, such as inference optimization and parameter-efficient domain adaptation. Experimental results show that the proposed model achieves performance comparable to modern E2E ASR models.【3】 Golden Gemini is All You Need: Finding the Sweet Spots for Speaker Verification标题：金双子座是你所需要的：为说话人验证寻找甜蜜点链接：http://arxiv.org/pdf/2312.03620v1作者：Tianchi Liu,Kong Aik Lee,Qiongqiong Wang,Haizhou Li备注：Submitted to IEEE TASLP in Oct., 2023. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：先前的研究表明，残差神经网络（ResNet）在说话人确认中表现出令人印象深刻的性能。ResNet模型同等对待时间和频率维度。它们遵循为图像识别设计的默认步幅配置，其中水平轴和垂直轴表现出相似性。这种方法忽略了语音表示中时间和频率不对称的事实。在本文中，我们解决这个问题，并寻找最佳的步幅配置专门为说话人验证。我们在网格图上表示步幅空间，并对时间和频率分辨率对性能的影响进行了系统的研究，并进一步确定了两个最佳点，即Golden Gemini，这是设计基于2D ResNet的说话人确认模型的指导原则。通过遵循这一原则，最先进的ResNet基线模型在VoxCeleb、SITW和CNCeleb数据集上获得了显着的性能提升，在不同的网络深度（ResNet 18、34、50和101）上，平均EER minDCF分别降低了7.70% 11.76%，同时参数数量减少了16.5%，FLOP减少了4.1%。我们称之为Gemini ResNet。进一步的调查揭示了在各种训练条件和架构下所提出的Golden Gemini操作点的有效性。此外，我们提出了一个新的基准，即双子座DF-ResNet，使用尖端的模型。摘要：Previous studies demonstrate the impressive performance of residual neural networks (ResNet) in speaker verification. The ResNet models treat the time and frequency dimensions equally. They follow the default stride configuration designed for image recognition, where the horizontal and vertical axes exhibit similarities. This approach ignores the fact that time and frequency are asymmetric in speech representation. In this paper, we address this issue and look for optimal stride configurations specifically tailored for speaker verification. We represent the stride space on a trellis diagram, and conduct a systematic study on the impact of temporal and frequency resolutions on the performance and further identify two optimal points, namely Golden Gemini, which serves as a guiding principle for designing 2D ResNet-based speaker verification models. By following the principle, a state-of-the-art ResNet baseline model gains a significant performance improvement on VoxCeleb, SITW, and CNCeleb datasets with 7.70% 11.76% average EER minDCF reductions, respectively, across different network depths (ResNet18, 34, 50, and 101), while reducing the number of parameters by 16.5% and FLOPs by 4.1%. We refer to it as Gemini ResNet. Further investigation reveals the efficacy of the proposed Golden Gemini operating points across various training conditions and architectures. Furthermore, we present a new benchmark, namely the Gemini DF-ResNet, using a cutting-edge model.【4】 Lightweight Speaker Verification Using Transformation Module with Feature Partition and Fusion链接：http://arxiv.org/pdf/2312.03324v1作者：Yanxiong Li,Zhongjie Jiang,Qisheng Huang,Wenchang Cao,Jialong Li备注：12 pages, 5 figures, 6 tables; accepted for publication in IEEE-ACM TASLP摘要：尽管人们在降低说话人确认模型复杂度方面做了很多努力，但在资源有限的终端上部署具有满意效果的说话人确认系统仍然具有挑战性。我们设计了一个转换模块，执行特征划分和融合，实现轻量级的说话人确认。转换模块由多个简单但有效的操作组成，如卷积、池化、均值、连接、归一化和逐元素求和。它以即插即用的方式工作，可以很容易地植入到各种各样的模型中，以降低模型的复杂性，同时保持模型的误差。首先，输入特征被分成几个低维的特征子集，以降低模型的复杂性。然后，通过融合特征子集间的相关性信息来更新特征子集，以增强特征子集的表征能力。最后，更新的特征子集被独立地馈送到模型的块（一个或几个层）中以供进一步处理。从模型的当前块输出的特征在被馈送到模型的下一个块之前根据上述步骤进行处理。实验数据选自两个公开语音语料库（VoxCeleb 1和VoxCeleb 2）。实验结果表明，将变换模块植入AMCRN、ResNet 34和ECAPA-TDNN三种说话人确认模型中，模型误差略有增加，模型复杂度显著降低。我们提出的方法优于基线方法在整体上的内存需求和计算复杂度与较低的等错误率。它还可以很好地推广到不同长度的截断段。摘要：Although many efforts have been made on decreasing the model complexity for speaker verification, it is still challenging to deploy speaker verification systems with satisfactory result on low-resource terminals. We design a transformation module that performs feature partition and fusion to implement lightweight speaker verification. The transformation module consists of multiple simple but effective operations, such as convolution, pooling, mean, concatenation, normalization, and element-wise summation. It works in a plug-and-play way, and can be easily implanted into a wide variety of models to reduce the model complexity while maintaining the model error. First, the input feature is split into several low-dimensional feature subsets for decreasing the model complexity. Then, each feature subset is updated by fusing it with the inter-feature-subsets correlational information to enhance its representational capability. Finally, the updated feature subsets are independently fed into the block (one or several layers) of the model for further processing. The features that are output from current block of the model are processed according to the steps above before they are fed into the next block of the model. Experimental data are selected from two public speech corpora (namely VoxCeleb1 and VoxCeleb2). Results show that implanting the transformation module into three models (namely AMCRN, ResNet34, and ECAPA-TDNN) for speaker verification slightly increases the model error and significantly decreases the model complexity. Our proposed method outperforms baseline methods on the whole in memory requirement and computational complexity with lower equal error rate. It also generalizes well across truncated segments with various lengths.【5】 Leveraging Laryngograph Data for Robust Voicing Detection in Speech标题：利用喉图数据进行语音中的健壮语音检测链接：http://arxiv.org/pdf/2312.03129v1作者：Yixuan Zhang,Heming Wang,DeLiang Wang摘要：准确地检测语音信号中的浊音间隔是基音跟踪中的关键步骤，并且具有许多应用。虽然传统的信号处理方法和深度学习算法已被提出用于此任务，但它们需要为不同的数据集微调阈值参数，并且泛化能力有限，这限制了它们在现实世界应用中的实用性。为了解决这些挑战，本研究提出了一个监督发声检测模型，利用记录的喉图数据。该模型是基于一个密集连接的卷积递归神经网络（DC-CRN），并训练的数据与参考发声决策提取的喉镜数据集。为了提高模型的泛化能力，还研究了预训练方法。该模型产生强大的浊音检测结果，优于其他强基线方法，并推广到看不见的数据集。所提出的模型与预训练的源代码一起提供了所使用的喉图数据集的列表，以方便在这一领域的进一步研究。摘要：Accurately detecting voiced intervals in speech signals is a critical step in pitch tracking and has numerous applications. While conventional signal processing methods and deep learning algorithms have been proposed for this task, their need to fine-tune threshold parameters for different datasets and limited generalization restrict their utility in real-world applications. To address these challenges, this study proposes a supervised voicing detection model that leverages recorded laryngograph data. The model is based on a densely-connected convolutional recurrent neural network (DC-CRN), and trained on data with reference voicing decisions extracted from laryngograph data sets. Pretraining is also investigated to improve the generalization ability of the model. The proposed model produces robust voicing detection results, outperforming other strong baseline methods, and generalizes well to unseen datasets. The source code of the proposed model with pretraining is provided along with the list of used laryngograph datasets to facilitate further research in this area.odels and significantly advance the usefulness of early fusion architectures.【6】 Distributed Speech Dereverberation Using Weighted Prediction Error链接：http://arxiv.org/pdf/2312.03034v1作者：Ziye Yang,Mengfei Zhang,Jie Chen摘要：语音去混响旨在减轻后期混响反射的负面影响。加权预测误差（WPE）方法是一种成熟的技术，以其在去混响方面的优越性能而闻名。然而，在麦克风节点分散的情况下，WPE方法的集中式方法需要聚合所有观测值以进行逆滤波，从而导致显著的计算负担。本文介绍了一种分布式的语音去混响方法，强调在每个节点的低计算复杂度。具体来说，我们利用分布式自适应节点特定的信号估计（DANSE）算法内的多通道线性预测（MCLP）过程。这种方法使每个节点执行本地操作，降低了复杂性，同时通过节点间的合作，实现全局性能。实验结果验证了该方法的有效性，展示了其在分散麦克风节点场景下实现高效语音去混响的能力。摘要：Speech dereverberation aims to alleviate the negative impact of late reverberant reflections. The weighted prediction error (WPE) method is a well-established technique known for its superior performance in dereverberation. However, in scenarios where microphone nodes are dispersed, the centralized approach of the WPE method requires aggregating all observations for inverse filtering, resulting in a significant computational burden. This paper introduces a distributed speech dereverberation method that emphasizes low computational complexity at each node. Specifically, we leverage the distributed adaptive node-specific signal estimation (DANSE) algorithm within the multichannel linear prediction (MCLP) process. This approach empowers each node to perform local operations with reduced complexity while achieving the global performance through inter-node cooperation. Experimental results validate the effectiveness of our proposed method, showcasing its ability to achieve efficient speech dereverberation in dispersed microphone node scenarios.【7】 Towards small and accurate convolutional neural networks for acoustic biodiversity monitoring
标题：用于声学生物多样性监测的小而准确的卷积神经网络
链接：http://arxiv.org/pdf/2312.03666v1
作者：Serge Zaugg,Mike van der Schaar,Florence Erbs,Antonio Sanchez,Joan V. Castell,Emiliano Ramallo,Michel André摘要：动物声音的自动分类是大规模监测生物多样性的先决条件。卷积神经网络（CNN）是最有前途的算法之一，但它们速度慢，通常在该领域的分类效果较差，并且通常需要大量的训练数据集。我们的目标是设计在推理时速度快的CNN，并在从中等规模的数据中学习时实现良好的分类性能。使用了来自雨林生态系统的记录。从20种鸟类的声音的开始和结束点手动注释。来自10秒片段的频谱图用作CNN输入。我们设计了具有频率解包层的简单CNN（SIMP-FU模型），使得任何输出单元都连接到所有频谱图频率，但仅连接到时间的子区域，即接收场（RF）。我们的模型允许使用不同的RF持续时间进行实验。模型要么使用编码声音的开始和结束点的时间索引标签，要么使用更简单的片段级标签。从时间索引标签学习的模型比它们的段级对应物表现得好得多。最好的分类性能，实现了与中间射频持续时间为1.5秒的模型。最好的SIMP-FU模型在测试集上的20个类别中的18个类别中实现了AUC超过0.95。在紧凑的低成本硬件上，最好的SIMP-FU模型的评估速度比实时数据采集快7倍。射频持续时间是分类性能的主要驱动因素。1.5s的最佳值与声音的持续时间相同。我们的模型在从中等规模的训练数据中学习的同时取得了良好的分类性能。这可以通过在训练期间使用时间索引标签和适当大小的RF来解释。结果证实了在紧凑型低成本设备上部署具有良好分类性能的小型CNN的可行性。摘要：Automated classification of animal sounds is a prerequisite for large-scale monitoring of biodiversity. Convolutional Neural Networks (CNNs) are among the most promising algorithms but they are slow, often achieve poor classification in the field and typically require large training data sets. Our objective was to design CNNs that are fast at inference time and achieve good classification performance while learning from moderate-sized data. Recordings from a rainforest ecosystem were used. Start and end-point of sounds from 20 bird species were manually annotated. Spectrograms from 10 second segments were used as CNN input. We designed simple CNNs with a frequency unwrapping layer (SIMP-FU models) such that any output unit was connected to all spectrogram frequencies but only to a sub-region of time, the Receptive Field (RF). Our models allowed experimentation with different RF durations. Models either used the time-indexed labels that encode start and end-point of sounds or simpler segment-level labels. Models learning from time-indexed labels performed considerably better than their segment-level counterparts. Best classification performances was achieved for models with intermediate RF duration of 1.5 seconds. The best SIMP-FU models achieved AUCs over 0.95 in 18 of 20 classes on the test set. On compact low-cost hardware the best SIMP-FU models evaluated up to seven times faster than real-time data acquisition. RF duration was a major driver of classification performance. The optimum of 1.5 s was in the same range as the duration of the sounds. Our models achieved good classification performance while learning from moderate-sized training data. This is explained by the usage of time-indexed labels during training and adequately sized RF. Results confirm the feasibility of deploying small CNNs with good classification performance on compact low-cost devices.
【8】 Multimodal Data and Resource Efficient Device-Directed Speech Detection with Large Foundation Models标题：利用大型基础模型进行多模式数据和资源高效的设备定向语音检测链接：http://arxiv.org/pdf/2312.03632v1作者：Dominik Wagner,Alexander Churchill,Siddharth Sigtia,Panayiotis Georgiou,Matt Mirsamadi,Aarshee Mishra,Erik Marchi摘要：与虚拟助理的交互通常以触发短语开始，然后是命令。在这项工作中，我们探讨了通过消除对触发短语的需要来使这些交互更自然的可能性。我们的目标是根据从设备麦克风记录的流式音频中获得的信号来确定用户是否对虚拟助理进行了寻址。我们解决这个任务相结合的1-最好的假设和解码器信号从一个自动语音识别系统与声学表示从音频编码器作为输入功能的大型语言模型（LLM）。特别是，我们对数据和资源高效的系统感兴趣，这些系统只需要少量的训练数据，并且可以在设备上只有单个冻结LLM可用的情况下运行。出于这个原因，我们的模型使用低秩自适应和前缀调整的组合在80 k或更少的多模态数据示例上进行训练。我们将所提出的系统与单峰基线进行了比较，并表明多模态方法实现了较低的等误差率（EER），同时仅使用一小部分训练数据。我们还表明，低维专门的音频表示导致较低的EER比高维一般的音频表示。摘要：Interactions with virtual assistants typically start with a trigger phrase followed by a command. In this work, we explore the possibility of making these interactions more natural by eliminating the need for a trigger phrase. Our goal is to determine whether a user addressed the virtual assistant based on signals obtained from the streaming audio recorded by the device microphone. We address this task by combining 1-best hypotheses and decoder signals from an automatic speech recognition system with acoustic representations from an audio encoder as input features to a large language model (LLM). In particular, we are interested in data and resource efficient systems that require only a small amount of training data and can operate in scenarios with only a single frozen LLM available on a device. For this reason, our model is trained on 80k or less examples of multimodal data using a combination of low-rank adaptation and prefix tuning. We compare the proposed system to unimodal baselines and show that the multimodal approach achieves lower equal-error-rates (EERs), while using only a fraction of the training data. We also show that low-dimensional specialized audio representations lead to lower EERs than high-dimensional general audio representations.【9】 Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis标题：基于Schrodinger Bridges差频扩散模型的文语合成链接：http://arxiv.org/pdf/2312.03491v1作者：Zehua Chen,Guande He,Kaiwen Zheng,Xu Tan,Jun Zhu摘要：在文语转换（TTS）合成中，扩散模型已经取得了很好的生成质量。然而，由于预定义的数据-噪声扩散过程，它们的先验分布被限制为噪声表示，这提供了很少的生成目标的信息。在这项工作中，我们提出了一种新的TTS系统，Bridge-TTS，第一次尝试用一个干净的和确定性的，它提供了强大的结构信息的目标，以取代噪声高斯先验在已建立的基于扩散的TTS方法。具体来说，我们利用从文本输入中获得的潜在表示作为先验知识，并在它和地面真实梅尔频谱图之间建立一个完全易于处理的薛定谔桥，从而实现数据到数据的过程。此外，我们的配方的易处理性和灵活性，使我们能够凭经验研究的设计空间，如噪声时间表，以及开发随机和确定性采样器。LJ-Speech数据集上的实验结果表明，我们的方法在合成质量和采样效率方面都是有效的，在50步 1000步合成中显着优于我们的扩散对应物Grad-TTS，在几步场景中显着优于快速TTS模型。项目页面：https: bridge-tts.github.io摘要：In text-to-speech (TTS) synthesis, diffusion models have achieved promising generation quality. However, because of the pre-defined data-to-noise diffusion process, their prior distribution is restricted to a noisy representation, which provides little information of the generation target. In this work, we present a novel TTS system, Bridge-TTS, making the first attempt to substitute the noisy Gaussian prior in established diffusion-based TTS methods with a clean and deterministic one, which provides strong structural information of the target. Specifically, we leverage the latent representation obtained from text input as our prior, and build a fully tractable Schrodinger bridge between it and the ground-truth mel-spectrogram, leading to a data-to-data process. Moreover, the tractability and flexibility of our formulation allow us to empirically study the design spaces such as noise schedules, as well as to develop stochastic and deterministic samplers. Experimental results on the LJ-Speech dataset illustrate the effectiveness of our method in terms of both synthesis quality and sampling efficiency, significantly outperforming our diffusion counterpart Grad-TTS in 50-step 1000-step synthesis and strong fast TTS models in few-step scenarios. Project page: https: bridge-tts.github.io【10】 JAMMIN-GPT: Text-based Improvisation using LLMs in Ableton Live标题：Jammin-GPT：在Ableton Live中使用LLMS进行基于文本的即兴表演链接：http://arxiv.org/pdf/2312.03479v1作者：Sven Hollowell,Tashi Namgyal,Paul Marshall备注：Conference: 24th International Society for Music Information Retrieval. Late Breaking Demo. 2023摘要：我们介绍了一个系统，允许用户Ableton现场创建MIDI剪辑命名与音乐的描述。用户可以直接在Ableton的剪辑视图中键入所需的音乐内容，然后由我们的集成系统插入。这使得用户可以在快速产生音乐创意的同时保持创作流程。该系统的工作原理是提示ChatGPT使用几种基于文本的音乐格式之一进行回复，例如ABC符号，和弦符号或鼓指谱。这是将生成式人工智能工具集成到现有音乐工作流程中的重要一步，对于喜欢通过描述性语言表达创意愿景的内容制作者来说可能很有价值。代码可在https: github.com supersational JAMMIN-GPT上获得。摘要：We introduce a system that allows users of Ableton Live to create MIDI-clips by naming them with musical descriptions. Users can compose by typing the desired musical content directly in Ableton's clip view, which is then inserted by our integrated system. This allows users to stay in the flow of their creative process while quickly generating musical ideas. The system works by prompting ChatGPT to reply using one of several text-based musical formats, such as ABC notation, chord symbols, or drum tablature. This is an important step in integrating generative AI tools into pre-existing musical workflows, and could be valuable for content makers who prefer to express their creative vision through descriptive language. Code is available at https: github.com supersational JAMMIN-GPT.【11】 Subnetwork-to-go: Elastic Neural Network with Dynamic Training and Customizable Inference标题：随行子网络：具有动态训练和可定制推理的弹性神经网络链接：http://arxiv.org/pdf/2312.03464v1作者：Kai Li,Yi Luo备注：5 pages, 3 figures摘要：将神经网络部署到不同的设备或平台通常具有挑战性，特别是当模型大小很大或模型复杂度很高时。尽管存在用于模型修剪或提炼的方法，但是通常需要执行完整轮的模型训练或微调过程，以便获得满足模型大小或复杂性约束的较小模型。受最近动态神经网络工作的启发，我们提出了一种简单的方法来训练一个大的网络，并灵活地从它提取一个子网络的模型大小或复杂性约束在推理过程中。我们引入了一种新的方法，允许在训练阶段以动态深度和宽度训练大型模型，并且在训练大型模型之后，我们可以在推理阶段从其中选择具有任意深度和宽度的子网络，与从头开始独立训练子网络相比，性能相对更好。在音乐源分离模型上的实验结果表明，该方法可以有效地提高单个大模型在不同子网络大小和复杂度下的分离性能，并且训练大模型所需的时间比训练所有不同子网络所需的时间要短得多。摘要：Deploying neural networks to different devices or platforms is in general challenging, especially when the model size is large or model complexity is high. Although there exist ways for model pruning or distillation, it is typically required to perform a full round of model training or finetuning procedure in order to obtain a smaller model that satisfies the model size or complexity constraints. Motivated by recent works on dynamic neural networks, we propose a simple way to train a large network and flexibly extract a subnetwork from it given a model size or complexity constraint during inference. We introduce a new way to allow a large model to be trained with dynamic depth and width during the training phase, and after the large model is trained we can select a subnetwork from it with arbitrary depth and width during the inference phase with a relatively better performance compared to training the subnetwork independently from scratch. Experiment results on a music source separation model show that our proposed method can effectively improve the separation performance across different subnetwork sizes and complexities with a single large model, and training the large model takes significantly shorter time than training all the different subnetworks.【12】 Data is Overrated: Perceptual Metrics Can Lead Learning in the Absence of Training Data标题：数据被高估了：感知能力可以在缺乏训练数据的情况下引导学习链接：http://arxiv.org/pdf/2312.03455v1作者：Tashi Namgyal,Alexander Hepburn,Raul Santos-Rodriguez,Valero Laparra,Jesus Malo备注：Machine Learning for Audio Workshop, NeurIPS 2023摘要：感知度量传统上用于评估自然信号（例如图像和音频）的质量。它们旨在模仿人类观察者的感知行为，通常反映自然信号中的结构。这促使它们用作训练生成模型的损失函数，以便模型将学习捕获度量中的结构。我们在音频领域将这一想法发挥到了极致，通过训练压缩自动编码器来重建均匀的噪声，而不是自然数据。我们表明，与标准欧几里得损失训练的模型相比，感知损失训练提高了测试时频谱图和重新合成音频的重建。这证明了在使用感知度量时对看不见的自然信号的更好的概括。摘要：Perceptual metrics are traditionally used to evaluate the quality of natural signals, such as images and audio. They are designed to mimic the perceptual behaviour of human observers and usually reflect structures found in natural signals. This motivates their use as loss functions for training generative models such that models will learn to capture the structure held in the metric. We take this idea to the extreme in the audio domain by training a compressive autoencoder to reconstruct uniform noise, in lieu of natural data. We show that training with perceptual losses improves the reconstruction of spectrograms and re-synthesized audio at test time over models trained with a standard Euclidean loss. This demonstrates better generalisation to unseen natural signals when using perceptual metrics.【13】 Detecting Voice Cloning Attacks via Timbre Watermarking链接：http://arxiv.org/pdf/2312.03410v1作者：Chang Liu,Jie Zhang,Tianwei Zhang,Xi Yang,Weiming Zhang,Nenghai Yu备注：NDSS 2024摘要：如今，向公众发布音频内容是常见的。然而，随着语音克隆技术的兴起，攻击者有可能在未经任何许可的情况下利用公开发布的音频轻松冒充特定的人。因此，检测所发布的音频内容的任何潜在误用并保护其音色不被模仿变得重要。为此，我们引入了一个新的概念，“音色水印”，它嵌入到目标个人的语音水印信息，最终击败了语音克隆攻击。为了保证水印对语音克隆模型的学习过程具有鲁棒性，我们设计了一个端到端的抗语音克隆检测框架。我们的解决方案的核心思想是嵌入和提取水印在频域中的时间不变的方式。为了在不同的语音克隆攻击中获得泛化，我们调整了它们的共享过程，并将其作为失真层集成到我们的框架中。实验表明，该算法能有效地抵抗各种语音克隆攻击，对各种自适应攻击（如基于重构的移除攻击、水印移除攻击），并在现实世界的服务中实现实用性，例如PaddleSpeech、Voice-Cloning-App和so-vits-svc。此外，还进行了消融研究以验证我们设计的有效性。一些音频样本可在https: timbrewatermarking.github.io samples上获得。摘要：Nowadays, it is common to release audio content to the public. However, with the rise of voice cloning technology, attackers have the potential to easily impersonate a specific person by utilizing his publicly released audio without any permission. Therefore, it becomes significant to detect any potential misuse of the released audio content and protect its timbre from being impersonated. To this end, we introduce a novel concept, "Timbre Watermarking", which embeds watermark information into the target individual's speech, eventually defeating the voice cloning attacks. To ensure the watermark is robust to the voice cloning model's learning process, we design an end-to-end voice cloning-resistant detection framework. The core idea of our solution is to embed and extract the watermark in the frequency domain in a temporally invariant manner. To acquire generalization across different voice cloning attacks, we modulate their shared process and integrate it into our framework as a distortion layer. Experiments demonstrate that the proposed timbre watermarking can defend against different voice cloning attacks, exhibit strong resistance against various adaptive attacks (e.g., reconstruction-based removal attacks, watermark overwriting attacks), and achieve practicality in real-world services such as PaddleSpeech, Voice-Cloning-App, and so-vits-svc. In addition, ablation studies are also conducted to verify the effectiveness of our design. Some audio samples are available at https: timbrewatermarking.github.io samples.【14】 Optimizing Two-Pass Cross-Lingual Transfer Learning: Phoneme Recognition and Phoneme to Grapheme Translation标题：优化两遍跨语言迁移学习：音素识别和音素到字素的翻译链接：http://arxiv.org/pdf/2312.03312v1作者：Wonjun Lee,Gary Geunbae Lee,Yunsu Kim备注：8 pages, ASRU 2023 Accepted摘要：该研究通过增强音素识别和音素到字素翻译模型来优化低资源语言中的两遍跨语言迁移学习。我们的方法优化了这两个阶段，以提高跨语言的语音识别。我们优化音素词汇覆盖率合并音素的基础上共享的发音特征，从而提高识别精度。此外，我们引入了一个全局音素噪声发生器，用于在音素到字素训练期间产生真实的ASR噪声，以减少错误传播。在CommonVoice 12.0数据集上的实验表明，低资源语言的单词错误率（WER）显著降低，突出了我们方法的有效性。这项研究有助于在低资源语言中推进双通道ASR系统，为改进跨语言迁移学习提供了潜力。摘要：This research optimizes two-pass cross-lingual transfer learning in low-resource languages by enhancing phoneme recognition and phoneme-to-grapheme translation models. Our approach optimizes these two stages to improve speech recognition across languages. We optimize phoneme vocabulary coverage by merging phonemes based on shared articulatory characteristics, thus improving recognition accuracy. Additionally, we introduce a global phoneme noise generator for realistic ASR noise during phoneme-to-grapheme training to reduce error propagation. Experiments on the CommonVoice 12.0 dataset show significant reductions in Word Error Rate (WER) for low-resource languages, highlighting the effectiveness of our approach. This research contributes to the advancements of two-pass ASR systems in low-resource languages, offering the potential for improved cross-lingual transfer learning.【15】Deep Multimodal Fusion for Surgical Feedback Classification标题：深度多模式融合用于手术反馈分类链接：http://arxiv.org/pdf/2312.03231v1作者：Rafal Kocielnik,Elyssa Y. Wong,Timothy N. Chu,Lydia Lin,De-An Huang,Jiayun Wang,Anima Anandkumar,Andrew J. HungJournal-ref：Published in Proceedings of Machine Learning for Health 2024摘要：在手术过程中，由经验丰富的外科医生向受训者提供的实时非正式反馈的量化对于手术训练中的技能提高是重要的。现场手术室中的这种反馈本质上是多模态的，包括口头对话（例如，问题和答案）以及非语言元素（例如，通过视觉线索，如指向解剖元素）。在这项工作中，我们利用临床验证的手术反馈的五类分类：“解剖”，“技术”，“程序”，“赞美”和“视觉辅助”。然后，我们开发了一个多标签机器学习模型，从文本，音频和视频模式的输入中对这五类手术反馈进行分类。我们工作的最终目标是帮助自动化大规模的实时上下文手术反馈的注释。我们对手术反馈的自动分类实现了AUC范围从71.5到77.6，融合将性能提高了3.1%。我们还表明，来自专家的反馈音频的高质量手动传输将AUC提高到76.5和96.2之间，这表明了未来改进的明确路径。从经验上讲，我们发现分阶段训练策略，首先分别对每种模态进行预训练，然后联合训练它们，比完全训练不同的模态更有效。我们还提出了直观的结果，不同的反馈类别的重要性的方式。这项工作提供了一个重要的第一次看的可行性，自动分类的真实世界的实时手术反馈的基础上，文本，音频和视频模式。摘要：Quantification of real-time informal feedback delivered by an experienced surgeon to a trainee during surgery is important for skill improvements in surgical training. Such feedback in the live operating room is inherently multimodal, consisting of verbal conversations (e.g., questions and answers) as well as non-verbal elements (e.g., through visual cues like pointing to anatomic elements). In this work, we leverage a clinically-validated five-category classification of surgical feedback: "Anatomic", "Technical", "Procedural", "Praise" and "Visual Aid". We then develop a multi-label machine learning model to classify these five categories of surgical feedback from inputs of text, audio, and video modalities. The ultimate goal of our work is to help automate the annotation of real-time contextual surgical feedback at scale. Our automated classification of surgical feedback achieves AUCs ranging from 71.5 to 77.6 with the fusion improving performance by 3.1%. We also show that high-quality manual transcriptions of feedback audio from experts improve AUCs to between 76.5 and 96.2, which demonstrates a clear path toward future improvements. Empirically, we find that the Staged training strategy, with first pre-training each modality separately and then training them jointly, is more effective than training different modalities altogether. We also present intuitive findings on the importance of modalities for different feedback categories. This work offers an important first look at the feasibility of automated classification of real-world live surgical feedback based on text, audio, and video modalities. 
每日语音速递