今日论文合集：cs.SD语音14篇，eess.AS音频处理17篇。【1】Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
标题：基于分层蒸馏的CIF语音识别器知识传递
链接：http://arxiv.org/pdf/2301.13003v1
作者：Minglun Han,Feilong Chen,Jing Shi,Shuang Xu,Bo Xu机构：Institute of Automation, Chinese Academy of Sciences, Beijing, China, School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China备注：5 pages; Keywords: speech recognition, continuous integrate-and-fire, knowledge distillation, contrastive learning, pre-trained language models摘要：大规模预训练语言模型具有强大的语言建模能力，在自然语言处理中得到了广泛的应用。对于自动语音识别（ASR），利用PLM来提高性能也成为一个很有前途的研究方向。然而，大多数先前的工作可能遭受PLM的不灵活的大小和结构，以及在PLM中的知识利用不足。为了解决这些问题，我们提出了基于连续积分点火（CIF）的ASR模型的层次化知识提取。具体地，我们通过在声学水平上应用具有对比损失的跨模态提取和在语言水平上应用具有回归损失的提取来将知识从PLM提取到ASR模型。在AISHELL-1数据集上，我们的方法实现了比原始基于CIF的模型减少15%的相对错误率，并实现了与最先进模型相当的性能（开发 测试上的3.8% 4.1%）。摘要：Large-scale pre-trained language models (PLMs) with powerful language modeling capabilities have been widely used in natural language processing. For automatic speech recognition (ASR), leveraging PLMs to improve performance has also become a promising research trend. However, most previous works may suffer from the inflexible sizes and structures of PLMs, along with the insufficient utilization of the knowledge in PLMs. To alleviate these problems, we propose the hierarchical knowledge distillation on the continuous integrate-and-fire (CIF) based ASR models. Specifically, we distill the knowledge from PLMs to the ASR model by applying cross-modal distillation with contrastive loss at the acoustic level and applying distillation with regression loss at the linguistic level. On the AISHELL-1 dataset, our method achieves 15% relative error rate reduction over the original CIF-based model and achieves comparable performance (3.8% 4.1% on dev test) to the state-of-the-art model.
【2】GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration
标题：GibbsDDRM：去噪扩散恢复盲逆问题的部分折叠Gibbs采样器
链接：http://arxiv.org/pdf/2301.12686v1
作者：Naoki Murata,Koichi Saito,Chieh-Hsin Lai,Yuhta Takida,Toshimitsu Uesaka,Yuki Mitsufuji,Stefano Ermon摘要：预先训练的扩散模型已经成功地用作各种线性逆问题中的先验，其中目标是从噪声线性测量重构信号。然而，现有的方法需要线性算子的知识。本文提出了GibbsDDRM，它是去噪扩散恢复模型（DDRM）在线性测量算子未知的盲背景下的一种扩展. GibbsDDRM通过对数据先验使用预先训练的扩散模型来构造数据、测量和线性算子的联合分布，并且它通过使用Gibbs采样器的有效变体的后验采样来解决该问题。所提出的方法是问题不可知的，这意味着预先训练的扩散模型可以应用于各种反问题而无需微调。在实验中，该算法在盲图像去模糊和语音去混响两个任务上都取得了很高的性能，尽管对底层线性算子使用了简单的通用先验。摘要：Pre-trained diffusion models have been successfully used as priors in a variety of linear inverse problems, where the goal is to reconstruct a signal from noisy linear measurements. However, existing approaches require knowledge of the linear operator. In this paper, we propose GibbsDDRM, an extension of Denoising Diffusion Restoration Models (DDRM) to a blind setting in which the linear measurement operator is unknown. GibbsDDRM constructs a joint distribution of the data, measurements, and linear operator by using a pre-trained diffusion model for the data prior, and it solves the problem by posterior sampling with an efficient variant of a Gibbs sampler. The proposed method is problem-agnostic, meaning that a pre-trained diffusion model can be applied to various inverse problems without fine tuning. In experiments, it achieved high performance on both blind image deblurring and vocal dereverberation tasks, despite the use of simple generic priors for the underlying linear operators.
【3】SingSong: Generating musical accompaniments from singing
标题：SING：从歌声中产生音乐伴奏
链接：http://arxiv.org/pdf/2301.12662v1
作者：Chris Donahue,Antoine Caillon,Adam Roberts,Ethan Manilow,Philippe Esling,Andrea Agostinelli,Mauro Verzetti,Ian Simon,Olivier Pietquin,Neil Zeghidour,Jesse Engel摘要：我们介绍SingSong，一个生成器乐来为输入人声伴奏的系统，它可能为音乐家和非音乐家提供一种直观的新方法来创作具有自己声音的音乐。为了实现这一点，我们建立在音乐源分离和音频生成的最新发展。具体来说，我们将最先进的源分离算法应用于大型音乐音频语料库，以产生对齐的人声和乐器源对。然后，我们采用AudioLM（Borsos等人，2022）--一种最先进的无条件音频生成方法--适合于有条件的“音频到音频”生成任务，并在源分离（声乐、器乐）对上训练它。在相同声音输入的两两比较中，与来自强检索基线的乐器相比，听者对由《歌唱》生成的乐器表现出显著的偏好。声音示例，请访问https: g.co magenta singsong摘要：We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be suitable for conditional "audio-to-audio" generation tasks, and train it on the source-separated (vocal, instrumental) pairs. In a pairwise comparison with the same vocal inputs, listeners expressed a significant preference for instrumentals generated by SingSong compared to those from a strong retrieval baseline. Sound examples at https: g.co magenta singsong
【4】Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
标题：制作音频：使用即时增强的扩散模型生成文本到音频
链接：http://arxiv.org/pdf/2301.12661v1
作者：Rongjie Huang,Jiawei Huang,Dongchao Yang,Yi Ren,Luping Liu,Mingze Li,Zhenhui Ye,Jinglin Liu,Xiang Yin,Zhou Zhao机构：20 2 2; HongEqual contribution 1Zhejiang University 2Peking University 3Speech & Audio Team备注：Audio samples are available at this https URL摘要：大规模的多模态生成式建模已经在文本到图像和文本到视频的生成中创造了里程碑。其在音频方面的应用仍然滞后，主要原因有两个：缺乏具有高质量文本-音频对的大规模数据集，以及对长的连续音频数据建模的复杂性。在本研究中，我们提出了一个即时增强扩散模型来解决这些问题：1）通过提取然后重新编程的方法来引入伪即时增强，它通过使用无语言的音频来缓解数量级概念组合的数据短缺;2）利用谱图自动编码器来预测自监督音频表示而不是波形。结合强大的对比语言音频预训练（CLAP）表示，Make-An-Audio在客观和主观基准评估中都取得了最先进的结果。此外，我们还介绍了它的可控性和通用性，用于“无模态遗留”的X到音频，首次解锁了在给定用户定义的模态输入的情况下生成高清晰度、高保真度音频的能力。音频样本可从www.example.com获得https: Text-to-Audio.github.io摘要：Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. In this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps by 1) introducing pseudo prompt enhancement with a distill-then-reprogram approach, it alleviates data scarcity with orders of magnitude concept compositions by using language-free audios; 2) leveraging spectrogram autoencoder to predict the self-supervised audio representation instead of waveforms. Together with robust contrastive language-audio pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art results in both objective and subjective benchmark evaluation. Moreover, we present its controllability and generalization for X-to-Audio with "No Modality Left Behind", for the first time unlocking the ability to generate high-definition, high-fidelity audios given a user-defined modality input. Audio samples are available at https: Text-to-Audio.github.io
【5】Composer's Assistant: Interactive Transformers for Multi-Track MIDI Infilling
标题：作曲家助手：用于多声道MIDI填充的交互式转换器
链接：http://arxiv.org/pdf/2301.12525v1
备注：16 pages, 7 figures, 3 tables摘要：我们考虑当任意（音轨，小节）信息对从MIDI文件的连续小节片段中删除时的多音轨MIDI填充任务。我们训练两个类似T5的模型来解决这个任务，一个使用基本的类似MIDI的事件词汇表，另一个使用这个词汇表的类似连接词的版本。我们介绍了一个新的测试集，从Lakh MIDI数据集创建，包括9个多轨MIDI填充任务。我们在这些任务上评估我们的模型，发现一个模型在一些任务上工作得更好，而另一个模型在其他任务上工作得更好。我们的结果对其他小词汇量领域的神经网络训练也有启示作用，如字节序列建模和蛋白质序列建模。我们发布了我们的源代码，并证明了我们的模型能够在REAPER数字音频工作站上实现实时人机交互合成摘要：We consider the task of multi-track MIDI infilling when arbitrary (track, measure) pairs of information have been deleted from a contiguous slice of measures from a MIDI file. We train two T5-like models to solve this task, one using a basic MIDI-like event vocabulary and one using a joined word-like version of this vocabulary. We introduce a new test set, created from the Lakh MIDI dataset, consisting of 9 multi-track MIDI infilling tasks. We evaluate our models on these tasks and find that one model works better on some tasks while the other works better on others. Our results have implications for the training of neural networks in other small-vocabulary domains, such as byte sequence modeling and protein sequence modeling. We release our source code, and we demonstrate that our models are capable of enabling real-time human-computer interactive composition in the REAPER digital audio workstation.
【6】AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
标题：AudioLDM：基于潜在扩散模型的文本到音频生成
链接：http://arxiv.org/pdf/2301.12503v1
作者：Haohe Liu,Zehua Chen,Yi Yuan,Xinhao Mei,Xubo Liu,Danilo Mandic,Wenwu Wang,Mark D. Plumbley机构：University of Surrey, UK 2Department of Electrical and Electronic Engineering备注：Demo and implementation at this https URL Evaluation toolbox at this https URL摘要：文本到音频（TTA）系统由于其能够基于文本描述合成通用音频而最近得到关注。然而，以往的TTA算法存在计算量大、发电质量差等问题.在本研究中，我们提出了AudioLDM，一个建立在潜在空间上的TTA系统，用于从对比语言音频预训练（CLAP）潜在空间中学习连续音频表征。预训练的CLAP模型使我们能够训练具有音频嵌入的LDM，同时在采样期间提供文本嵌入作为条件。AudioLDM通过学习音频信号及其组成的潜在表示而不对跨模态关系建模，在生成质量和计算效率方面都是有利的。AudioLDM在AudioCaps和单个GPU上进行了培训，可实现最先进的TTA性能（通过客观和主观指标衡量）（例如，弗雷谢距离）。此外，AudioLDM是第一个支持各种文本引导音频操作的TTA系统（例如，风格转移）。我们的实施和演示可在www.example.com上获得https: audioldm.github.io。摘要：Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https: audioldm.github.io.
【7】Artistic Curve Steganography Carried by Musical Audio
标题：音乐音频承载的艺术曲线隐写
链接：http://arxiv.org/pdf/2301.12354v1
备注：18 pages, 14 figures, in Proceedings of EvoMUSART 2023摘要：在这项工作中，我们创建了艺术闭环曲线，描绘出图像和3D形状，然后我们隐藏在音乐音频作为一种隐写术的形式。我们使用旅行推销员艺术来创建艺术平面环来描绘图像轮廓，我们使用三角形网格上的哈密顿循环来创建艺术空间环来填充3D表面。我们的嵌入方案被设计成在有损压缩后忠实地保留这些循环的几何形状，同时保持它们的存在不被音频收听者检测到。为了实现这一点，我们在不同频率中隐藏曲线的每个维度，并且我们扰动该频率的幅度的滑动窗口和以在该维度上最佳地匹配目标曲线，同时在该频率的相位中隐藏尺度信息。在这个过程中，我们利用曲线的几何特性来帮助更有效地隐藏和恢复它们。我们的方案是简单的和编码发生有效的非负最小二乘框架，而解码是平凡的。我们在大量的图像和音频数据集上定量验证了我们的技术，并展示了一个众包听力测试的结果，验证了隐藏的信息确实是不显眼的。摘要：In this work, we create artistic closed loop curves that trace out images and 3D shapes, which we then hide in musical audio as a form of steganography. We use traveling salesperson art to create artistic plane loops to trace out image contours, and we use Hamiltonian cycles on triangle meshes to create artistic space loops that fill out 3D surfaces. Our embedding scheme is designed to faithfully preserve the geometry of these loops after lossy compression, while keeping their presence undetectable to the audio listener. To accomplish this, we hide each dimension of the curve in a different frequency, and we perturb a sliding window sum of the magnitude of that frequency to best match the target curve at that dimension, while hiding scale information in that frequency's phase. In the process, we exploit geometric properties of the curves to help to more effectively hide and recover them. Our scheme is simple and encoding happens efficiently with a nonnegative least squares framework, while decoding is trivial. We validate our technique quantitatively on large datasets of images and audio, and we show results of a crowd sourced listening test that validate that the hidden information is indeed unobtrusive.
【8】Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model
标题：非自回归端到端ASR模型在识别时实现时间戳预测
链接：http://arxiv.org/pdf/2301.12343v1
作者：Xian Shi,Yanni Chen,Shiliang Zhang,Zhijie Yan机构：Speech Lab, Alibaba Group, China摘要：传统的ASR系统使用帧级音素后验进行力对齐（FA）并提供时间戳，而端到端的ASR系统尤其是基于AED的ASR系统缺乏这种能力。本文提出利用非自回归ASR模型Paraformer中的连续积分点火CIF机制，在识别的同时进行时间戳预测TP。针对CIF的起火点偏置问题，提出了包括起火点延迟和静音插入在内的后处理策略。此外，我们还提出使用scaled-CIF平滑CIF输出的权重，这对ASR和TP任务都是有益的。采用累积平均偏移量AAS和二值化误差率DER来衡量时间戳的质量，并将该系统与传统的混合力对准系统进行了比较。在人工标记时间戳测试集上的实验结果表明，提出的优化方法显著提高了CIF时间戳的准确性，AAS和DER分别降低了66.7%和82.1%。与用相同数据训练的Kaldi力对齐算法相比，优化CIF时间戳算法可使AAS相对减少12.3%。摘要：Conventional ASR systems use frame-level phoneme posterior to conduct force-alignment~(FA) and provide timestamps, while end-to-end ASR systems especially AED based ones are short of such ability. This paper proposes to perform timestamp prediction~(TP) while recognizing by utilizing continuous integrate-and-fire~(CIF) mechanism in non-autoregressive ASR model - Paraformer. Foucing on the fire place bias issue of CIF, we conduct post-processing strategies including fire-delay and silence insertion. Besides, we propose to use scaled-CIF to smooth the weights of CIF output, which is proved beneficial for both ASR and TP task. Accumulated averaging shift~(AAS) and diarization error rate~(DER) are adopted to measure the quality of timestamps and we compare these metrics of proposed system and conventional hybrid force-alignment system. The experiment results over manually-marked timestamps testset show that the proposed optimization methods significantly improve the accuracy of CIF timestamps, reducing 66.7 % and 82.1 % of AAS and DER respectively. Comparing to Kaldi force-alignment trained with the same data, optimized CIF timestamps achieved 12.3 % relative AAS reduction.
【9】Time out of Mind: Generating Emotionally Conditioned Rate of Speech
标题：心不在焉的时间：产生情绪调节的语速
链接：http://arxiv.org/pdf/2301.12331v1
作者：Navjot Kaur,Paige Tuttosi摘要：语音合成在过去的十年中有了显著的改进，产生了高度可理解的语音。进一步的研究已经产生了可以产生可变语音的模型，包括条件情感表达。然而，问题在于对短语级修改和韵律声乐特征的关注。使用CREMA-D数据集，我们已经训练了一个基于情感的GAN来为给定的输入文本生成有价值的长度。这些单词长度与中性语音有关，并且可以通过语音合成标记语言（SSML）提供给文本到语音（TTS）系统以生成更具表达力的语音。与开箱即用的模型相比，我们能够在中性语音的客观测量上实现更好的性能，并且在快乐语音的时间对准上实现更好的性能。摘要：Voice synthesis has seen significant improvements in the past decade resulting in highly intelligible voices. Further investigations have resulted in models that can produce variable speech, including conditional emotional expression. The problem lies, however, in a focus on phrase level modifications and prosodic vocal features. Using the CREMA-D dataset we have trained a GAN conditioned on emotion to generate worth lengths for a given input text. These word lengths are relative to neutral speech and can be provided, through speech synthesis markup language (SSML) to a text to speech (TTS) system to generate more expressive speech. We were able to achieve better performances on objective measures for neutral speech, and better time alignment for happy speech when compared to an out of box model.
【10】who is snoring? snore based user recognition
标题：谁在打呼噜？基于鼾声的用户识别
链接：http://arxiv.org/pdf/2301.12209v1
作者：Shenghao Li,Jagmohan Chauhan机构：University of Southampton摘要：打鼾是阻塞性睡眠呼吸暂停低通气综合征（OSAH）的最突出症状之一，OSAH是一种导致上呼吸道反复塌陷和停止的高度流行的疾病。因此，准确的鼾声监测和分析至关重要。然而，传统的监测方法多导睡眠图（PSG）需要患者在睡眠诊所呆上一整夜，并连接到许多件设备。另一种侵入性较小的方法是在家中或临床环境中使用智能手机进行被动监测。但是，存在一个挑战：环境可以由人共享，使得原始音频可以包含床伴或其他人的打鼾活动。打鼾活动的错误捕获可能导致严重的错误警报和患者的误诊。为了解决这一局限性，我们提出了一个假设，即鼾声包含唯一的身份信息，可以用于用户识别。我们分析了各种机器学习模型：高斯混合模型（GMM）、GMM-UBM（通用背景模型）和MPSSC上的深度神经网络（DNN）-用于评估我们假设有效性的开源打鼾数据集。我们的结果是有希望的，因为我们在识别和验证任务中实现了约90%的准确率。这项工作标志着了解基于打鼾的用户监测的实用性以实现多种医疗保健应用的第一步。摘要：Snoring is one of the most prominent symptoms of Obstructive Sleep Apnea-Hypopnea Syndrome (OSAH), a highly prevalent disease that causes repetitive collapse and cessation of the upper airway. Thus, accurate snore sound monitoring and analysis is crucial. However, the traditional monitoring method polysomnography (PSG) requires the patients to stay at a sleep clinic for the whole night and be connected to many pieces of equipment. An alternative and less invasive way is passive monitoring using a smartphone at home or in the clinical settings. But, there is a challenge: the environment may be shared by people such that the raw audio may contain the snore activities of the bed partner or other person. False capturing of the snoring activity could lead to critical false alarms and misdiagnosis of the patients. To address this limitation, we propose a hypothesis that snore sound contains unique identity information which can be used for user recognition. We analyzed various machine learning models: Gaussian Mixture Model (GMM), GMM-UBM (Universial Background Model), and a Deep Neural Network (DNN) on MPSSC - an open source snoring dataset to evaluate the validity of our hypothesis. Our results are promising as we achieved around 90% accuracy in identification and verification tasks. This work marks the first step towards understanding the practicality of snore based user monitoring to enable multiple healthcare applicaitons.
【11】Automated Arrangements of Multi-Part Music for Sets of Monophonic Instruments
标题：单音乐器套装多声部音乐的自动编排
链接：http://arxiv.org/pdf/2301.12084v1
作者：Matthew Mccloskey,Gabrielle Curcio,Amulya Badineni,Kevin Mcgrath,Dimitris Papamichail机构：ITRIS PAPAMICHAIL, The College of New Jersey, USA摘要：为一组不同的乐器编排音乐，这是一个传统的乏味和耗时的过程，由对特定乐器具有复杂知识的专家来执行，并涉及大量的实验。本文主要研究为单声道乐器或人声创作的音乐作品的自动编曲问题。我们设计并实现了一种算法，通过将音乐片段移调到不同的音阶、将分配的部分置换为乐器 人声、以及将各个部分移调一个或多个八度音阶，在可行的情况下，该算法总是可以产生音乐安排。我们还发布了用Python编写的开源软件，它可以处理MusicXML文件，并允许音乐家尝试音乐编排。这是我们的希望，我们的软件可以作为一个平台，为未来的扩展，将包括音乐减少和包括复调乐器。摘要：Arranging music for a different set of instruments that it was originally written for is traditionally a tedious and time-consuming process, performed by experts with intricate knowledge of the specific instruments and involving significant experimentation. In this paper we study the problem of automating music arrangements for music pieces written for monophonic instruments or voices. We designed and implemented an algorithm that can always produce a music arrangement when feasible by transposing the music piece to a different scale, permuting the assigned parts to instruments voices, and transposing individual parts by one or more octaves. We also published open source software written in Python that processes MusicXML files and allows musicians to experiment with music arrangements. It is our hope that our software can serve as a platform for future extensions that will include music reductions and inclusion of polyphonic instruments.
【12】Byte Pair Encoding for Symbolic Music
标题：符号音乐的字节对编码
链接：http://arxiv.org/pdf/2301.11975v1
作者：Nathan Fradet,Jean-Pierre Briot,Fabien Chhel,Amal El Fallah Seghrouchni,Nicolas Gutowski机构：Sorbonne University - CNRS, France 4University of Angers备注：Source code at this https URL摘要：符号音乐模态现在大多表示为离散的，并与序列模型（如Transformers）一起用于深度学习任务。最近的研究致力于标记化，即将数据转换为此类模型可理解的整数序列。这可以通过许多方式来实现，因为音乐可以由同时的音轨、具有几个属性的同时的音符组成。到目前为止，提出的标记化都是基于描述音符属性和时间事件的小词汇表，导致相当长的标记序列。在本文中，我们将展示字节对编码（BPE）如何在提高深度学习模型性能的同时改善其结果。我们在音乐生成和作曲家分类上进行了实验，研究了BPE对模型学习嵌入的影响，并表明它有助于增加嵌入的各向同性，即：它们在空间中的位置变化的均匀性。摘要：The symbolic music modality is nowadays mostly represented as discrete and used with sequential models such as Transformers, for deep learning tasks. Recent research put efforts on the tokenization, i.e. the conversion of data into sequences of integers intelligible to such models. This can be achieved by many ways as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations are based on small vocabularies describing the note attributes and time events, resulting in fairly long token sequences. In this paper, we show how Byte Pair Encoding (BPE) can improve the results of deep learning models while improving its performances. We experiment on music generation and composer classification, and study the impact of BPE on how models learn the embeddings, and show that it can help to increase their isotropy, i.e., the uniformity of the variance of their positions in the space.
【13】KalmanNet: A Learnable Kalman Filter for Acoustic Echo Cancellation
标题：KalmanNet：一种用于声回波消除的可学习卡尔曼滤波器
链接：http://arxiv.org/pdf/2301.12363v1
作者：Yixuan Zhang,Meng Yu,Hao Zhang,Dong Yu,DeLiang Wang机构：Tencent AI Lab, Bellevue, WA, USA, The Ohio State University, Columbus, OH, USA摘要：卡尔曼滤波器由于其对双端通话的鲁棒性和快速收敛而被广泛用于解决声学回声消除（AEC）问题。然而，不能对非线性进行建模以及需要调整控制参数对这种自适应滤波算法造成了限制。本文将频域卡尔曼滤波器（FDKF）和深度神经网络（DNNs）集成为一种混合方法，称为卡尔曼网（KalmanNet），以利用深度学习和自适应滤波算法的优势。具体地说，我们使用DNN来估计FDKF算法状态方程中的非线性失真远端信号、转移因子和非线性转移函数。实验结果表明，提出的卡尔曼网算法显著提高了FDKF的性能，优于强基线算法。摘要：The Kalman filter is widely used for addressing acoustic echo cancellation (AEC) problems due to their robustness to double-talk and fast convergence. However, the inability to model nonlinearity and the need to tune control parameters cast limitations on such adaptive filtering algorithms. In this paper, we integrate the frequency domain Kalman filter (FDKF) and deep neural networks (DNNs) into a hybrid method, called KalmanNet, to leverage the advantages of deep learning and adaptive filtering algorithms. Specifically, we employ a DNN to estimate nonlinearly distorted far-end signals, a transition factor, and the nonlinear transition function in the state equation of the FDKF algorithm. Experimental results show that the proposed KalmanNet improves the performance of FDKF significantly and outperforms strong baseline methods.
【14】Cross-domain Neural Pitch and Periodicity Estimation
标题：跨域神经网络基音周期估计
链接：http://arxiv.org/pdf/2301.12258v1
作者：Max Morrison,Caedon Hsieh,Nathan Pruyne,Bryan Pardo机构：andBryanPardoare with the Department of Computer Science摘要：音高是我们感知音频信号的一个基本方面。音高轮廓通常用于分析语音和音乐信号，并作为许多音频任务的输入特征，包括音乐转录、歌声合成和韵律编辑。在这篇文章中，我们描述了一套技术，以提高准确性的国家的最先进的神经节距和周期估计器。我们还介绍了一种新颖的基于熵的方法，用于从基于统计推断的音调估计器中提取周期性和每帧浊音-清音分类（例如，神经网络），并且示出了如何训练神经音调估计器以同时处理语音和音乐而不降低性能。虽然神经音调...在历史上比基于信号处理的音调...慢得多，但是我们的估计器实现方式在标准CPU上接近基于DSP的最新音调估计器的速度，但是具有明显更精确的音调和周期性估计。我们的实验表明，在没有硬件优化的情况下，用PyTorch编写的跳距为10毫秒的精确跨域基音和周期估计器在Intel i9- 9820 X 10核3.30 GHz CPU上的运行速度比实时快11.2倍，在NVIDIA GeForce RTX 3090 GPU上的运行速度比实时快408倍。我们将所有代码和模型发布为音调估计神经网络（pitch-Estimating Neural Networks，penn），这是一个开源的、可pip安装的Python模块，用于训练、评估音调和周期估计神经网络，并执行推理。宾州大学的代码可以在www.example.com上找到https: github.com interactiveaudiolab penn。摘要：Pitch is a foundational aspect of our perception of audio signals. Pitch contours are commonly used to analyze speech and music signals and as input features for many audio tasks, including music transcription, singing voice synthesis, and prosody editing. In this paper, we describe a set of techniques for improving the accuracy of state-of-the-art neural pitch and periodicity estimators. We also introduce a novel entropy-based method for extracting periodicity and per-frame voiced-unvoiced classifications from statistical inference-based pitch estimators (e.g., neural networks), and show how to train a neural pitch estimator to simultaneously handle speech and music without performance degradation. While neural pitch trackers have historically been significantly slower than signal processing based pitch trackers, our estimator implementations approach the speed of state-of-the-art DSP-based pitch estimators on a standard CPU, but with significantly more accurate pitch and periodicity estimation. Our experiments show that an accurate, cross-domain pitch and periodicity estimator written in PyTorch with a hopsize of ten milliseconds can run 11.2x faster than real-time on a Intel i9-9820X 10-core 3.30 GHz CPU or 408x faster than real-time on a NVIDIA GeForce RTX 3090 GPU without hardware optimization. We release all of our code and models as Pitch-Estimating Neural Networks (penn), an open-source, pip-installable Python module for training, evaluating, and performing inference with pitch- and periodicity-estimating neural networks. The code for penn is available at https: github.com interactiveaudiolab penn.
【1】MYRiAD: A Multi-Array Room Acoustic Database
标题：Myriad：一个多阵列房间声学数据库
链接：http://arxiv.org/pdf/2301.13057v1
作者：Thomas Dietzen,Randall Ali,Maja Taseska,Toon van Waterschoot备注：submitted for publication摘要：在声信号处理算法的发展过程中，在各种声环境中对算法进行评估是至关重要的。为了推进现实和可再现场景中的评估，多年来开发了几个高质量声学数据库。在本文中，我们提出了另一个声学记录的补充数据库，称为多阵列房间声学数据库（MYRiAD）。MYRiAD数据库的独特之处在于其麦克风配置的多样性，适用于广泛的增强和再现应用（诸如辅助听力、电话会议或声音分区）、两个记录空间的声学特性以及包括1214个房间脉冲响应的所包含信号的种类（RIR）、再现的语音、音乐和固定噪音，以及在两个房间举行的现场鸡尾酒会的录音。麦克风配置包括一个带有入耳式全向麦克风的仿真头（DH）、两个各配备2个全向麦克风的耳背式（BTE）麦克风、5个外置全向麦克风（XM）以及两个共包含12个全向麦克风的同心圆形麦克风阵列（CMA）。两个录音空间，即SONORA音频实验室（SAL）和Alamire交互实验室（AIL），混响时间分别为2.1秒和0.5秒。使用SAL中的10个可移动扬声器和AIL中的24个扬声器的内置阵列再现音频信号。MATLAB和Python脚本用于访问信号以及麦克风和扬声器坐标。该数据库可在[1]上公开获取。摘要：In the development of acoustic signal processing algorithms, their evaluation in various acoustic environments is of utmost importance. In order to advance evaluation in realistic and reproducible scenarios, several high-quality acoustic databases have been developed over the years. In this paper, we present another complementary database of acoustic recordings, referred to as the Multi-arraY Room Acoustic Database (MYRiAD). The MYRiAD database is unique in its diversity of microphone configurations suiting a wide range of enhancement and reproduction applications (such as assistive hearing, teleconferencing, or sound zoning), the acoustics of the two recording spaces, and the variety of contained signals including 1214 room impulse responses (RIRs), reproduced speech, music, and stationary noise, as well as recordings of live cocktail parties held in both rooms. The microphone configurations comprise a dummy head (DH) with in-ear omnidirectional microphones, two behind-the-ear (BTE) pieces equipped with 2 omnidirectional microphones each, 5 external omnidirectional microphones (XMs), and two concentric circular microphone arrays (CMAs) consisting of 12 omnidirectional microphones in total. The two recording spaces, namely the SONORA Audio Laboratory (SAL) and the Alamire Interactive Laboratory (AIL), have reverberation times of 2.1s and 0.5s, respectively. Audio signals were reproduced using 10 movable loudspeakers in the SAL and a built-in array of 24 loudspeakers in the AIL. MATLAB and Python scripts are included for accessing the signals as well as microphone and loudspeaker coordinates. The database is publicly available at [1].
【2】Real-Time Acoustic Perception for Automotive Applications
标题：汽车应用的实时声学感知
链接：http://arxiv.org/pdf/2301.12808v1
作者：Jun Yin,Stefano Damiano,Marian Verhelst,Toon van Waterschoot,Andre Guntoro机构：∗ESAT-MICAS KU Leuven, †ESAT-STADIUS KU Leuven, ‡Robert Bosch GmbH摘要：近年来，汽车行业一直在大力推动智能汽车的发展，智能汽车配备有多模态传感器来收集关于周围环境的信息，以便辅助人类驾驶员或做出自主决策。虽然焦点主要集中在视觉传感器上，但声学事件对于检测需要改变驾驶行为的情况也是至关重要的，例如汽车鸣笛或接近的紧急车辆的警笛。本文总结了Marie Sklodowska-Curie Actions（MSCA）欧洲工业博士学位（EID）项目智能超低功耗汽车信号处理（I-SPOT）迄今为止取得的成果。在算法方面，I-SPOT项目旨在通过联合开发专门针对汽车应用的麦克风阵列处理和深度学习技术，实现对环境音频信号的检测、定位和跟踪。开发了数据生成软件，以涵盖I-SPOT目标情景和研究挑战。该工具目前正被用于开发低复杂度的深度学习技术，用于紧急声音检测。在硬件方面，该目标推动硬件-算法协同设计的工作流，以简化对算法演进足够灵活而不放弃效率的架构的生成，以及实现算法决策的硬件含义的快速反馈。这是通过将硬件算法设计空间分解为合理子集的分层工作流来实现的，该工作流已针对边缘设备的最先进稳健声源定位的操作员级优化进行了测试。此外，为I-SPOT的下一阶段澄清了端到端系统面临的几个公开挑战。摘要：In recent years the automotive industry has been strongly promoting the development of smart cars, equipped with multi-modal sensors to gather information about the surroundings, in order to aid human drivers or make autonomous decisions. While the focus has mostly been on visual sensors, also acoustic events are crucial to detect situations that require a change in the driving behavior, such as a car honking, or the sirens of approaching emergency vehicles. In this paper, we summarize the results achieved so far in the Marie Sklodowska-Curie Actions (MSCA) European Industrial Doctorates (EID) project Intelligent Ultra Low-Power Signal Processing for Automotive (I-SPOT). On the algorithmic side, the I-SPOT Project aims to enable detecting, localizing and tracking environmental audio signals by jointly developing microphone array processing and deep learning techniques that specifically target automotive applications. Data generation software has been developed to cover the I-SPOT target scenarios and research challenges. This tool is currently being used to develop low-complexity deep learning techniques for emergency sound detection. On the hardware side, the goal impels workflows for hardware-algorithm co-design to ease the generation of architectures that are sufficiently flexible towards algorithmic evolutions without giving up on efficiency, as well as enable rapid feedback of hardware implications of algorithmic decision. This is pursued though a hierarchical workflow that breaks the hardware-algorithm design space into reasonable subsets, which has been tested for operator-level optimizations on state-of-the-art robust sound source localization for edge devices. Further, several open challenges towards an end-to-end system are clarified for the next stage of I-SPOT.
【3】Learning to Speak from Text: Zero-Shot Multilingual Text-to-Speech with Unsupervised Text Pretraining
标题：学习从文本说话：无监督文本预训练的零命中率多语言文本到语音转换
链接：http://arxiv.org/pdf/2301.12596v1
作者：Takaaki Saeki,Soumi Maiti,Xinjian Li,Shinji Watanabe,Shinnosuke Takamichi,Hiroshi Saruwatari机构：The University of Tokyo, Japan, Carnegie Mellon University, USA摘要：虽然神经文本到语音（TTS）已经实现了类似人类的自然合成语音，但是由于需要成对的文本和录音室质量的音频数据，多语言TTS系统被限制为资源丰富的语言。提出了一种使用纯文本数据的目标语言zero-shot多语种TTS方法。纯文本数据的使用允许为仅文本资源可用的低资源语言开发TTS系统，使得TTS可用于数千种语言。受多语言语言模型强大的跨语言可移植性的启发，我们的框架首先使用多语言纯文本数据执行掩蔽语言模型预训练。然后我们用成对数据以监督的方式训练这个模型，同时冻结一个语言感知的嵌入层。这甚至允许对不包括在配对数据中但存在于纯文本数据中的语言进行推断。评估结果表明，对于一种看不见的语言，zero-shot TTS具有低于12%的字符错误率。所有实验均使用公共数据集进行，并且实施将可用于重现性。摘要：While neural text-to-speech (TTS) has achieved human-like natural synthetic speech, multilingual TTS systems are limited to resource-rich languages due to the need for paired text and studio-quality audio data. This paper proposes a method for zero-shot multilingual TTS using text-only data for the target language. The use of text-only data allows the development of TTS systems for low-resource languages for which only textual resources are available, making TTS accessible to thousands of languages. Inspired by the strong cross-lingual transferability of multilingual language models, our framework first performs masked language model pretraining with multilingual text-only data. Then we train this model with a paired data in a supervised manner, while freezing a language-aware embedding layer. This allows inference even for languages not included in the paired data but present in the text-only data. Evaluation results demonstrate highly intelligible zero-shot TTS with a character error rate of less than 12% for an unseen language. All experiments were conducted using public datasets and the implementation will be made available for reproducibility.
【4】KalmanNet: A Learnable Kalman Filter for Acoustic Echo Cancellation
标题：KalmanNet：一种用于声回波消除的可学习卡尔曼滤波器
链接：http://arxiv.org/pdf/2301.12363v1
*与cs.SD语音【13】为同一篇
作者：Yixuan Zhang,Meng Yu,Hao Zhang,Dong Yu,DeLiang Wang机构：Tencent AI Lab, Bellevue, WA, USA, The Ohio State University, Columbus, OH, USA摘要：卡尔曼滤波器由于其对双端通话的鲁棒性和快速收敛而被广泛用于解决声学回声消除（AEC）问题。然而，不能对非线性进行建模以及需要调整控制参数对这种自适应滤波算法造成了限制。本文将频域卡尔曼滤波器（FDKF）和深度神经网络（DNNs）集成为一种混合方法，称为卡尔曼网（KalmanNet），以利用深度学习和自适应滤波算法的优势。具体地说，我们使用DNN来估计FDKF算法状态方程中的非线性失真远端信号、转移因子和非线性转移函数。实验结果表明，提出的卡尔曼网算法显著提高了FDKF的性能，优于强基线算法。摘要：The Kalman filter is widely used for addressing acoustic echo cancellation (AEC) problems due to their robustness to double-talk and fast convergence. However, the inability to model nonlinearity and the need to tune control parameters cast limitations on such adaptive filtering algorithms. In this paper, we integrate the frequency domain Kalman filter (FDKF) and deep neural networks (DNNs) into a hybrid method, called KalmanNet, to leverage the advantages of deep learning and adaptive filtering algorithms. Specifically, we employ a DNN to estimate nonlinearly distorted far-end signals, a transition factor, and the nonlinear transition function in the state equation of the FDKF algorithm. Experimental results show that the proposed KalmanNet improves the performance of FDKF significantly and outperforms strong baseline methods.
【5】Cross-domain Neural Pitch and Periodicity Estimation
标题：跨域神经网络基音周期估计
链接：http://arxiv.org/pdf/2301.12258v1
*与cs.SD语音【14】为同一篇
作者：Max Morrison,Caedon Hsieh,Nathan Pruyne,Bryan Pardo机构：andBryanPardoare with the Department of Computer Science摘要：音高是我们感知音频信号的一个基本方面。音高轮廓通常用于分析语音和音乐信号，并作为许多音频任务的输入特征，包括音乐转录、歌声合成和韵律编辑。在这篇文章中，我们描述了一套技术，以提高准确性的国家的最先进的神经节距和周期估计器。我们还介绍了一种新颖的基于熵的方法，用于从基于统计推断的音调估计器中提取周期性和每帧浊音-清音分类（例如，神经网络），并且示出了如何训练神经音调估计器以同时处理语音和音乐而不降低性能。虽然神经音调...在历史上比基于信号处理的音调...慢得多，但是我们的估计器实现方式在标准CPU上接近基于DSP的最新音调估计器的速度，但是具有明显更精确的音调和周期性估计。我们的实验表明，在没有硬件优化的情况下，用PyTorch编写的跳距为10毫秒的精确跨域基音和周期估计器在Intel i9- 9820 X 10核3.30 GHz CPU上的运行速度比实时快11.2倍，在NVIDIA GeForce RTX 3090 GPU上的运行速度比实时快408倍。我们将所有代码和模型发布为音调估计神经网络（pitch-Estimating Neural Networks，penn），这是一个开源的、可pip安装的Python模块，用于训练、评估音调和周期估计神经网络，并执行推理。宾州大学的代码可以在www.example.com上找到https: github.com interactiveaudiolab penn。摘要：Pitch is a foundational aspect of our perception of audio signals. Pitch contours are commonly used to analyze speech and music signals and as input features for many audio tasks, including music transcription, singing voice synthesis, and prosody editing. In this paper, we describe a set of techniques for improving the accuracy of state-of-the-art neural pitch and periodicity estimators. We also introduce a novel entropy-based method for extracting periodicity and per-frame voiced-unvoiced classifications from statistical inference-based pitch estimators (e.g., neural networks), and show how to train a neural pitch estimator to simultaneously handle speech and music without performance degradation. While neural pitch trackers have historically been significantly slower than signal processing based pitch trackers, our estimator implementations approach the speed of state-of-the-art DSP-based pitch estimators on a standard CPU, but with significantly more accurate pitch and periodicity estimation. Our experiments show that an accurate, cross-domain pitch and periodicity estimator written in PyTorch with a hopsize of ten milliseconds can run 11.2x faster than real-time on a Intel i9-9820X 10-core 3.30 GHz CPU or 408x faster than real-time on a NVIDIA GeForce RTX 3090 GPU without hardware optimization. We release all of our code and models as Pitch-Estimating Neural Networks (penn), an open-source, pip-installable Python module for training, evaluating, and performing inference with pitch- and periodicity-estimating neural networks. The code for penn is available at https: github.com interactiveaudiolab penn.
【6】Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
标题：基于分层蒸馏的CIF语音识别器知识传递
链接：http://arxiv.org/pdf/2301.13003v1
*与cs.SD语音【1】为同一篇
作者：Minglun Han,Feilong Chen,Jing Shi,Shuang Xu,Bo Xu机构：Institute of Automation, Chinese Academy of Sciences, Beijing, China, School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China备注：5 pages; Keywords: speech recognition, continuous integrate-and-fire, knowledge distillation, contrastive learning, pre-trained language models摘要：大规模预训练语言模型具有强大的语言建模能力，在自然语言处理中得到了广泛的应用。对于自动语音识别（ASR），利用PLM来提高性能也成为一个很有前途的研究方向。然而，大多数先前的工作可能遭受PLM的不灵活的大小和结构，以及在PLM中的知识利用不足。为了解决这些问题，我们提出了基于连续积分点火（CIF）的ASR模型的层次化知识提取。具体地，我们通过在声学水平上应用具有对比损失的跨模态提取和在语言水平上应用具有回归损失的提取来将知识从PLM提取到ASR模型。在AISHELL-1数据集上，我们的方法实现了比原始基于CIF的模型减少15%的相对错误率，并实现了与最先进模型相当的性能（开发 测试上的3.8% 4.1%）。摘要：Large-scale pre-trained language models (PLMs) with powerful language modeling capabilities have been widely used in natural language processing. For automatic speech recognition (ASR), leveraging PLMs to improve performance has also become a promising research trend. However, most previous works may suffer from the inflexible sizes and structures of PLMs, along with the insufficient utilization of the knowledge in PLMs. To alleviate these problems, we propose the hierarchical knowledge distillation on the continuous integrate-and-fire (CIF) based ASR models. Specifically, we distill the knowledge from PLMs to the ASR model by applying cross-modal distillation with contrastive loss at the acoustic level and applying distillation with regression loss at the linguistic level. On the AISHELL-1 dataset, our method achieves 15% relative error rate reduction over the original CIF-based model and achieves comparable performance (3.8% 4.1% on dev test) to the state-of-the-art model.
【7】GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration
标题：GibbsDDRM：去噪扩散恢复盲逆问题的部分折叠Gibbs采样器
链接：http://arxiv.org/pdf/2301.12686v1
*与cs.SD语音【2】为同一篇
作者：Naoki Murata,Koichi Saito,Chieh-Hsin Lai,Yuhta Takida,Toshimitsu Uesaka,Yuki Mitsufuji,Stefano Ermon摘要：预先训练的扩散模型已经成功地用作各种线性逆问题中的先验，其中目标是从噪声线性测量重构信号。然而，现有的方法需要线性算子的知识。本文提出了GibbsDDRM，它是去噪扩散恢复模型（DDRM）在线性测量算子未知的盲背景下的一种扩展. GibbsDDRM通过对数据先验使用预先训练的扩散模型来构造数据、测量和线性算子的联合分布，并且它通过使用Gibbs采样器的有效变体的后验采样来解决该问题。所提出的方法是问题不可知的，这意味着预先训练的扩散模型可以应用于各种反问题而无需微调。在实验中，该算法在盲图像去模糊和语音去混响两个任务上都取得了很高的性能，尽管对底层线性算子使用了简单的通用先验。摘要：Pre-trained diffusion models have been successfully used as priors in a variety of linear inverse problems, where the goal is to reconstruct a signal from noisy linear measurements. However, existing approaches require knowledge of the linear operator. In this paper, we propose GibbsDDRM, an extension of Denoising Diffusion Restoration Models (DDRM) to a blind setting in which the linear measurement operator is unknown. GibbsDDRM constructs a joint distribution of the data, measurements, and linear operator by using a pre-trained diffusion model for the data prior, and it solves the problem by posterior sampling with an efficient variant of a Gibbs sampler. The proposed method is problem-agnostic, meaning that a pre-trained diffusion model can be applied to various inverse problems without fine tuning. In experiments, it achieved high performance on both blind image deblurring and vocal dereverberation tasks, despite the use of simple generic priors for the underlying linear operators.
【8】SingSong: Generating musical accompaniments from singing
标题：SING：从歌声中产生音乐伴奏
链接：http://arxiv.org/pdf/2301.12662v1
*与cs.SD语音【3】为同一篇
作者：Chris Donahue,Antoine Caillon,Adam Roberts,Ethan Manilow,Philippe Esling,Andrea Agostinelli,Mauro Verzetti,Ian Simon,Olivier Pietquin,Neil Zeghidour,Jesse Engel摘要：我们介绍SingSong，一个生成器乐来为输入人声伴奏的系统，它可能为音乐家和非音乐家提供一种直观的新方法来创作具有自己声音的音乐。为了实现这一点，我们建立在音乐源分离和音频生成的最新发展。具体来说，我们将最先进的源分离算法应用于大型音乐音频语料库，以产生对齐的人声和乐器源对。然后，我们采用AudioLM（Borsos等人，2022）--一种最先进的无条件音频生成方法--适合于有条件的“音频到音频”生成任务，并在源分离（声乐、器乐）对上训练它。在相同声音输入的两两比较中，与来自强检索基线的乐器相比，听者对由《歌唱》生成的乐器表现出显著的偏好。声音示例，请访问https: g.co magenta singsong摘要：We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be suitable for conditional "audio-to-audio" generation tasks, and train it on the source-separated (vocal, instrumental) pairs. In a pairwise comparison with the same vocal inputs, listeners expressed a significant preference for instrumentals generated by SingSong compared to those from a strong retrieval baseline. Sound examples at https: g.co magenta singsong
【9】Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
标题：制作音频：使用即时增强的扩散模型生成文本到音频
链接：http://arxiv.org/pdf/2301.12661v1
*与cs.SD语音【4】为同一篇
作者：Rongjie Huang,Jiawei Huang,Dongchao Yang,Yi Ren,Luping Liu,Mingze Li,Zhenhui Ye,Jinglin Liu,Xiang Yin,Zhou Zhao机构：20 2 2; HongEqual contribution 1Zhejiang University 2Peking University 3Speech & Audio Team备注：Audio samples are available at this https URL摘要：大规模的多模态生成式建模已经在文本到图像和文本到视频的生成中创造了里程碑。其在音频方面的应用仍然滞后，主要原因有两个：缺乏具有高质量文本-音频对的大规模数据集，以及对长的连续音频数据建模的复杂性。在本研究中，我们提出了一个即时增强扩散模型来解决这些问题：1）通过提取然后重新编程的方法来引入伪即时增强，它通过使用无语言的音频来缓解数量级概念组合的数据短缺;2）利用谱图自动编码器来预测自监督音频表示而不是波形。结合强大的对比语言音频预训练（CLAP）表示，Make-An-Audio在客观和主观基准评估中都取得了最先进的结果。此外，我们还介绍了它的可控性和通用性，用于“无模态遗留”的X到音频，首次解锁了在给定用户定义的模态输入的情况下生成高清晰度、高保真度音频的能力。音频样本可从www.example.com获得https: Text-to-Audio.github.io摘要：Large-scale multimodal generative modeling has created milestones in text-to-image and text-to-video generation. Its application to audio still lags behind for two main reasons: the lack of large-scale datasets with high-quality text-audio pairs, and the complexity of modeling long continuous audio data. In this work, we propose Make-An-Audio with a prompt-enhanced diffusion model that addresses these gaps by 1) introducing pseudo prompt enhancement with a distill-then-reprogram approach, it alleviates data scarcity with orders of magnitude concept compositions by using language-free audios; 2) leveraging spectrogram autoencoder to predict the self-supervised audio representation instead of waveforms. Together with robust contrastive language-audio pretraining (CLAP) representations, Make-An-Audio achieves state-of-the-art results in both objective and subjective benchmark evaluation. Moreover, we present its controllability and generalization for X-to-Audio with "No Modality Left Behind", for the first time unlocking the ability to generate high-definition, high-fidelity audios given a user-defined modality input. Audio samples are available at https: Text-to-Audio.github.io
【10】Composer's Assistant: Interactive Transformers for Multi-Track MIDI Infilling
标题：作曲家助手：用于多声道MIDI填充的交互式转换器
链接：http://arxiv.org/pdf/2301.12525v1
*与cs.SD语音【5】为同一篇
备注：16 pages, 7 figures, 3 tables摘要：我们考虑当任意（音轨，小节）信息对从MIDI文件的连续小节片段中删除时的多音轨MIDI填充任务。我们训练两个类似T5的模型来解决这个任务，一个使用基本的类似MIDI的事件词汇表，另一个使用这个词汇表的类似连接词的版本。我们介绍了一个新的测试集，从Lakh MIDI数据集创建，包括9个多轨MIDI填充任务。我们在这些任务上评估我们的模型，发现一个模型在一些任务上工作得更好，而另一个模型在其他任务上工作得更好。我们的结果对其他小词汇量领域的神经网络训练也有启示作用，如字节序列建模和蛋白质序列建模。我们发布了我们的源代码，并证明了我们的模型能够在REAPER数字音频工作站上实现实时人机交互合成摘要：We consider the task of multi-track MIDI infilling when arbitrary (track, measure) pairs of information have been deleted from a contiguous slice of measures from a MIDI file. We train two T5-like models to solve this task, one using a basic MIDI-like event vocabulary and one using a joined word-like version of this vocabulary. We introduce a new test set, created from the Lakh MIDI dataset, consisting of 9 multi-track MIDI infilling tasks. We evaluate our models on these tasks and find that one model works better on some tasks while the other works better on others. Our results have implications for the training of neural networks in other small-vocabulary domains, such as byte sequence modeling and protein sequence modeling. We release our source code, and we demonstrate that our models are capable of enabling real-time human-computer interactive composition in the REAPER digital audio workstation.
【11】AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
标题：AudioLDM：基于潜在扩散模型的文本到音频生成
链接：http://arxiv.org/pdf/2301.12503v1
*与cs.SD语音【6】为同一篇
作者：Haohe Liu,Zehua Chen,Yi Yuan,Xinhao Mei,Xubo Liu,Danilo Mandic,Wenwu Wang,Mark D. Plumbley机构：University of Surrey, UK 2Department of Electrical and Electronic Engineering备注：Demo and implementation at this https URL Evaluation toolbox at this https URL摘要：文本到音频（TTA）系统由于其能够基于文本描述合成通用音频而最近得到关注。然而，以往的TTA算法存在计算量大、发电质量差等问题.在本研究中，我们提出了AudioLDM，一个建立在潜在空间上的TTA系统，用于从对比语言音频预训练（CLAP）潜在空间中学习连续音频表征。预训练的CLAP模型使我们能够训练具有音频嵌入的LDM，同时在采样期间提供文本嵌入作为条件。AudioLDM通过学习音频信号及其组成的潜在表示而不对跨模态关系建模，在生成质量和计算效率方面都是有利的。AudioLDM在AudioCaps和单个GPU上进行了培训，可实现最先进的TTA性能（通过客观和主观指标衡量）（例如，弗雷谢距离）。此外，AudioLDM是第一个支持各种文本引导音频操作的TTA系统（例如，风格转移）。我们的实施和演示可在www.example.com上获得https: audioldm.github.io。摘要：Text-to-audio (TTA) system has recently gained attention for its ability to synthesize general audio based on text descriptions. However, previous studies in TTA have limited generation quality with high computational costs. In this study, we propose AudioLDM, a TTA system that is built on a latent space to learn the continuous audio representations from contrastive language-audio pretraining (CLAP) latents. The pretrained CLAP models enable us to train LDMs with audio embedding while providing text embedding as a condition during sampling. By learning the latent representations of audio signals and their compositions without modeling the cross-modal relationship, AudioLDM is advantageous in both generation quality and computational efficiency. Trained on AudioCaps with a single GPU, AudioLDM achieves state-of-the-art TTA performance measured by both objective and subjective metrics (e.g., frechet distance). Moreover, AudioLDM is the first TTA system that enables various text-guided audio manipulations (e.g., style transfer) in a zero-shot fashion. Our implementation and demos are available at https: audioldm.github.io.
【12】Artistic Curve Steganography Carried by Musical Audio
标题：音乐音频承载的艺术曲线隐写
链接：http://arxiv.org/pdf/2301.12354v1
*与cs.SD语音【7】为同一篇
备注：18 pages, 14 figures, in Proceedings of EvoMUSART 2023摘要：在这项工作中，我们创建了艺术闭环曲线，描绘出图像和3D形状，然后我们隐藏在音乐音频作为一种隐写术的形式。我们使用旅行推销员艺术来创建艺术平面环来描绘图像轮廓，我们使用三角形网格上的哈密顿循环来创建艺术空间环来填充3D表面。我们的嵌入方案被设计成在有损压缩后忠实地保留这些循环的几何形状，同时保持它们的存在不被音频收听者检测到。为了实现这一点，我们在不同频率中隐藏曲线的每个维度，并且我们扰动该频率的幅度的滑动窗口和以在该维度上最佳地匹配目标曲线，同时在该频率的相位中隐藏尺度信息。在这个过程中，我们利用曲线的几何特性来帮助更有效地隐藏和恢复它们。我们的方案是简单的和编码发生有效的非负最小二乘框架，而解码是平凡的。我们在大量的图像和音频数据集上定量验证了我们的技术，并展示了一个众包听力测试的结果，验证了隐藏的信息确实是不显眼的。摘要：In this work, we create artistic closed loop curves that trace out images and 3D shapes, which we then hide in musical audio as a form of steganography. We use traveling salesperson art to create artistic plane loops to trace out image contours, and we use Hamiltonian cycles on triangle meshes to create artistic space loops that fill out 3D surfaces. Our embedding scheme is designed to faithfully preserve the geometry of these loops after lossy compression, while keeping their presence undetectable to the audio listener. To accomplish this, we hide each dimension of the curve in a different frequency, and we perturb a sliding window sum of the magnitude of that frequency to best match the target curve at that dimension, while hiding scale information in that frequency's phase. In the process, we exploit geometric properties of the curves to help to more effectively hide and recover them. Our scheme is simple and encoding happens efficiently with a nonnegative least squares framework, while decoding is trivial. We validate our technique quantitatively on large datasets of images and audio, and we show results of a crowd sourced listening test that validate that the hidden information is indeed unobtrusive.
【13】Achieving Timestamp Prediction While Recognizing with Non-Autoregressive End-to-End ASR Model
标题：非自回归端到端ASR模型在识别时实现时间戳预测
链接：http://arxiv.org/pdf/2301.12343v1
*与cs.SD语音【8】为同一篇
作者：Xian Shi,Yanni Chen,Shiliang Zhang,Zhijie Yan机构：Speech Lab, Alibaba Group, China摘要：传统的ASR系统使用帧级音素后验进行力对齐（FA）并提供时间戳，而端到端的ASR系统尤其是基于AED的ASR系统缺乏这种能力。本文提出利用非自回归ASR模型Paraformer中的连续积分点火CIF机制，在识别的同时进行时间戳预测TP。针对CIF的起火点偏置问题，提出了包括起火点延迟和静音插入在内的后处理策略。此外，我们还提出使用scaled-CIF平滑CIF输出的权重，这对ASR和TP任务都是有益的。采用累积平均偏移量AAS和二值化误差率DER来衡量时间戳的质量，并将该系统与传统的混合力对准系统进行了比较。在人工标记时间戳测试集上的实验结果表明，提出的优化方法显著提高了CIF时间戳的准确性，AAS和DER分别降低了66.7%和82.1%。与用相同数据训练的Kaldi力对齐算法相比，优化CIF时间戳算法可使AAS相对减少12.3%。摘要：Conventional ASR systems use frame-level phoneme posterior to conduct force-alignment~(FA) and provide timestamps, while end-to-end ASR systems especially AED based ones are short of such ability. This paper proposes to perform timestamp prediction~(TP) while recognizing by utilizing continuous integrate-and-fire~(CIF) mechanism in non-autoregressive ASR model - Paraformer. Foucing on the fire place bias issue of CIF, we conduct post-processing strategies including fire-delay and silence insertion. Besides, we propose to use scaled-CIF to smooth the weights of CIF output, which is proved beneficial for both ASR and TP task. Accumulated averaging shift~(AAS) and diarization error rate~(DER) are adopted to measure the quality of timestamps and we compare these metrics of proposed system and conventional hybrid force-alignment system. The experiment results over manually-marked timestamps testset show that the proposed optimization methods significantly improve the accuracy of CIF timestamps, reducing 66.7 % and 82.1 % of AAS and DER respectively. Comparing to Kaldi force-alignment trained with the same data, optimized CIF timestamps achieved 12.3 % relative AAS reduction.
【14】Time out of Mind: Generating Emotionally Conditioned Rate of Speech
标题：心不在焉的时间：产生情绪调节的语速
链接：http://arxiv.org/pdf/2301.12331v1
*与cs.SD语音【9】为同一篇
作者：Navjot Kaur,Paige Tuttosi摘要：语音合成在过去的十年中有了显著的改进，产生了高度可理解的语音。进一步的研究已经产生了可以产生可变语音的模型，包括条件情感表达。然而，问题在于对短语级修改和韵律声乐特征的关注。使用CREMA-D数据集，我们已经训练了一个基于情感的GAN来为给定的输入文本生成有价值的长度。这些单词长度与中性语音有关，并且可以通过语音合成标记语言（SSML）提供给文本到语音（TTS）系统以生成更具表达力的语音。与开箱即用的模型相比，我们能够在中性语音的客观测量上实现更好的性能，并且在快乐语音的时间对准上实现更好的性能。摘要：Voice synthesis has seen significant improvements in the past decade resulting in highly intelligible voices. Further investigations have resulted in models that can produce variable speech, including conditional emotional expression. The problem lies, however, in a focus on phrase level modifications and prosodic vocal features. Using the CREMA-D dataset we have trained a GAN conditioned on emotion to generate worth lengths for a given input text. These word lengths are relative to neutral speech and can be provided, through speech synthesis markup language (SSML) to a text to speech (TTS) system to generate more expressive speech. We were able to achieve better performances on objective measures for neutral speech, and better time alignment for happy speech when compared to an out of box model.
【15】who is snoring? snore based user recognition
标题：谁在打呼噜？基于鼾声的用户识别
链接：http://arxiv.org/pdf/2301.12209v1
*与cs.SD语音【10】为同一篇
作者：Shenghao Li,Jagmohan Chauhan机构：University of Southampton摘要：打鼾是阻塞性睡眠呼吸暂停低通气综合征（OSAH）的最突出症状之一，OSAH是一种导致上呼吸道反复塌陷和停止的高度流行的疾病。因此，准确的鼾声监测和分析至关重要。然而，传统的监测方法多导睡眠图（PSG）需要患者在睡眠诊所呆上一整夜，并连接到许多件设备。另一种侵入性较小的方法是在家中或临床环境中使用智能手机进行被动监测。但是，存在一个挑战：环境可以由人共享，使得原始音频可以包含床伴或其他人的打鼾活动。打鼾活动的错误捕获可能导致严重的错误警报和患者的误诊。为了解决这一局限性，我们提出了一个假设，即鼾声包含唯一的身份信息，可以用于用户识别。我们分析了各种机器学习模型：高斯混合模型（GMM）、GMM-UBM（通用背景模型）和MPSSC上的深度神经网络（DNN）-用于评估我们假设有效性的开源打鼾数据集。我们的结果是有希望的，因为我们在识别和验证任务中实现了约90%的准确率。这项工作标志着了解基于打鼾的用户监测的实用性以实现多种医疗保健应用的第一步。摘要：Snoring is one of the most prominent symptoms of Obstructive Sleep Apnea-Hypopnea Syndrome (OSAH), a highly prevalent disease that causes repetitive collapse and cessation of the upper airway. Thus, accurate snore sound monitoring and analysis is crucial. However, the traditional monitoring method polysomnography (PSG) requires the patients to stay at a sleep clinic for the whole night and be connected to many pieces of equipment. An alternative and less invasive way is passive monitoring using a smartphone at home or in the clinical settings. But, there is a challenge: the environment may be shared by people such that the raw audio may contain the snore activities of the bed partner or other person. False capturing of the snoring activity could lead to critical false alarms and misdiagnosis of the patients. To address this limitation, we propose a hypothesis that snore sound contains unique identity information which can be used for user recognition. We analyzed various machine learning models: Gaussian Mixture Model (GMM), GMM-UBM (Universial Background Model), and a Deep Neural Network (DNN) on MPSSC - an open source snoring dataset to evaluate the validity of our hypothesis. Our results are promising as we achieved around 90% accuracy in identification and verification tasks. This work marks the first step towards understanding the practicality of snore based user monitoring to enable multiple healthcare applicaitons.
【16】Automated Arrangements of Multi-Part Music for Sets of Monophonic Instruments
标题：单音乐器套装多声部音乐的自动编排
链接：http://arxiv.org/pdf/2301.12084v1
*与cs.SD语音【11】为同一篇
作者：Matthew Mccloskey,Gabrielle Curcio,Amulya Badineni,Kevin Mcgrath,Dimitris Papamichail机构：ITRIS PAPAMICHAIL, The College of New Jersey, USA摘要：为一组不同的乐器编排音乐，这是一个传统的乏味和耗时的过程，由对特定乐器具有复杂知识的专家来执行，并涉及大量的实验。本文主要研究为单声道乐器或人声创作的音乐作品的自动编曲问题。我们设计并实现了一种算法，通过将音乐片段移调到不同的音阶、将分配的部分置换为乐器 人声、以及将各个部分移调一个或多个八度音阶，在可行的情况下，该算法总是可以产生音乐安排。我们还发布了用Python编写的开源软件，它可以处理MusicXML文件，并允许音乐家尝试音乐编排。这是我们的希望，我们的软件可以作为一个平台，为未来的扩展，将包括音乐减少和包括复调乐器。摘要：Arranging music for a different set of instruments that it was originally written for is traditionally a tedious and time-consuming process, performed by experts with intricate knowledge of the specific instruments and involving significant experimentation. In this paper we study the problem of automating music arrangements for music pieces written for monophonic instruments or voices. We designed and implemented an algorithm that can always produce a music arrangement when feasible by transposing the music piece to a different scale, permuting the assigned parts to instruments voices, and transposing individual parts by one or more octaves. We also published open source software written in Python that processes MusicXML files and allows musicians to experiment with music arrangements. It is our hope that our software can serve as a platform for future extensions that will include music reductions and inclusion of polyphonic instruments.
【17】Byte Pair Encoding for Symbolic Music
标题：符号音乐的字节对编码
链接：http://arxiv.org/pdf/2301.11975v1
*与cs.SD语音【12】为同一篇
作者：Nathan Fradet,Jean-Pierre Briot,Fabien Chhel,Amal El Fallah Seghrouchni,Nicolas Gutowski机构：Sorbonne University - CNRS, France 4University of Angers备注：Source code at this https URL摘要：符号音乐模态现在大多表示为离散的，并与序列模型（如Transformers）一起用于深度学习任务。最近的研究致力于标记化，即将数据转换为此类模型可理解的整数序列。这可以通过许多方式来实现，因为音乐可以由同时的音轨、具有几个属性的同时的音符组成。到目前为止，提出的标记化都是基于描述音符属性和时间事件的小词汇表，导致相当长的标记序列。在本文中，我们将展示字节对编码（BPE）如何在提高深度学习模型性能的同时改善其结果。我们在音乐生成和作曲家分类上进行了实验，研究了BPE对模型学习嵌入的影响，并表明它有助于增加嵌入的各向同性，即：它们在空间中的位置变化的均匀性。摘要：The symbolic music modality is nowadays mostly represented as discrete and used with sequential models such as Transformers, for deep learning tasks. Recent research put efforts on the tokenization, i.e. the conversion of data into sequences of integers intelligible to such models. This can be achieved by many ways as music can be composed of simultaneous tracks, of simultaneous notes with several attributes. Until now, the proposed tokenizations are based on small vocabularies describing the note attributes and time events, resulting in fairly long token sequences. In this paper, we show how Byte Pair Encoding (BPE) can improve the results of deep learning models while improving its performances. We experiment on music generation and composer classification, and study the impact of BPE on how models learn the embeddings, and show that it can help to increase their isotropy, i.e., the uniformity of the variance of their positions in the space.
机器翻译，仅供参考
每日语音速递