今日论文合集：cs.SD语音3篇，eess.AS音频处理3篇。
【1】 Noise2Music: Text-conditioned Music Generation with Diffusion Models
标题：Noise2Music：基于扩散模型的文本条件音乐生成
链接：https://arxiv.org/abs/2302.03917
作者：Qingqing Huang,Daniel S. Park,Tao Wang,Timo I. Denk,Andy Ly,Nanxin Chen,Zhengdong Zhang,Zhishuai Zhang,Jiahui Yu,Christian Frank,Jesse Engel,Quoc V. Le,William Chan,Wei Han摘要：我们介绍Noise2Music，其中训练了一系列扩散模型，以便从文本提示生成高质量的30秒音乐剪辑。两种类型的扩散模型，即生成以文本为条件的中间表示的生成器模型，以及生成以中间表示和可能的文本为条件的高保真音频的级联器模型，被训练并被相继使用以生成高保真音乐。我们探索了两种中间表示方法，一种使用声谱图，另一种使用保真度较低的音频。我们发现，所生成的音频不仅能够忠实地反映文本提示的关键元素，如流派、节奏、乐器、语气和时代，而且超越了对提示的细粒度语义的基础。预先训练的大型语言模型在这个故事中起着关键作用--它们用于为训练集的音频生成配对文本，并提取由扩散模型摄取的文本提示的嵌入。 生成的示例：www.example.comhttps://google-research.github.io/noise2music摘要：We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. Two types of diffusion models, a generator model, which generates an intermediate representation conditioned on text, and a cascader model, which generates high-fidelity audio conditioned on the intermediate representation and possibly the text, are trained and utilized in succession to generate high-fidelity music. We explore two options for the intermediate representation, one using a spectrogram and the other using audio with lower fidelity. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era, but goes beyond to ground fine-grained semantics of the prompt. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models. Generated examples: https://google-research.github.io/noise2music【2】 A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
标题：一种基于真实自然语音的文语合成矢量量化方法
链接：https://arxiv.org/abs/2302.04215
作者：Li-Wei Chen,Shinji Watanabe,Alexander Rudnicky机构：Language Technologies Institute, Carnegie Mellon University摘要：近年来，基于阅读或表演语料库的文语转换系统已经达到了接近人类水平的自然度。然而，人类语音的多样性往往超出了这些语料库的覆盖范围。我们认为，处理这种多样性的能力对于人工智能系统实现人类层面的交流至关重要。我们的工作探索使用更丰富的真实世界数据来构建语音合成器。我们使用来自YouTube和播客的真实语音训练TTS系统。我们观察到在基于梅尔谱图的自回归模型中训练和推理对齐之间的失配，导致难以理解的合成，并证明在多个代码组中学习离散代码有效地解决了这个问题。我们介绍了我们的MQTTS系统，其架构设计用于多代码生成和单调对齐，以及使用干净的静音提示来提高合成质量。我们进行消融分析以确定我们方法的有效性。我们证明了MQTTS在一些客观和主观指标上优于现有的TTS系统。摘要：Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.【3】 Masking Kernel for Learning Energy-Efficient Speech Representation
标题：用于学习能量高效语音表示的掩蔽核
链接：https://arxiv.org/abs/2302.04161
作者：Apiwat Ditthapron,Emmanuel O. Agu,Adam C. Lammert机构：Worcester Polytechnic Institute, Institute Rd, Worcester, MA ,USA摘要：现代智能手机配备了功能强大的音频硬件和处理器，允许它们以高采样率采集和执行设备上的语音处理。然而，能耗仍然是一个问题，特别是对于资源密集型DNN。现有的移动语音处理通过压缩模型或经由超参数调整减少输入维度来减少计算复杂性，这降低了准确度或需要更多的训练迭代。本文提出了一种基于梯度下降法的节能语音记录格式（长度和采样率）优化方法。目标是减少输入大小，从而减少数据收集和推理能量。对于后向传递，使用具有非零导数（高斯、汉恩和汉明）的掩蔽函数作为窗口函数和低通滤波器。引入能量有效性惩罚来激励输入规模的减小。在说话人识别和创伤性脑损伤检测中，使用49%较短的持续时间，以较低的频率采样，所提出的掩蔽比基线好8.7%。摘要：Modern smartphones are equipped with powerful audio hardware and processors, allowing them to acquire and perform on-device speech processing at high sampling rates. However, energy consumption remains a concern, especially for resource-intensive DNNs. Prior mobile speech processing reduced computational complexity by compacting the model or reducing input dimensions via hyperparameter tuning, which reduced accuracy or required more training iterations. This paper proposes gradient descent for optimizing energy-efficient speech recording format (length and sampling rate). The goal is to reduce the input size, which reduces data collection and inference energy. For a backward pass, a masking function with non-zero derivatives (Gaussian, Hann, and Hamming) is used as a windowing function and a lowpass filter. An energy-efficient penalty is introduced to incentivize the reduction of the input size. The proposed masking outperformed baselines by 8.7% in speaker recognition and traumatic brain injury detection using 49% shorter duration, sampled at a lower frequency.【1】 A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech
标题：一种基于真实自然语音的文语合成矢量量化方法
链接：https://arxiv.org/abs/2302.04215
*与cs.SD语音【2】为同一篇
作者：Li-Wei Chen,Shinji Watanabe,Alexander Rudnicky机构：Language Technologies Institute, Carnegie Mellon University摘要：近年来，基于阅读或表演语料库的文语转换系统已经达到了接近人类水平的自然度。然而，人类语音的多样性往往超出了这些语料库的覆盖范围。我们认为，处理这种多样性的能力对于人工智能系统实现人类层面的交流至关重要。我们的工作探索使用更丰富的真实世界数据来构建语音合成器。我们使用来自YouTube和播客的真实语音训练TTS系统。我们观察到在基于梅尔谱图的自回归模型中训练和推理对齐之间的失配，导致难以理解的合成，并证明在多个代码组中学习离散代码有效地解决了这个问题。我们介绍了我们的MQTTS系统，其架构设计用于多代码生成和单调对齐，以及使用干净的静音提示来提高合成质量。我们进行消融分析以确定我们方法的有效性。我们证明了MQTTS在一些客观和主观指标上优于现有的TTS系统。摘要：Recent Text-to-Speech (TTS) systems trained on reading or acted corpora have achieved near human-level naturalness. The diversity of human speech, however, often goes beyond the coverage of these corpora. We believe the ability to handle such diversity is crucial for AI systems to achieve human-level communication. Our work explores the use of more abundant real-world data for building speech synthesizers. We train TTS systems using real-world speech from YouTube and podcasts. We observe the mismatch between training and inference alignments in mel-spectrogram based autoregressive models, leading to unintelligible synthesis, and demonstrate that learned discrete codes within multiple code groups effectively resolves this issue. We introduce our MQTTS system whose architecture is designed for multiple code generation and monotonic alignment, along with the use of a clean silence prompt to improve synthesis quality. We conduct ablation analyses to identify the efficacy of our methods. We show that MQTTS outperforms existing TTS systems in several objective and subjective measures.【2】 Masking Kernel for Learning Energy-Efficient Speech Representation
标题：用于学习能量高效语音表示的掩蔽核
链接：https://arxiv.org/abs/2302.04161
*与cs.SD语音【3】为同一篇
作者：Apiwat Ditthapron,Emmanuel O. Agu,Adam C. Lammert机构：Worcester Polytechnic Institute, Institute Rd, Worcester, MA ,USA摘要：现代智能手机配备了功能强大的音频硬件和处理器，允许它们以高采样率采集和执行设备上的语音处理。然而，能耗仍然是一个问题，特别是对于资源密集型DNN。现有的移动语音处理通过压缩模型或经由超参数调整减少输入维度来减少计算复杂性，这降低了准确度或需要更多的训练迭代。本文提出了一种基于梯度下降法的节能语音记录格式（长度和采样率）优化方法。目标是减少输入大小，从而减少数据收集和推理能量。对于后向传递，使用具有非零导数（高斯、汉恩和汉明）的掩蔽函数作为窗口函数和低通滤波器。引入能量有效性惩罚来激励输入规模的减小。在说话人识别和创伤性脑损伤检测中，使用49%较短的持续时间，以较低的频率采样，所提出的掩蔽比基线好8.7%。摘要：Modern smartphones are equipped with powerful audio hardware and processors, allowing them to acquire and perform on-device speech processing at high sampling rates. However, energy consumption remains a concern, especially for resource-intensive DNNs. Prior mobile speech processing reduced computational complexity by compacting the model or reducing input dimensions via hyperparameter tuning, which reduced accuracy or required more training iterations. This paper proposes gradient descent for optimizing energy-efficient speech recording format (length and sampling rate). The goal is to reduce the input size, which reduces data collection and inference energy. For a backward pass, a masking function with non-zero derivatives (Gaussian, Hann, and Hamming) is used as a windowing function and a lowpass filter. An energy-efficient penalty is introduced to incentivize the reduction of the input size. The proposed masking outperformed baselines by 8.7% in speaker recognition and traumatic brain injury detection using 49% shorter duration, sampled at a lower frequency.【3】 Noise2Music: Text-conditioned Music Generation with Diffusion Models
标题：Noise2Music：基于扩散模型的文本条件音乐生成
链接：https://arxiv.org/abs/2302.03917
*与cs.SD语音【1】为同一篇
作者：Qingqing Huang,Daniel S. Park,Tao Wang,Timo I. Denk,Andy Ly,Nanxin Chen,Zhengdong Zhang,Zhishuai Zhang,Jiahui Yu,Christian Frank,Jesse Engel,Quoc V. Le,William Chan,Wei Han摘要：We introduce Noise2Music, where a series of diffusion models is trained to generate high-quality 30-second music clips from text prompts. Two types of diffusion models, a generator model, which generates an intermediate representation conditioned on text, and a cascader model, which generates high-fidelity audio conditioned on the intermediate representation and possibly the text, are trained and utilized in succession to generate high-fidelity music. We explore two options for the intermediate representation, one using a spectrogram and the other using audio with lower fidelity. We find that the generated audio is not only able to faithfully reflect key elements of the text prompt such as genre, tempo, instruments, mood, and era, but goes beyond to ground fine-grained semantics of the prompt. Pretrained large language models play a key role in this story -- they are used to generate paired text for the audio of the training set and to extract embeddings of the text prompts ingested by the diffusion models. Generated examples: https://google-research.github.io/noise2music机器翻译，仅供参考
每日语音速递