语音/音频处理学术速递[2.8]

今日论文合集：cs.SD语音4篇，eess.AS音频处理4篇。

cs.SD语音

【1】 Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision

标题：说话、阅读和提示：高保真文本到语音转换，只需最少的监督

链接：https://arxiv.org/abs/2302.03540

作者：Eugene Kharitonov,Damien Vincent,Zalán Borsos,Raphaël Marinier,Sertan Girgin,Olivier Pietquin,Matt Sharifi,Marco Tagliasacchi,Neil Zeghidour

机构：Google Research

摘要：我们介绍SPEAR-TTS，一个多说话人文本到语音（TTS）系统，可以在最少的监督下进行训练。通过组合两种类型的离散语音表示，我们将TTS转换为两个序列到序列任务的组合：从文本到高级语义标记（类似于“阅读”）以及从语义标记到低级声学标记（“说话”）。将这两个任务分离开来，可以使用大量的纯音频数据来训练“说话”模块，并解锁预训练和反向翻译的高效组合，以减少训练“阅读”组件时对并行数据的需求。为了控制说话人身份，我们采用了示例提示，它允许SPEAR-TTS仅使用3秒的短样本就泛化到未见过的说话人，而不需要任何显式的说话人表示或说话人身份标签。我们的实验表明，SPEAR-TTS仅使用15分钟的并行数据就能达到与最先进方法相媲美的字符错误率，同时在主观测试中测量的自然度和声学质量方面与真实语音相匹配。

摘要：We introduce SPEAR-TTS, a multi-speaker text-to-speech (TTS) system that can be trained with minimal supervision. By combining two types of discrete speech representations, we cast TTS as a composition of two sequence-to-sequence tasks: from text to high-level semantic tokens (akin to "reading") and from semantic tokens to low-level acoustic tokens ("speaking"). Decoupling these two tasks enables training of the "speaking" module using abundant audio-only data, and unlocks the highly efficient combination of pretraining and backtranslation to reduce the need for parallel data when training the "reading" component. To control the speaker identity, we adopt example prompting, which allows SPEAR-TTS to generalize to unseen speakers using only a short sample of 3 seconds, without any explicit speaker representation or speaker-id labels. Our experiments demonstrate that SPEAR-TTS achieves a character error rate that is competitive with state-of-the-art methods using only 15 minutes of parallel data, while matching ground-truth speech in terms of naturalness and acoustic quality, as measured in subjective tests.

【2】 Revisiting Pre-training in Audio-Visual Learning

标题：再论视听学习中的前期训练

链接：https://arxiv.org/abs/2302.03533

作者：Ruoxuan Feng,Wenke Xia,Di Hu

机构：College of Computer Science and Electrical Engineering, Hunan University, Changsha, Gaoling School of Artificial Intelligence, Renmin University of China, Beijing

摘要：预训练技术在提高模型在各种任务上的性能方面取得了巨大的成功，但发现在一些单峰情况下比从头训练的性能更差。这启发我们思考：预先训练的模型在更复杂的多模态情况下，特别是对于诸如音频和视频模态之类的异构模态，是否总是有效？我们发现答案是否定的。具体来说，我们探讨了预先训练的模型对两个视听学习场景的影响：跨模态初始化和多模态联合学习。采用跨模态初始化时，Batchnorm参数异常会导致"死通道"现象，影响模型容量的利用。因此，我们提出了自适应Batchnorm重新初始化（ABRi），以更好地利用目标任务的预训练模型的容量。在多模态联合学习中，我们发现一个预先训练好的单模态编码器会对另一个模态的编码器产生负面影响。为了解决这一问题，我们引入了一种两阶段融合调节策略，在使单峰编码器与自适应掩蔽方法相配合的同时，更好地利用了预先训练的知识。实验结果表明，该方法可以进一步挖掘预训练模型的潜力，提高视听学习的效果。

摘要：Pre-training technique has gained tremendous success in enhancing model performance on various tasks, but found to perform worse than training from scratch in some uni-modal situations. This inspires us to think: are the pre-trained models always effective in the more complex multi-modal scenario, especially for the heterogeneous modalities such as audio and visual ones? We find that the answer is No. Specifically, we explore the effects of pre-trained models on two audio-visual learning scenarios: cross-modal initialization and multi-modal joint learning. When cross-modal initialization is applied, the phenomena of "dead channel" caused by abnormal Batchnorm parameters hinders the utilization of model capacity. Thus, we propose Adaptive Batchnorm Re-initialization (ABRi) to better exploit the capacity of pre-trained models for target tasks. In multi-modal joint learning, we find a strong pre-trained uni-modal encoder would bring negative effects on the encoder of another modality. To alleviate such problem, we introduce a two-stage Fusion Tuning strategy, taking better advantage of the pre-trained knowledge while making the uni-modal encoders cooperate with an adaptive masking method. The experiment results show that our methods could further exploit pre-trained models' potential and boost performance in audio-visual learning.

【3】 PAMP: A unified framework boosting low resource automatic speech recognition

标题：PAMP：一种支持低资源自动语音识别的统一框架

链接：https://arxiv.org/abs/2302.03498

作者：Zeping Min,Qian Ge,Zhong Li,Weinan E

机构：Peking University

摘要：针对低资源自动语音识别（ASR）任务，提出了一种新的文语转换（TTS）数据增强框架--音素音频混合（PAMP）. PAMP方法是高度可解释的，并且可以结合发音规则的先验知识。此外，PAMP可以很容易地部署在几乎任何语言，特别是低资源的ASR任务。大量实验已经证明PAMP在低资源ASR任务上的巨大有效性：我们在普通话ASR任务上实现了10.84的字符错误率（CER），与先前通过微调wav 2 vec 2预训练模型所实现的技术水平相比，带来了约30的较大相对改进.

摘要：We propose a novel text-to-speech (TTS) data augmentation framework for low resource automatic speech recognition (ASR) tasks, named phoneme audio mix up (PAMP). The PAMP method is highly interpretable and can incorporate prior knowledge of pronunciation rules. Furthermore, PAMP can be easily deployed in almost any language, extremely for low resource ASR tasks. Extensive experiments have demonstrated the great effectiveness of PAMP on low resource ASR tasks: we achieve a \textbf{10.84\%} character error rate (CER) on the common voice Cantonese ASR task, bringing a great relative improvement of about \textbf{30\%} compared to the previous state-of-the-art which was achieved by fine-tuning the wav2vec2 pretrained model.

【4】 Autodecompose: A generative self-supervised model for semantic decomposition

标题：自动分解：一种产生式自监督语义分解模型

链接：https://arxiv.org/abs/2302.03124

作者：Mohammad Reza Bonyadi

摘要：我们引入了Autodecomposition，一种新颖的自监督生成模型，它将数据分解为两个语义独立的属性：捕获数据的特定方面（例如，音频信号中的语音）的期望属性，以及聚集所有其它信息（例如，音频信号的内容）而不给出任何标签的上下文属性。自动分解使用两个互补的扩充，一个在保留所需属性的同时操作上下文，另一个在保留上下文的同时操作所需属性。数据的扩增变体由两个编码器编码并由解码器重构。我们证明了其中一个编码器嵌入了期望的属性，而另一个嵌入了上下文属性。我们将自动分解应用于音频信号，以编码声源（人声）和内容。我们在YouTube和LibriSpeech数据集上对模型进行了预训练，并在不暴露标签的情况下以自我监督的方式进行了微调。实验结果表明，使用预训练的自分解声源编码器，线性分类器仅用10 s的标记样本就能识别30个说话人的语音，F1得分为97.6%，而监督模型的F1得分为95.7%。此外，我们的实验表明，即使在小数据集上预训练大型模型时，Autodecompose也能有效防止过拟合。对3个说话人的60 s音频信号从头开始训练一个大规模的自分解模型，在识别其他未见语音中的3个说话人时取得了98.5%以上的F1分数.我们最后证明了上下文编码器嵌入了关于语音内容的信息，而忽略了声源信息。我们用于训练模型的示例代码以及使用预训练模型的示例可在此处获得：\url{https：//github.com/rezabonyadi/autodecompose}

摘要：We introduce Autodecompose, a novel self-supervised generative model that decomposes data into two semantically independent properties: the desired property, which captures a specific aspect of the data (e.g. the voice in an audio signal), and the context property, which aggregates all other information (e.g. the content of the audio signal), without any labels given. Autodecompose uses two complementary augmentations, one that manipulates the context while preserving the desired property and the other that manipulates the desired property while preserving the context. The augmented variants of the data are encoded by two encoders and reconstructed by a decoder. We prove that one of the encoders embeds the desired property while the other embeds the context property. We apply Autodecompose to audio signals to encode sound source (human voice) and content. We pre-trained the model on YouTube and LibriSpeech datasets and fine-tuned in a self-supervised manner without exposing the labels. Our results showed that, using the sound source encoder of pre-trained Autodecompose, a linear classifier achieves F1 score of 97.6\% in recognizing the voice of 30 speakers using only 10 seconds of labeled samples, compared to 95.7\% for supervised models. Additionally, our experiments showed that Autodecompose is robust against overfitting even when a large model is pre-trained on a small dataset. A large Autodecompose model was pre-trained from scratch on 60 seconds of audio from 3 speakers achieved over 98.5\% F1 score in recognizing those three speakers in other unseen utterances. We finally show that the context encoder embeds information about the content of the speech and ignores the sound source information. Our sample code for training the model, as well as examples for using the pre-trained models are available here: \url{https://github.com/rezabonyadi/autodecompose}

eess.AS音频处理