今日论文合集：cs.SD语音3篇，eess.AS音频处理4篇。【1】Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge标题：LIMMITS 2023挑战赛多说话人多语言VQTTS系统链接：https://arxiv.org/abs/2304.13121作者：Chenpeng Du,Yiwei Guo,Feiyu Shen,Kai Yu机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by ICASSP 2023 Special Session for Grand Challenges摘要：在本文中，我们描述了上海交通大学X-LANCE团队为LIMMITS 2023挑战赛开发的系统，我们主要关注赛道1的自然度获胜系统。这项挑战的目的是为马拉地语、印地语和泰卢固语建立一个多人多语言的文本到语音（TTS）系统。在给定的数据集中，每种语言都有一个男性和一个女性说话者。在音轨1中，仅可以选择来自每个说话者的5小时数据来训练TTS模型。我们的系统是基于最近提出的VQTTS，利用VQ声学特征，而不是梅尔频谱图。我们引入额外的发言人嵌入和语言嵌入VQTTS控制发言人和语言信息。在跨语言评估中，我们需要在跨语言扬声器的声音中合成语音，我们提供了母语扬声器的嵌入到声学模型和目标扬声器的嵌入到声码器。在自然度主观MOS听力测试中，本系统达到4.77，位居第一。摘要：In this paper, we describe the systems developed by the SJTU X-LANCE team for LIMMITS 2023 Challenge, and we mainly focus on the winning system on naturalness for track 1. The aim of this challenge is to build a multi-speaker multi-lingual text-to-speech (TTS) system for Marathi, Hindi and Telugu. Each of the languages has a male and a female speaker in the given dataset. In track 1, only 5 hours data from each speaker can be selected to train the TTS model. Our system is based on the recently proposed VQTTS that utilizes VQ acoustic feature rather than mel-spectrogram. We introduce additional speaker embeddings and language embeddings to VQTTS for controlling the speaker and language information. In the cross-lingual evaluations where we need to synthesize speech in a cross-lingual speaker's voice, we provide a native speaker's embedding to the acoustic model and the target speaker's embedding to the vocoder. In the subjective MOS listening test on naturalness, our system achieves 4.77 which ranks first.【2】 AI-Synthesized Voice Detection Using Neural Vocoder Artifacts标题：基于神经声码器伪影的人工合成语音检测链接：https://arxiv.org/abs/2304.13085作者：Chengzhe Sun,Shan Jia,Shuwei Hou,Siwei Lyu机构：University at Buffalo, State University of New York, NY, USA备注：arXiv admin note: substantial text overlap with arXiv:2302.09198摘要：人工智能合成人类声音的进步已经造成了越来越大的模仿和虚假信息的威胁，因此开发检测合成人类声音的方法至关重要。本研究提出了一种新的方法来识别合成人的声音，通过检测音频信号中的声码器的文物。大多数DeepFake音频合成模型使用神经声码器，这是一种神经网络，可以从梅尔频谱图等时间频率表示生成波形。通过识别音频中的神经声码器处理，我们可以确定样本是否被合成。为了检测合成的人类声音，我们引入了一个多任务学习框架的二进制类RawNet 2模型，共享的特征提取器与声码器识别模块。通过处理声码器识别作为一个借口任务，我们约束的特征提取器专注于声码器伪影，并提供最后的二进制分类器的区别性特征。实验结果表明，基于声码器识别的改进RawNet 2模型在二进制任务总体上取得了较高的分类性能。代码和数据可以在 url{https：github.com csun22 Synthetic-Voice-Detection-Vocoder-Artifacts}找到。摘要：Advancements in AI-synthesized human voices have created a growing threat of impersonation and disinformation, making it crucial to develop methods to detect synthetic human voices. This study proposes a new approach to identifying synthetic human voices by detecting artifacts of vocoders in audio signals. Most DeepFake audio synthesis models use a neural vocoder, a neural network that generates waveforms from temporal-frequency representations like mel-spectrograms. By identifying neural vocoder processing in audio, we can determine if a sample is synthesized. To detect synthetic human voices, we introduce a multi-task learning framework for a binary-class RawNet2 model that shares the feature extractor with a vocoder identification module. By treating vocoder identification as a pretext task, we constrain the feature extractor to focus on vocoder artifacts and provide discriminative features for the final binary classifier. Our experiments show that the improved RawNet2 model based on vocoder identification achieves high classification performance on the binary task overall. Codes and data can be found at url{https: github.com csun22 Synthetic-Voice-Detection-Vocoder-Artifacts}.【3】 Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity Speech Synthesis标题：高保真语音合成中基于信源滤波的生成性对抗性神经声码器链接：https://arxiv.org/abs/2304.13270作者：Ye-Xin Lu,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information, Processing, University of Science and Technology of China, Hefei, P. R. China备注：Accepted by NCMMSC 2022摘要：本文提出了一种基于源滤波器的生成式对抗神经声码器SF-GAN，通过将基于F0的源激励信号引入神经滤波器框架，从输入声学特征实现高保真波形生成。SF-GAN声码器由源模块和分辨率条件滤波器模块组成，并基于生成对抗策略进行训练。源模块从F0信息产生激励信号，然后分辨率卷积滤波器模块以各种时间分辨率将激励信号与经处理的声学特征组合，并最终重构原始波形。实验结果表明，我们提出的SF-GAN声码器优于国家的最先进的HiFi-GAN和Fre-GAN在分析-合成（AS）和文本到语音（TTS）的任务，SF-GAN的合成语音质量是地面实况音频相媲美。摘要：This paper proposes a source-filter-based generative adversarial neural vocoder named SF-GAN, which achieves high-fidelity waveform generation from input acoustic features by introducing F0-based source excitation signals to a neural filter framework. The SF-GAN vocoder is composed of a source module and a resolution-wise conditional filter module and is trained based on generative adversarial strategies. The source module produces an excitation signal from the F0 information, then the resolution-wise convolutional filter module combines the excitation signal with processed acoustic features at various temporal resolutions and finally reconstructs the raw waveform. The experimental results show that our proposed SF-GAN vocoder outperforms the state-of-the-art HiFi-GAN and Fre-GAN in both analysis-synthesis (AS) and text-to-speech (TTS) tasks, and the synthesized speech quality of SF-GAN is comparable to the ground-truth audio.【1】 All Information is Necessary: Integrating Speech Positive and Negative Information by Contrastive Learning for Speech Enhancement标题：所有信息都是必需的：通过对比学习整合语音正负信息进行语音增强链接：https://arxiv.org/abs/2304.13439
作者：Xinmeng Xu,Weiping Tu,Chang Han,Yuhong Yang机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China, Hubei Luojia Laboratory, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering摘要：单耳语音增强是一个不可逆的退化过程，是一个不适定的问题。实现SE任务的最近方法仅依赖于正面信息，例如，地面实况语音和语音相关特征。与上述不同的是，我们观察到的负面信息，如原始的语音混合和语音无关的功能，是有价值的指导SE模型的训练过程。在本研究中，我们提出了一个SE模型，整合了语音正面和负面信息，以提高SE绩效，采用对比学习，其中两个创新包括。(1)我们设计了一个协作模块（CM），它包含两个部分，对比注意分离相关和不相关的功能，通过对比学习和交互式的注意建立语音特征之间的相关性，在一个可学习和自适应的方式。(2)我们提出了一种基于对比学习的对比正则化（CR），以确保通过集成自监督模型将估计的语音拉近干净的语音，并远离表示空间中的嘈杂语音。我们提出的SE网络与CM和CR CMCR-Net。实验结果表明，我们的CMCR-Net实现了可比的和优越的性能，最近的方法。摘要：Monaural speech enhancement (SE) is an ill-posed problem due to the irreversible degradation process. Recent methods to achieve SE tasks rely solely on positive information, e.g., ground-truth speech and speech-relevant features. Different from the above, we observe that the negative information, such as original speech mixture and speech-irrelevant features, are valuable to guide the SE model training procedure. In this study, we propose a SE model that integrates both speech positive and negative information for improving SE performance by adopting contrastive learning, in which two innovations have consisted. (1) We design a collaboration module (CM), which contains two parts, contrastive attention for separating relevant and irrelevant features via contrastive learning and interactive attention for establishing the correlation between both speech features in a learnable and self-adaptive manner. (2) We propose a contrastive regularization (CR) built upon contrastive learning to ensure that the estimated speech is pulled closer to the clean speech and pushed far away from the noisy speech in the representation space by integrating self-supervised models. We term the proposed SE network with CM and CR as CMCR-Net. Experimental results demonstrate that our CMCR-Net achieves comparable and superior performance to recent approaches.
【2】 Source-Filter-Based Generative Adversarial Neural Vocoder for High Fidelity Speech Synthesis标题：高保真语音合成中基于信源滤波的生成性对抗性神经声码器链接：https://arxiv.org/abs/2304.13270作者：Ye-Xin Lu,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information, Processing, University of Science and Technology of China, Hefei, P. R. China备注：Accepted by NCMMSC 2022摘要：本文提出了一种基于源滤波器的生成式对抗神经声码器SF-GAN，通过将基于F0的源激励信号引入神经滤波器框架，从输入声学特征实现高保真波形生成。SF-GAN声码器由源模块和分辨率条件滤波器模块组成，并基于生成对抗策略进行训练。源模块从F0信息产生激励信号，然后分辨率卷积滤波器模块以各种时间分辨率将激励信号与经处理的声学特征组合，并最终重构原始波形。实验结果表明，我们提出的SF-GAN声码器优于国家的最先进的HiFi-GAN和Fre-GAN在分析-合成（AS）和文本到语音（TTS）的任务，SF-GAN的合成语音质量是地面实况音频相媲美。摘要：This paper proposes a source-filter-based generative adversarial neural vocoder named SF-GAN, which achieves high-fidelity waveform generation from input acoustic features by introducing F0-based source excitation signals to a neural filter framework. The SF-GAN vocoder is composed of a source module and a resolution-wise conditional filter module and is trained based on generative adversarial strategies. The source module produces an excitation signal from the F0 information, then the resolution-wise convolutional filter module combines the excitation signal with processed acoustic features at various temporal resolutions and finally reconstructs the raw waveform. The experimental results show that our proposed SF-GAN vocoder outperforms the state-of-the-art HiFi-GAN and Fre-GAN in both analysis-synthesis (AS) and text-to-speech (TTS) tasks, and the synthesized speech quality of SF-GAN is comparable to the ground-truth audio.【3】 Multi-Speaker Multi-Lingual VQTTS System for LIMMITS 2023 Challenge标题：LIMMITS 2023挑战赛多说话人多语言VQTTS系统链接：https://arxiv.org/abs/2304.13121作者：Chenpeng Du,Yiwei Guo,Feiyu Shen,Kai Yu机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by ICASSP 2023 Special Session for Grand Challenges摘要：在本文中，我们描述了上海交通大学X-LANCE团队为LIMMITS 2023挑战赛开发的系统，我们主要关注赛道1的自然度获胜系统。这项挑战的目的是为马拉地语、印地语和泰卢固语建立一个多人多语言的文本到语音（TTS）系统。在给定的数据集中，每种语言都有一个男性和一个女性说话者。在音轨1中，仅可以选择来自每个说话者的5小时数据来训练TTS模型。我们的系统是基于最近提出的VQTTS，利用VQ声学特征，而不是梅尔频谱图。我们引入额外的发言人嵌入和语言嵌入VQTTS控制发言人和语言信息。在跨语言评估中，我们需要在跨语言扬声器的声音中合成语音，我们提供了母语扬声器的嵌入到声学模型和目标扬声器的嵌入到声码器。在自然度主观MOS听力测试中，本系统达到4.77，位居第一。摘要：In this paper, we describe the systems developed by the SJTU X-LANCE team for LIMMITS 2023 Challenge, and we mainly focus on the winning system on naturalness for track 1. The aim of this challenge is to build a multi-speaker multi-lingual text-to-speech (TTS) system for Marathi, Hindi and Telugu. Each of the languages has a male and a female speaker in the given dataset. In track 1, only 5 hours data from each speaker can be selected to train the TTS model. Our system is based on the recently proposed VQTTS that utilizes VQ acoustic feature rather than mel-spectrogram. We introduce additional speaker embeddings and language embeddings to VQTTS for controlling the speaker and language information. In the cross-lingual evaluations where we need to synthesize speech in a cross-lingual speaker's voice, we provide a native speaker's embedding to the acoustic model and the target speaker's embedding to the vocoder. In the subjective MOS listening test on naturalness, our system achieves 4.77 which ranks first.
【4】 AI-Synthesized Voice Detection Using Neural Vocoder Artifacts标题：基于神经声码器伪影的人工合成语音检测链接：https://arxiv.org/abs/2304.13085作者：Chengzhe Sun,Shan Jia,Shuwei Hou,Siwei Lyu机构：University at Buffalo, State University of New York, NY, USA备注：arXiv admin note: substantial text overlap with arXiv:2302.09198摘要：人工智能合成人类声音的进步已经造成了越来越大的模仿和虚假信息的威胁，因此开发检测合成人类声音的方法至关重要。本研究提出了一种新的方法来识别合成人的声音，通过检测音频信号中的声码器的文物。大多数DeepFake音频合成模型使用神经声码器，这是一种神经网络，可以从梅尔频谱图等时间频率表示生成波形。通过识别音频中的神经声码器处理，我们可以确定样本是否被合成。为了检测合成的人类声音，我们引入了一个多任务学习框架的二进制类RawNet 2模型，共享的特征提取器与声码器识别模块。通过处理声码器识别作为一个借口任务，我们约束的特征提取器专注于声码器伪影，并提供最后的二进制分类器的区别性特征。实验结果表明，基于声码器识别的改进RawNet 2模型在二进制任务总体上取得了较高的分类性能。代码和数据可以在 url{https：github.com csun22 Synthetic-Voice-Detection-Vocoder-Artifacts}找到。摘要：Advancements in AI-synthesized human voices have created a growing threat of impersonation and disinformation, making it crucial to develop methods to detect synthetic human voices. This study proposes a new approach to identifying synthetic human voices by detecting artifacts of vocoders in audio signals. Most DeepFake audio synthesis models use a neural vocoder, a neural network that generates waveforms from temporal-frequency representations like mel-spectrograms. By identifying neural vocoder processing in audio, we can determine if a sample is synthesized. To detect synthetic human voices, we introduce a multi-task learning framework for a binary-class RawNet2 model that shares the feature extractor with a vocoder identification module. By treating vocoder identification as a pretext task, we constrain the feature extractor to focus on vocoder artifacts and provide discriminative features for the final binary classifier. Our experiments show that the improved RawNet2 model based on vocoder identification achieves high classification performance on the binary task overall. Codes and data can be found at url{https: github.com csun22 Synthetic-Voice-Detection-Vocoder-Artifacts}.机器翻译，仅供参考
每日语音速递