今日论文合集：cs.SD语音7篇，eess.AS音频处理11篇。
cs.SD语音
【1】 Learning Robust Self-attention Features for Speech Emotion Recognition  with Label-adaptive Mixup标题：标签自适应混合语音情感识别中的稳健自注意特征学习链接：https://arxiv.org/abs/2305.06273
作者：Lei Kang,Lichao Zhang,Dazhi Jiang机构：†Computer Science Dept., Shantou University, China, ‡Aeronautics Engineering College, Air Force Engineering University, China备注：Accepted to ICASSP 2023摘要：语音情感识别（SER）是在与机器的自然语言交互场景中识别人类的情感，由于人类情感的模糊性，这被认为是一个具有挑战性的问题。尽管SER最近取得了进展，但最先进的模型难以实现令人满意的性能。我们提出了一种基于自注意力的方法，结合使用标签自适应混合和中心损失。通过适应标签概率的混合和拟合中心损失的混合训练方案，我们提出的方法实现了优越的性能，以国家的最先进的方法。摘要：Speech Emotion Recognition (SER) is to recognize human emotions in a natural verbal interaction scenario with machines, which is considered as a challenging problem due to the ambiguous human emotions. Despite the recent progress in SER, state-of-the-art models struggle to achieve a satisfactory performance. We propose a self-attention based method with combined use of label-adaptive mixup and center loss. By adapting label probabilities in mixup and fitting center loss to the mixup training scheme, our proposed method achieves a superior performance to the state-of-the-art methods.
【2】 Enhancing Gappy Speech Audio Signals with Generative Adversarial  Networks标题：利用产生式对抗性网络增强Gappy语音音频信号链接：https://arxiv.org/abs/2305.05780
作者：Deniss Strods,Alan F. Smeaton机构：School of Computing, Dublin City University, Glasnevin, Dublin , Ireland, Insight Centre for Data Analytics备注：7 pages, 4 figures, 4 tables. 34th Irish Signals and Systems Conferences, 13-14 June 2023摘要：损坏音频的间隙、丢失和短片段是一个常见的问题，当它们发生在语音中时尤其令人讨厌。本文使用机器学习在音频语音信号中再生高达320 ms的间隙。通过将音频转换为Mel频谱图并使用图像修补来再生间隙，将音频再生转换为图像再生。然后使用Parallel-WaveGAN声码器将完整的Mel频谱图传输回音频，并集成到音频流中。使用从公开可用的LJSpeech数据集中获取的1到10秒的1300个语音音频片段的样本，我们的结果显示使用配备GPU的系统的GANs接近实时地再生音频间隙。正如预期的那样，音频中的间隙越小，填充间隙的质量越好。在240 ms的间隙上，表现最好的模型的平均平均意见得分（MOS）为3.737，在1（最差）到5（最好）的尺度上，这足以使人类感知为接近不间断的人类语音。摘要：Gaps, dropouts and short clips of corrupted audio are a common problem and particularly annoying when they occur in speech. This paper uses machine learning to regenerate gaps of up to 320ms in an audio speech signal. Audio regeneration is translated into image regeneration by transforming audio into a Mel-spectrogram and using image in-painting to regenerate the gaps. The full Mel-spectrogram is then transferred back to audio using the Parallel-WaveGAN vocoder and integrated into the audio stream. Using a sample of 1300 spoken audio clips of between 1 and 10 seconds taken from the publicly-available LJSpeech dataset our results show regeneration of audio gaps in close to real time using GANs with a GPU equipped system. As expected, the smaller the gap in the audio, the better the quality of the filled gaps. On a gap of 240ms the average mean opinion score (MOS) for the best performing models was 3.737, on a scale of 1 (worst) to 5 (best) which is sufficient for a human to perceive as close to uninterrupted human speech.
【3】 VSMask: Defending Against Voice Synthesis Attack via Real-Time  Predictive Perturbation标题：VSMAsk：利用实时预测扰动防御语音合成攻击链接：https://arxiv.org/abs/2305.05736
作者：Yuanda Wang,Hanqing Guo,Guangjing Wang,Bocheng Chen,Qiben Yan机构：Michigan State University, East Lansing, Michigan, USA摘要：基于深度学习的语音合成技术可以生成类似人类的人工语音，这些语音已被用于深度伪造或身份盗窃攻击。现有的防御机制注入微妙的对抗性扰动到原始语音音频误导语音合成模型。然而，优化对抗扰动不仅消耗大量的计算时间，而且还需要整个语音的可用性。因此，它们不适合保护实时语音流，例如语音消息或在线会议。在本文中，我们提出了VSMask，一个实时的保护机制，对语音合成攻击。与离线保护方案不同，VSMask利用预测神经网络来预测即将到来的流式传输语音的最有效扰动。VSMask引入了针对任意语音输入定制的通用扰动，以整体屏蔽实时语音。为了最大限度地减少受保护的语音内的音频失真，我们实现了基于权重的扰动约束，以减少增加的扰动的可感知性。我们全面评估了不同场景下的VSMask保护性能。实验结果表明，VSMask能够有效地防御3种流行的语音合成模型。没有合成语音可以欺骗说话者验证模型或具有VSMask保护的人耳。在一个物理世界的实验中，我们证明了VSMask成功地保障了实时语音注入扰动在空气中。摘要：Deep learning based voice synthesis technology generates artificial human-like speeches, which has been used in deepfakes or identity theft attacks. Existing defense mechanisms inject subtle adversarial perturbations into the raw speech audios to mislead the voice synthesis models. However, optimizing the adversarial perturbation not only consumes substantial computation time, but it also requires the availability of entire speech. Therefore, they are not suitable for protecting live speech streams, such as voice messages or online meetings. In this paper, we propose VSMask, a real-time protection mechanism against voice synthesis attacks. Different from offline protection schemes, VSMask leverages a predictive neural network to forecast the most effective perturbation for the upcoming streaming speech. VSMask introduces a universal perturbation tailored for arbitrary speech input to shield a real-time speech in its entirety. To minimize the audio distortion within the protected speech, we implement a weight-based perturbation constraint to reduce the perceptibility of the added perturbation. We comprehensively evaluate VSMask protection performance under different scenarios. The experimental results indicate that VSMask can effectively defend against 3 popular voice synthesis models. None of the synthetic voice could deceive the speaker verification models or human ears with VSMask protection. In a physical world experiment, we demonstrate that VSMask successfully safeguards the real-time speech by injecting the perturbation over the air.
【4】 Diffusion-based Signal Refiner for Speech Enhancement标题：一种基于扩散的语音增强信号精化器链接：https://arxiv.org/abs/2305.05857
作者：Masato Hirano,Shimada Kazuki,Yuichiro Koyama,Shusuke Takahashi,Yuhki Mitsufuji机构：Sony Group Corporation, Japan,  Sony Research, Japan备注：Under review摘要：我们已经开发了一个基于扩散的语音细化，提高了参考的感知质量的音频预测由以前的单通道语音分离模型。尽管现代基于深度神经网络的语音分离模型在基于参考的指标中表现出了高性能，但它们通常会产生感知上不自然的伪影。扩散模型的最新进展促使我们通过生成方法恢复初始分离的退化部分来解决这个问题。利用去噪扩散恢复模型（DDRM）为基础，我们提出了一个共享的DDRM为基础的精炼机，从任意语音分离模型的全局信息的前输出的条件下生成的样本。我们的实验表明，我们的细化器可以提供一个更清晰的谐波结构的语音和提高的参考自由度量的感知质量为任意的预处理模型架构。此外，我们根据先前的输出调整测量噪声的方差，这导致在无参考和基于参考的度量中的更高的分数。分离质量还可以通过混合鉴别输出和生成输出来进一步改善。摘要：We have developed a diffusion-based speech refiner that improves the reference-free perceptual quality of the audio predicted by preceding single-channel speech separation models. Although modern deep neural network-based speech separation models have show high performance in reference-based metrics, they often produce perceptually unnatural artifacts. The recent advancements made to diffusion models motivated us to tackle this problem by restoring the degraded parts of initial separations with a generative approach. Utilizing the denoising diffusion restoration model (DDRM) as a basis, we propose a shared DDRM-based refiner that generates samples conditioned on the global information of preceding outputs from arbitrary speech separation models. We experimentally show that our refiner can provide a clearer harmonic structure of speech and improves the reference-free metric of perceptual quality for arbitrary preceding model architectures. Furthermore, we tune the variance of the measurement noise based on preceding outputs, which results in higher scores in both reference-free and reference-based metrics. The separation quality can also be further improved by blending the discriminative and generative outputs.
eess.AS音频处理
【1】 Diffusion-based Signal Refiner for Speech Enhancement标题：一种基于扩散的语音增强信号精化器链接：https://arxiv.org/abs/2305.05857
*与cs.SD语音【4】为同一篇
作者：Masato Hirano,Shimada Kazuki,Yuichiro Koyama,Shusuke Takahashi,Yuhki Mitsufuji机构：Sony Group Corporation, Japan,  Sony Research, Japan备注：Under review摘要：我们已经开发了一个基于扩散的语音细化，提高了参考的感知质量的音频预测由以前的单通道语音分离模型。尽管现代基于深度神经网络的语音分离模型在基于参考的指标中表现出了高性能，但它们通常会产生感知上不自然的伪影。扩散模型的最新进展促使我们通过生成方法恢复初始分离的退化部分来解决这个问题。利用去噪扩散恢复模型（DDRM）为基础，我们提出了一个共享的DDRM为基础的精炼机，从任意语音分离模型的全局信息的前输出的条件下生成的样本。我们的实验表明，我们的细化器可以提供一个更清晰的谐波结构的语音和提高的参考自由度量的感知质量为任意的预处理模型架构。此外，我们根据先前的输出调整测量噪声的方差，这导致在无参考和基于参考的度量中的更高的分数。分离质量还可以通过混合鉴别输出和生成输出来进一步改善。摘要：We have developed a diffusion-based speech refiner that improves the reference-free perceptual quality of the audio predicted by preceding single-channel speech separation models. Although modern deep neural network-based speech separation models have show high performance in reference-based metrics, they often produce perceptually unnatural artifacts. The recent advancements made to diffusion models motivated us to tackle this problem by restoring the degraded parts of initial separations with a generative approach. Utilizing the denoising diffusion restoration model (DDRM) as a basis, we propose a shared DDRM-based refiner that generates samples conditioned on the global information of preceding outputs from arbitrary speech separation models. We experimentally show that our refiner can provide a clearer harmonic structure of speech and improves the reference-free metric of perceptual quality for arbitrary preceding model architectures. Furthermore, we tune the variance of the measurement noise based on preceding outputs, which results in higher scores in both reference-free and reference-based metrics. The separation quality can also be further improved by blending the discriminative and generative outputs.
【2】 Learning Robust Self-attention Features for Speech Emotion Recognition  with Label-adaptive Mixup标题：标签自适应混合语音情感识别中的稳健自注意特征学习链接：https://arxiv.org/abs/2305.06273
*与cs.SD语音【1】为同一篇
作者：Lei Kang,Lichao Zhang,Dazhi Jiang机构：†Computer Science Dept., Shantou University, China, ‡Aeronautics Engineering College, Air Force Engineering University, China备注：Accepted to ICASSP 2023摘要：语音情感识别（SER）是在与机器的自然语言交互场景中识别人类的情感，由于人类情感的模糊性，这被认为是一个具有挑战性的问题。尽管SER最近取得了进展，但最先进的模型难以实现令人满意的性能。我们提出了一种基于自注意力的方法，结合使用标签自适应混合和中心损失。通过适应标签概率的混合和拟合中心损失的混合训练方案，我们提出的方法实现了优越的性能，以国家的最先进的方法。摘要：Speech Emotion Recognition (SER) is to recognize human emotions in a natural verbal interaction scenario with machines, which is considered as a challenging problem due to the ambiguous human emotions. Despite the recent progress in SER, state-of-the-art models struggle to achieve a satisfactory performance. We propose a self-attention based method with combined use of label-adaptive mixup and center loss. By adapting label probabilities in mixup and fitting center loss to the mixup training scheme, our proposed method achieves a superior performance to the state-of-the-art methods.
【3】 Enhancing Gappy Speech Audio Signals with Generative Adversarial  Networks标题：利用产生式对抗性网络增强Gappy语音音频信号链接：https://arxiv.org/abs/2305.05780
*与cs.SD语音【2】为同一篇
作者：Deniss Strods,Alan F. Smeaton机构：School of Computing, Dublin City University, Glasnevin, Dublin , Ireland, Insight Centre for Data Analytics备注：7 pages, 4 figures, 4 tables. 34th Irish Signals and Systems Conferences, 13-14 June 2023摘要：损坏音频的间隙、丢失和短片段是一个常见的问题，当它们发生在语音中时尤其令人讨厌。本文使用机器学习在音频语音信号中再生高达320 ms的间隙。通过将音频转换为Mel频谱图并使用图像修补来再生间隙，将音频再生转换为图像再生。然后使用Parallel-WaveGAN声码器将完整的Mel频谱图传输回音频，并集成到音频流中。使用从公开可用的LJSpeech数据集中获取的1到10秒的1300个语音音频片段的样本，我们的结果显示使用配备GPU的系统的GANs接近实时地再生音频间隙。正如预期的那样，音频中的间隙越小，填充间隙的质量越好。在240 ms的间隙上，表现最好的模型的平均平均意见得分（MOS）为3.737，在1（最差）到5（最好）的尺度上，这足以使人类感知为接近不间断的人类语音。摘要：Gaps, dropouts and short clips of corrupted audio are a common problem and particularly annoying when they occur in speech. This paper uses machine learning to regenerate gaps of up to 320ms in an audio speech signal. Audio regeneration is translated into image regeneration by transforming audio into a Mel-spectrogram and using image in-painting to regenerate the gaps. The full Mel-spectrogram is then transferred back to audio using the Parallel-WaveGAN vocoder and integrated into the audio stream. Using a sample of 1300 spoken audio clips of between 1 and 10 seconds taken from the publicly-available LJSpeech dataset our results show regeneration of audio gaps in close to real time using GANs with a GPU equipped system. As expected, the smaller the gap in the audio, the better the quality of the filled gaps. On a gap of 240ms the average mean opinion score (MOS) for the best performing models was 3.737, on a scale of 1 (worst) to 5 (best) which is sufficient for a human to perceive as close to uninterrupted human speech.
【4】 VSMask: Defending Against Voice Synthesis Attack via Real-Time  Predictive Perturbation标题：VSMAsk：利用实时预测扰动防御语音合成攻击链接：https://arxiv.org/abs/2305.05736
*与cs.SD语音【3】为同一篇
作者：Yuanda Wang,Hanqing Guo,Guangjing Wang,Bocheng Chen,Qiben Yan机构：Michigan State University, East Lansing, Michigan, USA摘要：基于深度学习的语音合成技术可以生成类似人类的人工语音，这些语音已被用于深度伪造或身份盗窃攻击。现有的防御机制注入微妙的对抗性扰动到原始语音音频误导语音合成模型。然而，优化对抗扰动不仅消耗大量的计算时间，而且还需要整个语音的可用性。因此，它们不适合保护实时语音流，例如语音消息或在线会议。在本文中，我们提出了VSMask，一个实时的保护机制，对语音合成攻击。与离线保护方案不同，VSMask利用预测神经网络来预测即将到来的流式传输语音的最有效扰动。VSMask引入了针对任意语音输入定制的通用扰动，以整体屏蔽实时语音。为了最大限度地减少受保护的语音内的音频失真，我们实现了基于权重的扰动约束，以减少增加的扰动的可感知性。我们全面评估了不同场景下的VSMask保护性能。实验结果表明，VSMask能够有效地防御3种流行的语音合成模型。没有合成语音可以欺骗说话者验证模型或具有VSMask保护的人耳。在一个物理世界的实验中，我们证明了VSMask成功地保障了实时语音注入扰动在空气中。摘要：Deep learning based voice synthesis technology generates artificial human-like speeches, which has been used in deepfakes or identity theft attacks. Existing defense mechanisms inject subtle adversarial perturbations into the raw speech audios to mislead the voice synthesis models. However, optimizing the adversarial perturbation not only consumes substantial computation time, but it also requires the availability of entire speech. Therefore, they are not suitable for protecting live speech streams, such as voice messages or online meetings. In this paper, we propose VSMask, a real-time protection mechanism against voice synthesis attacks. Different from offline protection schemes, VSMask leverages a predictive neural network to forecast the most effective perturbation for the upcoming streaming speech. VSMask introduces a universal perturbation tailored for arbitrary speech input to shield a real-time speech in its entirety. To minimize the audio distortion within the protected speech, we implement a weight-based perturbation constraint to reduce the perceptibility of the added perturbation. We comprehensively evaluate VSMask protection performance under different scenarios. The experimental results indicate that VSMask can effectively defend against 3 popular voice synthesis models. None of the synthetic voice could deceive the speaker verification models or human ears with VSMask protection. In a physical world experiment, we demonstrate that VSMask successfully safeguards the real-time speech by injecting the perturbation over the air.
机器翻译，仅供参考