今日论文合集：cs.SD语音3篇，eess.AS音频处理3篇。
cs.SD语音
【1】Investigating the Design Space of Diffusion Models for Speech Enhancement
标题：语音增强扩散模型的设计空间研究
链接：http://arxiv.org/pdf/2312.04370v1
作者：Philippe Gonzalez,Zheng-Hua Tan,Jan Østergaard,Jesper Jensen,Tommy Sonne Alstrøm,Tobias May摘要：扩散模型是一类新的生成模型，在图像生成文献中表现出了出色的性能。因此，研究试图将扩散模型应用于其他任务，例如语音增强。一种流行的方法，在适应扩散模型的语音增强包括在建模的清洁和嘈杂的语音信号之间的渐进变换。然而，先前在图像生成文献中提出的一种流行的扩散模型框架没有考虑到这种朝向系统输入的变换，这阻止了现有的基于扩散的语音增强系统与前述扩散模型框架的关联。为了解决这个问题，我们扩展了这个框架，以考虑到清洁和嘈杂的语音信号之间的渐进变换。这使我们能够应用图像生成文献的最新发展，并系统地研究语音增强的扩散模型的设计方面，这些模型在很大程度上尚未探索，例如神经网络预处理，训练损失加权，随机微分方程（RSDs）或反向过程中注入的随机性。我们表明，以前的扩散为基础的语音增强系统的性能不能归因于清洁和嘈杂的语音信号之间的渐进变换。此外，我们表明，一个适当的选择预处理，训练损失加权，采样和采样器允许超越流行的基于扩散的语音增强系统的感知指标，同时使用更少的采样步骤，从而减少了四个因素的计算成本。摘要：Diffusion models are a new class of generative models that have shown outstanding performance in image generation literature. As a consequence, studies have attempted to apply diffusion models to other tasks, such as speech enhancement. A popular approach in adapting diffusion models to speech enhancement consists in modelling a progressive transformation between the clean and noisy speech signals. However, one popular diffusion model framework previously laid in image generation literature did not account for such a transformation towards the system input, which prevents from relating the existing diffusion-based speech enhancement systems with the aforementioned diffusion model framework. To address this, we extend this framework to account for the progressive transformation between the clean and noisy speech signals. This allows us to apply recent developments from image generation literature, and to systematically investigate design aspects of diffusion models that remain largely unexplored for speech enhancement, such as the neural network preconditioning, the training loss weighting, the stochastic differential equation (SDE), or the amount of stochasticity injected in the reverse process. We show that the performance of previous diffusion-based speech enhancement systems cannot be attributed to the progressive transformation between the clean and noisy speech signals. Moreover, we show that a proper choice of preconditioning, training loss weighting, SDE and sampler allows to outperform a popular diffusion-based speech enhancement system in terms of perceptual metrics while using fewer sampling steps, thus reducing the computational cost by a factor of four.
【2】DiaPer: End-to-End Neural Diarization with Perceiver-Based Attractors
标题：DiaPer：基于感知器的吸引子的端到端神经日志化链接：http://arxiv.org/pdf/2312.04324v1
作者：Federico Landini,Mireia Diez,Themos Stafylakis,Lukáš Burget摘要：直到最近，扬声器日记化领域一直由级联系统主导。由于其局限性，主要是关于重叠的语音和繁琐的管道，端到端模型最近得到了极大的普及。最成功的模型之一是端到端神经日志化与基于编码器-解码器的吸引子（EEND-EDA）。在这项工作中，我们将EDA模块替换为基于Perceiver的模块，并展示了其相对于EEND-EDA的优势;即在大量研究的Callhome数据集上获得更好的性能，更准确地找到对话中说话者的数量，并在长录音中几乎一半的时间内运行推理。此外，当与其他方法进行详尽的比较时，我们的模型DiaPer以非常轻巧的设计达到了卓越的性能。此外，我们与其他作品进行了比较，并在十多个公共宽带数据集的级联基线。与本出版物一起，我们发布了DiaPer的代码以及基于公共和免费数据训练的模型。摘要：Until recently, the field of speaker diarization was dominated by cascaded systems. Due to their limitations, mainly regarding overlapped speech and cumbersome pipelines, end-to-end models have gained great popularity lately. One of the most successful models is end-to-end neural diarization with encoder-decoder based attractors (EEND-EDA). In this work, we replace the EDA module with a Perceiver-based one and show its advantages over EEND-EDA; namely obtaining better performance on the largely studied Callhome dataset, finding the quantity of speakers in a conversation more accurately, and running inference on almost half of the time on long recordings. Furthermore, when exhaustively compared with other methods, our model, DiaPer, reaches remarkable performance with a very lightweight design. Besides, we perform comparisons with other works and a cascaded baseline across more than ten public wide-band datasets. Together with this publication, we release the code of DiaPer as well as models trained on public and free data.
【3】Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
标题：联合训练与非联合训练：视听说话人二元化中的预训练语音模型探讨
链接：http://arxiv.org/pdf/2312.04131v1
作者：Huan Zhao,Li Zhang,Yue Li,Yannan Wang,Hongji Wang,Wei Rao,Qing Wang,Lei Xie摘要：标记的视听数据集的稀缺性是训练优秀视听说话人日记系统的限制。为了提高视听说话人日记的性能，我们利用预训练的监督和自监督语音模型进行视听说话人日记。具体来说，我们采用监督的~（ResNet和ECAPA-TDNN）和自我监督的预训练模型~（WavLM和HuBERT）作为端到端视听扬声器日志化~（AVSD）系统中的扬声器和音频嵌入提取器。然后，我们探讨了不同的框架，包括Transformer，Conformer和交叉注意机制，在视听解码器的有效性。为了减轻单独训练造成的性能下降，我们联合训练AVSD系统中的音频编码器，扬声器编码器和视听解码器。在MISP数据集上的实验表明，该方法具有优异的性能，并在MISP Challenge 2022中获得第三名。摘要：The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems. To improve the performance of audio-visual speaker diarization, we leverage pre-trained supervised and self-supervised speech models for audio-visual speaker diarization. Specifically, we adopt supervised~(ResNet and ECAPA-TDNN) and self-supervised pre-trained models~(WavLM and HuBERT) as the speaker and audio embedding extractors in an end-to-end audio-visual speaker diarization~(AVSD) system. Then we explore the effectiveness of different frameworks, including Transformer, Conformer, and cross-attention mechanism, in the audio-visual decoder. To mitigate the degradation of performance caused by separate training, we jointly train the audio encoder, speaker encoder, and audio-visual decoder in the AVSD system. Experiments on the MISP dataset demonstrate that the proposed method achieves superior performance and obtained third place in MISP Challenge 2022.
eess.AS音频处理
【1】Investigating the Design Space of Diffusion Models for Speech Enhancement
标题：语音增强扩散模型的设计空间研究
链接：http://arxiv.org/pdf/2312.04370v1
作者：Philippe Gonzalez,Zheng-Hua Tan,Jan Østergaard,Jesper Jensen,Tommy Sonne Alstrøm,Tobias May摘要：扩散模型是一类新的生成模型，在图像生成文献中表现出了出色的性能。因此，研究试图将扩散模型应用于其他任务，例如语音增强。一种流行的方法，在适应扩散模型的语音增强包括在建模的清洁和嘈杂的语音信号之间的渐进变换。然而，先前在图像生成文献中提出的一种流行的扩散模型框架没有考虑到这种朝向系统输入的变换，这阻止了现有的基于扩散的语音增强系统与前述扩散模型框架的关联。为了解决这个问题，我们扩展了这个框架，以考虑到清洁和嘈杂的语音信号之间的渐进变换。这使我们能够应用图像生成文献的最新发展，并系统地研究语音增强的扩散模型的设计方面，这些模型在很大程度上尚未探索，例如神经网络预处理，训练损失加权，随机微分方程（RSDs）或反向过程中注入的随机性。我们表明，以前的扩散为基础的语音增强系统的性能不能归因于清洁和嘈杂的语音信号之间的渐进变换。此外，我们表明，一个适当的选择预处理，训练损失加权，采样和采样器允许超越流行的基于扩散的语音增强系统的感知指标，同时使用更少的采样步骤，从而减少了四个因素的计算成本。摘要：Diffusion models are a new class of generative models that have shown outstanding performance in image generation literature. As a consequence, studies have attempted to apply diffusion models to other tasks, such as speech enhancement. A popular approach in adapting diffusion models to speech enhancement consists in modelling a progressive transformation between the clean and noisy speech signals. However, one popular diffusion model framework previously laid in image generation literature did not account for such a transformation towards the system input, which prevents from relating the existing diffusion-based speech enhancement systems with the aforementioned diffusion model framework. To address this, we extend this framework to account for the progressive transformation between the clean and noisy speech signals. This allows us to apply recent developments from image generation literature, and to systematically investigate design aspects of diffusion models that remain largely unexplored for speech enhancement, such as the neural network preconditioning, the training loss weighting, the stochastic differential equation (SDE), or the amount of stochasticity injected in the reverse process. We show that the performance of previous diffusion-based speech enhancement systems cannot be attributed to the progressive transformation between the clean and noisy speech signals. Moreover, we show that a proper choice of preconditioning, training loss weighting, SDE and sampler allows to outperform a popular diffusion-based speech enhancement system in terms of perceptual metrics while using fewer sampling steps, thus reducing the computational cost by a factor of four.
【2】DiaPer: End-to-End Neural Diarization with Perceiver-Based Attractors
标题：DiaPer：基于感知器的吸引子的端到端神经日志化链接：http://arxiv.org/pdf/2312.04324v1
作者：Federico Landini,Mireia Diez,Themos Stafylakis,Lukáš Burget摘要：直到最近，扬声器日记化领域一直由级联系统主导。由于其局限性，主要是关于重叠的语音和繁琐的管道，端到端模型最近得到了极大的普及。最成功的模型之一是端到端神经日志化与基于编码器-解码器的吸引子（EEND-EDA）。在这项工作中，我们将EDA模块替换为基于Perceiver的模块，并展示了其相对于EEND-EDA的优势;即在大量研究的Callhome数据集上获得更好的性能，更准确地找到对话中说话者的数量，并在长录音中几乎一半的时间内运行推理。此外，当与其他方法进行详尽的比较时，我们的模型DiaPer以非常轻巧的设计达到了卓越的性能。此外，我们与其他作品进行了比较，并在十多个公共宽带数据集的级联基线。与本出版物一起，我们发布了DiaPer的代码以及基于公共和免费数据训练的模型。摘要：Until recently, the field of speaker diarization was dominated by cascaded systems. Due to their limitations, mainly regarding overlapped speech and cumbersome pipelines, end-to-end models have gained great popularity lately. One of the most successful models is end-to-end neural diarization with encoder-decoder based attractors (EEND-EDA). In this work, we replace the EDA module with a Perceiver-based one and show its advantages over EEND-EDA; namely obtaining better performance on the largely studied Callhome dataset, finding the quantity of speakers in a conversation more accurately, and running inference on almost half of the time on long recordings. Furthermore, when exhaustively compared with other methods, our model, DiaPer, reaches remarkable performance with a very lightweight design. Besides, we perform comparisons with other works and a cascaded baseline across more than ten public wide-band datasets. Together with this publication, we release the code of DiaPer as well as models trained on public and free data.
【3】Joint Training or Not: An Exploration of Pre-trained Speech Models in Audio-Visual Speaker Diarization
标题：联合训练与非联合训练：视听说话人二元化中的预训练语音模型探讨
链接：http://arxiv.org/pdf/2312.04131v1
作者：Huan Zhao,Li Zhang,Yue Li,Yannan Wang,Hongji Wang,Wei Rao,Qing Wang,Lei Xie摘要：标记的视听数据集的稀缺性是训练优秀视听说话人日记系统的限制。为了提高视听说话人日记的性能，我们利用预训练的监督和自监督语音模型进行视听说话人日记。具体来说，我们采用监督的~（ResNet和ECAPA-TDNN）和自我监督的预训练模型~（WavLM和HuBERT）作为端到端视听扬声器日志化~（AVSD）系统中的扬声器和音频嵌入提取器。然后，我们探讨了不同的框架，包括Transformer，Conformer和交叉注意机制，在视听解码器的有效性。为了减轻单独训练造成的性能下降，我们联合训练AVSD系统中的音频编码器，扬声器编码器和视听解码器。在MISP数据集上的实验表明，该方法具有优异的性能，并在MISP Challenge 2022中获得第三名。摘要：The scarcity of labeled audio-visual datasets is a constraint for training superior audio-visual speaker diarization systems. To improve the performance of audio-visual speaker diarization, we leverage pre-trained supervised and self-supervised speech models for audio-visual speaker diarization. Specifically, we adopt supervised~(ResNet and ECAPA-TDNN) and self-supervised pre-trained models~(WavLM and HuBERT) as the speaker and audio embedding extractors in an end-to-end audio-visual speaker diarization~(AVSD) system. Then we explore the effectiveness of different frameworks, including Transformer, Conformer, and cross-attention mechanism, in the audio-visual decoder. To mitigate the degradation of performance caused by separate training, we jointly train the audio encoder, speaker encoder, and audio-visual decoder in the AVSD system. Experiments on the MISP dataset demonstrate that the proposed method achieves superior performance and obtained third place in MISP Challenge 2022.
机器翻译由腾讯交互翻译提供，仅供参考