今日论文合集：cs.SD语音4篇，eess.AS音频处理5篇。
cs.SD语音
【1】 Integrating Plug-and-Play Data Priors with Weighted Prediction Error for  Speech Dereverberation标题：结合即插即用数据先验和加权预测误差的语音去混响算法链接：https://arxiv.org/abs/2312.02773
作者：Ziye Yang,Wenxing Yang,Kai Xie,Jie Chen摘要：语音去混响的目的是减轻后期混响分量的不利影响。虽然加权预测误差（WPE）方法在去混响方面表现出优越的性能，但在复杂和噪声环境中的性能和鲁棒性方面仍有进一步改进的空间。最近的研究强调了整合基于物理和数据驱动的方法的有效性，提高了各种信号处理任务的性能，同时保持了可解释性。受这些进步的启发，本文提出了一种新的去混响框架，它结合了数据驱动的方法来捕获语音先验的WPE框架内。即插即用策略（PIPE），特别是去噪正则化（RED）策略，用于在优化问题求解迭代期间将从数据中学习的语音先验信息合并。实验结果验证了该方法的有效性。摘要：Speech dereverberation aims to alleviate the detrimental effects of late-reverberant components. While the weighted prediction error (WPE) method has shown superior performance in dereverberation, there is still room for further improvement in terms of performance and robustness in complex and noisy environments. Recent research has highlighted the effectiveness of integrating physics-based and data-driven methods, enhancing the performance of various signal processing tasks while maintaining interpretability. Motivated by these advancements, this paper presents a novel dereverberation frame-work, which incorporates data-driven methods for capturing speech priors within the WPE framework. The plug-and-play strategy (PnP), specifically the regularization by denoising (RED) strategy, is utilized to incorporate speech prior information learnt from data during the optimization problem solving iterations. Experimental results validate the effectiveness of the proposed approach.
【2】 Synthetic Data Generation Techniques for Developing AI-based Speech  Assessments for Parkinson's Disease (A Comparative Study)标题：用于开发基于人工智能的帕金森病语音评估的合成数据生成技术(一项比较研究)链接：https://arxiv.org/abs/2312.02229
作者：Mahboobeh Parsapoor备注：6, 5 Tables, 5 Figures摘要：言语和语言的变化是帕金森病（PD）的第一个迹象。因此，临床医生多年来一直试图从声音中识别PD患者。由于人工智能（AI）的进步，医生可以利用基于AI的语音评估来发现PD。这样的人工智能系统可以使用机器学习分类器来开发，这些分类器已经使用个人的声音进行了训练。尽管一些研究已经显示出开发这种人工智能系统的合理结果，但这些系统需要更多的数据样本才能实现有希望的性能。本文探讨了使用基于深度学习的数据生成技术来提高机器学习分类器的准确性，这些分类器是此类系统的核心。摘要：Changes in speech and language are among the first signs of Parkinson's disease (PD). Thus, clinicians have tried to identify individuals with PD from their voices for years. Doctors can leverage AI-based speech assessments to spot PD thanks to advancements in artificial intelligence (AI). Such AI systems can be developed using machine learning classifiers that have been trained using individuals' voices. Although several studies have shown reasonable results in developing such AI systems, these systems would need more data samples to achieve promising performance. This paper explores using deep learning-based data generation techniques on the accuracy of machine learning classifiers that are the core of such systems.
【3】 Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions  Using a Heun-Based Sampler标题：基于Heun采样器的匹配和失配条件下的扩散语音增强链接：https://arxiv.org/abs/2312.02683
作者：Philippe Gonzalez,Zheng-Hua Tan,Jan Østergaard,Jesper Jensen,Tommy Sonne Alstrøm,Tobias May摘要：扩散模型是近年来成功应用于语音增强的一类新的生成模型。以前的作品已经证明了其优越的性能在不匹配的条件下相比，国家的最先进的判别模型。然而，这是用一个用于训练的数据库和另一个用于测试的数据库进行研究的，这使得结果高度依赖于特定的数据库。此外，最近的发展，从图像生成文学仍然在很大程度上未开发的语音增强。这些包括扩散模型的几个设计方面，例如噪声时间表或反向采样器。在这项工作中，我们系统地评估了基于扩散的语音增强模型的泛化性能，通过使用多个语音，噪声和双耳房间脉冲响应（BRIR）数据库来模拟不匹配的声学条件。我们还实验了噪声时间表和采样器，没有被应用到语音增强之前。我们表明，所提出的系统大大受益于使用多个数据库进行训练，并实现了卓越的性能相比，最先进的判别模型在匹配和不匹配的条件。我们还表明，Heun为基础的采样器实现了优越的性能，以较小的计算成本相比，通常用于语音增强的采样器。摘要：Diffusion models are a new class of generative models that have recently been applied to speech enhancement successfully. Previous works have demonstrated their superior performance in mismatched conditions compared to state-of-the art discriminative models. However, this was investigated with a single database for training and another one for testing, which makes the results highly dependent on the particular databases. Moreover, recent developments from the image generation literature remain largely unexplored for speech enhancement. These include several design aspects of diffusion models, such as the noise schedule or the reverse sampler. In this work, we systematically assess the generalization performance of a diffusion-based speech enhancement model by using multiple speech, noise and binaural room impulse response (BRIR) databases to simulate mismatched acoustic conditions. We also experiment with a noise schedule and a sampler that have not been applied to speech enhancement before. We show that the proposed system substantially benefits from using multiple databases for training, and achieves superior performance compared to state-of-the-art discriminative models in both matched and mismatched conditions. We also show that a Heun-based sampler achieves superior performance at a smaller computational cost compared to a sampler commonly used for speech enhancement.
【4】 Auralization based on multi-perspective ambisonic room impulse responses标题：基于多视角房间冲激响应的可听化链接：https://arxiv.org/abs/2312.02581
作者：Kaspar Müller,Franz Zotter备注：None摘要：大多数情况下，虚拟声学渲染采用实时更新的房间声学模拟来实现可变收听者视角的可听化。作为一种替代方案，我们提出并测试一种技术来内插房间脉冲响应，特别是高保真度立体声房间脉冲响应（ARIR），可在一个网格的空间分布的接收器的角度，测量或模拟在所需的声学环境。特别是，我们外推一个三元组的相邻ARIR的变量...的角度来看，他们的线性插值之前。通过将每个ARIR分解成局部声音事件并将它们的方向、时间和水平重新分配给在收听者视角下可以观察到的内容来实现外推，其中具有尽可能多的时间、方向和视角上下文。我们建议进行这种分解在两个层次：在早期ARIR的峰值分解成联合本地化的声音事件，根据时间差的到达观察到的ARIR三重，或所有ARIR观察到的直接声音。声音事件，不能联合本地化被视为残差，其不太精确的定位利用到达方向检测和估计的到达时间。对于插值渲染，找到合适的参数设置，通过评估所提出的方法在听实验中，使用测量和模拟的ARIR数据集，在静态和时变条件下。摘要：Most often, virtual acoustic rendering employs real-time updated room acoustic simulations to accomplish auralization for a variable listener perspective. As an alternative, we propose and test a technique to interpolate room impulse responses, specifically Ambisonic room impulse responses (ARIRs) available at a grid of spatially distributed receiver perspectives, measured or simulated in a desired acoustic environment. In particular, we extrapolate a triplet of neighboring ARIRs to the variable listener perspective, preceding their linear interpolation. The extrapolation is achieved by decomposing each ARIR into localized sound events and re-assigning their direction, time, and level to what could be observed at the listener perspective, with as much temporal, directional, and perspective context as possible. We propose to undertake this decomposition in two levels: Peaks in the early ARIRs are decomposed into jointly localized sound events, based on time differences of arrival observed in either an ARIR triplet, or all ARIRs observing the direct sound. Sound events that could not be jointly localized are treated as residuals whose less precise localization utilizes direction-of-arrival detection and the estimated time of arrival. For the interpolated rendering, suitable parameter settings are found by evaluating the proposed method in a listening experiment, using both measured and simulated ARIR data sets, under static and time-varying conditions.
eess.AS音频处理
【1】 Diffusion-Based Speech Enhancement in Matched and Mismatched Conditions  Using a Heun-Based Sampler标题：基于Heun采样器的匹配和失配条件下基于扩散的语音增强链接：https://arxiv.org/abs/2312.02683
作者：Philippe Gonzalez,Zheng-Hua Tan,Jan Østergaard,Jesper Jensen,Tommy Sonne Alstrøm,Tobias May摘要：None摘要：Diffusion models are a new class of generative models that have recently been applied to speech enhancement successfully. Previous works have demonstrated their superior performance in mismatched conditions compared to state-of-the art discriminative models. However, this was investigated with a single database for training and another one for testing, which makes the results highly dependent on the particular databases. Moreover, recent developments from the image generation literature remain largely unexplored for speech enhancement. These include several design aspects of diffusion models, such as the noise schedule or the reverse sampler. In this work, we systematically assess the generalization performance of a diffusion-based speech enhancement model by using multiple speech, noise and binaural room impulse response (BRIR) databases to simulate mismatched acoustic conditions. We also experiment with a noise schedule and a sampler that have not been applied to speech enhancement before. We show that the proposed system substantially benefits from using multiple databases for training, and achieves superior performance compared to state-of-the-art discriminative models in both matched and mismatched conditions. We also show that a Heun-based sampler achieves superior performance at a smaller computational cost compared to a sampler commonly used for speech enhancement.
【2】 Auralization based on multi-perspective ambisonic room impulse responses标题：基于多视角房间冲激响应的可听化链接：https://arxiv.org/abs/2312.02581
作者：Kaspar Müller,Franz Zotter备注：None摘要：大多数情况下，虚拟声学渲染采用实时更新的房间声学模拟来实现可变收听者视角的可听化。作为一种替代方案，我们提出并测试一种技术来内插房间脉冲响应，特别是高保真度立体声房间脉冲响应（ARIR），可在一个网格的空间分布的接收器的角度，测量或模拟在所需的声学环境。特别是，我们外推一个三元组的相邻ARIR的变量...的角度来看，他们的线性插值之前。通过将每个ARIR分解成局部声音事件并将它们的方向、时间和水平重新分配给在收听者视角下可以观察到的内容来实现外推，其中具有尽可能多的时间、方向和视角上下文。我们建议进行这种分解在两个层次：在早期ARIR的峰值分解成联合本地化的声音事件，根据时间差的到达观察到的ARIR三重，或所有ARIR观察到的直接声音。声音事件，不能联合本地化被视为残差，其不太精确的定位利用到达方向检测和估计的到达时间。对于插值渲染，找到合适的参数设置，通过评估所提出的方法在听实验中，使用测量和模拟的ARIR数据集，在静态和时变条件下。摘要：Most often, virtual acoustic rendering employs real-time updated room acoustic simulations to accomplish auralization for a variable listener perspective. As an alternative, we propose and test a technique to interpolate room impulse responses, specifically Ambisonic room impulse responses (ARIRs) available at a grid of spatially distributed receiver perspectives, measured or simulated in a desired acoustic environment. In particular, we extrapolate a triplet of neighboring ARIRs to the variable listener perspective, preceding their linear interpolation. The extrapolation is achieved by decomposing each ARIR into localized sound events and re-assigning their direction, time, and level to what could be observed at the listener perspective, with as much temporal, directional, and perspective context as possible. We propose to undertake this decomposition in two levels: Peaks in the early ARIRs are decomposed into jointly localized sound events, based on time differences of arrival observed in either an ARIR triplet, or all ARIRs observing the direct sound. Sound events that could not be jointly localized are treated as residuals whose less precise localization utilizes direction-of-arrival detection and the estimated time of arrival. For the interpolated rendering, suitable parameter settings are found by evaluating the proposed method in a listening experiment, using both measured and simulated ARIR data sets, under static and time-varying conditions.
【3】 Integrating Plug-and-Play Data Priors with Weighted Prediction Error for  Speech Dereverberation标题：基于即插即用数据先验和加权预测误差的语音去混响链接：https://arxiv.org/abs/2312.02773
作者：Ziye Yang,Wenxing Yang,Kai Xie,Jie Chen摘要：语音去混响的目的是减轻后期混响分量的不利影响。虽然加权预测误差（WPE）方法在去混响方面表现出优越的性能，但在复杂和噪声环境中的性能和鲁棒性方面仍有进一步改进的空间。最近的研究强调了整合基于物理和数据驱动的方法的有效性，提高了各种信号处理任务的性能，同时保持了可解释性。受这些进步的启发，本文提出了一种新的去混响框架，它结合了数据驱动的方法来捕获语音先验的WPE框架内。即插即用策略（PIPE），特别是去噪正则化（RED）策略，用于在优化问题求解迭代期间将从数据中学习的语音先验信息合并。实验结果验证了该方法的有效性。摘要：Speech dereverberation aims to alleviate the detrimental effects of late-reverberant components. While the weighted prediction error (WPE) method has shown superior performance in dereverberation, there is still room for further improvement in terms of performance and robustness in complex and noisy environments. Recent research has highlighted the effectiveness of integrating physics-based and data-driven methods, enhancing the performance of various signal processing tasks while maintaining interpretability. Motivated by these advancements, this paper presents a novel dereverberation frame-work, which incorporates data-driven methods for capturing speech priors within the WPE framework. The plug-and-play strategy (PnP), specifically the regularization by denoising (RED) strategy, is utilized to incorporate speech prior information learnt from data during the optimization problem solving iterations. Experimental results validate the effectiveness of the proposed approach.
【4】 AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation  with Unified Audio-Visual Speech Representation标题：AV2AV：具有统一视听语音表示的直接视听语音到视听语音翻译链接：https://arxiv.org/abs/2312.02512
作者：Jeongsoo Choi,Se Jin Park,Minsu Kim,Yong Man Ro摘要：本文提出了一种新的直接视听语音到视听语音翻译（AV 2AV）框架，其中系统的输入和输出是多模态的（即，音频和视频语音）。通过所提出的AV2AV，可以带来两个关键优势：1）我们可以通过使用我们自己的主要语言在虚拟会议中与世界各地的个人进行类似真实的对话。与仅在音频模态之间进行翻译的语音到语音翻译（A2A）相比，所提出的AV2AV直接在视听语音之间进行翻译。这种能力通过呈现与翻译的语音同步的嘴唇运动来增强对话体验。2)从而提高口语翻译系统的鲁棒性。通过利用视听语音的互补信息，即使在存在声学噪声的情况下，系统也可以有效地翻译口语，表现出鲁棒性。为了缓解缺乏并行AV2AV翻译数据集的问题，我们建议用A2A的仅音频数据集来训练我们的口语翻译系统。这是通过预先通过自我监督学习来学习统一的视听语音表示以训练翻译系统来完成的。此外，我们提出了一个AV渲染器，可以并行生成原始音频和视频。该方法采用zero-shot说话人建模，使源视听语音中的说话人保持在目标翻译视听语音中。AV2AV的有效性进行了评估，在一个多对多的语言翻译设置与广泛的实验。演示页面可在https: choijeongsoo.github.io av2av上找到。摘要：This paper proposes a novel direct Audio-Visual Speech to Audio-Visual Speech Translation (AV2AV) framework, where the input and output of the system are multimodal (i.e., audio and visual speech). With the proposed AV2AV, two key advantages can be brought: 1) We can perform real-like conversations with individuals worldwide in a virtual meeting by utilizing our own primary languages. In contrast to Speech-to-Speech Translation (A2A), which solely translates between audio modalities, the proposed AV2AV directly translates between audio-visual speech. This capability enhances the dialogue experience by presenting synchronized lip movements along with the translated speech. 2) We can improve the robustness of the spoken language translation system. By employing the complementary information of audio-visual speech, the system can effectively translate spoken language even in the presence of acoustic noise, showcasing robust performance. To mitigate the problem of the absence of a parallel AV2AV translation dataset, we propose to train our spoken language translation system with the audio-only dataset of A2A. This is done by learning unified audio-visual speech representations through self-supervised learning in advance to train the translation system. Moreover, we propose an AV-Renderer that can generate raw audio and video in parallel. It is designed with zero-shot speaker modeling, thus the speaker in source audio-visual speech can be maintained at the target translated audio-visual speech. The effectiveness of AV2AV is evaluated with extensive experiments in a many-to-many language translation setting. The demo page is available on https: choijeongsoo.github.io av2av.
【5】 Synthetic Data Generation Techniques for Developing AI-based Speech  Assessments for Parkinson's Disease (A Comparative Study)标题：用于开发基于人工智能的帕金森病语音评估的合成数据生成技术(一项比较研究)链接：https://arxiv.org/abs/2312.02229
作者：Mahboobeh Parsapoor备注：6, 5 Tables, 5 Figures摘要：言语和语言的变化是帕金森病（PD）的第一个迹象。因此，临床医生多年来一直试图从声音中识别PD患者。由于人工智能（AI）的进步，医生可以利用基于AI的语音评估来发现PD。这样的人工智能系统可以使用机器学习分类器来开发，这些分类器已经使用个人的声音进行了训练。尽管一些研究已经显示出开发这种人工智能系统的合理结果，但这些系统需要更多的数据样本才能实现有希望的性能。本文探讨了使用基于深度学习的数据生成技术来提高机器学习分类器的准确性，这些分类器是此类系统的核心。摘要：Changes in speech and language are among the first signs of Parkinson's disease (PD). Thus, clinicians have tried to identify individuals with PD from their voices for years. Doctors can leverage AI-based speech assessments to spot PD thanks to advancements in artificial intelligence (AI). Such AI systems can be developed using machine learning classifiers that have been trained using individuals' voices. Although several studies have shown reasonable results in developing such AI systems, these systems would need more data samples to achieve promising performance. This paper explores using deep learning-based data generation techniques on the accuracy of machine learning classifiers that are the core of such systems.
机器翻译由腾讯交互翻译提供，仅供参考