今日论文合集：cs.SD语音2篇，eess.AS音频处理5篇。
cs.SD语音

【1】 Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

标题：基于物理驱动的视频碰撞声合成扩散模型

链接：http://arxiv.org/pdf/2303.16897v1

作者：Kun Su,Kaizhi Qian,Eli Shlizerman,Antonio Torralba,Chuang Gan机构：University of Washington, MIT-IBM Watson AI Lab, UMass Amherst备注：CVPR 2023. Project page: this https URL摘要：对物理对象交互发出的声音进行建模对于真实和虚拟世界中的沉浸式感知体验至关重要。传统的冲击声合成方法是通过物理模拟来获得一组物理参数来表征和合成声音。然而，它们需要物体几何形状和撞击位置的精细细节，这在现实世界中很少可用，并且不能应用于从普通视频合成撞击声音。另一方面，现有的基于视频驱动的深度学习方法由于缺乏物理知识，只能捕获视觉内容和撞击声之间的弱对应关系。在这项工作中，我们提出了一个物理驱动的扩散模型，可以合成高保真的影响声音的无声的视频剪辑。除了视频内容，我们建议使用额外的物理先验来指导冲击声合成过程。物理先验包括直接从嘈杂的现实世界冲击声示例估计而无需复杂设置的物理参数和经由神经网络解释声音环境的学习的残差参数。我们进一步实现了一种新的扩散模型，具有特定的训练和推理策略，以结合物理先验和视觉信息进行冲击声合成。实验结果表明，我们的模型优于几个现有的系统在产生逼真的冲击声。更重要的是，基于物理的表示是完全可解释和透明的，从而使我们能够灵活地执行声音编辑。摘要：Modeling sounds emitted from physical object interactions is critical for immersive perceptual experiences in real and virtual worlds. Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound. However, they require fine details of both the object geometries and impact locations, which are rarely available in the real world and can not be applied to synthesize impact sounds from common videos. On the other hand, existing video-driven deep learning-based approaches could only capture the weak correspondence between visual content and impact sounds since they lack of physics knowledge. In this work, we propose a physics-driven diffusion model that can synthesize high-fidelity impact sound for a silent video clip. In addition to the video content, we propose to use additional physics priors to guide the impact sound synthesis procedure. The physics priors include both physics parameters that are directly estimated from noisy real-world impact sound examples without sophisticated setup and learned residual parameters that interpret the sound environment via neural networks. We further implement a novel diffusion model with specific training and inference strategies to combine physics priors and visual information for impact sound synthesis. Experimental results show that our model outperforms several existing systems in generating realistic impact sounds. More importantly, the physics-based representations are fully interpretable and transparent, thus enabling us to perform sound editing flexibly.

【2】 AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

标题：AVFormer：将视觉注入到Zero-ShotAV-ASR的冻结语音模型中

链接：http://arxiv.org/pdf/2303.16501v1

作者：Paul Hongsuck Seo,Arsha Nagrani,Cordelia Schmid机构：Google Research备注：CVPR 2023摘要：视听自动语音识别（AV-ASR）旨在通过引入视觉信息来提高语音识别系统的鲁棒性。然而，从头开始训练用于此任务的完全监督的多模态模型受到对大型标记视听数据集（在每个下游感兴趣域中）的需求的限制。我们提出了AVFormer，一个简单的方法，用于增强音频模型与视觉信息，同时执行轻量级域适应。我们通过（i）使用轻量级可训练适配器将视觉嵌入注入到冻结的ASR模型中来做到这一点。我们表明，这些可以训练少量的弱标记的视频数据，最小的额外的训练时间和参数。(ii)我们还介绍了一个简单的课程计划，在训练过程中，我们表明是至关重要的，使模型能够共同处理音频和视频信息有效;最后（iii）我们证明了我们的模型在三个不同的AV-ASR基准测试（How2，VisSpeech和Ego 4D）上实现了最先进的zero-shot结果，同时还在传统的纯音频语音识别基准测试（LibriSpeech）上保持了良好的性能。定性结果表明，我们的模型有效地利用视觉信息的强大的语音识别。摘要：Audiovisual automatic speech recognition (AV-ASR) aims to improve the robustness of a speech recognition system by incorporating visual information. Training fully supervised multimodal models for this task from scratch, however is limited by the need for large labelled audiovisual datasets (in each downstream domain of interest). We present AVFormer, a simple method for augmenting audio-only models with visual information, at the same time performing lightweight domain adaptation. We do this by (i) injecting visual embeddings into a frozen ASR model using lightweight trainable adaptors. We show that these can be trained on a small amount of weakly labelled video data with minimum additional training time and parameters. (ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech). Qualitative results show that our model effectively leverages visual information for robust speech recognition.
eess.AS音频处理

【1】 Joint unsupervised and supervised learning for context-aware language identification

标题：上下文感知语言识别的无监督和有监督联合学习

链接：http://arxiv.org/pdf/2303.16511v1

作者：Jinseok Park,Hyung Yong Kim,Jihwan Park,Byeong-Yeol Kim,Shukjae Choi,Yunkyu Lim机构：dot Inc., Seoul, Republic of Korea备注：Accepted by ICASSP 2023摘要：语言识别（LID）自动识别所讲话语的语言。根据最近的研究，使用自动语音识别（ASR）任务训练的LID模型比仅使用LID任务训练的模型表现更好。然而，我们需要额外的文本标签来训练模型来识别语音，并且获取文本标签的成本很高。为了克服这个问题，我们提出了上下文感知的语言识别使用无监督和监督学习的组合，没有任何文本标签。所提出的方法通过掩蔽语言模型（MLM）损失学习语音的上下文，同时训练以确定具有监督学习损失的话语的语言。研究发现，与仅在VoxLingua107数据集的一个子集上进行监督学习训练的相同结构模型相比，所提出的联合学习将错误率降低了15.6%，该数据集由11种语言的不到三秒的话语组成。摘要：Language identification (LID) recognizes the language of a spoken utterance automatically. According to recent studies, LID models trained with an automatic speech recognition (ASR) task perform better than those trained with a LID task only. However, we need additional text labels to train the model to recognize speech, and acquiring the text labels is a cost high. In order to overcome this problem, we propose context-aware language identification using a combination of unsupervised and supervised learning without any text labels. The proposed method learns the context of speech through masked language modeling (MLM) loss and simultaneously trains to determine the language of the utterance with supervised learning loss. The proposed joint learning was found to reduce the error rate by 15.6% compared to the same structure model trained by supervised-only learning on a subset of the VoxLingua107 dataset consisting of sub-three-second utterances in 11 languages.

【2】 Kernel-interpolation-based spatial active noise control with exterior radiation suppression

标题：基于核内插的外辐射抑制空间有源噪声控制

链接：http://arxiv.org/pdf/2303.16389v1

作者：Kazuyuki Arikawa,Shoichi Koyama,Hiroshi Saruwatari机构：and Hiroshi SARUWATARI( 1)( 1)Graduate School of Information Science and Technology, The University of Tokyo备注：Accepted to International Congress on Acoustics (ICAS) 2022摘要：提出了一种基于外辐射抑制声场核插值的空间有源噪声控制方法。空间ANC的目的是通过使用多个次级源和麦克风来减少目标区域上的传入噪声。基于声场的核插值的方法允许在具有任意几何形状的阵列的区域空间中的噪声衰减。成本函数被定义为声学势能，即，目标区域内的功率分布的区域积分。然而，该成本函数没有考虑二次源的外部辐射。因此，外部区域中的声功率可以通过次级源的输出来放大。我们提出了两个空间ANC方法与外部辐射抑制。第一种方法是基于最小化的成本函数，制定为内部声学势能和外部辐射功率的总和。第二种方法是基于最小化的内部声学势能与不等式约束的外部辐射功率。最小化的成本函数的自适应算法推导出的两种方法。数值实验结果表明，该方法可以在抑制外部辐射的同时，有效地降低车内区域噪声。摘要：A spatial active noise control (ANC) method based on kernel interpolation of a sound field with exterior radiation suppression is proposed. The aim of spatial ANC is to reduce incoming noise over a target region by using multiple secondary sources and microphones. The method based on kernel interpolation of a sound field allows noise attenuation in a regional space with an array of arbitrary geometry. The cost function is defined as the acoustic potential energy, i.e., the regional integral of the power distribution inside the target region. However, this cost function does not take into consideration the exterior radiation of secondary sources. Thus, the acoustic power in the exterior region can be amplified by the output of the secondary sources. We propose two spatial ANC methods with exterior radiation suppression. The first approach is based on the minimization of the cost function formulated as a sum of the interior acoustic potential energy and exterior radiation power. The second approach is based on the minimization of the interior acoustic potential energy with inequality constraints on the exterior radiation power. Adaptive algorithms for minimizing the cost function are derived for the two approaches. Numerical experimental results indicate that the proposed methods can reduce the interior regional noise while suppressing the exterior radiation.

【3】 Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos

标题：基于物理驱动的视频碰撞声合成扩散模型

链接：http://arxiv.org/pdf/2303.16897v1

*与cs.SD语音【1】为同一篇

作者：Kun Su,Kaizhi Qian,Eli Shlizerman,Antonio Torralba,Chuang Gan机构：University of Washington, MIT-IBM Watson AI Lab, UMass Amherst备注：CVPR 2023. Project page: this https URL摘要：对物理对象交互发出的声音进行建模对于真实和虚拟世界中的沉浸式感知体验至关重要。传统的冲击声合成方法是通过物理模拟来获得一组物理参数来表征和合成声音。然而，它们需要物体几何形状和撞击位置的精细细节，这在现实世界中很少可用，并且不能应用于从普通视频合成撞击声音。另一方面，现有的基于视频驱动的深度学习方法由于缺乏物理知识，只能捕获视觉内容和撞击声之间的弱对应关系。在这项工作中，我们提出了一个物理驱动的扩散模型，可以合成高保真的影响声音的无声的视频剪辑。除了视频内容，我们建议使用额外的物理先验来指导冲击声合成过程。物理先验包括直接从嘈杂的现实世界冲击声示例估计而无需复杂设置的物理参数和经由神经网络解释声音环境的学习的残差参数。我们进一步实现了一种新的扩散模型，具有特定的训练和推理策略，以结合物理先验和视觉信息进行冲击声合成。实验结果表明，我们的模型优于几个现有的系统在产生逼真的冲击声。更重要的是，基于物理的表示是完全可解释和透明的，从而使我们能够灵活地执行声音编辑。摘要：Modeling sounds emitted from physical object interactions is critical for immersive perceptual experiences in real and virtual worlds. Traditional methods of impact sound synthesis use physics simulation to obtain a set of physics parameters that could represent and synthesize the sound. However, they require fine details of both the object geometries and impact locations, which are rarely available in the real world and can not be applied to synthesize impact sounds from common videos. On the other hand, existing video-driven deep learning-based approaches could only capture the weak correspondence between visual content and impact sounds since they lack of physics knowledge. In this work, we propose a physics-driven diffusion model that can synthesize high-fidelity impact sound for a silent video clip. In addition to the video content, we propose to use additional physics priors to guide the impact sound synthesis procedure. The physics priors include both physics parameters that are directly estimated from noisy real-world impact sound examples without sophisticated setup and learned residual parameters that interpret the sound environment via neural networks. We further implement a novel diffusion model with specific training and inference strategies to combine physics priors and visual information for impact sound synthesis. Experimental results show that our model outperforms several existing systems in generating realistic impact sounds. More importantly, the physics-based representations are fully interpretable and transparent, thus enabling us to perform sound editing flexibly.

【4】 AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR

标题：AVFormer：将视觉注入到Zero-ShotAV-ASR的冻结语音模型中

链接：http://arxiv.org/pdf/2303.16501v1

*与cs.SD语音【2】为同一篇

作者：Paul Hongsuck Seo,Arsha Nagrani,Cordelia Schmid机构：Google Research备注：CVPR 2023摘要：视听自动语音识别（AV-ASR）旨在通过引入视觉信息来提高语音识别系统的鲁棒性。然而，从头开始训练用于此任务的完全监督的多模态模型受到对大型标记视听数据集（在每个下游感兴趣域中）的需求的限制。我们提出了AVFormer，一个简单的方法，用于增强音频模型与视觉信息，同时执行轻量级域适应。我们通过（i）使用轻量级可训练适配器将视觉嵌入注入到冻结的ASR模型中来做到这一点。我们表明，这些可以训练少量的弱标记的视频数据，最小的额外的训练时间和参数。(ii)我们还介绍了一个简单的课程计划，在训练过程中，我们表明是至关重要的，使模型能够共同处理音频和视频信息有效;最后（iii）我们证明了我们的模型在三个不同的AV-ASR基准测试（How2，VisSpeech和Ego 4D）上实现了最先进的zero-shot结果，同时还在传统的纯音频语音识别基准测试（LibriSpeech）上保持了良好的性能。定性结果表明，我们的模型有效地利用视觉信息的强大的语音识别。摘要：Audiovisual automatic speech recognition (AV-ASR) aims to improve the robustness of a speech recognition system by incorporating visual information. Training fully supervised multimodal models for this task from scratch, however is limited by the need for large labelled audiovisual datasets (in each downstream domain of interest). We present AVFormer, a simple method for augmenting audio-only models with visual information, at the same time performing lightweight domain adaptation. We do this by (i) injecting visual embeddings into a frozen ASR model using lightweight trainable adaptors. We show that these can be trained on a small amount of weakly labelled video data with minimum additional training time and parameters. (ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech). Qualitative results show that our model effectively leverages visual information for robust speech recognition.

【5】 Help the Blind See: Assistance for the Visually Impaired through Augmented Acoustic Simulation

标题：助盲：通过增强声学模拟为视障人士提供帮助

链接：http://arxiv.org/pdf/2303.13536v1

作者：Alexander Mehta,Ritik Jalisatgi机构：Theia Project, Sunnyvale, CA摘要：估计有2.53亿人有视力障碍。这些视觉障碍影响着日常生活，限制了他们对外界的理解。这可能会因跌倒或碰撞而对健康造成风险。我们提出了一个解决方案，通过声音快速和详细地传达环境空间几何形状，提供盲人和视力受损的人通过声音技术了解他们的空间环境的能力。该模型包括快速目标检测和3D环境映射，这是通过一系列快速的声音笔记进行交流。这些声音音符具有不同的频率、音高和排列，以便精确地传达环境中点的深度和位置。声音以音符的形式传递，以便易于识别和区分。使用独特的算法来分割对象，提供最小的精度损失和从正常O（n2）到O（n）的改进（这是重要的，因为点云中的N通常可以在105的范围内）。在测试中，我们在详细的对象上实现了0.866的R值，并且在具有大量噪声的夜间户外场景上实现了87.5%的准确度。我们还提供了我们的系统的补充视频演示。摘要：An estimated 253 million people have visual impairments. These visual impairments affect everyday lives, and limit their understanding of the outside world. This can pose a risk to health from falling or collisions. We propose a solution to this through quick and detailed communication of environmental spatial geometry through sound, providing the blind and visually impaired the ability to understand their spatial environment through sound technology. The model consists of fast object detection and 3D environmental mapping, which is communicated through a series of quick sound notes. These sound notes are at different frequencies, pitches, and arrangements in order to precisely communicate the depth and location of points within the environment. Sounds are communicated in the form of musical notes in order to be easily recognizable and distinguishable. A unique algorithm is used to segment objects, providing minimal accuracy loss and improvement from the normal O(n2 ) to O(n) (which is significant, as N in point clouds can often be in the range of 105 ). In testing, we achieved an R-value of 0.866 on detailed objects and an accuracy of 87.5% on an outdoor scene at night with large amounts of noise. We also provide a supplementary video demo of our system.

机器翻译，仅供参考