今日论文合集：cs.SD语音4篇，eess.AS音频处理5篇。
cs.SD语音
【1】Extending Whisper with prompt tuning to target-speaker ASR
标题：将Whisper扩展为目标扬声器ASR，并提供即时调谐功能链接：https://arxiv.org/abs/2312.08079
作者：Hao Ma,Zhiyuan Peng,Mingjie Shao,Jing Li,Ju Liu备注：ICASSP 2024摘要：目标说话人自动语音识别（ASR）的目的是从多个说话人的重叠话语中转录出目标说话人的期望语音。大多数现有的目标说话人ASR（TS-ASR）方法涉及从头开始训练或完全微调预训练模型，导致显著的训练成本并且变得不适用于大型基础模型。这项工作利用快速调整，一个参数有效的微调方法，扩展耳语，一个大规模的单说话者ASR模型，TS-ASR。实验结果表明，即时调整可以实现性能媲美国家的最先进的全微调方法，而只需要约1%的特定于任务的模型参数。值得注意的是，原始Whisper的功能，如反向文本规范化和时间戳预测，保留在目标扬声器ASR中，保持生成的transmittance自然和信息。摘要：Target-speaker automatic speech recognition (ASR) aims to transcribe the desired speech of a target speaker from multi-talker overlapped utterances. Most of the existing target-speaker ASR (TS-ASR) methods involve either training from scratch or fully fine-tuning a pre-trained model, leading to significant training costs and becoming inapplicable to large foundation models. This work leverages prompt tuning, a parameter-efficient fine-tuning approach, to extend Whisper, a large-scale single-talker ASR model, to TS-ASR. Experimental results show that prompt tuning can achieve performance comparable to state-of-the-art full fine-tuning approaches while only requiring about 1% of task-specific model parameters. Notably, the original Whisper's features, such as inverse text normalization and timestamp prediction, are retained in target-speaker ASR, keeping the generated transcriptions natural and informative.
【2】 Improving Spatial Resolution of First-order Ambisonics Using Sparse MDCT  Representation标题：利用稀疏MDCT表示提高一阶并列体的空间分辨率链接：https://arxiv.org/abs/2312.08069
作者：Denis Likhachov,Nick Petrovsky,Elias Azarov备注：None摘要：提出了一种提高一阶立体混响音频空间分辨率的方法。该方法基于音频的时间/频率分解，随后从每个频率分量中提取定向平面波。该方法发展了高角分辨率平面波展开（HARPEX）和方向性音频编码（DirAC）的基本思想，利用实值稀疏分解。实值频率分量与复值频率分量相反，引入了更简单和更稳定的到达方向估计，而稀疏分解引入了一种准确和统一的方法来描述从瞬态到音调声音的不同性质的声音。摘要：The paper presents a method for improving spatial resolution of first-order ambisonic audio. The method is based on time/frequency decomposition of the audio with subsequent extraction of a directed plane wave from each frequency component. The method develops the basic ideas of high angular resolution planewave expansion (HARPEX) and directional audio coding (DirAC) taking advantage of real-valued sparse decomposition. Real-valued frequency components as opposed to complex-valued introduce simpler and more stable direction of arrival estimates, while sparse decomposition introduces an accurate and unified approach to describing sounds of different nature from transient to tonal sounds.
【3】 Time Series Diffusion Method: A Denoising Diffusion Probabilistic Model  for Vibration Signal Generation标题：时间序列扩散法：一种用于振动信号产生的去噪扩散概率模型链接：https://arxiv.org/abs/2312.07981
作者：Haiming Yi,Lei Hou,Yuhong Jin,Nasser A. Saeed摘要：扩散模型在各个研究领域都表现出强大的数据生成能力。在本文中，时间序列扩散方法（TSDM）提出了振动信号生成，利用扩散模型的基本原理。TSDM采用改进的U-网结构和注意力块，有效地分割和提取一维时间序列数据的特征。它基于正向扩散和反向去噪过程进行时间序列生成。使用单频、多频数据集和轴承故障数据集进行了实验验证。结果表明，TSDM能够准确地生成时间序列中的单频和多频特征，并对轴承故障序列的扩散生成结果保留了基频特征。最后，将TSDM应用于三个公开轴承故障数据集的小样本故障诊断，结果表明，三个数据集的小样本故障诊断准确率最高分别提高了32.380%、18.355%和9.298摘要：Diffusion models have demonstrated robust data generation capabilities in various research fields. In this paper, a Time Series Diffusion Method (TSDM) is proposed for vibration signal generation, leveraging the foundational principles of diffusion models. The TSDM uses an improved U-net architecture with attention block to effectively segment and extract features from one-dimensional time series data. It operates based on forward diffusion and reverse denoising processes for time-series generation. Experimental validation is conducted using single-frequency, multi-frequency datasets, and bearing fault datasets. The results show that TSDM can accurately generate the single-frequency and multi-frequency features in the time series and retain the basic frequency features for the diffusion generation results of the bearing fault series. Finally, TSDM is applied to the small sample fault diagnosis of three public bearing fault datasets, and the results show that the accuracy of small sample fault diagnosis of the three datasets is improved by 32.380%, 18.355% and 9.298% at most, respectively
【4】 Reacting like Humans: Incorporating Intrinsic Human Behaviors into NAO  through Sound-Based Reactions for Enhanced Sociability标题：像人类一样反应：通过基于声音的反应将人类的内在行为融入NAO，以增强社交能力链接：https://arxiv.org/abs/2312.07671
作者：Ali Ghadami,Mohammadreza Taghimohammadi,Mohammad Mohammadzadeh,Mohammad Hosseinipour,Alireza Taheri备注：14 pages, 10 figures摘要：机器人在人类中的可接受性和社会性可以通过引入类似人类的反应来显着提高。人类可以对环境事件做出快速反应，而无需思考。人类表现出自然反应的一个例子是当他们遇到一个突然而响亮的声音，吓到或吓到他们。在这样的时刻，人们可能会本能地移动他们的手，转向声音的来源，并试图确定事件的原因。这种固有的行为促使我们探索社交机器人中研究较少的部分。在这项工作中，设计了一个由动作发生器，声音分类器和YOLO物体检测器组成的多模态系统，用于感知环境，并在突然的巨响中显示自然的人类恐惧反应，最后定位环境中引起恐惧的声源。这些独特而有效的运动和推理可以模仿人类的内在反应，提高机器人的社会性。对于运动生成，提出了一种基于LSTM和MDN网络的模型来合成各种运动。此外，在声音检测的情况下，优选使用声音信号的频谱图作为其输入的迁移学习模型。在开发了用于声音检测、运动生成和图像识别的各个模型之后，它们被集成到一个在NAO机器人上实现的综合恐惧模块中。最后，对恐惧模块进行了实际应用测试，并由专家组和非专家组填写了一份调查问卷，对机器人的性能进行了评估。鉴于我们有希望的结果，这项初步的探索性研究为社会机器人提供了一个新的视角，并可能成为机器人内在人类行为和情感建模的起点。摘要：Robots' acceptability among humans and their sociability can be significantly enhanced by incorporating human-like reactions. Humans can react to environmental events very quickly and without thinking. An instance where humans display natural reactions is when they encounter a sudden and loud sound that startles or frightens them. During such moments, individuals may instinctively move their hands, turn toward the origin of the sound, and try to determine the event's cause. This inherent behavior motivated us to explore this less-studied part of social robotics. In this work, a multi-modal system composed of an action generator, sound classifier, and YOLO object detector was designed to sense the environment and, in the presence of sudden loud sounds, show natural human fear reactions, and finally, locate the fear-causing sound source in the environment. These unique and valid generated motions and inferences could imitate intrinsic human reactions and enhance the sociability of robots. For motion generation, a model based on LSTM and MDN networks was proposed to synthesize various motions. Also, in the case of sound detection, a transfer learning model was preferred that used the spectrogram of sound signals as its input. After developing individual models for sound detection, motion generation, and image recognition, they were integrated into a comprehensive fear module that was implemented on the NAO robot. Finally, the fear module was tested in practical application and two groups of experts and non-experts filled out a questionnaire to evaluate the performance of the robot. Given our promising results, this preliminary exploratory research provides a fresh perspective on social robotics and could be a starting point for modeling intrinsic human behaviors and emotions in robots.
eess.AS音频处理
【1】 Ultra Low Complexity Deep Learning Based Noise Suppression标题：基于超低复杂度深度学习的噪声抑制链接：https://arxiv.org/abs/2312.08132
作者：Shrishti Saha Shetu,Soumitro Chakrabarty,Oliver Thiergart,Edwin Mabande摘要：本文介绍了一种创新方法，用于降低深度神经网络在资源受限设备上实时语音增强的计算复杂度。所提出的方法利用两阶段处理框架，采用通道特征重定向来减少卷积运算的计算负载。通过将其与改进的幂律压缩技术相结合以增强感知质量，这种方法实现了与最先进的方法相当的噪声抑制性能，并且计算要求显著降低。值得注意的是，我们的算法表现出3至4倍的计算复杂性和内存使用比现有的最先进的方法。摘要：This paper introduces an innovative method for reducing the computational complexity of deep neural networks in real-time speech enhancement on resource-constrained devices. The proposed approach utilizes a two-stage processing framework, employing channelwise feature reorientation to reduce the computational load of convolutional operations. By combining this with a modified power law compression technique for enhanced perceptual quality, this approach achieves noise suppression performance comparable to state-of-the-art methods with significantly less computational requirements. Notably, our algorithm exhibits 3 to 4 times less computational complexity and memory usage than prior state-of-the-art approaches.
【2】 Audio Deepfake Detection with Self-Supervised WavLM and Multi-Fusion  Attentive Classifier标题：基于自监督WavLM和多融合注意力分类器的音频深伪检测链接：https://arxiv.org/abs/2312.08089
作者：Yinlin Guo,Haofan Huang,Xi Chen,He Zhao,Yuehai Wang备注：5 pages, 1 figure摘要：随着语音合成和语音转换技术的快速发展，音频Deepfake已经成为自动说话人确认（ASV）系统的严重威胁。许多对策被提出来检测这种类型的攻击。在本文中，我们报告了我们将自监督WavLM模型和多融合注意分类器结合起来进行音频deepfake检测的努力。我们的方法利用WavLM模型来提取更有利于欺骗检测的特征。然后，我们提出了一种新的多融合注意（MFA）分类器的注意统计池（ASP）层的基础上。MFA在时间和层级别上捕获音频特征的补充信息。实验表明，我们的方法在ASVspoof 2021 DF集上实现了最先进的结果，并在ASVspoof 2019和2021 LA集上提供了有竞争力的结果。摘要：With the rapid development of speech synthesis and voice conversion technologies, Audio Deepfake has become a serious threat to the Automatic Speaker Verification (ASV) system. Numerous countermeasures are proposed to detect this type of attack. In this paper, we report our efforts to combine the self-supervised WavLM model and Multi-Fusion Attentive classifier for audio deepfake detection. Our method exploits the WavLM model to extract features that are more conducive to spoofing detection for the first time. Then, we propose a novel Multi-Fusion Attentive (MFA) classifier based on the Attentive Statistics Pooling (ASP) layer. The MFA captures the complementary information of audio features at both time and layer levels. Experiments demonstrate that our methods achieve state-of-the-art results on the ASVspoof 2021 DF set and provide competitive results on the ASVspoof 2019 and 2021 LA set.
【3】 Extending Whisper with prompt tuning to target-speaker ASR标题：通过快速调谐将Whisper扩展到目标说话人ASR链接：https://arxiv.org/abs/2312.08079
作者：Hao Ma,Zhiyuan Peng,Mingjie Shao,Jing Li,Ju Liu备注：ICASSP 2024摘要：目标说话人自动语音识别（ASR）的目的是从多个说话人的重叠话语中转录出目标说话人的期望语音。大多数现有的目标说话人ASR（TS-ASR）方法涉及从头开始训练或完全微调预训练模型，导致显著的训练成本并且变得不适用于大型基础模型。这项工作利用快速调整，一个参数有效的微调方法，扩展耳语，一个大规模的单说话者ASR模型，TS-ASR。实验结果表明，即时调整可以实现性能媲美国家的最先进的全微调方法，而只需要约1%的特定于任务的模型参数。值得注意的是，原始Whisper的功能，如反向文本规范化和时间戳预测，保留在目标扬声器ASR中，保持生成的transmittance自然和信息。摘要：Target-speaker automatic speech recognition (ASR) aims to transcribe the desired speech of a target speaker from multi-talker overlapped utterances. Most of the existing target-speaker ASR (TS-ASR) methods involve either training from scratch or fully fine-tuning a pre-trained model, leading to significant training costs and becoming inapplicable to large foundation models. This work leverages prompt tuning, a parameter-efficient fine-tuning approach, to extend Whisper, a large-scale single-talker ASR model, to TS-ASR. Experimental results show that prompt tuning can achieve performance comparable to state-of-the-art full fine-tuning approaches while only requiring about 1% of task-specific model parameters. Notably, the original Whisper's features, such as inverse text normalization and timestamp prediction, are retained in target-speaker ASR, keeping the generated transcriptions natural and informative.
【4】 Improving Spatial Resolution of First-order Ambisonics Using Sparse MDCT  Representation标题：利用稀疏MDCT表示提高一阶并列体的空间分辨率链接：https://arxiv.org/abs/2312.08069
作者：Denis Likhachov,Nick Petrovsky,Elias Azarov备注：None摘要：提出了一种提高一阶立体混响音频空间分辨率的方法。该方法基于音频的时间/频率分解，随后从每个频率分量中提取定向平面波。该方法发展了高角分辨率平面波展开（HARPEX）和方向性音频编码（DirAC）的基本思想，利用实值稀疏分解。实值频率分量与复值频率分量相反，引入了更简单和更稳定的到达方向估计，而稀疏分解引入了一种准确和统一的方法来描述从瞬态到音调声音的不同性质的声音。摘要：The paper presents a method for improving spatial resolution of first-order ambisonic audio. The method is based on time/frequency decomposition of the audio with subsequent extraction of a directed plane wave from each frequency component. The method develops the basic ideas of high angular resolution planewave expansion (HARPEX) and directional audio coding (DirAC) taking advantage of real-valued sparse decomposition. Real-valued frequency components as opposed to complex-valued introduce simpler and more stable direction of arrival estimates, while sparse decomposition introduces an accurate and unified approach to describing sounds of different nature from transient to tonal sounds.
【5】 Reacting like Humans: Incorporating Intrinsic Human Behaviors into NAO  through Sound-Based Reactions for Enhanced Sociability标题：像人类一样反应：通过基于声音的反应将人类的内在行为融入NAO，以增强社交能力链接：https://arxiv.org/abs/2312.07671
作者：Ali Ghadami,Mohammadreza Taghimohammadi,Mohammad Mohammadzadeh,Mohammad Hosseinipour,Alireza Taheri备注：14 pages, 10 figures摘要：机器人在人类中的可接受性和社会性可以通过引入类似人类的反应来显着提高。人类可以对环境事件做出快速反应，而无需思考。人类表现出自然反应的一个例子是当他们遇到一个突然而响亮的声音，吓到或吓到他们。在这样的时刻，人们可能会本能地移动他们的手，转向声音的来源，并试图确定事件的原因。这种固有的行为促使我们探索社交机器人中研究较少的部分。在这项工作中，设计了一个由动作发生器，声音分类器和YOLO物体检测器组成的多模态系统，用于感知环境，并在突然的巨响中显示自然的人类恐惧反应，最后定位环境中引起恐惧的声源。这些独特而有效的运动和推理可以模仿人类的内在反应，提高机器人的社会性。对于运动生成，提出了一种基于LSTM和MDN网络的模型来合成各种运动。此外，在声音检测的情况下，优选使用声音信号的频谱图作为其输入的迁移学习模型。在开发了用于声音检测、运动生成和图像识别的各个模型之后，它们被集成到一个在NAO机器人上实现的综合恐惧模块中。最后，对恐惧模块进行了实际应用测试，并由专家组和非专家组填写了一份调查问卷，对机器人的性能进行了评估。鉴于我们有希望的结果，这项初步的探索性研究为社会机器人提供了一个新的视角，并可能成为机器人内在人类行为和情感建模的起点。摘要：Robots' acceptability among humans and their sociability can be significantly enhanced by incorporating human-like reactions. Humans can react to environmental events very quickly and without thinking. An instance where humans display natural reactions is when they encounter a sudden and loud sound that startles or frightens them. During such moments, individuals may instinctively move their hands, turn toward the origin of the sound, and try to determine the event's cause. This inherent behavior motivated us to explore this less-studied part of social robotics. In this work, a multi-modal system composed of an action generator, sound classifier, and YOLO object detector was designed to sense the environment and, in the presence of sudden loud sounds, show natural human fear reactions, and finally, locate the fear-causing sound source in the environment. These unique and valid generated motions and inferences could imitate intrinsic human reactions and enhance the sociability of robots. For motion generation, a model based on LSTM and MDN networks was proposed to synthesize various motions. Also, in the case of sound detection, a transfer learning model was preferred that used the spectrogram of sound signals as its input. After developing individual models for sound detection, motion generation, and image recognition, they were integrated into a comprehensive fear module that was implemented on the NAO robot. Finally, the fear module was tested in practical application and two groups of experts and non-experts filled out a questionnaire to evaluate the performance of the robot. Given our promising results, this preliminary exploratory research provides a fresh perspective on social robotics and could be a starting point for modeling intrinsic human behaviors and emotions in robots.
机器翻译由腾讯交互翻译提供，仅供参考