今日论文合集：cs.SD语音5篇，eess.AS音频处理6篇。

cs.SD语音

【1】LCB-net: Long-Context Biasing for Audio-Visual Speech Recognition
标题：LCB-net：用于视听语音识别的长上下文偏置链接：https://arxiv.org/abs/2401.06390
作者：Fan Yu,Haoxu Wang,Xian Shi,Shiliang Zhang备注：Accepted by ICASPP 2024摘要：在线会议和课程的日益普及对利用视频幻灯片中丰富的文本信息来改进自动语音识别（ASR）提出了新的挑战。与罕见短语列表相比，视频中的幻灯片与演讲实时同步，从而能够提取长上下文偏差。因此，我们提出了一种新的长上下文偏置网络（LCB-net）的视听语音识别（AVSR），有效地利用视频中的长上下文信息。具体来说，我们采用双编码器架构，同时模拟音频和长上下文偏置。此外，我们还提出了一个偏置预测模块，利用二进制交叉熵（BCE）损失显式确定偏置短语的长上下文偏置。此外，我们引入了一个动态的上下文短语模拟，以提高我们的LCB网络的泛化和鲁棒性。实验结果表明，LCB-net在测试集上的相对WER/U-WER/B-WER约减率分别为9.4%/9.1%/10.9%，优于一般ASR模型，具有较高的无偏和有偏性能.此外，我们还评估了我们的模型在LibriSpeech语料库，导致23.8%/19.2%/35.4%的相对WER/U-WER/B-WER减少ASR模型。摘要：The growing prevalence of online conferences and courses presents a new challenge in improving automatic speech recognition (ASR) with enriched textual information from video slides. In contrast to rare phrase lists, the slides within videos are synchronized in real-time with the speech, enabling the extraction of long contextual bias. Therefore, we propose a novel long-context biasing network (LCB-net) for audio-visual speech recognition (AVSR) to leverage the long-context information available in videos effectively. Specifically, we adopt a bi-encoder architecture to simultaneously model audio and long-context biasing. Besides, we also propose a biasing prediction module that utilizes binary cross entropy (BCE) loss to explicitly determine biased phrases in the long-context biasing. Furthermore, we introduce a dynamic contextual phrases simulation to enhance the generalization and robustness of our LCB-net. Experiments on the SlideSpeech, a large-scale audio-visual corpus enriched with slides, reveal that our proposed LCB-net outperforms general ASR model by 9.4%/9.1%/10.9% relative WER/U-WER/B-WER reduction on test set, which enjoys high unbiased and biased performance. Moreover, we also evaluate our model on LibriSpeech corpus, leading to 23.8%/19.2%/35.4% relative WER/U-WER/B-WER reduction over the ASR model.
【2】 Dynamic Behaviour of Connectionist Speech Recognition with Strong  Latency Constraints标题：强延迟约束下连接式语音识别的动态行为链接：https://arxiv.org/abs/2401.06588
作者：Giampiero Salvi备注：None摘要：本文介绍了使用连接技术在语音识别的强延迟约束。通过将语音字符串馈送到发音合成器中，从语音信号实时导出合成面部的嘴唇运动的任务施加约束。特别注意已支付给分析之间的相互作用的时间演化模型学习的多层感知器和过渡模型所施加的维特比解码器，在不同的延迟条件。两个实验中，语言模型（LM）的时间依赖性进行了控制的参数。结果表明，所涉及的三个因素，即神经网络拓扑结构，LM和解码器延迟的时间依赖性的长度之间的强烈的相互作用。摘要：This paper describes the use of connectionist techniques in phonetic speech recognition with strong latency constraints. The constraints are imposed by the task of deriving the lip movements of a synthetic face in real time from the speech signal, by feeding the phonetic string into an articulatory synthesiser. Particular attention has been paid to analysing the interaction between the time evolution model learnt by the multi-layer perceptrons and the transition model imposed by the Viterbi decoder, in different latency conditions. Two experiments were conducted in which the time dependencies in the language model (LM) were controlled by a parameter. The results show a strong interaction between the three factors involved, namely the neural network topology, the length of time dependencies in the LM and the decoder latency.
【3】 Contrastive Learning With Audio Discrimination For Customizable Keyword  Spotting In Continuous Speech标题：连续语音中可定制关键词识别的对比学习和音频识别链接：https://arxiv.org/abs/2401.06485
作者：Yu Xi,Baochen Yang,Hao Li,Jiaqi Guo,Kai Yu备注：Accepted by ICASSP2024摘要：连续语音中的可定制关键词识别技术因其潜在的应用前景而受到越来越多的关注。虽然对比学习（CL）已被广泛用于提取关键词表示，以前的CL方法都是对预分割的孤立词进行操作，并且只采用音频-文本表示匹配策略。然而，对于连续语音中的KWS，协同发音和流式分词可以很容易地为不同的文本产生相似的音频模式，这可能因此触发假警报。为了解决这个问题，我们提出了一种新的CL与音频歧视（CLAD）的方法来学习关键字表示与音频文本匹配和音频音频歧视能力。这里，在训练期间，针对每个滑动窗口采用考虑音频-音频和音频-文本CL数据对的InfoNCE损失。对开源LibriPhrase数据集的评估表明，与以前的CL方法相比，使用滑动窗口级别的InfoNCE损失产生了相当的性能。此外，在连续语音数据集LibriSpeech上的实验表明，通过将音频歧视，CLAD实现了显着的性能增益CL没有音频歧视。同时，与两阶段KWS方法相比，采用CLAD的端到端KWS不仅具有更好的性能，而且具有显著的速度提升。摘要：Customizable keyword spotting (KWS) in continuous speech has attracted increasing attention due to its real-world application potential. While contrastive learning (CL) has been widely used to extract keyword representations, previous CL approaches all operate on pre-segmented isolated words and employ only audio-text representations matching strategy. However, for KWS in continuous speech, co-articulation and streaming word segmentation can easily yield similar audio patterns for different texts, which may consequently trigger false alarms. To address this issue, we propose a novel CL with Audio Discrimination (CLAD) approach to learning keyword representation with both audio-text matching and audio-audio discrimination ability. Here, an InfoNCE loss considering both audio-audio and audio-text CL data pairs is employed for each sliding window during training. Evaluations on the open-source LibriPhrase dataset show that the use of sliding-window level InfoNCE loss yields comparable performance compared to previous CL approaches. Furthermore, experiments on the continuous speech dataset LibriSpeech demonstrate that, by incorporating audio discrimination, CLAD achieves significant performance gain over CL without audio discrimination. Meanwhile, compared to two-stage KWS approaches, the end-to-end KWS with CLAD achieves not only better performance, but also significant speed-up.
【4】 Towards High-Quality and Efficient Speech Bandwidth Extension with  Parallel Amplitude and Phase Prediction标题：走向高质量高效的并行幅相预测语音带宽扩展链接：https://arxiv.org/abs/2401.06387
作者：Ye-Xin Lu,Yang Ai,Hui-Peng Du,Zhen-Hua Ling备注：Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：语音带宽扩展（BWE）是指拓宽语音信号的频带范围，使语音质量向更明亮、更饱满的方向发展。提出了一种基于生成对抗网络（GAN）的宽带语音波形生成模型AP-BWE，该模型具有幅度谱和相位谱的并行预测功能。所提出的AP-BWE发生器完全基于卷积神经网络（CNN）。它的特点是具有相互作用的双流结构，其中幅度流和相位流彼此通信，并分别从输入的窄带幅度和相位谱扩展高频分量。为了提高扩展语音信号的自然度，我们在波形级采用多周期滤波器，在谱级分别设计了一对多分辨率幅度和相位鉴别器。实验结果表明，我们提出的AP-BWE实现最先进的性能方面的语音质量BWE任务的目标采样率为16 kHz和48 kHz。在生成效率方面，由于全卷积架构和全帧级操作，拟议的AP-BWE可以在单个RTX 4090 GPU上生成比实时快292.3倍的48 kHz波形样本，在单个CPU上比实时快18.1倍。值得注意的是，据我们所知，AP-BWE是第一个实现高频相位谱直接扩展的方法，这有利于提高现有BWE方法的有效性。摘要：Speech bandwidth extension (BWE) refers to widening the frequency bandwidth range of speech signals, enhancing the speech quality towards brighter and fuller. This paper proposes a generative adversarial network (GAN) based BWE model with parallel prediction of Amplitude and Phase spectra, named AP-BWE, which achieves both high-quality and efficient wideband speech waveform generation. The proposed AP-BWE generator is entirely based on convolutional neural networks (CNNs). It features a dual-stream architecture with mutual interaction, where the amplitude stream and the phase stream communicate with each other and respectively extend the high-frequency components from the input narrowband amplitude and phase spectra. To improve the naturalness of the extended speech signals, we employ a multi-period discriminator at the waveform level and design a pair of multi-resolution amplitude and phase discriminators at the spectral level, respectively. Experimental results demonstrate that our proposed AP-BWE achieves state-of-the-art performance in terms of speech quality for BWE tasks targeting sampling rates of both 16 kHz and 48 kHz. In terms of generation efficiency, due to the all-convolutional architecture and all-frame-level operations, the proposed AP-BWE can generate 48 kHz waveform samples 292.3 times faster than real-time on a single RTX 4090 GPU and 18.1 times faster than real-time on a single CPU. Notably, to our knowledge, AP-BWE is the first to achieve the direct extension of the high-frequency phase spectrum, which is beneficial for improving the effectiveness of existing BWE methods.
【5】 Remixing Music for Hearing Aids Using Ensemble of Fine-Tuned Source  Separators标题：利用微调声源分离器组合的助听器混音链接：https://arxiv.org/abs/2401.06203
作者：Matthew Daly备注：2 pages, ICASSP 2024, Cadenza Grand Challenge摘要：本文介绍了我们为Cadenza ICASSP 2024 Grand Challenge提交的系统，该挑战提出了为助听器用户重新混合和增强音乐的问题。我们的系统在挑战中排名第一，在评估数据集上获得了最佳的助听器音频质量指数（HAAQI）平均得分。我们描述了这个系统，它使用了一系列深度学习音乐源分离器，这些分离器在挑战数据上进行了微调。我们通过挑战结果证明了我们系统的有效性，并通过消融研究分析了不同系统方面的重要性。摘要：This paper introduces our system submission for the Cadenza ICASSP 2024 Grand Challenge, which presents the problem of remixing and enhancing music for hearing aid users. Our system placed first in the challenge, achieving the best average Hearing-Aid Audio Quality Index (HAAQI) score on the evaluation data set. We describe the system, which uses an ensemble of deep learning music source separators that are fine tuned on the challenge data. We demonstrate the effectiveness of our system through the challenge results and analyze the importance of different system aspects through ablation studies.
eess.AS音频处理
【1】 Dynamic Behaviour of Connectionist Speech Recognition with Strong  Latency Constraints标题：强时延约束下联结主义语音识别的动态行为链接：https://arxiv.org/abs/2401.06588
作者：Giampiero Salvi备注：None摘要：本文介绍了使用连接技术在语音识别的强延迟约束。通过将语音字符串馈送到发音合成器中，从语音信号实时导出合成面部的嘴唇运动的任务施加约束。特别注意已支付给分析之间的相互作用的时间演化模型学习的多层感知器和过渡模型所施加的维特比解码器，在不同的延迟条件。两个实验中，语言模型（LM）的时间依赖性进行了控制的参数。结果表明，所涉及的三个因素，即神经网络拓扑结构，LM和解码器延迟的时间依赖性的长度之间的强烈的相互作用。摘要：This paper describes the use of connectionist techniques in phonetic speech recognition with strong latency constraints. The constraints are imposed by the task of deriving the lip movements of a synthetic face in real time from the speech signal, by feeding the phonetic string into an articulatory synthesiser. Particular attention has been paid to analysing the interaction between the time evolution model learnt by the multi-layer perceptrons and the transition model imposed by the Viterbi decoder, in different latency conditions. Two experiments were conducted in which the time dependencies in the language model (LM) were controlled by a parameter. The results show a strong interaction between the three factors involved, namely the neural network topology, the length of time dependencies in the LM and the decoder latency.
【2】 Contrastive Learning With Audio Discrimination For Customizable Keyword  Spotting In Continuous Speech标题：连续语音中可定制关键词识别的对比学习和音频识别链接：https://arxiv.org/abs/2401.06485
作者：Yu Xi,Baochen Yang,Hao Li,Jiaqi Guo,Kai Yu备注：Accepted by ICASSP2024摘要：连续语音中的可定制关键词识别技术因其潜在的应用前景而受到越来越多的关注。虽然对比学习（CL）已被广泛用于提取关键词表示，以前的CL方法都是对预分割的孤立词进行操作，并且只采用音频-文本表示匹配策略。然而，对于连续语音中的KWS，协同发音和流式分词可以很容易地为不同的文本产生相似的音频模式，这可能因此触发假警报。为了解决这个问题，我们提出了一种新的CL与音频歧视（CLAD）的方法来学习关键字表示与音频文本匹配和音频音频歧视能力。这里，在训练期间，针对每个滑动窗口采用考虑音频-音频和音频-文本CL数据对的InfoNCE损失。对开源LibriPhrase数据集的评估表明，与以前的CL方法相比，使用滑动窗口级别的InfoNCE损失产生了相当的性能。此外，在连续语音数据集LibriSpeech上的实验表明，通过将音频歧视，CLAD实现了显着的性能增益CL没有音频歧视。同时，与两阶段KWS方法相比，采用CLAD的端到端KWS不仅具有更好的性能，而且具有显著的速度提升。摘要：Customizable keyword spotting (KWS) in continuous speech has attracted increasing attention due to its real-world application potential. While contrastive learning (CL) has been widely used to extract keyword representations, previous CL approaches all operate on pre-segmented isolated words and employ only audio-text representations matching strategy. However, for KWS in continuous speech, co-articulation and streaming word segmentation can easily yield similar audio patterns for different texts, which may consequently trigger false alarms. To address this issue, we propose a novel CL with Audio Discrimination (CLAD) approach to learning keyword representation with both audio-text matching and audio-audio discrimination ability. Here, an InfoNCE loss considering both audio-audio and audio-text CL data pairs is employed for each sliding window during training. Evaluations on the open-source LibriPhrase dataset show that the use of sliding-window level InfoNCE loss yields comparable performance compared to previous CL approaches. Furthermore, experiments on the continuous speech dataset LibriSpeech demonstrate that, by incorporating audio discrimination, CLAD achieves significant performance gain over CL without audio discrimination. Meanwhile, compared to two-stage KWS approaches, the end-to-end KWS with CLAD achieves not only better performance, but also significant speed-up.
【3】 Towards High-Quality and Efficient Speech Bandwidth Extension with  Parallel Amplitude and Phase Prediction标题：走向高质量高效的并行幅相预测语音带宽扩展链接：https://arxiv.org/abs/2401.06387
作者：Ye-Xin Lu,Yang Ai,Hui-Peng Du,Zhen-Hua Ling备注：Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：语音带宽扩展（BWE）是指拓宽语音信号的频带范围，使语音质量向更明亮、更饱满的方向发展。提出了一种基于生成对抗网络（GAN）的宽带语音波形生成模型AP-BWE，该模型具有幅度谱和相位谱的并行预测功能。所提出的AP-BWE发生器完全基于卷积神经网络（CNN）。它的特点是具有相互作用的双流结构，其中幅度流和相位流彼此通信，并分别从输入的窄带幅度和相位谱扩展高频分量。为了提高扩展语音信号的自然度，我们在波形级采用多周期滤波器，在谱级分别设计了一对多分辨率幅度和相位鉴别器。实验结果表明，我们提出的AP-BWE实现最先进的性能方面的语音质量BWE任务的目标采样率为16 kHz和48 kHz。在生成效率方面，由于全卷积架构和全帧级操作，拟议的AP-BWE可以在单个RTX 4090 GPU上生成比实时快292.3倍的48 kHz波形样本，在单个CPU上比实时快18.1倍。值得注意的是，据我们所知，AP-BWE是第一个实现高频相位谱直接扩展的方法，这有利于提高现有BWE方法的有效性。摘要：Speech bandwidth extension (BWE) refers to widening the frequency bandwidth range of speech signals, enhancing the speech quality towards brighter and fuller. This paper proposes a generative adversarial network (GAN) based BWE model with parallel prediction of Amplitude and Phase spectra, named AP-BWE, which achieves both high-quality and efficient wideband speech waveform generation. The proposed AP-BWE generator is entirely based on convolutional neural networks (CNNs). It features a dual-stream architecture with mutual interaction, where the amplitude stream and the phase stream communicate with each other and respectively extend the high-frequency components from the input narrowband amplitude and phase spectra. To improve the naturalness of the extended speech signals, we employ a multi-period discriminator at the waveform level and design a pair of multi-resolution amplitude and phase discriminators at the spectral level, respectively. Experimental results demonstrate that our proposed AP-BWE achieves state-of-the-art performance in terms of speech quality for BWE tasks targeting sampling rates of both 16 kHz and 48 kHz. In terms of generation efficiency, due to the all-convolutional architecture and all-frame-level operations, the proposed AP-BWE can generate 48 kHz waveform samples 292.3 times faster than real-time on a single RTX 4090 GPU and 18.1 times faster than real-time on a single CPU. Notably, to our knowledge, AP-BWE is the first to achieve the direct extension of the high-frequency phase spectrum, which is beneficial for improving the effectiveness of existing BWE methods.
【4】 Remixing Music for Hearing Aids Using Ensemble of Fine-Tuned Source  Separators标题：利用微调声源分离器组合的助听器混音链接：https://arxiv.org/abs/2401.06203
作者：Matthew Daly备注：2 pages, ICASSP 2024, Cadenza Grand Challenge摘要：本文介绍了我们为Cadenza ICASSP 2024 Grand Challenge提交的系统，该挑战提出了为助听器用户重新混合和增强音乐的问题。我们的系统在挑战中排名第一，在评估数据集上获得了最佳的助听器音频质量指数（HAAQI）平均得分。我们描述了这个系统，它使用了一系列深度学习音乐源分离器，这些分离器在挑战数据上进行了微调。我们通过挑战结果证明了我们系统的有效性，并通过消融研究分析了不同系统方面的重要性。摘要：This paper introduces our system submission for the Cadenza ICASSP 2024 Grand Challenge, which presents the problem of remixing and enhancing music for hearing aid users. Our system placed first in the challenge, achieving the best average Hearing-Aid Audio Quality Index (HAAQI) score on the evaluation data set. We describe the system, which uses an ensemble of deep learning music source separators that are fine tuned on the challenge data. We demonstrate the effectiveness of our system through the challenge results and analyze the importance of different system aspects through ablation studies.
【5】 End to end Hindi to English speech conversion using Bark, mBART and a  finetuned XLSR Wav2Vec2标题：使用Bark、mBART和精调的XLSR Wav2Vec2进行端到端的印地语到英语语音转换链接：https://arxiv.org/abs/2401.06183
作者：Aniket Tathe,Anand Kamble,Suyash Kumbharkar,Atharva Bhandare,Anirban C. Mitra摘要：长期以来，语言一直是有效沟通和联系的障碍，在我们日益相互联系的世界中，这是一个持续存在的挑战。本文介绍了一个变革性的解决方案，这个持久的障碍，一个端到端的语音转换框架，为印地语到英语翻译量身定制，最终在英语音频的合成。通过集成诸如用于自动语音识别（ASR）的XLSR Wav2Vec2、用于神经机器翻译（NMT）的mBART以及文本到语音（TTS）合成组件等尖端技术，该框架为跨语言通信提供了统一且无缝的方法。我们深入研究了每个组件的复杂细节，阐明了它们各自的贡献，并探索了从印地语口语到合成英语音频的流畅过渡的协同作用。摘要：Speech has long been a barrier to effective communication and connection, persisting as a challenge in our increasingly interconnected world. This research paper introduces a transformative solution to this persistent obstacle an end-to-end speech conversion framework tailored for Hindi-to-English translation, culminating in the synthesis of English audio. By integrating cutting-edge technologies such as XLSR Wav2Vec2 for automatic speech recognition (ASR), mBART for neural machine translation (NMT), and a Text-to-Speech (TTS) synthesis component, this framework offers a unified and seamless approach to cross-lingual communication. We delve into the intricate details of each component, elucidating their individual contributions and exploring the synergies that enable a fluid transition from spoken Hindi to synthesized English audio.
【6】 LCB-net: Long-Context Biasing for Audio-Visual Speech Recognition标题：LCB-Net：视听语音识别中的长语境偏向链接：https://arxiv.org/abs/2401.06390
作者：Fan Yu,Haoxu Wang,Xian Shi,Shiliang Zhang备注：Accepted by ICASPP 2024摘要：在线会议和课程的日益普及对利用视频幻灯片中丰富的文本信息来改进自动语音识别（ASR）提出了新的挑战。与罕见短语列表相比，视频中的幻灯片与演讲实时同步，从而能够提取长上下文偏差。因此，我们提出了一种新的长上下文偏置网络（LCB-net）的视听语音识别（AVSR），有效地利用视频中的长上下文信息。具体来说，我们采用双编码器架构，同时模拟音频和长上下文偏置。此外，我们还提出了一个偏置预测模块，利用二进制交叉熵（BCE）损失显式确定偏置短语的长上下文偏置。此外，我们引入了一个动态的上下文短语模拟，以提高我们的LCB网络的泛化和鲁棒性。实验结果表明，LCB-net在测试集上的相对WER/U-WER/B-WER约减率分别为9.4%/9.1%/10.9%，优于一般ASR模型，具有较高的无偏和有偏性能.此外，我们还评估了我们的模型在LibriSpeech语料库，导致23.8%/19.2%/35.4%的相对WER/U-WER/B-WER减少ASR模型。摘要：The growing prevalence of online conferences and courses presents a new challenge in improving automatic speech recognition (ASR) with enriched textual information from video slides. In contrast to rare phrase lists, the slides within videos are synchronized in real-time with the speech, enabling the extraction of long contextual bias. Therefore, we propose a novel long-context biasing network (LCB-net) for audio-visual speech recognition (AVSR) to leverage the long-context information available in videos effectively. Specifically, we adopt a bi-encoder architecture to simultaneously model audio and long-context biasing. Besides, we also propose a biasing prediction module that utilizes binary cross entropy (BCE) loss to explicitly determine biased phrases in the long-context biasing. Furthermore, we introduce a dynamic contextual phrases simulation to enhance the generalization and robustness of our LCB-net. Experiments on the SlideSpeech, a large-scale audio-visual corpus enriched with slides, reveal that our proposed LCB-net outperforms general ASR model by 9.4%/9.1%/10.9% relative WER/U-WER/B-WER reduction on test set, which enjoys high unbiased and biased performance. Moreover, we also evaluate our model on LibriSpeech corpus, leading to 23.8%/19.2%/35.4% relative WER/U-WER/B-WER reduction over the ASR model.
机器翻译由腾讯交互翻译提供，仅供参考