今日论文合集：cs.SD语音6篇，eess.AS音频处理7篇。
【1】 Audiovisual Masked Autoencoders
标题：视听屏蔽式自动编码器
链接：https://arxiv.org/abs/2212.05922
作者：Mariana-Iuliana Georgescu,Eduardo Fonseca,Radu Tudor Ionescu,Mario Lucic,Cordelia Schmid,Anurag Arnab机构：Google Research, University of Bucharest摘要：Can we leverage the audiovisual information already present in video to improve self-supervised representation learning? To answer this question, we study various pretraining architectures and objectives within the masked autoencoding framework, motivated by the success of similar methods in natural language and image understanding. We show that we can achieve significant improvements on audiovisual downstream classification tasks, surpassing the state-of-the-art on VGGSound and AudioSet. Furthermore, we can leverage our audiovisual pretraining scheme for multiple unimodal downstream tasks using a single audiovisual pretrained model. We additionally demonstrate the transferability of our representations, achieving state-of-the-art audiovisual results on Epic Kitchens without pretraining specifically for this dataset.
【2】 Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features
标题：利用瓶颈特征实现无文本标注的直接语音转换
链接：https://arxiv.org/abs/2212.05805
作者：Junhui Zhang,Junjie Pan,Xiang Yin,Zejun Ma摘要：语-语翻译是一种在不同语言之间直接将一个语音话语翻译成另一个语音话语的翻译技术，在同声传译等任务中具有很大的潜力。现有技术的模型通常包含用于音素序列预测的辅助模块，并且这需要训练数据集的文本注释。提出了一种不需要任何文本标注或内容信息的直接语音到语音翻译模型。该模型不引入辅助音素预测任务，而是使用瓶颈特征作为中间训练目标，以保证系统的翻译性能。在普通话-粤语语音翻译系统上的实验结果表明，该方法是可行的，在翻译和合成质量方面可以与级联系统相媲美。摘要：Speech-to-speech translation directly translates a speech utterance to another between different languages, and has great potential in tasks such as simultaneous interpretation. State-of-art models usually contains an auxiliary module for phoneme sequences prediction, and this requires textual annotation of the training dataset. We propose a direct speech-to-speech translation model which can be trained without any textual annotation or content information. Instead of introducing an auxiliary phoneme prediction task in the model, we propose to use bottleneck features as intermediate training objectives for our model to ensure the translation performance of the system. Experiments on Mandarin-Cantonese speech translation demonstrate the feasibility of the proposed approach and the performance can match a cascaded system with respect of translation and synthesis qualities.
【3】 A Comparison of Audio Preprocessing Techniques and Deep Learning Algorithms for Raga Recognition
标题：用于RAGA识别的音频预处理技术和深度学习算法的比较
链接：https://arxiv.org/abs/2212.05335
作者：Devayani Hebbar,Vandana Jagtap机构：MIT World Peace, University, .备注：7 pages, 6 figures, 7 tables摘要：拉嘎构成印度古典音乐的基础。Raga Recognition的任务最近在音乐信息检索社区获得了牵引力，这可以归因于印度古典音乐的细微差别，这导致了计算领域的大量研究问题。在这项工作中，我们使用了两种不同的数字音频信号处理技术来预处理Carnatic经典Raga的音频样本，然后通过各种深度学习模型进行处理。比较他们的结果以推断哪种DASP技术更适合于raga识别任务。我们获得了最先进的结果，我们的最佳模型达到了98.1%的测试准确度。我们还比较了每个模型区分相似Raga的能力。摘要：Ragas form the foundation for Indian Classical Music. The task of Raga Recognition has gained traction in the Music Information Retrieval community in the recent past, which can be attributed to the nuances of Indian Classical Music that have resulted in a plethora of research problems in Computing. In this work, we used two different digital audio signal processing techniques to preprocess audio samples of Carnatic classical ragas that were then processed by various Deep Learning models. Their results were compared in order to infer which DASP technique is better suited to the task of raga recognition. We obtained state of the art results, with our best model reaching a testing accuracy of 98.1%. We also compared each model ability to distinguish between similar ragas.
【4】 Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning
标题：通过强化学习利用特定于通道的表示进行视听语音识别
链接：https://arxiv.org/abs/2212.05301
作者：Chen Chen,Yuchen Hu,Qiang Zhang,Heqing Zou,Beier Zhu,Eng Siong Chng机构：School of Computer Science and Engineering, Nanyang Technological University, ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, China, College of Computer Science and Technology, Zhejiang University摘要：视听语音识别（AVSR）在改善语音识别的抗噪性方面取得了显著的成功。主流方法集中于融合音频和视频输入以获得模态不变表示。然而，这样的表示倾向于过度依赖音频模态，因为在干净的条件下，音频模态比视频模态更容易识别。结果，AVSR模型在面对噪声污染时低估了视觉流的重要性。为此，我们利用视觉模态特定表示来为AVSR任务提供稳定的补充信息。具体地说，我们提出了一个基于强化学习（RL）的框架MSRL，其中智能体在自回归解码过程中动态协调模态不变和模态特定的表示。我们自定义与特定任务指标直接相关的奖励函数（即：字错误率），这促使MSRL有效地探索最佳整合策略。在LRS3数据集上的实验结果表明，该方法在干净和各种噪声环境下都能取得较好的分割效果.此外，当测试集含有未知噪声时，MSRL系统的通用性优于其他基线.摘要：Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant representations. However, such representations are prone to over-reliance on audio modality as it is much easier to recognize than video modality in clean conditions. As a result, the AVSR model underestimates the importance of visual stream in face of noise corruption. To this end, we leverage visual modality-specific representations to provide stable complementary information for the AVSR task. Specifically, we propose a reinforcement learning (RL) based framework called MSRL, where the agent dynamically harmonizes modality-invariant and modality-specific representations in the auto-regressive decoding process. We customize a reward function directly related to task-specific metrics (i.e., word error rate), which encourages the MSRL to effectively explore the optimal integration strategy. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions. Furthermore, we demonstrate the better generality of MSRL system than other baselines when test set contains unseen noises.
【5】 Variational Speech Waveform Compression to Catalyze Semantic Communications
标题：用于催化语义通信的变分语音波形压缩
链接：https://arxiv.org/abs/2212.05294
作者：Shengshi Yao,Zixuan Xiao,Sixian Wang,Jincheng Dai,Ping Zhang机构：∗Beijing University of Posts and Telecommunications, Beijing, China, †Peng Cheng Laboratory, Shenzhen, China摘要：提出了一种新的神经波形压缩方法来催化新兴的语音语义通信。通过引入非线性变换和变分建模，有效地捕捉了语音帧间的相关性，更准确地估计了语音特征的概率分布，从而提高了压缩性能。特别地，通过一对非线性变换来分析和合成语音信号，从而产生潜在特征。该方法首先建立超先验信息熵模型，获取潜在特征的概率分布，然后进行量化和熵编码。所提出的波形编解码器可以灵活地针对任意速率进行优化，并且另一个吸引人的特征是它可以针对任何可微分损失函数容易地进行优化，包括用于语义通信的感知损失。为了进一步提高保真度，我们引入残差编码来减轻潜在空间处量化失真引起的退化。实验结果表明，在达到相同性能的情况下，该方法比广泛使用的自适应多速率宽带（AMR-WB）编解码器和新兴的神经波形编码方法节省了高达27%的码率。摘要：We propose a novel neural waveform compression method to catalyze emerging speech semantic communications. By introducing nonlinear transform and variational modeling, we effectively capture the dependencies within speech frames and estimate the probabilistic distribution of the speech feature more accurately, giving rise to better compression performance. In particular, the speech signals are analyzed and synthesized by a pair of nonlinear transforms, yielding latent features. An entropy model with hyperprior is built to capture the probabilistic distribution of latent features, followed with quantization and entropy coding. The proposed waveform codec can be optimized flexibly towards arbitrary rate, and the other appealing feature is that it can be easily optimized for any differentiable loss function, including perceptual loss used in semantic communications. To further improve the fidelity, we incorporate residual coding to mitigate the degradation arising from quantization distortion at the latent space. Results indicate that achieving the same performance, the proposed method saves up to 27% coding rate than widely used adaptive multi-rate wideband (AMR-WB) codec as well as emerging neural waveform coding methods.
【6】 GPU-accelerated Guided Source Separation for Meeting Transcription
标题：用于会议转录的GPU加速引导源分离
链接：https://arxiv.org/abs/2212.05271
作者：Desh Raj,Daniel Povey,Sanjeev Khudanpur机构：CLSP & ,HLTCOE, Johns Hopkins University, Baltimore, USA; ,Xiaomi Corp., Beijing, China备注：7 pages, 4 figures. Code available at this https URL摘要：引导源分离（Guided Source Separation，GSS）是一种目标说话人提取方法，它依靠预先计算的说话人活动和盲源分离对重叠语音信号进行前端增强。它是在CHiME-5挑战赛期间首次提出的，与延迟求和波束成形基线相比有了显著改进。然而，尽管该方法具有优势，但主要由于其高计算时间，该方法在满足转录基准方面的采用有限。在本文中，我们介绍了GSS的改进实施，它利用了基于GPU的现代流水线的强大功能，包括频率和段的批处理，以提供比基于CPU的推理快300倍的速度。改进的推断时间允许我们对GSS算法的几个参数执行详细的消融研究--例如上下文持续时间、通道数量和噪声类别，仅举几例。我们提供端到端的可复制管道，用于常见会议基准的发言人属性转录：LibriCSS、AMI和阿里会议。我们的代码和配方是公开的：https://github.com/desh2608/gss。摘要：Guided source separation (GSS) is a type of target-speaker extraction method that relies on pre-computed speaker activities and blind source separation to perform front-end enhancement of overlapped speech signals. It was first proposed during the CHiME-5 challenge and provided significant improvements over the delay-and-sum beamforming baseline. Despite its strengths, however, the method has seen limited adoption for meeting transcription benchmarks primarily due to its high computation time. In this paper, we describe our improved implementation of GSS that leverages the power of modern GPU-based pipelines, including batched processing of frequencies and segments, to provide 300x speed-up over CPU-based inference. The improved inference time allows us to perform detailed ablation studies over several parameters of the GSS algorithm -- such as context duration, number of channels, and noise class, to name a few. We provide end-to-end reproducible pipelines for speaker-attributed transcription of popular meeting benchmarks: LibriCSS, AMI, and AliMeeting. Our code and recipes are publicly available: https://github.com/desh2608/gss.
【1】 Non-parallel Accent Conversion using Pseudo Siamese Disentanglement Network
标题：基于伪暹罗解缠网络的非并行重音转换
链接：https://arxiv.org/abs/2212.05751
作者：Dongya Jia,Qiao Tian,Jiaxin Li,Yuanzhe Chen,Kainan Peng,Mingbo Ma,Yuping Wang,Yuxuan Wang机构：Speech, Audio & Music Intelligence (SAMI), ByteDance摘要：重音转换（AC）的主要目标是在保持语音内容和音质的同时将语音的重音转换为目标重音。已有的基于参考的方法在推理阶段依赖于参考话语，这限制了其实际应用。此外，以往的无参考方法在训练阶段大多需要并行数据。本文从特征解纠缠的角度出发，提出了一种基于非平行数据的无参考方法。提出了一种伪连体解纠缠网络（PSDN），用于从内容表示中解纠缠重音信息，并对目标重音进行建模。此外，针对无目标重音数据的说话人，提出了一种音色增强方法，以提高说话人的音色保持能力。实验结果表明，该系统能够将美国英语语音的重音转换为印度口音，重音度（3.47）高于基线（2.75）和输入（1.19）。转换后的语音的自然度也与输入的语音的自然度相当。摘要：The main goal of accent conversion (AC) is to convert the accent of speech into the target accent while preserving the content and timbre. Previous reference-based methods rely on reference utterances in the inference phase, which limits their practical application. What's more, previous reference-free methods mostly require parallel data in the training phase. In this paper, we propose a reference-free method based on non-parallel data from the perspective of feature disentanglement. Pseudo Siamese Disentanglement Network (PSDN) is proposed to disentangle the accent information from the content representation and model the target accent. Besides, a timbre augmentation method is proposed to enhance the ability of timbre retaining for speakers without target-accent data. Experimental results show that the proposed system can convert the accent of native American English speech into Indian accent with higher accentedness (3.47) than the baseline (2.75) and input (1.19). The naturalness of converted speech is also comparable to that of the input.
【2】 Synthetic Wave-Geometric Impulse Responses for Improved Speech Dereverberation
标题：用于改进语音去混响的合成波形几何脉冲响应
链接：https://arxiv.org/abs/2212.05360
作者：Rohith Aralikatti,Zhenyu Tang,Dinesh Manocha机构：University of Maryland, College Park备注：Submitted to ICASSP 2023摘要：本文提出了一种新的方法来提高基于学习的语音去混响的性能使用准确的合成数据集。我们的方法被设计为从混响语音信号中恢复无混响信号。我们表明，准确模拟房间脉冲响应（RIR）的低频分量对实现良好的去混响非常重要。我们使用的GWA数据集包含以混合方式生成的合成RIR：使用精确的基于波的解算器来模拟较低的频率，并且几何射线跟踪方法模拟较高的频率。在四个真实RIR数据集上，我们证明了在混合合成RIR上训练的语音去混响模型优于在由先验几何射线跟踪方法生成的RIR上训练的模型。摘要：We present a novel approach to improve the performance of learning-based speech dereverberation using accurate synthetic datasets. Our approach is designed to recover the reverb-free signal from a reverberant speech signal. We show that accurately simulating the low-frequency components of Room Impulse Responses (RIRs) is important to achieving good dereverberation. We use the GWA dataset that consists of synthetic RIRs generated in a hybrid fashion: an accurate wave-based solver is used to simulate the lower frequencies and geometric ray tracing methods simulate the higher frequencies. We demonstrate that speech dereverberation models trained on hybrid synthetic RIRs outperform models trained on RIRs generated by prior geometric ray tracing methods on four real-world RIR datasets.
【3】 GPU-accelerated Guided Source Separation for Meeting Transcription
标题：用于会议转录的GPU加速引导源分离
链接：https://arxiv.org/abs/2212.05271
*与cs.SD语音【6】为同一篇
作者：Desh Raj,Daniel Povey,Sanjeev Khudanpur机构：CLSP & ,HLTCOE, Johns Hopkins University, Baltimore, USA; ,Xiaomi Corp., Beijing, China备注：7 pages, 4 figures. Code available at this https URL摘要：引导源分离（Guided Source Separation，GSS）是一种目标说话人提取方法，它依靠预先计算的说话人活动和盲源分离对重叠语音信号进行前端增强。它是在CHiME-5挑战赛期间首次提出的，与延迟求和波束成形基线相比有了显著改进。然而，尽管该方法具有优势，但主要由于其高计算时间，该方法在满足转录基准方面的采用有限。在本文中，我们介绍了GSS的改进实施，它利用了基于GPU的现代流水线的强大功能，包括频率和段的批处理，以提供比基于CPU的推理快300倍的速度。改进的推断时间允许我们对GSS算法的几个参数执行详细的消融研究--例如上下文持续时间、通道数量和噪声类别，仅举几例。我们提供端到端的可复制管道，用于常见会议基准的发言人属性转录：LibriCSS、AMI和阿里会议。我们的代码和配方是公开的：https://github.com/desh2608/gss。摘要：Guided source separation (GSS) is a type of target-speaker extraction method that relies on pre-computed speaker activities and blind source separation to perform front-end enhancement of overlapped speech signals. It was first proposed during the CHiME-5 challenge and provided significant improvements over the delay-and-sum beamforming baseline. Despite its strengths, however, the method has seen limited adoption for meeting transcription benchmarks primarily due to its high computation time. In this paper, we describe our improved implementation of GSS that leverages the power of modern GPU-based pipelines, including batched processing of frequencies and segments, to provide 300x speed-up over CPU-based inference. The improved inference time allows us to perform detailed ablation studies over several parameters of the GSS algorithm -- such as context duration, number of channels, and noise class, to name a few. We provide end-to-end reproducible pipelines for speaker-attributed transcription of popular meeting benchmarks: LibriCSS, AMI, and AliMeeting. Our code and recipes are publicly available: https://github.com/desh2608/gss.
【4】 Direct Speech-to-speech Translation without Textual Annotation using Bottleneck Features
标题：利用瓶颈特征实现无文本标注的直接语音转换
链接：https://arxiv.org/abs/2212.05805
*与cs.SD语音【2】为同一篇
作者：Junhui Zhang,Junjie Pan,Xiang Yin,Zejun Ma摘要：语-语翻译是一种在不同语言之间直接将一个语音话语翻译成另一个语音话语的翻译技术，在同声传译等任务中具有很大的潜力。现有技术的模型通常包含用于音素序列预测的辅助模块，并且这需要训练数据集的文本注释。提出了一种不需要任何文本标注或内容信息的直接语音到语音翻译模型。该模型不引入辅助音素预测任务，而是使用瓶颈特征作为中间训练目标，以保证系统的翻译性能。在普通话-粤语语音翻译系统上的实验结果表明，该方法是可行的，在翻译和合成质量方面可以与级联系统相媲美。摘要：Speech-to-speech translation directly translates a speech utterance to another between different languages, and has great potential in tasks such as simultaneous interpretation. State-of-art models usually contains an auxiliary module for phoneme sequences prediction, and this requires textual annotation of the training dataset. We propose a direct speech-to-speech translation model which can be trained without any textual annotation or content information. Instead of introducing an auxiliary phoneme prediction task in the model, we propose to use bottleneck features as intermediate training objectives for our model to ensure the translation performance of the system. Experiments on Mandarin-Cantonese speech translation demonstrate the feasibility of the proposed approach and the performance can match a cascaded system with respect of translation and synthesis qualities.
【5】 A Comparison of Audio Preprocessing Techniques and Deep Learning Algorithms for Raga Recognition
标题：用于RAGA识别的音频预处理技术和深度学习算法的比较
链接：https://arxiv.org/abs/2212.05335
*与cs.SD语音【3】为同一篇
作者：Devayani Hebbar,Vandana Jagtap机构：MIT World Peace, University, .备注：7 pages, 6 figures, 7 tables摘要：拉嘎构成印度古典音乐的基础。Raga Recognition的任务最近在音乐信息检索社区中获得了牵引力，这可以归因于印度古典音乐的细微差别，这导致了计算中过多的研究问题。在这项工作中，我们使用了两种不同的数字音频信号处理技术来预处理Carnatic经典Raga的音频样本，然后通过各种深度学习模型进行处理。比较他们的结果以推断哪种DASP技术更适合于raga识别任务。我们获得了最先进的结果，我们的最佳模型达到了98.1%的测试准确度。我们还比较了每个模型区分相似Raga的能力。摘要：Ragas form the foundation for Indian Classical Music. The task of Raga Recognition has gained traction in the Music Information Retrieval community in the recent past, which can be attributed to the nuances of Indian Classical Music that have resulted in a plethora of research problems in Computing. In this work, we used two different digital audio signal processing techniques to preprocess audio samples of Carnatic classical ragas that were then processed by various Deep Learning models. Their results were compared in order to infer which DASP technique is better suited to the task of raga recognition. We obtained state of the art results, with our best model reaching a testing accuracy of 98.1%. We also compared each model ability to distinguish between similar ragas.
【6】 Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning
标题：通过强化学习利用特定于通道的表示进行视听语音识别
链接：https://arxiv.org/abs/2212.05301
*与cs.SD语音【4】为同一篇
作者：Chen Chen,Yuchen Hu,Qiang Zhang,Heqing Zou,Beier Zhu,Eng Siong Chng机构：School of Computer Science and Engineering, Nanyang Technological University, ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou, China, College of Computer Science and Technology, Zhejiang University摘要：视听语音识别（AVSR）在改善语音识别的抗噪性方面取得了显著的成功。主流方法集中于融合音频和视频输入以获得模态不变表示。然而，这样的表示倾向于过度依赖音频模态，因为在干净的条件下，音频模态比视频模态更容易识别。结果，AVSR模型在面对噪声污染时低估了视觉流的重要性。为此，我们利用视觉模态特定表示来为AVSR任务提供稳定的补充信息。具体地说，我们提出了一个基于强化学习（RL）的框架MSRL，其中智能体在自回归解码过程中动态协调模态不变和模态特定的表示。我们自定义与特定任务指标直接相关的奖励函数（即：字错误率），这促使MSRL有效地探索最佳整合策略。在LRS3数据集上的实验结果表明，该方法在干净和各种噪声环境下都能取得较好的分割效果.此外，当测试集含有未知噪声时，MSRL系统的通用性优于其他基线.摘要：Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant representations. However, such representations are prone to over-reliance on audio modality as it is much easier to recognize than video modality in clean conditions. As a result, the AVSR model underestimates the importance of visual stream in face of noise corruption. To this end, we leverage visual modality-specific representations to provide stable complementary information for the AVSR task. Specifically, we propose a reinforcement learning (RL) based framework called MSRL, where the agent dynamically harmonizes modality-invariant and modality-specific representations in the auto-regressive decoding process. We customize a reward function directly related to task-specific metrics (i.e., word error rate), which encourages the MSRL to effectively explore the optimal integration strategy. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions. Furthermore, we demonstrate the better generality of MSRL system than other baselines when test set contains unseen noises.
【7】 Variational Speech Waveform Compression to Catalyze Semantic Communications
标题：用于催化语义通信的变分语音波形压缩
链接：https://arxiv.org/abs/2212.05294
*与cs.SD语音【5】为同一篇
作者：Shengshi Yao,Zixuan Xiao,Sixian Wang,Jincheng Dai,Ping Zhang机构：∗Beijing University of Posts and Telecommunications, Beijing, China, †Peng Cheng Laboratory, Shenzhen, China摘要：提出了一种新的神经波形压缩方法来催化新兴的语音语义通信。通过引入非线性变换和变分建模，有效地捕捉了语音帧间的相关性，更准确地估计了语音特征的概率分布，从而提高了压缩性能。特别地，通过一对非线性变换来分析和合成语音信号，从而产生潜在特征。该方法首先建立超先验信息熵模型，获取潜在特征的概率分布，然后进行量化和熵编码。所提出的波形编解码器可以灵活地针对任意速率进行优化，并且另一个吸引人的特征是它可以针对任何可微分损失函数容易地进行优化，包括用于语义通信的感知损失。为了进一步提高保真度，我们引入残差编码来减轻潜在空间处量化失真引起的退化。实验结果表明，在达到相同性能的情况下，该方法比广泛使用的自适应多速率宽带（AMR-WB）编解码器和新兴的神经波形编码方法节省了高达27%的码率。摘要：We propose a novel neural waveform compression method to catalyze emerging speech semantic communications. By introducing nonlinear transform and variational modeling, we effectively capture the dependencies within speech frames and estimate the probabilistic distribution of the speech feature more accurately, giving rise to better compression performance. In particular, the speech signals are analyzed and synthesized by a pair of nonlinear transforms, yielding latent features. An entropy model with hyperprior is built to capture the probabilistic distribution of latent features, followed with quantization and entropy coding. The proposed waveform codec can be optimized flexibly towards arbitrary rate, and the other appealing feature is that it can be easily optimized for any differentiable loss function, including perceptual loss used in semantic communications. To further improve the fidelity, we incorporate residual coding to mitigate the degradation arising from quantization distortion at the latent space. Results indicate that achieving the same performance, the proposed method saves up to 27% coding rate than widely used adaptive multi-rate wideband (AMR-WB) codec as well as emerging neural waveform coding methods.
机器翻译，仅供参考
每日语音速递