今日论文合集：cs.SD语音17篇，eess.AS音频处理19篇。【1】 Towards Measuring and Scoring Speaker Diarization Fairness
标题：关于说话人二元化公平性的测量和评分
链接：https://arxiv.org/abs/2302.09991
作者：Yannis Tevissen,Jérôme Boudy,Gérard Chollet,Frédéric Petitpont机构：SAMOVAR, Telecom SudParis, Institut Polytechnique de Paris, France, Newsbridge, France摘要：说话人日记化，或者说寻找“谁在说话，什么时候说话”的任务，现在几乎在每一个语音处理应用中都有使用。然而，由于没有协议逐一研究其偏倚，其公平性尚未得到评估。本文提出了一个协议和一个评分方法来评估说话人对话的公平性。该协议被应用于一个大的口语数据集，并根据说话人的性别、年龄、口音和口语句子的长度报告说话人日记化的性能。当我们应用最先进的说话人日记化方法时，一些由性别或说话人口音引起的偏见被识别出来。摘要：Speaker diarization, or the task of finding "who spoke and when", is now used in almost every speech processing application. Nevertheless, its fairness has not yet been evaluated because there was no protocol to study its biases one by one. In this paper we propose a protocol and a scoring method designed to evaluate speaker diarization fairness. This protocol is applied on a large dataset of spoken utterances and report the performances of speaker diarization depending on the gender, the age, the accent of the speaker and the length of the spoken sentence. Some biases induced by the gender, or the accent of the speaker were identified when we applied a state-of-the-art speaker diarization method.【2】 A Sidecar Separator Can Convert a Single-Speaker Speech Recognition System to a Multi-Speaker One
标题：Sidecar分离器可以将单说话人语音识别系统转换为多说话人语音识别系统
链接：https://arxiv.org/abs/2302.09908
作者：Lingwei Meng,Jiawen Kang,Mingyu Cui,Yuejiao Wang,Xixin Wu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023摘要：虽然自动语音识别（ASR）可以在常见的非重叠环境中很好地执行，但是在多说话人重叠语音识别中保持性能仍然是具有挑战性的。最近的研究表明，ASR模型的编码器在不同的层捕获不同层次的信息--较低的层往往具有更多的声学信息，而较高的层具有更多的语言信息。这启发了我们开发一种侧车分离器，通过分离嵌入在两个合适层之间的混合语音，使训练良好的ASR模型能够用于多说话人场景。我们使用基于wav 2 vec 2.0的ASR模型进行了试验，该模型安装了一个侧车。通过冻结原始模型的参数并且仅训练Sidecar（8.7M，所有参数的8.4%），所提出的方法对于2说话人混合的LibriMix数据集以较大的幅度优于先前的现有技术，达到10.36%的词错误率（WER）;并且在有限训练时获得了与LibriSpeechMix数据集相当的结果（7.56%）。摘要：Although automatic speech recognition (ASR) can perform well in common non-overlapping environments, sustaining performance in multi-speaker overlapping speech recognition remains challenging. Recent research revealed that ASR model's encoder captures different levels of information with different layers -- the lower layers tend to have more acoustic information, and the upper layers more linguistic. This inspires us to develop a Sidecar separator to empower a well-trained ASR model for multi-speaker scenarios by separating the mixed speech embedding between two suitable layers. We experimented with a wav2vec 2.0-based ASR model with a Sidecar mounted. By freezing the parameters of the original model and training only the Sidecar (8.7 M, 8.4% of all parameters), the proposed approach outperforms the previous state-of-the-art by a large margin for the 2-speaker mixed LibriMix dataset, reaching a word error rate (WER) of 10.36%; and obtains comparable results (7.56%) for LibriSpeechMix dataset when limited training.【3】 Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition
标题：基于知识感知的贝叶斯共注意多通道情绪识别
链接：https://arxiv.org/abs/2302.09856
作者：Zihan Zhao,Yu Wang,Yanfeng Wang机构：Cooperative Medianet Innovation Center, Shanghai Jiao Tong University ,Shanghai AI Laboratory备注：Accepted to IEEE ICASSP 2023摘要：多模态情感识别是一个具有挑战性的研究领域，旨在融合不同的模态来预测人类的情感。然而，大多数基于注意机制的现有模型难以独立地学习情绪相关部分。为了解决这个问题，我们提出将外部情绪相关知识加入到基于共同注意的预训练模型融合中。为了有效地结合这些知识，我们用贝叶斯注意模块（BAM）来增强共同注意模型，其中使用情绪相关知识来估计先验分布。在IEMOCAP数据集上的实验结果表明，该方法的未加权准确率（unweighted accuracy，UA）优于几种现有方法至少0.7%。摘要：Multimodal emotion recognition is a challenging research area that aims to fuse different modalities to predict human emotion. However, most existing models that are based on attention mechanisms have difficulty in learning emotionally relevant parts on their own. To solve this problem, we propose to incorporate external emotion-related knowledge in the co-attention based fusion of pre-trained models. To effectively incorporate this knowledge, we enhance the co-attention model with a Bayesian attention module (BAM) where a prior distribution is estimated using the emotion-related knowledge. Experimental results on the IEMOCAP dataset show that the proposed approach can outperform several state-of-the-art approaches by at least 0.7% unweighted accuracy (UA).【4】 Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
标题：强调未见词：端到端语音识别的新词汇习得
链接：https://arxiv.org/abs/2302.09723
作者：Leyuan Qu,Cornelius Weber,Stefan Wermter机构：Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany, Department of Artificial Intelligence, Zhejiang Lab, Hangzhou, China备注：Accepted by Neural Networks摘要：由于人类语言的动态性，自动语音识别系统需要不断地获取新词汇。词汇表外（OOV）单词，诸如趋势单词和新命名实体，给现代ASR系统带来了问题，现代ASR系统需要长的训练时间来适应它们的大量参数。与以往大多数研究侧重于语言模型后处理不同，本文从语言模型后处理的角度出发，消除声学建模中的偏差，从声学角度识别OOV词汇。我们建议使用文语转换系统来产生OOV字词，并重新调整损失以鼓励神经网路更多地注意OOV字词。具体地，当在合成音频上微调先前训练的模型时，我们扩大了用于训练包含OOV词的话语的神经网络参数的分类损失（句子级），或者重新缩放用于OOV词的反向传播的梯度（词级）。为了克服灾难性遗忘，我们还探索了损失重标度和模型正则化的组合，即L2正则化和弹性权重固结（EWC）。在LibriSpeech基准测试上的实验结果表明，与以往仅使用EWC对合成音频进行微调的方法相比，本文提出的损失重标方法能够显著提高召回率，而单词错误率只有轻微的下降。此外，词级重标度比话语级重标度更稳定，在OOV词识别中具有更高的召回率和准确率。此外，我们提出的组合损失重标度和权重合并方法可以支持ASR系统的持续学习。摘要：Due to the dynamic nature of human language, automatic speech recognition (ASR) systems need to continuously acquire new vocabulary. Out-Of-Vocabulary (OOV) words, such as trending words and new named entities, pose problems to modern ASR systems that require long training times to adapt their large numbers of parameters. Different from most previous research focusing on language model post-processing, we tackle this problem on an earlier processing level and eliminate the bias in acoustic modeling to recognize OOV words acoustically. We propose to generate OOV words using text-to-speech systems and to rescale losses to encourage neural networks to pay more attention to OOV words. Specifically, we enlarge the classification loss used for training neural networks' parameters of utterances containing OOV words (sentence-level), or rescale the gradient used for back-propagation for OOV words (word-level), when fine-tuning a previously trained model on synthetic audio. To overcome catastrophic forgetting, we also explore the combination of loss rescaling and model regularization, i.e. L2 regularization and elastic weight consolidation (EWC). Compared with previous methods that just fine-tune synthetic audio with EWC, the experimental results on the LibriSpeech benchmark reveal that our proposed loss rescaling approach can achieve significant improvement on the recall rate with only a slight decrease on word error rate. Moreover, word-level rescaling is more stable than utterance-level rescaling and leads to higher recall rates and precision on OOV word recognition. Furthermore, our proposed combined loss rescaling and weight consolidation methods can support continual learning of an ASR system.【5】 SSVMR: Saliency-based Self-training for Video-Music Retrieval
标题：SSVMR：基于显著度的视频音乐检索自我训练
链接：https://arxiv.org/abs/2302.09328
作者：Xuxin Cheng,Zhihong Zhu,Hongxiang Li,Yaowei Li,Yuexian Zou机构：ADSPLAB, School of ECE, Peking University备注：Accepted by ICASSP 2023摘要：随着短视频的兴起，为视频选择合适的背景音乐（BGM）的需求显著增加，视频-音乐检索（VMR）任务逐渐受到研究界的关注。与其他跨模态学习任务一样，现有的VMR方法通常试图在特征空间中度量视频和音乐之间的相似性。然而，它们（1）忽略了不可避免的标签噪声;（2）忽视增强关键视频片段的捕捉能力。本文提出了一种新的基于显著性的自训练框架SSVMR。具体而言，我们首先探索充分利用训练数据集所包含的信息，通过应用半监督方法来抑制标签噪声问题的不利影响，其中采用了自训练方法。此外，我们提出了通过在跨度级混合两个视频并保持两个原始视频的局部性来捕获视频的显著性。受自然语言处理中的反向翻译的启发，我们还进行了反向检索以获得更多的训练数据。在MVD数据集上的实验结果表明，SSVMR在R@1方面较之前的最佳模型提高了34.8%，大幅度地达到了最佳性能.摘要：With the rise of short videos, the demand for selecting appropriate background music (BGM) for a video has increased significantly, video-music retrieval (VMR) task gradually draws much attention by research community. As other cross-modal learning tasks, existing VMR approaches usually attempt to measure the similarity between the video and music in the feature space. However, they (1) neglect the inevitable label noise; (2) neglect to enhance the ability to capture critical video clips. In this paper, we propose a novel saliency-based self-training framework, which is termed SSVMR. Specifically, we first explore to fully make use of the information containing in the training dataset by applying a semi-supervised method to suppress the adverse impact of label noise problem, where a self-training approach is adopted. In addition, we propose to capture the saliency of the video by mixing two videos at span level and preserving the locality of the two original videos. Inspired by back translation in NLP, we also conduct back retrieval to obtain more training data. Experimental results on MVD dataset show that our SSVMR achieves the state-of-the-art performance by a large margin, obtaining a relative improvement of 34.8% over the previous best model in terms of R@1.【6】 Cost-effective Models for Detecting Depression from Speech
标题：从语音中检测抑郁的低成本模型
链接：https://arxiv.org/abs/2302.09214
作者：Mashrura Tasnim,Jekaterina Novikova机构：Winterlight Labs, Toronto, Canada备注：Accepted to ICMLA 2022摘要：抑郁症是最常见的心理障碍，被认为是世界范围内残疾和自杀的主要原因。一个能够检测人类语言中抑郁症迹象的自动化系统可以有助于确保为患有抑郁症的个人提供及时有效的心理健康护理。开发这样的自动化系统需要精确的机器学习模型，能够捕捉抑郁症的迹象。然而，基于深层声学表示的最新模型需要丰富的数据、细致的特征选择和严格的训练;该过程涉及巨大的计算资源。在这项工作中，我们探讨了两个不同的声学特征组-传统的手工策划和深度表征特征，从语音预测抑郁症的严重程度的有效性。我们探讨了可能的影响因素的相关性模型的表现，包括性别的个人，严重程度的障碍，内容和长度的讲话。我们的研究结果表明，与深度表征特征训练模型相比，传统声学特征训练模型表现得同样好或更好，但计算成本明显更低，而与其他因素无关，如内容和言语长度、说话者性别和障碍严重程度。这使得此类模型更适合于计算资源可用性受限的部署，例如智能设备中的实时抑郁症监测应用。摘要：Depression is the most common psychological disorder and is considered as a leading cause of disability and suicide worldwide. An automated system capable of detecting signs of depression in human speech can contribute to ensuring timely and effective mental health care for individuals suffering from the disorder. Developing such automated system requires accurate machine learning models, capable of capturing signs of depression. However, state-of-the-art models based on deep acoustic representations require abundant data, meticulous selection of features, and rigorous training; the procedure involves enormous computational resources. In this work, we explore the effectiveness of two different acoustic feature groups - conventional hand-curated and deep representation features, for predicting the severity of depression from speech. We explore the relevance of possible contributing factors to the models' performance, including gender of the individual, severity of the disorder, content and length of speech. Our findings suggest that models trained on conventional acoustic features perform equally well or better than the ones trained on deep representation features at significantly lower computational cost, irrespective of other factors, e.g. content and length of speech, gender of the speaker and severity of the disorder. This makes such models a better fit for deployment where availability of computational resources is restricted, such as real time depression monitoring applications in smart devices.【7】 Exposing AI-Synthesized Human Voices Using Neural Vocoder Artifacts
标题：利用神经声码器伪影显示人工智能合成的人声
链接：https://arxiv.org/abs/2302.09198
作者：Chengzhe Sun,Shan Jia,Shuwei Hou,Ehab AlBadawy,Siwei Lyu机构：Department of Computer Science and Engineering, University at Buffalo, SUNY, Buffalo, USA, Department of Electrical and Computer Engineering, University at Albany, SUNY, Albany, USA摘要：人工智能合成人类声音的进步带来了越来越大的假冒和虚假信息的威胁。因此，研究合成人声的检测方法具有重要的现实意义。本文提出了一种新的基于识别音频信号中神经声码器伪像的合成人声检测方法。神经声码器是特别设计的神经网络，其合成来自时间-频率表示的波形，梅尔谱图。神经声码器是大多数DeepFake音频合成模型中的核心组件。因此，神经声码器处理的识别意味着音频样本可能已经被合成。为了利用声码器伪像进行合成人声检测，我们引入了一个用于二进制类RawNet 2模型的多任务学习框架，该框架与声码器识别模块共享前端特征提取器。我们将声码器辨识当作一个借口任务，以限制前端的特征撷取器专注于声码器伪像，并为最后的二值分类器提供可辨识的特征。实验结果表明，基于声码器识别的改进RawNet 2模型对二进制任务具有较高的分类性能。摘要：The advancements of AI-synthesized human voices have introduced a growing threat of impersonation and disinformation. It is therefore of practical importance to developdetection methods for synthetic human voices. This work proposes a new approach to detect synthetic human voices based on identifying artifacts of neural vocoders in audio signals. A neural vocoder is a specially designed neural network that synthesizes waveforms from temporal-frequency representations, e.g., mel-spectrograms. The neural vocoder is a core component in most DeepFake audio synthesis models. Hence the identification of neural vocoder processing implies that an audio sample may have been synthesized. To take advantage of the vocoder artifacts for synthetic human voice detection, we introduce a multi-task learning framework for a binary-class RawNet2 model that shares the front-end feature extractor with a vocoder identification module. We treat the vocoder identification as a pretext task to constrain the front-end feature extractor to focus on vocoder artifacts and provide discriminative features for the final binary classifier. Our experiments show that the improved RawNet2 model based on vocoder identification achieves an overall high classification performance on the binary task.【8】 A DNN based Normalized Time-frequency Weighted Criterion for Robust Wideband DoA Estimation
标题：基于DNN归一化时频加权准则的宽带DOA稳健估计
链接：https://arxiv.org/abs/2302.10147
作者：Kuan-Lin Chen,Ching-Hua Lee,Bhaskar D. Rao,Harinath Garudadri机构：Department of Electrical and Computer Engineering, University of California, San Diego, Qualcomm Institute, University of California, San Diego备注：5 pages. Accepted at ICASSP 2023摘要：深度神经网络（DNNs）极大地提高了噪声环境中语音源定位的波达方向（DoA）估计方法。然而，由于易受非语音信号干扰，其定位精度仍远不能令人满意。为了提高抗干扰的鲁棒性，提出了一种基于DNN的归一化时频加权准则，该准则使得候选导向矢量与滤波后的快拍在T-F域的距离最小。我们的方法不需要特征分解，并使用简单的归一化来防止优化目标被噪声过滤快照误导。我们还研究了由DNN指导的T-F权的不同设计。我们发现，复制语音比率掩模的Hadamard乘积是高效的，并且优于所提出的方法中的其他技术，例如直接掩模和取平均值。然而，T-F权重的最佳性能设计一般是依赖于准则的。实验结果表明，该方法在噪声和混响环境下均优于常用的基于DNN的DOA估计方法，包括广泛使用的子空间方法。摘要：Deep neural networks (DNNs) have greatly benefited direction of arrival (DoA) estimation methods for speech source localization in noisy environments. However, their localization accuracy is still far from satisfactory due to the vulnerability to nonspeech interference. To improve the robustness against interference, we propose a DNN based normalized time-frequency (T-F) weighted criterion which minimizes the distance between the candidate steering vectors and the filtered snapshots in the T-F domain. Our method requires no eigendecomposition and uses a simple normalization to prevent the optimization objective from being misled by noisy filtered snapshots. We also study different designs of T-F weights guided by a DNN. We find that duplicating the Hadamard product of speech ratio masks is highly effective and better than other techniques such as direct masking and taking the mean in the proposed approach. However, the best-performing design of T-F weights is criterion-dependent in general. Experiments show that the proposed method outperforms popular DNN based DoA estimation methods including widely used subspace methods in noisy and reverberant environments.【9】 Stabilising and accelerating light gated recurrent units for automatic speech recognition
标题：用于自动语音识别的稳定和加速光门循环单元
链接：https://arxiv.org/abs/2302.10144
作者：Adel Moumen,Titouan Parcollet机构：Laboratoire Informatique d’Avignon, Avignon Universit´e, Cambridge Machine Learning Systems Lab, University of Cambridge摘要：光选通递归单元（Li-GRU）以在自动语音识别（ASR）任务中实现令人印象深刻的结果而闻名，同时比标准选通递归单元（GRU）更轻且训练更快。然而，其校正线性单元在候选递归门上的无界性质导致了一个重要的梯度爆炸现象，干扰了训练过程，并阻止了它应用于著名数据集。本文从理论和实验两方面推导了该结构稳定的必要条件，以及将其训练时间提高到原来五分之一的工程机制，并由此提出了一种新的结构SLi-GRU.然后，我们评估了它在一个玩具任务上的性能，说明了它新获得的能力，并评估了一组三个不同的ASR数据集，与更复杂的递归神经网络相比，显示了更低的单词错误率。摘要：The light gated recurrent units (Li-GRU) is well-known for achieving impressive results in automatic speech recognition (ASR) tasks while being lighter and faster to train than a standard gated recurrent units (GRU). However, the unbounded nature of its rectified linear unit on the candidate recurrent gate induces an important gradient exploding phenomenon disrupting the training process and preventing it from being applied to famous datasets. In this paper, we theoretically and empirically derive the necessary conditions for its stability as well as engineering mechanisms to speed up by a factor of five its training time, hence introducing a novel version of this architecture named SLi-GRU. Then, we evaluate its performance both on a toy task illustrating its newly acquired capabilities and a set of three different ASR datasets demonstrating lower word error rates compared to more complex recurrent neural networks.【10】 Personalized speech enhancement combining band-split RNN and speaker attentive module
标题：结合分带RNN和说话人注意模块的个性化语音增强
链接：https://arxiv.org/abs/2302.09953
作者：Xiaohuai Le,Zhongshu Hou,Li Chen,Chao He,Yiqing Guo,Cheng Chen,Xianjun Xia,Jing Lu机构：Key Laboratory of Modern Acoustics, Nanjing University, Nanjing , China, RTC Lab, ByteDance, China, NJU-Horizon Intelligent Audio Lab, Horizon Robotics, Beijing , China摘要：目标说话人信息可用于语音增强（SE）模型中以更有效地提取期望的语音。已有的研究工作是通过级联或仿射变换将说话人嵌入到语音增强模型中。本文提出了一个说话人注意力模型来计算说话人嵌入和中间特征之间的注意力分数，并利用这些分数对特征进行尺度变换。通过将此模块合并到最先进的SE模型中，我们为ICASSP信号处理大挑战赛构建了个性化SE模型：DNS挑战5（2023年）。我们的系统在track1的盲测试集上获得了0.529的最终得分，在track2上获得了0.549。摘要：Target speaker information can be utilized in speech enhancement (SE) models to more effectively extract the desired speech. Previous works introduce the speaker embedding into speech enhancement models by means of concatenation or affine transformation. In this paper, we propose a speaker attentive module to calculate the attention scores between the speaker embedding and the intermediate features, which are used to rescale the features. By merging this module in the state-of-the-art SE model, we construct the personalized SE model for ICASSP Signal Processing Grand Challenge: DNS Challenge 5 (2023). Our system achieves a final score of 0.529 on the blind test set of track1 and 0.549 on track2.【11】 Synergy between human and machine approaches to sound/scene recognition and processing: An overview of ICASSP special session
标题：声音/场景识别和处理的人机方法之间的协同作用：ICASSP特别会议综述
链接：https://arxiv.org/abs/2302.09719
作者：Laurie M. Heller,Benjamin Elizalde,Bhiksha Raj,Soham Deshmuk机构：∗Special session co-organizers, Department of Psychology, Carnegie Mellon University, Microsoft, Language Technologies Institute, Carnegie Mellon University, Mohammed bin Zayed University of AI备注：4 pages. Summary of Special Session planned for 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). this https URL摘要：通常形式化的机器监听试图执行一项任务，从我们的角度来看，该任务基本上是人类可执行的，并且由人类执行。当前机器听力的自动化模型从纯粹的数据驱动方法到模仿人类系统的方法各不相同。近年来，最有前途的方法是混合的，因为它们使用了由人类系统的感知、认知和语义过程的模型提供信息的数据驱动方法。人类感知和领域知识模型提供的指导不仅能够实现更好、更可推广的机器监听，相反，从这些模型中获得的经验教训可以用来验证或改进我们的人类感知模型本身。本文总结了这类混合方法的发展进展，从基于外围（人类）听觉过程模型的机器听力模型，到利用或推导编码在声音之间关系中的语义信息的模型。本文描述的研究在2023年ICASSP会议的“声音/场景识别和处理的人机协同方法”特别会议上进行了介绍。摘要：Machine Listening, as usually formalized, attempts to perform a task that is, from our perspective, fundamentally human-performable, and performed by humans. Current automated models of Machine Listening vary from purely data-driven approaches to approaches imitating human systems. In recent years, the most promising approaches have been hybrid in that they have used data-driven approaches informed by models of the perceptual, cognitive, and semantic processes of the human system. Not only does the guidance provided by models of human perception and domain knowledge enable better, and more generalizable Machine Listening, in the converse, the lessons learned from these models may be used to verify or improve our models of human perception themselves. This paper summarizes advances in the development of such hybrid approaches, ranging from Machine Listening models that are informed by models of peripheral (human) auditory processes, to those that employ or derive semantic information encoded in relations between sounds. The research described herein was presented in a special session on ``Synergy between human and machine approaches to sound/scene recognition and processing'' at the 2023 ICASSP meeting.【12】 Probabilistic Back-ends for Online Speaker Recognition and Clustering
标题：在线说话人识别和聚类的概率后端
链接：https://arxiv.org/abs/2302.09523
作者：Alexey Sholokhov,Nikita Kuzmin,Kong Aik Lee,Eng Siong Chng机构：Federal Research Center “Computer Science and Control”, of the Russian Academy of Sciences, Moscow, Russia, Nanyang Technological University, Singapore, Institute for Infocomm Research, A⋆STAR, Singapore备注：Accepted to ICASSP 2023摘要：针对在线说话人聚类任务中自然出现的多注册说话人识别问题，研究了该场景下不同评分后端的性能。首先，我们证明了流行的余弦评分遭受穷人的分数校准与不同数量的注册话语。第二，我们提出了一个简单的替代余弦评分的基础上极端约束版本的概率线性判别分析（PLDA）。该模型在保持一对一比较性能不变的情况下，对多注册识别的余弦评分进行了改进。最后，我们考虑了一个在线说话人聚类任务，其中每一步自然涉及多注册识别。我们提出了一种在线聚类算法，使我们能够从PLDA模型中获益，如处理不确定性和更好的分数校准的能力。实验证明了该算法的有效性。摘要：This paper focuses on multi-enrollment speaker recognition which naturally occurs in the task of online speaker clustering, and studies the properties of different scoring back-ends in this scenario. First, we show that popular cosine scoring suffers from poor score calibration with a varying number of enrollment utterances. Second, we propose a simple replacement for cosine scoring based on an extremely constrained version of probabilistic linear discriminant analysis (PLDA). The proposed model improves over the cosine scoring for multi-enrollment recognition while keeping the same performance in the case of one-to-one comparisons. Finally, we consider an online speaker clustering task where each step naturally involves multi-enrollment recognition. We propose an online clustering algorithm allowing us to take benefits from the PLDA model such as the ability to handle uncertainty and better score calibration. Our experiments demonstrate the effectiveness of the proposed algorithm.【13】 RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness
标题：RobustDistiller：压缩通用语音表示以增强环境健壮性
链接：https://arxiv.org/abs/2302.09437
作者：Heitor R. Guimarães,Arthur Pimentel,Anderson R. Avila,Mehdi Rezagholizadeh,Boxing Chen,Tiago H. Falk机构：INRS-EMT, Universit´e du Qu´ebec, Montr´eal, Canada, Huawei Noah’s Ark Lab, Montr´eal, Canada备注：Accepted for ICASSP 2023摘要：自监督语音预训练使得深度神经网络模型能够从原始波形信号中捕获有意义的和解纠缠的因素。学习的通用语音表示然后可以跨多个下游任务使用。然而，这些表示对由诸如噪声和/或房间混响之类的环境因素引起的分布偏移敏感。它们的大尺寸反过来又使它们不适用于边缘应用。在这项工作中，我们提出了一种知识提取方法，称为RobustDistiller，它通过多任务学习目标压缩普遍表示，同时使它们对环境伪像更鲁棒。基于三种公认的通用表示法以及三项下游任务，对拟定的逐层蒸馏配方进行了评估。实验结果表明，应用于WavLM Base+ teacher模型之上的所提出的方法在噪声类型和水平以及混响时间方面优于所有其他基准。通常，学生模型（24 M参数）获得的结果与教师模型（95 M）获得的结果一致。摘要：Self-supervised speech pre-training enables deep neural network models to capture meaningful and disentangled factors from raw waveform signals. The learned universal speech representations can then be used across numerous downstream tasks. These representations, however, are sensitive to distribution shifts caused by environmental factors, such as noise and/or room reverberation. Their large sizes, in turn, make them unfeasible for edge applications. In this work, we propose a knowledge distillation methodology termed RobustDistiller which compresses universal representations while making them more robust against environmental artifacts via a multi-task learning objective. The proposed layer-wise distillation recipe is evaluated on top of three well-established universal representations, as well as with three downstream tasks. Experimental results show the proposed methodology applied on top of the WavLM Base+ teacher model outperforming all other benchmarks across noise types and levels, as well as reverberation times. Oftentimes, the obtained results with the student model (24M parameters) achieved results inline with those of the teacher model (95M).【14】 Speaker and Language Change Detection using Wav2vec2 and Whisper
标题：基于Wav2ve2和Whisper的说话人和语言变化检测
链接：https://arxiv.org/abs/2302.09381
作者：Tijn Berns,Nik Vaessen,David A. van Leeuwen机构：Radboud University Nijmegen, The Netherlands, SpraakLab, The Netherlands备注：Going to be submitted, but rules do not allow to indicate where摘要：我们调查了最近的Transformer网络预训练的自动语音识别的能力，检测说话人和语言变化的语音。我们只需在标签中添加说话人（更改）或语言目标即可。对于Wav2vec2预训练网络，我们还研究了说话人改变符号的表示是否可以被调节以捕获说话人身份特征。使用大量构建的数据集，我们表明这些能力肯定存在，说话人识别的错误率为10%，语言检测的错误率为百分之几。我们将公布再现性代码。摘要：We investigate recent transformer networks pre-trained for automatic speech recognition for their ability to detect speaker and language changes in speech. We do this by simply adding speaker (change) or language targets to the labels. For Wav2vec2 pre-trained networks, we also investigate if the representation for the speaker change symbol can be conditioned to capture speaker identity characteristics. Using a number of constructed data sets we show that these capabilities are definitely there, with speaker recognition equal error rates of the order of 10% and language detection error rates of a few percent. We will publish the code for reproducibility.【15】 Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognition
标题：前端适配器：自适应语音自监督学习的前端输入用于语音识别
链接：https://arxiv.org/abs/2302.09331
作者：Xie Chen,Ziyang Ma,Changli Tang,Yujin Wang,Zhisheng Zheng机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Department of Electronic Engineering, Tsinghua University, Beijing, China摘要：近年来，在包括语音处理在内的各个领域中，自监督学习（SSL）蓬勃发展。基于语音的SSL模型在语音相关任务中表现出良好的性能。然而，SSL模型的训练在计算上是昂贵的，并且通常的实践是针对特定任务微调已发布的SSL模型。在预训练和微调过程中使用一致的前端输入至关重要。当最佳前端与预训练中使用的前端不相同时，这种一致性可能会引入潜在的问题。在本文中，我们提出了一种简单而有效的前端适配器来解决这种前端差异。通过最小化不同前端的输出之间的距离，滤波器组特征（Fbank）可以与用波形预训练的SSL模型兼容。实验结果证明了我们提出的前端适配器在几种流行的SSL模型上用于语音识别任务的有效性。摘要：Recent years have witnessed a boom in self-supervised learning (SSL) in various areas including speech processing. Speech based SSL models present promising performance in a range of speech related tasks. However, the training of SSL models is computationally expensive and a common practice is to fine-tune a released SSL model on the specific task. It is essential to use consistent front-end input during pre-training and fine-tuning. This consistency may introduce potential issues when the optimal front-end is not the same as that used in pre-training. In this paper, we propose a simple but effective front-end adapter to address this front-end discrepancy. By minimizing the distance between the outputs of different front-ends, the filterbank feature (Fbank) can be compatible with SSL models which are pre-trained with waveform. The experiment results demonstrate the effectiveness of our proposed front-end adapter on several popular SSL models for the speech recognition task.【16】 Multi-dimensional frequency dynamic convolution with confident mean teacher for sound event detection
标题：用于声事件检测的多维频率动态卷积与自信均值教师
链接：https://arxiv.org/abs/2302.09256
机构：Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, University of Chinese Academy of Sciences备注：accepted to ICASSP 2022摘要：近年来，卷积神经网络（convolutional neural networks，CNN）被广泛应用于声音事件检测（sound event detection，SED）中。然而，传统卷积在学习不同声音事件的时频域表示方面存在不足。为了解决这个问题，我们提出了多维频率动态卷积（MFDConv），一种新的设计，赋予卷积核在多个维度上的频率自适应动态特性。MFDConv利用一种新颖的多维注意机制，采用并行策略学习互补的频率自适应注意，大大增强了卷积核的特征提取能力.此外，为了提高均值教师的学习绩效，提出了置信均值教师，以提高教师伪标签的准确率，训练具有高置信度标签的学生。实验结果表明，该方法在DESED真实数据集上的PSDS1和PSDS2分别达到了0.470和0.692。摘要：Recently, convolutional neural networks (CNNs) have been widely used in sound event detection (SED). However, traditional convolution is deficient in learning time-frequency domain representation of different sound events. To address this issue, we propose multi-dimensional frequency dynamic convolution (MFDConv), a new design that endows convolutional kernels with frequency-adaptive dynamic properties along multiple dimensions. MFDConv utilizes a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary frequency-adaptive attentions, which substantially strengthen the feature extraction ability of convolutional kernels. Moreover, in order to promote the performance of mean teacher, we propose the confident mean teacher to increase the accuracy of pseudo-labels from the teacher and train the student with high confidence labels. Experimental results show that the proposed methods achieve 0.470 and 0.692 of PSDS1 and PSDS2 on the DESED real validation dataset.【17】 Deep AHS: A Deep Learning Approach to Acoustic Howling Suppression
标题：深度AHS：一种抑制声学嚎叫的深度学习方法
链接：https://arxiv.org/abs/2302.09252
作者：Hao Zhang,Meng Yu,Dong Yu机构：Tencent AI Lab, Bellevue, WA, USA备注：Accepted for publication in 2023 ICASSP摘要：本文将声啸叫抑制（acoustic howling suppression，AHS）问题描述为一个有监督的学习问题，提出了一种深度学习方法Deep AHS，采用教师强制的方式训练Deep AHS，将周期性的啸叫抑制过程转化为瞬时语音分离过程，以简化问题并加快模型训练。该方法利用合理设计的特征，训练基于注意力的递归神经网络（RNN）从麦克风录音中提取目标信号，从而抑制可能导致啸叫的回放信号。研究了不同的训练策略，并使用递归模式实现的流推理方法来评估所提出的方法的实时啸声抑制性能。Deep AHS避免了啸声检测，并从本质上禁止了啸声的发生，从而在音频系统的设计中实现了更大的灵活性。实验结果验证了该方法在不同场景下对啸声抑制的有效性。摘要：In this paper, we formulate acoustic howling suppression (AHS) as a supervised learning problem and propose a deep learning approach, called Deep AHS, to address it. Deep AHS is trained in a teacher forcing way which converts the recurrent howling suppression process into an instantaneous speech separation process to simplify the problem and accelerate the model training. The proposed method utilizes properly designed features and trains an attention based recurrent neural network (RNN) to extract the target signal from the microphone recording, thus attenuating the playback signal that may lead to howling. Different training strategies are investigated and a streaming inference method implemented in a recurrent mode used to evaluate the performance of the proposed method for real-time howling suppression. Deep AHS avoids howling detection and intrinsically prohibits howling from happening, allowing for more flexibility in the design of audio systems. Experimental results show the effectiveness of the proposed method for howling suppression under different scenarios.【1】 A DNN based Normalized Time-frequency Weighted Criterion for Robust Wideband DoA Estimation
标题：基于DNN归一化时频加权准则的宽带DOA稳健估计
链接：https://arxiv.org/abs/2302.10147
*与cs.SD语音【8】为同一篇
作者：Kuan-Lin Chen,Ching-Hua Lee,Bhaskar D. Rao,Harinath Garudadri机构：Department of Electrical and Computer Engineering, University of California, San Diego, Qualcomm Institute, University of California, San Diego备注：5 pages. Accepted at ICASSP 2023摘要：深度神经网络（DNNs）极大地提高了噪声环境中语音源定位的波达方向（DoA）估计方法。然而，由于易受非语音信号干扰，其定位精度仍远不能令人满意。为了提高抗干扰的鲁棒性，提出了一种基于DNN的归一化时频加权准则，该准则使得候选导向矢量与滤波后的快拍在T-F域的距离最小。我们的方法不需要特征分解，并使用简单的归一化来防止优化目标被噪声过滤快照误导。我们还研究了由DNN指导的T-F权的不同设计。我们发现，复制语音比率掩模的Hadamard乘积是高效的，并且优于所提出的方法中的其他技术，例如直接掩模和取平均值。然而，T-F权重的最佳性能设计一般是依赖于准则的。实验结果表明，该方法在噪声和混响环境下均优于常用的基于DNN的DOA估计方法，包括广泛使用的子空间方法。摘要：Deep neural networks (DNNs) have greatly benefited direction of arrival (DoA) estimation methods for speech source localization in noisy environments. However, their localization accuracy is still far from satisfactory due to the vulnerability to nonspeech interference. To improve the robustness against interference, we propose a DNN based normalized time-frequency (T-F) weighted criterion which minimizes the distance between the candidate steering vectors and the filtered snapshots in the T-F domain. Our method requires no eigendecomposition and uses a simple normalization to prevent the optimization objective from being misled by noisy filtered snapshots. We also study different designs of T-F weights guided by a DNN. We find that duplicating the Hadamard product of speech ratio masks is highly effective and better than other techniques such as direct masking and taking the mean in the proposed approach. However, the best-performing design of T-F weights is criterion-dependent in general. Experiments show that the proposed method outperforms popular DNN based DoA estimation methods including widely used subspace methods in noisy and reverberant environments.【2】 Stabilising and accelerating light gated recurrent units for automatic speech recognition
标题：用于自动语音识别的稳定和加速光门循环单元
链接：https://arxiv.org/abs/2302.10144
*与cs.SD语音【9】为同一篇
作者：Adel Moumen,Titouan Parcollet机构：Laboratoire Informatique d’Avignon, Avignon Universit´e, Cambridge Machine Learning Systems Lab, University of Cambridge摘要：光选通递归单元（Li-GRU）以在自动语音识别（ASR）任务中实现令人印象深刻的结果而闻名，同时比标准选通递归单元（GRU）更轻且训练更快。然而，其校正线性单元在候选递归门上的无界性质导致了一个重要的梯度爆炸现象，干扰了训练过程，并阻止了它应用于著名数据集。本文从理论和实验两方面推导了该结构稳定的必要条件，以及将其训练时间提高到原来五分之一的工程机制，并由此提出了一种新的结构SLi-GRU.然后，我们评估了它在一个玩具任务上的性能，说明了它新获得的能力，并评估了一组三个不同的ASR数据集，与更复杂的递归神经网络相比，显示了更低的单词错误率。摘要：The light gated recurrent units (Li-GRU) is well-known for achieving impressive results in automatic speech recognition (ASR) tasks while being lighter and faster to train than a standard gated recurrent units (GRU). However, the unbounded nature of its rectified linear unit on the candidate recurrent gate induces an important gradient exploding phenomenon disrupting the training process and preventing it from being applied to famous datasets. In this paper, we theoretically and empirically derive the necessary conditions for its stability as well as engineering mechanisms to speed up by a factor of five its training time, hence introducing a novel version of this architecture named SLi-GRU. Then, we evaluate its performance both on a toy task illustrating its newly acquired capabilities and a set of three different ASR datasets demonstrating lower word error rates compared to more complex recurrent neural networks.【3】 Learnable Frontends that do not Learn: Quantifying Sensitivity to Filterbank Initialisation
标题：不学习的可学习前沿：量化对过滤器组初始化的敏感度
链接：https://arxiv.org/abs/2302.10014
作者：Mark Anderson,Tomi Kinnunen,Naomi Harte机构：⋆ SIGMEDIA Lab, School of Engineering, Trinity College Dublin, Ireland, † School of Computing, University of Eastern Finland, Joensuu, Finland备注：Accepted at ICASSP 2023, 5 pages, 2 figures, 2 tables摘要：虽然许多现代语音和音频处理依赖于使用固定音频表示训练的深度神经网络，但最近的研究表明，与后端联合学习声学前端具有巨大潜力。在本研究中，我们特别关注可学习滤波器组。先前的研究已经报道，在使用初始化为mel尺度的可学习滤波器组的前端中，学习滤波器与其初始化没有实质性的不同。使用基于Gabor的滤波器组，我们研究了可学习滤波器组对其在两个音频任务上使用若干初始化策略进行的初始化的敏感性：语音活动检测和鸟类识别。我们使用Jensen-Shannon距离并分析训练前后的学习滤波器。我们表明，虽然整体性能得到改善，滤波器组表现出强烈的敏感性，他们的初始化策略。初始化值的有限移动表明，替代优化策略可以允许可学习前端达到更好的整体性能。摘要：While much of modern speech and audio processing relies on deep neural networks trained using fixed audio representations, recent studies suggest great potential in acoustic frontends learnt jointly with a backend. In this study, we focus specifically on learnable filterbanks. Prior studies have reported that in frontends using learnable filterbanks initialised to a mel scale, the learned filters do not differ substantially from their initialisation. Using a Gabor-based filterbank, we investigate the sensitivity of a learnable filterbank to its initialisation using several initialisation strategies on two audio tasks: voice activity detection and bird species identification. We use the Jensen-Shannon Distance and analysis of the learned filters before and after training. We show that although performance is overall improved, the filterbanks exhibit strong sensitivity to their initialisation strategy. The limited movement from initialised values suggests that alternate optimisation strategies may allow a learnable frontend to reach better overall performance.【4】 Personalized speech enhancement combining band-split RNN and speaker attentive module
标题：结合分带RNN和说话人注意模块的个性化语音增强
链接：https://arxiv.org/abs/2302.09953
*与cs.SD语音【10】为同一篇
作者：Xiaohuai Le,Zhongshu Hou,Li Chen,Chao He,Yiqing Guo,Cheng Chen,Xianjun Xia,Jing Lu机构：Key Laboratory of Modern Acoustics, Nanjing University, Nanjing , China, RTC Lab, ByteDance, China, NJU-Horizon Intelligent Audio Lab, Horizon Robotics, Beijing , China摘要：目标说话人信息可用于语音增强（SE）模型中以更有效地提取期望的语音。已有的研究工作是通过级联或仿射变换将说话人嵌入到语音增强模型中。本文提出了一个说话人注意力模型来计算说话人嵌入和中间特征之间的注意力分数，并利用这些分数对特征进行尺度变换。通过将此模块合并到最先进的SE模型中，我们为ICASSP信号处理大挑战赛构建了个性化SE模型：DNS挑战5（2023年）。我们的系统在track1的盲测试集上获得了0.529的最终得分，在track2上获得了0.549。摘要：Target speaker information can be utilized in speech enhancement (SE) models to more effectively extract the desired speech. Previous works introduce the speaker embedding into speech enhancement models by means of concatenation or affine transformation. In this paper, we propose a speaker attentive module to calculate the attention scores between the speaker embedding and the intermediate features, which are used to rescale the features. By merging this module in the state-of-the-art SE model, we construct the personalized SE model for ICASSP Signal Processing Grand Challenge: DNS Challenge 5 (2023). Our system achieves a final score of 0.529 on the blind test set of track1 and 0.549 on track2.【5】 An ASR-free Fluency Scoring Approach with Self-Supervised Learning
标题：一种无ASR的自监督学习流利性评分方法
链接：https://arxiv.org/abs/2302.09928
作者：Wei Liu,Kaiqi Fu,Xiaohai Tian,Shuju Shi,Wei Li,Zejun Ma,Tan Lee机构：Department of Electronic Engineering, The Chinese University of Hong Kong, ByteDance AI Lab Speech & Audio Team备注：Accepted by ICASSP 2023摘要：典型的流利性评分系统通常依赖于自动语音识别（ASR）系统来获得输入语音中的时间戳，用于流利性相关特征的后续计算或者利用端到端方法直接对语音流利性进行建模。提出了一种基于自监督学习（SSL）的自动流利度评估方法。具体地，使用wav2vec2.0来提取帧级语音特征，随后进行K均值聚类以向每个帧分配伪标签（聚类索引）。训练基于BLSTM的模型，以从帧级SSL特征和对应的聚类指数预测话语级流畅性得分。在所提出的系统中既不需要语音转录也不需要时间戳信息。它是无ASR的，并且可以潜在地避免ASR错误在实践中的影响。在非英语母语数据库上的实验结果表明，与已有方法相比，该方法在“开放式应答”场景下的性能有了显著提高，并与最近报道的“朗读”场景下的性能相匹配。摘要：A typical fluency scoring system generally relies on an automatic speech recognition (ASR) system to obtain time stamps in input speech for either the subsequent calculation of fluency-related features or directly modeling speech fluency with an end-to-end approach. This paper describes a novel ASR-free approach for automatic fluency assessment using self-supervised learning (SSL). Specifically, wav2vec2.0 is used to extract frame-level speech features, followed by K-means clustering to assign a pseudo label (cluster index) to each frame. A BLSTM-based model is trained to predict an utterance-level fluency score from frame-level SSL features and the corresponding cluster indexes. Neither speech transcription nor time stamp information is required in the proposed system. It is ASR-free and can potentially avoid the ASR errors effect in practice. Experimental results carried out on non-native English databases show that the proposed approach significantly improves the performance in the "open response" scenario as compared to previous methods and matches the recently reported performance in the "read aloud" scenario.【6】 Synergy between human and machine approaches to sound/scene recognition and processing: An overview of ICASSP special session
标题：声音/场景识别和处理的人机方法之间的协同作用：ICASSP特别会议综述
链接：https://arxiv.org/abs/2302.09719
*与cs.SD语音【11】为同一篇
作者：Laurie M. Heller,Benjamin Elizalde,Bhiksha Raj,Soham Deshmuk机构：∗Special session co-organizers, Department of Psychology, Carnegie Mellon University, Microsoft, Language Technologies Institute, Carnegie Mellon University, Mohammed bin Zayed University of AI备注：4 pages. Summary of Special Session planned for 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). this https URL摘要：通常形式化的机器监听试图执行一项任务，从我们的角度来看，该任务基本上是人类可执行的，并且由人类执行。当前机器听力的自动化模型从纯粹的数据驱动方法到模仿人类系统的方法各不相同。近年来，最有前途的方法是混合的，因为它们使用了由人类系统的感知、认知和语义过程的模型提供信息的数据驱动方法。人类感知和领域知识模型提供的指导不仅能够实现更好、更可推广的机器监听，相反，从这些模型中获得的经验教训可以用来验证或改进我们的人类感知模型本身。本文总结了这类混合方法的发展进展，从基于外围（人类）听觉过程模型的机器听力模型，到利用或推导编码在声音之间关系中的语义信息的模型。本文描述的研究在2023年ICASSP会议的“声音/场景识别和处理的人机协同方法”特别会议上进行了介绍。摘要：Machine Listening, as usually formalized, attempts to perform a task that is, from our perspective, fundamentally human-performable, and performed by humans. Current automated models of Machine Listening vary from purely data-driven approaches to approaches imitating human systems. In recent years, the most promising approaches have been hybrid in that they have used data-driven approaches informed by models of the perceptual, cognitive, and semantic processes of the human system. Not only does the guidance provided by models of human perception and domain knowledge enable better, and more generalizable Machine Listening, in the converse, the lessons learned from these models may be used to verify or improve our models of human perception themselves. This paper summarizes advances in the development of such hybrid approaches, ranging from Machine Listening models that are informed by models of peripheral (human) auditory processes, to those that employ or derive semantic information encoded in relations between sounds. The research described herein was presented in a special session on ``Synergy between human and machine approaches to sound/scene recognition and processing'' at the 2023 ICASSP meeting.【7】 Probabilistic Back-ends for Online Speaker Recognition and Clustering
标题：在线说话人识别和聚类的概率后端
链接：https://arxiv.org/abs/2302.09523
*与cs.SD语音【12】为同一篇
作者：Alexey Sholokhov,Nikita Kuzmin,Kong Aik Lee,Eng Siong Chng机构：Federal Research Center “Computer Science and Control”, of the Russian Academy of Sciences, Moscow, Russia, Nanyang Technological University, Singapore, Institute for Infocomm Research, A⋆STAR, Singapore备注：Accepted to ICASSP 2023摘要：针对在线说话人聚类任务中自然出现的多注册说话人识别问题，研究了该场景下不同评分后端的性能。首先，我们证明了流行的余弦评分遭受穷人的分数校准与不同数量的注册话语。第二，我们提出了一个简单的替代余弦评分的基础上极端约束版本的概率线性判别分析（PLDA）。该模型在保持一对一比较性能不变的情况下，对多注册识别的余弦评分进行了改进。最后，我们考虑了一个在线说话人聚类任务，其中每一步自然涉及多注册识别。我们提出了一种在线聚类算法，使我们能够从PLDA模型中获益，如处理不确定性和更好的分数校准的能力。实验证明了该算法的有效性。摘要：This paper focuses on multi-enrollment speaker recognition which naturally occurs in the task of online speaker clustering, and studies the properties of different scoring back-ends in this scenario. First, we show that popular cosine scoring suffers from poor score calibration with a varying number of enrollment utterances. Second, we propose a simple replacement for cosine scoring based on an extremely constrained version of probabilistic linear discriminant analysis (PLDA). The proposed model improves over the cosine scoring for multi-enrollment recognition while keeping the same performance in the case of one-to-one comparisons. Finally, we consider an online speaker clustering task where each step naturally involves multi-enrollment recognition. We propose an online clustering algorithm allowing us to take benefits from the PLDA model such as the ability to handle uncertainty and better score calibration. Our experiments demonstrate the effectiveness of the proposed algorithm.【8】 RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness
标题：RobustDistiller：压缩通用语音表示以增强环境健壮性
链接：https://arxiv.org/abs/2302.09437
*与cs.SD语音【13】为同一篇
作者：Heitor R. Guimarães,Arthur Pimentel,Anderson R. Avila,Mehdi Rezagholizadeh,Boxing Chen,Tiago H. Falk机构：INRS-EMT, Universit´e du Qu´ebec, Montr´eal, Canada, Huawei Noah’s Ark Lab, Montr´eal, Canada备注：Accepted for ICASSP 2023摘要：自监督语音预训练使得深度神经网络模型能够从原始波形信号中捕获有意义的和解纠缠的因素。学习的通用语音表示然后可以跨多个下游任务使用。然而，这些表示对由诸如噪声和/或房间混响之类的环境因素引起的分布偏移敏感。它们的大尺寸反过来又使它们不适用于边缘应用。在这项工作中，我们提出了一种知识提取方法，称为RobustDistiller，它通过多任务学习目标压缩普遍表示，同时使它们对环境伪像更鲁棒。基于三种公认的通用表示法以及三项下游任务，对拟定的逐层蒸馏配方进行了评估。实验结果表明，应用于WavLM Base+ teacher模型之上的所提出的方法在噪声类型和水平以及混响时间方面优于所有其他基准。通常，学生模型（24 M参数）获得的结果与教师模型（95 M）获得的结果一致。摘要：Self-supervised speech pre-training enables deep neural network models to capture meaningful and disentangled factors from raw waveform signals. The learned universal speech representations can then be used across numerous downstream tasks. These representations, however, are sensitive to distribution shifts caused by environmental factors, such as noise and/or room reverberation. Their large sizes, in turn, make them unfeasible for edge applications. In this work, we propose a knowledge distillation methodology termed RobustDistiller which compresses universal representations while making them more robust against environmental artifacts via a multi-task learning objective. The proposed layer-wise distillation recipe is evaluated on top of three well-established universal representations, as well as with three downstream tasks. Experimental results show the proposed methodology applied on top of the WavLM Base+ teacher model outperforming all other benchmarks across noise types and levels, as well as reverberation times. Oftentimes, the obtained results with the student model (24M parameters) achieved results inline with those of the teacher model (95M).【9】 Speaker and Language Change Detection using Wav2vec2 and Whisper
标题：基于Wav2ve2和Whisper的说话人和语言变化检测
链接：https://arxiv.org/abs/2302.09381
*与cs.SD语音【14】为同一篇
作者：Tijn Berns,Nik Vaessen,David A. van Leeuwen机构：Radboud University Nijmegen, The Netherlands, SpraakLab, The Netherlands备注：Going to be submitted, but rules do not allow to indicate where摘要：我们调查了最近的Transformer网络预训练的自动语音识别的能力，检测说话人和语言变化的语音。我们只需在标签中添加说话人（更改）或语言目标即可。对于Wav2vec2预训练网络，我们还研究了说话人改变符号的表示是否可以被调节以捕获说话人身份特征。使用大量构建的数据集，我们表明这些能力肯定存在，说话人识别的错误率为10%，语言检测的错误率为百分之几。我们将公布再现性代码。摘要：We investigate recent transformer networks pre-trained for automatic speech recognition for their ability to detect speaker and language changes in speech. We do this by simply adding speaker (change) or language targets to the labels. For Wav2vec2 pre-trained networks, we also investigate if the representation for the speaker change symbol can be conditioned to capture speaker identity characteristics. Using a number of constructed data sets we show that these capabilities are definitely there, with speaker recognition equal error rates of the order of 10% and language detection error rates of a few percent. We will publish the code for reproducibility.【10】 Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognition
标题：前端适配器：自适应语音自监督学习的前端输入用于语音识别
链接：https://arxiv.org/abs/2302.09331
*与cs.SD语音【15】为同一篇
作者：Xie Chen,Ziyang Ma,Changli Tang,Yujin Wang,Zhisheng Zheng机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Department of Electronic Engineering, Tsinghua University, Beijing, China摘要：近年来，在包括语音处理在内的各个领域中，自监督学习（SSL）蓬勃发展。基于语音的SSL模型在语音相关任务中表现出良好的性能。然而，SSL模型的训练在计算上是昂贵的，并且通常的实践是针对特定任务微调已发布的SSL模型。在预训练和微调过程中使用一致的前端输入至关重要。当最佳前端与预训练中使用的前端不相同时，这种一致性可能会引入潜在的问题。在本文中，我们提出了一种简单而有效的前端适配器来解决这种前端差异。通过最小化不同前端的输出之间的距离，滤波器组特征（Fbank）可以与用波形预训练的SSL模型兼容。实验结果证明了我们提出的前端适配器在几种流行的SSL模型上用于语音识别任务的有效性。摘要：Recent years have witnessed a boom in self-supervised learning (SSL) in various areas including speech processing. Speech based SSL models present promising performance in a range of speech related tasks. However, the training of SSL models is computationally expensive and a common practice is to fine-tune a released SSL model on the specific task. It is essential to use consistent front-end input during pre-training and fine-tuning. This consistency may introduce potential issues when the optimal front-end is not the same as that used in pre-training. In this paper, we propose a simple but effective front-end adapter to address this front-end discrepancy. By minimizing the distance between the outputs of different front-ends, the filterbank feature (Fbank) can be compatible with SSL models which are pre-trained with waveform. The experiment results demonstrate the effectiveness of our proposed front-end adapter on several popular SSL models for the speech recognition task.【11】 Multi-dimensional frequency dynamic convolution with confident mean teacher for sound event detection
标题：用于声事件检测的多维频率动态卷积与自信均值教师
链接：https://arxiv.org/abs/2302.09256
*与cs.SD语音【16】为同一篇
机构：Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, University of Chinese Academy of Sciences备注：accepted to ICASSP 2022摘要：近年来，卷积神经网络（convolutional neural networks，CNN）被广泛应用于声音事件检测（sound event detection，SED）中。然而，传统卷积在学习不同声音事件的时频域表示方面存在不足。为了解决这个问题，我们提出了多维频率动态卷积（MFDConv），一种新的设计，赋予卷积核在多个维度上的频率自适应动态特性。MFDConv利用一种新颖的多维注意机制，采用并行策略学习互补的频率自适应注意，大大增强了卷积核的特征提取能力.此外，为了提高均值教师的学习绩效，提出了置信均值教师，以提高教师伪标签的准确率，训练具有高置信度标签的学生。实验结果表明，该方法在DESED真实数据集上的PSDS1和PSDS2分别达到了0.470和0.692。摘要：Recently, convolutional neural networks (CNNs) have been widely used in sound event detection (SED). However, traditional convolution is deficient in learning time-frequency domain representation of different sound events. To address this issue, we propose multi-dimensional frequency dynamic convolution (MFDConv), a new design that endows convolutional kernels with frequency-adaptive dynamic properties along multiple dimensions. MFDConv utilizes a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary frequency-adaptive attentions, which substantially strengthen the feature extraction ability of convolutional kernels. Moreover, in order to promote the performance of mean teacher, we propose the confident mean teacher to increase the accuracy of pseudo-labels from the teacher and train the student with high confidence labels. Experimental results show that the proposed methods achieve 0.470 and 0.692 of PSDS1 and PSDS2 on the DESED real validation dataset.【12】 Deep AHS: A Deep Learning Approach to Acoustic Howling Suppression
标题：深度AHS：一种抑制声学嚎叫的深度学习方法
链接：https://arxiv.org/abs/2302.09252
*与cs.SD语音【17】为同一篇
作者：Hao Zhang,Meng Yu,Dong Yu机构：Tencent AI Lab, Bellevue, WA, USA备注：Accepted for publication in 2023 ICASSP摘要：本文将声啸叫抑制（acoustic howling suppression，AHS）问题描述为一个有监督的学习问题，提出了一种深度学习方法Deep AHS，采用教师强制的方式训练Deep AHS，将周期性的啸叫抑制过程转化为瞬时语音分离过程，以简化问题并加快模型训练。该方法利用合理设计的特征，训练基于注意力的递归神经网络（RNN）从麦克风录音中提取目标信号，从而抑制可能导致啸叫的回放信号。研究了不同的训练策略，并使用递归模式实现的流推理方法来评估所提出的方法的实时啸声抑制性能。Deep AHS避免了啸声检测，并从本质上禁止了啸声的发生，从而在音频系统的设计中实现了更大的灵活性。实验结果验证了该方法在不同场景下对啸声抑制的有效性。摘要：In this paper, we formulate acoustic howling suppression (AHS) as a supervised learning problem and propose a deep learning approach, called Deep AHS, to address it. Deep AHS is trained in a teacher forcing way which converts the recurrent howling suppression process into an instantaneous speech separation process to simplify the problem and accelerate the model training. The proposed method utilizes properly designed features and trains an attention based recurrent neural network (RNN) to extract the target signal from the microphone recording, thus attenuating the playback signal that may lead to howling. Different training strategies are investigated and a streaming inference method implemented in a recurrent mode used to evaluate the performance of the proposed method for real-time howling suppression. Deep AHS avoids howling detection and intrinsically prohibits howling from happening, allowing for more flexibility in the design of audio systems. Experimental results show the effectiveness of the proposed method for howling suppression under different scenarios.【13】 Towards Measuring and Scoring Speaker Diarization Fairness
标题：关于说话人二元化公平性的测量和评分
链接：https://arxiv.org/abs/2302.09991
*与cs.SD语音【1】为同一篇
作者：Yannis Tevissen,Jérôme Boudy,Gérard Chollet,Frédéric Petitpont机构：SAMOVAR, Telecom SudParis, Institut Polytechnique de Paris, France, Newsbridge, France摘要：说话人日记化，或者说寻找“谁在说话，什么时候说话”的任务，现在几乎在每一个语音处理应用中都有使用。然而，由于没有协议逐一研究其偏倚，其公平性尚未得到评估。本文提出了一个协议和一个评分方法来评估说话人对话的公平性。该协议被应用于一个大的口语数据集，并根据说话人的性别、年龄、口音和口语句子的长度报告说话人日记化的性能。当我们应用最先进的说话人日记化方法时，一些由性别或说话人口音引起的偏见被识别出来。摘要：Speaker diarization, or the task of finding "who spoke and when", is now used in almost every speech processing application. Nevertheless, its fairness has not yet been evaluated because there was no protocol to study its biases one by one. In this paper we propose a protocol and a scoring method designed to evaluate speaker diarization fairness. This protocol is applied on a large dataset of spoken utterances and report the performances of speaker diarization depending on the gender, the age, the accent of the speaker and the length of the spoken sentence. Some biases induced by the gender, or the accent of the speaker were identified when we applied a state-of-the-art speaker diarization method.【14】 A Sidecar Separator Can Convert a Single-Speaker Speech Recognition System to a Multi-Speaker One
标题：Sidecar分离器可以将单说话人语音识别系统转换为多说话人语音识别系统
链接：https://arxiv.org/abs/2302.09908
*与cs.SD语音【2】为同一篇
作者：Lingwei Meng,Jiawen Kang,Mingyu Cui,Yuejiao Wang,Xixin Wu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023摘要：虽然自动语音识别（ASR）可以在常见的非重叠环境中很好地执行，但是在多说话人重叠语音识别中保持性能仍然是具有挑战性的。最近的研究表明，ASR模型的编码器在不同的层捕获不同层次的信息--较低的层往往具有更多的声学信息，而较高的层具有更多的语言信息。这启发了我们开发一种侧车分离器，通过分离嵌入在两个合适层之间的混合语音，使训练良好的ASR模型能够用于多说话人场景。我们使用基于wav 2 vec 2.0的ASR模型进行了试验，该模型安装了一个侧车。通过冻结原始模型的参数并且仅训练Sidecar（8.7M，所有参数的8.4%），所提出的方法对于2说话人混合的LibriMix数据集以较大的幅度优于先前的现有技术，达到10.36%的词错误率（WER）;并且在有限训练时获得了与LibriSpeechMix数据集相当的结果（7.56%）。摘要：Although automatic speech recognition (ASR) can perform well in common non-overlapping environments, sustaining performance in multi-speaker overlapping speech recognition remains challenging. Recent research revealed that ASR model's encoder captures different levels of information with different layers -- the lower layers tend to have more acoustic information, and the upper layers more linguistic. This inspires us to develop a Sidecar separator to empower a well-trained ASR model for multi-speaker scenarios by separating the mixed speech embedding between two suitable layers. We experimented with a wav2vec 2.0-based ASR model with a Sidecar mounted. By freezing the parameters of the original model and training only the Sidecar (8.7 M, 8.4% of all parameters), the proposed approach outperforms the previous state-of-the-art by a large margin for the 2-speaker mixed LibriMix dataset, reaching a word error rate (WER) of 10.36%; and obtains comparable results (7.56%) for LibriSpeechMix dataset when limited training.【15】 Knowledge-aware Bayesian Co-attention for Multimodal Emotion Recognition
标题：基于知识感知的贝叶斯共注意多通道情绪识别
链接：https://arxiv.org/abs/2302.09856
*与cs.SD语音【3】为同一篇
作者：Zihan Zhao,Yu Wang,Yanfeng Wang机构：Cooperative Medianet Innovation Center, Shanghai Jiao Tong University ,Shanghai AI Laboratory备注：Accepted to IEEE ICASSP 2023摘要：多模态情感识别是一个具有挑战性的研究领域，旨在融合不同的模态来预测人类的情感。然而，大多数基于注意机制的现有模型难以独立地学习情绪相关部分。为了解决这个问题，我们提出将外部情绪相关知识加入到基于共同注意的预训练模型融合中。为了有效地结合这些知识，我们用贝叶斯注意模块（BAM）来增强共同注意模型，其中使用情绪相关知识来估计先验分布。在IEMOCAP数据集上的实验结果表明，该方法的未加权准确率（unweighted accuracy，UA）优于几种现有方法至少0.7%。摘要：Multimodal emotion recognition is a challenging research area that aims to fuse different modalities to predict human emotion. However, most existing models that are based on attention mechanisms have difficulty in learning emotionally relevant parts on their own. To solve this problem, we propose to incorporate external emotion-related knowledge in the co-attention based fusion of pre-trained models. To effectively incorporate this knowledge, we enhance the co-attention model with a Bayesian attention module (BAM) where a prior distribution is estimated using the emotion-related knowledge. Experimental results on the IEMOCAP dataset show that the proposed approach can outperform several state-of-the-art approaches by at least 0.7% unweighted accuracy (UA).【16】 Emphasizing Unseen Words: New Vocabulary Acquisition for End-to-End Speech Recognition
标题：强调未见词：端到端语音识别的新词汇习得
链接：https://arxiv.org/abs/2302.09723
*与cs.SD语音【4】为同一篇
作者：Leyuan Qu,Cornelius Weber,Stefan Wermter机构：Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany, Department of Artificial Intelligence, Zhejiang Lab, Hangzhou, China备注：Accepted by Neural Networks摘要：由于人类语言的动态性，自动语音识别系统需要不断地获取新词汇。词汇表外（OOV）单词，诸如趋势单词和新命名实体，给现代ASR系统带来了问题，现代ASR系统需要长的训练时间来适应它们的大量参数。与以往大多数研究侧重于语言模型后处理不同，本文从语言模型后处理的角度出发，消除声学建模中的偏差，从声学角度识别OOV词汇。我们建议使用文语转换系统来产生OOV字词，并重新调整损失以鼓励神经网路更多地注意OOV字词。具体地，当在合成音频上微调先前训练的模型时，我们扩大了用于训练包含OOV词的话语的神经网络参数的分类损失（句子级），或者重新缩放用于OOV词的反向传播的梯度（词级）。为了克服灾难性遗忘，我们还探索了损失重标度和模型正则化的组合，即L2正则化和弹性权重固结（EWC）。在LibriSpeech基准测试上的实验结果表明，与以往仅使用EWC对合成音频进行微调的方法相比，本文提出的损失重标方法能够显著提高召回率，而单词错误率只有轻微的下降。此外，词级重标度比话语级重标度更稳定，在OOV词识别中具有更高的召回率和准确率。此外，我们提出的组合损失重标度和权重合并方法可以支持ASR系统的持续学习。摘要：Due to the dynamic nature of human language, automatic speech recognition (ASR) systems need to continuously acquire new vocabulary. Out-Of-Vocabulary (OOV) words, such as trending words and new named entities, pose problems to modern ASR systems that require long training times to adapt their large numbers of parameters. Different from most previous research focusing on language model post-processing, we tackle this problem on an earlier processing level and eliminate the bias in acoustic modeling to recognize OOV words acoustically. We propose to generate OOV words using text-to-speech systems and to rescale losses to encourage neural networks to pay more attention to OOV words. Specifically, we enlarge the classification loss used for training neural networks' parameters of utterances containing OOV words (sentence-level), or rescale the gradient used for back-propagation for OOV words (word-level), when fine-tuning a previously trained model on synthetic audio. To overcome catastrophic forgetting, we also explore the combination of loss rescaling and model regularization, i.e. L2 regularization and elastic weight consolidation (EWC). Compared with previous methods that just fine-tune synthetic audio with EWC, the experimental results on the LibriSpeech benchmark reveal that our proposed loss rescaling approach can achieve significant improvement on the recall rate with only a slight decrease on word error rate. Moreover, word-level rescaling is more stable than utterance-level rescaling and leads to higher recall rates and precision on OOV word recognition. Furthermore, our proposed combined loss rescaling and weight consolidation methods can support continual learning of an ASR system.【17】 SSVMR: Saliency-based Self-training for Video-Music Retrieval
标题：SSVMR：基于显著度的视频音乐检索自我训练
链接：https://arxiv.org/abs/2302.09328
*与cs.SD语音【5】为同一篇
作者：Xuxin Cheng,Zhihong Zhu,Hongxiang Li,Yaowei Li,Yuexian Zou机构：ADSPLAB, School of ECE, Peking University备注：Accepted by ICASSP 2023摘要：随着短视频的兴起，为视频选择合适的背景音乐（BGM）的需求显著增加，视频-音乐检索（VMR）任务逐渐受到研究界的关注。与其他跨模态学习任务一样，现有的VMR方法通常试图在特征空间中度量视频和音乐之间的相似性。然而，它们（1）忽略了不可避免的标签噪声;（2）忽视增强关键视频片段的捕捉能力。本文提出了一种新的基于显著性的自训练框架SSVMR。具体而言，我们首先探索充分利用训练数据集所包含的信息，通过应用半监督方法来抑制标签噪声问题的不利影响，其中采用了自训练方法。此外，我们提出了通过在跨度级混合两个视频并保持两个原始视频的局部性来捕获视频的显著性。受自然语言处理中的反向翻译的启发，我们还进行了反向检索以获得更多的训练数据。在MVD数据集上的实验结果表明，SSVMR在R@1方面较之前的最佳模型提高了34.8%，大幅度地达到了最佳性能.摘要：With the rise of short videos, the demand for selecting appropriate background music (BGM) for a video has increased significantly, video-music retrieval (VMR) task gradually draws much attention by research community. As other cross-modal learning tasks, existing VMR approaches usually attempt to measure the similarity between the video and music in the feature space. However, they (1) neglect the inevitable label noise; (2) neglect to enhance the ability to capture critical video clips. In this paper, we propose a novel saliency-based self-training framework, which is termed SSVMR. Specifically, we first explore to fully make use of the information containing in the training dataset by applying a semi-supervised method to suppress the adverse impact of label noise problem, where a self-training approach is adopted. In addition, we propose to capture the saliency of the video by mixing two videos at span level and preserving the locality of the two original videos. Inspired by back translation in NLP, we also conduct back retrieval to obtain more training data. Experimental results on MVD dataset show that our SSVMR achieves the state-of-the-art performance by a large margin, obtaining a relative improvement of 34.8% over the previous best model in terms of R@1.【18】 Cost-effective Models for Detecting Depression from Speech
标题：从语音中检测抑郁的低成本模型
链接：https://arxiv.org/abs/2302.09214
*与cs.SD语音【6】为同一篇
作者：Mashrura Tasnim,Jekaterina Novikova机构：Winterlight Labs, Toronto, Canada备注：Accepted to ICMLA 2022摘要：抑郁症是最常见的心理障碍，被认为是世界范围内残疾和自杀的主要原因。一个能够检测人类语言中抑郁症迹象的自动化系统可以有助于确保为患有抑郁症的个人提供及时有效的心理健康护理。开发这样的自动化系统需要精确的机器学习模型，能够捕捉抑郁症的迹象。然而，基于深层声学表示的最新模型需要丰富的数据、细致的特征选择和严格的训练;该过程涉及巨大的计算资源。在这项工作中，我们探讨了两个不同的声学特征组-传统的手工策划和深度表征特征，从语音预测抑郁症的严重程度的有效性。我们探讨了可能的影响因素的相关性模型的表现，包括性别的个人，严重程度的障碍，内容和长度的讲话。我们的研究结果表明，与深度表征特征训练模型相比，传统声学特征训练模型表现得同样好或更好，但计算成本明显更低，而与其他因素无关，如内容和言语长度、说话者性别和障碍严重程度。这使得此类模型更适合于计算资源可用性受限的部署，例如智能设备中的实时抑郁症监测应用。摘要：Depression is the most common psychological disorder and is considered as a leading cause of disability and suicide worldwide. An automated system capable of detecting signs of depression in human speech can contribute to ensuring timely and effective mental health care for individuals suffering from the disorder. Developing such automated system requires accurate machine learning models, capable of capturing signs of depression. However, state-of-the-art models based on deep acoustic representations require abundant data, meticulous selection of features, and rigorous training; the procedure involves enormous computational resources. In this work, we explore the effectiveness of two different acoustic feature groups - conventional hand-curated and deep representation features, for predicting the severity of depression from speech. We explore the relevance of possible contributing factors to the models' performance, including gender of the individual, severity of the disorder, content and length of speech. Our findings suggest that models trained on conventional acoustic features perform equally well or better than the ones trained on deep representation features at significantly lower computational cost, irrespective of other factors, e.g. content and length of speech, gender of the speaker and severity of the disorder. This makes such models a better fit for deployment where availability of computational resources is restricted, such as real time depression monitoring applications in smart devices.【19】 Exposing AI-Synthesized Human Voices Using Neural Vocoder Artifacts
标题：利用神经声码器伪影显示人工智能合成的人声
链接：https://arxiv.org/abs/2302.09198
*与cs.SD语音【7】为同一篇
作者：Chengzhe Sun,Shan Jia,Shuwei Hou,Ehab AlBadawy,Siwei Lyu机构：Department of Computer Science and Engineering, University at Buffalo, SUNY, Buffalo, USA, Department of Electrical and Computer Engineering, University at Albany, SUNY, Albany, USA摘要：人工智能合成人类声音的进步带来了越来越大的假冒和虚假信息的威胁。因此，研究合成人声的检测方法具有重要的现实意义。本文提出了一种新的基于识别音频信号中神经声码器伪像的合成人声检测方法。神经声码器是特别设计的神经网络，其合成来自时间-频率表示的波形，梅尔谱图。神经声码器是大多数DeepFake音频合成模型中的核心组件。因此，神经声码器处理的识别意味着音频样本可能已经被合成。为了利用声码器伪像进行合成人声检测，我们引入了一个用于二进制类RawNet 2模型的多任务学习框架，该框架与声码器识别模块共享前端特征提取器。我们将声码器辨识当作一个借口任务，以限制前端的特征撷取器专注于声码器伪像，并为最后的二值分类器提供可辨识的特征。实验结果表明，基于声码器识别的改进RawNet 2模型对二进制任务具有较高的分类性能。摘要：The advancements of AI-synthesized human voices have introduced a growing threat of impersonation and disinformation. It is therefore of practical importance to developdetection methods for synthetic human voices. This work proposes a new approach to detect synthetic human voices based on identifying artifacts of neural vocoders in audio signals. A neural vocoder is a specially designed neural network that synthesizes waveforms from temporal-frequency representations, e.g., mel-spectrograms. The neural vocoder is a core component in most DeepFake audio synthesis models. Hence the identification of neural vocoder processing implies that an audio sample may have been synthesized. To take advantage of the vocoder artifacts for synthetic human voice detection, we introduce a multi-task learning framework for a binary-class RawNet2 model that shares the front-end feature extractor with a vocoder identification module. We treat the vocoder identification as a pretext task to constrain the front-end feature extractor to focus on vocoder artifacts and provide discriminative features for the final binary classifier. Our experiments show that the improved RawNet2 model based on vocoder identification achieves an overall high classification performance on the binary task.机器翻译，仅供参考
每日语音速递