今日论文合集：cs.SD语音21篇，eess.AS音频处理22篇。
【1】 Dynamic Kernels and Channel Attention with Multi-Layer Embedding Aggregation for Speaker Verification
标题：多层嵌入聚合的动态核函数和通道注意力用于说话人确认
链接：https://arxiv.org/abs/2211.02000
作者：Anna Ollerenshaw,Md Asif Jalal,Thomas Hain机构：Speech and Hearing Group, The University of Sheffield, Sheffield, UK备注：Submitted to ICASSP 2023摘要：现有技术的说话人确认框架通常集中于具有越来越深（更多层）和越来越宽（通道数量）的模型的语音增强技术，以提高其确认性能。相反，本文提出了一种在卷积神经网络中使用基于注意力的动态核来提高模型分辨率能力的方法，以适应特征条件下的模型参数。通过信道关注度和多层特征聚合进一步提取核上的关注度权重，以学习语音的全局特征。该方法为利用较少的数据资源提高表示能力提供了一种有效的解决方案。这是由于对模型参数的结构的输入的自适应。该模型在VoxCeleb 1测试集上的EER和miniDCF分别为1.62%和0.18%，与ECAPA-TDNN相比有17%的相对改善.摘要：State-of-the-art speaker verification frameworks have typically focused on speech enhancement techniques with increasingly deeper (more layers) and wider (number of channels) models to improve their verification performance. Instead, this paper proposes an approach to increase the model resolution capability using attention-based dynamic kernels in a convolutional neural network to adapt the model parameters to be feature-conditioned. The attention weights on the kernels are further distilled by channel attention and multi-layer feature aggregation to learn global features from speech. This approach provides an efficient solution to improving representation capacity with lower data resources. This is due to the self-adaptation to inputs of the structures of the model parameters. The proposed dynamic convolutional model achieved 1.62\% EER and 0.18 miniDCF on the VoxCeleb1 test set and has a 17\% relative improvement compared to the ECAPA-TDNN.
【2】 Probing Statistical Representations For End-To-End ASR
标题：探讨端到端ASR的统计表示法
链接：https://arxiv.org/abs/2211.01993
作者：Anna Ollerenshaw,Md Asif Jalal,Thomas Hain机构：Speech and Hearing Group, The University of Sheffield, Sheffield, UK备注：Submitted to ICASSP 2023摘要：端到端自动语音识别（ASR）模型旨在学习广义语音表示以执行识别。在这个领域中，很少有研究来分析内部表示依赖性及其与建模方法的关系。本文使用SVCCA研究了Transformer体系结构中的跨域语言模型依赖性，并使用这些见解来开发建模方法。研究发现，Transformer层内的特定神经表征表现出影响识别性能的相关行为。 总之，这项工作提供了对影响上下文依赖性和ASR性能的建模方法的分析，并可用于创建或调整性能更好的端到端ASR模型，也可用于下游任务。摘要：End-to-End automatic speech recognition (ASR) models aim to learn a generalised speech representation to perform recognition. In this domain there is little research to analyse internal representation dependencies and their relationship to modelling approaches. This paper investigates cross-domain language model dependencies within transformer architectures using SVCCA and uses these insights to exploit modelling approaches. It was found that specific neural representations within the transformer layers exhibit correlated behaviour which impacts recognition performance. Altogether, this work provides analysis of the modelling approaches affecting contextual dependencies and ASR performance, and can be used to create or adapt better performing End-to-End ASR models and also for downstream tasks.
【3】 MarginNCE: Robust Sound Localization with a Negative Margin
标题：MargiNCE：具有负边缘的稳健声音定位
链接：https://arxiv.org/abs/2211.01966
作者：Sooyoung Park,Arda Senocak,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, South Korea, Electronics and Telecommunications Research Institute, South Korea备注：Submitted to ICASSP 2023. SOTA performance in Audio-Visual Sound Localization. 5 Pages摘要：本研究的目的是利用一种自我监督的方法来定位视觉场景中的声源。声源定位背景下的对比学习利用了音频和视频信号之间的自然对应关系，其中来自相同源的音频-视频对被假定为正，而随机选择的对是负的。然而，这种方法带来了噪声对应;例如可能彼此不相关的正音频和视频对信号，或者可能包含与正信号语义相似的样本的负对。我们的主要贡献是证明了在对比学习中使用一个不太严格的决策边界可以减轻噪声对应对声源定位的影响。我们提出了一种简单而有效的方法，通过用负的边缘稍微修改对比损失。大量的实验结果表明，该方法的性能与现有的方法相当甚至更好.此外，我们证明了在现有方法中引入负裕度会导致性能的一致提高。摘要：The goal of this work is to localize sound sources in visual scenes with a self-supervised approach. Contrastive learning in the context of sound source localization leverages the natural correspondence between audio and visual signals where the audio-visual pairs from the same source are assumed as positive, while randomly selected pairs are negatives. However, this approach brings in noisy correspondences; for example, positive audio and visual pair signals that may be unrelated to each other, or negative pairs that may contain semantically similar samples to the positive one. Our key contribution in this work is to show that using a less strict decision boundary in contrastive learning can alleviate the effect of noisy correspondences in sound source localization. We propose a simple yet effective approach by slightly modifying the contrastive loss with a negative margin. Extensive experimental results show that our approach gives on-par or better performance than the state-of-the-art methods. Furthermore, we demonstrate that the introduction of a negative margin to existing methods results in a consistent improvement in performance.
【4】 HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks
标题：HyperSound：利用超网络生成音频信号的隐式神经表示
链接：https://arxiv.org/abs/2211.01839
作者：Filip Szatkowski,Karol J. Piczak,Przemysław Spurek,Jacek Tabor,Tomasz Trzciński机构：Tomasz Trzci´nski, Warsaw University of Technology, IDEAS NCBR, Tooploox, Jagiellonian University摘要：内隐神经表征（inplicit neural representation，INR）是一个快速发展的研究领域，它为多媒体信号的表征提供了一种新的方法。INR的最新应用包括图像超分辨率、高维信号的压缩或3D渲染。然而，这些解决方案通常集中于视觉数据，并且使它们适应于音频域并非是微不足道的。此外，它需要针对每个数据样本单独训练的模型。为了解决这一问题，我们提出了一种元学习方法--HyperSound，它利用超网络来产生训练时不可见的音频信号的INR。我们表明，我们的方法可以重建声波的质量可比的其他国家的最先进的模型。摘要：Implicit neural representations (INRs) are a rapidly growing research field, which provides alternative ways to represent multimedia signals. Recent applications of INRs include image super-resolution, compression of high-dimensional signals, or 3D rendering. However, these solutions usually focus on visual data, and adapting them to the audio domain is not trivial. Moreover, it requires a separately trained model for every data sample. To address this limitation, we propose HyperSound, a meta-learning method leveraging hypernetworks to produce INRs for audio signals unseen at training time. We show that our approach can reconstruct sound waves with quality comparable to other state-of-the-art models.
【5】 Iterative autoregression: a novel trick to improve your low-latency speech enhancement model
标题：迭代自回归：改进低延迟语音增强模型的新技巧
链接：https://arxiv.org/abs/2211.01751
作者：Pavel Andreev,Nicholas Babaev,Azat Saginbaev,Ivan Shchekotov机构：Samsung AI Center (Moscow)备注：Submitted to ICASSP 2023摘要：流模型是实时语音增强工具的重要组成部分。流传输机制将语音增强模型限制为仅使用未来信息的微小上下文，因此，低等待时间流传输设置通常被认为是具有挑战性的，并且对模型质量具有显著的负面影响。然而，由于流生成的顺序性质，它提供了自回归的自然可能性，即，在进行当前预测时使用以前的预测。本文提出了一种简单有效的自回归低时延语音增强模型的训练方法。我们证明了所提出的技术在不同的体系结构和训练场景中都能带来稳定的改进。摘要：Streaming models are an essential component of real-time speech enhancement tools. The streaming regime constrains speech enhancement models to use only a tiny context of future information, thus, the low-latency streaming setup is generally assumed to be challenging and has a significant negative effect on the model quality. However, due to the sequential nature of streaming generation, it provides a natural possibility for autoregression, i.e., using previous predictions when making current ones. In this paper, we present a simple, yet effective trick for training of autoregressive low-latency speech enhancement models. We demonstrate that the proposed technique leads to stable improvement across different architectures and training scenarios.
【6】 Hybrid-SD ($\text{H}_{\text{SD}}$) : A new hybrid evaluation metric for automatic speech recognition tasks
标题：混合SD($ext{H}_{ext{SD}}$)：一种新的用于自动语音识别任务的混合评价指标
链接：https://arxiv.org/abs/2211.01722
作者：Zitha Sasindran,Harsha Yelchuri,Supreeth Rao,T. V. Prabhakar机构：Department of Electronic Systems Engineering, Indian Institute of Science, Bengaluru, India,.摘要：许多研究已经检验了单词错误率（WER）作为自动语音识别（ASR）系统的评估度量的缺点，特别是当用于口语理解任务（例如意图识别和对话系统）时。本文提出了一种新的综合考虑语义正确性和错误率的混合评估方法Hybrid-SD（$\text{H}_{\text{SD}}$）.为了生成句子相异度得分（SD），我们利用提取技术建立了一个快速、轻量级的SNanoBERT模型.我们的实验表明，SNanoBERT模型比SRoBERTa小25.9倍，快38.8倍，同时在著名的基准测试中获得了相当的结果。因此，它适合在边缘设备上部署ASR模型。我们还表明，$\text{H}_{\text{SD}}$与下游任务（如意图识别和命名实体识别（NER））的相关性更强。摘要：Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems, particularly when used for spoken language understanding tasks such as intent recognition and dialogue systems. In this paper, we propose Hybrid-SD ($\text{H}_{\text{SD}}$), a new hybrid evaluation metric for ASR systems that takes into account both semantic correctness and error rate. To generate sentence dissimilarity scores (SD), we built a fast and lightweight SNanoBERT model using distillation techniques. Our experiments show that the SNanoBERT model is 25.9x smaller and 38.8x faster than SRoBERTa while achieving comparable results on well-known benchmarks. Hence, making it suitable for deploying with ASR models on edge devices. We also show that $\text{H}_{\text{SD}}$ correlates more strongly with downstream tasks such as intent recognition and named-entity recognition (NER).
【7】 The ISCSLP 2022 Intelligent Cockpit Speech Recognition Challenge (ICSRC): Dataset, Tracks, Baseline and Results
标题：ISCSLP 2022智能驾驶舱语音识别挑战赛(ICSRC)：数据集、轨迹、基线和结果
链接：https://arxiv.org/abs/2211.01585
作者：Ao Zhang,Fan Yu,Kaixun Huang,Lei Xie,Longbiao Wang,Eng Siong Chng,Hui Bu,Binbin Zhang,Wei Chen,Xin Xu机构：Northwestern Polytechnical University, Xi’an, China, Tianjin Key Laboratory of Cognitive Computing and Application,College of Intelligence and, Computing, Tianjin University, Tianjin, China备注：Accepted by ISCSLP2022摘要：本文总结了ISCSLP 2022智能驾驶舱语音识别挑战赛（ICSRC）的结果。首先阐述了这一挑战的必要性，然后介绍了从一辆新能源汽车（NEV）上收集的相关数据集，涵盖了各种驾驶舱声学条件和语言内容。然后，我们描述了轨道布置和基线系统。具体而言，我们根据允许的模型/系统大小设置了两个跟踪，分别针对车载嵌入式系统和云ASR系统，以研究资源受限和资源不受限的设置。最后，我们总结了挑战结果，并提供了来自提交系统的主要观察结果。摘要：This paper summarizes the outcomes from the ISCSLP 2022 Intelligent Cockpit Speech Recognition Challenge (ICSRC). We first address the necessity of the challenge and then introduce the associated dataset collected from a new-energy vehicle (NEV) covering a variety of cockpit acoustic conditions and linguistic contents. We then describe the track arrangement and the baseline system. Specifically, we set up two tracks in terms of allowed model/system size to investigate resource-constrained and -unconstrained setups, targeting to vehicle embedded as well as cloud ASR systems respectively. Finally we summarize the challenge results and provide the major observations from the submitted systems.
【8】 Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing
标题：损失可能是福：将自我监督的语音表示转向高效的多语言和多任务语音处理
链接：https://arxiv.org/abs/2211.01522
作者：Yonggan Fu,Yang Zhang,Kaizhi Qian,Zhifan Ye,Zhongzhi Yu,Cheng-I Lai,Yingyan Lin机构：Georgia Institute of Technology,MIT-IBM Watson AI Lab,Rice University,MIT CSAIL备注：Accepted at NeurIPS 2022摘要：针对丰富语音表示的自监督学习（SSL）已经在低资源自动语音识别（ASR）和其他语音处理任务中取得了经验上的成功，这可以减轻对大量转录语音的需要，并因此推动了对设备上ASR和其他语音处理的日益增长的需求。然而，高级语音SSL模型已经变得越来越大，这与有限的设备上资源相矛盾。在需要同时识别多种语言或执行多个语音处理任务的多语言/多任务场景中，这种差距可能更严重。此外，当在低资源语音语料库上进行微调时，强过参数化的语音SSL模型容易遭受过拟合。本文旨在通过我们提出的S$^3$-Router框架来提高语音SSL模型的实际应用，从而在提高效率和缓解过拟合方面实现双赢。该框架首次发现，通过对语音SSL模型的模型连接进行微调，只需丢弃不超过10%的模型权重，就可以在下游语音处理任务中获得比标准权重微调更好的精度。更重要的是，S$^3$-Router可以作为一种一体化技术，实现（1）一种新的微调方案，（2）一种有效的多语言/多任务解决方案，（3）一种最先进的ASR修剪技术，以及（4）一种定量分析学习到的语音表示的新工具。我们相信S$^3$-Router为语音SSL模型的实际部署提供了一个新的视角。我们的代码可从以下网址获得：https://github.com/GATECH-EIC/S3-Router。摘要：Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S$^3$-Router framework, which for the first time discovers that simply discarding no more than 10\% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S$^3$-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art ASR pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. We believe S$^3$-Router has provided a new perspective for practical deployment of speech SSL models. Our codes are available at: https://github.com/GATECH-EIC/S3-Router.
【9】 Towards Zero-Shot Code-Switched Speech Recognition
标题：迈向零发送码切换语音识别
链接：https://arxiv.org/abs/2211.01458
作者：Brian Yan,Matthew Wiesner,Ondrej Klejch,Preethi Jyothi,Shinji Watanabe机构：Carnegie Mellon University, US,Johns Hopkins University, US, University of Edinburgh, UK,Indian Institute of Technology Bombay, IN摘要：在此工作中，我们寻求建立有效的码切换（CS）自动语音识别系统（ASR）在zero-shot设置下，没有转录的CS语音数据可用于训练。先前提出的框架有条件地将双语任务分解成其组成的单语部分，是有效利用单语数据的一个有希望的起点。然而，这些方法需要单语模块来执行语言分割。也就是说，每个单语模块必须同时检测CS点并转录一种语言的语音片段，而忽略其他语言的语音片段--这不是一项微不足道的任务。我们建议简化每个单语模块，允许它们用单语脚本不加区别地转录所有语音片段（即音译）。这种简单的修改将CS点检测的责任传递给随后的双语模块，所述双语模块通过考虑多个单语音译以及外部语言模型信息来确定最终输出。我们将这种基于音译的方法应用于一个端到端的可微分神经网络，并在中英文SEAME测试集上证明了它对zero-shot CS ASR的有效性。摘要：In this work, we seek to build effective code-switched (CS) automatic speech recognition systems (ASR) under the zero-shot setting where no transcribed CS speech data is available for training. Previously proposed frameworks which conditionally factorize the bilingual task into its constituent monolingual parts are a promising starting point for leveraging monolingual data efficiently. However, these methods require the monolingual modules to perform language segmentation. That is, each monolingual module has to simultaneously detect CS points and transcribe speech segments of one language while ignoring those of other languages -- not a trivial task. We propose to simplify each monolingual module by allowing them to transcribe all speech segments indiscriminately with a monolingual script (i.e. transliteration). This simple modification passes the responsibility of CS point detection to subsequent bilingual modules which determine the final output by considering multiple monolingual transliterations along with external language model information. We apply this transliteration-based approach in an end-to-end differentiable neural network and demonstrate its efficacy for zero-shot CS ASR on Mandarin-English SEAME test sets.
【10】 Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing
标题：基于注意通道相关和标签平滑的自监督模型语音情感识别
链接：https://arxiv.org/abs/2211.01756
作者：Sofoklis Kakouros,Themos Stafylakis,Ladislav Mosner,Lukas Burget机构：University of Helsinki, Finland, Omilia - Conversational Intelligence, Athens, Greece备注：Submitted to IEEE-ICASSP 2023摘要：从语音中识别情感时，我们会遇到两个常见的问题：如何最佳地从语音信号中捕获情感相关信息以及如何最佳地量化或分类有噪声的主观情感标签。自监督的预训练表示可以鲁棒地从语音中捕获信息，使得在包括情感识别在内的许多下游任务中能够获得最先进的结果。然而，需要考虑在时间上聚集信息的更好方式，因为相关情感信息可能分段出现，并且在信号上不均匀。对于标签，我们需要考虑到存在来自主观人类注释的相当程度的噪声。本文提出了一种新的注意力池方法，该方法基于表征系数之间的相关性并结合标签平滑，旨在降低分类器对训练标签的置信度。我们在IEMOCAP基准数据集上对我们提出的方法进行了评估，并证明了我们的方法比文献中的方法具有更高的性能。可在www.example.com上找到重现结果的代码github.com/skakouros/s3prl_attentive_correlation。摘要：When recognizing emotions from speech, we encounter two common problems: how to optimally capture emotion-relevant information from the speech signal and how to best quantify or categorize the noisy subjective emotion labels. Self-supervised pre-trained representations can robustly capture information from speech enabling state-of-the-art results in many downstream tasks including emotion recognition. However, better ways of aggregating the information across time need to be considered as the relevant emotion information is likely to appear piecewise and not uniformly across the signal. For the labels, we need to take into account that there is a substantial degree of noise that comes from the subjective human annotations. In this paper, we propose a novel approach to attentive pooling based on correlations between the representations' coefficients combined with label smoothing, a method aiming to reduce the confidence of the classifier on the training labels. We evaluate our proposed approach on the benchmark dataset IEMOCAP, and demonstrate high performance surpassing that in the literature. The code to reproduce the results is available at github.com/skakouros/s3prl_attentive_correlation.
【11】 Discussion of Features for Acoustic Anomaly Detection under Industrial Disturbing Noise in an End-of-Line Test of Geared Motors
标题：齿轮电机尾线试验中工业干扰噪声下的声学异常检测特征探讨
链接：https://arxiv.org/abs/2211.01716
作者：Peter Wissbrock,David Pelkmann,Yvonne Richter机构：Lenze SE, Innovation, Aerzen, Germany, Institut für Systemdynamik und Mechatronik, Bielefeld University of Applied Sciences, Germany摘要：在减速电机的下线试验中，产品质量的评定是一个重要的环节。由于时间限制和变量的高度多样性，声学测量比振动测量更经济。然而，声学数据受到工业干扰噪声的影响。因此，本研究的目的是研究用于齿轮电机下线测试中异常检测的特征的鲁棒性。通过声阵列记录了具有典型故障和声干扰的真实世界数据集。这包括来自生产的工业噪声和系统产生的干扰，用于比较鲁棒性。总的来说，提出了将从对数包络频谱提取的特征与心理声学特征一起应用。异常检测采用隔离森林或更通用的Bagging随机挖掘器。大多数干扰都可以避免，而使用锤子或气压往往会引起问题。一般而言，这些结果对于基于声学或振动测量的状态监测任务非常重要。此外，还提出了一种真实世界问题的描述，以改进常见的信号处理和机器学习任务。摘要：In the end-of-line test of geared motors, the evaluation of product qual-ity is important. Due to time constraints and the high diversity of variants, acous-tic measurements are more economical than vibration measurements. However, the acoustic data is affected by industrial disturbing noise. Therefore, the aim of this study is to investigate the robustness of features used for anomaly detection in geared motor end-of-line testing. A real-world dataset with typical faults and acoustic disturbances is recorded by an acoustic array. This includes industrial noise from the production and systematically produced disturbances, used to compare the robustness. Overall, it is proposed to apply features extracted from a log-envelope spectrum together with psychoacoustic features. The anomaly de-tection is done by using the isolation forest or the more universal bagging random miner. Most disturbances can be circumvented, while the use of a hammer or air pressure often causes problems. In general, these results are important for condi-tion monitoring tasks that are based on acoustic or vibration measurements. Fur-thermore, a real-world problem description is presented to improve common sig-nal processing and machine learning tasks.
【12】 Cutting Through the Noise: An Empirical Comparison of Psychoacoustic and Envelope-based Features for Machinery Fault Detection
标题：穿透噪声：机械故障检测中心理声学特征和包络特征的实证比较
链接：https://arxiv.org/abs/2211.01704
作者：Peter Wißbrock,Yvonne Richter,David Pelkmann,Zhao Ren,Gregory Palmer机构：Lenze SE, Innovation, Aerzen, Germany, Institut für Systemdynamik und Mechatronik, Bielefeld University of Applied Sciences, Germany, L,S Research Center, Leibniz University of Hannover, Germany摘要：基于声学的故障检测技术在监测机械零件的健康状态方面具有很大的潜力。然而，工业环境的背景噪声可能负面地影响故障检测的性能。对提高故障检测对工业环境噪声的鲁棒性的关注有限。因此，我们提出了伦茨生产背景噪声（LPBN）真实世界数据集和一个自动化和噪声鲁棒的听觉检查（ARAI）系统，用于齿轮电机的生产线末端检查。声阵列用于从具有小故障、大故障或健康的电动机获取数据。提供基准以基于齿轮箱的专家知识将心理声学特征与不同类型的包络特征进行比较。据我们所知，我们是第一个将时变心理声学特征应用于故障检测的公司。我们训练了一个最先进的单类分类器，对健康电机的样本，并分离出故障的一个使用阈值的故障检测。性能最好的方法实现了0.87（对数包络）、0.86（时变心理声学）和0.91（两者的组合）的曲线下面积。摘要：Acoustic-based fault detection has a high potential to monitor the health condition of mechanical parts. However, the background noise of an industrial environment may negatively influence the performance of fault detection. Limited attention has been paid to improving the robustness of fault detection against industrial environmental noise. Therefore, we present the Lenze production background-noise (LPBN) real-world dataset and an automated and noise-robust auditory inspection (ARAI) system for the end-of-line inspection of geared motors. An acoustic array is used to acquire data from motors with a minor fault, major fault, or which are healthy. A benchmark is provided to compare the psychoacoustic features with different types of envelope features based on expert knowledge of the gearbox. To the best of our knowledge, we are the first to apply time-varying psychoacoustic features for fault detection. We train a state-of-the-art one-class-classifier, on samples from healthy motors and separate the faulty ones for fault detection using a threshold. The best-performing approaches achieve an area under curve of 0.87 (logarithm envelope), 0.86 (time-varying psychoacoustics), and 0.91 (combination of both).
【13】 Channel-Aware Pretraining of Joint Encoder-Decoder Self-Supervised Model for Telephonic-Speech ASR
标题：电话-语音ASR中联合编解码器自监督模型的信道感知预训练
链接：https://arxiv.org/abs/2211.01669
作者：Vrunda N. Sukhadia,A. Arunkumar,S. Umesh机构：Speech Lab, Electrical Engineering Department, IIT Madras, India摘要：本文提出了一种新的技术，当使用来自两个不同信道（窄带和宽带）的语音池进行训练时，从联合编解码器自监督模型获得更好的下游ASR性能。联合编码器-解码器自监督模型用Transformer解码器扩展了HuBERT模型。HuBERT执行特征聚类并预测每个输入帧的类别。在简单池中（这是我们的基准），无法识别通道信息。为了合并信道信息，我们已经提出了用于来自不同信道的语音的非重叠聚类ID。我们的方法给出了一个约5%的相对改善联合编码器-解码器自监督模型建立简单的数据池，作为我们的基线。摘要：This paper proposes a novel technique to obtain better downstream ASR performance from a joint encoder-decoder self-supervised model when trained with speech pooled from two different channels (narrow and wide band). The joint encoder-decoder self-supervised model extends the HuBERT model with a Transformer decoder. HuBERT performs clustering of features and predicts the class of every input frame. In simple pooling, which is our baseline, there is no way to identify the channel information. To incorporate channel information, we have proposed non-overlapping cluster IDs for speech from different channels. Our method gives a relative improvement of ~ 5% over the joint encoder-decoder self-supervised model built with simple pooling of data, which serves as our baseline.
【14】 Adversarial Data Augmentation Using VAE-GAN for Disordered Speech Recognition标题：基于VAE-GAN的无序语音识别对抗性数据增强链接：https://arxiv.org/abs/2211.01646
作者：Zengrui Jin,Xurong Xie,Mengzhe Geng,Tianzi Wang,Shujie Hu,Jiajun Deng,Guinan Li,Xunying Liu机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Institute of Software, Chinese Academy of Sciences, China备注：Submitted to ICASSP 2023摘要：无序语音的自动识别一直是一个极具挑战性的课题。潜在的神经运动疾病，通常与同时发生的身体残疾相结合，导致难以收集ASR系统开发所需的大量受损语音。提出了一种基于变分自动编码生成对抗网络（VAE-GAN）的个性化语音增强方法，该方法能够同时学习编码、生成和识别合成的受损语音。导出单独的潜在特征以学习构音障碍语音特征和音素上下文表示。还结合了自监督预训练的Wav 2 vec 2.0嵌入特征。在UASpeech语料库上进行的实验表明，所提出的对抗性数据增强方法在训练混合TDNN和端到端Conformer系统的情况下始终优于基线速度扰动和非VAE GAN增强方法。在LHUC说话者自适应之后，使用基于VAE-GAN的增强的最佳系统在16个构音障碍说话者的UASpeech测试集上产生了27.78%的总体WER，并且在具有“非常低”可懂度的说话者子集上产生了57.31%的最低公布WER。摘要：Automatic recognition of disordered speech remains a highly challenging task to date. The underlying neuro-motor conditions, often compounded with co-occurring physical disabilities, lead to the difficulty in collecting large quantities of impaired speech required for ASR system development. This paper presents novel variational auto-encoder generative adversarial network (VAE-GAN) based personalized disordered speech augmentation approaches that simultaneously learn to encode, generate and discriminate synthesized impaired speech. Separate latent features are derived to learn dysarthric speech characteristics and phoneme context representations. Self-supervised pre-trained Wav2vec 2.0 embedding features are also incorporated. Experiments conducted on the UASpeech corpus suggest the proposed adversarial data augmentation approach consistently outperformed the baseline speed perturbation and non-VAE GAN augmentation methods with trained hybrid TDNN and End-to-end Conformer systems. After LHUC speaker adaptation, the best system using VAE-GAN based augmentation produced an overall WER of 27.78% on the UASpeech test set of 16 dysarthric speakers, and the lowest published WER of 57.31% on the subset of speakers with "Very Low" intelligibility.
【15】 Leveraging Domain Features for Detecting Adversarial Attacks Against Deep Speech Recognition in Noise标题：利用域特征检测噪声中针对深度语音识别的敌意攻击链接：https://arxiv.org/abs/2211.01621
作者：Christian Heider Nielsen,Zheng-Hua Tan机构：Department of Electronic Systems, Aalborg University, Denmark摘要：近年来，基于深度模型的自动语音识别（ASR）技术取得了显著的进展，并在现实生活中得到了广泛的应用。与此同时，针对深度ASR系统的对抗性攻击也非常成功。已经提出了各种方法来保护ASR系统免受这些攻击。然而，现有的基于分类的方法主要集中在深度学习模型的设计上，而缺乏对领域特定特征的挖掘。这项工作利用基于过滤器组的特征来更好地捕获攻击的特征，以改进检测。此外，本文还分析了分别使用语音和非语音部分检测对抗性攻击的潜力。最后，考虑到ASR系统可能部署的不利环境，我们研究了各种类型的声学噪声和信噪比的影响。大量实验表明，逆滤波器组特征在干净和噪声环境下都有较好的检测性能，无论是语音还是非语音部分都能有效地进行检测，而声学噪声会大大降低检测性能。摘要：In recent years, significant progress has been made in deep model-based automatic speech recognition (ASR), leading to its widespread deployment in the real world. At the same time, adversarial attacks against deep ASR systems are highly successful. Various methods have been proposed to defend ASR systems from these attacks. However, existing classification based methods focus on the design of deep learning models while lacking exploration of domain specific features. This work leverages filter bank-based features to better capture the characteristics of attacks for improved detection. Furthermore, the paper analyses the potentials of using speech and non-speech parts separately in detecting adversarial attacks. In the end, considering adverse environments where ASR systems may be deployed, we study the impact of acoustic noise of various types and signal-to-noise ratios. Extensive experiments show that the inverse filter bank features generally perform better in both clean and noisy environments, the detection is effective using either speech or non-speech part, and the acoustic noise can largely degrade the detection performance.
【16】 Convolution channel separation and frequency sub-bands aggregation for music genre classification标题：用于音乐流派分类的卷积通道分离和频率子带聚合链接：https://arxiv.org/abs/2211.01599
作者：Jungwoo Heo,Hyun-seo Shin,Ju-ho Kim,Chan-yeong Lim,Ha-Jin Yu机构：School of Computer Science, University of Seoul摘要：在音乐中，音高和节奏等短期特征构成旋律和叙事等长期语义特征。音乐流派分类（MGC）系统应该能够分析这些特征。在本研究中，我们提出了一个新的架构，可以提取和聚合的短期和长期的特征层次。我们的框架基于ECAPA-TDNN，其中由于反向传播训练，提取短期特征的所有层都受到提取长期特征的层的影响。为了防止短期特征的失真，我们设计了卷积通道分离技术，将短期特征与长期特征提取路径分离。为了从我们的框架中提取更多的特征，我们引入了频率子带聚合方法，该方法沿着频率带宽划分输入谱图并处理每个段。我们使用Melon Playlist数据集对我们的框架进行了评估，该数据集是一个大规模数据集，包含的数据比MGC研究中广泛使用的数据集GTZAN多600倍。结果表明，该框架的准确率达到了70.4%，比传统框架提高了16.9%。摘要：In music, short-term features such as pitch and tempo constitute long-term semantic features such as melody and narrative. A music genre classification (MGC) system should be able to analyze these features. In this research, we propose a novel framework that can extract and aggregate both short- and long-term features hierarchically. Our framework is based on ECAPA-TDNN, where all the layers that extract short-term features are affected by the layers that extract long-term features because of the back-propagation training. To prevent the distortion of short-term features, we devised the convolution channel separation technique that separates short-term features from long-term feature extraction paths. To extract more diverse features from our framework, we incorporated the frequency sub-bands aggregation method, which divides the input spectrogram along frequency bandwidths and processes each segment. We evaluated our framework using the Melon Playlist dataset which is a large-scale dataset containing 600 times more data than GTZAN which is a widely used dataset in MGC studies. As the result, our framework achieved 70.4% accuracy, which was improved by 16.9% compared to a conventional framework.
【17】 Phonetic-assisted Multi-Target Units Modeling for Improving Conformer-Transducer ASR system标题：改进变压器-变送器ASR系统的语音辅助多目标单元建模链接：https://arxiv.org/abs/2211.01571
作者：Li Li,Dongxing Xu,Haoran Wei,Yanhua Long机构：Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai Normal University, Shanghai, China, Unisound AI Technology Co., Ltd., Beijing, China, Department of ECE, University of Texas at Dallas, Richardson, TX , USA备注：5 pages, 1 figures, submitted to ICASSP 2023摘要：在端到端的自动语音识别（ASR）中，开发有效的目标建模单元是非常重要的，也是一直关注的问题。本文提出一种语音辅助的多目标单元（PMU）建模方法，以渐进式表征学习的方式增强Conformer-Transducer ASR系统。具体来说，PMU首先使用语音辅助子词建模（PASM）和字节对编码（BPE）分别产生语音诱导和文本诱导的目标单元;在此基础上，提出了三种新的声学编码器增强框架，包括基本PMU、paraCTC和pcaCTC，它们集成了不同层次的PASM和BPE单元，用于CTC和传感器多任务训练。在LibriSpeech和重音ASR测试集上的实验结果表明，与传统的BPE相比，本文提出的PMU显著降低了LibriSpeech clean、other和6个重音ASR测试集的WER，分别降低了12.7%、6.0%和7.7%。摘要：Exploiting effective target modeling units is very important and has always been a concern in end-to-end automatic speech recognition (ASR). In this work, we propose a phonetic-assisted multi-target units (PMU) modeling approach, to enhance the Conformer-Transducer ASR system in a progressive representation learning manner. Specifically, PMU first uses the pronunciation-assisted subword modeling (PASM) and byte pair encoding (BPE) to produce phonetic-induced and text-induced target units separately; Then, three new frameworks are investigated to enhance the acoustic encoder, including a basic PMU, a paraCTC and a pcaCTC, they integrate the PASM and BPE units at different levels for CTC and transducer multi-task training. Experiments on both LibriSpeech and accented ASR tasks show that, the proposed PMU significantly outperforms the conventional BPE, it reduces the WER of LibriSpeech clean, other, and six accented ASR testsets by relative 12.7%, 6.0% and 7.7%, respectively.
【18】 SLICER: Learning universal audio representations using low-resource self-supervised pre-training标题：切片器：使用低资源自我监督预训练学习通用音频表示链接：https://arxiv.org/abs/2211.01519
作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha机构：University of Maryland, College Park, USA, Speech Lab, Department of Electrical Engineering, IIT Madras, Chennai, India备注：Submitted to ICASSP 2023摘要：本文提出了一种新的自监督学习（SSL）方法，用于在未标记音频数据上预训练编码器，从而减少了对用于音频和语音分类的大量标记数据的需要。我们的主要目标是在低资源的无标记音频预训练环境中学习能够在大量的语音和非语音任务中推广的音频表示。受聚类和对比学习范式在基于SSL的语音表征学习中取得的成功启发，我们提出了SLICER（Symmetrical Learning of Instance and Cluster-level Efficient Representations）算法，该算法结合了聚类和对比学习范式的优点。我们使用来自学生和教师编码器的潜在表征之间的对称损失，并同时解决实例和聚类水平的对比学习任务。通过将输入谱图投影到一个维数等于聚类数的输出子空间，在线获得聚类表示。此外，我们提出了一种新的基于mixup的mel谱图增强方法k-mix，它不需要标签，有助于音频的无监督表示学习。总的来说，SLICER在LAPE基准测试中取得了最先进的结果\cite{9868132}，明显优于DeLoRes-M和其他先前的方法，这些方法是在10倍于$的无监督数据上预先训练的。我们将在GitHub上提供所有代码。摘要：We present a new Self-Supervised Learning (SSL) approach to pre-train encoders on unlabeled audio data that reduces the need for large amounts of labeled data for audio and speech classification. Our primary aim is to learn audio representations that can generalize across a large variety of speech and non-speech tasks in a low-resource un-labeled audio pre-training setting. Inspired by the recent success of clustering and contrasting learning paradigms for SSL-based speech representation learning, we propose SLICER (Symmetrical Learning of Instance and Cluster-level Efficient Representations), which brings together the best of both clustering and contrasting learning paradigms. We use a symmetric loss between latent representations from student and teacher encoders and simultaneously solve instance and cluster-level contrastive learning tasks. We obtain cluster representations online by just projecting the input spectrogram into an output subspace with dimensions equal to the number of clusters. In addition, we propose a novel mel-spectrogram augmentation procedure, k-mix, based on mixup, which does not require labels and aids unsupervised representation learning for audio. Overall, SLICER achieves state-of-the-art results on the LAPE Benchmark \cite{9868132}, significantly outperforming DeLoRes-M and other prior approaches, which are pre-trained on $10\times$ larger of unsupervised data. We will make all our codes available on GitHub.
【19】 MAST: Multiscale Audio Spectrogram Transformers标题：MAST：多尺度音频频谱转换器链接：https://arxiv.org/abs/2211.01515
作者：Sreyan Ghosh,Ashish Seth,S. Umesh,Dinesh Manocha机构：University of Maryland, College Park, USA, Speech Lab, Department of Electrical Engineering, IIT Madras, Chennai, India备注：Submitted ICASSP 2023摘要：提出了一种用于音频分类的多尺度音频谱图Transformer（MAST），它将多尺度特征层次的概念引入到音频谱图变换器（AST）中。给定一个输入音频谱图，首先将其拼接并投影到一个初始的时间分辨率和嵌入维数上，然后在MAST中的多个阶段逐步扩展嵌入维数，同时降低输入的时间分辨率。我们使用金字塔结构，其允许MAST的早期层以高时间分辨率但低嵌入空间操作来建模简单的低级别声学信息，并且更深的时间粗糙层以高维嵌入来建模高级别声学信息。我们还扩展了我们的方法，提出了一种新的自我监督学习（SSL）方法，称为SS-MAST，该方法计算来自学生和教师编码器的潜在表征之间的对称对比损失。在实践中，MAST在LAPE Benchmark的8个语音和非语音任务中的平均准确率为3.4%，显著优于AST。此外，对于AST和MAST编码器，SS-MAST实现了相对于SSAST的2.6%的绝对平均改进。我们在发布时将所有代码发布在GitHub上。摘要：We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST). Given an input audio spectrogram we first patchify and project it into an initial temporal resolution and embedding dimension, post which the multiple stages in MAST progressively expand the embedding dimension while reducing the temporal resolution of the input. We use a pyramid structure that allows early layers of MAST operating at a high temporal resolution but low embedding space to model simple low-level acoustic information and deeper temporally coarse layers to model high-level acoustic information with high-dimensional embeddings. We also extend our approach to present a new Self-Supervised Learning (SSL) method called SS-MAST, which calculates a symmetric contrastive loss between latent representations from a student and a teacher encoder. In practice, MAST significantly outperforms AST by an average accuracy of 3.4% across 8 speech and non-speech tasks from the LAPE Benchmark. Moreover, SS-MAST achieves an absolute average improvement of 2.6% over SSAST for both AST and MAST encoders. We make all our codes available on GitHub at the time of publication.
【20】 Phoneme Segmentation Using Self-Supervised Speech Models标题：基于自监督语音模型的音素分割链接：https://arxiv.org/abs/2211.01461
作者：Luke Strgar,David Harwath机构：University of Texas at Austin, Department of Computer Science备注：Accepted to SLT 2022摘要：我们将迁移学习应用到音素切分任务中，并证明了在自我监督的预训练中学习到的表征对该任务的效用。我们的模型通过策略性放置的卷积来扩展变换器式编码器，这些卷积操作在预训练中学习到的特征。我们使用TIMIT和Buckeye语料库在有监督和无监督两种情况下对模型进行了训练和测试。后一种情况是通过向有噪声的标签集提供单独模型的预测来实现的，该模型已经以无监督的方式被训练。结果表明，我们的模型在两种设置和两个数据集上都超越了先前的最先进性能。最后，根据已发布代码评审期间的观察结果，并尝试重现过去的分割结果，我们发现需要消除广泛使用的评估指标的定义和实现的歧义。我们通过描述两种不同的评估方案并描述它们的细微差别来解决这种模糊性。摘要：We apply transfer learning to the task of phoneme segmentation and demonstrate the utility of representations learned in self-supervised pre-training for the task. Our model extends transformer-style encoders with strategically placed convolutions that manipulate features learned in pre-training. Using the TIMIT and Buckeye corpora we train and test the model in the supervised and unsupervised settings. The latter case is accomplished by furnishing a noisy label-set with the predictions of a separate model, it having been trained in an unsupervised fashion. Results indicate our model eclipses previous state-of-the-art performance in both settings and on both datasets. Finally, following observations during published code review and attempts to reproduce past segmentation results, we find a need to disambiguate the definition and implementation of widely-used evaluation metrics. We resolve this ambiguity by delineating two distinct evaluation schemes and describing their nuances.
【21】 Variable Attention Masking for Configurable Transformer Transducer Speech Recognition标题：可变注意掩蔽在可配置互感器语音识别中的应用链接：https://arxiv.org/abs/2211.01438
作者：Pawel Swietojanski,Stefan Braun,Dogan Can,Thiago Fraga da Silva,Arnab Ghoshal,Takaaki Hori,Roger Hsiao,Henry Mason,Erik McDermott,Honza Silovsky,Ruchir Travadi,Xiaodan Zhuang机构：Apple备注：5 pages, 4 figures, 2 Tables摘要：本文研究了在基于Transformer传感器的语音识别中使用注意力掩蔽来为不同的部署场景构建单个可配置模型。我们提供了一组全面的实验，在识别准确性和延迟方面比较固定掩蔽和分块掩蔽，其中固定掩蔽在每帧应用相同的注意力掩蔽，分块掩蔽在每帧的注意力掩蔽由分块边界确定。然后，我们探索了可变掩蔽的使用，其中注意力掩蔽是在训练时间从目标分布中采样的，以构建可以在不同配置下工作的模型。最后，我们研究了如何使用单个可配置模型来执行第一遍流识别和第二遍声学再评分。实验表明，无论有无FastEmit，与固定掩蔽相比，分块掩蔽实现了更好的准确性与延迟的折衷。我们还表明，在声学重新评分的情况下，可变掩蔽将准确度提高了8%。摘要：This work studies the use of attention masking in transformer transducer based speech recognition for building a single configurable model for different deployment scenarios. We present a comprehensive set of experiments comparing fixed masking, where the same attention mask is applied at every frame, with chunked masking, where the attention mask for each frame is determined by chunk boundaries, in terms of recognition accuracy and latency. We then explore the use of variable masking, where the attention masks are sampled from a target distribution at training time, to build models that can work in different configurations. Finally, we investigate how a single configurable model can be used to perform both first pass streaming recognition and second pass acoustic rescoring. Experiments show that chunked masking achieves a better accuracy vs latency trade-off compared to fixed masking, both with and without FastEmit. We also show that variable masking improves the accuracy by up to 8% relative in the acoustic re-scoring scenario.
【1】 Speech-based emotion recognition with self-supervised models using attentive channel-wise correlations and label smoothing标题：基于注意通道相关和标签平滑的自监督模型语音情感识别链接：https://arxiv.org/abs/2211.01756
* 与cs.SD语音【10】为同一篇
作者：Sofoklis Kakouros,Themos Stafylakis,Ladislav Mosner,Lukas Burget机构：University of Helsinki, Finland, Omilia - Conversational Intelligence, Athens, Greece备注：Submitted to IEEE-ICASSP 2023摘要：从语音中识别情感时，我们会遇到两个常见的问题：如何最佳地从语音信号中捕获情感相关信息以及如何最佳地量化或分类有噪声的主观情感标签。自监督的预训练表示可以鲁棒地从语音中捕获信息，使得在包括情感识别在内的许多下游任务中能够获得最先进的结果。然而，需要考虑在时间上聚集信息的更好方式，因为相关情感信息可能分段出现，并且在信号上不均匀。对于标签，我们需要考虑到存在来自主观人类注释的相当程度的噪声。本文提出了一种新的注意力池方法，该方法基于表征系数之间的相关性并结合标签平滑，旨在降低分类器对训练标签的置信度。我们在IEMOCAP基准数据集上对我们提出的方法进行了评估，并证明了我们的方法比文献中的方法具有更高的性能。可在www.example.com上找到重现结果的代码github.com/skakouros/s3prl_attentive_correlation。摘要：When recognizing emotions from speech, we encounter two common problems: how to optimally capture emotion-relevant information from the speech signal and how to best quantify or categorize the noisy subjective emotion labels. Self-supervised pre-trained representations can robustly capture information from speech enabling state-of-the-art results in many downstream tasks including emotion recognition. However, better ways of aggregating the information across time need to be considered as the relevant emotion information is likely to appear piecewise and not uniformly across the signal. For the labels, we need to take into account that there is a substantial degree of noise that comes from the subjective human annotations. In this paper, we propose a novel approach to attentive pooling based on correlations between the representations' coefficients combined with label smoothing, a method aiming to reduce the confidence of the classifier on the training labels. We evaluate our proposed approach on the benchmark dataset IEMOCAP, and demonstrate high performance surpassing that in the literature. The code to reproduce the results is available at github.com/skakouros/s3prl_attentive_correlation.
【2】 Discussion of Features for Acoustic Anomaly Detection under Industrial Disturbing Noise in an End-of-Line Test of Geared Motors标题：齿轮电机尾线试验中工业干扰噪声下的声学异常检测特征探讨链接：https://arxiv.org/abs/2211.01716
* 与cs.SD语音【11】为同一篇
作者：Peter Wissbrock,David Pelkmann,Yvonne Richter机构：Lenze SE, Innovation, Aerzen, Germany, Institut für Systemdynamik und Mechatronik, Bielefeld University of Applied Sciences, Germany摘要：在减速电机的下线试验中，产品质量的评定是一个重要的环节。由于时间限制和变量的高度多样性，声学测量比振动测量更经济。然而，声学数据受到工业干扰噪声的影响。因此，本研究的目的是研究用于齿轮电机下线测试中异常检测的特征的鲁棒性。通过声阵列记录了具有典型故障和声干扰的真实世界数据集。这包括来自生产的工业噪声和系统产生的干扰，用于比较鲁棒性。总的来说，提出了将从对数包络频谱提取的特征与心理声学特征一起应用。异常检测采用隔离森林或更通用的Bagging随机挖掘器。大多数干扰都可以避免，而使用锤子或气压往往会引起问题。一般而言，这些结果对于基于声学或振动测量的状态监测任务非常重要。此外，还提出了一种真实世界问题的描述，以改进常见的信号处理和机器学习任务。摘要：In the end-of-line test of geared motors, the evaluation of product qual-ity is important. Due to time constraints and the high diversity of variants, acous-tic measurements are more economical than vibration measurements. However, the acoustic data is affected by industrial disturbing noise. Therefore, the aim of this study is to investigate the robustness of features used for anomaly detection in geared motor end-of-line testing. A real-world dataset with typical faults and acoustic disturbances is recorded by an acoustic array. This includes industrial noise from the production and systematically produced disturbances, used to compare the robustness. Overall, it is proposed to apply features extracted from a log-envelope spectrum together with psychoacoustic features. The anomaly de-tection is done by using the isolation forest or the more universal bagging random miner. Most disturbances can be circumvented, while the use of a hammer or air pressure often causes problems. In general, these results are important for condi-tion monitoring tasks that are based on acoustic or vibration measurements. Fur-thermore, a real-world problem description is presented to improve common sig-nal processing and machine learning tasks.
【3】 Cutting Through the Noise: An Empirical Comparison of Psychoacoustic and Envelope-based Features for Machinery Fault Detection标题：穿透噪声：机械故障检测中心理声学特征和包络特征的实证比较链接：https://arxiv.org/abs/2211.01704
* 与cs.SD语音【12】为同一篇
作者：Peter Wißbrock,Yvonne Richter,David Pelkmann,Zhao Ren,Gregory Palmer机构：Lenze SE, Innovation, Aerzen, Germany, Institut für Systemdynamik und Mechatronik, Bielefeld University of Applied Sciences, Germany, L,S Research Center, Leibniz University of Hannover, Germany摘要：基于声学的故障检测技术在监测机械零件的健康状态方面具有很大的潜力。然而，工业环境的背景噪声可能负面地影响故障检测的性能。对提高故障检测对工业环境噪声的鲁棒性的关注有限。因此，我们提出了伦茨生产背景噪声（LPBN）真实世界数据集和一个自动化和噪声鲁棒的听觉检查（ARAI）系统，用于齿轮电机的生产线末端检查。声阵列用于从具有小故障、大故障或健康的电动机获取数据。提供基准以基于齿轮箱的专家知识将心理声学特征与不同类型的包络特征进行比较。据我们所知，我们是第一个将时变心理声学特征应用于故障检测的公司。我们训练了一个最先进的单类分类器，对健康电机的样本，并分离出故障的一个使用阈值的故障检测。性能最好的方法实现了0.87（对数包络）、0.86（时变心理声学）和0.91（两者的组合）的曲线下面积。摘要：Acoustic-based fault detection has a high potential to monitor the health condition of mechanical parts. However, the background noise of an industrial environment may negatively influence the performance of fault detection. Limited attention has been paid to improving the robustness of fault detection against industrial environmental noise. Therefore, we present the Lenze production background-noise (LPBN) real-world dataset and an automated and noise-robust auditory inspection (ARAI) system for the end-of-line inspection of geared motors. An acoustic array is used to acquire data from motors with a minor fault, major fault, or which are healthy. A benchmark is provided to compare the psychoacoustic features with different types of envelope features based on expert knowledge of the gearbox. To the best of our knowledge, we are the first to apply time-varying psychoacoustic features for fault detection. We train a state-of-the-art one-class-classifier, on samples from healthy motors and separate the faulty ones for fault detection using a threshold. The best-performing approaches achieve an area under curve of 0.87 (logarithm envelope), 0.86 (time-varying psychoacoustics), and 0.91 (combination of both).
【4】 Channel-Aware Pretraining of Joint Encoder-Decoder Self-Supervised Model for Telephonic-Speech ASR标题：电话-语音ASR中联合编解码器自监督模型的信道感知预训练链接：https://arxiv.org/abs/2211.01669
* 与cs.SD语音【13】为同一篇
作者：Vrunda N. Sukhadia,A. Arunkumar,S. Umesh机构：Speech Lab, Electrical Engineering Department, IIT Madras, India备注：5 pages, 5 figures摘要：本文提出了一种新的技术，当使用来自两个不同信道（窄带和宽带）的语音池进行训练时，从联合编解码器自监督模型获得更好的下游ASR性能。联合编码器-解码器自监督模型用Transformer解码器扩展了HuBERT模型。HuBERT执行特征聚类并预测每个输入帧的类别。在简单池中（这是我们的基准），无法识别通道信息。为了合并信道信息，我们已经提出了用于来自不同信道的语音的非重叠聚类ID。我们的方法给出了一个约5%的相对改善联合编码器-解码器自监督模型建立简单的数据池，作为我们的基线。摘要：This paper proposes a novel technique to obtain better downstream ASR performance from a joint encoder-decoder self-supervised model when trained with speech pooled from two different channels (narrow and wide band). The joint encoder-decoder self-supervised model extends the HuBERT model with a Transformer decoder. HuBERT performs clustering of features and predicts the class of every input frame. In simple pooling, which is our baseline, there is no way to identify the channel information. To incorporate channel information, we have proposed non-overlapping cluster IDs for speech from different channels. Our method gives a relative improvement of ~ 5% over the joint encoder-decoder self-supervised model built with simple pooling of data, which serves as our baseline.
【5】 Adversarial Data Augmentation Using VAE-GAN for Disordered Speech Recognition标题：基于VAE-GAN的无序语音识别对抗性数据增强链接：https://arxiv.org/abs/2211.01646
* 与cs.SD语音【14】为同一篇
作者：Zengrui Jin,Xurong Xie,Mengzhe Geng,Tianzi Wang,Shujie Hu,Jiajun Deng,Guinan Li,Xunying Liu机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Institute of Software, Chinese Academy of Sciences, China备注：Submitted to ICASSP 2023摘要：无序语音的自动识别一直是一个极具挑战性的课题。潜在的神经运动疾病，通常与同时发生的身体残疾相结合，导致难以收集ASR系统开发所需的大量受损语音。提出了一种基于变分自动编码生成对抗网络（VAE-GAN）的个性化语音增强方法，该方法能够同时学习编码、生成和识别合成的受损语音。导出单独的潜在特征以学习构音障碍语音特征和音素上下文表示。还结合了自监督预训练的Wav 2 vec 2.0嵌入特征。在UASpeech语料库上进行的实验表明，所提出的对抗性数据增强方法在训练混合TDNN和端到端Conformer系统的情况下始终优于基线速度扰动和非VAE GAN增强方法。在LHUC说话者自适应之后，使用基于VAE-GAN的增强的最佳系统在16个构音障碍说话者的UASpeech测试集上产生了27.78%的总体WER，并且在具有“非常低”可懂度的说话者子集上产生了57.31%的最低公布WER。摘要：Automatic recognition of disordered speech remains a highly challenging task to date. The underlying neuro-motor conditions, often compounded with co-occurring physical disabilities, lead to the difficulty in collecting large quantities of impaired speech required for ASR system development. This paper presents novel variational auto-encoder generative adversarial network (VAE-GAN) based personalized disordered speech augmentation approaches that simultaneously learn to encode, generate and discriminate synthesized impaired speech. Separate latent features are derived to learn dysarthric speech characteristics and phoneme context representations. Self-supervised pre-trained Wav2vec 2.0 embedding features are also incorporated. Experiments conducted on the UASpeech corpus suggest the proposed adversarial data augmentation approach consistently outperformed the baseline speed perturbation and non-VAE GAN augmentation methods with trained hybrid TDNN and End-to-end Conformer systems. After LHUC speaker adaptation, the best system using VAE-GAN based augmentation produced an overall WER of 27.78% on the UASpeech test set of 16 dysarthric speakers, and the lowest published WER of 57.31% on the subset of speakers with "Very Low" intelligibility.
【6】 Leveraging Domain Features for Detecting Adversarial Attacks Against Deep Speech Recognition in Noise标题：利用域特征检测噪声中针对深度语音识别的敌意攻击链接：https://arxiv.org/abs/2211.01621
* 与cs.SD语音【15】为同一篇
作者：Christian Heider Nielsen,Zheng-Hua Tan机构：Department of Electronic Systems, Aalborg University, Denmark摘要：近年来，基于深度模型的自动语音识别（ASR）技术取得了显著的进展，并在现实生活中得到了广泛的应用。与此同时，针对深度ASR系统的对抗性攻击也非常成功。已经提出了各种方法来保护ASR系统免受这些攻击。然而，现有的基于分类的方法主要集中在深度学习模型的设计上，而缺乏对领域特定特征的挖掘。这项工作利用基于过滤器组的特征来更好地捕获攻击的特征，以改进检测。此外，本文还分析了分别使用语音和非语音部分检测对抗性攻击的潜力。最后，考虑到ASR系统可能部署的不利环境，我们研究了各种类型的声学噪声和信噪比的影响。大量实验表明，逆滤波器组特征在干净和噪声环境下都有较好的检测性能，无论是语音还是非语音部分都能有效地进行检测，而声学噪声会大大降低检测性能。摘要：In recent years, significant progress has been made in deep model-based automatic speech recognition (ASR), leading to its widespread deployment in the real world. At the same time, adversarial attacks against deep ASR systems are highly successful. Various methods have been proposed to defend ASR systems from these attacks. However, existing classification based methods focus on the design of deep learning models while lacking exploration of domain specific features. This work leverages filter bank-based features to better capture the characteristics of attacks for improved detection. Furthermore, the paper analyses the potentials of using speech and non-speech parts separately in detecting adversarial attacks. In the end, considering adverse environments where ASR systems may be deployed, we study the impact of acoustic noise of various types and signal-to-noise ratios. Extensive experiments show that the inverse filter bank features generally perform better in both clean and noisy environments, the detection is effective using either speech or non-speech part, and the acoustic noise can largely degrade the detection performance.
【7】 Convolution channel separation and frequency sub-bands aggregation for music genre classification标题：用于音乐流派分类的卷积通道分离和频率子带聚合链接：https://arxiv.org/abs/2211.01599
* 与cs.SD语音【16】为同一篇
作者：Jungwoo Heo,Hyun-seo Shin,Ju-ho Kim,Chan-yeong Lim,Ha-Jin Yu机构：School of Computer Science, University of Seoul摘要：在音乐中，音高和节奏等短期特征构成旋律和叙事等长期语义特征。音乐流派分类（MGC）系统应该能够分析这些特征。在本研究中，我们提出了一个新的架构，可以提取和聚合的短期和长期的特征层次。我们的框架基于ECAPA-TDNN，其中由于反向传播训练，提取短期特征的所有层都受到提取长期特征的层的影响。为了防止短期特征的失真，我们设计了卷积通道分离技术，将短期特征与长期特征提取路径分离。为了从我们的框架中提取更多的特征，我们引入了频率子带聚合方法，该方法沿着频率带宽划分输入谱图并处理每个段。我们使用Melon Playlist数据集对我们的框架进行了评估，该数据集是一个大规模数据集，包含的数据比MGC研究中广泛使用的数据集GTZAN多600倍。结果表明，该框架的准确率达到了70.4%，比传统框架提高了16.9%。摘要：In music, short-term features such as pitch and tempo constitute long-term semantic features such as melody and narrative. A music genre classification (MGC) system should be able to analyze these features. In this research, we propose a novel framework that can extract and aggregate both short- and long-term features hierarchically. Our framework is based on ECAPA-TDNN, where all the layers that extract short-term features are affected by the layers that extract long-term features because of the back-propagation training. To prevent the distortion of short-term features, we devised the convolution channel separation technique that separates short-term features from long-term feature extraction paths. To extract more diverse features from our framework, we incorporated the frequency sub-bands aggregation method, which divides the input spectrogram along frequency bandwidths and processes each segment. We evaluated our framework using the Melon Playlist dataset which is a large-scale dataset containing 600 times more data than GTZAN which is a widely used dataset in MGC studies. As the result, our framework achieved 70.4% accuracy, which was improved by 16.9% compared to a conventional framework.
【8】 Phonetic-assisted Multi-Target Units Modeling for Improving Conformer-Transducer ASR system标题：改进变压器-变送器ASR系统的语音辅助多目标单元建模链接：https://arxiv.org/abs/2211.01571
* 与cs.SD语音【17】为同一篇
作者：Li Li,Dongxing Xu,Haoran Wei,Yanhua Long机构：Shanghai Engineering Research Center of Intelligent Education and Bigdata, Shanghai Normal University, Shanghai, China, Unisound AI Technology Co., Ltd., Beijing, China, Department of ECE, University of Texas at Dallas, Richardson, TX , USA备注：5 pages, 1 figures, submitted to ICASSP 2023摘要：在端到端的自动语音识别（ASR）中，开发有效的目标建模单元是非常重要的，也是一直关注的问题。本文提出一种语音辅助的多目标单元（PMU）建模方法，以渐进式表征学习的方式增强Conformer-Transducer ASR系统。具体来说，PMU首先使用语音辅助子词建模（PASM）和字节对编码（BPE）分别产生语音诱导和文本诱导的目标单元;在此基础上，提出了三种新的声学编码器增强框架，包括基本PMU、paraCTC和pcaCTC，它们集成了不同层次的PASM和BPE单元，用于CTC和传感器多任务训练。在LibriSpeech和重音ASR测试集上的实验结果表明，与传统的BPE相比，本文提出的PMU显著降低了LibriSpeech clean、other和6个重音ASR测试集的WER，分别降低了12.7%、6.0%和7.7%。摘要：Exploiting effective target modeling units is very important and has always been a concern in end-to-end automatic speech recognition (ASR). In this work, we propose a phonetic-assisted multi-target units (PMU) modeling approach, to enhance the Conformer-Transducer ASR system in a progressive representation learning manner. Specifically, PMU first uses the pronunciation-assisted subword modeling (PASM) and byte pair encoding (BPE) to produce phonetic-induced and text-induced target units separately; Then, three new frameworks are investigated to enhance the acoustic encoder, including a basic PMU, a paraCTC and a pcaCTC, they integrate the PASM and BPE units at different levels for CTC and transducer multi-task training. Experiments on both LibriSpeech and accented ASR tasks show that, the proposed PMU significantly outperforms the conventional BPE, it reduces the WER of LibriSpeech clean, other, and six accented ASR testsets by relative 12.7%, 6.0% and 7.7%, respectively.
【9】 SLICER: Learning universal audio representations using low-resource self-supervised pre-training标题：切片器：使用低资源自我监督预训练学习通用音频表示链接：https://arxiv.org/abs/2211.01519
* 与cs.SD语音【18】为同一篇
作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha机构：University of Maryland, College Park, USA, Speech Lab, Department of Electrical Engineering, IIT Madras, Chennai, India备注：Submitted to ICASSP 2023摘要：本文提出了一种新的自监督学习（SSL）方法，用于在未标记音频数据上预训练编码器，从而减少了对用于音频和语音分类的大量标记数据的需要。我们的主要目标是在低资源的无标记音频预训练环境中学习能够在大量的语音和非语音任务中推广的音频表示。受聚类和对比学习范式在基于SSL的语音表征学习中取得的成功启发，我们提出了SLICER（Symmetrical Learning of Instance and Cluster-level Efficient Representations）算法，该算法结合了聚类和对比学习范式的优点。我们使用来自学生和教师编码器的潜在表征之间的对称损失，并同时解决实例和聚类水平的对比学习任务。通过将输入谱图投影到一个维数等于聚类数的输出子空间，在线获得聚类表示。此外，我们提出了一种新的基于mixup的mel谱图增强方法k-mix，它不需要标签，有助于音频的无监督表示学习。总的来说，SLICER在LAPE基准测试中取得了最先进的结果\cite{9868132}，明显优于DeLoRes-M和其他先前的方法，这些方法是在10倍于$的无监督数据上预先训练的。我们将在GitHub上提供所有代码。摘要：We present a new Self-Supervised Learning (SSL) approach to pre-train encoders on unlabeled audio data that reduces the need for large amounts of labeled data for audio and speech classification. Our primary aim is to learn audio representations that can generalize across a large variety of speech and non-speech tasks in a low-resource un-labeled audio pre-training setting. Inspired by the recent success of clustering and contrasting learning paradigms for SSL-based speech representation learning, we propose SLICER (Symmetrical Learning of Instance and Cluster-level Efficient Representations), which brings together the best of both clustering and contrasting learning paradigms. We use a symmetric loss between latent representations from student and teacher encoders and simultaneously solve instance and cluster-level contrastive learning tasks. We obtain cluster representations online by just projecting the input spectrogram into an output subspace with dimensions equal to the number of clusters. In addition, we propose a novel mel-spectrogram augmentation procedure, k-mix, based on mixup, which does not require labels and aids unsupervised representation learning for audio. Overall, SLICER achieves state-of-the-art results on the LAPE Benchmark \cite{9868132}, significantly outperforming DeLoRes-M and other prior approaches, which are pre-trained on $10\times$ larger of unsupervised data. We will make all our codes available on GitHub.
【10】 MAST: Multiscale Audio Spectrogram Transformers标题：MAST：多尺度音频频谱转换器链接：https://arxiv.org/abs/2211.01515
* 与cs.SD语音【19】为同一篇
作者：Sreyan Ghosh,Ashish Seth,S. Umesh,Dinesh Manocha机构：University of Maryland, College Park, USA, Speech Lab, Department of Electrical Engineering, IIT Madras, Chennai, India备注：Submitted ICASSP 2023摘要：提出了一种用于音频分类的多尺度音频谱图Transformer（MAST），它将多尺度特征层次的概念引入到音频谱图变换器（AST）中。给定一个输入音频谱图，首先将其拼接并投影到一个初始的时间分辨率和嵌入维数上，然后在MAST中的多个阶段逐步扩展嵌入维数，同时降低输入的时间分辨率。我们使用金字塔结构，其允许MAST的早期层以高时间分辨率但低嵌入空间操作来建模简单的低级别声学信息，并且更深的时间粗糙层以高维嵌入来建模高级别声学信息。我们还扩展了我们的方法，提出了一种新的自我监督学习（SSL）方法，称为SS-MAST，该方法计算来自学生和教师编码器的潜在表征之间的对称对比损失。在实践中，MAST在LAPE Benchmark的8个语音和非语音任务中的平均准确率为3.4%，显著优于AST。此外，对于AST和MAST编码器，SS-MAST实现了相对于SSAST的2.6%的绝对平均改进。我们在发布时将所有代码发布在GitHub上。摘要：We present Multiscale Audio Spectrogram Transformer (MAST) for audio classification, which brings the concept of multiscale feature hierarchies to the Audio Spectrogram Transformer (AST). Given an input audio spectrogram we first patchify and project it into an initial temporal resolution and embedding dimension, post which the multiple stages in MAST progressively expand the embedding dimension while reducing the temporal resolution of the input. We use a pyramid structure that allows early layers of MAST operating at a high temporal resolution but low embedding space to model simple low-level acoustic information and deeper temporally coarse layers to model high-level acoustic information with high-dimensional embeddings. We also extend our approach to present a new Self-Supervised Learning (SSL) method called SS-MAST, which calculates a symmetric contrastive loss between latent representations from a student and a teacher encoder. In practice, MAST significantly outperforms AST by an average accuracy of 3.4% across 8 speech and non-speech tasks from the LAPE Benchmark. Moreover, SS-MAST achieves an absolute average improvement of 2.6% over SSAST for both AST and MAST encoders. We make all our codes available on GitHub at the time of publication.
【11】 Phoneme Segmentation Using Self-Supervised Speech Models标题：基于自监督语音模型的音素分割链接：https://arxiv.org/abs/2211.01461
* 与cs.SD语音【20】为同一篇
作者：Luke Strgar,David Harwath机构：University of Texas at Austin, Department of Computer Science备注：Accepted to SLT 2022摘要：我们将迁移学习应用到音素切分任务中，并证明了在自我监督的预训练中学习到的表征对该任务的效用。我们的模型通过策略性放置的卷积来扩展变换器式编码器，这些卷积操作在预训练中学习到的特征。我们使用TIMIT和Buckeye语料库在有监督和无监督两种情况下对模型进行了训练和测试。后一种情况是通过向有噪声的标签集提供单独模型的预测来实现的，该模型已经以无监督的方式被训练。结果表明，我们的模型在两种设置和两个数据集上都超越了先前的最先进性能。最后，根据已发布代码评审期间的观察结果，并尝试重现过去的分割结果，我们发现需要消除广泛使用的评估指标的定义和实现的歧义。我们通过描述两种不同的评估方案并描述它们的细微差别来解决这种模糊性。摘要：We apply transfer learning to the task of phoneme segmentation and demonstrate the utility of representations learned in self-supervised pre-training for the task. Our model extends transformer-style encoders with strategically placed convolutions that manipulate features learned in pre-training. Using the TIMIT and Buckeye corpora we train and test the model in the supervised and unsupervised settings. The latter case is accomplished by furnishing a noisy label-set with the predictions of a separate model, it having been trained in an unsupervised fashion. Results indicate our model eclipses previous state-of-the-art performance in both settings and on both datasets. Finally, following observations during published code review and attempts to reproduce past segmentation results, we find a need to disambiguate the definition and implementation of widely-used evaluation metrics. We resolve this ambiguity by delineating two distinct evaluation schemes and describing their nuances.
【12】 Variable Attention Masking for Configurable Transformer Transducer Speech Recognition标题：可变注意掩蔽在可配置互感器语音识别中的应用链接：https://arxiv.org/abs/2211.01438
* 与cs.SD语音【21】为同一篇
作者：Pawel Swietojanski,Stefan Braun,Dogan Can,Thiago Fraga da Silva,Arnab Ghoshal,Takaaki Hori,Roger Hsiao,Henry Mason,Erik McDermott,Honza Silovsky,Ruchir Travadi,Xiaodan Zhuang机构：Apple备注：5 pages, 4 figures, 2 Tables摘要：本文研究了在基于Transformer传感器的语音识别中使用注意力掩蔽来为不同的部署场景构建单个可配置模型。我们提供了一组全面的实验，在识别准确性和延迟方面比较固定掩蔽和分块掩蔽，其中固定掩蔽在每帧应用相同的注意力掩蔽，分块掩蔽在每帧的注意力掩蔽由分块边界确定。然后，我们探索了可变掩蔽的使用，其中注意力掩蔽是在训练时间从目标分布中采样的，以构建可以在不同配置下工作的模型。最后，我们研究了如何使用单个可配置模型来执行第一遍流识别和第二遍声学再评分。实验表明，无论有无FastEmit，与固定掩蔽相比，分块掩蔽实现了更好的准确性与延迟的折衷。我们还表明，在声学重新评分的情况下，可变掩蔽将准确度提高了8%。摘要：This work studies the use of attention masking in transformer transducer based speech recognition for building a single configurable model for different deployment scenarios. We present a comprehensive set of experiments comparing fixed masking, where the same attention mask is applied at every frame, with chunked masking, where the attention mask for each frame is determined by chunk boundaries, in terms of recognition accuracy and latency. We then explore the use of variable masking, where the attention masks are sampled from a target distribution at training time, to build models that can work in different configurations. Finally, we investigate how a single configurable model can be used to perform both first pass streaming recognition and second pass acoustic rescoring. Experiments show that chunked masking achieves a better accuracy vs latency trade-off compared to fixed masking, both with and without FastEmit. We also show that variable masking improves the accuracy by up to 8% relative in the acoustic re-scoring scenario.
【13】 Dynamic Kernels and Channel Attention with Multi-Layer Embedding Aggregation for Speaker Verification标题：多层嵌入聚合的动态核函数和通道注意力用于说话人确认链接：https://arxiv.org/abs/2211.02000
* 与cs.SD语音【1】为同一篇
作者：Anna Ollerenshaw,Md Asif Jalal,Thomas Hain机构：Speech and Hearing Group, The University of Sheffield, Sheffield, UK备注：Submitted to ICASSP 2023摘要：现有技术的说话人确认框架通常集中于具有越来越深（更多层）和越来越宽（通道数量）的模型的语音增强技术，以提高其确认性能。相反，本文提出了一种在卷积神经网络中使用基于注意力的动态核来提高模型分辨率能力的方法，以适应特征条件下的模型参数。通过信道关注度和多层特征聚合进一步提取核上的关注度权重，以学习语音的全局特征。该方法为利用较少的数据资源提高表示能力提供了一种有效的解决方案。这是由于对模型参数的结构的输入的自适应。该模型在VoxCeleb 1测试集上的EER和miniDCF分别为1.62%和0.18%，与ECAPA-TDNN相比有17%的相对改善.摘要：State-of-the-art speaker verification frameworks have typically focused on speech enhancement techniques with increasingly deeper (more layers) and wider (number of channels) models to improve their verification performance. Instead, this paper proposes an approach to increase the model resolution capability using attention-based dynamic kernels in a convolutional neural network to adapt the model parameters to be feature-conditioned. The attention weights on the kernels are further distilled by channel attention and multi-layer feature aggregation to learn global features from speech. This approach provides an efficient solution to improving representation capacity with lower data resources. This is due to the self-adaptation to inputs of the structures of the model parameters. The proposed dynamic convolutional model achieved 1.62\% EER and 0.18 miniDCF on the VoxCeleb1 test set and has a 17\% relative improvement compared to the ECAPA-TDNN.
【14】 Probing Statistical Representations For End-To-End ASR标题：探讨端到端ASR的统计表示法链接：https://arxiv.org/abs/2211.01993
* 与cs.SD语音【2】为同一篇
作者：Anna Ollerenshaw,Md Asif Jalal,Thomas Hain机构：Speech and Hearing Group, The University of Sheffield, Sheffield, UK备注：Submitted to ICASSP 2023摘要：端到端自动语音识别（ASR）模型旨在学习广义语音表示以执行识别。在这个领域中，很少有研究来分析内部表示依赖性及其与建模方法的关系。本文使用SVCCA研究了Transformer体系结构中的跨域语言模型依赖性，并使用这些见解来开发建模方法。研究发现，Transformer层内的特定神经表征表现出影响识别性能的相关行为。 总之，这项工作提供了对影响上下文依赖性和ASR性能的建模方法的分析，并可用于创建或调整性能更好的端到端ASR模型，也可用于下游任务。摘要：End-to-End automatic speech recognition (ASR) models aim to learn a generalised speech representation to perform recognition. In this domain there is little research to analyse internal representation dependencies and their relationship to modelling approaches. This paper investigates cross-domain language model dependencies within transformer architectures using SVCCA and uses these insights to exploit modelling approaches. It was found that specific neural representations within the transformer layers exhibit correlated behaviour which impacts recognition performance. Altogether, this work provides analysis of the modelling approaches affecting contextual dependencies and ASR performance, and can be used to create or adapt better performing End-to-End ASR models and also for downstream tasks.
【15】 MarginNCE: Robust Sound Localization with a Negative Margin标题：MargiNCE：具有负边缘的稳健声音定位链接：https://arxiv.org/abs/2211.01966
* 与cs.SD语音【3】为同一篇
作者：Sooyoung Park,Arda Senocak,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, South Korea, Electronics and Telecommunications Research Institute, South Korea备注：Submitted to ICASSP 2023. SOTA performance in Audio-Visual Sound Localization. 5 Pages摘要：本研究的目的是利用一种自我监督的方法来定位视觉场景中的声源。声源定位背景下的对比学习利用了音频和视频信号之间的自然对应关系，其中来自相同源的音频-视频对被假定为正，而随机选择的对是负的。然而，这种方法带来了噪声对应;例如可能彼此不相关的正音频和视频对信号，或者可能包含与正信号语义相似的样本的负对。我们的主要贡献是证明了在对比学习中使用一个不太严格的决策边界可以减轻噪声对应对声源定位的影响。我们提出了一种简单而有效的方法，通过用负的边缘稍微修改对比损失。大量的实验结果表明，该方法的性能与现有的方法相当甚至更好.此外，我们证明了在现有方法中引入负裕度会导致性能的一致提高。摘要：The goal of this work is to localize sound sources in visual scenes with a self-supervised approach. Contrastive learning in the context of sound source localization leverages the natural correspondence between audio and visual signals where the audio-visual pairs from the same source are assumed as positive, while randomly selected pairs are negatives. However, this approach brings in noisy correspondences; for example, positive audio and visual pair signals that may be unrelated to each other, or negative pairs that may contain semantically similar samples to the positive one. Our key contribution in this work is to show that using a less strict decision boundary in contrastive learning can alleviate the effect of noisy correspondences in sound source localization. We propose a simple yet effective approach by slightly modifying the contrastive loss with a negative margin. Extensive experimental results show that our approach gives on-par or better performance than the state-of-the-art methods. Furthermore, we demonstrate that the introduction of a negative margin to existing methods results in a consistent improvement in performance.
【16】 HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks标题：HyperSound：利用超网络生成音频信号的隐式神经表示链接：https://arxiv.org/abs/2211.01839
* 与cs.SD语音【4】为同一篇
作者：Filip Szatkowski,Karol J. Piczak,Przemysław Spurek,Jacek Tabor,Tomasz Trzciński机构：Tomasz Trzci´nski, Warsaw University of Technology, IDEAS NCBR, Tooploox, Jagiellonian University摘要：内隐神经表征（inplicit neural representation，INR）是一个快速发展的研究领域，它为多媒体信号的表征提供了一种新的方法。INR的最新应用包括图像超分辨率、高维信号的压缩或3D渲染。然而，这些解决方案通常集中于视觉数据，并且使它们适应于音频域并非是微不足道的。此外，它需要针对每个数据样本单独训练的模型。为了解决这一问题，我们提出了一种元学习方法--HyperSound，它利用超网络来产生训练时不可见的音频信号的INR。我们表明，我们的方法可以重建声波的质量可比的其他国家的最先进的模型。摘要：Implicit neural representations (INRs) are a rapidly growing research field, which provides alternative ways to represent multimedia signals. Recent applications of INRs include image super-resolution, compression of high-dimensional signals, or 3D rendering. However, these solutions usually focus on visual data, and adapting them to the audio domain is not trivial. Moreover, it requires a separately trained model for every data sample. To address this limitation, we propose HyperSound, a meta-learning method leveraging hypernetworks to produce INRs for audio signals unseen at training time. We show that our approach can reconstruct sound waves with quality comparable to other state-of-the-art models.
【17】 Iterative autoregression: a novel trick to improve your low-latency speech enhancement model标题：迭代自回归：改进低延迟语音增强模型的新技巧链接：https://arxiv.org/abs/2211.01751
* 与cs.SD语音【5】为同一篇
作者：Pavel Andreev,Nicholas Babaev,Azat Saginbaev,Ivan Shchekotov机构：Samsung AI Center (Moscow)备注：Submitted to ICASSP 2023摘要：流模型是实时语音增强工具的重要组成部分。流传输机制将语音增强模型限制为仅使用未来信息的微小上下文，因此，低等待时间流传输设置通常被认为是具有挑战性的，并且对模型质量具有显著的负面影响。然而，由于流生成的顺序性质，它提供了自回归的自然可能性，即，在进行当前预测时使用以前的预测。本文提出了一种简单有效的自回归低时延语音增强模型的训练方法。我们证明了所提出的技术在不同的体系结构和训练场景中都能带来稳定的改进。摘要：Streaming models are an essential component of real-time speech enhancement tools. The streaming regime constrains speech enhancement models to use only a tiny context of future information, thus, the low-latency streaming setup is generally assumed to be challenging and has a significant negative effect on the model quality. However, due to the sequential nature of streaming generation, it provides a natural possibility for autoregression, i.e., using previous predictions when making current ones. In this paper, we present a simple, yet effective trick for training of autoregressive low-latency speech enhancement models. We demonstrate that the proposed technique leads to stable improvement across different architectures and training scenarios.
【18】 Hybrid-SD ($\text{H}_{\text{SD}}$) : A new hybrid evaluation metric for automatic speech recognition tasks标题：混合SD($ext{H}_{ext{SD}}$)：一种新的用于自动语音识别任务的混合评价指标链接：https://arxiv.org/abs/2211.01722
* 与cs.SD语音【6】为同一篇
作者：Zitha Sasindran,Harsha Yelchuri,Supreeth Rao,T. V. Prabhakar机构：Department of Electronic Systems Engineering, Indian Institute of Science, Bengaluru, India,.摘要：许多研究已经检验了单词错误率（WER）作为自动语音识别（ASR）系统的评估度量的缺点，特别是当用于口语理解任务（例如意图识别和对话系统）时。本文提出了一种新的综合考虑语义正确性和错误率的混合评估方法Hybrid-SD（$\text{H}_{\text{SD}}$）.为了生成句子相异度得分（SD），我们利用提取技术建立了一个快速、轻量级的SNanoBERT模型.我们的实验表明，SNanoBERT模型比SRoBERTa小25.9倍，快38.8倍，同时在著名的基准测试中获得了相当的结果。因此，它适合在边缘设备上部署ASR模型。我们还表明，$\text{H}_{\text{SD}}$与下游任务（如意图识别和命名实体识别（NER））的相关性更强。摘要：Many studies have examined the shortcomings of word error rate (WER) as an evaluation metric for automatic speech recognition (ASR) systems, particularly when used for spoken language understanding tasks such as intent recognition and dialogue systems. In this paper, we propose Hybrid-SD ($\text{H}_{\text{SD}}$), a new hybrid evaluation metric for ASR systems that takes into account both semantic correctness and error rate. To generate sentence dissimilarity scores (SD), we built a fast and lightweight SNanoBERT model using distillation techniques. Our experiments show that the SNanoBERT model is 25.9x smaller and 38.8x faster than SRoBERTa while achieving comparable results on well-known benchmarks. Hence, making it suitable for deploying with ASR models on edge devices. We also show that $\text{H}_{\text{SD}}$ correlates more strongly with downstream tasks such as intent recognition and named-entity recognition (NER).
【19】 The ISCSLP 2022 Intelligent Cockpit Speech Recognition Challenge (ICSRC): Dataset, Tracks, Baseline and Results标题：ISCSLP 2022智能驾驶舱语音识别挑战赛(ICSRC)：数据集、轨迹、基线和结果链接：https://arxiv.org/abs/2211.01585
* 与cs.SD语音【7】为同一篇
作者：Ao Zhang,Fan Yu,Kaixun Huang,Lei Xie,Longbiao Wang,Eng Siong Chng,Hui Bu,Binbin Zhang,Wei Chen,Xin Xu机构：Northwestern Polytechnical University, Xi’an, China, Tianjin Key Laboratory of Cognitive Computing and Application,College of Intelligence and, Computing, Tianjin University, Tianjin, China备注：Accepted by ISCSLP2022摘要：本文总结了ISCSLP 2022智能驾驶舱语音识别挑战赛（ICSRC）的结果。首先阐述了这一挑战的必要性，然后介绍了从一辆新能源汽车（NEV）上收集的相关数据集，涵盖了各种驾驶舱声学条件和语言内容。然后，我们描述了轨道布置和基线系统。具体而言，我们根据允许的模型/系统大小设置了两个跟踪，分别针对车载嵌入式系统和云ASR系统，以研究资源受限和资源不受限的设置。最后，我们总结了挑战结果，并提供了来自提交系统的主要观察结果。摘要：This paper summarizes the outcomes from the ISCSLP 2022 Intelligent Cockpit Speech Recognition Challenge (ICSRC). We first address the necessity of the challenge and then introduce the associated dataset collected from a new-energy vehicle (NEV) covering a variety of cockpit acoustic conditions and linguistic contents. We then describe the track arrangement and the baseline system. Specifically, we set up two tracks in terms of allowed model/system size to investigate resource-constrained and -unconstrained setups, targeting to vehicle embedded as well as cloud ASR systems respectively. Finally we summarize the challenge results and provide the major observations from the submitted systems.
【20】 Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing标题：损失可能是福：将自我监督的语音表示转向高效的多语言和多任务语音处理链接：https://arxiv.org/abs/2211.01522
* 与cs.SD语音【8】为同一篇
作者：Yonggan Fu,Yang Zhang,Kaizhi Qian,Zhifan Ye,Zhongzhi Yu,Cheng-I Lai,Yingyan Lin机构：Georgia Institute of Technology,MIT-IBM Watson AI Lab,Rice University,MIT CSAIL备注：Accepted at NeurIPS 2022摘要：针对丰富语音表示的自监督学习（SSL）已经在低资源自动语音识别（ASR）和其他语音处理任务中取得了经验上的成功，这可以减轻对大量转录语音的需要，并因此推动了对设备上ASR和其他语音处理的日益增长的需求。然而，高级语音SSL模型已经变得越来越大，这与有限的设备上资源相矛盾。在需要同时识别多种语言或执行多个语音处理任务的多语言/多任务场景中，这种差距可能更严重。此外，当在低资源语音语料库上进行微调时，强过参数化的语音SSL模型容易遭受过拟合。本文旨在通过我们提出的S$^3$-Router框架来提高语音SSL模型的实际应用，从而在提高效率和缓解过拟合方面实现双赢。该框架首次发现，通过对语音SSL模型的模型连接进行微调，只需丢弃不超过10%的模型权重，就可以在下游语音处理任务中获得比标准权重微调更好的精度。更重要的是，S$^3$-Router可以作为一种一体化技术，实现（1）一种新的微调方案，（2）一种有效的多语言/多任务解决方案，（3）一种最先进的ASR修剪技术，以及（4）一种定量分析学习到的语音表示的新工具。我们相信S$^3$-Router为语音SSL模型的实际部署提供了一个新的视角。我们的代码可从以下网址获得：https://github.com/GATECH-EIC/S3-Router。摘要：Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S$^3$-Router framework, which for the first time discovers that simply discarding no more than 10\% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S$^3$-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art ASR pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. We believe S$^3$-Router has provided a new perspective for practical deployment of speech SSL models. Our codes are available at: https://github.com/GATECH-EIC/S3-Router.
【21】 Towards Zero-Shot Code-Switched Speech Recognition标题：迈向零发送码切换语音识别链接：https://arxiv.org/abs/2211.01458
* 与cs.SD语音【9】为同一篇
作者：Brian Yan,Matthew Wiesner,Ondrej Klejch,Preethi Jyothi,Shinji Watanabe机构：Carnegie Mellon University, US,Johns Hopkins University, US, University of Edinburgh, UK,Indian Institute of Technology Bombay, IN备注：5 pages摘要：在此工作中，我们寻求建立有效的码切换（CS）自动语音识别系统（ASR）在zero-shot设置下，没有转录的CS语音数据可用于训练。先前提出的框架有条件地将双语任务分解成其组成的单语部分，是有效利用单语数据的一个有希望的起点。然而，这些方法需要单语模块来执行语言分割。也就是说，每个单语模块必须同时检测CS点并转录一种语言的语音片段，而忽略其他语言的语音片段--这不是一项微不足道的任务。我们建议简化每个单语模块，允许它们用单语脚本不加区别地转录所有语音片段（即音译）。这种简单的修改将CS点检测的责任传递给随后的双语模块，所述双语模块通过考虑多个单语音译以及外部语言模型信息来确定最终输出。我们将这种基于音译的方法应用于一个端到端的可微分神经网络，并在中英文SEAME测试集上证明了它对zero-shot CS ASR的有效性。摘要：In this work, we seek to build effective code-switched (CS) automatic speech recognition systems (ASR) under the zero-shot setting where no transcribed CS speech data is available for training. Previously proposed frameworks which conditionally factorize the bilingual task into its constituent monolingual parts are a promising starting point for leveraging monolingual data efficiently. However, these methods require the monolingual modules to perform language segmentation. That is, each monolingual module has to simultaneously detect CS points and transcribe speech segments of one language while ignoring those of other languages -- not a trivial task. We propose to simplify each monolingual module by allowing them to transcribe all speech segments indiscriminately with a monolingual script (i.e. transliteration). This simple modification passes the responsibility of CS point detection to subsequent bilingual modules which determine the final output by considering multiple monolingual transliterations along with external language model information. We apply this transliteration-based approach in an end-to-end differentiable neural network and demonstrate its efficacy for zero-shot CS ASR on Mandarin-English SEAME test sets.
【22】 M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval标题：M-SpeechCLIP：利用大规模、预先训练的模型进行多语言语音到图像检索链接：https://arxiv.org/abs/2211.01180
作者：Layne Berry,Yi-Jen Shih,Hsuan-Fu Wang,Heng-Jui Chang,Hung-yi Lee,David Harwath机构：University of Texas at Austin, National Taiwan University, MIT CSAIL备注：Submitted to ICASSP 2023摘要：本文研究了大规模预训练模型（CLIP和HuBERT）在多语言语音图像检索中的应用。对于非英语语音图像检索，当为每种语言训练单独的模型时，我们的表现远远优于当前最先进的性能，并且表明处理所有三种语言的语音的单个模型仍然实现了与现有最先进的检索分数相当的检索分数。我们确定了英语和非英语设置之间的模型行为和性能的关键差异，这大概是由于CLIP和HuBERT只接受英语的预培训。最后，我们证明了我们的模型可以用于单语和跨语语音-文本检索和跨语语音-语音检索，尽管在训练过程中从未见过任何并行的语音-文本或语音-语音数据。摘要：This work investigates the use of large-scale, pre-trained models (CLIP and HuBERT) for multilingual speech-image retrieval. For non-English speech-image retrieval, we outperform the current state-of-the-art performance by a wide margin when training separate models for each language, and show that a single model which processes speech in all three languages still achieves retrieval scores comparable with the prior state-of-the-art. We identify key differences in model behavior and performance between English and non-English settings, presumably attributable to the English-only pre-training of CLIP and HuBERT. Finally, we show that our models can be used for mono- and cross-lingual speech-text retrieval and cross-lingual speech-speech retrieval, despite never having seen any parallel speech-text or speech-speech data during training.
机器翻译，仅供参考
每日语音速递