今日论文合集：cs.SD语音9篇，eess.AS音频处理15篇。cs.SD语音【1】 Take the aTrain. Introducing an Interface for the Accessible Transcription of Interviews标题：坐aTrain吧。介绍一个可供访问的采访转录界面链接：https://arxiv.org/abs/2310.11967作者：Armin Haberl,Jürgen Fleiß,Dominik Kowald,Stefan Thalmann机构：Business Analytics and Data Science-Center, University of Graz, Universit¨atsstraße, Know Center & Graz University of Technology, Sandgasse , Graz, Austria备注：Install via Microsoft store: apps.microsoft.com/store/detail/atrain/9N15Q44SZNS2. Github: github.com/BANDAS-Center/aTrain摘要：aTrain是一个开源的离线工具，用于在CPU和NVIDIA GPU支持下以多种语言转录音频数据。它是专门为研究人员设计的，使用从与研究参与者的各种形式的语音交互中生成的定性数据。aTrain不需要编程技能，在大多数计算机上运行，不需要互联网连接，并且经过验证不会将数据上传到任何服务器。aTrain将OpenAI的Whisper模型与说话人识别相结合，提供与流行的定性数据分析软件工具MAXQDA和ATLAS.ti集成的输出。它具有易于使用的图形界面，并通过Microsoft Store作为Windows应用程序提供，允许研究人员简单安装。源代码在GitHub上免费提供。在开发了一个专注于本地计算机速度的aTrain之后，我们发现，使用最高精度的转录模型，当前移动CPU上的转录时间大约是音频文件持续时间的2到3倍。如果有入门级显卡可用，转录速度将增加到音频持续时间的20%。摘要：aTrain is an open-source and offline tool for transcribing audio data in multiple languages with CPU and NVIDIA GPU support. It is specifically designed for researchers using qualitative data generated from various forms of speech interactions with research participants. aTrain requires no programming skills, runs on most computers, does not require an internet connection, and was verified not to upload data to any server. aTrain combines OpenAI's Whisper model with speaker recognition to provide output that integrates with the popular qualitative data analysis software tools MAXQDA and ATLAS.ti. It has an easy-to-use graphical interface and is provided as a Windows-App through the Microsoft Store allowing for simple installation by researchers. The source code is freely available on GitHub. Having developed aTrain with a focus on speed on local computers, we show that the transcription time on current mobile CPUs is around 2 to 3 times the duration of the audio file using the highest-accuracy transcription models. If an entry-level graphics card is available, the transcription speed increases to 20% of the audio duration.【2】 BUT CHiME-7 system description标题：但CHINE-7系统描述链接：https://arxiv.org/abs/2310.11921作者：Martin Karafiát,Karel Veselý,Igor Szöke,Ladislav Mošner,Karel Beneš,Marcin Witkowski,Germán Barchi,Leonardo Pepino机构：Brno University of Technology, AGH University of Krakow, University of Buenos Aires备注：6 pages, Chime-7 challenge 2023摘要：本文介绍了布尔诺理工大学（BUT），克拉科夫AGH大学和布宜诺斯艾利斯大学为CHiME-7挑战赛开发自动语音识别系统的共同努力。我们使用多个工具包对各种端到端模型进行训练和评估。我们严重依赖引导源分离（GSS）将多声道音频转换为单声道。ASR利用了通过自监督学习预训练的模型的语音表示，我们融合了几个ASR系统。此外，我们修改了LibriSpeech语料库中的外部数据，使其成为一个封闭的域，并将其添加到训练中。我们的努力集中在任务1 -远距离自动语音识别（DASR）的远场声学鲁棒性子轨道上，我们的系统使用oracle分割。摘要：This paper describes the joint effort of Brno University of Technology (BUT), AGH University of Krakow and University of Buenos Aires on the development of Automatic Speech Recognition systems for the CHiME-7 Challenge. We train and evaluate various end-to-end models with several toolkits. We heavily relied on Guided Source Separation (GSS) to convert multi-channel audio to single channel. The ASR is leveraging speech representations from models pre-trained by self-supervised learning, and we do a fusion of several ASR systems. In addition, we modified external data from the LibriSpeech corpus to become a close domain and added it to the training. Our efforts were focused on the far-field acoustic robustness sub-track of Task 1 - Distant Automatic Speech Recognition (DASR), our systems use oracle segmentation.【3】 CLARA: Multilingual Contrastive Learning for Audio Representation Acquisition标题：Clara：用于音频表征习得的多语言对比学习链接：https://arxiv.org/abs/2310.11830作者：Kari A Noriy,Xiaosong Yang,Marcin Budka,Jian Jun Zhang机构：∗Centre for Digital Entertainment, †Computer Science and Informatics, ‡National Centre for Computer Animation, Bournemouth University摘要：本文提出了一种新的框架，多语言的语音和声音表示学习使用对比学习。缺乏大规模的标记数据集阻碍了跨语言的语音处理研究。对比学习的最新进展提供了从未标记数据中学习的自监督技术。出于减少数据依赖和提高不同语言和条件下的泛化能力，我们开发了一个多语言对比框架。该框架使模型能够获得跨语言的共享表示，从而促进了有限目标语言数据的跨语言迁移。 此外，由于主观感知评估，捕获语音中的情感线索具有挑战性。通过以自我监督的方式从不同的多语言数据中学习表达性表征，我们的方法旨在开发编码情感维度的语音表征。 我们的方法在一个大型的多语言音频数据语料库上训练编码器。数据扩充技术被用来扩展数据集。对比学习方法训练模型以最大化正对之间的一致性，并最小化负对之间的一致性。大量的实验证明了所提出的模型在情感识别，音频分类和检索基准下的最先进的性能zero-shot和Few-Shot条件。这提供了一种有效的方法，用于在编码潜在的情感维度的同时获取跨语言和声学条件的共享和广义语音表示。摘要：This paper proposes a novel framework for multilingual speech and sound representation learning using contrastive learning. The lack of sizeable labelled datasets hinders speech-processing research across languages. Recent advances in contrastive learning provide self-supervised techniques to learn from unlabelled data. Motivated by reducing data dependence and improving generalisation across diverse languages and conditions, we develop a multilingual contrastive framework. This framework enables models to acquire shared representations across languages, facilitating cross-lingual transfer with limited target language data. Additionally, capturing emotional cues within speech is challenging due to subjective perceptual assessments. By learning expressive representations from diverse, multilingual data in a self-supervised manner, our approach aims to develop speech representations that encode emotive dimensions. Our method trains encoders on a large corpus of multi-lingual audio data. Data augmentation techniques are employed to expand the dataset. The contrastive learning approach trains the model to maximise agreement between positive pairs and minimise agreement between negative pairs. Extensive experiments demonstrate state-of-the-art performance of the proposed model on emotion recognition, audio classification, and retrieval benchmarks under zero-shot and few-shot conditions. This provides an effective approach for acquiring shared and generalised speech representations across languages and acoustic conditions while encoding latent emotional dimensions.【4】 Physics-informed Neural Network for Acoustic Resonance Analysis标题：用于声学共振分析的物理信息神经网络链接：https://arxiv.org/abs/2310.11804作者：Kazuya Yokota,Takahiko Kurahashi,Masajiro Abe机构：Department of Mechanical Engineering, Nagaoka University of Technology, Nagaoka, Niigata,-, Japan, Department of System Safety Engineering, Nagaoka University of Technology备注：11 pages, 14 figures. The following article has been submitted to the Journal of the Acoustical Society of America. After it is published, it will be found at this https URL摘要：本研究提出了物理信息神经网络（PINN）框架来求解声波共振分析的波动方程。本研究中提出的分析模型ResoNet除了传统的PINN损失函数外，还最大限度地减少了周期解的损失函数，从而有效地利用了神经网络的函数逼近能力，同时进行共振分析。此外，它可以很容易地应用到反问题。本文分析了一维声管中的共振现象。通过对含能量损失项波动方程的正、反演分析，验证了该方法的有效性。在正演分析中，通过与有限差分法的比较，评价了PINN法对共振问题的适用性。反演分析，其中包括识别的能量损失项的波动方程和声管的设计优化，具有良好的精度。摘要：This study proposes the physics-informed neural network (PINN) framework to solve the wave equation for acoustic resonance analysis. ResoNet, the analytical model proposed in this study, minimizes the loss function for periodic solutions, in addition to conventional PINN loss functions, thereby effectively using the function approximation capability of neural networks, while performing resonance analysis. Additionally, it can be easily applied to inverse problems. Herein, the resonance in a one-dimensional acoustic tube was analyzed. The effectiveness of the proposed method was validated through the forward and inverse analyses of the wave equation with energy-loss terms. In the forward analysis, the applicability of PINN to the resonance problem was evaluated by comparison with the finite-difference method. The inverse analysis, which included the identification of the energy loss term in the wave equation and design optimization of the acoustic tube, was performed with good accuracy.【5】 Blind estimation of audio effects using an auto-encoder approach and differentiable signal processing标题：基于自动编码法和可微信号处理的音频效果盲估计链接：https://arxiv.org/abs/2310.11781作者：Côme Peladeau,Geoffroy Peeters机构：LTCI, T´el´ecom-Paris, Institut Polytechnique de Paris, France摘要：音频效果盲估计（BE-AFX）旨在仅基于经处理的音频样本来估计应用于原始未经处理的音频样本的音频效果（AFX）。为了训练这样的系统，传统方法优化地面实况和估计的AFX参数之间的损失。这涉及到了解用于流程的AFX的确切实现。在这项工作中，我们提出了一种替代解决方案，消除了了解这种实现的要求。相反，我们引入了一种自动编码器方法，它优化了音频质量指标。我们探索，建议和比较常用的掌握AFX的各种实现，使用差分信号处理或神经近似。我们的研究结果表明，与传统的基于参数的方法相比，我们的自动编码器方法对AFX链产生的音频质量产生了更好的估计，即使后者提供了更准确的参数估计。摘要：Blind Estimation of Audio Effects (BE-AFX) aims at estimating the Audio Effects (AFXs) applied to an original, unprocessed audio sample solely based on the processed audio sample. To train such a system traditional approaches optimize a loss between ground truth and estimated AFX parameters. This involves knowing the exact implementation of the AFXs used for the process. In this work, we propose an alternative solution that eliminates the requirement for knowing this implementation. Instead, we introduce an auto-encoder approach, which optimizes an audio quality metric. We explore, suggest, and compare various implementations of commonly used mastering AFXs, using differential signal processing or neural approximations. Our findings demonstrate that our auto-encoder approach yields superior estimates of the audio quality produced by a chain of AFXs, compared to the traditional parameter-based approach, even if the latter provides a more accurate parameter estimation.【6】 Unintended Memorization in Large ASR Models, and How to Mitigate It标题：大型ASR模型中的意外记忆以及如何减少这种记忆链接：https://arxiv.org/abs/2310.11739作者：Lun Wang,Om Thakkar,Rajiv Mathews机构：Google摘要：众所周知，神经网络可能会无意中记住它们的训练样本，从而引起隐私问题。然而，审计记忆在大型非自回归自动语音识别（ASR）模型一直具有挑战性，由于现有的方法，如硬度校准的高计算成本。在这项工作中，我们设计了一个简单的审计方法来衡量大型ASR模型的记忆，而无需额外的计算开销。具体地说，我们加快了随机生成的话语，以创建语音和文本信息之间的映射，这是很难从典型的训练示例中学习的。因此，仅针对加速训练示例的准确预测可以作为记忆的明确证据，并且相应的准确度可以用于测量记忆。使用所提出的方法，我们展示了最先进的ASR模型中的记忆。为了减轻记忆，我们在训练过程中尝试了梯度裁剪，以限制任何单个示例对最终模型的影响。我们的经验表明，裁剪每个例子的梯度可以减轻记忆的加速训练的例子，在训练集上重复多达16次。此外，我们证明了在大规模分布式训练中，削减每个计算核心上的平均梯度可以保持中性模型质量和计算成本，同时提供强大的隐私保护。摘要：It is well-known that neural networks can unintentionally memorize their training examples, causing privacy concerns. However, auditing memorization in large non-auto-regressive automatic speech recognition (ASR) models has been challenging due to the high compute cost of existing methods such as hardness calibration. In this work, we design a simple auditing method to measure memorization in large ASR models without the extra compute overhead. Concretely, we speed up randomly-generated utterances to create a mapping between vocal and text information that is difficult to learn from typical training examples. Hence, accurate predictions only for sped-up training examples can serve as clear evidence for memorization, and the corresponding accuracy can be used to measure memorization. Using the proposed method, we showcase memorization in the state-of-the-art ASR models. To mitigate memorization, we tried gradient clipping during training to bound the influence of any individual example on the final model. We empirically show that clipping each example's gradient can mitigate memorization for sped-up training examples with up to 16 repetitions in the training set. Furthermore, we show that in large-scale distributed training, clipping the average gradient on each compute core maintains neutral model quality and compute cost while providing strong privacy protection.【7】 EchoScan: Scanning Complex Indoor Geometries via Acoustic Echoes标题：回声扫描：通过声学回波扫描复杂的室内几何图形链接：https://arxiv.org/abs/2310.11728作者：Inmo Yeon,Iljoo Jeong,Seungchul Lee,Jung-Woo Choi机构：the Institute of Civil-Military Tech-nology Cooperation funded by the Defense Acquisition Program Ad-ministration and Ministry of Trade备注：9 pages, 8 figures摘要：准确估计室内空间几何形状对于构建精确的数字孪生模型至关重要，其广泛的工业应用包括在陌生环境中的导航和有效的疏散规划，特别是在低光照条件下。这项研究介绍了EchoScan，这是一种深度神经网络模型，它利用声学回声来进行房间几何推断。传统的基于声音的技术依赖于估计几何形状相关的房间参数，例如墙壁位置和房间大小，从而限制了可推断的房间几何形状的多样性。实际上，EchoScan通过直接推断房间平面图和高度来克服这一限制，从而使其能够处理具有任意形状的房间，包括弯曲的墙壁。EchoScan的关键创新在于它能够使用多聚合模块分析房间脉冲响应（RIR）中低阶和高阶反射之间的复杂关系。高阶反射的分析还使其能够在回声无法从音频设备的位置观察到时推断复杂的房间形状。在此，EchoScan使用从复杂环境（包括曼哈顿和亚特兰大布局）合成的RIR进行训练和评估，采用与商用现成设备兼容的实用音频设备配置。与基于视觉的方法相比，EchoScan在各种形状的房间中表现出出色的几何估计性能。摘要：Accurate estimation of indoor space geometries is vital for constructing precise digital twins, whose broad industrial applications include navigation in unfamiliar environments and efficient evacuation planning, particularly in low-light conditions. This study introduces EchoScan, a deep neural network model that utilizes acoustic echoes to perform room geometry inference. Conventional sound-based techniques rely on estimating geometry-related room parameters such as wall position and room size, thereby limiting the diversity of inferable room geometries. Contrarily, EchoScan overcomes this limitation by directly inferring room floorplans and heights, thereby enabling it to handle rooms with arbitrary shapes, including curved walls. The key innovation of EchoScan is its ability to analyze the complex relationship between low- and high-order reflections in room impulse responses (RIRs) using a multi-aggregation module. The analysis of high-order reflections also enables it to infer complex room shapes when echoes are unobservable from the position of an audio device. Herein, EchoScan was trained and evaluated using RIRs synthesized from complex environments, including the Manhattan and Atlanta layouts, employing a practical audio device configuration compatible with commercial, off-the-shelf devices. Compared with vision-based methods, EchoScan demonstrated outstanding geometry estimation performance in rooms with various shapes.【8】 Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation标题：分离看不见的声音走向通用视听场景感知声音分离链接：https://arxiv.org/abs/2310.11713作者：Yiyang Su,Ali Vosoughi,Shijian Deng,Yapeng Tian,Chenliang Xu机构：Michigan State University, University of Rochester, University of Texas at Dallas备注：Accepted at ICCV 2023 - AV4D, 4 figures, 3 tables摘要：视听声音分离字段假设视频中的可见源，但这排除了摄像机视野之外的不可见声音。目前的方法难以处理缺乏可见线索的这种声音。本文介绍了一种新的“视听场景感知分离”（AVSA-Sep）框架。它包括一个用于可见和不可见声音的语义解析器和一个用于场景分离的分离器。AVSA-Sep成功地分离了两种声音类型，联合训练和跨模态对齐提高了效果。摘要：The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds lacking visible cues. This paper introduces a novel "Audio-Visual Scene-Aware Separation" (AVSA-Sep) framework. It includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation. AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness.【9】 Experimental Results of Underwater Sound Speed Profile Inversion by Few-shot Multi-task Learning标题：基于Few-Shot多任务学习的水下声速剖面反演实验结果链接：https://arxiv.org/abs/2310.11708作者：Wei Huang,Fan Gao,Junting Wang,Hao Zhang机构：• To achieve good accuracy performance of SSP inversion under few-shot learning situation, we propose the MTL, approach. Through learning on multi-task (different kinds of SSPs), the common features of SSPs are extracted摘要：水下声速剖面分布对声信号的传播模式有很大影响，因此快速准确地估计出水下声速剖面分布对建立水下观测系统具有重要意义。现有的SSP反演方法包括匹配场处理（MFP）、压缩感知（CS）和前馈神经网络（FNN）等，其中FNN在保持相同精度的同时，具有更好的实时性。然而，模糊神经网络的训练需要大量的历史SSP样本，这在许多海洋领域很难得到满足。这种情况称为Few-Shot学习。为了解决这个问题，我们提出了一个多任务学习（MTL）模型与部分参数共享之间的不同训练任务。通过MTL提取共同特征，加快了对给定任务的学习过程，减少了对参考样本的需求，从而提高了Few-Shot学习的泛化能力。为了验证MTL的可行性和有效性，2023年4月在南海进行了深海实验。结果表明，MTL在SSP反演精度方面优于现有方法，同时在反演阶段继承了FNN的实时性优势。摘要：Underwater Sound Speed Profile (SSP) distribution has great influence on the propagation mode of acoustic signal, thus the fast and accurate estimation of SSP is of great importance in building underwater observation systems. The state-of-the-art SSP inversion methods include frameworks of matched field processing (MFP), compressive sensing (CS), and feedforeward neural networks (FNN), among which the FNN shows better real-time performance while maintain the same level of accuracy. However, the training of FNN needs quite a lot historical SSP samples, which is diffcult to be satisfied in many ocean areas. This situation is called few-shot learning. To tackle this issue, we propose a multi-task learning (MTL) model with partial parameter sharing among different traning tasks. By MTL, common features could be extracted, thus accelerating the learning process on given tasks, and reducing the demand for reference samples, so as to enhance the generalization ability in few-shot learning. To verify the feasibility and effectiveness of MTL, a deep-ocean experiment was held in April 2023 at the South China Sea. Results shows that MTL outperforms the state-of-the-art methods in terms of accuracy for SSP inversion, while inherits the real-time advantage of FNN during the inversion stage.eess.AS音频处理【1】 DASA: Difficulty-Aware Semantic Augmentation for Speaker Verification标题：DASA：说话人确认中的难度感知语义增强链接：https://arxiv.org/abs/2310.12111作者：Yuanyuan Wang,Yang Zhang,Zhiyong Wu,Zhihan Yang,Tao Wei,Kun Zou,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Ping An Technology, Shenzhen, China, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by ICASSP 2023摘要：数据增强对于深度神经网络（DNN）模型的泛化能力和鲁棒性至关重要。现有的说话人确认增强方法需要对原始信号进行处理，耗时长，增强后的样本缺乏多样性。在本文中，我们提出了一种新的困难意识的语义增强（DASA）的说话人确认方法，它可以产生多样化的训练样本在说话人嵌入空间，可以忽略不计的额外计算成本。首先，我们通过沿着语义方向扰动说话人嵌入来增加训练样本，这些嵌入是从说话人协方差矩阵中获得的。其次，准确的协方差矩阵估计从鲁棒的说话人嵌入在训练过程中，所以我们引入困难意识的附加余量softmax（DAAM-Softmax），以获得最佳的说话人嵌入。最后，我们假设增强样本的数量趋于无穷大，并推导出封闭形式的DASA，实现兼容性和效率的预期损失的上界。大量的实验表明，所提出的方法可以实现显着的性能改善。最佳结果在CN-Celeb评估集上实现了EER度量的14.6%相对降低。摘要：Data augmentation is vital to the generalization ability and robustness of deep neural networks (DNNs) models. Existing augmentation methods for speaker verification manipulate the raw signal, which are time-consuming and the augmented samples lack diversity. In this paper, we present a novel difficulty-aware semantic augmentation (DASA) approach for speaker verification, which can generate diversified training samples in speaker embedding space with negligible extra computing cost. Firstly, we augment training samples by perturbing speaker embeddings along semantic directions, which are obtained from speaker-wise covariance matrices. Secondly, accurate covariance matrices are estimated from robust speaker embeddings during training, so we introduce difficultyaware additive margin softmax (DAAM-Softmax) to obtain optimal speaker embeddings. Finally, we assume the number of augmented samples goes to infinity and derive a closed-form upper bound of the expected loss with DASA, which achieves compatibility and efficiency. Extensive experiments demonstrate the proposed approach can achieve a remarkable performance improvement. The best result achieves a 14.6% relative reduction in EER metric on CN-Celeb evaluation set.【2】 Enhancing Spoofing Speech Detection Using Rhythm Information标题：利用韵律信息增强欺骗语音检测链接：https://arxiv.org/abs/2310.12014作者：Jingze Lu,Yuxiang Zhang,Wenchao Wang,Zengqiang Shang,Pengyuan Zhang机构：Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, University of Chinese Academy of Sciences, China备注：5 pages, 2 figures, submitted to icassp2024摘要：欺骗语音检测是一个热门的研究领域。然而，目前的欺骗语音检测系统缺乏令人信服的证据。为了提高检测系统的可靠性，本文分析了TTS生成语音中固有的节奏信息缺陷。TTS模型以文本作为输入，利用声学模型来预测节奏信息，这在节奏信息中引入了伪影。通过滤除声道反应，保留的声门流与节奏信息保留检测能力TTS生成的语音。基于这些分析，节奏扰动模块，提出了加强复制合成数据增强方法。该方法产生的虚假语音迫使检测模型关注节奏信息中的伪影，有效地提高了反欺骗对策对TTS生成语音的检测能力。摘要：Spoofing speech detection is a hot and in-demand research field. However, current spoofing speech detection systems is lack of convincing evidence. In this paper, to increase the reliability of detection systems, the flaws of rhythm information inherent in the TTS-generated speech are analyzed. TTS models take text as input and utilize acoustic models to predict rhythm information, which introduces artifacts in the rhythm information. By filtering out vocal tract response, the remaining glottal flow with rhythm information retains detection ability for TTS-generated speech. Based on these analyses, a rhythm perturbation module is proposed to enhance the copy-synthesis data augmentation method. Fake utterances generated by the proposed method force the detecting model to pay attention to the artifacts in rhythm information and effectively improve the ability to detect TTS-generated speech of the anti-spoofing countermeasures.【3】 Experimental Results of Underwater Sound Speed Profile Inversion by Few-shot Multi-task Learning标题：基于Few-Shot多任务学习的水下声速剖面反演实验结果链接：https://arxiv.org/abs/2310.11708作者：Wei Huang,Fan Gao,Junting Wang,Hao Zhang机构：• To achieve good accuracy performance of SSP inversion under few-shot learning situation, we propose the MTL, approach. Through learning on multi-task (different kinds of SSPs), the common features of SSPs are extracted摘要：水下声速剖面分布对声信号的传播模式有很大影响，因此快速准确地估计出水下声速剖面分布对建立水下观测系统具有重要意义。现有的SSP反演方法包括匹配场处理（MFP）、压缩感知（CS）和前馈神经网络（FNN）等，其中FNN在保持相同精度的同时，具有更好的实时性。然而，模糊神经网络的训练需要大量的历史SSP样本，这在许多海洋领域很难得到满足。这种情况称为Few-Shot学习。为了解决这个问题，我们提出了一个多任务学习（MTL）模型与部分参数共享之间的不同训练任务。通过MTL提取共同特征，加快了对给定任务的学习过程，减少了对参考样本的需求，从而提高了Few-Shot学习的泛化能力。为了验证MTL的可行性和有效性，2023年4月在南海进行了深海实验。结果表明，MTL在SSP反演精度方面优于现有方法，同时在反演阶段继承了FNN的实时性优势。摘要：Underwater Sound Speed Profile (SSP) distribution has great influence on the propagation mode of acoustic signal, thus the fast and accurate estimation of SSP is of great importance in building underwater observation systems. The state-of-the-art SSP inversion methods include frameworks of matched field processing (MFP), compressive sensing (CS), and feedforeward neural networks (FNN), among which the FNN shows better real-time performance while maintain the same level of accuracy. However, the training of FNN needs quite a lot historical SSP samples, which is diffcult to be satisfied in many ocean areas. This situation is called few-shot learning. To tackle this issue, we propose a multi-task learning (MTL) model with partial parameter sharing among different traning tasks. By MTL, common features could be extracted, thus accelerating the learning process on given tasks, and reducing the demand for reference samples, so as to enhance the generalization ability in few-shot learning. To verify the feasibility and effectiveness of MTL, a deep-ocean experiment was held in April 2023 at the South China Sea. Results shows that MTL outperforms the state-of-the-art methods in terms of accuracy for SSP inversion, while inherits the real-time advantage of FNN during the inversion stage.【4】 End-to-End real time tracking of children's reading with pointer network标题：利用指针网络实现端到端的儿童阅读实时跟踪链接：https://arxiv.org/abs/2310.11486作者：Vishal Sunder,Beulah Karrolla,Eric Fosler-Lussier机构：The Ohio State University备注：5 pages, 3 figures摘要：在这项工作中，我们将探讨如何实时阅读跟踪器可以有效地建立儿童的声音。虽然以前提出的阅读跟踪器专注于基于ASR的级联方法，但我们提出了一个完全端到端的模型，使其不太容易出现语音跟踪滞后。我们采用了一个指针网络，它直接学习预测以流媒体语音为条件的地面真实文本中的位置。为了训练这个指针网络，我们通过使用在训练集上读取的语音和正在读取的文本之间的强制对齐来生成地面实况训练信号。探索不同的强制对齐模型，我们发现基于神经注意力的模型在对齐精度上至少与蒙特利尔强制对齐器一样接近，但令人惊讶的是，它是指针网络的更好的训练信号。我们的研究结果报告一个成人的语音数据（TIMIT）和两个儿童的语音数据集（CMU儿童和阅读比赛）。我们最好的模型可以准确地跟踪成人语音，准确率为87.8%，更难和不流利的儿童语音，CMU Kids数据的准确率为77.1%，Reading Races数据集的准确率为65.3%。摘要：In this work, we explore how a real time reading tracker can be built efficiently for children's voices. While previously proposed reading trackers focused on ASR-based cascaded approaches, we propose a fully end-to-end model making it less prone to lags in voice tracking. We employ a pointer network that directly learns to predict positions in the ground truth text conditioned on the streaming speech. To train this pointer network, we generate ground truth training signals by using forced alignment between the read speech and the text being read on the training set. Exploring different forced alignment models, we find a neural attention based model is at least as close in alignment accuracy to the Montreal Forced Aligner, but surprisingly is a better training signal for the pointer network. Our results are reported on one adult speech data (TIMIT) and two children's speech datasets (CMU Kids and Reading Races). Our best model can accurately track adult speech with 87.8% accuracy and the much harder and disfluent children's speech with 77.1% accuracy on CMU Kids data and a 65.3% accuracy on the Reading Races dataset.【5】 Take the aTrain. Introducing an Interface for the Accessible Transcription of Interviews标题：坐aTrain吧。介绍一个可供访问的采访转录界面链接：https://arxiv.org/abs/2310.11967作者：Armin Haberl,Jürgen Fleiß,Dominik Kowald,Stefan Thalmann机构：Business Analytics and Data Science-Center, University of Graz, Universit¨atsstraße, Know Center & Graz University of Technology, Sandgasse , Graz, Austria备注：Install via Microsoft store: apps.microsoft.com/store/detail/atrain/9N15Q44SZNS2. Github: github.com/BANDAS-Center/aTrain摘要：aTrain是一个开源的离线工具，用于在CPU和NVIDIA GPU支持下以多种语言转录音频数据。它是专门为研究人员设计的，使用从与研究参与者的各种形式的语音交互中生成的定性数据。aTrain不需要编程技能，在大多数计算机上运行，不需要互联网连接，并且经过验证不会将数据上传到任何服务器。aTrain将OpenAI的Whisper模型与说话人识别相结合，提供与流行的定性数据分析软件工具MAXQDA和ATLAS.ti集成的输出。它具有易于使用的图形界面，并通过Microsoft Store作为Windows应用程序提供，允许研究人员简单安装。源代码在GitHub上免费提供。在开发了一个专注于本地计算机速度的aTrain之后，我们发现，使用最高精度的转录模型，当前移动CPU上的转录时间大约是音频文件持续时间的2到3倍。如果有入门级显卡可用，转录速度将增加到音频持续时间的20%。摘要：aTrain is an open-source and offline tool for transcribing audio data in multiple languages with CPU and NVIDIA GPU support. It is specifically designed for researchers using qualitative data generated from various forms of speech interactions with research participants. aTrain requires no programming skills, runs on most computers, does not require an internet connection, and was verified not to upload data to any server. aTrain combines OpenAI's Whisper model with speaker recognition to provide output that integrates with the popular qualitative data analysis software tools MAXQDA and ATLAS.ti. It has an easy-to-use graphical interface and is provided as a Windows-App through the Microsoft Store allowing for simple installation by researchers. The source code is freely available on GitHub. Having developed aTrain with a focus on speed on local computers, we show that the transcription time on current mobile CPUs is around 2 to 3 times the duration of the audio file using the highest-accuracy transcription models. If an entry-level graphics card is available, the transcription speed increases to 20% of the audio duration.【6】 MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models标题：MusicAgent：一个面向大型语言模型的音乐理解和生成人工智能代理链接：https://arxiv.org/abs/2310.11954作者：Dingyao Yu,Kaitao Song,Peiling Lu,Tianyu He,Xu Tan,Wei Ye,Shikun Zhang,Jiang Bian机构：Peking University, Microsoft Research Asia摘要：人工智能授权的音乐处理是一个多样化的领域，包括数十个任务，从生成任务（例如，音色合成）到理解任务（例如，音乐分类）。对于开发人员和业余爱好者来说，很难掌握所有这些任务来满足他们在音乐处理方面的需求，特别是考虑到音乐数据的表示方法和各种任务之间跨平台的模型适用性的巨大差异。因此，有必要建立一个系统来组织和集成这些任务，从而帮助从业者自动分析他们的需求，并调用合适的工具作为解决方案，以满足他们的需求。受最近大语言模型（LLM）在任务自动化方面取得的成功的启发，我们开发了一个名为MusicAgent的系统，该系统集成了许多与音乐相关的工具和自主工作流，以满足用户的需求。更具体地说，我们构建了1）工具集，收集了来自不同来源的工具，包括Hugging Face，GitHub和Web API等。2）由LLM授权的自主工作流（例如，ChatGPT）来组织这些工具，并自动将用户请求分解为多个子任务，调用相应的音乐工具。该系统的主要目标是将用户从复杂的人工智能音乐工具中解放出来，使他们能够专注于创作方面。通过赋予用户轻松组合工具的自由，该系统提供了无缝和丰富的音乐体验。摘要：AI-empowered music processing is a diverse field that encompasses dozens of tasks, ranging from generation tasks (e.g., timbre synthesis) to comprehension tasks (e.g., music classification). For developers and amateurs, it is very difficult to grasp all of these task to satisfy their requirements in music processing, especially considering the huge differences in the representations of music data and the model applicability across platforms among various tasks. Consequently, it is necessary to build a system to organize and integrate these tasks, and thus help practitioners to automatically analyze their demand and call suitable tools as solutions to fulfill their requirements. Inspired by the recent success of large language models (LLMs) in task automation, we develop a system, named MusicAgent, which integrates numerous music-related tools and an autonomous workflow to address user requirements. More specifically, we build 1) toolset that collects tools from diverse sources, including Hugging Face, GitHub, and Web API, etc. 2) an autonomous workflow empowered by LLMs (e.g., ChatGPT) to organize these tools and automatically decompose user requests into multiple sub-tasks and invoke corresponding music tools. The primary goal of this system is to free users from the intricacies of AI-music tools, enabling them to concentrate on the creative aspect. By granting users the freedom to effortlessly combine tools, the system offers a seamless and enriching music experience.【7】 BUT CHiME-7 system description标题：但CHINE-7系统描述链接：https://arxiv.org/abs/2310.11921作者：Martin Karafiát,Karel Veselý,Igor Szöke,Ladislav Mošner,Karel Beneš,Marcin Witkowski,Germán Barchi,Leonardo Pepino机构：Brno University of Technology, AGH University of Krakow, University of Buenos Aires备注：6 pages, Chime-7 challenge 2023摘要：本文介绍了布尔诺理工大学（BUT），克拉科夫AGH大学和布宜诺斯艾利斯大学为CHiME-7挑战赛开发自动语音识别系统的共同努力。我们使用多个工具包对各种端到端模型进行训练和评估。我们严重依赖引导源分离（GSS）将多声道音频转换为单声道。ASR利用了通过自监督学习预训练的模型的语音表示，我们融合了几个ASR系统。此外，我们修改了LibriSpeech语料库中的外部数据，使其成为一个封闭的域，并将其添加到训练中。我们的努力集中在任务1 -远距离自动语音识别（DASR）的远场声学鲁棒性子轨道上，我们的系统使用oracle分割。摘要：This paper describes the joint effort of Brno University of Technology (BUT), AGH University of Krakow and University of Buenos Aires on the development of Automatic Speech Recognition systems for the CHiME-7 Challenge. We train and evaluate various end-to-end models with several toolkits. We heavily relied on Guided Source Separation (GSS) to convert multi-channel audio to single channel. The ASR is leveraging speech representations from models pre-trained by self-supervised learning, and we do a fusion of several ASR systems. In addition, we modified external data from the LibriSpeech corpus to become a close domain and added it to the training. Our efforts were focused on the far-field acoustic robustness sub-track of Task 1 - Distant Automatic Speech Recognition (DASR), our systems use oracle segmentation.【8】 CLARA: Multilingual Contrastive Learning for Audio Representation Acquisition标题：Clara：用于音频表征习得的多语言对比学习链接：https://arxiv.org/abs/2310.11830作者：Kari A Noriy,Xiaosong Yang,Marcin Budka,Jian Jun Zhang机构：∗Centre for Digital Entertainment, †Computer Science and Informatics, ‡National Centre for Computer Animation, Bournemouth University摘要：本文提出了一种新的框架，多语言的语音和声音表示学习使用对比学习。缺乏大规模的标记数据集阻碍了跨语言的语音处理研究。对比学习的最新进展提供了从未标记数据中学习的自监督技术。出于减少数据依赖和提高不同语言和条件下的泛化能力，我们开发了一个多语言对比框架。该框架使模型能够获得跨语言的共享表示，从而促进了有限目标语言数据的跨语言迁移。 此外，由于主观感知评估，捕获语音中的情感线索具有挑战性。通过以自我监督的方式从不同的多语言数据中学习表达性表征，我们的方法旨在开发编码情感维度的语音表征。 我们的方法在一个大型的多语言音频数据语料库上训练编码器。数据扩充技术被用来扩展数据集。对比学习方法训练模型以最大化正对之间的一致性，并最小化负对之间的一致性。大量的实验证明了所提出的模型在情感识别，音频分类和检索基准下的最先进的性能zero-shot和Few-Shot条件。这提供了一种有效的方法，用于在编码潜在的情感维度的同时获取跨语言和声学条件的共享和广义语音表示。摘要：This paper proposes a novel framework for multilingual speech and sound representation learning using contrastive learning. The lack of sizeable labelled datasets hinders speech-processing research across languages. Recent advances in contrastive learning provide self-supervised techniques to learn from unlabelled data. Motivated by reducing data dependence and improving generalisation across diverse languages and conditions, we develop a multilingual contrastive framework. This framework enables models to acquire shared representations across languages, facilitating cross-lingual transfer with limited target language data. Additionally, capturing emotional cues within speech is challenging due to subjective perceptual assessments. By learning expressive representations from diverse, multilingual data in a self-supervised manner, our approach aims to develop speech representations that encode emotive dimensions. Our method trains encoders on a large corpus of multi-lingual audio data. Data augmentation techniques are employed to expand the dataset. The contrastive learning approach trains the model to maximise agreement between positive pairs and minimise agreement between negative pairs. Extensive experiments demonstrate state-of-the-art performance of the proposed model on emotion recognition, audio classification, and retrieval benchmarks under zero-shot and few-shot conditions. This provides an effective approach for acquiring shared and generalised speech representations across languages and acoustic conditions while encoding latent emotional dimensions.【9】 Physics-informed Neural Network for Acoustic Resonance Analysis标题：用于声学共振分析的物理信息神经网络链接：https://arxiv.org/abs/2310.11804作者：Kazuya Yokota,Takahiko Kurahashi,Masajiro Abe机构：Department of Mechanical Engineering, Nagaoka University of Technology, Nagaoka, Niigata,-, Japan, Department of System Safety Engineering, Nagaoka University of Technology备注：11 pages, 14 figures. The following article has been submitted to the Journal of the Acoustical Society of America. After it is published, it will be found at this https URL摘要：本研究提出了物理信息神经网络（PINN）框架来求解声波共振分析的波动方程。本研究中提出的分析模型ResoNet除了传统的PINN损失函数外，还最大限度地减少了周期解的损失函数，从而有效地利用了神经网络的函数逼近能力，同时进行共振分析。此外，它可以很容易地应用到反问题。本文分析了一维声管中的共振现象。通过对含能量损失项波动方程的正、反演分析，验证了该方法的有效性。在正演分析中，通过与有限差分法的比较，评价了PINN法对共振问题的适用性。反演分析，其中包括识别的能量损失项的波动方程和声管的设计优化，具有良好的精度。摘要：This study proposes the physics-informed neural network (PINN) framework to solve the wave equation for acoustic resonance analysis. ResoNet, the analytical model proposed in this study, minimizes the loss function for periodic solutions, in addition to conventional PINN loss functions, thereby effectively using the function approximation capability of neural networks, while performing resonance analysis. Additionally, it can be easily applied to inverse problems. Herein, the resonance in a one-dimensional acoustic tube was analyzed. The effectiveness of the proposed method was validated through the forward and inverse analyses of the wave equation with energy-loss terms. In the forward analysis, the applicability of PINN to the resonance problem was evaluated by comparison with the finite-difference method. The inverse analysis, which included the identification of the energy loss term in the wave equation and design optimization of the acoustic tube, was performed with good accuracy.【10】 Blind estimation of audio effects using an auto-encoder approach and differentiable signal processing标题：基于自动编码法和可微信号处理的音频效果盲估计链接：https://arxiv.org/abs/2310.11781作者：Côme Peladeau,Geoffroy Peeters机构：LTCI, T´el´ecom-Paris, Institut Polytechnique de Paris, France摘要：音频效果盲估计（BE-AFX）旨在仅基于经处理的音频样本来估计应用于原始未经处理的音频样本的音频效果（AFX）。为了训练这样的系统，传统方法优化地面实况和估计的AFX参数之间的损失。这涉及到了解用于流程的AFX的确切实现。在这项工作中，我们提出了一种替代解决方案，消除了了解这种实现的要求。相反，我们引入了一种自动编码器方法，它优化了音频质量指标。我们探索，建议和比较常用的掌握AFX的各种实现，使用差分信号处理或神经近似。我们的研究结果表明，与传统的基于参数的方法相比，我们的自动编码器方法对AFX链产生的音频质量产生了更好的估计，即使后者提供了更准确的参数估计。摘要：Blind Estimation of Audio Effects (BE-AFX) aims at estimating the Audio Effects (AFXs) applied to an original, unprocessed audio sample solely based on the processed audio sample. To train such a system traditional approaches optimize a loss between ground truth and estimated AFX parameters. This involves knowing the exact implementation of the AFXs used for the process. In this work, we propose an alternative solution that eliminates the requirement for knowing this implementation. Instead, we introduce an auto-encoder approach, which optimizes an audio quality metric. We explore, suggest, and compare various implementations of commonly used mastering AFXs, using differential signal processing or neural approximations. Our findings demonstrate that our auto-encoder approach yields superior estimates of the audio quality produced by a chain of AFXs, compared to the traditional parameter-based approach, even if the latter provides a more accurate parameter estimation.【11】 Unintended Memorization in Large ASR Models, and How to Mitigate It标题：大型ASR模型中的意外记忆以及如何减少这种记忆链接：https://arxiv.org/abs/2310.11739作者：Lun Wang,Om Thakkar,Rajiv Mathews机构：Google摘要：众所周知，神经网络可能会无意中记住它们的训练样本，从而引起隐私问题。然而，审计记忆在大型非自回归自动语音识别（ASR）模型一直具有挑战性，由于现有的方法，如硬度校准的高计算成本。在这项工作中，我们设计了一个简单的审计方法来衡量大型ASR模型的记忆，而无需额外的计算开销。具体地说，我们加快了随机生成的话语，以创建语音和文本信息之间的映射，这是很难从典型的训练示例中学习的。因此，仅针对加速训练示例的准确预测可以作为记忆的明确证据，并且相应的准确度可以用于测量记忆。使用所提出的方法，我们展示了最先进的ASR模型中的记忆。为了减轻记忆，我们在训练过程中尝试了梯度裁剪，以限制任何单个示例对最终模型的影响。我们的经验表明，裁剪每个例子的梯度可以减轻记忆的加速训练的例子，在训练集上重复多达16次。此外，我们证明了在大规模分布式训练中，削减每个计算核心上的平均梯度可以保持中性模型质量和计算成本，同时提供强大的隐私保护。摘要：It is well-known that neural networks can unintentionally memorize their training examples, causing privacy concerns. However, auditing memorization in large non-auto-regressive automatic speech recognition (ASR) models has been challenging due to the high compute cost of existing methods such as hardness calibration. In this work, we design a simple auditing method to measure memorization in large ASR models without the extra compute overhead. Concretely, we speed up randomly-generated utterances to create a mapping between vocal and text information that is difficult to learn from typical training examples. Hence, accurate predictions only for sped-up training examples can serve as clear evidence for memorization, and the corresponding accuracy can be used to measure memorization. Using the proposed method, we showcase memorization in the state-of-the-art ASR models. To mitigate memorization, we tried gradient clipping during training to bound the influence of any individual example on the final model. We empirically show that clipping each example's gradient can mitigate memorization for sped-up training examples with up to 16 repetitions in the training set. Furthermore, we show that in large-scale distributed training, clipping the average gradient on each compute core maintains neutral model quality and compute cost while providing strong privacy protection.【12】 EchoScan: Scanning Complex Indoor Geometries via Acoustic Echoes标题：回声扫描：通过声学回波扫描复杂的室内几何图形链接：https://arxiv.org/abs/2310.11728作者：Inmo Yeon,Iljoo Jeong,Seungchul Lee,Jung-Woo Choi机构：the Institute of Civil-Military Tech-nology Cooperation funded by the Defense Acquisition Program Ad-ministration and Ministry of Trade备注：9 pages, 8 figures摘要：准确估计室内空间几何形状对于构建精确的数字孪生模型至关重要，其广泛的工业应用包括在陌生环境中的导航和有效的疏散规划，特别是在低光照条件下。这项研究介绍了EchoScan，这是一种深度神经网络模型，它利用声学回声来进行房间几何推断。传统的基于声音的技术依赖于估计几何形状相关的房间参数，例如墙壁位置和房间大小，从而限制了可推断的房间几何形状的多样性。实际上，EchoScan通过直接推断房间平面图和高度来克服这一限制，从而使其能够处理具有任意形状的房间，包括弯曲的墙壁。EchoScan的关键创新在于它能够使用多聚合模块分析房间脉冲响应（RIR）中低阶和高阶反射之间的复杂关系。高阶反射的分析还使其能够在回声无法从音频设备的位置观察到时推断复杂的房间形状。在此，EchoScan使用从复杂环境（包括曼哈顿和亚特兰大布局）合成的RIR进行训练和评估，采用与商用现成设备兼容的实用音频设备配置。与基于视觉的方法相比，EchoScan在各种形状的房间中表现出出色的几何估计性能。摘要：Accurate estimation of indoor space geometries is vital for constructing precise digital twins, whose broad industrial applications include navigation in unfamiliar environments and efficient evacuation planning, particularly in low-light conditions. This study introduces EchoScan, a deep neural network model that utilizes acoustic echoes to perform room geometry inference. Conventional sound-based techniques rely on estimating geometry-related room parameters such as wall position and room size, thereby limiting the diversity of inferable room geometries. Contrarily, EchoScan overcomes this limitation by directly inferring room floorplans and heights, thereby enabling it to handle rooms with arbitrary shapes, including curved walls. The key innovation of EchoScan is its ability to analyze the complex relationship between low- and high-order reflections in room impulse responses (RIRs) using a multi-aggregation module. The analysis of high-order reflections also enables it to infer complex room shapes when echoes are unobservable from the position of an audio device. Herein, EchoScan was trained and evaluated using RIRs synthesized from complex environments, including the Manhattan and Atlanta layouts, employing a practical audio device configuration compatible with commercial, off-the-shelf devices. Compared with vision-based methods, EchoScan demonstrated outstanding geometry estimation performance in rooms with various shapes.【13】 Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation标题：分离看不见的声音走向通用视听场景感知声音分离链接：https://arxiv.org/abs/2310.11713作者：Yiyang Su,Ali Vosoughi,Shijian Deng,Yapeng Tian,Chenliang Xu机构：Michigan State University, University of Rochester, University of Texas at Dallas备注：Accepted at ICCV 2023 - AV4D, 4 figures, 3 tables摘要：视听声音分离字段假设视频中的可见源，但这排除了摄像机视野之外的不可见声音。目前的方法难以处理缺乏可见线索的这种声音。本文介绍了一种新的“视听场景感知分离”（AVSA-Sep）框架。它包括一个用于可见和不可见声音的语义解析器和一个用于场景分离的分离器。AVSA-Sep成功地分离了两种声音类型，联合训练和跨模态对齐提高了效果。摘要：The audio-visual sound separation field assumes visible sources in videos, but this excludes invisible sounds beyond the camera's view. Current methods struggle with such sounds lacking visible cues. This paper introduces a novel "Audio-Visual Scene-Aware Separation" (AVSA-Sep) framework. It includes a semantic parser for visible and invisible sounds and a separator for scene-informed separation. AVSA-Sep successfully separates both sound types, with joint training and cross-modal alignment enhancing effectiveness.【14】 MUST&P-SRL: Multi-lingual and Unified Syllabification in Text and Phonetic Domains for Speech Representation Learning标题：MASH&P-SRL：用于语音表征学习的语音域多语种统一音节化链接：https://arxiv.org/abs/2310.11541作者：Noé Tits机构：Flowchase, ISIA Lab, Numediart Institute, University of Mons备注：Accepted for publication at EMNLP 2023摘要：在本文中，我们提出了一种语言特征提取的方法，特别侧重于自动音节化的话，在多种语言中，与强制对齐工具，蒙特利尔强制对齐（MFA）的设计是兼容的。在文本和语音领域，我们的方法侧重于从文本，重音标记，和一个统一的自动音节（在文本和语音领域）的语音音标的提取。该系统使用开源组件和资源构建。通过消融研究，我们证明了我们的方法在自动音节化的话，从几种语言（英语，法语和西班牙语）的有效性。此外，我们将该技术应用于CMU ARCTIC数据集的transmittance，生成在线提供的有价值的注释，这些注释非常适合语音表示学习，语音单元发现和几个语音相关领域的语音因素的解开。摘要：In this paper, we present a methodology for linguistic feature extraction, focusing particularly on automatically syllabifying words in multiple languages, with a design to be compatible with a forced-alignment tool, the Montreal Forced Aligner (MFA). In both the textual and phonetic domains, our method focuses on the extraction of phonetic transcriptions from text, stress marks, and a unified automatic syllabification (in text and phonetic domains). The system was built with open-source components and resources. Through an ablation study, we demonstrate the efficacy of our approach in automatically syllabifying words from several languages (English, French and Spanish). Additionally, we apply the technique to the transcriptions of the CMU ARCTIC dataset, generating valuable annotations available online\footnote{\url{https://github.com/noetits/MUST_P-SRL}} that are ideal for speech representation learning, speech unit discovery, and disentanglement of speech factors in several speech-related fields.【15】 Multi-stage Large Language Model Correction for Speech Recognition标题：用于语音识别的多阶段大语言模型校正链接：https://arxiv.org/abs/2310.11532作者：Jie Pu,Thai-Son Nguyen,Sebastian Stüker机构：Zoom Video Communications, Karlsruhe, Germany备注：Submitted to ICASSP 2024摘要：在本文中，我们研究了使用大型语言模型（LLM），以提高竞争的语音识别系统的性能。与传统语言模型专注于单个数据域不同，LLM的兴起为我们带来了突破最先进ASR性能极限的机会，同时实现更高的鲁棒性并有效地跨多个域进行泛化。基于此，我们提出了一种新的多阶段方法，将传统的语言模型重新评分和LLM提示相结合。具体而言，所提出的方法具有两个阶段：第一阶段使用语言模型对ASR假设的N最佳列表重新评分并运行置信度检查;第二阶段使用对LLM的提示对来自第一阶段的不太可信的结果执行ASR错误校正。我们的实验结果表明，所提出的方法的有效性，表现出10% ~ 20%的相对改善，在竞争力的ASR系统-跨多个测试域。摘要：In this paper, we investigate the usage of large language models (LLMs) to improve the performance of competitive speech recognition systems. Different from traditional language models that focus on one single data domain, the rise of LLMs brings us the opportunity to push the limit of state-of-the-art ASR performance, and at the same time to achieve higher robustness and generalize effectively across multiple domains. Motivated by this, we propose a novel multi-stage approach to combine traditional language model re-scoring and LLM prompting. Specifically, the proposed method has two stages: the first stage uses a language model to re-score an N-best list of ASR hypotheses and run a confidence check; The second stage uses prompts to a LLM to perform ASR error correction on less confident results from the first stage. Our experimental results demonstrate the effectiveness of the proposed method by showing a 10% ~ 20% relative improvement in WER over a competitive ASR system -- across multiple test domains.
每日语音速递
