今日论文合集：cs.SD语音9篇，eess.AS音频处理9篇。
cs.SD语音
【1】 Indonesian Automatic Speech Recognition with XLSR-53标题：基于XLSR-53的印尼语自动语音识别链接：https://arxiv.org/abs/2308.11589
作者：Panji Arisaputra,Amalia Zahra机构：Computer Science Department, BINUS Graduate Program, Master of Computer Science, Bina Nusantara University, Jakarta, Indonesia摘要：本研究的重点是发展印尼语自动语音识别（ASR）使用XLSR-53预训练模型，XLSR代表跨语言语音表征。使用此XLSR-53预训练模型是为了显著减少非英语语言的训练数据量，以实现有竞争力的错误字率（WER）。本研究使用的数据总量为24小时18分1秒：（1）TITML-IDN 14小时31分钟;（2）神奇数据3小时33分;（3）普通语音6小时14分1秒。WER为20%，本研究中建立的模型可以与使用Common Voice数据集分裂测试的类似模型竞争。使用语言模型可以将WER降低约8%，导致WER从20%降低到12%。因此，本研究的结果成功地完善了以前的研究，有助于创建一个更好的印度尼西亚ASR与更少的数据量。摘要：This study focuses on the development of Indonesian Automatic Speech Recognition (ASR) using the XLSR-53 pre-trained model, the XLSR stands for cross-lingual speech representations. The use of this XLSR-53 pre-trained model is to significantly reduce the amount of training data in non-English languages required to achieve a competitive Word Error Rate (WER). The total amount of data used in this study is 24 hours, 18 minutes, and 1 second: (1) TITML-IDN 14 hours and 31 minutes; (2) Magic Data 3 hours and 33 minutes; and (3) Common Voice 6 hours, 14 minutes, and 1 second. With a WER of 20%, the model built in this study can compete with similar models using the Common Voice dataset split test. WER can be decreased by around 8% using a language model, resulted in WER from 20% to 12%. Thus, the results of this study have succeeded in perfecting previous research in contributing to the creation of a better Indonesian ASR with a smaller amount of data.
【2】 Furnishing Sound Event Detection with Language Model Abilities标题：赋予声音事件检测以语言模型能力链接：https://arxiv.org/abs/2308.11530
作者：Hualei Wang,Jianguo Mao,Zhifang Guo,Jiarui Wan,Hong Liu,Xiangdong Wang机构：Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China, Beijing Jiaotong University, Beijing, China备注：8 pages,2 figures,published to AAAI摘要：近年来，语言模型的能力在视觉跨模态领域引起了越来越多的关注。在本文中，我们进一步探讨的生成能力的LM声音事件检测（SED），超越视觉域。具体来说，我们提出了一个优雅的方法，对齐音频功能和文本功能，以完成声音事件分类和时间定位。该框架包括一个声学编码器，一个对比模块，对齐相应的表示的文本和音频，和一个解耦的语言解码器，生成时间和事件序列的音频特性。与传统的作品相比，需要复杂的处理和几乎没有利用有限的音频功能，我们的模型是更简洁和全面的，因为语言模型直接利用其语义能力来生成序列。我们调查不同的解耦模块，以证明时间戳捕获和事件分类的有效性。评估结果表明，该方法实现了准确的声音事件检测序列。摘要：Recently, the ability of language models (LMs) has attracted increasing attention in visual cross-modality. In this paper, we further explore the generation capacity of LMs for sound event detection (SED), beyond the visual domain. Specifically, we propose an elegant method that aligns audio features and text features to accomplish sound event classification and temporal location. The framework consists of an acoustic encoder, a contrastive module that align the corresponding representations of the text and audio, and a decoupled language decoder that generates temporal and event sequences from the audio characteristic. Compared with conventional works that require complicated processing and barely utilize limited audio features, our model is more concise and comprehensive since language model directly leverage its semantic capabilities to generate the sequences. We investigate different decoupling modules to demonstrate the effectiveness for timestamps capture and event classification. Evaluation results show that the proposed method achieves accurate sequences of sound event detection.
【3】 Deep learning-based denoising streamed from mobile phones improves  speech-in-noise understanding for hearing aid users标题：基于深度学习的手机流去噪改善助听器用户对噪声中语音的理解链接：https://arxiv.org/abs/2308.11456
作者：Peter Udo Diehl,Hannes Zilly,Felix Sattler,Yosef Singer,Kevin Kepp,Mark Berry,Henning Hasemann,Marlene Zippel,Müge Kaya,Paul Meyer-Rachner,Annett Pudszuhn,Veit M. Hofmann,Matthias Vormann,Elias Sprengel机构：Audatic, Friedrichstr. , Berlin, Germany, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-, Universität zu Berlin, and Berlin Institute of Health, Department of Otorhinolaryngology, Head and摘要：近5亿人的听力损失通常用助听器治疗。然而，目前的助听器在现实世界的嘈杂环境中通常不能很好地工作。我们提出了一个基于深度学习的去噪系统，该系统在iPhone 7和三星Galaxy S10上实时运行（25ms算法延迟）。降噪后的音频被流式传输到助听器，导致大约75ms的总延迟。在对患有中度至重度听力损失的助听器用户进行的测试中，我们的去噪系统在三项测试中改善了音频：1）收听主观音频评级，2）收听客观语音可懂度，以及3）在嘈杂环境中的实况对话，用于主观评级。与作为基线的安装助听器相比，听力测试和现场对话的主观评级增加了40%以上。语音接收阈值，测量噪声中的语音理解，提高了1.6 dB SRT。我们是第一个在移动终端上实现的去噪系统，仅使用单个通道作为音频输入直接传输到用户的助听器，同时提高用户对所有测试方面的满意度，包括语音清晰度。这包括去噪和流式传输的信号相对于助听器的总体偏好，从而接受较高的等待时间以显著改善语音理解。摘要：The hearing loss of almost half a billion people is commonly treated with hearing aids. However, current hearing aids often do not work well in real-world noisy environments. We present a deep learning based denoising system that runs in real time on iPhone 7 and Samsung Galaxy S10 (25ms algorithmic latency). The denoised audio is streamed to the hearing aid, resulting in a total delay of around 75ms. In tests with hearing aid users having moderate to severe hearing loss, our denoising system improves audio across three tests: 1) listening for subjective audio ratings, 2) listening for objective speech intelligibility, and 3) live conversations in a noisy environment for subjective ratings. Subjective ratings increase by more than 40%, for both the listening test and the live conversation compared to a fitted hearing aid as a baseline. Speech reception thresholds, measuring speech understanding in noise, improve by 1.6 dB SRT. Ours is the first denoising system that is implemented on a mobile device, streamed directly to users' hearing aids using only a single channel as audio input while improving user satisfaction on all tested aspects, including speech intelligibility. This includes overall preference of the denoised and streamed signal over the hearing aid, thereby accepting the higher latency for the significant improvement in speech understanding.
【4】 Convoifilter: A case study of doing cocktail party speech recognition标题：Convoifilter：鸡尾酒会语音识别的实例研究链接：https://arxiv.org/abs/2308.11380
作者：Thai-Binh Nguyen,Alexander Waibel机构：Karlsruhe Institute of Technology, Carnegie Mellon University备注：6 pages摘要：本文提出了一种端到端的模型，旨在提高自动语音识别（ASR）的特定扬声器在拥挤，嘈杂的环境。该模型利用单通道语音增强模块，该模块将说话者的声音与背景噪声隔离，以及ASR模块。通过这种方法，该模型能够将ASR的单词错误率（WER）从80%降低到26.4%。通常，由于数据要求的变化，这两个分量被独立地调整。然而，语音增强可能产生降低ASR效率的异常。通过实施一个联合微调策略，该模型可以减少WER从26.4%，在单独调整到14.5%，在联合调整。摘要：This paper presents an end-to-end model designed to improve automatic speech recognition (ASR) for a particular speaker in a crowded, noisy environment. The model utilizes a single-channel speech enhancement module that isolates the speaker's voice from background noise, along with an ASR module. Through this approach, the model is able to decrease the word error rate (WER) of ASR from 80% to 26.4%. Typically, these two components are adjusted independently due to variations in data requirements. However, speech enhancement can create anomalies that decrease ASR efficiency. By implementing a joint fine-tuning strategy, the model can reduce the WER from 26.4% in separate tuning to 14.5% in joint tuning.
【5】 Music Understanding LLaMA: Advancing Text-to-Music Generation with  Question Answering and Captioning标题：音乐理解骆驼：通过问题回答和字幕推动文本到音乐的生成链接：https://arxiv.org/abs/2308.11276
作者：Shansong Liu,Atin Sakkeer Hussain,Chenshuo Sun,Ying Shan机构：ARC Lab, Tencent PCG,  National University of Singapore摘要：文本到音乐生成（T2 M-Gen）面临着一个主要的障碍，由于缺乏大规模的公共可用的音乐数据集与自然语言字幕。为了解决这个问题，我们提出了音乐理解LLaMA（MU-LLaMA），能够回答音乐相关的问题，并生成音乐文件的字幕。我们的模型利用音频表示从一个预先训练的MERT模型提取音乐功能。然而，获得用于训练MU-LLaMA模型的合适数据集仍然具有挑战性，因为现有的可公开访问的音频问答数据集缺乏开放式音乐问答所需的深度。为了填补这一空白，我们提出了一种从现有的音频字幕数据集生成问答对的方法，并介绍了MusicQA数据集，旨在回答开放式音乐相关的问题。实验表明，所提出的MU-LLaMA模型，在我们设计的MusicQA数据集上训练，在音乐问答和音乐字幕生成方面实现了出色的性能，在这两个领域都优于当前的最先进的（SOTA）模型，并在T2 M-Gen研究领域提供了有前途的进步。摘要：Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.
【6】 An Effective Transformer-based Contextual Model and Temporal Gate  Pooling for Speaker Identification标题：一种有效的基于变换的语境模型和时间门池的说话人识别链接：https://arxiv.org/abs/2308.11241
作者：Harunori Kawano,Sota Shimizu机构：Shibaura Institute of Technology, Japan摘要：Wav 2 vec 2已经成功地将Transformer架构和自监督学习应用于语音识别。最近，这些已经开始不仅用于语音识别，而且用于整个语音处理。本文介绍了一种有效的端到端说话人识别模型，应用基于Transformer的上下文模型。我们探讨了参数和性能之间的关系，以辨别一个有效的模型的结构。此外，我们提出了一种池化方法，时间门池，具有强大的学习能力的说话人识别。我们应用Conformer作为编码器和BEST-RQ进行预训练，并利用VoxCeleb 1的说话人识别进行评估。所提出的方法已经实现了85.9%的准确度与28.5M的参数，表现出相当的精度wav 2 vec 2与317.7M的参数。代码可在https://github.com/HarunoriKawano/speaker-identification-with-tgp上获得。摘要：Wav2vec2 has achieved success in applying Transformer architecture and self-supervised learning to speech recognition. Recently, these have come to be used not only for speech recognition but also for the entire speech processing. This paper introduces an effective end-to-end speaker identification model applied Transformer-based contextual model. We explored the relationship between the parameters and the performance in order to discern the structure of an effective model. Furthermore, we propose a pooling method, Temporal Gate Pooling, with powerful learning ability for speaker identification. We applied Conformer as encoder and BEST-RQ for pre-training and conducted an evaluation utilizing the speaker identification of VoxCeleb1. The proposed method has achieved an accuracy of 85.9% with 28.5M parameters, demonstrating comparable precision to wav2vec2 with 317.7M parameters. Code is available at https://github.com/HarunoriKawano/speaker-identification-with-tgp.
【7】 PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice  Conversion标题：PMVC：基于数据增强的表现性语音转换韵律建模链接：https://arxiv.org/abs/2308.11084
作者：Yimin Deng,Huaizhen Tang,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., University of Science and Technology, of China, Huya Inc (Shenzhen) Co., Ltd.备注：Accepted by the 31st ACM International Conference on Multimedia (MM2023)摘要：语音转换作为应用于语音的风格转换任务，是指将一个人的语音转换成听起来像另一个人的新语音。到目前为止，已经有大量的研究致力于更好地执行VC任务。然而，一个好的语音转换模型不仅要与目标说话人的音色信息相匹配，还要与韵律、步速、停顿等表达信息相匹配在这种情况下，韵律建模是至关重要的，以实现表达的声音转换，听起来自然和令人信服的。不幸的是，韵律建模是重要的，但具有挑战性的，特别是没有文本转录。在本文中，我们首先提出了一种新的语音转换框架，名为“PMVC”，有效地分离和模型的内容，音色和韵律信息的语音没有文本转录。特别地，我们介绍了一种新的语音增强算法的鲁棒韵律提取。在此基础上，采用屏蔽和预测机制对韵律信息和内容信息进行分离。在AIShell-3语料库上的实验结果支持我们对转换语音的自然度和相似度的改进。摘要：Voice conversion as the style transfer task applied to speech, refers to converting one person's speech into a new speech that sounds like another person's. Up to now, there has been a lot of research devoted to better implementation of VC tasks. However, a good voice conversion model should not only match the timbre information of the target speaker, but also expressive information such as prosody, pace, pause, etc. In this context, prosody modeling is crucial for achieving expressive voice conversion that sounds natural and convincing. Unfortunately, prosody modeling is important but challenging, especially without text transcriptions. In this paper, we firstly propose a novel voice conversion framework named 'PMVC', which effectively separates and models the content, timbre, and prosodic information from the speech without text transcriptions. Specially, we introduce a new speech augmentation algorithm for robust prosody extraction. And building upon this, mask and predict mechanism is applied in the disentanglement of prosody and content information. The experimental results on the AIShell-3 corpus supports our improvement of naturalness and similarity of converted speech.
【8】 Evaluation of the Speech Resynthesis Capabilities of the VoicePrivacy  Challenge Baseline B1标题：语音隐私挑战基线B1的语音再合成能力评估链接：https://arxiv.org/abs/2308.11337
作者：Ünal Ege Gaznepoglu,Nils Peters机构：International Audio Laboratories, Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Germany备注：Accepted to 3rd Symposium on Security and Privacy of Speech Communication摘要：说话者匿名化系统继续改进其混淆语音信号中的原始说话者特性的能力，但通常创建处理伪像和不自然的发声语音作为折衷。这些系统中的许多源于VoicePrivacy Challenge（VPC）Baseline B1，使用神经声码器从F0、x向量和瓶颈特征为基础的语音表示合成语音。受此启发，我们调查了上述基线的再现能力，以评估共享方法在合成类人语音方面的成功程度。我们使用四个客观指标来测量语音质量，波形相似性，F0相似性。我们的研究结果表明，语音表示和声码器引入伪影，造成不自然的感知。对18名受试者的MUSHRA样听力测试证实了我们的发现，激发了对VPC基线B1的分析和合成组件的进一步研究。摘要：Speaker anonymization systems continue to improve their ability to obfuscate the original speaker characteristics in a speech signal, but often create processing artifacts and unnatural sounding voices as a tradeoff. Many of those systems stem from the VoicePrivacy Challenge (VPC) Baseline B1, using a neural vocoder to synthesize speech from an F0, x-vectors and bottleneck features-based speech representation. Inspired by this, we investigate the reproduction capabilities of the aforementioned baseline, to assess how successful the shared methodology is in synthesizing human-like speech. We use four objective metrics to measure speech quality, waveform similarity, and F0 similarity. Our findings indicate that both the speech representation and the vocoder introduces artifacts, causing an unnatural perception. A MUSHRA-like listening test on 18 subjects corroborate our findings, motivating further research on the analysis and synthesis components of the VPC Baseline B1.
【9】 Ultra Dual-Path Compression For Joint Echo Cancellation And Noise  Suppression标题：用于回波抵消和噪声抑制的超双路径压缩链接：https://arxiv.org/abs/2308.11053
作者：Hangting Chen,Jianwei Yu,Yi Luo,Rongzhi Gu,Weihua Li,Zhuocheng Lu,Chao Weng机构：Tencent AI Lab, Audio and Speech Signal Processing Oteam备注：Accepted by Interspeech 2023摘要：回波消除和噪声降低对于全双工通信是必不可少的，然而大多数现有的神经网络具有高计算成本并且在调整模型复杂性方面是不灵活的。在本文中，我们引入时频双路径压缩，以实现一个宽范围的压缩比的计算成本。具体地，对于频率压缩，可训练滤波器用于代替手动设计的滤波器以减小尺寸。对于时间压缩，仅使用帧跳过预测导致大的性能下降，这可以通过具有全序列建模的后处理网络来缓解。我们已经发现，在固定的压缩比下，双路径压缩结合时间和频率的方法将提供进一步的性能改善，覆盖压缩比从4倍到32倍，模型大小的变化很小。此外，与快速FullSubNet和DeepFilterNet相比，所提出的模型表现出竞争力。演示页面可在hangtingchen.github.io/ultra_dual_path_compression.github.io/找到。摘要：Echo cancellation and noise reduction are essential for full-duplex communication, yet most existing neural networks have high computational costs and are inflexible in tuning model complexity. In this paper, we introduce time-frequency dual-path compression to achieve a wide range of compression ratios on computational cost. Specifically, for frequency compression, trainable filters are used to replace manually designed filters for dimension reduction. For time compression, only using frame skipped prediction causes large performance degradation, which can be alleviated by a post-processing network with full sequence modeling. We have found that under fixed compression ratios, dual-path compression combining both the time and frequency methods will give further performance improvement, covering compression ratios from 4x to 32x with little model size change. Moreover, the proposed models show competitive performance compared with fast FullSubNet and DeepFilterNet. A demo page can be found at hangtingchen.github.io/ultra_dual_path_compression.github.io/.
eess.AS音频处理
【1】 Evaluation of the Speech Resynthesis Capabilities of the VoicePrivacy  Challenge Baseline B1标题：语音隐私挑战基线B1的语音再合成能力评估链接：https://arxiv.org/abs/2308.11337
作者：Ünal Ege Gaznepoglu,Nils Peters机构：International Audio Laboratories, Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Germany备注：Accepted to 3rd Symposium on Security and Privacy of Speech Communication摘要：说话者匿名化系统继续改进其混淆语音信号中的原始说话者特性的能力，但通常创建处理伪像和不自然的发声语音作为折衷。这些系统中的许多源于VoicePrivacy Challenge（VPC）Baseline B1，使用神经声码器从F0、x向量和瓶颈特征为基础的语音表示合成语音。受此启发，我们调查了上述基线的再现能力，以评估共享方法在合成类人语音方面的成功程度。我们使用四个客观指标来测量语音质量，波形相似性，F0相似性。我们的研究结果表明，语音表示和声码器引入伪影，造成不自然的感知。对18名受试者的MUSHRA样听力测试证实了我们的发现，激发了对VPC基线B1的分析和合成组件的进一步研究。摘要：Speaker anonymization systems continue to improve their ability to obfuscate the original speaker characteristics in a speech signal, but often create processing artifacts and unnatural sounding voices as a tradeoff. Many of those systems stem from the VoicePrivacy Challenge (VPC) Baseline B1, using a neural vocoder to synthesize speech from an F0, x-vectors and bottleneck features-based speech representation. Inspired by this, we investigate the reproduction capabilities of the aforementioned baseline, to assess how successful the shared methodology is in synthesizing human-like speech. We use four objective metrics to measure speech quality, waveform similarity, and F0 similarity. Our findings indicate that both the speech representation and the vocoder introduces artifacts, causing an unnatural perception. A MUSHRA-like listening test on 18 subjects corroborate our findings, motivating further research on the analysis and synthesis components of the VPC Baseline B1.
【2】 Ultra Dual-Path Compression For Joint Echo Cancellation And Noise  Suppression标题：用于回波抵消和噪声抑制的超双路径压缩链接：https://arxiv.org/abs/2308.11053
作者：Hangting Chen,Jianwei Yu,Yi Luo,Rongzhi Gu,Weihua Li,Zhuocheng Lu,Chao Weng机构：Tencent AI Lab, Audio and Speech Signal Processing Oteam备注：Accepted by Interspeech 2023摘要：回波消除和噪声降低对于全双工通信是必不可少的，然而大多数现有的神经网络具有高计算成本并且在调整模型复杂性方面是不灵活的。在本文中，我们引入时频双路径压缩，以实现一个宽范围的压缩比的计算成本。具体地，对于频率压缩，可训练滤波器用于代替手动设计的滤波器以减小尺寸。对于时间压缩，仅使用帧跳过预测导致大的性能下降，这可以通过具有全序列建模的后处理网络来缓解。我们已经发现，在固定的压缩比下，双路径压缩结合时间和频率的方法将提供进一步的性能改善，覆盖压缩比从4倍到32倍，模型大小的变化很小。此外，与快速FullSubNet和DeepFilterNet相比，所提出的模型表现出竞争力。演示页面可在hangtingchen.github.io/ultra_dual_path_compression.github.io/找到。摘要：Echo cancellation and noise reduction are essential for full-duplex communication, yet most existing neural networks have high computational costs and are inflexible in tuning model complexity. In this paper, we introduce time-frequency dual-path compression to achieve a wide range of compression ratios on computational cost. Specifically, for frequency compression, trainable filters are used to replace manually designed filters for dimension reduction. For time compression, only using frame skipped prediction causes large performance degradation, which can be alleviated by a post-processing network with full sequence modeling. We have found that under fixed compression ratios, dual-path compression combining both the time and frequency methods will give further performance improvement, covering compression ratios from 4x to 32x with little model size change. Moreover, the proposed models show competitive performance compared with fast FullSubNet and DeepFilterNet. A demo page can be found at hangtingchen.github.io/ultra_dual_path_compression.github.io/.
【3】 Indonesian Automatic Speech Recognition with XLSR-53标题：基于XLSR-53的印尼语自动语音识别链接：https://arxiv.org/abs/2308.11589
作者：Panji Arisaputra,Amalia Zahra机构：Computer Science Department, BINUS Graduate Program, Master of Computer Science, Bina Nusantara University, Jakarta, Indonesia摘要：本研究的重点是发展印尼语自动语音识别（ASR）使用XLSR-53预训练模型，XLSR代表跨语言语音表征。使用此XLSR-53预训练模型是为了显著减少非英语语言的训练数据量，以实现有竞争力的错误字率（WER）。本研究使用的数据总量为24小时18分1秒：（1）TITML-IDN 14小时31分钟;（2）神奇数据3小时33分;（3）普通语音6小时14分1秒。WER为20%，本研究中建立的模型可以与使用Common Voice数据集分裂测试的类似模型竞争。使用语言模型可以将WER降低约8%，导致WER从20%降低到12%。因此，本研究的结果成功地完善了以前的研究，有助于创建一个更好的印度尼西亚ASR与更少的数据量。摘要：This study focuses on the development of Indonesian Automatic Speech Recognition (ASR) using the XLSR-53 pre-trained model, the XLSR stands for cross-lingual speech representations. The use of this XLSR-53 pre-trained model is to significantly reduce the amount of training data in non-English languages required to achieve a competitive Word Error Rate (WER). The total amount of data used in this study is 24 hours, 18 minutes, and 1 second: (1) TITML-IDN 14 hours and 31 minutes; (2) Magic Data 3 hours and 33 minutes; and (3) Common Voice 6 hours, 14 minutes, and 1 second. With a WER of 20%, the model built in this study can compete with similar models using the Common Voice dataset split test. WER can be decreased by around 8% using a language model, resulted in WER from 20% to 12%. Thus, the results of this study have succeeded in perfecting previous research in contributing to the creation of a better Indonesian ASR with a smaller amount of data.
【4】 Furnishing Sound Event Detection with Language Model Abilities标题：赋予声音事件检测以语言模型能力链接：https://arxiv.org/abs/2308.11530
作者：Hualei Wang,Jianguo Mao,Zhifang Guo,Jiarui Wan,Hong Liu,Xiangdong Wang机构：Beijing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China, Beijing Jiaotong University, Beijing, China备注：8 pages,2 figures,published to AAAI摘要：近年来，语言模型的能力在视觉跨模态领域引起了越来越多的关注。在本文中，我们进一步探讨的生成能力的LM声音事件检测（SED），超越视觉域。具体来说，我们提出了一个优雅的方法，对齐音频功能和文本功能，以完成声音事件分类和时间定位。该框架包括一个声学编码器，一个对比模块，对齐相应的表示的文本和音频，和一个解耦的语言解码器，生成时间和事件序列的音频特性。与传统的作品相比，需要复杂的处理和几乎没有利用有限的音频功能，我们的模型是更简洁和全面的，因为语言模型直接利用其语义能力来生成序列。我们调查不同的解耦模块，以证明时间戳捕获和事件分类的有效性。评估结果表明，该方法实现了准确的声音事件检测序列。摘要：Recently, the ability of language models (LMs) has attracted increasing attention in visual cross-modality. In this paper, we further explore the generation capacity of LMs for sound event detection (SED), beyond the visual domain. Specifically, we propose an elegant method that aligns audio features and text features to accomplish sound event classification and temporal location. The framework consists of an acoustic encoder, a contrastive module that align the corresponding representations of the text and audio, and a decoupled language decoder that generates temporal and event sequences from the audio characteristic. Compared with conventional works that require complicated processing and barely utilize limited audio features, our model is more concise and comprehensive since language model directly leverage its semantic capabilities to generate the sequences. We investigate different decoupling modules to demonstrate the effectiveness for timestamps capture and event classification. Evaluation results show that the proposed method achieves accurate sequences of sound event detection.
【5】 Deep learning-based denoising streamed from mobile phones improves  speech-in-noise understanding for hearing aid users标题：基于深度学习的手机流去噪改善助听器用户对噪声中语音的理解链接：https://arxiv.org/abs/2308.11456
作者：Peter Udo Diehl,Hannes Zilly,Felix Sattler,Yosef Singer,Kevin Kepp,Mark Berry,Henning Hasemann,Marlene Zippel,Müge Kaya,Paul Meyer-Rachner,Annett Pudszuhn,Veit M. Hofmann,Matthias Vormann,Elias Sprengel机构：Audatic, Friedrichstr. , Berlin, Germany, Charité – Universitätsmedizin Berlin, corporate member of Freie Universität Berlin, Humboldt-, Universität zu Berlin, and Berlin Institute of Health, Department of Otorhinolaryngology, Head and摘要：近5亿人的听力损失通常用助听器治疗。然而，目前的助听器在现实世界的嘈杂环境中通常不能很好地工作。我们提出了一个基于深度学习的去噪系统，该系统在iPhone 7和三星Galaxy S10上实时运行（25ms算法延迟）。降噪后的音频被流式传输到助听器，导致大约75ms的总延迟。在对患有中度至重度听力损失的助听器用户进行的测试中，我们的去噪系统在三项测试中改善了音频：1）收听主观音频评级，2）收听客观语音可懂度，以及3）在嘈杂环境中的实况对话，用于主观评级。与作为基线的安装助听器相比，听力测试和现场对话的主观评级增加了40%以上。语音接收阈值，测量噪声中的语音理解，提高了1.6 dB SRT。我们是第一个在移动终端上实现的去噪系统，仅使用单个通道作为音频输入直接传输到用户的助听器，同时提高用户对所有测试方面的满意度，包括语音清晰度。这包括去噪和流式传输的信号相对于助听器的总体偏好，从而接受较高的等待时间以显著改善语音理解。摘要：The hearing loss of almost half a billion people is commonly treated with hearing aids. However, current hearing aids often do not work well in real-world noisy environments. We present a deep learning based denoising system that runs in real time on iPhone 7 and Samsung Galaxy S10 (25ms algorithmic latency). The denoised audio is streamed to the hearing aid, resulting in a total delay of around 75ms. In tests with hearing aid users having moderate to severe hearing loss, our denoising system improves audio across three tests: 1) listening for subjective audio ratings, 2) listening for objective speech intelligibility, and 3) live conversations in a noisy environment for subjective ratings. Subjective ratings increase by more than 40%, for both the listening test and the live conversation compared to a fitted hearing aid as a baseline. Speech reception thresholds, measuring speech understanding in noise, improve by 1.6 dB SRT. Ours is the first denoising system that is implemented on a mobile device, streamed directly to users' hearing aids using only a single channel as audio input while improving user satisfaction on all tested aspects, including speech intelligibility. This includes overall preference of the denoised and streamed signal over the hearing aid, thereby accepting the higher latency for the significant improvement in speech understanding.
【6】 Convoifilter: A case study of doing cocktail party speech recognition标题：Convoifilter：鸡尾酒会语音识别的实例研究链接：https://arxiv.org/abs/2308.11380
作者：Thai-Binh Nguyen,Alexander Waibel机构：Karlsruhe Institute of Technology, Carnegie Mellon University备注：6 pages摘要：本文提出了一种端到端的模型，旨在提高自动语音识别（ASR）的特定扬声器在拥挤，嘈杂的环境。该模型利用单通道语音增强模块，该模块将说话者的声音与背景噪声隔离，以及ASR模块。通过这种方法，该模型能够将ASR的单词错误率（WER）从80%降低到26.4%。通常，由于数据要求的变化，这两个分量被独立地调整。然而，语音增强可能产生降低ASR效率的异常。通过实施一个联合微调策略，该模型可以减少WER从26.4%，在单独调整到14.5%，在联合调整。摘要：This paper presents an end-to-end model designed to improve automatic speech recognition (ASR) for a particular speaker in a crowded, noisy environment. The model utilizes a single-channel speech enhancement module that isolates the speaker's voice from background noise, along with an ASR module. Through this approach, the model is able to decrease the word error rate (WER) of ASR from 80% to 26.4%. Typically, these two components are adjusted independently due to variations in data requirements. However, speech enhancement can create anomalies that decrease ASR efficiency. By implementing a joint fine-tuning strategy, the model can reduce the WER from 26.4% in separate tuning to 14.5% in joint tuning.
【7】 Music Understanding LLaMA: Advancing Text-to-Music Generation with  Question Answering and Captioning标题：音乐理解骆驼：通过问题回答和字幕推动文本到音乐的生成链接：https://arxiv.org/abs/2308.11276
作者：Shansong Liu,Atin Sakkeer Hussain,Chenshuo Sun,Ying Shan机构：ARC Lab, Tencent PCG,  National University of Singapore摘要：文本到音乐生成（T2 M-Gen）面临着一个主要的障碍，由于缺乏大规模的公共可用的音乐数据集与自然语言字幕。为了解决这个问题，我们提出了音乐理解LLaMA（MU-LLaMA），能够回答音乐相关的问题，并生成音乐文件的字幕。我们的模型利用音频表示从一个预先训练的MERT模型提取音乐功能。然而，获得用于训练MU-LLaMA模型的合适数据集仍然具有挑战性，因为现有的可公开访问的音频问答数据集缺乏开放式音乐问答所需的深度。为了填补这一空白，我们提出了一种从现有的音频字幕数据集生成问答对的方法，并介绍了MusicQA数据集，旨在回答开放式音乐相关的问题。实验表明，所提出的MU-LLaMA模型，在我们设计的MusicQA数据集上训练，在音乐问答和音乐字幕生成方面实现了出色的性能，在这两个领域都优于当前的最先进的（SOTA）模型，并在T2 M-Gen研究领域提供了有前途的进步。摘要：Text-to-music generation (T2M-Gen) faces a major obstacle due to the scarcity of large-scale publicly available music datasets with natural language captions. To address this, we propose the Music Understanding LLaMA (MU-LLaMA), capable of answering music-related questions and generating captions for music files. Our model utilizes audio representations from a pretrained MERT model to extract music features. However, obtaining a suitable dataset for training the MU-LLaMA model remains challenging, as existing publicly accessible audio question answering datasets lack the necessary depth for open-ended music question answering. To fill this gap, we present a methodology for generating question-answer pairs from existing audio captioning datasets and introduce the MusicQA Dataset designed for answering open-ended music-related questions. The experiments demonstrate that the proposed MU-LLaMA model, trained on our designed MusicQA dataset, achieves outstanding performance in both music question answering and music caption generation across various metrics, outperforming current state-of-the-art (SOTA) models in both fields and offering a promising advancement in the T2M-Gen research field.
【8】 An Effective Transformer-based Contextual Model and Temporal Gate  Pooling for Speaker Identification标题：一种有效的基于变换的语境模型和时间门池的说话人识别链接：https://arxiv.org/abs/2308.11241
作者：Harunori Kawano,Sota Shimizu机构：Shibaura Institute of Technology, Japan摘要：Wav 2 vec 2已经成功地将Transformer架构和自监督学习应用于语音识别。最近，这些已经开始不仅用于语音识别，而且用于整个语音处理。本文介绍了一种有效的端到端说话人识别模型，应用基于Transformer的上下文模型。我们探讨了参数和性能之间的关系，以辨别一个有效的模型的结构。此外，我们提出了一种池化方法，时间门池，具有强大的学习能力的说话人识别。我们应用Conformer作为编码器和BEST-RQ进行预训练，并利用VoxCeleb 1的说话人识别进行评估。所提出的方法已经实现了85.9%的准确度与28.5M的参数，表现出相当的精度wav 2 vec 2与317.7M的参数。代码可在https://github.com/HarunoriKawano/speaker-identification-with-tgp上获得。摘要：Wav2vec2 has achieved success in applying Transformer architecture and self-supervised learning to speech recognition. Recently, these have come to be used not only for speech recognition but also for the entire speech processing. This paper introduces an effective end-to-end speaker identification model applied Transformer-based contextual model. We explored the relationship between the parameters and the performance in order to discern the structure of an effective model. Furthermore, we propose a pooling method, Temporal Gate Pooling, with powerful learning ability for speaker identification. We applied Conformer as encoder and BEST-RQ for pre-training and conducted an evaluation utilizing the speaker identification of VoxCeleb1. The proposed method has achieved an accuracy of 85.9% with 28.5M parameters, demonstrating comparable precision to wav2vec2 with 317.7M parameters. Code is available at https://github.com/HarunoriKawano/speaker-identification-with-tgp.
【9】 PMVC: Data Augmentation-Based Prosody Modeling for Expressive Voice  Conversion标题：PMVC：基于数据增强的表现性语音转换韵律建模链接：https://arxiv.org/abs/2308.11084
作者：Yimin Deng,Huaizhen Tang,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., University of Science and Technology, of China, Huya Inc (Shenzhen) Co., Ltd.备注：Accepted by the 31st ACM International Conference on Multimedia (MM2023)摘要：语音转换作为应用于语音的风格转换任务，是指将一个人的语音转换成听起来像另一个人的新语音。到目前为止，已经有大量的研究致力于更好地执行VC任务。然而，一个好的语音转换模型不仅要与目标说话人的音色信息相匹配，还要与韵律、步速、停顿等表达信息相匹配在这种情况下，韵律建模是至关重要的，以实现表达的声音转换，听起来自然和令人信服的。不幸的是，韵律建模是重要的，但具有挑战性的，特别是没有文本转录。在本文中，我们首先提出了一种新的语音转换框架，名为“PMVC”，有效地分离和模型的内容，音色和韵律信息的语音没有文本转录。特别地，我们介绍了一种新的语音增强算法的鲁棒韵律提取。在此基础上，采用屏蔽和预测机制对韵律信息和内容信息进行分离。在AIShell-3语料库上的实验结果支持我们对转换语音的自然度和相似度的改进。摘要：Voice conversion as the style transfer task applied to speech, refers to converting one person's speech into a new speech that sounds like another person's. Up to now, there has been a lot of research devoted to better implementation of VC tasks. However, a good voice conversion model should not only match the timbre information of the target speaker, but also expressive information such as prosody, pace, pause, etc. In this context, prosody modeling is crucial for achieving expressive voice conversion that sounds natural and convincing. Unfortunately, prosody modeling is important but challenging, especially without text transcriptions. In this paper, we firstly propose a novel voice conversion framework named 'PMVC', which effectively separates and models the content, timbre, and prosodic information from the speech without text transcriptions. Specially, we introduce a new speech augmentation algorithm for robust prosody extraction. And building upon this, mask and predict mechanism is applied in the disentanglement of prosody and content information. The experimental results on the AIShell-3 corpus supports our improvement of naturalness and similarity of converted speech.
机器翻译，仅供参考