今日论文合集：cs.SD语音13篇，eess.AS音频处理13篇。
cs.SD语音
【1】 Device-Robust Acoustic Scene Classification via Impulse Response  Augmentation标题：基于脉冲响应增强的设备稳健声场景分类链接：https://arxiv.org/abs/2305.07499
作者：Tobias Morocutti,Florian Schmid,Khaled Koutini,Gerhard Widmer机构：Institute of Computational Perception,LIT Artificial Intelligence Lab, Johannes Kepler University, Linz, Austria备注：Submitted to the 31st European Signal Processing Conference, EUSIPCO 2023. Source Code available at: this https URL摘要：推广到各种记录设备的能力是音频分类模型的关键性能因素。不同类型的麦克风的特性由于其变化的频率响应而在数字化音频信号中引入分布偏移。如果在训练过程中不考虑这种域偏移，则当将其应用于由看不见的设备记录的信号时，模型的性能可能会严重降低。特别地，在用少量不同麦克风记录的音频信号上训练模型可能使得难以推广到看不见的设备。为了解决这个问题，我们将训练集中的音频信号与预先记录的设备脉冲响应（DIR）进行卷积，以人为地增加记录设备的多样性。我们系统地研究了DIR增强对使用CNN和音频频谱图Transformers的声学场景分类任务的影响。结果表明，DIR增强在隔离执行类似的国家的最先进的方法Freq-MixStyle。然而，我们还表明，DIR增强和Freq-MixStyle是互补的，在训练期间看不见的设备记录的信号上实现了新的最先进的性能。摘要：The ability to generalize to a wide range of recording devices is a crucial performance factor for audio classification models. The characteristics of different types of microphones introduce distributional shifts in the digitized audio signals due to their varying frequency responses. If this domain shift is not taken into account during training, the model's performance could degrade severely when it is applied to signals recorded by unseen devices. In particular, training a model on audio signals recorded with a small number of different microphones can make generalization to unseen devices difficult. To tackle this problem, we convolve audio signals in the training set with pre-recorded device impulse responses (DIRs) to artificially increase the diversity of recording devices. We systematically study the effect of DIR augmentation on the task of Acoustic Scene Classification using CNNs and Audio Spectrogram Transformers. The results show that DIR augmentation in isolation performs similarly to the state-of-the-art method Freq-MixStyle. However, we also show that DIR augmentation and Freq-MixStyle are complementary, achieving a new state-of-the-art performance on signals recorded by devices unseen during training.
【2】 Benchmarks and leaderboards for sound demixing tasks标题：合理分离任务的基准和排行榜链接：https://arxiv.org/abs/2305.07489
作者：Roman Solovyev,Alexander Stempkovskiy,Tatiana Habruseva机构：Institute for Design Problems in Microelectronics of Russian Academy of Sciences, Moscow, Russian Federation, Independent researcher, Cork, Ireland摘要：音乐去混是将来自给定单个音频信号的不同轨道分离成分量（诸如来自伴奏的其余部分的鼓、低音和人声）的任务。源的分离对于一系列领域是有用的，包括娱乐和助听器。在本文中，我们介绍了两个新的基准的声源分离任务，并比较流行的模型的声音混音，以及他们的合奏，在这些基准。对于模型的评估，我们在https：//mvsep.com/quality_checker/上提供了排行榜，对一系列模型进行了比较。新的基准数据集可供下载。我们还开发了一种新的方法，音频分离的基础上，不同的模型，最适合特定的干的合奏。在2023年音乐分层挑战赛的背景下对所提出的解决方案进行了评估，并在挑战赛的不同赛道中取得了最佳成绩。代码和方法在GitHub上开源。摘要：Music demixing is the task of separating different tracks from the given single audio signal into components, such as drums, bass, and vocals from the rest of the accompaniment. Separation of sources is useful for a range of areas, including entertainment and hearing aids. In this paper, we introduce two new benchmarks for the sound source separation tasks and compare popular models for sound demixing, as well as their ensembles, on these benchmarks. For the models' assessments, we provide the leaderboard at https://mvsep.com/quality_checker/, giving a comparison for a range of models. The new benchmark datasets are available for download. We also develop a novel approach for audio separation, based on the ensembling of different models that are suited best for the particular stem. The proposed solution was evaluated in the context of the Music Demixing Challenge 2023 and achieved top results in different tracks of the challenge. The code and the approach are open-sourced on GitHub.
【3】 Improving Cascaded Unsupervised Speech Translation with Denoising  Back-translation标题：用去噪反译法改进级联无监督语音翻译链接：https://arxiv.org/abs/2305.07455
作者：Yu-Kuan Fu,Liang-Hsuan Tseng,Jiatong Shi,Chen-An Li,Tsu-Yuan Hsu,Shinji Watanabe,Hung-yi Lee机构：College of Electrical Engineering and Computer Science, National Taiwan University, Language Technologies Institute, Carnegie Mellon University摘要：大多数语音翻译模型严重依赖于并行数据，这是很难收集，特别是对于低资源的语言。为了解决这个问题，我们建议建立一个级联的语音翻译系统，而不利用任何类型的配对数据。我们使用完全未配对的数据来训练我们的无监督系统，并在CoVoST 2和CVSS上评估我们的结果。结果表明，我们的工作是与其他一些早期的监督方法在一些语言对。虽然级联系统总是遭受严重的错误传播问题，我们提出了去噪反向翻译（DBT），一种新的方法来建立强大的无监督神经机器翻译（UNMT）。DBT在所有三个平移方向上成功地将BLEU分数提高了0.7- 0.9。此外，我们简化了我们的级联系统的流水线，以减少推理延迟，并进行了全面的分析，我们的工作的每一个部分。我们还展示了我们的无监督的语音翻译结果上建立的网站。摘要：Most of the speech translation models heavily rely on parallel data, which is hard to collect especially for low-resource languages. To tackle this issue, we propose to build a cascaded speech translation system without leveraging any kind of paired data. We use fully unpaired data to train our unsupervised systems and evaluate our results on CoVoST 2 and CVSS. The results show that our work is comparable with some other early supervised methods in some language pairs. While cascaded systems always suffer from severe error propagation problems, we proposed denoising back-translation (DBT), a novel approach to building robust unsupervised neural machine translation (UNMT). DBT successfully increases the BLEU score by 0.7--0.9 in all three translation directions. Moreover, we simplified the pipeline of our cascaded system to reduce inference latency and conducted a comprehensive analysis of every part of our work. We also demonstrate our unsupervised speech translation results on the established website.
【4】 Universal Source Separation with Weakly Labelled Data标题：弱标记数据的泛源分离链接：https://arxiv.org/abs/2305.07447
作者：Qiuqiang Kong,Ke Chen,Haohe Liu,Xingjian Du,Taylor Berg-Kirkpatrick,Shlomo Dubnov,Mark D. Plumbley摘要：通用源分离是计算听觉场景分析的一项基础性研究任务，其目的是将单声道录音分离成单独的源音轨。有三个潜在的挑战等待解决方案的音频源分离任务。首先，先前的音频源分离系统主要集中于分离一个或有限数量的特定源。目前缺乏建立一个统一的系统，可以通过一个单一的模型分离任意源的研究。其次，大多数以前的系统需要干净的源数据来训练分离器，而干净的源数据是稀缺的。第三，缺乏可以自动检测和分离分层级别中的活动声音类别的USS系统。为了使用大规模弱标记/未标记音频数据进行音频源分离，我们提出了一种通用音频源分离框架，该框架包含：1）在弱标记数据上训练的音频标记模型作为查询网;以及2）条件源分离模型，其将查询网络输出作为分离任意声源的条件。我们调查了各种查询网络，源分离模型和训练策略，并提出了一个分层USS策略，自动检测和分离的声音类AudioSet本体。通过单独利用弱标记的AudioSet，我们的USS系统成功地分离了各种各样的声音类，包括声音事件分离，音乐源分离和语音增强。USS系统在AudioSet的527个声音等级上实现了5.57dB的平均信号失真比改善（SDRi）; DCASE 2018任务2数据集上为10.57 dB; MUSDB 18数据集上为8.12 dB; Slakh 2100数据集上的SDRi为7.28dB;以及在语音库需求数据集上的9.00dB的SSNR。我们在www.example.com上发布源代码https://github.com/bytedance/uss摘要：Universal source separation (USS) is a fundamental research task for computational auditory scene analysis, which aims to separate mono recordings into individual source tracks. There are three potential challenges awaiting the solution to the audio source separation task. First, previous audio source separation systems mainly focus on separating one or a limited number of specific sources. There is a lack of research on building a unified system that can separate arbitrary sources via a single model. Second, most previous systems require clean source data to train a separator, while clean source data are scarce. Third, there is a lack of USS system that can automatically detect and separate active sound classes in a hierarchical level. To use large-scale weakly labeled/unlabeled audio data for audio source separation, we propose a universal audio source separation framework containing: 1) an audio tagging model trained on weakly labeled data as a query net; and 2) a conditional source separation model that takes query net outputs as conditions to separate arbitrary sound sources. We investigate various query nets, source separation models, and training strategies and propose a hierarchical USS strategy to automatically detect and separate sound classes from the AudioSet ontology. By solely leveraging the weakly labelled AudioSet, our USS system is successful in separating a wide variety of sound classes, including sound event separation, music source separation, and speech enhancement. The USS system achieves an average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over 527 sound classes of AudioSet; 10.57 dB on the DCASE 2018 Task 2 dataset; 8.12 dB on the MUSDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of 9.00 dB on the voicebank-demand dataset. We release the source code at https://github.com/bytedance/uss
【5】 Investigating the Sensitivity of Automatic Speech Recognition Systems to  Phonetic Variation in L2 Englishes标题：自动语音识别系统对二语英语语音变化的敏感性研究链接：https://arxiv.org/abs/2305.07389
作者：Emma O'Neill,Julie Carson-Berndsen机构：In an investigation into the use of ASR for the purposes of linguis-This research was conducted with the financial support of the School of Computer Science at the ADAPTSFI Research Centre at University College Dublin摘要：自动语音识别（ASR）系统在与其被训练的语音相似的语音上表现出最佳性能。因此，代表性不足的品种，包括地区方言，少数民族语言和低资源语言，比那些被视为“声望”，“主流”或“标准”的品种看到更高的单词错误率（WERs）。这可能会成为将ASR技术纳入大规模语言学研究的注释过程的障碍，因为手动纠正错误的自动转录本可能与手动转录一样耗时耗力。因此，从语音技术的角度来看，在提高ASR准确性方面，以及从注释的角度来看，对ASR系统的行为的更深入理解是有益的，其中，知道ASR系统可能产生的错误可以有助于这种手动校正。这项工作展示了一种方法，探测ASR系统，以发现它如何处理语音变化在一些第二语言英语。具体地，在系统的训练数据中罕见或不存在的特定语音实现如何能够导致音素水平的误识别并有助于更高的WER。它表明，ASR的行为是系统的和一致的扬声器具有类似的口语品种（在这种情况下，相同的L1）和音素替换错误通常是在协议与人类注释。通过识别有问题的制作，可以通过获取用于训练和微调的这种实现来解决特定的弱点，从而使系统对发音变化更鲁棒。摘要：Automatic Speech Recognition (ASR) systems exhibit the best performance on speech that is similar to that on which it was trained. As such, underrepresented varieties including regional dialects, minority-speakers, and low-resource languages, see much higher word error rates (WERs) than those varieties seen as 'prestigious', 'mainstream', or 'standard'. This can act as a barrier to incorporating ASR technology into the annotation process for large-scale linguistic research since the manual correction of the erroneous automated transcripts can be just as time and resource consuming as manual transcriptions. A deeper understanding of the behaviour of an ASR system is thus beneficial from a speech technology standpoint, in terms of improving ASR accuracy, and from an annotation standpoint, where knowing the likely errors made by an ASR system can aid in this manual correction. This work demonstrates a method of probing an ASR system to discover how it handles phonetic variation across a number of L2 Englishes. Specifically, how particular phonetic realisations which were rare or absent in the system's training data can lead to phoneme level misrecognitions and contribute to higher WERs. It is demonstrated that the behaviour of the ASR is systematic and consistent across speakers with similar spoken varieties (in this case the same L1) and phoneme substitution errors are typically in agreement with human annotators. By identifying problematic productions specific weaknesses can be addressed by sourcing such realisations for training and fine-tuning thus making the system more robust to pronunciation variation.
【6】 Music Rearrangement Using Hierarchical Segmentation标题：基于层次分割的音乐重排技术链接：https://arxiv.org/abs/2305.07347
作者：Christos Plachouras,Marius Miron机构：Music Technology Group, Universitat Pompeu Fabra, Spain摘要：音乐重排涉及重新编排、删除和重复音乐作品的部分，目的是产生具有不同持续时间的独立版本。这是一项创造性和耗时的任务，通常由专业音乐工程师执行。在本文中，我们提出了一种自动重新安排音乐录音，考虑到录音的层次结构的方法。先前的方法仅关注于识别音频中的可能导致平滑过渡的切割点。相反，我们利用深度音频表示来分层地分割作品，并根据片段的边界和音乐功能定义切点搜索。我们根据它们的相似性和它们所属的段对合适的入口和出口点对进行评分，并定义最佳路径搜索。实验结果表明，所选择的切割点是最常见的听众难以察觉的，并导致更一致的音乐发展与更少的分心重复。摘要：Music rearrangement involves reshuffling, deleting, and repeating sections of a music piece with the goal of producing a standalone version that has a different duration. It is a creative and time-consuming task commonly performed by an expert music engineer. In this paper, we propose a method for automatically rearranging music recordings that takes into account the hierarchical structure of the recording. Previous approaches focus solely on identifying cut-points in the audio that could result in smooth transitions. We instead utilize deep audio representations to hierarchically segment the piece and define a cut-point search subject to the boundaries and musical functions of the segments. We score suitable entry- and exit-point pairs based on their similarity and the segments they belong to, and define an optimal path search. Experimental results demonstrate the selected cut-points are most commonly imperceptible by listeners and result in more consistent musical development with less distracting repetitions.
【7】 Better speech synthesis through scaling标题：通过缩放改进语音合成链接：https://arxiv.org/abs/2305.07243
作者：James Betker摘要：近年来，自回归Transformers和DDPM的应用使图像生成领域发生了革命性的变化。这些方法将图像生成过程建模为逐步概率过程，并利用大量的计算和数据来学习图像分布。这种提高性能的方法不需要局限于图像。本文介绍了一种方法，适用于图像生成领域的语音合成的进展。其结果是TorToise -一个富有表现力的多语音文本到语音系统。  所有模型代码和训练权重都已在www.example.com上开源https://github.com/neonbjb/tortoise-tts。摘要：In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise -- an expressive, multi-voice text-to-speech system.  All model code and trained weights have been open-sourced at https://github.com/neonbjb/tortoise-tts.
【8】 Hear to Segment: Unmixing the Audio to Guide the Semantic Segmentation标题：Hear to Segment：分解音频以指导语义切分链接：https://arxiv.org/abs/2305.07223
作者：Yuhang Ling,Yuxi Li,Zhenye Gan,Jiangning Zhang,Mingmin Chi,Yabiao Wang机构：Fudan University, Shanghai, China, Youtu lab, Tencent备注：11 pages, 7 figures摘要：在本文中，我们专注于最近提出的一个新的任务，称为视听分割（AVS），其中需要建立音频流和图像像素之间的细粒度的对应关系。然而，学习这种对应关系面临两个关键挑战：（1）音频信号固有地表现出高度的信息密度，因为由多个对象产生的声音在同一音频流内纠缠;（2）同一类别的对象的音频信号的频率趋于相似，这妨碍了目标对象的区分，从而导致不明确的分割结果。为此，我们提出了一个音频解混和语义分割网络（AUSS），它鼓励解混复杂的音频信号和区分类似的声音。从技术上讲，我们的AUSS将音频信号解混成一组音频查询，并通过掩蔽注意机制将它们与视觉特征进行交互。为了鼓励这些音频查询捕获嵌入在音频中的独特特征，还引入了两个自监督损失作为类和掩码级别的额外监督。AVSBench基准测试的大量实验结果表明，我们的AUSS在单源和多源子集中都具有新的最先进水平，证明了我们的AUSS在弥合音频和视觉模式之间的差距方面的有效性。摘要：In this paper, we focus on a recently proposed novel task called Audio-Visual Segmentation (AVS), where the fine-grained correspondence between audio stream and image pixels is required to be established. However, learning such correspondence faces two key challenges: (1) audio signals inherently exhibit a high degree of information density, as sounds produced by multiple objects are entangled within the same audio stream; (2) the frequency of audio signals from objects with the same category tends to be similar, which hampers the distinction of target object and consequently leads to ambiguous segmentation results. Toward this end, we propose an Audio Unmixing and Semantic Segmentation Network (AUSS), which encourages unmixing complicated audio signals and distinguishing similar sounds. Technically, our AUSS unmixs the audio signals into a set of audio queries, and interacts them with visual features by masked attention mechanisms. To encourage these audio queries to capture distinctive features embedded within the audio, two self-supervised losses are also introduced as additional supervision at both class and mask levels. Extensive experimental results on the AVSBench benchmark show that our AUSS sets a new state-of-the-art in both single-source and multi-source subsets, demonstrating the effectiveness of our AUSS in bridging the gap between audio and vision modalities.
【9】 Versatile Audio-Visual Learning for Handling Single and Multi Modalities  in Emotion Regression and Classification Tasks标题：情绪回归和分类任务中处理单通道和多通道的多功能视听学习链接：https://arxiv.org/abs/2305.07216
作者：Lucas Goncalves,Seong-Gyun Leem,Wei-Cheng Lin,Berrak Sisman,Carlos Busso机构：Multimodal Signal Processing (MSP) Lab, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas备注：14 pages, 2 Figures, 2 tables摘要：目前大多数视听情感识别模型缺乏在实际应用中部署所需的灵活性。我们设想一个多模态系统，即使只有一种模态是可用的，并且可以互换地实现用于预测情感属性或识别分类情感。在多模态情感识别系统中实现这样的灵活性是困难的，这是由于在准确地解释和集成不同的数据源的固有挑战。在允许回归和分类任务之间的直接切换的同时稳健地处理缺失或部分信息也是一个挑战。本研究提出了一个通用的视听学习（VAVL）的框架，用于处理情绪回归和情绪分类任务的单峰和多模态系统。我们实现了一种视听框架，即使在视听配对数据对于训练集的一部分不可用时（即，仅存在音频或仅存在视频）。我们实现了这种有效的表示学习与视听共享层，剩余连接共享层，和一个单峰重建任务。我们的实验结果表明，我们的架构显着优于强大的基线CREMA-D和MSP-IMPROV语料库。值得注意的是，VAVL在MSP-IMPROV语料库上的情感属性预测任务中获得了新的最先进的性能。代码可在：https://github.com/ilucasgoncalves/VAVL摘要：Most current audio-visual emotion recognition models lack the flexibility needed for deployment in practical applications. We envision a multimodal system that works even when only one modality is available and can be implemented interchangeably for either predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is difficult due to the inherent challenges in accurately interpreting and integrating varied data sources. It is also a challenge to robustly handle missing or partial information while allowing direct switch between regression and classification tasks. This study proposes a \emph{versatile audio-visual learning} (VAVL) framework for handling unimodal and multimodal systems for emotion regression and emotion classification tasks. We implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set (i.e., audio only or only video is present). We achieve this effective representation learning with audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Our experimental results reveal that our architecture significantly outperforms strong baselines on both the CREMA-D and MSP-IMPROV corpora. Notably, VAVL attains a new state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus. Code available at: https://github.com/ilucasgoncalves/VAVL
【10】 Tackling Interpretability in Audio Classification Networks with  Non-negative Matrix Factorization标题：利用非负矩阵分解解决音频分类网络中的可译性问题链接：https://arxiv.org/abs/2305.07132
作者：Jayneel Parekh,Sanjeel Parekh,Pavlo Mozharovskyi,Gaël Richard,Florence d'Alché-Buc备注：Under submission at IEEE/ACM TASLP. arXiv admin note: text overlap with arXiv:2202.11479摘要：本文解决了两个主要的问题设置的音频处理网络的可解释性，事后和设计的解释。对于事后的解释，我们的目标是解释的决定，网络的高级音频对象，也是listenable的终端用户。这是扩展到目前一个固有的解释模型，具有高性能。为此，我们提出了一种新的解释器设计，采用非负矩阵分解（NMF）。特别地，解释器被训练为从目标网络的隐藏层生成正则化的中间嵌入，作为预先学习的NMF字典的时间激活来学习。我们的方法使我们能够生成直观的基于音频的解释，显式地增强输入信号的部分最相关的网络的决定。我们证明了我们的方法的适用性在各种分类任务，包括多标签数据的真实世界的音频和音乐。摘要：This paper tackles two major problem settings for interpretability of audio processing networks, post-hoc and by-design interpretation. For post-hoc interpretation, we aim to interpret decisions of a network in terms of high-level audio objects that are also listenable for the end-user. This is extended to present an inherently interpretable model with high performance. To this end, we propose a novel interpreter design that incorporates non-negative matrix factorization (NMF). In particular, an interpreter is trained to generate a regularized intermediate embedding from hidden layers of a target network, learnt as time-activations of a pre-learnt NMF dictionary. Our methodology allows us to generate intuitive audio-based interpretations that explicitly enhance parts of the input signal most relevant for a network's decision. We demonstrate our method's applicability on a variety of classification tasks, including multi-label data for real-world audio and music.
【11】 QVoice: Arabic Speech Pronunciation Learning Application标题：QVoice：阿拉伯语语音发音学习应用程序链接：https://arxiv.org/abs/2305.07445
作者：Yassine El Kheir,Fouad Khnaisser,Shammur Absar Chowdhury,Hamdy Mubarak,Shazia Afzal,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar备注：None摘要：本文介绍了一种新颖的阿拉伯语发音学习应用程序QVoice，配备了端到端的发音错误检测和反馈生成器模块。该应用程序旨在支持非母语阿拉伯语使用者提高他们的发音技能，同时也帮助母语使用者减轻区域方言对其现代标准阿拉伯语（MSA）发音的任何潜在影响。QVoice采用各种学习线索来帮助学习者理解含义，与他们现有的英语知识建立联系，并提供详细的发音纠正反馈，以及展示单词用法的上下文示例。QVoice中的学习线索包含了广泛的有意义的信息，例如短语/单词及其翻译的可视化，以及音标和音译。QVoice提供字符级别的发音反馈，并在单词级别评估表现。摘要：This paper introduces a novel Arabic pronunciation learning application QVoice, powered with end-to-end mispronunciation detection and feedback generator module. The application is designed to support non-native Arabic speakers in enhancing their pronunciation skills, while also helping native speakers mitigate any potential influence from regional dialects on their Modern Standard Arabic (MSA) pronunciation. QVoice employs various learning cues to aid learners in comprehending meaning, drawing connections with their existing knowledge of English language, and offers detailed feedback for pronunciation correction, along with contextual examples showcasing word usage. The learning cues featured in QVoice encompass a wide range of meaningful information, such as visualizations of phrases/words and their translations, as well as phonetic transcriptions and transliterations. QVoice provides pronunciation feedback at the character level and assesses performance at the word level.
【12】 Multi-level Temporal-channel Speaker Retrieval for Robust Zero-shot  Voice Conversion标题：用于稳健零发声转换的多级时间通道说话人检索链接：https://arxiv.org/abs/2305.07204
作者：Zhichao Wang,Liumeng Xue,Qiuqiang Kong,Lei Xie,Yuanzhe Chen,Qiao Tian,Yuping Wang机构：and Lei Xie are with the School of ComputerScience,  Northwestern Polytechnical University备注：Submitted to TASLP摘要：Zero-shot语音转换（VC）仅使用说话者的一个话语将源语音转换成任何期望说话者的语音，而不需要额外的模型更新。典型的方法使用来自预先训练的说话者验证（SV）模型的说话者表示或在VC训练期间学习说话者表示以实现zero-shot VC。然而，现有的说话人建模方法忽略了说话人信息丰富度在语音的时间和频率通道维度上的变化。这种不充分的说话者建模阻碍了VC模型准确地表示不在训练数据集中的未看到的说话者的能力。在这项研究中，我们提出了一个强大的zero-shot VC模型与多级时间通道检索，称为MTCR-VC。具体来说，灵活地适应动态变化的扬声器特性的时间和通道轴的语音，我们提出了一种新的细粒度扬声器建模方法，称为时间通道检索（TCR），找出何时何地扬声器信息出现在语音中。它检索可变长度的说话人表示从时间和通道维度的指导下，一个预先训练的SV模型。此外，受人类语音产生的分层过程的启发，MTCR扬声器模块堆叠多个TCR块以从多粒度级别提取扬声器表示。此外，为了实现更好的语音解纠缠和重构，我们引入了一个基于循环的训练策略来模拟zero-shot推理。我们在内容、风格和演讲者三个方面采取永久性约束来推动这一过程。实验结果表明，MTCR-VC方法优于以往的zero-shot VC方法在建模说话人音色，同时保持良好的语音自然度。摘要：Zero-shot voice conversion (VC) converts source speech into the voice of any desired speaker using only one utterance of the speaker without requiring additional model updates. Typical methods use a speaker representation from a pre-trained speaker verification (SV) model or learn speaker representation during VC training to achieve zero-shot VC. However, existing speaker modeling methods overlook the variation of speaker information richness in temporal and frequency channel dimensions of speech. This insufficient speaker modeling hampers the ability of the VC model to accurately represent unseen speakers who are not in the training dataset. In this study, we present a robust zero-shot VC model with multi-level temporal-channel retrieval, referred to as MTCR-VC. Specifically, to flexibly adapt to the dynamic-variant speaker characteristic in the temporal and channel axis of the speech, we propose a novel fine-grained speaker modeling method, called temporal-channel retrieval (TCR), to find out when and where speaker information appears in speech. It retrieves variable-length speaker representation from both temporal and channel dimensions under the guidance of a pre-trained SV model. Besides, inspired by the hierarchical process of human speech production, the MTCR speaker module stacks several TCR blocks to extract speaker representations from multi-granularity levels. Furthermore, to achieve better speech disentanglement and reconstruction, we introduce a cycle-based training strategy to simulate zero-shot inference recurrently. We adopt perpetual constraints on three aspects, including content, style, and speaker, to drive this process. Experiments demonstrate that MTCR-VC is superior to the previous zero-shot VC methods in modeling speaker timbre while maintaining good speech naturalness.
【13】 Quran Recitation Recognition using End-to-End Deep Learning标题：基于端到端深度学习的古兰经背诵识别链接：https://arxiv.org/abs/2305.07034
作者：Ahmad Al Harere,Khloud Al Jallad机构：Department of Information and Communication Engineering, Arab International University, Daraa, Syria.摘要：《古兰经》是伊斯兰教的圣典，它的背诵是伊斯兰教的重要方面。自动识别古兰经的背诵是一项具有挑战性的任务，因为它的独特规则不适用于正常的演讲。在这一领域已经做了大量的研究，但以前的作品检测背诵错误作为一个分类任务或使用传统的自动语音识别（ASR）。在本文中，我们提出了一种新的端到端深度学习模型，用于识别古兰经的背诵。该模型是一个CNN双向GRU编码器，使用CTC作为目标函数，和基于字符的解码器，这是一个波束搜索解码器。此外，所有以前的作品都是在小型私人数据集上完成的，包括简短的诗句和古兰经的几个章节。由于使用了私有数据集，因此未进行比较。为了克服这个问题，我们使用了一个最近发布的公共数据集（Ar-DAD），包含大约37个章节，由30个背诵者背诵，具有不同的背诵速度和不同类型的发音规则。该模型的性能进行了评估，使用最常见的评价指标，在语音识别，字错误率（WER），字符错误率（CER）。结果为8.34%的WER和2.42%的CER。我们希望这项研究将成为与这个公共新数据集（Ar-DAD）的未来研究进行比较的基线。摘要：The Quran is the holy scripture of Islam, and its recitation is an important aspect of the religion. Recognizing the recitation of the Holy Quran automatically is a challenging task due to its unique rules that are not applied in normal speaking speeches. A lot of research has been done in this domain, but previous works have detected recitation errors as a classification task or used traditional automatic speech recognition (ASR). In this paper, we proposed a novel end-to-end deep learning model for recognizing the recitation of the Holy Quran. The proposed model is a CNN-Bidirectional GRU encoder that uses CTC as an objective function, and a character-based decoder which is a beam search decoder. Moreover, all previous works were done on small private datasets consisting of short verses and a few chapters of the Holy Quran. As a result of using private datasets, no comparisons were done. To overcome this issue, we used a public dataset that has recently been published (Ar-DAD) and contains about 37 chapters that were recited by 30 reciters, with different recitation speeds and different types of pronunciation rules. The proposed model performance was evaluated using the most common evaluation metrics in speech recognition, word error rate (WER), and character error rate (CER). The results were 8.34% WER and 2.42% CER. We hope this research will be a baseline for comparisons with future research on this public new dataset (Ar-DAD).
eess.AS音频处理
【1】 QVoice: Arabic Speech Pronunciation Learning Application标题：QVoice：阿拉伯语语音发音学习应用程序链接：https://arxiv.org/abs/2305.07445
*与cs.SD语音【11】为同一篇
作者：Yassine El Kheir,Fouad Khnaisser,Shammur Absar Chowdhury,Hamdy Mubarak,Shazia Afzal,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar备注：None摘要：本文介绍了一种新颖的阿拉伯语发音学习应用程序QVoice，配备了端到端的发音错误检测和反馈生成器模块。该应用程序旨在支持非母语阿拉伯语使用者提高他们的发音技能，同时也帮助母语使用者减轻区域方言对其现代标准阿拉伯语（MSA）发音的任何潜在影响。QVoice采用各种学习线索来帮助学习者理解含义，与他们现有的英语知识建立联系，并提供详细的发音纠正反馈，以及展示单词用法的上下文示例。QVoice中的学习线索包含了广泛的有意义的信息，例如短语/单词及其翻译的可视化，以及音标和音译。QVoice提供字符级别的发音反馈，并在单词级别评估表现。摘要：This paper introduces a novel Arabic pronunciation learning application QVoice, powered with end-to-end mispronunciation detection and feedback generator module. The application is designed to support non-native Arabic speakers in enhancing their pronunciation skills, while also helping native speakers mitigate any potential influence from regional dialects on their Modern Standard Arabic (MSA) pronunciation. QVoice employs various learning cues to aid learners in comprehending meaning, drawing connections with their existing knowledge of English language, and offers detailed feedback for pronunciation correction, along with contextual examples showcasing word usage. The learning cues featured in QVoice encompass a wide range of meaningful information, such as visualizations of phrases/words and their translations, as well as phonetic transcriptions and transliterations. QVoice provides pronunciation feedback at the character level and assesses performance at the word level.
【2】 Multi-level Temporal-channel Speaker Retrieval for Robust Zero-shot  Voice Conversion标题：用于稳健零发声转换的多级时间通道说话人检索链接：https://arxiv.org/abs/2305.07204
*与cs.SD语音【12】为同一篇
作者：Zhichao Wang,Liumeng Xue,Qiuqiang Kong,Lei Xie,Yuanzhe Chen,Qiao Tian,Yuping Wang机构：and Lei Xie are with the School of ComputerScience,  Northwestern Polytechnical University备注：Submitted to TASLP摘要：Zero-shot语音转换（VC）仅使用说话者的一个话语将源语音转换成任何期望说话者的语音，而不需要额外的模型更新。典型的方法使用来自预先训练的说话者验证（SV）模型的说话者表示或在VC训练期间学习说话者表示以实现zero-shot VC。然而，现有的说话人建模方法忽略了说话人信息丰富度在语音的时间和频率通道维度上的变化。这种不充分的说话者建模阻碍了VC模型准确地表示不在训练数据集中的未看到的说话者的能力。在这项研究中，我们提出了一个强大的zero-shot VC模型与多级时间通道检索，称为MTCR-VC。具体来说，灵活地适应动态变化的扬声器特性的时间和通道轴的语音，我们提出了一种新的细粒度扬声器建模方法，称为时间通道检索（TCR），找出何时何地扬声器信息出现在语音中。它检索可变长度的说话人表示从时间和通道维度的指导下，一个预先训练的SV模型。此外，受人类语音产生的分层过程的启发，MTCR扬声器模块堆叠多个TCR块以从多粒度级别提取扬声器表示。此外，为了实现更好的语音解纠缠和重构，我们引入了一个基于循环的训练策略来模拟zero-shot推理。我们在内容、风格和演讲者三个方面采取永久性约束来推动这一过程。实验结果表明，MTCR-VC方法优于以往的zero-shot VC方法在建模说话人音色，同时保持良好的语音自然度。摘要：Zero-shot voice conversion (VC) converts source speech into the voice of any desired speaker using only one utterance of the speaker without requiring additional model updates. Typical methods use a speaker representation from a pre-trained speaker verification (SV) model or learn speaker representation during VC training to achieve zero-shot VC. However, existing speaker modeling methods overlook the variation of speaker information richness in temporal and frequency channel dimensions of speech. This insufficient speaker modeling hampers the ability of the VC model to accurately represent unseen speakers who are not in the training dataset. In this study, we present a robust zero-shot VC model with multi-level temporal-channel retrieval, referred to as MTCR-VC. Specifically, to flexibly adapt to the dynamic-variant speaker characteristic in the temporal and channel axis of the speech, we propose a novel fine-grained speaker modeling method, called temporal-channel retrieval (TCR), to find out when and where speaker information appears in speech. It retrieves variable-length speaker representation from both temporal and channel dimensions under the guidance of a pre-trained SV model. Besides, inspired by the hierarchical process of human speech production, the MTCR speaker module stacks several TCR blocks to extract speaker representations from multi-granularity levels. Furthermore, to achieve better speech disentanglement and reconstruction, we introduce a cycle-based training strategy to simulate zero-shot inference recurrently. We adopt perpetual constraints on three aspects, including content, style, and speaker, to drive this process. Experiments demonstrate that MTCR-VC is superior to the previous zero-shot VC methods in modeling speaker timbre while maintaining good speech naturalness.
【3】 Quran Recitation Recognition using End-to-End Deep Learning标题：基于端到端深度学习的古兰经背诵识别链接：https://arxiv.org/abs/2305.07034
*与cs.SD语音【13】为同一篇
作者：Ahmad Al Harere,Khloud Al Jallad机构：Department of Information and Communication Engineering, Arab International University, Daraa, Syria.摘要：《古兰经》是伊斯兰教的圣典，它的背诵是伊斯兰教的重要方面。自动识别古兰经的背诵是一项具有挑战性的任务，因为它的独特规则不适用于正常的演讲。在这一领域已经做了大量的研究，但以前的作品检测背诵错误作为一个分类任务或使用传统的自动语音识别（ASR）。在本文中，我们提出了一种新的端到端深度学习模型，用于识别古兰经的背诵。该模型是一个CNN双向GRU编码器，使用CTC作为目标函数，和基于字符的解码器，这是一个波束搜索解码器。此外，所有以前的作品都是在小型私人数据集上完成的，包括简短的诗句和古兰经的几个章节。由于使用了私有数据集，因此未进行比较。为了克服这个问题，我们使用了一个最近发布的公共数据集（Ar-DAD），包含大约37个章节，由30个背诵者背诵，具有不同的背诵速度和不同类型的发音规则。该模型的性能进行了评估，使用最常见的评价指标，在语音识别，字错误率（WER），字符错误率（CER）。结果为8.34%的WER和2.42%的CER。我们希望这项研究将成为与这个公共新数据集（Ar-DAD）的未来研究进行比较的基线。摘要：The Quran is the holy scripture of Islam, and its recitation is an important aspect of the religion. Recognizing the recitation of the Holy Quran automatically is a challenging task due to its unique rules that are not applied in normal speaking speeches. A lot of research has been done in this domain, but previous works have detected recitation errors as a classification task or used traditional automatic speech recognition (ASR). In this paper, we proposed a novel end-to-end deep learning model for recognizing the recitation of the Holy Quran. The proposed model is a CNN-Bidirectional GRU encoder that uses CTC as an objective function, and a character-based decoder which is a beam search decoder. Moreover, all previous works were done on small private datasets consisting of short verses and a few chapters of the Holy Quran. As a result of using private datasets, no comparisons were done. To overcome this issue, we used a public dataset that has recently been published (Ar-DAD) and contains about 37 chapters that were recited by 30 reciters, with different recitation speeds and different types of pronunciation rules. The proposed model performance was evaluated using the most common evaluation metrics in speech recognition, word error rate (WER), and character error rate (CER). The results were 8.34% WER and 2.42% CER. We hope this research will be a baseline for comparisons with future research on this public new dataset (Ar-DAD).
【4】 Device-Robust Acoustic Scene Classification via Impulse Response  Augmentation标题：基于脉冲响应增强的设备稳健声场景分类链接：https://arxiv.org/abs/2305.07499
*与cs.SD语音【1】为同一篇
作者：Tobias Morocutti,Florian Schmid,Khaled Koutini,Gerhard Widmer机构：Institute of Computational Perception,LIT Artificial Intelligence Lab, Johannes Kepler University, Linz, Austria备注：Submitted to the 31st European Signal Processing Conference, EUSIPCO 2023. Source Code available at: this https URL摘要：推广到各种记录设备的能力是音频分类模型的关键性能因素。不同类型的麦克风的特性由于其变化的频率响应而在数字化音频信号中引入分布偏移。如果在训练过程中不考虑这种域偏移，则当将其应用于由看不见的设备记录的信号时，模型的性能可能会严重降低。特别地，在用少量不同麦克风记录的音频信号上训练模型可能使得难以推广到看不见的设备。为了解决这个问题，我们将训练集中的音频信号与预先记录的设备脉冲响应（DIR）进行卷积，以人为地增加记录设备的多样性。我们系统地研究了DIR增强对使用CNN和音频频谱图Transformers的声学场景分类任务的影响。结果表明，DIR增强在隔离执行类似的国家的最先进的方法Freq-MixStyle。然而，我们还表明，DIR增强和Freq-MixStyle是互补的，在训练期间看不见的设备记录的信号上实现了新的最先进的性能。摘要：The ability to generalize to a wide range of recording devices is a crucial performance factor for audio classification models. The characteristics of different types of microphones introduce distributional shifts in the digitized audio signals due to their varying frequency responses. If this domain shift is not taken into account during training, the model's performance could degrade severely when it is applied to signals recorded by unseen devices. In particular, training a model on audio signals recorded with a small number of different microphones can make generalization to unseen devices difficult. To tackle this problem, we convolve audio signals in the training set with pre-recorded device impulse responses (DIRs) to artificially increase the diversity of recording devices. We systematically study the effect of DIR augmentation on the task of Acoustic Scene Classification using CNNs and Audio Spectrogram Transformers. The results show that DIR augmentation in isolation performs similarly to the state-of-the-art method Freq-MixStyle. However, we also show that DIR augmentation and Freq-MixStyle are complementary, achieving a new state-of-the-art performance on signals recorded by devices unseen during training.
【5】 Benchmarks and leaderboards for sound demixing tasks标题：合理分离任务的基准和排行榜链接：https://arxiv.org/abs/2305.07489
*与cs.SD语音【2】为同一篇
作者：Roman Solovyev,Alexander Stempkovskiy,Tatiana Habruseva机构：Institute for Design Problems in Microelectronics of Russian Academy of Sciences, Moscow, Russian Federation, Independent researcher, Cork, Ireland摘要：音乐去混是将来自给定单个音频信号的不同轨道分离成分量（诸如来自伴奏的其余部分的鼓、低音和人声）的任务。源的分离对于一系列领域是有用的，包括娱乐和助听器。在本文中，我们介绍了两个新的基准的声源分离任务，并比较流行的模型的声音混音，以及他们的合奏，在这些基准。对于模型的评估，我们在https：//mvsep.com/quality_checker/上提供了排行榜，对一系列模型进行了比较。新的基准数据集可供下载。我们还开发了一种新的方法，音频分离的基础上，不同的模型，最适合特定的干的合奏。在2023年音乐分层挑战赛的背景下对所提出的解决方案进行了评估，并在挑战赛的不同赛道中取得了最佳成绩。代码和方法在GitHub上开源。摘要：Music demixing is the task of separating different tracks from the given single audio signal into components, such as drums, bass, and vocals from the rest of the accompaniment. Separation of sources is useful for a range of areas, including entertainment and hearing aids. In this paper, we introduce two new benchmarks for the sound source separation tasks and compare popular models for sound demixing, as well as their ensembles, on these benchmarks. For the models' assessments, we provide the leaderboard at https://mvsep.com/quality_checker/, giving a comparison for a range of models. The new benchmark datasets are available for download. We also develop a novel approach for audio separation, based on the ensembling of different models that are suited best for the particular stem. The proposed solution was evaluated in the context of the Music Demixing Challenge 2023 and achieved top results in different tracks of the challenge. The code and the approach are open-sourced on GitHub.
【6】 Improving Cascaded Unsupervised Speech Translation with Denoising  Back-translation标题：用去噪反译法改进级联无监督语音翻译链接：https://arxiv.org/abs/2305.07455
*与cs.SD语音【3】为同一篇
作者：Yu-Kuan Fu,Liang-Hsuan Tseng,Jiatong Shi,Chen-An Li,Tsu-Yuan Hsu,Shinji Watanabe,Hung-yi Lee机构：College of Electrical Engineering and Computer Science, National Taiwan University, Language Technologies Institute, Carnegie Mellon University摘要：大多数语音翻译模型严重依赖于并行数据，这是很难收集，特别是对于低资源的语言。为了解决这个问题，我们建议建立一个级联的语音翻译系统，而不利用任何类型的配对数据。我们使用完全未配对的数据来训练我们的无监督系统，并在CoVoST 2和CVSS上评估我们的结果。结果表明，我们的工作是与其他一些早期的监督方法在一些语言对。虽然级联系统总是遭受严重的错误传播问题，我们提出了去噪反向翻译（DBT），一种新的方法来建立强大的无监督神经机器翻译（UNMT）。DBT在所有三个平移方向上成功地将BLEU分数提高了0.7- 0.9。此外，我们简化了我们的级联系统的流水线，以减少推理延迟，并进行了全面的分析，我们的工作的每一个部分。我们还展示了我们的无监督的语音翻译结果上建立的网站。摘要：Most of the speech translation models heavily rely on parallel data, which is hard to collect especially for low-resource languages. To tackle this issue, we propose to build a cascaded speech translation system without leveraging any kind of paired data. We use fully unpaired data to train our unsupervised systems and evaluate our results on CoVoST 2 and CVSS. The results show that our work is comparable with some other early supervised methods in some language pairs. While cascaded systems always suffer from severe error propagation problems, we proposed denoising back-translation (DBT), a novel approach to building robust unsupervised neural machine translation (UNMT). DBT successfully increases the BLEU score by 0.7--0.9 in all three translation directions. Moreover, we simplified the pipeline of our cascaded system to reduce inference latency and conducted a comprehensive analysis of every part of our work. We also demonstrate our unsupervised speech translation results on the established website.
【7】 Universal Source Separation with Weakly Labelled Data标题：弱标记数据的泛源分离链接：https://arxiv.org/abs/2305.07447
*与cs.SD语音【4】为同一篇
作者：Qiuqiang Kong,Ke Chen,Haohe Liu,Xingjian Du,Taylor Berg-Kirkpatrick,Shlomo Dubnov,Mark D. Plumbley摘要：通用源分离是计算听觉场景分析的一项基础性研究任务，其目的是将单声道录音分离成单独的源音轨。有三个潜在的挑战等待解决方案的音频源分离任务。首先，先前的音频源分离系统主要集中于分离一个或有限数量的特定源。目前缺乏建立一个统一的系统，可以通过一个单一的模型分离任意源的研究。其次，大多数以前的系统需要干净的源数据来训练分离器，而干净的源数据是稀缺的。第三，缺乏可以自动检测和分离分层级别中的活动声音类别的USS系统。为了使用大规模弱标记/未标记音频数据进行音频源分离，我们提出了一种通用音频源分离框架，该框架包含：1）在弱标记数据上训练的音频标记模型作为查询网;以及2）条件源分离模型，其将查询网络输出作为分离任意声源的条件。我们调查了各种查询网络，源分离模型和训练策略，并提出了一个分层USS策略，自动检测和分离的声音类AudioSet本体。通过单独利用弱标记的AudioSet，我们的USS系统成功地分离了各种各样的声音类，包括声音事件分离，音乐源分离和语音增强。USS系统在AudioSet的527个声音等级上实现了5.57dB的平均信号失真比改善（SDRi）; DCASE 2018任务2数据集上为10.57 dB; MUSDB 18数据集上为8.12 dB; Slakh 2100数据集上的SDRi为7.28dB;以及在语音库需求数据集上的9.00dB的SSNR。我们在www.example.com上发布源代码https://github.com/bytedance/uss摘要：Universal source separation (USS) is a fundamental research task for computational auditory scene analysis, which aims to separate mono recordings into individual source tracks. There are three potential challenges awaiting the solution to the audio source separation task. First, previous audio source separation systems mainly focus on separating one or a limited number of specific sources. There is a lack of research on building a unified system that can separate arbitrary sources via a single model. Second, most previous systems require clean source data to train a separator, while clean source data are scarce. Third, there is a lack of USS system that can automatically detect and separate active sound classes in a hierarchical level. To use large-scale weakly labeled/unlabeled audio data for audio source separation, we propose a universal audio source separation framework containing: 1) an audio tagging model trained on weakly labeled data as a query net; and 2) a conditional source separation model that takes query net outputs as conditions to separate arbitrary sound sources. We investigate various query nets, source separation models, and training strategies and propose a hierarchical USS strategy to automatically detect and separate sound classes from the AudioSet ontology. By solely leveraging the weakly labelled AudioSet, our USS system is successful in separating a wide variety of sound classes, including sound event separation, music source separation, and speech enhancement. The USS system achieves an average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over 527 sound classes of AudioSet; 10.57 dB on the DCASE 2018 Task 2 dataset; 8.12 dB on the MUSDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of 9.00 dB on the voicebank-demand dataset. We release the source code at https://github.com/bytedance/uss
【8】 Investigating the Sensitivity of Automatic Speech Recognition Systems to  Phonetic Variation in L2 Englishes标题：自动语音识别系统对二语英语语音变化的敏感性研究链接：https://arxiv.org/abs/2305.07389
*与cs.SD语音【5】为同一篇
作者：Emma O'Neill,Julie Carson-Berndsen机构：In an investigation into the use of ASR for the purposes of linguis-This research was conducted with the financial support of the School of Computer Science at the ADAPTSFI Research Centre at University College Dublin摘要：自动语音识别（ASR）系统在与其被训练的语音相似的语音上表现出最佳性能。因此，代表性不足的品种，包括地区方言，少数民族语言和低资源语言，比那些被视为“声望”，“主流”或“标准”的品种看到更高的单词错误率（WERs）。这可能会成为将ASR技术纳入大规模语言学研究的注释过程的障碍，因为手动纠正错误的自动转录本可能与手动转录一样耗时耗力。因此，从语音技术的角度来看，在提高ASR准确性方面，以及从注释的角度来看，对ASR系统的行为的更深入理解是有益的，其中，知道ASR系统可能产生的错误可以有助于这种手动校正。这项工作展示了一种方法，探测ASR系统，以发现它如何处理语音变化在一些第二语言英语。具体地，在系统的训练数据中罕见或不存在的特定语音实现如何能够导致音素水平的误识别并有助于更高的WER。它表明，ASR的行为是系统的和一致的扬声器具有类似的口语品种（在这种情况下，相同的L1）和音素替换错误通常是在协议与人类注释。通过识别有问题的制作，可以通过获取用于训练和微调的这种实现来解决特定的弱点，从而使系统对发音变化更鲁棒。摘要：Automatic Speech Recognition (ASR) systems exhibit the best performance on speech that is similar to that on which it was trained. As such, underrepresented varieties including regional dialects, minority-speakers, and low-resource languages, see much higher word error rates (WERs) than those varieties seen as 'prestigious', 'mainstream', or 'standard'. This can act as a barrier to incorporating ASR technology into the annotation process for large-scale linguistic research since the manual correction of the erroneous automated transcripts can be just as time and resource consuming as manual transcriptions. A deeper understanding of the behaviour of an ASR system is thus beneficial from a speech technology standpoint, in terms of improving ASR accuracy, and from an annotation standpoint, where knowing the likely errors made by an ASR system can aid in this manual correction. This work demonstrates a method of probing an ASR system to discover how it handles phonetic variation across a number of L2 Englishes. Specifically, how particular phonetic realisations which were rare or absent in the system's training data can lead to phoneme level misrecognitions and contribute to higher WERs. It is demonstrated that the behaviour of the ASR is systematic and consistent across speakers with similar spoken varieties (in this case the same L1) and phoneme substitution errors are typically in agreement with human annotators. By identifying problematic productions specific weaknesses can be addressed by sourcing such realisations for training and fine-tuning thus making the system more robust to pronunciation variation.
【9】 Music Rearrangement Using Hierarchical Segmentation标题：基于层次分割的音乐重排技术链接：https://arxiv.org/abs/2305.07347
*与cs.SD语音【6】为同一篇
作者：Christos Plachouras,Marius Miron机构：Music Technology Group, Universitat Pompeu Fabra, Spain摘要：音乐重排涉及重新编排、删除和重复音乐作品的部分，目的是产生具有不同持续时间的独立版本。这是一项创造性和耗时的任务，通常由专业音乐工程师执行。在本文中，我们提出了一种自动重新安排音乐录音，考虑到录音的层次结构的方法。先前的方法仅关注于识别音频中的可能导致平滑过渡的切割点。相反，我们利用深度音频表示来分层地分割作品，并根据片段的边界和音乐功能定义切点搜索。我们根据它们的相似性和它们所属的段对合适的入口和出口点对进行评分，并定义最佳路径搜索。实验结果表明，所选择的切割点是最常见的听众难以察觉的，并导致更一致的音乐发展与更少的分心重复。摘要：Music rearrangement involves reshuffling, deleting, and repeating sections of a music piece with the goal of producing a standalone version that has a different duration. It is a creative and time-consuming task commonly performed by an expert music engineer. In this paper, we propose a method for automatically rearranging music recordings that takes into account the hierarchical structure of the recording. Previous approaches focus solely on identifying cut-points in the audio that could result in smooth transitions. We instead utilize deep audio representations to hierarchically segment the piece and define a cut-point search subject to the boundaries and musical functions of the segments. We score suitable entry- and exit-point pairs based on their similarity and the segments they belong to, and define an optimal path search. Experimental results demonstrate the selected cut-points are most commonly imperceptible by listeners and result in more consistent musical development with less distracting repetitions.
【10】 Better speech synthesis through scaling标题：通过缩放改进语音合成链接：https://arxiv.org/abs/2305.07243
*与cs.SD语音【7】为同一篇
作者：James Betker摘要：近年来，自回归Transformers和DDPM的应用使图像生成领域发生了革命性的变化。这些方法将图像生成过程建模为逐步概率过程，并利用大量的计算和数据来学习图像分布。这种提高性能的方法不需要局限于图像。本文介绍了一种方法，适用于图像生成领域的语音合成的进展。其结果是TorToise -一个富有表现力的多语音文本到语音系统。  所有模型代码和训练权重都已在www.example.com上开源https://github.com/neonbjb/tortoise-tts。摘要：In recent years, the field of image generation has been revolutionized by the application of autoregressive transformers and DDPMs. These approaches model the process of image generation as a step-wise probabilistic processes and leverage large amounts of compute and data to learn the image distribution. This methodology of improving performance need not be confined to images. This paper describes a way to apply advances in the image generative domain to speech synthesis. The result is TorToise -- an expressive, multi-voice text-to-speech system.  All model code and trained weights have been open-sourced at https://github.com/neonbjb/tortoise-tts.
【11】 Hear to Segment: Unmixing the Audio to Guide the Semantic Segmentation标题：Hear to Segment：分解音频以指导语义切分链接：https://arxiv.org/abs/2305.07223
*与cs.SD语音【8】为同一篇
作者：Yuhang Ling,Yuxi Li,Zhenye Gan,Jiangning Zhang,Mingmin Chi,Yabiao Wang机构：Fudan University, Shanghai, China, Youtu lab, Tencent备注：11 pages, 7 figures摘要：在本文中，我们专注于最近提出的一个新的任务，称为视听分割（AVS），其中需要建立音频流和图像像素之间的细粒度的对应关系。然而，学习这种对应关系面临两个关键挑战：（1）音频信号固有地表现出高度的信息密度，因为由多个对象产生的声音在同一音频流内纠缠;（2）同一类别的对象的音频信号的频率趋于相似，这妨碍了目标对象的区分，从而导致不明确的分割结果。为此，我们提出了一个音频解混和语义分割网络（AUSS），它鼓励解混复杂的音频信号和区分类似的声音。从技术上讲，我们的AUSS将音频信号解混成一组音频查询，并通过掩蔽注意机制将它们与视觉特征进行交互。为了鼓励这些音频查询捕获嵌入在音频中的独特特征，还引入了两个自监督损失作为类和掩码级别的额外监督。AVSBench基准测试的大量实验结果表明，我们的AUSS在单源和多源子集中都具有新的最先进水平，证明了我们的AUSS在弥合音频和视觉模式之间的差距方面的有效性。摘要：In this paper, we focus on a recently proposed novel task called Audio-Visual Segmentation (AVS), where the fine-grained correspondence between audio stream and image pixels is required to be established. However, learning such correspondence faces two key challenges: (1) audio signals inherently exhibit a high degree of information density, as sounds produced by multiple objects are entangled within the same audio stream; (2) the frequency of audio signals from objects with the same category tends to be similar, which hampers the distinction of target object and consequently leads to ambiguous segmentation results. Toward this end, we propose an Audio Unmixing and Semantic Segmentation Network (AUSS), which encourages unmixing complicated audio signals and distinguishing similar sounds. Technically, our AUSS unmixs the audio signals into a set of audio queries, and interacts them with visual features by masked attention mechanisms. To encourage these audio queries to capture distinctive features embedded within the audio, two self-supervised losses are also introduced as additional supervision at both class and mask levels. Extensive experimental results on the AVSBench benchmark show that our AUSS sets a new state-of-the-art in both single-source and multi-source subsets, demonstrating the effectiveness of our AUSS in bridging the gap between audio and vision modalities.
【12】 Versatile Audio-Visual Learning for Handling Single and Multi Modalities  in Emotion Regression and Classification Tasks标题：情绪回归和分类任务中处理单通道和多通道的多功能视听学习链接：https://arxiv.org/abs/2305.07216
*与cs.SD语音【9】为同一篇
作者：Lucas Goncalves,Seong-Gyun Leem,Wei-Cheng Lin,Berrak Sisman,Carlos Busso机构：Multimodal Signal Processing (MSP) Lab, Erik Jonsson School of Engineering and Computer Science, The University of Texas at Dallas备注：14 pages, 2 Figures, 2 tables摘要：目前大多数视听情感识别模型缺乏在实际应用中部署所需的灵活性。我们设想一个多模态系统，即使只有一种模态是可用的，并且可以互换地实现用于预测情感属性或识别分类情感。在多模态情感识别系统中实现这样的灵活性是困难的，这是由于在准确地解释和集成不同的数据源的固有挑战。在允许回归和分类任务之间的直接切换的同时稳健地处理缺失或部分信息也是一个挑战。本研究提出了一个通用的视听学习（VAVL）的框架，用于处理情绪回归和情绪分类任务的单峰和多模态系统。我们实现了一种视听框架，即使在视听配对数据对于训练集的一部分不可用时（即，仅存在音频或仅存在视频）。我们实现了这种有效的表示学习与视听共享层，剩余连接共享层，和一个单峰重建任务。我们的实验结果表明，我们的架构显着优于强大的基线CREMA-D和MSP-IMPROV语料库。值得注意的是，VAVL在MSP-IMPROV语料库上的情感属性预测任务中获得了新的最先进的性能。代码可在：https://github.com/ilucasgoncalves/VAVL摘要：Most current audio-visual emotion recognition models lack the flexibility needed for deployment in practical applications. We envision a multimodal system that works even when only one modality is available and can be implemented interchangeably for either predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is difficult due to the inherent challenges in accurately interpreting and integrating varied data sources. It is also a challenge to robustly handle missing or partial information while allowing direct switch between regression and classification tasks. This study proposes a \emph{versatile audio-visual learning} (VAVL) framework for handling unimodal and multimodal systems for emotion regression and emotion classification tasks. We implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set (i.e., audio only or only video is present). We achieve this effective representation learning with audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Our experimental results reveal that our architecture significantly outperforms strong baselines on both the CREMA-D and MSP-IMPROV corpora. Notably, VAVL attains a new state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus. Code available at: https://github.com/ilucasgoncalves/VAVL
【13】 Tackling Interpretability in Audio Classification Networks with  Non-negative Matrix Factorization标题：利用非负矩阵分解解决音频分类网络中的可译性问题链接：https://arxiv.org/abs/2305.07132
*与cs.SD语音【10】为同一篇
作者：Jayneel Parekh,Sanjeel Parekh,Pavlo Mozharovskyi,Gaël Richard,Florence d'Alché-Buc备注：Under submission at IEEE/ACM TASLP. arXiv admin note: text overlap with arXiv:2202.11479摘要：本文解决了两个主要的问题设置的音频处理网络的可解释性，事后和设计的解释。对于事后的解释，我们的目标是解释的决定，网络的高级音频对象，也是listenable的终端用户。这是扩展到目前一个固有的解释模型，具有高性能。为此，我们提出了一种新的解释器设计，采用非负矩阵分解（NMF）。特别地，解释器被训练为从目标网络的隐藏层生成正则化的中间嵌入，作为预先学习的NMF字典的时间激活来学习。我们的方法使我们能够生成直观的基于音频的解释，显式地增强输入信号的部分最相关的网络的决定。我们证明了我们的方法的适用性在各种分类任务，包括多标签数据的真实世界的音频和音乐。摘要：This paper tackles two major problem settings for interpretability of audio processing networks, post-hoc and by-design interpretation. For post-hoc interpretation, we aim to interpret decisions of a network in terms of high-level audio objects that are also listenable for the end-user. This is extended to present an inherently interpretable model with high performance. To this end, we propose a novel interpreter design that incorporates non-negative matrix factorization (NMF). In particular, an interpreter is trained to generate a regularized intermediate embedding from hidden layers of a target network, learnt as time-activations of a pre-learnt NMF dictionary. Our methodology allows us to generate intuitive audio-based interpretations that explicitly enhance parts of the input signal most relevant for a network's decision. We demonstrate our method's applicability on a variety of classification tasks, including multi-label data for real-world audio and music.
机器翻译，仅供参考