今日论文合集：cs.SD语音8篇，eess.AS音频处理9篇。
cs.SD语音
【1】 CDSD: Chinese Dysarthria Speech Database标题：CDSD：汉语构音障碍语音库链接：https://arxiv.org/abs/2310.15930
作者：Mengyi Sun,Ming Gao,Xinchen Kang,Shiru Wang,Jun Du,Dengfeng Yao,Su-Jing Wang备注：9 pages, 3 figures摘要：我们介绍了中国构音障碍语音数据库（CDSD）作为构音障碍研究的一个有价值的资源。该数据库包括来自24名患有构音障碍的参与者的语音数据。在这些参与者中，一人额外记录了10小时的演讲数据，而每个人记录了一小时，从而产生了34小时的演讲材料。为了适应不同认知水平的参与者，我们的文本库主要由AISHELL-1数据集的内容和中小学生的演讲组成。当与会者阅读这些文本时，他们必须使用移动终端或ZOOM F8 n多轨现场录音机来记录他们的发言。在本文中，我们阐明了数据收集和注释过程，并提出了一种建立构音障碍语音识别基线的方法。此外，我们进行了一个依赖于说话者的构音障碍语音识别实验，使用额外的10个小时的语音数据，从我们的参与者之一。我们的研究结果表明，通过广泛的数据驱动的模型训练，微调有限数量的特定个人数据在说话者依赖性构音障碍语音识别中产生了值得称赞的结果。然而，我们观察到不同构音障碍的扬声器之间的识别结果的显着变化。这些见解提供了有价值的参考点依赖于说话人构音障碍的语音识别。摘要：We present the Chinese Dysarthria Speech Database (CDSD) as a valuable resource for dysarthria research. This database comprises speech data from 24 participants with dysarthria. Among these participants, one recorded an additional 10 hours of speech data, while each recorded one hour, resulting in 34 hours of speech material. To accommodate participants with varying cognitive levels, our text pool primarily consists of content from the AISHELL-1 dataset and speeches by primary and secondary school students. When participants read these texts, they must use a mobile device or the ZOOM F8n multi-track field recorder to record their speeches. In this paper, we elucidate the data collection and annotation processes and present an approach for establishing a baseline for dysarthric speech recognition. Furthermore, we conducted a speaker-dependent dysarthric speech recognition experiment using an additional 10 hours of speech data from one of our participants. Our research findings indicate that, through extensive data-driven model training, fine-tuning limited quantities of specific individual data yields commendable results in speaker-dependent dysarthric speech recognition. However, we observe significant variations in recognition results among different dysarthric speakers. These insights provide valuable reference points for speaker-dependent dysarthric speech recognition.
【2】 How Much Context Does My Attention-Based ASR System Need?标题：我的基于注意力的ASR系统需要多少上下文？链接：https://arxiv.org/abs/2310.15672
作者：Robert Flynn,Anton Ragni摘要：对于语音识别任务，在训练期间使用超过30秒的声学上下文是不常见的，并且在文献中研究不足。在这项工作中，我们研究的效果缩放序列长度用于训练/评估（密集注意力为基础的）声学和语言模型的语音识别性能。对于这些实验，使用了大约100，000个伪标记的Spotify播客的数据集，探索了5秒到1小时的上下文长度。对长格式数据集Earnings-22和Teddom的Zero-shot评估表明，使用大约80秒的声学背景进行训练是有益的，与有限的背景基线相比，相对提高了14.9%。此外，我们执行一个系统的组合与长上下文Transformer语言模型通过波束搜索一个完整的长上下文ASR系统，与当前的最先进的竞争力的结果。摘要：For the task of speech recognition, the use of more than 30 seconds of acoustic context during training is uncommon, and under-investigated in literature. In this work, we examine the effect of scaling the sequence length used to train/evaluate (dense-attention based) acoustic and language models on speech recognition performance. For these experiments a dataset of roughly 100,000 pseudo-labelled Spotify podcasts is used, with context lengths of 5 seconds to 1 hour being explored. Zero-shot evaluations on long-format datasets Earnings-22 and Tedlium demonstrate a benefit from training with around 80 seconds of acoustic context, showing up to a 14.9% relative improvement from a limited context baseline. Furthermore, we perform a system combination with long-context transformer language models via beam search for a fully long-context ASR system, with results that are competitive with the current state-of-the-art.
【3】 Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio  Models标题：动态卷积神经网络作为高效的预训练音频模型链接：https://arxiv.org/abs/2310.15648
作者：Florian Schmid,Khaled Koutini,Gerhard Widmer备注：Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing. Source Code available at: this https URL摘要：大规模音频数据集（如AudioSet）的引入为Transformers征服音频领域并取代CNN成为许多任务的最先进神经网络架构铺平了道路。Audio Spectrogram Transformers在利用大型数据集方面非常出色，可以创建功能强大的预训练模型，当对下游任务进行微调时，这些模型将超过CNN。然而，与CNN相比，当前流行的音频频谱图Transformers在计算复杂度方面要求较高。最近，我们已经证明，通过采用Transformer-to-CNN知识蒸馏，高效的CNN可以在大型数据集上赶上甚至超过Transformers。在这项工作中，我们扩展了这一研究领域，并通过引入动态CNN块来增加高效CNN的容量，这些CNN块由动态非线性、动态卷积和注意力机制构成。我们表明，在大规模AudioSet上的音频标记任务中，这些动态CNN在性能复杂度权衡和参数效率方面优于传统的高效CNN。我们的实验进一步表明，引入的动态CNN在下游任务上实现了更好的性能，并且可以很好地扩展，达到Transformer性能，甚至在AudioSet和几个下游任务上优于它们。摘要：The introduction of large-scale audio datasets, such as AudioSet, paved the way for Transformers to conquer the audio domain and replace CNNs as the state-of-the-art neural network architecture for many tasks. Audio Spectrogram Transformers are excellent at exploiting large datasets, creating powerful pre-trained models that surpass CNNs when fine-tuned on downstream tasks. However, current popular Audio Spectrogram Transformers are demanding in terms of computational complexity compared to CNNs. Recently, we have shown that, by employing Transformer-to-CNN Knowledge Distillation, efficient CNNs can catch up with and even outperform Transformers on large datasets. In this work, we extend this line of research and increase the capacity of efficient CNNs by introducing dynamic CNN blocks, constructed of dynamic non-linearities, dynamic convolutions and attention mechanisms. We show that these dynamic CNNs outperform traditional efficient CNNs, in terms of the performance-complexity trade-off and parameter efficiency, at the task of audio tagging on the large-scale AudioSet. Our experiments further indicate that the introduced dynamic CNNs achieve better performance on downstream tasks and scale up well, attaining Transformer performance and even outperforming them on AudioSet and several downstream tasks.
【4】 Modality Dropout for Multimodal Device Directed Speech Detection using  Verbal and Non-Verbal Features标题：基于语言和非语言特征的多通道设备导向语音检测的通道丢失链接：https://arxiv.org/abs/2310.15261
作者：Gautam Krishna,Sameer Dharur,Oggi Rudovic,Pranay Dighe,Saurabh Adya,Ahmed Hussen Abdelaziz,Ahmed H Tewfik备注：5 pages摘要：设备导向语音检测（DDSD）是区分针对语音助理的查询与侧面对话或背景语音的二进制分类任务。现有技术的DDSD系统使用口头提示（例如，声学、文本和/或自动语音识别系统（ASR）特征）来将语音分类为设备引导的或其他的，并且当部署在现实世界设置中时，通常必须应对这些模态中的一个或多个不可用。在本文中，我们研究DDSD系统的融合方案，可以更强大的丢失模态。同时，我们研究了非语言线索的使用，特别是韵律功能，除了言语线索DDSD。我们提出了不同的方法来结合分数和嵌入从韵律与相应的口头线索，发现韵律提高DDSD性能高达8.5%的错误接受率（FA）在一个给定的固定操作点通过非线性中间融合，而我们使用的模态丢弃技术提高了这些模型的性能7.4%在推理时间内使用缺失模态进行评估时，摘要：Device-directed speech detection (DDSD) is the binary classification task of distinguishing between queries directed at a voice assistant versus side conversation or background speech. State-of-the-art DDSD systems use verbal cues, e.g acoustic, text and/or automatic speech recognition system (ASR) features, to classify speech as device-directed or otherwise, and often have to contend with one or more of these modalities being unavailable when deployed in real-world settings. In this paper, we investigate fusion schemes for DDSD systems that can be made more robust to missing modalities. Concurrently, we study the use of non-verbal cues, specifically prosody features, in addition to verbal cues for DDSD. We present different approaches to combine scores and embeddings from prosody with the corresponding verbal cues, finding that prosody improves DDSD performance by upto 8.5% in terms of false acceptance rate (FA) at a given fixed operating point via non-linear intermediate fusion, while our use of modality dropout techniques improves the performance of these models by 7.4% in terms of FA when evaluated with missing modalities during inference time.
【5】 SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis标题：SyncFusion：多模式同步视频到音频Foley合成链接：https://arxiv.org/abs/2310.15247
作者：Marco Comunità,Riccardo F. Gramaccioni,Emilian Postolache,Emanuele Rodolà,Danilo Comminiello,Joshua D. Reiss摘要：声音设计涉及创造性地选择，录制和编辑各种媒体的声音效果，如电影，视频游戏和虚拟/增强现实。设计声音时最耗时的步骤之一是同步音频和视频。在某些情况下，可以使用视频拍摄的环境记录，这可以帮助这个过程。然而，在视频游戏和动画中，不存在参考音频，需要从视频手动注释事件定时。我们提出了一个从视频中提取重复动作起始的系统，然后将其与音频或文本嵌入结合使用，以调节经过训练的扩散模型，从而生成新的同步音效音轨。通过这种方式，我们将完全的创意控制权留给了声音设计师，同时消除了与视频同步的负担。此外，编辑起始音轨或改变条件嵌入需要比编辑音轨本身少得多的努力，从而简化发声过程。我们提供可靠的示例、源代码和预训练模型，以促进可重复性摘要：Sound design involves creatively selecting, recording, and editing sound effects for various media like cinema, video games, and virtual/augmented reality. One of the most time-consuming steps when designing sound is synchronizing audio with video. In some cases, environmental recordings from video shoots are available, which can aid in the process. However, in video games and animations, no reference audio exists, requiring manual annotation of event timings from the video. We propose a system to extract repetitive actions onsets from a video, which are then used - in conjunction with audio or textual embeddings - to condition a diffusion model trained to generate a new synchronized sound effects audio track. In this way, we leave complete creative control to the sound designer while removing the burden of synchronization with video. Furthermore, editing the onset track or changing the conditioning embedding requires much less effort than editing the audio track itself, simplifying the sonification process. We provide sound examples, source code, and pretrained models to faciliate reproducibility
【6】 FOLEY-VAE: Generación de efectos de audio para cine con inteligencia  artificial标题：Foley-VAE：人工智能音响设备链接：https://arxiv.org/abs/2310.15663
作者：Mateo Cámara,José Luis Blanco备注：9 pages, in Spanish, Tecniac\'ustica摘要：在这项研究中，我们提出了一个基于可变自动编码器的接口，该接口使用各种自然声音进行训练，以创新地创建Foley效果。该模型可以将新的声音特征实时传输到预先录制的音频或麦克风捕获的语音中。此外，它还允许对潜在变量进行交互式修改，便于进行精确和定制的艺术调整。以我们之前在去年同一次大会上提出的变分自编码器研究为起点，我们分析了现有的实现：RAVE [1]。该模型经过专门的音频效果制作训练。各种音频效果已经成功地产生，从电磁，科幻小说，和水的声音，其中包括与这项工作出版。这种创新的方法是第一部西班牙短片的艺术创作的基础，该短片具有人工智能辅助的声音效果。这一里程碑清楚地说明了这项技术在电影行业的变革潜力，为声音创作和提高电影制作的艺术质量打开了新的可能性。摘要：In this research, we present an interface based on Variational Autoencoders trained with a wide range of natural sounds for the innovative creation of Foley effects. The model can transfer new sound features to prerecorded audio or microphone-captured speech in real time. In addition, it allows interactive modification of latent variables, facilitating precise and customized artistic adjustments. Taking as a starting point our previous study on Variational Autoencoders presented at this same congress last year, we analyzed an existing implementation: RAVE [1]. This model has been specifically trained for audio effects production. Various audio effects have been successfully generated, ranging from electromagnetic, science fiction, and water sounds, among others published with this work. This innovative approach has been the basis for the artistic creation of the first Spanish short film with sound effects assisted by artificial intelligence. This milestone illustrates palpably the transformative potential of this technology in the film industry, opening the door to new possibilities for sound creation and the improvement of artistic quality in film productions.
【7】 The Mason-Alberta Phonetic Segmenter: A forced alignment system based on  deep neural networks and interpolation标题：Mason-Alberta语音分割器：一种基于深度神经网络和内插的强制对齐系统链接：https://arxiv.org/abs/2310.15425
作者：Matthew C. Kelley,Scott James Perry,Benjamin V. Tucker备注：submitted for publication摘要：强制对齐系统自动确定语音数据中的段之间的边界，给定正交转录。这些工具在语音学中是常见的，以便于使用无法手动转录和分段的语音数据。在本文中，我们描述了一个新的基于神经网络的强制对齐系统，梅森-阿尔伯塔语音分段器（MAPS）。MAPS对准器作为一个测试平台，用于我们对强制对准系统进行的两种可能的改进。第一种是将强制对准器中的声学模型视为标记任务，而不是分类任务，其动机是普遍理解语音中的片段不是真正离散的并且通常重叠。第二种是插值技术，允许边界比现代强制对齐系统中常见的10 ms限制更精确。我们比较我们的系统配置的最先进的系统，蒙特利尔强制对齐。标记方法通常不会产生比蒙特利尔强制对齐器更好的结果。然而，具有插值技术的系统在测试集上的目标的10 ms内的边界量方面相对于Montreal Forced Aligner增加了27.92%。我们还反映了在强制对齐声学建模的任务和训练过程中，突出这些模型的输出目标如何不匹配语音学家的概念之间的相似性电话和这种紧张局势的和解可能需要重新思考的任务和输出目标或语音本身应该如何分割。摘要：Forced alignment systems automatically determine boundaries between segments in speech data, given an orthographic transcription. These tools are commonplace in phonetics to facilitate the use of speech data that would be infeasible to manually transcribe and segment. In the present paper, we describe a new neural network-based forced alignment system, the Mason-Alberta Phonetic Segmenter (MAPS). The MAPS aligner serves as a testbed for two possible improvements we pursue for forced alignment systems. The first is treating the acoustic model in a forced aligner as a tagging task, rather than a classification task, motivated by the common understanding that segments in speech are not truly discrete and commonly overlap. The second is an interpolation technique to allow boundaries more precise than the common 10 ms limit in modern forced alignment systems. We compare configurations of our system to a state-of-the-art system, the Montreal Forced Aligner. The tagging approach did not generally yield improved results over the Montreal Forced Aligner. However, a system with the interpolation technique had a 27.92% increase relative to the Montreal Forced Aligner in the amount of boundaries within 10 ms of the target on the test set. We also reflect on the task and training process for acoustic modeling in forced alignment, highlighting how the output targets for these models do not match phoneticians' conception of similarity between phones and that reconciliation of this tension may require rethinking the task and output targets or how speech itself should be segmented.
【8】 GESI: Gammachirp Envelope Similarity Index for Predicting  Intelligibility of Simulated Hearing Loss Sounds标题：预测模拟听力损失声音清晰度的Gammachirp包络相似性指数链接：https://arxiv.org/abs/2310.15399
作者：Ayako Yamamoto,Toshio Irino,Fuki Miyazaki,Honoka Tamaru备注：This paper was submitted to Speech Communication on September 14, 2023摘要：本文提出了一种新的客观可懂度指标Gammachirp包络相似性指数（GESI），用于预测正常听力（NH）受试者模拟听力损失（HL）声音的言语可懂度（SI）。GESI是一种侵入式方法，它使用伽马线性滤波器组（GCFB）、调制滤波器组和扩展余弦相似性度量来计算SI度量。GESI可以接受参考声和测试声的水平不对称性，并反映HI听者的听力水平，因为它出现在听力图上。GESI的一个独特功能是它能够将个人参与者的听力条件纳入SI预测。我们在实验室和众包远程环境中对男性和女性的语音进行了四次SI实验。然后，我们评估了GESI和传统的OIMs，STOI，ESTOI，MBSTOI和HASPI，他们的能力，预测平均和个人SI值与和不使用模拟HL声音。GESI在所有评价中的表现都优于其他OIM。即使使用模拟HL声音，STOI、ESTOI和MBSTOI也根本不能预测SI。HASPI没有预测男性语音和个人SI值的实验室和远程实验之间的差异。GESI可以提供一个第一步SI预测个别HI听众的HL是由外周功能障碍引起的。摘要：We proposed a new objective intelligibility measure (OIM), called the Gammachirp Envelope Similarity Index (GESI), which can predict the speech intelligibility (SI) of simulated hearing loss (HL) sounds for normal hearing (NH) listeners. GESI is an intrusive method that computes the SI metric using the gammachirp filterbank (GCFB), the modulation filterbank, and the extended cosine similarity measure. GESI can accept the level asymmetry of the reference and test sounds and reflect the HI listener's hearing level as it appears on the audiogram. A unique feature of GESI is its ability to incorporate an individual participant's listening condition into the SI prediction. We conducted four SI experiments on male and female speech sounds in both laboratory and crowdsourced remote environments. We then evaluated GESI and the conventional OIMs, STOI, ESTOI, MBSTOI, and HASPI, for their ability to predict mean and individual SI values with and without the use of simulated HL sounds. GESI outperformed the other OIMs in all evaluations. STOI, ESTOI, and MBSTOI did not predict SI at all, even when using the simulated HL sounds. HASPI did not predict the difference between the laboratory and remote experiments on male speech sounds and the individual SI values. GESI may provide a first step toward SI prediction for individual HI listeners whose HL is caused solely by peripheral dysfunction.
eess.AS音频处理
【1】 Pre-training Music Classification Models via Music Source Separation标题：基于音源分离的预训练音乐分类模型链接：https://arxiv.org/abs/2310.15845
作者：Christos Garoufis,Athanasia Zlatintsi,Petros Maragos备注：5 pages (4+references), 3 figures. ICASSP-24 submission摘要：在本文中，我们研究音乐源分离是否可以作为一个预训练策略的音乐表示学习，针对音乐分类任务。为此，我们首先在各种音乐源分离目标下预训练U-Net网络，例如从音乐作品中分离声乐或乐器源;然后，我们将卷积尾网络附加到预训练的U-Net并共同微调整个网络。分离网络学习到的特征也通过跳过连接传播到尾部网络。在两个广泛使用和公开可用的数据集上的实验结果表明，与从头开始训练整个网络和使用尾网络作为两个音乐分类任务的独立任务相比，使用音乐源分离目标预训练U-Net可以提高性能：使用声乐分离时的音乐自动标记，以及多源分离情况下的音乐流派分类。摘要：In this paper, we study whether music source separation can be used as a pre-training strategy for music representation learning, targeted at music classification tasks. To this end, we first pre-train U-Net networks under various music source separation objectives, such as the isolation of vocal or instrumental sources from a musical piece; afterwards, we attach a convolutional tail network to the pre-trained U-Net and jointly finetune the whole network. The features learned by the separation network are also propagated to the tail network through skip connections. Experimental results in two widely used and publicly available datasets indicate that pre-training the U-Nets with a music source separation objective can improve performance compared to both training the whole network from scratch and using the tail network as a standalone in two music classification tasks: music auto-tagging, when vocal separation is used, and music genre classification for the case of multi-source separation.
【2】 FOLEY-VAE: Generación de efectos de audio para cine con inteligencia  artificial标题：Foley-VAE：人工智能音响设备链接：https://arxiv.org/abs/2310.15663
作者：Mateo Cámara,José Luis Blanco备注：9 pages, in Spanish, Tecniac\'ustica摘要：在这项研究中，我们提出了一个基于可变自动编码器的接口，该接口使用各种自然声音进行训练，以创新地创建Foley效果。该模型可以将新的声音特征实时传输到预先录制的音频或麦克风捕获的语音中。此外，它还允许对潜在变量进行交互式修改，便于进行精确和定制的艺术调整。以我们之前在去年同一次大会上提出的变分自编码器研究为起点，我们分析了现有的实现：RAVE [1]。该模型经过专门的音频效果制作训练。各种音频效果已经成功地产生，从电磁，科幻小说，和水的声音，其中包括与这项工作出版。这种创新的方法是第一部西班牙短片的艺术创作的基础，该短片具有人工智能辅助的声音效果。这一里程碑清楚地说明了这项技术在电影行业的变革潜力，为声音创作和提高电影制作的艺术质量打开了新的可能性。摘要：In this research, we present an interface based on Variational Autoencoders trained with a wide range of natural sounds for the innovative creation of Foley effects. The model can transfer new sound features to prerecorded audio or microphone-captured speech in real time. In addition, it allows interactive modification of latent variables, facilitating precise and customized artistic adjustments. Taking as a starting point our previous study on Variational Autoencoders presented at this same congress last year, we analyzed an existing implementation: RAVE [1]. This model has been specifically trained for audio effects production. Various audio effects have been successfully generated, ranging from electromagnetic, science fiction, and water sounds, among others published with this work. This innovative approach has been the basis for the artistic creation of the first Spanish short film with sound effects assisted by artificial intelligence. This milestone illustrates palpably the transformative potential of this technology in the film industry, opening the door to new possibilities for sound creation and the improvement of artistic quality in film productions.
【3】 The Mason-Alberta Phonetic Segmenter: A forced alignment system based on  deep neural networks and interpolation标题：Mason-Alberta语音分割器：一种基于深度神经网络和内插的强制对齐系统链接：https://arxiv.org/abs/2310.15425
作者：Matthew C. Kelley,Scott James Perry,Benjamin V. Tucker备注：submitted for publication摘要：None摘要：Forced alignment systems automatically determine boundaries between segments in speech data, given an orthographic transcription. These tools are commonplace in phonetics to facilitate the use of speech data that would be infeasible to manually transcribe and segment. In the present paper, we describe a new neural network-based forced alignment system, the Mason-Alberta Phonetic Segmenter (MAPS). The MAPS aligner serves as a testbed for two possible improvements we pursue for forced alignment systems. The first is treating the acoustic model in a forced aligner as a tagging task, rather than a classification task, motivated by the common understanding that segments in speech are not truly discrete and commonly overlap. The second is an interpolation technique to allow boundaries more precise than the common 10 ms limit in modern forced alignment systems. We compare configurations of our system to a state-of-the-art system, the Montreal Forced Aligner. The tagging approach did not generally yield improved results over the Montreal Forced Aligner. However, a system with the interpolation technique had a 27.92% increase relative to the Montreal Forced Aligner in the amount of boundaries within 10 ms of the target on the test set. We also reflect on the task and training process for acoustic modeling in forced alignment, highlighting how the output targets for these models do not match phoneticians' conception of similarity between phones and that reconciliation of this tension may require rethinking the task and output targets or how speech itself should be segmented.
【4】 GESI: Gammachirp Envelope Similarity Index for Predicting  Intelligibility of Simulated Hearing Loss Sounds标题：预测模拟听力损失声音清晰度的Gammachirp包络相似性指数链接：https://arxiv.org/abs/2310.15399
作者：Ayako Yamamoto,Toshio Irino,Fuki Miyazaki,Honoka Tamaru备注：This paper was submitted to Speech Communication on September 14, 2023摘要：本文提出了一种新的客观可懂度指标Gammachirp包络相似性指数（GESI），用于预测正常听力（NH）受试者模拟听力损失（HL）声音的言语可懂度（SI）。GESI是一种侵入式方法，它使用伽马线性滤波器组（GCFB）、调制滤波器组和扩展余弦相似性度量来计算SI度量。GESI可以接受参考声和测试声的水平不对称性，并反映HI听者的听力水平，因为它出现在听力图上。GESI的一个独特功能是它能够将个人参与者的听力条件纳入SI预测。我们在实验室和众包远程环境中对男性和女性的语音进行了四次SI实验。然后，我们评估了GESI和传统的OIMs，STOI，ESTOI，MBSTOI和HASPI，他们的能力，预测平均和个人SI值与和不使用模拟HL声音。GESI在所有评价中的表现都优于其他OIM。即使使用模拟HL声音，STOI、ESTOI和MBSTOI也根本不能预测SI。HASPI没有预测男性语音和个人SI值的实验室和远程实验之间的差异。GESI可以提供一个第一步SI预测个别HI听众的HL是由外周功能障碍引起的。摘要：We proposed a new objective intelligibility measure (OIM), called the Gammachirp Envelope Similarity Index (GESI), which can predict the speech intelligibility (SI) of simulated hearing loss (HL) sounds for normal hearing (NH) listeners. GESI is an intrusive method that computes the SI metric using the gammachirp filterbank (GCFB), the modulation filterbank, and the extended cosine similarity measure. GESI can accept the level asymmetry of the reference and test sounds and reflect the HI listener's hearing level as it appears on the audiogram. A unique feature of GESI is its ability to incorporate an individual participant's listening condition into the SI prediction. We conducted four SI experiments on male and female speech sounds in both laboratory and crowdsourced remote environments. We then evaluated GESI and the conventional OIMs, STOI, ESTOI, MBSTOI, and HASPI, for their ability to predict mean and individual SI values with and without the use of simulated HL sounds. GESI outperformed the other OIMs in all evaluations. STOI, ESTOI, and MBSTOI did not predict SI at all, even when using the simulated HL sounds. HASPI did not predict the difference between the laboratory and remote experiments on male speech sounds and the individual SI values. GESI may provide a first step toward SI prediction for individual HI listeners whose HL is caused solely by peripheral dysfunction.
【5】 CDSD: Chinese Dysarthria Speech Database标题：CDSD：汉语构音障碍语音库链接：https://arxiv.org/abs/2310.15930
作者：Mengyi Sun,Ming Gao,Xinchen Kang,Shiru Wang,Jun Du,Dengfeng Yao,Su-Jing Wang备注：9 pages, 3 figures摘要：我们介绍了中国构音障碍语音数据库（CDSD）作为构音障碍研究的一个有价值的资源。该数据库包括来自24名患有构音障碍的参与者的语音数据。在这些参与者中，一人额外记录了10小时的演讲数据，而每个人记录了一小时，从而产生了34小时的演讲材料。为了适应不同认知水平的参与者，我们的文本库主要由AISHELL-1数据集的内容和中小学生的演讲组成。当与会者阅读这些文本时，他们必须使用移动终端或ZOOM F8 n多轨现场录音机来记录他们的发言。在本文中，我们阐明了数据收集和注释过程，并提出了一种建立构音障碍语音识别基线的方法。此外，我们进行了一个依赖于说话者的构音障碍语音识别实验，使用额外的10个小时的语音数据，从我们的参与者之一。我们的研究结果表明，通过广泛的数据驱动的模型训练，微调有限数量的特定个人数据在说话者依赖性构音障碍语音识别中产生了值得称赞的结果。然而，我们观察到不同构音障碍的扬声器之间的识别结果的显着变化。这些见解提供了有价值的参考点依赖于说话人构音障碍的语音识别。摘要：We present the Chinese Dysarthria Speech Database (CDSD) as a valuable resource for dysarthria research. This database comprises speech data from 24 participants with dysarthria. Among these participants, one recorded an additional 10 hours of speech data, while each recorded one hour, resulting in 34 hours of speech material. To accommodate participants with varying cognitive levels, our text pool primarily consists of content from the AISHELL-1 dataset and speeches by primary and secondary school students. When participants read these texts, they must use a mobile device or the ZOOM F8n multi-track field recorder to record their speeches. In this paper, we elucidate the data collection and annotation processes and present an approach for establishing a baseline for dysarthric speech recognition. Furthermore, we conducted a speaker-dependent dysarthric speech recognition experiment using an additional 10 hours of speech data from one of our participants. Our research findings indicate that, through extensive data-driven model training, fine-tuning limited quantities of specific individual data yields commendable results in speaker-dependent dysarthric speech recognition. However, we observe significant variations in recognition results among different dysarthric speakers. These insights provide valuable reference points for speaker-dependent dysarthric speech recognition.
【6】 How Much Context Does My Attention-Based ASR System Need?标题：我的基于注意力的ASR系统需要多少背景？链接：https://arxiv.org/abs/2310.15672
作者：Robert Flynn,Anton Ragni摘要：对于语音识别任务，在训练期间使用超过30秒的声学上下文是不常见的，并且在文献中研究不足。在这项工作中，我们研究的效果缩放序列长度用于训练/评估（密集注意力为基础的）声学和语言模型的语音识别性能。对于这些实验，使用了大约100，000个伪标记的Spotify播客的数据集，探索了5秒到1小时的上下文长度。对长格式数据集Earnings-22和Teddom的Zero-shot评估表明，使用大约80秒的声学背景进行训练是有益的，与有限的背景基线相比，相对提高了14.9%。此外，我们执行一个系统的组合与长上下文Transformer语言模型通过波束搜索一个完整的长上下文ASR系统，与当前的最先进的竞争力的结果。摘要：For the task of speech recognition, the use of more than 30 seconds of acoustic context during training is uncommon, and under-investigated in literature. In this work, we examine the effect of scaling the sequence length used to train/evaluate (dense-attention based) acoustic and language models on speech recognition performance. For these experiments a dataset of roughly 100,000 pseudo-labelled Spotify podcasts is used, with context lengths of 5 seconds to 1 hour being explored. Zero-shot evaluations on long-format datasets Earnings-22 and Tedlium demonstrate a benefit from training with around 80 seconds of acoustic context, showing up to a 14.9% relative improvement from a limited context baseline. Furthermore, we perform a system combination with long-context transformer language models via beam search for a fully long-context ASR system, with results that are competitive with the current state-of-the-art.
【7】 Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio  Models标题：动态卷积神经网络作为高效的预训练音频模型链接：https://arxiv.org/abs/2310.15648
作者：Florian Schmid,Khaled Koutini,Gerhard Widmer备注：Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing. Source Code available at: this https URL摘要：大规模音频数据集（如AudioSet）的引入为Transformers征服音频领域并取代CNN成为许多任务的最先进神经网络架构铺平了道路。Audio Spectrogram Transformers在利用大型数据集方面非常出色，可以创建功能强大的预训练模型，当对下游任务进行微调时，这些模型将超过CNN。然而，与CNN相比，当前流行的音频频谱图Transformers在计算复杂度方面要求较高。最近，我们已经证明，通过采用Transformer-to-CNN知识蒸馏，高效的CNN可以在大型数据集上赶上甚至超过Transformers。在这项工作中，我们扩展了这一研究领域，并通过引入动态CNN块来增加高效CNN的容量，这些CNN块由动态非线性、动态卷积和注意力机制构成。我们表明，在大规模AudioSet上的音频标记任务中，这些动态CNN在性能复杂度权衡和参数效率方面优于传统的高效CNN。我们的实验进一步表明，引入的动态CNN在下游任务上实现了更好的性能，并且可以很好地扩展，达到Transformer性能，甚至在AudioSet和几个下游任务上优于它们。摘要：The introduction of large-scale audio datasets, such as AudioSet, paved the way for Transformers to conquer the audio domain and replace CNNs as the state-of-the-art neural network architecture for many tasks. Audio Spectrogram Transformers are excellent at exploiting large datasets, creating powerful pre-trained models that surpass CNNs when fine-tuned on downstream tasks. However, current popular Audio Spectrogram Transformers are demanding in terms of computational complexity compared to CNNs. Recently, we have shown that, by employing Transformer-to-CNN Knowledge Distillation, efficient CNNs can catch up with and even outperform Transformers on large datasets. In this work, we extend this line of research and increase the capacity of efficient CNNs by introducing dynamic CNN blocks, constructed of dynamic non-linearities, dynamic convolutions and attention mechanisms. We show that these dynamic CNNs outperform traditional efficient CNNs, in terms of the performance-complexity trade-off and parameter efficiency, at the task of audio tagging on the large-scale AudioSet. Our experiments further indicate that the introduced dynamic CNNs achieve better performance on downstream tasks and scale up well, attaining Transformer performance and even outperforming them on AudioSet and several downstream tasks.
【8】 Modality Dropout for Multimodal Device Directed Speech Detection using  Verbal and Non-Verbal Features标题：基于语言和非语言特征的多通道设备导向语音检测的通道丢失链接：https://arxiv.org/abs/2310.15261
作者：Gautam Krishna,Sameer Dharur,Oggi Rudovic,Pranay Dighe,Saurabh Adya,Ahmed Hussen Abdelaziz,Ahmed H Tewfik备注：5 pages摘要：设备导向语音检测（DDSD）是区分针对语音助理的查询与侧面对话或背景语音的二进制分类任务。现有技术的DDSD系统使用口头提示（例如，声学、文本和/或自动语音识别系统（ASR）特征）来将语音分类为设备引导的或其他的，并且当部署在现实世界设置中时，通常必须应对这些模态中的一个或多个不可用。在本文中，我们研究DDSD系统的融合方案，可以更强大的丢失模态。同时，我们研究了非语言线索的使用，特别是韵律功能，除了言语线索DDSD。我们提出了不同的方法来结合分数和嵌入从韵律与相应的口头线索，发现韵律提高DDSD性能高达8.5%的错误接受率（FA）在一个给定的固定操作点通过非线性中间融合，而我们使用的模态丢弃技术提高了这些模型的性能7.4%在推理时间内使用缺失模态进行评估时，摘要：Device-directed speech detection (DDSD) is the binary classification task of distinguishing between queries directed at a voice assistant versus side conversation or background speech. State-of-the-art DDSD systems use verbal cues, e.g acoustic, text and/or automatic speech recognition system (ASR) features, to classify speech as device-directed or otherwise, and often have to contend with one or more of these modalities being unavailable when deployed in real-world settings. In this paper, we investigate fusion schemes for DDSD systems that can be made more robust to missing modalities. Concurrently, we study the use of non-verbal cues, specifically prosody features, in addition to verbal cues for DDSD. We present different approaches to combine scores and embeddings from prosody with the corresponding verbal cues, finding that prosody improves DDSD performance by upto 8.5% in terms of false acceptance rate (FA) at a given fixed operating point via non-linear intermediate fusion, while our use of modality dropout techniques improves the performance of these models by 7.4% in terms of FA when evaluated with missing modalities during inference time.
【9】 SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis标题：SyncFusion：多模式同步视频到音频Foley合成链接：https://arxiv.org/abs/2310.15247
作者：Marco Comunità,Riccardo F. Gramaccioni,Emilian Postolache,Emanuele Rodolà,Danilo Comminiello,Joshua D. Reiss摘要：声音设计涉及创造性地选择，录制和编辑各种媒体的声音效果，如电影，视频游戏和虚拟/增强现实。设计声音时最耗时的步骤之一是同步音频和视频。在某些情况下，可以使用视频拍摄的环境记录，这可以帮助这个过程。然而，在视频游戏和动画中，不存在参考音频，需要从视频手动注释事件定时。我们提出了一个从视频中提取重复动作起始的系统，然后将其与音频或文本嵌入结合使用，以调节经过训练的扩散模型，从而生成新的同步音效音轨。通过这种方式，我们将完全的创意控制权留给了声音设计师，同时消除了与视频同步的负担。此外，编辑起始音轨或改变条件嵌入需要比编辑音轨本身少得多的努力，从而简化发声过程。我们提供可靠的示例、源代码和预训练模型，以促进可重复性摘要：Sound design involves creatively selecting, recording, and editing sound effects for various media like cinema, video games, and virtual/augmented reality. One of the most time-consuming steps when designing sound is synchronizing audio with video. In some cases, environmental recordings from video shoots are available, which can aid in the process. However, in video games and animations, no reference audio exists, requiring manual annotation of event timings from the video. We propose a system to extract repetitive actions onsets from a video, which are then used - in conjunction with audio or textual embeddings - to condition a diffusion model trained to generate a new synchronized sound effects audio track. In this way, we leave complete creative control to the sound designer while removing the burden of synchronization with video. Furthermore, editing the onset track or changing the conditioning embedding requires much less effort than editing the audio track itself, simplifying the sonification process. We provide sound examples, source code, and pretrained models to faciliate reproducibility
机器翻译，仅供参考