今日论文合集：cs.SD语音8篇，eess.AS音频处理9篇。
cs.SD语音
【1】 SynVox2: Towards a privacy-friendly VoxCeleb2 dataset标题：SynVox2：迈向隐私友好的VoxCeleb2数据集链接：https://arxiv.org/abs/2309.06141
作者：Xiaoxiao Miao,Xin Wang,Erica Cooper,Junichi Yamagishi,Nicholas Evans,Massimiliano Todisco,Jean-François Bonastre,Mickael Rouvier备注：conference摘要：深度学习在说话人识别中的成功在很大程度上依赖于大型数据集的使用。然而，深度学习方法的数据饥渴性质已经受到质疑，因为在使用从真实人类说话者收集的大规模自然语音数据集时会出现伦理、隐私和法律问题。例如，用于说话人识别的广泛使用的VoxCeleb2数据集不再可以从官方网站访问。为了减轻这些担忧，这项工作提出了一项倡议，以生成一个隐私友好的合成VoxCeleb2数据集，确保所生成的语音在隐私，实用性和公平性方面的质量。我们还讨论了使用合成数据的说话人验证的下游任务的挑战。摘要：The success of deep learning in speaker recognition relies heavily on the use of large datasets. However, the data-hungry nature of deep learning methods has already being questioned on account the ethical, privacy, and legal concerns that arise when using large-scale datasets of natural speech collected from real human speakers. For example, the widely-used VoxCeleb2 dataset for speaker recognition is no longer accessible from the official website. To mitigate these concerns, this work presents an initiative to generate a privacy-friendly synthetic VoxCeleb2 dataset that ensures the quality of the generated speech in terms of privacy, utility, and fairness. We also discuss the challenges of using synthetic data for the downstream task of speaker verification.
【2】 CleanUNet 2: A Hybrid Speech Denoising Model on Waveform and Spectrogram标题：CleanUNet 2：一种基于波形和语谱图的混合语音去噪模型链接：https://arxiv.org/abs/2309.05975
作者：Zhifeng Kong,Wei Ping,Ambrish Dantrey,Bryan Catanzaro机构：‡UCSD, †NVIDIA备注：None摘要：在这项工作中，我们提出了CleanUNet 2，一个语音去噪模型，结合了波形去噪器和频谱图去噪器的优点，实现了两全其美。CleanUNet 2使用两阶段框架，灵感来自流行的语音合成方法，由波形模型和频谱图模型组成。具体来说，CleanUNet 2建立在CleanUNet（最先进的波形降噪器）的基础上，并通过将来自频谱图降噪器的预测频谱图作为输入来进一步提高其性能。我们证明了CleanUNet 2在各种客观和主观评价方面优于以前的方法。摘要：In this work, we present CleanUNet 2, a speech denoising model that combines the advantages of waveform denoiser and spectrogram denoiser and achieves the best of both worlds. CleanUNet 2 uses a two-stage framework inspired by popular speech synthesis methods that consist of a waveform model and a spectrogram model. Specifically, CleanUNet 2 builds upon CleanUNet, the state-of-the-art waveform denoiser, and further boosts its performance by taking predicted spectrograms from a spectrogram denoiser as the input. We demonstrate that CleanUNet 2 outperforms previous methods in terms of various objective and subjective evaluations.
【3】 Energy Preservation and Stability of Random Filterbanks标题：随机滤波器组的能量守恒与稳定性链接：https://arxiv.org/abs/2309.05855
作者：Daniel Haider,Vincent Lostanlen,Martin Ehler,Peter Balazs备注：4 pages, 5 figures, 1 page appendix摘要：是什么让基于波形的深度学习如此困难？尽管在训练卷积神经网络（convnets）用于滤波器组设计方面进行了许多尝试，但它们往往无法超越手工制作的基线。这更令人惊讶，因为这些基线是线性时不变系统：因此，它们的传递函数可以由具有大接收场的卷积网络精确地表示。在这篇文章中，我们从随机卷积算子的数学角度阐述了简单卷积网的统计特性。我们发现，随机高斯权重的FIR滤波器组是病态的大滤波器和本地周期性的输入信号，这两者都是典型的音频信号处理应用。此外，我们观察到，预期的能量保存的随机滤波器组是不够的数值稳定性，并推导出其预期的框架边界的理论界。摘要：What makes waveform-based deep learning so hard? Despite numerous attempts at training convolutional neural networks (convnets) for filterbank design, they often fail to outperform hand-crafted baselines. This is all the more surprising because these baselines are linear time-invariant systems: as such, their transfer functions could be accurately represented by a convnet with a large receptive field. In this article, we elaborate on the statistical properties of simple convnets from the mathematical perspective of random convolutional operators. We find that FIR filterbanks with random Gaussian weights are ill-conditioned for large filters and locally periodic input signals, which both are typical in audio signal processing applications. Furthermore, we observe that expected energy preservation of a random filterbank is not sufficient for numerical stability and derive theoretical bounds for its expected frame bounds.
【4】 Optimizing Audio Augmentations for Contrastive Learning of  Health-Related Acoustic Signals标题：用于健康相关声学信号对比学习的优化音频增强链接：https://arxiv.org/abs/2309.05843
作者：Louis Blankemeier,Sebastien Baur,Wei-Hung Weng,Jake Garrison,Yossi Matias,Shruthi Prabhakara,Diego Ardila,Zaid Nabulsi机构：Google Research, USA备注：7 pages, 2 pages appendix, 2 figures, 5 appendix tables摘要：健康相关的声学信号，诸如咳嗽和呼吸声，与医学诊断和连续健康监测相关。大多数现有的用于健康声学的机器学习方法都是针对特定任务进行训练和评估的，这限制了它们在各种医疗保健应用中的通用性。在本文中，我们利用自我监督学习框架，SimCLR与慢速NFNet骨干，健康声学的对比学习。针对该应用优化慢速NFNet的关键方面在于识别有效的音频增强。我们对各种音频增强策略进行了深入的分析，并证明了适当的增强策略可以提高慢速NFNet音频编码器在不同的健康声学任务集的性能。我们的研究结果表明，当增强剂结合起来时，它们可以产生协同效应，超过单独应用时看到的益处。摘要：Health-related acoustic signals, such as cough and breathing sounds, are relevant for medical diagnosis and continuous health monitoring. Most existing machine learning approaches for health acoustics are trained and evaluated on specific tasks, limiting their generalizability across various healthcare applications. In this paper, we leverage a self-supervised learning framework, SimCLR with a Slowfast NFNet backbone, for contrastive learning of health acoustics. A crucial aspect of optimizing Slowfast NFNet for this application lies in identifying effective audio augmentations. We conduct an in-depth analysis of various audio augmentation strategies and demonstrate that an appropriate augmentation strategy enhances the performance of the Slowfast NFNet audio encoder across a diverse set of health acoustic tasks. Our findings reveal that when augmentations are combined, they can produce synergistic effects that exceed the benefits seen when each is applied individually.
【5】 Natural Language Supervision for General-Purpose Audio Representations标题：用于通用音频表示的自然语言监督链接：https://arxiv.org/abs/2309.05767
作者：Benjamin Elizalde,Soham Deshmukh,Huaming Wang机构：Microsoft摘要：音频语言模型联合学习多模态文本和音频表示，使Zero-Shot推理。模型依赖于编码器来创建输入的强大表示，并将其推广到从声音、音乐和语音等多种任务。虽然模型已经取得了显着的性能，仍然有一个性能差距的任务特定模型。在本文中，我们提出了一个对比语言音频预训练模型，该模型使用两个创新的编码器进行Zero-Shot推理，使用4.6M音频文本对的各种集合进行预训练。为了学习音频表示，我们在22个音频任务上训练了音频编码器，而不是声音事件分类的标准训练。为了学习语言表示，我们训练了一个自回归解码器模型，而不是标准的编码器模型。然后，音频和语言表示被带入一个联合的多模态空间使用对比学习。我们使用我们的编码器，以提高下游性能的利润。我们广泛评估了我们的代表性的26个下游任务，在文献中最大的概括。我们的模型实现了最先进的结果，在几个任务导致的通用音频表示的方式。摘要：Audio-Language models jointly learn multimodal text and audio representations that enable Zero-Shot inference. Models rely on the encoders to create powerful representations of the input and generalize to multiple tasks ranging from sounds, music, and speech. Although models have achieved remarkable performance, there is still a performance gap with task-specific models. In this paper, we propose a Contrastive Language-Audio Pretraining model that is pretrained with a diverse collection of 4.6M audio-text pairs employing two innovative encoders for Zero-Shot inference. To learn audio representations, we trained an audio encoder on 22 audio tasks, instead of the standard training of sound event classification. To learn language representations, we trained an autoregressive decoder-only model instead of the standard encoder-only models. Then, the audio and language representations are brought into a joint multimodal space using Contrastive Learning. We used our encoders to improve the downstream performance by a margin. We extensively evaluated the generalization of our representations on 26 downstream tasks, the largest in the literature. Our model achieves state of the art results in several tasks leading the way towards general-purpose audio representations.
【6】 Assessing the Generalization Gap of Learning-Based Speech Enhancement  Systems in Noisy and Reverberant Environments标题：基于学习的语音增强系统在噪声和混响环境中的泛化差距评估链接：https://arxiv.org/abs/2309.06183
作者：Philippe Gonzalez,Tommy Sonne Alstrøm,Tobias May备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：噪声和混响语音混合的声学可变性受到多种因素的影响，例如目标说话人和干扰噪声的频谱-时间特性、信噪比（SNR）和房间特性。这种大的可变性对基于学习的语音增强系统提出了重大挑战，因为训练和测试条件之间的不匹配会大大降低系统的性能。通常通过用与训练期间使用的数据库不同的新语音、噪声或双耳房间脉冲响应（BRIR）数据库测试系统来评估对看不见的条件的泛化。然而，语音增强任务的难度可以在数据库之间改变，这可以实质上影响结果。本研究介绍了一个泛化评估框架，使用一个参考模型训练的测试条件下，这样它可以被用来作为一个代理的测试条件的难度。这允许解开的任务难度的变化的影响，从处理新的数据的效果，从而定义一个新的衡量泛化性能称为泛化差距。通过循环通过多个语音、噪声和BRIR数据库以交叉验证的方式重复该过程，以准确地估计泛化间隙。建议的框架被施加到评估的泛化潜力的前馈神经网络（FFNN），Conv-TasNet，DCCRN和方式。我们发现，对于所有的模型，性能下降最语音不匹配，而良好的噪声和房间的泛化可以通过在多个数据库上训练。此外，虽然最近的模型在匹配条件下表现出更高的性能，但它们的性能在失配条件下大幅下降，并且可能变得不如基于FFNN的系统。摘要：The acoustic variability of noisy and reverberant speech mixtures is influenced by multiple factors, such as the spectro-temporal characteristics of the target speaker and the interfering noise, the signal-to-noise ratio (SNR) and the room characteristics. This large variability poses a major challenge for learning-based speech enhancement systems, since a mismatch between the training and testing conditions can substantially reduce the performance of the system. Generalization to unseen conditions is typically assessed by testing the system with a new speech, noise or binaural room impulse response (BRIR) database different from the one used during training. However, the difficulty of the speech enhancement task can change across databases, which can substantially influence the results. The present study introduces a generalization assessment framework that uses a reference model trained on the test condition, such that it can be used as a proxy for the difficulty of the test condition. This allows to disentangle the effect of the change in task difficulty from the effect of dealing with new data, and thus to define a new measure of generalization performance termed the generalization gap. The procedure is repeated in a cross-validation fashion by cycling through multiple speech, noise, and BRIR databases to accurately estimate the generalization gap. The proposed framework is applied to evaluate the generalization potential of a feedforward neural network (FFNN), Conv-TasNet, DCCRN and MANNER. We find that for all models, the performance degrades the most in speech mismatches, while good noise and room generalization can be achieved by training on multiple databases. Moreover, while recent models show higher performance in matched conditions, their performance substantially decreases in mismatched conditions and can become inferior to that of the FFNN-based system.
【7】 Can large-scale vocoded spoofed data improve speech spoofing  countermeasure with a self-supervised front end?标题：大规模声码欺骗数据能否通过自我监督的前端提高语音欺骗对抗能力？链接：https://arxiv.org/abs/2309.06014
作者：Xin Wang,Junichi Yamagishi备注：submitted to icassp 2024. code on github: this https URL摘要：区分看不见的欺骗数据和真实数据的语音欺骗对策（CM）需要不同的训练数据。虽然许多数据集使用由语音合成系统生成的欺骗数据，但最近发现由神经声码器声码化的数据也有效地作为欺骗训练数据。由于许多神经声码器的构建和生成速度很快，因此本研究使用了多个神经声码器，并在VoxCeleb 2语料库的基础上创建了超过9,000小时的声码数据。本研究探讨了这种大规模的声码数据如何可以改善使用数据饥饿的自我监督学习（SSL）模型的欺骗对策。实验表明，多个测试集上的整体CM性能提高时，使用的SSL模型不断训练的声码数据提取的特征。当使用从连续训练之前和之后的两个SSL中提取的新SSL时，观察到进一步的改善。具有蒸馏SSL的CM在具有挑战性的看不见的测试集上表现优于以前的最佳模型，包括ASVspoof 2019逻辑访问，WaveFake和In-the-Wild。摘要：A speech spoofing countermeasure (CM) that discriminates between unseen spoofed and bona fide data requires diverse training data. While many datasets use spoofed data generated by speech synthesis systems, it was recently found that data vocoded by neural vocoders were also effective as the spoofed training data. Since many neural vocoders are fast in building and generation, this study used multiple neural vocoders and created more than 9,000 hours of vocoded data on the basis of the VoxCeleb2 corpus. This study investigates how this large-scale vocoded data can improve spoofing countermeasures that use data-hungry self-supervised learning (SSL) models. Experiments demonstrated that the overall CM performance on multiple test sets improved when using features extracted by an SSL model continually trained on the vocoded data. Further improvement was observed when using a new SSL distilled from the two SSLs before and after the continual training. The CM with the distilled SSL outperformed the previous best model on challenging unseen test sets, including the ASVspoof 2019 logical access, WaveFake, and In-the-Wild.
【8】 Smartwatch-derived Acoustic Markers for Deficits in Cognitively Relevant  Everyday Functioning标题：智能手表衍生的日常认知功能缺陷的声学标记链接：https://arxiv.org/abs/2309.05777
作者：Yasunori Yamada,Kaoru Shinkawa,Masatomo Kobayashi,Miyuki Nemoto,Miho Ota,Kiyotaka Nemoto,Tetsuaki Arai机构：Digital Health, IBM Research, Tokyo, Japan, Department of Psychiatry, Division of Clinical Medicine, Institute of Medicine, University of Tsukuba, Ibaraki, Japan备注：None摘要：检测由于认知障碍导致的日常功能的细微缺陷对于神经退行性疾病，特别是阿尔茨海默病的早期检测是重要的。然而，目前评估日常功能的标准是基于定性的主观评级。言语已被证明为认知障碍提供了良好的客观标记，但与认知相关的日常功能的关联仍然没有调查。在这项研究中，我们展示了使用基于智能手表的应用程序来收集声学特征作为检测日常功能缺陷的客观标记的可行性。我们收集了54名老年人在执行认知任务和日常对话时的语音数据，作为可能的应用场景，以及日常功能的测量。使用声学特征的机器学习模型可以检测出日常功能缺陷的个体，准确率高达77.8%，高于标准神经心理学测试的68.5%。我们还确定了常见的声学特征，用于在两种类型的语音数据（认知任务和日常对话）中强有力地区分日常功能的缺陷。我们的研究结果表明，从不同类型的语音数据中提取的共同的声学特征可以被用来作为标记的赤字在日常运作。摘要：Detection of subtle deficits in everyday functioning due to cognitive impairment is important for early detection of neurodegenerative diseases, particularly Alzheimer's disease. However, current standards for assessment of everyday functioning are based on qualitative, subjective ratings. Speech has been shown to provide good objective markers for cognitive impairments, but the association with cognition-relevant everyday functioning remains uninvestigated. In this study, we demonstrate the feasibility of using a smartwatch-based application to collect acoustic features as objective markers for detecting deficits in everyday functioning. We collected voice data during the performance of cognitive tasks and daily conversation, as possible application scenarios, from 54 older adults, along with a measure of everyday functioning. Machine learning models using acoustic features could detect individuals with deficits in everyday functioning with up to 77.8% accuracy, which was higher than the 68.5% accuracy with standard neuropsychological tests. We also identified common acoustic features for robustly discriminating deficits in everyday functioning across both types of voice data (cognitive tasks and daily conversation). Our results suggest that common acoustic features extracted from different types of voice data can be used as markers for deficits in everyday functioning.
eess.AS音频处理
【1】 Assessing the Generalization Gap of Learning-Based Speech Enhancement  Systems in Noisy and Reverberant Environments标题：基于学习的语音增强系统在噪声和混响环境中的泛化差距评估链接：https://arxiv.org/abs/2309.06183
作者：Philippe Gonzalez,Tommy Sonne Alstrøm,Tobias May备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：噪声和混响语音混合的声学可变性受到多种因素的影响，例如目标说话人和干扰噪声的频谱-时间特性、信噪比（SNR）和房间特性。这种大的可变性对基于学习的语音增强系统提出了重大挑战，因为训练和测试条件之间的不匹配会大大降低系统的性能。通常通过用与训练期间使用的数据库不同的新语音、噪声或双耳房间脉冲响应（BRIR）数据库测试系统来评估对看不见的条件的泛化。然而，语音增强任务的难度可以在数据库之间改变，这可以实质上影响结果。本研究介绍了一个泛化评估框架，使用一个参考模型训练的测试条件下，这样它可以被用来作为一个代理的测试条件的难度。这允许解开的任务难度的变化的影响，从处理新的数据的效果，从而定义一个新的衡量泛化性能称为泛化差距。通过循环通过多个语音、噪声和BRIR数据库以交叉验证的方式重复该过程，以准确地估计泛化间隙。建议的框架被施加到评估的泛化潜力的前馈神经网络（FFNN），Conv-TasNet，DCCRN和方式。我们发现，对于所有的模型，性能下降最语音不匹配，而良好的噪声和房间的泛化可以通过在多个数据库上训练。此外，虽然最近的模型在匹配条件下表现出更高的性能，但它们的性能在失配条件下大幅下降，并且可能变得不如基于FFNN的系统。摘要：The acoustic variability of noisy and reverberant speech mixtures is influenced by multiple factors, such as the spectro-temporal characteristics of the target speaker and the interfering noise, the signal-to-noise ratio (SNR) and the room characteristics. This large variability poses a major challenge for learning-based speech enhancement systems, since a mismatch between the training and testing conditions can substantially reduce the performance of the system. Generalization to unseen conditions is typically assessed by testing the system with a new speech, noise or binaural room impulse response (BRIR) database different from the one used during training. However, the difficulty of the speech enhancement task can change across databases, which can substantially influence the results. The present study introduces a generalization assessment framework that uses a reference model trained on the test condition, such that it can be used as a proxy for the difficulty of the test condition. This allows to disentangle the effect of the change in task difficulty from the effect of dealing with new data, and thus to define a new measure of generalization performance termed the generalization gap. The procedure is repeated in a cross-validation fashion by cycling through multiple speech, noise, and BRIR databases to accurately estimate the generalization gap. The proposed framework is applied to evaluate the generalization potential of a feedforward neural network (FFNN), Conv-TasNet, DCCRN and MANNER. We find that for all models, the performance degrades the most in speech mismatches, while good noise and room generalization can be achieved by training on multiple databases. Moreover, while recent models show higher performance in matched conditions, their performance substantially decreases in mismatched conditions and can become inferior to that of the FFNN-based system.
【2】 iPhonMatchNet: Zero-Shot User-Defined Keyword Spotting Using Implicit  Acoustic Echo Cancellation标题：IPhonMatchNet：使用隐式声学回波消除的零射用户定义关键字检测链接：https://arxiv.org/abs/2309.06096
作者：Yong-Hyeok Lee,Namhyun Cho机构：Speech AI Lab., NCSOFT Corporation, South Korea备注：Submitted to ICASSP 2024摘要：在响应越来越多的兴趣在人机通信在各个领域，本文介绍了一种新的方法，称为iPhonMatchNet，它解决了在场景中，其中用户语音与设备播放音频重叠，从而创建一个自引用的问题，闯入的挑战。该模型利用隐式声学回声消除（iAEC）技术，以提高用户定义的关键字定位模型的效率，实现了显着的平均绝对误差减少95%，模型大小（0.13%）的最小增加相比，基线模型，PhonMatchNet。我们还提出了一个有效的模型结构，并展示了它的能力，学习iAEC功能，而不需要一个干净的信号。我们的研究结果表明，所提出的模型在现实世界的部署条件下的智能设备实现竞争力的表现。摘要：In response to the increasing interest in human--machine communication across various domains, this paper introduces a novel approach called iPhonMatchNet, which addresses the challenge of barge-in scenarios, wherein user speech overlaps with device playback audio, thereby creating a self-referencing problem. The proposed model leverages implicit acoustic echo cancellation (iAEC) techniques to increase the efficiency of user-defined keyword spotting models, achieving a remarkable 95% reduction in mean absolute error with a minimal increase in model size (0.13%) compared to the baseline model, PhonMatchNet. We also present an efficient model structure and demonstrate its capability to learn iAEC functionality without requiring a clean signal. The findings of our study indicate that the proposed model achieves competitive performance in real-world deployment conditions of smart devices.
【3】 Can large-scale vocoded spoofed data improve speech spoofing  countermeasure with a self-supervised front end?标题：大规模声码欺骗数据能否通过自我监督的前端提高语音欺骗对抗能力？链接：https://arxiv.org/abs/2309.06014
作者：Xin Wang,Junichi Yamagishi备注：submitted to icassp 2024. code on github: this https URL摘要：区分看不见的欺骗数据和真实数据的语音欺骗对策（CM）需要不同的训练数据。虽然许多数据集使用由语音合成系统生成的欺骗数据，但最近发现由神经声码器声码化的数据也有效地作为欺骗训练数据。由于许多神经声码器的构建和生成速度很快，因此本研究使用了多个神经声码器，并在VoxCeleb 2语料库的基础上创建了超过9,000小时的声码数据。本研究探讨了这种大规模的声码数据如何可以改善使用数据饥饿的自我监督学习（SSL）模型的欺骗对策。实验表明，多个测试集上的整体CM性能提高时，使用的SSL模型不断训练的声码数据提取的特征。当使用从连续训练之前和之后的两个SSL中提取的新SSL时，观察到进一步的改善。具有蒸馏SSL的CM在具有挑战性的看不见的测试集上表现优于以前的最佳模型，包括ASVspoof 2019逻辑访问，WaveFake和In-the-Wild。摘要：A speech spoofing countermeasure (CM) that discriminates between unseen spoofed and bona fide data requires diverse training data. While many datasets use spoofed data generated by speech synthesis systems, it was recently found that data vocoded by neural vocoders were also effective as the spoofed training data. Since many neural vocoders are fast in building and generation, this study used multiple neural vocoders and created more than 9,000 hours of vocoded data on the basis of the VoxCeleb2 corpus. This study investigates how this large-scale vocoded data can improve spoofing countermeasures that use data-hungry self-supervised learning (SSL) models. Experiments demonstrated that the overall CM performance on multiple test sets improved when using features extracted by an SSL model continually trained on the vocoded data. Further improvement was observed when using a new SSL distilled from the two SSLs before and after the continual training. The CM with the distilled SSL outperformed the previous best model on challenging unseen test sets, including the ASVspoof 2019 logical access, WaveFake, and In-the-Wild.
【4】 Smartwatch-derived Acoustic Markers for Deficits in Cognitively Relevant  Everyday Functioning标题：智能手表衍生的日常认知功能缺陷的声学标记链接：https://arxiv.org/abs/2309.05777
作者：Yasunori Yamada,Kaoru Shinkawa,Masatomo Kobayashi,Miyuki Nemoto,Miho Ota,Kiyotaka Nemoto,Tetsuaki Arai机构：Digital Health, IBM Research, Tokyo, Japan, Department of Psychiatry, Division of Clinical Medicine, Institute of Medicine, University of Tsukuba, Ibaraki, Japan备注：None摘要：检测由于认知障碍导致的日常功能的细微缺陷对于神经退行性疾病，特别是阿尔茨海默病的早期检测是重要的。然而，目前评估日常功能的标准是基于定性的主观评级。言语已被证明为认知障碍提供了良好的客观标记，但与认知相关的日常功能的关联仍然没有调查。在这项研究中，我们展示了使用基于智能手表的应用程序来收集声学特征作为检测日常功能缺陷的客观标记的可行性。我们收集了54名老年人在执行认知任务和日常对话时的语音数据，作为可能的应用场景，以及日常功能的测量。使用声学特征的机器学习模型可以检测出日常功能缺陷的个体，准确率高达77.8%，高于标准神经心理学测试的68.5%。我们还确定了常见的声学特征，用于在两种类型的语音数据（认知任务和日常对话）中强有力地区分日常功能的缺陷。我们的研究结果表明，从不同类型的语音数据中提取的共同的声学特征可以被用来作为标记的赤字在日常运作。摘要：Detection of subtle deficits in everyday functioning due to cognitive impairment is important for early detection of neurodegenerative diseases, particularly Alzheimer's disease. However, current standards for assessment of everyday functioning are based on qualitative, subjective ratings. Speech has been shown to provide good objective markers for cognitive impairments, but the association with cognition-relevant everyday functioning remains uninvestigated. In this study, we demonstrate the feasibility of using a smartwatch-based application to collect acoustic features as objective markers for detecting deficits in everyday functioning. We collected voice data during the performance of cognitive tasks and daily conversation, as possible application scenarios, from 54 older adults, along with a measure of everyday functioning. Machine learning models using acoustic features could detect individuals with deficits in everyday functioning with up to 77.8% accuracy, which was higher than the 68.5% accuracy with standard neuropsychological tests. We also identified common acoustic features for robustly discriminating deficits in everyday functioning across both types of voice data (cognitive tasks and daily conversation). Our results suggest that common acoustic features extracted from different types of voice data can be used as markers for deficits in everyday functioning.
【5】 SynVox2: Towards a privacy-friendly VoxCeleb2 dataset标题：SynVox2：迈向隐私友好的VoxCeleb2数据集链接：https://arxiv.org/abs/2309.06141
作者：Xiaoxiao Miao,Xin Wang,Erica Cooper,Junichi Yamagishi,Nicholas Evans,Massimiliano Todisco,Jean-François Bonastre,Mickael Rouvier备注：conference摘要：深度学习在说话人识别中的成功在很大程度上依赖于大型数据集的使用。然而，深度学习方法的数据饥渴性质已经受到质疑，因为在使用从真实人类说话者收集的大规模自然语音数据集时会出现伦理、隐私和法律问题。例如，用于说话人识别的广泛使用的VoxCeleb2数据集不再可以从官方网站访问。为了减轻这些担忧，这项工作提出了一项倡议，以生成一个隐私友好的合成VoxCeleb2数据集，确保所生成的语音在隐私，实用性和公平性方面的质量。我们还讨论了使用合成数据的说话人验证的下游任务的挑战。摘要：The success of deep learning in speaker recognition relies heavily on the use of large datasets. However, the data-hungry nature of deep learning methods has already being questioned on account the ethical, privacy, and legal concerns that arise when using large-scale datasets of natural speech collected from real human speakers. For example, the widely-used VoxCeleb2 dataset for speaker recognition is no longer accessible from the official website. To mitigate these concerns, this work presents an initiative to generate a privacy-friendly synthetic VoxCeleb2 dataset that ensures the quality of the generated speech in terms of privacy, utility, and fairness. We also discuss the challenges of using synthetic data for the downstream task of speaker verification.
【6】 CleanUNet 2: A Hybrid Speech Denoising Model on Waveform and Spectrogram标题：CleanUNet 2：一种基于波形和语谱图的混合语音去噪模型链接：https://arxiv.org/abs/2309.05975
作者：Zhifeng Kong,Wei Ping,Ambrish Dantrey,Bryan Catanzaro机构：‡UCSD, †NVIDIA备注：None摘要：在这项工作中，我们提出了CleanUNet 2，一个语音去噪模型，结合了波形去噪器和频谱图去噪器的优点，实现了两全其美。CleanUNet 2使用两阶段框架，灵感来自流行的语音合成方法，由波形模型和频谱图模型组成。具体来说，CleanUNet 2建立在CleanUNet（最先进的波形降噪器）的基础上，并通过将来自频谱图降噪器的预测频谱图作为输入来进一步提高其性能。我们证明了CleanUNet 2在各种客观和主观评价方面优于以前的方法。摘要：In this work, we present CleanUNet 2, a speech denoising model that combines the advantages of waveform denoiser and spectrogram denoiser and achieves the best of both worlds. CleanUNet 2 uses a two-stage framework inspired by popular speech synthesis methods that consist of a waveform model and a spectrogram model. Specifically, CleanUNet 2 builds upon CleanUNet, the state-of-the-art waveform denoiser, and further boosts its performance by taking predicted spectrograms from a spectrogram denoiser as the input. We demonstrate that CleanUNet 2 outperforms previous methods in terms of various objective and subjective evaluations.
【7】 Energy Preservation and Stability of Random Filterbanks标题：随机滤波器组的能量守恒与稳定性链接：https://arxiv.org/abs/2309.05855
作者：Daniel Haider,Vincent Lostanlen,Martin Ehler,Peter Balazs备注：4 pages, 5 figures, 1 page appendix摘要：是什么让基于波形的深度学习如此困难？尽管在训练卷积神经网络（convnets）用于滤波器组设计方面进行了许多尝试，但它们往往无法超越手工制作的基线。这更令人惊讶，因为这些基线是线性时不变系统：因此，它们的传递函数可以由具有大接收场的卷积网络精确地表示。在这篇文章中，我们从随机卷积算子的数学角度阐述了简单卷积网的统计特性。我们发现，随机高斯权重的FIR滤波器组是病态的大滤波器和本地周期性的输入信号，这两者都是典型的音频信号处理应用。此外，我们观察到，预期的能量保存的随机滤波器组是不够的数值稳定性，并推导出其预期的框架边界的理论界。摘要：What makes waveform-based deep learning so hard? Despite numerous attempts at training convolutional neural networks (convnets) for filterbank design, they often fail to outperform hand-crafted baselines. This is all the more surprising because these baselines are linear time-invariant systems: as such, their transfer functions could be accurately represented by a convnet with a large receptive field. In this article, we elaborate on the statistical properties of simple convnets from the mathematical perspective of random convolutional operators. We find that FIR filterbanks with random Gaussian weights are ill-conditioned for large filters and locally periodic input signals, which both are typical in audio signal processing applications. Furthermore, we observe that expected energy preservation of a random filterbank is not sufficient for numerical stability and derive theoretical bounds for its expected frame bounds.
【8】 Optimizing Audio Augmentations for Contrastive Learning of  Health-Related Acoustic Signals标题：用于健康相关声学信号对比学习的优化音频增强链接：https://arxiv.org/abs/2309.05843
作者：Louis Blankemeier,Sebastien Baur,Wei-Hung Weng,Jake Garrison,Yossi Matias,Shruthi Prabhakara,Diego Ardila,Zaid Nabulsi机构：Google Research, USA备注：7 pages, 2 pages appendix, 2 figures, 5 appendix tables摘要：健康相关的声学信号，诸如咳嗽和呼吸声，与医学诊断和连续健康监测相关。大多数现有的用于健康声学的机器学习方法都是针对特定任务进行训练和评估的，这限制了它们在各种医疗保健应用中的通用性。在本文中，我们利用自我监督学习框架，SimCLR与慢速NFNet骨干，健康声学的对比学习。针对该应用优化慢速NFNet的关键方面在于识别有效的音频增强。我们对各种音频增强策略进行了深入的分析，并证明了适当的增强策略可以提高慢速NFNet音频编码器在不同的健康声学任务集的性能。我们的研究结果表明，当增强剂结合起来时，它们可以产生协同效应，超过单独应用时看到的益处。摘要：Health-related acoustic signals, such as cough and breathing sounds, are relevant for medical diagnosis and continuous health monitoring. Most existing machine learning approaches for health acoustics are trained and evaluated on specific tasks, limiting their generalizability across various healthcare applications. In this paper, we leverage a self-supervised learning framework, SimCLR with a Slowfast NFNet backbone, for contrastive learning of health acoustics. A crucial aspect of optimizing Slowfast NFNet for this application lies in identifying effective audio augmentations. We conduct an in-depth analysis of various audio augmentation strategies and demonstrate that an appropriate augmentation strategy enhances the performance of the Slowfast NFNet audio encoder across a diverse set of health acoustic tasks. Our findings reveal that when augmentations are combined, they can produce synergistic effects that exceed the benefits seen when each is applied individually.
【9】 Natural Language Supervision for General-Purpose Audio Representations标题：用于通用音频表示的自然语言监督链接：https://arxiv.org/abs/2309.05767
作者：Benjamin Elizalde,Soham Deshmukh,Huaming Wang机构：Microsoft摘要：音频语言模型联合学习多模态文本和音频表示，使Zero-Shot推理。模型依赖于编码器来创建输入的强大表示，并将其推广到从声音、音乐和语音等多种任务。虽然模型已经取得了显着的性能，仍然有一个性能差距的任务特定模型。在本文中，我们提出了一个对比语言音频预训练模型，该模型使用两个创新的编码器进行Zero-Shot推理，使用4.6M音频文本对的各种集合进行预训练。为了学习音频表示，我们在22个音频任务上训练了音频编码器，而不是声音事件分类的标准训练。为了学习语言表示，我们训练了一个自回归解码器模型，而不是标准的编码器模型。然后，音频和语言表示被带入一个联合的多模态空间使用对比学习。我们使用我们的编码器，以提高下游性能的利润。我们广泛评估了我们的代表性的26个下游任务，在文献中最大的概括。我们的模型实现了最先进的结果，在几个任务导致的通用音频表示的方式。摘要：Audio-Language models jointly learn multimodal text and audio representations that enable Zero-Shot inference. Models rely on the encoders to create powerful representations of the input and generalize to multiple tasks ranging from sounds, music, and speech. Although models have achieved remarkable performance, there is still a performance gap with task-specific models. In this paper, we propose a Contrastive Language-Audio Pretraining model that is pretrained with a diverse collection of 4.6M audio-text pairs employing two innovative encoders for Zero-Shot inference. To learn audio representations, we trained an audio encoder on 22 audio tasks, instead of the standard training of sound event classification. To learn language representations, we trained an autoregressive decoder-only model instead of the standard encoder-only models. Then, the audio and language representations are brought into a joint multimodal space using Contrastive Learning. We used our encoders to improve the downstream performance by a margin. We extensively evaluated the generalization of our representations on 26 downstream tasks, the largest in the literature. Our model achieves state of the art results in several tasks leading the way towards general-purpose audio representations.
机器翻译，仅供参考