今日论文合集：cs.SD语音9篇，eess.AS音频处理9篇。
cs.SD语音
【1】CED: Consistent ensemble distillation for audio tagging
标题：CED：用于音频标记的一致集成蒸馏
链接：http://arxiv.org/pdf/2308.11957v1
作者：Heinrich Dinkel,Yongqing Wang,Zhiyong Yan,Junbo Zhang,Yujun Wang机构：Xiaomi Corporation, Beijing, China摘要：增强和知识蒸馏（KD）是音频分类任务领域中采用的成熟技术，旨在提高性能并减小广泛认可的Audioset（AS）基准上的模型大小。虽然这两种技术单独使用都很有效，但它们的结合使用，称为一致性教学，以前还没有被探索过。本文提出了CED，一个简单的培训框架，从大型教师合奏与一致的教学中提炼出学生模型。为了实现这一点，CED有效地将logits以及增强方法存储在磁盘上，使其可扩展到大规模数据集。CED的功效的核心是其无标签性质，这意味着仅存储的logits用于学生模型的优化，仅需要0.3 %的额外磁盘空间用于AS。该研究训练了各种基于变压器的模型，包括在AS上实现49.0平均平均精度（mAP）的10M参数模型。预训练模型和代码可在https: github.com RicherMans CED上获得。摘要：Augmentation and knowledge distillation (KD) are well-established techniques employed in the realm of audio classification tasks, aimed at enhancing performance and reducing model sizes on the widely recognized Audioset (AS) benchmark. Although both techniques are effective individually, their combined use, called consistent teaching, hasn't been explored before. This paper proposes CED, a simple training framework that distils student models from large teacher ensembles with consistent teaching. To achieve this, CED efficiently stores logits as well as the augmentation methods on disk, making it scalable to large-scale datasets. Central to CED's efficacy is its label-free nature, meaning that only the stored logits are used for the optimization of a student model only requiring 0.3 % additional disk space for AS. The study trains various transformer-based models, including a 10M parameter model achieving a 49.0 mean average precision (mAP) on AS. Pretrained models and code are available at https: github.com RicherMans CED.
【2】Audio Generation with Multiple Conditional Diffusion Model标题：基于多条件扩散模型的音频生成链接：http://arxiv.org/pdf/2308.11940v1
作者：Zhifang Guo,Jianguo Mao,Rui Tao,Long Yan,Kazushige Ouchi,Hong Liu,Xiangdong Wang机构：Bejing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China, Toshiba China R&D Center, Beijing, China备注：Submitted to AAAI 2024摘要：基于文本的音频生成模型具有局限性，因为它们不能包含音频中的所有信息，从而导致当仅依赖于文本时受限的可控性。为了解决这个问题，我们提出了一种新的模型，该模型提高了现有的预训练文本到音频模型的可控性，通过纳入额外的条件，包括内容（时间戳）和风格（音高轮廓和能量轮廓）作为补充的文本。该方法实现了对所生成的音频的时间顺序、音高和能量的细粒度控制。为了保持生成的多样性，我们采用了一个可训练的控制条件编码器，该编码器由一个大的语言模型和一个可训练的融合网来编码和融合额外的条件，同时保持预训练的文本到音频模型的权重冻结。由于缺乏合适的数据集和评估指标，我们将现有的数据集合并为一个新的数据集，包括音频和相应的条件，并使用一系列的评估指标来评估可控性性能。实验结果表明，我们的模型成功地实现了细粒度控制，以实现可控的音频生成。音频样本和我们的数据集可在https: conditionaudiogen.github.io conditionaudiogen 上公开获取摘要：Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing datasets into a new dataset comprising the audio and corresponding conditions and use a series of evaluation metrics to evaluate the controllability performance. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation. Audio samples and our dataset are publicly available at https: conditionaudiogen.github.io conditionaudiogen
【3】Complex-valued neural networks for voice anti-spoofing
标题：用于语音反欺骗的复值神经网络
链接：http://arxiv.org/pdf/2308.11800v1
作者：Nicolas M. Müller,Philip Sperl,Konstantin Böttinger机构：Fraunhofer AISEC备注：Interspeech 2023摘要：当前的反欺骗和音频深度伪造检测系统使用基于幅度谱图的特征（例如CQT或Melspectrograms）或通过卷积或sinc-layers处理的原始音频。这两种方法都有缺点：幅度谱图丢弃影响音频自然度的相位信息，并且基于原始特征的模型不能使用传统的可解释AI方法。本文提出了一种新的方法，结合了这两种方法的好处，通过使用复值神经网络处理的复值，CQT频域表示的输入音频。该方法保留相位信息，并允许可解释的AI方法。结果表明，这种方法优于以前的方法上的“在野外”反欺骗数据集，并使解释的结果，通过可解释的AI。消融研究证实，模型已经学会使用相位信息来检测语音欺骗。摘要：Current anti-spoofing and audio deepfake detection systems use either magnitude spectrogram-based features (such as CQT or Melspectrograms) or raw audio processed through convolution or sinc-layers. Both methods have drawbacks: magnitude spectrograms discard phase information, which affects audio naturalness, and raw-feature-based models cannot use traditional explainable AI methods. This paper proposes a new approach that combines the benefits of both methods by using complex-valued neural networks to process the complex-valued, CQT frequency-domain representation of the input audio. This method retains phase information and allows for explainable AI methods. Results show that this approach outperforms previous methods on the "In-the-Wild" anti-spoofing dataset and enables interpretation of the results through explainable AI. Ablation studies confirm that the model has learned to use phase information to detect voice spoofing.
【4】Identifying depression-related topics in smartphone-collected free-response speech recordings using an automatic speech recognition system and a deep learning topic model标题：使用自动语音识别系统和深度学习主题模型在智能手机收集的自由反应语音记录中识别与抑郁相关的主题链接：http://arxiv.org/pdf/2308.11773v1
作者：Yuezhou Zhang,Amos A Folarin,Judith Dineley,Pauline Conde,Valeria de Angel,Shaoxiong Sun,Yatharth Ranjan,Zulqarnain Rashid,Callum Stewart,Petroula Laiou,Heet Sankesara,Linglong Qian,Faith Matcham,Katie M White,Carolin Oetzmann,Femke Lamers,Sara Siddi,Sara Simblett,Björn W. Schuller,Srinivasan Vairavan,Til Wykes,Josep Maria Haro,Brenda WJH Penninx,Vaibhav A Narayan,Matthew Hotopf,Richard JB Dobson,Nicholas Cummins,RADAR-CNS consortium机构：King's College London, London, UK, South London and Maudsley NHS Foundation Trust, London, UK, Health Data Research UK London, University College London, London, UK, University of Augsburg, Augsburg, Germany摘要：语言使用已被证明与抑郁症相关，但需要大规模的验证。传统的方法如临床研究是昂贵的。因此，自然语言处理已经被用于社交媒体来预测抑郁症，但其局限性仍然存在-缺乏有效的标签，有偏见的用户样本，以及没有上下文。我们的研究确定了29个主题的3919智能手机收集的语音录音从265名参与者使用Whisper工具和BERTopic模型。中位PHQ-8大于或等于10的6个主题被视为抑郁症的风险主题：没有期望，睡眠，心理治疗，理发，学习和课程。为了阐明主题的出现和与抑郁症的关联，我们比较了已确定主题的行为（从可穿戴设备）和语言特征。话题转移和抑郁症严重程度随时间的变化之间的相关性也进行了调查，表明纵向监测语言使用的重要性。我们还在一个类似的较小数据集（来自57名参与者的356个语音记录）上测试了BERTopic模型，获得了一些一致的结果。总之，我们的研究结果表明，特定的言语主题可能指示抑郁症的严重程度。所提出的数据驱动的工作流提供了一种实用的方法来收集和分析来自真实世界的数字健康研究设置的大规模语音数据。摘要：Language use has been shown to correlate with depression, but large-scale validation is needed. Traditional methods like clinic studies are expensive. So, natural language processing has been employed on social media to predict depression, but limitations remain-lack of validated labels, biased user samples, and no context. Our study identified 29 topics in 3919 smartphone-collected speech recordings from 265 participants using the Whisper tool and BERTopic model. Six topics with a median PHQ-8 greater than or equal to 10 were regarded as risk topics for depression: No Expectations, Sleep, Mental Therapy, Haircut, Studying, and Coursework. To elucidate the topic emergence and associations with depression, we compared behavioral (from wearables) and linguistic characteristics across identified topics. The correlation between topic shifts and changes in depression severity over time was also investigated, indicating the importance of longitudinally monitoring language use. We also tested the BERTopic model on a similar smaller dataset (356 speech recordings from 57 participants), obtaining some consistent results. In summary, our findings demonstrate specific speech topics may indicate depression severity. The presented data-driven workflow provides a practical approach to collecting and analyzing large-scale speech data from real-world settings for digital health research.
【5】Analysis of XLS-R for Speech Quality Assessment
标题：用于语音质量评估的XLS-R分析
链接：http://arxiv.org/pdf/2308.12077v1
作者：Bastiaan Tamm,Rik Vandenberghe,Hugo Van hamme机构：Laboratory for Cognitive Neurology (LCN), Department of Neurosciences, KU Leuven, Belgium, Processing Speech and Images (PSI), Department of Electrical Engineering, KU Leuven, Belgium备注：5 pages, submitted to WASPAA 2023摘要：在在线会议应用中，估计音频信号的感知质量对于确保最终用户的高质量体验至关重要。评估语音信号质量的最可靠方法是通过平均意见得分（MOS）度量形式的人类判断。然而，这种方法是劳动密集型的，并且对于大规模应用是不可行的。因此，重点已经转向通过深度神经网络的端到端训练进行自动语音质量评估。最近，研究表明，利用预训练的基于wav 2 vec的XLS-R嵌入可以为语音质量预测任务带来最先进的性能。在本文中，我们对预训练模型进行了深入分析。首先，我们分析了从XLS-R的每一层提取的嵌入的性能，以及模型的每个大小（300 M，1B，2B参数）。令人惊讶的是，我们发现了两个用于特征提取的最佳区域：一个在低级特征中，一个在高级特征中。接下来，我们调查两个不同的最优的原因。我们假设，较低级别的功能捕捉噪声和室内声学特性，而高级别的功能集中在语音内容和清晰度。为了调查这一点，我们分析了MOS预测的敏感性，就不同程度的腐败在每个类别。之后，我们尝试融合两个最佳特征深度，以确定它们是否包含用于MOS预测的互补信息。最后，我们比较了所提出的模型的性能，并评估了模型在未知数据集上的泛化能力。摘要：In online conferencing applications, estimating the perceived quality of an audio signal is crucial to ensure high quality of experience for the end user. The most reliable way to assess the quality of a speech signal is through human judgments in the form of the mean opinion score (MOS) metric. However, such an approach is labor intensive and not feasible for large-scale applications. The focus has therefore shifted towards automated speech quality assessment through end-to-end training of deep neural networks. Recently, it was shown that leveraging pre-trained wav2vec-based XLS-R embeddings leads to state-of-the-art performance for the task of speech quality prediction. In this paper, we perform an in-depth analysis of the pre-trained model. First, we analyze the performance of embeddings extracted from each layer of XLS-R and also for each size of the model (300M, 1B, 2B parameters). Surprisingly, we find two optimal regions for feature extraction: one in the lower-level features and one in the high-level features. Next, we investigate the reason for the two distinct optima. We hypothesize that the lower-level features capture characteristics of noise and room acoustics, whereas the high-level features focus on speech content and intelligibility. To investigate this, we analyze the sensitivity of the MOS predictions with respect to different levels of corruption in each category. Afterwards, we try fusing the two optimal feature depths to determine if they contain complementary information for MOS prediction. Finally, we compare the performance of the proposed models and assess the generalizability of the models on unseen datasets.
【6】Joint Prediction of Audio Event and Annoyance Rating in an Urban Soundscape by Hierarchical Graph Representation Learning
标题：基于层次图表示学习的城市声环境中音频事件和烦扰等级联合预测
链接：http://arxiv.org/pdf/2308.11980v1
作者：Yuanbo Hou,Siyang Song,Cheng Luo,Andrew Mitchell,Qiaoqiao Ren,Weicheng Xie,Jian Kang,Wenwu Wang,Dick Botteldooren机构：WAVES Research Group, Ghent University, Belgium. ,University of Leicester, UK., Shenzhen University, China. ,University College London, UK., AIRO-IDlab, Ghent University-Imec, Belgium. ,University of Surrey, UK.备注：INTERSPEECH 2023, Code and models: this https URL摘要：日常生活中的声音事件承载着丰富的客观世界信息。这些声音的构成影响了声景中人们的情绪。大多数先前的方法仅关注于对音频事件和场景进行分类和检测，但是可能忽略它们的感知质量，这可能影响人类对环境的收听情绪，例如，音频事件和场景。烦恼为此，本文提出了一种新的分层图表示学习（HGRL）的方法，该方法将客观的音频事件（AE）与人类感知的声景的主观烦恼评级（AR）联系起来。分层图由具有单类事件语义的细粒度事件（fAE）嵌入、具有多类事件语义的粗粒度事件（cAE）嵌入和AR嵌入组成。实验结果表明，所提出的HGRL成功地集成了AE与AR的AEC和ARP任务，同时协调cAE和fAE之间的关系，并进一步对齐的两个不同颗粒的AE信息与AR。摘要：Sound events in daily life carry rich information about the objective world. The composition of these sounds affects the mood of people in a soundscape. Most previous approaches only focus on classifying and detecting audio events and scenes, but may ignore their perceptual quality that may impact humans' listening mood for the environment, e.g. annoyance. To this end, this paper proposes a novel hierarchical graph representation learning (HGRL) approach which links objective audio events (AE) with subjective annoyance ratings (AR) of the soundscape perceived by humans. The hierarchical graph consists of fine-grained event (fAE) embeddings with single-class event semantics, coarse-grained event (cAE) embeddings with multi-class event semantics, and AR embeddings. Experiments show the proposed HGRL successfully integrates AE with AR for AEC and ARP tasks, while coordinating the relations between cAE and fAE and further aligning the two different grains of AE information with the AR.
【7】Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement
标题：利用相似-差异解缠的音频差异字幕
链接：http://arxiv.org/pdf/2308.11923v1
作者：Daiki Takeuchi,Yasunori Ohishi,Daisuke Niizumi,Noboru Harada,Kunio Kashino机构：NTT Corporation, Japan备注：Accepted to DCASE2023 Workshop摘要：我们提出了音频差异字幕（ADC）作为一个新的扩展任务的音频字幕描述相似，但略有不同的音频片段的输入对之间的语义差异。ADC解决了传统的音频字幕有时会为类似的音频片段生成类似的字幕，无法描述内容差异的问题。我们还提出了一个交叉注意力集中的Transformer编码器提取差异，通过比较一对音频剪辑和相似性-差异解纠缠强调潜在空间中的差异。为了评估所提出的方法，我们建立了一个AudioDiffCaps数据集，该数据集由两对相似但略有不同的音频剪辑组成，并带有人类注释的差异描述。AudioDiffCaps数据集的实验表明，所提出的方法有效地解决了ADC任务，并提高了注意力权重，通过在Transformer编码器中可视化它们来提取差异。摘要：We proposed Audio Difference Captioning (ADC) as a new extension task of audio captioning for describing the semantic differences between input pairs of similar but slightly different audio clips. The ADC solves the problem that conventional audio captioning sometimes generates similar captions for similar audio clips, failing to describe the difference in content. We also propose a cross-attention-concentrated transformer encoder to extract differences by comparing a pair of audio clips and a similarity-discrepancy disentanglement to emphasize the difference in the latent space. To evaluate the proposed methods, we built an AudioDiffCaps dataset consisting of pairs of similar but slightly different audio clips with human-annotated descriptions of their differences. The experiment with the AudioDiffCaps dataset showed that the proposed methods solve the ADC task effectively and improve the attention weights to extract the difference by visualizing them in the transformer encoder.
【8】KinSPEAK: Improving speech recognition for Kinyarwanda via semi-supervised learning methods标题：KinSPEAK：利用半监督学习方法改进基尼亚卢旺达语语音识别
链接：http://arxiv.org/pdf/2308.11863v1
作者：Antoine Nzeyimana机构：KINLP R&D, Kigali, Rwanda备注：9 pages, 2 figures, 5 tables摘要：尽管最近可用的大型转录基尼亚卢旺达语语音数据，实现鲁棒的基尼亚卢旺达语语音识别仍然是具有挑战性的。在这项工作中，我们表明，使用自监督预训练，在微调过程中遵循简单的课程安排，并使用半监督学习来利用大型未标记的语音数据显着提高基尼亚卢旺达语的语音识别性能。我们的方法侧重于只使用公共领域的数据。从公共网站收集一个新的工作室质量的语音数据集，然后用于训练一个干净的基线模型。然后使用干净的基线模型对来自更多样化和嘈杂的公共数据集的示例进行排名，定义简单的课程训练计划。最后，我们应用半监督学习来标记并从连续四代的大型未标记数据中学习。我们的最终模型在新数据集上实现了3.2%的单词错误率（WER），在Mozilla Common Voice基准测试中实现了15.9%的WER，这是我们所知的最先进的。我们的实验还表明，使用音节，而不是基于字符的令牌化结果在更好的语音识别性能为基尼亚卢旺达语。摘要：Despite recent availability of large transcribed Kinyarwanda speech data, achieving robust speech recognition for Kinyarwanda is still challenging. In this work, we show that using self-supervised pre-training, following a simple curriculum schedule during fine-tuning and using semi-supervised learning to leverage large unlabelled speech data significantly improve speech recognition performance for Kinyarwanda. Our approach focuses on using public domain data only. A new studio-quality speech dataset is collected from a public website, then used to train a clean baseline model. The clean baseline model is then used to rank examples from a more diverse and noisy public dataset, defining a simple curriculum training schedule. Finally, we apply semi-supervised learning to label and learn from large unlabelled data in four successive generations. Our final model achieves 3.2% word error rate (WER) on the new dataset and 15.9% WER on Mozilla Common Voice benchmark, which is state-of-the-art to the best of our knowledge. Our experiments also indicate that using syllabic rather than character-based tokenization results in better speech recognition performance for Kinyarwanda.
【9】Example-Based Framework for Perceptually Guided Audio Texture Generation标题：基于实例的感知制导音频纹理生成框架
链接：http://arxiv.org/pdf/2308.11859v1
作者：Purnima Kamath,Chitralekha Gupta,Lonce Wyse,Suranga Nanayakkara机构：†Universitat Pompeu Fabra, ∗National University of Singapore摘要：用于合成音频纹理的生成模型通过用标记数据调节模型来显式地编码可控性。虽然音频纹理的数据集可以很容易地在野外记录，但语义标记它们是昂贵的，耗时的，并且由于人类注释者的主观性而容易出错。因此，为了控制生成，需要在对未标记的纹理进行建模的同时自动推断生成模型的潜在空间中的变化的用户定义的感知因素。在本文中，我们提出了一个基于实例的框架，以确定向量，指导纹理生成的基础上，用户定义的语义属性。通过合成一些合成的例子来指示语义属性的存在或不存在，我们可以推断生成模型的潜在空间中的指导向量，以在生成期间控制该属性。我们的研究结果表明，我们的方法是能够找到感知相关的和确定性的指导矢量可控生成离散以及连续纹理。此外，我们展示了该方法的应用程序的其他任务，如选择性的语义属性转移。摘要：Generative models for synthesizing audio textures explicitly encode controllability by conditioning the model with labelled data. While datasets for audio textures can be easily recorded in-the-wild, semantically labeling them is expensive, time-consuming, and prone to errors due to human annotator subjectivity. Thus, to control generation, there is a need to automatically infer user-defined perceptual factors of variation in the latent space of a generative model while modelling unlabeled textures. In this paper, we propose an example-based framework to determine vectors to guide texture generation based on user-defined semantic attributes. By synthesizing a few synthetic examples to indicate the presence or absence of a semantic attribute, we can infer the guidance vectors in the latent space of a generative model to control that attribute during generation. Our results show that our method is capable of finding perceptually relevant and deterministic guidance vectors for controllable generation for both discrete as well as continuous textures. Furthermore, we demonstrate the application of this method to other tasks such as selective semantic attribute transfer.
eess.AS音频处理
【1】Analysis of XLS-R for Speech Quality Assessment
标题：用于语音质量评估的XLS-R分析
链接：http://arxiv.org/pdf/2308.12077v1
*与cs.SD语音【5】为同一篇
作者：Bastiaan Tamm,Rik Vandenberghe,Hugo Van hamme机构：Laboratory for Cognitive Neurology (LCN), Department of Neurosciences, KU Leuven, Belgium, Processing Speech and Images (PSI), Department of Electrical Engineering, KU Leuven, Belgium备注：5 pages, submitted to WASPAA 2023摘要：在在线会议应用中，估计音频信号的感知质量对于确保最终用户的高质量体验至关重要。评估语音信号质量的最可靠方法是通过平均意见得分（MOS）度量形式的人类判断。然而，这种方法是劳动密集型的，并且对于大规模应用是不可行的。因此，重点已经转向通过深度神经网络的端到端训练进行自动语音质量评估。最近，研究表明，利用预训练的基于wav 2 vec的XLS-R嵌入可以为语音质量预测任务带来最先进的性能。在本文中，我们对预训练模型进行了深入分析。首先，我们分析了从XLS-R的每一层提取的嵌入的性能，以及模型的每个大小（300 M，1B，2B参数）。令人惊讶的是，我们发现了两个用于特征提取的最佳区域：一个在低级特征中，一个在高级特征中。接下来，我们调查两个不同的最优的原因。我们假设，较低级别的功能捕捉噪声和室内声学特性，而高级别的功能集中在语音内容和清晰度。为了调查这一点，我们分析了MOS预测的敏感性，就不同程度的腐败在每个类别。之后，我们尝试融合两个最佳特征深度，以确定它们是否包含用于MOS预测的互补信息。最后，我们比较了所提出的模型的性能，并评估了模型在未知数据集上的泛化能力。摘要：In online conferencing applications, estimating the perceived quality of an audio signal is crucial to ensure high quality of experience for the end user. The most reliable way to assess the quality of a speech signal is through human judgments in the form of the mean opinion score (MOS) metric. However, such an approach is labor intensive and not feasible for large-scale applications. The focus has therefore shifted towards automated speech quality assessment through end-to-end training of deep neural networks. Recently, it was shown that leveraging pre-trained wav2vec-based XLS-R embeddings leads to state-of-the-art performance for the task of speech quality prediction. In this paper, we perform an in-depth analysis of the pre-trained model. First, we analyze the performance of embeddings extracted from each layer of XLS-R and also for each size of the model (300M, 1B, 2B parameters). Surprisingly, we find two optimal regions for feature extraction: one in the lower-level features and one in the high-level features. Next, we investigate the reason for the two distinct optima. We hypothesize that the lower-level features capture characteristics of noise and room acoustics, whereas the high-level features focus on speech content and intelligibility. To investigate this, we analyze the sensitivity of the MOS predictions with respect to different levels of corruption in each category. Afterwards, we try fusing the two optimal feature depths to determine if they contain complementary information for MOS prediction. Finally, we compare the performance of the proposed models and assess the generalizability of the models on unseen datasets.
【2】Joint Prediction of Audio Event and Annoyance Rating in an Urban Soundscape by Hierarchical Graph Representation Learning
标题：基于层次图表示学习的城市声环境中音频事件和烦扰等级联合预测
链接：http://arxiv.org/pdf/2308.11980v1
*与cs.SD语音【6】为同一篇
作者：Yuanbo Hou,Siyang Song,Cheng Luo,Andrew Mitchell,Qiaoqiao Ren,Weicheng Xie,Jian Kang,Wenwu Wang,Dick Botteldooren机构：WAVES Research Group, Ghent University, Belgium. ,University of Leicester, UK., Shenzhen University, China. ,University College London, UK., AIRO-IDlab, Ghent University-Imec, Belgium. ,University of Surrey, UK.备注：INTERSPEECH 2023, Code and models: this https URL摘要：日常生活中的声音事件承载着丰富的客观世界信息。这些声音的构成影响了声景中人们的情绪。大多数先前的方法仅关注于对音频事件和场景进行分类和检测，但是可能忽略它们的感知质量，这可能影响人类对环境的收听情绪，例如，音频事件和场景。烦恼为此，本文提出了一种新的分层图表示学习（HGRL）的方法，该方法将客观的音频事件（AE）与人类感知的声景的主观烦恼评级（AR）联系起来。分层图由具有单类事件语义的细粒度事件（fAE）嵌入、具有多类事件语义的粗粒度事件（cAE）嵌入和AR嵌入组成。实验结果表明，所提出的HGRL成功地集成了AE与AR的AEC和ARP任务，同时协调cAE和fAE之间的关系，并进一步对齐的两个不同颗粒的AE信息与AR。摘要：Sound events in daily life carry rich information about the objective world. The composition of these sounds affects the mood of people in a soundscape. Most previous approaches only focus on classifying and detecting audio events and scenes, but may ignore their perceptual quality that may impact humans' listening mood for the environment, e.g. annoyance. To this end, this paper proposes a novel hierarchical graph representation learning (HGRL) approach which links objective audio events (AE) with subjective annoyance ratings (AR) of the soundscape perceived by humans. The hierarchical graph consists of fine-grained event (fAE) embeddings with single-class event semantics, coarse-grained event (cAE) embeddings with multi-class event semantics, and AR embeddings. Experiments show the proposed HGRL successfully integrates AE with AR for AEC and ARP tasks, while coordinating the relations between cAE and fAE and further aligning the two different grains of AE information with the AR.
【3】Audio Difference Captioning Utilizing Similarity-Discrepancy Disentanglement
标题：利用相似-差异解缠的音频差异字幕
链接：http://arxiv.org/pdf/2308.11923v1
*与cs.SD语音【7】为同一篇
作者：Daiki Takeuchi,Yasunori Ohishi,Daisuke Niizumi,Noboru Harada,Kunio Kashino机构：NTT Corporation, Japan备注：Accepted to DCASE2023 Workshop摘要：我们提出了音频差异字幕（ADC）作为一个新的扩展任务的音频字幕描述相似，但略有不同的音频片段的输入对之间的语义差异。ADC解决了传统的音频字幕有时会为类似的音频片段生成类似的字幕，无法描述内容差异的问题。我们还提出了一个交叉注意力集中的Transformer编码器提取差异，通过比较一对音频剪辑和相似性-差异解纠缠强调潜在空间中的差异。为了评估所提出的方法，我们建立了一个AudioDiffCaps数据集，该数据集由两对相似但略有不同的音频剪辑组成，并带有人类注释的差异描述。AudioDiffCaps数据集的实验表明，所提出的方法有效地解决了ADC任务，并提高了注意力权重，通过在Transformer编码器中可视化它们来提取差异。摘要：We proposed Audio Difference Captioning (ADC) as a new extension task of audio captioning for describing the semantic differences between input pairs of similar but slightly different audio clips. The ADC solves the problem that conventional audio captioning sometimes generates similar captions for similar audio clips, failing to describe the difference in content. We also propose a cross-attention-concentrated transformer encoder to extract differences by comparing a pair of audio clips and a similarity-discrepancy disentanglement to emphasize the difference in the latent space. To evaluate the proposed methods, we built an AudioDiffCaps dataset consisting of pairs of similar but slightly different audio clips with human-annotated descriptions of their differences. The experiment with the AudioDiffCaps dataset showed that the proposed methods solve the ADC task effectively and improve the attention weights to extract the difference by visualizing them in the transformer encoder.
【4】KinSPEAK: Improving speech recognition for Kinyarwanda via semi-supervised learning methods标题：KinSPEAK：利用半监督学习方法改进基尼亚卢旺达语语音识别
链接：http://arxiv.org/pdf/2308.11863v1
*与cs.SD语音【8】为同一篇
作者：Antoine Nzeyimana机构：KINLP R&D, Kigali, Rwanda备注：9 pages, 2 figures, 5 tables摘要：尽管最近可用的大型转录基尼亚卢旺达语语音数据，实现鲁棒的基尼亚卢旺达语语音识别仍然是具有挑战性的。在这项工作中，我们表明，使用自监督预训练，在微调过程中遵循简单的课程安排，并使用半监督学习来利用大型未标记的语音数据显着提高基尼亚卢旺达语的语音识别性能。我们的方法侧重于只使用公共领域的数据。从公共网站收集一个新的工作室质量的语音数据集，然后用于训练一个干净的基线模型。然后使用干净的基线模型对来自更多样化和嘈杂的公共数据集的示例进行排名，定义简单的课程训练计划。最后，我们应用半监督学习来标记并从连续四代的大型未标记数据中学习。我们的最终模型在新数据集上实现了3.2%的单词错误率（WER），在Mozilla Common Voice基准测试中实现了15.9%的WER，这是我们所知的最先进的。我们的实验还表明，使用音节，而不是基于字符的令牌化结果在更好的语音识别性能为基尼亚卢旺达语。摘要：Despite recent availability of large transcribed Kinyarwanda speech data, achieving robust speech recognition for Kinyarwanda is still challenging. In this work, we show that using self-supervised pre-training, following a simple curriculum schedule during fine-tuning and using semi-supervised learning to leverage large unlabelled speech data significantly improve speech recognition performance for Kinyarwanda. Our approach focuses on using public domain data only. A new studio-quality speech dataset is collected from a public website, then used to train a clean baseline model. The clean baseline model is then used to rank examples from a more diverse and noisy public dataset, defining a simple curriculum training schedule. Finally, we apply semi-supervised learning to label and learn from large unlabelled data in four successive generations. Our final model achieves 3.2% word error rate (WER) on the new dataset and 15.9% WER on Mozilla Common Voice benchmark, which is state-of-the-art to the best of our knowledge. Our experiments also indicate that using syllabic rather than character-based tokenization results in better speech recognition performance for Kinyarwanda.
【5】Example-Based Framework for Perceptually Guided Audio Texture Generation标题：基于实例的感知制导音频纹理生成框架
链接：http://arxiv.org/pdf/2308.11859v1
*与cs.SD语音【9】为同一篇
作者：Purnima Kamath,Chitralekha Gupta,Lonce Wyse,Suranga Nanayakkara机构：†Universitat Pompeu Fabra, ∗National University of Singapore摘要：用于合成音频纹理的生成模型通过用标记数据调节模型来显式地编码可控性。虽然音频纹理的数据集可以很容易地在野外记录，但语义标记它们是昂贵的，耗时的，并且由于人类注释者的主观性而容易出错。因此，为了控制生成，需要在对未标记的纹理进行建模的同时自动推断生成模型的潜在空间中的变化的用户定义的感知因素。在本文中，我们提出了一个基于实例的框架，以确定向量，指导纹理生成的基础上，用户定义的语义属性。通过合成一些合成的例子来指示语义属性的存在或不存在，我们可以推断生成模型的潜在空间中的指导向量，以在生成期间控制该属性。我们的研究结果表明，我们的方法是能够找到感知相关的和确定性的指导矢量可控生成离散以及连续纹理。此外，我们展示了该方法的应用程序的其他任务，如选择性的语义属性转移。摘要：Generative models for synthesizing audio textures explicitly encode controllability by conditioning the model with labelled data. While datasets for audio textures can be easily recorded in-the-wild, semantically labeling them is expensive, time-consuming, and prone to errors due to human annotator subjectivity. Thus, to control generation, there is a need to automatically infer user-defined perceptual factors of variation in the latent space of a generative model while modelling unlabeled textures. In this paper, we propose an example-based framework to determine vectors to guide texture generation based on user-defined semantic attributes. By synthesizing a few synthetic examples to indicate the presence or absence of a semantic attribute, we can infer the guidance vectors in the latent space of a generative model to control that attribute during generation. Our results show that our method is capable of finding perceptually relevant and deterministic guidance vectors for controllable generation for both discrete as well as continuous textures. Furthermore, we demonstrate the application of this method to other tasks such as selective semantic attribute transfer.
【6】CED: Consistent ensemble distillation for audio tagging
标题：CED：用于音频标记的一致集成蒸馏
链接：http://arxiv.org/pdf/2308.11957v1
*与cs.SD语音【1】为同一篇
作者：Heinrich Dinkel,Yongqing Wang,Zhiyong Yan,Junbo Zhang,Yujun Wang机构：Xiaomi Corporation, Beijing, China摘要：增强和知识蒸馏（KD）是音频分类任务领域中采用的成熟技术，旨在提高性能并减小广泛认可的Audioset（AS）基准上的模型大小。虽然这两种技术单独使用都很有效，但它们的结合使用，称为一致性教学，以前还没有被探索过。本文提出了CED，一个简单的培训框架，从大型教师合奏与一致的教学中提炼出学生模型。为了实现这一点，CED有效地将logits以及增强方法存储在磁盘上，使其可扩展到大规模数据集。CED的功效的核心是其无标签性质，这意味着仅存储的logits用于学生模型的优化，仅需要0.3 %的额外磁盘空间用于AS。该研究训练了各种基于变压器的模型，包括在AS上实现49.0平均平均精度（mAP）的10M参数模型。预训练模型和代码可在https: github.com RicherMans CED上获得。摘要：Augmentation and knowledge distillation (KD) are well-established techniques employed in the realm of audio classification tasks, aimed at enhancing performance and reducing model sizes on the widely recognized Audioset (AS) benchmark. Although both techniques are effective individually, their combined use, called consistent teaching, hasn't been explored before. This paper proposes CED, a simple training framework that distils student models from large teacher ensembles with consistent teaching. To achieve this, CED efficiently stores logits as well as the augmentation methods on disk, making it scalable to large-scale datasets. Central to CED's efficacy is its label-free nature, meaning that only the stored logits are used for the optimization of a student model only requiring 0.3 % additional disk space for AS. The study trains various transformer-based models, including a 10M parameter model achieving a 49.0 mean average precision (mAP) on AS. Pretrained models and code are available at https: github.com RicherMans CED.
【7】Audio Generation with Multiple Conditional Diffusion Model标题：基于多条件扩散模型的音频生成链接：http://arxiv.org/pdf/2308.11940v1
*与cs.SD语音【2】为同一篇
作者：Zhifang Guo,Jianguo Mao,Rui Tao,Long Yan,Kazushige Ouchi,Hong Liu,Xiangdong Wang机构：Bejing Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China, Toshiba China R&D Center, Beijing, China备注：Submitted to AAAI 2024摘要：基于文本的音频生成模型具有局限性，因为它们不能包含音频中的所有信息，从而导致当仅依赖于文本时受限的可控性。为了解决这个问题，我们提出了一种新的模型，该模型提高了现有的预训练文本到音频模型的可控性，通过纳入额外的条件，包括内容（时间戳）和风格（音高轮廓和能量轮廓）作为补充的文本。该方法实现了对所生成的音频的时间顺序、音高和能量的细粒度控制。为了保持生成的多样性，我们采用了一个可训练的控制条件编码器，该编码器由一个大的语言模型和一个可训练的融合网来编码和融合额外的条件，同时保持预训练的文本到音频模型的权重冻结。由于缺乏合适的数据集和评估指标，我们将现有的数据集合并为一个新的数据集，包括音频和相应的条件，并使用一系列的评估指标来评估可控性性能。实验结果表明，我们的模型成功地实现了细粒度控制，以实现可控的音频生成。音频样本和我们的数据集可在https: conditionaudiogen.github.io conditionaudiogen 上公开获取摘要：Text-based audio generation models have limitations as they cannot encompass all the information in audio, leading to restricted controllability when relying solely on text. To address this issue, we propose a novel model that enhances the controllability of existing pre-trained text-to-audio models by incorporating additional conditions including content (timestamp) and style (pitch contour and energy contour) as supplements to the text. This approach achieves fine-grained control over the temporal order, pitch, and energy of generated audio. To preserve the diversity of generation, we employ a trainable control condition encoder that is enhanced by a large language model and a trainable Fusion-Net to encode and fuse the additional conditions while keeping the weights of the pre-trained text-to-audio model frozen. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing datasets into a new dataset comprising the audio and corresponding conditions and use a series of evaluation metrics to evaluate the controllability performance. Experimental results demonstrate that our model successfully achieves fine-grained control to accomplish controllable audio generation. Audio samples and our dataset are publicly available at https: conditionaudiogen.github.io conditionaudiogen
【8】Complex-valued neural networks for voice anti-spoofing
标题：用于语音反欺骗的复值神经网络
链接：http://arxiv.org/pdf/2308.11800v1
*与cs.SD语音【3】为同一篇
作者：Nicolas M. Müller,Philip Sperl,Konstantin Böttinger机构：Fraunhofer AISEC备注：Interspeech 2023摘要：当前的反欺骗和音频深度伪造检测系统使用基于幅度谱图的特征（例如CQT或Melspectrograms）或通过卷积或sinc-layers处理的原始音频。这两种方法都有缺点：幅度谱图丢弃影响音频自然度的相位信息，并且基于原始特征的模型不能使用传统的可解释AI方法。本文提出了一种新的方法，结合了这两种方法的好处，通过使用复值神经网络处理的复值，CQT频域表示的输入音频。该方法保留相位信息，并允许可解释的AI方法。结果表明，这种方法优于以前的方法上的“在野外”反欺骗数据集，并使解释的结果，通过可解释的AI。消融研究证实，模型已经学会使用相位信息来检测语音欺骗。摘要：Current anti-spoofing and audio deepfake detection systems use either magnitude spectrogram-based features (such as CQT or Melspectrograms) or raw audio processed through convolution or sinc-layers. Both methods have drawbacks: magnitude spectrograms discard phase information, which affects audio naturalness, and raw-feature-based models cannot use traditional explainable AI methods. This paper proposes a new approach that combines the benefits of both methods by using complex-valued neural networks to process the complex-valued, CQT frequency-domain representation of the input audio. This method retains phase information and allows for explainable AI methods. Results show that this approach outperforms previous methods on the "In-the-Wild" anti-spoofing dataset and enables interpretation of the results through explainable AI. Ablation studies confirm that the model has learned to use phase information to detect voice spoofing.
【9】Identifying depression-related topics in smartphone-collected free-response speech recordings using an automatic speech recognition system and a deep learning topic model标题：使用自动语音识别系统和深度学习主题模型在智能手机收集的自由反应语音记录中识别与抑郁相关的主题链接：http://arxiv.org/pdf/2308.11773v1
*与cs.SD语音【4】为同一篇
作者：Yuezhou Zhang,Amos A Folarin,Judith Dineley,Pauline Conde,Valeria de Angel,Shaoxiong Sun,Yatharth Ranjan,Zulqarnain Rashid,Callum Stewart,Petroula Laiou,Heet Sankesara,Linglong Qian,Faith Matcham,Katie M White,Carolin Oetzmann,Femke Lamers,Sara Siddi,Sara Simblett,Björn W. Schuller,Srinivasan Vairavan,Til Wykes,Josep Maria Haro,Brenda WJH Penninx,Vaibhav A Narayan,Matthew Hotopf,Richard JB Dobson,Nicholas Cummins,RADAR-CNS consortium机构：King's College London, London, UK, South London and Maudsley NHS Foundation Trust, London, UK, Health Data Research UK London, University College London, London, UK, University of Augsburg, Augsburg, Germany摘要：语言使用已被证明与抑郁症相关，但需要大规模的验证。传统的方法如临床研究是昂贵的。因此，自然语言处理已经被用于社交媒体来预测抑郁症，但其局限性仍然存在-缺乏有效的标签，有偏见的用户样本，以及没有上下文。我们的研究确定了29个主题的3919智能手机收集的语音录音从265名参与者使用Whisper工具和BERTopic模型。中位PHQ-8大于或等于10的6个主题被视为抑郁症的风险主题：没有期望，睡眠，心理治疗，理发，学习和课程。为了阐明主题的出现和与抑郁症的关联，我们比较了已确定主题的行为（从可穿戴设备）和语言特征。话题转移和抑郁症严重程度随时间的变化之间的相关性也进行了调查，表明纵向监测语言使用的重要性。我们还在一个类似的较小数据集（来自57名参与者的356个语音记录）上测试了BERTopic模型，获得了一些一致的结果。总之，我们的研究结果表明，特定的言语主题可能指示抑郁症的严重程度。所提出的数据驱动的工作流提供了一种实用的方法来收集和分析来自真实世界的数字健康研究设置的大规模语音数据。摘要：Language use has been shown to correlate with depression, but large-scale validation is needed. Traditional methods like clinic studies are expensive. So, natural language processing has been employed on social media to predict depression, but limitations remain-lack of validated labels, biased user samples, and no context. Our study identified 29 topics in 3919 smartphone-collected speech recordings from 265 participants using the Whisper tool and BERTopic model. Six topics with a median PHQ-8 greater than or equal to 10 were regarded as risk topics for depression: No Expectations, Sleep, Mental Therapy, Haircut, Studying, and Coursework. To elucidate the topic emergence and associations with depression, we compared behavioral (from wearables) and linguistic characteristics across identified topics. The correlation between topic shifts and changes in depression severity over time was also investigated, indicating the importance of longitudinally monitoring language use. We also tested the BERTopic model on a similar smaller dataset (356 speech recordings from 57 participants), obtaining some consistent results. In summary, our findings demonstrate specific speech topics may indicate depression severity. The presented data-driven workflow provides a practical approach to collecting and analyzing large-scale speech data from real-world settings for digital health research.
机器翻译，仅供参考