今日论文合集：cs.SD语音7篇，eess.AS音频处理8篇。cs.SD语音【1】 Low-latency Speech Enhancement via Speech Token Generation标题：通过语音令牌生成实现低延迟语音增强链接：https://arxiv.org/abs/2310.08981作者：Huaying Xue,Xiulian Peng,Yan Lu机构：Microsoft Research Asia, Beijing, China备注：5 pages摘要：现有的基于深度学习的语音增强主要采用数据驱动的方法，该方法利用具有各种噪声类型的大量数据来实现从含噪信号中去除噪声。然而，对数据的高度依赖限制了它对现实环境中不可见的复杂噪声的推广。在本文中，我们专注于低延迟的情况下，把语音增强作为一个语音生成问题的条件下的噪声信号，我们生成干净的语音，而不是识别和去除噪声。具体来说，我们提出了一个有条件的语音增强生成框架，该框架通过神经语音编解码器的声学代码对干净的语音进行建模，并以自回归的方式生成以过去的噪声帧为条件的语音代码。此外，我们提出了一种显式对齐方法，以调整噪声帧与生成的语音令牌，以提高不同的输入长度的鲁棒性和可扩展性。与其他利用多个阶段来生成语音代码的方法不同，我们利用基于TF-Codec神经编解码器的单阶段语音生成方法来实现低延迟的高语音质量。在人工合成和真实语音测试集上的实验结果表明，该方法在噪声鲁棒性和语音时间相干性方面优于数据驱动方法。摘要：Existing deep learning based speech enhancement mainly employ a data-driven approach, which leverage large amounts of data with a variety of noise types to achieve noise removal from noisy signal. However, the high dependence on the data limits its generalization on the unseen complex noises in real-life environment. In this paper, we focus on the low-latency scenario and regard speech enhancement as a speech generation problem conditioned on the noisy signal, where we generate clean speech instead of identifying and removing noises. Specifically, we propose a conditional generative framework for speech enhancement, which models clean speech by acoustic codes of a neural speech codec and generates the speech codes conditioned on past noisy frames in an auto-regressive way. Moreover, we propose an explicit-alignment approach to align noisy frames with the generated speech tokens to improve the robustness and scalability to different input lengths. Different from other methods that leverage multiple stages to generate speech codes, we leverage a single-stage speech generation approach based on the TF-Codec neural codec to achieve high speech quality with low latency. Extensive results on both synthetic and real-recorded test set show its superiority over data-driven approaches in terms of noise robustness and temporal speech coherence.【2】 Transformer-based Autoencoder with ID Constraint for Unsupervised Anomalous Sound Detection标题：基于ID约束的Transformer式自动编码器无监督异常音检测链接：https://arxiv.org/abs/2310.08950作者：Jian Guan,Youde Liu,Qiuqiang Kong,Feiyang Xiao,Qiaoxi Zhu,Jiantong Tian,Wenwu Wang机构：College of Computer Science and Technology, Harbin Engineering, University, Harbin, Heilongjiang, China., Bytedance, Shanghai, China., Centre for Audio, Acoustics and Vibration, University of Technology, Sydney, Ultimo, NSW , NSW, Australia.备注：Accepted by EURASIP Journal on Audio, Speech, and Music Processing摘要：无监督异常声音检测（ASD）的目的是在只有正常声音数据可用时检测设备的未知异常声音。基于自动编码器（AE）和自监督学习的方法是两种主流方法。然而，基于AE的方法可能受到限制，因为从正常声音学习的特征也可以与异常声音相适应，从而降低了模型从声音中检测异常的能力。自监督方法并不总是稳定的，并且即使对于相同类型的机器也会有不同的表现。此外，异常声音可能是短暂的，使其更难与正常声音区分开来。本文提出了一种基于ID约束的Transformer-based自动编码器（IDC-transAE）架构与加权异常分数计算的无监督ASD。通过引入一个简单的机器ID分类器来学习同一机器类型的分布差异，并增强模型区分异常声音的能力，机器ID被用来约束基于Transformer-based autoencoder（transAE）的潜在空间。此外，加权异常分数计算被引入到突出的异常事件，只出现了很短的时间的异常分数。在DCASE 2020 Challenge Task 2开发数据集上进行的实验证明了该方法的有效性和优越性。摘要：Unsupervised anomalous sound detection (ASD) aims to detect unknown anomalous sounds of devices when only normal sound data is available. The autoencoder (AE) and self-supervised learning based methods are two mainstream methods. However, the AE-based methods could be limited as the feature learned from normal sounds can also fit with anomalous sounds, reducing the ability of the model in detecting anomalies from sound. The self-supervised methods are not always stable and perform differently, even for machines of the same type. In addition, the anomalous sound may be short-lived, making it even harder to distinguish from normal sound. This paper proposes an ID constrained Transformer-based autoencoder (IDC-TransAE) architecture with weighted anomaly score computation for unsupervised ASD. Machine ID is employed to constrain the latent space of the Transformer-based autoencoder (TransAE) by introducing a simple ID classifier to learn the difference in the distribution for the same machine type and enhance the ability of the model in distinguishing anomalous sound. Moreover, weighted anomaly score computation is introduced to highlight the anomaly scores of anomalous events that only appear for a short time. Experiments performed on DCASE 2020 Challenge Task2 development dataset demonstrate the effectiveness and superiority of our proposed method.【3】 Differential Evolution Algorithm based Hyper-Parameters Selection of Convolutional Neural Network for Speech Command Recognition标题：基于微分进化算法的卷积神经网络超参数选择用于语音命令识别链接：https://arxiv.org/abs/2310.08914作者：Sandipan Dhar,Anuvab Sen,Aritra Bandyopadhyay,Nanda Dulal Jana,Arjun Ghosh,Zahra Sarayloo机构：Computer Science and Engineering, National Institute of Technology, Durgapur, West Bengal, India, Electronics and Telecommunication, Indian Institute of Engineering Science and Technology, Shibpur, Howrah, India备注：8 Pages, 7 Figures, 4 Tables, Accepted by the 15th International Joint Conference on Computational Intelligence (IJCCI 2023), November 13-15, 2023, Rome, Italy摘要：语音命令识别（SCR）处理短语音命令的识别，对于包括物联网设备和辅助技术在内的各种应用至关重要。尽管卷积神经网络（CNN）在SCR任务中表现出了良好的前景，但它们的功效在很大程度上依赖于超参数选择，而手动选择通常既费力又耗时。本文介绍了一种基于差分进化（DE）算法的CNN超参数选择方法，旨在提高SCR任务的性能。通过对Google Speech Command（GSC）数据集的训练和测试，该方法在语音命令分类方面表现出了有效性。此外，与基于遗传算法的选择和其他深度CNN（DCNN）模型的比较分析突出了所提出的DE算法在SCR任务中CNN的超参数选择中的效率。摘要：Speech Command Recognition (SCR), which deals with identification of short uttered speech commands, is crucial for various applications, including IoT devices and assistive technology. Despite the promise shown by Convolutional Neural Networks (CNNs) in SCR tasks, their efficacy relies heavily on hyper-parameter selection, which is typically laborious and time-consuming when done manually. This paper introduces a hyper-parameter selection method for CNNs based on the Differential Evolution (DE) algorithm, aiming to enhance performance in SCR tasks. Training and testing with the Google Speech Command (GSC) dataset, the proposed approach showed effectiveness in classifying speech commands. Moreover, a comparative analysis with Genetic Algorithm based selections and other deep CNN (DCNN) models highlighted the efficiency of the proposed DE algorithm in hyper-parameter selection for CNNs in SCR tasks.【4】 Learning to Behave Like Clean Speech: Dual-Branch Knowledge Distillation for Noise-Robust Fake Audio Detection标题：学会表现得像干净的语音：用于抗噪声伪音频检测的双分支知识提取链接：https://arxiv.org/abs/2310.08869作者：Cunhang Fan,Mingming Ding,Jianhua Tao,Ruibo Fu,Jiangyan Yi,Zhengqi Wen,Zhao Lv摘要：虚假音频检测（FAD）的大多数研究都集中在提高标准无噪声数据集的性能上。然而，在实际应用中，FAD系统中通常存在噪声干扰，这将导致系统性能的显著下降。为了提高对噪声的鲁棒性，提出了一种双分支知识提取的虚假音频检测方法。具体而言，设计了干净教师分支和嘈杂学生分支的并行数据流，并提出了交互式融合和基于响应的师生范式，从数据分布和决策的角度指导嘈杂数据的训练。在噪声分支中，首先引入语音增强进行去噪，降低了强噪声的干扰。交互式融合结合了去噪特征和噪声特征，以降低语音失真的影响，并寻求与干净分支数据分布的一致性。师生范式将学生的决策空间映射到教师的决策空间，使嘈杂的演讲表现得像干净的一样。此外，联合训练方法被用来优化两个分支，以实现全局最优。基于多个数据集的实验结果表明，该方法在噪声环境中表现良好，并保持在跨数据集实验的性能。摘要：Most research in fake audio detection (FAD) focuses on improving performance on standard noise-free datasets. However, in actual situations, there is usually noise interference, which will cause significant performance degradation in FAD systems. To improve the noise robustness, we propose a dual-branch knowledge distillation fake audio detection (DKDFAD) method. Specifically, a parallel data flow of the clean teacher branch and the noisy student branch is designed, and interactive fusion and response-based teacher-student paradigms are proposed to guide the training of noisy data from the data distribution and decision-making perspectives. In the noise branch, speech enhancement is first introduced for denoising, which reduces the interference of strong noise. The proposed interactive fusion combines denoising features and noise features to reduce the impact of speech distortion and seek consistency with the data distribution of clean branch. The teacher-student paradigm maps the student's decision space to the teacher's decision space, making noisy speech behave as clean. In addition, a joint training method is used to optimize the two branches to achieve global optimality. Experimental results based on multiple datasets show that the proposed method performs well in noisy environments and maintains performance in cross-dataset experiments.【5】 CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models标题：COPA：解决音频语言模型中成分推理的差距链接：https://arxiv.org/abs/2310.08753作者：Sreyan Ghosh,Ashish Seth,Sonal Kumar,Utkarsh Tyagi,Chandra Kiran Evuru,S. Ramaneswaran,S. Sakshi,Oriol Nieto,Ramani Duraiswami,Dinesh Manocha机构：♠University of Maryland, College Park, USA, ⊡Indian Institute Of Technology, Madras, India, ♥NVIDIA, Bangalore, India, ♣UMass, Amherst, USA, ♦Adobe, USA备注：Pre-print under review摘要：音频的一个基本特征是其组成性质。使用对比方法训练的音频语言模型（ALM）（例如，CLAP）学习音频和语言模态之间的共享表示，其在许多下游应用（包括zero-shot音频分类、音频检索等）中提高了性能。然而，这些模型有效地执行组合推理的能力在很大程度上尚未探索，需要进行额外的研究。在本文中，我们提出了CompA，一个集合的两个专家注释的基准与大多数的真实世界的音频样本，以评估组合推理在ALM。我们提出的CompA-order评估ALM如何理解音频中声学事件的顺序或发生，CompA-attribute评估声学事件的属性绑定。来自任一基准的实例由两个音频-字幕对组成，其中两个音频具有相同的声学事件但具有不同的合成。ALM的评估取决于它将正确的音频与正确的字幕匹配得有多好。使用这个基准测试，我们首先表明，目前的ALMs执行仅略好于随机机会，从而挣扎与组合推理。接下来，我们提出了CompA-CLAP，我们使用一种新的学习方法来微调CLAP，以提高其组合推理能力。为了训练CompA-CLAP，我们首先提出了对具有成分感知的硬否定的对比训练的改进，从而允许更集中的训练。接下来，我们提出了一种新的模块化对比损失，帮助模型学习细粒度的组成理解，并克服公开可用的组成音频的严重稀缺性。CompA-CLAP在CompA基准测试中显著改善了我们所有的基线模型，表明其优越的组合推理能力。摘要：A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM is evaluated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, we propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to improve its compositional reasoning abilities. To train CompA-CLAP, we first propose improvements to contrastive training with composition-aware hard negatives, allowing for more focused training. Next, we propose a novel modular contrastive loss that helps the model learn fine-grained compositional understanding and overcomes the acute scarcity of openly available compositional audios. CompA-CLAP significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.【6】 Toward Joint Language Modeling for Speech Units and Text标题：面向语音单元和语篇的联合语言建模链接：https://arxiv.org/abs/2310.08715作者：Ju-Chieh Chou,Chung-Ming Chien,Wei-Ning Hsu,Karen Livescu,Arun Babu,Alexis Conneau,Alexei Baevski,Michael Auli机构：Toyota Technological Institute at Chicago ,Meta AI ,OpenAI ,Character AI备注：EMNLP findings 2023摘要：言语和语篇是人类语言的两种主要形式。多年来，研究界一直专注于将语音映射到文本或反之亦然。然而，在语言建模领域，很少有人对它们进行联合建模。有鉴于此，我们探讨了语音单元和文本的联合语言建模。具体来说，我们比较不同的语音标记器将连续语音信号转换为离散单元，并使用不同的方法来构建混合语音文本数据。我们引入自动度量来评估联合LM如何混合语音和文本。我们还微调LM在下游口语理解（SLU）任务与不同的方式（语音或文本），并测试其性能，以评估模型的学习共享表示。我们的研究结果表明，通过将语音单元和文本与我们提出的混合技术混合，联合LM在SLU任务上的语音基线上有所改善，并显示出zero-shot跨模态可转移性。摘要：Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability.【7】 End-to-end Online Speaker Diarization with Target Speaker Tracking标题：使用目标说话人跟踪的端到端在线说话人二值化链接：https://arxiv.org/abs/2310.08696作者：Weiqing Wang,Ming Li机构：such as overlap speechWeiqing Wang and Ming Li are with the Department of Electrical andComputer Engineering at Duke University, Ming Li is also with Data ScienceResearch Center at Duke Kunshan University备注：Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：本文提出了一种在线目标说话人语音活动检测系统的说话人日记任务，它不需要从基于聚类的日记系统的先验知识，以获得目标说话人嵌入。通过调整传统的目标说话人语音活动检测的实时操作，该框架可以识别说话人活动使用自生成的嵌入，从而在推理阶段没有排列不一致的一致性能。在推理过程中，我们采用了一个前端模型来提取帧级的扬声器嵌入信号的每个未来的块。接下来，我们根据这些帧级说话人嵌入和先前估计的目标说话人嵌入来预测每个说话人的检测状态。然后，通过根据当前块中的预测聚合这些帧级说话人嵌入来更新目标说话人嵌入。我们的模型预测每个块的结果，并更新目标说话人的嵌入，直到到达信号的结尾。在DIHARD III和AliMeeting数据集上的实验结果表明，该方法的性能优于基于离线聚类的日志化系统.所提出的方法进一步扩展到多通道数据，达到类似的性能与国家的最先进的离线日志系统。摘要：This paper proposes an online target speaker voice activity detection system for speaker diarization tasks, which does not require a priori knowledge from the clustering-based diarization system to obtain the target speaker embeddings. By adapting the conventional target speaker voice activity detection for real-time operation, this framework can identify speaker activities using self-generated embeddings, resulting in consistent performance without permutation inconsistencies in the inference phase. During the inference process, we employ a front-end model to extract the frame-level speaker embeddings for each coming block of a signal. Next, we predict the detection state of each speaker based on these frame-level speaker embeddings and the previously estimated target speaker embedding. Then, the target speaker embeddings are updated by aggregating these frame-level speaker embeddings according to the predictions in the current block. Our model predicts the results for each block and updates the target speakers' embeddings until reaching the end of the signal. Experimental results show that the proposed method outperforms the offline clustering-based diarization system on the DIHARD III and AliMeeting datasets. The proposed method is further extended to multi-channel data, which achieves similar performance with the state-of-the-art offline diarization systems.eess.AS音频处理【1】 Speaking rate attention-based duration prediction for speed control TTS标题：基于语速注意力的语速控制时长预测链接：https://arxiv.org/abs/2310.08846作者：Jesuraj Bandekar,Sathvik Udupa,Abhayjeet Singh,Anjali Jayakumar,Deekshitha G,Sandhya Badiger,Saurabh Kumar,Pooja VH,Prasanta Kumar Ghosh机构：Electrical Engineering Department, Indian Institute of Science (IISc), Bangalore-, India备注：\c{opyright} 20XX IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：None摘要：With the advent of high-quality speech synthesis, there is a lot of interest in controlling various prosodic attributes of speech. Speaking rate is an essential attribute towards modelling the expressivity of speech. In this work, we propose a novel approach to control the speaking rate for non-autoregressive TTS. We achieve this by conditioning the speaking rate inside the duration predictor, allowing implicit speaking rate control. We show the benefits of this approach by synthesising audio at various speaking rate factors and measuring the quality of speaking rate-controlled synthesised speech. Further, we study the effect of the speaking rate distribution of the training data towards effective rate control. Finally, we fine-tune a baseline pretrained TTS model to obtain speaking rate control TTS. We provide various analyses to showcase the benefits of using this proposed approach, along with objective as well as subjective metrics. We find that the proposed methods have higher subjective scores and lower speaker rate errors across many speaking rate factors over the baseline.【2】 End-to-end Online Speaker Diarization with Target Speaker Tracking标题：使用目标说话人跟踪的端到端在线说话人二值化链接：https://arxiv.org/abs/2310.08696作者：Weiqing Wang,Ming Li机构：such as overlap speechWeiqing Wang and Ming Li are with the Department of Electrical andComputer Engineering at Duke University, Ming Li is also with Data ScienceResearch Center at Duke Kunshan University备注：Submitted to IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：本文提出了一种在线目标说话人语音活动检测系统的说话人日记任务，它不需要从基于聚类的日记系统的先验知识，以获得目标说话人嵌入。通过调整传统的目标说话人语音活动检测的实时操作，该框架可以识别说话人活动使用自生成的嵌入，从而在推理阶段没有排列不一致的一致性能。在推理过程中，我们采用了一个前端模型来提取帧级的扬声器嵌入信号的每个未来的块。接下来，我们根据这些帧级说话人嵌入和先前估计的目标说话人嵌入来预测每个说话人的检测状态。然后，通过根据当前块中的预测聚合这些帧级说话人嵌入来更新目标说话人嵌入。我们的模型预测每个块的结果，并更新目标说话人的嵌入，直到到达信号的结尾。在DIHARD III和AliMeeting数据集上的实验结果表明，该方法的性能优于基于离线聚类的日志化系统.所提出的方法进一步扩展到多通道数据，达到类似的性能与国家的最先进的离线日志系统。摘要：This paper proposes an online target speaker voice activity detection system for speaker diarization tasks, which does not require a priori knowledge from the clustering-based diarization system to obtain the target speaker embeddings. By adapting the conventional target speaker voice activity detection for real-time operation, this framework can identify speaker activities using self-generated embeddings, resulting in consistent performance without permutation inconsistencies in the inference phase. During the inference process, we employ a front-end model to extract the frame-level speaker embeddings for each coming block of a signal. Next, we predict the detection state of each speaker based on these frame-level speaker embeddings and the previously estimated target speaker embedding. Then, the target speaker embeddings are updated by aggregating these frame-level speaker embeddings according to the predictions in the current block. Our model predicts the results for each block and updates the target speakers' embeddings until reaching the end of the signal. Experimental results show that the proposed method outperforms the offline clustering-based diarization system on the DIHARD III and AliMeeting datasets. The proposed method is further extended to multi-channel data, which achieves similar performance with the state-of-the-art offline diarization systems.【3】 Low-latency Speech Enhancement via Speech Token Generation标题：通过语音令牌生成实现低延迟语音增强链接：https://arxiv.org/abs/2310.08981作者：Huaying Xue,Xiulian Peng,Yan Lu机构：Microsoft Research Asia, Beijing, China备注：5 pages摘要：现有的基于深度学习的语音增强主要采用数据驱动的方法，该方法利用具有各种噪声类型的大量数据来实现从含噪信号中去除噪声。然而，对数据的高度依赖限制了它对现实环境中不可见的复杂噪声的推广。在本文中，我们专注于低延迟的情况下，把语音增强作为一个语音生成问题的条件下的噪声信号，我们生成干净的语音，而不是识别和去除噪声。具体来说，我们提出了一个有条件的语音增强生成框架，该框架通过神经语音编解码器的声学代码对干净的语音进行建模，并以自回归的方式生成以过去的噪声帧为条件的语音代码。此外，我们提出了一种显式对齐方法，以调整噪声帧与生成的语音令牌，以提高不同的输入长度的鲁棒性和可扩展性。与其他利用多个阶段来生成语音代码的方法不同，我们利用基于TF-Codec神经编解码器的单阶段语音生成方法来实现低延迟的高语音质量。在人工合成和真实语音测试集上的实验结果表明，该方法在噪声鲁棒性和语音时间相干性方面优于数据驱动方法。摘要：Existing deep learning based speech enhancement mainly employ a data-driven approach, which leverage large amounts of data with a variety of noise types to achieve noise removal from noisy signal. However, the high dependence on the data limits its generalization on the unseen complex noises in real-life environment. In this paper, we focus on the low-latency scenario and regard speech enhancement as a speech generation problem conditioned on the noisy signal, where we generate clean speech instead of identifying and removing noises. Specifically, we propose a conditional generative framework for speech enhancement, which models clean speech by acoustic codes of a neural speech codec and generates the speech codes conditioned on past noisy frames in an auto-regressive way. Moreover, we propose an explicit-alignment approach to align noisy frames with the generated speech tokens to improve the robustness and scalability to different input lengths. Different from other methods that leverage multiple stages to generate speech codes, we leverage a single-stage speech generation approach based on the TF-Codec neural codec to achieve high speech quality with low latency. Extensive results on both synthetic and real-recorded test set show its superiority over data-driven approaches in terms of noise robustness and temporal speech coherence.【4】 Transformer-based Autoencoder with ID Constraint for Unsupervised Anomalous Sound Detection标题：基于ID约束的Transformer式自动编码器无监督异常音检测链接：https://arxiv.org/abs/2310.08950作者：Jian Guan,Youde Liu,Qiuqiang Kong,Feiyang Xiao,Qiaoxi Zhu,Jiantong Tian,Wenwu Wang机构：College of Computer Science and Technology, Harbin Engineering, University, Harbin, Heilongjiang, China., Bytedance, Shanghai, China., Centre for Audio, Acoustics and Vibration, University of Technology, Sydney, Ultimo, NSW , NSW, Australia.备注：Accepted by EURASIP Journal on Audio, Speech, and Music Processing摘要：无监督异常声音检测（ASD）的目的是在只有正常声音数据可用时检测设备的未知异常声音。基于自动编码器（AE）和自监督学习的方法是两种主流方法。然而，基于AE的方法可能受到限制，因为从正常声音学习的特征也可以与异常声音相适应，从而降低了模型从声音中检测异常的能力。自监督方法并不总是稳定的，并且即使对于相同类型的机器也会有不同的表现。此外，异常声音可能是短暂的，使其更难与正常声音区分开来。本文提出了一种基于ID约束的Transformer-based自动编码器（IDC-transAE）架构与加权异常分数计算的无监督ASD。通过引入一个简单的机器ID分类器来学习同一机器类型的分布差异，并增强模型区分异常声音的能力，机器ID被用来约束基于Transformer-based autoencoder（transAE）的潜在空间。此外，加权异常分数计算被引入到突出的异常事件，只出现了很短的时间的异常分数。在DCASE 2020 Challenge Task 2开发数据集上进行的实验证明了该方法的有效性和优越性。摘要：Unsupervised anomalous sound detection (ASD) aims to detect unknown anomalous sounds of devices when only normal sound data is available. The autoencoder (AE) and self-supervised learning based methods are two mainstream methods. However, the AE-based methods could be limited as the feature learned from normal sounds can also fit with anomalous sounds, reducing the ability of the model in detecting anomalies from sound. The self-supervised methods are not always stable and perform differently, even for machines of the same type. In addition, the anomalous sound may be short-lived, making it even harder to distinguish from normal sound. This paper proposes an ID constrained Transformer-based autoencoder (IDC-TransAE) architecture with weighted anomaly score computation for unsupervised ASD. Machine ID is employed to constrain the latent space of the Transformer-based autoencoder (TransAE) by introducing a simple ID classifier to learn the difference in the distribution for the same machine type and enhance the ability of the model in distinguishing anomalous sound. Moreover, weighted anomaly score computation is introduced to highlight the anomaly scores of anomalous events that only appear for a short time. Experiments performed on DCASE 2020 Challenge Task2 development dataset demonstrate the effectiveness and superiority of our proposed method.【5】 Differential Evolution Algorithm based Hyper-Parameters Selection of Convolutional Neural Network for Speech Command Recognition标题：基于微分进化算法的卷积神经网络超参数选择用于语音命令识别链接：https://arxiv.org/abs/2310.08914作者：Sandipan Dhar,Anuvab Sen,Aritra Bandyopadhyay,Nanda Dulal Jana,Arjun Ghosh,Zahra Sarayloo机构：Computer Science and Engineering, National Institute of Technology, Durgapur, West Bengal, India, Electronics and Telecommunication, Indian Institute of Engineering Science and Technology, Shibpur, Howrah, India备注：8 Pages, 7 Figures, 4 Tables, Accepted by the 15th International Joint Conference on Computational Intelligence (IJCCI 2023), November 13-15, 2023, Rome, Italy摘要：语音命令识别（SCR）处理短语音命令的识别，对于包括物联网设备和辅助技术在内的各种应用至关重要。尽管卷积神经网络（CNN）在SCR任务中表现出了良好的前景，但它们的功效在很大程度上依赖于超参数选择，而手动选择通常既费力又耗时。本文介绍了一种基于差分进化（DE）算法的CNN超参数选择方法，旨在提高SCR任务的性能。通过对Google Speech Command（GSC）数据集的训练和测试，该方法在语音命令分类方面表现出了有效性。此外，与基于遗传算法的选择和其他深度CNN（DCNN）模型的比较分析突出了所提出的DE算法在SCR任务中CNN的超参数选择中的效率。摘要：Speech Command Recognition (SCR), which deals with identification of short uttered speech commands, is crucial for various applications, including IoT devices and assistive technology. Despite the promise shown by Convolutional Neural Networks (CNNs) in SCR tasks, their efficacy relies heavily on hyper-parameter selection, which is typically laborious and time-consuming when done manually. This paper introduces a hyper-parameter selection method for CNNs based on the Differential Evolution (DE) algorithm, aiming to enhance performance in SCR tasks. Training and testing with the Google Speech Command (GSC) dataset, the proposed approach showed effectiveness in classifying speech commands. Moreover, a comparative analysis with Genetic Algorithm based selections and other deep CNN (DCNN) models highlighted the efficiency of the proposed DE algorithm in hyper-parameter selection for CNNs in SCR tasks.【6】 Learning to Behave Like Clean Speech: Dual-Branch Knowledge Distillation for Noise-Robust Fake Audio Detection标题：学会表现得像干净的语音：用于抗噪声伪音频检测的双分支知识提取链接：https://arxiv.org/abs/2310.08869作者：Cunhang Fan,Mingming Ding,Jianhua Tao,Ruibo Fu,Jiangyan Yi,Zhengqi Wen,Zhao Lv摘要：None摘要：Most research in fake audio detection (FAD) focuses on improving performance on standard noise-free datasets. However, in actual situations, there is usually noise interference, which will cause significant performance degradation in FAD systems. To improve the noise robustness, we propose a dual-branch knowledge distillation fake audio detection (DKDFAD) method. Specifically, a parallel data flow of the clean teacher branch and the noisy student branch is designed, and interactive fusion and response-based teacher-student paradigms are proposed to guide the training of noisy data from the data distribution and decision-making perspectives. In the noise branch, speech enhancement is first introduced for denoising, which reduces the interference of strong noise. The proposed interactive fusion combines denoising features and noise features to reduce the impact of speech distortion and seek consistency with the data distribution of clean branch. The teacher-student paradigm maps the student's decision space to the teacher's decision space, making noisy speech behave as clean. In addition, a joint training method is used to optimize the two branches to achieve global optimality. Experimental results based on multiple datasets show that the proposed method performs well in noisy environments and maintains performance in cross-dataset experiments.【7】 CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models标题：COPA：解决音频语言模型中成分推理的差距链接：https://arxiv.org/abs/2310.08753作者：Sreyan Ghosh,Ashish Seth,Sonal Kumar,Utkarsh Tyagi,Chandra Kiran Evuru,S. Ramaneswaran,S. Sakshi,Oriol Nieto,Ramani Duraiswami,Dinesh Manocha机构：♠University of Maryland, College Park, USA, ⊡Indian Institute Of Technology, Madras, India, ♥NVIDIA, Bangalore, India, ♣UMass, Amherst, USA, ♦Adobe, USA备注：Pre-print under review摘要：音频的一个基本特征是其组成性质。使用对比方法训练的音频语言模型（ALM）（例如，CLAP）学习音频和语言模态之间的共享表示，其在许多下游应用（包括zero-shot音频分类、音频检索等）中提高了性能。然而，这些模型有效地执行组合推理的能力在很大程度上尚未探索，需要进行额外的研究。在本文中，我们提出了CompA，一个集合的两个专家注释的基准与大多数的真实世界的音频样本，以评估组合推理在ALM。我们提出的CompA-order评估ALM如何理解音频中声学事件的顺序或发生，CompA-attribute评估声学事件的属性绑定。来自任一基准的实例由两个音频-字幕对组成，其中两个音频具有相同的声学事件但具有不同的合成。ALM的评估取决于它将正确的音频与正确的字幕匹配得有多好。使用这个基准测试，我们首先表明，目前的ALMs执行仅略好于随机机会，从而挣扎与组合推理。接下来，我们提出了CompA-CLAP，我们使用一种新的学习方法来微调CLAP，以提高其组合推理能力。为了训练CompA-CLAP，我们首先提出了对具有成分感知的硬否定的对比训练的改进，从而允许更集中的训练。接下来，我们提出了一种新的模块化对比损失，帮助模型学习细粒度的组成理解，并克服公开可用的组成音频的严重稀缺性。CompA-CLAP在CompA基准测试中显著改善了我们所有的基线模型，表明其优越的组合推理能力。摘要：A fundamental characteristic of audio is its compositional nature. Audio-language models (ALMs) trained using a contrastive approach (e.g., CLAP) that learns a shared representation between audio and language modalities have improved performance in many downstream applications, including zero-shot audio classification, audio retrieval, etc. However, the ability of these models to effectively perform compositional reasoning remains largely unexplored and necessitates additional research. In this paper, we propose CompA, a collection of two expert-annotated benchmarks with a majority of real-world audio samples, to evaluate compositional reasoning in ALMs. Our proposed CompA-order evaluates how well an ALM understands the order or occurrence of acoustic events in audio, and CompA-attribute evaluates attribute binding of acoustic events. An instance from either benchmark consists of two audio-caption pairs, where both audios have the same acoustic events but with different compositions. An ALM is evaluated on how well it matches the right audio to the right caption. Using this benchmark, we first show that current ALMs perform only marginally better than random chance, thereby struggling with compositional reasoning. Next, we propose CompA-CLAP, where we fine-tune CLAP using a novel learning method to improve its compositional reasoning abilities. To train CompA-CLAP, we first propose improvements to contrastive training with composition-aware hard negatives, allowing for more focused training. Next, we propose a novel modular contrastive loss that helps the model learn fine-grained compositional understanding and overcomes the acute scarcity of openly available compositional audios. CompA-CLAP significantly improves over all our baseline models on the CompA benchmark, indicating its superior compositional reasoning capabilities.【8】 Toward Joint Language Modeling for Speech Units and Text标题：面向语音单元和语篇的联合语言建模链接：https://arxiv.org/abs/2310.08715作者：Ju-Chieh Chou,Chung-Ming Chien,Wei-Ning Hsu,Karen Livescu,Arun Babu,Alexis Conneau,Alexei Baevski,Michael Auli机构：Toyota Technological Institute at Chicago ,Meta AI ,OpenAI ,Character AI备注：EMNLP findings 2023摘要：言语和语篇是人类语言的两种主要形式。多年来，研究界一直专注于将语音映射到文本或反之亦然。然而，在语言建模领域，很少有人对它们进行联合建模。有鉴于此，我们探讨了语音单元和文本的联合语言建模。具体来说，我们比较不同的语音标记器将连续语音信号转换为离散单元，并使用不同的方法来构建混合语音文本数据。我们引入自动度量来评估联合LM如何混合语音和文本。我们还微调LM在下游口语理解（SLU）任务与不同的方式（语音或文本），并测试其性能，以评估模型的学习共享表示。我们的研究结果表明，通过将语音单元和文本与我们提出的混合技术混合，联合LM在SLU任务上的语音基线上有所改善，并显示出zero-shot跨模态可转移性。摘要：Speech and text are two major forms of human language. The research community has been focusing on mapping speech to text or vice versa for many years. However, in the field of language modeling, very little effort has been made to model them jointly. In light of this, we explore joint language modeling for speech units and text. Specifically, we compare different speech tokenizers to transform continuous speech signals into discrete units and use different methods to construct mixed speech-text data. We introduce automatic metrics to evaluate how well the joint LM mixes speech and text. We also fine-tune the LM on downstream spoken language understanding (SLU) tasks with different modalities (speech or text) and test its performance to assess the model's learning of shared representations. Our results show that by mixing speech units and text with our proposed mixing techniques, the joint LM improves over a speech-only baseline on SLU tasks and shows zero-shot cross-modal transferability.
每日语音速递
