今日论文合集：cs.SD语音16篇，eess.AS音频处理16篇。
cs.SD语音
【1】 LCANets++: Robust Audio Classification using Multi-layer Neural Networks  with Lateral Competition标题：LCANets++：基于横向竞争的多层神经网络稳健音频分类链接：https://arxiv.org/abs/2308.12882
作者：Sayanton V. Dibbo,Juston S. Moore,Garrett T. Kenyon,Michael A. Teti机构：§ Los Alamos National Laboratory, Los Alamos, NM, USA, †Dartmouth College, Hanover, NH, USA备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：音频分类旨在识别音频信号，包括语音命令或声音事件。然而，当前的音频分类器容易受到扰动和对抗攻击。此外，现实世界的音频分类任务经常遭受有限的标记数据。为了帮助弥合这些差距，先前的工作开发了具有稀疏编码的神经启发卷积神经网络（CNN），通过第一层中的局部竞争算法（LCA）（即，LCANets）用于计算机视觉。LCANets结合了监督学习和无监督学习，减少了对标记样本的依赖。由于听觉皮层也是稀疏的，我们将LCANets扩展到音频识别任务，并引入LCANets++，这是通过LCA在多个层中执行稀疏编码的CNN。我们证明了LCANet ++比标准CNN和LCANet对扰动更鲁棒，例如：背景噪声以及黑盒和白盒攻击，例如，快速梯度符号（FGSM）攻击。摘要：Audio classification aims at recognizing audio signals, including speech commands or sound events. However, current audio classifiers are susceptible to perturbations and adversarial attacks. In addition, real-world audio classification tasks often suffer from limited labeled data. To help bridge these gaps, previous work developed neuro-inspired convolutional neural networks (CNNs) with sparse coding via the Locally Competitive Algorithm (LCA) in the first layer (i.e., LCANets) for computer vision. LCANets learn in a combination of supervised and unsupervised learning, reducing dependency on labeled samples. Motivated by the fact that auditory cortex is also sparse, we extend LCANets to audio recognition tasks and introduce LCANets++, which are CNNs that perform sparse coding in multiple layers via LCA. We demonstrate that LCANets++ are more robust than standard CNNs and LCANets against perturbations, e.g., background noise, as well as black-box and white-box attacks, e.g., evasion and fast gradient sign (FGSM) attacks.
【2】 Towards Automated Animal Density Estimation with Acoustic Spatial  Capture-Recapture标题：基于声学空间捕获-再捕获的动物密度自动估测链接：https://arxiv.org/abs/2308.12859
作者：Yuheng Wang,Juan Ye,David L. Borchers机构：Centre for Research into Ecological and Environmental Modelling, School of Mathematics, and Statistics, University of St Andrews, The Observatory, St Andrews, Fife, KY,LZ, School of Computer Science, University of St Andrews, North Haugh, St Andrews, Fife备注：35 pages, 5 figures摘要：被动声学监测可以是监测声学活跃但难以目视调查的野生动物种群的有效方式。数字记录器使调查员能够以低成本收集大量数据，但在这些数据中识别目标物种的发声并不简单。机器学习（ML）方法通常用于进行识别。它们可以快速处理大量数据，但它们不能检测所有的发声，并且会产生一些假阳性（不是来自目标物种的发声）。现有的野生动物丰度调查方法是专门设计来处理这些错误中的第一个，但目前处理假阳性的方法还没有得到很好的发展。他们不考虑个人发声的特征，其中一些更有可能是假阳性。我们提出了三种声学空间捕获-再捕获推断方法，将ML发声识别的个人水平置信度措施集成到可能性中，从而将ML不确定性集成到推断中。所述方法包括混合模型，其中物种身份是潜在变量。我们通过模拟测试的方法，并发现，在一个场景的基础上，从海南长臂猿的声学数据，其中忽略假阳性的结果在17%的正偏差，我们的方法可以忽略不计的偏差和覆盖概率接近标称95%的水平。摘要：Passive acoustic monitoring can be an effective way of monitoring wildlife populations that are acoustically active but difficult to survey visually. Digital recorders allow surveyors to gather large volumes of data at low cost, but identifying target species vocalisations in these data is non-trivial. Machine learning (ML) methods are often used to do the identification. They can process large volumes of data quickly, but they do not detect all vocalisations and they do generate some false positives (vocalisations that are not from the target species). Existing wildlife abundance survey methods have been designed specifically to deal with the first of these mistakes, but current methods of dealing with false positives are not well-developed. They do not take account of features of individual vocalisations, some of which are more likely to be false positives than others. We propose three methods for acoustic spatial capture-recapture inference that integrate individual-level measures of confidence from ML vocalisation identification into the likelihood and hence integrate ML uncertainty into inference. The methods include a mixture model in which species identity is a latent variable. We test the methods by simulation and find that in a scenario based on acoustic data from Hainan gibbons, in which ignoring false positives results in 17% positive bias, our methods give negligible bias and coverage probabilities that are close to the nominal 95% level.
【3】 Sparks of Large Audio Models: A Survey and Outlook标题：大型音频模型的火花：回顾与展望链接：https://arxiv.org/abs/2308.12792
作者：Siddique Latif,Moazzam Shoukat,Fahad Shamshad,Muhammad Usama,Heriberto Cuayáhuitl,Björn W. Schuller机构：Heriberto Cuay´ahuitl, and Bj¨orn W. Schuller备注：work in progress, Repo URL: this https URL摘要：这篇调查论文全面概述了将大型语言模型应用于音频信号处理领域的最新进展和挑战。音频处理以其多样化的信号表示和广泛的来源-从人类声音到乐器和环境声音-提出了与传统自然语言处理场景中不同的挑战。尽管如此，\texit {Large Audio Models}，由基于变压器的架构所体现，在这一领域中已经显示出显著的功效。通过利用大量数据，这些模型在各种音频任务中表现出了强大的能力，从自动语音识别和文本到语音到音乐生成等。值得注意的是，最近这些基础音频模型，如SeamlessM 4 T，已经开始显示出充当通用翻译器的能力，支持多达100种语言的多种语音任务，而不依赖于单独的任务特定系统。本文深入分析了\textit{基础大型音频模型}的最新方法、其性能基准及其在现实场景中的适用性。我们还强调了当前的局限性，并提供了对\textit{大型音频模型}领域未来潜在研究方向的见解，旨在引发进一步的讨论，从而促进下一代音频处理系统的创新。此外，为了应对这一领域的快速发展，我们将在https://github.com/EmulationAI/awesome-large-audio-models上不断更新相关的存储库，其中包含相关的最新文章及其开源实现。摘要：This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, \textit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding \textit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of \textit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.
【4】 WavMark: Watermarking for Audio Generation标题：WavMark：用于音频生成的水印链接：https://arxiv.org/abs/2308.12770
作者：Guangyu Chen,Yu Wu,Shujie Liu,Tao Liu,Xiaoyong Du,Furu Wei机构：Microsoft Research Asia‡, Renmin University of China†摘要：最近在zero-shot语音合成方面的突破使得能够仅使用几秒钟的录音来模仿说话者的声音，同时保持高水平的真实感。除了潜在的好处，这项强大的技术也带来了显著的风险，包括语音欺诈和说话人模仿。与仅依赖被动方法来检测合成数据的传统方法不同，水印提供了针对这些迫在眉睫的风险的主动且强大的防御机制。本文介绍了一种创新的音频水印框架，编码高达32位的水印在短短1秒的音频片段。该水印对人的感官是不可感知的，并且对各种攻击具有很强的抵抗力。它可以作为一个有效的识别合成的声音，并具有更广泛的应用潜力，在音频版权保护。此外，该框架具有很高的灵活性，允许多个水印段的组合，以实现增强的鲁棒性和扩展的容量。利用10至20秒的音频作为主机，我们的方法表明，平均误码率（BER）为0.48\%，在10个常见的攻击，显着减少了超过2800\%的BER相比，国家的最先进的水印工具。请访问https://aka.ms/wavmark以获取我们工作的演示。摘要：Recent breakthroughs in zero-shot voice synthesis have enabled imitating a speaker's voice using just a few seconds of recording while maintaining a high level of realism. Alongside its potential benefits, this powerful technology introduces notable risks, including voice fraud and speaker impersonation. Unlike the conventional approach of solely relying on passive methods for detecting synthetic data, watermarking presents a proactive and robust defence mechanism against these looming risks. This paper introduces an innovative audio watermarking framework that encodes up to 32 bits of watermark within a mere 1-second audio snippet. The watermark is imperceptible to human senses and exhibits strong resilience against various attacks. It can serve as an effective identifier for synthesized voices and holds potential for broader applications in audio copyright protection. Moreover, this framework boasts high flexibility, allowing for the combination of multiple watermark segments to achieve heightened robustness and expanded capacity. Utilizing 10 to 20-second audio as the host, our approach demonstrates an average Bit Error Rate (BER) of 0.48\% across ten common attacks, a remarkable reduction of over 2800\% in BER compared to the state-of-the-art watermarking tool. See https://aka.ms/wavmark for demos of our work.
【5】 Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion标题：用于DeepFake语音转换的人工智能语音实时检测链接：https://arxiv.org/abs/2308.12734
作者：Jordan J. Bird,Ahmad Lotfi机构：Nottingham Trent University, Nottingham, UK摘要：在语音领域，围绕生成式人工智能的影响越来越大，它可以实现语音克隆和从一个人到另一个人的实时语音转换。这项技术构成了重大的道德威胁，可能导致侵犯隐私和虚假陈述，因此迫切需要实时检测DeepFake语音转换的AI生成的语音。为了解决上述新出现的问题，在本研究中生成了DEEP-VOICE数据集，该数据集由来自八个知名人物的真实人类语音组成，并且使用基于检索的语音转换将他们的语音转换为彼此。呈现为语音是真实的还是AI生成的二元分类问题，通过t检验对时间音频特征的统计分析表明存在显著不同的分布。针对机器学习模型实现超参数优化以识别语音源。通过10倍交叉验证对208个单独的机器学习模型进行训练后，发现极端梯度提升模型可以实现99.3%的平均分类准确率，并且可以实时分类语音，每秒钟语音约为0.004毫秒。本研究生成的所有数据都将公开发布，用于未来的人工智能语音检测研究。摘要：There are growing implications surrounding generative AI in the speech domain that enable voice cloning and real-time voice conversion from one individual to another. This technology poses a significant ethical threat and could lead to breaches of privacy and misrepresentation, thus there is an urgent need for real-time detection of AI-generated speech for DeepFake Voice Conversion. To address the above emerging issues, the DEEP-VOICE dataset is generated in this study, comprised of real human speech from eight well-known figures and their speech converted to one another using Retrieval-based Voice Conversion. Presenting as a binary classification problem of whether the speech is real or AI-generated, statistical analysis of temporal audio features through t-testing reveals that there are significantly different distributions. Hyperparameter optimisation is implemented for machine learning models to identify the source of speech. Following the training of 208 individual machine learning models over 10-fold cross validation, it is found that the Extreme Gradient Boosting model can achieve an average classification accuracy of 99.3% and can classify speech in real-time, at around 0.004 milliseconds given one second of speech. All data generated for this study is released publicly for future research on AI speech detection.
【6】 Whombat: An open-source annotation tool for machine learning development  in bioacoustics标题：WhomBat：一个用于生物声学机器学习开发的开源标注工具链接：https://arxiv.org/abs/2308.12688
作者：Santiago Martinez Balvanera,Oisin Mac Aodha,Matthew J. Weldy,Holly Pringle,Ella Browning,Kate E. Jones机构：|, Centre for Biodiversity and Environment, Research, Department of Genetics, Evolution and Environment, University, College London, London, WC,E ,BT, United Kingdom, School of Informatics, University of, Edinburgh, Edinburgh, EH,AB, United备注：17 pages, 2 figures, 2 tables, to be submitted to Methods in Ecology and Evolution摘要：1.使用机器学习（ML）方法对生物声学记录进行自动分析有可能大大扩展生物多样性监测工作。将ML用于高风险应用（如保护研究）需要以数据为中心的方法，重点是利用相关且具有代表性的仔细注释和策划的评估和培训数据。创建声音记录的注释数据集提出了许多挑战，例如管理具有相关元数据的大量记录，开发可以适应不同生物体的各种发声特征的灵活注释工具，以及解决专家注释者的稀缺性。  2.我们提出了Whombat一个用户友好的，基于浏览器的界面，用于管理录音和注释项目，有几个可视化，探索和注释工具。它使用户能够快速注释，查看和共享注释，以及可视化和评估数据集上的一组机器学习预测。该工具促进了迭代工作流，其中用户注释和机器学习预测反馈，以提高模型性能和注释质量。  3.我们通过展示两个不同的用例来展示Whombat的灵活性：一个项目旨在加强英国蝙蝠保护信托基金（BCT）的自动蝙蝠呼叫识别，以及美国农业部林务局和俄勒冈州立大学研究人员在美国太平洋西北部探索生物声学应用和扩展自动鸟类分类模型的合作努力。  4. Whombat是一个灵活的工具，可以有效地解决生物声学研究注释的挑战。它可以用于个人和协作工作，托管在共享服务器上或远程访问，或在个人计算机上运行，而无需编码技能。摘要：1. Automated analysis of bioacoustic recordings using machine learning (ML) methods has the potential to greatly scale biodiversity monitoring efforts. The use of ML for high-stakes applications, such as conservation research, demands a data-centric approach with a focus on utilizing carefully annotated and curated evaluation and training data that is relevant and representative. Creating annotated datasets of sound recordings presents a number of challenges, such as managing large collections of recordings with associated metadata, developing flexible annotation tools that can accommodate the diverse range of vocalization profiles of different organisms, and addressing the scarcity of expert annotators.  2. We present Whombat a user-friendly, browser-based interface for managing audio recordings and annotation projects, with several visualization, exploration, and annotation tools. It enables users to quickly annotate, review, and share annotations, as well as visualize and evaluate a set of machine learning predictions on a dataset. The tool facilitates an iterative workflow where user annotations and machine learning predictions feedback to enhance model performance and annotation quality.  3. We demonstrate the flexibility of Whombat by showcasing two distinct use cases: an project aimed at enhancing automated UK bat call identification at the Bat Conservation Trust (BCT), and a collaborative effort among the USDA Forest Service and Oregon State University researchers exploring bioacoustic applications and extending automated avian classification models in the Pacific Northwest, USA.  4. Whombat is a flexible tool that can effectively address the challenges of annotation for bioacoustic research. It can be used for individual and collaborative work, hosted on a shared server or accessed remotely, or run on a personal computer without the need for coding skills.
【7】 Naaloss: Rethinking the objective of speech enhancement标题：纳尔罗斯：重新思考语音增强的目标链接：https://arxiv.org/abs/2308.12615
作者：Kuan-Hsun Ho,En-Lun Yu,Jeih-weih Hung,Berlin Chen机构：† Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei, Taiwan, ⋆ Department of Electrical Engineering, National Chi Nan University, Nantou, Taiwan摘要：减少噪声干扰对于现实世界场景中的自动语音识别（ASR）至关重要。然而，大多数单通道语音增强（SE）生成负面影响ASR性能的“处理伪像”。因此，在这项研究中，我们提出了一个噪声和文物意识的损失函数，NAaLoss，从一个新的角度来改善文物的影响。NAaLoss考虑估计的损失、去伪影和噪声忽略，使得学习的SE能够单独地对语音、伪影和噪声进行建模。我们研究了两个SE模型（简单/高级）学习NAaLoss在各种输入场景（干净/嘈杂）下使用两种配置的ASR系统（有/无噪声鲁棒性）。实验表明，NAaLoss显着提高了ASR性能的大多数设置，同时保持质量的SE对感知和可理解性。此外，我们通过波形和频谱图可视化文物，并解释其对ASR的影响。摘要：Reducing noise interference is crucial for automatic speech recognition (ASR) in a real-world scenario. However, most single-channel speech enhancement (SE) generates "processing artifacts" that negatively affect ASR performance. Hence, in this study, we suggest a Noise- and Artifacts-aware loss function, NAaLoss, to ameliorate the influence of artifacts from a novel perspective. NAaLoss considers the loss of estimation, de-artifact, and noise ignorance, enabling the learned SE to individually model speech, artifacts, and noise. We examine two SE models (simple/advanced) learned with NAaLoss under various input scenarios (clean/noisy) using two configurations of the ASR system (with/without noise robustness). Experiments reveal that NAaLoss significantly improves the ASR performance of most setups while preserving the quality of SE toward perception and intelligibility. Furthermore, we visualize artifacts through waveforms and spectrograms, and explain their impact on ASR.
【8】 Emotion-Aligned Contrastive Learning Between Images and Music标题：图像与音乐的情绪对齐对比学习链接：https://arxiv.org/abs/2308.12610
作者：Shanti Stewart,Tiantian Feng,Kleanthis Avramidis,Shrikanth Narayanan机构：University of Southern California, Los Angeles, California, USA备注：Under review摘要：传统的音乐搜索引擎依赖于将自然语言查询与音乐元数据相匹配的检索方法。已经有越来越多的努力来扩展检索方法，以考虑音乐本身的音频特性，使用包括文本，视频和语音的各种形式的查询。大多数方法的目标是匹配一般的音乐语义的输入查询，而只有少数专注于情感品质。我们解决的任务，检索情感相关的音乐从图像查询，提出了一个框架，学习图像和音乐音频之间的情感对齐。我们的方法专注于学习图像和音乐之间的情感对齐的联合嵌入空间。该联合嵌入空间通过情感监督对比学习来学习，使用SupCon损失的适应性跨模态版本。我们直接评估的联合嵌入与跨模态检索任务（图像到音乐和音乐到图像）的基础上的情感标签。此外，我们调查的泛化学习的音乐嵌入自动音乐标记作为下游任务。我们的实验表明，我们的方法成功地对齐图像和音乐，学习的嵌入空间是有效的跨模态检索应用程序。摘要：Traditional music search engines rely on retrieval methods that match natural language queries with music metadata. There have been increasing efforts to expand retrieval methods to consider the audio characteristics of music itself, using queries of various modalities including text, video, and speech. Most approaches aim to match general music semantics to the input queries, while only a few focus on affective qualities. We address the task of retrieving emotionally-relevant music from image queries by proposing a framework for learning an affective alignment between images and music audio. Our approach focuses on learning an emotion-aligned joint embedding space between images and music. This joint embedding space is learned via emotion-supervised contrastive learning, using an adapted cross-modal version of the SupCon loss. We directly evaluate the joint embeddings with cross-modal retrieval tasks (image-to-music and music-to-image) based on emotion labels. In addition, we investigate the generalizability of the learned music embeddings with automatic music tagging as a downstream task. Our experiments show that our approach successfully aligns images and music, and that the learned embedding space is effective for cross-modal retrieval applications.
【9】 Exploiting Time-Frequency Conformers for Music Audio Enhancement标题：利用时频一致性进行音乐音频增强链接：https://arxiv.org/abs/2308.12599
作者：Yunkee Chae,Junghyun Koo,Sungho Lee,Kyogu Lee机构：Seoul National University, Seoul, Republic of Korea备注：Accepted by ACM Multimedia 2023摘要：随着互联网上视频平台的激增，通过移动设备记录音乐表演已经变得司空见惯。然而，这些录音经常遭受诸如噪声和混响的劣化，这对收听体验产生负面影响。因此，对音乐音频增强（从这一点开始称为音乐增强）的必要性已经激增，以增强听觉体验，所述音乐音频增强涉及将降级的音频记录转换成原始的高质量音乐。为了解决这个问题，我们提出了一个音乐增强系统的基础上的Conformer架构，已表现出出色的性能，在语音增强任务。我们的方法探讨了注意机制的一致性，并检查他们的表现，以发现最好的方法，音乐增强任务。我们的实验结果表明，我们提出的模型实现了国家的最先进的单干音乐增强性能。此外，我们的系统可以执行一般的音乐增强与多轨道的混合物，这在以前的工作中还没有被检查。摘要：With the proliferation of video platforms on the internet, recording musical performances by mobile devices has become commonplace. However, these recordings often suffer from degradation such as noise and reverberation, which negatively impact the listening experience. Consequently, the necessity for music audio enhancement (referred to as music enhancement from this point onward), involving the transformation of degraded audio recordings into pristine high-quality music, has surged to augment the auditory experience. To address this issue, we propose a music enhancement system based on the Conformer architecture that has demonstrated outstanding performance in speech enhancement tasks. Our approach explores the attention mechanisms of the Conformer and examines their performance to discover the best approach for the music enhancement task. Our experimental results show that our proposed model achieves state-of-the-art performance on single-stem music enhancement. Furthermore, our system can perform general music enhancement with multi-track mixtures, which has not been examined in previous work.
【10】 MultiPA: a multi-task speech pronunciation assessment system for a  closed and open response scenario标题：MultiPA：一个适用于封闭式和开放式响应场景的多任务语音发音评估系统链接：https://arxiv.org/abs/2308.12490
作者：Yu-Wen Chen,Zhou Yu,Julia Hirschberg机构：Department of Computer Science, Columbia University, United States摘要：自动语音发音评估的设计可以分为封闭式和开放式响应场景，每种场景都有优势和局限性。一个能够在这两种情况下发挥作用的系统可以满足不同的学习需求，并提供更精确和全面的发音技能评估。在这项研究中，我们提出了一个多任务发音评估模型称为MultiPA。MultiPA为基于Kaldi的系统提供了一种替代方案，因为它具有更简单的格式要求和与其他神经网络模型更好的兼容性。与以前的开放式回答系统相比，MultiPA提供了更广泛的评估，包括句子和单词级别的评估。我们的实验结果表明，MultiPA在封闭响应场景中工作时实现了相当的性能，并且在直接用于开放响应时保持了更强大的性能。摘要：The design of automatic speech pronunciation assessment can be categorized into closed and open response scenarios, each with strengths and limitations. A system with the ability to function in both scenarios can cater to diverse learning needs and provide a more precise and holistic assessment of pronunciation skills. In this study, we propose a Multi-task Pronunciation Assessment model called MultiPA. MultiPA provides an alternative to Kaldi-based systems in that it has simpler format requirements and better compatibility with other neural network models. Compared with previous open response systems, MultiPA provides a wider range of evaluations, encompassing assessments at both the sentence and word-level. Our experimental results show that MultiPA achieves comparable performance when working in closed response scenarios and maintains more robust performance when directly used for open responses.
【11】 Attention-Based Acoustic Feature Fusion Network for Depression Detection标题：基于注意力的声学特征融合网络用于抑郁症检测链接：https://arxiv.org/abs/2308.12478
作者：Xiao Xu,Yang Wang,Xinru Wei,Fei Wang,Xizhe Zhang机构：School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, P.R., Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing摘要：抑郁症是一种常见的精神疾病，对个体有着显著的影响，并产生了相当大的社会影响。疾病的复杂性和异质性需要迅速和有效的检测，然而，这带来了困难的挑战。这种情况突出了对改进的检测方法的迫切需求。通过先进的机器学习范式利用听觉数据提出了有前途的研究方向。然而，现有的技术主要依赖于单维特征模型，潜在地忽略了隐藏在各种语音特征中的丰富信息。为了纠正这一点，我们提出了新的基于注意力的声学特征融合网络（ABAFnet）抑郁症检测。ABAFnet将四种不同的声学特征组合成一个全面的深度学习模型，从而有效地集成和混合多层特征。我们提出了一种新的权重调整模块，用于后期融合，通过有效地合成这些功能来提高性能。通过对两个临床语音数据库CNRAC和CS-NRAC的广泛验证，证实了我们方法的有效性，从而在抑郁症检测和亚型分类方面优于以前的方法。进一步的深入分析证实了每个功能的关键作用，并强调了MFCC相关功能在基于语音的抑郁症检测中的重要性。摘要：Depression, a common mental disorder, significantly influences individuals and imposes considerable societal impacts. The complexity and heterogeneity of the disorder necessitate prompt and effective detection, which nonetheless, poses a difficult challenge. This situation highlights an urgent requirement for improved detection methods. Exploiting auditory data through advanced machine learning paradigms presents promising research directions. Yet, existing techniques mainly rely on single-dimensional feature models, potentially neglecting the abundance of information hidden in various speech characteristics. To rectify this, we present the novel Attention-Based Acoustic Feature Fusion Network (ABAFnet) for depression detection. ABAFnet combines four different acoustic features into a comprehensive deep learning model, thereby effectively integrating and blending multi-tiered features. We present a novel weight adjustment module for late fusion that boosts performance by efficaciously synthesizing these features. The effectiveness of our approach is confirmed via extensive validation on two clinical speech databases, CNRAC and CS-NRAC, thereby outperforming previous methods in depression detection and subtype classification. Further in-depth analysis confirms the key role of each feature and highlights the importance of MFCCrelated features in speech-based depression detection.
【12】 An Initial Exploration: Learning to Generate Realistic Audio for Silent  Video标题：初步探索：学习为无声视频生成逼真的音频链接：https://arxiv.org/abs/2308.12408
作者：Matthew Martel,Jackson Wagner摘要：为电影和其他媒体生成逼真的音频效果是一项具有挑战性的任务，目前主要通过称为Foley艺术的物理技术来完成。Foley艺术家用普通物体（例如，拳击手套，碎玻璃）与视频同步，因为它正在播放以生成迷人的音轨。在这项工作中，我们的目标是开发一个基于深度学习的框架，该框架可以做很多相同的事情-观察视频的自然序列，并生成逼真的音频来陪伴它。值得注意的是，我们有理由相信这是可实现的，这是由于以其他输入（例如，音频）为条件的真实音频生成技术的进步。以文本为条件的Wavenet）。我们探索了几种不同的模型架构来完成这一任务，处理先前生成的音频和视频上下文。这些包括深度融合CNN，具有视觉上下文的扩张Wavenet CNN和基于变换器的架构。我们发现，基于变压器的架构产生最有前途的结果，匹配低频视觉模式有效，但未能产生更细致入微的波形。摘要：Generating realistic audio effects for movies and other media is a challenging task that is accomplished today primarily through physical techniques known as Foley art. Foley artists create sounds with common objects (e.g., boxing gloves, broken glass) in time with video as it is playing to generate captivating audio tracks. In this work, we aim to develop a deep-learning based framework that does much the same - observes video in it's natural sequence and generates realistic audio to accompany it. Notably, we have reason to believe this is achievable due to advancements in realistic audio generation techniques conditioned on other inputs (e.g., Wavenet conditioned on text). We explore several different model architectures to accomplish this task that process both previously-generated audio and video context. These include deep-fusion CNN, dilated Wavenet CNN with visual context, and transformer-based architectures. We find that the transformer-based architecture yields the most promising results, matching low-frequencies to visual patterns effectively, but failing to generate more nuanced waveforms.
【13】 AdVerb: Visually Guided Audio Dereverberation标题：副词：视觉引导的音频去混响链接：https://arxiv.org/abs/2308.12370
作者：Sanjoy Chowdhury,Sreyan Ghosh,Subhrajyoti Dasgupta,Anton Ratnarajah,Utkarsh Tyagi,Dinesh Manocha机构：University of Maryland, College Park, Mila and Universit´e de Montr´eal备注：Accepted at ICCV 2023. For project page, see this https URL摘要：我们提出了AdVerb，一种新的视听去混响框架，使用视觉线索，除了混响的声音估计干净的音频。虽然只有音频去混响是一个很好的研究问题，我们的方法采用了互补的视觉模态来执行音频去混响。给定混响声音信号已被记录的环境的图像，AdVerb采用新颖的几何感知跨模态Transformer架构，其捕获场景几何形状和视听跨模态关系以生成复杂的理想比率掩模，其在应用于混响音频时预测干净的声音。通过广泛的定量和定性评估，证明了我们的方法的有效性。我们的方法在三个下游任务上显著优于传统的纯音频和视听基线：语音增强、语音识别和说话人验证，在LibriSpeech测试干净集上的相对改进在18% - 82%的范围内。我们还在AVSpeech数据集上获得了非常令人满意的RT 60错误分数。摘要：We present AdVerb, a novel audio-visual dereverberation framework that uses visual cues in addition to the reverberant sound to estimate clean audio. Although audio-only dereverberation is a well-studied problem, our approach incorporates the complementary visual modality to perform audio dereverberation. Given an image of the environment where the reverberated sound signal has been recorded, AdVerb employs a novel geometry-aware cross-modal transformer architecture that captures scene geometry and audio-visual cross-modal relationship to generate a complex ideal ratio mask, which, when applied to the reverberant audio predicts the clean sound. The effectiveness of our method is demonstrated through extensive quantitative and qualitative evaluations. Our approach significantly outperforms traditional audio-only and audio-visual baselines on three downstream tasks: speech enhancement, speech recognition, and speaker verification, with relative improvements in the range of 18% - 82% on the LibriSpeech test-clean set. We also achieve highly satisfactory RT60 error scores on the AVSpeech dataset.
【14】 Modeling Bends in Popular Music Guitar Tablatures标题：流行音乐吉他表中的弯曲造型链接：https://arxiv.org/abs/2308.12307
作者：Alexandre D'Hooge,Louis Bigo,Ken Déguernel机构：Univ. Lille, CNRS, Centrale Lille, UMR , CRIStAL, F-, Lille, France备注：None摘要：指法记谱法广泛用于流行音乐中，以转录和共享吉他音乐内容。作为一个补充，标准的分数符号，tablatures转录性能手势信息，包括手指位置和各种吉他特定的演奏技术，如幻灯片，锤上/拉下或benders.This本文重点介绍弯曲，使逐步转移的注意到音高，从而规避物理限制的离散指板。在本文中，我们提出了一组25个高层次的功能，计算每个音符的指法，研究如何弯曲的发生可以预测他们的过去和未来的短期背景。实验进行了语料库的932个流行音乐的主音吉他tablatures，并表明，决策树成功地预测弯曲发生的F1得分为0.71和有限的假阳性预测量，展示了有前途的应用程序，以协助安排非吉他音乐吉他tablatures。摘要：Tablature notation is widely used in popular music to transcribe and share guitar musical content. As a complement to standard score notation, tablatures transcribe performance gesture information including finger positions and a variety of guitar-specific playing techniques such as slides, hammer-on/pull-off or bends.This paper focuses on bends, which enable to progressively shift the pitch of a note, therefore circumventing physical limitations of the discrete fretted fingerboard. In this paper, we propose a set of 25 high-level features, computed for each note of the tablature, to study how bend occurrences can be predicted from their past and future short-term context. Experiments are performed on a corpus of 932 lead guitar tablatures of popular music and show that a decision tree successfully predicts bend occurrences with an F1 score of 0.71 anda limited amount of false positive predictions, demonstrating promising applications to assist the arrangement of non-guitar music into guitar tablatures.
【15】 Hybrid noise shaping for audio coding using perfectly overlapped window标题：使用完美重叠窗口的音频编码混合噪声整形链接：https://arxiv.org/abs/2308.12566
作者：Byeongho Jo,Seungkwon Beack机构：Electronics and Telecommunications Research Institute, South Korea备注：accepted to WASPAA (IEEE Workshop on Applications of Signal Processing to Audio and Acoustics) 2023摘要：近年来，已基于并入线性预测译码（LPC）的若干框架来标准化音频译码技术。然而，使用频域LP残差信号对瞬态信号进行编码仍然是一个挑战。为了解决这个问题，时间噪声整形（TNS）可以被适配，尽管它不能被有效地操作，因为在修改的离散余弦变换（MDCT）域中的估计的时间包络伴随有时域混叠（TDA）项。在这项研究中，我们提出了调制复重叠变换为基础的编码框架集成变换编码的激励（TCX）和复杂的LPC为基础的TNS（CTNS）。我们的方法使用一个50%的重叠窗口和切换方案的CTNS，以提高编码效率。此外，提出了基于量化LPC系数使用频率包络信息的子带的目标比特的自适应计算。为了最小化两种模式之间的量化失配，提出了实值和复值的集成量化以及在切换操作期间补偿人工生成的TDA分量的TDA增强方法。所提出的编码框架在客观指标和主观听力测试中表现出优异的性能，从而证明了其低比特率音频编码。摘要：In recent years, audio coding technology has been standardized based on several frameworks that incorporate linear predictive coding (LPC). However, coding the transient signal using frequency-domain LP residual signals remains a challenge. To address this, temporal noise shaping (TNS) can be adapted, although it cannot be effectively operated since the estimated temporal envelope in the modified discrete cosine transform (MDCT) domain is accompanied by the time-domain aliasing (TDA) terms. In this study, we propose the modulated complex lapped transform-based coding framework integrated with transform coded excitation (TCX) and complex LPC-based TNS (CTNS). Our approach uses a 50\% overlap window and switching scheme for the CTNS to improve the coding efficiency. Additionally, an adaptive calculation of the target bits for the sub-bands using the frequency envelope information based on the quantized LPC coefficients is proposed. To minimize the quantization mismatch between both modes, an integrated quantization for real and complex values and a TDA augmentation method that compensates for the artificially generated TDA components during switching operations are proposed. The proposed coding framework shows a superior performance in both objective metrics and subjective listening tests, thereby demonstrating its low bit-rate audio coding.
【16】 UNISOUND System for VoxCeleb Speaker Recognition Challenge 2023标题：2023年VoxCeleb说话人识别挑战赛UNISOUND系统链接：https://arxiv.org/abs/2308.12526
作者：Yu Zheng,Yajun Zhang,Chuanying Niu,Yibin Zhan,Yanhua Long,Dongxing Xu机构：Unisound AI Technology Co., Ltd., Beijing, China, SHNU-Unisound Joint Laboratory of Natural Human-Computer Interaction, Shanghai Normal, University, Shanghai, China摘要：本报告描述了UNISOUND提交的VoxCeleb Speaker Recognition Challenge 2023（VoxSRC 2023）的Track 1和Track 2。我们在轨道1和轨道2上提交了相同的系统，仅使用VoxCeleb 2-dev进行训练。大规模ResNet和RepVGG架构是为应对挑战而开发的。我们提出了一个一致性感知的分数校准方法，利用稳定的音频声纹的相似性分数的一致性度量因子（CMF）。CMF在这一挑战中带来了巨大的性能提升。我们的最终系统融合了六个模型，并在VoxSRC 2023的Track 1和Track 2中获得第一名和第二名。我们提交的最小DCF为0.0855，EER为1.5880%。摘要：This report describes the UNISOUND submission for Track1 and Track2 of VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC 2023). We submit the same system on Track 1 and Track 2, which is trained with only VoxCeleb2-dev. Large-scale ResNet and RepVGG architectures are developed for the challenge. We propose a consistency-aware score calibration method, which leverages the stability of audio voiceprints in similarity score by a Consistency Measure Factor (CMF). CMF brings a huge performance boost in this challenge. Our final system is a fusion of six models and achieves the first place in Track 1 and second place in Track 2 of VoxSRC 2023. The minDCF of our submission is 0.0855 and the EER is 1.5880%.
eess.AS音频处理
【1】 Hybrid noise shaping for audio coding using perfectly overlapped window标题：使用完美重叠窗口的音频编码混合噪声整形链接：https://arxiv.org/abs/2308.12566
作者：Byeongho Jo,Seungkwon Beack机构：Electronics and Telecommunications Research Institute, South Korea备注：accepted to WASPAA (IEEE Workshop on Applications of Signal Processing to Audio and Acoustics) 2023摘要：近年来，已基于并入线性预测译码（LPC）的若干框架来标准化音频译码技术。然而，使用频域LP残差信号对瞬态信号进行编码仍然是一个挑战。为了解决这个问题，时间噪声整形（TNS）可以被适配，尽管它不能被有效地操作，因为在修改的离散余弦变换（MDCT）域中的估计的时间包络伴随有时域混叠（TDA）项。在这项研究中，我们提出了调制复重叠变换为基础的编码框架集成变换编码的激励（TCX）和复杂的LPC为基础的TNS（CTNS）。我们的方法使用一个50%的重叠窗口和切换方案的CTNS，以提高编码效率。此外，提出了基于量化LPC系数使用频率包络信息的子带的目标比特的自适应计算。为了最小化两种模式之间的量化失配，提出了实值和复值的集成量化以及在切换操作期间补偿人工生成的TDA分量的TDA增强方法。所提出的编码框架在客观指标和主观听力测试中表现出优异的性能，从而证明了其低比特率音频编码。摘要：In recent years, audio coding technology has been standardized based on several frameworks that incorporate linear predictive coding (LPC). However, coding the transient signal using frequency-domain LP residual signals remains a challenge. To address this, temporal noise shaping (TNS) can be adapted, although it cannot be effectively operated since the estimated temporal envelope in the modified discrete cosine transform (MDCT) domain is accompanied by the time-domain aliasing (TDA) terms. In this study, we propose the modulated complex lapped transform-based coding framework integrated with transform coded excitation (TCX) and complex LPC-based TNS (CTNS). Our approach uses a 50\% overlap window and switching scheme for the CTNS to improve the coding efficiency. Additionally, an adaptive calculation of the target bits for the sub-bands using the frequency envelope information based on the quantized LPC coefficients is proposed. To minimize the quantization mismatch between both modes, an integrated quantization for real and complex values and a TDA augmentation method that compensates for the artificially generated TDA components during switching operations are proposed. The proposed coding framework shows a superior performance in both objective metrics and subjective listening tests, thereby demonstrating its low bit-rate audio coding.
【2】 UNISOUND System for VoxCeleb Speaker Recognition Challenge 2023标题：2023年VoxCeleb说话人识别挑战赛UNISOUND系统链接：https://arxiv.org/abs/2308.12526
作者：Yu Zheng,Yajun Zhang,Chuanying Niu,Yibin Zhan,Yanhua Long,Dongxing Xu机构：Unisound AI Technology Co., Ltd., Beijing, China, SHNU-Unisound Joint Laboratory of Natural Human-Computer Interaction, Shanghai Normal, University, Shanghai, China摘要：本报告描述了UNISOUND提交的VoxCeleb Speaker Recognition Challenge 2023（VoxSRC 2023）的Track 1和Track 2。我们在轨道1和轨道2上提交了相同的系统，仅使用VoxCeleb 2-dev进行训练。大规模ResNet和RepVGG架构是为应对挑战而开发的。我们提出了一个一致性感知的分数校准方法，利用稳定的音频声纹的相似性分数的一致性度量因子（CMF）。CMF在这一挑战中带来了巨大的性能提升。我们的最终系统融合了六个模型，并在VoxSRC 2023的Track 1和Track 2中获得第一名和第二名。我们提交的最小DCF为0.0855，EER为1.5880%。摘要：This report describes the UNISOUND submission for Track1 and Track2 of VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC 2023). We submit the same system on Track 1 and Track 2, which is trained with only VoxCeleb2-dev. Large-scale ResNet and RepVGG architectures are developed for the challenge. We propose a consistency-aware score calibration method, which leverages the stability of audio voiceprints in similarity score by a Consistency Measure Factor (CMF). CMF brings a huge performance boost in this challenge. Our final system is a fusion of six models and achieves the first place in Track 1 and second place in Track 2 of VoxSRC 2023. The minDCF of our submission is 0.0855 and the EER is 1.5880%.
【3】 LCANets++: Robust Audio Classification using Multi-layer Neural Networks  with Lateral Competition标题：LCANets++：基于横向竞争的多层神经网络稳健音频分类链接：https://arxiv.org/abs/2308.12882
作者：Sayanton V. Dibbo,Juston S. Moore,Garrett T. Kenyon,Michael A. Teti机构：§ Los Alamos National Laboratory, Los Alamos, NM, USA, †Dartmouth College, Hanover, NH, USA备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：音频分类旨在识别音频信号，包括语音命令或声音事件。然而，当前的音频分类器容易受到扰动和对抗攻击。此外，现实世界的音频分类任务经常遭受有限的标记数据。为了帮助弥合这些差距，先前的工作开发了具有稀疏编码的神经启发卷积神经网络（CNN），通过第一层中的局部竞争算法（LCA）（即，LCANets）用于计算机视觉。LCANets结合了监督学习和无监督学习，减少了对标记样本的依赖。由于听觉皮层也是稀疏的，我们将LCANets扩展到音频识别任务，并引入LCANets++，这是通过LCA在多个层中执行稀疏编码的CNN。我们证明了LCANet ++比标准CNN和LCANet对扰动更鲁棒，例如：背景噪声以及黑盒和白盒攻击，例如，快速梯度符号（FGSM）攻击。摘要：Audio classification aims at recognizing audio signals, including speech commands or sound events. However, current audio classifiers are susceptible to perturbations and adversarial attacks. In addition, real-world audio classification tasks often suffer from limited labeled data. To help bridge these gaps, previous work developed neuro-inspired convolutional neural networks (CNNs) with sparse coding via the Locally Competitive Algorithm (LCA) in the first layer (i.e., LCANets) for computer vision. LCANets learn in a combination of supervised and unsupervised learning, reducing dependency on labeled samples. Motivated by the fact that auditory cortex is also sparse, we extend LCANets to audio recognition tasks and introduce LCANets++, which are CNNs that perform sparse coding in multiple layers via LCA. We demonstrate that LCANets++ are more robust than standard CNNs and LCANets against perturbations, e.g., background noise, as well as black-box and white-box attacks, e.g., evasion and fast gradient sign (FGSM) attacks.
【4】 Towards Automated Animal Density Estimation with Acoustic Spatial  Capture-Recapture标题：基于声学空间捕获-再捕获的动物密度自动估测链接：https://arxiv.org/abs/2308.12859
作者：Yuheng Wang,Juan Ye,David L. Borchers机构：Centre for Research into Ecological and Environmental Modelling, School of Mathematics, and Statistics, University of St Andrews, The Observatory, St Andrews, Fife, KY,LZ, School of Computer Science, University of St Andrews, North Haugh, St Andrews, Fife备注：35 pages, 5 figures摘要：被动声学监测可以是监测声学活跃但难以目视调查的野生动物种群的有效方式。数字记录器使调查员能够以低成本收集大量数据，但在这些数据中识别目标物种的发声并不简单。机器学习（ML）方法通常用于进行识别。它们可以快速处理大量数据，但它们不能检测所有的发声，并且会产生一些假阳性（不是来自目标物种的发声）。现有的野生动物丰度调查方法是专门设计来处理这些错误中的第一个，但目前处理假阳性的方法还没有得到很好的发展。他们不考虑个人发声的特征，其中一些更有可能是假阳性。我们提出了三种声学空间捕获-再捕获推断方法，将ML发声识别的个人水平置信度措施集成到可能性中，从而将ML不确定性集成到推断中。所述方法包括混合模型，其中物种身份是潜在变量。我们通过模拟测试的方法，并发现，在一个场景的基础上，从海南长臂猿的声学数据，其中忽略假阳性的结果在17%的正偏差，我们的方法可以忽略不计的偏差和覆盖概率接近标称95%的水平。摘要：Passive acoustic monitoring can be an effective way of monitoring wildlife populations that are acoustically active but difficult to survey visually. Digital recorders allow surveyors to gather large volumes of data at low cost, but identifying target species vocalisations in these data is non-trivial. Machine learning (ML) methods are often used to do the identification. They can process large volumes of data quickly, but they do not detect all vocalisations and they do generate some false positives (vocalisations that are not from the target species). Existing wildlife abundance survey methods have been designed specifically to deal with the first of these mistakes, but current methods of dealing with false positives are not well-developed. They do not take account of features of individual vocalisations, some of which are more likely to be false positives than others. We propose three methods for acoustic spatial capture-recapture inference that integrate individual-level measures of confidence from ML vocalisation identification into the likelihood and hence integrate ML uncertainty into inference. The methods include a mixture model in which species identity is a latent variable. We test the methods by simulation and find that in a scenario based on acoustic data from Hainan gibbons, in which ignoring false positives results in 17% positive bias, our methods give negligible bias and coverage probabilities that are close to the nominal 95% level.
【5】 Sparks of Large Audio Models: A Survey and Outlook标题：大型音频模型的火花：回顾与展望链接：https://arxiv.org/abs/2308.12792
作者：Siddique Latif,Moazzam Shoukat,Fahad Shamshad,Muhammad Usama,Heriberto Cuayáhuitl,Björn W. Schuller机构：Heriberto Cuay´ahuitl, and Bj¨orn W. Schuller备注：work in progress, Repo URL: this https URL摘要：这篇调查论文全面概述了将大型语言模型应用于音频信号处理领域的最新进展和挑战。音频处理以其多样化的信号表示和广泛的来源-从人类声音到乐器和环境声音-提出了与传统自然语言处理场景中不同的挑战。尽管如此，\texit {Large Audio Models}，由基于变压器的架构所体现，在这一领域中已经显示出显著的功效。通过利用大量数据，这些模型在各种音频任务中表现出了强大的能力，从自动语音识别和文本到语音到音乐生成等。值得注意的是，最近这些基础音频模型，如SeamlessM 4 T，已经开始显示出充当通用翻译器的能力，支持多达100种语言的多种语音任务，而不依赖于单独的任务特定系统。本文深入分析了\textit{基础大型音频模型}的最新方法、其性能基准及其在现实场景中的适用性。我们还强调了当前的局限性，并提供了对\textit{大型音频模型}领域未来潜在研究方向的见解，旨在引发进一步的讨论，从而促进下一代音频处理系统的创新。此外，为了应对这一领域的快速发展，我们将在https://github.com/EmulationAI/awesome-large-audio-models上不断更新相关的存储库，其中包含相关的最新文章及其开源实现。摘要：This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, \textit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding \textit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of \textit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.
【6】 WavMark: Watermarking for Audio Generation标题：WavMark：用于音频生成的水印链接：https://arxiv.org/abs/2308.12770
作者：Guangyu Chen,Yu Wu,Shujie Liu,Tao Liu,Xiaoyong Du,Furu Wei机构：Microsoft Research Asia‡, Renmin University of China†摘要：最近在zero-shot语音合成方面的突破使得能够仅使用几秒钟的录音来模仿说话者的声音，同时保持高水平的真实感。除了潜在的好处，这项强大的技术也带来了显著的风险，包括语音欺诈和说话人模仿。与仅依赖被动方法来检测合成数据的传统方法不同，水印提供了针对这些迫在眉睫的风险的主动且强大的防御机制。本文介绍了一种创新的音频水印框架，编码高达32位的水印在短短1秒的音频片段。该水印对人的感官是不可感知的，并且对各种攻击具有很强的抵抗力。它可以作为一个有效的识别合成的声音，并具有更广泛的应用潜力，在音频版权保护。此外，该框架具有很高的灵活性，允许多个水印段的组合，以实现增强的鲁棒性和扩展的容量。利用10至20秒的音频作为主机，我们的方法表明，平均误码率（BER）为0.48\%，在10个常见的攻击，显着减少了超过2800\%的BER相比，国家的最先进的水印工具。请访问https://aka.ms/wavmark以获取我们工作的演示。摘要：Recent breakthroughs in zero-shot voice synthesis have enabled imitating a speaker's voice using just a few seconds of recording while maintaining a high level of realism. Alongside its potential benefits, this powerful technology introduces notable risks, including voice fraud and speaker impersonation. Unlike the conventional approach of solely relying on passive methods for detecting synthetic data, watermarking presents a proactive and robust defence mechanism against these looming risks. This paper introduces an innovative audio watermarking framework that encodes up to 32 bits of watermark within a mere 1-second audio snippet. The watermark is imperceptible to human senses and exhibits strong resilience against various attacks. It can serve as an effective identifier for synthesized voices and holds potential for broader applications in audio copyright protection. Moreover, this framework boasts high flexibility, allowing for the combination of multiple watermark segments to achieve heightened robustness and expanded capacity. Utilizing 10 to 20-second audio as the host, our approach demonstrates an average Bit Error Rate (BER) of 0.48\% across ten common attacks, a remarkable reduction of over 2800\% in BER compared to the state-of-the-art watermarking tool. See https://aka.ms/wavmark for demos of our work.
【7】 Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion标题：用于DeepFake语音转换的人工智能语音实时检测链接：https://arxiv.org/abs/2308.12734
作者：Jordan J. Bird,Ahmad Lotfi机构：Nottingham Trent University, Nottingham, UK摘要：在语音领域，围绕生成式人工智能的影响越来越大，它可以实现语音克隆和从一个人到另一个人的实时语音转换。这项技术构成了重大的道德威胁，可能导致侵犯隐私和虚假陈述，因此迫切需要实时检测DeepFake语音转换的AI生成的语音。为了解决上述新出现的问题，在本研究中生成了DEEP-VOICE数据集，该数据集由来自八个知名人物的真实人类语音组成，并且使用基于检索的语音转换将他们的语音转换为彼此。呈现为语音是真实的还是AI生成的二元分类问题，通过t检验对时间音频特征的统计分析表明存在显著不同的分布。针对机器学习模型实现超参数优化以识别语音源。通过10倍交叉验证对208个单独的机器学习模型进行训练后，发现极端梯度提升模型可以实现99.3%的平均分类准确率，并且可以实时分类语音，每秒钟语音约为0.004毫秒。本研究生成的所有数据都将公开发布，用于未来的人工智能语音检测研究。摘要：There are growing implications surrounding generative AI in the speech domain that enable voice cloning and real-time voice conversion from one individual to another. This technology poses a significant ethical threat and could lead to breaches of privacy and misrepresentation, thus there is an urgent need for real-time detection of AI-generated speech for DeepFake Voice Conversion. To address the above emerging issues, the DEEP-VOICE dataset is generated in this study, comprised of real human speech from eight well-known figures and their speech converted to one another using Retrieval-based Voice Conversion. Presenting as a binary classification problem of whether the speech is real or AI-generated, statistical analysis of temporal audio features through t-testing reveals that there are significantly different distributions. Hyperparameter optimisation is implemented for machine learning models to identify the source of speech. Following the training of 208 individual machine learning models over 10-fold cross validation, it is found that the Extreme Gradient Boosting model can achieve an average classification accuracy of 99.3% and can classify speech in real-time, at around 0.004 milliseconds given one second of speech. All data generated for this study is released publicly for future research on AI speech detection.
【8】 Whombat: An open-source annotation tool for machine learning development  in bioacoustics标题：WhomBat：一个用于生物声学机器学习开发的开源标注工具链接：https://arxiv.org/abs/2308.12688
作者：Santiago Martinez Balvanera,Oisin Mac Aodha,Matthew J. Weldy,Holly Pringle,Ella Browning,Kate E. Jones机构：|, Centre for Biodiversity and Environment, Research, Department of Genetics, Evolution and Environment, University, College London, London, WC,E ,BT, United Kingdom, School of Informatics, University of, Edinburgh, Edinburgh, EH,AB, United备注：17 pages, 2 figures, 2 tables, to be submitted to Methods in Ecology and Evolution摘要：1.使用机器学习（ML）方法对生物声学记录进行自动分析有可能大大扩展生物多样性监测工作。将ML用于高风险应用（如保护研究）需要以数据为中心的方法，重点是利用相关且具有代表性的仔细注释和策划的评估和培训数据。创建声音记录的注释数据集提出了许多挑战，例如管理具有相关元数据的大量记录，开发可以适应不同生物体的各种发声特征的灵活注释工具，以及解决专家注释者的稀缺性。  2.我们提出了Whombat一个用户友好的，基于浏览器的界面，用于管理录音和注释项目，有几个可视化，探索和注释工具。它使用户能够快速注释，查看和共享注释，以及可视化和评估数据集上的一组机器学习预测。该工具促进了迭代工作流，其中用户注释和机器学习预测反馈，以提高模型性能和注释质量。  3.我们通过展示两个不同的用例来展示Whombat的灵活性：一个项目旨在加强英国蝙蝠保护信托基金（BCT）的自动蝙蝠呼叫识别，以及美国农业部林务局和俄勒冈州立大学研究人员在美国太平洋西北部探索生物声学应用和扩展自动鸟类分类模型的合作努力。  4. Whombat是一个灵活的工具，可以有效地解决生物声学研究注释的挑战。它可以用于个人和协作工作，托管在共享服务器上或远程访问，或在个人计算机上运行，而无需编码技能。摘要：1. Automated analysis of bioacoustic recordings using machine learning (ML) methods has the potential to greatly scale biodiversity monitoring efforts. The use of ML for high-stakes applications, such as conservation research, demands a data-centric approach with a focus on utilizing carefully annotated and curated evaluation and training data that is relevant and representative. Creating annotated datasets of sound recordings presents a number of challenges, such as managing large collections of recordings with associated metadata, developing flexible annotation tools that can accommodate the diverse range of vocalization profiles of different organisms, and addressing the scarcity of expert annotators.  2. We present Whombat a user-friendly, browser-based interface for managing audio recordings and annotation projects, with several visualization, exploration, and annotation tools. It enables users to quickly annotate, review, and share annotations, as well as visualize and evaluate a set of machine learning predictions on a dataset. The tool facilitates an iterative workflow where user annotations and machine learning predictions feedback to enhance model performance and annotation quality.  3. We demonstrate the flexibility of Whombat by showcasing two distinct use cases: an project aimed at enhancing automated UK bat call identification at the Bat Conservation Trust (BCT), and a collaborative effort among the USDA Forest Service and Oregon State University researchers exploring bioacoustic applications and extending automated avian classification models in the Pacific Northwest, USA.  4. Whombat is a flexible tool that can effectively address the challenges of annotation for bioacoustic research. It can be used for individual and collaborative work, hosted on a shared server or accessed remotely, or run on a personal computer without the need for coding skills.
【9】 Naaloss: Rethinking the objective of speech enhancement标题：纳尔罗斯：重新思考语音增强的目标链接：https://arxiv.org/abs/2308.12615
作者：Kuan-Hsun Ho,En-Lun Yu,Jeih-weih Hung,Berlin Chen机构：† Department of Computer Science and Information Engineering, National Taiwan Normal University, Taipei, Taiwan, ⋆ Department of Electrical Engineering, National Chi Nan University, Nantou, Taiwan摘要：减少噪声干扰对于现实世界场景中的自动语音识别（ASR）至关重要。然而，大多数单通道语音增强（SE）生成负面影响ASR性能的“处理伪像”。因此，在这项研究中，我们提出了一个噪声和文物意识的损失函数，NAaLoss，从一个新的角度来改善文物的影响。NAaLoss考虑估计的损失、去伪影和噪声忽略，使得学习的SE能够单独地对语音、伪影和噪声进行建模。我们研究了两个SE模型（简单/高级）学习NAaLoss在各种输入场景（干净/嘈杂）下使用两种配置的ASR系统（有/无噪声鲁棒性）。实验表明，NAaLoss显着提高了ASR性能的大多数设置，同时保持质量的SE对感知和可理解性。此外，我们通过波形和频谱图可视化文物，并解释其对ASR的影响。摘要：Reducing noise interference is crucial for automatic speech recognition (ASR) in a real-world scenario. However, most single-channel speech enhancement (SE) generates "processing artifacts" that negatively affect ASR performance. Hence, in this study, we suggest a Noise- and Artifacts-aware loss function, NAaLoss, to ameliorate the influence of artifacts from a novel perspective. NAaLoss considers the loss of estimation, de-artifact, and noise ignorance, enabling the learned SE to individually model speech, artifacts, and noise. We examine two SE models (simple/advanced) learned with NAaLoss under various input scenarios (clean/noisy) using two configurations of the ASR system (with/without noise robustness). Experiments reveal that NAaLoss significantly improves the ASR performance of most setups while preserving the quality of SE toward perception and intelligibility. Furthermore, we visualize artifacts through waveforms and spectrograms, and explain their impact on ASR.
【10】 Emotion-Aligned Contrastive Learning Between Images and Music标题：图像与音乐的情绪对齐对比学习链接：https://arxiv.org/abs/2308.12610
作者：Shanti Stewart,Tiantian Feng,Kleanthis Avramidis,Shrikanth Narayanan机构：University of Southern California, Los Angeles, California, USA备注：Under review摘要：传统的音乐搜索引擎依赖于将自然语言查询与音乐元数据相匹配的检索方法。已经有越来越多的努力来扩展检索方法，以考虑音乐本身的音频特性，使用包括文本，视频和语音的各种形式的查询。大多数方法的目标是匹配一般的音乐语义的输入查询，而只有少数专注于情感品质。我们解决的任务，检索情感相关的音乐从图像查询，提出了一个框架，学习图像和音乐音频之间的情感对齐。我们的方法专注于学习图像和音乐之间的情感对齐的联合嵌入空间。该联合嵌入空间通过情感监督对比学习来学习，使用SupCon损失的适应性跨模态版本。我们直接评估的联合嵌入与跨模态检索任务（图像到音乐和音乐到图像）的基础上的情感标签。此外，我们调查的泛化学习的音乐嵌入自动音乐标记作为下游任务。我们的实验表明，我们的方法成功地对齐图像和音乐，学习的嵌入空间是有效的跨模态检索应用程序。摘要：Traditional music search engines rely on retrieval methods that match natural language queries with music metadata. There have been increasing efforts to expand retrieval methods to consider the audio characteristics of music itself, using queries of various modalities including text, video, and speech. Most approaches aim to match general music semantics to the input queries, while only a few focus on affective qualities. We address the task of retrieving emotionally-relevant music from image queries by proposing a framework for learning an affective alignment between images and music audio. Our approach focuses on learning an emotion-aligned joint embedding space between images and music. This joint embedding space is learned via emotion-supervised contrastive learning, using an adapted cross-modal version of the SupCon loss. We directly evaluate the joint embeddings with cross-modal retrieval tasks (image-to-music and music-to-image) based on emotion labels. In addition, we investigate the generalizability of the learned music embeddings with automatic music tagging as a downstream task. Our experiments show that our approach successfully aligns images and music, and that the learned embedding space is effective for cross-modal retrieval applications.
【11】 Exploiting Time-Frequency Conformers for Music Audio Enhancement标题：利用时频一致性进行音乐音频增强链接：https://arxiv.org/abs/2308.12599
作者：Yunkee Chae,Junghyun Koo,Sungho Lee,Kyogu Lee机构：Seoul National University, Seoul, Republic of Korea备注：Accepted by ACM Multimedia 2023摘要：随着互联网上视频平台的激增，通过移动设备记录音乐表演已经变得司空见惯。然而，这些录音经常遭受诸如噪声和混响的劣化，这对收听体验产生负面影响。因此，对音乐音频增强（从这一点开始称为音乐增强）的必要性已经激增，以增强听觉体验，所述音乐音频增强涉及将降级的音频记录转换成原始的高质量音乐。为了解决这个问题，我们提出了一个音乐增强系统的基础上的Conformer架构，已表现出出色的性能，在语音增强任务。我们的方法探讨了注意机制的一致性，并检查他们的表现，以发现最好的方法，音乐增强任务。我们的实验结果表明，我们提出的模型实现了国家的最先进的单干音乐增强性能。此外，我们的系统可以执行一般的音乐增强与多轨道的混合物，这在以前的工作中还没有被检查。摘要：With the proliferation of video platforms on the internet, recording musical performances by mobile devices has become commonplace. However, these recordings often suffer from degradation such as noise and reverberation, which negatively impact the listening experience. Consequently, the necessity for music audio enhancement (referred to as music enhancement from this point onward), involving the transformation of degraded audio recordings into pristine high-quality music, has surged to augment the auditory experience. To address this issue, we propose a music enhancement system based on the Conformer architecture that has demonstrated outstanding performance in speech enhancement tasks. Our approach explores the attention mechanisms of the Conformer and examines their performance to discover the best approach for the music enhancement task. Our experimental results show that our proposed model achieves state-of-the-art performance on single-stem music enhancement. Furthermore, our system can perform general music enhancement with multi-track mixtures, which has not been examined in previous work.
【12】 MultiPA: a multi-task speech pronunciation assessment system for a  closed and open response scenario标题：MultiPA：一个适用于封闭式和开放式响应场景的多任务语音发音评估系统链接：https://arxiv.org/abs/2308.12490
作者：Yu-Wen Chen,Zhou Yu,Julia Hirschberg机构：Department of Computer Science, Columbia University, United States摘要：自动语音发音评估的设计可以分为封闭式和开放式响应场景，每种场景都有优势和局限性。一个能够在这两种情况下发挥作用的系统可以满足不同的学习需求，并提供更精确和全面的发音技能评估。在这项研究中，我们提出了一个多任务发音评估模型称为MultiPA。MultiPA为基于Kaldi的系统提供了一种替代方案，因为它具有更简单的格式要求和与其他神经网络模型更好的兼容性。与以前的开放式回答系统相比，MultiPA提供了更广泛的评估，包括句子和单词级别的评估。我们的实验结果表明，MultiPA在封闭响应场景中工作时实现了相当的性能，并且在直接用于开放响应时保持了更强大的性能。摘要：The design of automatic speech pronunciation assessment can be categorized into closed and open response scenarios, each with strengths and limitations. A system with the ability to function in both scenarios can cater to diverse learning needs and provide a more precise and holistic assessment of pronunciation skills. In this study, we propose a Multi-task Pronunciation Assessment model called MultiPA. MultiPA provides an alternative to Kaldi-based systems in that it has simpler format requirements and better compatibility with other neural network models. Compared with previous open response systems, MultiPA provides a wider range of evaluations, encompassing assessments at both the sentence and word-level. Our experimental results show that MultiPA achieves comparable performance when working in closed response scenarios and maintains more robust performance when directly used for open responses.
【13】 Attention-Based Acoustic Feature Fusion Network for Depression Detection标题：基于注意力的声学特征融合网络用于抑郁症检测链接：https://arxiv.org/abs/2308.12478
作者：Xiao Xu,Yang Wang,Xinru Wei,Fei Wang,Xizhe Zhang机构：School of Biomedical Engineering and Informatics, Nanjing Medical University, Nanjing, P.R., Early Intervention Unit, Department of Psychiatry, The Affiliated Brain Hospital of Nanjing摘要：抑郁症是一种常见的精神疾病，对个体有着显著的影响，并产生了相当大的社会影响。疾病的复杂性和异质性需要迅速和有效的检测，然而，这带来了困难的挑战。这种情况突出了对改进的检测方法的迫切需求。通过先进的机器学习范式利用听觉数据提出了有前途的研究方向。然而，现有的技术主要依赖于单维特征模型，潜在地忽略了隐藏在各种语音特征中的丰富信息。为了纠正这一点，我们提出了新的基于注意力的声学特征融合网络（ABAFnet）抑郁症检测。ABAFnet将四种不同的声学特征组合成一个全面的深度学习模型，从而有效地集成和混合多层特征。我们提出了一种新的权重调整模块，用于后期融合，通过有效地合成这些功能来提高性能。通过对两个临床语音数据库CNRAC和CS-NRAC的广泛验证，证实了我们方法的有效性，从而在抑郁症检测和亚型分类方面优于以前的方法。进一步的深入分析证实了每个功能的关键作用，并强调了MFCC相关功能在基于语音的抑郁症检测中的重要性。摘要：Depression, a common mental disorder, significantly influences individuals and imposes considerable societal impacts. The complexity and heterogeneity of the disorder necessitate prompt and effective detection, which nonetheless, poses a difficult challenge. This situation highlights an urgent requirement for improved detection methods. Exploiting auditory data through advanced machine learning paradigms presents promising research directions. Yet, existing techniques mainly rely on single-dimensional feature models, potentially neglecting the abundance of information hidden in various speech characteristics. To rectify this, we present the novel Attention-Based Acoustic Feature Fusion Network (ABAFnet) for depression detection. ABAFnet combines four different acoustic features into a comprehensive deep learning model, thereby effectively integrating and blending multi-tiered features. We present a novel weight adjustment module for late fusion that boosts performance by efficaciously synthesizing these features. The effectiveness of our approach is confirmed via extensive validation on two clinical speech databases, CNRAC and CS-NRAC, thereby outperforming previous methods in depression detection and subtype classification. Further in-depth analysis confirms the key role of each feature and highlights the importance of MFCCrelated features in speech-based depression detection.
【14】 An Initial Exploration: Learning to Generate Realistic Audio for Silent  Video标题：初步探索：学习为无声视频生成逼真的音频链接：https://arxiv.org/abs/2308.12408
作者：Matthew Martel,Jackson Wagner摘要：为电影和其他媒体生成逼真的音频效果是一项具有挑战性的任务，目前主要通过称为Foley艺术的物理技术来完成。Foley艺术家用普通物体（例如，拳击手套，碎玻璃）与视频同步，因为它正在播放以生成迷人的音轨。在这项工作中，我们的目标是开发一个基于深度学习的框架，该框架可以做很多相同的事情-观察视频的自然序列，并生成逼真的音频来陪伴它。值得注意的是，我们有理由相信这是可实现的，这是由于以其他输入（例如，音频）为条件的真实音频生成技术的进步。以文本为条件的Wavenet）。我们探索了几种不同的模型架构来完成这一任务，处理先前生成的音频和视频上下文。这些包括深度融合CNN，具有视觉上下文的扩张Wavenet CNN和基于变换器的架构。我们发现，基于变压器的架构产生最有前途的结果，匹配低频视觉模式有效，但未能产生更细致入微的波形。摘要：Generating realistic audio effects for movies and other media is a challenging task that is accomplished today primarily through physical techniques known as Foley art. Foley artists create sounds with common objects (e.g., boxing gloves, broken glass) in time with video as it is playing to generate captivating audio tracks. In this work, we aim to develop a deep-learning based framework that does much the same - observes video in it's natural sequence and generates realistic audio to accompany it. Notably, we have reason to believe this is achievable due to advancements in realistic audio generation techniques conditioned on other inputs (e.g., Wavenet conditioned on text). We explore several different model architectures to accomplish this task that process both previously-generated audio and video context. These include deep-fusion CNN, dilated Wavenet CNN with visual context, and transformer-based architectures. We find that the transformer-based architecture yields the most promising results, matching low-frequencies to visual patterns effectively, but failing to generate more nuanced waveforms.
【15】 AdVerb: Visually Guided Audio Dereverberation标题：副词：视觉引导的音频去混响链接：https://arxiv.org/abs/2308.12370
作者：Sanjoy Chowdhury,Sreyan Ghosh,Subhrajyoti Dasgupta,Anton Ratnarajah,Utkarsh Tyagi,Dinesh Manocha机构：University of Maryland, College Park, Mila and Universit´e de Montr´eal备注：Accepted at ICCV 2023. For project page, see this https URL摘要：我们提出了AdVerb，一种新的视听去混响框架，使用视觉线索，除了混响的声音估计干净的音频。虽然只有音频去混响是一个很好的研究问题，我们的方法采用了互补的视觉模态来执行音频去混响。给定混响声音信号已被记录的环境的图像，AdVerb采用新颖的几何感知跨模态Transformer架构，其捕获场景几何形状和视听跨模态关系以生成复杂的理想比率掩模，其在应用于混响音频时预测干净的声音。通过广泛的定量和定性评估，证明了我们的方法的有效性。我们的方法在三个下游任务上显著优于传统的纯音频和视听基线：语音增强、语音识别和说话人验证，在LibriSpeech测试干净集上的相对改进在18% - 82%的范围内。我们还在AVSpeech数据集上获得了非常令人满意的RT 60错误分数。摘要：We present AdVerb, a novel audio-visual dereverberation framework that uses visual cues in addition to the reverberant sound to estimate clean audio. Although audio-only dereverberation is a well-studied problem, our approach incorporates the complementary visual modality to perform audio dereverberation. Given an image of the environment where the reverberated sound signal has been recorded, AdVerb employs a novel geometry-aware cross-modal transformer architecture that captures scene geometry and audio-visual cross-modal relationship to generate a complex ideal ratio mask, which, when applied to the reverberant audio predicts the clean sound. The effectiveness of our method is demonstrated through extensive quantitative and qualitative evaluations. Our approach significantly outperforms traditional audio-only and audio-visual baselines on three downstream tasks: speech enhancement, speech recognition, and speaker verification, with relative improvements in the range of 18% - 82% on the LibriSpeech test-clean set. We also achieve highly satisfactory RT60 error scores on the AVSpeech dataset.
【16】 Modeling Bends in Popular Music Guitar Tablatures标题：流行音乐吉他表中的弯曲造型链接：https://arxiv.org/abs/2308.12307
作者：Alexandre D'Hooge,Louis Bigo,Ken Déguernel机构：Univ. Lille, CNRS, Centrale Lille, UMR , CRIStAL, F-, Lille, France备注：None摘要：指法记谱法广泛用于流行音乐中，以转录和共享吉他音乐内容。作为一个补充，标准的分数符号，tablatures转录性能手势信息，包括手指位置和各种吉他特定的演奏技术，如幻灯片，锤上/拉下或benders.This本文重点介绍弯曲，使逐步转移的注意到音高，从而规避物理限制的离散指板。在本文中，我们提出了一组25个高层次的功能，计算每个音符的指法，研究如何弯曲的发生可以预测他们的过去和未来的短期背景。实验进行了语料库的932个流行音乐的主音吉他tablatures，并表明，决策树成功地预测弯曲发生的F1得分为0.71和有限的假阳性预测量，展示了有前途的应用程序，以协助安排非吉他音乐吉他tablatures。摘要：Tablature notation is widely used in popular music to transcribe and share guitar musical content. As a complement to standard score notation, tablatures transcribe performance gesture information including finger positions and a variety of guitar-specific playing techniques such as slides, hammer-on/pull-off or bends.This paper focuses on bends, which enable to progressively shift the pitch of a note, therefore circumventing physical limitations of the discrete fretted fingerboard. In this paper, we propose a set of 25 high-level features, computed for each note of the tablature, to study how bend occurrences can be predicted from their past and future short-term context. Experiments are performed on a corpus of 932 lead guitar tablatures of popular music and show that a decision tree successfully predicts bend occurrences with an F1 score of 0.71 anda limited amount of false positive predictions, demonstrating promising applications to assist the arrangement of non-guitar music into guitar tablatures.
机器翻译，仅供参考