今日论文合集：cs.SD语音15篇，eess.AS音频处理15篇。
cs.SD语音
【1】 ImageBind-LLM: Multi-modality Instruction Tuning标题：ImageBind-LLM：多通道指令调优链接：https://arxiv.org/abs/2309.03905
作者：Jiaming Han,Renrui Zhang,Wenqi Shao,Peng Gao,Peng Xu,Han Xiao,Kaipeng Zhang,Chris Liu,Song Wen,Ziyu Guo,Xudong Lu,Shuai Ren,Yafei Wen,Xiaoxin Chen,Xiangyu Yue,Hongsheng Li,Yu Qiao备注：Code is available at this https URL摘要：我们提出了ImageBind-LLM，一种通过ImageBind-LLM的多模态指令调优方法。现有的工作主要集中在语言和图像指令调整，不同的是，我们的ImageBind-LLM可以响应多模态条件，包括音频，3D点云，视频，以及他们的嵌入空间算法，只有图像-文本对齐训练。在训练过程中，我们采用一个可学习的绑定网络来对齐LLaMA和ImageBind的图像编码器之间的嵌入空间。然后，由绑定网络变换的图像特征被添加到LLaMA中的所有层的单词令牌，其通过无注意力和零初始化的门控机制逐步注入视觉指令。在ImageBind的联合嵌入的帮助下，简单的图像-文本训练使我们的模型能够表现出卓越的多模态指令跟随能力。在推理过程中，多模态输入被馈送到相应的ImageBind编码器中，并由所提出的视觉缓存模型进行处理，以进一步增强跨模态嵌入。免训练缓存模型从ImageBind提取的300万个图像特征中检索，有效地减轻了训练-推理模态差异。值得注意的是，使用我们的方法，ImageBind-LLM可以响应各种形式的指令，并表现出显着的语言生成质量。代码发布于https://github.com/OpenGVLab/LLaMA-Adapter。摘要：We present ImageBind-LLM, a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. Existing works mainly focus on language and image instruction tuning, different from which, our ImageBind-LLM can respond to multi-modality conditions, including audio, 3D point clouds, video, and their embedding-space arithmetic by only image-text alignment training. During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder. Then, the image features transformed by the bind network are added to word tokens of all layers in LLaMA, which progressively injects visual instructions via an attention-free and zero-initialized gating mechanism. Aided by the joint embedding of ImageBind, the simple image-text training enables our model to exhibit superior multi-modality instruction-following capabilities. During inference, the multi-modality inputs are fed into the corresponding ImageBind encoders, and processed by a proposed visual cache model for further cross-modal embedding enhancement. The training-free cache model retrieves from three million image features extracted by ImageBind, which effectively mitigates the training-inference modality discrepancy. Notably, with our approach, ImageBind-LLM can respond to instructions of diverse modalities and demonstrate significant language generation quality. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.
【2】 Zero-Shot Audio Captioning via Audibility Guidance标题：通过可听度制导实现Zero-Shot音频字幕链接：https://arxiv.org/abs/2309.03884
作者：Tal Shaharabany,Ariel Shaulov,Lior Wolf摘要：音频字幕的任务在本质上类似于诸如图像和视频字幕的任务。然而，它受到的关注要少得多。我们提出了三个desiderata的音频字幕-（i）流畅的生成文本，（ii）忠实的生成的文本输入音频，和有点相关的（iii）可听度，这是能够被感知的质量只基于音频。我们的方法是zero-shot方法，即，我们不学习字幕。相反，字幕作为推理过程发生，涉及对应于三个期望质量的三个网络：（i）大型语言模型，在我们的情况下，为了方便起见，GPT-2，（ii）提供音频文件和文本之间匹配分数的模型，为此，我们使用称为ImageBind的多模态匹配网络，以及（iii）文本分类器，使用我们通过指示GPT-4自动收集的数据集训练，提示旨在指导可听和听不见的句子的生成。我们提出了我们的结果AudioCap数据集，表明可听度指导显着提高性能相比，基线，这缺乏这个目标。摘要：The task of audio captioning is similar in essence to tasks such as image and video captioning. However, it has received much less attention. We propose three desiderata for captioning audio -- (i) fluency of the generated text, (ii) faithfulness of the generated text to the input audio, and the somewhat related (iii) audibility, which is the quality of being able to be perceived based only on audio. Our method is a zero-shot method, i.e., we do not learn to perform captioning. Instead, captioning occurs as an inference process that involves three networks that correspond to the three desired qualities: (i) A Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A model that provides a matching score between an audio file and a text, for which we use a multimodal matching network called ImageBind, and (iii) A text classifier, trained using a dataset we collected automatically by instructing GPT-4 with prompts designed to direct the generation of both audible and inaudible sentences. We present our results on the AudioCap dataset, demonstrating that audibility guidance significantly enhances performance compared to the baseline, which lacks this objective.
【3】 Spiking Structured State Space Model for Monaural Speech Enhancement标题：单声道语音增强的尖峰结构状态空间模型链接：https://arxiv.org/abs/2309.03641
作者：Yu Du,Xu Liu,Yansong Chua摘要：语音增强试图从噪声信号中提取干净的语音。传统的深度学习方法面临两个挑战：有效地使用长语音序列中的信息和高计算成本。为了解决这些问题，我们引入了尖峰结构化状态空间模型（Spiking-S4）。这种方法将尖峰神经网络（SNN）的能量效率与结构化状态空间模型（S4）的远程序列建模功能相结合，提供了一种引人注目的解决方案。对DNS挑战和VoiceBank+需求数据集的评估证实，Spiking-S4可与现有的人工神经网络（ANN）方法相媲美，但计算资源更少，如减少的参数和浮点运算（FLOPs）所证明的。摘要：Speech enhancement seeks to extract clean speech from noisy signals. Traditional deep learning methods face two challenges: efficiently using information in long speech sequences and high computational costs. To address these, we introduce the Spiking Structured State Space Model (Spiking-S4). This approach merges the energy efficiency of Spiking Neural Networks (SNN) with the long-range sequence modeling capabilities of Structured State Space Models (S4), offering a compelling solution. Evaluation on the DNS Challenge and VoiceBank+Demand Datasets confirms that Spiking-S4 rivals existing Artificial Neural Network (ANN) methods but with fewer computational resources, as evidenced by reduced parameters and Floating Point Operations (FLOPs).
【4】 Understanding Self-Supervised Learning of Speech Representation via  Invariance and Redundancy Reduction标题：通过不变性和冗余性减少理解语音表示的自监督学习链接：https://arxiv.org/abs/2309.03619
作者：Yusuf Brima,Ulf Krumnack,Simone Pika,Gunther Heidemann备注：6 pages, 1 figure, in submission to ICASSP 2024摘要：目标函数的选择对于从自监督学习中产生高质量的表示至关重要。本文探讨了如何不同配方的巴洛双胞胎（BT）的目标影响下游任务性能的语音数据。我们提出了改进的巴洛双胞胎（MBT）与归一化的潜伏，以加强尺度不变性和评估说话人识别，性别识别和关键词发现任务。我们的研究结果表明，MBT提高了代表性的推广，特别是在有限的目标数据进行微调。这突出了设计目标的重要性，鼓励不变和可转移的表示。我们的分析提供了深入了解如何BT学习目标可以定制，以产生语音表示，当适应新的下游任务时表现出色。这项研究是一个重要的一步，发展可重复使用的自我监督的语音表示。摘要：The choice of the objective function is crucial in emerging high-quality representations from self-supervised learning. This paper investigates how different formulations of the Barlow Twins (BT) objective impact downstream task performance for speech data. We propose Modified Barlow Twins (MBT) with normalized latents to enforce scale-invariance and evaluate on speaker identification, gender recognition and keyword spotting tasks. Our results show MBT improves representation generalization over original BT, especially when fine-tuning with limited target data. This highlights the importance of designing objectives that encourage invariant and transferable representations. Our analysis provides insights into how the BT learning objective can be tailored to produce speech representations that excel when adapted to new downstream tasks. This study is an important step towards developing reusable self-supervised speech representations.
【5】 MVD:A Novel Methodology and Dataset for Acoustic Vehicle Type  Classification标题：MVD：一种新的声学车辆类型分类方法和数据集链接：https://arxiv.org/abs/2309.03544
作者：Mohd Ashhad,Omar Ahmed,Sooraj K. Ambat,Zeeshan Ali Haq,Mansaf Alam摘要：城市人口的增加导致车辆使用量激增，使得交通监控和管理变得不可或缺。声学流量监测（ATM）提供了一个成本效益和有效的替代方案，以更计算昂贵的流量监测方法，如涉及计算机视觉技术。在本文中，我们提出MVD和MVDA：两个开放的数据集用于声学交通监测和车辆类型分类算法的开发，其中包含移动车辆的音频记录。该数据集包含四个类-卡车，汽车，摩托车和无车辆类。此外，我们提出了一种新的和有效的方法来准确地分类这些声信号，使用倒谱和基于频谱的本地和全球的音频功能，和一个多输入神经网络。实验结果表明，我们的方法提高了以前的作品建立的基线，并实现了MVD和MVDA数据集的准确率分别为91.98%和96.66%。最后，通过Android应用程序部署所提出的模型，使其可访问的测试和证明其有效性。摘要：Rising urban populations have led to a surge in vehicle use and made traffic monitoring and management indispensable. Acoustic traffic monitoring (ATM) offers a cost-effective and efficient alternative to more computationally expensive methods of monitoring traffic such as those involving computer vision technologies. In this paper, we present MVD and MVDA: two open datasets for the development of acoustic traffic monitoring and vehicle-type classification algorithms, which contain audio recordings of moving vehicles. The dataset contain four classes- Trucks, Cars, Motorbikes, and a No-vehicle class. Additionally, we propose a novel and efficient way to accurately classify these acoustic signals using cepstrum and spectrum based local and global audio features, and a multi-input neural network. Experimental results show that our methodology improves upon the established baselines of previous works and achieves an accuracy of 91.98% and 96.66% on MVD and MVDA Datasets, respectively. Finally, the proposed model was deployed through an Android application to make it accessible for testing and demonstrate its efficacy.
【6】 Topological fingerprints for audio identification标题：用于音频识别的拓扑指纹链接：https://arxiv.org/abs/2309.03516
作者：Wojciech Reise,Ximena Fernández,Maria Dominguez,Heather A. Harrington,Mariano Beguerisse-Díaz备注：26 pages摘要：我们提出了一种拓扑音频指纹方法，用于鲁棒地识别重复的音轨。我们的方法适用于持久的同源性的音频信号的局部频谱分解，使用从梅尔频谱图计算的过滤立方复合物。通过在本地Betti曲线编码音频内容，我们的拓扑音频指纹能够准确检测时间对齐的音频匹配。实验结果表明，我们的算法在相同的音频内容的轨道检测的准确性，即使受到各种混淆。我们的方法优于现有的方法，涉及拓扑失真的情况下，如时间拉伸和音高变化。摘要：We present a topological audio fingerprinting approach for robustly identifying duplicate audio tracks. Our method applies persistent homology on local spectral decompositions of audio signals, using filtered cubical complexes computed from mel-spectrograms. By encoding the audio content in terms of local Betti curves, our topological audio fingerprints enable accurate detection of time-aligned audio matchings. Experimental results demonstrate the accuracy of our algorithm in the detection of tracks with the same audio content, even when subjected to various obfuscations. Our approach outperforms existing methods in scenarios involving topological distortions, such as time stretching and pitch shifting.
【7】 Cross-domain Sound Recognition for Efficient Underwater Data Analysis标题：用于高效水下数据分析的跨域声音识别链接：https://arxiv.org/abs/2309.03451
作者：Jeongsoo Park,Dong-Gyun Han,Hyoung Sul La,Sangmin Lee,Yoonchang Han,Eun-Jin Yang备注：Accepted to APSIPA 2023摘要：本文提出了一种新的深度学习方法，通过利用在广泛的非水下（空中）声音上训练的模型来分析大量的水下声学数据。认识到在标记大量的水下数据的挑战，我们提出了一个双重的方法来加速这个劳动密集型的过程。  我们的方法的第一部分涉及PCA和UMAP可视化的水下数据使用的特征向量的空中声音识别模型。这使我们能够在二维空间中对数据进行聚类，并监听这些聚类中的点，以了解它们的定义特征。这种创新的方法简化了选择候选标签进行进一步训练的过程。  在第二部分中，我们使用选定的水下数据和非水下数据集训练神经网络模型。我们进行了定量分析，以衡量我们的模型识别气枪的声音，一种常见的水下声音的精度，召回率和F1得分。我们的模型达到的F1得分超过84.3%，证明了我们的方法在分析水声数据的有效性。  本文提出的方法具有显着的潜力，以减少水下数据分析所需的劳动量，并开辟了新的可能性，在该领域的跨域数据分析的进一步研究。摘要：This paper presents a novel deep learning approach for analyzing massive underwater acoustic data by leveraging a model trained on a broad spectrum of non-underwater (aerial) sounds. Recognizing the challenge in labeling vast amounts of underwater data, we propose a two-fold methodology to accelerate this labor-intensive procedure.  The first part of our approach involves PCA and UMAP visualization of the underwater data using the feature vectors of an aerial sound recognition model. This enables us to cluster the data in a two dimensional space and listen to points within these clusters to understand their defining characteristics. This innovative method simplifies the process of selecting candidate labels for further training.  In the second part, we train a neural network model using both the selected underwater data and the non-underwater dataset. We conducted a quantitative analysis to measure the precision, recall, and F1 score of our model for recognizing airgun sounds, a common type of underwater sound. The F1 score achieved by our model exceeded 84.3%, demonstrating the effectiveness of our approach in analyzing underwater acoustic data.  The methodology presented in this paper holds significant potential to reduce the amount of labor required in underwater data analysis and opens up new possibilities for further research in the field of cross-domain data analysis.
【8】 RoDia: A New Dataset for Romanian Dialect Identification from Speech标题：Rodia：一种新的罗马尼亚方言语音识别数据集链接：https://arxiv.org/abs/2309.03378
作者：Codrut Rotaru,Nicolae-Catalin Ristea,Radu Tudor Ionescu摘要：方言识别是语音处理和语言技术中的一项重要任务，它促进了语音识别、说话人验证等各种应用。虽然大多数研究都致力于广泛使用的语言中的方言识别，但对低资源语言（如罗马尼亚语）中的方言识别的关注有限。为了解决这一研究空白，我们介绍RoDia，第一个数据集罗马尼亚方言识别语音。RoDia数据集包括来自罗马尼亚五个不同地区的语音样本的各种汇编，涵盖城市和农村环境，总计2小时的手动注释语音数据。随着我们的数据集，我们引入了一组竞争模型，作为未来研究的基线。得分最高的模型实现了59.83%的宏观F1得分和62.08%的微观F1得分，表明任务具有挑战性。因此，我们相信RoDia是一个宝贵的资源，将刺激研究，旨在解决罗马尼亚方言识别的挑战。我们在https://github.com/codrut2/RoDia公开发布我们的数据集和代码。摘要：Dialect identification is a critical task in speech processing and language technology, enhancing various applications such as speech recognition, speaker verification, and many others. While most research studies have been dedicated to dialect identification in widely spoken languages, limited attention has been given to dialect identification in low-resource languages, such as Romanian. To address this research gap, we introduce RoDia, the first dataset for Romanian dialect identification from speech. The RoDia dataset includes a varied compilation of speech samples from five distinct regions of Romania, covering both urban and rural environments, totaling 2 hours of manually annotated speech data. Along with our dataset, we introduce a set of competitive models to be used as baselines for future research. The top scoring model achieves a macro F1 score of 59.83% and a micro F1 score of 62.08%, indicating that the task is challenging. We thus believe that RoDia is a valuable resource that will stimulate research aiming to address the challenges of Romanian dialect identification. We publicly release our dataset and code at https://github.com/codrut2/RoDia.
【9】 Highly Controllable Diffusion-based Any-to-Any Voice Conversion Model  with Frame-level Prosody Feature标题：具有帧级韵律特征的基于高度可控扩散的任意语音转换模型链接：https://arxiv.org/abs/2309.03364
作者：Kyungguen Byun,Sunkuk Moon,Erik Visser备注：5 pages, 3 figures, submitted to ICASSP 2024摘要：我们提出了一个高度可控的语音操作系统，可以同时执行任何到任何语音转换（VC）和韵律调制。最先进的VC系统可以传输句子级别的特征，例如说话者、情感和说话风格。然而，操纵帧级韵律，如音高，能量和语速，仍然是具有挑战性的。我们提出的模型利用帧级韵律功能，有效地转移这些属性。具体地，音高和能量轨迹被集成在韵律调节模块中，然后与扬声器和内容嵌入一起被馈送到基于扩散的解码器，生成转换后的语音梅尔频谱图。为了调整说话速率，我们的系统包括一个基于自我监督模型的后处理步骤，允许改进的可控性。所提出的模型表现出可比的语音质量和提高的清晰度相比，SOTA的方法。它可以覆盖基频（F0）、能量和速度调制的变化范围，同时保持转换后的语音质量。摘要：We propose a highly controllable voice manipulation system that can perform any-to-any voice conversion (VC) and prosody modulation simultaneously. State-of-the-art VC systems can transfer sentence-level characteristics such as speaker, emotion, and speaking style. However, manipulating the frame-level prosody, such as pitch, energy and speaking rate, still remains challenging. Our proposed model utilizes a frame-level prosody feature to effectively transfer such properties. Specifically, pitch and energy trajectories are integrated in a prosody conditioning module and then fed alongside speaker and contents embeddings to a diffusion-based decoder generating a converted speech mel-spectrogram. To adjust the speaking rate, our system includes a self-supervised model based post-processing step which allows improved controllability. The proposed model showed comparable speech quality and improved intelligibility compared to a SOTA approach. It can cover a varying range of fundamental frequency (F0), energy and speed modulation while maintaining converted speech quality.
【10】 Parameter Efficient Audio Captioning With Faithful Guidance Using  Audio-text Shared Latent Representation标题：基于音文共享隐含表示的参数高效音频字幕链接：https://arxiv.org/abs/2309.03340
作者：Arvind Krishna Sridhar,Yinyi Guo,Erik Visser,Rehana Mahfuz备注：5 pages, 5 tables, 1 figure摘要：针对多模态文本生成任务开发预训练的Transformer架构已经有了重要的研究。尽管性能有所提高，但这些模型经常被过度参数化，因此会出现幻觉和大内存占用，这使得它们难以在边缘设备上部署。在本文中，我们解决了这两个问题的应用自动音频字幕。首先，我们提出了一种数据增强技术，用于产生幻觉的音频字幕，并表明，相似性的基础上的音频文本共享潜在空间是适合检测幻觉。然后，我们提出了一个参数高效的推理时间忠实解码算法，使较小的音频字幕模型的性能等同于用更多的数据训练的较大模型。在波束解码步骤期间，较小的模型利用音频-文本共享潜在表示来将生成的文本与对应的输入音频在语义上对齐。通过引入贪婪滚动中间波束和音频剪辑的潜在表示投影之间的余弦相似性，将忠实引导引入波束概率。我们显示了我们的算法的功效，基准数据集和评估所提出的方案对基线使用传统的音频字幕和语义相似性指标，同时说明性能和复杂性之间的权衡。摘要：There has been significant research on developing pretrained transformer architectures for multimodal-to-text generation tasks. Albeit performance improvements, such models are frequently overparameterized, hence suffer from hallucination and large memory footprint making them challenging to deploy on edge devices. In this paper, we address both these issues for the application of automated audio captioning. First, we propose a data augmentation technique for generating hallucinated audio captions and show that similarity based on an audio-text shared latent space is suitable for detecting hallucination. Then, we propose a parameter efficient inference time faithful decoding algorithm that enables smaller audio captioning models with performance equivalent to larger models trained with more data. During the beam decoding step, the smaller model utilizes an audio-text shared latent representation to semantically align the generated text with corresponding input audio. Faithful guidance is introduced into the beam probability by incorporating the cosine similarity between latent representation projections of greedy rolled out intermediate beams and audio clip. We show the efficacy of our algorithm on benchmark datasets and evaluate the proposed scheme against baselines using conventional audio captioning and semantic similarity metrics while illustrating tradeoffs between performance and complexity.
【11】 Presenting the SWTC: A Symbolic Corpus of Themes from John Williams'  Star Wars Episodes I-IX标题：呈现SWTC：约翰·威廉姆斯《星球大战》第一至第九集的象征性主题语料库链接：https://arxiv.org/abs/2309.03298
作者：Claire Arthur,Frank Lehman,John McNamara备注：Corpus report (5000 words)摘要：本文提出了一个新的象征性语料库的音乐主题，从完整的星球大战三部曲（第一至第九集）的约翰威廉姆斯。语料库文件以多种格式（.krn、.sib和.musicxml）提供，包括旋律、和声和正式信息。《星球大战主题语料库》（SWTC）共包含64个独特的、反复出现的、具有象征意义的主题和主题，通常被称为主旋律。通过这个语料库，我们还介绍了一个新的单调的非功能性和声编码标准，**harte，基于哈特（2005年，2010年）。本报告详细介绍了动机，描述了转录和编码过程，并提供了一些简要的统计数据。虽然规模相对较小，但SWTC代表了20世纪最多产和最有影响力的作曲家之一的统一集合，以及一般研究不足的电影和多媒体音乐材料子集。我们希望SWTC将提供深入了解约翰·威廉姆斯的作曲风格，以及证明与其他主题语料库从电影和超越比较有用。摘要：This paper presents a new symbolic corpus of musical themes from the complete Star Wars trilogies (Episodes I-IX) by John Williams. The corpus files are made available in multiple formats (.krn, .sib, and .musicxml) and include melodic, harmonic, and formal information. The Star Wars Thematic Corpus (SWTC) contains a total of 64 distinctive, recurring, and symbolically meaningful themes and motifs, commonly referred to as leitmotifs. Through this corpus we also introduce a new humdrum standard for non-functional harmony encodings, **harte, based on Harte (2005, 2010). This report details the motivation, describes the transcription and encoding processes, and provides some brief summary statistics. While relatively small in scale, the SWTC represents a unified collection from one of the most prolific and influential composers of the 20th century, and the under-studied subset of film and multimedia musical material in general. We hope the SWTC will provide insights into John Williams' compositional style, as well as prove useful in comparisons against other thematic corpora from film and beyond.
【12】 Implicit Design Choices and Their Impact on Emotion Recognition Model  Development and Evaluation标题：内隐设计选择及其对情绪识别模型开发和评估的影响链接：https://arxiv.org/abs/2309.03238
作者：Mimansa Jaiswal备注：PhD Thesis摘要：情感识别是一项复杂的任务，由于情感的感知和产生都具有内在的主观性。情感的主观性在开发准确和鲁棒的计算模型方面提出了重大挑战。本论文探讨情绪识别的关键方面，首先收集各种数据集，占情绪产生的心理因素。  为了应对非代表性训练数据的挑战，这项工作收集了多模态应激情绪数据集，该数据集在数据收集过程中引入受控压力源，以更好地代表现实世界对情绪产生的影响。为了解决标签主观性的问题，本研究全面分析了数据增强技术和注释方案如何影响情绪感知和注释者标签。它通过采用对抗性网络来进一步处理自然的混杂变量和变化，以在模型训练期间从学习的情绪表示中隔离压力等关键因素。为了解决敏感人口统计变量泄漏的问题，这项工作利用对抗学习从多模态编码中剥离敏感人口统计信息。此外，它提出了优化的社会学评估指标，以符合成本效益，现实世界的模型测试需求。  这项研究通过对数据集、标签、建模、人口统计和成员变量编码和评估中的挑战进行多方面的研究，推进了稳健、实用的情感识别。已经为成本效益高、可推广的情绪识别模型奠定了基础，这些模型不太可能编码敏感的人口统计信息。摘要：Emotion recognition is a complex task due to the inherent subjectivity in both the perception and production of emotions. The subjectivity of emotions poses significant challenges in developing accurate and robust computational models. This thesis examines critical facets of emotion recognition, beginning with the collection of diverse datasets that account for psychological factors in emotion production.  To handle the challenge of non-representative training data, this work collects the Multimodal Stressed Emotion dataset, which introduces controlled stressors during data collection to better represent real-world influences on emotion production. To address issues with label subjectivity, this research comprehensively analyzes how data augmentation techniques and annotation schemes impact emotion perception and annotator labels. It further handles natural confounding variables and variations by employing adversarial networks to isolate key factors like stress from learned emotion representations during model training. For tackling concerns about leakage of sensitive demographic variables, this work leverages adversarial learning to strip sensitive demographic information from multimodal encodings. Additionally, it proposes optimized sociological evaluation metrics aligned with cost-effective, real-world needs for model testing.  This research advances robust, practical emotion recognition through multifaceted studies of challenges in datasets, labels, modeling, demographic and membership variable encoding in representations, and evaluation. The groundwork has been laid for cost-effective, generalizable emotion recognition models that are less likely to encode sensitive demographic information.
【13】 Causal Signal-Based DCCRN with Overlapped-Frame Prediction for Online  Speech Enhancement标题：基于因果信号的重叠帧预测DCCRN在线语音增强链接：https://arxiv.org/abs/2309.03684
作者：Julitta Bartolewska,Stanisław Kacprzak,Konrad Kowalczyk备注：None摘要：语音增强的目的是从有噪声的麦克风信号中提高语音信号的质量和可懂度。在许多应用中，关键是实现具有小计算复杂度和关于访问未来信号样本的最小要求的处理（前瞻）。本文提出了基于信号的因果DCCRN，提高了在线单通道语音增强，减少所需的前瞻和网络参数的数量。所提出的修改包括信号的复杂滤波、重叠帧预测的应用、因果卷积和去卷积以及损失函数的修改。实验结果表明，所提出的模型与重叠的信号预测和额外的调整，实现类似或更好的性能比原来的DCCRN在各种语音增强指标，而它减少了约30%的延迟和网络参数的数量。摘要：The aim of speech enhancement is to improve speech signal quality and intelligibility from a noisy microphone signal. In many applications, it is crucial to enable processing with small computational complexity and minimal requirements regarding access to future signal samples (look-ahead). This paper presents signal-based causal DCCRN that improves online single-channel speech enhancement by reducing the required look-ahead and the number of network parameters. The proposed modifications include complex filtering of the signal, application of overlapped-frame prediction, causal convolutions and deconvolutions, and modification of the loss function. Results of performed experiments indicate that the proposed model with overlapped signal prediction and additional adjustments, achieves similar or better performance than the original DCCRN in terms of various speech enhancement metrics, while it reduces the latency and network parameter number by around 30%.
【14】 Simulating room transfer functions between transducers mounted on audio  devices using a modified image source method标题：使用改进的图像源法模拟安装在音频设备上的换能器之间的房间传递函数链接：https://arxiv.org/abs/2309.03486
作者：Zeyu Xu,Adrian Herzog,Alexander Lodermeyer,Emanuël A. P. Habets,Albert G. Prinn备注：The following article has been submitted to the Journal of the Acoustical Society of America (JASA). After it is published, it will be found at this http URL摘要：镜像源法（ISM）由于其易于使用和计算效率，经常被用来模拟房间声学。标准ISM仅限于模拟点源和全向接收器之间的房间脉冲响应。在这项工作中，ISM扩展使用球谐方向性系数，包括由于源和接收器换能器安装在物理设备上，这是通常在实际情况下遇到的声衍射效应。所提出的方法进行了验证，在一个矩形的房间里使用各种扬声器和麦克风配置的有限元模拟。结果表明，所提出的方法的准确性是有关的大小，形状，数量，和房间内的设备的位置。所提出的方法，可以显着减少计算工作量的简化版本，也提出了。所提出的方法及其简化版本可以更准确地模拟房间传递函数比目前可用的图像源的方法，可以帮助语音和声信号处理算法的开发和评估，包括语音增强，声学场景分析，声学参数估计。摘要：The image source method (ISM) is often used to simulate room acoustics due to its ease of use and computational efficiency. The standard ISM is limited to simulations of room impulse responses between point sources and omnidirectional receivers. In this work, the ISM is extended using spherical harmonic directivity coefficients to include acoustic diffraction effects due to source and receiver transducers mounted on physical devices, which are typically encountered in practical situations. The proposed method is verified using finite element simulations of various loudspeaker and microphone configurations in a rectangular room. It is shown that the accuracy of the proposed method is related to the sizes, shapes, number, and positions of the devices inside a room. A simplified version of the proposed method, which can significantly reduce computational effort, is also presented. The proposed method and its simplified version can simulate room transfer functions more accurately than currently available image source methods and can aid the development and evaluation of speech and acoustic signal processing algorithms, including speech enhancement, acoustic scene analysis, and acoustic parameter estimation.
【15】 Leveraging Geometrical Acoustic Simulations of Spatial Room Impulse  Responses for Improved Sound Event Detection and Localization标题：利用空间房间脉冲响应的几何声学模拟改进声事件检测和定位链接：https://arxiv.org/abs/2309.03337
作者：Christopher Ick,Brian McFee备注：5 pages, 3 figures, 3 tables, presented in the Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events 2023 Workshop (DCASE2023)摘要：随着针对声音事件定位和检测（SELD）的任务开发更深和更复杂的模型，对带注释的空间音频数据的需求持续增加。用360美元的视频注释现场记录需要训练有素的注释者花费许多小时，而在运动跟踪实验室内记录事件则受到成本和专业知识的限制。正因为如此，定位模型依赖于空间房间脉冲响应（SRIR）数据集形式的相对有限量的空间音频数据，这限制了越来越深的基于神经网络的方法的进展。在这项工作中，我们证明，模拟几何声学可以提供一个有吸引力的解决方案这个问题。我们使用模拟几何声学生成一个新的SRIR数据集，可以训练SELD模型，以提供类似的性能，一个真正的SRIR数据集。此外，我们演示了使用模拟数据来增强现有的数据集，提高了由国家的最先进的SELD模型设定的基准。我们探讨了几何声学模拟的定位和事件检测的潜力和局限性。我们还提出了进一步的研究，以验证这种方法的局限性，以及进一步的方法来生成SELD任务的合成数据，而不需要记录更多的数据。摘要：As deeper and more complex models are developed for the task of sound event localization and detection (SELD), the demand for annotated spatial audio data continues to increase. Annotating field recordings with 360$^{\circ}$ video takes many hours from trained annotators, while recording events within motion-tracked laboratories are bounded by cost and expertise. Because of this, localization models rely on a relatively limited amount of spatial audio data in the form of spatial room impulse response (SRIR) datasets, which limits the progress of increasingly deep neural network based approaches. In this work, we demonstrate that simulated geometrical acoustics can provide an appealing solution to this problem. We use simulated geometrical acoustics to generate a novel SRIR dataset that can train a SELD model to provide similar performance to that of a real SRIR dataset. Furthermore, we demonstrate using simulated data to augment existing datasets, improving on benchmarks set by state of the art SELD models. We explore the potential and limitations of geometric acoustic simulation for localization and event detection. We also propose further studies to verify the limitations of this method, as well as further methods to generate synthetic data for SELD tasks without the need to record more data.
eess.AS音频处理
【1】 Causal Signal-Based DCCRN with Overlapped-Frame Prediction for Online  Speech Enhancement标题：基于因果信号的重叠帧预测DCCRN在线语音增强链接：https://arxiv.org/abs/2309.03684
作者：Julitta Bartolewska,Stanisław Kacprzak,Konrad Kowalczyk备注：None摘要：语音增强的目的是从有噪声的麦克风信号中提高语音信号的质量和可懂度。在许多应用中，关键是实现具有小计算复杂度和关于访问未来信号样本的最小要求的处理（前瞻）。本文提出了基于信号的因果DCCRN，提高了在线单通道语音增强，减少所需的前瞻和网络参数的数量。所提出的修改包括信号的复杂滤波、重叠帧预测的应用、因果卷积和去卷积以及损失函数的修改。实验结果表明，所提出的模型与重叠的信号预测和额外的调整，实现类似或更好的性能比原来的DCCRN在各种语音增强指标，而它减少了约30%的延迟和网络参数的数量。摘要：The aim of speech enhancement is to improve speech signal quality and intelligibility from a noisy microphone signal. In many applications, it is crucial to enable processing with small computational complexity and minimal requirements regarding access to future signal samples (look-ahead). This paper presents signal-based causal DCCRN that improves online single-channel speech enhancement by reducing the required look-ahead and the number of network parameters. The proposed modifications include complex filtering of the signal, application of overlapped-frame prediction, causal convolutions and deconvolutions, and modification of the loss function. Results of performed experiments indicate that the proposed model with overlapped signal prediction and additional adjustments, achieves similar or better performance than the original DCCRN in terms of various speech enhancement metrics, while it reduces the latency and network parameter number by around 30%.
【2】 Simulating room transfer functions between transducers mounted on audio  devices using a modified image source method标题：使用改进的图像源法模拟安装在音频设备上的换能器之间的房间传递函数链接：https://arxiv.org/abs/2309.03486
作者：Zeyu Xu,Adrian Herzog,Alexander Lodermeyer,Emanuël A. P. Habets,Albert G. Prinn备注：The following article has been submitted to the Journal of the Acoustical Society of America (JASA). After it is published, it will be found at this http URL摘要：镜像源法（ISM）由于其易于使用和计算效率，经常被用来模拟房间声学。标准ISM仅限于模拟点源和全向接收器之间的房间脉冲响应。在这项工作中，ISM扩展使用球谐方向性系数，包括由于源和接收器换能器安装在物理设备上，这是通常在实际情况下遇到的声衍射效应。所提出的方法进行了验证，在一个矩形的房间里使用各种扬声器和麦克风配置的有限元模拟。结果表明，所提出的方法的准确性是有关的大小，形状，数量，和房间内的设备的位置。所提出的方法，可以显着减少计算工作量的简化版本，也提出了。所提出的方法及其简化版本可以更准确地模拟房间传递函数比目前可用的图像源的方法，可以帮助语音和声信号处理算法的开发和评估，包括语音增强，声学场景分析，声学参数估计。摘要：The image source method (ISM) is often used to simulate room acoustics due to its ease of use and computational efficiency. The standard ISM is limited to simulations of room impulse responses between point sources and omnidirectional receivers. In this work, the ISM is extended using spherical harmonic directivity coefficients to include acoustic diffraction effects due to source and receiver transducers mounted on physical devices, which are typically encountered in practical situations. The proposed method is verified using finite element simulations of various loudspeaker and microphone configurations in a rectangular room. It is shown that the accuracy of the proposed method is related to the sizes, shapes, number, and positions of the devices inside a room. A simplified version of the proposed method, which can significantly reduce computational effort, is also presented. The proposed method and its simplified version can simulate room transfer functions more accurately than currently available image source methods and can aid the development and evaluation of speech and acoustic signal processing algorithms, including speech enhancement, acoustic scene analysis, and acoustic parameter estimation.
【3】 Leveraging Geometrical Acoustic Simulations of Spatial Room Impulse  Responses for Improved Sound Event Detection and Localization标题：利用空间房间脉冲响应的几何声学模拟改进声事件检测和定位链接：https://arxiv.org/abs/2309.03337
作者：Christopher Ick,Brian McFee备注：5 pages, 3 figures, 3 tables, presented in the Proceedings of the 8th Detection and Classification of Acoustic Scenes and Events 2023 Workshop (DCASE2023)摘要：随着针对声音事件定位和检测（SELD）的任务开发更深和更复杂的模型，对带注释的空间音频数据的需求持续增加。用360美元的视频注释现场记录需要训练有素的注释者花费许多小时，而在运动跟踪实验室内记录事件则受到成本和专业知识的限制。正因为如此，定位模型依赖于空间房间脉冲响应（SRIR）数据集形式的相对有限量的空间音频数据，这限制了越来越深的基于神经网络的方法的进展。在这项工作中，我们证明，模拟几何声学可以提供一个有吸引力的解决方案这个问题。我们使用模拟几何声学生成一个新的SRIR数据集，可以训练SELD模型，以提供类似的性能，一个真正的SRIR数据集。此外，我们演示了使用模拟数据来增强现有的数据集，提高了由国家的最先进的SELD模型设定的基准。我们探讨了几何声学模拟的定位和事件检测的潜力和局限性。我们还提出了进一步的研究，以验证这种方法的局限性，以及进一步的方法来生成SELD任务的合成数据，而不需要记录更多的数据。摘要：As deeper and more complex models are developed for the task of sound event localization and detection (SELD), the demand for annotated spatial audio data continues to increase. Annotating field recordings with 360$^{\circ}$ video takes many hours from trained annotators, while recording events within motion-tracked laboratories are bounded by cost and expertise. Because of this, localization models rely on a relatively limited amount of spatial audio data in the form of spatial room impulse response (SRIR) datasets, which limits the progress of increasingly deep neural network based approaches. In this work, we demonstrate that simulated geometrical acoustics can provide an appealing solution to this problem. We use simulated geometrical acoustics to generate a novel SRIR dataset that can train a SELD model to provide similar performance to that of a real SRIR dataset. Furthermore, we demonstrate using simulated data to augment existing datasets, improving on benchmarks set by state of the art SELD models. We explore the potential and limitations of geometric acoustic simulation for localization and event detection. We also propose further studies to verify the limitations of this method, as well as further methods to generate synthetic data for SELD tasks without the need to record more data.
【4】 ImageBind-LLM: Multi-modality Instruction Tuning标题：ImageBind-LLM：多通道指令调优链接：https://arxiv.org/abs/2309.03905
作者：Jiaming Han,Renrui Zhang,Wenqi Shao,Peng Gao,Peng Xu,Han Xiao,Kaipeng Zhang,Chris Liu,Song Wen,Ziyu Guo,Xudong Lu,Shuai Ren,Yafei Wen,Xiaoxin Chen,Xiangyu Yue,Hongsheng Li,Yu Qiao备注：Code is available at this https URL摘要：我们提出了ImageBind-LLM，一种通过ImageBind-LLM的多模态指令调优方法。现有的工作主要集中在语言和图像指令调整，不同的是，我们的ImageBind-LLM可以响应多模态条件，包括音频，3D点云，视频，以及他们的嵌入空间算法，只有图像-文本对齐训练。在训练过程中，我们采用一个可学习的绑定网络来对齐LLaMA和ImageBind的图像编码器之间的嵌入空间。然后，由绑定网络变换的图像特征被添加到LLaMA中的所有层的单词令牌，其通过无注意力和零初始化的门控机制逐步注入视觉指令。在ImageBind的联合嵌入的帮助下，简单的图像-文本训练使我们的模型能够表现出卓越的多模态指令跟随能力。在推理过程中，多模态输入被馈送到相应的ImageBind编码器中，并由所提出的视觉缓存模型进行处理，以进一步增强跨模态嵌入。免训练缓存模型从ImageBind提取的300万个图像特征中检索，有效地减轻了训练-推理模态差异。值得注意的是，使用我们的方法，ImageBind-LLM可以响应各种形式的指令，并表现出显着的语言生成质量。代码发布于https://github.com/OpenGVLab/LLaMA-Adapter。摘要：We present ImageBind-LLM, a multi-modality instruction tuning method of large language models (LLMs) via ImageBind. Existing works mainly focus on language and image instruction tuning, different from which, our ImageBind-LLM can respond to multi-modality conditions, including audio, 3D point clouds, video, and their embedding-space arithmetic by only image-text alignment training. During training, we adopt a learnable bind network to align the embedding space between LLaMA and ImageBind's image encoder. Then, the image features transformed by the bind network are added to word tokens of all layers in LLaMA, which progressively injects visual instructions via an attention-free and zero-initialized gating mechanism. Aided by the joint embedding of ImageBind, the simple image-text training enables our model to exhibit superior multi-modality instruction-following capabilities. During inference, the multi-modality inputs are fed into the corresponding ImageBind encoders, and processed by a proposed visual cache model for further cross-modal embedding enhancement. The training-free cache model retrieves from three million image features extracted by ImageBind, which effectively mitigates the training-inference modality discrepancy. Notably, with our approach, ImageBind-LLM can respond to instructions of diverse modalities and demonstrate significant language generation quality. Code is released at https://github.com/OpenGVLab/LLaMA-Adapter.
【5】 Zero-Shot Audio Captioning via Audibility Guidance标题：通过可听度制导实现Zero-Shot音频字幕链接：https://arxiv.org/abs/2309.03884
作者：Tal Shaharabany,Ariel Shaulov,Lior Wolf摘要：音频字幕的任务在本质上类似于诸如图像和视频字幕的任务。然而，它受到的关注要少得多。我们提出了三个desiderata的音频字幕-（i）流畅的生成文本，（ii）忠实的生成的文本输入音频，和有点相关的（iii）可听度，这是能够被感知的质量只基于音频。我们的方法是zero-shot方法，即，我们不学习字幕。相反，字幕作为推理过程发生，涉及对应于三个期望质量的三个网络：（i）大型语言模型，在我们的情况下，为了方便起见，GPT-2，（ii）提供音频文件和文本之间匹配分数的模型，为此，我们使用称为ImageBind的多模态匹配网络，以及（iii）文本分类器，使用我们通过指示GPT-4自动收集的数据集训练，提示旨在指导可听和听不见的句子的生成。我们提出了我们的结果AudioCap数据集，表明可听度指导显着提高性能相比，基线，这缺乏这个目标。摘要：The task of audio captioning is similar in essence to tasks such as image and video captioning. However, it has received much less attention. We propose three desiderata for captioning audio -- (i) fluency of the generated text, (ii) faithfulness of the generated text to the input audio, and the somewhat related (iii) audibility, which is the quality of being able to be perceived based only on audio. Our method is a zero-shot method, i.e., we do not learn to perform captioning. Instead, captioning occurs as an inference process that involves three networks that correspond to the three desired qualities: (i) A Large Language Model, in our case, for reasons of convenience, GPT-2, (ii) A model that provides a matching score between an audio file and a text, for which we use a multimodal matching network called ImageBind, and (iii) A text classifier, trained using a dataset we collected automatically by instructing GPT-4 with prompts designed to direct the generation of both audible and inaudible sentences. We present our results on the AudioCap dataset, demonstrating that audibility guidance significantly enhances performance compared to the baseline, which lacks this objective.
【6】 Spiking Structured State Space Model for Monaural Speech Enhancement标题：单声道语音增强的尖峰结构状态空间模型链接：https://arxiv.org/abs/2309.03641
作者：Yu Du,Xu Liu,Yansong Chua摘要：语音增强试图从噪声信号中提取干净的语音。传统的深度学习方法面临两个挑战：有效地使用长语音序列中的信息和高计算成本。为了解决这些问题，我们引入了尖峰结构化状态空间模型（Spiking-S4）。这种方法将尖峰神经网络（SNN）的能量效率与结构化状态空间模型（S4）的远程序列建模功能相结合，提供了一种引人注目的解决方案。对DNS挑战和VoiceBank+需求数据集的评估证实，Spiking-S4可与现有的人工神经网络（ANN）方法相媲美，但计算资源更少，如减少的参数和浮点运算（FLOPs）所证明的。摘要：Speech enhancement seeks to extract clean speech from noisy signals. Traditional deep learning methods face two challenges: efficiently using information in long speech sequences and high computational costs. To address these, we introduce the Spiking Structured State Space Model (Spiking-S4). This approach merges the energy efficiency of Spiking Neural Networks (SNN) with the long-range sequence modeling capabilities of Structured State Space Models (S4), offering a compelling solution. Evaluation on the DNS Challenge and VoiceBank+Demand Datasets confirms that Spiking-S4 rivals existing Artificial Neural Network (ANN) methods but with fewer computational resources, as evidenced by reduced parameters and Floating Point Operations (FLOPs).
【7】 Understanding Self-Supervised Learning of Speech Representation via  Invariance and Redundancy Reduction标题：通过不变性和冗余性减少理解语音表示的自监督学习链接：https://arxiv.org/abs/2309.03619
作者：Yusuf Brima,Ulf Krumnack,Simone Pika,Gunther Heidemann备注：6 pages, 1 figure, in submission to ICASSP 2024摘要：目标函数的选择对于从自监督学习中产生高质量的表示至关重要。本文探讨了如何不同配方的巴洛双胞胎（BT）的目标影响下游任务性能的语音数据。我们提出了改进的巴洛双胞胎（MBT）与归一化的潜伏，以加强尺度不变性和评估说话人识别，性别识别和关键词发现任务。我们的研究结果表明，MBT提高了代表性的推广，特别是在有限的目标数据进行微调。这突出了设计目标的重要性，鼓励不变和可转移的表示。我们的分析提供了深入了解如何BT学习目标可以定制，以产生语音表示，当适应新的下游任务时表现出色。这项研究是一个重要的一步，发展可重复使用的自我监督的语音表示。摘要：The choice of the objective function is crucial in emerging high-quality representations from self-supervised learning. This paper investigates how different formulations of the Barlow Twins (BT) objective impact downstream task performance for speech data. We propose Modified Barlow Twins (MBT) with normalized latents to enforce scale-invariance and evaluate on speaker identification, gender recognition and keyword spotting tasks. Our results show MBT improves representation generalization over original BT, especially when fine-tuning with limited target data. This highlights the importance of designing objectives that encourage invariant and transferable representations. Our analysis provides insights into how the BT learning objective can be tailored to produce speech representations that excel when adapted to new downstream tasks. This study is an important step towards developing reusable self-supervised speech representations.
【8】 MVD:A Novel Methodology and Dataset for Acoustic Vehicle Type  Classification标题：MVD：一种新的声学车辆类型分类方法和数据集链接：https://arxiv.org/abs/2309.03544
作者：Mohd Ashhad,Omar Ahmed,Sooraj K. Ambat,Zeeshan Ali Haq,Mansaf Alam摘要：城市人口的增加导致车辆使用量激增，使得交通监控和管理变得不可或缺。声学流量监测（ATM）提供了一个成本效益和有效的替代方案，以更计算昂贵的流量监测方法，如涉及计算机视觉技术。在本文中，我们提出MVD和MVDA：两个开放的数据集用于声学交通监测和车辆类型分类算法的开发，其中包含移动车辆的音频记录。该数据集包含四个类-卡车，汽车，摩托车和无车辆类。此外，我们提出了一种新的和有效的方法来准确地分类这些声信号，使用倒谱和基于频谱的本地和全球的音频功能，和一个多输入神经网络。实验结果表明，我们的方法提高了以前的作品建立的基线，并实现了MVD和MVDA数据集的准确率分别为91.98%和96.66%。最后，通过Android应用程序部署所提出的模型，使其可访问的测试和证明其有效性。摘要：Rising urban populations have led to a surge in vehicle use and made traffic monitoring and management indispensable. Acoustic traffic monitoring (ATM) offers a cost-effective and efficient alternative to more computationally expensive methods of monitoring traffic such as those involving computer vision technologies. In this paper, we present MVD and MVDA: two open datasets for the development of acoustic traffic monitoring and vehicle-type classification algorithms, which contain audio recordings of moving vehicles. The dataset contain four classes- Trucks, Cars, Motorbikes, and a No-vehicle class. Additionally, we propose a novel and efficient way to accurately classify these acoustic signals using cepstrum and spectrum based local and global audio features, and a multi-input neural network. Experimental results show that our methodology improves upon the established baselines of previous works and achieves an accuracy of 91.98% and 96.66% on MVD and MVDA Datasets, respectively. Finally, the proposed model was deployed through an Android application to make it accessible for testing and demonstrate its efficacy.
【9】 Topological fingerprints for audio identification标题：用于音频识别的拓扑指纹链接：https://arxiv.org/abs/2309.03516
作者：Wojciech Reise,Ximena Fernández,Maria Dominguez,Heather A. Harrington,Mariano Beguerisse-Díaz备注：26 pages摘要：我们提出了一种拓扑音频指纹方法，用于鲁棒地识别重复的音轨。我们的方法适用于持久的同源性的音频信号的局部频谱分解，使用从梅尔频谱图计算的过滤立方复合物。通过在本地Betti曲线编码音频内容，我们的拓扑音频指纹能够准确检测时间对齐的音频匹配。实验结果表明，我们的算法在相同的音频内容的轨道检测的准确性，即使受到各种混淆。我们的方法优于现有的方法，涉及拓扑失真的情况下，如时间拉伸和音高变化。摘要：We present a topological audio fingerprinting approach for robustly identifying duplicate audio tracks. Our method applies persistent homology on local spectral decompositions of audio signals, using filtered cubical complexes computed from mel-spectrograms. By encoding the audio content in terms of local Betti curves, our topological audio fingerprints enable accurate detection of time-aligned audio matchings. Experimental results demonstrate the accuracy of our algorithm in the detection of tracks with the same audio content, even when subjected to various obfuscations. Our approach outperforms existing methods in scenarios involving topological distortions, such as time stretching and pitch shifting.
【10】 Cross-domain Sound Recognition for Efficient Underwater Data Analysis标题：用于高效水下数据分析的跨域声音识别链接：https://arxiv.org/abs/2309.03451
作者：Jeongsoo Park,Dong-Gyun Han,Hyoung Sul La,Sangmin Lee,Yoonchang Han,Eun-Jin Yang备注：Accepted to APSIPA 2023摘要：本文提出了一种新的深度学习方法，通过利用在广泛的非水下（空中）声音上训练的模型来分析大量的水下声学数据。认识到在标记大量的水下数据的挑战，我们提出了一个双重的方法来加速这个劳动密集型的过程。  我们的方法的第一部分涉及PCA和UMAP可视化的水下数据使用的特征向量的空中声音识别模型。这使我们能够在二维空间中对数据进行聚类，并监听这些聚类中的点，以了解它们的定义特征。这种创新的方法简化了选择候选标签进行进一步训练的过程。  在第二部分中，我们使用选定的水下数据和非水下数据集训练神经网络模型。我们进行了定量分析，以衡量我们的模型识别气枪的声音，一种常见的水下声音的精度，召回率和F1得分。我们的模型达到的F1得分超过84.3%，证明了我们的方法在分析水声数据的有效性。  本文提出的方法具有显着的潜力，以减少水下数据分析所需的劳动量，并开辟了新的可能性，在该领域的跨域数据分析的进一步研究。摘要：This paper presents a novel deep learning approach for analyzing massive underwater acoustic data by leveraging a model trained on a broad spectrum of non-underwater (aerial) sounds. Recognizing the challenge in labeling vast amounts of underwater data, we propose a two-fold methodology to accelerate this labor-intensive procedure.  The first part of our approach involves PCA and UMAP visualization of the underwater data using the feature vectors of an aerial sound recognition model. This enables us to cluster the data in a two dimensional space and listen to points within these clusters to understand their defining characteristics. This innovative method simplifies the process of selecting candidate labels for further training.  In the second part, we train a neural network model using both the selected underwater data and the non-underwater dataset. We conducted a quantitative analysis to measure the precision, recall, and F1 score of our model for recognizing airgun sounds, a common type of underwater sound. The F1 score achieved by our model exceeded 84.3%, demonstrating the effectiveness of our approach in analyzing underwater acoustic data.  The methodology presented in this paper holds significant potential to reduce the amount of labor required in underwater data analysis and opens up new possibilities for further research in the field of cross-domain data analysis.
【11】 The Role of Communication and Reference Songs in the Mixing Process:  Insights from Professional Mix Engineers标题：交流和参考歌曲在混音过程中的作用：专业混音工程师的见解链接：https://arxiv.org/abs/2309.03404
作者：Soumya Sai Vanka,MAryam Safi,Jean-Baptiste Rolland,György Fazekas备注：Submitted to Journal of Audio Engineering Society in July 2023. Awaiting reviews and acceptance notifications摘要：有效的音乐混音需要技术和创意技巧，但与客户的清晰沟通至关重要。混音工程师必须掌握客户的期望和偏好，并合作以实现所需的声音。对于混音的期望声音的默契通常使用在艺术家和工程师之间交换的参考歌曲和演示混音之类的指南来建立，并且有时使用语义术语来口头表达。本文介绍了一个两阶段的探索性研究的结果，旨在了解专业的搅拌工程师如何与客户互动，并利用他们的反馈来指导搅拌过程。对于第一阶段，半结构化的采访进行了五个混合工程师的目的是收集他们的沟通策略，创造性的过程中，和决策标准的见解。基于这些访谈的推论，设计了一个在线调查问卷，并在第二阶段向22名混合工程师的更大群体进行管理。这项研究的结果揭示了合作，同理心和意图在混合过程中的重要性，并可以为更好地支持这些实践的智能多轨道混合系统的开发提供信息。通过强调这些研究结果的意义，本文有助于越来越多的研究机构的协作性质的音乐制作，并提供了可操作的建议，创新的混合工具的设计和实施。摘要：Effective music mixing requires technical and creative finesse, but clear communication with the client is crucial. The mixing engineer must grasp the client's expectations, and preferences, and collaborate to achieve the desired sound. The tacit agreement for the desired sound of the mix is often established using guides like reference songs and demo mixes exchanged between the artist and the engineer and sometimes verbalised using semantic terms. This paper presents the findings of a two-phased exploratory study aimed at understanding how professional mixing engineers interact with clients and use their feedback to guide the mixing process. For phase one, semi-structured interviews were conducted with five mixing engineers with the aim of gathering insights about their communication strategies, creative processes, and decision-making criteria. Based on the inferences from these interviews, an online questionnaire was designed and administered to a larger group of 22 mixing engineers during the second phase. The results of this study shed light on the importance of collaboration, empathy, and intention in the mixing process, and can inform the development of smart multi-track mixing systems that better support these practices. By highlighting the significance of these findings, this paper contributes to the growing body of research on the collaborative nature of music production and provides actionable recommendations for the design and implementation of innovative mixing tools.
【12】 RoDia: A New Dataset for Romanian Dialect Identification from Speech标题：Rodia：一种新的罗马尼亚方言语音识别数据集链接：https://arxiv.org/abs/2309.03378
作者：Codrut Rotaru,Nicolae-Catalin Ristea,Radu Tudor Ionescu摘要：方言识别是语音处理和语言技术中的一项重要任务，它促进了语音识别、说话人验证等各种应用。虽然大多数研究都致力于广泛使用的语言中的方言识别，但对低资源语言（如罗马尼亚语）中的方言识别的关注有限。为了解决这一研究空白，我们介绍RoDia，第一个数据集罗马尼亚方言识别语音。RoDia数据集包括来自罗马尼亚五个不同地区的语音样本的各种汇编，涵盖城市和农村环境，总计2小时的手动注释语音数据。随着我们的数据集，我们引入了一组竞争模型，作为未来研究的基线。得分最高的模型实现了59.83%的宏观F1得分和62.08%的微观F1得分，表明任务具有挑战性。因此，我们相信RoDia是一个宝贵的资源，将刺激研究，旨在解决罗马尼亚方言识别的挑战。我们在https://github.com/codrut2/RoDia公开发布我们的数据集和代码。摘要：Dialect identification is a critical task in speech processing and language technology, enhancing various applications such as speech recognition, speaker verification, and many others. While most research studies have been dedicated to dialect identification in widely spoken languages, limited attention has been given to dialect identification in low-resource languages, such as Romanian. To address this research gap, we introduce RoDia, the first dataset for Romanian dialect identification from speech. The RoDia dataset includes a varied compilation of speech samples from five distinct regions of Romania, covering both urban and rural environments, totaling 2 hours of manually annotated speech data. Along with our dataset, we introduce a set of competitive models to be used as baselines for future research. The top scoring model achieves a macro F1 score of 59.83% and a micro F1 score of 62.08%, indicating that the task is challenging. We thus believe that RoDia is a valuable resource that will stimulate research aiming to address the challenges of Romanian dialect identification. We publicly release our dataset and code at https://github.com/codrut2/RoDia.
【13】 Highly Controllable Diffusion-based Any-to-Any Voice Conversion Model  with Frame-level Prosody Feature标题：具有帧级韵律特征的基于高度可控扩散的任意语音转换模型链接：https://arxiv.org/abs/2309.03364
作者：Kyungguen Byun,Sunkuk Moon,Erik Visser备注：5 pages, 3 figures, submitted to ICASSP 2024摘要：我们提出了一个高度可控的语音操作系统，可以同时执行任何到任何语音转换（VC）和韵律调制。最先进的VC系统可以传输句子级别的特征，例如说话者、情感和说话风格。然而，操纵帧级韵律，如音高，能量和语速，仍然是具有挑战性的。我们提出的模型利用帧级韵律功能，有效地转移这些属性。具体地，音高和能量轨迹被集成在韵律调节模块中，然后与扬声器和内容嵌入一起被馈送到基于扩散的解码器，生成转换后的语音梅尔频谱图。为了调整说话速率，我们的系统包括一个基于自我监督模型的后处理步骤，允许改进的可控性。所提出的模型表现出可比的语音质量和提高的清晰度相比，SOTA的方法。它可以覆盖基频（F0）、能量和速度调制的变化范围，同时保持转换后的语音质量。摘要：We propose a highly controllable voice manipulation system that can perform any-to-any voice conversion (VC) and prosody modulation simultaneously. State-of-the-art VC systems can transfer sentence-level characteristics such as speaker, emotion, and speaking style. However, manipulating the frame-level prosody, such as pitch, energy and speaking rate, still remains challenging. Our proposed model utilizes a frame-level prosody feature to effectively transfer such properties. Specifically, pitch and energy trajectories are integrated in a prosody conditioning module and then fed alongside speaker and contents embeddings to a diffusion-based decoder generating a converted speech mel-spectrogram. To adjust the speaking rate, our system includes a self-supervised model based post-processing step which allows improved controllability. The proposed model showed comparable speech quality and improved intelligibility compared to a SOTA approach. It can cover a varying range of fundamental frequency (F0), energy and speed modulation while maintaining converted speech quality.
【14】 Presenting the SWTC: A Symbolic Corpus of Themes from John Williams'  Star Wars Episodes I-IX标题：呈现SWTC：约翰·威廉姆斯《星球大战》第一至第九集的象征性主题语料库链接：https://arxiv.org/abs/2309.03298
作者：Claire Arthur,Frank Lehman,John McNamara备注：Corpus report (5000 words)摘要：本文提出了一个新的象征性语料库的音乐主题，从完整的星球大战三部曲（第一至第九集）的约翰威廉姆斯。语料库文件以多种格式（.krn、.sib和.musicxml）提供，包括旋律、和声和正式信息。《星球大战主题语料库》（SWTC）共包含64个独特的、反复出现的、具有象征意义的主题和主题，通常被称为主旋律。通过这个语料库，我们还介绍了一个新的单调的非功能性和声编码标准，**harte，基于哈特（2005年，2010年）。本报告详细介绍了动机，描述了转录和编码过程，并提供了一些简要的统计数据。虽然规模相对较小，但SWTC代表了20世纪最多产和最有影响力的作曲家之一的统一集合，以及一般研究不足的电影和多媒体音乐材料子集。我们希望SWTC将提供深入了解约翰·威廉姆斯的作曲风格，以及证明与其他主题语料库从电影和超越比较有用。摘要：This paper presents a new symbolic corpus of musical themes from the complete Star Wars trilogies (Episodes I-IX) by John Williams. The corpus files are made available in multiple formats (.krn, .sib, and .musicxml) and include melodic, harmonic, and formal information. The Star Wars Thematic Corpus (SWTC) contains a total of 64 distinctive, recurring, and symbolically meaningful themes and motifs, commonly referred to as leitmotifs. Through this corpus we also introduce a new humdrum standard for non-functional harmony encodings, **harte, based on Harte (2005, 2010). This report details the motivation, describes the transcription and encoding processes, and provides some brief summary statistics. While relatively small in scale, the SWTC represents a unified collection from one of the most prolific and influential composers of the 20th century, and the under-studied subset of film and multimedia musical material in general. We hope the SWTC will provide insights into John Williams' compositional style, as well as prove useful in comparisons against other thematic corpora from film and beyond.
【15】 Implicit Design Choices and Their Impact on Emotion Recognition Model  Development and Evaluation标题：内隐设计选择及其对情绪识别模型开发和评估的影响链接：https://arxiv.org/abs/2309.03238
作者：Mimansa Jaiswal备注：PhD Thesis摘要：情感识别是一项复杂的任务，由于情感的感知和产生都具有内在的主观性。情感的主观性在开发准确和鲁棒的计算模型方面提出了重大挑战。本论文探讨情绪识别的关键方面，首先收集各种数据集，占情绪产生的心理因素。  为了应对非代表性训练数据的挑战，这项工作收集了多模态应激情绪数据集，该数据集在数据收集过程中引入受控压力源，以更好地代表现实世界对情绪产生的影响。为了解决标签主观性的问题，本研究全面分析了数据增强技术和注释方案如何影响情绪感知和注释者标签。它通过采用对抗性网络来进一步处理自然的混杂变量和变化，以在模型训练期间从学习的情绪表示中隔离压力等关键因素。为了解决敏感人口统计变量泄漏的问题，这项工作利用对抗学习从多模态编码中剥离敏感人口统计信息。此外，它提出了优化的社会学评估指标，以符合成本效益，现实世界的模型测试需求。  这项研究通过对数据集、标签、建模、人口统计和成员变量编码和评估中的挑战进行多方面的研究，推进了稳健、实用的情感识别。已经为成本效益高、可推广的情绪识别模型奠定了基础，这些模型不太可能编码敏感的人口统计信息。摘要：Emotion recognition is a complex task due to the inherent subjectivity in both the perception and production of emotions. The subjectivity of emotions poses significant challenges in developing accurate and robust computational models. This thesis examines critical facets of emotion recognition, beginning with the collection of diverse datasets that account for psychological factors in emotion production.  To handle the challenge of non-representative training data, this work collects the Multimodal Stressed Emotion dataset, which introduces controlled stressors during data collection to better represent real-world influences on emotion production. To address issues with label subjectivity, this research comprehensively analyzes how data augmentation techniques and annotation schemes impact emotion perception and annotator labels. It further handles natural confounding variables and variations by employing adversarial networks to isolate key factors like stress from learned emotion representations during model training. For tackling concerns about leakage of sensitive demographic variables, this work leverages adversarial learning to strip sensitive demographic information from multimodal encodings. Additionally, it proposes optimized sociological evaluation metrics aligned with cost-effective, real-world needs for model testing.  This research advances robust, practical emotion recognition through multifaceted studies of challenges in datasets, labels, modeling, demographic and membership variable encoding in representations, and evaluation. The groundwork has been laid for cost-effective, generalizable emotion recognition models that are less likely to encode sensitive demographic information.
机器翻译，仅供参考