今日论文合集：cs.SD语音10篇，eess.AS音频处理12篇。
cs.SD语音
【1】 Proceedings of the 5th International Workshop on Reading Music Systems标题：第五届国际音乐阅读系统研讨会论文集链接：https://arxiv.org/abs/2311.04091
作者：Jorge Calvo-Zaragoza,Alexander Pacha,Elona Shatri备注：Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri摘要：阅读音乐系统国际研讨会（WoRMS）是一个研讨会，试图将开发阅读音乐系统的研究人员（如光学音乐识别领域）与其他研究人员和从业者（如图书馆员或音乐学家）联系起来。研讨会的相关主题包括但不限于：音乐阅读系统;光学音乐识别;数据集和性能评估;乐谱图像处理;作者识别;乐谱的创作、编辑、存储和呈现系统;多模式系统;用于制作书面音乐的新颖音乐输入方法;基于网络的音乐信息检索服务;应用和项目;与书面音乐有关的案件。  这些是2023年11月4日在意大利米兰举行的第五届国际阅读音乐系统研讨会的会议记录。摘要：The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music.  These are the proceedings of the 5th International Workshop on Reading Music Systems, held in Milan, Italy on Nov. 4th 2023.
【2】 Can CLIP Help Sound Source Localization?标题：CLIP能帮助声源本地化吗？链接：https://arxiv.org/abs/2311.04066
作者：Sooyoung Park,Arda Senocak,Joon Son Chung备注：WACV 2024摘要：大规模预训练的图像-文本模型在不同的任务中表现出显着的多功能性，受益于其强大的表示能力和有效的多模态对齐。我们将这些模型的应用，特别是CLIP，扩展到声源定位领域。与传统方法不同，我们采用预训练的CLIP模型，而无需显式文本输入，仅依赖于视听对应关系。为此，我们引入了一个框架，该框架将音频信号转换为与CLIP的文本编码器兼容的令牌，从而产生音频驱动的嵌入。通过直接使用这些嵌入，我们的方法为所提供的音频生成音频接地掩码，从突出显示的区域中提取音频接地图像特征，并使用视听对应目标将它们与音频驱动的嵌入对齐。我们的研究结果表明，利用预先训练的图像-文本模型使我们的模型能够为发声对象生成更完整和紧凑的定位地图。大量的实验表明，我们的方法优于国家的最先进的方法的显着保证金。摘要：Large-scale pre-trained image-text models demonstrate remarkable versatility across diverse tasks, benefiting from their robust representational capabilities and effective multimodal alignment. We extend the application of these models, specifically CLIP, to the domain of sound source localization. Unlike conventional approaches, we employ the pre-trained CLIP model without explicit text input, relying solely on the audio-visual correspondence. To this end, we introduce a framework that translates audio signals into tokens compatible with CLIP's text encoder, yielding audio-driven embeddings. By directly using these embeddings, our method generates audio-grounded masks for the provided audio, extracts audio-grounded image features from the highlighted regions, and aligns them with the audio-driven embeddings using the audio-visual correspondence objective. Our findings suggest that utilizing pre-trained image-text models enable our model to generate more complete and compact localization maps for the sounding objects. Extensive experiments show that our method outperforms state-of-the-art approaches by a significant margin.
【3】 Rethinking and Improving Multi-task Learning for End-to-end Speech  Translation标题：端到端语音翻译多任务学习的反思与改进链接：https://arxiv.org/abs/2311.03810
作者：Yuhao Zhang,Chen Xu,Bei Li,Hao Chen,Tong Xiao,Chunliang Zhang,Jingbo Zhu备注：Accepted to EMNLP2023 main conference摘要：通过应用多任务学习，已经实现了端到端语音翻译（ST）的显著改进。然而，辅助任务在多大程度上与ST任务高度一致，以及这种方法真正有多大帮助，还没有得到彻底的研究。在本文中，我们研究了不同的任务之间的一致性，考虑不同的时间和模块。我们发现，文本编码器主要促进跨模态转换，但语音中的噪声的存在阻碍了文本和语音表示之间的一致性。此外，我们提出了一种改进的多任务学习（IMTL）方法的ST任务，它通过减轻长度和表示的差异来弥合模态差距。我们在Must-C数据集上进行了实验。结果表明，我们的方法达到国家的最先进的结果。此外，当使用额外的数据时，我们在MuST-C英语到西班牙语任务上实现了新的SOTA结果，其训练时间为当前SOTA方法所需的20.8%。摘要：Significant improvements in end-to-end speech translation (ST) have been achieved through the application of multi-task learning. However, the extent to which auxiliary tasks are highly consistent with the ST task, and how much this approach truly helps, have not been thoroughly studied. In this paper, we investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. Furthermore, we propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation. We conduct experiments on the MuST-C dataset. The results demonstrate that our method attains state-of-the-art results. Moreover, when additional data is used, we achieve the new SOTA result on MuST-C English to Spanish task with 20.8% of the training time required by the current SOTA method.
【4】 Are Words Enough? On the semantic conditioning of affective music  generation标题：光说一句就够了吗？论情感音乐生成的语义制约链接：https://arxiv.org/abs/2311.03624
作者：Jorge Forero,Gilberto Bernardes,Mónica Mendes摘要：音乐被公认为表达情感的一种手段。从这个意义上说，一场激烈的争论出现在需要用语言表达音乐情感的问题上。考虑到使用深度学习模型的自然语言处理的指数级增长，这种关注在今天似乎非常相关，在这种情况下，可以提示语义命题来自动生成音乐。本文旨在分析和讨论情感制约下音乐生成的可能性。为了解决这个问题，我们提出了一个历史的角度来看，包括不同的学科和方法有助于这个话题。详细地说，我们回顾了自动音乐生成中采用的两种主要范式：基于规则的模型和机器学习模型。值得注意的是深度学习架构，旨在从文本描述中生成高保真音乐。这些模型提出了关于音乐表现力的基本问题，包括情感是否可以用文字表达或通过文字表达。我们的结论是，克服语言的局限性和模糊性，通过音乐表达情感，自然语言的深度学习的一些使用有可能通过提供强大的工具来促进和产生新的音乐作品，从而影响创意产业。摘要：Music has been commonly recognized as a means of expressing emotions. In this sense, an intense debate emerges from the need to verbalize musical emotions. This concern seems highly relevant today, considering the exponential growth of natural language processing using deep learning models where it is possible to prompt semantic propositions to generate music automatically. This scoping review aims to analyze and discuss the possibilities of music generation conditioned by emotions. To address this topic, we propose a historical perspective that encompasses the different disciplines and methods contributing to this topic. In detail, we review two main paradigms adopted in automatic music generation: rules-based and machine-learning models. Of note are the deep learning architectures that aim to generate high-fidelity music from textual descriptions. These models raise fundamental questions about the expressivity of music, including whether emotions can be represented with words or expressed through them. We conclude that overcoming the limitation and ambiguity of language to express emotions through music, some of the use of deep learning with natural language has the potential to impact the creative industries by providing powerful tools to prompt and generate new musical works.
【5】 Exploring Latent Spaces of Tonal Music using Variational Autoencoders标题：利用变分自动编码器探索调性音乐的潜在空间链接：https://arxiv.org/abs/2311.03621
作者：Nádia Carvalho,Gilberto Bernardes摘要：变分自动编码器（VAE）已被证明是产生认知和语义值的潜在表示的有效模型。我们评估的程度VAE训练的原型音调音乐语料库的371巴赫的合唱定义潜在的空间代表的第五圈和层次关系的每个关键组成部分音高在音乐认知。详细地说，我们比较了潜在的空间不同的VAE语料库编码-钢琴辊，pitch，ABC，Tonnetz，DFT的音高，和音高类分布-在提供一个音高空间的关键关系，符合认知距离。我们使用客观指标来评估这些编码的模型性能，以捕获精度，均方误差（MSE），KL发散和计算成本。ABC编码在重构原始数据方面表现最好，而基音DFT似乎从潜在空间捕获更多信息。此外，客观评价12个主要或次要的换位每件被用来量化对齐1）内部和每键段间的距离和2）的关键距离的认知音高空间。我们的研究结果表明，音高DFT VAE潜在空间与认知空间最佳对齐，并提供了一个共同的音调空间，其中键内的重叠对象是模糊聚类，这强加了结构重要性或稳定性的明确定义的顺序-即，音调层次不同键的音调层次结构可以用于测量键距离和它们在多个层次结构处的键内分量的关系（例如，音符和和弦）。我们的VAE和编码框架的实现可以在线获得。摘要：Variational Autoencoders (VAEs) have proven to be effective models for producing latent representations of cognitive and semantic value. We assess the degree to which VAEs trained on a prototypical tonal music corpus of 371 Bach's chorales define latent spaces representative of the circle of fifths and the hierarchical relation of each key component pitch as drawn in music cognition. In detail, we compare the latent space of different VAE corpus encodings -- Piano roll, MIDI, ABC, Tonnetz, DFT of pitch, and pitch class distributions -- in providing a pitch space for key relations that align with cognitive distances. We evaluate the model performance of these encodings using objective metrics to capture accuracy, mean square error (MSE), KL-divergence, and computational cost. The ABC encoding performs the best in reconstructing the original data, while the Pitch DFT seems to capture more information from the latent space. Furthermore, an objective evaluation of 12 major or minor transpositions per piece is adopted to quantify the alignment of 1) intra- and inter-segment distances per key and 2) the key distances to cognitive pitch spaces. Our results show that Pitch DFT VAE latent spaces align best with cognitive spaces and provide a common-tone space where overlapping objects within a key are fuzzy clusters, which impose a well-defined order of structural significance or stability -- i.e., a tonal hierarchy. Tonal hierarchies of different keys can be used to measure key distances and the relationships of their in-key components at multiple hierarchies (e.g., notes and chords). The implementation of our VAE and the encodings framework are made available online.
【6】 SoundCam: A Dataset for Finding Humans Using Room Acoustics标题：SoundCam：使用房间声学寻找人类的数据集链接：https://arxiv.org/abs/2311.03517
作者：Mason Wang,Samuel Clarke,Jui-Hsien Wang,Ruohan Gao,Jiajun Wu备注：In NeurIPS 2023 Datasets and Benchmarks Track. Project page: this https URL Wang and Clarke contributed equally to this work摘要：房间的声学特性是房间的几何形状、房间内的对象及其特定位置的产物。房间的声学特性可以通过声源和收听者位置之间的脉冲响应（RIR）来表征，或者从房间中存在的自然信号的记录中粗略推断。房间中物体位置的变化可以影响房间声学特性的可测量变化，如RIR所表征的。现有的RIR数据集要么不系统地改变环境中对象的位置，要么只由模拟的RIR组成。我们介绍了SoundCam，这是迄今为止公开发布的野生房间中最大的独特RIR数据集。它包括5，000个10通道的房间脉冲响应真实世界测量和2，000个10通道的音乐录音，这些录音在三个不同的房间中进行，包括一个受控的声学实验室，一个野外客厅和一个会议室，每个房间都有不同的人。我们表明，这些测量可以用于有趣的任务，如检测和识别人类，并跟踪他们的位置。摘要：A room's acoustic properties are a product of the room's geometry, the objects within the room, and their specific positions. A room's acoustic properties can be characterized by its impulse response (RIR) between a source and listener location, or roughly inferred from recordings of natural signals present in the room. Variations in the positions of objects in a room can effect measurable changes in the room's acoustic properties, as characterized by the RIR. Existing datasets of RIRs either do not systematically vary positions of objects in an environment, or they consist of only simulated RIRs. We present SoundCam, the largest dataset of unique RIRs from in-the-wild rooms publicly released to date. It includes 5,000 10-channel real-world measurements of room impulse responses and 2,000 10-channel recordings of music in three different rooms, including a controlled acoustic lab, an in-the-wild living room, and a conference room, with different humans in positions throughout each room. We show that these measurements can be used for interesting tasks, such as detecting and identifying humans, and tracking their positions.
【7】 MFAAN: Unveiling Audio Deepfakes with a Multi-Feature Authenticity  Network标题：MFAAN：用多功能真实性网络揭开音频Deepfake的面纱链接：https://arxiv.org/abs/2311.03509
作者：Karthik Sivarama Krishnan,Koushik Sivarama Krishnan摘要：在当代数字时代，deepfake的扩散对信息传播的神圣性提出了严峻的挑战。特别是音频deepfake，可能具有欺骗性的现实，在错误信息活动中构成重大风险。为了解决这一威胁，我们引入了多特征音频真实性网络（MFAAN），这是一种专为检测伪造音频内容而设计的高级架构。MFAAN包含多个并行路径，旨在利用不同音频表示的优势，包括梅尔频率倒谱系数（MFCC），线性频率倒谱系数（LFCC）和色度短时傅立叶变换（Chroma-STFT）。通过协同融合这些功能，MFAAN实现了对音频内容的细致入微的理解，促进了真实录音和操纵录音之间的鲁棒区分。MFAAN在两个基准数据集“野外”音频Deepfake Data和The Fake or Real Dataset上的初步评估表明，其性能优越，准确率分别为98.93%和94.47%。这些结果不仅强调了MFAAN的有效性，还突出了其作为对抗deepfake音频内容的关键工具的潜力。摘要：In the contemporary digital age, the proliferation of deepfakes presents a formidable challenge to the sanctity of information dissemination. Audio deepfakes, in particular, can be deceptively realistic, posing significant risks in misinformation campaigns. To address this threat, we introduce the Multi-Feature Audio Authenticity Network (MFAAN), an advanced architecture tailored for the detection of fabricated audio content. MFAAN incorporates multiple parallel paths designed to harness the strengths of different audio representations, including Mel-frequency cepstral coefficients (MFCC), linear-frequency cepstral coefficients (LFCC), and Chroma Short Time Fourier Transform (Chroma-STFT). By synergistically fusing these features, MFAAN achieves a nuanced understanding of audio content, facilitating robust differentiation between genuine and manipulated recordings. Preliminary evaluations of MFAAN on two benchmark datasets, 'In-the-Wild' Audio Deepfake Data and The Fake-or-Real Dataset, demonstrate its superior performance, achieving accuracies of 98.93% and 94.47% respectively. Such results not only underscore the efficacy of MFAAN but also highlight its potential as a pivotal tool in the ongoing battle against deepfake audio content.
【8】 Combinatorial Hodge Theory in Simplicial Signal Processing -- DAFx2023  Lecture Notes标题：单纯信号处理中的组合Hodge理论--DAFx 2023讲义链接：https://arxiv.org/abs/2311.03469
作者：Georg Essl备注：33 pages, 40 figures. arXiv admin note: substantial text overlap with arXiv:2211.05821摘要：在丹麦哥本哈根举行的国际数字音频效果会议（DAFx-23）上举行的单纯信号处理中的组合霍奇理论教程的讲义。摘要：Lecture notes of a tutorial on Combinatorial Hodge Theory in Simplicial Signal Processing held at international conference for digital audio effects (DAFx-23) in Copenhagen, Denmark.
【9】 Personalizing Keyword Spotting with Speaker Information标题：使用说话人信息个性化关键词识别链接：https://arxiv.org/abs/2311.03419
作者：Beltrán Labrador,Pai Zhu,Guanlong Zhao,Angelo Scorza Scarpati,Quan Wang,Alicia Lozano-Diez,Alex Park,Ignacio López Moreno摘要：关键词识别系统通常很难推广到具有各种口音和年龄组的不同人群。为了应对这一挑战，我们提出了一种新的方法，将说话人信息集成到关键字定位使用智能线性调制（FILM），最近的方法从多个信息源学习。我们探索文本相关和文本无关的说话人识别系统来提取说话人信息，我们实验从输入音频和预先注册的用户音频中提取此信息。我们在不同的数据集上评估了我们的系统，并在关键字检测准确性方面取得了实质性的改进，特别是在代表性不足的扬声器组中。此外，我们提出的方法只需要在参数数量上增加1%，对延迟和计算成本的影响最小，这使得它成为现实世界应用的实用解决方案。摘要：Keyword spotting systems often struggle to generalize to a diverse population with various accents and age groups. To address this challenge, we propose a novel approach that integrates speaker information into keyword spotting using Feature-wise Linear Modulation (FiLM), a recent method for learning from multiple sources of information. We explore both Text-Dependent and Text-Independent speaker recognition systems to extract speaker information, and we experiment on extracting this information from both the input audio and pre-enrolled user audio. We evaluate our systems on a diverse dataset and achieve a substantial improvement in keyword detection accuracy, particularly among underrepresented speaker groups. Moreover, our proposed approach only requires a small 1% increase in the number of parameters, with a minimum impact on latency and computational cost, which makes it a practical solution for real-world applications.
【10】 Learning Disentangled Speech Representations标题：学习解开纠缠的语音表示链接：https://arxiv.org/abs/2311.03389
作者：Yusuf Brima,Ulf Krumnack,Simone Pika,Gunther Heidemann摘要：尽管从语音中分离表征学习在许多应用领域中很重要，但它仍然是有限的。一个关键的挑战是缺乏具有已知生成因素的语音数据集来评估方法。本文提出了SynSpeech：一种新的合成语音数据集，具有地面真实因素，可以研究解开语音表示。我们计划提出一个全面的研究评估监督技术，使用已建立的监督解纠缠指标。这个基准数据集和框架解决了严格评估最先进的分离语音表示学习方法的差距。我们的研究结果将为推进这一未充分探索的领域提供见解，并实现更强大的语音表示。摘要：Disentangled representation learning from speech remains limited despite its importance in many application domains. A key challenge is the lack of speech datasets with known generative factors to evaluate methods. This paper proposes SynSpeech: a novel synthetic speech dataset with ground truth factors enabling research on disentangling speech representations. We plan to present a comprehensive study evaluating supervised techniques using established supervised disentanglement metrics. This benchmark dataset and framework address the gap in the rigorous evaluation of state-of-the-art disentangled speech representation learning methods. Our findings will provide insights to advance this underexplored area and enable more robust speech representations.
eess.AS音频处理
【1】 Fine-tuning convergence model in Bengali speech recognition标题：孟加拉语音识别中的微调收敛模型链接：https://arxiv.org/abs/2311.04122
作者：Zhu Ruiying,Shen Meng摘要：语音识别的研究吸引了相当大的兴趣，由于分割不间断的语音的困难任务。在各种语言中，孟加拉语具有独特的节奏模式和音调，这使得它特别难以识别，并且缺乏有效的商业识别方法。为了改进孟加拉语的自动语音识别模型，我们的团队选择使用wave2vec 2.0预训练模型，该模型已经过收敛微调。在单词错误率（WER）方面，对学习率和辍学率参数进行了微调，并在模型训练稳定后，尝试扩大训练集比例，提高了模型的性能。因此，在公开上市的官方数据集的测试集上，WER从0.508显著提高到0.437。之后，将训练集和验证集合并，创建一个综合数据集，用作训练集，实现了0.436的显著WER。摘要：Research on speech recognition has attracted considerable interest due to the difficult task of segmenting uninterrupted speech. Among various languages, Bengali features distinct rhythmic patterns and tones, making it particularly difficult to recognize and lacking an efficient commercial recognition method. In order to improve the automatic speech recognition model for Bengali, our team has chosen to utilize the wave2vec 2.0 pre-trained model, which has undergone convergence for fine-tuning. Regarding Word Error Rate (WER), the learning rate and dropout parameters were fine-tuned, and after the model training was stable, attempts were made to enlarge the training set ratio, which improved the model's performance. Consequently, there was a notable enhancement in the WER from 0.508 to 0.437 on the test set of the publicly listed official dataset. Afterwards, the training and validation sets were merged, creating a comprehensive dataset that was used as the training set, achieving a remarkable WER of 0.436.
【2】 HRTF Estimation in the Wild标题：野外HRTF估计链接：https://arxiv.org/abs/2311.03560
作者：Vivek Jayaram,Ira Kemelmacher-Shlizerman,Steven M. Seitz备注：9 Pages. Presented at UIST '23摘要：头部相关传递函数（HRTF）在创建沉浸式空间音频体验方面起着至关重要的作用。然而，HRTF因人而异，并且用于估计个性化HRTF的传统方法是昂贵的、耗时的，并且需要专门的设备。我们设想这样一个世界：在日常环境中，通过耳塞捕获数据，即可确定您的个性化HRTF。在本文中，我们提出了一种新的方法来获得个性化的HRTF，只依赖于在野生双耳录音和头部跟踪数据。通过分析用户在不同噪声源的不同环境中旋转头部时声音的变化，我们可以准确地估计他们的个性化HRTF。我们的研究结果表明，我们预测的HRTF密切匹配地面实况HRTF在消声室测量。此外，听力研究表明，我们的个性化HRTF显着提高声音定位，并减少在虚拟环境中的前后混淆。我们的方法提供了一个有效的和可访问的方法来获得个性化的HRTF，并有可能大大提高空间音频体验。摘要：Head Related Transfer Functions (HRTFs) play a crucial role in creating immersive spatial audio experiences. However, HRTFs differ significantly from person to person, and traditional methods for estimating personalized HRTFs are expensive, time-consuming, and require specialized equipment. We imagine a world where your personalized HRTF can be determined by capturing data through earbuds in everyday environments. In this paper, we propose a novel approach for deriving personalized HRTFs that only relies on in-the-wild binaural recordings and head tracking data. By analyzing how sounds change as the user rotates their head through different environments with different noise sources, we can accurately estimate their personalized HRTF. Our results show that our predicted HRTFs closely match ground-truth HRTFs measured in an anechoic chamber. Furthermore, listening studies demonstrate that our personalized HRTFs significantly improve sound localization and reduce front-back confusion in virtual environments. Our approach offers an efficient and accessible method for deriving personalized HRTFs and has the potential to greatly improve spatial audio experiences.
【3】 Combinatorial Hodge Theory in Simplicial Signal Processing -- DAFx2023  Lecture Notes标题：单纯信号处理中的组合霍奇理论--DAFx2023讲义链接：https://arxiv.org/abs/2311.03469
作者：Georg Essl备注：33 pages, 40 figures. arXiv admin note: substantial text overlap with arXiv:2211.05821摘要：在丹麦哥本哈根举行的国际数字音频效果会议（DAFx-23）上举行的单纯信号处理中的组合霍奇理论教程的讲义。摘要：Lecture notes of a tutorial on Combinatorial Hodge Theory in Simplicial Signal Processing held at international conference for digital audio effects (DAFx-23) in Copenhagen, Denmark.
【4】 Personalizing Keyword Spotting with Speaker Information标题：利用说话人信息进行个性化关键词识别链接：https://arxiv.org/abs/2311.03419
作者：Beltrán Labrador,Pai Zhu,Guanlong Zhao,Angelo Scorza Scarpati,Quan Wang,Alicia Lozano-Diez,Alex Park,Ignacio López Moreno摘要：关键词识别系统通常很难推广到具有各种口音和年龄组的不同人群。为了应对这一挑战，我们提出了一种新的方法，将说话人信息集成到关键字定位使用智能线性调制（FILM），最近的方法从多个信息源学习。我们探索文本相关和文本无关的说话人识别系统来提取说话人信息，我们实验从输入音频和预先注册的用户音频中提取此信息。我们在不同的数据集上评估了我们的系统，并在关键字检测准确性方面取得了实质性的改进，特别是在代表性不足的扬声器组中。此外，我们提出的方法只需要在参数数量上增加1%，对延迟和计算成本的影响最小，这使得它成为现实世界应用的实用解决方案。摘要：Keyword spotting systems often struggle to generalize to a diverse population with various accents and age groups. To address this challenge, we propose a novel approach that integrates speaker information into keyword spotting using Feature-wise Linear Modulation (FiLM), a recent method for learning from multiple sources of information. We explore both Text-Dependent and Text-Independent speaker recognition systems to extract speaker information, and we experiment on extracting this information from both the input audio and pre-enrolled user audio. We evaluate our systems on a diverse dataset and achieve a substantial improvement in keyword detection accuracy, particularly among underrepresented speaker groups. Moreover, our proposed approach only requires a small 1% increase in the number of parameters, with a minimum impact on latency and computational cost, which makes it a practical solution for real-world applications.
【5】 Learning Disentangled Speech Representations标题：学习解开纠缠的语音表示链接：https://arxiv.org/abs/2311.03389
作者：Yusuf Brima,Ulf Krumnack,Simone Pika,Gunther Heidemann摘要：尽管从语音中分离表征学习在许多应用领域中很重要，但它仍然是有限的。一个关键的挑战是缺乏具有已知生成因素的语音数据集来评估方法。本文提出了SynSpeech：一种新的合成语音数据集，具有地面真实因素，可以研究解开语音表示。我们计划提出一个全面的研究评估监督技术，使用已建立的监督解纠缠指标。这个基准数据集和框架解决了严格评估最先进的分离语音表示学习方法的差距。我们的研究结果将为推进这一未充分探索的领域提供见解，并实现更强大的语音表示。摘要：Disentangled representation learning from speech remains limited despite its importance in many application domains. A key challenge is the lack of speech datasets with known generative factors to evaluate methods. This paper proposes SynSpeech: a novel synthetic speech dataset with ground truth factors enabling research on disentangling speech representations. We plan to present a comprehensive study evaluating supervised techniques using established supervised disentanglement metrics. This benchmark dataset and framework address the gap in the rigorous evaluation of state-of-the-art disentangled speech representation learning methods. Our findings will provide insights to advance this underexplored area and enable more robust speech representations.
【6】 Proceedings of the 5th International Workshop on Reading Music Systems标题：第五届国际音乐阅读系统研讨会论文集链接：https://arxiv.org/abs/2311.04091
作者：Jorge Calvo-Zaragoza,Alexander Pacha,Elona Shatri备注：Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri摘要：阅读音乐系统国际研讨会（WoRMS）是一个研讨会，试图将开发阅读音乐系统的研究人员（如光学音乐识别领域）与其他研究人员和从业者（如图书馆员或音乐学家）联系起来。研讨会的相关主题包括但不限于：音乐阅读系统;光学音乐识别;数据集和性能评估;乐谱图像处理;作者识别;乐谱的创作、编辑、存储和呈现系统;多模式系统;用于制作书面音乐的新颖音乐输入方法;基于网络的音乐信息检索服务;应用和项目;与书面音乐有关的案件。  这些是2023年11月4日在意大利米兰举行的第五届国际阅读音乐系统研讨会的会议记录。摘要：The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music.  These are the proceedings of the 5th International Workshop on Reading Music Systems, held in Milan, Italy on Nov. 4th 2023.
【7】 Can CLIP Help Sound Source Localization?标题：CLIP能帮助声源本地化吗？链接：https://arxiv.org/abs/2311.04066
作者：Sooyoung Park,Arda Senocak,Joon Son Chung备注：WACV 2024摘要：大规模预训练的图像-文本模型在不同的任务中表现出显着的多功能性，受益于其强大的表示能力和有效的多模态对齐。我们将这些模型的应用，特别是CLIP，扩展到声源定位领域。与传统方法不同，我们采用预训练的CLIP模型，而无需显式文本输入，仅依赖于视听对应关系。为此，我们引入了一个框架，该框架将音频信号转换为与CLIP的文本编码器兼容的令牌，从而产生音频驱动的嵌入。通过直接使用这些嵌入，我们的方法为所提供的音频生成音频接地掩码，从突出显示的区域中提取音频接地图像特征，并使用视听对应目标将它们与音频驱动的嵌入对齐。我们的研究结果表明，利用预先训练的图像-文本模型使我们的模型能够为发声对象生成更完整和紧凑的定位地图。大量的实验表明，我们的方法优于国家的最先进的方法的显着保证金。摘要：Large-scale pre-trained image-text models demonstrate remarkable versatility across diverse tasks, benefiting from their robust representational capabilities and effective multimodal alignment. We extend the application of these models, specifically CLIP, to the domain of sound source localization. Unlike conventional approaches, we employ the pre-trained CLIP model without explicit text input, relying solely on the audio-visual correspondence. To this end, we introduce a framework that translates audio signals into tokens compatible with CLIP's text encoder, yielding audio-driven embeddings. By directly using these embeddings, our method generates audio-grounded masks for the provided audio, extracts audio-grounded image features from the highlighted regions, and aligns them with the audio-driven embeddings using the audio-visual correspondence objective. Our findings suggest that utilizing pre-trained image-text models enable our model to generate more complete and compact localization maps for the sounding objects. Extensive experiments show that our method outperforms state-of-the-art approaches by a significant margin.
【8】 Rethinking and Improving Multi-task Learning for End-to-end Speech  Translation标题：端到端语音翻译多任务学习的反思与改进链接：https://arxiv.org/abs/2311.03810
作者：Yuhao Zhang,Chen Xu,Bei Li,Hao Chen,Tong Xiao,Chunliang Zhang,Jingbo Zhu备注：Accepted to EMNLP2023 main conference摘要：通过应用多任务学习，已经实现了端到端语音翻译（ST）的显著改进。然而，辅助任务在多大程度上与ST任务高度一致，以及这种方法真正有多大帮助，还没有得到彻底的研究。在本文中，我们研究了不同的任务之间的一致性，考虑不同的时间和模块。我们发现，文本编码器主要促进跨模态转换，但语音中的噪声的存在阻碍了文本和语音表示之间的一致性。此外，我们提出了一种改进的多任务学习（IMTL）方法的ST任务，它通过减轻长度和表示的差异来弥合模态差距。我们在Must-C数据集上进行了实验。结果表明，我们的方法达到国家的最先进的结果。此外，当使用额外的数据时，我们在MuST-C英语到西班牙语任务上实现了新的SOTA结果，其训练时间为当前SOTA方法所需的20.8%。摘要：Significant improvements in end-to-end speech translation (ST) have been achieved through the application of multi-task learning. However, the extent to which auxiliary tasks are highly consistent with the ST task, and how much this approach truly helps, have not been thoroughly studied. In this paper, we investigate the consistency between different tasks, considering different times and modules. We find that the textual encoder primarily facilitates cross-modal conversion, but the presence of noise in speech impedes the consistency between text and speech representations. Furthermore, we propose an improved multi-task learning (IMTL) approach for the ST task, which bridges the modal gap by mitigating the difference in length and representation. We conduct experiments on the MuST-C dataset. The results demonstrate that our method attains state-of-the-art results. Moreover, when additional data is used, we achieve the new SOTA result on MuST-C English to Spanish task with 20.8% of the training time required by the current SOTA method.
【9】 Are Words Enough? On the semantic conditioning of affective music  generation标题：光说一句就够了吗？论情感音乐生成的语义制约链接：https://arxiv.org/abs/2311.03624
作者：Jorge Forero,Gilberto Bernardes,Mónica Mendes摘要：音乐被公认为表达情感的一种手段。从这个意义上说，一场激烈的争论出现在需要用语言表达音乐情感的问题上。考虑到使用深度学习模型的自然语言处理的指数级增长，这种关注在今天似乎非常相关，在这种情况下，可以提示语义命题来自动生成音乐。本文旨在分析和讨论情感制约下音乐生成的可能性。为了解决这个问题，我们提出了一个历史的角度来看，包括不同的学科和方法有助于这个话题。详细地说，我们回顾了自动音乐生成中采用的两种主要范式：基于规则的模型和机器学习模型。值得注意的是深度学习架构，旨在从文本描述中生成高保真音乐。这些模型提出了关于音乐表现力的基本问题，包括情感是否可以用文字表达或通过文字表达。我们的结论是，克服语言的局限性和模糊性，通过音乐表达情感，自然语言的深度学习的一些使用有可能通过提供强大的工具来促进和产生新的音乐作品，从而影响创意产业。摘要：Music has been commonly recognized as a means of expressing emotions. In this sense, an intense debate emerges from the need to verbalize musical emotions. This concern seems highly relevant today, considering the exponential growth of natural language processing using deep learning models where it is possible to prompt semantic propositions to generate music automatically. This scoping review aims to analyze and discuss the possibilities of music generation conditioned by emotions. To address this topic, we propose a historical perspective that encompasses the different disciplines and methods contributing to this topic. In detail, we review two main paradigms adopted in automatic music generation: rules-based and machine-learning models. Of note are the deep learning architectures that aim to generate high-fidelity music from textual descriptions. These models raise fundamental questions about the expressivity of music, including whether emotions can be represented with words or expressed through them. We conclude that overcoming the limitation and ambiguity of language to express emotions through music, some of the use of deep learning with natural language has the potential to impact the creative industries by providing powerful tools to prompt and generate new musical works.
【10】 Exploring Latent Spaces of Tonal Music using Variational Autoencoders标题：利用变分自动编码器探索调性音乐的潜在空间链接：https://arxiv.org/abs/2311.03621
作者：Nádia Carvalho,Gilberto Bernardes摘要：变分自动编码器（VAE）已被证明是产生认知和语义值的潜在表示的有效模型。我们评估的程度VAE训练的原型音调音乐语料库的371巴赫的合唱定义潜在的空间代表的第五圈和层次关系的每个关键组成部分音高在音乐认知。详细地说，我们比较了潜在的空间不同的VAE语料库编码-钢琴辊，pitch，ABC，Tonnetz，DFT的音高，和音高类分布-在提供一个音高空间的关键关系，符合认知距离。我们使用客观指标来评估这些编码的模型性能，以捕获精度，均方误差（MSE），KL发散和计算成本。ABC编码在重构原始数据方面表现最好，而基音DFT似乎从潜在空间捕获更多信息。此外，客观评价12个主要或次要的换位每件被用来量化对齐1）内部和每键段间的距离和2）的关键距离的认知音高空间。我们的研究结果表明，音高DFT VAE潜在空间与认知空间最佳对齐，并提供了一个共同的音调空间，其中键内的重叠对象是模糊聚类，这强加了结构重要性或稳定性的明确定义的顺序-即，音调层次不同键的音调层次结构可以用于测量键距离和它们在多个层次结构处的键内分量的关系（例如，音符和和弦）。我们的VAE和编码框架的实现可以在线获得。摘要：Variational Autoencoders (VAEs) have proven to be effective models for producing latent representations of cognitive and semantic value. We assess the degree to which VAEs trained on a prototypical tonal music corpus of 371 Bach's chorales define latent spaces representative of the circle of fifths and the hierarchical relation of each key component pitch as drawn in music cognition. In detail, we compare the latent space of different VAE corpus encodings -- Piano roll, MIDI, ABC, Tonnetz, DFT of pitch, and pitch class distributions -- in providing a pitch space for key relations that align with cognitive distances. We evaluate the model performance of these encodings using objective metrics to capture accuracy, mean square error (MSE), KL-divergence, and computational cost. The ABC encoding performs the best in reconstructing the original data, while the Pitch DFT seems to capture more information from the latent space. Furthermore, an objective evaluation of 12 major or minor transpositions per piece is adopted to quantify the alignment of 1) intra- and inter-segment distances per key and 2) the key distances to cognitive pitch spaces. Our results show that Pitch DFT VAE latent spaces align best with cognitive spaces and provide a common-tone space where overlapping objects within a key are fuzzy clusters, which impose a well-defined order of structural significance or stability -- i.e., a tonal hierarchy. Tonal hierarchies of different keys can be used to measure key distances and the relationships of their in-key components at multiple hierarchies (e.g., notes and chords). The implementation of our VAE and the encodings framework are made available online.
【11】 SoundCam: A Dataset for Finding Humans Using Room Acoustics标题：SoundCam：使用房间声学寻找人类的数据集链接：https://arxiv.org/abs/2311.03517
作者：Mason Wang,Samuel Clarke,Jui-Hsien Wang,Ruohan Gao,Jiajun Wu备注：In NeurIPS 2023 Datasets and Benchmarks Track. Project page: this https URL Wang and Clarke contributed equally to this work摘要：房间的声学特性是房间的几何形状、房间内的对象及其特定位置的产物。房间的声学特性可以通过声源和收听者位置之间的脉冲响应（RIR）来表征，或者从房间中存在的自然信号的记录中粗略推断。房间中物体位置的变化可以影响房间声学特性的可测量变化，如RIR所表征的。现有的RIR数据集要么不系统地改变环境中对象的位置，要么只由模拟的RIR组成。我们介绍了SoundCam，这是迄今为止公开发布的野生房间中最大的独特RIR数据集。它包括5，000个10通道的房间脉冲响应真实世界测量和2，000个10通道的音乐录音，这些录音在三个不同的房间中进行，包括一个受控的声学实验室，一个野外客厅和一个会议室，每个房间都有不同的人。我们表明，这些测量可以用于有趣的任务，如检测和识别人类，并跟踪他们的位置。摘要：A room's acoustic properties are a product of the room's geometry, the objects within the room, and their specific positions. A room's acoustic properties can be characterized by its impulse response (RIR) between a source and listener location, or roughly inferred from recordings of natural signals present in the room. Variations in the positions of objects in a room can effect measurable changes in the room's acoustic properties, as characterized by the RIR. Existing datasets of RIRs either do not systematically vary positions of objects in an environment, or they consist of only simulated RIRs. We present SoundCam, the largest dataset of unique RIRs from in-the-wild rooms publicly released to date. It includes 5,000 10-channel real-world measurements of room impulse responses and 2,000 10-channel recordings of music in three different rooms, including a controlled acoustic lab, an in-the-wild living room, and a conference room, with different humans in positions throughout each room. We show that these measurements can be used for interesting tasks, such as detecting and identifying humans, and tracking their positions.
【12】 MFAAN: Unveiling Audio Deepfakes with a Multi-Feature Authenticity  Network标题：MFAAN：用多功能真实性网络揭开音频Deepfake的面纱链接：https://arxiv.org/abs/2311.03509
作者：Karthik Sivarama Krishnan,Koushik Sivarama Krishnan摘要：在当代数字时代，deepfake的扩散对信息传播的神圣性提出了严峻的挑战。特别是音频deepfake，可能具有欺骗性的现实，在错误信息活动中构成重大风险。为了解决这一威胁，我们引入了多特征音频真实性网络（MFAAN），这是一种专为检测伪造音频内容而设计的高级架构。MFAAN包含多个并行路径，旨在利用不同音频表示的优势，包括梅尔频率倒谱系数（MFCC），线性频率倒谱系数（LFCC）和色度短时傅立叶变换（Chroma-STFT）。通过协同融合这些功能，MFAAN实现了对音频内容的细致入微的理解，促进了真实录音和操纵录音之间的鲁棒区分。MFAAN在两个基准数据集“野外”音频Deepfake Data和The Fake or Real Dataset上的初步评估表明，其性能优越，准确率分别为98.93%和94.47%。这些结果不仅强调了MFAAN的有效性，还突出了其作为对抗deepfake音频内容的关键工具的潜力。摘要：In the contemporary digital age, the proliferation of deepfakes presents a formidable challenge to the sanctity of information dissemination. Audio deepfakes, in particular, can be deceptively realistic, posing significant risks in misinformation campaigns. To address this threat, we introduce the Multi-Feature Audio Authenticity Network (MFAAN), an advanced architecture tailored for the detection of fabricated audio content. MFAAN incorporates multiple parallel paths designed to harness the strengths of different audio representations, including Mel-frequency cepstral coefficients (MFCC), linear-frequency cepstral coefficients (LFCC), and Chroma Short Time Fourier Transform (Chroma-STFT). By synergistically fusing these features, MFAAN achieves a nuanced understanding of audio content, facilitating robust differentiation between genuine and manipulated recordings. Preliminary evaluations of MFAAN on two benchmark datasets, 'In-the-Wild' Audio Deepfake Data and The Fake-or-Real Dataset, demonstrate its superior performance, achieving accuracies of 98.93% and 94.47% respectively. Such results not only underscore the efficacy of MFAAN but also highlight its potential as a pivotal tool in the ongoing battle against deepfake audio content.
机器翻译，仅供参考