今日论文合集：cs.SD语音9篇，eess.AS音频处理9篇。【1】 Improved Vehicle Sub-type Classification for Acoustic Traffic Monitoring
标题：一种改进的声学交通监测车辆亚型分类
链接：https://arxiv.org/abs/2302.02945
作者：Mohd Ashhad,Umang Goenka,Aaryan Jagetia,Parwin Akhtari,Sooraj K. Ambat,Mary Samuel机构：Dept. of Computer Science & Engineering, Jamia Hamdard, New Delhi, India, Dept. of Information Technology, IIIT Lucknow, Lucknow, India, Dept. of Mechanical Engineering, NIT Rourkeela, Roorkela, India, Naval Physical & Oceanographic Laboratory, Defence R&D Organization备注：Accepted at Twenty-Ninth National Conference on Communications(NCC) 23 - 26 February, Indian Institute of Technology Guwahati摘要：道路上车辆的检测与分类是交通监控的一项重要任务。通常，计算机视觉（CV）算法在道路上的车辆分类任务中占主导地位，但是CV方法可能在较差的照明条件下受到影响，并且需要更大量的计算能力。此外，在敏感和安全区域安装摄像头还存在隐私问题。相比之下，声学交通监控成本效益高，并且可以提供更高的准确性，特别是在低照明条件下和在不能安装摄像机的地方。在本文中，我们考虑声学车辆子类型分类的任务，其中我们将声学信号分为4类：小汽车卡车自行车没有车。我们实验了梅尔谱图，MFCC和GFCC作为特征，并执行数据预处理，以训练一个简单的，良好优化的CNN，执行良好的任务。当使用MFCC作为特征和仔细的数据预处理时，我们提出的方法以98.95%的准确率改进了IDMT交通数据集上建立的最先进的基线。摘要：The detection and classification of vehicles on the road is a crucial task for traffic monitoring. Usually, Computer Vision (CV) algorithms dominate the task of vehicle classification on the road, but CV methodologies might suffer in poor lighting conditions and require greater amounts of computational power. Additionally, there is a privacy concern with installing cameras in sensitive and secure areas. In contrast, acoustic traffic monitoring is cost-effective, and can provide greater accuracy, particularly in low lighting conditions and in places where cameras cannot be installed. In this paper, we consider the task of acoustic vehicle sub-type classification, where we classify acoustic signals into 4 classes: car, truck, bike, and no vehicle. We experimented with Mel spectrograms, MFCC and GFCC as features and performed data pre-processing to train a simple, well optimized CNN that performs well at the task. When used with MFCC as features and careful data pre-processing, our proposed methodology improves upon the established state-of-the-art baseline on the IDMT Traffic dataset with an accuracy of 98.95%.
【2】 Audio Representation Learning by Distilling Video as Privileged Information
标题：提取视频作为特权信息的音频表征学习
链接：https://arxiv.org/abs/2302.02845
作者：Amirhossein Hajavi,Ali Etemad机构：The Authors of this work are in Electrical and Computer EngineeringDepartment of the Queen’s University at Kingston摘要：与单模态方法相比，使用多模态视听数据的深度音频表示学习通常导致更好的性能。然而，在现实世界场景中，两种模态在推理时并不总是可用的，从而导致针对多模态推理训练的模型的性能下降。在这项工作中，我们提出了一种新的方法，用于深度音频表示学习使用视听数据时，视频模态不存在的推理。为此，我们在使用特权信息学习（LUPI）的框架下采用师生知识蒸馏。虽然先前针对LUPI提出的方法使用教师生成的软标签，但在我们提出的方法中，我们使用教师学习的嵌入来训练学生网络。我们在两个不同的设置中集成我们的方法：连续数据，其中特征在整个时间上被划分为多个段;以及非连续数据，其中整个特征被视为一个完整段。在非顺序设置中，教师和学生网络都由编码器组件和任务头组成。我们使用由教师的编码器组件产生的嵌入来训练学生的编码器，同时使用地面实况标签来训练学生的任务报头。在顺序设置中，网络具有放置在编码器和任务报头之间的附加聚合组件。我们使用由教师的编码器和聚合组件产生的两组嵌入来训练学生。类似于非顺序设置，使用地面实况标签来训练学生网络的任务报头。我们在两个不同的音视频任务上测试了我们的框架，即说话人识别和语音情感识别，并显示出比单独的基于音频的识别以及使用LUPI的现有工作有很大的改进。摘要：Deep audio representation learning using multi-modal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI). While the previous methods proposed for LUPI use soft-labels generated by the teacher, in our proposed method we use embeddings learned by the teacher to train the student network. We integrate our method in two different settings: sequential data where the features are divided into multiple segments throughout time, and non-sequential data where the entire features are treated as one whole segment. In the non-sequential setting both the teacher and student networks are comprised of an encoder component and a task header. We use the embeddings produced by the encoder component of the teacher to train the encoder of the student, while the task header of the student is trained using ground-truth labels. In the sequential setting, the networks have an additional aggregation component that is placed between the encoder and task header. We use two sets of embeddings produced by the encoder and aggregation component of the teacher to train the student. Similar to the non-sequential setting, the task header of the student network is trained using ground-truth labels. We test our framework on two different audio-visual tasks, namely speaker recognition and speech emotion recognition and show considerable improvements over sole audio-based recognition as well as prior works that use LUPI.
【3】 deep learning of segment-level feature representation for speech emotion recognition in conversations
标题：用于对话中语音情感识别的段级特征表示深度学习
链接：https://arxiv.org/abs/2302.02419
作者：Jiachen Luo,Huy Phan,Joshua Reiss机构：Centre for Digital Music, Queen Mary University of London, UK, Amazon Alexa, Cambridge, MA, USA摘要：由于会话中情感的复杂性和动态性，准确检测会话中的情感是一项必要而又具有挑战性的任务。说话人的情绪状态受许多因素的影响，如对话者刺激、对话场景和话题。在这项工作中，我们提出了一个会话语音情感识别方法，以处理捕捉注意上下文依赖和说话人敏感的互动。首先，我们使用一个预先训练好的VGGish模型来提取单个话语中基于片段的音频表示。其次，注意双向门控循环单元（GRU）对语境敏感信息进行建模，并以动态的方式联合探索说话人内部和说话人之间的依存关系。在标准会话数据集MELD上进行的实验表明，与现有方法相比，所提方法是有效的。摘要：Accurately detecting emotions in conversation is a necessary yet challenging task due to the complexity of emotions and dynamics in dialogues. The emotional state of a speaker can be influenced by many different factors, such as interlocutor stimulus, dialogue scene, and topic. In this work, we propose a conversational speech emotion recognition method to deal with capturing attentive contextual dependency and speaker-sensitive interactions. First, we use a pretrained VGGish model to extract segment-based audio representation in individual utterances. Second, an attentive bi-directional gated recurrent unit (GRU) models contextual-sensitive information and explores intra- and inter-speaker dependencies jointly in a dynamic manner. The experiments conducted on the standard conversational dataset MELD demonstrate the effectiveness of the proposed method when compared against state-of the-art methods.
【4】 Multi-Source Diffusion Models for Simultaneous Music Generation and Separation
标题：同时产生和分离音乐的多源扩散模型
链接：https://arxiv.org/abs/2302.02257
作者：Giorgio Mariani,Irene Tallini,Emilian Postolache,Michele Mancusi,Luca Cosmo,Emanuele Rodolà机构：1Department of Computer Science, Sapienza Universityof Rome, Italy 2Department of Computer Science, Ca’ FoscariUniversity备注：Demo page: this https URL摘要：在这项工作中，我们定义了一个基于扩散的生成模型，通过学习共享上下文的源的联合概率密度的得分，该模型能够同时进行音乐合成和源分离。除了经典的完全推理任务（即生成混合、分离源）之外，我们还介绍并实验了源插补的部分推理任务，其中我们生成给定其他源的子集（例如，演奏与鼓配合良好的钢琴音轨）。此外，我们还介绍了一种新的分离任务推理方法。我们在音乐源分离的标准数据集Slakh2100上训练我们的模型，在生成设置中提供定性结果，并在分离设置中展示有竞争力的定量结果。我们的方法是可以处理生成和分离任务的单个模型的第一个例子，因此代表了向通用音频模型的一步。摘要：In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e. generating a mixture, separating the sources), we also introduce and experiment on the partial inference task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.
【5】 AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
标题：用于真实视听场景合成的学习神经场
链接：https://arxiv.org/abs/2302.02088
作者：Susan Liang,Chao Huang,Yapeng Tian,Anurag Kumar,Chenliang Xu机构：University of Rochester, University of Texas at Dallas, Meta Reality Labs Research摘要：人类对复杂世界的感知依赖于对多模态信号的综合分析，而音频和视频信号的同现为人类提供了丰富的线索。本文主要研究现实世界中新颖的视听场景合成。给定视听场景的视频记录，任务是沿着该视听场景中任意新颖的摄像机轨迹合成具有空间音频的新视频。直接使用基于NeRF的模型进行音频合成是不够的，因为其缺乏先验知识和声学监督。为了应对这些挑战，我们首先提出了一种声学感知的音频生成模块，该模块将我们对音频传播的先验知识集成到NeRF中，其中我们将音频生成与视觉环境的3D几何结构相关联。此外，我们提出了一个坐标转换模块，用于表示相对于声源的观看方向。这样的方向变换帮助模型学习以声源为中心的声场。此外，我们利用头部相关脉冲响应函数来合成伪双耳音频，用于增强训练的数据扩增。我们定性和定量地证明了我们的模型在真实世界视听场景中的优势。我们建议感兴趣的读者查看我们的视频结果，以进行令人信服的比较。摘要：Human perception of the complex world relies on a comprehensive analysis of multi-modal signals, and the co-occurrences of audio and video signals provide humans with rich cues. This paper focuses on novel audio-visual scene synthesis in the real world. Given a video recording of an audio-visual scene, the task is to synthesize new videos with spatial audios along arbitrary novel camera trajectories in that audio-visual scene. Directly using a NeRF-based model for audio synthesis is insufficient due to its lack of prior knowledge and acoustic supervision. To tackle the challenges, we first propose an acoustic-aware audio generation module that integrates our prior knowledge of audio propagation into NeRF, in which we associate audio generation with the 3D geometry of the visual environment. In addition, we propose a coordinate transformation module that expresses a viewing direction relative to the sound source. Such a direction transformation helps the model learn sound source-centric acoustic fields. Moreover, we utilize a head-related impulse response function to synthesize pseudo binaural audio for data augmentation that strengthens training. We qualitatively and quantitatively demonstrate the advantage of our model on real-world audio-visual scenes. We refer interested readers to view our video results for convincing comparisons.
【6】 PSST! Prosodic Speech Segmentation with Transformers
标题：嘘！基于变形器的韵律语音分割
链接：https://arxiv.org/abs/2302.01984
作者：Nathan Roll,Calbert Graham,Simon Todd机构：University of California, Santa Barbara, USA, University of Cambridge, UK备注：5 pages, 3 figures. For associated repository, see this https URL摘要：自注意机制使Transformers在许多语音到文本（STT）任务上取得了超人级别的性能，但自动韵律分割的挑战仍然没有得到解决。本文对预训练的STT模型Whisper进行了微调，通过重新利用低频标记来标注语调单元（IU）边界。我们的方法达到了95.8%的准确率，优于以往的方法，而不需要大规模的标记数据或企业级计算资源。我们还通过应用一系列滤波器来减少输入信号，发现3.2 kHz水平的低通滤波器改善了样本外和分布外环境中的分割性能。我们发布我们的模型作为转录工具和基线进一步改善韵律分割。摘要：Self-attention mechanisms have enabled transformers to achieve superhuman-level performance on many speech-to-text (STT) tasks, yet the challenge of automatic prosodic segmentation has remained unsolved. In this paper we finetune Whisper, a pretrained STT model, to annotate intonation unit (IU) boundaries by repurposing low-frequency tokens. Our approach achieves an accuracy of 95.8%, outperforming previous methods without the need for large-scale labeled data or enterprise grade compute resources. We also diminish input signals by applying a series of filters, finding that low pass filters at a 3.2 kHz level improve segmentation performance in out of sample and out of distribution contexts. We release our model as both a transcription tool and a baseline for further improvements in prosodic segmentation.
【7】 Scene2BIR: Material-aware learning-based binaural impulse response generator for reconstructed real-world 3D scenes
标题：Scene2BIR：用于重建真实世界3D场景的基于材质感知学习的双耳脉冲响应发生器
链接：https://arxiv.org/abs/2302.02809
作者：Anton Jeran Ratnarajah,Dinesh Manocha机构：Real-world environment , D Scene, Channel, Left, Right, Geometric-based BIR, Our Scene,BIR备注：Project page: this https URL摘要：我们提出一个端对端的双耳脉冲响应发生器（BIR）来产生真实世界模型的实时似真声音。我们的方法使用一种新颖的基于神经网络的BIR生成器（Scene2BIR）用于重建3D模型。我们提出一个图形神经网络，它利用三维场景的材质和拓扑信息，生成一个场景潜在向量。此外，我们使用一个条件生成对抗网络（CGAN）从场景潜在向量生成BIRs。我们的网络能够处理重建的3D网格模型中的孔或其他伪影。我们提出了一个有效的成本函数的发生器网络，以纳入空间音频效果。给定源和收听者位置，我们的方法可以在NVIDIA GeForce RTX 2080 Ti GPU上在0.1毫秒内生成BIR，并且可以轻松处理多个源。我们已经用真实世界捕获的BIRs和交互式几何声音传播算法评估了我们的方法的准确性。摘要：We present an end-to-end binaural impulse response generator (BIR) to generate plausible sounds in real-time for real-world models. Our approach uses a novel neural-network-based BIR generator (Scene2BIR) for the reconstructed 3D model. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate BIRs from the scene latent vector. Our network is able to handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function to the generator network to incorporate spatial audio effects. Given the source and the listener position, our approach can generate a BIR in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU and can easily handle multiple sources. We have evaluated the accuracy of our approach with real-world captured BIRs and an interactive geometric sound propagation algorithm.
【8】 Residual Information in Deep Speaker Embedding Architectures
标题：深度说话人嵌入结构中的残差信息
链接：https://arxiv.org/abs/2302.02742
机构：Department of Communications, Technical University of Cluj-Napoca, Cluj-Napoca, Romania;摘要：说话者嵌入表示从语音信号中提取代表性矢量表示的手段，使得该表示仅与说话者身份有关。嵌入通常用于在不同说话者之间进行分类和区分。然而，目前还没有客观的方法来评估说话人嵌入将说话人身份与其他语音特征分离的能力。这意味着嵌入远非理想，高度依赖于训练语料库，并且仍然包括与诸如话语的语言内容、记录条件或说话风格之类的因素有关的一定程度的残留信息。本文介绍了对用一些最新的和高性能的DNN结构提取的六组说话人嵌入的分析，特别是它们能够真正从语音信号中分离说话人身份的程度。为了正确地评估体系结构，使用大的多说话者并行语音数据集。该数据集包括46个说话者发出相同的一组提示，记录在专业工作室或他们的家庭环境中。该分析着眼于在不同嵌入集上计算的说话者内和说话者间相似性度量，以及简单分类和回归方法是否能够从说话者嵌入中提取若干残差信息因子。结果表明，所分析的嵌入结构具有很高的区分能力，但在所有分析的结构中，残留信息仍然以与录音条件、语言内容和话语时长高度相关的形式存在于表征中。摘要：Speaker embeddings represent a means to extract representative vectorial representations from a speech signal such that the representation pertains to the speaker identity alone. The embeddings are commonly used to classify and discriminate between different speakers. However, there is no objective measure to evaluate the ability of a speaker embedding to disentangle the speaker identity from the other speech characteristics. This means that the embeddings are far from ideal, highly dependent on the training corpus and still include a degree of residual information pertaining to factors such as linguistic content, recording conditions or speaking style of the utterance. This paper introduces an analysis over six sets of speaker embeddings extracted with some of the most recent and high-performing DNN architectures, and in particular, the degree to which they are able to truly disentangle the speaker identity from the speech signal. To correctly evaluate the architectures, a large multi-speaker parallel speech dataset is used. The dataset includes 46 speakers uttering the same set of prompts, recorded in either a professional studio or their home environments. The analysis looks into the intra- and inter-speaker similarity measures computed over the different embedding sets, as well as if simple classification and regression methods are able to extract several residual information factors from the speaker embeddings. The results show that the discriminative power of the analyzed embeddings is very high, yet across all the analyzed architectures, residual information is still present in the representations in the form of a high correlation to the recording conditions, linguistic contents and utterance duration.
【9】 cross-modal fusion techniques for utterance-level emotion recognition from text and speech
标题：文本和语音语音级情感识别的跨模式融合技术
链接：https://arxiv.org/abs/2302.02447
作者：Jiachen Luo,Huy Phan,Joshua Reiss机构：Centre for Digital Music, Queen Mary University of London, UK, Amazon Alexa, Cambridge, MA, USA摘要：由于人类情感表达的不确定性和不同模态间的异质性差异，多模态情感识别是一个复杂的基础性研究问题。音频和文本模态对于人类参与者理解情绪尤其重要。尽管已经为MER设计了许多成功的尝试，但是仍然存在要解决的多个挑战：1）弥合多模态特征之间的异质性差距，并对多个模态的模态间和模态内交互进行建模;2）有效且高效地对会话序列中的上下文动态进行建模。本文提出了一种基于跨模态RoBERTa（CM-RoBERTa）模型的语音情感检测方法.作为CM-RoBERTa的核心单元，并行自注意和交叉注意被设计用于动态捕捉音频和文本的模态间和模态内交互。特别地，中层融合和残差模块被用来建模长期上下文依赖和学习模态特定模式。在MELD数据集上对该方法进行了测试，实验结果表明，该方法在MELD数据集上取得了较好的性能。摘要：Multimodal emotion recognition (MER) is a fundamental complex research problem due to the uncertainty of human emotional expression and the heterogeneity gap between different modalities. Audio and text modalities are particularly important for a human participant in understanding emotions. Although many successful attempts have been designed multimodal representations for MER, there still exist multiple challenges to be addressed: 1) bridging the heterogeneity gap between multimodal features and model inter- and intra-modal interactions of multiple modalities; 2) effectively and efficiently modelling the contextual dynamics in the conversation sequence. In this paper, we propose Cross-Modal RoBERTa (CM-RoBERTa) model for emotion detection from spoken audio and corresponding transcripts. As the core unit of the CM-RoBERTa, parallel self- and cross- attention is designed to dynamically capture inter- and intra-modal interactions of audio and text. Specially, the mid-level fusion and residual module are employed to model long-term contextual dependencies and learn modality-specific patterns. We evaluate the approach on the MELD dataset and the experimental results show the proposed approach achieves the state-of-art performance on the dataset.
【1】 Scene2BIR: Material-aware learning-based binaural impulse response generator for reconstructed real-world 3D scenes
标题：Scene2BIR：用于重建真实世界3D场景的基于材质感知学习的双耳脉冲响应发生器
链接：https://arxiv.org/abs/2302.02809
*与cs.SD语音【7】为同一篇
作者：Anton Jeran Ratnarajah,Dinesh Manocha机构：Real-world environment , D Scene, Channel, Left, Right, Geometric-based BIR, Our Scene,BIR备注：Project page: this https URL摘要：我们提出一个端对端的双耳脉冲响应发生器（BIR）来产生真实世界模型的实时似真声音。我们的方法使用一种新颖的基于神经网络的BIR生成器（Scene2BIR）用于重建3D模型。我们提出一个图形神经网络，它利用三维场景的材质和拓扑信息，生成一个场景潜在向量。此外，我们使用一个条件生成对抗网络（CGAN）从场景潜在向量生成BIRs。我们的网络能够处理重建的3D网格模型中的孔或其他伪影。我们提出了一个有效的成本函数的发生器网络，以纳入空间音频效果。给定源和收听者位置，我们的方法可以在NVIDIA GeForce RTX 2080 Ti GPU上在0.1毫秒内生成BIR，并且可以轻松处理多个源。我们已经用真实世界捕获的BIRs和交互式几何声音传播算法评估了我们的方法的准确性。摘要：We present an end-to-end binaural impulse response generator (BIR) to generate plausible sounds in real-time for real-world models. Our approach uses a novel neural-network-based BIR generator (Scene2BIR) for the reconstructed 3D model. We propose a graph neural network that uses both the material and the topology information of the 3D scenes and generates a scene latent vector. Moreover, we use a conditional generative adversarial network (CGAN) to generate BIRs from the scene latent vector. Our network is able to handle holes or other artifacts in the reconstructed 3D mesh model. We present an efficient cost function to the generator network to incorporate spatial audio effects. Given the source and the listener position, our approach can generate a BIR in 0.1 milliseconds on an NVIDIA GeForce RTX 2080 Ti GPU and can easily handle multiple sources. We have evaluated the accuracy of our approach with real-world captured BIRs and an interactive geometric sound propagation algorithm.
【2】 Residual Information in Deep Speaker Embedding Architectures
标题：深度说话人嵌入结构中的残差信息
链接：https://arxiv.org/abs/2302.02742
*与cs.SD语音【8】为同一篇
机构：Department of Communications, Technical University of Cluj-Napoca, Cluj-Napoca, Romania;摘要：说话者嵌入表示从语音信号中提取代表性矢量表示的手段，使得该表示仅与说话者身份有关。嵌入通常用于在不同说话者之间进行分类和区分。然而，目前还没有客观的方法来评估说话人嵌入将说话人身份与其他语音特征分离的能力。这意味着嵌入远非理想，高度依赖于训练语料库，并且仍然包括与诸如话语的语言内容、记录条件或说话风格之类的因素有关的一定程度的残留信息。本文介绍了对用一些最新的和高性能的DNN结构提取的六组说话人嵌入的分析，特别是它们能够真正从语音信号中分离说话人身份的程度。为了正确地评估体系结构，使用大的多说话者并行语音数据集。该数据集包括46个说话者发出相同的一组提示，记录在专业工作室或他们的家庭环境中。该分析着眼于在不同嵌入集上计算的说话者内和说话者间相似性度量，以及简单分类和回归方法是否能够从说话者嵌入中提取若干残差信息因子。结果表明，所分析的嵌入结构具有很高的区分能力，但在所有分析的结构中，残留信息仍然以与录音条件、语言内容和话语时长高度相关的形式存在于表征中。摘要：Speaker embeddings represent a means to extract representative vectorial representations from a speech signal such that the representation pertains to the speaker identity alone. The embeddings are commonly used to classify and discriminate between different speakers. However, there is no objective measure to evaluate the ability of a speaker embedding to disentangle the speaker identity from the other speech characteristics. This means that the embeddings are far from ideal, highly dependent on the training corpus and still include a degree of residual information pertaining to factors such as linguistic content, recording conditions or speaking style of the utterance. This paper introduces an analysis over six sets of speaker embeddings extracted with some of the most recent and high-performing DNN architectures, and in particular, the degree to which they are able to truly disentangle the speaker identity from the speech signal. To correctly evaluate the architectures, a large multi-speaker parallel speech dataset is used. The dataset includes 46 speakers uttering the same set of prompts, recorded in either a professional studio or their home environments. The analysis looks into the intra- and inter-speaker similarity measures computed over the different embedding sets, as well as if simple classification and regression methods are able to extract several residual information factors from the speaker embeddings. The results show that the discriminative power of the analyzed embeddings is very high, yet across all the analyzed architectures, residual information is still present in the representations in the form of a high correlation to the recording conditions, linguistic contents and utterance duration.
【3】 cross-modal fusion techniques for utterance-level emotion recognition from text and speech
标题：文本和语音语音级情感识别的跨模式融合技术
链接：https://arxiv.org/abs/2302.02447
*与cs.SD语音【9】为同一篇
作者：Jiachen Luo,Huy Phan,Joshua Reiss机构：Centre for Digital Music, Queen Mary University of London, UK, Amazon Alexa, Cambridge, MA, USA摘要：由于人类情感表达的不确定性和不同模态间的异质性差异，多模态情感识别是一个复杂的基础性研究问题。音频和文本模态对于人类参与者理解情绪尤其重要。尽管已经为MER设计了许多成功的尝试，但是仍然存在要解决的多个挑战：1）弥合多模态特征之间的异质性差距，并对多个模态的模态间和模态内交互进行建模;2）有效且高效地对会话序列中的上下文动态进行建模。本文提出了一种基于跨模态RoBERTa（CM-RoBERTa）模型的语音情感检测方法.作为CM-RoBERTa的核心单元，并行自注意和交叉注意被设计用于动态捕捉音频和文本的模态间和模态内交互。特别地，中层融合和残差模块被用来建模长期上下文依赖和学习模态特定模式。在MELD数据集上对该方法进行了测试，实验结果表明，该方法在MELD数据集上取得了较好的性能。摘要：Multimodal emotion recognition (MER) is a fundamental complex research problem due to the uncertainty of human emotional expression and the heterogeneity gap between different modalities. Audio and text modalities are particularly important for a human participant in understanding emotions. Although many successful attempts have been designed multimodal representations for MER, there still exist multiple challenges to be addressed: 1) bridging the heterogeneity gap between multimodal features and model inter- and intra-modal interactions of multiple modalities; 2) effectively and efficiently modelling the contextual dynamics in the conversation sequence. In this paper, we propose Cross-Modal RoBERTa (CM-RoBERTa) model for emotion detection from spoken audio and corresponding transcripts. As the core unit of the CM-RoBERTa, parallel self- and cross- attention is designed to dynamically capture inter- and intra-modal interactions of audio and text. Specially, the mid-level fusion and residual module are employed to model long-term contextual dependencies and learn modality-specific patterns. We evaluate the approach on the MELD dataset and the experimental results show the proposed approach achieves the state-of-art performance on the dataset.
【4】 Improved Vehicle Sub-type Classification for Acoustic Traffic Monitoring
标题：一种改进的声学交通监测车辆亚型分类
链接：https://arxiv.org/abs/2302.02945
*与cs.SD语音【1】为同一篇
作者：Mohd Ashhad,Umang Goenka,Aaryan Jagetia,Parwin Akhtari,Sooraj K. Ambat,Mary Samuel机构：Dept. of Computer Science & Engineering, Jamia Hamdard, New Delhi, India, Dept. of Information Technology, IIIT Lucknow, Lucknow, India, Dept. of Mechanical Engineering, NIT Rourkeela, Roorkela, India, Naval Physical & Oceanographic Laboratory, Defence R&D Organization备注：Accepted at Twenty-Ninth National Conference on Communications(NCC) 23 - 26 February, Indian Institute of Technology Guwahati摘要：道路上车辆的检测与分类是交通监控的一项重要任务。通常，计算机视觉（CV）算法在道路上的车辆分类任务中占主导地位，但是CV方法可能在较差的照明条件下受到影响，并且需要更大量的计算能力。此外，在敏感和安全区域安装摄像头还存在隐私问题。相比之下，声学交通监控成本效益高，并且可以提供更高的准确性，特别是在低照明条件下和在不能安装摄像机的地方。在本文中，我们考虑声学车辆子类型分类的任务，其中我们将声学信号分为4类：小汽车卡车自行车没有车。我们实验了梅尔谱图，MFCC和GFCC作为特征，并执行数据预处理，以训练一个简单的，良好优化的CNN，执行良好的任务。当使用MFCC作为特征和仔细的数据预处理时，我们提出的方法以98.95%的准确率改进了IDMT交通数据集上建立的最先进的基线。摘要：The detection and classification of vehicles on the road is a crucial task for traffic monitoring. Usually, Computer Vision (CV) algorithms dominate the task of vehicle classification on the road, but CV methodologies might suffer in poor lighting conditions and require greater amounts of computational power. Additionally, there is a privacy concern with installing cameras in sensitive and secure areas. In contrast, acoustic traffic monitoring is cost-effective, and can provide greater accuracy, particularly in low lighting conditions and in places where cameras cannot be installed. In this paper, we consider the task of acoustic vehicle sub-type classification, where we classify acoustic signals into 4 classes: car, truck, bike, and no vehicle. We experimented with Mel spectrograms, MFCC and GFCC as features and performed data pre-processing to train a simple, well optimized CNN that performs well at the task. When used with MFCC as features and careful data pre-processing, our proposed methodology improves upon the established state-of-the-art baseline on the IDMT Traffic dataset with an accuracy of 98.95%.
【5】 Audio Representation Learning by Distilling Video as Privileged Information
标题：提取视频作为特权信息的音频表征学习
链接：https://arxiv.org/abs/2302.02845
*与cs.SD语音【2】为同一篇
作者：Amirhossein Hajavi,Ali Etemad机构：The Authors of this work are in Electrical and Computer EngineeringDepartment of the Queen’s University at Kingston摘要：与单模态方法相比，使用多模态视听数据的深度音频表示学习通常导致更好的性能。然而，在现实世界场景中，两种模态在推理时并不总是可用的，从而导致针对多模态推理训练的模型的性能下降。在这项工作中，我们提出了一种新的方法，用于深度音频表示学习使用视听数据时，视频模态不存在的推理。为此，我们在使用特权信息学习（LUPI）的框架下采用师生知识蒸馏。虽然先前针对LUPI提出的方法使用教师生成的软标签，但在我们提出的方法中，我们使用教师学习的嵌入来训练学生网络。我们在两个不同的设置中集成我们的方法：连续数据，其中特征在整个时间上被划分为多个段;以及非连续数据，其中整个特征被视为一个完整段。在非顺序设置中，教师和学生网络都由编码器组件和任务头组成。我们使用由教师的编码器组件产生的嵌入来训练学生的编码器，同时使用地面实况标签来训练学生的任务报头。在顺序设置中，网络具有放置在编码器和任务报头之间的附加聚合组件。我们使用由教师的编码器和聚合组件产生的两组嵌入来训练学生。类似于非顺序设置，使用地面实况标签来训练学生网络的任务报头。我们在两个不同的音视频任务上测试了我们的框架，即说话人识别和语音情感识别，并显示出比单独的基于音频的识别以及使用LUPI的现有工作有很大的改进。摘要：Deep audio representation learning using multi-modal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI). While the previous methods proposed for LUPI use soft-labels generated by the teacher, in our proposed method we use embeddings learned by the teacher to train the student network. We integrate our method in two different settings: sequential data where the features are divided into multiple segments throughout time, and non-sequential data where the entire features are treated as one whole segment. In the non-sequential setting both the teacher and student networks are comprised of an encoder component and a task header. We use the embeddings produced by the encoder component of the teacher to train the encoder of the student, while the task header of the student is trained using ground-truth labels. In the sequential setting, the networks have an additional aggregation component that is placed between the encoder and task header. We use two sets of embeddings produced by the encoder and aggregation component of the teacher to train the student. Similar to the non-sequential setting, the task header of the student network is trained using ground-truth labels. We test our framework on two different audio-visual tasks, namely speaker recognition and speech emotion recognition and show considerable improvements over sole audio-based recognition as well as prior works that use LUPI.
【6】 deep learning of segment-level feature representation for speech emotion recognition in conversations
标题：用于对话中语音情感识别的段级特征表示深度学习
链接：https://arxiv.org/abs/2302.02419
*与cs.SD语音【3】为同一篇
作者：Jiachen Luo,Huy Phan,Joshua Reiss机构：Centre for Digital Music, Queen Mary University of London, UK, Amazon Alexa, Cambridge, MA, USA摘要：由于会话中情感的复杂性和动态性，准确检测会话中的情感是一项必要而又具有挑战性的任务。说话人的情绪状态受许多因素的影响，如对话者刺激、对话场景和话题。在这项工作中，我们提出了一个会话语音情感识别方法，以处理捕捉注意上下文依赖和说话人敏感的互动。首先，我们使用一个预先训练好的VGGish模型来提取单个话语中基于片段的音频表示。其次，注意双向门控循环单元（GRU）对语境敏感信息进行建模，并以动态的方式联合探索说话人内部和说话人之间的依存关系。在标准会话数据集MELD上进行的实验表明，与现有方法相比，所提方法是有效的。摘要：Accurately detecting emotions in conversation is a necessary yet challenging task due to the complexity of emotions and dynamics in dialogues. The emotional state of a speaker can be influenced by many different factors, such as interlocutor stimulus, dialogue scene, and topic. In this work, we propose a conversational speech emotion recognition method to deal with capturing attentive contextual dependency and speaker-sensitive interactions. First, we use a pretrained VGGish model to extract segment-based audio representation in individual utterances. Second, an attentive bi-directional gated recurrent unit (GRU) models contextual-sensitive information and explores intra- and inter-speaker dependencies jointly in a dynamic manner. The experiments conducted on the standard conversational dataset MELD demonstrate the effectiveness of the proposed method when compared against state-of the-art methods.
【7】 Multi-Source Diffusion Models for Simultaneous Music Generation and Separation
标题：同时产生和分离音乐的多源扩散模型
链接：https://arxiv.org/abs/2302.02257
*与cs.SD语音【4】为同一篇
作者：Giorgio Mariani,Irene Tallini,Emilian Postolache,Michele Mancusi,Luca Cosmo,Emanuele Rodolà机构：1Department of Computer Science, Sapienza Universityof Rome, Italy 2Department of Computer Science, Ca’ FoscariUniversity备注：Demo page: this https URL摘要：在这项工作中，我们定义了一个基于扩散的生成模型，通过学习共享上下文的源的联合概率密度的得分，该模型能够同时进行音乐合成和源分离。除了经典的完全推理任务（即生成混合、分离源）之外，我们还介绍并实验了源插补的部分推理任务，其中我们生成给定其他源的子集（例如，演奏与鼓配合良好的钢琴音轨）。此外，我们还介绍了一种新的分离任务推理方法。我们在音乐源分离的标准数据集Slakh2100上训练我们的模型，在生成设置中提供定性结果，并在分离设置中展示有竞争力的定量结果。我们的方法是可以处理生成和分离任务的单个模型的第一个例子，因此代表了向通用音频模型的一步。摘要：In this work, we define a diffusion-based generative model capable of both music synthesis and source separation by learning the score of the joint probability density of sources sharing a context. Alongside the classic total inference tasks (i.e. generating a mixture, separating the sources), we also introduce and experiment on the partial inference task of source imputation, where we generate a subset of the sources given the others (e.g., play a piano track that goes well with the drums). Additionally, we introduce a novel inference method for the separation task. We train our model on Slakh2100, a standard dataset for musical source separation, provide qualitative results in the generation settings, and showcase competitive quantitative results in the separation setting. Our method is the first example of a single model that can handle both generation and separation tasks, thus representing a step toward general audio models.
【8】 AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis
标题：用于真实视听场景合成的学习神经场
链接：https://arxiv.org/abs/2302.02088
*与cs.SD语音【5】为同一篇
作者：Susan Liang,Chao Huang,Yapeng Tian,Anurag Kumar,Chenliang Xu机构：University of Rochester, University of Texas at Dallas, Meta Reality Labs Research摘要：人类对复杂世界的感知依赖于对多模态信号的综合分析，而音频和视频信号的同现为人类提供了丰富的线索。本文主要研究现实世界中新颖的视听场景合成。给定视听场景的视频记录，任务是沿着该视听场景中任意新颖的摄像机轨迹合成具有空间音频的新视频。直接使用基于NeRF的模型进行音频合成是不够的，因为其缺乏先验知识和声学监督。为了应对这些挑战，我们首先提出了一种声学感知的音频生成模块，该模块将我们对音频传播的先验知识集成到NeRF中，其中我们将音频生成与视觉环境的3D几何结构相关联。此外，我们提出了一个坐标转换模块，用于表示相对于声源的观看方向。这样的方向变换帮助模型学习以声源为中心的声场。此外，我们利用头部相关脉冲响应函数来合成伪双耳音频，用于增强训练的数据扩增。我们定性和定量地证明了我们的模型在真实世界视听场景中的优势。我们建议感兴趣的读者查看我们的视频结果，以进行令人信服的比较。摘要：Human perception of the complex world relies on a comprehensive analysis of multi-modal signals, and the co-occurrences of audio and video signals provide humans with rich cues. This paper focuses on novel audio-visual scene synthesis in the real world. Given a video recording of an audio-visual scene, the task is to synthesize new videos with spatial audios along arbitrary novel camera trajectories in that audio-visual scene. Directly using a NeRF-based model for audio synthesis is insufficient due to its lack of prior knowledge and acoustic supervision. To tackle the challenges, we first propose an acoustic-aware audio generation module that integrates our prior knowledge of audio propagation into NeRF, in which we associate audio generation with the 3D geometry of the visual environment. In addition, we propose a coordinate transformation module that expresses a viewing direction relative to the sound source. Such a direction transformation helps the model learn sound source-centric acoustic fields. Moreover, we utilize a head-related impulse response function to synthesize pseudo binaural audio for data augmentation that strengthens training. We qualitatively and quantitatively demonstrate the advantage of our model on real-world audio-visual scenes. We refer interested readers to view our video results for convincing comparisons.
【9】 PSST! Prosodic Speech Segmentation with Transformers
标题：嘘！基于变形器的韵律语音分割
链接：https://arxiv.org/abs/2302.01984
*与cs.SD语音【6】为同一篇
作者：Nathan Roll,Calbert Graham,Simon Todd机构：University of California, Santa Barbara, USA, University of Cambridge, UK备注：5 pages, 3 figures. For associated repository, see this https URL摘要：自注意机制使Transformers在许多语音到文本（STT）任务上取得了超人级别的性能，但自动韵律分割的挑战仍然没有得到解决。本文对预训练的STT模型Whisper进行了微调，通过重新利用低频标记来标注语调单元（IU）边界。我们的方法达到了95.8%的准确率，优于以往的方法，而不需要大规模的标记数据或企业级计算资源。我们还通过应用一系列滤波器来减少输入信号，发现3.2 kHz水平的低通滤波器改善了样本外和分布外环境中的分割性能。我们发布我们的模型作为转录工具和基线进一步改善韵律分割。摘要：Self-attention mechanisms have enabled transformers to achieve superhuman-level performance on many speech-to-text (STT) tasks, yet the challenge of automatic prosodic segmentation has remained unsolved. In this paper we finetune Whisper, a pretrained STT model, to annotate intonation unit (IU) boundaries by repurposing low-frequency tokens. Our approach achieves an accuracy of 95.8%, outperforming previous methods without the need for large-scale labeled data or enterprise grade compute resources. We also diminish input signals by applying a series of filters, finding that low pass filters at a 3.2 kHz level improve segmentation performance in out of sample and out of distribution contexts. We release our model as both a transcription tool and a baseline for further improvements in prosodic segmentation.
机器翻译，仅供参考
每日语音速递