今日论文合集：cs.SD语音18篇，eess.AS音频处理19篇。【1】Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism
标题：神经文本发音：实现听觉和照片真实感的深层文本到视听语音合成链接：https://arxiv.org/abs/2312.06613作者：Georgios Milis,Panagiotis P. Filntisis,Anastasios Roussos,Petros Maragos摘要：最近在序列数据深度学习方面的进展已经产生了快速而强大的模型，可以生成逼真的人类对话视频。在说话的脸生成的艺术状态主要集中在嘴唇同步，被限制在音频剪辑。然而，具有从文本传输而不是音频合成说话人的能力对于许多应用特别有益，并且随着最近在大型语言模型方面的突破，预计将受到越来越多的关注。为此，大多数方法实现了文本到语音模块的级联2级架构，然后是音频驱动的说话面部生成器，但这忽略了说话期间发生的音频和视频流之间高度复杂的相互作用。在本文中，我们提出了第一，据我们所知，文本驱动的视听语音合成器，使用Transformers，不遵循级联的方法。我们的方法，我们称之为神经文本到ARticulate谈话（NEUTART），是一个说话的脸生成器，使用联合视听特征空间，以及语音知情的3D面部重建和唇读损失的视觉监督。该模型产生逼真的说话人脸视频与人类一样的清晰度和良好同步的视听流。我们在视听数据集以及野外视频上的实验揭示了在客观指标和人类评估方面最先进的生成质量。摘要：Recent advances in deep learning for sequential data have given rise to fast and powerful models that produce realistic videos of talking humans. The state of the art in talking face generation focuses mainly on lip-syncing, being conditioned on audio clips. However, having the ability to synthesize talking humans from text transcriptions rather than audio is particularly beneficial for many applications and is expected to receive more and more attention, following the recent breakthroughs in large language models. For that, most methods implement a cascaded 2-stage architecture of a text-to-speech module followed by an audio-driven talking face generator, but this ignores the highly complex interplay between audio and visual streams that occurs during speaking. In this paper, we propose the first, to the best of our knowledge, text-driven audiovisual speech synthesizer that uses Transformers and does not follow a cascaded approach. Our method, which we call NEUral Text to ARticulate Talk (NEUTART), is a talking face generator that uses a joint audiovisual feature space, as well as speech-informed 3D facial reconstructions and a lip-reading loss for visual supervision. The proposed model produces photorealistic talking face videos with human-like articulation and well-synced audiovisual streams. Our experiments on audiovisual datasets as well as in-the-wild videos reveal state-of-the-art generation quality both in terms of objective metrics and human evaluation.【2】 Deep Photonic Reservoir Computer for Speech Recognition标题：用于语音识别的深度光子库计算机链接：https://arxiv.org/abs/2312.06558作者：Enrico Picco,Alessandro Lupo,Serge Massar摘要：语音识别是人工智能领域的一项关键任务，由于大型复杂的神经网络，其训练过程通常需要大量的标记数据和计算密集型操作，语音识别已经取得了显着的进步。另一种范例，水库计算，是能源效率，并很好地适应在物理基板上的实现，但表现出的性能相比，更资源密集型的机器学习算法的限制。在这项工作中，我们通过调查互连水库的不同架构来应对这一挑战，所有这些都属于深层水库计算的范畴。我们提出了一种基于光子的深水库计算机，并评估其在不同的语音识别任务的有效性。我们展示了具体的设计选择，旨在简化水库计算机的实际实现，同时实现高维音频信号的高速处理。总的来说，通过目前的工作，我们希望能够帮助低功耗和高性能的神经形态硬件的进步。摘要：Speech recognition is a critical task in the field of artificial intelligence and has witnessed remarkable advancements thanks to large and complex neural networks, whose training process typically requires massive amounts of labeled data and computationally intensive operations. An alternative paradigm, reservoir computing, is energy efficient and is well adapted to implementation in physical substrates, but exhibits limitations in performance when compared to more resource-intensive machine learning algorithms. In this work we address this challenge by investigating different architectures of interconnected reservoirs, all falling under the umbrella of deep reservoir computing. We propose a photonic-based deep reservoir computer and evaluate its effectiveness on different speech recognition tasks. We show specific design choices that aim to simplify the practical implementation of a reservoir computer while simultaneously achieving high-speed processing of high-dimensional audio signals. Overall, with the present work we hope to help the advancement of low-power and high-performance neuromorphic hardware.【3】 Towards Domain-Specific Cross-Corpus Speech Emotion Recognition Approach标题：面向特定领域的跨语料库语音情感识别方法链接：https://arxiv.org/abs/2312.06466作者：Yan Zhao,Yuan Zong,Hailun Lian,Cheng Lu,Jingang Shi,Wenming Zheng摘要：跨语料库语音情感识别（SER）提出了一个挑战，由于特征分布不匹配，潜在地降低了既定的SER方法的性能。在本文中，我们提出了一种新的转移子空间学习方法，称为声学知识引导转移线性回归（AKTLR）来应对这一挑战。与现有的方法，这往往忽略了特定领域的知识相关的SER和简单地处理跨语料库SER作为一个通用的迁移学习任务，我们的AKTLR方法是建立在一个精心设计的声学知识指导的双稀疏约束机制。这种机制强调了最小的声学参数特征集的潜力，以减轻分类器过度适应，这是经验验证的声学知识在SER，使优越的泛化跨语料库SER任务相比，使用大的特征集。通过这种机制，我们扩展了一个简单的转移线性回归模型到AKTLR。该扩展利用其全部能力，以寻求emotiondiscriminative和语料库不变的功能，从建立的声学参数特征集用于描述语音信号在两个尺度：贡献的声学参数组和每个贡献组内的组成元素。我们所提出的方法进行了评估，通过广泛的跨语料库SER实验三个广泛使用的语音情感语料库：ENUDB，ENTERFACE，和CASIA。结果证实了我们方法的有效性和优越性能，优于最近最先进的迁移子空间学习和基于深度迁移学习的跨语料库SER方法。此外，我们的工作提供了实验证据，支持将特定领域的知识迁移学习模型，以解决跨语料库SER任务的可行性和优越性。摘要：Cross-corpus speech emotion recognition (SER) poses a challenge due to feature distribution mismatch, potentially degrading the performance of established SER methods. In this paper, we tackle this challenge by proposing a novel transfer subspace learning method called acoustic knowledgeguided transfer linear regression (AKTLR). Unlike existing approaches, which often overlook domain-specific knowledge related to SER and simply treat cross-corpus SER as a generic transfer learning task, our AKTLR method is built upon a well-designed acoustic knowledge-guided dual sparsity constraint mechanism. This mechanism emphasizes the potential of minimalistic acoustic parameter feature sets to alleviate classifier overadaptation, which is empirically validated acoustic knowledge in SER, enabling superior generalization in cross-corpus SER tasks compared to using large feature sets. Through this mechanism, we extend a simple transfer linear regression model to AKTLR. This extension harnesses its full capability to seek emotiondiscriminative and corpus-invariant features from established acoustic parameter feature sets used for describing speech signals across two scales: contributive acoustic parameter groups and constituent elements within each contributive group. Our proposed method is evaluated through extensive cross-corpus SER experiments on three widely-used speech emotion corpora: EmoDB, eNTERFACE, and CASIA. The results confirm the effectiveness and superior performance of our method, outperforming recent state-of-the-art transfer subspace learning and deep transfer learning-based cross-corpus SER methods. Furthermore, our work provides experimental evidence supporting the feasibility and superiority of incorporating domain-specific knowledge into the transfer learning model to address cross-corpus SER tasks.【4】 Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation标题：合作很重要：探索视听分割的多阶双边关系链接：https://arxiv.org/abs/2312.06462作者：Qi Yang,Xing Nie,Tong Li,Pengfei Gao,Ying Guo,Cheng Zhen,Pengfei Yan,Shiming Xiang备注：13 pages, 10 figures摘要：最近，已经引入了视听分割（AVS）任务，旨在将给定视频内的具有发声对象的像素分组。这项任务需要对场景进行首次音频驱动的像素级理解，这带来了重大挑战。在本文中，我们提出了一个创新的视听Transformer框架，称为COMBO，多阶双边关系的合作的缩写。第一次，我们的框架探讨了AVS中的三种双边纠缠：像素纠缠，模态纠缠和时间纠缠。关于像素纠缠，我们采用了Siam编码器模块（SEM），该模块利用先验知识从基础模型中生成更精确的视觉特征。对于模态纠缠，我们设计了一个双边融合模块（BFM），使COMBO能够双向对齐相应的视觉和听觉信号。在时间纠缠方面，根据时间纠缠的内在规律，提出了一种自适应的帧间一致性丢失算法。对AVSBench-object（S4上84.7 mIoU，MS 3上59.2 mIou）和AVSBench-semantic（AVSS上42.1 mIoU）数据集的综合实验和消融研究表明，COMBO超过了以前最先进的方法。代码和更多结果将在https://combo-avs.github.io/上公开。摘要：Recently, an audio-visual segmentation (AVS) task has been introduced, aiming to group pixels with sounding objects within a given video. This task necessitates a first-ever audio-driven pixel-level understanding of the scene, posing significant challenges. In this paper, we propose an innovative audio-visual transformer framework, termed COMBO, an acronym for COoperation of Multi-order Bilateral relatiOns. For the first time, our framework explores three types of bilateral entanglements within AVS: pixel entanglement, modality entanglement, and temporal entanglement. Regarding pixel entanglement, we employ a Siam-Encoder Module (SEM) that leverages prior knowledge to generate more precise visual features from the foundational model. For modality entanglement, we design a Bilateral-Fusion Module (BFM), enabling COMBO to align corresponding visual and auditory signals bi-directionally. As for temporal entanglement, we introduce an innovative adaptive inter-frame consistency loss according to the inherent rules of temporal. Comprehensive experiments and ablation studies on AVSBench-object (84.7 mIoU on S4, 59.2 mIou on MS3) and AVSBench-semantic (42.1 mIoU on AVSS) datasets demonstrate that COMBO surpasses previous state-of-the-art methods. Code and more results will be publicly available at https://combo-avs.github.io/.【5】 Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations标题：深度不平衡学习在对话中多通道情绪识别中的应用链接：https://arxiv.org/abs/2312.06337作者：Tao Meng,Yuntao Shou,Wei Ai,Nan Yin,Keqin Li备注：16 pages, 9 figures摘要：会话中的多模态情感识别（MERC）的主要任务是识别模态中的情感，例如，文本、音频、图像和视频，这是实现机器智能的一个重要发展方向。然而，MERC中的许多数据自然地表现出情感类别的不平衡分布，研究人员忽略了不平衡数据对情感识别的负面影响。针对这一问题，本文从数据扩充、损失敏感性和采样策略三个方面进行了系统分析，提出了类边界增强表示学习（CBERL）模型。具体来说，我们首先设计了一个多模态生成对抗网络，以解决原始数据中{情感}类别分布不平衡的问题。其次，提出了一种深度联合变分自编码器，以融合跨模态的互补语义信息，并获得有区别的特征表示。最后，我们实现了一个多任务的图神经网络与掩模重建和分类优化，以解决类边界学习中的过拟合和欠拟合问题，并实现跨模态情感识别。我们在IEMOCAP和MELD基准数据集上进行了大量的实验，结果表明CBERL在情感识别的有效性上取得了一定的性能提升。特别是在少数类别的恐惧和厌恶情绪标签上，我们的模型将准确率和F1值提高了10%到20%。摘要：The main task of Multimodal Emotion Recognition in Conversations (MERC) is to identify the emotions in modalities, e.g., text, audio, image and video, which is a significant development direction for realizing machine intelligence. However, many data in MERC naturally exhibit an imbalanced distribution of emotion categories, and researchers ignore the negative impact of imbalanced data on emotion recognition. To tackle this problem, we systematically analyze it from three aspects: data augmentation, loss sensitivity, and sampling strategy, and propose the Class Boundary Enhanced Representation Learning (CBERL) model. Concretely, we first design a multimodal generative adversarial network to address the imbalanced distribution of {emotion} categories in raw data. Secondly, a deep joint variational autoencoder is proposed to fuse complementary semantic information across modalities and obtain discriminative feature representations. Finally, we implement a multi-task graph neural network with mask reconstruction and classification optimization to solve the problem of overfitting and underfitting in class boundary learning, and achieve cross-modal emotion recognition. We have conducted extensive experiments on the IEMOCAP and MELD benchmark datasets, and the results show that CBERL has achieved a certain performance improvement in the effectiveness of emotion recognition. Especially on the minority class fear and disgust emotion labels, our model improves the accuracy and F1 value by 10% to 20%.【6】 Transformer Attractors for Robust and Efficient End-to-End Neural Diarization标题：用于稳健有效的端到端神经网络二值化的Transformer吸引子链接：https://arxiv.org/abs/2312.06253作者：Lahiru Samarakoon,Samuel J. Broughton,Marc Harkönen,Ivan Fung备注：8 pages, 1 figure, ASRU2023摘要：基于吸引子的端到端神经日志化（End-to-end neural diarization with encoder-decoder based attractor，EEND-EDA）是一种在单个神经网络中执行日志化的方法。EDA通过使用基于LSTM的编码器-解码器来处理灵活数量的扬声器的日记化，该编码器-解码器以自回归方式生成一组扬声器吸引子。在本文中，我们建议用基于变换的吸引子计算（TA）模块来取代EDA。TA由Combiner模块和Transformer解码器组成。组合器块的主要功能是通过将学习到的会话信息合并到嵌入的全局集合中来生成会话相关（CD）嵌入。然后，这些CD嵌入将用作Transformer解码器的输入。在公开数据集上的结果表明，EEND-TA比EEND-EDA实现了2.68%的绝对DER改进。EEND-TA的推理速度是EEND-EDA的1.28倍。摘要：End-to-end neural diarization with encoder-decoder based attractors (EEND-EDA) is a method to perform diarization in a single neural network. EDA handles the diarization of a flexible number of speakers by using an LSTM-based encoder-decoder that generates a set of speaker-wise attractors in an autoregressive manner. In this paper, we propose to replace EDA with a transformer-based attractor calculation (TA) module. TA is composed of a Combiner block and a Transformer decoder. The main function of the combiner block is to generate conversational dependent (CD) embeddings by incorporating learned conversational information into a global set of embeddings. These CD embeddings will then serve as the input for the transformer decoder. Results on public datasets show that EEND-TA achieves 2.68% absolute DER improvement over EEND-EDA. EEND-TA inference is 1.28 times faster than that of EEND-EDA.【7】 Music-PAW: Learning Music Representations via Hierarchical Part-whole Interaction and Contrast标题：音乐-PAW：通过层次化的部分-整体互动和对比学习音乐表征链接：https://arxiv.org/abs/2312.06197作者：Dong Yao,Shengyu Zhang,Zhou Zhao,Jieming Zhu,Liqun Deng,Wenqiao Zhang,Zhenhua Dong,Ruiming Tang,Xin Jiang摘要：最近的自监督学习方法在各种下游任务上的出色表现引起了学术界和工业界的极大关注。最近的一些研究致力于自我监督的音乐表征学习。然而，他们中的大多数人学会了在波形或频谱图中表示相等大小的音乐片段。尽管在某些任务中是有效的，但以这种方式学习音乐表征在很大程度上忽视了音乐固有的部分-整体层次结构。由于听觉皮层的分层性质[24]，理解音乐的自下而上的结构，即，不同的部分如何在不同的层次上构成整体，是音乐理解和表征学习的关键。这项工作追求分层音乐表示学习，并引入了Music-PAW框架，该框架使裁剪的音乐片段与部分-整体层次结构的特征交互成为可能。从技术的角度来看，我们提出了一个基于transformer的部分-整体交互模块，逐步原因部分-整体音乐片段之间的结构关系在相邻的水平。此外，为了建立一个多层次的表示空间，我们设计了一个分层的对比学习目标，以对齐相邻层次中的部分-整体音乐表示。从部分-整体层次结构中学习音频表示的优点已经在各种下游任务中得到验证，包括音乐分类（单标签和多标签），翻唱歌曲识别和声学场景分类。摘要：The excellent performance of recent self-supervised learning methods on various downstream tasks has attracted great attention from academia and industry. Some recent research efforts have been devoted to self-supervised music representation learning. Nevertheless, most of them learn to represent equally-sized music clips in the waveform or a spectrogram. Despite being effective in some tasks, learning music representations in such a manner largely neglect the inherent part-whole hierarchies of music. Due to the hierarchical nature of the auditory cortex [24], understanding the bottom-up structure of music, i.e., how different parts constitute the whole at different levels, is essential for music understanding and representation learning. This work pursues hierarchical music representation learning and introduces the Music-PAW framework, which enables feature interactions of cropped music clips with part-whole hierarchies. From a technical perspective, we propose a transformer-based part-whole interaction module to progressively reason the structural relationships between part-whole music clips at adjacent levels. Besides, to create a multi-hierarchy representation space, we devise a hierarchical contrastive learning objective to align part-whole music representations in adjacent hierarchies. The merits of audio representation learning from part-whole hierarchies have been validated on various downstream tasks, including music classification (single-label and multi-label), cover song identification and acoustic scene classification.【8】 ROSE: A Recognition-Oriented Speech Enhancement Framework in Air Traffic Control Using Multi-Objective Learning标题：ROSE：一种基于多目标学习的空中交通管制语音增强框架链接：https://arxiv.org/abs/2312.06118作者：Xincheng Yu,Dongyue Guo,Jianwei Zhang,Yi Lin摘要：无线电语音回波是空中交通管制（ATC）领域的一种特殊现象，它会降低语音质量，进而影响自动语音识别（ASR）的准确性。在这项工作中，提出了一个面向语音增强（ROSE）的框架，以提高语音清晰度，并提高ASR的准确性，它作为一个即插即用的工具，在ATC的情况下，不需要额外的重新训练的ASR模型。具体而言，提出了一个基于编码器-解码器的U-Net框架，以消除无线电语音回声的基础上，真实世界收集的语料库。通过结合面向SE和面向ASR的损失，ROSE通过学习两个优化目标的共享表示以多目标方式实现。一个基于注意力的跳跃融合（ABSF）机制被应用到跳过连接，以细化的功能。创新性地设计了通道和序列注意（CSAtt）块，以引导模型专注于信息表示并抑制干扰特征。实验结果表明，ROSE显着优于其他国家的最先进的方法的SE和ASR任务。此外，所提出的方法可以有助于在公共数据集上实现所需的性能改进。摘要：Radio speech echo is a specific phenomenon in the air traffic control (ATC) domain, which degrades speech quality and further impacts automatic speech recognition (ASR) accuracy. In this work, a recognition-oriented speech enhancement (ROSE) framework is proposed to improve speech intelligibility and also advance ASR accuracy, which serves as a plug-and-play tool in ATC scenarios and does not require additional retraining of the ASR model. Specifically, an encoder-decoder-based U-Net framework is proposed to eliminate the radio speech echo based on the real-world collected corpus. By incorporating the SE-oriented and ASR-oriented loss, ROSE is implemented in a multi-objective manner by learning shared representations across the two optimization objectives. An attention-based skip-fusion (ABSF) mechanism is applied to skip connections to refine the features. A channel and sequence attention (CSAtt) block is innovatively designed to guide the model to focus on informative representations and suppress disturbing features. The experimental results show that the ROSE significantly outperforms other state-of-the-art methods for both the SE and ASR tasks. In addition, the proposed approach can contribute to the desired performance improvements on public datasets.【9】 Speaker-Text Retrieval via Contrastive Learning标题：基于对比学习的说话人-文本检索链接：https://arxiv.org/abs/2312.06055作者：Xuechen Liu,Xin Wang,Erica Cooper,Xiaoxiao Miao,Junichi Yamagishi备注：Submitted to IEEE Signal Processing Letters摘要：在这项研究中，我们介绍了一种新的跨模态检索任务，涉及说话人的描述和相应的音频样本。利用预先训练的扬声器和文本编码器，我们提出了一个简单的基于对比学习的学习框架。此外，我们还探讨了将说话人标签纳入训练过程的影响。我们的研究结果建立了有效的连接扬声器和文本信息的任务，英语和日语，在不同的数据配置。额外的视觉分析揭示了说话人聚类和检索性能之间潜在的细微差别。摘要：In this study, we introduce a novel cross-modal retrieval task involving speaker descriptions and their corresponding audio samples. Utilizing pre-trained speaker and text encoders, we present a simple learning framework based on contrastive learning. Additionally, we explore the impact of incorporating speaker labels into the training process. Our findings establish the effectiveness of linking speaker and text information for the task for both English and Japanese languages, across diverse data configurations. Additional visual analysis unveils potential nuanced associations between speaker clustering and retrieval performance.【10】 A Practical Survey on Emerging Threats from AI-driven Voice Attacks: How Vulnerable are Commercial Voice Control Systems?标题：人工智能驱动的语音攻击新威胁的实践调查：商业语音控制系统有多脆弱？链接：https://arxiv.org/abs/2312.06010作者：Yuanda Wang,Qiben Yan,Nikolay Ivanov,Xun Chen备注：14 pages摘要：人工智能（AI）驱动的音频攻击的出现揭示了语音控制系统中新的安全漏洞。虽然研究人员已经引入了多种针对语音控制系统（VoIP）的攻击策略，但VoIP的不断进步已经减少了许多此类攻击的影响。认识到这一动态格局，我们的研究努力全面评估商业语音控制系统对恶意音频攻击的弹性。通过广泛的实验，我们评估了六种主要的攻击技术，包括语音控制接口和设备。与流行的说法相反，我们的研究结果表明，商业语音控制系统对现有威胁表现出更强的抵抗力。特别是，我们的研究强调了白盒攻击在黑盒场景中的无效性。此外，对手在与商业系统（如Apple Siri和Samsung Bixby）进行基于查询的交互时，在获得精确的梯度估计方面遇到了巨大的障碍。同时，我们发现，目前的防御策略并不能完全免疫于高级攻击。我们的研究结果有助于提高防御机制的重要见解。通过这项调查，我们的目标是提高学术界对网络安全问题的认识，并倡导在这一关键领域继续研究。摘要：The emergence of Artificial Intelligence (AI)-driven audio attacks has revealed new security vulnerabilities in voice control systems. While researchers have introduced a multitude of attack strategies targeting voice control systems (VCS), the continual advancements of VCS have diminished the impact of many such attacks. Recognizing this dynamic landscape, our study endeavors to comprehensively assess the resilience of commercial voice control systems against a spectrum of malicious audio attacks. Through extensive experimentation, we evaluate six prominent attack techniques across a collection of voice control interfaces and devices. Contrary to prevailing narratives, our results suggest that commercial voice control systems exhibit enhanced resistance to existing threats. Particularly, our research highlights the ineffectiveness of white-box attacks in black-box scenarios. Furthermore, the adversaries encounter substantial obstacles in obtaining precise gradient estimations during query-based interactions with commercial systems, such as Apple Siri and Samsung Bixby. Meanwhile, we find that current defense strategies are not completely immune to advanced attacks. Our findings contribute valuable insights for enhancing defense mechanisms in VCS. Through this survey, we aim to raise awareness within the academic community about the security concerns of VCS and advocate for continued research in this crucial area.【11】 mir_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks标题：mir_ref：一个音乐信息检索任务的表示评估框架链接：https://arxiv.org/abs/2312.05994作者：Christos Plachouras,Pablo Alonson-Jiménez,Dmitry Bogdanov备注：Machine Learning for Audio Workshop, Neural Information Processing Systems (NeurIPS) 2023, New Orleans, LA摘要：音乐信息检索（MIR）研究越来越多地利用表示学习来获得更紧凑，更强大的音乐音频表示，用于各种下游MIR任务。然而，由于音频和标签预处理、下游模型和度量实现、数据可用性和计算资源的差异，当前的表示评估方法是分散的，通常导致不一致和有限的结果。在这项工作中，我们介绍了mir_ref，MIR表示评估框架专注于无缝，透明，本地优先的实验编排，以支持表示开发。它实现了各种组件，如MIR数据集，任务，嵌入模型以及用于结果分析和可视化的工具，同时促进了自定义组件的实现。为了证明它的实用性，我们使用它对各种任务和数据集的几个嵌入模型进行了广泛的评估，包括评估它们对各种音频扰动的鲁棒性以及从中提取相关信息的容易性。摘要：Music Information Retrieval (MIR) research is increasingly leveraging representation learning to obtain more compact, powerful music audio representations for various downstream MIR tasks. However, current representation evaluation methods are fragmented due to discrepancies in audio and label preprocessing, downstream model and metric implementations, data availability, and computational resources, often leading to inconsistent and limited results. In this work, we introduce mir_ref, an MIR Representation Evaluation Framework focused on seamless, transparent, local-first experiment orchestration to support representation development. It features implementations of a variety of components such as MIR datasets, tasks, embedding models, and tools for result analysis and visualization, while facilitating the implementation of custom components. To demonstrate its utility, we use it to conduct an extensive evaluation of several embedding models across various tasks and datasets, including evaluating their robustness to various audio perturbations and the ease of extracting relevant information from them.【12】 Voice Activity Detection (VAD) in Noisy Environments标题：噪声环境中的语音活动检测(VAD)链接：https://arxiv.org/abs/2312.05815作者：Joshua Ball备注：7 pages摘要：在数字音频处理领域，语音活动检测（VAD）在区分语音和非语音元素方面起着关键作用，这是一项在嘈杂环境中变得越来越复杂的任务。本文详细介绍了VAD系统的开发和实施，专门设计，以保持在各种环境噪声的存在下的高精度。我们引入了一种新的算法，增强了一个专门设计的过滤技术，有效地隔离语音，即使在不同的背景声音。我们全面的测试和验证证明了该系统的鲁棒性，突出了它的能力，以显着的精度从噪声中辨别语音。探索深入到：（1）VAD的核心原理及其在现代音频处理中的关键作用;（2）我们用于过滤环境噪声的方法;（3）证明我们的系统在嘈杂环境中具有卓越性能的证据。摘要：In the realm of digital audio processing, Voice Activity Detection (VAD) plays a pivotal role in distinguishing speech from non-speech elements, a task that becomes increasingly complex in noisy environments. This paper details the development and implementation of a VAD system, specifically engineered to maintain high accuracy in the presence of various ambient noises. We introduce a novel algorithm enhanced with a specially designed filtering technique, effectively isolating speech even amidst diverse background sounds. Our comprehensive testing and validation demonstrate the system's robustness, highlighting its capability to discern speech from noise with remarkable precision. The exploration delves into: (1) the core principles underpinning VAD and its crucial role in modern audio processing; (2) the methodologies we employed to filter ambient noise; and (3) a presentation of evidence affirming our system's superior performance in noisy conditions.【13】 Neural Speech Embeddings for Speech Synthesis Based on Deep Generative Networks标题：基于深度生成网络的神经语音嵌入语音合成链接：https://arxiv.org/abs/2312.05814作者：Seo-Hyun Lee,Young-Eun Lee,Soowon Kim,Byung-Kwan Ko,Jun-Young Kim,Seong-Whan Lee备注：4 pages摘要：脑语转换技术代表了人工智能、脑机接口和语音合成等跨学科应用的融合。基于意图解码和语音合成的神经表征学习将神经活动与人类语言交际手段直接联系起来，可以大大提高交际的自然性。随着表征学习理论的不断发展和语音合成技术的不断进步，将脑电信号直接转换为语音已经显示出了巨大的潜力。特别是，当使用深度生成模型从大脑信号生成语音时，给予神经网络的处理后的输入特征和神经语音嵌入在整体性能中起着重要作用。在本文中，我们介绍了目前的脑到语音技术与语音合成的可能性，从大脑信号，这可能最终促进非语言交流的创新。此外，我们进行了全面的分析，神经功能和神经语音嵌入的神经生理激活，而执行的语音，这可能会发挥重要作用的语音合成工作。摘要：Brain-to-speech technology represents a fusion of interdisciplinary applications encompassing fields of artificial intelligence, brain-computer interfaces, and speech synthesis. Neural representation learning based intention decoding and speech synthesis directly connects the neural activity to the means of human linguistic communication, which may greatly enhance the naturalness of communication. With the current discoveries on representation learning and the development of the speech synthesis technologies, direct translation of brain signals into speech has shown great promise. Especially, the processed input features and neural speech embeddings which are given to the neural network play a significant role in the overall performance when using deep generative models for speech generation from brain signals. In this paper, we introduce the current brain-to-speech technology with the possibility of speech synthesis from brain signals, which may ultimately facilitate innovation in non-verbal communication. Also, we perform comprehensive analysis on the neural features and neural speech embeddings underlying the neurophysiological activation while performing speech, which may play a significant role in the speech synthesis works.【14】 Keyword spotting -- Detecting commands in speech using deep learning标题：关键词识别--利用深度学习检测语音中的命令链接：https://arxiv.org/abs/2312.05640作者：Sumedha Rai,Tong Li,Bella Lyu摘要：语音识别已成为机器学习和人工智能发展的重要课题。在这项研究中，我们探索了使用语音识别机器学习和深度学习技术进行关键字识别的重要任务。我们通过将原始波形转换为Mel频率倒谱系数（MFCC）来实现特征工程，我们将其用作模型的输入。我们实验了几种不同的算法，如高斯混合的隐马尔可夫模型，卷积神经网络和递归神经网络的变体，包括长短期记忆和注意力机制。在我们的实验中，带有BiLSTM和Attention的RNN实现了最佳性能，准确率为93.9%摘要：Speech recognition has become an important task in the development of machine learning and artificial intelligence. In this study, we explore the important task of keyword spotting using speech recognition machine learning and deep learning techniques. We implement feature engineering by converting raw waveforms to Mel Frequency Cepstral Coefficients (MFCCs), which we use as inputs to our models. We experiment with several different algorithms such as Hidden Markov Model with Gaussian Mixture, Convolutional Neural Networks and variants of Recurrent Neural Networks including Long Short-Term Memory and the Attention mechanism. In our experiments, RNN with BiLSTM and Attention achieves the best performance with an accuracy of 93.9 %【15】 An Experimental Study: Assessing the Combined Framework of WavLM and BEST-RQ for Text-to-Speech Synthesis标题：WavLM与BEST-RQ相结合的文语合成框架的实验研究链接：https://arxiv.org/abs/2312.05415作者：Via Nielson,Steven Hillis备注：8 pages, 1 figure, 4 tables摘要：我们提出了一种新的模型架构，特别适合于文本到语音（TTS）模型。我们结合WavLM，一个预先训练的自监督学习（SSL）语音模型，和BEST-RQ矢量量化框架。我们评估在何种程度上更任务不可知的WavLM，再加上优越的适用性的简单化BEST-RQ框架的下游任务的更广泛的阵列，产生有利的结果。在LibriSpeech数据集上进行的SUPERB基准测试表明，所提出的模型表现明显不佳。我们推测这种性能的根本原因与量化原始音频波形和频谱图之间的差异有关。我们讨论了这种方法的局限性，以更好地指导未来的发展TTS。摘要：We propose a new model architecture specifically suited for text-to-speech (TTS) models. We combine WavLM, a pre-trained self-supervised learning (SSL) speech model, and the BEST-RQ vector quantization framework. We assess the extent to which the more task-agnostic WavLM, coupled with the superior suitability of the simplistic BEST-RQ framework for a wider array of downstream tasks, yields favorable outcomes. Experiments on the LibriSpeech dataset with SUPERB benchmarking assert that the proposed model significantly underperforms. We speculate the underlying reason for this performance is related to the difference between featurizing raw audio waveforms and spectrograms with a quantizer. We discuss the limitations of this approach to better guide future advancements in TTS.【16】 CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling标题：CMMD：用于视频-音频条件建模的对比多模扩散链接：https://arxiv.org/abs/2312.05412作者：Ruihan Yang,Hannes Gamper,Sebastian Braun摘要：我们介绍了一个多模态扩散模型定制的双向条件生成的视频和音频。认识到多模态生成任务中视频和音频事件之间准确对齐的重要性，我们提出了一种联合对比训练损失，以增强视觉和听觉事件之间的同步。我们的研究方法包括在多个数据集上进行综合实验，以彻底评估我们提出的模型的有效性。生成质量和对齐性能的评估是从各个角度进行的，包括客观和主观的度量。我们的研究结果表明，所提出的模型优于基线，证实其有效性和效率。值得注意的是，对比度损失的引入导致视听对齐的改善，特别是在高相关性视频到音频生成任务中。这些结果表明，我们提出的模型作为一个强大的解决方案，提高多模态生成的质量和对齐的潜力，从而有助于视频和音频条件生成系统的进步。摘要：We introduce a multi-modal diffusion model tailored for the bi-directional conditional generation of video and audio. Recognizing the importance of accurate alignment between video and audio events in multi-modal generation tasks, we propose a joint contrastive training loss to enhance the synchronization between visual and auditory occurrences. Our research methodology involves conducting comprehensive experiments on multiple datasets to thoroughly evaluate the efficacy of our proposed model. The assessment of generation quality and alignment performance is carried out from various angles, encompassing both objective and subjective metrics. Our findings demonstrate that the proposed model outperforms the baseline, substantiating its effectiveness and efficiency. Notably, the incorporation of the contrastive loss results in improvements in audio-visual alignment, particularly in the high-correlation video-to-audio generation task. These results indicate the potential of our proposed model as a robust solution for improving the quality and alignment of multi-modal generation, thereby contributing to the advancement of video and audio conditional generation systems.【17】 Testing Speech Emotion Recognition Machine Learning Models标题：测试语音情感识别机器学习模型链接：https://arxiv.org/abs/2312.06270作者：Anna Derington,Hagen Wierstorf,Ali Özkil,Florian Eyben,Felix Burkhardt,Björn W. Schuller摘要：用于语音情感识别（SER）的机器学习模型可以针对不同的任务进行训练，并且通常基于每个任务的几个可用数据集进行评估。任务可以包括唤醒、效价、支配、情绪类别或语调。这些模型主要根据相关性或召回率进行评估，并且总是在预测中显示出一些错误。错误表现在模型行为中，即使模型实现了相同的回忆或相关性，模型行为在不同的维度上也可能非常不同。本文研究了语音情感识别模型的行为与测试框架，该框架要求模型满足正确性，公平性和鲁棒性方面的条件。摘要：Machine learning models for speech emotion recognition (SER) can be trained for different tasks and are usually evaluated on the basis of a few available datasets per task. Tasks could include arousal, valence, dominance, emotional categories, or tone of voice. Those models are mainly evaluated in terms of correlation or recall, and always show some errors in their predictions. The errors manifest themselves in model behaviour, which can be very different along different dimensions even if the same recall or correlation is achieved by the model. This paper investigates behavior of speech emotion recognition models with a testing framework which requires models to fulfill conditions in terms of correctness, fairness, and robustness.【18】 EEND-DEMUX: End-to-End Neural Speaker Diarization via Demultiplexed Speaker Embeddings标题：EEND-DEMUX：通过多路分解话者嵌入实现端到端神经话者二值化链接：https://arxiv.org/abs/2312.06065作者：Sung Hwan Mun,Min Hyun Han,Canyeong Moon,Nam Soo Kim备注：Submitted to IEEE Signal Processing Letters摘要：近年来，有研究进一步改善端到端神经说话人日志化（EEND）系统。这封信提出了EEND-DEMUX模型，一种新的框架，利用解复用的扬声器嵌入。在这项工作中，我们专注于解开说话人相关的信息在潜在的空间，然后将每个分离的潜在变量转化为相应的语音活动。EEND-DEMUX在推理阶段通过解复用操作直接获得分离的说话人嵌入，而无需外部说话人日志化系统、嵌入提取器或启发式解码技术。此外，我们采用了多头交叉注意机制，有效地捕捉混合和分离的说话人嵌入之间的相关性。我们制定了三个损失函数的基础上匹配，正交性和稀疏性的限制，学习强大的解复用扬声器嵌入。LibriMix数据集上的实验结果显示，在固定和灵活的扬声器数量情况下，性能都得到了持续改善。摘要：In recent years, there have been studies to further improve the end-to-end neural speaker diarization (EEND) systems. This letter proposes the EEND-DEMUX model, a novel framework utilizing demultiplexed speaker embeddings. In this work, we focus on disentangling speaker-relevant information in the latent space and then transform each separated latent variable into its corresponding speech activity. EEND-DEMUX can directly obtain separated speaker embeddings through the demultiplexing operation in the inference phase without an external speaker diarization system, an embedding extractor, or a heuristic decoding technique. Furthermore, we employ a multi-head cross-attention mechanism to capture the correlation between mixture and separated speaker embeddings effectively. We formulate three loss functions based on matching, orthogonality, and sparsity constraints to learn robust demultiplexed speaker embeddings. The experimental results on the LibriMix dataset show consistently improved performance in both a fixed and flexible number of speakers scenarios.【1】 Testing Speech Emotion Recognition Machine Learning Models标题：测试语音情感识别机器学习模型链接：https://arxiv.org/abs/2312.06270作者：Anna Derington,Hagen Wierstorf,Ali Özkil,Florian Eyben,Felix Burkhardt,Björn W. Schuller摘要：用于语音情感识别（SER）的机器学习模型可以针对不同的任务进行训练，并且通常基于每个任务的几个可用数据集进行评估。任务可以包括唤醒、效价、支配、情绪类别或语调。这些模型主要根据相关性或召回率进行评估，并且总是在预测中显示出一些错误。错误表现在模型行为中，即使模型实现了相同的回忆或相关性，模型行为在不同的维度上也可能非常不同。本文研究了语音情感识别模型的行为与测试框架，该框架要求模型满足正确性，公平性和鲁棒性方面的条件。摘要：Machine learning models for speech emotion recognition (SER) can be trained for different tasks and are usually evaluated on the basis of a few available datasets per task. Tasks could include arousal, valence, dominance, emotional categories, or tone of voice. Those models are mainly evaluated in terms of correlation or recall, and always show some errors in their predictions. The errors manifest themselves in model behaviour, which can be very different along different dimensions even if the same recall or correlation is achieved by the model. This paper investigates behavior of speech emotion recognition models with a testing framework which requires models to fulfill conditions in terms of correctness, fairness, and robustness.【2】 EEND-DEMUX: End-to-End Neural Speaker Diarization via Demultiplexed Speaker Embeddings标题：EEND-DEMUX：通过多路分解话者嵌入实现端到端神经话者二值化链接：https://arxiv.org/abs/2312.06065作者：Sung Hwan Mun,Min Hyun Han,Canyeong Moon,Nam Soo Kim备注：Submitted to IEEE Signal Processing Letters摘要：近年来，有研究进一步改善端到端神经说话人日志化（EEND）系统。这封信提出了EEND-DEMUX模型，一种新的框架，利用解复用的扬声器嵌入。在这项工作中，我们专注于解开说话人相关的信息在潜在的空间，然后将每个分离的潜在变量转化为相应的语音活动。EEND-DEMUX在推理阶段通过解复用操作直接获得分离的说话人嵌入，而无需外部说话人日志化系统、嵌入提取器或启发式解码技术。此外，我们采用了多头交叉注意机制，有效地捕捉混合和分离的说话人嵌入之间的相关性。我们制定了三个损失函数的基础上匹配，正交性和稀疏性的限制，学习强大的解复用扬声器嵌入。LibriMix数据集上的实验结果显示，在固定和灵活的扬声器数量情况下，性能都得到了持续改善。摘要：In recent years, there have been studies to further improve the end-to-end neural speaker diarization (EEND) systems. This letter proposes the EEND-DEMUX model, a novel framework utilizing demultiplexed speaker embeddings. In this work, we focus on disentangling speaker-relevant information in the latent space and then transform each separated latent variable into its corresponding speech activity. EEND-DEMUX can directly obtain separated speaker embeddings through the demultiplexing operation in the inference phase without an external speaker diarization system, an embedding extractor, or a heuristic decoding technique. Furthermore, we employ a multi-head cross-attention mechanism to capture the correlation between mixture and separated speaker embeddings effectively. We formulate three loss functions based on matching, orthogonality, and sparsity constraints to learn robust demultiplexed speaker embeddings. The experimental results on the LibriMix dataset show consistently improved performance in both a fixed and flexible number of speakers scenarios.【3】 Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism标题：神经文本发音：实现听觉和照片真实感的深层文本到视听语音合成链接：https://arxiv.org/abs/2312.06613作者：Georgios Milis,Panagiotis P. Filntisis,Anastasios Roussos,Petros Maragos摘要：最近在序列数据深度学习方面的进展已经产生了快速而强大的模型，可以生成逼真的人类对话视频。在说话的脸生成的艺术状态主要集中在嘴唇同步，被限制在音频剪辑。然而，具有从文本传输而不是音频合成说话人的能力对于许多应用特别有益，并且随着最近在大型语言模型方面的突破，预计将受到越来越多的关注。为此，大多数方法实现了文本到语音模块的级联2级架构，然后是音频驱动的说话面部生成器，但这忽略了说话期间发生的音频和视频流之间高度复杂的相互作用。在本文中，我们提出了第一，据我们所知，文本驱动的视听语音合成器，使用Transformers，不遵循级联的方法。我们的方法，我们称之为神经文本到ARticulate谈话（NEUTART），是一个说话的脸生成器，使用联合视听特征空间，以及语音知情的3D面部重建和唇读损失的视觉监督。该模型产生逼真的说话人脸视频与人类一样的清晰度和良好同步的视听流。我们在视听数据集以及野外视频上的实验揭示了在客观指标和人类评估方面最先进的生成质量。摘要：Recent advances in deep learning for sequential data have given rise to fast and powerful models that produce realistic videos of talking humans. The state of the art in talking face generation focuses mainly on lip-syncing, being conditioned on audio clips. However, having the ability to synthesize talking humans from text transcriptions rather than audio is particularly beneficial for many applications and is expected to receive more and more attention, following the recent breakthroughs in large language models. For that, most methods implement a cascaded 2-stage architecture of a text-to-speech module followed by an audio-driven talking face generator, but this ignores the highly complex interplay between audio and visual streams that occurs during speaking. In this paper, we propose the first, to the best of our knowledge, text-driven audiovisual speech synthesizer that uses Transformers and does not follow a cascaded approach. Our method, which we call NEUral Text to ARticulate Talk (NEUTART), is a talking face generator that uses a joint audiovisual feature space, as well as speech-informed 3D facial reconstructions and a lip-reading loss for visual supervision. The proposed model produces photorealistic talking face videos with human-like articulation and well-synced audiovisual streams. Our experiments on audiovisual datasets as well as in-the-wild videos reveal state-of-the-art generation quality both in terms of objective metrics and human evaluation.【4】 Deep Photonic Reservoir Computer for Speech Recognition标题：用于语音识别的深度光子库计算机链接：https://arxiv.org/abs/2312.06558作者：Enrico Picco,Alessandro Lupo,Serge Massar摘要：语音识别是人工智能领域的一项关键任务，由于大型复杂的神经网络，其训练过程通常需要大量的标记数据和计算密集型操作，语音识别已经取得了显着的进步。另一种范例，水库计算，是能源效率，并很好地适应在物理基板上的实现，但表现出的性能相比，更资源密集型的机器学习算法的限制。在这项工作中，我们通过调查互连水库的不同架构来应对这一挑战，所有这些都属于深层水库计算的范畴。我们提出了一种基于光子的深水库计算机，并评估其在不同的语音识别任务的有效性。我们展示了具体的设计选择，旨在简化水库计算机的实际实现，同时实现高维音频信号的高速处理。总的来说，通过目前的工作，我们希望能够帮助低功耗和高性能的神经形态硬件的进步。摘要：Speech recognition is a critical task in the field of artificial intelligence and has witnessed remarkable advancements thanks to large and complex neural networks, whose training process typically requires massive amounts of labeled data and computationally intensive operations. An alternative paradigm, reservoir computing, is energy efficient and is well adapted to implementation in physical substrates, but exhibits limitations in performance when compared to more resource-intensive machine learning algorithms. In this work we address this challenge by investigating different architectures of interconnected reservoirs, all falling under the umbrella of deep reservoir computing. We propose a photonic-based deep reservoir computer and evaluate its effectiveness on different speech recognition tasks. We show specific design choices that aim to simplify the practical implementation of a reservoir computer while simultaneously achieving high-speed processing of high-dimensional audio signals. Overall, with the present work we hope to help the advancement of low-power and high-performance neuromorphic hardware.【5】 Towards Domain-Specific Cross-Corpus Speech Emotion Recognition Approach标题：面向特定领域的跨语料库语音情感识别方法链接：https://arxiv.org/abs/2312.06466作者：Yan Zhao,Yuan Zong,Hailun Lian,Cheng Lu,Jingang Shi,Wenming Zheng摘要：跨语料库语音情感识别（SER）提出了一个挑战，由于特征分布不匹配，潜在地降低了既定的SER方法的性能。在本文中，我们提出了一种新的转移子空间学习方法，称为声学知识引导转移线性回归（AKTLR）来应对这一挑战。与现有的方法，这往往忽略了特定领域的知识相关的SER和简单地处理跨语料库SER作为一个通用的迁移学习任务，我们的AKTLR方法是建立在一个精心设计的声学知识指导的双稀疏约束机制。这种机制强调了最小的声学参数特征集的潜力，以减轻分类器过度适应，这是经验验证的声学知识在SER，使优越的泛化跨语料库SER任务相比，使用大的特征集。通过这种机制，我们扩展了一个简单的转移线性回归模型到AKTLR。该扩展利用其全部能力，以寻求emotiondiscriminative和语料库不变的功能，从建立的声学参数特征集用于描述语音信号在两个尺度：贡献的声学参数组和每个贡献组内的组成元素。我们所提出的方法进行了评估，通过广泛的跨语料库SER实验三个广泛使用的语音情感语料库：ENUDB，ENTERFACE，和CASIA。结果证实了我们方法的有效性和优越性能，优于最近最先进的迁移子空间学习和基于深度迁移学习的跨语料库SER方法。此外，我们的工作提供了实验证据，支持将特定领域的知识迁移学习模型，以解决跨语料库SER任务的可行性和优越性。摘要：Cross-corpus speech emotion recognition (SER) poses a challenge due to feature distribution mismatch, potentially degrading the performance of established SER methods. In this paper, we tackle this challenge by proposing a novel transfer subspace learning method called acoustic knowledgeguided transfer linear regression (AKTLR). Unlike existing approaches, which often overlook domain-specific knowledge related to SER and simply treat cross-corpus SER as a generic transfer learning task, our AKTLR method is built upon a well-designed acoustic knowledge-guided dual sparsity constraint mechanism. This mechanism emphasizes the potential of minimalistic acoustic parameter feature sets to alleviate classifier overadaptation, which is empirically validated acoustic knowledge in SER, enabling superior generalization in cross-corpus SER tasks compared to using large feature sets. Through this mechanism, we extend a simple transfer linear regression model to AKTLR. This extension harnesses its full capability to seek emotiondiscriminative and corpus-invariant features from established acoustic parameter feature sets used for describing speech signals across two scales: contributive acoustic parameter groups and constituent elements within each contributive group. Our proposed method is evaluated through extensive cross-corpus SER experiments on three widely-used speech emotion corpora: EmoDB, eNTERFACE, and CASIA. The results confirm the effectiveness and superior performance of our method, outperforming recent state-of-the-art transfer subspace learning and deep transfer learning-based cross-corpus SER methods. Furthermore, our work provides experimental evidence supporting the feasibility and superiority of incorporating domain-specific knowledge into the transfer learning model to address cross-corpus SER tasks.【6】 Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation标题：合作确实很重要：探索视听分割的多层次双边关系链接：https://arxiv.org/abs/2312.06462作者：Qi Yang,Xing Nie,Tong Li,Pengfei Gao,Ying Guo,Cheng Zhen,Pengfei Yan,Shiming Xiang备注：13 pages, 10 figures摘要：最近，已经引入了视听分割（AVS）任务，旨在将给定视频内的具有发声对象的像素分组。这项任务需要对场景进行首次音频驱动的像素级理解，这带来了重大挑战。在本文中，我们提出了一个创新的视听Transformer框架，称为COMBO，多阶双边关系的合作的缩写。第一次，我们的框架探讨了AVS中的三种双边纠缠：像素纠缠，模态纠缠和时间纠缠。关于像素纠缠，我们采用了Siam编码器模块（SEM），该模块利用先验知识从基础模型中生成更精确的视觉特征。对于模态纠缠，我们设计了一个双边融合模块（BFM），使COMBO能够双向对齐相应的视觉和听觉信号。在时间纠缠方面，根据时间纠缠的内在规律，提出了一种自适应的帧间一致性丢失算法。对AVSBench-object（S4上84.7 mIoU，MS 3上59.2 mIou）和AVSBench-semantic（AVSS上42.1 mIoU）数据集的综合实验和消融研究表明，COMBO超过了以前最先进的方法。代码和更多结果将在https://combo-avs.github.io/上公开。摘要：Recently, an audio-visual segmentation (AVS) task has been introduced, aiming to group pixels with sounding objects within a given video. This task necessitates a first-ever audio-driven pixel-level understanding of the scene, posing significant challenges. In this paper, we propose an innovative audio-visual transformer framework, termed COMBO, an acronym for COoperation of Multi-order Bilateral relatiOns. For the first time, our framework explores three types of bilateral entanglements within AVS: pixel entanglement, modality entanglement, and temporal entanglement. Regarding pixel entanglement, we employ a Siam-Encoder Module (SEM) that leverages prior knowledge to generate more precise visual features from the foundational model. For modality entanglement, we design a Bilateral-Fusion Module (BFM), enabling COMBO to align corresponding visual and auditory signals bi-directionally. As for temporal entanglement, we introduce an innovative adaptive inter-frame consistency loss according to the inherent rules of temporal. Comprehensive experiments and ablation studies on AVSBench-object (84.7 mIoU on S4, 59.2 mIou on MS3) and AVSBench-semantic (42.1 mIoU on AVSS) datasets demonstrate that COMBO surpasses previous state-of-the-art methods. Code and more results will be publicly available at https://combo-avs.github.io/.【7】 Deep Imbalanced Learning for Multimodal Emotion Recognition in Conversations标题：深度不平衡学习在对话中多通道情绪识别中的应用链接：https://arxiv.org/abs/2312.06337作者：Tao Meng,Yuntao Shou,Wei Ai,Nan Yin,Keqin Li备注：16 pages, 9 figures摘要：会话中的多模态情感识别（MERC）的主要任务是识别模态中的情感，例如，文本、音频、图像和视频，这是实现机器智能的一个重要发展方向。然而，MERC中的许多数据自然地表现出情感类别的不平衡分布，研究人员忽略了不平衡数据对情感识别的负面影响。针对这一问题，本文从数据扩充、损失敏感性和采样策略三个方面进行了系统分析，提出了类边界增强表示学习（CBERL）模型。具体来说，我们首先设计了一个多模态生成对抗网络，以解决原始数据中{情感}类别分布不平衡的问题。其次，提出了一种深度联合变分自编码器，以融合跨模态的互补语义信息，并获得有区别的特征表示。最后，我们实现了一个多任务的图神经网络与掩模重建和分类优化，以解决类边界学习中的过拟合和欠拟合问题，并实现跨模态情感识别。我们在IEMOCAP和MELD基准数据集上进行了大量的实验，结果表明CBERL在情感识别的有效性上取得了一定的性能提升。特别是在少数类别的恐惧和厌恶情绪标签上，我们的模型将准确率和F1值提高了10%到20%。摘要：The main task of Multimodal Emotion Recognition in Conversations (MERC) is to identify the emotions in modalities, e.g., text, audio, image and video, which is a significant development direction for realizing machine intelligence. However, many data in MERC naturally exhibit an imbalanced distribution of emotion categories, and researchers ignore the negative impact of imbalanced data on emotion recognition. To tackle this problem, we systematically analyze it from three aspects: data augmentation, loss sensitivity, and sampling strategy, and propose the Class Boundary Enhanced Representation Learning (CBERL) model. Concretely, we first design a multimodal generative adversarial network to address the imbalanced distribution of {emotion} categories in raw data. Secondly, a deep joint variational autoencoder is proposed to fuse complementary semantic information across modalities and obtain discriminative feature representations. Finally, we implement a multi-task graph neural network with mask reconstruction and classification optimization to solve the problem of overfitting and underfitting in class boundary learning, and achieve cross-modal emotion recognition. We have conducted extensive experiments on the IEMOCAP and MELD benchmark datasets, and the results show that CBERL has achieved a certain performance improvement in the effectiveness of emotion recognition. Especially on the minority class fear and disgust emotion labels, our model improves the accuracy and F1 value by 10% to 20%.【8】 Creating Spoken Dialog Systems in Ultra-Low Resourced Settings标题：在超低资源设置中创建口语对话系统链接：https://arxiv.org/abs/2312.06266作者：Moayad Elamin,Muhammad Omer,Yonas Chanie,Henslaac Ndlovu备注：12 pages, 15 figures摘要：自动语音识别（ASR）系统是一项关键技术，目前用于设计各种应用程序，最值得注意的是智能助手，如Alexa。ASR系统本质上是使用口语理解（SLU）从语音中提取有意义的信息的对话系统。设计这样的系统的主要挑战是，它们需要大量标记的干净数据来竞争性地执行，这样的数据非常难以收集和注释到相应的SLU任务，此外，当为低资源语言设计这样的系统时，其中数据非常有限，问题的严重性加剧。在本文中，我们专注于一个相当流行的SLU任务，即意图分类，同时使用低资源语言，即佛兰芒语。意图分类是一项与理解用户与系统交互的意图有关的任务。我们建立在现有的光模型在佛兰芒语的意图分类，我们的主要贡献是在两个层面上应用不同的增强技术-语音水平，和语音成绩单水平-现有的模型，以应对稀缺的问题标记的数据在低资源的语言。我们发现，我们的数据增强技术，在这两个层面上，提高了模型的性能在一些任务。摘要：Automatic Speech Recognition (ASR) systems are a crucial technology that is used today to design a wide variety of applications, most notably, smart assistants, such as Alexa. ASR systems are essentially dialogue systems that employ Spoken Language Understanding (SLU) to extract meaningful information from speech. The main challenge with designing such systems is that they require a huge amount of labeled clean data to perform competitively, such data is extremely hard to collect and annotate to respective SLU tasks, furthermore, when designing such systems for low resource languages, where data is extremely limited, the severity of the problem intensifies. In this paper, we focus on a fairly popular SLU task, that is, Intent Classification while working with a low resource language, namely, Flemish. Intent Classification is a task concerned with understanding the intents of the user interacting with the system. We build on existing light models for intent classification in Flemish, and our main contribution is applying different augmentation techniques on two levels -- the voice level, and the phonetic transcripts level -- to the existing models to counter the problem of scarce labeled data in low-resource languages. We find that our data augmentation techniques, on both levels, have improved the model performance on a number of tasks.【9】 Transformer Attractors for Robust and Efficient End-to-End Neural Diarization标题：用于稳健有效的端到端神经网络二值化的Transformer吸引子链接：https://arxiv.org/abs/2312.06253作者：Lahiru Samarakoon,Samuel J. Broughton,Marc Harkönen,Ivan Fung备注：8 pages, 1 figure, ASRU2023摘要：基于吸引子的端到端神经日志化（End-to-end neural diarization with encoder-decoder based attractor，EEND-EDA）是一种在单个神经网络中执行日志化的方法。EDA通过使用基于LSTM的编码器-解码器来处理灵活数量的扬声器的日记化，该编码器-解码器以自回归方式生成一组扬声器吸引子。在本文中，我们建议用基于变换的吸引子计算（TA）模块来取代EDA。TA由Combiner模块和Transformer解码器组成。组合器块的主要功能是通过将学习到的会话信息合并到嵌入的全局集合中来生成会话相关（CD）嵌入。然后，这些CD嵌入将用作Transformer解码器的输入。在公开数据集上的结果表明，EEND-TA比EEND-EDA实现了2.68%的绝对DER改进。EEND-TA的推理速度是EEND-EDA的1.28倍。摘要：End-to-end neural diarization with encoder-decoder based attractors (EEND-EDA) is a method to perform diarization in a single neural network. EDA handles the diarization of a flexible number of speakers by using an LSTM-based encoder-decoder that generates a set of speaker-wise attractors in an autoregressive manner. In this paper, we propose to replace EDA with a transformer-based attractor calculation (TA) module. TA is composed of a Combiner block and a Transformer decoder. The main function of the combiner block is to generate conversational dependent (CD) embeddings by incorporating learned conversational information into a global set of embeddings. These CD embeddings will then serve as the input for the transformer decoder. Results on public datasets show that EEND-TA achieves 2.68% absolute DER improvement over EEND-EDA. EEND-TA inference is 1.28 times faster than that of EEND-EDA.【10】 Music-PAW: Learning Music Representations via Hierarchical Part-whole Interaction and Contrast标题：音乐-PAW：通过层次化的部分-整体互动和对比学习音乐表征链接：https://arxiv.org/abs/2312.06197作者：Dong Yao,Shengyu Zhang,Zhou Zhao,Jieming Zhu,Liqun Deng,Wenqiao Zhang,Zhenhua Dong,Ruiming Tang,Xin Jiang摘要：最近的自监督学习方法在各种下游任务上的出色表现引起了学术界和工业界的极大关注。最近的一些研究致力于自我监督的音乐表征学习。然而，他们中的大多数人学会了在波形或频谱图中表示相等大小的音乐片段。尽管在某些任务中是有效的，但以这种方式学习音乐表征在很大程度上忽视了音乐固有的部分-整体层次结构。由于听觉皮层的分层性质[24]，理解音乐的自下而上的结构，即，不同的部分如何在不同的层次上构成整体，是音乐理解和表征学习的关键。这项工作追求分层音乐表示学习，并引入了Music-PAW框架，该框架使裁剪的音乐片段与部分-整体层次结构的特征交互成为可能。从技术的角度来看，我们提出了一个基于transformer的部分-整体交互模块，逐步原因部分-整体音乐片段之间的结构关系在相邻的水平。此外，为了建立一个多层次的表示空间，我们设计了一个分层的对比学习目标，以对齐相邻层次中的部分-整体音乐表示。从部分-整体层次结构中学习音频表示的优点已经在各种下游任务中得到验证，包括音乐分类（单标签和多标签），翻唱歌曲识别和声学场景分类。摘要：The excellent performance of recent self-supervised learning methods on various downstream tasks has attracted great attention from academia and industry. Some recent research efforts have been devoted to self-supervised music representation learning. Nevertheless, most of them learn to represent equally-sized music clips in the waveform or a spectrogram. Despite being effective in some tasks, learning music representations in such a manner largely neglect the inherent part-whole hierarchies of music. Due to the hierarchical nature of the auditory cortex [24], understanding the bottom-up structure of music, i.e., how different parts constitute the whole at different levels, is essential for music understanding and representation learning. This work pursues hierarchical music representation learning and introduces the Music-PAW framework, which enables feature interactions of cropped music clips with part-whole hierarchies. From a technical perspective, we propose a transformer-based part-whole interaction module to progressively reason the structural relationships between part-whole music clips at adjacent levels. Besides, to create a multi-hierarchy representation space, we devise a hierarchical contrastive learning objective to align part-whole music representations in adjacent hierarchies. The merits of audio representation learning from part-whole hierarchies have been validated on various downstream tasks, including music classification (single-label and multi-label), cover song identification and acoustic scene classification.【11】 Speaker-Text Retrieval via Contrastive Learning标题：基于对比学习的说话人文本检索链接：https://arxiv.org/abs/2312.06055作者：Xuechen Liu,Xin Wang,Erica Cooper,Xiaoxiao Miao,Junichi Yamagishi备注：Submitted to IEEE Signal Processing Letters摘要：在这项研究中，我们介绍了一种新的跨模态检索任务，涉及说话人的描述和相应的音频样本。利用预先训练的扬声器和文本编码器，我们提出了一个简单的基于对比学习的学习框架。此外，我们还探讨了将说话人标签纳入训练过程的影响。我们的研究结果建立了有效的连接扬声器和文本信息的任务，英语和日语，在不同的数据配置。额外的视觉分析揭示了说话人聚类和检索性能之间潜在的细微差别。摘要：In this study, we introduce a novel cross-modal retrieval task involving speaker descriptions and their corresponding audio samples. Utilizing pre-trained speaker and text encoders, we present a simple learning framework based on contrastive learning. Additionally, we explore the impact of incorporating speaker labels into the training process. Our findings establish the effectiveness of linking speaker and text information for the task for both English and Japanese languages, across diverse data configurations. Additional visual analysis unveils potential nuanced associations between speaker clustering and retrieval performance.【12】 A Practical Survey on Emerging Threats from AI-driven Voice Attacks: How Vulnerable are Commercial Voice Control Systems?标题：人工智能驱动的语音攻击新威胁的实践调查：商业语音控制系统有多脆弱？链接：https://arxiv.org/abs/2312.06010作者：Yuanda Wang,Qiben Yan,Nikolay Ivanov,Xun Chen备注：14 pages摘要：人工智能（AI）驱动的音频攻击的出现揭示了语音控制系统中新的安全漏洞。虽然研究人员已经引入了多种针对语音控制系统（VoIP）的攻击策略，但VoIP的不断进步已经减少了许多此类攻击的影响。认识到这一动态格局，我们的研究努力全面评估商业语音控制系统对恶意音频攻击的弹性。通过广泛的实验，我们评估了六种主要的攻击技术，包括语音控制接口和设备。与流行的说法相反，我们的研究结果表明，商业语音控制系统对现有威胁表现出更强的抵抗力。特别是，我们的研究强调了白盒攻击在黑盒场景中的无效性。此外，对手在与商业系统（如Apple Siri和Samsung Bixby）进行基于查询的交互时，在获得精确的梯度估计方面遇到了巨大的障碍。同时，我们发现，目前的防御策略并不能完全免疫于高级攻击。我们的研究结果有助于提高防御机制的重要见解。通过这项调查，我们的目标是提高学术界对网络安全问题的认识，并倡导在这一关键领域继续研究。摘要：The emergence of Artificial Intelligence (AI)-driven audio attacks has revealed new security vulnerabilities in voice control systems. While researchers have introduced a multitude of attack strategies targeting voice control systems (VCS), the continual advancements of VCS have diminished the impact of many such attacks. Recognizing this dynamic landscape, our study endeavors to comprehensively assess the resilience of commercial voice control systems against a spectrum of malicious audio attacks. Through extensive experimentation, we evaluate six prominent attack techniques across a collection of voice control interfaces and devices. Contrary to prevailing narratives, our results suggest that commercial voice control systems exhibit enhanced resistance to existing threats. Particularly, our research highlights the ineffectiveness of white-box attacks in black-box scenarios. Furthermore, the adversaries encounter substantial obstacles in obtaining precise gradient estimations during query-based interactions with commercial systems, such as Apple Siri and Samsung Bixby. Meanwhile, we find that current defense strategies are not completely immune to advanced attacks. Our findings contribute valuable insights for enhancing defense mechanisms in VCS. Through this survey, we aim to raise awareness within the academic community about the security concerns of VCS and advocate for continued research in this crucial area.【13】 mir_ref: A Representation Evaluation Framework for Music Information Retrieval Tasks标题：mir_ref：一个音乐信息检索任务的表示评估框架链接：https://arxiv.org/abs/2312.05994作者：Christos Plachouras,Pablo Alonson-Jiménez,Dmitry Bogdanov备注：Machine Learning for Audio Workshop, Neural Information Processing Systems (NeurIPS) 2023, New Orleans, LA摘要：音乐信息检索（MIR）研究越来越多地利用表示学习来获得更紧凑，更强大的音乐音频表示，用于各种下游MIR任务。然而，由于音频和标签预处理、下游模型和度量实现、数据可用性和计算资源的差异，当前的表示评估方法是分散的，通常导致不一致和有限的结果。在这项工作中，我们介绍了mir_ref，MIR表示评估框架专注于无缝，透明，本地优先的实验编排，以支持表示开发。它实现了各种组件，如MIR数据集，任务，嵌入模型以及用于结果分析和可视化的工具，同时促进了自定义组件的实现。为了证明它的实用性，我们使用它对各种任务和数据集的几个嵌入模型进行了广泛的评估，包括评估它们对各种音频扰动的鲁棒性以及从中提取相关信息的容易性。摘要：Music Information Retrieval (MIR) research is increasingly leveraging representation learning to obtain more compact, powerful music audio representations for various downstream MIR tasks. However, current representation evaluation methods are fragmented due to discrepancies in audio and label preprocessing, downstream model and metric implementations, data availability, and computational resources, often leading to inconsistent and limited results. In this work, we introduce mir_ref, an MIR Representation Evaluation Framework focused on seamless, transparent, local-first experiment orchestration to support representation development. It features implementations of a variety of components such as MIR datasets, tasks, embedding models, and tools for result analysis and visualization, while facilitating the implementation of custom components. To demonstrate its utility, we use it to conduct an extensive evaluation of several embedding models across various tasks and datasets, including evaluating their robustness to various audio perturbations and the ease of extracting relevant information from them.【14】 Voice Activity Detection (VAD) in Noisy Environments标题：噪声环境中的语音活动检测(VAD)链接：https://arxiv.org/abs/2312.05815作者：Joshua Ball备注：7 pages摘要：在数字音频处理领域，语音活动检测（VAD）在区分语音和非语音元素方面起着关键作用，这是一项在嘈杂环境中变得越来越复杂的任务。本文详细介绍了VAD系统的开发和实施，专门设计，以保持在各种环境噪声的存在下的高精度。我们引入了一种新的算法，增强了一个专门设计的过滤技术，有效地隔离语音，即使在不同的背景声音。我们全面的测试和验证证明了该系统的鲁棒性，突出了它的能力，以显着的精度从噪声中辨别语音。探索深入到：（1）VAD的核心原理及其在现代音频处理中的关键作用;（2）我们用于过滤环境噪声的方法;（3）证明我们的系统在嘈杂环境中具有卓越性能的证据。摘要：In the realm of digital audio processing, Voice Activity Detection (VAD) plays a pivotal role in distinguishing speech from non-speech elements, a task that becomes increasingly complex in noisy environments. This paper details the development and implementation of a VAD system, specifically engineered to maintain high accuracy in the presence of various ambient noises. We introduce a novel algorithm enhanced with a specially designed filtering technique, effectively isolating speech even amidst diverse background sounds. Our comprehensive testing and validation demonstrate the system's robustness, highlighting its capability to discern speech from noise with remarkable precision. The exploration delves into: (1) the core principles underpinning VAD and its crucial role in modern audio processing; (2) the methodologies we employed to filter ambient noise; and (3) a presentation of evidence affirming our system's superior performance in noisy conditions.【15】 Neural Speech Embeddings for Speech Synthesis Based on Deep Generative Networks标题：基于深度生成网络的神经语音嵌入语音合成链接：https://arxiv.org/abs/2312.05814作者：Seo-Hyun Lee,Young-Eun Lee,Soowon Kim,Byung-Kwan Ko,Jun-Young Kim,Seong-Whan Lee备注：4 pages摘要：脑语转换技术代表了人工智能、脑机接口和语音合成等跨学科应用的融合。基于意图解码和语音合成的神经表征学习将神经活动与人类语言交际手段直接联系起来，可以大大提高交际的自然性。随着表征学习理论的不断发展和语音合成技术的不断进步，将脑电信号直接转换为语音已经显示出了巨大的潜力。特别是，当使用深度生成模型从大脑信号生成语音时，给予神经网络的处理后的输入特征和神经语音嵌入在整体性能中起着重要作用。在本文中，我们介绍了目前的脑到语音技术与语音合成的可能性，从大脑信号，这可能最终促进非语言交流的创新。此外，我们进行了全面的分析，神经功能和神经语音嵌入的神经生理激活，而执行的语音，这可能会发挥重要作用的语音合成工作。摘要：Brain-to-speech technology represents a fusion of interdisciplinary applications encompassing fields of artificial intelligence, brain-computer interfaces, and speech synthesis. Neural representation learning based intention decoding and speech synthesis directly connects the neural activity to the means of human linguistic communication, which may greatly enhance the naturalness of communication. With the current discoveries on representation learning and the development of the speech synthesis technologies, direct translation of brain signals into speech has shown great promise. Especially, the processed input features and neural speech embeddings which are given to the neural network play a significant role in the overall performance when using deep generative models for speech generation from brain signals. In this paper, we introduce the current brain-to-speech technology with the possibility of speech synthesis from brain signals, which may ultimately facilitate innovation in non-verbal communication. Also, we perform comprehensive analysis on the neural features and neural speech embeddings underlying the neurophysiological activation while performing speech, which may play a significant role in the speech synthesis works.【16】 Keyword spotting -- Detecting commands in speech using deep learning标题：关键词识别--利用深度学习检测语音中的命令链接：https://arxiv.org/abs/2312.05640作者：Sumedha Rai,Tong Li,Bella Lyu摘要：语音识别已成为机器学习和人工智能发展的重要课题。在这项研究中，我们探索了使用语音识别机器学习和深度学习技术进行关键字识别的重要任务。我们通过将原始波形转换为Mel频率倒谱系数（MFCC）来实现特征工程，我们将其用作模型的输入。我们实验了几种不同的算法，如高斯混合的隐马尔可夫模型，卷积神经网络和递归神经网络的变体，包括长短期记忆和注意力机制。在我们的实验中，带有BiLSTM和Attention的RNN实现了最佳性能，准确率为93.9%摘要：Speech recognition has become an important task in the development of machine learning and artificial intelligence. In this study, we explore the important task of keyword spotting using speech recognition machine learning and deep learning techniques. We implement feature engineering by converting raw waveforms to Mel Frequency Cepstral Coefficients (MFCCs), which we use as inputs to our models. We experiment with several different algorithms such as Hidden Markov Model with Gaussian Mixture, Convolutional Neural Networks and variants of Recurrent Neural Networks including Long Short-Term Memory and the Attention mechanism. In our experiments, RNN with BiLSTM and Attention achieves the best performance with an accuracy of 93.9 %【17】 An Experimental Study: Assessing the Combined Framework of WavLM and BEST-RQ for Text-to-Speech Synthesis标题：WavLM与BEST-RQ相结合的文语合成框架的实验研究链接：https://arxiv.org/abs/2312.05415作者：Via Nielson,Steven Hillis备注：8 pages, 1 figure, 4 tables摘要：我们提出了一种新的模型架构，特别适合于文本到语音（TTS）模型。我们结合WavLM，一个预先训练的自监督学习（SSL）语音模型，和BEST-RQ矢量量化框架。我们评估在何种程度上更任务不可知的WavLM，再加上优越的适用性的简单化BEST-RQ框架的下游任务的更广泛的阵列，产生有利的结果。在LibriSpeech数据集上进行的SUPERB基准测试表明，所提出的模型表现明显不佳。我们推测这种性能的根本原因与量化原始音频波形和频谱图之间的差异有关。我们讨论了这种方法的局限性，以更好地指导未来的发展TTS。摘要：We propose a new model architecture specifically suited for text-to-speech (TTS) models. We combine WavLM, a pre-trained self-supervised learning (SSL) speech model, and the BEST-RQ vector quantization framework. We assess the extent to which the more task-agnostic WavLM, coupled with the superior suitability of the simplistic BEST-RQ framework for a wider array of downstream tasks, yields favorable outcomes. Experiments on the LibriSpeech dataset with SUPERB benchmarking assert that the proposed model significantly underperforms. We speculate the underlying reason for this performance is related to the difference between featurizing raw audio waveforms and spectrograms with a quantizer. We discuss the limitations of this approach to better guide future advancements in TTS.【18】 CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling标题：CMMD：用于视音频条件建模的对比性多模扩散链接：https://arxiv.org/abs/2312.05412作者：Ruihan Yang,Hannes Gamper,Sebastian Braun摘要：我们介绍了一个多模态扩散模型定制的双向条件生成的视频和音频。认识到多模态生成任务中视频和音频事件之间准确对齐的重要性，我们提出了一种联合对比训练损失，以增强视觉和听觉事件之间的同步。我们的研究方法包括在多个数据集上进行综合实验，以彻底评估我们提出的模型的有效性。生成质量和对齐性能的评估是从各个角度进行的，包括客观和主观的度量。我们的研究结果表明，所提出的模型优于基线，证实其有效性和效率。值得注意的是，对比度损失的引入导致视听对齐的改善，特别是在高相关性视频到音频生成任务中。这些结果表明，我们提出的模型作为一个强大的解决方案，提高多模态生成的质量和对齐的潜力，从而有助于视频和音频条件生成系统的进步。摘要：We introduce a multi-modal diffusion model tailored for the bi-directional conditional generation of video and audio. Recognizing the importance of accurate alignment between video and audio events in multi-modal generation tasks, we propose a joint contrastive training loss to enhance the synchronization between visual and auditory occurrences. Our research methodology involves conducting comprehensive experiments on multiple datasets to thoroughly evaluate the efficacy of our proposed model. The assessment of generation quality and alignment performance is carried out from various angles, encompassing both objective and subjective metrics. Our findings demonstrate that the proposed model outperforms the baseline, substantiating its effectiveness and efficiency. Notably, the incorporation of the contrastive loss results in improvements in audio-visual alignment, particularly in the high-correlation video-to-audio generation task. These results indicate the potential of our proposed model as a robust solution for improving the quality and alignment of multi-modal generation, thereby contributing to the advancement of video and audio conditional generation systems.【19】 Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features标题：基于隐私兼容特征的野外多模式群体情感识别链接：https://arxiv.org/abs/2312.05265作者：Anderson Augusma,Dominique Vaufreydaz,Frédérique Letué备注：None摘要：本文探讨了隐私兼容的组级情感识别“在野外”在2023年的挑战。群体级情感识别在许多领域都很有用，包括社交机器人、会话代理、电子教练和学习分析。这项研究只使用全局特征，而不使用单个特征，即所有可用于识别或跟踪视频中的人的特征（面部标志、身体姿势、音频日志等）。提出的多模态模型是由一个视频和一个音频分支之间的交叉注意模态。视频分支基于微调的ViT架构。音频分支提取Mel-频谱图，并通过CNN块将它们馈送到Transformer编码器中。我们的训练范例包括一个生成的合成数据集，以数据驱动的方式增加我们的模型对图像中面部表情的敏感性。大量的实验表明了我们的方法的意义。我们的隐私合规提案在CITTIW挑战中表现相当好，最佳模型的验证和测试集的准确率分别为79.24%和75.13%。值得注意的是，我们的研究结果强调，仅使用均匀分布在视频上的5帧，就可以通过符合隐私的功能达到这种准确性水平。摘要：This paper explores privacy-compliant group-level emotion recognition ''in-the-wild'' within the EmotiW Challenge 2023. Group-level emotion recognition can be useful in many fields including social robotics, conversational agents, e-coaching and learning analytics. This research imposes itself using only global features avoiding individual ones, i.e. all features that can be used to identify or track people in videos (facial landmarks, body poses, audio diarization, etc.). The proposed multimodal model is composed of a video and an audio branches with a cross-attention between modalities. The video branch is based on a fine-tuned ViT architecture. The audio branch extracts Mel-spectrograms and feed them through CNN blocks into a transformer encoder. Our training paradigm includes a generated synthetic dataset to increase the sensitivity of our model on facial expression within the image in a data-driven way. The extensive experiments show the significance of our methodology. Our privacy-compliant proposal performs fairly on the EmotiW challenge, with 79.24% and 75.13% of accuracy respectively on validation and test set for the best models. Noticeably, our findings highlight that it is possible to reach this accuracy level with privacy-compliant features using only 5 frames uniformly distributed on the video. 
每日语音速递
