今日论文合集：cs.SD语音12篇，eess.AS音频处理12篇。

cs.SD语音

【1】Masked Audio Generation using a Single Non-Autoregressive Transformer
标题：用单台非自回归Transformer产生掩蔽音频链接：https://arxiv.org/abs/2401.04577
作者：Alon Ziv,Itai Gat,Gael Le Lan,Tal Remez,Felix Kreuk,Alexandre Défossez,Jade Copet,Gabriel Synnaeve,Yossi Adi摘要：我们介绍MAGNeT，一种屏蔽生成序列建模方法，直接在几个音频令牌流上操作。与以前的工作不同，MAGNeT是由一个单级，非自回归Transformer。在训练过程中，我们预测从掩码调度器获得的掩码令牌的跨度，而在推理过程中，我们使用几个解码步骤逐步构建输出序列。为了进一步提高生成的音频的质量，我们引入了一种新的重新评分方法，在该方法中，我们利用外部预训练模型对来自MAGNeT的预测进行重新评分和排名，然后将其用于后续的解码步骤。最后，我们探索了MAGNeT的混合版本，在该版本中，我们在自回归和非自回归模型之间进行融合，以自回归方式生成前几秒，而序列的其余部分则被并行解码。我们证明了MAGNeT的文本到音乐和文本到音频生成任务的效率，并进行了广泛的实证评估，同时考虑到客观指标和人类研究。所提出的方法与所评估的基线相当，同时明显更快（比自回归基线快7倍）。通过消融研究和分析，我们揭示了组成MAGNeT的每个组件的重要性，同时指出了自回归和非自回归建模之间的权衡，考虑到延迟，吞吐量和生成质量。样品可在我们的演示页面https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT上获得。摘要：We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAGNeT is comprised of a single-stage, non-autoregressive transformer. During training, we predict spans of masked tokens obtained from a masking scheduler, while during inference we gradually construct the output sequence using several decoding steps. To further enhance the quality of the generated audio, we introduce a novel rescoring method in which, we leverage an external pre-trained model to rescore and rank predictions from MAGNeT, which will be then used for later decoding steps. Lastly, we explore a hybrid version of MAGNeT, in which we fuse between autoregressive and non-autoregressive models to generate the first few seconds in an autoregressive manner while the rest of the sequence is being decoded in parallel. We demonstrate the efficiency of MAGNeT for the task of text-to-music and text-to-audio generation and conduct an extensive empirical evaluation, considering both objective metrics and human studies. The proposed approach is comparable to the evaluated baselines, while being significantly faster (x7 faster than the autoregressive baseline). Through ablation studies and analysis, we shed light on the importance of each of the components comprising MAGNeT, together with pointing to the trade-offs between autoregressive and non-autoregressive modeling, considering latency, throughput, and generation quality. Samples are available on our demo page https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
【2】 HyperGANStrument: Instrument Sound Synthesis and Editing with  Pitch-Invariant Hypernetworks标题：HyperGANStrument：基于基音不变超网络的乐器声音合成与编辑链接：https://arxiv.org/abs/2401.04558
作者：Zhe Zhang,Taketo Akama备注：5 pages, 3 figures, Accepted for 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Audio examples: this https URL摘要：GANStrument利用具有音高不变特征提取器和实例调节技术的GAN，在合成逼真的乐器声音方面表现出了卓越的能力。为了进一步提高重建能力和音高精度，以增强用户提供的声音的可编辑性，我们提出了HyperGANStrument，它引入了一个音高不变的超网络来调制预先训练的GANStrument生成器的权重，给定一个单次声音作为输入。超网络调制在输入声音的重建中为发生器提供反馈。此外，我们利用对抗微调计划的超网络，以提高重建保真度和发电机的发电多样性。实验结果表明，该模型不仅增强了GANStrument的生成能力，而且显著提高了合成声音的可编辑性。在线演示页面提供音频示例。摘要：GANStrument, exploiting GANs with a pitch-invariant feature extractor and instance conditioning technique, has shown remarkable capabilities in synthesizing realistic instrument sounds. To further improve the reconstruction ability and pitch accuracy to enhance the editability of user-provided sound, we propose HyperGANStrument, which introduces a pitch-invariant hypernetwork to modulate the weights of a pre-trained GANStrument generator, given a one-shot sound as input. The hypernetwork modulation provides feedback for the generator in the reconstruction of the input sound. In addition, we take advantage of an adversarial fine-tuning scheme for the hypernetwork to improve the reconstruction fidelity and generation diversity of the generator. Experimental results show that the proposed model not only enhances the generation capability of GANStrument but also significantly improves the editability of synthesized sounds. Audio examples are available at the online demo page.
【3】 SonicVisionLM: Playing Sound with Vision Language Models标题：SonicVisionLM：用视觉语言模型播放声音链接：https://arxiv.org/abs/2401.04394
作者：Zhifeng Xie,Shengye Yu,Mengtian Li,Qile He,Chaofeng Chen,Yu-Gang Jiang摘要：人们对为无声视频生成声音的任务越来越感兴趣，主要是因为它在简化视频后期制作方面的实用性。然而，用于视频-声音生成的现有方法试图从视觉表示直接创建声音，由于难以将视觉表示与音频表示对准，这可能是具有挑战性的。在本文中，我们提出了SonicVisionLM，一个新的框架，旨在通过利用视觉语言模型产生广泛的声音效果。我们使用强大的视觉语言模型（VLM）的功能，而不是直接从视频生成音频。当提供无声视频时，我们的方法首先使用VLM来识别视频内的事件，以建议与视频内容匹配的可能声音。这种方法的转变将图像和音频对齐的挑战性任务转化为通过流行的扩散模型对齐图像到文本和文本到音频的更深入研究的子问题。为了提高LLM音频推荐的质量，我们收集了一个广泛的数据集，将文本描述映射到特定的声音效果，并开发了时间控制的音频适配器。我们的方法超越了当前最先进的视频转换为音频的方法，从而增强了与视觉效果的同步，并改善了音频和视频组件之间的对齐。项目页面：https://yusiissy.github.io/SonicVisionLM.github.io/摘要：There has been a growing interest in the task of generating sound for silent videos, primarily because of its practicality in streamlining video post-production. However, existing methods for video-sound generation attempt to directly create sound from visual representations, which can be challenging due to the difficulty of aligning visual representations with audio representations. In this paper, we present SonicVisionLM, a novel framework aimed at generating a wide range of sound effects by leveraging vision language models. Instead of generating audio directly from video, we use the capabilities of powerful vision language models (VLMs). When provided with a silent video, our approach first identifies events within the video using a VLM to suggest possible sounds that match the video content. This shift in approach transforms the challenging task of aligning image and audio into more well-studied sub-problems of aligning image-to-text and text-to-audio through the popular diffusion models. To improve the quality of audio recommendations with LLMs, we have collected an extensive dataset that maps text descriptions to specific sound effects and developed temporally controlled audio adapters. Our approach surpasses current state-of-the-art methods for converting video to audio, resulting in enhanced synchronization with the visuals and improved alignment between audio and video components. Project page: https://yusiissy.github.io/SonicVisionLM.github.io/
【4】 RaD-Net: A Repairing and Denoising Network for Speech Signal Improvement标题：RaD-Net：一种用于语音信号改善的修复和去噪网络链接：https://arxiv.org/abs/2401.04389
作者：Mingshuai Liu,Zhuangqi Chen,Xiaopeng Yan,Yuanjun Lv,Xianjun Xia,Chuanzeng Huang,Yijian Xiao,Lei Xie备注：submitted to ICASSP 2024摘要：本文介绍了我们为ICASSP 2024语音信号改善（SSI）挑战赛开发的修复和去噪网络（RaD-Net）。我们扩展了我们以前的框架的基础上，两阶段的网络，并提出了一个升级的模型。具体来说，我们用TEA-PSE中的COM-Net替换了修复网络。此外，在训练阶段采用了多分辨率鉴别器和多波段鉴别器。最后，我们使用三步训练策略来优化我们的模型。我们提交了两个模型，具有不同的参数集，以满足两个轨道的RTF要求。根据官方结果，拟议的系统在轨道1中排名第二，在轨道2中排名第三。摘要：This paper introduces our repairing and denoising network (RaD-Net) for the ICASSP 2024 Speech Signal Improvement (SSI) Challenge. We extend our previous framework based on a two-stage network and propose an upgraded model. Specifically, we replace the repairing network with COM-Net from TEA-PSE. In addition, multi-resolution discriminators and multi-band discriminators are adopted in the training stage. Finally, we use a three-step training strategy to optimize our model. We submit two models with different sets of parameters to meet the RTF requirement of the two tracks. According to the official results, the proposed systems rank 2nd in track 1 and 3rd in track 2.
【5】 High-precision Voice Search Query Correction via Retrievable Speech-text  Embedings标题：基于可检索语音-文本嵌入的高精度语音搜索查询校正链接：https://arxiv.org/abs/2401.04235
作者：Christopher Li,Gary Wang,Kyle Kastner,Heng Su,Allen Chen,Andrew Rosenberg,Zhehuai Chen,Zelin Wu,Leonid Velikovich,Pat Rondon,Diamantino Caseiro,Petar Aleksic摘要：自动语音识别（ASR）系统可能由于各种原因（诸如嘈杂的音频、缺乏足够的训练数据等）而遭受不良召回。  以前的工作表明，召回可以通过检索重写候选人从一个大的数据库中的可能的，上下文相关的替代品的假设文本使用最近的邻居搜索嵌入的ASR假设文本，以纠正和候选人的更正。  然而，基于ASR假设的检索可以产生差的精度，如果文本假设是太语音不相似的成绩单的真相。在本文中，我们消除了假设音频不匹配的问题，直接使用嵌入来自话语音频查询的校正数据库;嵌入的话语音频和候选人的更正产生的多模态语音文本嵌入网络训练放置嵌入的音频的话语和嵌入其相应的文本转录接近在一起。  在使用最近邻搜索定位适当的校正候选者之后，我们在将候选者添加到原始的n-最佳列表之前，用其语音-文本嵌入距离对候选者进行评分。  我们显示了相对字错误率（WER）减少6%的话语，其成绩单出现在候选人集，而不增加WER的一般话语。摘要：Automatic speech recognition (ASR) systems can suffer from poor recall for various reasons, such as noisy audio, lack of sufficient training data, etc.  Previous work has shown that recall can be improved by retrieving rewrite candidates from a large database of likely, contextually-relevant alternatives to the hypothesis text using nearest-neighbors search over embeddings of the ASR hypothesis text to correct and candidate corrections.  However, ASR-hypothesis-based retrieval can yield poor precision if the textual hypotheses are too phonetically dissimilar to the transcript truth. In this paper, we eliminate the hypothesis-audio mismatch problem by querying the correction database directly using embeddings derived from the utterance audio; the embeddings of the utterance audio and candidate corrections are produced by multimodal speech-text embedding networks trained to place the embedding of the audio of an utterance and the embedding of its corresponding textual transcript close together.  After locating an appropriate correction candidate using nearest-neighbor search, we score the candidate with its speech-text embedding distance before adding the candidate to the original n-best list.  We show a relative word error rate (WER) reduction of 6% on utterances whose transcripts appear in the candidate set, without increasing WER on general utterances.
【6】 FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild标题：FunnyNet-W：野外视频中搞笑时刻的多模式学习链接：https://arxiv.org/abs/2401.04210
作者：Zhi-Song Liu,Robin Courant,Vicky Kalogeiton备注：22 pages, 14 figures摘要：自动理解有趣的时刻（即，当观看喜剧时，让人发笑的时刻是具有挑战性的，因为它们涉及到各种特征，如肢体语言，对话和文化。在本文中，我们提出了FunnyNet-W，这是一种依赖于视觉，音频和文本数据的交叉和自我注意力来预测视频中有趣时刻的模型。与大多数依赖于字幕形式的地面实况数据的方法不同，在这项工作中，我们利用了视频中自然出现的模式：（a）视频帧，因为它们包含对于场景理解必不可少的视觉信息，（b）音频，因为它包含与有趣时刻相关联的高级线索，例如语调，音高和停顿，以及（c）用语音到文本模型自动提取的文本，因为它在由大型语言模型处理时可以提供丰富的信息。为了获取标签进行训练，我们提出了一种无监督的方法，可以发现和标记有趣的音频时刻。我们提供了五个数据集上的实验：情景喜剧TBBT，MHD，MUSTARD，朋友，和TED谈话UR-Funny。大量的实验和分析表明，FunnyNet-W成功地利用视觉，听觉和文本线索来识别有趣的时刻，而我们的研究结果揭示了FunnyNet-W的预测有趣的时刻在野外的能力。FunnyNet-W在所有数据集上使用多模态线索进行有趣时刻检测，使用和不使用地面真实信息。摘要：Automatically understanding funny moments (i.e., the moments that make people laugh) when watching comedy is challenging, as they relate to various features, such as body language, dialogues and culture. In this paper, we propose FunnyNet-W, a model that relies on cross- and self-attention for visual, audio and text data to predict funny moments in videos. Unlike most methods that rely on ground truth data in the form of subtitles, in this work we exploit modalities that come naturally with videos: (a) video frames as they contain visual information indispensable for scene understanding, (b) audio as it contains higher-level cues associated with funny moments, such as intonation, pitch and pauses and (c) text automatically extracted with a speech-to-text model as it can provide rich information when processed by a Large Language Model. To acquire labels for training, we propose an unsupervised approach that spots and labels funny audio moments. We provide experiments on five datasets: the sitcoms TBBT, MHD, MUStARD, Friends, and the TED talk UR-Funny. Extensive experiments and analysis show that FunnyNet-W successfully exploits visual, auditory and textual cues to identify funny moments, while our findings reveal FunnyNet-W's ability to predict funny moments in the wild. FunnyNet-W sets the new state of the art for funny moment detection with multimodal cues on all datasets with and without using ground truth information.
【7】 Efficient Selective Audio Masked Multimodal Bottleneck Transformer for  Audio-Video Classification标题：用于音视频分类的高效选择性音频掩蔽多模态瓶颈Transformer链接：https://arxiv.org/abs/2401.04154
作者：Wentao Zhu备注：Accepted by WACV 2024; well-formatted PDF is in this https URL arXiv admin note: text overlap with arXiv:2401.04023摘要：音频和视频是主流媒体平台中最常见的两种形式，例如，YouTube.为了有效地从多模态视频中学习，在这项工作中，我们提出了一种新的音频-视频识别方法，称为音频视频Transformer，AVT，利用视频Transformer的有效时空表示，以提高动作识别的准确性。对于多模态融合，在跨模态Transformer中简单地连接多模态标记需要大量的计算和存储资源，相反，我们通过音频-视频瓶颈Transformer来降低跨模态复杂度。为了提高多模态Transformer的学习效率，我们集成了自监督目标，即，音频-视频对比学习、音频-视频匹配和掩蔽音频和视频学习，到AVT训练中，其将不同的音频和视频表示映射到公共的多模态表示空间中。我们进一步提出了一个掩蔽的音频段损失，学习语义音频活动的AVT。在三个公共数据集和两个内部数据集上的大量实验和消融研究一致地证明了所提出的AVT的有效性。具体来说，AVT在Kinetics-Sounds上的表现比之前最先进的同类产品高出8%。AVT还通过利用音频信号，在VGGSound上超过了之前最先进的视频Transformers [25] 10%。与先前最先进的多模式方法之一MBT [32]相比，AVT在FLOP方面的效率提高了1.3%，并且在Epic-Kitchen-100上的准确性提高了3.8%。摘要：Audio and video are two most common modalities in the mainstream media platforms, e.g., YouTube. To learn from multimodal videos effectively, in this work, we propose a novel audio-video recognition approach termed audio video Transformer, AVT, leveraging the effective spatio-temporal representation by the video Transformer to improve action recognition accuracy. For multimodal fusion, simply concatenating multimodal tokens in a cross-modal Transformer requires large computational and memory resources, instead we reduce the cross-modality complexity through an audio-video bottleneck Transformer. To improve the learning efficiency of multimodal Transformer, we integrate self-supervised objectives, i.e., audio-video contrastive learning, audio-video matching, and masked audio and video learning, into AVT training, which maps diverse audio and video representations into a common multimodal representation space. We further propose a masked audio segment loss to learn semantic audio activities in AVT. Extensive experiments and ablation studies on three public datasets and two in-house datasets consistently demonstrate the effectiveness of the proposed AVT. Specifically, AVT outperforms its previous state-of-the-art counterparts on Kinetics-Sounds by 8%. AVT also surpasses one of the previous state-of-the-art video Transformers [25] by 10% on VGGSound by leveraging the audio signal. Compared to one of the previous state-of-the-art multimodal methods, MBT [32], AVT is 1.3% more efficient in terms of FLOPs and improves the accuracy by 3.8% on Epic-Kitchens-100.
【8】 Cross-Speaker Encoding Network for Multi-Talker Speech Recognition标题：用于多说话人语音识别的跨话者编码网络链接：https://arxiv.org/abs/2401.04152
作者：Jiawen Kang,Lingwei Meng,Mingyu Cui,Haohan Guo,Xixin Wu,Xunying Liu,Helen Meng备注：Accepted by ICASSP2024摘要：端到端的多说话人语音识别作为一种直接转录多个说话人重叠语音的有效方法已经引起了人们的极大兴趣。当前的方法通常采用1）具有分支编码器的单输入多输出（SIMO）模型，或者2）基于具有串行化输出训练（SOT）的基于注意力的编码器-解码器架构的单输入单输出（SISO）模型。在这项工作中，我们提出了一个跨扬声器编码（CSE）网络，以解决SIMO模型的限制，通过聚合跨扬声器表示。此外，CSE模型与SOT集成，以利用SIMO和SISO的优点，同时减轻它们的缺点。据我们所知，这项工作是一个早期的努力，集成SIMO和SISO多说话人语音识别。在两个说话人的LibrispeechMix数据集上的实验表明，CES模型比SIMO基线降低了8%的单词错误率（WER）。与SOT模型相比，CSE-SOT模型总体上将WER降低了10%，在高重叠语音上降低了16%。摘要：End-to-end multi-talker speech recognition has garnered great interest as an effective approach to directly transcribe overlapped speech from multiple speakers. Current methods typically adopt either 1) single-input multiple-output (SIMO) models with a branched encoder, or 2) single-input single-output (SISO) models based on attention-based encoder-decoder architecture with serialized output training (SOT). In this work, we propose a Cross-Speaker Encoding (CSE) network to address the limitations of SIMO models by aggregating cross-speaker representations. Furthermore, the CSE model is integrated with SOT to leverage both the advantages of SIMO and SISO while mitigating their drawbacks. To the best of our knowledge, this work represents an early effort to integrate SIMO and SISO for multi-talker speech recognition. Experiments on the two-speaker LibrispeechMix dataset show that the CES model reduces word error rate (WER) by 8% over the SIMO baseline. The CSE-SOT model reduces WER by 10% overall and by 16% on high-overlap speech compared to the SOT model.
【9】 Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement标题：基于说话人解缠的Zero-Shot音频情感转换链接：https://arxiv.org/abs/2401.04511
作者：Soumya Dutta,Sriram Ganapathy备注：5 pages, 3 figures, accepted at ICASSP 2024摘要：音频到音频（A2A）风格转换的问题涉及用来自目标音频的风格特征替换源音频的风格特征，同时保留源音频的内容相关属性。在本文中，我们提出了一种有效的方法，称为Zero-shot情感风格转移（ZEST），允许转移的情感内容存在于给定的源音频与嵌入在目标音频，同时保留扬声器和语音内容的源。所提出的系统建立在将语音分解为语义标记、说话者表示和情感嵌入的基础上。利用这些因素，我们提出了一个框架来重建给定的语音信号的音高轮廓和训练解码器，重建语音信号。该模型使用基于重建损失的自我监督来训练。在转换期间，情感嵌入仅从目标音频导出，而其余因素从源音频导出。在我们的实验中，我们表明，即使不使用来自源或目标音频的并行训练数据或标签，我们也使用客观和主观质量评估来说明所提出的ZEST模型的zero shot情感传输能力。摘要：The problem of audio-to-audio (A2A) style transfer involves replacing the style features of the source audio with those from the target audio while preserving the content related attributes of the source audio. In this paper, we propose an efficient approach, termed as Zero-shot Emotion Style Transfer (ZEST), that allows the transfer of emotional content present in the given source audio with the one embedded in the target audio while retaining the speaker and speech content from the source. The proposed system builds upon decomposing speech into semantic tokens, speaker representations and emotion embeddings. Using these factors, we propose a framework to reconstruct the pitch contour of the given speech signal and train a decoder that reconstructs the speech signal. The model is trained using a self-supervision based reconstruction loss. During conversion, the emotion embedding is alone derived from the target audio, while rest of the factors are derived from the source audio. In our experiments, we show that, even without using parallel training data or labels from the source or target audio, we illustrate zero shot emotion transfer capabilities of the proposed ZEST model using objective and subjective quality evaluations.
【10】 Class-Incremental Learning for Multi-Label Audio Classification标题：基于类增量学习的多标签音频分类方法链接：https://arxiv.org/abs/2401.04447
作者：Manjunath Mulimani,Annamaria Mesaros备注：Accepted to ICASSP 2024摘要：在本文中，我们提出了一种用于潜在重叠声音的类增量学习的方法，用于解决一系列多标签音频分类任务。我们设计了一个增量学习器，它独立于旧类学习新类。为了保留有关旧类的知识，我们提出了一种基于余弦相似性的蒸馏损失，最大限度地减少后续学习者的特征表示的差异，并将其与基于Kullback-Leibler发散的蒸馏损失一起使用，最大限度地减少各自输出的差异。实验在具有50个声音类的数据集上进行，初始分类任务包含30个基本类和4个增量阶段，每个阶段5个类。在每个阶段之后，系统将使用到目前为止学习的整个类集进行多标签分类测试。所提出的方法在五个阶段中获得了40.9%的平均F1分数，范围从30个类的阶段0中的45.2%到50个类的阶段4中的36.3%。增量阶段的平均性能下降仅为初始F1分数45.2%的0.7个百分点。摘要：In this paper, we propose a method for class-incremental learning of potentially overlapping sounds for solving a sequence of multi-label audio classification tasks. We design an incremental learner that learns new classes independently of the old classes. To preserve knowledge about the old classes, we propose a cosine similarity-based distillation loss that minimizes discrepancy in the feature representations of subsequent learners, and use it along with a Kullback-Leibler divergence-based distillation loss that minimizes discrepancy in their respective outputs. Experiments are performed on a dataset with 50 sound classes, with an initial classification task containing 30 base classes and 4 incremental phases of 5 classes each. After each phase, the system is tested for multi-label classification with the entire set of classes learned so far. The proposed method obtains an average F1-score of 40.9% over the five phases, ranging from 45.2% in phase 0 on 30 classes, to 36.3% in phase 4 on 50 classes. Average performance degradation over incremental phases is only 0.7 percentage points from the initial F1-score of 45.2%.
【11】 FADI-AEC: Fast Score Based Diffusion Model Guided by Far-end Signal for  Acoustic Echo Cancellation标题：FADI-AEC：远端信号引导的快速分数扩散模型用于声回波消除链接：https://arxiv.org/abs/2401.04283
作者：Yang Liu,Li Wan,Yun Li,Yiteng Huang,Ming Sun,James Luan,Yangyang Shi,Xin Lei摘要：尽管扩散模型在语音增强中具有潜力，但其在声学回声消除（AEC）中的部署受到限制。在本文中，我们提出了DI-AEC，开创了一个基于扩散的随机再生方法，致力于AEC。此外，我们提出了FADI-AEC，快速基于分数的扩散AEC框架，以节省计算需求，使其有利于边缘设备。它通过每帧运行一次得分模型而脱颖而出，从而实现了处理效率的显著提升。除此之外，我们还介绍了一种新的噪声生成技术，其中利用远端信号，将远端和近端信号结合起来，以提高分数模型的准确性。我们在ICASSP 2023 Microsoft深度回声消除挑战评估数据集上测试了我们提出的方法，其中我们的方法优于一些端到端方法和其他基于扩散的回声消除方法。摘要：Despite the potential of diffusion models in speech enhancement, their deployment in Acoustic Echo Cancellation (AEC) has been restricted. In this paper, we propose DI-AEC, pioneering a diffusion-based stochastic regeneration approach dedicated to AEC. Further, we propose FADI-AEC, fast score-based diffusion AEC framework to save computational demands, making it favorable for edge devices. It stands out by running the score model once per frame, achieving a significant surge in processing efficiency. Apart from that, we introduce a novel noise generation technique where far-end signals are utilized, incorporating both far-end and near-end signals to refine the score model's accuracy. We test our proposed method on the ICASSP2023 Microsoft deep echo cancellation challenge evaluation dataset, where our method outperforms some of the end-to-end methods and other diffusion based echo cancellation methods.
【12】 Using perceptive subbands analysis to perform audio scenes cartography标题：利用感知子带分析进行音频场景制图链接：https://arxiv.org/abs/2401.04127
作者：Laurent Millot,Gérard Pelé,Mohammed Elliq备注：None摘要：真实或模拟立体声录音的音频场景制图。该音频场景分析是连续进行的：感知10-子带分析，使用短时恒定场景假设和允许跟随移动源移动的通道间相关性计算每个子带的两个通道之间的相对延迟和增益的时间规律，计算其峰值给出固定源的发生率信息的全局和子带直方图。由2到4个固定源或一个固定源和一个移动源组成的音频场景已经成功测试。进一步的扩展和应用将被讨论。音频场景的音频插图，子带分析和实时立体声录音模拟的演示将被给出。论文6340在音频工程学会第118届会议上提出，巴塞罗那，2005年摘要：Audio scene cartography for real or simulated stereo recordings is presented. This audio scene analysis is performed doing successively: a perceptive 10-subbands analysis, calculation of temporal laws for relative delays and gains between both channels of each subband using a short-time cons\-tant scene assumption and channels inter-correlation which permit to follow a mobile source in its moves, calculation of global and subbands histograms whose peaks give the incidence information for fixed sources. Audio scenes composed of 2 to 4 fixed sources or with a fixed source and a mobile one have been already successfully tested. Further extensions and applications will be discussed. Audio illustrations of audio scenes, subband analysis and demonstration of real-time stereo recording simulations will be given.Paper 6340 presented at the 118th Convention of the Audio Engineering Society, Barcelona, 2005
eess.AS音频处理
【1】 Zero Shot Audio to Audio Emotion Transfer With Speaker Disentanglement标题：零拍摄音频到音频情感转移与扬声器解开链接：https://arxiv.org/abs/2401.04511
作者：Soumya Dutta,Sriram Ganapathy备注：5 pages, 3 figures, accepted at ICASSP 2024摘要：音频到音频（A2A）风格转换的问题涉及用来自目标音频的风格特征替换源音频的风格特征，同时保留源音频的内容相关属性。在本文中，我们提出了一种有效的方法，称为Zero-shot情感风格转移（ZEST），允许转移的情感内容存在于给定的源音频与嵌入在目标音频，同时保留扬声器和语音内容的源。所提出的系统建立在将语音分解为语义标记、说话者表示和情感嵌入的基础上。利用这些因素，我们提出了一个框架来重建给定的语音信号的音高轮廓和训练解码器，重建语音信号。该模型使用基于重建损失的自我监督来训练。在转换期间，情感嵌入仅从目标音频导出，而其余因素从源音频导出。在我们的实验中，我们表明，即使不使用来自源或目标音频的并行训练数据或标签，我们也使用客观和主观质量评估来说明所提出的ZEST模型的zero shot情感传输能力。摘要：The problem of audio-to-audio (A2A) style transfer involves replacing the style features of the source audio with those from the target audio while preserving the content related attributes of the source audio. In this paper, we propose an efficient approach, termed as Zero-shot Emotion Style Transfer (ZEST), that allows the transfer of emotional content present in the given source audio with the one embedded in the target audio while retaining the speaker and speech content from the source. The proposed system builds upon decomposing speech into semantic tokens, speaker representations and emotion embeddings. Using these factors, we propose a framework to reconstruct the pitch contour of the given speech signal and train a decoder that reconstructs the speech signal. The model is trained using a self-supervision based reconstruction loss. During conversion, the emotion embedding is alone derived from the target audio, while rest of the factors are derived from the source audio. In our experiments, we show that, even without using parallel training data or labels from the source or target audio, we illustrate zero shot emotion transfer capabilities of the proposed ZEST model using objective and subjective quality evaluations.
【2】 Class-Incremental Learning for Multi-Label Audio Classification标题：基于类增量学习的多标签音频分类方法链接：https://arxiv.org/abs/2401.04447
作者：Manjunath Mulimani,Annamaria Mesaros备注：Accepted to ICASSP 2024摘要：在本文中，我们提出了一种用于潜在重叠声音的类增量学习的方法，用于解决一系列多标签音频分类任务。我们设计了一个增量学习器，它独立于旧类学习新类。为了保留有关旧类的知识，我们提出了一种基于余弦相似性的蒸馏损失，最大限度地减少后续学习者的特征表示的差异，并将其与基于Kullback-Leibler发散的蒸馏损失一起使用，最大限度地减少各自输出的差异。实验在具有50个声音类的数据集上进行，初始分类任务包含30个基本类和4个增量阶段，每个阶段5个类。在每个阶段之后，系统将使用到目前为止学习的整个类集进行多标签分类测试。所提出的方法在五个阶段中获得了40.9%的平均F1分数，范围从30个类的阶段0中的45.2%到50个类的阶段4中的36.3%。增量阶段的平均性能下降仅为初始F1分数45.2%的0.7个百分点。摘要：In this paper, we propose a method for class-incremental learning of potentially overlapping sounds for solving a sequence of multi-label audio classification tasks. We design an incremental learner that learns new classes independently of the old classes. To preserve knowledge about the old classes, we propose a cosine similarity-based distillation loss that minimizes discrepancy in the feature representations of subsequent learners, and use it along with a Kullback-Leibler divergence-based distillation loss that minimizes discrepancy in their respective outputs. Experiments are performed on a dataset with 50 sound classes, with an initial classification task containing 30 base classes and 4 incremental phases of 5 classes each. After each phase, the system is tested for multi-label classification with the entire set of classes learned so far. The proposed method obtains an average F1-score of 40.9% over the five phases, ranging from 45.2% in phase 0 on 30 classes, to 36.3% in phase 4 on 50 classes. Average performance degradation over incremental phases is only 0.7 percentage points from the initial F1-score of 45.2%.
【3】 FADI-AEC: Fast Score Based Diffusion Model Guided by Far-end Signal for  Acoustic Echo Cancellation标题：FADI-AEC：远端信号引导的快速分数扩散模型用于声回波消除链接：https://arxiv.org/abs/2401.04283
作者：Yang Liu,Li Wan,Yun Li,Yiteng Huang,Ming Sun,James Luan,Yangyang Shi,Xin Lei摘要：尽管扩散模型在语音增强中具有潜力，但其在声学回声消除（AEC）中的部署受到限制。在本文中，我们提出了DI-AEC，开创了一个基于扩散的随机再生方法，致力于AEC。此外，我们提出了FADI-AEC，快速基于分数的扩散AEC框架，以节省计算需求，使其有利于边缘设备。它通过每帧运行一次得分模型而脱颖而出，从而实现了处理效率的显著提升。除此之外，我们还介绍了一种新的噪声生成技术，其中利用远端信号，将远端和近端信号结合起来，以提高分数模型的准确性。我们在ICASSP 2023 Microsoft深度回声消除挑战评估数据集上测试了我们提出的方法，其中我们的方法优于一些端到端方法和其他基于扩散的回声消除方法。摘要：Despite the potential of diffusion models in speech enhancement, their deployment in Acoustic Echo Cancellation (AEC) has been restricted. In this paper, we propose DI-AEC, pioneering a diffusion-based stochastic regeneration approach dedicated to AEC. Further, we propose FADI-AEC, fast score-based diffusion AEC framework to save computational demands, making it favorable for edge devices. It stands out by running the score model once per frame, achieving a significant surge in processing efficiency. Apart from that, we introduce a novel noise generation technique where far-end signals are utilized, incorporating both far-end and near-end signals to refine the score model's accuracy. We test our proposed method on the ICASSP2023 Microsoft deep echo cancellation challenge evaluation dataset, where our method outperforms some of the end-to-end methods and other diffusion based echo cancellation methods.
【4】 Using perceptive subbands analysis to perform audio scenes cartography标题：利用感知子带分析进行音频场景制图链接：https://arxiv.org/abs/2401.04127
作者：Laurent Millot,Gérard Pelé,Mohammed Elliq备注：None摘要：真实或模拟立体声录音的音频场景制图。该音频场景分析是连续进行的：感知10-子带分析，使用短时恒定场景假设和允许跟随移动源移动的通道间相关性计算每个子带的两个通道之间的相对延迟和增益的时间规律，计算其峰值给出固定源的发生率信息的全局和子带直方图。由2到4个固定源或一个固定源和一个移动源组成的音频场景已经成功测试。进一步的扩展和应用将被讨论。音频场景的音频插图，子带分析和实时立体声录音模拟的演示将被给出。论文6340在音频工程学会第118届会议上提出，巴塞罗那，2005年摘要：Audio scene cartography for real or simulated stereo recordings is presented. This audio scene analysis is performed doing successively: a perceptive 10-subbands analysis, calculation of temporal laws for relative delays and gains between both channels of each subband using a short-time cons\-tant scene assumption and channels inter-correlation which permit to follow a mobile source in its moves, calculation of global and subbands histograms whose peaks give the incidence information for fixed sources. Audio scenes composed of 2 to 4 fixed sources or with a fixed source and a mobile one have been already successfully tested. Further extensions and applications will be discussed. Audio illustrations of audio scenes, subband analysis and demonstration of real-time stereo recording simulations will be given.Paper 6340 presented at the 118th Convention of the Audio Engineering Society, Barcelona, 2005
【5】 Masked Audio Generation using a Single Non-Autoregressive Transformer标题：使用单个非自回归Transformer的掩蔽音频生成链接：https://arxiv.org/abs/2401.04577
作者：Alon Ziv,Itai Gat,Gael Le Lan,Tal Remez,Felix Kreuk,Alexandre Défossez,Jade Copet,Gabriel Synnaeve,Yossi Adi摘要：我们介绍MAGNeT，一种屏蔽生成序列建模方法，直接在几个音频令牌流上操作。与以前的工作不同，MAGNeT是由一个单级，非自回归Transformer。在训练过程中，我们预测从掩码调度器获得的掩码令牌的跨度，而在推理过程中，我们使用几个解码步骤逐步构建输出序列。为了进一步提高生成的音频的质量，我们引入了一种新的重新评分方法，在该方法中，我们利用外部预训练模型对来自MAGNeT的预测进行重新评分和排名，然后将其用于后续的解码步骤。最后，我们探索了MAGNeT的混合版本，在该版本中，我们在自回归和非自回归模型之间进行融合，以自回归方式生成前几秒，而序列的其余部分则被并行解码。我们证明了MAGNeT的文本到音乐和文本到音频生成任务的效率，并进行了广泛的实证评估，同时考虑到客观指标和人类研究。所提出的方法与所评估的基线相当，同时明显更快（比自回归基线快7倍）。通过消融研究和分析，我们揭示了组成MAGNeT的每个组件的重要性，同时指出了自回归和非自回归建模之间的权衡，考虑到延迟，吞吐量和生成质量。样品可在我们的演示页面https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT上获得。摘要：We introduce MAGNeT, a masked generative sequence modeling method that operates directly over several streams of audio tokens. Unlike prior work, MAGNeT is comprised of a single-stage, non-autoregressive transformer. During training, we predict spans of masked tokens obtained from a masking scheduler, while during inference we gradually construct the output sequence using several decoding steps. To further enhance the quality of the generated audio, we introduce a novel rescoring method in which, we leverage an external pre-trained model to rescore and rank predictions from MAGNeT, which will be then used for later decoding steps. Lastly, we explore a hybrid version of MAGNeT, in which we fuse between autoregressive and non-autoregressive models to generate the first few seconds in an autoregressive manner while the rest of the sequence is being decoded in parallel. We demonstrate the efficiency of MAGNeT for the task of text-to-music and text-to-audio generation and conduct an extensive empirical evaluation, considering both objective metrics and human studies. The proposed approach is comparable to the evaluated baselines, while being significantly faster (x7 faster than the autoregressive baseline). Through ablation studies and analysis, we shed light on the importance of each of the components comprising MAGNeT, together with pointing to the trade-offs between autoregressive and non-autoregressive modeling, considering latency, throughput, and generation quality. Samples are available on our demo page https://pages.cs.huji.ac.il/adiyoss-lab/MAGNeT.
【6】 HyperGANStrument: Instrument Sound Synthesis and Editing with  Pitch-Invariant Hypernetworks标题：HyperGANStrument：基于基音不变超网络的乐器声音合成与编辑链接：https://arxiv.org/abs/2401.04558
作者：Zhe Zhang,Taketo Akama备注：5 pages, 3 figures, Accepted for 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Audio examples: this https URL摘要：GANStrument利用具有音高不变特征提取器和实例调节技术的GAN，在合成逼真的乐器声音方面表现出了卓越的能力。为了进一步提高重建能力和音高精度，以增强用户提供的声音的可编辑性，我们提出了HyperGANStrument，它引入了一个音高不变的超网络来调制预先训练的GANStrument生成器的权重，给定一个单次声音作为输入。超网络调制在输入声音的重建中为发生器提供反馈。此外，我们利用对抗微调计划的超网络，以提高重建保真度和发电机的发电多样性。实验结果表明，该模型不仅增强了GANStrument的生成能力，而且显著提高了合成声音的可编辑性。在线演示页面提供音频示例。摘要：GANStrument, exploiting GANs with a pitch-invariant feature extractor and instance conditioning technique, has shown remarkable capabilities in synthesizing realistic instrument sounds. To further improve the reconstruction ability and pitch accuracy to enhance the editability of user-provided sound, we propose HyperGANStrument, which introduces a pitch-invariant hypernetwork to modulate the weights of a pre-trained GANStrument generator, given a one-shot sound as input. The hypernetwork modulation provides feedback for the generator in the reconstruction of the input sound. In addition, we take advantage of an adversarial fine-tuning scheme for the hypernetwork to improve the reconstruction fidelity and generation diversity of the generator. Experimental results show that the proposed model not only enhances the generation capability of GANStrument but also significantly improves the editability of synthesized sounds. Audio examples are available at the online demo page.
【7】 SonicVisionLM: Playing Sound with Vision Language Models标题：SonicVisionLM：用视觉语言模型播放声音链接：https://arxiv.org/abs/2401.04394
作者：Zhifeng Xie,Shengye Yu,Mengtian Li,Qile He,Chaofeng Chen,Yu-Gang Jiang摘要：人们对为无声视频生成声音的任务越来越感兴趣，主要是因为它在简化视频后期制作方面的实用性。然而，用于视频-声音生成的现有方法试图从视觉表示直接创建声音，由于难以将视觉表示与音频表示对准，这可能是具有挑战性的。在本文中，我们提出了SonicVisionLM，一个新的框架，旨在通过利用视觉语言模型产生广泛的声音效果。我们使用强大的视觉语言模型（VLM）的功能，而不是直接从视频生成音频。当提供无声视频时，我们的方法首先使用VLM来识别视频内的事件，以建议与视频内容匹配的可能声音。这种方法的转变将图像和音频对齐的挑战性任务转化为通过流行的扩散模型对齐图像到文本和文本到音频的更深入研究的子问题。为了提高LLM音频推荐的质量，我们收集了一个广泛的数据集，将文本描述映射到特定的声音效果，并开发了时间控制的音频适配器。我们的方法超越了当前最先进的视频转换为音频的方法，从而增强了与视觉效果的同步，并改善了音频和视频组件之间的对齐。项目页面：https://yusiissy.github.io/SonicVisionLM.github.io/摘要：There has been a growing interest in the task of generating sound for silent videos, primarily because of its practicality in streamlining video post-production. However, existing methods for video-sound generation attempt to directly create sound from visual representations, which can be challenging due to the difficulty of aligning visual representations with audio representations. In this paper, we present SonicVisionLM, a novel framework aimed at generating a wide range of sound effects by leveraging vision language models. Instead of generating audio directly from video, we use the capabilities of powerful vision language models (VLMs). When provided with a silent video, our approach first identifies events within the video using a VLM to suggest possible sounds that match the video content. This shift in approach transforms the challenging task of aligning image and audio into more well-studied sub-problems of aligning image-to-text and text-to-audio through the popular diffusion models. To improve the quality of audio recommendations with LLMs, we have collected an extensive dataset that maps text descriptions to specific sound effects and developed temporally controlled audio adapters. Our approach surpasses current state-of-the-art methods for converting video to audio, resulting in enhanced synchronization with the visuals and improved alignment between audio and video components. Project page: https://yusiissy.github.io/SonicVisionLM.github.io/
【8】 RaD-Net: A Repairing and Denoising Network for Speech Signal Improvement标题：Rad-Net：一种用于语音信号改善的修复和去噪网络链接：https://arxiv.org/abs/2401.04389
作者：Mingshuai Liu,Zhuangqi Chen,Xiaopeng Yan,Yuanjun Lv,Xianjun Xia,Chuanzeng Huang,Yijian Xiao,Lei Xie备注：submitted to ICASSP 2024摘要：本文介绍了我们为ICASSP 2024语音信号改善（SSI）挑战赛开发的修复和去噪网络（RaD-Net）。我们扩展了我们以前的框架的基础上，两阶段的网络，并提出了一个升级的模型。具体来说，我们用TEA-PSE中的COM-Net替换了修复网络。此外，在训练阶段采用了多分辨率鉴别器和多波段鉴别器。最后，我们使用三步训练策略来优化我们的模型。我们提交了两个模型，具有不同的参数集，以满足两个轨道的RTF要求。根据官方结果，拟议的系统在轨道1中排名第二，在轨道2中排名第三。摘要：This paper introduces our repairing and denoising network (RaD-Net) for the ICASSP 2024 Speech Signal Improvement (SSI) Challenge. We extend our previous framework based on a two-stage network and propose an upgraded model. Specifically, we replace the repairing network with COM-Net from TEA-PSE. In addition, multi-resolution discriminators and multi-band discriminators are adopted in the training stage. Finally, we use a three-step training strategy to optimize our model. We submit two models with different sets of parameters to meet the RTF requirement of the two tracks. According to the official results, the proposed systems rank 2nd in track 1 and 3rd in track 2.
【9】 High-precision Voice Search Query Correction via Retrievable Speech-text  Embedings标题：基于可检索语音-文本嵌入的高精度语音搜索查询校正链接：https://arxiv.org/abs/2401.04235
作者：Christopher Li,Gary Wang,Kyle Kastner,Heng Su,Allen Chen,Andrew Rosenberg,Zhehuai Chen,Zelin Wu,Leonid Velikovich,Pat Rondon,Diamantino Caseiro,Petar Aleksic摘要：自动语音识别（ASR）系统可能由于各种原因（诸如嘈杂的音频、缺乏足够的训练数据等）而遭受不良召回。  以前的工作表明，召回可以通过检索重写候选人从一个大的数据库中的可能的，上下文相关的替代品的假设文本使用最近的邻居搜索嵌入的ASR假设文本，以纠正和候选人的更正。  然而，基于ASR假设的检索可以产生差的精度，如果文本假设是太语音不相似的成绩单的真相。在本文中，我们消除了假设音频不匹配的问题，直接使用嵌入来自话语音频查询的校正数据库;嵌入的话语音频和候选人的更正产生的多模态语音文本嵌入网络训练放置嵌入的音频的话语和嵌入其相应的文本转录接近在一起。  在使用最近邻搜索定位适当的校正候选者之后，我们在将候选者添加到原始的n-最佳列表之前，用其语音-文本嵌入距离对候选者进行评分。  我们显示了相对字错误率（WER）减少6%的话语，其成绩单出现在候选人集，而不增加WER的一般话语。摘要：Automatic speech recognition (ASR) systems can suffer from poor recall for various reasons, such as noisy audio, lack of sufficient training data, etc.  Previous work has shown that recall can be improved by retrieving rewrite candidates from a large database of likely, contextually-relevant alternatives to the hypothesis text using nearest-neighbors search over embeddings of the ASR hypothesis text to correct and candidate corrections.  However, ASR-hypothesis-based retrieval can yield poor precision if the textual hypotheses are too phonetically dissimilar to the transcript truth. In this paper, we eliminate the hypothesis-audio mismatch problem by querying the correction database directly using embeddings derived from the utterance audio; the embeddings of the utterance audio and candidate corrections are produced by multimodal speech-text embedding networks trained to place the embedding of the audio of an utterance and the embedding of its corresponding textual transcript close together.  After locating an appropriate correction candidate using nearest-neighbor search, we score the candidate with its speech-text embedding distance before adding the candidate to the original n-best list.  We show a relative word error rate (WER) reduction of 6% on utterances whose transcripts appear in the candidate set, without increasing WER on general utterances.
【10】 FunnyNet-W: Multimodal Learning of Funny Moments in Videos in the Wild标题：FunnyNet-W：野外视频中搞笑时刻的多模式学习链接：https://arxiv.org/abs/2401.04210
作者：Zhi-Song Liu,Robin Courant,Vicky Kalogeiton备注：22 pages, 14 figures摘要：自动理解有趣的时刻（即，当观看喜剧时，让人发笑的时刻是具有挑战性的，因为它们涉及到各种特征，如肢体语言，对话和文化。在本文中，我们提出了FunnyNet-W，这是一种依赖于视觉，音频和文本数据的交叉和自我注意力来预测视频中有趣时刻的模型。与大多数依赖于字幕形式的地面实况数据的方法不同，在这项工作中，我们利用了视频中自然出现的模式：（a）视频帧，因为它们包含对于场景理解必不可少的视觉信息，（b）音频，因为它包含与有趣时刻相关联的高级线索，例如语调，音高和停顿，以及（c）用语音到文本模型自动提取的文本，因为它在由大型语言模型处理时可以提供丰富的信息。为了获取标签进行训练，我们提出了一种无监督的方法，可以发现和标记有趣的音频时刻。我们提供了五个数据集上的实验：情景喜剧TBBT，MHD，MUSTARD，朋友，和TED谈话UR-Funny。大量的实验和分析表明，FunnyNet-W成功地利用视觉，听觉和文本线索来识别有趣的时刻，而我们的研究结果揭示了FunnyNet-W的预测有趣的时刻在野外的能力。FunnyNet-W在所有数据集上使用多模态线索进行有趣时刻检测，使用和不使用地面真实信息。摘要：Automatically understanding funny moments (i.e., the moments that make people laugh) when watching comedy is challenging, as they relate to various features, such as body language, dialogues and culture. In this paper, we propose FunnyNet-W, a model that relies on cross- and self-attention for visual, audio and text data to predict funny moments in videos. Unlike most methods that rely on ground truth data in the form of subtitles, in this work we exploit modalities that come naturally with videos: (a) video frames as they contain visual information indispensable for scene understanding, (b) audio as it contains higher-level cues associated with funny moments, such as intonation, pitch and pauses and (c) text automatically extracted with a speech-to-text model as it can provide rich information when processed by a Large Language Model. To acquire labels for training, we propose an unsupervised approach that spots and labels funny audio moments. We provide experiments on five datasets: the sitcoms TBBT, MHD, MUStARD, Friends, and the TED talk UR-Funny. Extensive experiments and analysis show that FunnyNet-W successfully exploits visual, auditory and textual cues to identify funny moments, while our findings reveal FunnyNet-W's ability to predict funny moments in the wild. FunnyNet-W sets the new state of the art for funny moment detection with multimodal cues on all datasets with and without using ground truth information.
【11】 Efficient Selective Audio Masked Multimodal Bottleneck Transformer for  Audio-Video Classification标题：用于音视频分类的高效选择性音频掩蔽多模式瓶颈转换器链接：https://arxiv.org/abs/2401.04154
作者：Wentao Zhu备注：Accepted by WACV 2024; well-formatted PDF is in this https URL arXiv admin note: text overlap with arXiv:2401.04023摘要：音频和视频是主流媒体平台中最常见的两种形式，例如，YouTube.为了有效地从多模态视频中学习，在这项工作中，我们提出了一种新的音频-视频识别方法，称为音频视频Transformer，AVT，利用视频Transformer的有效时空表示，以提高动作识别的准确性。对于多模态融合，在跨模态Transformer中简单地连接多模态标记需要大量的计算和存储资源，相反，我们通过音频-视频瓶颈Transformer来降低跨模态复杂度。为了提高多模态Transformer的学习效率，我们集成了自监督目标，即，音频-视频对比学习、音频-视频匹配和掩蔽音频和视频学习，到AVT训练中，其将不同的音频和视频表示映射到公共的多模态表示空间中。我们进一步提出了一个掩蔽的音频段损失，学习语义音频活动的AVT。在三个公共数据集和两个内部数据集上的大量实验和消融研究一致地证明了所提出的AVT的有效性。具体来说，AVT在Kinetics-Sounds上的表现比之前最先进的同类产品高出8%。AVT还通过利用音频信号，在VGGSound上超过了之前最先进的视频Transformers [25] 10%。与先前最先进的多模式方法之一MBT [32]相比，AVT在FLOP方面的效率提高了1.3%，并且在Epic-Kitchen-100上的准确性提高了3.8%。摘要：Audio and video are two most common modalities in the mainstream media platforms, e.g., YouTube. To learn from multimodal videos effectively, in this work, we propose a novel audio-video recognition approach termed audio video Transformer, AVT, leveraging the effective spatio-temporal representation by the video Transformer to improve action recognition accuracy. For multimodal fusion, simply concatenating multimodal tokens in a cross-modal Transformer requires large computational and memory resources, instead we reduce the cross-modality complexity through an audio-video bottleneck Transformer. To improve the learning efficiency of multimodal Transformer, we integrate self-supervised objectives, i.e., audio-video contrastive learning, audio-video matching, and masked audio and video learning, into AVT training, which maps diverse audio and video representations into a common multimodal representation space. We further propose a masked audio segment loss to learn semantic audio activities in AVT. Extensive experiments and ablation studies on three public datasets and two in-house datasets consistently demonstrate the effectiveness of the proposed AVT. Specifically, AVT outperforms its previous state-of-the-art counterparts on Kinetics-Sounds by 8%. AVT also surpasses one of the previous state-of-the-art video Transformers [25] by 10% on VGGSound by leveraging the audio signal. Compared to one of the previous state-of-the-art multimodal methods, MBT [32], AVT is 1.3% more efficient in terms of FLOPs and improves the accuracy by 3.8% on Epic-Kitchens-100.
【12】 Cross-Speaker Encoding Network for Multi-Talker Speech Recognition标题：用于多说话人语音识别的跨话者编码网络链接：https://arxiv.org/abs/2401.04152
作者：Jiawen Kang,Lingwei Meng,Mingyu Cui,Haohan Guo,Xixin Wu,Xunying Liu,Helen Meng备注：Accepted by ICASSP2024摘要：端到端的多说话人语音识别作为一种直接转录多个说话人重叠语音的有效方法已经引起了人们的极大兴趣。当前的方法通常采用1）具有分支编码器的单输入多输出（SIMO）模型，或者2）基于具有串行化输出训练（SOT）的基于注意力的编码器-解码器架构的单输入单输出（SISO）模型。在这项工作中，我们提出了一个跨扬声器编码（CSE）网络，以解决SIMO模型的限制，通过聚合跨扬声器表示。此外，CSE模型与SOT集成，以利用SIMO和SISO的优点，同时减轻它们的缺点。据我们所知，这项工作是一个早期的努力，集成SIMO和SISO多说话人语音识别。在两个说话人的LibrispeechMix数据集上的实验表明，CES模型比SIMO基线降低了8%的单词错误率（WER）。与SOT模型相比，CSE-SOT模型总体上将WER降低了10%，在高重叠语音上降低了16%。摘要：End-to-end multi-talker speech recognition has garnered great interest as an effective approach to directly transcribe overlapped speech from multiple speakers. Current methods typically adopt either 1) single-input multiple-output (SIMO) models with a branched encoder, or 2) single-input single-output (SISO) models based on attention-based encoder-decoder architecture with serialized output training (SOT). In this work, we propose a Cross-Speaker Encoding (CSE) network to address the limitations of SIMO models by aggregating cross-speaker representations. Furthermore, the CSE model is integrated with SOT to leverage both the advantages of SIMO and SISO while mitigating their drawbacks. To the best of our knowledge, this work represents an early effort to integrate SIMO and SISO for multi-talker speech recognition. Experiments on the two-speaker LibrispeechMix dataset show that the CES model reduces word error rate (WER) by 8% over the SIMO baseline. The CSE-SOT model reduces WER by 10% overall and by 16% on high-overlap speech compared to the SOT model.
机器翻译由腾讯交互翻译提供，仅供参考