每日语音速递本文经arXiv每日学术速递授权转载
链接：https://arxiv.org/abs/2412.19351
摘要：近年来，文本到音频（TTA）合成取得了重大进展，使用户能够通过从自然语言提示生成的合成音频来丰富他们的创意工作流程。尽管取得了这一进展，但数据、模型架构、训练目标函数和采样策略对目标基准的影响还没有得到很好的理解。为了全面了解TTA模型的设计空间，我们建立了一个大规模的经验实验，重点是扩散和流量匹配模型。我们的贡献包括：1）AF-Synthetic，从音频理解模型中获得的高质量合成字幕的大型数据集; 2）TTA模型的不同架构、训练和推理设计选择的系统比较; 3）采样方法及其帕累托曲线分析相对于生成质量和推理速度。我们利用从这种广泛的分析中获得的知识，提出了我们最好的模型，称为阐明文本到音频（ETTA）。当在AudioCaps和MusicCaps上进行评估时，ETTA提供了对公开数据训练的基线的改进，同时与专有数据训练的模型竞争。最后，我们展示了ETTA在复杂和富有想象力的字幕之后生成创意音频的改进能力-这是一项比当前基准更具挑战性的任务。
摘要：Recent years have seen significant progress in Text-To-Audio (TTA) synthesis,enabling users to enrich their creative workflows with synthetic audiogenerated from natural language prompts. Despite this progress, the effects ofdata, model architecture, training objective functions, and sampling strategieson target benchmarks are not well understood. With the purpose of providing aholistic understanding of the design space of TTA models, we set up alarge-scale empirical experiment focused on diffusion and flow matching models.Our contributions include: 1) AF-Synthetic, a large dataset of high qualitysynthetic captions obtained from an audio understanding model; 2) a systematiccomparison of different architectural, training, and inference design choicesfor TTA models; 3) an analysis of sampling methods and their Pareto curves withrespect to generation quality and inference speed. We leverage the knowledgeobtained from this extensive analysis to propose our best model dubbedElucidated Text-To-Audio (ETTA). When evaluated on AudioCaps and MusicCaps,ETTA provides improvements over the baselines trained on publicly availabledata, while being competitive with models trained on proprietary data. Finally,we show ETTA's improved ability to generate creative audio following complexand imaginative captions -- a task that is more challenging than currentbenchmarks.
标题： 提高人工智能合成语音检测的通用性
链接：https://arxiv.org/abs/2412.19279
备注：AAAI25
摘要：人工智能合成语音技术有可能为有益的应用创造真实的人类声音，但它也可能被滥用于恶意目的。虽然现有的人工智能合成语音检测模型在域内评估方面表现出色，但它们在跨不同领域推广方面面临挑战，随着新的语音生成器的出现，它们可能会过时。当前的解决方案使用不同的数据和先进的机器学习技术（例如，域不变表示、自监督学习），但是受限于预定义的声码器以及对像背景噪声和说话者身份这样的因素的敏感性。在这项工作中，我们引入了一个创新的解纠缠框架，旨在提取域不可知的文物功能相关的声码器。利用这些功能，我们增强了平坦损失环境中的模型学习，从而能够避免次优解决方案并提高泛化能力。在基准测试上的大量实验表明，我们的方法优于最先进的方法，在域内的等错误率度量和跨域评估中分别实现了高达5.12%和7.59%的改进。
摘要：AI-synthesized voice technology has the potential to create realistic humanvoices for beneficial applications, but it can also be misused for maliciouspurposes. While existing AI-synthesized voice detection models excel inintra-domain evaluation, they face challenges in generalizing across differentdomains, potentially becoming obsolete as new voice generators emerge. Currentsolutions use diverse data and advanced machine learning techniques (e.g.,domain-invariant representation, self-supervised learning), but are limited bypredefined vocoders and sensitivity to factors like background noise andspeaker identity. In this work, we introduce an innovative disentanglementframework aimed at extracting domain-agnostic artifact features related tovocoders. Utilizing these features, we enhance model learning in a flat losslandscape, enabling escape from suboptimal solutions and improvinggeneralization. Extensive experiments on benchmarks show our approachoutperforms state-of-the-art methods, achieving up to 5.12% improvement in theequal error rate metric in intra-domain and 7.59% in cross-domain evaluations.
标题： 基于双尺度注意力的元学习的个性化动态音乐情感识别
链接：https://arxiv.org/abs/2412.19200
备注：Accepted by the 39th AAAI Conference on Artificial Intelligence (AAAI-25)
摘要：动态音乐情感识别（DMER）旨在预测音乐中不同时刻的情感，在音乐信息检索中起着至关重要的作用。现有的DMER方法在处理序列数据时难以捕获长期依赖性，这限制了它们的性能。此外，这些方法往往忽略了个体差异对情绪感知的影响，即使每个人都有自己的个性化情绪感知在现实世界中。出于这些问题的动机，我们探索更有效的序列处理方法，并引入个性化DMER（PDMER）问题，这需要模型来预测与个性化感知一致的情绪。具体来说，我们提出了一种基于双尺度注意力的元学习（DSAML）方法。该方法融合了双尺度特征提取器的特征，并使用双尺度注意力Transformer捕获短期和长期依赖关系，提高了传统DMER的性能。为了实现PDMER，我们设计了一种新的任务构造策略，划分任务的注释。任务中的样本由相同的注释器注释，确保一致的感知。利用这种策略和元学习，DSAML可以预测个性化的情绪感知，只需一个个性化的注释样本。我们的客观和主观实验表明，我们的方法可以实现国家的最先进的性能在传统的DMER和PDMER。
摘要：Dynamic Music Emotion Recognition (DMER) aims to predict the emotion ofdifferent moments in music, playing a crucial role in music informationretrieval. The existing DMER methods struggle to capture long-term dependencieswhen dealing with sequence data, which limits their performance. Furthermore,these methods often overlook the influence of individual differences on emotionperception, even though everyone has their own personalized emotionalperception in the real world. Motivated by these issues, we explore moreeffective sequence processing methods and introduce the Personalized DMER(PDMER) problem, which requires models to predict emotions that align withpersonalized perception. Specifically, we propose a Dual-Scale Attention-BasedMeta-Learning (DSAML) method. This method fuses features from a dual-scalefeature extractor and captures both short and long-term dependencies using adual-scale attention transformer, improving the performance in traditionalDMER. To achieve PDMER, we design a novel task construction strategy thatdivides tasks by annotators. Samples in a task are annotated by the sameannotator, ensuring consistent perception. Leveraging this strategy alongsidemeta-learning, DSAML can predict personalized perception of emotions with justone personalized annotation sample. Our objective and subjective experimentsdemonstrate that our method can achieve state-of-the-art performance in bothtraditional DMER and PDMER.
标题： 凝聚舞者：通过音乐驱动的凝聚力分解增强互动群舞生成
链接：https://arxiv.org/abs/2412.19123
摘要：舞蹈生成是至关重要和具有挑战性的，特别是在舞蹈表演和虚拟游戏等领域。在目前的文献中，大多数方法论都集中在独奏音乐2Dance上。虽然有针对集体音乐舞蹈的努力，但这些努力往往缺乏连贯性，导致舞蹈表演美学上的缺陷。因此，我们介绍Cohedancers，一个新的框架，音乐驱动的互动群舞生成。Cohedancers旨在通过将其分解为三个关键方面来增强集体舞的连贯性：同步性，自然性和流动性。相应地，我们开发了基于周期一致性的舞蹈同步策略来促进音乐与舞蹈的对应，基于自回归的暴露偏差纠正策略来增强生成舞蹈的流动性，以及对抗性训练策略来增强自然性团体舞输出。总的来说，这些策略使CohdeDancers能够产生高质量的高度连贯的集体舞蹈。此外，为了为Group Music 2Dance建立更好的基准，我们构建了迄今为止最多元化、最全面的开源数据集I-Dancers，具有丰富的舞者互动功能，并创建了全面的评估指标。对I-Dancers和其他现存数据集的实验评估证实了CoheDancers实现了前所未有的最先进性能。将发布代码。
摘要：Dance generation is crucial and challenging, particularly in domains likedance performance and virtual gaming. In the current body of literature, mostmethodologies focus on Solo Music2Dance. While there are efforts directedtowards Group Music2Dance, these often suffer from a lack of coherence,resulting in aesthetically poor dance performances. Thus, we introduceCoheDancers, a novel framework for Music-Driven Interactive Group DanceGeneration. CoheDancers aims to enhance group dance generation coherence bydecomposing it into three key aspects: synchronization, naturalness, andfluidity. Correspondingly, we develop a Cycle Consistency based DanceSynchronization strategy to foster music-dance correspondences, anAuto-Regressive-based Exposure Bias Correction strategy to enhance the fluidityof the generated dances, and an Adversarial Training Strategy to augment thenaturalness of the group dance output. Collectively, these strategies enableCohdeDancers to produce highly coherent group dances with superior quality.Furthermore, to establish better benchmarks for Group Music2Dance, we constructthe most diverse and comprehensive open-source dataset to date, I-Dancers,featuring rich dancer interactions, and create comprehensive evaluationmetrics. Experimental evaluations on I-Dancers and other extant datasetssubstantiate that CoheDancers achieves unprecedented state-of-the-artperformance. Code will be released.
标题： BSDB-Net：带分裂双分支网络，具有选择性状态空间机制，用于单耳语音增强
链接：https://arxiv.org/abs/2412.19099
备注：Accepted by AAAI 2025
摘要：虽然基于复谱的语音增强（SE）方法已经取得了显著的性能，但是幅度和相位的耦合会导致补偿效应，其中牺牲幅度信息来补偿对SE有害的相位。此外，为了进一步提高SE的性能，许多模块堆叠在SE上，导致模型复杂度增加，限制了SE的应用。为了解决这些问题，提出了一种基于Mamba压缩频率的双路径网络。首先，通过并行双分支提取幅度和相位信息。该方法利用结构化复频谱隐式地捕获相位信息，并通过解耦幅度和相位来解决补偿效应，网络中加入交互模块来抑制不必要的部分并从另一分支恢复丢失的分量。其次，为了降低网络复杂度，该网络引入了频带分割策略来压缩频率维。为了在保持良好性能的同时进一步降低复杂度，我们设计了一个基于Mamba的模块，该模块在线性复杂度下对时间和频率维度进行建模。最后，与基准相比，我们的模型在保持卓越性能的同时，平均将计算复杂度降低了8.3倍。此外，与基于变换器的模型相比，它实现了25倍的复杂度降低。
摘要：Although the complex spectrum-based speech enhancement(SE) methods haveachieved significant performance, coupling amplitude and phase can lead to acompensation effect, where amplitude information is sacrificed to compensatefor the phase that is harmful to SE. In addition, to further improve theperformance of SE, many modules are stacked onto SE, resulting in increasedmodel complexity that limits the application of SE. To address these problems,we proposed a dual-path network based on compressed frequency using Mamba.First, we extract amplitude and phase information through parallel dualbranches. This approach leverages structured complex spectra to implicitlycapture phase information and solves the compensation effect by decouplingamplitude and phase, and the network incorporates an interaction module tosuppress unnecessary parts and recover missing components from the otherbranch. Second, to reduce network complexity, the network introduces aband-split strategy to compress the frequency dimension. To further reducecomplexity while maintaining good performance, we designed a Mamba-based modulethat models the time and frequency dimensions under linear complexity. Finally,compared to baselines, our model achieves an average 8.3 times reduction incomputational complexity while maintaining superior performance. Furthermore,it achieves a 25 times reduction in complexity compared to transformer-basedmodels.
标题： 利用多语言STEN-TTC和Bert LID的印度尼西亚-英语代码转换语音合成器
链接：https://arxiv.org/abs/2412.19043
备注：Accepted at O-COCOSDA 2024
摘要：多语言文本到语音系统跨多种语言将文本转换为语音。在许多情况下，文本句子可能包含不同语言的片段，这种现象称为语码转换。这在印度尼西亚尤其常见，尤其是在印度尼西亚语和英语之间。尽管它的意义，还没有研究开发出一个多语种的TTS系统能够处理这两种语言之间的代码切换。本研究旨在探讨STEN-TTS中的汉英语码转换现象。关键的修改包括使用微调的BERT进行每个单词的语言识别，以及从基本模型中删除语言嵌入，将语言识别组件添加到文本到音素转换中。实验结果表明，代码转换模型实现了优越的自然度和改善的语音清晰度相比，印尼和英语基线STEN-TTS模型。
摘要：Multilingual text-to-speech systems convert text into speech across multiplelanguages. In many cases, text sentences may contain segments in differentlanguages, a phenomenon known as code-switching. This is particularly common inIndonesia, especially between Indonesian and English. Despite its significance,no research has yet developed a multilingual TTS system capable of handlingcode-switching between these two languages. This study addressesIndonesian-English code-switching in STEN-TTS. Key modifications include addinga language identification component to the text-to-phoneme conversion usingfinetuned BERT for per-word language identification, as well as removinglanguage embedding from the base model. Experimental results demonstrate thatthe code-switching model achieves superior naturalness and improved speechintelligibility compared to the Indonesian and English baseline STEN-TTSmodels.
标题： Leave-One-Equival：减轻对比音乐表示中与不变性相关的信息损失
链接：https://arxiv.org/abs/2412.18955
摘要：对比学习在自我监督的音乐表征学习中被证明是有效的，特别是对于音乐信息检索（MIR）任务。然而，依赖于增强链的对比视图生成和由此产生的学习不变性提出了挑战时，不同的下游任务需要对某些音乐属性的敏感性。为了解决这个问题，我们提出了Leave One Equivariant（LOEV）框架，与以前的工作相比，该框架通过选择性地保留有关特定增强的信息，引入了一种灵活的任务自适应方法，允许模型保持任务相关的等方差。我们证明了LOEV消除了与学习不变性相关的信息损失，提高了增强相关任务和检索的性能，而不会牺牲一般表示质量。此外，我们引入了LOEV的一个变体LOEV++，它通过设计以自监督的方式构建了一个解纠缠的潜在空间，并基于增强相关属性实现了有针对性的检索。
摘要：Contrastive learning has proven effective in self-supervised musicalrepresentation learning, particularly for Music Information Retrieval (MIR)tasks. However, reliance on augmentation chains for contrastive view generationand the resulting learnt invariances pose challenges when different downstreamtasks require sensitivity to certain musical attributes. To address this, wepropose the Leave One EquiVariant (LOEV) framework, which introduces aflexible, task-adaptive approach compared to previous work by selectivelypreserving information about specific augmentations, allowing the model tomaintain task-relevant equivariances. We demonstrate that LOEV alleviatesinformation loss related to learned invariances, improving performance onaugmentation related tasks and retrieval without sacrificing generalrepresentation quality. Furthermore, we introduce a variant of LOEV, LOEV++,which builds a disentangled latent space by design in a self-supervised manner,and enables targeted retrieval based on augmentation related attributes.
标题： 稳健的目标说话人到达方向估计
链接：https://arxiv.org/abs/2412.18913
摘要：在多说话人环境中，目标说话人的波达方向（DOA）是提高语音清晰度和提取目标说话人语音的关键。然而，传统的DOA估计方法往往在噪声、混响的存在下，特别是当存在竞争扬声器时，会遇到困难。为了解决这些挑战，我们提出了RTS-DOA，一个强大的实时DOA估计系统。该系统创新性地使用目标说话人的注册语音作为参考，并利用来自麦克风阵列的全频带和子频带频谱信息来估计目标说话人的语音的DOA。具体地，该系统包括用于初始改善语音质量的语音增强模块、用于学习空间信息的空间模块和用于提取声纹特征的说话者模块。LibriSpeech数据集上的实验结果表明，我们的RTS-DOA系统有效地处理多说话人的情况下，建立了新的最佳基准。
摘要：In multi-speaker environments the direction of arrival (DOA) of a targetspeaker is key for improving speech clarity and extracting target speaker'svoice. However, traditional DOA estimation methods often struggle in thepresence of noise, reverberation, and particularly when competing speakers arepresent. To address these challenges, we propose RTS-DOA, a robust real-timeDOA estimation system. This system innovatively uses the registered speech ofthe target speaker as a reference and leverages full-band and sub-band spectralinformation from a microphone array to estimate the DOA of the target speaker'svoice. Specifically, the system comprises a speech enhancement module forinitially improving speech quality, a spatial module for learning spatialinformation, and a speaker module for extracting voiceprint features.Experimental results on the LibriSpeech dataset demonstrate that our RTS-DOAsystem effectively tackles multi-speaker scenarios and established new optimalbenchmarks.
标题： 用于声学回声消除的注意力增强型短期维纳解决方案
链接：https://arxiv.org/abs/2412.18851
摘要：声学回声消除（AEC）是一种重要的语音信号处理技术，它可以消除麦克风输入的回声，以促进自然发声的全双工通信。目前，基于深度学习的AEC方法主要集中在细化模型架构，经常忽略传统过滤器理论的知识。本文提出了一种创新的方法，AEC通过引入注意力增强的短时维纳解决方案。我们的方法策略性地利用注意力机制来减轻双端谈话干扰的影响，从而优化知识利用的效率。短期维纳解决方案，它适应经典维纳解决方案的有限输入因果关系的推导，集成了从滤波器理论到这种方法建立的见解。实验结果证实了我们提出的方法的有效性，超过其他基线模型的性能和泛化。官方代码可在https://github.com/ZhaoF-i/ASTWS-AEC上获得
摘要：Acoustic Echo Cancellation (AEC) is an essential speech signal processingtechnology that removes echoes from microphone inputs to facilitatenatural-sounding full-duplex communication. Currently, deep learning-based AECmethods primarily focus on refining model architectures, frequently neglectingthe incorporation of knowledge from traditional filter theory. This paperpresents an innovative approach to AEC by introducing an attention-enhancedshort-time Wiener solution. Our method strategically harnesses attentionmechanisms to mitigate the impact of double-talk interference, therebyoptimizing the efficiency of knowledge utilization. The derivation of theshort-term Wiener solution, which adapts classical Wiener solutions to finiteinput causality, integrates established insights from filter theory into thismethod. The experimental outcomes corroborate the effectiveness of our proposedapproach, surpassing other baseline models in performance and generalization.The official code is available at https://github.com/ZhaoF-i/ASTWS-AEC
标题： 利用新型方法和MultiNAM数据集推进NAM到语音的转换
链接：https://arxiv.org/abs/2412.18839
备注：Accepted at IEEE ICASSP 2025
摘要：目前的非可听杂音（NAM）到语音技术依赖于语音克隆来模拟成对耳语的地面真实语音。然而，模拟的语音往往缺乏可懂度，并且不能很好地概括不同的说话者。为了解决这个问题，我们专注于从成对的耳语和文本中学习音素级对齐，并采用文本到语音（TTS）系统来模拟地面实况。为了减少对耳语的依赖，我们直接从NAM学习音素对齐，尽管质量受到可用训练数据的限制。为了进一步减轻对NAM/耳语数据的依赖，我们建议结合嘴唇模态来推断语音，并引入一种新的基于扩散的方法，该方法利用了嘴唇到语音技术的最新进展。此外，我们发布了MultiNAM数据集，其中包含来自两个扬声器的配对NAM，耳语，视频和文本数据，并在此数据集上对所有方法进行基准测试。语音样本和数据集可在\url{https：//diff-nam.github.io/DiffNAM/}
摘要：Current Non-Audible Murmur (NAM)-to-speech techniques rely on voice cloningto simulate ground-truth speech from paired whispers. However, the simulatedspeech often lacks intelligibility and fails to generalize well acrossdifferent speakers. To address this issue, we focus on learning phoneme-levelalignments from paired whispers and text and employ a Text-to-Speech (TTS)system to simulate the ground-truth. To reduce dependence on whispers, we learnphoneme alignments directly from NAMs, though the quality is constrained by theavailable training data. To further mitigate reliance on NAM/whisper data forground-truth simulation, we propose incorporating the lip modality to inferspeech and introduce a novel diffusion-based method that leverages recentadvancements in lip-to-speech technology. Additionally, we release the MultiNAMdataset with over $7.96$ hours of paired NAM, whisper, video, and text datafrom two speakers and benchmark all methods on this dataset. Speech samples andthe dataset are available at \url{https://diff-nam.github.io/DiffNAM/}
标题： MRI 2 Speech：实时MRI记录的关节运动的语音合成
链接：https://arxiv.org/abs/2412.18836
备注：Accepted at IEEE ICASSP 2025
摘要：以前的实时MRI（rtMRI）为基础的语音合成模型严重依赖于嘈杂的地面实况语音。直接在地面真实梅尔频谱图上应用损失会使语音内容与MRI噪声纠缠在一起，导致可懂度差。我们引入了一种新的方法，该方法采用多模态自监督AV-HuBERT模型进行rtMRI文本预测，并采用了一种新的基于流的持续时间预测器进行特定于说话人的对齐。然后，语音解码器使用预测的文本和持续时间来合成任何新语音中的对齐语音。我们在两个数据集上进行了彻底的实验，并证明了我们的方法对看不见的说话者的泛化能力。我们通过掩蔽rtMRI视频的一部分来评估我们框架的性能，以评估不同发音器对文本预测的影响。我们的方法在USC-TIMIT MRI语料库上实现了15.18\%$的字错误率（WER），这标志着对当前最先进技术的巨大改进。语音样本可在\url{https：//mri2speech.github.io/MRI2Speech/}获得
摘要：Previous real-time MRI (rtMRI)-based speech synthesis models depend heavilyon noisy ground-truth speech. Applying loss directly over ground truthmel-spectrograms entangles speech content with MRI noise, resulting in poorintelligibility. We introduce a novel approach that adapts the multi-modalself-supervised AV-HuBERT model for text prediction from rtMRI and incorporatesa new flow-based duration predictor for speaker-specific alignment. Thepredicted text and durations are then used by a speech decoder to synthesizealigned speech in any novel voice. We conduct thorough experiments on twodatasets and demonstrate our method's generalization ability to unseenspeakers. We assess our framework's performance by masking parts of the rtMRIvideo to evaluate the impact of different articulators on text prediction. Ourmethod achieves a $15.18\%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus,marking a huge improvement over the current state-of-the-art. Speech samplesare available at \url{https://mri2speech.github.io/MRI2Speech/}
标题： 通过多尺度多模式上下文交互实现具有表达力的视频配音
链接：https://arxiv.org/abs/2412.18748
备注：Accepted by ICSSP 2025
摘要：自动视频配音（AVD）从脚本中生成与嘴唇运动和面部情绪一致的语音。近年来的研究主要集中在对多模态语境进行建模以增强韵律表达能力，但忽略了两个关键问题：1）语境中的多尺度韵律表达属性对当前句子的韵律产生影响。2)上下文中的韵律线索与当前句子相互作用，影响最终的韵律表达力。为了应对这些挑战，我们提出了M2 CI-Dubber，一个多尺度多模态上下文交互方案AVD。该方案包括两个共享的M2 CI编码器，用于对多尺度多模态上下文进行建模，并促进其与当前句子的深度交互。该方法通过提取上下文中每种模态的全局和局部特征，利用基于注意力的聚合和交互机制，采用基于交互的图注意力网络进行融合，增强了当前句子合成语音的韵律表达能力。在Chem数据集上的实验表明，我们的模型在配音表现力方面优于基线。代码和演示可在\textcolor[rgb]{0.93，0.0，0.47}{https：//github.com/AI-S2-Lab/M2CI-Dubber}获得。
摘要：Automatic Video Dubbing (AVD) generates speech aligned with lip motion andfacial emotion from scripts. Recent research focuses on modeling multimodalcontext to enhance prosody expressiveness but overlooks two key issues: 1)Multiscale prosody expression attributes in the context influence the currentsentence's prosody. 2) Prosody cues in context interact with the currentsentence, impacting the final prosody expressiveness. To tackle thesechallenges, we propose M2CI-Dubber, a Multiscale Multimodal Context Interactionscheme for AVD. This scheme includes two shared M2CI encoders to model themultiscale multimodal context and facilitate its deep interaction with thecurrent sentence. By extracting global and local features for each modality inthe context, utilizing attention-based mechanisms for aggregation andinteraction, and employing an interaction-based graph attention network forfusion, the proposed approach enhances the prosody expressiveness ofsynthesized speech for the current sentence. Experiments on the Chem datasetshow our model outperforms baselines in dubbing expressiveness. The code anddemos are available at\textcolor[rgb]{0.93,0.0,0.47}{https://github.com/AI-S2-Lab/M2CI-Dubber}.
标题： 用于对话语音合成的模式内和模式间上下文交互建模
链接：https://arxiv.org/abs/2412.18733
备注：Accepted by ICASSP 2025
摘要：会话语音合成（CSS）旨在有效地利用多模态对话历史（MDH）生成具有适当会话韵律的语音。CSS的关键挑战是对MDH和目标话语之间的交互进行建模。MDH中的语篇模态和言语模态各有其独特的影响，它们相辅相成，对目标语产生综合影响。以前的作品没有明确的模型，这样的模态内和模态间的相互作用。为了解决这个问题，我们提出了一个新的模态内和模态间的上下文交互模式为基础的CSS系统，称为III-CSS。具体而言，在训练阶段，我们将MDH与目标话语中的文本和语音模态相结合，以获得四种模态组合，包括历史文本-下一文本，历史语音-下一语音，历史文本-下一语音和历史语音-下一文本。然后，我们设计了两个基于对比学习的模态内和两个模态间交互模块，以深入学习模态内和模态间的上下文交互。在推理阶段，我们采取MDH和采用训练的交互模块，充分推断目标话语的文本内容的语音韵律。在DailyTalk数据集上进行的主观和客观实验表明，III-CSS在韵律表现力方面优于高级基线。代码和语音示例可在https://github.com/AI-S2-Lab/I3CSS上获得。
摘要：Conversational Speech Synthesis (CSS) aims to effectively take the multimodaldialogue history (MDH) to generate speech with appropriate conversationalprosody for target utterance. The key challenge of CSS is to model theinteraction between the MDH and the target utterance. Note that text and speechmodalities in MDH have their own unique influences, and they complement eachother to produce a comprehensive impact on the target utterance. Previous worksdid not explicitly model such intra-modal and inter-modal interactions. Toaddress this issue, we propose a new intra-modal and inter-modal contextinteraction scheme-based CSS system, termed III-CSS. Specifically, in thetraining phase, we combine the MDH with the text and speech modalities in thetarget utterance to obtain four modal combinations, including HistoricalText-Next Text, Historical Speech-Next Speech, Historical Text-Next Speech, andHistorical Speech-Next Text. Then, we design two contrastive learning-basedintra-modal and two inter-modal interaction modules to deeply learn theintra-modal and inter-modal context interaction. In the inference phase, wetake MDH and adopt trained interaction modules to fully infer the speechprosody of the target utterance's text content. Subjective and objectiveexperiments on the DailyTalk dataset show that III-CSS outperforms the advancedbaselines in terms of prosody expressiveness. Code and speech samples areavailable at https://github.com/AI-S2-Lab/I3CSS.
标题： Simi-SFX：一种基于相似度的可控音效合成方法
链接：https://arxiv.org/abs/2412.18710
摘要：生成具有可控变化的声音效果是一项具有挑战性的任务，传统上使用需要深入了解信号处理参数和算法的复杂物理模型来解决。在生成和大型语言模型的时代，文本已经成为控制声音合成的常见的、人类可解释的界面。然而，语言标记的离散和定性性质使得难以捕捉不同声音之间的细微音色变化。在这项研究中，我们提出了一种新型的基于相似性的声音合成调节方法，利用可微分数字信号处理（DDSP）。该方法结合使用潜在空间学习和控制音频音色与直观的指导向量，在范围[0，1]内归一化，以编码分类声学信息。通过利用预先训练的音频表示模型，我们的方法实现了表达和细粒度的音色控制。为了对我们的方法进行基准测试，我们引入了两个声音效果数据集-足迹集和影响集-旨在评估可控性和声音质量。回归分析表明，建议的相似性得分有效地控制音色的变化，并使创造性的应用，如离散类之间的音色插值。我们的工作为音效合成提供了一个强大而通用的框架，弥合了传统信号处理和现代机器学习技术之间的差距。
摘要：Generating sound effects with controllable variations is a challenging task,traditionally addressed using sophisticated physical models that requirein-depth knowledge of signal processing parameters and algorithms. In the eraof generative and large language models, text has emerged as a common,human-interpretable interface for controlling sound synthesis. However, thediscrete and qualitative nature of language tokens makes it difficult tocapture subtle timbral variations across different sounds. In this research, wepropose a novel similarity-based conditioning method for sound synthesis,leveraging differentiable digital signal processing (DDSP). This approachcombines the use of latent space for learning and controlling audio timbre withan intuitive guiding vector, normalized within the range [0,1], to encodecategorical acoustic information. By utilizing pre-trained audio representationmodels, our method achieves expressive and fine-grained timbre control. Tobenchmark our approach, we introduce two sound effect datasets--Footstep-setand Impact-set--designed to evaluate both controllability and sound quality.Regression analysis demonstrates that the proposed similarity score effectivelycontrols timbre variations and enables creative applications such as timbreinterpolation between discrete classes. Our work provides a robust andversatile framework for sound effect synthesis, bridging the gap betweentraditional signal processing and modern machine learning techniques.
标题： 基于元学习的无延迟子带自适应过滤器利用复自注意力进行主动噪音控制
链接：https://arxiv.org/abs/2412.19471
备注：31 pages, 8 figures
摘要：有源噪声控制通常采用自适应滤波来产生次级噪声，其中最小均方算法是最广泛使用的。然而，传统的更新规则是线性的，在解决非线性环境和非平稳噪声表现出有限的有效性。为了应对这一挑战，我们将有源噪声控制问题重新表述为元学习问题，并提出了一种基于元学习的无延迟子带自适应滤波器和深度神经网络。其核心思想是利用神经网络作为自适应算法，可以适应不同的环境和噪声类型。神经网络将在噪声观测下进行训练，这意味着它可以在没有真实标签的情况下识别优化的更新规则。设计了一种单头注意力递归神经网络，该网络具有可学习的特征嵌入，可以有效地更新自适应滤波器的权重，从而能够精确计算次级源以衰减不需要的初级噪声。为了放松对更新自适应滤波器权重的时间约束，采用无延迟子带架构，这将允许系统随着下采样因子的增加而不太频繁地更新。此外，无延迟子带架构不会在有源噪声控制系统中引入额外的时间延迟。引入跳过更新策略，进一步降低更新频率，使资源有限的机器有更多的可能性登上我们的基于元学习的模型。广泛的多条件训练确保了对各种类型的噪声和环境的泛化和鲁棒性。仿真结果表明，与传统方法相比，我们的基于元学习的模型具有更好的降噪性能。
摘要：Active noise control typically employs adaptive filtering to generatesecondary noise, where the least mean square algorithm is the most widely used.However, traditional updating rules are linear and exhibit limitedeffectiveness in addressing nonlinear environments and nonstationary noise. Totackle this challenge, we reformulate the active noise control problem as ameta-learning problem and propose a meta-learning-based delayless subbandadaptive filter with deep neural networks. The core idea is to utilize a neuralnetwork as an adaptive algorithm that can adapt to different environments andtypes of noise. The neural network will train under noisy observations,implying that it recognizes the optimized updating rule without true labels. Asingle-headed attention recurrent neural network is devised with learnablefeature embedding to update the adaptive filter weight efficiently, enablingaccurate computation of the secondary source to attenuate the unwanted primarynoise. In order to relax the time constraint on updating the adaptive filterweights, the delayless subband architecture is employed, which will allow thesystem to be updated less frequently as the downsampling factor increases. Inaddition, the delayless subband architecture does not introduce additional timedelays in active noise control systems. A skip updating strategy is introducedto decrease the updating frequency further so that machines with limitedresources have more possibility to board our meta-learning-based model.Extensive multi-condition training ensures generalization and robustnessagainst various types of noise and environments. Simulation results demonstratethat our meta-learning-based model achieves superior noise reductionperformance compared to traditional methods.
标题： SecureDiT：用于环境感知语音合成的双条件扩散Transformer
链接：https://arxiv.org/abs/2412.19259
备注：Accepted to ICASSP 2025
摘要：我们提出了VoiceDiT，一个多模态生成模型，用于从文本和视觉提示中产生环境感知的语音和音频。虽然将语音与文本对齐对于可理解的语音至关重要，但在嘈杂条件下实现这种对齐仍然是该领域的一个重大挑战。为了解决这个问题，我们提出了一种新的音频生成流水线命名为VoiceDiT。该管道包括三个关键组成部分：（1）创建用于预训练的大规模合成语音数据集和用于微调的精炼的真实世界语音数据集，（2）Dual-DiT，一种被设计为有效地保留对齐的语音信息同时准确地反映环境条件的模型，以及（3）基于扩散的图像到音频转换器，其允许模型桥接音频与图像之间的间隙，促进与多模态提示对准的环境声音的产生。大量的实验结果表明，VoiceDiT在现实世界的数据集上优于以前的模型，在音频质量和模态集成方面都有显着的改进。
摘要：We present VoiceDiT, a multi-modal generative model for producingenvironment-aware speech and audio from text and visual prompts. While aligningspeech with text is crucial for intelligible speech, achieving this alignmentin noisy conditions remains a significant and underexplored challenge in thefield. To address this, we present a novel audio generation pipeline namedVoiceDiT. This pipeline includes three key components: (1) the creation of alarge-scale synthetic speech dataset for pre-training and a refined real-worldspeech dataset for fine-tuning, (2) the Dual-DiT, a model designed toefficiently preserve aligned speech information while accurately reflectingenvironmental conditions, and (3) a diffusion-based Image-to-Audio Translatorthat allows the model to bridge the gap between audio and image, facilitatingthe generation of environmental sound that aligns with the multi-modal prompts.Extensive experimental results demonstrate that VoiceDiT outperforms previousmodels on real-world datasets, showcasing significant improvements in bothaudio quality and modality integration.
标题： 基于量化自我监督学习特征的预测语义的因果语音增强
链接：https://arxiv.org/abs/2412.19248
备注：Accepted for ICASSP 2025, 5 pages
摘要：实时语音增强是在线语音通信的关键。因果SE模型只使用以前的上下文，而预测未来的信息，如音素延续，可能有助于执行因果SE。语音信息通常通过量化自监督学习（SSL）模型的潜在特征来表示。这项工作是第一个将SSL功能与因果关系纳入SE模型。因果SSL特征被编码并使用特征线性调制与频谱图特征组合以估计用于增强噪声输入语音的掩模。同时，我们使用矢量量化来表示语音特征作为语义令牌的因果SSL功能。该模型不仅对SSL特征进行编码，而且可以预测多任务学习（MTL）中的未来语义标记。使用VoiceBank + DEMAND数据集的实验结果表明，我们提出的方法在PESQ中达到2.88，特别是语义预测MTL，其中我们证实了语义预测在因果SE中发挥了重要作用。
摘要：Real-time speech enhancement (SE) is essential to online speechcommunication. Causal SE models use only the previous context while predictingfuture information, such as phoneme continuation, may help performing causalSE. The phonetic information is often represented by quantizing latent featuresof self-supervised learning (SSL) models. This work is the first to incorporateSSL features with causality into an SE model. The causal SSL features areencoded and combined with spectrogram features using feature-wise linearmodulation to estimate a mask for enhancing the noisy input speech.Simultaneously, we quantize the causal SSL features using vector quantizationto represent phonetic characteristics as semantic tokens. The model not onlyencodes SSL features but also predicts the future semantic tokens in multi-tasklearning (MTL). The experimental results using VoiceBank + DEMAND dataset showthat our proposed method achieves 2.88 in PESQ, especially with semanticprediction MTL, in which we confirm that the semantic prediction played animportant role in causal SE.
标题： 攻击具有增强特征和说话者身份差异的语音合成系统
链接：https://arxiv.org/abs/2412.19068
备注：2 pages, submitted to ICASSP 2025 GC-7: The First VoicePrivacy Attacker Challenge (by invitation)
摘要：本研究的重点是ICASSP 2025信号处理大挑战赛中的第一个语音隐私攻击者挑战赛，该挑战赛旨在开发能够确定两个匿名语音信号是否来自同一说话人的说话人验证系统。然而，原始和匿名语音的特征分布之间的差异使这项任务复杂化。为了解决这一挑战，我们提出了一个攻击系统，结合数据增强增强的特征表示和扬声器身份差异增强分类器，以提高验证性能，称为DA-SID。具体而言，数据增强策略（即，数据融合和SpecAugment）来减小特征分布差距，而概率线性判别分析（PLDA）用于进一步增强说话人身份差异。我们的系统显著优于基线，在各种语音匿名系统中表现出卓越的有效性和鲁棒性，最终确保了挑战的前5名。
摘要：This study focuses on the First VoicePrivacy Attacker Challenge within theICASSP 2025 Signal Processing Grand Challenge, which aims to develop speakerverification systems capable of determining whether two anonymized speechsignals are from the same speaker. However, differences between featuredistributions of original and anonymized speech complicate this task. Toaddress this challenge, we propose an attacker system that combines DataAugmentation enhanced feature representation and Speaker Identity Differenceenhanced classifier to improve verification performance, termed DA-SID.Specifically, data augmentation strategies (i.e., data fusion and SpecAugment)are utilized to mitigate feature distribution gaps, while probabilistic lineardiscriminant analysis (PLDA) is employed to further enhance speaker identitydifference. Our system significantly outperforms the baseline, demonstratingexceptional effectiveness and robustness against various voice anonymizationsystems, ultimately securing a top-5 ranking in the challenge.
标题： 用于发音障碍和老年人语音识别的基础模型的结构化说话人缺陷适应
链接：https://arxiv.org/abs/2412.18832
摘要：语音基础模型（SFM）的数据密集型微调，以稀缺和多样的构音障碍和老年人的讲话导致数据偏见和穷人的泛化看不见的扬声器。本文提出了新的结构化的说话人不足的适应方法，SSL预训练的SFM等数据。在监督自适应微调阶段构建了说话人和语音缺陷不变的SFM，以减少对训练数据说话人的不适当偏见，并作为测试时间无监督自适应的更中立和鲁棒的起点。语音变异归因于扬声器的身份和语音障碍的严重程度，或老化引起的神经认知下降，建模使用单独的适配器，可以结合在一起，以模拟任何可见或不可见的扬声器。在UASpeech构音障碍和DementiaBank Pitt老年人语音语料库上的实验表明，HuBERT和Wav 2 vec 2-conformer模型的结构化说话人缺陷适应始终优于使用以下任一种的基线SFM：a）没有适配器; b）所有说话人共享全局适配器;或c）单属性适配器单独建模扬声器或缺陷标签，统计上显著的WER减少高达3.01%和1.50%绝对值（10.86%和6.94%）。最低公布的WER为19.45%（49.34%的非常低的可懂度，33.17%的看不见的话）是在UASpeech测试集的16个构音障碍的扬声器。
摘要：Data-intensive fine-tuning of speech foundation models (SFMs) to scarce anddiverse dysarthric and elderly speech leads to data bias and poorgeneralization to unseen speakers. This paper proposes novel structuredspeaker-deficiency adaptation approaches for SSL pre-trained SFMs on such data.Speaker and speech deficiency invariant SFMs were constructed in theirsupervised adaptive fine-tuning stage to reduce undue bias to training dataspeakers, and serves as a more neutral and robust starting point for test timeunsupervised adaptation. Speech variability attributed to speaker identity andspeech impairment severity, or aging induced neurocognitive decline, aremodelled using separate adapters that can be combined together to model anyseen or unseen speaker. Experiments on the UASpeech dysarthric and DementiaBankPitt elderly speech corpora suggest structured speaker-deficiency adaptation ofHuBERT and Wav2vec2-conformer models consistently outperforms baseline SFMsusing either: a) no adapters; b) global adapters shared among all speakers; orc) single attribute adapters modelling speaker or deficiency labels alone bystatistically significant WER reductions up to 3.01% and 1.50% absolute (10.86%and 6.94% relative) on the two tasks respectively. The lowest published WER of19.45% (49.34% on very low intelligibility, 33.17% on unseen words) is obtainedon the UASpeech test set of 16 dysarthric speakers.
标题： 埃塞俄比亚东正教Tewahedo教堂圣歌中Yaredawi YeZema淤泥的计算分析
链接：https://arxiv.org/abs/2412.18788
备注：6 pages
摘要：尽管其音乐学，文化和宗教意义，埃塞俄比亚东正教Tewahedo教会（EOTC）圣歌在音乐研究中的代表性相对不足。历史记录，包括手稿、研究论文和口头传统，证实了圣雅列在6世纪建立了三种典型的EOTC念诵模式。本文试图利用音乐信息检索技术对EOTC圣歌进行研究。在有关EOTC圣歌分析和理解的研究问题中，Yaredawi YeZema Silt，即遵循圣亚里德标准的圣歌模式，是至关重要的。因此，我们认为在EOTC圣歌Yaredawi YeZema淤泥分类的任务，通过引入一个新的数据集，并展示了一系列的分类实验，这项任务。结果表明，使用稳定的音高轮廓的分布作为一个简单的基于神经网络的分类器的特征表示成为一个有效的解决方案。本研究将借由与先前关于EOTC圣歌的民族音乐学文献的比较研究，进一步探讨这些结果的音乐学意涵与洞见。通过公开这一数据集，我们的目标是促进EOTC圣歌的未来探索和分析，并突出进一步研究的潜在方向，从而促进对这一独特精神和文化遗产的更深入理解和保护。
摘要：Despite its musicological, cultural, and religious significance, theEthiopian Orthodox Tewahedo Church (EOTC) chant is relatively underrepresentedin music research. Historical records, including manuscripts, research papers,and oral traditions, confirm Saint Yared's establishment of three canonicalEOTC chanting modes during the 6th century. This paper attempts to investigatethe EOTC chants using music information retrieval (MIR) techniques. Among theresearch questions regarding the analysis and understanding of EOTC chants,Yaredawi YeZema Silt, namely the mode of chanting adhering to Saint Yared'sstandards, is of primary importance. Therefore, we consider the task ofYaredawi YeZema Silt classification in EOTC chants by introducing a new datasetand showcasing a series of classification experiments for this task. Resultsshow that using the distribution of stabilized pitch contours as the featurerepresentation on a simple neural network-based classifier becomes an effectivesolution. The musicological implications and insights of such results arefurther discussed through a comparative study with the previous ethnomusicologyliterature on EOTC chants. By making this dataset publicly accessible, we aimto promote future exploration and analysis of EOTC chants and highlightpotential directions for further research, thereby fostering a deeperunderstanding and preservation of this unique spiritual and cultural heritage.
标题： 基于元学习的无延迟子带自适应过滤器利用复自注意力进行主动噪音控制
链接：https://arxiv.org/abs/2412.19471
备注：31 pages, 8 figures
摘要：有源噪声控制通常采用自适应滤波来产生次级噪声，其中最小均方算法是最广泛使用的。然而，传统的更新规则是线性的，在解决非线性环境和非平稳噪声表现出有限的有效性。为了应对这一挑战，我们将有源噪声控制问题重新表述为元学习问题，并提出了一种基于元学习的无延迟子带自适应滤波器和深度神经网络。其核心思想是利用神经网络作为自适应算法，可以适应不同的环境和噪声类型。神经网络将在噪声观测下进行训练，这意味着它可以在没有真实标签的情况下识别优化的更新规则。设计了一种单头注意力递归神经网络，该网络具有可学习的特征嵌入，可以有效地更新自适应滤波器的权重，从而能够精确计算次级源以衰减不需要的初级噪声。为了放松对更新自适应滤波器权重的时间约束，采用无延迟子带架构，这将允许系统随着下采样因子的增加而不太频繁地更新。此外，无延迟子带架构不会在有源噪声控制系统中引入额外的时间延迟。引入跳过更新策略，进一步降低更新频率，使资源有限的机器有更多的可能性登上我们的基于元学习的模型。广泛的多条件训练确保了对各种类型的噪声和环境的泛化和鲁棒性。仿真结果表明，与传统方法相比，我们的基于元学习的模型具有更好的降噪性能。
摘要：Active noise control typically employs adaptive filtering to generatesecondary noise, where the least mean square algorithm is the most widely used.However, traditional updating rules are linear and exhibit limitedeffectiveness in addressing nonlinear environments and nonstationary noise. Totackle this challenge, we reformulate the active noise control problem as ameta-learning problem and propose a meta-learning-based delayless subbandadaptive filter with deep neural networks. The core idea is to utilize a neuralnetwork as an adaptive algorithm that can adapt to different environments andtypes of noise. The neural network will train under noisy observations,implying that it recognizes the optimized updating rule without true labels. Asingle-headed attention recurrent neural network is devised with learnablefeature embedding to update the adaptive filter weight efficiently, enablingaccurate computation of the secondary source to attenuate the unwanted primarynoise. In order to relax the time constraint on updating the adaptive filterweights, the delayless subband architecture is employed, which will allow thesystem to be updated less frequently as the downsampling factor increases. Inaddition, the delayless subband architecture does not introduce additional timedelays in active noise control systems. A skip updating strategy is introducedto decrease the updating frequency further so that machines with limitedresources have more possibility to board our meta-learning-based model.Extensive multi-condition training ensures generalization and robustnessagainst various types of noise and environments. Simulation results demonstratethat our meta-learning-based model achieves superior noise reductionperformance compared to traditional methods.
标题： 建立一个推广到无序语音的单一ASB模型
链接：https://arxiv.org/abs/2412.19315
备注：Accepted at ICASSP 2025
摘要：本研究调查了将无序语音记录（1,000小时）的数据集整合到接近最先进的ASR基线系统的微调中的影响。与人们所期望的相反，尽管数据不到ASR系统训练数据的1%，但我们发现无序语音识别准确率有了相当大的提高。具体来说，我们观察到提示语音有33%的改善，而新收集的自发的，无序语音的会话数据集有26%的改善。重要的是，在标准语音识别基准测试中没有显着的性能下降。此外，我们观察到，拟议的调优策略有助于缩小基线系统和个性化模型之间的差距64%，突出了重大进展以及改进的空间。考虑到我们的研究结果的巨大好处，这个实验表明，从公平的角度来看，将一小部分高质量的无序语音数据纳入训练配方是一个简单的步骤，可以使语音技术更容易为语言障碍用户所用。
摘要：This study investigates the impact of integrating a dataset of disorderedspeech recordings ($\sim$1,000 hours) into the fine-tuning of a nearstate-of-the-art ASR baseline system. Contrary to what one might expect,despite the data being less than 1% of the training data of the ASR system, wefind a considerable improvement in disordered speech recognition accuracy.Specifically, we observe a 33% improvement on prompted speech, and a 26%improvement on a newly gathered spontaneous, conversational dataset ofdisordered speech. Importantly, there is no significant performance decline onstandard speech recognition benchmarks. Further, we observe that the proposedtuning strategy helps close the gap between the baseline system andpersonalized models by 64% highlighting the significant progress as well as theroom for improvement. Given the substantial benefits of our findings, thisexperiment suggests that from a fairness perspective, incorporating a smallfraction of high quality disordered speech data in a training recipe is an easystep that could be done to make speech technology more accessible for userswith speech disabilities.
标题： SecureDiT：用于环境感知语音合成的双条件扩散Transformer
链接：https://arxiv.org/abs/2412.19259
备注：Accepted to ICASSP 2025
摘要：我们提出了VoiceDiT，一个多模态生成模型，用于从文本和视觉提示中产生环境感知的语音和音频。虽然将语音与文本对齐对于可理解的语音至关重要，但在嘈杂条件下实现这种对齐仍然是该领域的一个重大挑战。为了解决这个问题，我们提出了一种新的音频生成流水线命名为VoiceDiT。该管道包括三个关键组成部分：（1）创建用于预训练的大规模合成语音数据集和用于微调的精炼的真实世界语音数据集，（2）Dual-DiT，一种被设计为有效地保留对齐的语音信息同时准确地反映环境条件的模型，以及（3）基于扩散的图像到音频转换器，其允许模型桥接音频与图像之间的间隙，促进与多模态提示对准的环境声音的产生。大量的实验结果表明，VoiceDiT在现实世界的数据集上优于以前的模型，在音频质量和模态集成方面都有显着的改进。
摘要：We present VoiceDiT, a multi-modal generative model for producingenvironment-aware speech and audio from text and visual prompts. While aligningspeech with text is crucial for intelligible speech, achieving this alignmentin noisy conditions remains a significant and underexplored challenge in thefield. To address this, we present a novel audio generation pipeline namedVoiceDiT. This pipeline includes three key components: (1) the creation of alarge-scale synthetic speech dataset for pre-training and a refined real-worldspeech dataset for fine-tuning, (2) the Dual-DiT, a model designed toefficiently preserve aligned speech information while accurately reflectingenvironmental conditions, and (3) a diffusion-based Image-to-Audio Translatorthat allows the model to bridge the gap between audio and image, facilitatingthe generation of environmental sound that aligns with the multi-modal prompts.Extensive experimental results demonstrate that VoiceDiT outperforms previousmodels on real-world datasets, showcasing significant improvements in bothaudio quality and modality integration.
标题： 基于量化自我监督学习特征的预测语义的因果语音增强
链接：https://arxiv.org/abs/2412.19248
备注：Accepted for ICASSP 2025, 5 pages
摘要：实时语音增强是在线语音通信的关键。因果SE模型只使用以前的上下文，而预测未来的信息，如音素延续，可能有助于执行因果SE。语音信息通常通过量化自监督学习（SSL）模型的潜在特征来表示。这项工作是第一个将SSL功能与因果关系纳入SE模型。因果SSL特征被编码并使用特征线性调制与频谱图特征组合以估计用于增强噪声输入语音的掩模。同时，我们使用矢量量化来表示语音特征作为语义令牌的因果SSL功能。该模型不仅对SSL特征进行编码，而且可以预测多任务学习（MTL）中的未来语义标记。使用VoiceBank + DEMAND数据集的实验结果表明，我们提出的方法在PESQ中达到2.88，特别是语义预测MTL，其中我们证实了语义预测在因果SE中发挥了重要作用。
摘要：Real-time speech enhancement (SE) is essential to online speechcommunication. Causal SE models use only the previous context while predictingfuture information, such as phoneme continuation, may help performing causalSE. The phonetic information is often represented by quantizing latent featuresof self-supervised learning (SSL) models. This work is the first to incorporateSSL features with causality into an SE model. The causal SSL features areencoded and combined with spectrogram features using feature-wise linearmodulation to estimate a mask for enhancing the noisy input speech.Simultaneously, we quantize the causal SSL features using vector quantizationto represent phonetic characteristics as semantic tokens. The model not onlyencodes SSL features but also predicts the future semantic tokens in multi-tasklearning (MTL). The experimental results using VoiceBank + DEMAND dataset showthat our proposed method achieves 2.88 in PESQ, especially with semanticprediction MTL, in which we confirm that the semantic prediction played animportant role in causal SE.
标题： 利用预训练模型进行图形增强的双流特征融合用于声学交通监测
链接：https://arxiv.org/abs/2412.19078
备注：Shitong Fan and Feiyang Xiao contributed equally. Accepted by the IEEE International Conference on Acoustics, Speech, and Signal Processing(ICASSP)2025
摘要：麦克风阵列技术广泛应用于声源定位和基于声学的智慧城市交通监控中，但由于标记的真实世界交通音频数据的稀缺性以及应用场景的复杂性和多样性，这些应用面临着重大挑战。DCASE挑战赛的任务10侧重于使用多声道音频信号来计算车辆（汽车或商用车）并识别它们的方向（从左到右或反之亦然）。在本文中，我们提出了一个图形增强的双流特征融合网络（GEDF-Net）的声学交通监测，同时考虑车辆类型和方向，以提高检测。我们提出了一种图增强的双流特征融合策略，该策略由车辆类型特征提取（VTFE）分支，车辆方向特征提取（VDFE）分支，和一个帧级特征融合模块相结合的类型和方向的功能，以提高性能。在VTFE分支中使用预训练模型（PANN）来缓解数据稀缺性并增强类型特征，然后使用图形注意力机制来利用时间关系并突出显示这些特征中的重要音频事件。方向和类型特征的帧级融合实现了细粒度的特征表示，从而获得更好的检测性能。实验证明了该方法的有效性。GEDF-Net是我们在DCASE 2024挑战任务10中获得第一名的作品。
摘要：Microphone array techniques are widely used in sound source localization andsmart city acoustic-based traffic monitoring, but these applications facesignificant challenges due to the scarcity of labeled real-world traffic audiodata and the complexity and diversity of application scenarios. The DCASEChallenge's Task 10 focuses on using multi-channel audio signals to countvehicles (cars or commercial vehicles) and identify their directions(left-to-right or vice versa). In this paper, we propose a graph-enhanceddual-stream feature fusion network (GEDF-Net) for acoustic traffic monitoring,which simultaneously considers vehicle type and direction to improve detection.We propose a graph-enhanced dual-stream feature fusion strategy which consistsof a vehicle type feature extraction (VTFE) branch, a vehicle direction featureextraction (VDFE) branch, and a frame-level feature fusion module to combinethe type and direction feature for enhanced performance. A pre-trained model(PANNs) is used in the VTFE branch to mitigate data scarcity and enhance thetype features, followed by a graph attention mechanism to exploit temporalrelationships and highlight important audio events within these features. Theframe-level fusion of direction and type features enables fine-grained featurerepresentation, resulting in better detection performance. Experimentsdemonstrate the effectiveness of our proposed method. GEDF-Net is oursubmission that achieved 1st place in the DCASE 2024 Challenge Task 10.
标题： 用于抑郁症筛查的稳健语音和自然语言处理模型
链接：https://arxiv.org/abs/2412.19072
备注：None
摘要：抑郁症是一个全球性的健康问题，迫切需要增加患者筛查。语音技术为远程筛查提供了优势，但必须在患者中稳健地执行。我们已经描述了为此目的开发的两个深度学习模型。一个模型基于声学;另一个基于自然语言处理。这两种模型都采用了迁移学习。使用来自抑郁标记语料库的数据，其中11，000个独特的用户使用会话语音与人机应用程序进行交互。二元抑郁症分类的结果表明，这两种模型在没有说话者重叠的情况下，对看不见的数据的AUC=0.80或以上。性能作为测试子集特征的函数进行了进一步分析，发现这些模型通常对扬声器和会话变量具有鲁棒性。我们的结论是，基于这些方法的模型提供了广义的自动抑郁症筛查的承诺。
摘要：Depression is a global health concern with a critical need for increasedpatient screening. Speech technology offers advantages for remote screening butmust perform robustly across patients. We have described two deep learningmodels developed for this purpose. One model is based on acoustics; the otheris based on natural language processing. Both models employ transfer learning.Data from a depression-labeled corpus in which 11,000 unique users interactedwith a human-machine application using conversational speech is used. Resultson binary depression classification have shown that both models perform at orabove AUC=0.80 on unseen data with no speaker overlap. Performance is furtheranalyzed as a function of test subset characteristics, finding that the modelsare generally robust over speaker and session variables. We conclude thatmodels based on these approaches offer promise for generalized automateddepression screening.
标题： 攻击具有增强特征和说话者身份差异的语音合成系统
链接：https://arxiv.org/abs/2412.19068
备注：2 pages, submitted to ICASSP 2025 GC-7: The First VoicePrivacy Attacker Challenge (by invitation)
摘要：本研究的重点是ICASSP 2025信号处理大挑战赛中的第一个语音隐私攻击者挑战赛，该挑战赛旨在开发能够确定两个匿名语音信号是否来自同一说话人的说话人验证系统。然而，原始和匿名语音的特征分布之间的差异使这项任务复杂化。为了解决这一挑战，我们提出了一个攻击系统，结合数据增强增强的特征表示和扬声器身份差异增强分类器，以提高验证性能，称为DA-SID。具体而言，数据增强策略（即，数据融合和SpecAugment）来减小特征分布差距，而概率线性判别分析（PLDA）用于进一步增强说话人身份差异。我们的系统显著优于基线，在各种语音匿名系统中表现出卓越的有效性和鲁棒性，最终确保了挑战的前5名。
摘要：This study focuses on the First VoicePrivacy Attacker Challenge within theICASSP 2025 Signal Processing Grand Challenge, which aims to develop speakerverification systems capable of determining whether two anonymized speechsignals are from the same speaker. However, differences between featuredistributions of original and anonymized speech complicate this task. Toaddress this challenge, we propose an attacker system that combines DataAugmentation enhanced feature representation and Speaker Identity Differenceenhanced classifier to improve verification performance, termed DA-SID.Specifically, data augmentation strategies (i.e., data fusion and SpecAugment)are utilized to mitigate feature distribution gaps, while probabilistic lineardiscriminant analysis (PLDA) is employed to further enhance speaker identitydifference. Our system significantly outperforms the baseline, demonstratingexceptional effectiveness and robustness against various voice anonymizationsystems, ultimately securing a top-5 ranking in the challenge.
标题： 通过双焦点偏好优化增强视听语音识别
链接：https://arxiv.org/abs/2412.19005
备注：Accepted by AAAI 2025
摘要：视听自动语音识别（AV-ASR）旨在通过利用视觉信号来提高语音识别的准确性。由于噪声声学环境、自发语音和视觉信息的不确定使用，在各个领域的不受约束的真实世界场景中，这是特别具有挑战性的。大多数以前的作品都在视听数据集上微调纯音频ASR模型，针对传统的ASR目标进行优化。然而，他们往往忽略了视觉功能和常见的错误，在无约束的视频场景。在本文中，我们提出使用偏好优化策略来提高真实世界视频的语音识别准确率。首先，我们通过从两个焦点模拟AV-ASR中发生的常见错误来创建偏好数据：操纵音频或视觉输入并重写输出转录。其次，我们提出了BPO-AVASR，一种双焦点偏好优化方法，通过利用输入侧和输出侧偏好来改进AV-ASR模型。大量的实验表明，我们的方法显着提高了语音识别的准确性在各个领域，优于以前的国家的最先进的模型对现实世界的视频语音识别。
摘要：Audiovisual Automatic Speech Recognition (AV-ASR) aims to improve speechrecognition accuracy by leveraging visual signals. It is particularlychallenging in unconstrained real-world scenarios across various domains due tonoisy acoustic environments, spontaneous speech, and the uncertain use ofvisual information. Most previous works fine-tune audio-only ASR models onaudiovisual datasets, optimizing them for conventional ASR objectives. However,they often neglect visual features and common errors in unconstrained videoscenarios. In this paper, we propose using a preference optimization strategyto improve speech recognition accuracy for real-world videos. First, we createpreference data via simulating common errors that occurred in AV-ASR from twofocals: manipulating the audio or vision input and rewriting the outputtranscript. Second, we propose BPO-AVASR, a Bifocal Preference Optimizationmethod to improve AV-ASR models by leveraging both input-side and output-sidepreference. Extensive experiments demonstrate that our approach significantlyimproves speech recognition accuracy across various domains, outperformingprevious state-of-the-art models on real-world video speech recognition.
标题： 防止主动噪音控制中的输出饱和：一种输出约束的卡尔曼过滤器方法
链接：https://arxiv.org/abs/2412.18887
摘要：与最小均方（LMS）方法相比，基于卡尔曼滤波器（KF）的有源噪声控制（ANC）系统具有更好的跟踪能力和更快的收敛速度，尤其是在动态噪声消除场景中。然而，在噪声水平极高的环境中，由于硬件限制，控制信号的功率可能超过系统的额定输出功率，导致输出饱和和随后的非线性。为了缓解这个问题，提出了一种具有输出约束的修正KF。在这种方法中，被视为测量的干扰被重新缩放的约束因子，这是由系统的额定功率，次级路径增益，和干扰功率。结果，系统的输出功率，即控制信号，被间接地约束在系统的最大输出内，从而确保稳定性。仿真结果表明，该算法不仅实现了动态噪声的快速抑制，而且有效地防止了输出饱和引起的非线性，具有一定的实用意义。
摘要：The Kalman filter (KF)-based active noise control (ANC) system demonstratessuperior tracking and faster convergence compared to the least mean square(LMS) method, particularly in dynamic noise cancellation scenarios. However, inenvironments with extremely high noise levels, the power of the control signalcan exceed the system's rated output power due to hardware limitations, leadingto output saturation and subsequent non-linearity. To mitigate this issue, amodified KF with an output constraint is proposed. In this approach, thedisturbance treated as an measurement is re-scaled by a constraint factor,which is determined by the system's rated power, the secondary path gain, andthe disturbance power. As a result, the output power of the system, i.e. thecontrol signal, is indirectly constrained within the maximum output of thesystem, ensuring stability. Simulation results indicate that the proposedalgorithm not only achieves rapid suppression of dynamic noise but alsoeffectively prevents non-linearity due to output saturation, highlighting itspractical significance.
标题： 用于发音障碍和老年人语音识别的基础模型的结构化说话人缺陷适应
链接：https://arxiv.org/abs/2412.18832
摘要：语音基础模型（SFM）的数据密集型微调，以稀缺和多样的构音障碍和老年人的讲话导致数据偏见和穷人的泛化看不见的扬声器。本文提出了新的结构化的说话人不足的适应方法，SSL预训练的SFM等数据。在监督自适应微调阶段构建了说话人和语音缺陷不变的SFM，以减少对训练数据说话人的不适当偏见，并作为测试时间无监督自适应的更中立和鲁棒的起点。语音变异归因于扬声器的身份和语音障碍的严重程度，或老化引起的神经认知下降，建模使用单独的适配器，可以结合在一起，以模拟任何可见或不可见的扬声器。在UASpeech构音障碍和DementiaBank Pitt老年人语音语料库上的实验表明，HuBERT和Wav 2 vec 2-conformer模型的结构化说话人缺陷适应始终优于使用以下任一种的基线SFM：a）没有适配器; b）所有说话人共享全局适配器;或c）单属性适配器单独建模扬声器或缺陷标签，统计上显著的WER减少高达3.01%和1.50%绝对值（10.86%和6.94%）。最低公布的WER为19.45%（49.34%的非常低的可懂度，33.17%的看不见的话）是在UASpeech测试集的16个构音障碍的扬声器。
摘要：Data-intensive fine-tuning of speech foundation models (SFMs) to scarce anddiverse dysarthric and elderly speech leads to data bias and poorgeneralization to unseen speakers. This paper proposes novel structuredspeaker-deficiency adaptation approaches for SSL pre-trained SFMs on such data.Speaker and speech deficiency invariant SFMs were constructed in theirsupervised adaptive fine-tuning stage to reduce undue bias to training dataspeakers, and serves as a more neutral and robust starting point for test timeunsupervised adaptation. Speech variability attributed to speaker identity andspeech impairment severity, or aging induced neurocognitive decline, aremodelled using separate adapters that can be combined together to model anyseen or unseen speaker. Experiments on the UASpeech dysarthric and DementiaBankPitt elderly speech corpora suggest structured speaker-deficiency adaptation ofHuBERT and Wav2vec2-conformer models consistently outperforms baseline SFMsusing either: a) no adapters; b) global adapters shared among all speakers; orc) single attribute adapters modelling speaker or deficiency labels alone bystatistically significant WER reductions up to 3.01% and 1.50% absolute (10.86%and 6.94% relative) on the two tasks respectively. The lowest published WER of19.45% (49.34% on very low intelligibility, 33.17% on unseen words) is obtainedon the UASpeech test set of 16 dysarthric speakers.
标题： 埃塞俄比亚东正教Tewahedo教堂圣歌中Yaredawi YeZema淤泥的计算分析
链接：https://arxiv.org/abs/2412.18788
备注：6 pages
摘要：尽管其音乐学，文化和宗教意义，埃塞俄比亚东正教Tewahedo教会（EOTC）圣歌在音乐研究中的代表性相对不足。历史记录，包括手稿、研究论文和口头传统，证实了圣雅列在6世纪建立了三种典型的EOTC念诵模式。本文试图利用音乐信息检索技术对EOTC圣歌进行研究。在关于EOTC圣歌的分析和理解的研究问题中，Yaredawi YeZema Silt，即遵循圣雅列标准的圣歌模式，是首要的。因此，我们认为在EOTC圣歌Yaredawi YeZema淤泥分类的任务，通过引入一个新的数据集，并展示了一系列的分类实验，这项任务。结果表明，使用稳定的音高轮廓的分布作为一个简单的基于神经网络的分类器的特征表示成为一个有效的解决方案。通过与之前关于EOTC圣歌的民族音乐学文献的比较研究，进一步讨论了这些结果的音乐学含义和见解。通过公开这一数据集，我们的目标是促进EOTC圣歌的未来探索和分析，并突出进一步研究的潜在方向，从而促进对这一独特精神和文化遗产的更深入理解和保护。
摘要：Despite its musicological, cultural, and religious significance, theEthiopian Orthodox Tewahedo Church (EOTC) chant is relatively underrepresentedin music research. Historical records, including manuscripts, research papers,and oral traditions, confirm Saint Yared's establishment of three canonicalEOTC chanting modes during the 6th century. This paper attempts to investigatethe EOTC chants using music information retrieval (MIR) techniques. Among theresearch questions regarding the analysis and understanding of EOTC chants,Yaredawi YeZema Silt, namely the mode of chanting adhering to Saint Yared'sstandards, is of primary importance. Therefore, we consider the task ofYaredawi YeZema Silt classification in EOTC chants by introducing a new datasetand showcasing a series of classification experiments for this task. Resultsshow that using the distribution of stabilized pitch contours as the featurerepresentation on a simple neural network-based classifier becomes an effectivesolution. The musicological implications and insights of such results arefurther discussed through a comparative study with the previous ethnomusicologyliterature on EOTC chants. By making this dataset publicly accessible, we aimto promote future exploration and analysis of EOTC chants and highlightpotential directions for further research, thereby fostering a deeperunderstanding and preservation of this unique spiritual and cultural heritage.
标题： Zema数据集：Yaredawi Zema的全面研究，重点关注钟表圣歌
链接：https://arxiv.org/abs/2412.18784
备注：6 pages
摘要：计算音乐研究在推动音乐制作、发行和理解全球各种音乐风格方面发挥着至关重要的作用。尽管具有巨大的文化和宗教意义，但埃塞俄比亚东正教Tewahedo教会（EOTC）的圣歌在计算音乐研究中的代表性相对不足。本文通过介绍一个专门用于分析EOTC圣歌的新数据集（也称为Yaredawi Zema），为这一领域做出了贡献。这项工作提供了一个10小时的数据集，369个实例，创建和策展过程，包括严格的质量保证措施的全面概述。我们的数据集有一个详细的词级时间边界和阅读音调注释以及相应的诵经模式标签的音频。此外，我们还通过相应的注释，确定了手稿中与多个念诵符号相关的念诵选项。我们向公众提供此数据集的目的是鼓励更多对EOTC圣歌的研究和学习，包括歌词转录，歌词到音频对齐和音乐生成任务。这种研究工作将促进知识和努力，以保护这种独特的礼仪音乐，这是埃塞俄比亚人民的无价文化手工艺品。
摘要：Computational music research plays a critical role in advancing musicproduction, distribution, and understanding across various musical stylesworldwide. Despite the immense cultural and religious significance, theEthiopian Orthodox Tewahedo Church (EOTC) chants are relativelyunderrepresented in computational music research. This paper contributes tothis field by introducing a new dataset specifically tailored for analyzingEOTC chants, also known as Yaredawi Zema. This work provides a comprehensiveoverview of a 10-hour dataset, 369 instances, creation, and curation process,including rigorous quality assurance measures. Our dataset has a detailedword-level temporal boundary and reading tone annotation along with thecorresponding chanting mode label of audios. Moreover, we have also identifiedthe chanting options associated with multiple chanting notations in themanuscript by annotating them accordingly. Our goal in making this datasetavailable to the public 1 is to encourage more research and study of EOTCchants, including lyrics transcription, lyric-to-audio alignment, and musicgeneration tasks. Such research work will advance knowledge and efforts topreserve this distinctive liturgical music, a priceless cultural artifact forthe Ethiopian people.
标题： 调查声学-文本情感不一致信息以自动抑郁检测
链接：https://arxiv.org/abs/2412.18614
摘要：先前的研究已经证明，来自单个声学情感标签的情感特征可以提高抑郁症诊断的准确性。此外，根据情绪情境不敏感理论和我们的初步研究，抑郁症患者可能会以出乎意料的平静方式传达负面情绪内容，在自然对话中表现出高度的情绪表达不一致。到目前为止，很少有研究认识到并利用情绪表达不一致性进行抑郁检测。本文提出了一种多模态交叉注意方法来捕获声音-文本情感不一致（ATEI）信息。这是通过分析声学和文本域中情感表达的复杂的局部和长期依赖性，以及两个域中情感内容之间的不匹配来实现的。然后，提出了一种基于变换器的模型，将此ATEI信息与各种融合策略相结合，用于检测抑郁症。此外，采用缩放技术来调整ATEI特征度在融合过程中，从而提高模型的能力，以区分不同程度的抑郁症患者的严重程度。据我们所知，这项工作是第一次将情绪表达不一致的信息纳入抑郁症检测。在一个咨询会话数据集上的实验结果说明了该方法的有效性。
摘要：Previous studies have demonstrated that emotional features from a singleacoustic sentiment label can enhance depression diagnosis accuracy.Additionally, according to the Emotion Context-Insensitivity theory and ourpilot study, individuals with depression might convey negative emotionalcontent in an unexpectedly calm manner, showing a high degree of inconsistencyin emotional expressions during natural conversations. So far, few studies haverecognized and leveraged the emotional expression inconsistency for depressiondetection. In this paper, a multimodal cross-attention method is presented tocapture the Acoustic-Textual Emotional Inconsistency (ATEI) information. Thisis achieved by analyzing the intricate local and long-term dependencies ofemotional expressions across acoustic and textual domains, as well as themismatch between the emotional content within both domains. A Transformer-basedmodel is then proposed to integrate this ATEI information with various fusionstrategies for detecting depression. Furthermore, a scaling technique isemployed to adjust the ATEI feature degree during the fusion process, therebyenhancing the model's ability to discern patients with depression acrossvarying levels of severity. To best of our knowledge, this work is the first toincorporate emotional expression inconsistency information into depressiondetection. Experimental results on a counseling conversational datasetillustrate the effectiveness of our method.
标题： 通过预算调整和代币化提高印度语言Whisper的准确性和速度
链接：https://arxiv.org/abs/2412.19785
备注：Accepted at ICASSP 2025, 5 pages, 1 figures, 5 tables
摘要：自动语音识别最近在大型基础模型（如Whisper）方面取得了重大进展。然而，这些模型通常难以在低资源语言（如印度语言）中表现良好。本文探讨了两种新的方法，以提高耳语的多语种语音识别性能在印度语言。首先，我们提出了与语言家族信息，这提高了耳语的准确性，在语言相似的语言调整。其次，我们引入了一种新的标记器，它减少了生成的标记的数量，从而加快了Whisper的推理速度。我们广泛的实验表明，标记器显着减少了推理时间，而快速调整提高了各种Whisper模型大小的准确性，包括小型，中型和大型。总之，这些技术实现了最佳WER和推理速度之间的平衡。
摘要：Automatic speech recognition has recently seen a significant advancement withlarge foundational models such as Whisper. However, these models often struggleto perform well in low-resource languages, such as Indian languages. This paperexplores two novel approaches to enhance Whisper's multilingual speechrecognition performance in Indian languages. First, we propose prompt-tuningwith language family information, which enhances Whisper's accuracy inlinguistically similar languages. Second, we introduce a novel tokenizer thatreduces the number of generated tokens, thereby accelerating Whisper'sinference speed. Our extensive experiments demonstrate that the tokenizersignificantly reduces inference time, while prompt-tuning enhances accuracyacross various Whisper model sizes, including Small, Medium, and Large.Together, these techniques achieve a balance between optimal WER and inferencespeed.
标题： ETTA：阐明文本到音频模型的设计空间
链接：https://arxiv.org/abs/2412.19351
摘要：近年来，文本到音频（TTA）合成取得了重大进展，使用户能够通过从自然语言提示生成的合成音频来丰富他们的创意工作流程。尽管取得了这一进展，但数据、模型架构、训练目标函数和采样策略对目标基准的影响还没有得到很好的理解。为了全面了解TTA模型的设计空间，我们建立了一个大规模的经验实验，重点是扩散和流量匹配模型。我们的贡献包括：1）AF-Synthetic，从音频理解模型中获得的高质量合成字幕的大型数据集; 2）TTA模型的不同架构、训练和推理设计选择的系统比较; 3）采样方法及其帕累托曲线分析相对于生成质量和推理速度。我们利用从这种广泛的分析中获得的知识，提出了我们最好的模型，称为阐明文本到音频（ETTA）。当在AudioCaps和MusicCaps上进行评估时，ETTA提供了对公开数据训练的基线的改进，同时与专有数据训练的模型竞争。最后，我们展示了ETTA在复杂和富有想象力的字幕之后生成创意音频的改进能力-这是一项比当前基准更具挑战性的任务。
摘要：Recent years have seen significant progress in Text-To-Audio (TTA) synthesis,enabling users to enrich their creative workflows with synthetic audiogenerated from natural language prompts. Despite this progress, the effects ofdata, model architecture, training objective functions, and sampling strategieson target benchmarks are not well understood. With the purpose of providing aholistic understanding of the design space of TTA models, we set up alarge-scale empirical experiment focused on diffusion and flow matching models.Our contributions include: 1) AF-Synthetic, a large dataset of high qualitysynthetic captions obtained from an audio understanding model; 2) a systematiccomparison of different architectural, training, and inference design choicesfor TTA models; 3) an analysis of sampling methods and their Pareto curves withrespect to generation quality and inference speed. We leverage the knowledgeobtained from this extensive analysis to propose our best model dubbedElucidated Text-To-Audio (ETTA). When evaluated on AudioCaps and MusicCaps,ETTA provides improvements over the baselines trained on publicly availabledata, while being competitive with models trained on proprietary data. Finally,we show ETTA's improved ability to generate creative audio following complexand imaginative captions -- a task that is more challenging than currentbenchmarks.
标题： 提高人工智能合成语音检测的通用性
链接：https://arxiv.org/abs/2412.19279
备注：AAAI25
摘要：人工智能合成语音技术有可能为有益的应用创造真实的人类声音，但它也可能被滥用于恶意目的。虽然现有的人工智能合成语音检测模型在域内评估方面表现出色，但它们在跨不同领域推广方面面临挑战，随着新的语音生成器的出现，它们可能会过时。当前的解决方案使用不同的数据和先进的机器学习技术（例如，域不变表示、自监督学习），但是受限于预定义的声码器以及对像背景噪声和说话者身份这样的因素的敏感性。在这项工作中，我们引入了一个创新的解纠缠框架，旨在提取域不可知的文物功能相关的声码器。利用这些功能，我们增强了平坦损失环境中的模型学习，从而能够避免次优解决方案并提高泛化能力。在基准测试上的大量实验表明，我们的方法优于最先进的方法，在域内的等错误率度量和跨域评估中分别实现了高达5.12%和7.59%的改进。
摘要：AI-synthesized voice technology has the potential to create realistic humanvoices for beneficial applications, but it can also be misused for maliciouspurposes. While existing AI-synthesized voice detection models excel inintra-domain evaluation, they face challenges in generalizing across differentdomains, potentially becoming obsolete as new voice generators emerge. Currentsolutions use diverse data and advanced machine learning techniques (e.g.,domain-invariant representation, self-supervised learning), but are limited bypredefined vocoders and sensitivity to factors like background noise andspeaker identity. In this work, we introduce an innovative disentanglementframework aimed at extracting domain-agnostic artifact features related tovocoders. Utilizing these features, we enhance model learning in a flat losslandscape, enabling escape from suboptimal solutions and improvinggeneralization. Extensive experiments on benchmarks show our approachoutperforms state-of-the-art methods, achieving up to 5.12% improvement in theequal error rate metric in intra-domain and 7.59% in cross-domain evaluations.
标题： 基于双尺度注意力的元学习的个性化动态音乐情感识别
链接：https://arxiv.org/abs/2412.19200
备注：Accepted by the 39th AAAI Conference on Artificial Intelligence (AAAI-25)
摘要：动态音乐情感识别（DMER）旨在预测音乐中不同时刻的情感，在音乐信息检索中起着至关重要的作用。现有的DMER方法在处理序列数据时难以捕获长期依赖性，这限制了它们的性能。此外，这些方法往往忽略了个体差异对情绪感知的影响，即使每个人都有自己的个性化情绪感知在现实世界中。出于这些问题的动机，我们探索更有效的序列处理方法，并引入个性化DMER（PDMER）问题，这需要模型来预测与个性化感知一致的情绪。具体来说，我们提出了一种基于双尺度注意力的元学习（DSAML）方法。该方法融合了双尺度特征提取器的特征，并使用双尺度注意力Transformer捕获短期和长期依赖关系，提高了传统DMER的性能。为了实现PDMER，我们设计了一种新的任务构造策略，划分任务的注释。任务中的样本由相同的注释器注释，确保一致的感知。利用这种策略和元学习，DSAML可以预测个性化的情绪感知，只需一个个性化的注释样本。我们的客观和主观实验表明，我们的方法可以实现国家的最先进的性能在传统的DMER和PDMER。
摘要：Dynamic Music Emotion Recognition (DMER) aims to predict the emotion ofdifferent moments in music, playing a crucial role in music informationretrieval. The existing DMER methods struggle to capture long-term dependencieswhen dealing with sequence data, which limits their performance. Furthermore,these methods often overlook the influence of individual differences on emotionperception, even though everyone has their own personalized emotionalperception in the real world. Motivated by these issues, we explore moreeffective sequence processing methods and introduce the Personalized DMER(PDMER) problem, which requires models to predict emotions that align withpersonalized perception. Specifically, we propose a Dual-Scale Attention-BasedMeta-Learning (DSAML) method. This method fuses features from a dual-scalefeature extractor and captures both short and long-term dependencies using adual-scale attention transformer, improving the performance in traditionalDMER. To achieve PDMER, we design a novel task construction strategy thatdivides tasks by annotators. Samples in a task are annotated by the sameannotator, ensuring consistent perception. Leveraging this strategy alongsidemeta-learning, DSAML can predict personalized perception of emotions with justone personalized annotation sample. Our objective and subjective experimentsdemonstrate that our method can achieve state-of-the-art performance in bothtraditional DMER and PDMER.
标题： 凝聚舞者：通过音乐驱动的凝聚力分解增强互动群舞生成
链接：https://arxiv.org/abs/2412.19123
摘要：舞蹈生成是至关重要和具有挑战性的，特别是在舞蹈表演和虚拟游戏等领域。在目前的文献中，大多数方法论都集中在独奏音乐2Dance上。虽然有针对集体音乐舞蹈的努力，但这些努力往往缺乏连贯性，导致舞蹈表演美学上的缺陷。因此，我们介绍Cohedancers，一个新的框架，音乐驱动的互动群舞生成。Cohedancers旨在通过将其分解为三个关键方面来增强集体舞的连贯性：同步性，自然性和流动性。相应地，我们开发了基于周期一致性的舞蹈同步策略来促进音乐与舞蹈的对应，基于自回归的暴露偏差纠正策略来增强生成舞蹈的流动性，以及对抗性训练策略来增强自然性团体舞输出。总的来说，这些策略使CohdeDancers能够产生高质量的高度连贯的集体舞蹈。此外，为了为Group Music 2Dance建立更好的基准，我们构建了迄今为止最多样化和最全面的开源数据集I-Dancers，具有丰富的舞者互动，并创建了全面的评估指标。对I-Dancers和其他现存数据集的实验评估证实了CoheDancers实现了前所未有的最先进性能。将发布代码。
摘要：Dance generation is crucial and challenging, particularly in domains likedance performance and virtual gaming. In the current body of literature, mostmethodologies focus on Solo Music2Dance. While there are efforts directedtowards Group Music2Dance, these often suffer from a lack of coherence,resulting in aesthetically poor dance performances. Thus, we introduceCoheDancers, a novel framework for Music-Driven Interactive Group DanceGeneration. CoheDancers aims to enhance group dance generation coherence bydecomposing it into three key aspects: synchronization, naturalness, andfluidity. Correspondingly, we develop a Cycle Consistency based DanceSynchronization strategy to foster music-dance correspondences, anAuto-Regressive-based Exposure Bias Correction strategy to enhance the fluidityof the generated dances, and an Adversarial Training Strategy to augment thenaturalness of the group dance output. Collectively, these strategies enableCohdeDancers to produce highly coherent group dances with superior quality.Furthermore, to establish better benchmarks for Group Music2Dance, we constructthe most diverse and comprehensive open-source dataset to date, I-Dancers,featuring rich dancer interactions, and create comprehensive evaluationmetrics. Experimental evaluations on I-Dancers and other extant datasetssubstantiate that CoheDancers achieves unprecedented state-of-the-artperformance. Code will be released.
标题： BSDB-Net：带分裂双分支网络，具有选择性状态空间机制，用于单耳语音增强
链接：https://arxiv.org/abs/2412.19099
备注：Accepted by AAAI 2025
摘要：虽然基于复谱的语音增强（SE）方法已经取得了显著的性能，但是幅度和相位的耦合会导致补偿效应，其中牺牲幅度信息来补偿对SE有害的相位。此外，为了进一步提高SE的性能，许多模块堆叠在SE上，导致模型复杂度增加，限制了SE的应用。为了解决这些问题，提出了一种基于Mamba压缩频率的双路径网络。首先，通过并行双分支提取幅度和相位信息。该方法利用结构化复频谱隐式地捕获相位信息，并通过解耦幅度和相位来解决补偿效应，网络中加入交互模块来抑制不必要的部分并从另一分支恢复丢失的分量。其次，为了降低网络复杂度，该网络引入了频带分割策略来压缩频率维。为了在保持良好性能的同时进一步降低复杂度，我们设计了一个基于Mamba的模块，该模块在线性复杂度下对时间和频率维度进行建模。最后，与基准相比，我们的模型在保持卓越性能的同时，平均将计算复杂度降低了8.3倍。此外，与基于变换器的模型相比，它实现了25倍的复杂度降低。
摘要：Although the complex spectrum-based speech enhancement(SE) methods haveachieved significant performance, coupling amplitude and phase can lead to acompensation effect, where amplitude information is sacrificed to compensatefor the phase that is harmful to SE. In addition, to further improve theperformance of SE, many modules are stacked onto SE, resulting in increasedmodel complexity that limits the application of SE. To address these problems,we proposed a dual-path network based on compressed frequency using Mamba.First, we extract amplitude and phase information through parallel dualbranches. This approach leverages structured complex spectra to implicitlycapture phase information and solves the compensation effect by decouplingamplitude and phase, and the network incorporates an interaction module tosuppress unnecessary parts and recover missing components from the otherbranch. Second, to reduce network complexity, the network introduces aband-split strategy to compress the frequency dimension. To further reducecomplexity while maintaining good performance, we designed a Mamba-based modulethat models the time and frequency dimensions under linear complexity. Finally,compared to baselines, our model achieves an average 8.3 times reduction incomputational complexity while maintaining superior performance. Furthermore,it achieves a 25 times reduction in complexity compared to transformer-basedmodels.
标题： 利用多语言STEN-TTC和Bert LID的印度尼西亚-英语代码转换语音合成器
链接：https://arxiv.org/abs/2412.19043
备注：Accepted at O-COCOSDA 2024
摘要：多语言文本到语音系统跨多种语言将文本转换为语音。在许多情况下，文本句子可能包含不同语言的片段，这种现象称为语码转换。这在印度尼西亚尤其常见，尤其是在印度尼西亚语和英语之间。尽管它的意义，还没有研究开发出一个多语种的TTS系统能够处理这两种语言之间的代码切换。本研究旨在探讨STEN-TTS中的汉英语码转换现象。关键的修改包括使用微调的BERT进行每个单词的语言识别，以及从基本模型中删除语言嵌入，将语言识别组件添加到文本到音素转换中。实验结果表明，代码转换模型实现了优越的自然度和改善的语音清晰度相比，印尼和英语基线STEN-TTS模型。
摘要：Multilingual text-to-speech systems convert text into speech across multiplelanguages. In many cases, text sentences may contain segments in differentlanguages, a phenomenon known as code-switching. This is particularly common inIndonesia, especially between Indonesian and English. Despite its significance,no research has yet developed a multilingual TTS system capable of handlingcode-switching between these two languages. This study addressesIndonesian-English code-switching in STEN-TTS. Key modifications include addinga language identification component to the text-to-phoneme conversion usingfinetuned BERT for per-word language identification, as well as removinglanguage embedding from the base model. Experimental results demonstrate thatthe code-switching model achieves superior naturalness and improved speechintelligibility compared to the Indonesian and English baseline STEN-TTSmodels.
标题： Leave-One-Equival：减轻对比音乐表示中与不变性相关的信息损失
链接：https://arxiv.org/abs/2412.18955
摘要：对比学习在自我监督的音乐表征学习中被证明是有效的，特别是对于音乐信息检索（MIR）任务。然而，依赖于增强链的对比视图生成和由此产生的学习不变性提出了挑战时，不同的下游任务需要对某些音乐属性的敏感性。为了解决这个问题，我们提出了Leave One Equivariant（LOEV）框架，与以前的工作相比，该框架通过选择性地保留有关特定增强的信息，引入了一种灵活的任务自适应方法，允许模型保持任务相关的等方差。我们证明了LOEV消除了与学习不变性相关的信息损失，提高了增强相关任务和检索的性能，而不会牺牲一般表示质量。此外，我们引入了LOEV的一个变体LOEV++，它通过设计以自监督的方式构建了一个解纠缠的潜在空间，并基于增强相关属性实现了有针对性的检索。
摘要：Contrastive learning has proven effective in self-supervised musicalrepresentation learning, particularly for Music Information Retrieval (MIR)tasks. However, reliance on augmentation chains for contrastive view generationand the resulting learnt invariances pose challenges when different downstreamtasks require sensitivity to certain musical attributes. To address this, wepropose the Leave One EquiVariant (LOEV) framework, which introduces aflexible, task-adaptive approach compared to previous work by selectivelypreserving information about specific augmentations, allowing the model tomaintain task-relevant equivariances. We demonstrate that LOEV alleviatesinformation loss related to learned invariances, improving performance onaugmentation related tasks and retrieval without sacrificing generalrepresentation quality. Furthermore, we introduce a variant of LOEV, LOEV++,which builds a disentangled latent space by design in a self-supervised manner,and enables targeted retrieval based on augmentation related attributes.
标题： 稳健的目标说话人到达方向估计
链接：https://arxiv.org/abs/2412.18913
摘要：在多说话人环境中，目标说话人的波达方向（DOA）是提高语音清晰度和提取目标说话人语音的关键。然而，传统的DOA估计方法往往在噪声、混响的存在下，特别是当存在竞争扬声器时，会遇到困难。为了解决这些挑战，我们提出了RTS-DOA，一个强大的实时DOA估计系统。该系统创新性地使用目标说话人的注册语音作为参考，并利用来自麦克风阵列的全频带和子频带频谱信息来估计目标说话人的语音的DOA。具体地，该系统包括用于初始改善语音质量的语音增强模块、用于学习空间信息的空间模块和用于提取声纹特征的说话者模块。LibriSpeech数据集上的实验结果表明，我们的RTS-DOA系统有效地处理多说话人的情况下，建立了新的最佳基准。
摘要：In multi-speaker environments the direction of arrival (DOA) of a targetspeaker is key for improving speech clarity and extracting target speaker'svoice. However, traditional DOA estimation methods often struggle in thepresence of noise, reverberation, and particularly when competing speakers arepresent. To address these challenges, we propose RTS-DOA, a robust real-timeDOA estimation system. This system innovatively uses the registered speech ofthe target speaker as a reference and leverages full-band and sub-band spectralinformation from a microphone array to estimate the DOA of the target speaker'svoice. Specifically, the system comprises a speech enhancement module forinitially improving speech quality, a spatial module for learning spatialinformation, and a speaker module for extracting voiceprint features.Experimental results on the LibriSpeech dataset demonstrate that our RTS-DOAsystem effectively tackles multi-speaker scenarios and established new optimalbenchmarks.
标题： 利用新型方法和MultiNAM数据集推进NAM到语音的转换
链接：https://arxiv.org/abs/2412.18839
备注：Accepted at IEEE ICASSP 2025
摘要：目前的非可听杂音（NAM）到语音技术依赖于语音克隆来模拟成对耳语的地面真实语音。然而，模拟的语音往往缺乏可懂度，并且不能很好地概括不同的说话者。为了解决这个问题，我们专注于从成对的耳语和文本中学习音素级对齐，并采用文本到语音（TTS）系统来模拟地面实况。为了减少对耳语的依赖，我们直接从NAM学习音素对齐，尽管质量受到可用训练数据的限制。为了进一步减轻对NAM/耳语数据的依赖，我们建议结合嘴唇模态来推断语音，并引入一种新的基于扩散的方法，该方法利用了嘴唇到语音技术的最新进展。此外，我们发布了MultiNAM数据集，其中包含来自两个扬声器的配对NAM，耳语，视频和文本数据，并在此数据集上对所有方法进行基准测试。语音样本和数据集可在\url{https：//diff-nam.github.io/DiffNAM/}
摘要：Current Non-Audible Murmur (NAM)-to-speech techniques rely on voice cloningto simulate ground-truth speech from paired whispers. However, the simulatedspeech often lacks intelligibility and fails to generalize well acrossdifferent speakers. To address this issue, we focus on learning phoneme-levelalignments from paired whispers and text and employ a Text-to-Speech (TTS)system to simulate the ground-truth. To reduce dependence on whispers, we learnphoneme alignments directly from NAMs, though the quality is constrained by theavailable training data. To further mitigate reliance on NAM/whisper data forground-truth simulation, we propose incorporating the lip modality to inferspeech and introduce a novel diffusion-based method that leverages recentadvancements in lip-to-speech technology. Additionally, we release the MultiNAMdataset with over $7.96$ hours of paired NAM, whisper, video, and text datafrom two speakers and benchmark all methods on this dataset. Speech samples andthe dataset are available at \url{https://diff-nam.github.io/DiffNAM/}
标题： MRI 2 Speech：实时MRI记录的关节运动的语音合成
链接：https://arxiv.org/abs/2412.18836
备注：Accepted at IEEE ICASSP 2025
摘要：以前的实时MRI（rtMRI）为基础的语音合成模型严重依赖于嘈杂的地面实况语音。直接在地面真实梅尔频谱图上应用损失会使语音内容与MRI噪声纠缠在一起，导致可懂度差。我们引入了一种新的方法，该方法采用多模态自监督AV-HuBERT模型进行rtMRI文本预测，并采用了一种新的基于流的持续时间预测器进行特定于说话人的对齐。然后，语音解码器使用预测的文本和持续时间来合成任何新语音中的对齐语音。我们在两个数据集上进行了彻底的实验，并证明了我们的方法对看不见的说话者的泛化能力。我们通过掩蔽rtMRI视频的一部分来评估我们框架的性能，以评估不同发音器对文本预测的影响。我们的方法在USC-TIMIT MRI语料库上实现了15.18\%$的字错误率（WER），这标志着对当前最先进技术的巨大改进。语音样本可在\url{https：//mri2speech.github.io/MRI2Speech/}获得
摘要：Previous real-time MRI (rtMRI)-based speech synthesis models depend heavilyon noisy ground-truth speech. Applying loss directly over ground truthmel-spectrograms entangles speech content with MRI noise, resulting in poorintelligibility. We introduce a novel approach that adapts the multi-modalself-supervised AV-HuBERT model for text prediction from rtMRI and incorporatesa new flow-based duration predictor for speaker-specific alignment. Thepredicted text and durations are then used by a speech decoder to synthesizealigned speech in any novel voice. We conduct thorough experiments on twodatasets and demonstrate our method's generalization ability to unseenspeakers. We assess our framework's performance by masking parts of the rtMRIvideo to evaluate the impact of different articulators on text prediction. Ourmethod achieves a $15.18\%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus,marking a huge improvement over the current state-of-the-art. Speech samplesare available at \url{https://mri2speech.github.io/MRI2Speech/}
标题： 通过多尺度多模式上下文交互实现具有表达力的视频配音
链接：https://arxiv.org/abs/2412.18748
备注：Accepted by ICSSP 2025
摘要：自动视频配音（AVD）从脚本中生成与嘴唇运动和面部情绪一致的语音。近年来的研究主要集中在对多模态语境进行建模以增强韵律表达能力，但忽略了两个关键问题：1）语境中的多尺度韵律表达属性对当前句子的韵律产生影响。2)上下文中的韵律线索与当前句子相互作用，影响最终的韵律表达力。为了应对这些挑战，我们提出了M2 CI-Dubber，一个多尺度多模态上下文交互方案AVD。该方案包括两个共享的M2 CI编码器，用于对多尺度多模态上下文进行建模，并促进其与当前句子的深度交互。该方法通过提取上下文中每种模态的全局和局部特征，利用基于注意力的聚合和交互机制，采用基于交互的图注意力网络进行融合，增强了当前句子合成语音的韵律表达能力。在Chem数据集上的实验表明，我们的模型在配音表现力方面优于基线。代码和演示可在\textcolor[rgb]{0.93，0.0，0.47}{https：//github.com/AI-S2-Lab/M2CI-Dubber}获得。
摘要：Automatic Video Dubbing (AVD) generates speech aligned with lip motion andfacial emotion from scripts. Recent research focuses on modeling multimodalcontext to enhance prosody expressiveness but overlooks two key issues: 1)Multiscale prosody expression attributes in the context influence the currentsentence's prosody. 2) Prosody cues in context interact with the currentsentence, impacting the final prosody expressiveness. To tackle thesechallenges, we propose M2CI-Dubber, a Multiscale Multimodal Context Interactionscheme for AVD. This scheme includes two shared M2CI encoders to model themultiscale multimodal context and facilitate its deep interaction with thecurrent sentence. By extracting global and local features for each modality inthe context, utilizing attention-based mechanisms for aggregation andinteraction, and employing an interaction-based graph attention network forfusion, the proposed approach enhances the prosody expressiveness ofsynthesized speech for the current sentence. Experiments on the Chem datasetshow our model outperforms baselines in dubbing expressiveness. The code anddemos are available at\textcolor[rgb]{0.93,0.0,0.47}{https://github.com/AI-S2-Lab/M2CI-Dubber}.
标题： 用于对话语音合成的模式内和模式间上下文交互建模
链接：https://arxiv.org/abs/2412.18733
备注：Accepted by ICASSP 2025
摘要：会话语音合成（CSS）旨在有效地利用多模态对话历史（MDH）生成具有适当会话韵律的语音。CSS的关键挑战是对MDH和目标话语之间的交互进行建模。MDH中的语篇模态和言语模态各有其独特的影响，它们相辅相成，对目标语产生综合影响。以前的作品没有明确的模型，这样的模态内和模态间的相互作用。为了解决这个问题，我们提出了一个新的模态内和模态间的上下文交互模式为基础的CSS系统，称为III-CSS。具体而言，在训练阶段，我们将MDH与目标话语中的文本和语音模态相结合，以获得四种模态组合，包括历史文本-下一文本，历史语音-下一语音，历史文本-下一语音和历史语音-下一文本。然后，我们设计了两个基于对比学习的模态内和两个模态间交互模块，以深入学习模态内和模态间的上下文交互。在推理阶段，我们采取MDH和采用训练的交互模块，充分推断目标话语的文本内容的语音韵律。在DailyTalk数据集上进行的主观和客观实验表明，III-CSS在韵律表现力方面优于高级基线。代码和语音示例可在https://github.com/AI-S2-Lab/I3CSS上获得。
摘要：Conversational Speech Synthesis (CSS) aims to effectively take the multimodaldialogue history (MDH) to generate speech with appropriate conversationalprosody for target utterance. The key challenge of CSS is to model theinteraction between the MDH and the target utterance. Note that text and speechmodalities in MDH have their own unique influences, and they complement eachother to produce a comprehensive impact on the target utterance. Previous worksdid not explicitly model such intra-modal and inter-modal interactions. Toaddress this issue, we propose a new intra-modal and inter-modal contextinteraction scheme-based CSS system, termed III-CSS. Specifically, in thetraining phase, we combine the MDH with the text and speech modalities in thetarget utterance to obtain four modal combinations, including HistoricalText-Next Text, Historical Speech-Next Speech, Historical Text-Next Speech, andHistorical Speech-Next Text. Then, we design two contrastive learning-basedintra-modal and two inter-modal interaction modules to deeply learn theintra-modal and inter-modal context interaction. In the inference phase, wetake MDH and adopt trained interaction modules to fully infer the speechprosody of the target utterance's text content. Subjective and objectiveexperiments on the DailyTalk dataset show that III-CSS outperforms the advancedbaselines in terms of prosody expressiveness. Code and speech samples areavailable at https://github.com/AI-S2-Lab/I3CSS.
标题： Simi-SFX：一种基于相似度的可控音效合成方法
链接：https://arxiv.org/abs/2412.18710
摘要：生成具有可控变化的声音效果是一项具有挑战性的任务，传统上使用需要深入了解信号处理参数和算法的复杂物理模型来解决。在生成和大型语言模型的时代，文本已经成为控制声音合成的通用、人类可解释的界面。然而，语言标记的离散和定性性质使得难以捕捉不同声音之间的细微音色变化。在这项研究中，我们提出了一种新的相似性为基础的空调方法的声音合成，利用可微数字信号处理（DDSP）。该方法结合使用潜在空间学习和控制音频音色与直观的指导向量，在范围[0，1]内归一化，以编码分类声学信息。通过利用预先训练的音频表示模型，我们的方法实现了表达和细粒度的音色控制。为了对我们的方法进行基准测试，我们引入了两个声音效果数据集-足迹集和影响集-旨在评估可控性和声音质量。回归分析表明，建议的相似性得分有效地控制音色的变化，并使创造性的应用，如离散类之间的音色插值。我们的工作为音效合成提供了一个强大而通用的框架，弥合了传统信号处理和现代机器学习技术之间的差距。
摘要：Generating sound effects with controllable variations is a challenging task,traditionally addressed using sophisticated physical models that requirein-depth knowledge of signal processing parameters and algorithms. In the eraof generative and large language models, text has emerged as a common,human-interpretable interface for controlling sound synthesis. However, thediscrete and qualitative nature of language tokens makes it difficult tocapture subtle timbral variations across different sounds. In this research, wepropose a novel similarity-based conditioning method for sound synthesis,leveraging differentiable digital signal processing (DDSP). This approachcombines the use of latent space for learning and controlling audio timbre withan intuitive guiding vector, normalized within the range [0,1], to encodecategorical acoustic information. By utilizing pre-trained audio representationmodels, our method achieves expressive and fine-grained timbre control. Tobenchmark our approach, we introduce two sound effect datasets--Footstep-setand Impact-set--designed to evaluate both controllability and sound quality.Regression analysis demonstrates that the proposed similarity score effectivelycontrols timbre variations and enables creative applications such as timbreinterpolation between discrete classes. Our work provides a robust andversatile framework for sound effect synthesis, bridging the gap betweentraditional signal processing and modern machine learning techniques.
标题： 下一个面向多模式智能的代币预测：全面调查
链接：https://arxiv.org/abs/2412.18619
备注：69 papes, 18 figures, repo at this https URL
摘要：基于自然语言处理中语言建模的基础，下一个标记预测（NTP）已经发展成为各种模式机器学习任务的通用训练目标，取得了相当大的成功。随着大型语言模型（LLM）在文本模态中统一理解和生成任务的发展，最近的研究表明，来自不同模态的任务也可以有效地封装在NTP框架中，将多模态信息转换为令牌，并在给定上下文的情况下预测下一个。这项调查介绍了一个全面的分类，统一的理解和生成多模态学习通过NTP的镜头。提出的分类法涵盖了五个关键方面：多模式标记化，MMNTP模型架构，统一任务表示，数据集\评估和开放的挑战。这种新的分类法旨在帮助研究人员探索多模态智能。收集最新论文和repos的相关GitHub存储库可在https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction上获得
摘要：Building on the foundations of language modeling in natural languageprocessing, Next Token Prediction (NTP) has evolved into a versatile trainingobjective for machine learning tasks across various modalities, achievingconsiderable success. As Large Language Models (LLMs) have advanced to unifyunderstanding and generation tasks within the textual modality, recent researchhas shown that tasks from different modalities can also be effectivelyencapsulated within the NTP framework, transforming the multimodal informationinto tokens and predict the next one given the context. This survey introducesa comprehensive taxonomy that unifies both understanding and generation withinmultimodal learning through the lens of NTP. The proposed taxonomy covers fivekey aspects: Multimodal tokenization, MMNTP model architectures, unified taskrepresentation, datasets \& evaluation, and open challenges. This new taxonomyaims to aid researchers in their exploration of multimodal intelligence. Anassociated GitHub repository collecting the latest papers and repos isavailable at https://github.com/LMM101/Awesome-Multimodal-Next-Token-Prediction
