语音/音频处理学术速递[12.1]

今日论文合集：cs.SD语音5篇，eess.AS音频处理9篇。

cs.SD语音

【1】CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation

标题：CoDI-2：情景中、交错和交互式任意对任意一代链接：https://arxiv.org/abs/2311.18775

作者：Zineng Tang,Ziyi Yang,Mahmoud Khademi,Yang Liu,Chenguang Zhu,Mohit Bansal备注：Project Page: this https URL摘要：我们提出了CoDi-2，一个多功能和交互式的多模态大型语言模型（MLLM），可以遵循复杂的多模态交错指令，进行上下文学习（ICL），推理，聊天，编辑等，在任意对任意输入输出模态范例中。通过将模态与编码和生成的语言对齐，CoDi-2使大型语言模型（LLM）不仅能够理解复杂的模态交织指令和上下文示例，而且还可以在连续特征空间中自回归生成接地和连贯的多模态输出。为了训练CoDi-2，我们构建了一个大规模的生成数据集，包括文本，视觉和音频的上下文多模态指令。CoDi-2展示了多模态生成的广泛zero-shot能力，例如通过多轮交互式对话的上下文学习，推理和任意到任意模态生成的组合性。CoDi-2在主题驱动的图像生成、视觉转换和音频编辑等任务上超越了以前的特定领域模型。CoDi-2标志着在开发一个全面的多模态基础模型方面的重大突破，该模型擅长解释上下文中的语言-视觉-音频交织指令并产生多模态输出。摘要：We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any input-output modality paradigm. By aligning modalities with language for both encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not only understand complex modality-interleaved instructions and in-context examples, but also autoregressively generate grounded and coherent multimodal outputs in the continuous feature space. To train CoDi-2, we build a large-scale generation dataset encompassing in-context multimodal instructions across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot capabilities for multimodal generation, such as in-context learning, reasoning, and compositionality of any-to-any modality generation through multi-round interactive conversation. CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing. CoDi-2 signifies a substantial breakthrough in developing a comprehensive multimodal foundation model adept at interpreting in-context language-vision-audio interleaved instructions and producing multimodal outputs.

【2】 Barwise Music Structure Analysis with the Correlation Block-Matching Segmentation Algorithm标题：基于相关块匹配分割算法的巴氏音乐结构分析链接：https://arxiv.org/abs/2311.18604

作者：Axel Marmoret,Jérémy E. Cohen,Frédéric Bimbot备注：None摘要：音乐结构分析（MSA）是一种音乐信息检索任务，其包括通过将歌曲分解成通常对应于"合唱“、"主歌”、"独唱“等的部分来以简化的、有组织的方式表示歌曲。在这项工作中，我们扩展了由（Marmoret等人，2020年，2022年b）。CBM算法是一种动态规划算法，它分割自相似矩阵，自相似矩阵是MSA和许多其他应用中使用的标准描述。在这项工作中，自相似矩阵计算的音频信号的特征表示和时间采样在酒吧尺度。本研究探讨三种不同的标准相似性函数的自相似矩阵的计算。结果表明，在最佳条件下，所提出的算法实现了一个水平的性能，这是有竞争力的监督国家的最先进的方法，而只需要知识的酒吧的位置。此外，该算法是开源的，具有高度可定制性。摘要：Music Structure Analysis (MSA) is a Music Information Retrieval task consisting of representing a song in a simplified, organized manner by breaking it down into sections typically corresponding to ``chorus'', ``verse'', ``solo'', etc. In this work, we extend an MSA algorithm called the Correlation Block-Matching (CBM) algorithm introduced by (Marmoret et al., 2020, 2022b). The CBM algorithm is a dynamic programming algorithm that segments self-similarity matrices, which are a standard description used in MSA and in numerous other applications. In this work, self-similarity matrices are computed from the feature representation of an audio signal and time is sampled at the bar-scale. This study examines three different standard similarity functions for the computation of self-similarity matrices. Results show that, in optimal conditions, the proposed algorithm achieves a level of performance which is competitive with supervised state-of-the-art methods while only requiring knowledge of bar positions. In addition, the algorithm is made open-source and is highly customizable.

【3】 String Sound Synthesizer on GPU-accelerated Finite Difference Scheme标题：基于GPU加速有限差分格式的弦声合成器链接：https://arxiv.org/abs/2311.18505

作者：Jin Woo Lee,Min Jun Choi,Kyogu Lee备注：Submitted to ICASSP 2024摘要：本文介绍了一种非线性弦声合成器，它是基于有限差分法模拟弦在各种激励下的动力学行为。所提出的合成器具有一个多功能的字符串模拟引擎能够随机参数化，包括基本频率调制，刚度，张力，频率相关的损失，和激励控制。这个开源的物理模型模拟器不仅有利于音频信号处理社区，而且通过作为一种新的数据集构建工具，为新兴的基于神经网络的音频合成领域做出了贡献。该合成器在PyTorch中实现，提供了灵活性，促进了CPU和GPU的利用率，从而增强了其作为模拟器的适用性。GPU利用率通过跨空间和批处理维度的并行化操作来加速计算，进一步增强其作为数据生成器的实用性。摘要：This paper introduces a nonlinear string sound synthesizer, based on a finite difference simulation of the dynamic behavior of strings under various excitations. The presented synthesizer features a versatile string simulation engine capable of stochastic parameterization, encompassing fundamental frequency modulation, stiffness, tension, frequency-dependent loss, and excitation control. This open-source physical model simulator not only benefits the audio signal processing community but also contributes to the burgeoning field of neural network-based audio synthesis by serving as a novel dataset construction tool. Implemented in PyTorch, this synthesizer offers flexibility, facilitating both CPU and GPU utilization, thereby enhancing its applicability as a simulator. GPU utilization expedites computation by parallelizing operations across spatial and batch dimensions, further enhancing its utility as a data generator.

【4】 Subspace Hybrid MVDR Beamforming for Augmented Hearing标题：用于增强听力的子空间混合MVDR波束形成链接：https://arxiv.org/abs/2311.18689

作者：Sina Hafezi,Alastair H. Moore,Pierre H. Guiraud,Patrick A. Naylor,Jacob Donley,Vladimir Tourbabin,Thomas Lunner备注：14 pages, 10 figures, submitted for IEEE/ACM Transactions on Audio, Speech, and Language Processing on 23-Nov-2023摘要：当声学场景（无论是真实世界还是模拟的）在声源数量、环境声场及其动态方面是直接的时，信号相关波束形成器优于信号无关波束形成器。然而，在使用头戴式麦克风阵列的增强现实音频的上下文中，遇到的声学场景通常远非简单。针对这种情况设计鲁棒、高性能、自适应的波束形成器是一个持续的挑战。这是由于违反了对噪声场的通常所需的假设，所述噪声场由例如由复杂声学环境和/或收听者头部的旋转引起的快速变化引起。这项工作提出了一种多通道语音增强算法，利用信号相关的波束形成器的适应性，同时仍然受益于信号无关的超定向波束形成器的计算效率和鲁棒性能。该算法有两个阶段。(i)第一阶段是基于对应于一组噪声场模型的权重字典的混合波束形成器。(ii)第二级是宽带子空间后滤波器，以去除由（i）产生的任何伪影。该算法使用现实世界的录音和模拟的鸡尾酒会的情况下进行评估。噪声抑制，可懂度和语音质量的结果表明，所提出的算法相比，基线超定向波束形成器的显着性能改善。噪声场字典的数据驱动的实现被示出为提供更多的噪声抑制，和类似的语音清晰度和质量，相比于参数字典。摘要：Signal-dependent beamformers are advantageous over signal-independent beamformers when the acoustic scenario - be it real-world or simulated - is straightforward in terms of the number of sound sources, the ambient sound field and their dynamics. However, in the context of augmented reality audio using head-worn microphone arrays, the acoustic scenarios encountered are often far from straightforward. The design of robust, high-performance, adaptive beamformers for such scenarios is an on-going challenge. This is due to the violation of the typically required assumptions on the noise field caused by, for example, rapid variations resulting from complex acoustic environments, and/or rotations of the listener's head. This work proposes a multi-channel speech enhancement algorithm which utilises the adaptability of signal-dependent beamformers while still benefiting from the computational efficiency and robust performance of signal-independent super-directive beamformers. The algorithm has two stages. (i) The first stage is a hybrid beamformer based on a dictionary of weights corresponding to a set of noise field models. (ii) The second stage is a wide-band subspace post-filter to remove any artifacts resulting from (i). The algorithm is evaluated using both real-world recordings and simulations of a cocktail-party scenario. Noise suppression, intelligibility and speech quality results show a significant performance improvement by the proposed algorithm compared to the baseline super-directive beamformer. A data-driven implementation of the noise field dictionary is shown to provide more noise suppression, and similar speech intelligibility and quality, compared to a parametric dictionary.

【5】 Audio Prompt Tuning for Universal Sound Separation标题：通用声音分离的音频提示调谐链接：https://arxiv.org/abs/2311.18399

作者：Yuzhuo Liu,Xubo Liu,Yan Zhao,Yuanyuan Wang,Rui Xia,Pingchuan Tain,Yuxuan Wang摘要：通用声音分离（USS）是从音频混合中分离任意声音的任务。现有的USS系统能够分离任意源，给出了目标源作为查询的几个示例。然而，用单个系统分离任意声音是具有挑战性的，并且鲁棒性并不总是得到保证。在这项工作中，我们提出了音频提示调谐（APT），一个简单而有效的方法来提高现有的USS系统。具体而言，APT通过用有限的音频样本训练少量提示参数来提高特定源的分离性能，同时通过保持USS模型的参数冻结来保持USS模型的泛化。我们在MUSDB 18和ESC-50数据集上评估了所提出的方法。与基线模型相比，使用两个数据集的全训练集，APT可以将信号失真比性能提高0.67 dB和2.06 dB。此外，只有5个音频样本的APT甚至优于使用ESC-50数据集上的完整训练数据的基线系统，表明Few-Shot APT的巨大潜力。摘要：Universal sound separation (USS) is a task to separate arbitrary sounds from an audio mixture. Existing USS systems are capable of separating arbitrary sources, given a few examples of the target sources as queries. However, separating arbitrary sounds with a single system is challenging, and the robustness is not always guaranteed. In this work, we propose audio prompt tuning (APT), a simple yet effective approach to enhance existing USS systems. Specifically, APT improves the separation performance of specific sources through training a small number of prompt parameters with limited audio samples, while maintaining the generalization of the USS model by keeping its parameters frozen. We evaluate the proposed method on MUSDB18 and ESC-50 datasets. Compared with the baseline model, APT can improve the signal-to-distortion ratio performance by 0.67 dB and 2.06 dB using the full training set of two datasets. Moreover, APT with only 5 audio samples even outperforms the baseline systems utilizing full training data on the ESC-50 dataset, indicating the great potential of few-shot APT.

eess.AS音频处理

【1】 An Aliasing-Free Hybrid Digital-Analog Polyphonic Synthesizer标题：一种无混叠的数模混合复调合成器链接：https://arxiv.org/abs/2311.18774

作者：Jonas Roth,Domenic Keller,Oscar Castañeda,Christoph Studer备注：Presented at DAFx23摘要：与数字仿真相比，模拟减法合成器通常被认为提供更好的音质。然而，模拟电路需要校准，并且在生成各种波形时遭受老化、温度不稳定和有限的灵活性。数字合成可以缓解许多这些缺点，但生成任意无混叠波形仍然具有挑战性。在本文中，我们提出了+合成器，一个混合数字模拟八语音合成器的原型，结合了两个世界的最佳。该合成器的核心是大傅立叶振荡器（BFO），这是一种新型的数字超大规模集成电路（VLSI）设计，利用加法合成来生成各种无混叠波形。每个BFO产生两个声音，每个声音使用四个振荡器。单个振荡器可以产生多达1024个可自由配置的分音（谐波或非谐波），这些分音是使用坐标旋转数字计算机（CORDIC）计算的。BFO制造为65 nm CMOS定制专用集成电路（ASIC），集成在+-synth中，同时生成多达32768个分音。然后，四个24位96 kHz立体声DAC将八个声音转换为模拟域，然后进行数字控制的模拟低通滤波和放大。+-synth原型的测量结果表明高保真度和低延迟。摘要：Analog subtractive synthesizers are generally considered to provide superior sound quality compared to digital emulations. However, analog circuitry requires calibration and suffers from aging, temperature instability, and limited flexibility in generating a wide variety of waveforms. Digital synthesis can mitigate many of these drawbacks, but generating arbitrary aliasing-free waveforms remains challenging. In this paper, we present the +-synth, a hybrid digital-analog eight-voice polyphonic synthesizer prototype that combines the best of both worlds. At the heart of the synthesizer is the big Fourier oscillator (BFO), a novel digital very-large scale integration (VLSI) design that utilizes additive synthesis to generate a wide variety of aliasing-free waveforms. Each BFO produces two voices, using four oscillators per voice. A single oscillator can generate up to 1024 freely configurable partials (harmonic or inharmonic), which are calculated using coordinate rotation digital computers (CORDICs). The BFOs were fabricated as 65nm CMOS custom application-specific integrated circuits (ASICs), which are integrated in the +-synth to simultaneously generate up to 32768 partials. Four 24-bit 96kHz stereo DACs then convert the eight voices into the analog domain, followed by digitally controlled analog low-pass filtering and amplification. Measurement results of the +-synth prototype demonstrate high fidelity and low latency.

【2】 Subspace Hybrid MVDR Beamforming for Augmented Hearing标题：用于增强听力的子空间混合MVDR波束形成链接：https://arxiv.org/abs/2311.18689

【3】 Audio Prompt Tuning for Universal Sound Separation标题：通用声音分离的音频提示调谐链接：https://arxiv.org/abs/2311.18399

【4】 Leveraging cache to enable SLU on tiny devices标题：利用缓存在微型设备上启用SLU链接：https://arxiv.org/abs/2311.18188

作者：Afsara Benazir,Zhiming Xu,Felix Xiaozhu Lin备注：submitted to Mobisys 2024摘要：None摘要：This paper addresses spoken language understanding (SLU) on microcontroller-like embedded devices, integrating on-device execution with cloud offloading in a novel fashion. We exploit temporal locality in a device's speech inputs and accordingly reuse recent SLU inferences. Our idea is simple: let the device match new inputs against cached results, and only offload unmatched inputs to the cloud for full inference. Realization of this idea, however, is non-trivial: the device needs to compare acoustic features in a robust, low-cost way. To this end, we present XYZ, a speech cache for tiny devices. It matches speech inputs at two levels of representations: first by clustered sequences of raw sound units, then as sequences of phonemes. Working in tandem, the two representations offer complementary cost/accuracy tradeoffs. To further boost accuracy, our cache is learning: with the mismatched and then offloaded inputs, it continuously finetunes the device's feature extractors (with the assistance of the cloud). We implement XYZ on an off-the-shelf STM32 microcontroller. The resultant implementation has a small memory footprint of 2MB. Evaluated on challenging speech benchmarks, our system resolves 45%--90% of inputs on device, reducing the average latency by up to 80% compared to offloading to popular cloud speech services. Our benefit is pronounced even in adversarial settings -- noisy environments, cold cache, or one device shared by a number of users.

【5】 Enhanced Generative Adversarial Networks for Unseen Word Generation from EEG Signals标题：用于从脑电信号生成未见词的增强型生成性对抗网络链接：https://arxiv.org/abs/2311.17923

作者：Young-Eun Lee,Seo-Hyun Lee,Soowon Kim,Jung-Sun Lee,Deok-Seon Kim,Seong-Whan Lee备注：5 pages, 2 figures摘要：脑-机接口（BCI）技术的最新进展，特别是基于生成对抗网络（GAN）的脑-机接口（BCI）技术的最新进展，已经显示出改善BCI解码性能的巨大前景。在脑机接口（BCI）领域，GANs在许多领域都有应用。它们是数据增强的宝贵工具，可以解决有限数据可用性和综合的挑战，有效地扩展数据集并创建新的数据格式，从而增强BCI系统的鲁棒性和适应性。与言语有关的范例的研究已经大大扩展，对辅助技术的进步和对言语障碍者的沟通支持产生了重要影响。在这项研究中，研究了GANs，特别是BCI领域，并应用于从EEG信号生成文本。GANs可以概括所有主题并解码看不见的单词，这表明它能够捕获不同个体之间一致的潜在语音模式。该方法在基于神经信号的语音识别系统和语音障碍者的通信辅助中具有实际应用。摘要：Recent advances in brain-computer interface (BCI) technology, particularly based on generative adversarial networks (GAN), have shown great promise for improving decoding performance for BCI. Within the realm of Brain-Computer Interfaces (BCI), GANs find application in addressing many areas. They serve as a valuable tool for data augmentation, which can solve the challenge of limited data availability, and synthesis, effectively expanding the dataset and creating novel data formats, thus enhancing the robustness and adaptability of BCI systems. Research in speech-related paradigms has significantly expanded, with a critical impact on the advancement of assistive technologies and communication support for individuals with speech impairments. In this study, GANs were investigated, particularly for the BCI field, and applied to generate text from EEG signals. The GANs could generalize all subjects and decode unseen words, indicating its ability to capture underlying speech patterns consistent across different individuals. The method has practical applications in neural signal-based speech recognition systems and communication aids for individuals with speech difficulties.

【6】 CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation标题：CoDI-2：情景中、交错和交互式任意对任意一代链接：https://arxiv.org/abs/2311.18775

【7】 Barwise Music Structure Analysis with the Correlation Block-Matching Segmentation Algorithm标题：基于相关块匹配分割算法的巴氏音乐结构分析链接：https://arxiv.org/abs/2311.18604

【8】 String Sound Synthesizer on GPU-accelerated Finite Difference Scheme标题：基于GPU加速有限差分格式的弦声合成器链接：https://arxiv.org/abs/2311.18505

【9】 Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications标题：概率语音驱动的3D人脸运动合成：新的基准、方法和应用链接：https://arxiv.org/abs/2311.18168

作者：Karren D. Yang,Anurag Ranjan,Jen-Hao Rick Chang,Raviteja Vemulapalli,Oncel Tuzel摘要：我们认为，从语音信号的三维人脸几何动画的任务。现有的工作主要是确定性的，专注于学习一个一对一的映射，从语音信号到3D人脸网格上的小数据集有限的扬声器。虽然这些模型可以为训练集中的说话者实现高质量的嘴唇清晰度，但它们无法捕捉真实世界中伴随语音的3D面部运动的完整和多样化分布。重要的是，语音和面部运动之间的关系是一对多，包含说话者之间和说话者内的变化，需要一个概率的方法。在本文中，我们确定并解决了迄今为止限制概率模型发展的关键挑战：缺乏适合于训练和评估它们的数据集和指标，以及设计一个模型的困难，该模型生成不同的结果，同时保持忠实于一个强大的条件信号，如语音。我们首先提出了适合于概率建模的大规模基准数据集和度量。然后，我们展示了一个概率模型，实现了多样性和保真度的语音，优于其他方法在提出的基准。最后，我们展示了在这些大规模数据集上训练的概率模型的有用应用：我们可以生成各种语音驱动的3D面部运动，这些运动与从参考剪辑中提取的看不见的说话者风格相匹配;我们的合成网格可用于提高下游视听模型的性能。摘要：We consider the task of animating 3D facial geometry from speech signal. Existing works are primarily deterministic, focusing on learning a one-to-one mapping from speech signal to 3D face meshes on small datasets with limited speakers. While these models can achieve high-quality lip articulation for speakers in the training set, they are unable to capture the full and diverse distribution of 3D facial motions that accompany speech in the real world. Importantly, the relationship between speech and facial motion is one-to-many, containing both inter-speaker and intra-speaker variations and necessitating a probabilistic approach. In this paper, we identify and address key challenges that have so far limited the development of probabilistic models: lack of datasets and metrics that are suitable for training and evaluating them, as well as the difficulty of designing a model that generates diverse results while remaining faithful to a strong conditioning signal as speech. We first propose large-scale benchmark datasets and metrics suitable for probabilistic modeling. Then, we demonstrate a probabilistic model that achieves both diversity and fidelity to speech, outperforming other methods across the proposed benchmarks. Finally, we showcase useful applications of probabilistic models trained on these large-scale datasets: we can generate diverse speech-driven 3D facial motion that matches unseen speaker styles extracted from reference clips; and our synthetic meshes can be used to improve the performance of downstream audio-visual models.

机器翻译由腾讯交互翻译提供，仅供参考

语音/音频处理学术速递[12.1]

评论 0