语音/音频处理学术速递[8.15]

今日论文合集：cs.SD语音18篇，eess.AS音频处理19篇。

cs.SD语音

【1】 DiffSED: Sound Event Detection with Denoising Diffusion标题：DiffSED：带去噪扩散的声事件检测链接：https://arxiv.org/abs/2308.07293

作者：Swapnil Bhosale,Sauradip Nag,Diptesh Kanojia,Jiankang Deng,Xiatian Zhu机构：University of Surrey, UK, Imperial College London, UK摘要：声音事件检测（SED）的目的是在给定不受约束的音频样本的情况下预测所有感兴趣的事件及其类别标签的时间边界。采用分裂-分类（即，帧级）策略或更有原则的事件级建模方法，所有现有方法都从判别式学习的角度考虑SED问题。在这项工作中，我们重新制定的SED问题，采取生成学习的角度来看。具体而言，我们的目标是从噪声的建议，在去噪扩散过程中，以目标音频样本为条件，生成声音的时间边界。在训练过程中，我们的模型通过将嘈杂的潜在查询转换为优雅的Transformer解码器框架中的地面实况版本来学习逆转噪声过程。这样做使模型能够在推理期间从甚至是嘈杂的查询中生成准确的事件边界。在Urban-SED和EPIC-Sounds数据集上进行的大量实验表明，我们的模型明显优于现有的替代方案，在训练中收敛速度快了40%以上。摘要：Sound Event Detection (SED) aims to predict the temporal boundaries of all the events of interest and their class labels, given an unconstrained audio sample. Taking either the splitand-classify (i.e., frame-level) strategy or the more principled event-level modeling approach, all existing methods consider the SED problem from the discriminative learning perspective. In this work, we reformulate the SED problem by taking a generative learning perspective. Specifically, we aim to generate sound temporal boundaries from noisy proposals in a denoising diffusion process, conditioned on a target audio sample. During training, our model learns to reverse the noising process by converting noisy latent queries to the groundtruth versions in the elegant Transformer decoder framework. Doing so enables the model generate accurate event boundaries from even noisy queries during inference. Extensive experiments on the Urban-SED and EPIC-Sounds datasets demonstrate that our model significantly outperforms existing alternatives, with 40+% faster convergence in training.

【2】 AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes标题：AudioFormer：音频转换器从离散的声学代码中学习音频特征表示链接：https://arxiv.org/abs/2308.07221

作者：Zhaohui Li,Haitao Wang,Xinghua Jiang机构：YiWise Inc. ,Xidian University备注：9 pages, 4 figures摘要：我们提出了一种名为AudioFormer的方法，该方法通过获取离散声学代码来学习音频特征表示，并随后对它们进行微调以用于音频分类任务。最初，我们引入了一个新的视角，将音频分类任务视为自然语言理解（NLU）的一种形式。利用现有的神经音频编解码器模型，我们生成离散的声学代码，并利用它们来训练掩蔽语言模型（MLM），从而获得音频特征表示。此外，我们开创了\textbf{M}ulti-\textbf{P}主动样本\textbf{C}对比（MPC）学习方法的集成。该方法使得能够学习相同音频输入内的多个离散声学代码之间的联合表示。在我们的实验中，我们将离散的声学代码视为文本数据，并使用类似完形填空的方法来训练掩蔽的语言模型，最终获得高质量的音频表示。值得注意的是，MPC学习技术有效地捕获不同的正样本之间的协作表示。我们的研究结果表明，AudioFormer在多个数据集上与当前的单模态音频分类模型相比，性能得到了显着提高，甚至在选定的数据集上优于视听多模态分类模型。具体来说，我们的方法在包括AudioSet（2 M，20 K）和FSD 50 K在内的数据集上取得了显着的结果，性能得分分别为53.9，45.1和65.6。我们公开分享了代码和模型：\url{https：//github.com/LZH-0225/AudioFormer.git}.摘要：We propose a method named AudioFormer, which learns audio feature representations through the acquisition of discrete acoustic codes and subsequently fine-tunes them for audio classification tasks. Initially, we introduce a novel perspective by considering the audio classification task as a form of natural language understanding (NLU). Leveraging an existing neural audio codec model, we generate discrete acoustic codes and utilize them to train a masked language model (MLM), thereby obtaining audio feature representations. Furthermore, we pioneer the integration of a \textbf{M}ulti-\textbf{P}ositive sample \textbf{C}ontrastive (MPC) learning approach. This method enables the learning of joint representations among multiple discrete acoustic codes within the same audio input. In our experiments, we treat discrete acoustic codes as textual data and train a masked language model using a cloze-like methodology, ultimately deriving high-quality audio representations. Notably, the MPC learning technique effectively captures collaborative representations among distinct positive samples. Our research outcomes demonstrate that AudioFormer attains significantly improved performance compared to prevailing monomodal audio classification models across multiple datasets, and even outperforms audio-visual multimodal classification models on select datasets. Specifically, our approach achieves remarkable results on datasets including AudioSet (2M, 20K), and FSD50K, with performance scores of 53.9, 45.1, and 65.6, respectively. We have openly shared both the code and models: \url{https://github.com/LZH-0225/AudioFormer.git}.

【3】 PitchNet: A Fully Convolutional Neural Network for Pitch Estimation标题：PitchNet：一种用于基音估计的全卷积神经网络链接：https://arxiv.org/abs/2308.07170

作者：Jeremy Cochoy摘要：在音乐和声音处理领域，基音提取起着举足轻重的作用。本研究介绍了“PitchNet”，这是一种卷积神经网络，用于从人类歌唱声音中提取音高，包括无伴奏合唱表演。PitchNet将自相关与深度学习技术相结合，旨在优化基音检测的准确性。包括合成声音，歌剧录音和时间拉伸元音的数据集的评估证明了其有效性。这项工作为在音乐和语音设置中增强音高提取铺平了道路。摘要：In the domain of music and sound processing, pitch extraction plays a pivotal role. This research introduces "PitchNet", a convolutional neural network tailored for pitch extraction from the human singing voice, including acapella performances. Integrating autocorrelation with deep learning techniques, PitchNet aims to optimize the accuracy of pitch detection. Evaluation across datasets comprising synthetic sounds, opera recordings, and time-stretched vowels demonstrates its efficacy. This work paves the way for enhanced pitch extraction in both music and voice settings.

【4】 Active Bird2Vec: Towards End-to-End Bird Sound Monitoring with Transformers标题：Active Bird2Vec：用Transformer实现端到端的鸟声监测链接：https://arxiv.org/abs/2308.07121

作者：Lukas Rauch,Raphael Schwinger,Moritz Wirth,Bernhard Sick,Sven Tomforde,Christoph Scholz机构：IES, University of Kassel, IS, University of Kiel, IEE, Fraunhofer-Institut备注：Accepted @AI4S ECAI2023. This is the author's version of the work摘要：我们建议通过结合自监督（SSL）和深度主动学习（DAL），在鸟类声音监测中转向端到端学习。利用Transformer模型，我们的目标是绕过传统的频谱图转换，实现直接的原始音频处理。ActiveBird2Vec将通过SSL生成高质量的鸟类声音表示，这可能会加速对风电场环境变化和决策过程的评估。此外，我们试图通过DAL利用各种各样的鸟类发声，减少对人类专家广泛标记的数据集的依赖。我们计划通过Huggingface数据集策划一系列全面的任务，提高未来生物声学研究的可比性和可重复性。将进行各种Transformer模型之间的比较分析，以评估它们在鸟声识别任务中的熟练程度。我们的目标是加速鸟类生物声学研究的进展，并为更有效的保护策略做出贡献。摘要：We propose a shift towards end-to-end learning in bird sound monitoring by combining self-supervised (SSL) and deep active learning (DAL). Leveraging transformer models, we aim to bypass traditional spectrogram conversions, enabling direct raw audio processing. ActiveBird2Vec is set to generate high-quality bird sound representations through SSL, potentially accelerating the assessment of environmental changes and decision-making processes for wind farms. Additionally, we seek to utilize the wide variety of bird vocalizations through DAL, reducing the reliance on extensively labeled datasets by human experts. We plan to curate a comprehensive set of tasks through Huggingface Datasets, enhancing future comparability and reproducibility of bioacoustic research. A comparative analysis between various transformer models will be conducted to evaluate their proficiency in bird sound recognition tasks. We aim to accelerate the progression of avian bioacoustic research and contribute to more effective conservation strategies.

【5】 iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN标题：ISTFTNet2：使用1D-2D CNN的更快、更轻量级的基于iSTFT的神经声码器链接：https://arxiv.org/abs/2308.07117

作者：Takuhiro Kaneko,Hirokazu Kameoka,Kou Tanaka,Shogo Seki机构：NTT Communication Science Laboratories, NTT Corporation, Japan备注：Accepted to Interspeech 2023. Project page: this https URL摘要：逆短时傅立叶变换网络（iSTFTNet）由于其快速、轻量和高保真的语音合成而引起关注。它使用快速且轻量级的1D CNN作为主干并使用iSTFT替换一些神经过程来获得这些特征。由于一维CNN难以建模高维频谱图，频率维度通过时间上采样来降低。然而，这种策略损害了提高速度的潜力。因此，我们提出了iSTFTNet 2，iSTFTNet的改进变体，具有1D-2D CNN，采用1D和2D CNN来建模时间和频谱图结构，分别。我们设计了一个2D CNN，执行频率上采样后转换在一个少频率空间。这种设计有利于在不影响速度的情况下对高维谱图进行建模。结果表明，iSTFTNet 2使iSTFTNet更快，更轻便，语音质量相当。音频样本可在https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet2/上获得。摘要：The inverse short-time Fourier transform network (iSTFTNet) has garnered attention owing to its fast, lightweight, and high-fidelity speech synthesis. It obtains these characteristics using a fast and lightweight 1D CNN as the backbone and replacing some neural processes with iSTFT. Owing to the difficulty of a 1D CNN to model high-dimensional spectrograms, the frequency dimension is reduced via temporal upsampling. However, this strategy compromises the potential to enhance the speed. Therefore, we propose iSTFTNet2, an improved variant of iSTFTNet with a 1D-2D CNN that employs 1D and 2D CNNs to model temporal and spectrogram structures, respectively. We designed a 2D CNN that performs frequency upsampling after conversion in a few-frequency space. This design facilitates the modeling of high-dimensional spectrograms without compromising the speed. The results demonstrated that iSTFTNet2 made iSTFTNet faster and more lightweight with comparable speech quality. Audio samples are available at https://www.kecl.ntt.co.jp/people/kaneko.takuhiro/projects/istftnet2/.

【6】 BigWavGAN: A Wave-To-Wave Generative Adversarial Network for Music Super-Resolution标题：BigWavGan：一种针对音乐超分辨率的波对波生成性对抗网络链接：https://arxiv.org/abs/2308.06483

作者：Yenan Zhang,Hiroshi Watanabe机构：Graduate School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan摘要：通常，深度神经网络（DNN）在其模型大小较大时被期望具有高性能。然而，大型模型未能产生与其在音乐超分辨率（SR）中的规模相称的高质量结果。我们将此归因于DNN无法从标准均方误差损失中学习与其大小相称的信息。为了释放大型DNN模型在音乐SR中的潜力，我们提出了BigWavGAN，它结合了Demucs，一个大规模的波到波模型，具有最先进的（SOTA）鉴别器和对抗训练策略。我们的鉴别器包括多尺度鉴别器（MSD）和多分辨率鉴别器（MRD）。在推断期间，由于仅利用生成器，因此与基线模型Demucs相比，不需要额外的参数或计算资源。客观评价肯定了BigWavGAN在音乐SR中的有效性。主观评估表明，BigWavGAN可以生成比基线模型具有显著高感知质量的音乐。值得注意的是，BigWavGAN在模拟和现实场景中都超过了SOTA音乐SR模型。此外，BigWavGAN代表了其解决分布外数据的卓越泛化能力。进行的消融研究揭示了我们的鉴别器和训练策略的重要性。演示页面上提供了示例。摘要：Generally, Deep Neural Networks (DNNs) are expected to have high performance when their model size is large. However, large models failed to produce high-quality results commensurate with their scale in music Super-Resolution (SR). We attribute this to that DNNs cannot learn information commensurate with their size from standard mean square error losses. To unleash the potential of large DNN models in music SR, we propose BigWavGAN, which incorporates Demucs, a large-scale wave-to-wave model, with State-Of-The-Art (SOTA) discriminators and adversarial training strategies. Our discriminator consists of Multi-Scale Discriminator (MSD) and Multi-Resolution Discriminator (MRD). During inference, since only the generator is utilized, there are no additional parameters or computational resources required compared to the baseline model Demucs. Objective evaluation affirms the effectiveness of BigWavGAN in music SR. Subjective evaluations indicate that BigWavGAN can generate music with significantly high perceptual quality over the baseline model. Notably, BigWavGAN surpasses the SOTA music SR model in both simulated and real-world scenarios. Moreover, BigWavGAN represents its superior generalization ability to address out-of-distribution data. The conducted ablation study reveals the importance of our discriminators and training strategies. Samples are available on the demo page.

【7】 Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding标题：基于同构音文嵌入的柔性关键词检测链接：https://arxiv.org/abs/2308.06472

作者：Kumari Nishu,Minsik Cho,Paul Dixon,Devang Naik机构：Apple摘要：发现在文本中表示的用户定义的/灵活的关键字经常使用昂贵的文本编码器与嵌入空间中的音频编码器进行联合分析，这可能遭受异构模态表示（即，大的失配）和增加的复杂性。在这项工作中，我们提出了一种新的架构，以有效地检测任意关键字的音频兼容的文本编码器的基础上，固有地具有同质的表示与音频嵌入，它也是比兼容的文本编码器小得多。我们的文本编码器将文本转换为音素使用的字形到音素（G2 P）模型，然后使用代表性的音素向量，提取从丰富的语音数据集的配对音频编码器的嵌入。我们进一步增加我们的方法与混淆的关键字生成开发一个音频文本嵌入验证器具有很强的辨别力。实验结果表明，我们的计划优于国家的最先进的结果Librphrase硬数据集，增加ROC曲线下面积（AUC）指标从84.21%到92.7%，降低等错误率（EER）指标从23.36%到14.4%。摘要：Spotting user-defined/flexible keywords represented in text frequently uses an expensive text encoder for joint analysis with an audio encoder in an embedding space, which can suffer from heterogeneous modality representation (i.e., large mismatch) and increased complexity. In this work, we propose a novel architecture to efficiently detect arbitrary keywords based on an audio-compliant text encoder which inherently has homogeneous representation with audio embedding, and it is also much smaller than a compatible text encoder. Our text encoder converts the text to phonemes using a grapheme-to-phoneme (G2P) model, and then to an embedding using representative phoneme vectors, extracted from the paired audio encoder on rich speech datasets. We further augment our method with confusable keyword generation to develop an audio-text embedding verifier with strong discriminative power. Experimental results show that our scheme outperforms the state-of-the-art results on Libriphrase hard dataset, increasing Area Under the ROC Curve (AUC) metric from 84.21% to 92.7% and reducing Equal-Error-Rate (EER) metric from 23.36% to 14.4%.

【8】 Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion标题：音素幻想器：通过集合扩展实现一次语音转换链接：https://arxiv.org/abs/2308.06382

作者：Siyuan Shan,Yang Li,Amartya Banerjee,Junier B. Oliva机构：Department of Computer Science, UNC-Chapel Hill, NC, USA备注：under review摘要：语音转换（VC）的目的是改变一个人的声音，使其听起来类似于另一个人的声音，同时保留语言内容。现有方法在内容可懂度和说话者相似性之间陷入困境;也就是说，具有较高可懂度的方法通常具有较低的说话人相似性，而具有较高说话人相似性的方法通常需要大量的目标说话人语音数据来实现高可懂度。在这项工作中，我们提出了一种新的方法\textit{Phoneme Hallucinator}，实现了两全其美。Phoneme Hallucinator是一个一次性VC模型;它采用了一种新的模型来仅基于短的目标说话人语音（例如，3秒）。然后利用产生幻觉的音素来执行基于邻居的语音转换。我们的模型是一个无文本的，任何到任何VC模型，不需要文本注释，并支持转换到任何看不见的扬声器。客观和主观评估表明\texit {Phoneme Hallucinator}在可懂度和说话人相似度方面优于现有VC方法。摘要：Voice conversion (VC) aims at altering a person's voice to make it sound similar to the voice of another person while preserving linguistic content. Existing methods suffer from a dilemma between content intelligibility and speaker similarity; i.e., methods with higher intelligibility usually have a lower speaker similarity, while methods with higher speaker similarity usually require plenty of target speaker voice data to achieve high intelligibility. In this work, we propose a novel method \textit{Phoneme Hallucinator} that achieves the best of both worlds. Phoneme Hallucinator is a one-shot VC model; it adopts a novel model to hallucinate diversified and high-fidelity target speaker phonemes based just on a short target speaker voice (e.g. 3 seconds). The hallucinated phonemes are then exploited to perform neighbor-based voice conversion. Our model is a text-free, any-to-any VC model that requires no text annotations and supports conversion to any unseen speaker. Objective and subjective evaluations show that \textit{Phoneme Hallucinator} outperforms existing VC methods for both intelligibility and speaker similarity.

【9】 Localization of DOA trajectories -- Beyond the grid标题：定位DOA轨迹--超越网格链接：https://arxiv.org/abs/2308.07265

作者：Ruchi Pandey,Santosh Nannuru机构：Signal Processing and Communications Research Center, IIIT Hyderabad, India摘要：波达方向（DOA）估计算法是声源定位的关键。传统的定位方法依赖于块级处理来从一起处理的多个测量中提取方向信息。然而，这些方法假设DOA在整个块中保持恒定，这在实际场景中可能不是真的。此外，当真实参数不位于参数搜索网格上时，定位方法的性能受到限制。在本文中，我们提出了两个轨迹模型，即多项式和带限轨迹模型，捕获的DOA动态。为了估计轨迹参数，我们采用两种无网格算法：i）滑动Frank-Wolfe（SFW），其解决了Beurling LASSO问题，以及ii）牛顿化正交匹配追踪（NOMP），其使用循环细化来改进OMP。此外，我们扩展我们的分析，包括宽带处理。仿真结果表明，所提出的轨迹定位算法表现出更好的性能相比，基于网格的方法在分辨率，鲁棒性噪声，和计算效率。摘要：The direction of arrival (DOA) estimation algorithms are crucial in localizing acoustic sources. Traditional localization methods rely on block-level processing to extract the directional information from multiple measurements processed together. However, these methods assume that DOA remains constant throughout the block, which may not be true in practical scenarios. Also, the performance of localization methods is limited when the true parameters do not lie on the parameter search grid. In this paper we propose two trajectory models, namely the polynomial and bandlimited trajectory models, to capture the DOA dynamics. To estimate trajectory parameters, we adopt two gridless algorithms: i) Sliding Frank-Wolfe (SFW), which solves the Beurling LASSO problem and ii) Newtonized Orthogonal Matching Pursuit (NOMP), which improves over OMP using cyclic refinement. Furthermore, we extend our analysis to include wideband processing. The simulation results indicate that the proposed trajectory localization algorithms exhibit improved performance compared to grid-based methods in terms of resolution, robustness to noise, and computational efficiency.

【10】 Compositional nonlinear audio signal processing with Volterra series标题：基于Volterra级数的合成非线性音频信号处理链接：https://arxiv.org/abs/2308.07229

作者：Jake Araujo-Simon机构：Department of Music Technology, Montr´eal, Canada, A thesis submitted to McGill University in partial fulfillment of requirements for the degree of, Masters of Arts., arXiv:,.,v, [eess.SP] , Aug备注：Master's thesis摘要：我们开发了一个非线性音频信号处理的Volterra级数的分类的基础上的合成理论。我们增加的经典定义的Volterra系列是函的相对于一个基本范畴，其对象是温带分布和其态射是某些线性变换。这导致公式描述的结果，非线性变换的影响，如果他们的输入信号首先进行线性处理。然后，我们考虑如何非线性音频系统的变化，并介绍作为一个模型的概念，态射的Volterra系列。我们展示了如何态射可以被参数化，并用于生成Volterra级数，这是非常适合模型非平稳或随时间变化的非线性现象的索引家庭。我们描述了如何Volterra系列和它们的态射组织成一个函子范畴，伏特，其对象是Volterra系列和其态射是自然变换。我们展示的操作的总和，产品，和一系列的组成Volterra系列monoidal产品上伏，并确定，每个反过来，其相应的普遍属性。我们表明，特别是，Volterra级数的系列组成是结合的。然后，我们在我们的框架和音频信号处理的核心主题之间建立桥梁：时频分析具体而言，我们证明了一类二阶Volterra级数和双线性时频分布（TFD）之间的等价性可以扩展到某些高阶Volterra级数和所谓的多项式TFD之间。最后，我们展望了未来的工作，包括非线性系统识别技术和扩展我们的理论的组成图和拓扑音频信号处理的设置。摘要：We develop a compositional theory of nonlinear audio signal processing based on a categorification of the Volterra series. We augment the classical definition of the Volterra series to be functorial with respect to a base category whose objects are temperate distributions and whose morphisms are certain linear transformations. This leads to formulae describing how the outcomes of nonlinear transformations are affected if their input signals are first linearly processed. We then consider how nonlinear audio systems change, and introduce as a model thereof the notion of morphism of Volterra series. We show how morphisms can be parameterized and used to generate indexed families of Volterra series, which are well-suited to model nonstationary or time-varying nonlinear phenomena. We describe how Volterra series and their morphisms organize into a functor category, Volt, whose objects are Volterra series and whose morphisms are natural transformations. We exhibit the operations of sum, product, and series composition of Volterra series as monoidal products on Volt and identify, for each in turn, its corresponding universal property. We show, in particular, that the series composition of Volterra series is associative. We then bridge between our framework and a subject at the heart of audio signal processing: time-frequency analysis. Specifically, we show that an equivalence between a certain class of second-order Volterra series and the bilinear time-frequency distributions (TFDs) can be extended to one between certain higher-order Volterra series and the so-called polynomial TFDs. We end with prospects for future work, including the incorporation of nonlinear system identification techniques and the extension of our theory to the settings of compositional graph and topological audio signal processing.

【11】 Integrating Emotion Recognition with Speech Recognition and Speaker Diarisation for Conversations标题：将情感识别与语音识别和说话人识别相结合用于对话链接：https://arxiv.org/abs/2308.07145

作者：Wen Wu,Chao Zhang,Philip C. Woodland机构：Department of Engineering, University of Cambridge, Cambridge, UK, Department of Electrical Engineering, Tsinghua University, Beijing, China备注：Interspeech 2023摘要：虽然自动情感识别（AER）最近引起了显着的研究兴趣，大多数目前的AER研究使用手动分割的话语，这通常是不可用的对话系统。本文提出了将AER与自动语音识别（ASR）和扬声器diarisation（SD）在一个联合训练的系统。基于共享编码器，为包括AER、ASR、语音活动检测和说话人分类的四个子任务构建不同的输出层。以对话的音频作为输入，集成系统找到所有的语音段，并转录相应的情感类，单词序列，和扬声器的身份。提出了两个指标来评估AER性能与自动分割的基础上，时间加权的情感和说话人分类错误。IEMOCAP数据集上的结果表明，该系统始终优于两个基线分别训练的单任务系统的AER，ASR和SD。摘要：Although automatic emotion recognition (AER) has recently drawn significant research interest, most current AER studies use manually segmented utterances, which are usually unavailable for dialogue systems. This paper proposes integrating AER with automatic speech recognition (ASR) and speaker diarisation (SD) in a jointly-trained system. Distinct output layers are built for four sub-tasks including AER, ASR, voice activity detection and speaker classification based on a shared encoder. Taking the audio of a conversation as input, the integrated system finds all speech segments and transcribes the corresponding emotion classes, word sequences, and speaker identities. Two metrics are proposed to evaluate AER performance with automatic segmentation based on time-weighted emotion and speaker classification errors. Results on the IEMOCAP dataset show that the proposed system consistently outperforms two baselines with separately trained single-task systems on AER, ASR and SD.

【12】 VoxSnap: X-Large Speaker Verification Dataset on Camera标题：VoxSnap：摄像机上的X-大型说话人验证数据集链接：https://arxiv.org/abs/2308.07056

作者：Yuke Lin,Xiaoyi Qin,Ming Cheng,Ning Jiang,Guoqing Zhao,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China, Mashang Consumer Finance Co., Ltd备注：submit to ICASSP2023摘要：在本文中，我们贡献了一个新的和广泛的数据集说话人验证，其中包含嘈杂的38 k身份/1.45M话语（VoxSnap）和相对清洁的18 k身份/1.02M（VoxSnap-Clean）话语训练。首先，我们收集了60 K+用户的名单以及他们的头像，并在YouTube上下载他们的短视频。然后，设计了一个自动管道来提取目标用户的语音片段和视频，这是高效和可扩展的。据我们所知，VoxSnap数据集是最大的说话人识别数据集。其次，我们开发了一系列的实验，基于VoxSnap-clean和VoxCeleb 2。我们的研究结果突出了性能的显着提高，范围从15%到30%，跨不同的骨干架构，集成我们的数据集进行训练。数据集将很快发布。摘要：In this paper, we contribute a novel and extensive dataset for speaker verification, which contains noisy 38k identities/1.45M utterances (VoxSnap) and relatively cleaned 18k identities/1.02M (VoxSnap-Clean) utterances for training. Firstly, we collect a 60K+ users' list as well as their avatar and download their SHORT videos on the YouTube. Then, an automatically pipeline is devised to extract target user's speech segments and videos, which is efficient and scalable. To the best of our knowledge, the VoxSnap dataset is the largest speaker recognition dataset. Secondly, we develop a series of experiments based on VoxSnap-clean together with VoxCeleb2. Our findings highlight a notable improvement in performance, ranging from 15% to 30%, across different backbone architectures, upon integrating our dataset for training. The dataset will be released SOON~.

【13】 The Sound Demixing Challenge 2023 $\unicode{x2013}$ Cinematic Demixing Track链接：https://arxiv.org/abs/2308.06981

作者：Stefan Uhlich,Giorgio Fabbro,Masato Hirano,Shusuke Takahashi,Gordon Wichern,Jonathan Le Roux,Dipam Chakraborty,Sharada Mohanty,Kai Li,Yi Luo,Jianwei Yu,Rongzhi Gu,Roman Solovyev,Alexander Stempkovskiy,Tatiana Habruseva,Mikhail Sukhovei,Yuki Mitsufuji机构：ChinaDepartment of Computer Science and Technology, ††Institute for Design Problems in Microelectronics of RussianAcademy of Sciences备注：under review摘要：本文总结了Sound Demixing Challenge 2023（SDX'23）的电影混音（CDX）轨道。我们提供了挑战设置的全面总结，详细介绍了比赛的结构和使用的数据集。特别是，我们详细介绍了CDXDB23，一个新的隐藏数据集，从真实的电影，用于排名的意见书。该文件还提供了最成功的参与者所采用的方法的见解。与鸡尾酒餐叉基线相比，仅在模拟的Divide and Remaster（DnR）数据集上训练的性能最好的系统在SDR中实现了1.8dB的改进，而在开放排行榜上表现最好的系统，其中任何数据都可以用于训练，看到了5.7dB的显着改进。摘要：This paper summarizes the cinematic demixing (CDX) track of the Sound Demixing Challenge 2023 (SDX'23). We provide a comprehensive summary of the challenge setup, detailing the structure of the competition and the datasets used. Especially, we detail CDXDB23, a new hidden dataset constructed from real movies that was used to rank the submissions. The paper also offers insights into the most successful approaches employed by participants. Compared to the cocktail-fork baseline, the best-performing system trained exclusively on the simulated Divide and Remaster (DnR) dataset achieved an improvement of 1.8dB in SDR whereas the top performing system on the open leaderboard, where any data could be used for training, saw a significant improvement of 5.7dB.

【14】 The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track链接：https://arxiv.org/abs/2308.06979

作者：Giorgio Fabbro,Stefan Uhlich,Chieh-Hsin Lai,Woosung Choi,Marco Martínez-Ramírez,Weihsiang Liao,Igor Gadelha,Geraldo Ramos,Eddie Hsu,Hugo Rodrigues,Fabian-Robert Stöter,Alexandre Défossez,Yi Luo,Jianwei Yu,Dipam Chakraborty,Sharada Mohanty,Roman Solovyev,Alexander Stempkovskiy,Tatiana Habruseva,Nabarun Goswami,Tatsuya Harada,Minseok Kim,Jun Hyung Lee,Yuanliang Dong,Xinran Zhang,Jiafeng Liu,Yuki Mitsufuji备注：under review摘要：本文总结了声音分层挑战赛（SDX'23）的音乐分层（MDX）曲目。我们提供了挑战设置的总结，并介绍了鲁棒音乐源分离（MSS）的任务，即，在训练数据中存在误差的情况下训练MSS模型。我们提出了MSS系统训练数据集设计中可能发生的错误的形式化，并引入了两个模拟此类错误的新数据集：SDXDB23_LabelNoise和SDXDB23_Bleeding1。我们描述的方法，在竞争中取得了最高分。此外，我们还与上一届挑战赛（2021年音乐分层挑战赛）进行了直接比较：当在MDXDB 21上进行评估时，在标准MSS公式下的最佳性能系统在信号失真比方面比先前竞争的获胜者实现了超过1.6dB的改进。除了依靠信号失真比作为客观指标外，我们还与知名制作人/音乐家进行了听力测试，以研究系统的感知质量，并在此报告结果。最后，我们提供了我们的见解，竞争的组织和我们对未来版本的前景。摘要：This paper summarizes the music demixing (MDX) track of the Sound Demixing Challenge (SDX'23). We provide a summary of the challenge setup and introduce the task of robust music source separation (MSS), i.e., training MSS models in the presence of errors in the training data. We propose a formalization of the errors that can occur in the design of a training dataset for MSS systems and introduce two new datasets that simulate such errors: SDXDB23_LabelNoise and SDXDB23_Bleeding1. We describe the methods that achieved the highest scores in the competition. Moreover, we present a direct comparison with the previous edition of the challenge (the Music Demixing Challenge 2021): the best performing system under the standard MSS formulation achieved an improvement of over 1.6dB in signal-to-distortion ratio over the winner of the previous competition, when evaluated on MDXDB21. Besides relying on the signal-to-distortion ratio as objective metric, we also performed a listening test with renowned producers/musicians to study the perceptual quality of the systems and report here the results. Finally, we provide our insights into the organization of the competition and our prospects for future editions.

【15】 SpeechX: Neural Codec Language Model as a Versatile Speech Transformer标题：SpeechX：作为通用语音转换器的神经编解码器语言模型链接：https://arxiv.org/abs/2308.06873

作者：Xiaofei Wang,Manthan Thakker,Zhuo Chen,Naoyuki Kanda,Sefik Emre Eskimez,Sanyuan Chen,Min Tang,Shujie Liu,Jinyu Li,Takuya Yoshioka备注：See this https URL for demo samples摘要：基于音频文本提示的生成语音模型的最新进展已经实现了卓越的创新，如高质量zero-shot文本到语音。然而，现有的模型在处理涉及变换输入语音和处理在不利的声学条件下捕获的音频的各种音频文本语音生成任务中仍然面临限制。本文介绍了SpeechX，一个多功能的语音生成模型能够zero-shot TTS和各种语音转换任务，处理干净和嘈杂的信号。SpeechX将神经编解码器语言建模与使用任务相关提示的多任务学习相结合，实现了统一和可扩展的建模，并为在语音增强和转换任务中利用文本输入提供了一致的方式。实验结果显示了SpeechX在各种任务中的功效，包括zero-shot TTS，噪声抑制，目标说话人提取，语音删除和有或没有背景噪声的语音编辑，实现了与跨任务的专用模型相当或更高的性能。有关演示示例，请访问https://aka.ms/speechx。摘要：Recent advancements in generative speech models based on audio-text prompts have enabled remarkable innovations like high-quality zero-shot text-to-speech. However, existing models still face limitations in handling diverse audio-text speech generation tasks involving transforming input speech and processing audio captured in adverse acoustic conditions. This paper introduces SpeechX, a versatile speech generation model capable of zero-shot TTS and various speech transformation tasks, dealing with both clean and noisy signals. SpeechX combines neural codec language modeling with multi-task learning using task-dependent prompting, enabling unified and extensible modeling and providing a consistent way for leveraging textual input in speech enhancement and transformation tasks. Experimental results show SpeechX's efficacy in various tasks, including zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing with or without background noise, achieving comparable or superior performance to specialized models across tasks. See https://aka.ms/speechx for demo samples.

【16】 Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition标题：半监督自动语音识别的交替伪标记法链接：https://arxiv.org/abs/2308.06547

作者：Han Zhu,Dongji Gao,Gaofeng Cheng,Daniel Povey,Pengyuan Zhang,Yonghong Yan备注：Accepted by IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 2023摘要：在标记数据不足的情况下，利用伪标记技术进行半监督学习，可以显著提高自动语音识别的性能。然而，伪标签通常是嘈杂的，包含许多不正确的标记。将噪声标签作为损失函数中的地面实况导致次优性能。以前的工作试图通过过滤掉最嘈杂的伪标签或提高伪标签的整体质量来缓解这个问题。虽然这些方法在一定程度上是有效的，但完全消除伪标签中的不正确标记是不现实的。在这项工作中，我们提出了一个新的框架，称为替代伪标签，以解决问题的噪声伪标签的角度来看，训练目标。该框架包括几个组成部分。首先，一个广义的CTC损失函数被引入处理噪声伪标签，通过接受替代令牌的位置不正确的令牌。在伪标记中应用该损失函数需要检测预测的伪标记中的不正确标记。在这项工作中，我们采用了一个基于置信度的错误检测方法，通过比较他们的置信度分数与给定的阈值来识别不正确的令牌，从而有必要的置信度分数是有区别的。因此，第二个提出的技术是对比CTC损失函数，其扩大了正确和不正确预测的令牌之间的置信度差距，从而提高错误检测能力。此外，利用基于置信度的错误检测获得令人满意的性能通常需要广泛的阈值调整。相反，我们提出了一个自动阈值的方法，使用标记的数据作为代理确定阈值，从而节省了手动调整的痛苦。摘要：When labeled data is insufficient, semi-supervised learning with the pseudo-labeling technique can significantly improve the performance of automatic speech recognition. However, pseudo-labels are often noisy, containing numerous incorrect tokens. Taking noisy labels as ground-truth in the loss function results in suboptimal performance. Previous works attempted to mitigate this issue by either filtering out the nosiest pseudo-labels or improving the overall quality of pseudo-labels. While these methods are effective to some extent, it is unrealistic to entirely eliminate incorrect tokens in pseudo-labels. In this work, we propose a novel framework named alternative pseudo-labeling to tackle the issue of noisy pseudo-labels from the perspective of the training objective. The framework comprises several components. Firstly, a generalized CTC loss function is introduced to handle noisy pseudo-labels by accepting alternative tokens in the positions of incorrect tokens. Applying this loss function in pseudo-labeling requires detecting incorrect tokens in the predicted pseudo-labels. In this work, we adopt a confidence-based error detection method that identifies the incorrect tokens by comparing their confidence scores with a given threshold, thus necessitating the confidence score to be discriminative. Hence, the second proposed technique is the contrastive CTC loss function that widens the confidence gap between the correctly and incorrectly predicted tokens, thereby improving the error detection ability. Additionally, obtaining satisfactory performance with confidence-based error detection typically requires extensive threshold tuning. Instead, we propose an automatic thresholding method that uses labeled data as a proxy for determining the threshold, thus saving the pain of manual tuning.

【17】 Knowledge Distilled Ensemble Model for sEMG-based Silent Speech Interface标题：基于表面肌电信号的无声语音接口知识提取集成模型链接：https://arxiv.org/abs/2308.06533

作者：Wenqiang Lai,Qihan Yang,Ye Mao,Endong Sun,Jiangnan Ye机构：Electrical & Electronic Eng, Imperial College London, London, UK备注：6 pages, 5 figures摘要：声音障碍影响着全世界数百万人。基于表面肌电图的无声语音接口（基于sEMG的SSI）已经作为潜在的解决方案被探索了几十年。然而，以前的工作受限于小词汇表和手动提取的原始数据的功能。为了解决这些限制，我们提出了一种轻量级的深度学习知识蒸馏集成模型，用于基于sEMG的SSI（KDE-SSI）。我们的模型可以对26个NATO拼音字母数据集进行分类，其中包含3900个数据样本，可以通过拼写明确地生成任何英语单词。大量的实验验证了KDE-SSI的有效性，达到了85.9%的测试精度。我们的研究结果还揭示了便携式实用设备的端到端系统。摘要：Voice disorders affect millions of people worldwide. Surface electromyography-based Silent Speech Interfaces (sEMG-based SSIs) have been explored as a potential solution for decades. However, previous works were limited by small vocabularies and manually extracted features from raw data. To address these limitations, we propose a lightweight deep learning knowledge-distilled ensemble model for sEMG-based SSI (KDE-SSI). Our model can classify a 26 NATO phonetic alphabets dataset with 3900 data samples, enabling the unambiguous generation of any English word through spelling. Extensive experiments validate the effectiveness of KDE-SSI, achieving a test accuracy of 85.9\%. Our findings also shed light on an end-to-end system for portable, practical equipment.

【18】 Bilingual Streaming ASR with Grapheme units and Auxiliary Monolingual Loss标题：具有字素单位和辅助性单语丢失的双语流媒体ASR链接：https://arxiv.org/abs/2308.06327

作者：Mohammad Soleymanpour,Mahmoud Al Ismail,Fahimeh Bahmaninezhad,Kshitiz Kumar,Jian Wu机构：Microsoft Speech and Language Group摘要：我们介绍了一个双语的解决方案，以支持英语作为次要地区的混合自动语音识别（ASR）设置中的大多数主要地区。我们的主要发展包括：（a）具有字素单元而不是音素单元的发音词典，（b）完全双语对齐模型和随后的双语流Transformer模型，（c）具有语言识别（LID）损失的并行编码器结构，（d）具有用于单语投影的辅助损失的并行编码器。我们的结论是，在LID损失相比，我们提出的辅助损失是优越的，专门的并行编码器各自的单语地区，并有助于更强的双语学习。我们评估我们的工作，大规模的培训和测试任务的双语西班牙语（ES）和双语意大利语（IT）的应用程序。我们的双语模型表现出强大的英语代码混合能力。特别是，双语IT模型提高了代码混合IT任务的单词错误率（WER）从46.5%到13.8%，同时也实现了接近奇偶校验（9.6%）与单语IT模型（9.5%）超过IT测试。摘要：We introduce a bilingual solution to support English as secondary locale for most primary locales in hybrid automatic speech recognition (ASR) settings. Our key developments constitute: (a) pronunciation lexicon with grapheme units instead of phone units, (b) a fully bilingual alignment model and subsequently bilingual streaming transformer model, (c) a parallel encoder structure with language identification (LID) loss, (d) parallel encoder with an auxiliary loss for monolingual projections. We conclude that in comparison to LID loss, our proposed auxiliary loss is superior in specializing the parallel encoders to respective monolingual locales, and that contributes to stronger bilingual learning. We evaluate our work on large-scale training and test tasks for bilingual Spanish (ES) and bilingual Italian (IT) applications. Our bilingual models demonstrate strong English code-mixing capability. In particular, the bilingual IT model improves the word error rate (WER) for a code-mix IT task from 46.5% to 13.8%, while also achieving a close parity (9.6%) with the monolingual IT model (9.5%) over IT tests.

eess.AS音频处理

【1】 Localization of DOA trajectories -- Beyond the grid标题：定位DOA轨迹--超越网格链接：https://arxiv.org/abs/2308.07265

【2】 Compositional nonlinear audio signal processing with Volterra series标题：基于Volterra级数的合成非线性音频信号处理链接：https://arxiv.org/abs/2308.07229

【3】 Integrating Emotion Recognition with Speech Recognition and Speaker Diarisation for Conversations标题：将情感识别与语音识别和说话人识别相结合用于对话链接：https://arxiv.org/abs/2308.07145

【4】 VoxSnap: X-Large Speaker Verification Dataset on Camera标题：VoxSnap：摄像机上的X-大型说话人验证数据集链接：https://arxiv.org/abs/2308.07056

【5】 The Sound Demixing Challenge 2023 $\unicode{x2013}$ Cinematic Demixing Track链接：https://arxiv.org/abs/2308.06981

【6】 The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track链接：https://arxiv.org/abs/2308.06979

【7】 SpeechX: Neural Codec Language Model as a Versatile Speech Transformer标题：SpeechX：作为通用语音转换器的神经编解码器语言模型链接：https://arxiv.org/abs/2308.06873

【8】 Alternative Pseudo-Labeling for Semi-Supervised Automatic Speech Recognition标题：半监督自动语音识别的交替伪标记法链接：https://arxiv.org/abs/2308.06547

【9】 Knowledge Distilled Ensemble Model for sEMG-based Silent Speech Interface标题：基于表面肌电信号的无声语音接口知识提取集成模型链接：https://arxiv.org/abs/2308.06533

【10】 Bilingual Streaming ASR with Grapheme units and Auxiliary Monolingual Loss标题：具有字素单位和辅助性单语丢失的双语流媒体ASR链接：https://arxiv.org/abs/2308.06327

【11】 DiffSED: Sound Event Detection with Denoising Diffusion标题：DiffSED：带去噪扩散的声事件检测链接：https://arxiv.org/abs/2308.07293

【12】 AudioFormer: Audio Transformer learns audio feature representations from discrete acoustic codes标题：AudioFormer：音频转换器从离散的声学代码中学习音频特征表示链接：https://arxiv.org/abs/2308.07221

【13】 PitchNet: A Fully Convolutional Neural Network for Pitch Estimation标题：PitchNet：一种用于基音估计的全卷积神经网络链接：https://arxiv.org/abs/2308.07170

【14】 Active Bird2Vec: Towards End-to-End Bird Sound Monitoring with Transformers标题：Active Bird2Vec：用Transformer实现端到端的鸟声监测链接：https://arxiv.org/abs/2308.07121

【15】 iSTFTNet2: Faster and More Lightweight iSTFT-Based Neural Vocoder Using 1D-2D CNN标题：ISTFTNet2：使用1D-2D CNN的更快、更轻量级的基于iSTFT的神经声码器链接：https://arxiv.org/abs/2308.07117

【16】 BigWavGAN: A Wave-To-Wave Generative Adversarial Network for Music Super-Resolution标题：BigWavGan：一种针对音乐超分辨率的波对波生成性对抗网络链接：https://arxiv.org/abs/2308.06483

【17】 Flexible Keyword Spotting based on Homogeneous Audio-Text Embedding标题：基于同构音文嵌入的柔性关键词检测链接：https://arxiv.org/abs/2308.06472

【18】 Neural Latent Aligner: Cross-trial Alignment for Learning Representations of Complex, Naturalistic Neural Data标题：神经潜伏期比对：学习复杂自然神经数据表示的交叉试验比对链接：https://arxiv.org/abs/2308.06443

作者：Cheol Jun Cho,Edward F. Chang,Gopala K. Anumanchipalli机构：and successful applications have provided significant in-Equal contribution 1Department of Electrical Engineering andComputer Sciences, University of California, USA 2Department of Neurological Surgery, University of Cal-ifornia备注：None摘要：理解复杂人类行为的神经实现是神经科学的主要目标之一。为此，找到神经数据的真实表示是至关重要的，由于行为的高复杂性和信号的低信噪比（SNR），这是具有挑战性的。在这里，我们提出了一种新的无监督学习框架，神经潜在对齐（NLA），找到良好的约束，行为相关的神经表示的复杂行为。关键思想是在重复试验之间对齐表示，以学习跨试验一致性信息。此外，我们提出了一个新的，完全可微的时间扭曲模型（TWM），以解决时间错位的试验。当应用于自然说话的颅内皮层电图（ECoG），我们的模型学习更好的表示解码行为比基线模型，特别是在低维空间。TWM通过测量对齐试验之间的行为一致性来经验验证。所提出的框架学习更多的跨试验一致性表示比基线，当可视化时，流形揭示了跨试验共享的神经轨迹。摘要：Understanding the neural implementation of complex human behaviors is one of the major goals in neuroscience. To this end, it is crucial to find a true representation of the neural data, which is challenging due to the high complexity of behaviors and the low signal-to-ratio (SNR) of the signals. Here, we propose a novel unsupervised learning framework, Neural Latent Aligner (NLA), to find well-constrained, behaviorally relevant neural representations of complex behaviors. The key idea is to align representations across repeated trials to learn cross-trial consistent information. Furthermore, we propose a novel, fully differentiable time warping model (TWM) to resolve the temporal misalignment of trials. When applied to intracranial electrocorticography (ECoG) of natural speaking, our model learns better representations for decoding behaviors than the baseline models, especially in lower dimensional space. The TWM is empirically validated by measuring behavioral coherence between aligned trials. The proposed framework learns more cross-trial consistent representations than the baselines, and when visualized, the manifold reveals shared neural trajectories across trials.

【19】 Phoneme Hallucinator: One-shot Voice Conversion via Set Expansion标题：音素幻想器：通过集合扩展实现一次语音转换链接：https://arxiv.org/abs/2308.06382

机器翻译，仅供参考

语音/音频处理学术速递[8.15]

评论 0