今日论文合集：cs.SD语音44篇，eess.AS音频处理44篇。由于两个种类的内容一致，今天不分开列举。
【1】 LyricJam Sonic: A Generative System for Real-Time Composition and Musical Improvisation标题：LyricJam Sonic：一个实时作曲和音乐即兴创作的生成系统链接：https://arxiv.org/abs/2210.15638
作者：Olga Vechtomova,Gaurav Sahu机构：University of Waterloo, Waterloo, ON, Canada备注：15 pages, 9 figures, 2 tables摘要：电子音乐艺术家和声音设计师具有独特的工作流实践，这使得开发音乐信息检索和创造性支持工具的专门方法成为必要。此外，诸如模块化合成器之类的电子乐器具有近乎无限的声音创建可能性，并且可以组合以创建独特且复杂的音频路径。发现有趣声音的过程往往是偶然的，不可能复制。出于这个原因，许多电子音乐家在录音室工作时一直在录制音频输出。因此，艺术家很难从数千小时的录音中重新发现可能适合用于他们的作品的音频片段。在本文中，我们描述了LyricJam Sonic --一个新颖的创作工具，它可以让音乐家重新发现他们以前的录音，将它们与其他录音重新联系起来，并实时创作原创的现场音乐作品。双模态人工智能驱动的方法使用生成的歌词线从艺术家过去的录音室录音中找到匹配的音频剪辑，并使用它们生成新的歌词线，这些歌词线又用于找到其他剪辑，从而创建连续和不断变化的音乐和歌词流。其目的是让艺术家们保持一种有利于音乐创作的创造性流动状态，而不是让他们进入一种故意寻找过去音频片段的分析/批判状态。该系统可以在没有用户输入的完全自主模式下运行，也可以在现场表演模式下运行，在现场表演模式下，艺术家播放现场音乐，而系统“收听”并作为响应创建音乐和歌词的连续流。摘要：Electronic music artists and sound designers have unique workflow practices that necessitate specialized approaches for developing music information retrieval and creativity support tools. Furthermore, electronic music instruments, such as modular synthesizers, have near-infinite possibilities for sound creation and can be combined to create unique and complex audio paths. The process of discovering interesting sounds is often serendipitous and impossible to replicate. For this reason, many musicians in electronic genres record audio output at all times while they work in the studio. Subsequently, it is difficult for artists to rediscover audio segments that might be suitable for use in their compositions from thousands of hours of recordings. In this paper, we describe LyricJam Sonic -- a novel creative tool for musicians to rediscover their previous recordings, re-contextualize them with other recordings, and create original live music compositions in real-time. A bi-modal AI-driven approach uses generated lyric lines to find matching audio clips from the artist's past studio recordings, and uses them to generate new lyric lines, which in turn are used to find other clips, thus creating a continuous and evolving stream of music and lyrics. The intent is to keep the artists in a state of creative flow conducive to music creation rather than taking them into an analytical/critical state of deliberately searching for past audio segments. The system can run in either a fully autonomous mode without user input, or in a live performance mode, where the artist plays live music, while the system "listens" and creates a continuous stream of music and lyrics in response.
【2】 Multimodal Transformer Distillation for Audio-Visual Synchronization标题：用于视听同步的多模Transformer蒸馏链接：https://arxiv.org/abs/2210.15563
作者：Xuanjun Chen,Haibin Wu,Chung-Che Wang,Hung-yi Lee,Jyh-Shing Roger Jang机构：Department of Computer Science and Information Engineering, National Taiwan University, Graduate Institute of Communication Engineering, National Taiwan University备注：Submitted to ICASSP 2023摘要：视听同步的目的是确定视频中的嘴部运动和语音是否同步。VocaLiST通过结合多模式Transformers来模拟视听交互信息，达到了最先进的性能。然而，它需要大量的计算资源，使得它对于现实世界的应用不切实际。提出了一种MTDVocaLiST模型，该模型利用我们提出的多峰Transformer蒸馏（MTD）损耗进行训练。MTD损失使得MTDVocaLiST模型能够更深入地模拟VocaLiST的Transformer中的交叉注意力分布和价值关系。我们提出的方法在两个方面是有效的：从蒸馏方法的角度来看，MTD损失优于其他强蒸馏基线。从提取模型的性能角度来看：1）MTDVocaLiST的性能比相同规模的SOTA模型、SyncNet模型和PM模型分别提高了15.69%和3.39%; 2）MTDVocaLiST将VocaLiST的模型大小减少了83.52%，但仍保持了类似的性能。摘要：Audio-visual synchronization aims to determine whether the mouth movements and speech in the video are synchronized. VocaLiST reaches state-of-the-art performance by incorporating multimodal Transformers to model audio-visual interact information. However, it requires high computing resources, making it impractical for real-world applications. This paper proposed an MTDVocaLiST model, which is trained by our proposed multimodal Transformer distillation (MTD) loss. MTD loss enables MTDVocaLiST model to deeply mimic the cross-attention distribution and value-relation in the Transformer of VocaLiST. Our proposed method is effective in two aspects: From the distillation method perspective, MTD loss outperforms other strong distillation baselines. From the distilled model's performance perspective: 1) MTDVocaLiST outperforms similar-size SOTA models, SyncNet, and PM models by 15.69% and 3.39%; 2) MTDVocaLiST reduces the model size of VocaLiST by 83.52%, yet still maintaining similar performance.
【3】 Source-Filter HiFi-GAN: Fast and Pitch Controllable High-Fidelity Neural Vocoder标题：源-滤波高保真-GaN：快速、音调可控的高保真神经声码器链接：https://arxiv.org/abs/2210.15533
作者：Reo Yoneyama,Yi-Chiao Wu,Tomoki Toda机构：Nagoya University, Japan,Meta Reality Labs Research, USA备注：Submitted to ICASSP 2023摘要：我们的前期工作--统一源滤波器GAN（uSFGAN）声码器，将一种基于源滤波器理论的新型结构引入到并行波形生成对抗网络中，以实现高音质和基音可控性。然而，高时间分辨率输入导致高计算成本。尽管HiFi-GAN声码器由于基于上采样的高效生成器架构而实现了快速高保真语音生成，但音调可控性受到严重限制。为了实现一种快速、基音可控的高保真神经声码器，将源滤波器理论引入HiFi-GAN，通过对源激励信息的合理估计，分层调节谐振滤波网络，实现了一种快速、基音可控的高保真神经声码器。实验结果表明，该方法在单CPU上的语音质量和合成速度上均优于HiFi-GAN和uSFGAN.此外，与uSFGAN声码器不同，所提出的方法可以容易地在实时应用和端到端系统中采用/集成。摘要：Our previous work, the unified source-filter GAN (uSFGAN) vocoder, introduced a novel architecture based on the source-filter theory into the parallel waveform generative adversarial network to achieve high voice quality and pitch controllability. However, the high temporal resolution inputs result in high computation costs. Although the HiFi-GAN vocoder achieves fast high-fidelity voice generation thanks to the efficient upsampling-based generator architecture, the pitch controllability is severely limited. To realize a fast and pitch-controllable high-fidelity neural vocoder, we introduce the source-filter theory into HiFi-GAN by hierarchically conditioning the resonance filtering network on a well-estimated source excitation information. According to the experimental results, our proposed method outperforms HiFi-GAN and uSFGAN on a singing voice generation in voice quality and synthesis speed on a single CPU. Furthermore, unlike the uSFGAN vocoder, the proposed method can be easily adopted/integrated in real-time applications and end-to-end systems.
【4】 Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech标题：Virtuoso：面向语音合成的大规模多语言语音-文本联合半监督学习链接：https://arxiv.org/abs/2210.15447
作者：Takaaki Saeki,Heiga Zen,Zhehuai Chen,Nobuyuki Morioka,Gary Wang,Yu Zhang,Ankur Bapna,Andrew Rosenberg,Bhuvana Ramabhadran机构：Google, Japan, Google, USA, The University of Tokyo, Japan备注：Submitted to ICASSP 2023摘要：本文提出了一个面向文本语音合成（TTS）模型的大规模多语言语音-文本联合半监督学习框架Virtuoso。现有的多语言TTS通常支持数十种语言，这是世界上数千种语言中的一小部分。将多语言TTS扩展到数百种语言的一个困难是收集低资源语言的高质量语音-文本配对数据。本研究将自动语音识别（ASR）的语音-文本联合预训练框架Maestro扩展到语音生成任务。为了从各种类型的语音和文本数据训练TTS模型，设计不同的训练方案来处理监督（成对的TTS和ASR数据）和非监督（未转录的语音和未说出的文本）数据集。实验结果表明：1）在Virtuoso平台上训练的多语种TTS模型在可见语言中的自然度和可懂度明显优于基准模型; 2）在没有高质量配对TTS数据的情况下，它们可以为不可见语言合成出合理可懂且听起来自然的语音。摘要：This paper proposes Virtuoso, a massively multilingual speech-text joint semi-supervised learning framework for text-to-speech synthesis (TTS) models. Existing multilingual TTS typically supports tens of languages, which are a small fraction of the thousands of languages in the world. One difficulty to scale multilingual TTS to hundreds of languages is collecting high-quality speech-text paired data in low-resource languages. This study extends Maestro, a speech-text joint pretraining framework for automatic speech recognition (ASR), to speech generation tasks. To train a TTS model from various types of speech and text data, different training schemes are designed to handle supervised (paired TTS and ASR data) and unsupervised (untranscribed speech and unspoken text) datasets. Experimental evaluation shows that 1) multilingual TTS models trained on Virtuoso can achieve significantly better naturalness and intelligibility than baseline ones in seen languages, and 2) they can synthesize reasonably intelligible and naturally sounding speech for unseen languages where no high-quality paired TTS data is available.
【5】 Toroidal Probabilistic Spherical Discriminant Analysis标题：环形概率球面判别式分析链接：https://arxiv.org/abs/2210.15441
作者：Anna Silnova,Niko Brümmer,Albert Swart,Lukáš Burget机构：Amazon Alexa, South Africa, Speechly, Finland备注：Submitted to ICASSP 2023摘要：在说话人识别中，将语音片段映射到单位超球面上的嵌入，通常使用两种评分后端，即余弦评分和PLDA。我们最近提出了PSDA，它是PLDA的一种类似物，使用Von Mises-Fisher分布而不是高斯分布。本文提出了一种环形PSDA（T-PSDA）算法。它扩展了PSDA，使其能够在超球面的环形子流形中建模说话人内部和说话人之间的变量。与PLDA和PSDA一样，该模型允许对训练进行闭合式评分和闭合式EM更新。在VoxCeleb上，我们发现T-PSDA准确性与余弦评分相当，而PLDA准确性较差。在NIST SRE'21中，我们发现与余弦评分和PLDA相比，T-PSDA提供了较大的准确性增益。摘要：In speaker recognition, where speech segments are mapped to embeddings on the unit hypersphere, two scoring back-ends are commonly used, namely cosine scoring and PLDA. We have recently proposed PSDA, an analog to PLDA that uses Von Mises-Fisher distributions instead of Gaussians. In this paper, we present toroidal PSDA (T-PSDA). It extends PSDA with the ability to model within and between-speaker variabilities in toroidal submanifolds of the hypersphere. Like PLDA and PSDA, the model allows closed-form scoring and closed-form EM updates for training. On VoxCeleb, we find T-PSDA accuracy on par with cosine scoring, while PLDA accuracy is inferior. On NIST SRE'21 we find that T-PSDA gives large accuracy gains compared to both cosine scoring and PLDA.
【6】 FreeVC: Towards High-Quality Text-Free One-Shot Voice Conversion标题：FreeVC：迈向高质量的无文本一次语音转换链接：https://arxiv.org/abs/2210.15418
作者：Jingyi li,Weiping tu,Li xiao机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan , China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University,Wuhan , China摘要：语音转换（VC）可以通过首先提取源内容信息和目标说话人信息，然后利用这些信息重构波形来实现。然而，现有的方法通常要么提取泄漏了说话人信息的脏内容信息，要么需要大量的注释数据用于训练。此外，转换模型与声码器之间的不匹配也会降低重构波形的质量。本文采用了端到端的VITS框架来实现高质量的波形重构，并提出了无需文本标注的干净内容信息提取策略。通过在WavLM特征中引入信息瓶颈，对内容信息进行分解，并提出基于谱图大小调整的数据增强方法，以提高提取内容信息的纯度。实验结果表明，与最新的基于标注数据的VC模型相比，该方法具有更好的鲁棒性。摘要：Voice conversion (VC) can be achieved by first extracting source content information and target speaker information, and then reconstructing waveform with these information. However, current approaches normally either extract dirty content information with speaker information leaked in, or demand a large amount of annotated data for training. Besides, the quality of reconstructed waveform can be degraded by the mismatch between conversion model and vocoder. In this paper, we adopt the end-to-end framework of VITS for high-quality waveform reconstruction, and propose strategies for clean content information extraction without text annotation. We disentangle content information by imposing an information bottleneck to WavLM features, and propose the spectrogram-resize based data augmentation to improve the purity of extracted content information. Experimental results show that the proposed method outperforms the latest VC models trained with annotated data and has greater robustness.
【7】 Make More of Your Data: Minimal Effort Data Augmentation for Automatic Speech Recognition and Translation标题：充分利用您的数据：为自动语音识别和翻译提供最省力的数据增强链接：https://arxiv.org/abs/2210.15398
作者：Tsz Kin Lam,Shigehiko Schamoni,Stefan Riezler机构：⋆Computational Linguistics & †IWR, Heidelberg University, Germany摘要：数据扩充是一种基于现有数据生成新训练数据的技术。我们评估了连接原始数据实例以构建新训练实例的简单且经济高效的方法。使用这种增强数据继续训练能够改进仅在原始数据上优化的现成Transformer和构象器模型。我们展示了LibriSpeech-960 h测试集的显著改进（测试-清洁和测试-其他的WER 2.83和6.87），这些改进延续到与浅融合相结合的模型（WER 2.55和6.27）。我们的连续训练方法还导致CoVoST-2的ASR部分对于四种非英语语言的提高高达0.9 WER，并且我们观察到增益高度依赖于原始训练数据的大小。我们比较了不同的拼接策略，发现我们的方法不需要说话人信息来实现其改进。最后，我们在两个数据集上证明了我们的方法也适用于语音翻译任务。摘要：Data augmentation is a technique to generate new training data based on existing data. We evaluate the simple and cost-effective method of concatenating the original data examples to build new training instances. Continued training with such augmented data is able to improve off-the-shelf Transformer and Conformer models that were optimized on the original data only. We demonstrate considerable improvements on the LibriSpeech-960h test sets (WER 2.83 and 6.87 for test-clean and test-other), which carry over to models combined with shallow fusion (WER 2.55 and 6.27). Our method of continued training also leads to improvements of up to 0.9 WER on the ASR part of CoVoST-2 for four non English languages, and we observe that the gains are highly dependent on the size of the original training data. We compare different concatenation strategies and found that our method does not need speaker information to achieve its improvements. Finally, we demonstrate on two datasets that our methods also works for speech translation tasks.
【8】 Automatic Severity Assessment of Dysarthric speech by using Self-supervised Model with Multi-task Learning标题：基于多任务学习的自监督模型自动评估节律障碍语音严重程度链接：https://arxiv.org/abs/2210.15387
作者：Eun Jung Yeo,Kwanghee Choi,Sunhee Kim,Minhwa Chung机构：Department of Linguistics, Seoul National University, Republic of Korea, Department of Computer Science and Engineering, Sogang University, Republic of Korea, Department of French Language Education, Seoul National University, Republic of Korea摘要：构音障碍言语的自动评估对于持续治疗和康复至关重要。然而，获得非典型语音是具有挑战性的，常常导致数据稀缺问题。针对这一问题，提出了一种新的构音障碍语音严重度自动评估方法，该方法将自监督模型与多任务学习相结合。Wav 2 vec 2.0 XLS-R针对两种不同任务进行联合训练：严重性级别分类和辅助自动语音识别（ASR）。对于基线实验，我们采用了手工制作的特征，如eGeMaps和语言特征，以及SVM、MLP和XGBoost分类器。在韩国构音障碍语音QoLT数据库上的实验结果表明，该模型的分类准确率相对提高了4.79%，优于传统的基线方法。此外，所提出的模型优于未使用ASR头部训练的模型，实现了10.09%的相对百分比改进。此外，通过分析多任务学习的潜在表征和正则化效应，给出了多任务学习对严重度分类性能的影响.摘要：Automatic assessment of dysarthric speech is essential for sustained treatments and rehabilitation. However, obtaining atypical speech is challenging, often leading to data scarcity issues. To tackle the problem, we propose a novel automatic severity assessment method for dysarthric speech, using the self-supervised model in conjunction with multi-task learning. Wav2vec 2.0 XLS-R is jointly trained for two different tasks: severity level classification and an auxilary automatic speech recognition (ASR). For the baseline experiments, we employ hand-crafted features such as eGeMaps and linguistic features, and SVM, MLP, and XGBoost classifiers. Explored on the Korean dysarthric speech QoLT database, our model outperforms the traditional baseline methods, with a relative percentage increase of 4.79% for classification accuracy. In addition, the proposed model surpasses the model trained without ASR head, achieving 10.09% relative percentage improvements. Furthermore, we present how multi-task learning affects the severity classification performance by analyzing the latent representations and regularization effect.
【9】 Opening the Black Box of wav2vec Feature Encoder标题：打开Wave2vec特征编码器的黑盒链接：https://arxiv.org/abs/2210.15386
作者：Kwanghee Choi,Eun Jung Yeo机构：Department of Computer Science and Engineering, Sogang University, Republic of Korea, Department of Linguistics, Seoul National University, Republic of Korea摘要：自监督模型，即wav2vec及其变体，已经在语音域中的各种下游任务中显示出有希望的结果。然而，它们的内部运作却知之甚少，需要对模型学习的内容进行深入分析。在这篇论文中，我们集中在卷积特征编码器，其中它的潜在空间通常被推测为代表离散的声学单元。为了以简化的方式分析嵌入空间，我们馈送合成的音频信号，其是简单正弦波的总和。通过大量的实验，我们得出结论，各种信息被嵌入在特征编码器表示中：（1）基频，（2）共振峰，和（3）振幅，用（4）足够的时间细节打包。此外，包含在潜在表示中的信息类似于频谱图，但是具有基本的差别：潜在表示构建度量空间，使得更接近的表示暗示声学相似性。摘要：Self-supervised models, namely, wav2vec and its variants, have shown promising results in various downstream tasks in the speech domain. However, their inner workings are poorly understood, calling for in-depth analyses on what the model learns. In this paper, we concentrate on the convolutional feature encoder where its latent space is often speculated to represent discrete acoustic units. To analyze the embedding space in a reductive manner, we feed the synthesized audio signals, which is the summation of simple sine waves. Through extensive experiments, we conclude that various information is embedded inside the feature encoder representations: (1) fundamental frequency, (2) formants, and (3) amplitude, packed with (4) sufficient temporal detail. Further, the information incorporated inside the latent representations is analogous to spectrograms but with a fundamental difference: latent representations construct a metric space so that closer representations imply acoustic similarity.
【10】 CasNet: Investigating Channel Robustness for Speech Separation标题：CasNet：研究语音分离的信道稳健性链接：https://arxiv.org/abs/2210.15370
作者：Fan-Lin Wang,Yao-Fei Cheng,Hung-Shin Lee,Yu Tsao,Hsin-Min Wang机构：North Co., Ltd., Taiwan, Institute of Information Science, Academia Sinica, Research Center for Information Technology Innovation, Academia Sinica备注：Submitted to ICASSP 2023摘要：训练和测试条件之间的记录通道失配已经被证明是语音分离的一个严重问题。这种情况大大降低了分离性能，不能满足日常使用的要求。在本研究中，我们继承了我们先前构建的TAT-2 mix语料库的使用，通过提出一种用于端到端时域语音分离的深度学习框架--通道感知音频分离网络（CasNet）来解决通道失配问题。CasNet是在TasNet之上实现的。信道编码器产生的信道嵌入（在多个话语的混合中表征信道信息）通过FILM技术被引入分离模块。通过两种训练策略，我们探讨了渠道嵌入可能扮演的两个角色：1）现实生活中的噪声干扰，使模型更鲁棒，或2）指导，指示分离模型保留期望的信道信息。在TAT-2 mix上的实验结果表明，采用这两种训练策略训练的CasNet性能优于未使用信道嵌入的TasNet基线。摘要：Recording channel mismatch between training and testing conditions has been shown to be a serious problem for speech separation. This situation greatly reduces the separation performance, and cannot meet the requirement of daily use. In this study, inheriting the use of our previously constructed TAT-2mix corpus, we address the channel mismatch problem by proposing a channel-aware audio separation network (CasNet), a deep learning framework for end-to-end time-domain speech separation. CasNet is implemented on top of TasNet. Channel embedding (characterizing channel information in a mixture of multiple utterances) generated by Channel Encoder is introduced into the separation module by the FiLM technique. Through two training strategies, we explore two roles that channel embedding may play: 1) a real-life noise disturbance, making the model more robust, or 2) a guide, instructing the separation model to retain the desired channel information. Experimental results on TAT-2mix show that CasNet trained with both training strategies outperforms the TasNet baseline, which does not use channel embeddings.
【11】 A Teacher-student Framework for Unsupervised Speech Enhancement Using Noise Remixing Training and Two-stage Inference标题：一种基于噪声混合训练和两阶段推理的无监督语音增强师生框架链接：https://arxiv.org/abs/2210.15368
作者：Li-Wei Chen,Yao-Fei Cheng,Hung-Shin Lee,Yu Tsao,Hsin-Min Wang机构：North Co., Ltd., Taiwan, Institute of Information Science, Academia Sinica, Research Center for Information Technology Innovation, Academia Sinica备注：Submitted to ICASSP 2023摘要：干净语音的缺乏是语音增强系统发展的一个实际挑战，这意味着神经网络模型的训练必须在无监督的方式下进行，并且它们的训练标准和评价标准之间不可避免地存在不匹配。针对这一不利情况，本文提出了一种不需要任何主观/客观语音质量指标作为学习参考的师生训练策略，通过改进先前提出的噪声目标训练（NyTT）。由于域内噪声和外部噪声之间的同质性是NyTT有效性的关键，因此我们通过重新混合教师模型的估计语音和噪声来训练各种学生模型，以进行干净目标训练，或者通过重新混合原始噪声语音和教师模型的估计噪声来训练噪声目标训练。我们使用NyTT模型作为初始教师模型。实验结果表明，该方法优于几种基线，尤其是在两阶段推理中，通过bootstrap模型和final student模型相继得到干净的语音。摘要：The lack of clean speech is a practical challenge to the development of speech enhancement systems, which means that the training of neural network models must be done in an unsupervised manner, and there is an inevitable mismatch between their training criterion and evaluation metric. In response to this unfavorable situation, we propose a teacher-student training strategy that does not require any subjective/objective speech quality metrics as learning reference by improving the previously proposed noisy-target training (NyTT). Because homogeneity between in-domain noise and extraneous noise is the key to the effectiveness of NyTT, we train various student models by remixing the teacher model's estimated speech and noise for clean-target training or raw noisy speech and the teacher model's estimated noise for noisy-target training. We use the NyTT model as the initial teacher model. Experimental results show that our proposed method outperforms several baselines, especially with two-stage inference, where clean speech is derived successively through the bootstrap model and the final student model.
【12】 Explicit Intensity Control for Accented Text-to-speech标题：带重音的文语转换的显式强度控制链接：https://arxiv.org/abs/2210.15364
作者：Rui Liu,Haolin Zuo,De Hu,Guanglai Gao,Haizhou Li机构：Inner Mongolia University, China , The Chinese University of Hong Kong, Shenzhen, China备注：5 pages, 3 figures. Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2209.10804摘要：带重音的文本到语音（TTS）合成寻求生成带有重音（L2）的语音作为标准版本（L1）的变体。如何在TTS过程中控制重音的强度是一个非常有趣的研究方向，并引起了越来越多的关注。最近的工作设计了一个说话人对抗损失，以解开说话人和口音信息，然后调整损失的权重，以控制口音强度。然而，这样的控制方法缺乏可解释性，并且在控制因素和自然重音强度之间没有直接的相关性。为此，本文提出了一种新的直观、显式的重音强度控制方案。具体地说，我们首先从L1语音识别模型中提取后验概率，称为'发音优度（GoP）''，以量化重音语音的音素重音强度，然后设计一个基于FastSpeech 2的TTS模型，称为Ai-TTS，以在语音生成过程中考虑重音强度的表达。实验结果表明，该方法在重音渲染和亮度控制方面优于基线模型。摘要：Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1). How to control the intensity of accent in the process of TTS is a very interesting research direction, and has attracted more and more attention. Recent work design a speaker-adversarial loss to disentangle the speaker and accent information, and then adjust the loss weight to control the accent intensity. However, such a control method lacks interpretability, and there is no direct correlation between the controlling factor and natural accent intensity. To this end, this paper propose a new intuitive and explicit accent intensity control scheme for accented TTS. Specifically, we first extract the posterior probability, called as ``goodness of pronunciation (GoP)'' from the L1 speech recognition model to quantify the phoneme accent intensity for accented speech, then design a FastSpeech2 based TTS model, named Ai-TTS, to take the accent intensity expression into account during speech generation. Experiments show that the our method outperforms the baseline model in terms of accent rendering and intensity control.
【13】 FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesis标题：FCTalker：用于表现性会话语音合成的细粒度和粗粒度上下文建模链接：https://arxiv.org/abs/2210.15360
作者：Yifan Hu,Rui Liu,Guanglai Gao,Haizhou Li机构：Inner Mongolia University, China , The Chinese University of Hong Kong, Shenzhen, China备注：5 pages, 4 figures, 1 table. Submitted to ICASSP 2023. We release the source code at: this https URL摘要：会话文语转换（TTS）是在会话语境中合成具有正确的语言韵律和情感韵律的话语。该方法利用当前话语与对话历史在话语级的相关性来提高合成语音的表达能力。然而，词层面的对话历史中的细粒度信息对话语的韵律表达也有重要影响，这一点在以往的研究中还没有得到很好的研究。因此，本文提出了一种新的表达性会话TTS模型FCTalker，该模型在语音生成过程中同时学习细粒度和粗粒度的上下文依赖关系。具体地，FCTalker包括细粒度和粗粒度编码器，以利用单词和话语级上下文依赖性。为了对话语和其对话历史之间的词级依赖性进行建模，在对话BERT模型之上构建了细粒度对话编码器。实验结果表明，该方法的语音识别性能优于所有基准语音识别方法，并能生成更符合上下文的语音。我们在以下位置发布源代码：https://github.com/walker-hyf/FCTalker。摘要：Conversational Text-to-Speech (TTS) aims to synthesis an utterance with the right linguistic and affective prosody in a conversational context. The correlation between the current utterance and the dialogue history at the utterance level was used to improve the expressiveness of synthesized speech. However, the fine-grained information in the dialogue history at the word level also has an important impact on the prosodic expression of an utterance, which has not been well studied in the prior work. Therefore, we propose a novel expressive conversational TTS model, termed as FCTalker, that learn the fine and coarse grained context dependency at the same time during speech generation. Specifically, the FCTalker includes fine and coarse grained encoders to exploit the word and utterance-level context dependency. To model the word-level dependencies between an utterance and its dialogue history, the fine-grained dialogue encoder is built on top of a dialogue BERT model. The experimental results show that the proposed method outperforms all baselines and generates more expressive speech that is contextually appropriate. We release the source code at: https://github.com/walker-hyf/FCTalker.
【14】 Convolutive Block-Matching Segmentation Algorithm with Application to Music Structure Analysis标题：卷积块匹配分割算法及其在音乐结构分析中的应用链接：https://arxiv.org/abs/2210.15356
作者：Axel Marmoret,Jérémy E. Cohen,Frédéric Bimbot机构：Univ. Rennes , Inria, CNRS, IRISA, France., CREATIS, Univ Lyon, CNRS.备注：4 pages, 5 figures, 1 table. Submitted at ICASSP 2023. The associated toolbox is available at this https URL摘要：音乐结构分析（MSA）是将一首歌曲分成几个部分（如合唱、独唱、独唱等）来表示，可以看作是对歌曲的简化组织的检索。本文提出了一种新的卷积块匹配（Convolutive Block-Matching，CBM）算法，用于MSA的匹配。特别地，CBM算法是应用于自相似矩阵的动态规划算法，自相似矩阵是MSA中的标准工具。在这项工作中，从音频信号的特征表示计算自相似矩阵，并在条形标尺上采样时间。我们研究了三种不同的相似性函数来计算自相似矩阵。我们报告说，所提出的算法在4个指标中的3个指标上实现了与监督的最先进方法相竞争的性能水平，同时完全无监督。摘要：Music Structure Analysis (MSA) consists of representing a song in sections (such as ``chorus'', ``verse'', ``solo'' etc), and can be seen as the retrieval of a simplified organization of the song. This work presents a new algorithm, called Convolutive Block-Matching (CBM) algorithm, devoted to MSA. In particular, the CBM algorithm is a dynamic programming algorithm, applying on autosimilarity matrices, a standard tool in MSA. In this work, autosimilarity matrices are computed from the feature representation of an audio signal, and time is sampled on the barscale. We study three different similarity functions for the computation of autosimilarity matrices. We report that the proposed algorithm achieves a level of performance competitive to that of supervised state-of-the-art methods on 3 among 4 metrics, while being fully unsupervised.
【15】 Rigid-Body Sound Synthesis with Differentiable Modal Resonators标题：不同振型谐振器的刚体声音合成链接：https://arxiv.org/abs/2210.15306
作者：R. Diaz,B. Hayes,C. Saitis,G. Fazekas,M. Sandler机构：Queen Mary University of LondonABSTRACTPhysical models of rigid bodies are used for sound synthesisin applications from virtual environments to music produc-tion备注：5 pages摘要：在从虚拟环境到音乐制作的应用中，刚体的物理模型用于声音合成。模态综合等传统方法通常依赖于计算成本高昂的数值求解器，而最近的深度学习方法则受限于其结果的后处理。在这项工作中，我们提出了一个新的端到端的框架，用于训练一个深度神经网络，以生成一个给定的2D形状和材料的模态谐振器，使用一组可微分IIR滤波器。我们在合成对象的数据集上演示了我们的方法，但使用音频域目标来训练我们的模型，为直接从真实世界对象的记录中学习物理信息合成器铺平了道路。摘要：Physical models of rigid bodies are used for sound synthesis in applications from virtual environments to music production. Traditional methods such as modal synthesis often rely on computationally expensive numerical solvers, while recent deep learning approaches are limited by post-processing of their results. In this work we present a novel end-to-end framework for training a deep neural network to generate modal resonators for a given 2D shape and material, using a bank of differentiable IIR filters. We demonstrate our method on a dataset of synthetic objects, but train our model using an audio-domain objective, paving the way for physically-informed synthesisers to be learned directly from recordings of real-world objects.
【16】 Deformable Temporal Convolutional Networks for Monaural Noisy Reverberant Speech Separation标题：用于单声道噪声混响语音分离的可变形时间卷积网络链接：https://arxiv.org/abs/2210.15305
作者：William Ravenscroft,Stefan Goetze,Thomas Hain机构：Department of Computer Science, The University of Sheffield, Sheffield, United Kingdom备注：Submitted to ICASSP 2023摘要：在许多语音处理应用中，语音分离模型被用于隔离各个说话者。深度学习模型已被证明可在多个语音分离基准测试中产生最先进的（SOTA）结果。一种称为时间卷积网络（TCN）的此类模型已经显示出用于语音分离任务的有希望的结果。这些模型的局限性在于它们具有固定的感受野（RF）。语音去混响的最新研究表明，TCN的最佳RF随语音信号的混响特性而变化。在这项工作中，提出了可变形卷积作为一种解决方案，以允许TCN模型具有动态RF，该RF可以适应用于混响语音分离的各种混响时间。在WHAMR基准测试中，所提出的模型能够实现比输入信号平均11.1dB的尺度不变信号失真比（SISRD）改善。提出了一种1.3M参数的相对较小的可变形TCN模型，其给出了与较大且计算更复杂的模型相当的分离性能。摘要：Speech separation models are used for isolating individual speakers in many speech processing applications. Deep learning models have been shown to lead to state-of-the-art (SOTA) results on a number of speech separation benchmarks. One such class of models known as temporal convolutional networks (TCNs) has shown promising results for speech separation tasks. A limitation of these models is that they have a fixed receptive field (RF). Recent research in speech dereverberation has shown that the optimal RF of a TCN varies with the reverberation characteristics of the speech signal. In this work deformable convolution is proposed as a solution to allow TCN models to have dynamic RFs that can adapt to various reverberation times for reverberant speech separation. The proposed models are capable of achieving an 11.1 dB average scale-invariant signal-to-distortion ratio (SISDR) improvement over the input signal on the WHAMR benchmark. A relatively small deformable TCN model of 1.3M parameters is proposed which gives comparable separation performance to larger and more computationally complex models.
【17】 SAN: a robust end-to-end ASR model architecture标题：SAN：一种健壮的端到端ASR模型架构链接：https://arxiv.org/abs/2210.15285
作者：Zeping Min,Qian Ge,Guanhua Huang机构：Peking University, University of Science and Technology of China摘要：针对模糊音频识别的困难，提出了一种新的用于自动语音识别的暹罗对抗网络（SAN）结构。具体地说，SAN构造两个子网络以区分音频特征输入，然后引入损耗以统一这些子网络的输出分布。对抗性学习使网络能够捕获更多的基本声学特征，并帮助模型在遇到模糊音频输入时获得更好的性能。我们使用SAN模型在几个数据集上进行了自动语音识别任务的数值实验。实验结果表明，连体对抗网显著降低了字符错误率（CER）.具体而言，我们在AISHELL-1数据集上实现了新的最新技术水平的4.37 CER（无语言模型），这导致CER相对减少约5%。为了揭示连体对抗网络的通用性，我们还在音素识别任务上进行了实验，这也显示了连体对抗网络的优越性。摘要：In this paper, we propose a novel Siamese Adversarial Network (SAN) architecture for automatic speech recognition, which aims at solving the difficulty of fuzzy audio recognition. Specifically, SAN constructs two sub-networks to differentiate the audio feature input and then introduces a loss to unify the output distribution of these sub-networks. Adversarial learning enables the network to capture more essential acoustic features and helps the models achieve better performance when encountering fuzzy audio input. We conduct numerical experiments with the SAN model on several datasets for the automatic speech recognition task. All experimental results show that the siamese adversarial nets significantly reduce the character error rate (CER). Specifically, we achieve a new state of art 4.37 CER without language model on the AISHELL-1 dataset, which leads to around 5% relative CER reduction. To reveal the generality of the siamese adversarial net, we also conduct experiments on the phoneme recognition task, which also shows the superiority of the siamese adversarial network.
【18】 On Out-of-Distribution Detection for Audio with Deep Nearest Neighbors标题：深度近邻音频的非分布检测研究链接：https://arxiv.org/abs/2210.15283
作者：Zaharah Bukhsh,Aaqib Saeed机构：Eindhoven University of Technology, Eindhoven, The Netherlands, Philips Research, Eindhoven, The Netherlands摘要：分布外（OOD）检测涉及识别与模型的训练数据不属于同一分布的数据点。为了在现实环境中安全部署预测模型，避免对OOD输入进行自信预测是至关重要的，因为这可能导致潜在的危险后果。然而，OOD检测在很大程度上仍然是音频（和语音）域中未充分探索的领域。尽管音频是许多任务的中心模态，例如说话者日记化、自动语音识别和声音事件检测，但这是不考虑的事实。为了解决这个问题，我们提出利用具有深度k-最近邻的模型的特征空间来检测OOD样本。我们证明了这种简单而灵活的方法可以有效地检测到广泛类别的音频（和语音）数据集上的OOD输入。具体地，与其它现有技术相比，它将假阳性率（FPR@TPR95）提高了17%，将AUROC评分提高了7%。摘要：Out-of-distribution (OOD) detection is concerned with identifying data points that do not belong to the same distribution as the model's training data. For the safe deployment of predictive models in a real-world environment, it is critical to avoid making confident predictions on OOD inputs as it can lead to potentially dangerous consequences. However, OOD detection largely remains an under-explored area in the audio (and speech) domain. This is despite the fact that audio is a central modality for many tasks, such as speaker diarization, automatic speech recognition, and sound event detection. To address this, we propose to leverage feature-space of the model with deep k-nearest neighbors to detect OOD samples. We show that this simple and flexible method effectively detects OOD inputs across a broad category of audio (and speech) datasets. Specifically, it improves the false positive rate (FPR@TPR95) by 17% and the AUROC score by 7% than other prior techniques.
【19】 A knowledge-driven vowel-based approach of depression classification from speech using data augmentation标题：一种基于知识驱动的基于元音的语音抑郁分类方法链接：https://arxiv.org/abs/2210.15261
作者：Kexin Feng,Theodora Chaspari机构：Computer Science and Engineering, Texas A&M University摘要：我们提出了一种新的可解释的机器学习（ML）模型，该模型通过建模跨话语的时间依赖性并利用元音水平的谱时间信息来从语音中识别抑郁症。我们的方法首先使用具有空间金字塔池层的卷积神经网络（“元音CNN”）将本地级别的可变长度话语建模为基于固定长度元音的嵌入。接着，从一组元音CNN嵌入中在全局水平上对抑郁进行分类，所述元音CNN嵌入用作另一个1D CNN（“抑郁CNN”）的输入。对元音CNN和抑压CNN的训练分别设计了不同的数据扩充方法。我们研究了当分别对应于10、21和42个话语的短、中和长分析窗口建模时，所提出的系统在不同时间粒度下的性能。所提出的方法达到了与先前的最先进方法相当的性能，并描述了关于抑郁结果的可解释特性。这项工作的发现可能有利于临床医生在联合人类ML决策任务期间提供额外的直觉。摘要：We propose a novel explainable machine learning (ML) model that identifies depression from speech, by modeling the temporal dependencies across utterances and utilizing the spectrotemporal information at the vowel level. Our method first models the variable-length utterances at the local-level into a fixed-size vowel-based embedding using a convolutional neural network with a spatial pyramid pooling layer ("vowel CNN"). Following that, the depression is classified at the global-level from a group of vowel CNN embeddings that serve as the input of another 1D CNN ("depression CNN"). Different data augmentation methods are designed for both the training of vowel CNN and depression CNN. We investigate the performance of the proposed system at various temporal granularities when modeling short, medium, and long analysis windows, corresponding to 10, 21, and 42 utterances, respectively. The proposed method reaches comparable performance with previous state-of-the-art approaches and depicts explainable properties with respect to the depression outcome. The findings from this work may benefit clinicians by providing additional intuitions during joint human-ML decision-making tasks.
【20】 Iterative pseudo-forced alignment by acoustic CTC loss for self-supervised ASR domain adaptation标题：基于声学CTC损失的自监督ASR域自适应迭代伪强迫对齐链接：https://arxiv.org/abs/2210.15226
作者：Fernando López,Jordi Luque机构：Telef´onica I+D, Universidad Aut´onoma de Madrid备注：5 pages, 4 figures, IberSPEECH2022摘要：来自特定领域的高质量数据标记是昂贵的且耗费人力时间的。在此工作中，我们提出了一种基于迭代伪强制对齐算法的自监督域自适应方法。所产生的比对被用于定制端到端自动语音识别（ASR）并被迭代地改进。该算法由种子ASR产生的逐帧字符后验值作为输入，用域外数据进行训练，并在连接主义时态分类（CTC）丢失过程中进行优化。在广播TV的语料库上迭代地计算对准。通过减少要对齐的文本的数量或扩展对齐窗口来重复该过程，直到找到最佳可能的音频-文本对齐。基于最后对准的话语的置信度得分来唯一地产生开始时间戳或时间锚。该分数是用CTC比对矩阵的路径计算的。使用这种方法，不需要人工修改的文本引用。来自长音频文件的低质量转录（如电视字幕）的对齐将通过置信度分数过滤掉，并准备好进行进一步的ASR适配。在西班牙RTVE 2022和CommonVoice数据库上获得的结果支持使用基于CTC的系统执行以下操作的可行性：高度精确的音频-文本对齐、域自适应和端到端ASR的半监督训练。摘要：High-quality data labeling from specific domains is costly and human time-consuming. In this work, we propose a self-supervised domain adaptation method, based upon an iterative pseudo-forced alignment algorithm. The produced alignments are employed to customize an end-to-end Automatic Speech Recognition (ASR) and iteratively refined. The algorithm is fed with frame-wise character posteriors produced by a seed ASR, trained with out-of-domain data, and optimized throughout a Connectionist Temporal Classification (CTC) loss. The alignments are computed iteratively upon a corpus of broadcast TV. The process is repeated by reducing the quantity of text to be aligned or expanding the alignment window until finding the best possible audio-text alignment. The starting timestamps, or temporal anchors, are produced uniquely based on the confidence score of the last aligned utterance. This score is computed with the paths of the CTC-alignment matrix. With this methodology, no human-revised text references are required. Alignments from long audio files with low-quality transcriptions, like TV captions, are filtered out by confidence score and ready for further ASR adaptation. The obtained results, on both the Spanish RTVE2022 and CommonVoice databases, underpin the feasibility of using CTC-based systems to perform: highly accurate audio-text alignments, domain adaptation and semi-supervised training of end-to-end ASR.
【21】 Articulation GAN: Unsupervised modeling of articulatory learning标题：发音GAN：发音学习的无监督建模链接：https://arxiv.org/abs/2210.15173
作者：Gašper Beguš,Alan Zhou,Peter Wu,Gopala K Anumanchipalli机构：University of California, Berkeley,Johns Hopkins University摘要：生成式深度神经网络被广泛用于语音合成，但大多数现有模型直接生成波形或频谱输出。然而，人类通过控制发音器官来产生语音，这导致通过声音传播的物理特性来产生语音声音。我们提出了一种新的无监督的语音产生/合成的生成模型，该模型包括发音表示，从而更接近地模仿人类语音产生。我们将发音发生器引入生成对抗网络范式。Articulatory Generator需要学习以完全无监督的方式生成发音表示（电磁发音描记术或EMA），而无需访问EMA数据。然后，一个单独的预先训练的物理模型（ema 2 wav）将生成的EMA表示转换为语音波形，该语音波形被发送到鉴别器进行评估。对生成的EMA表征的发音分析表明，网络学习以在言语产生期间紧密跟随人类发音器的方式控制发音器。对输出的声学分析表明，网络学习生成作为训练数据的一部分的单词以及训练数据中不存在的新颖的创新单词。因此，我们提出的体系结构允许以完全无监督的方式利用来自原始音频输入的深度神经网络对发音学习进行建模。此外，我们还讨论了发音表征对人类语言和一般语音技术认知模型的影响。摘要：Generative deep neural networks are widely used for speech synthesis, but most existing models directly generate waveforms or spectral outputs. Humans, however, produce speech by controlling articulators, which results in the production of speech sounds through physical properties of sound propagation. We propose a new unsupervised generative model of speech production/synthesis that includes articulatory representations and thus more closely mimics human speech production. We introduce the Articulatory Generator to the Generative Adversarial Network paradigm. The Articulatory Generator needs to learn to generate articulatory representations (electromagnetic articulography or EMA) in a fully unsupervised manner without ever accessing EMA data. A separate pre-trained physical model (ema2wav) then transforms the generated EMA representations to speech waveforms, which get sent to the Discriminator for evaluation. Articulatory analysis of the generated EMA representations suggests that the network learns to control articulators in a manner that closely follows human articulators during speech production. Acoustic analysis of the outputs suggest that the network learns to generate words that are part of training data as well as novel innovative words that are absent from training data. Our proposed architecture thus allows modeling of articulatory learning with deep neural networks from raw audio inputs in a fully unsupervised manner. We additionally discuss implications of articulatory representations for cognitive models of human language and speech technology in general.
【22】 Audio Signal Enhancement with Learning from Positive and Unlabelled Data标题：基于正值和未标记数据学习的音频信号增强链接：https://arxiv.org/abs/2210.15143
作者：Nobutaka Ito,Masashi Sugiyama机构：The University of Tokyo, Japan摘要：监督学习是一种主流的音频信号增强方法，它需要由噪声信号和相应的干净信号组成的并行训练数据。这样的数据只能合成，因此与实际数据不匹配，这会导致较差的性能。而且，通常难以/不可能获得干净的信号，这使得难以/不可能应用该方法。这里，我们使用由噪声信号片段和噪声片段组成的非平行训练数据来探索SE，这些数据可以很容易地记录下来。我们将阳性（P）和阴性（N）类别分别定义为信号缺失和存在。我们观察到噪声片段的谱图面片可以用作P数据，而有噪信号片段的谱图面片可以用作未标记数据。这些数据使得卷积神经网络能够通过从正的和未标记的数据学习来学习将每个谱图补丁分类为P或N以用于SE。摘要：Supervised learning is a mainstream approach to audio signal enhancement (SE) and requires parallel training data consisting of noisy signals and the corresponding clean signals. Such data can only be synthesised and are thus mismatched with real data, which can result in poor performance. Moreover, it is often difficult/impossible to obtain clean signals, which makes it difficult/impossible to apply the approach. Here we explore SE using non-parallel training data consisting of noisy signal clips and noise clips, which can be easily recorded. We define the positive (P) and the negative (N) classes as signal absence and presence, respectively. We observe that the spectrogram patches of noise clips can be used as P data and those of noisy signal clips as unlabelled data. These data enable a convolutional neural network to learn to classify each spectrogram patch as P or N for SE through learning from positive and unlabelled data.
【23】 V-Cloak: Intelligibility-, Naturalness- & Timbre-Preserving Real-Time Voice Anonymization标题：V-Cloak：可理解性、自然性和保留音色的实时语音匿名化链接：https://arxiv.org/abs/2210.15140
作者：Jiangyi Deng,Fei Teng,Yanjiao Chen,Xiaofu Chen,Zhaohui Wang,Wenyuan Xu机构：Zhejiang University,Wuhan University备注：Accepted by USENIX Security Symposium 2023摘要：即时消息或社交媒体应用程序上生成的语音数据包含独特的用户声纹，恶意对手可能会滥用这些声纹进行身份推断或身份盗窃。现有的语音匿名化技术，信号处理和语音转换/合成受到感知质量下降的影响。本文开发了一个语音匿名系统V-Cloak，该系统在保持语音可懂度、自然度和音质的同时，实现了实时语音匿名。我们设计的匿名器具有一个一次性生成模型，该模型在不同的频率水平上调制原始音频的特征。我们用精心设计的损失函数训练匿名者。除了匿名性损失外，我们还考虑了可懂度损失和基于心理声学的自然性损失。匿名者可以实现无目标和有目标的匿名化，以达到不可识别和不可链接的匿名目标。 我们在四个数据集上进行了广泛的实验，LibriSpeech（英语）、AISHELL（中文）、CommonVoice（法语）和CommonVoice（意大利语），五个自动说话人验证（ASV）系统（包括两个基于DNN的、两个统计的和一个商业的ASV），以及十一个自动语音识别（ASR）系统（针对不同语言）。实验结果表明，V-Cloak在匿名性能方面优于5个基线.我们还证明了仅在VoxCeleb 1数据集上针对ECAPA-TDNN ASV和DeepSpeech 2 ASR进行训练的V-Cloak相对于其他ASV具有可转移匿名性，并且对于其他ASR具有跨语言可理解性。此外，我们还验证了V-Cloak对各种去噪技术和自适应攻击的鲁棒性。希望V-Cloak能为我们在棱镜世界里提供一件斗篷。摘要：Voice data generated on instant messaging or social media applications contains unique user voiceprints that may be abused by malicious adversaries for identity inference or identity theft. Existing voice anonymization techniques, e.g., signal processing and voice conversion/synthesis, suffer from degradation of perceptual quality. In this paper, we develop a voice anonymization system, named V-Cloak, which attains real-time voice anonymization while preserving the intelligibility, naturalness and timbre of the audio. Our designed anonymizer features a one-shot generative model that modulates the features of the original audio at different frequency levels. We train the anonymizer with a carefully-designed loss function. Apart from the anonymity loss, we further incorporate the intelligibility loss and the psychoacoustics-based naturalness loss. The anonymizer can realize untargeted and targeted anonymization to achieve the anonymity goals of unidentifiability and unlinkability. We have conducted extensive experiments on four datasets, i.e., LibriSpeech (English), AISHELL (Chinese), CommonVoice (French) and CommonVoice (Italian), five Automatic Speaker Verification (ASV) systems (including two DNN-based, two statistical and one commercial ASV), and eleven Automatic Speech Recognition (ASR) systems (for different languages). Experiment results confirm that V-Cloak outperforms five baselines in terms of anonymity performance. We also demonstrate that V-Cloak trained only on the VoxCeleb1 dataset against ECAPA-TDNN ASV and DeepSpeech2 ASR has transferable anonymity against other ASVs and cross-language intelligibility for other ASRs. Furthermore, we verify the robustness of V-Cloak against various de-noising techniques and adaptive attacks. Hopefully, V-Cloak may provide a cloak for us in a prism world.
【24】 Training Autoregressive Speech Recognition Models with Limited in-domain Supervision标题：有限域内监督下的自回归语音识别模型训练链接：https://arxiv.org/abs/2210.15135
作者：Chak-Fai Li,Francis Keith,William Hartmann,Matthew Snover机构：Raytheon BBN, Cambridge MA, USA备注：Submitted to IEEE ICASSP 2023摘要：自监督学习的进步已经显著减少了训练所需的转录音频的量。然而，该领域的大部分工作都集中在朗读语音上。我们在会话言语领域探讨了有限监督。虽然我们假设域内数据的数量是有限的，但我们用开源的读语音数据来扩充模型。XLS-R模型已被证明在有限的适应数据下表现良好，并可作为一个强有力的基线。我们使用非转录数据进行自回归编码器-解码器模型中的自监督学习和半监督训练。我们证明，通过使用假转录的XLS-R模型，当转录的域内数据有限时，小得多的自回归模型可以优于微调的XLS-R模型，将WER绝对值减少多达8%。摘要：Advances in self-supervised learning have significantly reduced the amount of transcribed audio required for training. However, the majority of work in this area is focused on read speech. We explore limited supervision in the domain of conversational speech. While we assume the amount of in-domain data is limited, we augment the model with open source read speech data. The XLS-R model has been shown to perform well with limited adaptation data and serves as a strong baseline. We use untranscribed data for self-supervised learning and semi-supervised training in an autoregressive encoder-decoder model. We demonstrate that by using the XLS-R model for pseudotranscription, a much smaller autoregressive model can outperform a finetuned XLS-R model when transcribed in-domain data is limited, reducing WER by as much as 8% absolute.
【25】 Towards High-Quality Neural TTS for Low-Resource Languages by Learning Compact Speech Representations标题：通过学习紧凑的语音表示实现低资源语言的高质量神经TTS链接：https://arxiv.org/abs/2210.15131
作者：Haohan Guo,Fenglong Xie,Xixin Wu,Hui Lu,Helen Meng机构：∗The Chinese University of Hong Kong, Hong Kong SAR, China, †Xiaohongshu Inc., Shanghai, China备注：Submitted to ICASSP 2023摘要：本文旨在通过使用紧凑语音表示来减少训练数据需求，从而增强低资源TTS。训练多级多码本（MSMC）VQ-GAN以学习表示MSMCR，并将其解码为波形。随后，我们训练多阶段预测器，以从文本中预测MSMCR，用于TTS合成。此外，我们通过利用更多的音频来更好地学习低资源语言的MSMCR，从而优化了训练策略。该算法利用说话人相似性度量从其他语言中选择音频来扩充训练集，并应用迁移学习来提高训练质量。在MOS测试中，所提系统在标准和低资源场景下的性能明显优于FastSpeech和VITS，表现出更低的数据需求。所提出的训练策略有效地提高了MSMCR的波形重构能力。它进一步提高了TTS的性能，在对只有15分钟配对数据的低资源TTS的偏好测试中赢得了77%的投票。摘要：This paper aims to enhance low-resource TTS by reducing training data requirements using compact speech representations. A Multi-Stage Multi-Codebook (MSMC) VQ-GAN is trained to learn the representation, MSMCR, and decode it to waveforms. Subsequently, we train the multi-stage predictor to predict MSMCRs from the text for TTS synthesis. Moreover, we optimize the training strategy by leveraging more audio to learn MSMCRs better for low-resource languages. It selects audio from other languages using speaker similarity metric to augment the training set, and applies transfer learning to improve training quality. In MOS tests, the proposed system significantly outperforms FastSpeech and VITS in standard and low-resource scenarios, showing lower data requirements. The proposed training strategy effectively enhances MSMCRs on waveform reconstruction. It improves TTS performance further, which wins 77% votes in the preference test for the low-resource TTS with only 15 minutes of paired data.
【26】 Knowledge Transfer For On-Device Speech Emotion Recognition with Neural Structured Learning标题：神经结构学习在设备语音情感识别中的知识传递链接：https://arxiv.org/abs/2210.14977
作者：Yi Chang,Zhao Ren,Thanh Tam Nguyen,Kun Qian,Björn W. Schuller机构：GLAM – Group on Language, Audio, & Music, Imperial College London, United Kingdom, L,S Research Center, Leibniz University Hannover, Germany, Griffith University, Australia, Laboratory of Brain Health Engineering (BHE), Beijing Institute of Technology, China备注：Submitted to ICASSP 2023摘要：语音情感识别是人机交互领域的一个研究热点。随着边缘器件的迅速兴起，将SER应用于边缘器件对于大量的HCI应用来说是有前途的。虽然已经研究了深度学习以通过训练复杂模型来提高SER的性能，但是边缘设备的存储空间和计算能力代表了嵌入深度学习模型的约束。通过构建合成图，提出了一种神经结构化学习（NSL）框架. SER模型在源数据集上训练，并用于在目标数据集上构建图。然后，用语音样本和图一起作为输入来训练轻量模型。实验结果表明，在包含语音样本和图的目标数据集上训练一个轻量级的SER模型，不仅可以生成较小的SER模型，而且与仅包含语音样本的模型相比，模型的性能得到了提高.摘要：Speech emotion recognition (SER) has been a popular research topic in human-computer interaction (HCI). As edge devices are rapidly springing up, applying SER to edge devices is promising for a huge number of HCI applications. Although deep learning has been investigated to improve the performance of SER by training complex models, the memory space and computational capability of edge devices represents a constraint for embedding deep learning models. We propose a neural structured learning (NSL) framework through building synthesized graphs. An SER model is trained on a source dataset and used to build graphs on a target dataset. A lightweight model is then trained with the speech samples and graphs together as the input. Our experiments demonstrate that training a lightweight SER model on the target dataset with speech samples and graphs can not only produce small SER models, but also enhance the model performance over models with speech samples only.
【27】 Exploring Effective Distillation of Self-Supervised Speech Models for Automatic Speech Recognition标题：探索用于自动语音识别的自监督语音模型的有效提取链接：https://arxiv.org/abs/2210.15631
作者：Yujin Wang,Changli Tang,Ziyang Ma,Zhisheng Zheng,Xie Chen,Wei-Qiang Zhang机构：Department of Electronic Engineering, Tsinghua University, MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Peng Cheng Laboratory, Shenzhen, China备注：Submitted to ICASSP 2023摘要：近年来，语音处理领域的自监督学习（SSL）取得了长足的发展。SSL模型通常在大量未标记的数据上预先训练，并且优选大的模型大小以增加建模能力。然而，由于超大模型引入的昂贵的计算和存储成本，这可能限制其潜在的应用。SSL模型的小型化已成为具有实用价值的重要研究方向。为此，我们探索了基于HuBERT的SSL模型在自动语音识别（ASR）中的有效提炼。首先，为了建立一个强有力的基线，对不同的学生模型结构进行了全面的研究。在此基础上，作为对以往广泛采用的回归损失的补充，引入了区分损失，以提高HuBERT的精馏性能，尤其是在低资源条件下。此外，我们设计了一种简单有效的算法，将前端输入的波形提取为Fbank特征，在性能略有下降的情况下，参数减少了17%，推理速度提高了一倍。摘要：Recent years have witnessed great strides in self-supervised learning (SSL) on the speech processing. The SSL model is normally pre-trained on a great variety of unlabelled data and a large model size is preferred to increase the modeling capacity. However, this might limit its potential applications due to the expensive computation and memory costs introduced by the oversize model. Miniaturization for SSL models has become an important research direction of practical value. To this end, we explore the effective distillation of HuBERT-based SSL models for automatic speech recognition (ASR). First, in order to establish a strong baseline, a comprehensive study on different student model structures is conducted. On top of this, as a supplement to the regression loss widely adopted in previous works, a discriminative loss is introduced for HuBERT to enhance the distillation performance, especially in low-resource scenarios. In addition, we design a simple and effective algorithm to distill the front-end input from waveform to Fbank feature, resulting in 17% parameter reduction and doubling inference speed, at marginal performance degradation.
【28】 Exploiting spatial information with the informed complex-valued spatial autoencoder for target speaker extraction标题：利用空间信息的信息复值空间自动编码器提取目标说话人链接：https://arxiv.org/abs/2210.15512
作者：Annika Briegleb,Mhd Modar Halimeh,Walter Kellermann机构：Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Erlangen, Germany备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece. 5 pages, 3 figures摘要：在常规的多声道音频信号增强中，空间滤波和频谱滤波通常是顺序执行的。相反，已经表明，对于神经空间滤波，光谱-空间滤波的联合方法更有益。在这篇文章中，我们研究了训练目标对这种时变光谱-空间滤波器的空间选择性的影响。我们扩展了最近提出的复值空间自动编码器（COSPA）用于目标说话人提取，利用其可解释的结构并有目的地通知网络目标说话人的位置。因此，该方法使用能够处理空间和频谱信息的多通道复值神经网络架构，从而提供了一种有效的神经空间滤波方法--通知COSPA（iCOSPA）。我们对iCOSPA进行训练，以实现不同空间处理量的多个训练目标，并分析网络的空间滤波能力。我们发现，所提出的架构确实能够学习不同的空间选择模式，以达到不同的训练目标。摘要：In conventional multichannel audio signal enhancement, spatial and spectral filtering are often performed sequentially. In contrast, it has been shown that for neural spatial filtering a joint approach of spectro-spatial filtering is more beneficial. In this contribution, we investigate the influence of the training target on the spatial selectivity of such a time-varying spectro-spatial filter. We extend the recently proposed complex-valued spatial autoencoder (COSPA) for target speaker extraction by leveraging its interpretable structure and purposefully informing the network of the target speaker's position. Consequently, this approach uses a multichannel complex-valued neural network architecture that is capable of processing spatial and spectral information rendering informed COSPA (iCOSPA) an effective neural spatial filtering method. We train iCOSPA for several training targets that enforce different amounts of spatial processing and analyze the network's spatial filtering capacity. We find that the proposed architecture is indeed capable of learning different spatial selectivity patterns to attain the different training targets.
【29】 Efficient Use of Large Pre-Trained Models for Low Resource ASR标题：针对低资源ASR的大型预训练模型的有效使用链接：https://arxiv.org/abs/2210.15445
作者：Peter Vieting,Christoph Lüscher,Julian Dierkes,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition Group, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：Submitted to ICASSP 2023摘要：自动语音识别（ASR）已经被确立为用于许多其中大量标记数据可用的场景的性能良好的技术。此外，无监督的表征学习最近帮助处理有限数据的任务。在此之后，硬件限制和应用提出了如何有效地利用大型预训练模型并降低其下游任务的复杂性的问题。在这项工作中，我们研究了一个具有挑战性的低资源会话电话语音语料库从医疗领域的越南语和德语。我们展示了使用非监督技术的好处，而不仅仅是对大型预训练模型进行简单的微调，讨论了如何使它们适应包括带宽传输在内的实际电话任务，并研究了预训练和微调的不同数据条件。使用预培训技术，我们比项目基准高出22%。通过改进架构和训练可以进一步获得29%的增益，通过添加0.8h的域内自适应数据可以进一步获得6%的增益。摘要：Automatic speech recognition (ASR) has been established as a well-performing technique for many scenarios where lots of labeled data is available. Additionally, unsupervised representation learning recently helped to tackle tasks with limited data. Following this, hardware limitations and applications give rise to the question how to efficiently take advantage of large pretrained models and reduce their complexity for downstream tasks. In this work, we study a challenging low resource conversational telephony speech corpus from the medical domain in Vietnamese and German. We show the benefits of using unsupervised techniques beyond simple fine-tuning of large pre-trained models, discuss how to adapt them to a practical telephony task including bandwidth transfer and investigate different data conditions for pre-training and fine-tuning. We outperform the project baselines by 22% relative using pretraining techniques. Further gains of 29% can be achieved by refinements of architecture and training and 6% by adding 0.8 h of in-domain adaptation data.
【30】 Time-Domain Based Embeddings for Spoofed Audio Representation标题：一种基于时间域的音频伪装嵌入方法链接：https://arxiv.org/abs/2210.15428
作者：Matan Karo,Arie Yeredor,Itshak Lapidot机构：Tel Aviv University (TAU), School of Electrical Engineering, Israel, Afeka Tel Aviv Academic College of Engineering, ACLP, Israel, Avignon University, LIA, France摘要：反欺骗是语音认证的任务。也就是说，与欺骗语音相比，识别真实的人类语音。本文的主要工作是基于音频波形幅度的概率质量函数（PMF）估计，提出新的真实语音和欺骗语音的表示。 本文介绍了一种新的语音音频信号特征提取方法：与传统方法不同，我们的方法基于时域音频样本的直接处理。通过设计基于不同PMF距离和相似性度量的特征提取器来利用PMF。作为附加步骤，我们使用了滤波器组预处理，这显著地影响了特征的区别性特征，并且促进了对欺骗攻击的可能聚类的方便可视化。此外，我们使用扩散映射来揭示数据所在的基础流形。 所建议的嵌入允许使用简单的线性分隔符来实现良好的性能。此外，我们还提供了一种方便的方法来可视化数据，这有助于评估不同欺骗技术的效率。 实验结果表明，除了使用扩散图作为分析工具和嵌入工具的好处之外，使用基于多通道PMF的特征用于反欺骗任务的潜力。摘要：Anti-spoofing is the task of speech authentication. That is, identifying genuine human speech compared to spoofed speech. The main focus of this paper is to suggest new representations for genuine and spoofed speech, based on the probability mass function (PMF) estimation of the audio waveforms' amplitude. We introduce a new feature extraction method for speech audio signals: unlike traditional methods, our method is based on direct processing of time-domain audio samples. The PMF is utilized by designing a feature extractor based on different PMF distances and similarity measures. As an additional step, we used filter-bank preprocessing, which significantly affects the discriminative characteristics of the features and facilitates convenient visualization of possible clustering of spoofing attacks. Furthermore, we use diffusion maps to reveal the underlying manifold on which the data lies. The suggested embeddings allow the use of simple linear separators to achieve decent performance. In addition, we present a convenient way to visualize the data, which helps to assess the efficiency of different spoofing techniques. The experimental results show the potential of using multi-channel PMF based features for the anti-spoofing task, in addition to the benefits of using diffusion maps both as an analysis tool and as an embedding tool.
【31】 HEiMDaL: Highly Efficient Method for Detection and Localization of wake-words标题：Heimdal：一种高效的唤醒词检测与定位方法链接：https://arxiv.org/abs/2210.15425
作者：Arnav Kundu,Mohammad Samragh Razlighi,Minsik Cho,Priyanka Padmanabhan,Devang Naik摘要：流式关键字识别是一种广泛使用的激活语音助手的解决方案。基于具有隐马尔可夫模型的深度神经网络（DNN-HMM）的方法已经被证明是有效的并且在该空间中被广泛采用，这主要是因为以低计算成本检测和识别唤醒字的开始和结束的能力。然而，当DNN和HMM被独立训练时，这种混合系统遭受损失度量失配。由于运算的固有马尔可夫风格，序列判别训练不能完全减轻损失度量失配。提出了一种低足迹CNN模型HEiMDaL，用于流媒体条件下的关键词检测和定位.我们引入了基于对齐的分类损失来检测关键字的出现，以及偏移损失来预测关键字的开始。HEiMDaL显示出检测度量减少了73%，同时具有等同的定位精度，并且对于给定的唤醒字具有与现有DNN-HMM类型模型相同的存储器占用。摘要：Streaming keyword spotting is a widely used solution for activating voice assistants. Deep Neural Networks with Hidden Markov Model (DNN-HMM) based methods have proven to be efficient and widely adopted in this space, primarily because of the ability to detect and identify the start and end of the wake-up word at low compute cost. However, such hybrid systems suffer from loss metric mismatch when the DNN and HMM are trained independently. Sequence discriminative training cannot fully mitigate the loss-metric mismatch due to the inherent Markovian style of the operation. We propose an low footprint CNN model, called HEiMDaL, to detect and localize keywords in streaming conditions. We introduce an alignment-based classification loss to detect the occurrence of the keyword along with an offset loss to predict the start of the keyword. HEiMDaL shows 73% reduction in detection metrics along with equivalent localization accuracy and with the same memory footprint as existing DNN-HMM style models for a given wake-word.
【32】 Self-Supervised Training of Speaker Encoder with Multi-Modal Diverse Positive Pairs标题：多模不同正对说话人编码者的自监督训练链接：https://arxiv.org/abs/2210.15385
作者：Ruijie Tao,Kong Aik Lee,Rohan Kumar Das,Ville Hautamäki,Haizhou Li机构：Kong Aik Lee is with the Institute for Infocomm Research备注：13 pages摘要：本文研究了一种新的说话人编码器的神经网络结构及其训练策略，用于不使用任何身份标签的说话人识别。说话人编码器被训练成从各种长度的口语话语中提取固定大小的说话人嵌入。对比学习是一种典型的自监督学习技术。然而，说话人编码器的质量在很大程度上取决于正负对的采样策略。我们通常从同一话语中采样一对正片段。不幸的是，这种穷人的正对（PPP）缺乏训练鲁棒编码器所必需的多样性。在本研究中，我们提出了一个多模态对比学习技术与新颖的抽样策略。通过语音和人脸数据的交叉参考，研究了一种寻找多样正对（DPP）进行对比学习的方法，从而提高了说话人编码器的鲁棒性。在VoxCeleb 2数据集上训练说话人编码器，在VoxCeleb 1的三个测试集上，在所提出的渐进聚类策略下，平均错误率分别为2.89%，3.17%和6.27%，在伪标记两阶段学习策略下，平均错误率分别为1.44%，1.77%和3.27%.该方法的性能大大优于现有的自监督学习方法，同时也取得了与监督学习方法相当的效果。我们还在说话人信息未知的LRS 2和LRW数据集上评估了我们的自监督学习技术。所有的实验表明，所提出的神经网络结构和采样策略在数据集上是鲁棒的。摘要：We study a novel neural architecture and its training strategies of speaker encoder for speaker recognition without using any identity labels. The speaker encoder is trained to extract a fixed-size speaker embedding from a spoken utterance of various length. Contrastive learning is a typical self-supervised learning technique. However, the quality of the speaker encoder depends very much on the sampling strategy of positive and negative pairs. It is common that we sample a positive pair of segments from the same utterance. Unfortunately, such poor-man's positive pairs (PPP) lack necessary diversity for the training of a robust encoder. In this work, we propose a multi-modal contrastive learning technique with novel sampling strategies. By cross-referencing between speech and face data, we study a method that finds diverse positive pairs (DPP) for contrastive learning, thus improving the robustness of the speaker encoder. We train the speaker encoder on the VoxCeleb2 dataset without any speaker labels, and achieve an equal error rate (EER) of 2.89\%, 3.17\% and 6.27\% under the proposed progressive clustering strategy, and an EER of 1.44\%, 1.77\% and 3.27\% under the two-stage learning strategy with pseudo labels, on the three test sets of VoxCeleb1. This novel solution outperforms the state-of-the-art self-supervised learning methods by a large margin, at the same time, achieves comparable results with the supervised learning counterpart. We also evaluate our self-supervised learning technique on LRS2 and LRW datasets, where the speaker information is unknown. All experiments suggest that the proposed neural architecture and sampling strategies are robust across datasets.
【33】 Multi-dimensional Edge-based Audio Event Relational Graph Representation Learning for Acoustic Scene Classification标题：基于多维边缘的音频事件关联图表示学习用于声学场景分类链接：https://arxiv.org/abs/2210.15366
作者：Yuanbo Hou,Siyang Song,Chuang Yu,Yuxin Song,Wenwu Wang,Dick Botteldooren机构：Ghent University, Belgium., University of Cambridge, UK., University of Manchester, UK., Baidu Inc., China., University of Surrey, UK.摘要：现有的基于深度学习的声学场景分类（ASC）方法大多直接利用从声谱图中提取的表示来识别目标场景.然而，这些方法很少关注场景中发生的音频事件，尽管它们提供了关键的语义信息。本文首次研究了仅基于描述有限数量音频事件的特征是否能够可靠地识别现实生活中的声学场景。为了对粗粒度声学场景和细粒度音频事件之间的任务相关关系进行建模，提出了一种面向ASC的事件关系图表示学习（ERGL）框架.具体地，ERGL从输入音频中学习声学场景的图形表示，其中每个事件的嵌入被视为节点，而从每对事件嵌入导出的关系线索由学习的多维边缘特征描述。在一个多音声学场景数据集上的实验表明，ERGL算法在ASC上仅使用有限数量的音频事件嵌入，无需任何数据扩充，就能获得较好的性能. ERGL框架的有效性证明了基于事件关系图识别多种声学场景的可行性。我们的代码可以在项目主页上找到（https：//github.com/Yuanbo2020/ERGL）。摘要：Most existing deep learning-based acoustic scene classification (ASC) approaches directly utilize representations extracted from spectrograms to identify target scenes. However, these approaches pay little attention to the audio events occurring in the scene despite they provide crucial semantic information. This paper conducts the first study to investigate whether real-life acoustic scenes can be reliably recognized based only on the features that describe a limited number of audio events. To model the task-specific relationships between coarse-grained acoustic scenes and fine-grained audio events, we propose an event relational graph representation learning (ERGL) framework for ASC. Specifically, the ERGL learns a graph representation of an acoustic scene from the input audio, where the embedding of each event is treated as a node, while the relationship cues derived from each pair of event embeddings are described by a learned multi-dimensional edge feature. Experiments on a polyphonic acoustic scene dataset show that the proposed ERGL achieves competitive performance on ASC by using only a limited number of embeddings of audio events without any data augmentations. The validity of the proposed ERGL framework proves the feasibility of recognizing diverse acoustic scenes based on the event relational graph. Our code is available on project homepage (https://github.com/Yuanbo2020/ERGL).
【34】 Multi-class Detection of Pathological Speech with Latent Features: How does it perform on unseen data?标题：具有潜在特征的病态语音的多类检测：它如何在看不见的数据上执行？链接：https://arxiv.org/abs/2210.15336
作者：Dominik Wagner,Ilja Baumann,Franziska Braun,Sebastian P. Bayerl,Elmar Nöth,Korbinian Riedhammer,Tobias Bocklet机构：†Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, ‡Friedrich-Alexander-University Erlangen-N¨urnberg, Pattern Recognition Lab, Erlangen, Germany, Intel Labs备注：Submitted to ICASSP 2023摘要：从语音特征中检测病理通常被定义为二元分类任务，其中一类表示特定病理，另一类表示健康语音。在这项工作中，我们训练神经网络、大间隔分类器和树提升机器来区分四种不同的病理：帕金森氏病、喉癌、唇腭裂、口腔鳞状细胞癌。我们证明了在预先训练的wav2vec 2.0系统的不同层提取的潜在表示可以有效地用于分类这些类型的病态嗓音。我们通过将房间脉冲响应加到测试数据中并将其应用到未看过的语音语料库中来评估我们的分类器的鲁棒性。根据所使用的模型和噪声条件，我们的方法实现了74.1%和96.4%之间的未加权平均F1分数。该系统对从各种不同源采样的健康说话者的未见数据进行了概括并表现良好。摘要：The detection of pathologies from speech features is usually defined as a binary classification task with one class representing a specific pathology and the other class representing healthy speech. In this work, we train neural networks, large margin classifiers, and tree boosting machines to distinguish between four different pathologies: Parkinson's disease, laryngeal cancer, cleft lip and palate, and oral squamous cell carcinoma. We demonstrate that latent representations extracted at different layers of a pre-trained wav2vec 2.0 system can be effectively used to classify these types of pathological voices. We evaluate the robustness of our classifiers by adding room impulse responses to the test data and by applying them to unseen speech corpora. Our approach achieves unweighted average F1-Scores between 74.1% and 96.4%, depending on the model and the noise conditions used. The systems generalize and perform well on unseen data of healthy speakers sampled from a variety of different sources.
【35】 Robust Data2vec: Noise-robust Speech Representation Learning for ASR by Combining Regression and Improved Contrastive Learning标题：Robust Data2vec：结合回归和改进对比学习的ASR抗噪语音表示学习链接：https://arxiv.org/abs/2210.15324
作者：Qiu-Shi Zhu,Long Zhou,Jie Zhang,Shu-Jie Liu,Yu-Chen Hu,Li-Rong Dai机构：NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China, Microsoft Research Asia, Nanyang Technological University, Singapore备注：Submitted to ICASSP 2023摘要：基于对比学习或回归任务的自监督预训练方法可以利用更多的未标记数据来提高自动语音识别（ASR）的性能。然而，将两种预训练任务相结合并构造不同的负样本进行对比学习，对鲁棒性的影响仍不清楚。本文提出了一种用于自监督语音表示学习的抗噪data 2 vec算法，该算法在预训练阶段联合优化对比学习和回归任务.此外，我们提出了两种改进的方法来促进对比学习。更具体地说，我们首先提出构造基于块的非语义负样本来增强预训练模型的噪声鲁棒性，这是通过将特征划分成不同大小的块来实现的（即，所谓的负样本）。其次，通过分析正负样本的分布，提出了去除易区分的负样本，以提高预训练模型的区分能力。在CHiME-4数据集上的实验结果表明，该方法能够提高预训练模型在噪声环境下的性能.我们发现，对比学习和回归任务的联合训练比只训练回归任务能在一定程度上避免模型崩溃。摘要：Self-supervised pre-training methods based on contrastive learning or regression tasks can utilize more unlabeled data to improve the performance of automatic speech recognition (ASR). However, the robustness impact of combining the two pre-training tasks and constructing different negative samples for contrastive learning still remains unclear. In this paper, we propose a noise-robust data2vec for self-supervised speech representation learning by jointly optimizing the contrastive learning and regression tasks in the pre-training stage. Furthermore, we present two improved methods to facilitate contrastive learning. More specifically, we first propose to construct patch-based non-semantic negative samples to boost the noise robustness of the pre-training model, which is achieved by dividing the features into patches at different sizes (i.e., so-called negative samples). Second, by analyzing the distribution of positive and negative samples, we propose to remove the easily distinguishable negative samples to improve the discriminative capacity for pre-training models. Experimental results on the CHiME-4 dataset show that our method is able to improve the performance of the pre-trained model in noisy scenarios. We find that joint training of the contrastive learning and regression tasks can avoid the model collapse to some extent compared to only training the regression task.
【36】 Learning Music Representations with wav2vec 2.0标题：用Wave2vec 2.0学习音乐表示法链接：https://arxiv.org/abs/2210.15310
作者：Alessandro Ragano,Emmanouil Benetos,Andrew Hines机构：School of CS, University College Dublin, Ireland, School of EECS, Queen Mary University of London, UK备注：Submitted to ICASSP 2023摘要：学习通用的音乐表示提供了使用较小数据集微调若干下游任务的灵活性。wav2vec 2.0语音表示模型在许多下游语音任务中显示出有希望的结果，但在适应音乐时效果较差。在本文中，我们评估了直接在音乐数据上预训练wav2vec 2.0是否比微调语音模型更好。我们说明了当在音乐数据上进行预训练时，离散的潜在表示能够编码音乐概念的语义含义，例如音高和乐器。我们的结果表明，在音乐数据上预训练的微调wav2vec 2.0使我们能够在音乐分类任务上获得有希望的结果，这与先前在音频表示上的工作具有竞争力。另外，在语音嵌入上的实验结果优于预先训练的模型，说明在音乐数据上预先训练的wav2vec 2.0是一种有前途的音乐表示模型。摘要：Learning music representations that are general-purpose offers the flexibility to finetune several downstream tasks using smaller datasets. The wav2vec 2.0 speech representation model showed promising results in many downstream speech tasks, but has been less effective when adapted to music. In this paper, we evaluate whether pre-training wav2vec 2.0 directly on music data can be a better solution instead of finetuning the speech model. We illustrate that when pre-training on music data, the discrete latent representations are able to encode the semantic meaning of musical concepts such as pitch and instrument. Our results show that finetuning wav2vec 2.0 pre-trained on music data allows us to achieve promising results on music classification tasks that are competitive with prior work on audio representations. In addition, the results are superior to the pre-trained model on speech embeddings, demonstrating that wav2vec 2.0 pre-trained on music data can be a promising music representation model.
【37】 Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition标题：加权平均：克服语音识别中灾难性遗忘的一种简单而有效的方法链接：https://arxiv.org/abs/2210.15282
作者：Steven Vander Eeckt,Hugo Van hamme机构：KU Leuven, Department Electrical Engineering ESAT-PSI, Leuven, Belgium备注：Submitted to ICASSP 2023. 5 pages摘要：自动语音识别（ASR）模型在适应新任务时会导致对旧任务的灾难性遗忘，限制了模型的连续学习能力和扩展到新说话人、方言、语言等的能力。针对端到端ASR，本文提出了一种简单有效的方法来克服灾难性遗忘：加权平均。通过简单地取前一个模型和自适应模型的平均值，我们的方法在旧任务和新任务上都获得了高性能。通过在自适应过程中引入知识蒸馏损失，可以进一步改进该方法。最后，我们以单语和多语ASR为例说明了该方法的有效性。在这两种情况下，我们的方法都远远优于所有基线，即使是最简单的形式。摘要：Adapting a trained Automatic Speech Recognition (ASR) model to new tasks results in catastrophic forgetting of old tasks, limiting the model's ability to learn continually and to be extended to new speakers, dialects, languages, etc. Focusing on End-to-End ASR, in this paper, we propose a simple yet effective method to overcome catastrophic forgetting: weight averaging. By simply taking the average of the previous and the adapted model, our method achieves high performance on both the old and new tasks. It can be further improved by introducing a knowledge distillation loss during the adaptation. We illustrate the effectiveness of our method on both monolingual and multilingual ASR. In both cases, our method strongly outperforms all baselines, even in its simplest form.
【38】 A Fast and Accurate Pitch Estimation Algorithm Based on the Pseudo Wigner-Ville Distribution标题：一种基于伪Wigner-Ville分布的快速准确基音估计算法链接：https://arxiv.org/abs/2210.15272
作者：Yisi Liu,Peter Wu,Alan W Black,Gopala K. Anumanchipalli机构：Department of Electronic and Information Engineering, University of Chinese Academy of Sciences, Language Technologies Institute, Carnegie Mellon University, Berkeley Artificial Intelligence Research Lab, University of California, Berkeley摘要：语音信号浊音段基频（F0）的估计，也称为基音跟踪，在基音同步语音分析、语音合成和语音处理中起着至关重要的作用。本文利用伪Wigner-Ville分布（PWVD）的高时频分辨率，提出了一种基于PWVD的基音估计方法。我们设计了一种有效的算法来快速计算PWVD，并使用基于倒谱的预滤波来避免交叉项干扰。在语音和电声门图（EGG）记录的数据库上评估我们的方法产生了大约4 Hz的最先进的平均绝对误差（MAE）。我们的方法在浊音/清音分类和处理突然的频率变化方面也是有效的。摘要：Estimation of fundamental frequency (F0) in voiced segments of speech signals, also known as pitch tracking, plays a crucial role in pitch synchronous speech analysis, speech synthesis, and speech manipulation. In this paper, we capitalize on the high time and frequency resolution of the pseudo Wigner-Ville distribution (PWVD) and propose a new PWVD-based pitch estimation method. We devise an efficient algorithm to compute PWVD faster and use cepstrum-based pre-filtering to avoid cross-term interference. Evaluating our approach on a database with speech and electroglottograph (EGG) recordings yields a state-of-the-art mean absolute error (MAE) of around 4Hz. Our approach is also effective at voiced/unvoiced classification and handling sudden frequency changes.
【39】 Solving Audio Inverse Problems with a Diffusion Model标题：用扩散模型求解音频逆问题链接：https://arxiv.org/abs/2210.15228
作者：Eloi Moliner,Jaakko Lehtinen,Vesa Välimäki机构：Acoustics Lab, Dept. Signal Processing & Acoustics, Department of Computer Science, Aalto University, Espoo, Finland备注：Submitted to ICASSP 2023摘要：提出了一种基于数据驱动的生成式音频模型CQT-Diff，该模型经过训练后，可以在问题不可知的环境下求解各种不同的音频逆问题。CQT-Diff是一种神经扩散模型，其架构经过精心构建，以利用音乐中的音高等变对称性。这是通过用可逆常数Q变换（CQT）预处理模型来实现的，CQT的对数间隔频率轴将音调等方差表示为平移等方差。在三个不同的和变化的任务中，用客观和主观度量来评估所提出的方法：音频带宽扩展、修复和去削波。结果表明，CQT-Diff在音频带宽扩展方面优于对比的基线和消融，并且在不进行再训练的情况下，在音频修复和去削波方面提供了与现代基线相比具有竞争力的性能。这项工作代表了第一个基于扩散的通用框架，用于解决音频处理中的逆问题。摘要：This paper presents CQT-Diff, a data-driven generative audio model that can, once trained, be used for solving various different audio inverse problems in a problem-agnostic setting. CQT-Diff is a neural diffusion model with an architecture that is carefully constructed to exploit pitch-equivariant symmetries in music. This is achieved by preconditioning the model with an invertible Constant-Q Transform (CQT), whose logarithmically-spaced frequency axis represents pitch equivariance as translation equivariance. The proposed method is evaluated with objective and subjective metrics in three different and varied tasks: audio bandwidth extension, inpainting, and declipping. The results show that CQT-Diff outperforms the compared baselines and ablations in audio bandwidth extension and, without retraining, delivers competitive performance against modern baselines in audio inpainting and declipping. This work represents the first diffusion-based general framework for solving inverse problems in audio processing.
【40】 HRTF Field: Unifying Measured HRTF Magnitude Representation with Neural Fields标题：HRTF场：统一测量的HRTF震级表示和神经场链接：https://arxiv.org/abs/2210.15196
作者：You Zhang,Yuxiang Wang,Zhiyao Duan机构：University of Rochester, Rochester, NY, USA备注：5 pages, submitted to ICASSP 2023摘要：头部相关传递函数（HRTF）是描述外耳的空间滤波效应的一组频率函数（即，躯干、头部和耳廓）以不同的方位角和仰角投射到声源上。它们被广泛地用于空间音频渲染。虽然方位角和仰角在本质上是连续的，但现有数据集中测量的HRTF采用不同的空间采样方案，使得难以跨数据集建模HRTF。在这项工作中，我们建议使用神经场，一种通过神经网络的函数的可微表示，来模拟具有任意空间采样方案的HRTF。这种表示在具有不同空间采样方案的数据集上是统一的，并且可以从这种表示中导出任意方位角和仰角的HRTF。进一步引入HRTF场生成模型，学习HRTF神经场的潜在空间。我们在HRTF插值和生成任务上展示了有前途的性能，并指出了潜在的未来工作。摘要：Head-related transfer functions (HRTFs) are a set of functions of frequency describing the spatial filtering effect of the outer ear (i.e., torso, head, and pinnae) onto sound sources at different azimuth and elevation angles. They are widely used in spatial audio rendering. While the azimuth and elevation angles are intrinsically continuous, measured HRTFs in existing datasets employ different spatial sampling schemes, making it difficult to model HRTFs across datasets. In this work, we propose to use neural fields, a differentiable representation of functions through neural networks, to model HRTFs with arbitrary spatial sampling schemes. Such representation is unified across datasets with different spatial sampling schemes and HRTFs for arbitrary azimuth and elevation angles can be derived from this representation. We further introduce a generative model named HRTF field to learn a latent space of the HRTF neural fields. We demonstrate promising performance on HRTF interpolation and generation tasks and point out potential future work.
【41】 Masked Autoencoders Are Articulatory Learners标题：蒙面自动编码器是节能型学习者链接：https://arxiv.org/abs/2210.15195
作者：Ahmed Adel Attia,Carol Espy-Wilson机构：University Of Maryland, College Park, Institute For Systems Research, Maryland, USA摘要：发音记录跟踪沿声道的不同发音器的位置和运动，并且被广泛用于研究语音产生和开发语音技术，例如基于发音的语音合成器和语音反转系统。威斯康星大学X射线微束（XRMB）数据集是提供与音频记录同步的发音记录的各种数据集之一。XRMB发音记录采用放置在多个发音器上的小球，这些小球可以被微束跟踪。然而，发音记录的很大一部分是错误的，到目前为止是无法起诉的。在这项工作中，我们提出了一种基于深度学习的方法，使用掩蔽自动编码器来准确重建XRMB数据集47个说话者中的41个的错误发音记录。我们的模型能够重建发音轨迹，与真实情况非常接近，即使八个发音器中有三个轨迹错误，也能从3.4小时以前无法使用的记录中检索出3.28个。摘要：Articulatory recordings track the positions and motion of different articulators along the vocal tract and are widely used to study speech production and to develop speech technologies such as articulatory based speech synthesizers and speech inversion systems. The University of Wisconsin X-Ray microbeam (XRMB) dataset is one of various datasets that provide articulatory recordings synced with audio recordings. The XRMB articulatory recordings employ pellets placed on a number of articulators which can be tracked by the microbeam. However, a significant portion of the articulatory recordings are mistracked, and have been so far unsuable. In this work, we present a deep learning based approach using Masked Autoencoders to accurately reconstruct the mistracked articulatory recordings for 41 out of 47 speakers of the XRMB dataset. Our model is able to reconstruct articulatory trajectories that closely match ground truth, even when three out of eight articulators are mistracked, and retrieve 3.28 out of 3.4 hours of previously unusable recordings.
【42】 Streaming Voice Conversion Via Intermediate Bottleneck Features And Non-streaming Teacher Guidance标题：通过中间瓶颈功能和非流媒体教师指导实现流媒体语音转换链接：https://arxiv.org/abs/2210.15158
作者：Yuanzhe Chen,Ming Tu,Tang Li,Xin Li,Qiuqiang Kong,Jiaxin Li,Zhichao Wang,Qiao Tian,Yuping Wang,Yuxuan Wang机构：Speech, Audio & Music Intelligence (SAMI), ByteDance备注：The paper has been submitted to ICASSP2023摘要：流式语音转换（VC）是将一个人的语音实时转换为另一个人的语音的任务。先前的流式VC方法使用从自动语音识别（ASR）系统提取的语音后验图（PPG）来表示说话者无关的信息。然而，PPG缺乏源说话人的韵律和发声信息，并且流式PPG包含不期望的源说话人的泄漏音色。本文提出用中间瓶颈特征（IBFs）来代替PPG。用IBF训练的VC系统保留了更多的源说话人的韵律和发声信息。此外，我们提出了一个非流式教师指导（TG）框架，以解决音色泄漏问题。实验结果表明，在160ms的未来感受野下，我们提出的IBF和TG框架获得了3.85的流VC自然度、3.77的内容一致性和3.77的音色相似度，明显优于现有的流VC系统.摘要：Streaming voice conversion (VC) is the task of converting the voice of one person to another in real-time. Previous streaming VC methods use phonetic posteriorgrams (PPGs) extracted from automatic speech recognition (ASR) systems to represent speaker-independent information. However, PPGs lack the prosody and vocalization information of the source speaker, and streaming PPGs contain undesired leaked timbre of the source speaker. In this paper, we propose to use intermediate bottleneck features (IBFs) to replace PPGs. VC systems trained with IBFs retain more prosody and vocalization information of the source speaker. Furthermore, we propose a non-streaming teacher guidance (TG) framework that addresses the timbre leakage problem. Experiments show that our proposed IBFs and the TG framework achieve a state-of-the-art streaming VC naturalness of 3.85, a content consistency of 3.77, and a timbre similarity of 3.77 under a future receptive field of 160 ms which significantly outperform previous streaming VC systems.
【43】 Acoustically-Driven Phoneme Removal That Preserves Vocal Affect Cues标题：保留声音情感线索的声学驱动音素去除链接：https://arxiv.org/abs/2210.15001
作者：Camille Noufi,Jonathan Berger,Michael Frank,Karen Parker,Daniel L. Bowling机构：Stanford University, Center for Computer Research, in Music and Acoustics, Stanford, CA, USA, Stanford School of Medicine, Department of Psychiatry, and Behavioral Sciences备注：Submitted to the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：本文提出了一种从语音中去除语言信息的方法，以分离副语言的情感指标。该方法的直接效用在于对声音情感敏感性的临床测试，该测试不受语言的混淆，而语言在各种临床人群中受损。该方法基于语音音频和电声门图（EGG）信号的同时记录。语音音频信号被用于估计平均声道滤波器响应和幅度包络。EGG信号提供了与语音源活动的直接相关，其大部分独立于语音清晰度。这些信号被用于创建第三信号，该第三信号被设计成从发声系统捕获尽可能多的副语言信息--最大化要影响的生物声学线索的保留--同时消除对言语意义的语音线索。为了评估该方法的成功性，我们在一个在线听众的情感评级实验中研究了相应语音音频和转换后的EGG信号的感知。实验结果表明，匹配信号在感知情感上具有较高的相似度，表明该方法是有效的。摘要：In this paper, we propose a method for removing linguistic information from speech for the purpose of isolating paralinguistic indicators of affect. The immediate utility of this method lies in clinical tests of sensitivity to vocal affect that are not confounded by language, which is impaired in a variety of clinical populations. The method is based on simultaneous recordings of speech audio and electroglottographic (EGG) signals. The speech audio signal is used to estimate the average vocal tract filter response and amplitude envelop. The EGG signal supplies a direct correlate of voice source activity that is mostly independent of phonetic articulation. These signals are used to create a third signal designed to capture as much paralinguistic information from the vocal production system as possible -- maximizing the retention of bioacoustic cues to affect -- while eliminating phonetic cues to verbal meaning. To evaluate the success of this method, we studied the perception of corresponding speech audio and transformed EGG signals in an affect rating experiment with online listeners. The results show a high degree of similarity in the perceived affect of matched signals, indicating that our method is effective.
【44】 Privacy-preserving Automatic Speaker Diarization标题：保护隐私的自动说话人二值化链接：https://arxiv.org/abs/2210.14995
作者：Francisco Teixeira,Alberto Abad,Bhiksha Raj,Isabel Trancoso机构：INESC-IDIST, University of Lisbon, Portugal,LTI, Carnegie Mellon University, USA摘要：自动扬声器拨号（ASD）是一种具有众多应用的使能技术，它处理多个扬声器的录音，在隐私方面引起了特别的关注。事实上，在远程设置中，录音与服务器共享，客户端不仅放弃了他们谈话的隐私，而且放弃了可以从他们的声音推断出的所有信息。然而，据我们所知，迄今为止，保护隐私的ASD系统的开发一直被忽视。在本文中，我们使用两种密码技术的组合来解决这个问题，安全多方计算（SMC）和安全模块哈希，并将它们应用于级联ASD系统的两个主要步骤：说话人嵌入提取和凝聚层次聚类。我们的系统能够在性能和效率之间实现合理的折衷，对于两种不同的SMC安全设置，呈现1.1和1.6的实时因子。摘要：Automatic Speaker Diarization (ASD) is an enabling technology with numerous applications, which deals with recordings of multiple speakers, raising special concerns in terms of privacy. In fact, in remote settings, where recordings are shared with a server, clients relinquish not only the privacy of their conversation, but also of all the information that can be inferred from their voices. However, to the best of our knowledge, the development of privacy-preserving ASD systems has been overlooked thus far. In this work, we tackle this problem using a combination of two cryptographic techniques, Secure Multiparty Computation (SMC) and Secure Modular Hashing, and apply them to the two main steps of a cascaded ASD system: speaker embedding extraction and agglomerative hierarchical clustering. Our system is able to achieve a reasonable trade-off between performance and efficiency, presenting real-time factors of 1.1 and 1.6, for two different SMC security settings.
机器翻译，仅供参考
每日语音速递