今日论文合集：cs.SD语音12篇，eess.AS音频处理15篇。
cs.SD语音
【1】GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Speech Emotion Recognition
标题：GEMO-CLAP：用于语音情感识别的性别增强型对比语言-音频预训练
链接：http://arxiv.org/pdf/2306.07848v1
作者：Yu Pan,Yanni Hu,Yuguang Yang,Jixun Yao,Wen Fei,Lei Ma,Heng Lu机构：University of Alberta, Edmonton, Canada, The University of Tokyo, Japan, Ximalaya Inc., ShangHai, China, Shanghai Jiao Tong University, ShangHai, China备注：5 pages摘要：对比语言音频预训练（CLAP）最近在不同领域取得了令人印象深刻的成功。在本文中，我们提出了GEmo-CLAP，一种有效的性别属性增强CLAP模型的语音情感识别（SER）。具体来说，我们首先建立一个有效的情感CLAP模型，称为Emo-CLAP SER，利用各种基于自监督学习的预训练模型。然后，考虑到性别属性在语音情感建模中的重要性，进一步提出了两种GEmo-CLAP方法，以融合语音信号的情感和性别信息，形成更合理的目标。在IEMOCAP语料库上进行的大量实验表明，我们提出的两种GEmo-CLAP方法在不同的预训练模型下始终优于基线Emo-CLAP，同时与其他最先进的方法相比，也实现了更好的识别性能。摘要：Contrastive Language-Audio Pretraining (CLAP) has recently exhibited impressive success in diverse fields. In this paper, we propose GEmo-CLAP, a kind of efficient gender-attribute-enhanced CLAP model for speech emotion recognition (SER). Specifically, we first build an effective emotion CLAP model termed Emo-CLAP for SER, utilizing various self-supervised learning based pre-trained models. Then, considering the importance of the gender attribute in speech emotion modeling, two GEmo-CLAP approaches are further proposed to integrate the emotion and gender information of speech signals, forming more reasonable objectives. Extensive experiments conducted on the IEMOCAP corpus demonstrate that our proposed two GEmo-CLAP approaches consistently outperform the baseline Emo-CLAP with different pre-trained models, while also achieving superior recognition performance compared with other state-of-the-art methods.
【2】Unlocking Foundation Models for Privacy-Enhancing Speech Understanding: An Early Study on Low Resource Speech Training Leveraging Label-guided Synthetic Speech Content
标题：解锁隐私增强语音理解的基础模型：利用标签引导的合成语音内容的低资源语音训练的早期研究
链接：http://arxiv.org/pdf/2306.07791v1
作者：Tiantian Feng,Digbalay Bose,Xuan Shi,Shrikanth Narayanan机构：Signal Analysis and Interpretation Laboratory, University of Southern California, USA摘要：自动语音理解（ASU）利用深度学习模型的强大功能来准确解释人类语音，从而产生广泛的语音应用，丰富人类体验。然而，训练一个强大的ASU模型需要管理大量的语音样本，这会带来隐私泄露的风险。在这项工作中，我们调查使用基础模型，以协助隐私增强语音计算。与主要关注数据扰动或分布式算法的传统作品不同，我们的工作研究了使用预训练的生成模型来合成语音内容作为训练数据的可能性，仅使用标签指导。我们表明，zero-shot学习与训练标签引导的合成语音内容仍然是一项具有挑战性的任务。另一方面，我们的研究结果表明，用合成语音样本训练的模型为低资源ASU训练提供了有效的初始化点。这一结果揭示了通过减少用户数据收集但使用标签引导的合成语音内容来增强隐私的潜力。摘要：Automatic Speech Understanding (ASU) leverages the power of deep learning models for accurate interpretation of human speech, leading to a wide range of speech applications that enrich the human experience. However, training a robust ASU model requires the curation of a large number of speech samples, creating risks for privacy breaches. In this work, we investigate using foundation models to assist privacy-enhancing speech computing. Unlike conventional works focusing primarily on data perturbation or distributed algorithms, our work studies the possibilities of using pre-trained generative models to synthesize speech content as training data with just label guidance. We show that zero-shot learning with training label-guided synthetic speech content remains a challenging task. On the other hand, our results demonstrate that the model trained with synthetic speech samples provides an effective initialization point for low-resource ASU training. This result reveals the potential to enhance privacy by reducing user data collection but using label-guided synthetic speech content.
【3】Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages
标题：基于对比学习的多语言音歌词对齐
链接：http://arxiv.org/pdf/2306.07744v1
作者：Simon Durand,Daniel Stoller,Sebastian Ewert机构：Spotify Equal contributionJournal-ref：ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5摘要：近年来，歌词对齐获得了相当大的关注。最先进的系统要么重新使用已建立的语音识别工具包，要么设计涉及连接主义时间分类（CTC）损失的端到端解决方案。然而，这两种方法都有特定的缺点：工具包以其复杂性而闻名，CTC系统使用为转录设计的损失，这可能限制比对准确性。在本文中，我们使用的是一个对比学习过程，得出跨模态嵌入链接的音频和文本域。通过这种方式，我们获得了一个新的系统，该系统易于端到端训练，可以利用弱注释的训练数据，联合学习强大的文本模型，并针对对齐量身定制。该系统不仅是第一个在标准Jamendo数据集上产生低于0.2秒的平均绝对误差的系统，而且它对其他语言也是鲁棒的，即使只在英语数据上训练。最后，我们发布了JamendoLyrics Multi-Lang数据集的单词级对齐。摘要：Lyrics alignment gained considerable attention in recent years. State-of-the-art systems either re-use established speech recognition toolkits, or design end-to-end solutions involving a Connectionist Temporal Classification (CTC) loss. However, both approaches suffer from specific weaknesses: toolkits are known for their complexity, and CTC systems use a loss designed for transcription which can limit alignment accuracy. In this paper, we use instead a contrastive learning procedure that derives cross-modal embeddings linking the audio and text domains. This way, we obtain a novel system that is simple to train end-to-end, can make use of weakly annotated training data, jointly learns a powerful text model, and is tailored to alignment. The system is not only the first to yield an average absolute error below 0.2 seconds on the standard Jamendo dataset but it is also robust to other languages, even when trained on English data only. Finally, we release word-level alignments for the JamendoLyrics Multi-Lang dataset.
【4】Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation
标题：情态适应还是正规化？端到端语音翻译的个案研究
链接：http://arxiv.org/pdf/2306.07650v1
作者：JYuchen Han,Chen Xu,Tong Xiao,Jingbo Zhu机构：School of Computer Science and Engineering, Northeastern University, Shenyang, China, NiuTrans Research, Shenyang, China备注：ACL 2023 Main Conference摘要：预训练和微调是缓解端到端语音翻译（E2 E ST）中的数据稀缺问题的范例。语音和文本数据之间常见的“模态间隙”通常导致预训练和微调之间的输入不一致。然而，我们观察到这种差距出现在微调的早期阶段，但不会对最终性能产生重大影响。另一方面，我们发现还有另一个缺口，我们称之为“能力缺口”：高资源任务（如ASR和MT）通常需要一个大的模型来拟合，当模型被重用于低资源任务（E2 E ST）时，由于过度拟合，它将获得次优性能。在一个案例研究中，我们发现正则化比精心设计的模态自适应方法起着更重要的作用，在MuST-C数据集上，en-de的正则化方法达到了29.0，en-fr的正则化方法达到了40.3。代码和模型可在www.example.com上获得https: github.com hannlp TAB。摘要：Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in end-to-end speech translation (E2E ST). The commonplace "modality gap" between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning. However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance. On the other hand, we find that there has another gap, which we call the "capacity gap": high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the over-fitting. In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for en-fr on the MuST-C dataset. Code and models are available at https: github.com hannlp TAB.
【5】UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
标题：UniCATS：一种支持上下文VQ扩散和语音编码的统一上下文感知文语转换框架
链接：http://arxiv.org/pdf/2306.07547v1
作者：Chenpeng Du,Yiwei Guo,Feiyu Shen,Zhijun Liu,Zheng Liang,Xie Chen,Shuai Wang,Hui Zhang,Kai Yu机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Shenzhen Research Institute of Big Data, Shenzhen, China, AISpeech Ltd, Beijing, China摘要：利用离散语音令牌，分为语义令牌和声学令牌，已被证明优于传统的声学特征梅尔频谱图的自然性和鲁棒性的文本到语音（TTS）合成。最近流行的模型，如VALL-E和SPEAR-TTS，允许通过自回归（AR）连续的声学令牌提取的短语音提示zero-shot扬声器自适应。然而，这些AR模型被限制为仅在从左到右的方向上生成语音，使得它们不适合于其中提供了先前和随后的上下文两者的语音编辑。此外，这些模型依赖于声学令牌，其具有由音频编解码器模型的性能施加的音频质量限制。在这项研究中，我们提出了一个统一的上下文感知的TTS框架称为UniCATS，这是能够语音延续和编辑。UniCATS包括两个组件，声学模型CTX-txt 2 vec和声码器CTX-vec 2 wav。CTX-txt 2 vec采用上下文VQ扩散来预测来自输入文本的语义令牌，使其能够结合语义上下文并保持与周围上下文的无缝连接。之后，CTX-vec 2 wav利用上下文声码将这些语义标记转换为波形，同时考虑到声学上下文。我们的实验结果表明，CTX-vec 2 wav优于HifiGAN和AudioLM的语音再合成从语义令牌。此外，我们表明，UniCATS实现国家的最先进的性能，在语音延续和编辑。摘要：The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing.
【6】FOOCTTS: Generating Arabic Speech with Acoustic Environment for Football Commentator
标题：FOOCTTS：足球解说员使用声学环境生成阿拉伯语语音
链接：http://arxiv.org/pdf/2306.07936v1
作者：Massa Baali,Ahmed Ali机构：Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Qatar Computing Research Institute, HBKU, Doha, Qatar备注：Accepted at Interspeech 2023 Show & Tell Demo Session摘要：本文介绍了FOOCTTS，一个自动流水线的足球评论员，产生语音与背景人群噪声。应用程序从用户处获取文本，应用文本预处理（如元音化），然后是评论员的语音合成器。我们的管道包括阿拉伯语自动语音识别的数据标签，CTC分割，转录元音匹配语音，并微调TTS。我们的系统能够在有限的15分钟的足球评论员录音的声学环境中生成语音。我们的原型是可推广的，可以很容易地应用到不同的领域和语言。摘要：This paper presents FOOCTTS, an automatic pipeline for a football commentator that generates speech with background crowd noise. The application gets the text from the user, applies text pre-processing such as vowelization, followed by the commentator's speech synthesizer. Our pipeline included Arabic automatic speech recognition for data labeling, CTC segmentation, transcription vowelization to match speech, and fine-tuning the TTS. Our system is capable of generating speech with its acoustic environment within limited 15 minutes of football commentator recording. Our prototype is generalizable and can be easily applied to different domains and languages.
【7】A Theory of Unsupervised Speech Recognition
标题：一种无监督语音识别理论
链接：http://arxiv.org/pdf/2306.07926v1
作者：Liming Wang,Mark Hasegawa-Johnson,Chang D. Yoo机构：University of Illinois Urbana-Champaign, Korea Advanced Institute of Science Technology摘要：无监督语音识别（ASR-U）是从非配对的纯语音和纯文本语料库中学习自动语音识别（ASR）系统的问题。虽然存在各种算法来解决这个问题，但在研究它们的属性和解决对超参数的敏感性和训练不稳定性等问题时，缺少理论框架。本文基于随机矩阵理论和神经切核理论，提出了一个研究ASR-U系统性质的一般理论框架。这样的框架使我们能够证明各种可学习性条件和样本的复杂性范围的ASR-U。在具有三类转换图的合成语言上进行的广泛的ASR-U实验为我们的理论提供了强有力的经验证据（代码可在cactuswiththoughts UnsupASRTheory.git获得）。摘要：Unsupervised speech recognition (ASR-U) is the problem of learning automatic speech recognition (ASR) systems from unpaired speech-only and text-only corpora. While various algorithms exist to solve this problem, a theoretical framework is missing from studying their properties and addressing such issues as sensitivity to hyperparameters and training instability. In this paper, we proposed a general theoretical framework to study the properties of ASR-U systems based on random matrix theory and the theory of neural tangent kernels. Such a framework allows us to prove various learnability conditions and sample complexity bounds of ASR-U. Extensive ASR-U experiments on synthetic languages with three classes of transition graphs provide strong empirical evidence for our theory (code available at cactuswiththoughts UnsupASRTheory.git).
【8】Unsupervised speech enhancement with deep dynamical generative speech and noise models
标题：基于深度动态生成语音和噪声模型的非监督语音增强
链接：http://arxiv.org/pdf/2306.07820v1
作者：Xiaoyu Lin,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda机构：Inria Grenoble Rhˆone-Alpes, Univ. Grenoble Alpes, France, CentraleSup´elec, IETR (UMR CNRS ,), France, Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France摘要：这项工作建立在以前的工作上的无监督语音增强使用动态变分自动编码器（DVAE）作为干净的语音模型和非负矩阵分解（NMF）作为噪声模型。我们建议用深度动态生成模型（DDGM）取代NMF噪声模型，该模型取决于DVAE潜变量，或噪声观测，或两者兼而有之。该DDGM可以在三种配置中进行训练：噪声不可知、噪声相关和噪声相关训练后的噪声适应。实验结果表明，所提出的方法相比，国家的最先进的无监督语音增强方法，实现了有竞争力的性能，而噪声相关的训练配置产生一个更有效的时间推理过程。摘要：This work builds on a previous work on unsupervised speech enhancement using a dynamical variational autoencoder (DVAE) as the clean speech model and non-negative matrix factorization (NMF) as the noise model. We propose to replace the NMF noise model with a deep dynamical generative model (DDGM) depending either on the DVAE latent variables, or on the noisy observations, or on both. This DDGM can be trained in three configurations: noise-agnostic, noise-dependent and noise adaptation after noise-dependent training. Experimental results show that the proposed method achieves competitive performance compared to state-of-the-art unsupervised speech enhancement methods, while the noise-dependent training configuration yields a much more time-efficient inference process.
【9】StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
标题：StyleTTS 2：通过大语音语言模型的风格扩散和对抗性训练实现人类级别的文语转换
链接：http://arxiv.org/pdf/2306.07691v1
作者：Yinghao Aaron Li,Cong Han,Vinay S. Raghavan,Gavin Mischler,Nima Mesgarani机构：Columbia University摘要：在本文中，我们提出了StyleTTS 2，一个文本到语音（TTS）模型，利用风格扩散和对抗训练与大型语音语言模型（SLM），以实现人类水平的TTS合成。StyleTTS 2与其前身的不同之处在于通过扩散模型将风格建模为潜在随机变量，以生成最适合文本的风格，而无需参考语音，实现高效的潜在扩散，同时受益于扩散模型提供的多样化语音合成。此外，我们采用大型的预训练SLM，如WavLM，作为鉴别器，我们的新型可微分持续时间建模用于端到端训练，从而提高了语音的自然度。StyleTTS 2在单说话者LJSpeech数据集上超越了人类录音，并在多说话者VCTK数据集上与之相匹配，由英语母语者判断。此外，当在LibriTTS数据集上进行训练时，我们的模型在zero-shot扬声器自适应方面优于以前公开的模型。这项工作在单说话者和多说话者数据集上实现了第一个人类水平的TTS，展示了使用大型SLM进行风格扩散和对抗性训练的潜力。音频演示和源代码可在www.example.com上获得https: styletts2.github.io 。摘要：In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https: styletts2.github.io .
【10】Statistical Beamformer Exploiting Non-stationarity and Sparsity with Spatially Constrained ICA for Robust Speech Recognition
标题：基于空间约束ICA的统计波束形成器非平稳性和稀疏性的稳健语音识别
链接：http://arxiv.org/pdf/2306.07562v1
作者：Ui-Hyeop Shin,Hyung-Min Park机构：This work was partly supported by Institute of Information & communi-cations Technology Planning & evaluation (IITP) grant funded by the Koreagovernment (MSIT) (No摘要：在本文中，我们提出了一种统计波束形成算法作为鲁棒自动语音识别（ASR）的预处理步骤。通过将目标语音建模为非平稳拉普拉斯分布，提出了一种基于掩模的统计波束形成算法，以利用其输出和掩模输入方差进行波束形成器的鲁棒估计。此外，我们还提出了一种方法的导向矢量估计（SVE）的基础上获得的目标和噪声输出的独立分量分析（ICA）的噪声功率比。为了在相同的ICA框架中更新波束形成器，我们推导出目标语音上具有无失真和零约束的ICA，其分别在目标输出处产生波束形成语音和在其他输出处产生噪声。目标输出的解混权重导致具有加权空间协方差矩阵（wSCM）的统计波束形成器，其使用由源模型表征的加权函数。为了提高SVE，严格的零约束施加的拉格朗日乘子方法放宽广义惩罚权参数，而严格的无失真的约束保持。此外，我们推导出一个在线算法的基础上的递归最小二乘（RLS）的实际应用的优化技术。使用CHiME-4和LibriCSS数据集在各种环境下的实验结果表明，相比传统的波束形成和盲源提取（BSE）ICA的基础上的批处理和在线处理所提出的算法的有效性。摘要：In this paper, we present a statistical beamforming algorithm as a pre-processing step for robust automatic speech recognition (ASR). By modeling the target speech as a non-stationary Laplacian distribution, a mask-based statistical beamforming algorithm is proposed to exploit both its output and masked input variance for robust estimation of the beamformer. In addition, we also present a method for steering vector estimation (SVE) based on a noise power ratio obtained from the target and noise outputs in independent component analysis (ICA). To update the beamformer in the same ICA framework, we derive ICA with distortionless and null constraints on target speech, which yields beamformed speech at the target output and noises at the other outputs, respectively. The demixing weights for the target output result in a statistical beamformer with the weighted spatial covariance matrix (wSCM) using a weighting function characterized by a source model. To enhance the SVE, the strict null constraints imposed by the Lagrange multiplier methods are relaxed by generalized penalties with weight parameters, while the strict distortionless constraints are maintained. Furthermore, we derive an online algorithm based on an optimization technique of recursive least squares (RLS) for practical applications. Experimental results on various environments using CHiME-4 and LibriCSS datasets demonstrate the effectiveness of the presented algorithm compared to conventional beamforming and blind source extraction (BSE) based on ICA on both batch and online processing.
【11】Speaker Verification Across Ages: Investigating Deep Speaker Embedding Sensitivity to Age Mismatch in Enrollment and Test Speech
标题：跨年龄说话人验证：研究深度说话人在注册和测试演讲中对年龄不匹配的敏感性
链接：http://arxiv.org/pdf/2306.07501v1
作者：Vishwanath Pratap Singh,Md Sahidullah,Tomi Kinnunen机构：University of Eastern Finland, Finland, Institute for Advancing Intelligence, TCG CREST, IndiaJournal-ref：Interspeech 2023摘要：在本文中，我们研究了老化对现代基于深度说话人嵌入的自动说话人验证（ASV）系统的影响。我们选择了两个不同的数据集来检查最先进的ECAPA-TDNN系统的老化。第一个数据集是VoxCeleb，用于解决在不受控制的条件下的短期老化（注册和测试之间长达10年的时间差）。第二个数据集，用于解决长期老化效应（高达40年的差异）的芬兰语扬声器下一个更受控的设置，是纵向语料库芬兰语口语在赫尔辛基（LCFSH）。我们的研究提供了新的见解扬声器老化对现代ASV系统的影响。具体来说，我们建立了一个定量的衡量老化和ASV分数。此外，我们的研究表明，老龄化影响女性英语发言人在更大程度上比男性英语发言人，而在芬兰的情况下，它有一个更大的影响男性发言人比女性发言人。摘要：In this paper, we study the impact of the ageing on modern deep speaker embedding based automatic speaker verification (ASV) systems. We have selected two different datasets to examine ageing on the state-of-the-art ECAPA-TDNN system. The first dataset, used for addressing short-term ageing (up to 10 years time difference between enrollment and test) under uncontrolled conditions, is VoxCeleb. The second dataset, used for addressing long-term ageing effect (up to 40 years difference) of Finnish speakers under a more controlled setup, is Longitudinal Corpus of Finnish Spoken in Helsinki (LCFSH). Our study provides new insights into the impact of speaker ageing on modern ASV systems. Specifically, we establish a quantitative measure between ageing and ASV scores. Further, our research indicates that ageing affects female English speakers to a greater degree than male English speakers, while in the case of Finnish, it has a greater impact on male speakers than female speakers.
【12】PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling
标题：PauseSpeech：基于预训练语言模型和基于停顿的韵律建模的自然语音合成
链接：http://arxiv.org/pdf/2306.07489v1
作者：Ji-Sang Hwang,Sang-Hoon Lee,Seong-Whan Lee机构：Department of Artificial Intelligence, Korea University, Seoul, Korea备注：13 pages, 4 figures, 3 tables, under reivew摘要：虽然文本到语音（TTS）系统已经显著改进，但是大多数TTS系统在合成具有适当措辞的语音方面仍然具有限制。对于自然语音合成，重要的是合成具有基于语义信息将单词分组为短语的短语结构的语音。在本文中，我们提出了PuaseSpeech，一个语音合成系统与预训练的语言模型和基于停顿的韵律建模。首先，我们引入了一个短语结构编码器，它利用了来自预训练语言模型的上下文表示。在短语结构编码器中，我们从上下文表示中提取依赖于说话者的句法表示，然后预测将输入文本分成短语的暂停序列。此外，我们还引入了一个基于停顿的单词编码器来建立基于停顿序列的单词级韵律模型。实验结果表明，PauseSpeech在自然度方面优于以前的模型。此外，在客观评价方面，我们可以观察到，我们提出的方法有助于模型减少地面实况和合成语音之间的距离。音频样本可在www.example.com上获得https: jisang93.github.io pausespeech-demo。摘要：Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https: jisang93.github.io pausespeech-demo .
eess.AS音频处理
【1】Improving Frame-level Classifier for Word Timings with Non-peaky CTC in End-to-End Automatic Speech Recognition
标题：端到端自动语音识别中非峰值CTC改进的帧级单词计时分类器
链接：http://arxiv.org/pdf/2306.07949v1
作者：Xianzhao Chen,Yist Y. Lin,Kang Wang,Yi He,Zejun Ma机构：ByteDance备注：To appear in the proceedings of INTERSPEECH 2023摘要：端到端（E2 E）系统已经显示出与用于自动语音识别（ASR）的混合系统相当的性能。作为ASR的副产品，单词计时在许多应用中是必不可少的，特别是对于字幕和计算机辅助发音训练。在本文中，我们改进了帧级分类器在E2 E系统中的字定时，通过引入标签先验的连接时间分类（CTC）损失，这是通过从以前的作品，并结合低级别的梅尔规模滤波器组与高级别的ASR编码器输出作为输入功能。在内部中文语料库上，该方法的词计时准确率分别为95.68%和94.18%，而混合系统的词计时准确率分别为93.0%和90.22%.它还超过了以前的E2 E方法，在7种语言的指标上绝对增加了4.80% 8.02%。此外，我们进一步提高字定时精度延迟CTC峰值与帧知识蒸馏，虽然只实验LibriSpeech。摘要：End-to-end (E2E) systems have shown comparable performance to hybrid systems for automatic speech recognition (ASR). Word timings, as a by-product of ASR, are essential in many applications, especially for subtitling and computer-aided pronunciation training. In this paper, we improve the frame-level classifier for word timings in E2E system by introducing label priors in connectionist temporal classification (CTC) loss, which is adopted from prior works, and combining low-level Mel-scale filter banks with high-level ASR encoder output as input feature. On the internal Chinese corpus, the proposed method achieves 95.68% 94.18% compared to the hybrid system 93.0% 90.22% on the word timing accuracy metrics. It also surpass a previous E2E approach with an absolute increase of 4.80% 8.02% on the metrics on 7 languages. In addition, we further improve word timing accuracy by delaying CTC peaks with frame-wise knowledge distillation, though only experimenting on LibriSpeech.
【2】Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding标题：用于语音理解的语音-文本适配器和语音-实体检索器增强的LLMS
链接：http://arxiv.org/pdf/2306.07944v1
作者：Mingqiu Wang,Izhak Shafran,Hagen Soltau,Wei Han,Yuan Cao,Dian Yu,Laurent El Shafey机构：Google DeepMind摘要：大型语言模型（LLM）已经应用于语音领域，经常由于语音和语言表示之间的不对齐而导致性能下降。为了弥合这一差距，我们提出了一个联合语音和语言模型（SLM）使用Speech2Text适配器，它将语音映射到文本令牌嵌入空间，而不会丢失语音信息。此外，使用基于CTC的空白滤波，我们可以减少语音序列长度的文本。在语音MultiWoz数据集（DSTC11挑战）中，SLM大大提高了对话状态跟踪（DST）性能（准确率为24.7%至28.4%）。进一步解决罕见的实体上的错误，我们增加SLM与Speech2Entity检索器，它使用语音检索相关的实体，然后将它们添加到原始SLM输入作为前缀。有了这种检索增强SLM（ReSLM），DST性能跃升至34.6%的准确度。此外，增强ASR任务与对话理解任务提高ASR性能从9.4%到8.5% WER。摘要：Large Language Models (LLMs) have been applied in the speech domain, often incurring a performance drop due to misaligned between speech and language representations. To bridge this gap, we propose a joint speech and language model (SLM) using a Speech2Text adapter, which maps speech into text token embedding space without speech information loss. Additionally, using a CTC-based blank-filtering, we can reduce the speech sequence length to that of text. In speech MultiWoz dataset (DSTC11 challenge), SLM largely improves the dialog state tracking (DST) performance (24.7% to 28.4% accuracy). Further to address errors on rare entities, we augment SLM with a Speech2Entity retriever, which uses speech to retrieve relevant entities, and then adds them to the original SLM input as a prefix. With this retrieval-augmented SLM (ReSLM), the DST performance jumps to 34.6% accuracy. Moreover, augmenting the ASR task with the dialog understanding task improves the ASR performance from 9.4% to 8.5% WER.
【3】FOOCTTS: Generating Arabic Speech with Acoustic Environment for Football Commentator
标题：FOOCTTS：足球解说员使用声学环境生成阿拉伯语语音
链接：http://arxiv.org/pdf/2306.07936v1
*与cs.SD语音【6】为同一篇
作者：Massa Baali,Ahmed Ali机构：Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE, Qatar Computing Research Institute, HBKU, Doha, Qatar备注：Accepted at Interspeech 2023 Show & Tell Demo Session摘要：本文介绍了FOOCTTS，一个自动流水线的足球评论员，产生语音与背景人群噪声。应用程序从用户处获取文本，应用文本预处理（如元音化），然后是评论员的语音合成器。我们的管道包括阿拉伯语自动语音识别的数据标签，CTC分割，转录元音匹配语音，并微调TTS。我们的系统能够在有限的15分钟的足球评论员录音的声学环境中生成语音。我们的原型是可推广的，可以很容易地应用到不同的领域和语言。摘要：This paper presents FOOCTTS, an automatic pipeline for a football commentator that generates speech with background crowd noise. The application gets the text from the user, applies text pre-processing such as vowelization, followed by the commentator's speech synthesizer. Our pipeline included Arabic automatic speech recognition for data labeling, CTC segmentation, transcription vowelization to match speech, and fine-tuning the TTS. Our system is capable of generating speech with its acoustic environment within limited 15 minutes of football commentator recording. Our prototype is generalizable and can be easily applied to different domains and languages.
【4】A Theory of Unsupervised Speech Recognition
标题：一种无监督语音识别理论
链接：http://arxiv.org/pdf/2306.07926v1
*与cs.SD语音【7】为同一篇
作者：Liming Wang,Mark Hasegawa-Johnson,Chang D. Yoo机构：University of Illinois Urbana-Champaign, Korea Advanced Institute of Science Technology摘要：无监督语音识别（ASR-U）是从非配对的纯语音和纯文本语料库中学习自动语音识别（ASR）系统的问题。虽然存在各种算法来解决这个问题，但在研究它们的属性和解决对超参数的敏感性和训练不稳定性等问题时，缺少理论框架。本文基于随机矩阵理论和神经切核理论，提出了一个研究ASR-U系统性质的一般理论框架。这样的框架使我们能够证明各种可学习性条件和样本的复杂性范围的ASR-U。在具有三类转换图的合成语言上进行的广泛的ASR-U实验为我们的理论提供了强有力的经验证据（代码可在cactuswiththoughts UnsupASRTheory.git获得）。摘要：Unsupervised speech recognition (ASR-U) is the problem of learning automatic speech recognition (ASR) systems from unpaired speech-only and text-only corpora. While various algorithms exist to solve this problem, a theoretical framework is missing from studying their properties and addressing such issues as sensitivity to hyperparameters and training instability. In this paper, we proposed a general theoretical framework to study the properties of ASR-U systems based on random matrix theory and the theory of neural tangent kernels. Such a framework allows us to prove various learnability conditions and sample complexity bounds of ASR-U. Extensive ASR-U experiments on synthetic languages with three classes of transition graphs provide strong empirical evidence for our theory (code available at cactuswiththoughts UnsupASRTheory.git).
【5】Unsupervised speech enhancement with deep dynamical generative speech and noise models
标题：基于深度动态生成语音和噪声模型的非监督语音增强
链接：http://arxiv.org/pdf/2306.07820v1
*与cs.SD语音【8】为同一篇
作者：Xiaoyu Lin,Simon Leglaive,Laurent Girin,Xavier Alameda-Pineda机构：Inria Grenoble Rhˆone-Alpes, Univ. Grenoble Alpes, France, CentraleSup´elec, IETR (UMR CNRS ,), France, Univ. Grenoble Alpes, CNRS, Grenoble-INP, GIPSA-lab, France摘要：这项工作建立在以前的工作上的无监督语音增强使用动态变分自动编码器（DVAE）作为干净的语音模型和非负矩阵分解（NMF）作为噪声模型。我们建议用深度动态生成模型（DDGM）取代NMF噪声模型，该模型取决于DVAE潜变量，或噪声观测，或两者兼而有之。该DDGM可以在三种配置中进行训练：噪声不可知、噪声相关和噪声相关训练后的噪声适应。实验结果表明，所提出的方法相比，国家的最先进的无监督语音增强方法，实现了有竞争力的性能，而噪声相关的训练配置产生一个更有效的时间推理过程。摘要：This work builds on a previous work on unsupervised speech enhancement using a dynamical variational autoencoder (DVAE) as the clean speech model and non-negative matrix factorization (NMF) as the noise model. We propose to replace the NMF noise model with a deep dynamical generative model (DDGM) depending either on the DVAE latent variables, or on the noisy observations, or on both. This DDGM can be trained in three configurations: noise-agnostic, noise-dependent and noise adaptation after noise-dependent training. Experimental results show that the proposed method achieves competitive performance compared to state-of-the-art unsupervised speech enhancement methods, while the noise-dependent training configuration yields a much more time-efficient inference process.
【6】StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
标题：StyleTTS 2：通过大语音语言模型的风格扩散和对抗性训练实现人类级别的文语转换
链接：http://arxiv.org/pdf/2306.07691v1
*与cs.SD语音【9】为同一篇
作者：Yinghao Aaron Li,Cong Han,Vinay S. Raghavan,Gavin Mischler,Nima Mesgarani机构：Columbia University摘要：在本文中，我们提出了StyleTTS 2，一个文本到语音（TTS）模型，利用风格扩散和对抗训练与大型语音语言模型（SLM），以实现人类水平的TTS合成。StyleTTS 2与其前身的不同之处在于通过扩散模型将风格建模为潜在随机变量，以生成最适合文本的风格，而无需参考语音，实现高效的潜在扩散，同时受益于扩散模型提供的多样化语音合成。此外，我们采用大型的预训练SLM，如WavLM，作为鉴别器，我们的新型可微分持续时间建模用于端到端训练，从而提高了语音的自然度。StyleTTS 2在单说话者LJSpeech数据集上超越了人类录音，并在多说话者VCTK数据集上与之相匹配，由英语母语者判断。此外，当在LibriTTS数据集上进行训练时，我们的模型在zero-shot扬声器自适应方面优于以前公开的模型。这项工作在单说话者和多说话者数据集上实现了第一个人类水平的TTS，展示了使用大型SLM进行风格扩散和对抗性训练的潜力。音频演示和源代码可在www.example.com上获得https: styletts2.github.io 。摘要：In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https: styletts2.github.io .
【7】 Malafide: a novel adversarial convolutive noise attack against deepfake and spoofing detection systems标题：恶意：一种新的对抗深度假冒和欺骗检测系统的对抗性卷积噪声攻击
链接：http://arxiv.org/pdf/2306.07655v1
作者：Michele Panariello,Wanying Ge,Hemlata Tak,Massimiliano Todisco,Nicholas Evans机构：EURECOM, Sophia Antipolis, France备注：Accepted at INTERSPEECH 2023摘要：我们提出了Malafide，一个通用的对抗攻击自动说话人验证（ASV）欺骗对策（CMs）。通过使用优化的线性时不变滤波器引入卷积噪声，Malafide攻击可以用于损害CM可靠性，同时保留其他语音属性，例如质量和说话者的声音。与最近提出的其他对抗性攻击相比，Malafide滤波器独立于输入话语和持续时间进行优化，而是针对潜在的欺骗攻击进行调整，并且仅需要少量滤波器系数的优化。即便如此，它们也会使CM性能估计降低一个数量级，即使在黑盒设置中也是如此，并且还可以被配置为克服集成的CM和ASV子系统。然而，使用自监督学习CM的集成解决方案在黑盒和白盒设置下都更健壮。摘要：We present Malafide, a universal adversarial attack against automatic speaker verification (ASV) spoofing countermeasures (CMs). By introducing convolutional noise using an optimised linear time-invariant filter, Malafide attacks can be used to compromise CM reliability while preserving other speech attributes such as quality and the speaker's voice. In contrast to other adversarial attacks proposed recently, Malafide filters are optimised independently of the input utterance and duration, are tuned instead to the underlying spoofing attack, and require the optimisation of only a small number of filter coefficients. Even so, they degrade CM performance estimates by an order of magnitude, even in black-box settings, and can also be configured to overcome integrated CM and ASV subsystems. Integrated solutions that use self-supervised learning CMs, however, are more robust, under both black-box and white-box settings.
【8】Statistical Beamformer Exploiting Non-stationarity and Sparsity with Spatially Constrained ICA for Robust Speech Recognition
标题：基于空间约束ICA的统计波束形成器非平稳性和稀疏性的稳健语音识别
链接：http://arxiv.org/pdf/2306.07562v1
*与cs.SD语音【10】为同一篇
作者：Ui-Hyeop Shin,Hyung-Min Park机构：This work was partly supported by Institute of Information & communi-cations Technology Planning & evaluation (IITP) grant funded by the Koreagovernment (MSIT) (No摘要：在本文中，我们提出了一种统计波束形成算法作为鲁棒自动语音识别（ASR）的预处理步骤。通过将目标语音建模为非平稳拉普拉斯分布，提出了一种基于掩模的统计波束形成算法，以利用其输出和掩模输入方差进行波束形成器的鲁棒估计。此外，我们还提出了一种方法的导向矢量估计（SVE）的基础上获得的目标和噪声输出的独立分量分析（ICA）的噪声功率比。为了在相同的ICA框架中更新波束形成器，我们推导出目标语音上具有无失真和零约束的ICA，其分别在目标输出处产生波束形成语音和在其他输出处产生噪声。目标输出的解混权重导致具有加权空间协方差矩阵（wSCM）的统计波束形成器，其使用由源模型表征的加权函数。为了提高SVE，严格的零约束施加的拉格朗日乘子方法放宽广义惩罚权参数，而严格的无失真的约束保持。此外，我们推导出一个在线算法的基础上的递归最小二乘（RLS）的实际应用的优化技术。使用CHiME-4和LibriCSS数据集在各种环境下的实验结果表明，相比传统的波束形成和盲源提取（BSE）ICA的基础上的批处理和在线处理所提出的算法的有效性。摘要：In this paper, we present a statistical beamforming algorithm as a pre-processing step for robust automatic speech recognition (ASR). By modeling the target speech as a non-stationary Laplacian distribution, a mask-based statistical beamforming algorithm is proposed to exploit both its output and masked input variance for robust estimation of the beamformer. In addition, we also present a method for steering vector estimation (SVE) based on a noise power ratio obtained from the target and noise outputs in independent component analysis (ICA). To update the beamformer in the same ICA framework, we derive ICA with distortionless and null constraints on target speech, which yields beamformed speech at the target output and noises at the other outputs, respectively. The demixing weights for the target output result in a statistical beamformer with the weighted spatial covariance matrix (wSCM) using a weighting function characterized by a source model. To enhance the SVE, the strict null constraints imposed by the Lagrange multiplier methods are relaxed by generalized penalties with weight parameters, while the strict distortionless constraints are maintained. Furthermore, we derive an online algorithm based on an optimization technique of recursive least squares (RLS) for practical applications. Experimental results on various environments using CHiME-4 and LibriCSS datasets demonstrate the effectiveness of the presented algorithm compared to conventional beamforming and blind source extraction (BSE) based on ICA on both batch and online processing.
【9】Speaker Verification Across Ages: Investigating Deep Speaker Embedding Sensitivity to Age Mismatch in Enrollment and Test Speech
标题：跨年龄说话人验证：研究深度说话人在注册和测试演讲中对年龄不匹配的敏感性
链接：http://arxiv.org/pdf/2306.07501v1
*与cs.SD语音【11】为同一篇
作者：Vishwanath Pratap Singh,Md Sahidullah,Tomi Kinnunen机构：University of Eastern Finland, Finland, Institute for Advancing Intelligence, TCG CREST, IndiaJournal-ref：Interspeech 2023摘要：在本文中，我们研究了老化对现代基于深度说话人嵌入的自动说话人验证（ASV）系统的影响。我们选择了两个不同的数据集来检查最先进的ECAPA-TDNN系统的老化。第一个数据集是VoxCeleb，用于解决在不受控制的条件下的短期老化（注册和测试之间长达10年的时间差）。第二个数据集，用于解决长期老化效应（高达40年的差异）的芬兰语扬声器下一个更受控的设置，是纵向语料库芬兰语口语在赫尔辛基（LCFSH）。我们的研究提供了新的见解扬声器老化对现代ASV系统的影响。具体来说，我们建立了一个定量的衡量老化和ASV分数。此外，我们的研究表明，老龄化影响女性英语发言人在更大程度上比男性英语发言人，而在芬兰的情况下，它有一个更大的影响男性发言人比女性发言人。摘要：In this paper, we study the impact of the ageing on modern deep speaker embedding based automatic speaker verification (ASV) systems. We have selected two different datasets to examine ageing on the state-of-the-art ECAPA-TDNN system. The first dataset, used for addressing short-term ageing (up to 10 years time difference between enrollment and test) under uncontrolled conditions, is VoxCeleb. The second dataset, used for addressing long-term ageing effect (up to 40 years difference) of Finnish speakers under a more controlled setup, is Longitudinal Corpus of Finnish Spoken in Helsinki (LCFSH). Our study provides new insights into the impact of speaker ageing on modern ASV systems. Specifically, we establish a quantitative measure between ageing and ASV scores. Further, our research indicates that ageing affects female English speakers to a greater degree than male English speakers, while in the case of Finnish, it has a greater impact on male speakers than female speakers.
【10】PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling
标题：PauseSpeech：基于预训练语言模型和基于停顿的韵律建模的自然语音合成
链接：http://arxiv.org/pdf/2306.07489v1
*与cs.SD语音【12】为同一篇
作者：Ji-Sang Hwang,Sang-Hoon Lee,Seong-Whan Lee机构：Department of Artificial Intelligence, Korea University, Seoul, Korea备注：13 pages, 4 figures, 3 tables, under reivew摘要：虽然文本到语音（TTS）系统已经显著改进，但是大多数TTS系统在合成具有适当措辞的语音方面仍然具有限制。对于自然语音合成，重要的是合成具有基于语义信息将单词分组为短语的短语结构的语音。在本文中，我们提出了PuaseSpeech，一个语音合成系统与预训练的语言模型和基于停顿的韵律建模。首先，我们引入了一个短语结构编码器，它利用了来自预训练语言模型的上下文表示。在短语结构编码器中，我们从上下文表示中提取依赖于说话者的句法表示，然后预测将输入文本分成短语的暂停序列。此外，我们还引入了一个基于停顿的单词编码器来建立基于停顿序列的单词级韵律模型。实验结果表明，PauseSpeech在自然度方面优于以前的模型。此外，在客观评价方面，我们可以观察到，我们提出的方法有助于模型减少地面实况和合成语音之间的距离。音频样本可在www.example.com上获得https: jisang93.github.io pausespeech-demo。摘要：Although text-to-speech (TTS) systems have significantly improved, most TTS systems still have limitations in synthesizing speech with appropriate phrasing. For natural speech synthesis, it is important to synthesize the speech with a phrasing structure that groups words into phrases based on semantic information. In this paper, we propose PuaseSpeech, a speech synthesis system with a pre-trained language model and pause-based prosody modeling. First, we introduce a phrasing structure encoder that utilizes a context representation from the pre-trained language model. In the phrasing structure encoder, we extract a speaker-dependent syntactic representation from the context representation and then predict a pause sequence that separates the input text into phrases. Furthermore, we introduce a pause-based word encoder to model word-level prosody based on pause sequence. Experimental results show PauseSpeech outperforms previous models in terms of naturalness. Furthermore, in terms of objective evaluations, we can observe that our proposed methods help the model decrease the distance between ground-truth and synthesized speech. Audio samples are available at https: jisang93.github.io pausespeech-demo .
【11】GEmo-CLAP: Gender-Attribute-Enhanced Contrastive Language-Audio Pretraining for Speech Emotion Recognition
标题：GEMO-CLAP：用于语音情感识别的性别增强型对比语言-音频预训练
链接：http://arxiv.org/pdf/2306.07848v1
*与cs.SD语音【1】为同一篇
作者：Yu Pan,Yanni Hu,Yuguang Yang,Jixun Yao,Wen Fei,Lei Ma,Heng Lu机构：University of Alberta, Edmonton, Canada, The University of Tokyo, Japan, Ximalaya Inc., ShangHai, China, Shanghai Jiao Tong University, ShangHai, China备注：5 pages摘要：对比语言音频预训练（CLAP）最近在不同领域取得了令人印象深刻的成功。在本文中，我们提出了GEmo-CLAP，一种有效的性别属性增强CLAP模型的语音情感识别（SER）。具体来说，我们首先建立一个有效的情感CLAP模型，称为Emo-CLAP SER，利用各种基于自监督学习的预训练模型。然后，考虑到性别属性在语音情感建模中的重要性，进一步提出了两种GEmo-CLAP方法，以融合语音信号的情感和性别信息，形成更合理的目标。在IEMOCAP语料库上进行的大量实验表明，我们提出的两种GEmo-CLAP方法在不同的预训练模型下始终优于基线Emo-CLAP，同时与其他最先进的方法相比，也实现了更好的识别性能。摘要：Contrastive Language-Audio Pretraining (CLAP) has recently exhibited impressive success in diverse fields. In this paper, we propose GEmo-CLAP, a kind of efficient gender-attribute-enhanced CLAP model for speech emotion recognition (SER). Specifically, we first build an effective emotion CLAP model termed Emo-CLAP for SER, utilizing various self-supervised learning based pre-trained models. Then, considering the importance of the gender attribute in speech emotion modeling, two GEmo-CLAP approaches are further proposed to integrate the emotion and gender information of speech signals, forming more reasonable objectives. Extensive experiments conducted on the IEMOCAP corpus demonstrate that our proposed two GEmo-CLAP approaches consistently outperform the baseline Emo-CLAP with different pre-trained models, while also achieving superior recognition performance compared with other state-of-the-art methods.
【12】Unlocking Foundation Models for Privacy-Enhancing Speech Understanding: An Early Study on Low Resource Speech Training Leveraging Label-guided Synthetic Speech Content
标题：解锁隐私增强语音理解的基础模型：利用标签引导的合成语音内容的低资源语音训练的早期研究
链接：http://arxiv.org/pdf/2306.07791v1
*与cs.SD语音【2】为同一篇
作者：Tiantian Feng,Digbalay Bose,Xuan Shi,Shrikanth Narayanan机构：Signal Analysis and Interpretation Laboratory, University of Southern California, USA摘要：自动语音理解（ASU）利用深度学习模型的强大功能来准确解释人类语音，从而产生广泛的语音应用，丰富人类体验。然而，训练一个强大的ASU模型需要管理大量的语音样本，这会带来隐私泄露的风险。在这项工作中，我们调查使用基础模型，以协助隐私增强语音计算。与主要关注数据扰动或分布式算法的传统作品不同，我们的工作研究了使用预训练的生成模型来合成语音内容作为训练数据的可能性，仅使用标签指导。我们表明，zero-shot学习与训练标签引导的合成语音内容仍然是一项具有挑战性的任务。另一方面，我们的研究结果表明，用合成语音样本训练的模型为低资源ASU训练提供了有效的初始化点。这一结果揭示了通过减少用户数据收集但使用标签引导的合成语音内容来增强隐私的潜力。摘要：Automatic Speech Understanding (ASU) leverages the power of deep learning models for accurate interpretation of human speech, leading to a wide range of speech applications that enrich the human experience. However, training a robust ASU model requires the curation of a large number of speech samples, creating risks for privacy breaches. In this work, we investigate using foundation models to assist privacy-enhancing speech computing. Unlike conventional works focusing primarily on data perturbation or distributed algorithms, our work studies the possibilities of using pre-trained generative models to synthesize speech content as training data with just label guidance. We show that zero-shot learning with training label-guided synthetic speech content remains a challenging task. On the other hand, our results demonstrate that the model trained with synthetic speech samples provides an effective initialization point for low-resource ASU training. This result reveals the potential to enhance privacy by reducing user data collection but using label-guided synthetic speech content.
【13】Contrastive Learning-Based Audio to Lyrics Alignment for Multiple Languages
标题：基于对比学习的多语言音歌词对齐
链接：http://arxiv.org/pdf/2306.07744v1
*与cs.SD语音【3】为同一篇
作者：Simon Durand,Daniel Stoller,Sebastian Ewert机构：Spotify Equal contributionJournal-ref：ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5摘要：近年来，歌词对齐获得了相当大的关注。最先进的系统要么重新使用已建立的语音识别工具包，要么设计涉及连接主义时间分类（CTC）损失的端到端解决方案。然而，这两种方法都有特定的缺点：工具包以其复杂性而闻名，CTC系统使用为转录设计的损失，这可能限制比对准确性。在本文中，我们使用的是一个对比学习过程，得出跨模态嵌入链接的音频和文本域。通过这种方式，我们获得了一个新的系统，该系统易于端到端训练，可以利用弱注释的训练数据，联合学习强大的文本模型，并针对对齐量身定制。该系统不仅是第一个在标准Jamendo数据集上产生低于0.2秒的平均绝对误差的系统，而且它对其他语言也是鲁棒的，即使只在英语数据上训练。最后，我们发布了JamendoLyrics Multi-Lang数据集的单词级对齐。摘要：Lyrics alignment gained considerable attention in recent years. State-of-the-art systems either re-use established speech recognition toolkits, or design end-to-end solutions involving a Connectionist Temporal Classification (CTC) loss. However, both approaches suffer from specific weaknesses: toolkits are known for their complexity, and CTC systems use a loss designed for transcription which can limit alignment accuracy. In this paper, we use instead a contrastive learning procedure that derives cross-modal embeddings linking the audio and text domains. This way, we obtain a novel system that is simple to train end-to-end, can make use of weakly annotated training data, jointly learns a powerful text model, and is tailored to alignment. The system is not only the first to yield an average absolute error below 0.2 seconds on the standard Jamendo dataset but it is also robust to other languages, even when trained on English data only. Finally, we release word-level alignments for the JamendoLyrics Multi-Lang dataset.
【14】Modality Adaption or Regularization? A Case Study on End-to-End Speech Translation
标题：情态适应还是正规化？端到端语音翻译的个案研究
链接：http://arxiv.org/pdf/2306.07650v1
*与cs.SD语音【4】为同一篇
作者：JYuchen Han,Chen Xu,Tong Xiao,Jingbo Zhu机构：School of Computer Science and Engineering, Northeastern University, Shenyang, China, NiuTrans Research, Shenyang, China备注：ACL 2023 Main Conference摘要：预训练和微调是缓解端到端语音翻译（E2 E ST）中的数据稀缺问题的范例。语音和文本数据之间常见的“模态间隙”通常导致预训练和微调之间的输入不一致。然而，我们观察到这种差距出现在微调的早期阶段，但不会对最终性能产生重大影响。另一方面，我们发现还有另一个缺口，我们称之为“能力缺口”：高资源任务（如ASR和MT）通常需要一个大的模型来拟合，当模型被重用于低资源任务（E2 E ST）时，由于过度拟合，它将获得次优性能。在一个案例研究中，我们发现正则化比精心设计的模态自适应方法起着更重要的作用，在MuST-C数据集上，en-de的正则化方法达到了29.0，en-fr的正则化方法达到了40.3。代码和模型可在www.example.com上获得https: github.com hannlp TAB。摘要：Pre-training and fine-tuning is a paradigm for alleviating the data scarcity problem in end-to-end speech translation (E2E ST). The commonplace "modality gap" between speech and text data often leads to inconsistent inputs between pre-training and fine-tuning. However, we observe that this gap occurs in the early stages of fine-tuning, but does not have a major impact on the final performance. On the other hand, we find that there has another gap, which we call the "capacity gap": high resource tasks (such as ASR and MT) always require a large model to fit, when the model is reused for a low resource task (E2E ST), it will get a sub-optimal performance due to the over-fitting. In a case study, we find that the regularization plays a more important role than the well-designed modality adaption method, which achieves 29.0 for en-de and 40.3 for en-fr on the MuST-C dataset. Code and models are available at https: github.com hannlp TAB.
【15】UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
标题：UniCATS：一种支持上下文VQ扩散和语音编码的统一上下文感知文语转换框架
链接：http://arxiv.org/pdf/2306.07547v1
*与cs.SD语音【5】为同一篇
作者：Chenpeng Du,Yiwei Guo,Feiyu Shen,Zhijun Liu,Zheng Liang,Xie Chen,Shuai Wang,Hui Zhang,Kai Yu机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Shenzhen Research Institute of Big Data, Shenzhen, China, AISpeech Ltd, Beijing, China摘要：利用离散语音令牌，分为语义令牌和声学令牌，已被证明优于传统的声学特征梅尔频谱图的自然性和鲁棒性的文本到语音（TTS）合成。最近流行的模型，如VALL-E和SPEAR-TTS，允许通过自回归（AR）连续的声学令牌提取的短语音提示zero-shot扬声器自适应。然而，这些AR模型被限制为仅在从左到右的方向上生成语音，使得它们不适合于其中提供了先前和随后的上下文两者的语音编辑。此外，这些模型依赖于声学令牌，其具有由音频编解码器模型的性能施加的音频质量限制。在这项研究中，我们提出了一个统一的上下文感知的TTS框架称为UniCATS，这是能够语音延续和编辑。UniCATS包括两个组件，声学模型CTX-txt 2 vec和声码器CTX-vec 2 wav。CTX-txt 2 vec采用上下文VQ扩散来预测来自输入文本的语义令牌，使其能够结合语义上下文并保持与周围上下文的无缝连接。之后，CTX-vec 2 wav利用上下文声码将这些语义标记转换为波形，同时考虑到声学上下文。我们的实验结果表明，CTX-vec 2 wav优于HifiGAN和AudioLM的语音再合成从语义令牌。此外，我们表明，UniCATS实现国家的最先进的性能，在语音延续和编辑。摘要：The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing.
机器翻译，仅供参考