今日论文合集：cs.SD语音7篇，eess.AS音频处理8篇。
cs.SD语音
【1】 Parallel and Limited Data Voice Conversion Using Stochastic Variational  Deep Kernel Learning标题：基于随机变分深度核学习的并行有限数据语音转换链接：https://arxiv.org/abs/2309.04420
作者：Mohamadreza Jafaryani,Hamid Sheikhzadeh,Vahid Pourahmadi机构：a Department of Electrical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Iran备注：None摘要：通常，语音转换被视为具有有限训练数据的工程问题。对大量数据的依赖阻碍了深度学习方法的实际适用性，近年来这些方法得到了广泛的研究。另一方面，统计方法是有效的有限的数据，但有困难的建模复杂的映射函数。本文提出了一种基于随机变分深度核学习（SVDKL）的语音转换方法。同时，SVDKL能够利用深度神经网络的表达能力以及高斯过程作为贝叶斯和非参数方法的高度灵活性。当常规核与深度神经网络相结合时，可以估计非平滑和更复杂的函数。此外，该模型的稀疏变分高斯过程解决了可扩展性问题，并且，不像精确高斯过程，允许整个声学空间的全局映射函数的学习。所提出的方案的最重要的方面之一是，模型参数的训练使用边际似然优化，它考虑了数据拟合和模型的复杂性。考虑到模型的复杂性，通过增加对过拟合的阻力来减少训练数据量。为了评估所提出的方案，我们检查了模型的性能与大约80秒的训练数据。结果表明，我们的方法获得了更高的平均意见得分，更小的频谱失真，和更好的偏好测试比比较的方法。摘要：Typically, voice conversion is regarded as an engineering problem with limited training data. The reliance on massive amounts of data hinders the practical applicability of deep learning approaches, which have been extensively researched in recent years. On the other hand, statistical methods are effective with limited data but have difficulties in modelling complex mapping functions. This paper proposes a voice conversion method that works with limited data and is based on stochastic variational deep kernel learning (SVDKL). At the same time, SVDKL enables the use of deep neural networks' expressive capability as well as the high flexibility of the Gaussian process as a Bayesian and non-parametric method. When the conventional kernel is combined with the deep neural network, it is possible to estimate non-smooth and more complex functions. Furthermore, the model's sparse variational Gaussian process solves the scalability problem and, unlike the exact Gaussian process, allows for the learning of a global mapping function for the entire acoustic space. One of the most important aspects of the proposed scheme is that the model parameters are trained using marginal likelihood optimization, which considers both data fitting and model complexity. Considering the complexity of the model reduces the amount of training data by increasing the resistance to overfitting. To evaluate the proposed scheme, we examined the model's performance with approximately 80 seconds of training data. The results indicated that our method obtained a higher mean opinion score, smaller spectral distortion, and better preference tests than the compared methods.
【2】 A Long-Tail Friendly Representation Framework for Artist and Music  Similarity标题：一种长尾友好的艺术家和音乐相似度表示框架链接：https://arxiv.org/abs/2309.04182
作者：Haoran Xiang,Junyu Dai,Xuchen Song,Furao Shen机构：National Key Laboratory for Novel Software Technology, Nanjing University, China, ByteDance摘要：研究艺术家和音乐之间的相似性是音乐检索和推荐的关键，解决长尾现象的挑战也越来越重要。本文提出了一个长尾友好表示框架（LTFRF），利用神经网络建模的相似性关系。我们的方法集成音乐，用户，元数据和关系数据到一个统一的度量学习框架，并采用元一致性关系作为一个常规术语引入多关系损失。与图神经网络（GNN）相比，我们提出的框架提高了长尾场景中的表示性能，其特征是艺术家和音乐之间的稀疏关系。我们在AllMusic数据集上进行了实验和分析，结果表明，我们的框架提供了一个良好的艺术家和音乐表示的泛化。具体而言，在类似的艺术家/音乐推荐任务中，LTFRF在命中率@10中比基线高出9.69%/19.42%，在长尾情况下，框架在Consistent@10中比基线高出11.05%/14.14%。摘要：The investigation of the similarity between artists and music is crucial in music retrieval and recommendation, and addressing the challenge of the long-tail phenomenon is increasingly important. This paper proposes a Long-Tail Friendly Representation Framework (LTFRF) that utilizes neural networks to model the similarity relationship. Our approach integrates music, user, metadata, and relationship data into a unified metric learning framework, and employs a meta-consistency relationship as a regular term to introduce the Multi-Relationship Loss. Compared to the Graph Neural Network (GNN), our proposed framework improves the representation performance in long-tail scenarios, which are characterized by sparse relationships between artists and music. We conduct experiments and analysis on the AllMusic dataset, and the results demonstrate that our framework provides a favorable generalization of artist and music representation. Specifically, on similar artist/music recommendation tasks, the LTFRF outperforms the baseline by 9.69%/19.42% in Hit Ratio@10, and in long-tail cases, the framework achieves 11.05%/14.14% higher than the baseline in Consistent@10.
【3】 Cross-Utterance Conditioned VAE for Speech Generation标题：用于语音生成的交叉发音条件VAE链接：https://arxiv.org/abs/2309.04156
作者：Yang Li,Cheng Yu,Guangzhi Sun,Weiqin Zu,Zheng Tian,Ying Wen,Wei Pan,Chao Zhang,Jun Wang,Yang Yang,Fanglei Sun机构：Shanghai Jiao Tong University备注：13 pages;摘要：由神经网络驱动的语音合成系统有望用于多媒体制作，但经常面临产生表达性语音和无缝编辑的问题。作为回应，我们提出了跨话语条件变分自动编码器语音合成（CUC-VAE S2）框架，以增强韵律，并确保自然的语音生成。该框架利用了预训练语言模型的强大表示能力和变分自动编码器（VAE）的重新表达能力。CUC-VAE S2框架的核心组件是跨话语CVAE，其从周围的句子中提取声学、说话者和文本特征以生成上下文敏感的韵律特征，更准确地模拟人类韵律生成。我们进一步提出了两个实用的算法，为不同的语音合成应用程序量身定制：CUC-VAE TTS用于文本到语音转换，CUC-VAE SE用于语音编辑。CUC-VAE TTS是该框架的直接应用，旨在生成具有来自周围文本的上下文韵律的音频。另一方面，CUC-VAE SE算法利用以上下文信息为条件的真实梅尔频谱图采样，产生紧密反映真实声音的音频，从而促进基于文本的灵活语音编辑，例如删除、插入和替换。LibriTTS数据集上的实验结果表明，我们提出的模型显着增强语音合成和编辑，产生更自然和表达的语音。摘要：Speech synthesis systems powered by neural networks hold promise for multimedia production, but frequently face issues with producing expressive speech and seamless editing. In response, we present the Cross-Utterance Conditioned Variational Autoencoder speech synthesis (CUC-VAE S2) framework to enhance prosody and ensure natural speech generation. This framework leverages the powerful representational capabilities of pre-trained language models and the re-expression abilities of variational autoencoders (VAEs). The core component of the CUC-VAE S2 framework is the cross-utterance CVAE, which extracts acoustic, speaker, and textual features from surrounding sentences to generate context-sensitive prosodic features, more accurately emulating human prosody generation. We further propose two practical algorithms tailored for distinct speech synthesis applications: CUC-VAE TTS for text-to-speech and CUC-VAE SE for speech editing. The CUC-VAE TTS is a direct application of the framework, designed to generate audio with contextual prosody derived from surrounding texts. On the other hand, the CUC-VAE SE algorithm leverages real mel spectrogram sampling conditioned on contextual information, producing audio that closely mirrors real sound and thereby facilitating flexible speech editing based on text such as deletion, insertion, and replacement. Experimental results on the LibriTTS datasets demonstrate that our proposed models significantly enhance speech synthesis and editing, producing more natural and expressive speech.
【4】 A Two-Stage Training Framework for Joint Speech Compression and  Enhancement标题：一种联合语音压缩和增强的两阶段训练框架链接：https://arxiv.org/abs/2309.04132
作者：Jiayi Huang,Zeyu Yan,Wenbin Jiang,Fei Wen机构：School of Electronic Infor-mation and Electrical Engineering,  Shanghai Jiao Tong University摘要：本文研究了含噪语音信号的联合压缩与增强问题。最近，SoundStream编解码器，其依赖于端到端的联合训练的编码器-解码器对和残余矢量量化器的对抗和重建损失的组合，已经显示出非常有前途的性能，特别是在主观感知质量。在这项工作中，我们提供了一个理论结果表明，同时实现低失真和高感知的噪声存在下，存在一个最佳的两阶段优化过程的联合压缩和增强问题。该过程首先仅使用失真损失来优化编码器-解码器对，然后固定编码器以使用感知损失来优化感知解码器。在此基础上，我们构建了一个两阶段的训练框架，用于含噪语音信号的联合压缩和增强。与现有的启发式训练方法不同，所提出的两阶段训练方法具有理论基础。最后，提供了各种噪声和比特率条件下的实验结果。结果表明，由所提出的框架训练的编解码器可以在客观和主观评价指标方面优于SoundStream和其他代表性编解码器。代码可在\textit{https：//github.com/jscscloris/SEStream}获得。摘要：This paper considers the joint compression and enhancement problem for speech signal in the presence of noise. Recently, the SoundStream codec, which relies on end-to-end joint training of an encoder-decoder pair and a residual vector quantizer by a combination of adversarial and reconstruction losses,has shown very promising performance, especially in subjective perception quality. In this work, we provide a theoretical result to show that, to simultaneously achieve low distortion and high perception in the presence of noise, there exist an optimal two-stage optimization procedure for the joint compression and enhancement problem. This procedure firstly optimizes an encoder-decoder pair using only distortion loss and then fixes the encoder to optimize a perceptual decoder using perception loss. Based on this result, we construct a two-stage training framework for joint compression and enhancement of noisy speech signal. Unlike existing training methods which are heuristic, the proposed two-stage training method has a theoretical foundation. Finally, experimental results for various noise and bit-rate conditions are provided. The results demonstrate that a codec trained by the proposed framework can outperform SoundStream and other representative codecs in terms of both objective and subjective evaluation metrics. Code is available at \textit{https://github.com/jscscloris/SEStream}.
【5】 Multiple Representation Transfer from Large Language Models to  End-to-End ASR Systems标题：从大型语言模型到端到端ASR系统的多表示转换链接：https://arxiv.org/abs/2309.04031
作者：Takuma Udagawa,Masayuki Suzuki,Gakuto Kurata,Masayasu Muraoka,George Saon机构：IBM Research AI备注：Submitted to ICASSP 2024摘要：转移大语言模型（LLM）的知识是一种很有前途的技术，将语言知识纳入端到端的自动语音识别（ASR）系统。然而，现有的工作仅传递LLM的单个表示（例如，LLM的一个或多个表示）。预训练BERT的最后一层），而文本的表示本质上是非唯一的，并且可以从不同的层、上下文和模型中以各种方式获得。在这项工作中，我们探索了广泛的技术，以获得和转移到一个基于传感器的ASR系统的LLM的多个表示。虽然在概念上很简单，我们表明，传输多个表示的LLM可以是一个有效的替代方法，只传输一个单一的表示。摘要：Transferring the knowledge of large language models (LLMs) is a promising technique to incorporate linguistic knowledge into end-to-end automatic speech recognition (ASR) systems. However, existing works only transfer a single representation of LLM (e.g. the last layer of pretrained BERT), while the representation of a text is inherently non-unique and can be obtained variously from different layers, contexts and models. In this work, we explore a wide range of techniques to obtain and transfer multiple representations of LLMs into a transducer-based ASR system. While being conceptually simple, we show that transferring multiple representations of LLMs can be an effective alternative to transferring only a single representation.
【6】 LanSER: Language-Model Supported Speech Emotion Recognition标题：Lanser：支持语言模型的语音情感识别链接：https://arxiv.org/abs/2309.03978
作者：Taesik Gong,Josh Belanich,Krishna Somandepalli,Arsha Nagrani,Brian Eoff,Brendan Jou机构：Jou, KAIST, Republic of Korea, Google Research备注：None摘要：语音情感识别（SER）模型通常依赖于昂贵的人工标记数据进行训练，使得缩放方法到大型语音数据集和细微差别的情感分类变得困难。我们提出了LanSER，这是一种通过弱监督学习通过预训练的大型语言模型推断弱情感标签来使用未标记数据的方法。为了推断弱标签约束到分类，我们使用文本蕴涵的方法，选择一个情感标签的最高蕴涵分数提取的语音转录通过自动语音识别。我们的实验结果表明，模型预训练的大型数据集，这种弱监督优于其他基线模型的标准SER数据集时，微调，并显示出提高的标签效率。尽管被预先训练的标签，仅从文本，我们表明，所产生的表示似乎建模的韵律内容的语音。摘要：Speech emotion recognition (SER) models typically rely on costly human-labeled data for training, making scaling methods to large speech datasets and nuanced emotion taxonomies difficult. We present LanSER, a method that enables the use of unlabeled data by inferring weak emotion labels via pre-trained large language models through weakly-supervised learning. For inferring weak labels constrained to a taxonomy, we use a textual entailment approach that selects an emotion label with the highest entailment score for a speech transcript extracted via automatic speech recognition. Our experimental results show that models pre-trained on large datasets with this weak supervision outperform other baseline models on standard SER datasets when fine-tuned, and show improved label efficiency. Despite being pre-trained on labels derived only from text, we show that the resulting representations appear to model the prosodic content of speech.
【7】 Large-Scale Automatic Audiobook Creation标题：大规模自动有声读物创作链接：https://arxiv.org/abs/2309.03926
作者：Brendan Walsh,Mark Hamilton,Greg Newby,Xi Wang,Serena Ruan,Sheng Zhao,Lei He,Shaofei Zhang,Eric Dettinger,William T. Freeman,Markus Weimer机构：Microsoft,MIT,Project Gutenberg,Google, Equal Contribution摘要：有声读物可以极大地提高文学作品的可访问性，并提高读者的参与度。然而，有声读物可能需要数百小时的人力来创建、编辑和发布。在这项工作中，我们提出了一个系统，可以自动生成高品质的有声读物从在线电子书。特别是，我们利用神经文本到语音的最新进展，从古腾堡项目电子书收藏中创建和发布数千本人类质量的开放许可有声读物。我们的方法可以识别适当的电子书内容的子集，以阅读广泛收集的不同结构的书籍，并可以在数百本书并行操作。我们的系统允许用户自定义有声读物的语速和风格，情感语调，甚至可以匹配所需的声音使用少量的样本音频。这项工作贡献了超过5000本开放许可证的有声读物和一个交互式演示，允许用户快速创建自己的定制有声读物。要收听有声读物集，请访问\url{https：//aka.ms/audiobook}。摘要：An audiobook can dramatically improve a work of literature's accessibility and improve reader engagement. However, audiobooks can take hundreds of hours of human effort to create, edit, and publish. In this work, we present a system that can automatically generate high-quality audiobooks from online e-books. In particular, we leverage recent advances in neural text-to-speech to create and release thousands of human-quality, open-license audiobooks from the Project Gutenberg e-book collection. Our method can identify the proper subset of e-book content to read for a wide collection of diversely structured books and can operate on hundreds of books in parallel. Our system allows users to customize an audiobook's speaking speed and style, emotional intonation, and can even match a desired voice using a small amount of sample audio. This work contributed over five thousand open-license audiobooks and an interactive demo that allows users to quickly create their own customized audiobooks. To listen to the audiobook collection visit \url{https://aka.ms/audiobook}.
eess.AS音频处理
【1】 Asymmetric Clean Segments-Guided Self-Supervised Learning for Robust  Speaker Verification标题：非对称清洁分段引导的稳健说话人确认自监督学习链接：https://arxiv.org/abs/2309.04265
作者：Chong-Xin Gan,Man-Wai Mak,Weiwei Lin,Jen-Tzung Chien机构：Dept. of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR, Inst. of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan备注：5 pages, 2 figures, submitted to ICASSP 2024摘要：对比自监督学习（CSL）说话人验证（SV）由于其能够利用无标记数据而引起了越来越多的关注。在原始波形上执行数据增强，例如添加噪声或混响，在SV中实现有希望的结果方面起着关键作用。然而，数据增强需要细致的校准，以确保完整的特定于说话者的信息，这是很难实现没有说话者标签。为了解决这个问题，我们引入了一个新的框架，通过将清洁和增强段的对比训练管道。干净段被重新利用以与有噪声段配对以形成附加的正对和负对。此外，对比损失被加权以增加不同扬声器的干净嵌入和增强嵌入之间的差异。在Voxceleb1上的实验结果表明，该框架可以实现比传统方法显着的19%的改进，它超越了许多现有的国家的最先进的技术。摘要：Contrastive self-supervised learning (CSL) for speaker verification (SV) has drawn increasing interest recently due to its ability to exploit unlabeled data. Performing data augmentation on raw waveforms, such as adding noise or reverberation, plays a pivotal role in achieving promising results in SV. Data augmentation, however, demands meticulous calibration to ensure intact speaker-specific information, which is difficult to achieve without speaker labels. To address this issue, we introduce a novel framework by incorporating clean and augmented segments into the contrastive training pipeline. The clean segments are repurposed to pair with noisy segments to form additional positive and negative pairs. Moreover, the contrastive loss is weighted to increase the difference between the clean and augmented embeddings of different speakers. Experimental results on Voxceleb1 suggest that the proposed framework can achieve a remarkable 19% improvement over the conventional methods, and it surpasses many existing state-of-the-art techniques.
【2】 Parallel and Limited Data Voice Conversion Using Stochastic Variational  Deep Kernel Learning标题：基于随机变分深度核学习的并行有限数据语音转换链接：https://arxiv.org/abs/2309.04420
作者：Mohamadreza Jafaryani,Hamid Sheikhzadeh,Vahid Pourahmadi机构：a Department of Electrical Engineering, Amirkabir University of Technology (Tehran Polytechnic), Iran备注：None摘要：通常，语音转换被视为具有有限训练数据的工程问题。对大量数据的依赖阻碍了深度学习方法的实际适用性，近年来这些方法得到了广泛的研究。另一方面，统计方法是有效的有限的数据，但有困难的建模复杂的映射函数。本文提出了一种基于随机变分深度核学习（SVDKL）的语音转换方法。同时，SVDKL能够利用深度神经网络的表达能力以及高斯过程作为贝叶斯和非参数方法的高度灵活性。当常规核与深度神经网络相结合时，可以估计非平滑和更复杂的函数。此外，该模型的稀疏变分高斯过程解决了可扩展性问题，并且，不像精确高斯过程，允许整个声学空间的全局映射函数的学习。所提出的方案的最重要的方面之一是，模型参数的训练使用边际似然优化，它考虑了数据拟合和模型的复杂性。考虑到模型的复杂性，通过增加对过拟合的阻力来减少训练数据量。为了评估所提出的方案，我们检查了模型的性能与大约80秒的训练数据。结果表明，我们的方法获得了更高的平均意见得分，更小的频谱失真，和更好的偏好测试比比较的方法。摘要：Typically, voice conversion is regarded as an engineering problem with limited training data. The reliance on massive amounts of data hinders the practical applicability of deep learning approaches, which have been extensively researched in recent years. On the other hand, statistical methods are effective with limited data but have difficulties in modelling complex mapping functions. This paper proposes a voice conversion method that works with limited data and is based on stochastic variational deep kernel learning (SVDKL). At the same time, SVDKL enables the use of deep neural networks' expressive capability as well as the high flexibility of the Gaussian process as a Bayesian and non-parametric method. When the conventional kernel is combined with the deep neural network, it is possible to estimate non-smooth and more complex functions. Furthermore, the model's sparse variational Gaussian process solves the scalability problem and, unlike the exact Gaussian process, allows for the learning of a global mapping function for the entire acoustic space. One of the most important aspects of the proposed scheme is that the model parameters are trained using marginal likelihood optimization, which considers both data fitting and model complexity. Considering the complexity of the model reduces the amount of training data by increasing the resistance to overfitting. To evaluate the proposed scheme, we examined the model's performance with approximately 80 seconds of training data. The results indicated that our method obtained a higher mean opinion score, smaller spectral distortion, and better preference tests than the compared methods.
【3】 A Long-Tail Friendly Representation Framework for Artist and Music  Similarity标题：一种长尾友好的艺术家和音乐相似度表示框架链接：https://arxiv.org/abs/2309.04182
作者：Haoran Xiang,Junyu Dai,Xuchen Song,Furao Shen机构：National Key Laboratory for Novel Software Technology, Nanjing University, China, ByteDance摘要：研究艺术家和音乐之间的相似性是音乐检索和推荐的关键，解决长尾现象的挑战也越来越重要。本文提出了一个长尾友好表示框架（LTFRF），利用神经网络建模的相似性关系。我们的方法集成音乐，用户，元数据和关系数据到一个统一的度量学习框架，并采用元一致性关系作为一个常规术语引入多关系损失。与图神经网络（GNN）相比，我们提出的框架提高了长尾场景中的表示性能，其特征是艺术家和音乐之间的稀疏关系。我们在AllMusic数据集上进行了实验和分析，结果表明，我们的框架提供了一个良好的艺术家和音乐表示的泛化。具体而言，在类似的艺术家/音乐推荐任务中，LTFRF在命中率@10中比基线高出9.69%/19.42%，在长尾情况下，框架在Consistent@10中比基线高出11.05%/14.14%。摘要：The investigation of the similarity between artists and music is crucial in music retrieval and recommendation, and addressing the challenge of the long-tail phenomenon is increasingly important. This paper proposes a Long-Tail Friendly Representation Framework (LTFRF) that utilizes neural networks to model the similarity relationship. Our approach integrates music, user, metadata, and relationship data into a unified metric learning framework, and employs a meta-consistency relationship as a regular term to introduce the Multi-Relationship Loss. Compared to the Graph Neural Network (GNN), our proposed framework improves the representation performance in long-tail scenarios, which are characterized by sparse relationships between artists and music. We conduct experiments and analysis on the AllMusic dataset, and the results demonstrate that our framework provides a favorable generalization of artist and music representation. Specifically, on similar artist/music recommendation tasks, the LTFRF outperforms the baseline by 9.69%/19.42% in Hit Ratio@10, and in long-tail cases, the framework achieves 11.05%/14.14% higher than the baseline in Consistent@10.
【4】 Cross-Utterance Conditioned VAE for Speech Generation标题：用于语音生成的交叉发音条件VAE链接：https://arxiv.org/abs/2309.04156
作者：Yang Li,Cheng Yu,Guangzhi Sun,Weiqin Zu,Zheng Tian,Ying Wen,Wei Pan,Chao Zhang,Jun Wang,Yang Yang,Fanglei Sun机构：Shanghai Jiao Tong University备注：13 pages;摘要：由神经网络驱动的语音合成系统有望用于多媒体制作，但经常面临产生表达性语音和无缝编辑的问题。作为回应，我们提出了跨话语条件变分自动编码器语音合成（CUC-VAE S2）框架，以增强韵律，并确保自然的语音生成。该框架利用了预训练语言模型的强大表示能力和变分自动编码器（VAE）的重新表达能力。CUC-VAE S2框架的核心组件是跨话语CVAE，其从周围的句子中提取声学、说话者和文本特征以生成上下文敏感的韵律特征，更准确地模拟人类韵律生成。我们进一步提出了两个实用的算法，为不同的语音合成应用程序量身定制：CUC-VAE TTS用于文本到语音转换，CUC-VAE SE用于语音编辑。CUC-VAE TTS是该框架的直接应用，旨在生成具有来自周围文本的上下文韵律的音频。另一方面，CUC-VAE SE算法利用以上下文信息为条件的真实梅尔频谱图采样，产生紧密反映真实声音的音频，从而促进基于文本的灵活语音编辑，例如删除、插入和替换。LibriTTS数据集上的实验结果表明，我们提出的模型显着增强语音合成和编辑，产生更自然和表达的语音。摘要：Speech synthesis systems powered by neural networks hold promise for multimedia production, but frequently face issues with producing expressive speech and seamless editing. In response, we present the Cross-Utterance Conditioned Variational Autoencoder speech synthesis (CUC-VAE S2) framework to enhance prosody and ensure natural speech generation. This framework leverages the powerful representational capabilities of pre-trained language models and the re-expression abilities of variational autoencoders (VAEs). The core component of the CUC-VAE S2 framework is the cross-utterance CVAE, which extracts acoustic, speaker, and textual features from surrounding sentences to generate context-sensitive prosodic features, more accurately emulating human prosody generation. We further propose two practical algorithms tailored for distinct speech synthesis applications: CUC-VAE TTS for text-to-speech and CUC-VAE SE for speech editing. The CUC-VAE TTS is a direct application of the framework, designed to generate audio with contextual prosody derived from surrounding texts. On the other hand, the CUC-VAE SE algorithm leverages real mel spectrogram sampling conditioned on contextual information, producing audio that closely mirrors real sound and thereby facilitating flexible speech editing based on text such as deletion, insertion, and replacement. Experimental results on the LibriTTS datasets demonstrate that our proposed models significantly enhance speech synthesis and editing, producing more natural and expressive speech.
【5】 A Two-Stage Training Framework for Joint Speech Compression and  Enhancement标题：一种联合语音压缩和增强的两阶段训练框架链接：https://arxiv.org/abs/2309.04132
作者：Jiayi Huang,Zeyu Yan,Wenbin Jiang,Fei Wen机构：School of Electronic Infor-mation and Electrical Engineering,  Shanghai Jiao Tong University摘要：本文研究了含噪语音信号的联合压缩与增强问题。最近，SoundStream编解码器，其依赖于端到端的联合训练的编码器-解码器对和残余矢量量化器的对抗和重建损失的组合，已经显示出非常有前途的性能，特别是在主观感知质量。在这项工作中，我们提供了一个理论结果表明，同时实现低失真和高感知的噪声存在下，存在一个最佳的两阶段优化过程的联合压缩和增强问题。该过程首先仅使用失真损失来优化编码器-解码器对，然后固定编码器以使用感知损失来优化感知解码器。在此基础上，我们构建了一个两阶段的训练框架，用于含噪语音信号的联合压缩和增强。与现有的启发式训练方法不同，所提出的两阶段训练方法具有理论基础。最后，提供了各种噪声和比特率条件下的实验结果。结果表明，由所提出的框架训练的编解码器可以在客观和主观评价指标方面优于SoundStream和其他代表性编解码器。代码可在\textit{https：//github.com/jscscloris/SEStream}获得。摘要：This paper considers the joint compression and enhancement problem for speech signal in the presence of noise. Recently, the SoundStream codec, which relies on end-to-end joint training of an encoder-decoder pair and a residual vector quantizer by a combination of adversarial and reconstruction losses,has shown very promising performance, especially in subjective perception quality. In this work, we provide a theoretical result to show that, to simultaneously achieve low distortion and high perception in the presence of noise, there exist an optimal two-stage optimization procedure for the joint compression and enhancement problem. This procedure firstly optimizes an encoder-decoder pair using only distortion loss and then fixes the encoder to optimize a perceptual decoder using perception loss. Based on this result, we construct a two-stage training framework for joint compression and enhancement of noisy speech signal. Unlike existing training methods which are heuristic, the proposed two-stage training method has a theoretical foundation. Finally, experimental results for various noise and bit-rate conditions are provided. The results demonstrate that a codec trained by the proposed framework can outperform SoundStream and other representative codecs in terms of both objective and subjective evaluation metrics. Code is available at \textit{https://github.com/jscscloris/SEStream}.
【6】 Multiple Representation Transfer from Large Language Models to  End-to-End ASR Systems标题：从大型语言模型到端到端ASR系统的多表示转换链接：https://arxiv.org/abs/2309.04031
作者：Takuma Udagawa,Masayuki Suzuki,Gakuto Kurata,Masayasu Muraoka,George Saon机构：IBM Research AI备注：Submitted to ICASSP 2024摘要：转移大语言模型（LLM）的知识是一种很有前途的技术，将语言知识纳入端到端的自动语音识别（ASR）系统。然而，现有的工作仅传递LLM的单个表示（例如，LLM的一个或多个表示）。预训练BERT的最后一层），而文本的表示本质上是非唯一的，并且可以从不同的层、上下文和模型中以各种方式获得。在这项工作中，我们探索了广泛的技术，以获得和转移到一个基于传感器的ASR系统的LLM的多个表示。虽然在概念上很简单，我们表明，传输多个表示的LLM可以是一个有效的替代方法，只传输一个单一的表示。摘要：Transferring the knowledge of large language models (LLMs) is a promising technique to incorporate linguistic knowledge into end-to-end automatic speech recognition (ASR) systems. However, existing works only transfer a single representation of LLM (e.g. the last layer of pretrained BERT), while the representation of a text is inherently non-unique and can be obtained variously from different layers, contexts and models. In this work, we explore a wide range of techniques to obtain and transfer multiple representations of LLMs into a transducer-based ASR system. While being conceptually simple, we show that transferring multiple representations of LLMs can be an effective alternative to transferring only a single representation.
【7】 LanSER: Language-Model Supported Speech Emotion Recognition标题：Lanser：支持语言模型的语音情感识别链接：https://arxiv.org/abs/2309.03978
作者：Taesik Gong,Josh Belanich,Krishna Somandepalli,Arsha Nagrani,Brian Eoff,Brendan Jou机构：Jou, KAIST, Republic of Korea, Google Research备注：None摘要：语音情感识别（SER）模型通常依赖于昂贵的人工标记数据进行训练，使得缩放方法到大型语音数据集和细微差别的情感分类变得困难。我们提出了LanSER，这是一种通过弱监督学习通过预训练的大型语言模型推断弱情感标签来使用未标记数据的方法。为了推断弱标签约束到分类，我们使用文本蕴涵的方法，选择一个情感标签的最高蕴涵分数提取的语音转录通过自动语音识别。我们的实验结果表明，模型预训练的大型数据集，这种弱监督优于其他基线模型的标准SER数据集时，微调，并显示出提高的标签效率。尽管被预先训练的标签，仅从文本，我们表明，所产生的表示似乎建模的韵律内容的语音。摘要：Speech emotion recognition (SER) models typically rely on costly human-labeled data for training, making scaling methods to large speech datasets and nuanced emotion taxonomies difficult. We present LanSER, a method that enables the use of unlabeled data by inferring weak emotion labels via pre-trained large language models through weakly-supervised learning. For inferring weak labels constrained to a taxonomy, we use a textual entailment approach that selects an emotion label with the highest entailment score for a speech transcript extracted via automatic speech recognition. Our experimental results show that models pre-trained on large datasets with this weak supervision outperform other baseline models on standard SER datasets when fine-tuned, and show improved label efficiency. Despite being pre-trained on labels derived only from text, we show that the resulting representations appear to model the prosodic content of speech.
【8】 Large-Scale Automatic Audiobook Creation标题：大规模自动有声读物创作链接：https://arxiv.org/abs/2309.03926
作者：Brendan Walsh,Mark Hamilton,Greg Newby,Xi Wang,Serena Ruan,Sheng Zhao,Lei He,Shaofei Zhang,Eric Dettinger,William T. Freeman,Markus Weimer机构：Microsoft,MIT,Project Gutenberg,Google, Equal Contribution摘要：有声读物可以极大地提高文学作品的可访问性，并提高读者的参与度。然而，有声读物可能需要数百小时的人力来创建、编辑和发布。在这项工作中，我们提出了一个系统，可以自动生成高品质的有声读物从在线电子书。特别是，我们利用神经文本到语音的最新进展，从古腾堡项目电子书收藏中创建和发布数千本人类质量的开放许可有声读物。我们的方法可以识别适当的电子书内容的子集，以阅读广泛收集的不同结构的书籍，并可以在数百本书并行操作。我们的系统允许用户自定义有声读物的语速和风格，情感语调，甚至可以匹配所需的声音使用少量的样本音频。这项工作贡献了超过5000本开放许可证的有声读物和一个交互式演示，允许用户快速创建自己的定制有声读物。要收听有声读物集，请访问\url{https：//aka.ms/audiobook}。摘要：An audiobook can dramatically improve a work of literature's accessibility and improve reader engagement. However, audiobooks can take hundreds of hours of human effort to create, edit, and publish. In this work, we present a system that can automatically generate high-quality audiobooks from online e-books. In particular, we leverage recent advances in neural text-to-speech to create and release thousands of human-quality, open-license audiobooks from the Project Gutenberg e-book collection. Our method can identify the proper subset of e-book content to read for a wide collection of diversely structured books and can operate on hundreds of books in parallel. Our system allows users to customize an audiobook's speaking speed and style, emotional intonation, and can even match a desired voice using a small amount of sample audio. This work contributed over five thousand open-license audiobooks and an interactive demo that allows users to quickly create their own customized audiobooks. To listen to the audiobook collection visit \url{https://aka.ms/audiobook}.
机器翻译，仅供参考