【1】 Puffin: pitch-synchronous neural waveform generation for fullband speech on modest devices
标题：Puffin：在普通设备上生成全频带语音的基音同步神经波形
链接：https://arxiv.org/abs/2211.14130
作者：Oliver Watts,Lovisa Wihlborg,Cassia Valentini-Botinhao机构：∗SpeakUnique Ltd., †University of Edinburgh备注：ICASSP 2023 submission摘要：我们提出了一个神经声码器设计与低功率的替代和增强通信设备的想法。通过将成功的现代声码器的元素与老一代技术的既定理念相结合，我们的系统能够在神经声码器过于复杂的设备上产生48 kHz的高质量合成语音。该系统采用可微分基音同步重叠相加进行逆训练，并采用基音同步短时傅里叶逆变换（ISTFT）生成语音样本，降低了系统复杂度。我们的系统以强大的（HiFi-GAN）基线实现了相当的质量，同时仅使用了一小部分计算。我们给出了感知评估的结果以及系统复杂度的分析。摘要：We present a neural vocoder designed with low-powered Alternative and Augmentative Communication devices in mind. By combining elements of successful modern vocoders with established ideas from an older generation of technology, our system is able to produce high quality synthetic speech at 48kHz on devices where neural vocoders are otherwise prohibitively complex. The system is trained adversarially using differentiable pitch synchronous overlap add, and reduces complexity by relying on pitch synchronous Inverse Short-Time Fourier Transform (ISTFT) to generate speech samples. Our system achieves comparable quality with a strong (HiFi-GAN) baseline while using only a fraction of the compute. We present results of a perceptual evaluation as well as an analysis of system complexity.
【2】 Learning General Audio Representations with Large-Scale Training of Patchout Audio Transformers
标题：通过Patchout音频Transformer的大规模训练学习通用音频表示
链接：https://arxiv.org/abs/2211.13956
作者：Khaled Koutini,Shahed Masoudian,Florian Schmid,Hamid Eghbal-zadeh,Jan Schlüter,Gerhard Widmer机构：Jan Schl¨uter, Institute of Computational Perception, & LIT AI Lab, Johannes Kepler University, Linz, Austria, Editor: Joseph Turian, Bj¨orn W. Schuller, Dorien Herremans, Katrin Kirchoff, Paola Garcia Perera, Philippe Esling备注：will apear in HEAR: Holistic Evaluation of Audio Representations Proceedings of Machine Learning Research PMLR 166. Source code: this https URL摘要：监督式深度学习方法的成功很大程度上归功于它们从原始数据中学习相关特征的能力。在大规模数据集上训练的深度神经网络（DNN）能够捕获一组多样的特征，并学习一种表示，该表示可以推广到来自同一领域的未知任务和数据集。因此，对于训练数据量不足以从头开始学习端到端模型的较小任务和数据集，这些模型可以与作为分类器的较浅模型组合用作强大的特征提取器。在过去的几年中，卷积神经网络（CNNs）已经在很大程度上成为用于音频处理的选择方法。然而，最近基于注意力的Transformer模型在监督设置中表现出了巨大的潜力，优于CNNs。在这项工作中，我们研究了在大规模数据集上训练的音频Transformers的使用，以学习通用表示。我们研究了这些音频Transformers中的不同设置如何影响其嵌入质量。我们根据HEAR 2021 NeurIPS挑战评估设置，对模型的时间分辨率、提取的嵌入水平和感受野进行实验，以了解它们如何影响各种任务和数据集的性能。我们的结果表明，由音频Transformers提取的表示优于CNN表示。此外，我们还将展示在Audioset上训练的Transformers可以成为非常有效的表示提取器，用于各种下游任务。摘要：The success of supervised deep learning methods is largely due to their ability to learn relevant features from raw data. Deep Neural Networks (DNNs) trained on large-scale datasets are capable of capturing a diverse set of features, and learning a representation that can generalize onto unseen tasks and datasets that are from the same domain. Hence, these models can be used as powerful feature extractors, in combination with shallower models as classifiers, for smaller tasks and datasets where the amount of training data is insufficient for learning an end-to-end model from scratch. During the past years, Convolutional Neural Networks (CNNs) have largely been the method of choice for audio processing. However, recently attention-based transformer models have demonstrated great potential in supervised settings, outperforming CNNs. In this work, we investigate the use of audio transformers trained on large-scale datasets to learn general-purpose representations. We study how the different setups in these audio transformers affect the quality of their embeddings. We experiment with the models' time resolution, extracted embedding level, and receptive fields in order to see how they affect performance on a variety of tasks and datasets, following the HEAR 2021 NeurIPS challenge evaluation setup. Our results show that representations extracted by audio transformers outperform CNN representations. Furthermore, we will show that transformers trained on Audioset can be extremely effective representation extractors for a wide range of downstream tasks.
【3】 Efficient Incremental Text-to-Speech on GPUs
标题：基于GPU的高效增量式文本语音合成
链接：https://arxiv.org/abs/2211.13939
作者：Muyang Du,Chuan Liu,Jiaxing Qi,Junjie Lai摘要：增量式文本到语音转换，也称为流式TTS，已越来越多地应用于需要超低响应延迟以提供最佳用户体验的在线语音应用。然而，目前大多数基于GPU的语音合成流水线仍然是非增量式的，这在高并发场景下存在局限性，尤其是在使用端到端神经网络模型构建流水线时.为了解决这一问题，提出了一种高效的方法，通过即时请求池和模块式动态批处理在GPU上执行实时增量TTS。实验结果表明，该方法能够在单GPU上以100 QPS的速度生成高质量的语音，首块延迟小于80 ms，在并发性和延迟上均明显优于非增量孪生算法.我们的工作揭示了高性能增量TTS在GPU上的有效性。摘要：Incremental text-to-speech, also known as streaming TTS, has been increasingly applied to online speech applications that require ultra-low response latency to provide an optimal user experience. However, most of the existing speech synthesis pipelines deployed on GPU are still non-incremental, which uncovers limitations in high-concurrency scenarios, especially when the pipeline is built with end-to-end neural network models. To address this issue, we present a highly efficient approach to perform real-time incremental TTS on GPUs with Instant Request Pooling and Module-wise Dynamic Batching. Experimental results demonstrate that the proposed method is capable of producing high-quality speech with a first-chunk latency lower than 80ms under 100 QPS on a single NVIDIA A10 GPU and significantly outperforms the non-incremental twin in both concurrency and latency. Our work reveals the effectiveness of high-performance incremental TTS on GPUs.
【4】 Can Knowledge of End-to-End Text-to-Speech Models Improve Neural MIDI-to-Audio Synthesis Systems?
标题：端到端文本到语音模型的知识能改进神经MIDI到音频合成系统吗？
链接：https://arxiv.org/abs/2211.13868
作者：Xuan Shi,Erica Cooper,Xin Wang,Junichi Yamagishi,Shrikanth Narayanan机构：University of Southern California, USA ,National Institute of Informatics, Japan备注：Submitted to ICASSP 2023摘要：随着音乐与语音合成的相似性以及文本到语音（TTS）技术的迅速发展，借鉴TTS技术来提高MIDI到音频的性能是一个值得探索的问题。本研究分析了基于TTS的MIDI转音频系统的不足，并从特征计算、模型选择和训练策略等方面进行了改进，旨在合成出高自然度的音频。此外，我们还通过听力测试、音高测量和声谱图分析进行了广泛的模型评估。这项工作不仅展示了高度自然的音乐合成，但提供了一个彻底的分析方法和有益的结果，为社区。Our code and pre-trained models are open sourced at https://github.com/nii-yamagishilab/midi-to-audio.摘要：With the similarity between music and speech synthesis from symbolic input and the rapid development of text-to-speech (TTS) techniques, it is worthwhile to explore ways to improve the MIDI-to-audio performance by borrowing from TTS techniques. In this study, we analyze the shortcomings of a TTS-based MIDI-to-audio system and improve it in terms of feature computation, model selection, and training strategy, aiming to synthesize highly natural-sounding audio. Moreover, we conducted an extensive model evaluation through listening tests, pitch measurement, and spectrogram analysis. This work demonstrates not only synthesis of highly natural music but offers a thorough analytical approach and useful outcomes for the community. Our code and pre-trained models are open sourced at https://github.com/nii-yamagishilab/midi-to-audio.
【5】 Multitask Learning for Low Resource Spoken Language Understanding
标题：低资源口语理解的多任务学习
链接：https://arxiv.org/abs/2211.13703
作者：Quentin Meeus,Marie-Francine Moens,Hugo Van hamme机构：KU Leuven, Dept. Computer Sciences CS-LIIR, KU Leuven, Dept. Electrical Engineering ESAT-PSI摘要：我们探索了多任务学习为语音处理提供的好处，因为我们在自动语音识别和意图分类或情感分类的双重目标上训练模型。我们的模型虽然规模不大，但显示出比在意图分类上进行端到端训练的模型有所改进。我们比较不同的设置，以找出每个任务模块的最佳配置。最后，通过对模型进行训练，研究了模型在低资源情况下的性能.我们表明，在这些场景中的多任务学习与基于文本特征训练的基线模型竞争，并且比管道模型表现得更好。在情感分类方面，我们使用十倍于端到端模型的参数来匹配性能。我们考虑了荷兰语和英语的4个任务和4个数据集。摘要：We explore the benefits that multitask learning offer to speech processing as we train models on dual objectives with automatic speech recognition and intent classification or sentiment classification. Our models, although being of modest size, show improvements over models trained end-to-end on intent classification. We compare different settings to find the optimal disposition of each task module compared to one another. Finally, we study the performance of the models in low-resource scenario by training the models with as few as one example per class. We show that multitask learning in these scenarios compete with a baseline model trained on text features and performs considerably better than a pipeline model. On sentiment classification, we match the performance of an end-to-end model with ten times as many parameters. We consider 4 tasks and 4 datasets in Dutch and English.
【6】 TESSP: Text-Enhanced Self-Supervised Speech Pre-training
标题：TESSP：文本增强的自我监督语音预训练
链接：https://arxiv.org/abs/2211.13443
作者：Zhuoyuan Yao,Shuo Ren,Sanyuan Chen,Ziyang Ma,Pengcheng Guo,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Microsoft Corporation摘要：自监督语音预训练使模型具有语音信号中固有的上下文结构，而自监督文本预训练使模型具有语言信息。这两种方法都有利于下游语音任务，如ASR。然而，不同的预训练目标使得在同一模型中联合优化语音和文本表示具有挑战性。针对这一问题，提出了文本增强的自监督语音预训练（TESSP），旨在将语言信息融入语音预训练中。该模型由三部分组成，语音编码器、文本编码器和共享编码器。该模型以无监督语音和文本数据作为输入，分别利用了常见的HuBERT和MLM损失。我们还提出了音素上采样和表示交换，以实现语音和文本信息的联合建模。具体地，为了解决语音和文本数据之间的长度不匹配问题，我们对文本序列进行音素化，并利用从一小组监督数据中提取的对齐信息对音素进行上采样。此外，为了缩小所学习的语音和文本表示之间的差距，我们根据对准信息将文本表示与由相应专用编码器提取的语音表示交换。在Librispeech数据集上的实验结果表明，与WavLM相比，TESSP模型在测试-清洁和测试-其他两个数据集上的性能提高了10%以上。在SUPERB基准测试上对该模型进行了测试，结果表明，该模型在音素识别、声学语音识别和语音翻译等方面的性能均优于WavLM模型。摘要：Self-supervised speech pre-training empowers the model with the contextual structure inherent in the speech signal while self-supervised text pre-training empowers the model with linguistic information. Both of them are beneficial for downstream speech tasks such as ASR. However, the distinct pre-training objectives make it challenging to jointly optimize the speech and text representation in the same model. To solve this problem, we propose Text-Enhanced Self-Supervised Speech Pre-training (TESSP), aiming to incorporate the linguistic information into speech pre-training. Our model consists of three parts, i.e., a speech encoder, a text encoder and a shared encoder. The model takes unsupervised speech and text data as the input and leverages the common HuBERT and MLM losses respectively. We also propose phoneme up-sampling and representation swapping to enable joint modeling of the speech and text information. Specifically, to fix the length mismatching problem between speech and text data, we phonemize the text sequence and up-sample the phonemes with the alignment information extracted from a small set of supervised data. Moreover, to close the gap between the learned speech and text representations, we swap the text representation with the speech representation extracted by the respective private encoders according to the alignment information. Experiments on the Librispeech dataset shows the proposed TESSP model achieves more than 10% improvement compared with WavLM on the test-clean and test-other sets. We also evaluate our model on the SUPERB benchmark, showing our model has better performance on Phoneme Recognition, Acoustic Speech Recognition and Speech Translation compared with WavLM.
【7】 Proceedings of the 4th International Workshop on Reading Music Systems
标题：第四届国际音乐阅读系统研讨会论文集
链接：https://arxiv.org/abs/2211.13285
作者：Jorge Calvo-Zaragoza,Alexander Pacha,Elona Shatri备注：Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri摘要：阅读音乐系统国际研讨会（英语：International Workshop on Reading Music Systems，简称WoRMS）是一个尝试将开发音乐阅读系统的研究人员（例如光学音乐识别领域的研究人员）与其他研究人员和从业人员（例如图书管理员或音乐学家）联系起来的研讨会。研讨会的相关主题包括但不限于：音乐阅读系统光学音乐识别数据集和性能评估;乐谱图像处理写入者标识;乐谱的创作、编辑、存储和呈现系统;多模态系统产生书面音乐的音乐输入新方法基于Web的音乐信息检索服务;申请和项目;与书面音乐相关的用例。这些是2022年11月18日在线举行的第四届阅读音乐系统国际研讨会的会议记录。摘要：The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 4th International Workshop on Reading Music Systems, held online on Nov. 18th 2022.
【8】 Voice-preserving Zero-shot Multiple Accent Conversion
标题：保声零点多重重音转换
链接：https://arxiv.org/abs/2211.13282
作者：Mumin Jin,Prashant Serai,Jilong Wu,Andros Tjandra,Vimal Manohar,Qing He备注：Submitted to IEEE ICASSP 2023摘要：大多数试图学习外语的人都会在理解或用母语口音说话时遇到困难。对于以英语为母语的人来说，理解或说一种新的口音同样是一项困难的任务。改变说话者的口音但保留说话者的声音特征（例如音色和音调）的口音转换系统具有用于一系列应用（例如通信、语言学习和娱乐）的潜力。现有的口音转换模型往往同时改变说话人身份和口音。这里，我们使用对抗学习来解开重音依赖特征，同时保留其他声学特征。我们的工作与现有的口音转换模型不同之处在于能够将看不见的说话者的话语转换为多种口音，同时保留其原始语音身份。主观评估表明，我们的模型生成的音频听起来更接近目标口音，像原始说话者。摘要：Most people who have tried to learn a foreign language would have experienced difficulties understanding or speaking with a native speaker's accent. For native speakers, understanding or speaking a new accent is likewise a difficult task. An accent conversion system that changes a speaker's accent but preserves that speaker's voice identity, such as timbre and pitch, has the potential for a range of applications, such as communication, language learning, and entertainment. Existing accent conversion models tend to change the speaker identity and accent at the same time. Here, we use adversarial learning to disentangle accent dependent features while retaining other acoustic characteristics. What sets our work apart from existing accent conversion models is the capability to convert an unseen speaker's utterance to multiple accents while preserving its original voice identity. Subjective evaluations show that our model generates audio that sound closer to the target accent and like the original speaker.
【9】 Device Directedness with Contextual Cues for Spoken Dialog Systems
标题：使用语音对话系统的上下文提示的设备指导性
链接：https://arxiv.org/abs/2211.13280
作者：Dhanush Bekal,Sundararajan Srinivasan,Sravan Bodapati,Srikanth Ronanki,Katrin Kirchhoff摘要：在本工作中，我们将插嘴验证定义为一个监督学习任务，其中仅使用音频信息来将用户口头对话分类为真插嘴和假插嘴。在预训练模型成功之后，我们使用来自自监督表示学习模型的低级别语音表示来进行我们的下游分类任务。进一步地，我们提出了一种新的技术，将词汇信息直接注入到语音表示中，以改善在预训练过程中隐式学习的特定领域语言信息。在口头对话数据上进行的实验表明，与使用音频和自动语音识别（ASR）1-最佳假设两者的基线LSTM模型相比，被训练成完全根据语音表示来验证插嘴的我们提出的模型相对快38%，并且实现了4.5%的相对F1分数改进。除此之外，我们提出的最佳模型与词汇注入表征以及上下文特征一起提供了5.7%的F1得分进一步相对改善，但仅比基线快22%。摘要：In this work, we define barge-in verification as a supervised learning task where audio-only information is used to classify user spoken dialogue into true and false barge-ins. Following the success of pre-trained models, we use low-level speech representations from a self-supervised representation learning model for our downstream classification task. Further, we propose a novel technique to infuse lexical information directly into speech representations to improve the domain-specific language information implicitly learned during pre-training. Experiments conducted on spoken dialog data show that our proposed model trained to validate barge-in entirely from speech representations is faster by 38% relative and achieves 4.5% relative F1 score improvement over a baseline LSTM model that uses both audio and Automatic Speech Recognition (ASR) 1-best hypotheses. On top of this, our best proposed model with lexically infused representations along with contextual features provides a further relative improvement of 5.7% in the F1 score but only 22% faster than the baseline.
【10】 Prosody-controllable spontaneous TTS with neural HMMs
标题：韵律可控的自发性TTS与神经HMM
链接：https://arxiv.org/abs/2211.13533
作者：Harm Lameris,Shivam Mehta,Gustav Eje Henter,Joakim Gustafson,Éva Székely机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 3 figures, Submitted to ICASSP 2023摘要：自发言语具有许多情感和语用功能，这些功能在文语转换系统（TTS）中的建模是有趣和具有挑战性的。然而，降低的清晰度、填充词、重复和其他不流畅的存在意味着文本和声学比朗读时更不一致。这对于基于注意力的TTS是有问题的。我们提出了一种TTS体系结构，它特别适合于从不规则和小数据集快速学习说话，同时还再现了自发语音中存在的表达现象的多样性。具体地说，我们改进了现有的基于神经HMM的TTS系统，该系统能够对自发语音进行稳定、单调的对齐，并增加了话语级的韵律控制，从而使系统能够代表自发语音语料库中的大范围自然变异。我们客观地评估了控制精度，并进行了主观听力测试，以与没有韵律控制的系统进行比较。为了举例说明结合中级韵律控制和生态学有效数据再现复杂的自发语音现象的能力，我们评估了系统合成两种类型的咯吱发声的能力。音频样本可从www.example.com获得https://hfkml.github.io/pc_nhmm_tts/摘要：Spontaneous speech has many affective and pragmatic functions that are interesting and challenging to model in TTS (text-to-speech). However, the presence of reduced articulation, fillers, repetitions, and other disfluencies mean that text and acoustics are less well aligned than in read speech. This is problematic for attention-based TTS. We propose a TTS architecture that is particularly suited for rapidly learning to speak from irregular and small datasets while also reproducing the diversity of expressive phenomena present in spontaneous speech. Specifically, we modify an existing neural HMM-based TTS system, which is capable of stable, monotonic alignments for spontaneous speech, and add utterance-level prosody control, so that the system can represent the wide range of natural variability in a spontaneous speech corpus. We objectively evaluate control accuracy and perform a subjective listening test to compare to a system without prosody control. To exemplify the power of combining mid-level prosody control and ecologically valid data for reproducing intricate spontaneous speech phenomena, we evaluate the system's capability of synthesizing two types of creaky phonation. Audio samples are available at https://hfkml.github.io/pc_nhmm_tts/
【11】 A new Speech Feature Fusion method with cross gate parallel CNN for Speaker Recognition
标题：一种用于说话人识别的交叉门并行CNN语音特征融合新方法
链接：https://arxiv.org/abs/2211.13377
作者：Jiacheng Zhang,Wenyi Yan,Ye Zhang机构：Zhang are with the Department of Electronicand Information Engineering Nanchang University摘要：提出了一种基于交叉门并行卷积神经网络（CG-PCNN）的语音特征融合方法，用于说话人识别。利用多个三角滤波器组，从说话人语音的每个语音帧中提取不同频率分辨率的Mel滤波器组特征（MFBFs），其中Mel滤波器组中三角滤波器的个数不同。由于这些MBF的频率分辨率不同，因此这些MBF具有一些互补性。利用CG-PCNN提取MBF的深层特征，并采用交叉门机制捕获互补特征，提高说话人识别系统的性能。然后，将这些深度特征连接起来得到融合特征，用于说话人识别。实验结果表明，基于该方法的说话人识别系统是有效的，其性能略优于现有的说话人识别系统。摘要：In this paper, a new speech feature fusion method is proposed for speaker recognition on the basis of the cross gate parallel convolutional neural network (CG-PCNN). The Mel filter bank features (MFBFs) of different frequency resolutions can be extracted from each speech frame of a speaker's speech by several Mel filter banks, where the numbers of the triangular filters in the Mel filter banks are different. Due to the frequency resolutions of these MFBFs are different, there are some complementaries for these MFBFs. The CG-PCNN is utilized to extract the deep features from these MFBFs, which applies a cross gate mechanism to capture the complementaries for improving the performance of the speaker recognition system. Then, the fusion feature can be obtained by concatenating these deep features for speaker recognition. The experimental results show that the speaker recognition system with the proposed speech feature fusion method is effective, and marginally outperforms the existing state-of-the-art systems.
【12】 Adjusting Pleasure-Arousal-Dominance for Continuous Emotional Text-to-speech Synthesizer
标题：连续情感文语合成器的快感唤醒优势调节
链接：https://arxiv.org/abs/1906.05507
作者：Azam Rabiee,Tae-Ho Kim,Soo-Young Lee机构：KAIST Institute for Artificial Intelligence, Korea Advanced Institute of Technology, Daejeon, Korea备注：Interspeech2019, Show and Tell demonstration this https URL&feature=youtu.be摘要：情绪并不局限于快乐、悲伤、愤怒、恐惧、厌恶、惊讶等离散的类别，而是被投射到一系列几乎独立的维度中，这些维度被命名为快乐（或效价）、唤醒和支配，即PAD。每个维度的值从-1变化到1，使得中性情感处于中心，具有全零值。在独立维度上训练情感连续文本到语音（TTS）合成器提供了具有无限情感类别的情感语音合成的可能性。我们的端到端神经语音合成器是基于著名的Tacotron。根据经验，我们已经找到了注入3D PAD的最佳网络架构。此外，为了语音合成的目的，调整PAD值。摘要：Emotion is not limited to discrete categories of happy, sad, angry, fear, disgust, surprise, and so on. Instead, each emotion category is projected into a set of nearly independent dimensions, named pleasure (or valence), arousal, and dominance, known as PAD. The value of each dimension varies from -1 to 1, such that the neutral emotion is in the center with all-zero values. Training an emotional continuous text-to-speech (TTS) synthesizer on the independent dimensions provides the possibility of emotional speech synthesis with unlimited emotion categories. Our end-to-end neural speech synthesizer is based on the well-known Tacotron. Empirically, we have found the optimum network architecture for injecting the 3D PADs. Moreover, the PAD values are adjusted for the speech synthesis purpose.
【13】 A Fully Time-domain Neural Model for Subband-based Speech Synthesizer
标题：一种用于子带语音合成器的全时域神经模型
链接：https://arxiv.org/abs/1810.05319
作者：Azam Rabiee,Geonmin Kim,Tae-Ho Kim,Soo-Young Lee机构：KAIST Institute for Artificial Intelligence, Department of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon, Korea摘要：介绍了一种用于子带语音合成的深度神经网络模型。该模型利用子带信号的短带宽来降低时域语音发生器的复杂度。采用多级小波分析/合成方法对信号进行时域分解/重构。受WaveNet的启发，卷积神经网络（CNN）模型对子带语音信号进行时域全预测。由于子带的带宽较短，简单的网络架构足以精确地训练子带的简单模式。在教师强迫的地面真实实验中，子带合成器在主观和客观测量方面都显著优于全带模型。此外，通过使用发音词典将模型条件化到音素序列上，我们实现了基于子带的接近端到端的文语合成器的全时域神经网络模型。子带TTS生成的语音质量与全带TTS相当，但每个子带的网络结构较轻。摘要：This paper introduces a deep neural network model for subband-based speech synthesizer. The model benefits from the short bandwidth of the subband signals to reduce the complexity of the time-domain speech generator. We employed the multi-level wavelet analysis/synthesis to decompose/reconstruct the signal into subbands in time domain. Inspired from the WaveNet, a convolutional neural network (CNN) model predicts subband speech signals fully in time domain. Due to the short bandwidth of the subbands, a simple network architecture is enough to train the simple patterns of the subbands accurately. In the ground truth experiments with teacher-forcing, the subband synthesizer outperforms the fullband model significantly in terms of both subjective and objective measures. In addition, by conditioning the model on the phoneme sequence using a pronunciation dictionary, we have achieved the fully time-domain neural model for subband-based text-to-speech (TTS) synthesizer, which is nearly end-to-end. The generated speech of the subband TTS shows comparable quality as the fullband one with a slighter network architecture for each subband.
【1】 Prosody-controllable spontaneous TTS with neural HMMs
标题：韵律可控的自发性TTS与神经HMM
链接：https://arxiv.org/abs/2211.13533
* 与cs.SD语音【10】为同一篇
作者：Harm Lameris,Shivam Mehta,Gustav Eje Henter,Joakim Gustafson,Éva Székely机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 3 figures, Submitted to ICASSP 2023摘要：自发言语具有许多情感和语用功能，这些功能在文语转换系统（TTS）中的建模是有趣和具有挑战性的。然而，降低的清晰度、填充词、重复和其他不流畅的存在意味着文本和声学比朗读时更不一致。这对于基于注意力的TTS是有问题的。我们提出了一种TTS体系结构，它特别适合于从不规则和小数据集快速学习说话，同时还再现了自发语音中存在的表达现象的多样性。具体地说，我们改进了现有的基于神经HMM的TTS系统，该系统能够对自发语音进行稳定、单调的对齐，并增加了话语级的韵律控制，使系统能够代表自发语音语料库中的大范围自然变异。我们客观地评估了控制精度，并进行了主观听力测试，以与没有韵律控制的系统进行比较。为了举例说明结合中级韵律控制和生态学有效数据再现复杂的自发语音现象的能力，我们评估了系统合成两种类型的咯吱发声的能力。音频样本可从www.example.com获得https://hfkml.github.io/pc_nhmm_tts/摘要：Spontaneous speech has many affective and pragmatic functions that are interesting and challenging to model in TTS (text-to-speech). However, the presence of reduced articulation, fillers, repetitions, and other disfluencies mean that text and acoustics are less well aligned than in read speech. This is problematic for attention-based TTS. We propose a TTS architecture that is particularly suited for rapidly learning to speak from irregular and small datasets while also reproducing the diversity of expressive phenomena present in spontaneous speech. Specifically, we modify an existing neural HMM-based TTS system, which is capable of stable, monotonic alignments for spontaneous speech, and add utterance-level prosody control, so that the system can represent the wide range of natural variability in a spontaneous speech corpus. We objectively evaluate control accuracy and perform a subjective listening test to compare to a system without prosody control. To exemplify the power of combining mid-level prosody control and ecologically valid data for reproducing intricate spontaneous speech phenomena, we evaluate the system's capability of synthesizing two types of creaky phonation. Audio samples are available at https://hfkml.github.io/pc_nhmm_tts/
【2】 A new Speech Feature Fusion method with cross gate parallel CNN for Speaker Recognition
标题：一种用于说话人识别的交叉门并行CNN语音特征融合新方法
链接：https://arxiv.org/abs/2211.13377
* 与cs.SD语音【11】为同一篇
作者：Jiacheng Zhang,Wenyi Yan,Ye Zhang机构：Zhang are with the Department of Electronicand Information Engineering Nanchang University摘要：提出了一种基于交叉门并行卷积神经网络（CG-PCNN）的语音特征融合方法，用于说话人识别。利用多个三角滤波器组，从说话人语音的每个语音帧中提取不同频率分辨率的Mel滤波器组特征（MFBFs），其中Mel滤波器组中三角滤波器的个数不同。由于这些MBF的频率分辨率不同，因此这些MBF具有一些互补性。利用CG-PCNN提取MBF的深层特征，并采用交叉门机制捕获互补特征，提高说话人识别系统的性能。然后，将这些深度特征连接起来得到融合特征，用于说话人识别。实验结果表明，基于该方法的说话人识别系统是有效的，其性能略优于现有的说话人识别系统。摘要：In this paper, a new speech feature fusion method is proposed for speaker recognition on the basis of the cross gate parallel convolutional neural network (CG-PCNN). The Mel filter bank features (MFBFs) of different frequency resolutions can be extracted from each speech frame of a speaker's speech by several Mel filter banks, where the numbers of the triangular filters in the Mel filter banks are different. Due to the frequency resolutions of these MFBFs are different, there are some complementaries for these MFBFs. The CG-PCNN is utilized to extract the deep features from these MFBFs, which applies a cross gate mechanism to capture the complementaries for improving the performance of the speaker recognition system. Then, the fusion feature can be obtained by concatenating these deep features for speaker recognition. The experimental results show that the speaker recognition system with the proposed speech feature fusion method is effective, and marginally outperforms the existing state-of-the-art systems.
【3】 Puffin: pitch-synchronous neural waveform generation for fullband speech on modest devices
标题：Puffin：在普通设备上生成全频带语音的基音同步神经波形
链接：https://arxiv.org/abs/2211.14130
* 与cs.SD语音【1】为同一篇
作者：Oliver Watts,Lovisa Wihlborg,Cassia Valentini-Botinhao机构：∗SpeakUnique Ltd., †University of Edinburgh备注：ICASSP 2023 submission摘要：我们提出了一个神经声码器设计与低功率的替代和增强通信设备的想法。通过将成功的现代声码器的元素与老一代技术的既定理念相结合，我们的系统能够在神经声码器过于复杂的设备上产生48kHz的高质量合成语音。该系统采用可微分基音同步重叠相加进行逆训练，并采用基音同步短时傅里叶逆变换（ISTFT）生成语音样本，降低了系统复杂度。我们的系统以强大的（HiFi-GAN）基线实现了相当的质量，同时仅使用了一小部分计算。我们给出了感知评估的结果以及系统复杂度的分析。摘要：We present a neural vocoder designed with low-powered Alternative and Augmentative Communication devices in mind. By combining elements of successful modern vocoders with established ideas from an older generation of technology, our system is able to produce high quality synthetic speech at 48kHz on devices where neural vocoders are otherwise prohibitively complex. The system is trained adversarially using differentiable pitch synchronous overlap add, and reduces complexity by relying on pitch synchronous Inverse Short-Time Fourier Transform (ISTFT) to generate speech samples. Our system achieves comparable quality with a strong (HiFi-GAN) baseline while using only a fraction of the compute. We present results of a perceptual evaluation as well as an analysis of system complexity.
【4】 Learning General Audio Representations with Large-Scale Training of Patchout Audio Transformers
标题：通过Patchout音频Transformer的大规模训练学习通用音频表示
链接：https://arxiv.org/abs/2211.13956
* 与cs.SD语音【2】为同一篇
作者：Khaled Koutini,Shahed Masoudian,Florian Schmid,Hamid Eghbal-zadeh,Jan Schlüter,Gerhard Widmer机构：Jan Schl¨uter, Institute of Computational Perception, & LIT AI Lab, Johannes Kepler University, Linz, Austria, Editor: Joseph Turian, Bj¨orn W. Schuller, Dorien Herremans, Katrin Kirchoff, Paola Garcia Perera, Philippe Esling备注：will apear in HEAR: Holistic Evaluation of Audio Representations Proceedings of Machine Learning Research PMLR 166. Source code: this https URL摘要：监督式深度学习方法的成功很大程度上归功于它们从原始数据中学习相关特征的能力。在大规模数据集上训练的深度神经网络（DNN）能够捕获一组多样的特征，并学习一种表示，该表示可以推广到来自同一领域的未知任务和数据集。因此，对于训练数据量不足以从头开始学习端到端模型的较小任务和数据集，这些模型可以与作为分类器的较浅模型组合用作强大的特征提取器。在过去的几年中，卷积神经网络（CNNs）已经在很大程度上成为用于音频处理的选择方法。然而，最近基于注意力的转换器模型在监督设置中表现出了巨大的潜力，优于CNNs。在这项工作中，我们研究了在大规模数据集上训练的音频转换器的使用，以学习通用表示。我们研究了这些音频转换器中的不同设置如何影响其嵌入质量。我们根据HEAR 2021 NeurIPS挑战评估设置，对模型的时间分辨率、提取的嵌入水平和感受野进行实验，以了解它们如何影响各种任务和数据集的性能。我们的结果表明，由音频变换器提取的表示优于CNN表示。此外，我们还将展示在Audioset上训练的变压器可以成为非常有效的表示提取器，用于各种下游任务。摘要：The success of supervised deep learning methods is largely due to their ability to learn relevant features from raw data. Deep Neural Networks (DNNs) trained on large-scale datasets are capable of capturing a diverse set of features, and learning a representation that can generalize onto unseen tasks and datasets that are from the same domain. Hence, these models can be used as powerful feature extractors, in combination with shallower models as classifiers, for smaller tasks and datasets where the amount of training data is insufficient for learning an end-to-end model from scratch. During the past years, Convolutional Neural Networks (CNNs) have largely been the method of choice for audio processing. However, recently attention-based transformer models have demonstrated great potential in supervised settings, outperforming CNNs. In this work, we investigate the use of audio transformers trained on large-scale datasets to learn general-purpose representations. We study how the different setups in these audio transformers affect the quality of their embeddings. We experiment with the models' time resolution, extracted embedding level, and receptive fields in order to see how they affect performance on a variety of tasks and datasets, following the HEAR 2021 NeurIPS challenge evaluation setup. Our results show that representations extracted by audio transformers outperform CNN representations. Furthermore, we will show that transformers trained on Audioset can be extremely effective representation extractors for a wide range of downstream tasks.
【5】 Efficient Incremental Text-to-Speech on GPUs
标题：基于GPU的高效增量式文本语音合成
链接：https://arxiv.org/abs/2211.13939
* 与cs.SD语音【3】为同一篇
作者：Muyang Du,Chuan Liu,Jiaxing Qi,Junjie Lai摘要：增量式文本到语音转换，也称为流式TTS，已越来越多地应用于需要超低响应延迟以提供最佳用户体验的在线语音应用。然而，现有的大多数基于GPU的语音合成流水线仍然是非增量式的，这在高并发场景下暴露出了局限性，尤其是当流水线是由端到端的神经网络模型构建时。为了解决这一问题，本文提出了一种高效的方法，通过即时请求池和模块式动态批处理在GPU上执行实时增量TTS。实验结果表明，该方法能够在单GPU上以100 QPS的速度生成高质量的语音，首块延迟小于80 ms，在并发性和延迟上均明显优于非增量孪生算法.我们的工作揭示了高性能增量TTS在GPU上的有效性。摘要：Incremental text-to-speech, also known as streaming TTS, has been increasingly applied to online speech applications that require ultra-low response latency to provide an optimal user experience. However, most of the existing speech synthesis pipelines deployed on GPU are still non-incremental, which uncovers limitations in high-concurrency scenarios, especially when the pipeline is built with end-to-end neural network models. To address this issue, we present a highly efficient approach to perform real-time incremental TTS on GPUs with Instant Request Pooling and Module-wise Dynamic Batching. Experimental results demonstrate that the proposed method is capable of producing high-quality speech with a first-chunk latency lower than 80ms under 100 QPS on a single NVIDIA A10 GPU and significantly outperforms the non-incremental twin in both concurrency and latency. Our work reveals the effectiveness of high-performance incremental TTS on GPUs.
【6】 Can Knowledge of End-to-End Text-to-Speech Models Improve Neural MIDI-to-Audio Synthesis Systems?
标题：端到端文本到语音模型的知识能改进神经MIDI到音频合成系统吗？
链接：https://arxiv.org/abs/2211.13868
* 与cs.SD语音【4】为同一篇
作者：Xuan Shi,Erica Cooper,Xin Wang,Junichi Yamagishi,Shrikanth Narayanan机构：University of Southern California, USA ,National Institute of Informatics, Japan备注：Submitted to ICASSP 2023摘要：随着音乐与语音合成的相似性以及文本到语音（TTS）技术的迅速发展，借鉴TTS技术来提高MIDI到音频的性能是一个值得探索的问题。本研究分析了基于TTS的MIDI转音频系统的不足，并从特征计算、模型选择和训练策略等方面进行了改进，旨在合成出高自然度的音频。此外，我们还通过听力测试、音高测量和声谱图分析进行了广泛的模型评估。这项工作不仅展示了高度自然的音乐合成，但提供了一个彻底的分析方法和有益的结果，为社区。我们的代码和预先训练的模型是开源的，请访问https://github.com/nii-yamagishilab/midi-to-audio。摘要：With the similarity between music and speech synthesis from symbolic input and the rapid development of text-to-speech (TTS) techniques, it is worthwhile to explore ways to improve the MIDI-to-audio performance by borrowing from TTS techniques. In this study, we analyze the shortcomings of a TTS-based MIDI-to-audio system and improve it in terms of feature computation, model selection, and training strategy, aiming to synthesize highly natural-sounding audio. Moreover, we conducted an extensive model evaluation through listening tests, pitch measurement, and spectrogram analysis. This work demonstrates not only synthesis of highly natural music but offers a thorough analytical approach and useful outcomes for the community. Our code and pre-trained models are open sourced at https://github.com/nii-yamagishilab/midi-to-audio.
【7】 Multitask Learning for Low Resource Spoken Language Understanding
标题：低资源口语理解的多任务学习
链接：https://arxiv.org/abs/2211.13703
* 与cs.SD语音【5】为同一篇
作者：Quentin Meeus,Marie-Francine Moens,Hugo Van hamme机构：KU Leuven, Dept. Computer Sciences CS-LIIR, KU Leuven, Dept. Electrical Engineering ESAT-PSI摘要：我们探索了多任务学习为语音处理提供的好处，因为我们在自动语音识别和意图分类或情感分类的双重目标上训练模型。我们的模型虽然规模不大，但显示出比在意图分类上进行端到端训练的模型有所改进。我们比较不同的设置，以找出每个任务模块的最佳配置。最后，通过对模型进行训练，研究了模型在低资源情况下的性能.我们表明，在这些场景中的多任务学习与基于文本特征训练的基线模型竞争，并且比管道模型表现得更好。在情感分类方面，我们使用十倍于端到端模型的参数来匹配性能。我们考虑了荷兰语和英语的4个任务和4个数据集。摘要：We explore the benefits that multitask learning offer to speech processing as we train models on dual objectives with automatic speech recognition and intent classification or sentiment classification. Our models, although being of modest size, show improvements over models trained end-to-end on intent classification. We compare different settings to find the optimal disposition of each task module compared to one another. Finally, we study the performance of the models in low-resource scenario by training the models with as few as one example per class. We show that multitask learning in these scenarios compete with a baseline model trained on text features and performs considerably better than a pipeline model. On sentiment classification, we match the performance of an end-to-end model with ten times as many parameters. We consider 4 tasks and 4 datasets in Dutch and English.
【8】 TESSP: Text-Enhanced Self-Supervised Speech Pre-training
标题：TESSP：文本增强的自我监督语音预训练
链接：https://arxiv.org/abs/2211.13443
* 与cs.SD语音【6】为同一篇
作者：Zhuoyuan Yao,Shuo Ren,Sanyuan Chen,Ziyang Ma,Pengcheng Guo,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, Microsoft Corporation摘要：自监督语音预训练使模型具有语音信号中固有的上下文结构，而自监督文本预训练使模型具有语言信息。这两种方法都有利于诸如ASR之类的下游语音任务。然而，不同的预训练目标使得在同一模型中联合优化语音和文本表示具有挑战性。针对这一问题，提出了文本增强的自监督语音预训练（TESSP），旨在将语言信息融入语音预训练中。该模型由三部分组成，语音编码器、文本编码器和共享编码器。该模型以无监督语音和文本数据作为输入，分别利用了常见的HuBERT和MLM损失。我们还提出了音素上采样和表示交换，以实现语音和文本信息的联合建模。具体地，为了解决语音和文本数据之间的长度不匹配问题，我们对文本序列进行音素化，并利用从一小组监督数据中提取的对齐信息对音素进行上采样。此外，为了缩小所学习的语音和文本表示之间的差距，我们根据对准信息将文本表示与由相应专用编码器提取的语音表示交换。在Librispeech数据集上的实验结果表明，与WavLM相比，TESSP模型在测试-清洁和测试-其他两个数据集上的性能提高了10%以上。在SUPERB基准测试上对该模型进行了测试，结果表明，该模型在音素识别、声学语音识别和语音翻译等方面的性能均优于WavLM模型。摘要：Self-supervised speech pre-training empowers the model with the contextual structure inherent in the speech signal while self-supervised text pre-training empowers the model with linguistic information. Both of them are beneficial for downstream speech tasks such as ASR. However, the distinct pre-training objectives make it challenging to jointly optimize the speech and text representation in the same model. To solve this problem, we propose Text-Enhanced Self-Supervised Speech Pre-training (TESSP), aiming to incorporate the linguistic information into speech pre-training. Our model consists of three parts, i.e., a speech encoder, a text encoder and a shared encoder. The model takes unsupervised speech and text data as the input and leverages the common HuBERT and MLM losses respectively. We also propose phoneme up-sampling and representation swapping to enable joint modeling of the speech and text information. Specifically, to fix the length mismatching problem between speech and text data, we phonemize the text sequence and up-sample the phonemes with the alignment information extracted from a small set of supervised data. Moreover, to close the gap between the learned speech and text representations, we swap the text representation with the speech representation extracted by the respective private encoders according to the alignment information. Experiments on the Librispeech dataset shows the proposed TESSP model achieves more than 10% improvement compared with WavLM on the test-clean and test-other sets. We also evaluate our model on the SUPERB benchmark, showing our model has better performance on Phoneme Recognition, Acoustic Speech Recognition and Speech Translation compared with WavLM.
【9】 Proceedings of the 4th International Workshop on Reading Music Systems
标题：第四届国际音乐阅读系统研讨会论文集
链接：https://arxiv.org/abs/2211.13285
* 与cs.SD语音【7】为同一篇
作者：Jorge Calvo-Zaragoza,Alexander Pacha,Elona Shatri备注：Proceedings edited by Jorge Calvo-Zaragoza, Alexander Pacha and Elona Shatri摘要：阅读音乐系统国际研讨会（英语：International Workshop on Reading Music Systems，简称WoRMS）是一个尝试将开发音乐阅读系统的研究人员（例如光学音乐识别领域的研究人员）与其他研究人员和从业人员（例如图书管理员或音乐学家）联系起来的研讨会。研讨会的相关主题包括但不限于：音乐阅读系统光学音乐识别数据集和性能评估;乐谱图像处理写入者标识;乐谱的创作、编辑、存储和呈现系统;多模态系统产生书面音乐的音乐输入新方法基于Web的音乐信息检索服务;申请和项目;与书面音乐相关的用例。这些是2022年11月18日在线举行的第四届阅读音乐系统国际研讨会的会议记录。摘要：The International Workshop on Reading Music Systems (WoRMS) is a workshop that tries to connect researchers who develop systems for reading music, such as in the field of Optical Music Recognition, with other researchers and practitioners that could benefit from such systems, like librarians or musicologists. The relevant topics of interest for the workshop include, but are not limited to: Music reading systems; Optical music recognition; Datasets and performance evaluation; Image processing on music scores; Writer identification; Authoring, editing, storing and presentation systems for music scores; Multi-modal systems; Novel input-methods for music to produce written music; Web-based Music Information Retrieval services; Applications and projects; Use-cases related to written music. These are the proceedings of the 4th International Workshop on Reading Music Systems, held online on Nov. 18th 2022.
【10】 Voice-preserving Zero-shot Multiple Accent Conversion
标题：保声零点多重重音转换
链接：https://arxiv.org/abs/2211.13282
* 与cs.SD语音【8】为同一篇
作者：Mumin Jin,Prashant Serai,Jilong Wu,Andros Tjandra,Vimal Manohar,Qing He备注：Submitted to IEEE ICASSP 2023摘要：大多数试图学习外语的人都会在理解或用母语口音说话时遇到困难。对于以英语为母语的人来说，理解或说一种新的口音同样是一项困难的任务。改变说话者的口音但保留说话者的声音特征（例如音色和音调）的口音转换系统具有用于一系列应用（例如通信、语言学习和娱乐）的潜力。现有的口音转换模型往往同时改变说话人身份和口音。这里，我们使用对抗学习来解开重音依赖特征，同时保留其他声学特征。我们的工作与现有的口音转换模型不同之处在于能够将看不见的说话者的话语转换为多种口音，同时保留其原始语音身份。主观评估表明，我们的模型生成的音频听起来更接近目标口音，像原始说话者。摘要：Most people who have tried to learn a foreign language would have experienced difficulties understanding or speaking with a native speaker's accent. For native speakers, understanding or speaking a new accent is likewise a difficult task. An accent conversion system that changes a speaker's accent but preserves that speaker's voice identity, such as timbre and pitch, has the potential for a range of applications, such as communication, language learning, and entertainment. Existing accent conversion models tend to change the speaker identity and accent at the same time. Here, we use adversarial learning to disentangle accent dependent features while retaining other acoustic characteristics. What sets our work apart from existing accent conversion models is the capability to convert an unseen speaker's utterance to multiple accents while preserving its original voice identity. Subjective evaluations show that our model generates audio that sound closer to the target accent and like the original speaker.
【11】 Device Directedness with Contextual Cues for Spoken Dialog Systems
标题：使用语音对话系统的上下文提示的设备指导性
链接：https://arxiv.org/abs/2211.13280
* 与cs.SD语音【9】为同一篇
作者：Dhanush Bekal,Sundararajan Srinivasan,Sravan Bodapati,Srikanth Ronanki,Katrin Kirchhoff摘要：在本工作中，我们将插嘴验证定义为一个监督学习任务，其中仅使用音频信息来将用户口头对话分类为真插嘴和假插嘴。在预训练模型成功之后，我们使用来自自监督表示学习模型的低级别语音表示来进行我们的下游分类任务。进一步地，我们提出了一种新的技术，将词汇信息直接注入到语音表示中，以改善在预训练过程中隐式学习的特定领域语言信息。在口头对话数据上进行的实验表明，与使用音频和自动语音识别（ASR）1-最佳假设两者的基线LSTM模型相比，被训练成完全根据语音表示来验证插嘴的我们提出的模型相对快38%，并且实现了4.5%的相对F1分数改进。除此之外，我们提出的最佳模型与词汇注入表征以及上下文特征一起提供了5.7%的F1得分进一步相对改善，但仅比基线快22%。摘要：In this work, we define barge-in verification as a supervised learning task where audio-only information is used to classify user spoken dialogue into true and false barge-ins. Following the success of pre-trained models, we use low-level speech representations from a self-supervised representation learning model for our downstream classification task. Further, we propose a novel technique to infuse lexical information directly into speech representations to improve the domain-specific language information implicitly learned during pre-training. Experiments conducted on spoken dialog data show that our proposed model trained to validate barge-in entirely from speech representations is faster by 38% relative and achieves 4.5% relative F1 score improvement over a baseline LSTM model that uses both audio and Automatic Speech Recognition (ASR) 1-best hypotheses. On top of this, our best proposed model with lexically infused representations along with contextual features provides a further relative improvement of 5.7% in the F1 score but only 22% faster than the baseline.
机器翻译，仅供参考
每日语音速递