今天跟大家分享一篇语音相关的论文合集：cs.SD语音9篇，eess.AS音频处理12篇。
【1】 CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement
标题：CMGAN：基于Conform的Metric-GaN单声道语音增强
链接：https://arxiv.org/abs/2209.11112
作者：Sherif Abdulatif,Ruizhe Cao,Bin Yang机构：This limitation is usually reflected as artifacts in theThe authors are with the Institute of Signal Processing and Sys-tem Theory, University of Stuttgart备注：16 pages, 10 figures and 5 tables. arXiv admin note: text overlap with arXiv:2203.15149摘要：卷积增强Transformers（Conformer）最近被提出用于各种语音领域的应用，例如自动语音识别（ASR）和语音分离，因为它们可以捕获局部和全局依赖性。本文提出了一种基于保形器的度量生成对抗网络（CMGAN），用于时频域语音增强。该发生器使用两级构象异构器模块对幅度和复频谱图信息进行编码，以对时间和频率依赖性进行建模。解码器然后将估计解耦到幅度掩码解码器分支和复数细化分支中，幅度掩码解码器分支用于滤除不想要的失真，复数细化分支用于进一步改进幅度估计并隐含地增强相位信息。另外，我们包括度量鉴别器，以通过相对于相应的评估分数优化生成器来减轻度量失配。客观和主观评价结果表明，CMGAN在3种语音增强任务（去噪、去混响和超分辨率）中均能表现出比现有方法更好的性能。例如，对Voice Bank+DEMAND数据集的定量去噪分析表明，CMGAN优于各种先前的模型，具有一定余量，即，PESQ为3.41，SSNR为11.10 dB。摘要：Convolution-augmented transformers (Conformers) are recently proposed in various speech-domain applications, such as automatic speech recognition (ASR) and speech separation, as they can capture both local and global dependencies. In this paper, we propose a conformer-based metric generative adversarial network (CMGAN) for speech enhancement (SE) in the time-frequency (TF) domain. The generator encodes the magnitude and complex spectrogram information using two-stage conformer blocks to model both time and frequency dependencies. The decoder then decouples the estimation into a magnitude mask decoder branch to filter out unwanted distortions and a complex refinement branch to further improve the magnitude estimation and implicitly enhance the phase information. Additionally, we include a metric discriminator to alleviate metric mismatch by optimizing the generator with respect to a corresponding evaluation score. Objective and subjective evaluations illustrate that CMGAN is able to show superior performance compared to state-of-the-art methods in three speech enhancement tasks (denoising, dereverberation and super-resolution). For instance, quantitative denoising analysis on Voice Bank+DEMAND dataset indicates that CMGAN outperforms various previous models with a margin, i.e., PESQ of 3.41 and SSNR of 11.10 dB.
【2】 Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networks
标题：使用并行评级数据和反对称双神经网络预测TTS音频刺激之间的成对偏好
链接：https://arxiv.org/abs/2209.11003
作者：Cassia Valentini-Botinhao,Manuel Sam Ribeiro,Oliver Watts,Korin Richmond,Gustav Eje Henter机构：The Centre for Speech Technology Research, University of Edinburgh, UK, SpeakUnique Ltd., UK, Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden摘要：主观听力测试结果的自动预测是一项具有挑战性的任务。即使听众的偏好是一致的，评级也可能因人而异。虽然以前的工作集中在预测听众对单个刺激的评分（平均意见得分），但我们集中在预测同一文本的两个语音刺激的主观偏好这一更简单的任务上。我们提出了一个基于反对称孪生神经网络的模型，该模型在成对的波形和它们对应的偏好得分上进行训练。我们研究了注意神经网络和递归神经网络，以解释一对刺激中的刺激不是时间对齐的事实。为了获得一个大的训练集，我们将听众的评分从MUSHRA测试转换为反映成对刺激中一个刺激比另一个刺激评分高的频率的值。具体而言，我们评估了从五年来进行的12次MUSHRA评估中获得的数据的性能，这些评估包含不同的TTS系统，这些系统是根据不同说话者的数据构建的。我们的结果优于最先进的模型训练预测MOS评分。摘要：Automatically predicting the outcome of subjective listening tests is a challenging task. Ratings may vary from person to person even if preferences are consistent across listeners. While previous work has focused on predicting listeners' ratings (mean opinion scores) of individual stimuli, we focus on the simpler task of predicting subjective preference given two speech stimuli for the same text. We propose a model based on anti-symmetric twin neural networks, trained on pairs of waveforms and their corresponding preference scores. We explore both attention and recurrent neural nets to account for the fact that stimuli in a pair are not time aligned. To obtain a large training set we convert listeners' ratings from MUSHRA tests to values that reflect how often one stimulus in the pair was rated higher than the other. Specifically, we evaluate performance on data obtained from twelve MUSHRA evaluations conducted over five years, containing different TTS systems, built from data of different speakers. Our results compare favourably to a state-of-the-art model trained to predict MOS scores.
【3】 Maths, Computation and Flamenco: overview and challenges
标题：数学、计算和弗拉门戈：概述和挑战
链接：https://arxiv.org/abs/2209.10970
作者：José-Miguel Díaz-Báñez,Nadine Kroher摘要：弗拉门戈是一种来自西班牙南部的以表演为导向的艺术音乐流派，吸引了全球越来越多的爱好者。由于其即兴和表现的性质，其独特的音乐特征，以及事实上，这一流派是很大程度上没有文件，弗拉门戈提出了一些有趣的数学和计算的挑战。音乐信息检索（MIR）中的大多数现有方法是在流行音乐或古典音乐的背景下开发的，并且通常不能很好地推广到非西方音乐传统，特别是当潜在的音乐理论假设不适用于这些流派时。在最近十年中，已经定义了与弗拉门戈音乐的自动分析相关的许多计算问题，并且已经提出了解决各种音乐方面的若干方法。本文概述了弗拉门戈音乐计算分析的挑战，并概述了现有的方法。摘要：Flamenco is a rich performance-oriented art music genre from Southern Spain which attracts a growing community of aficionados around the globe. Due to its improvisational and expressive nature, its unique musical characteristics, and the fact that the genre is largely undocumented, flamenco poses a number of interesting mathematical and computational challenges. Most existing approaches in Musical Information Retrieval (MIR) were developed in the context of popular or classical music and do often not generalize well to non-Western music traditions, in particular when the underlying music theoretical assumptions do not hold for these genres. Over the recent decade, a number of computational problems related to the automatic analysis of flamenco music have been defined and several methods addressing a variety of musical aspects have been proposed. This paper provides an overview of the challenges which arise in the context of computational analysis of flamenco music and outlines an overview of existing approaches.
【4】 A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS
标题：一种用于高性能神经TTS的多级多码本VQ-VAE方法
链接：https://arxiv.org/abs/2209.10887
作者：Haohan Guo,Fenglong Xie,Frank K. Soong,Xixin Wu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, ‡ Xiaohongshu Inc., Shanghai, China, †Microsoft Research Asia, Beijing, China摘要：我们提出了一种多阶段、多码本（MSMC）的高性能神经TTS合成方法。基于矢量量化变分自动编码器（VQ-VAE）的特征分析器对语音训练数据的Mel谱图进行多级渐进下采样，得到不同时间分辨率的MSMC表示（MSMCR），并分别用多个VQ码本进行量化。通过最小化重建均方误差（MSE）和“三元组损失”的组合损失，训练多阶段预测器以将输入文本序列渐进地映射到MSMCR。在合成中，神经声码器将预测的MSMCR转换成最终的语音波形。在一个16小时的英语TTS语料库上，由一位女性说话人对该方法进行了训练和测试。申报TTS的MOS评分为4.41，优于基线的MOS评分3.62。所提出的TTS的具有少得多的参数的紧凑版本仍然可以保持高MOS分数。消融研究表明，多级和多个码本都能有效地实现高TTS性能。摘要：We propose a Multi-Stage, Multi-Codebook (MSMC) approach to high-performance neural TTS synthesis. A vector-quantized, variational autoencoder (VQ-VAE) based feature analyzer is used to encode Mel spectrograms of speech training data by down-sampling progressively in multiple stages into MSMC Representations (MSMCRs) with different time resolutions, and quantizing them with multiple VQ codebooks, respectively. Multi-stage predictors are trained to map the input text sequence to MSMCRs progressively by minimizing a combined loss of the reconstruction Mean Square Error (MSE) and "triplet loss". In synthesis, the neural vocoder converts the predicted MSMCRs into final speech waveforms. The proposed approach is trained and tested with an English TTS database of 16 hours by a female speaker. The proposed TTS achieves an MOS score of 4.41, which outperforms the baseline with an MOS of 3.62. Compact versions of the proposed TTS with much less parameters can still preserve high MOS scores. Ablation studies show that both multiple stages and multiple codebooks are effective for achieving high TTS performance.
【5】 MnTTS: An Open-Source Mongolian Text-to-Speech Synthesis Dataset and Accompanied Baseline
标题：MnTTS：一个开源的蒙古语文语合成数据集和伴随基线
链接：https://arxiv.org/abs/2209.10848
作者：Yifan Hu,Pengkai Yin,Rui Liu,Feilong Bao,Guanglai Gao机构：College of Computer Science, Inner Mongolia University, Hohhot, China备注：Accepted at the 2022 International Conference on Asian Language Processing (IALP2022)摘要：本文介绍了一个高质量的蒙古语文语转换（TTS）合成数据集，该数据集是一个开源的高质量数据集。该数据集名为MnTTS，由一位22岁的蒙古族职业女播音员所讲的大约8小时的转录音频记录组成。它是第一个公开的数据集，旨在促进蒙古语TTS在学术界和工业界的应用。本文通过介绍数据集的开发过程和面临的挑战，分享了我们的经验。为了验证我们的数据集的可靠性，我们基于FastSpeech 2模型和HiFi-GAN声码器构建了一个功能强大的非自回归基线系统，并使用主观平均意见得分（MOS）和实时因子（RTF）度量对其进行了评估。评估结果表明，在我们的数据集上训练的强大的基线系统达到了4以上的MOS和约3.30 × 10 ^{-1}$的RTF，这使它适合于实际使用。数据集、训练方法和预训练的TTS模型可免费获得。摘要：This paper introduces a high-quality open-source text-to-speech (TTS) synthesis dataset for Mongolian, a low-resource language spoken by over 10 million people worldwide. The dataset, named MnTTS, consists of about 8 hours of transcribed audio recordings spoken by a 22-year-old professional female Mongolian announcer. It is the first publicly available dataset developed to promote Mongolian TTS applications in both academia and industry. In this paper, we share our experience by describing the dataset development procedures and faced challenges. To demonstrate the reliability of our dataset, we built a powerful non-autoregressive baseline system based on FastSpeech2 model and HiFi-GAN vocoder, and evaluated it using the subjective mean opinion score (MOS) and real time factor (RTF) metrics. Evaluation results show that the powerful baseline system trained on our dataset achieves MOS above 4 and RTF about $3.30\times10^{-1}$, which makes it applicable for practical use. The dataset, training recipe, and pretrained TTS models are freely available \footnote{\label{github}\url{https://github.com/walker-hyf/MnTTS}}.
【6】 The SpeakIn System Description for CNSRC2022
标题：CNSRC2022的扬声器系统描述
链接：https://arxiv.org/abs/2209.10846
作者：Yu Zheng,Yihao Chen,Jinghan Peng,Yajun Zhang,Min Liu,Minqiang Xu机构：SpeakIn Technologies Co. Ltd.摘要：本报告描述了我们为2022年CN-Celeb说话人识别挑战赛（CNSRC 2022）任务而开发的说话人验证系统。该挑战包括两个任务，即说话人确认（SV）和说话人检索（SR）。SV任务涉及两个轨道：固定轨道和开放轨道。在固定轨道中，我们仅使用CN-Celeb.T作为训练集。对于SV任务和SR任务的开放音轨，我们添加了我们的开源音频数据。基于ResNet、基于RepVGG和基于TDNN的架构就是为应对这一挑战而开发的。采用全局统计池结构和MQMHA池结构对帧级特征进行跨时间聚合，得到话语级特征表示。采用AM-Softmax和AAM-Softmax结合Sub-Center方法对嵌入结果进行分类。我们还使用了大裕度微调策略来进一步提高模型性能。在后端，使用次平均值和AS-范数。在SV任务固定航迹中，系统融合了5个模型;在SV任务开放航迹中，融合了2个模型。我们在SR任务中使用了单个系统。我们的方法在SV任务的开放轨道中获得了第一名，在SV任务的固定轨道中获得了第二名，在SR任务中获得了第三名。摘要：This report describes our speaker verification systems for the tasks of the CN-Celeb Speaker Recognition Challenge 2022 (CNSRC 2022). This challenge includes two tasks, namely speaker verification(SV) and speaker retrieval(SR). The SV task involves two tracks: fixed track and open track. In the fixed track, we only used CN-Celeb.T as the training set. For the open track of the SV task and SR task, we added our open-source audio data. The ResNet-based, RepVGG-based, and TDNN-based architectures were developed for this challenge. Global statistic pooling structure and MQMHA pooling structure were used to aggregate the frame-level features across time to obtain utterance-level representation. We adopted AM-Softmax and AAM-Softmax combined with the Sub-Center method to classify the resulting embeddings. We also used the Large-Margin Fine-Tuning strategy to further improve the model performance. In the backend, Sub-Mean and AS-Norm were used. In the SV task fixed track, our system was a fusion of five models, and two models were fused in the SV task open track. And we used a single system in the SR task. Our approach leads to superior performance and comes the 1st place in the open track of the SV task, the 2nd place in the fixed track of the SV task, and the 3rd place in the SR task.
【7】 Controllable Accented Text-to-Speech Synthesis
标题：可控带重音的文语合成
链接：https://arxiv.org/abs/2209.10804
作者：Rui Liu,Berrak Sisman,Guanglai Gao,Haizhou Li机构：Guanglai Gao is with the Department of Computer Science, Inner MongoliaUniversity备注：To be submitted for possible journal publication摘要：带重音的文本到语音（TTS）合成寻求生成带有重音（L2）的语音作为标准版本（L1）的变体。由于L2在语音渲染和韵律模式方面都不同于L1，因此重音TTS合成具有挑战性。此外，没有简单的解决方案来控制话语中的重音强度。在这项工作中，我们提出了一个神经TTS架构，它允许我们在推理过程中控制重音及其强度。这是通过三个新颖的机制实现的，1）重音变化适配器用三个韵律控制因子，即音高、能量和持续时间来模拟复杂的重音变化; 2）重音强度建模策略，用于量化重音强度; 3）一致性约束模块，用于鼓励TTS系统以精细水平呈现预期重音强度。实验结果表明，该系统在重音渲染和亮度控制方面的性能优于基线模型。据我们所知，这是第一次研究重音TTS合成与显式强度控制。摘要：Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1). Accented TTS synthesis is challenging as L2 is different from L1 in both in terms of phonetic rendering and prosody pattern. Furthermore, there is no easy solution to the control of the accent intensity in an utterance. In this work, we propose a neural TTS architecture, that allows us to control the accent and its intensity during inference. This is achieved through three novel mechanisms, 1) an accent variance adaptor to model the complex accent variance with three prosody controlling factors, namely pitch, energy and duration; 2) an accent intensity modeling strategy to quantify the accent intensity; 3) a consistency constraint module to encourage the TTS system to render the expected accent intensity at a fine level. Experiments show that the proposed system attains superior performance to the baseline models in terms of accent rendering and intensity control. To our best knowledge, this is the first study of accented TTS synthesis with explicit intensity control.
【8】 Homophone Reveals the Truth: A Reality Check for Speech2Vec
标题：同音字揭示真相：Speech2Vec的现实检验
链接：https://arxiv.org/abs/2209.10791
机构：Renmin University of China摘要：生成具有语义信息的口语单词嵌入是一个令人感兴趣的课题。与基于文本的嵌入方式相比，它同时覆盖了语音和语义特征，能够提供更丰富的信息，对提高ASR和语音翻译系统具有潜在的帮助。在本文中，我们回顾并检验了这一领域一项开创性工作的真实性：语音2矢量首先，提出了一种基于同音异义词的检测方法，对Speech 2 Vec作者发布的语音嵌入进行检测。没有迹象表明这些嵌入由Speech 2 Vec模型生成。此外，通过对词汇构成的进一步分析，我们怀疑一个基于文本的模型制造了这些嵌入。最后，我们参考官方代码和原论文中的优化设置，重新生成了Speech 2 Vec模型。实验结果表明，该模型不能有效地学习语义嵌入。在单词相似度基准测试中，它在MEN测试中获得了0.08的相关性得分，在WS-353-SIM测试中获得了0.15的相关性得分，这比原论文中描述的相关性得分低了0.5以上。我们的数据和代码是可用的。摘要：Generating spoken word embeddings that possess semantic information is a fascinating topic. Compared with text-based embeddings, they cover both phonetic and semantic characteristics, which can provide richer information and are potentially helpful for improving ASR and speech translation systems. In this paper, we review and examine the authenticity of a seminal work in this field: Speech2Vec. First, a homophone-based inspection method is proposed to check the speech embeddings released by the author of Speech2Vec. There is no indication that these embeddings are generated by the Speech2Vec model. Moreover, through further analysis of the vocabulary composition, we suspect that a text-based model fabricates these embeddings. Finally, we reproduce the Speech2Vec model, referring to the official code and optimal settings in the original paper. Experiments showed that this model failed to learn effective semantic embeddings. In word similarity benchmarks, it gets a correlation score of 0.08 in MEN and 0.15 in WS-353-SIM tests, which is over 0.5 lower than those described in the original paper. Our data and code are available.
【9】 Modeling Perceptual Loudness of Piano Tone: Theory and Applications
标题：钢琴声调知觉响度模型的理论与应用
链接：https://arxiv.org/abs/2209.10674
作者：Yang Qu,Yutian Qin,Lecheng Chao,Hangkai Qian,Ziyu Wang,Gus Xia机构：Music X Lab, NYU Shanghai, City University of Hong Kong, New York University, Mohamed Bin Zayed University of Artificial Intelligence备注：Accepted to ISMIR 2022摘要：感知响度与声音物理属性之间的关系是计算机音乐和心理声学研究的重要课题。“等响线”的早期研究可以追溯到20世纪20年代，从那时起，测量的响度在强度和频率方面已经被多次修正。然而，大多数的研究仅仅集中在合成音上，而对于复杂音色的自然音调的归纳理论却很少得到证实。为此，本文通过对钢琴音色的建模，对自然音色响度感知的理论和应用进行了研究。理论部分包括：1）音高的钢琴音调等响轮廓的精确测量，以及2）能够纯粹基于在人类对象测量上训练的频谱特征来推断响度的机器学习模型。至于应用，我们将我们的理论应用到钢琴控制转移中，其中我们在两个不同的演奏者钢琴上（在不同的声学环境中）调整MIDI速度，以实现相同的感知效果。实验结果表明，理论响度模型和相应的性能控制传递算法均明显优于基线。摘要：The relationship between perceptual loudness and physical attributes of sound is an important subject in both computer music and psychoacoustics. Early studies of "equal-loudness contour" can trace back to the 1920s and the measured loudness with respect to intensity and frequency has been revised many times since then. However, most studies merely focus on synthesized sound, and the induced theories on natural tones with complex timbre have rarely been justified. To this end, we investigate both theory and applications of natural-tone loudness perception in this paper via modeling piano tone. The theory part contains: 1) an accurate measurement of piano-tone equal-loudness contour of pitches, and 2) a machine-learning model capable of inferring loudness purely based on spectral features trained on human subject measurements. As for the application, we apply our theory to piano control transfer, in which we adjust the MIDI velocities on two different player pianos (in different acoustic environments) to achieve the same perceptual effect. Experiments show that both our theoretical loudness modeling and the corresponding performance control transfer algorithm significantly outperform their baselines.
【1】 Cross-domain Voice Activity Detection with Self-Supervised Representations
标题：基于自监督表示的跨域语音活动检测
链接：https://arxiv.org/abs/2209.11061
作者：Sina Alisamir,Fabien Ringeval,Francois Portet机构：Atos & LIG, Grenoble, France, Grenoble INP, LIG, Univ. Grenoble Alpes, Inria, CNRS, François Portet摘要：语音活动检测（VAD）旨在检测音频信号上的语音片段，这是当今许多基于语音的应用所必需的第一步。当前最先进的方法集中于训练神经网络，该神经网络利用直接包含在声学中的特征，诸如梅尔滤波器组（MFB）。因此，这样的方法需要额外的归一化步骤以适应声学受到影响的新域，这可能仅仅是由于扬声器、麦克风或环境的改变。另外，该归一化步骤通常是相当基本的方法，其具有某些限制，诸如高度易受新域可用的数据量的影响。在这里，我们利用众包的共同语音（CV）语料库来表明基于自监督学习（SSL）的表示可以很好地适应不同的领域，因为它们是用跨多个领域的语音的上下文化表示来计算的。与基于手工制作的表示（MFB）和现成VAD的系统相比，SSL表示还实现了更好的结果，在跨域设置方面有了显著改进。摘要：Voice Activity Detection (VAD) aims at detecting speech segments on an audio signal, which is a necessary first step for many today's speech based applications. Current state-of-the-art methods focus on training a neural network exploiting features directly contained in the acoustics, such as Mel Filter Banks (MFBs). Such methods therefore require an extra normalisation step to adapt to a new domain where the acoustics is impacted, which can be simply due to a change of speaker, microphone, or environment. In addition, this normalisation step is usually a rather rudimentary method that has certain limitations, such as being highly susceptible to the amount of data available for the new domain. Here, we exploited the crowd-sourced Common Voice (CV) corpus to show that representations based on Self-Supervised Learning (SSL) can adapt well to different domains, because they are computed with contextualised representations of speech across multiple domains. SSL representations also achieve better results than systems based on hand-crafted representations (MFBs), and off-the-shelf VADs, with significant improvement in cross-domain settings.
【2】 EPIC TTS Models: Empirical Pruning Investigations Characterizing Text-To-Speech Models
标题：EPIC TTS模型：表征文本到语音模型的经验剪枝研究
链接：https://arxiv.org/abs/2209.10890
作者：Perry Lam,Huayun Zhang,Nancy F. Chen,Berrak Sisman机构：Singapore University of Technology and Design, Institute for Infocomm Research, ASTAR, Singapore摘要：已知神经模型是过度参数化的，并且最近的工作已经表明稀疏文本到语音（TTS）模型可以胜过密集模型。虽然在其他领域已经提出了大量的稀疏方法，但这些方法很少应用于TTS。在这项工作中，我们试图回答这样一个问题：在性能和模型复杂度方面，所选的稀疏技术有什么特点？我们比较了Tacotron2基线和应用五种技术的结果。然后，我们通过自然度、可懂度和韵律等因素来评估性能，同时报告模型大小和训练时间。作为对先前研究的补充，我们发现在训练之前或期间进行修剪可以实现与训练之后进行修剪类似的性能，并且可以更快地训练，而移除整个神经元比移除参数更严重地降低性能。据我们所知，这是第一个比较稀疏性范例在文本到语音合成的工作。摘要：Neural models are known to be over-parameterized, and recent work has shown that sparse text-to-speech (TTS) models can outperform dense models. Although a plethora of sparse methods has been proposed for other domains, such methods have rarely been applied in TTS. In this work, we seek to answer the question: what are the characteristics of selected sparse techniques on the performance and model complexity? We compare a Tacotron2 baseline and the results of applying five techniques. We then evaluate the performance via the factors of naturalness, intelligibility and prosody, while reporting model size and training time. Complementary to prior research, we find that pruning before or during training can achieve similar performance to pruning after training and can be trained much faster, while removing entire neurons degrades performance much more than removing parameters. To our best knowledge, this is the first work that compares sparsity paradigms in text-to-speech synthesis.
【3】 Assessing ASR Model Quality on Disordered Speech using BERTScore
标题：用BERTScore评估语音障碍的ASR模型质量
链接：https://arxiv.org/abs/2209.10591
作者：Jimmy Tobin,Qisheng Li,Subhashini Venugopalan,Katie Seaver,Richard Cave,Katrin Tomanek机构：Tomanek, Google Research, USA;, University of Washington, USA;, Leonard Florence Center for Living, USA;, MND Association, UK备注：Accepted to Interspeech 2022 Workshop on Speech for Social Good摘要：误字率（WER）是用于评估自动语音识别（ASR）模型质量的主要度量。研究表明，ASR模型在有言语障碍的说话者身上的WER往往比典型的英语说话者高得多。在如此高的错误率下，很难确定模型是否有用。本研究调查了BERTScore的使用，BERTScore是一种用于文本生成的评估指标，以提供一种更有信息的ASR模型质量和有用性的测量。将BERTScore和WER与言语语言病理学家手动注释的预测错误进行比较，以确定错误类型和评估。BERT分数与错误类型和评估的人为评估更相关。BERTScore尤其对保留意义的正字法变化（缩写和规范化错误）更为稳健。此外，使用有序logistic回归和Akaike信息标准（AIC）测量时，BERT评分比WER更适合错误评估。总体而言，我们的研究结果表明，当从实际角度评估ASR模型性能时，BERTScore可以补充WER，特别是对于可访问性应用，其中模型即使在比典型语音更低的准确度下也是有用的。摘要：Word Error Rate (WER) is the primary metric used to assess automatic speech recognition (ASR) model quality. It has been shown that ASR models tend to have much higher WER on speakers with speech impairments than typical English speakers. It is hard to determine if models can be be useful at such high error rates. This study investigates the use of BERTScore, an evaluation metric for text generation, to provide a more informative measure of ASR model quality and usefulness. Both BERTScore and WER were compared to prediction errors manually annotated by Speech Language Pathologists for error type and assessment. BERTScore was found to be more correlated with human assessment of error type and assessment. BERTScore was specifically more robust to orthographic changes (contraction and normalization errors) where meaning was preserved. Furthermore, BERTScore was a better fit of error assessment than WER, as measured using an ordinal logistic regression and the Akaike's Information Criterion (AIC). Overall, our findings suggest that BERTScore can complement WER when assessing ASR model performance from a practical perspective, especially for accessibility applications where models are useful even at lower accuracy than for typical speech.
【4】 CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement
标题：CMGAN：基于Conform的Metric-GaN单声道语音增强
链接：https://arxiv.org/abs/2209.11112
* 与cs.SD语音【1】为同一篇
作者：Sherif Abdulatif,Ruizhe Cao,Bin Yang机构：This limitation is usually reflected as artifacts in theThe authors are with the Institute of Signal Processing and Sys-tem Theory, University of Stuttgart备注：16 pages, 10 figures and 5 tables. arXiv admin note: text overlap with arXiv:2203.15149摘要：卷积增强Transformers（Conformer）最近被提出用于各种语音领域的应用，例如自动语音识别（ASR）和语音分离，因为它们可以捕获局部和全局依赖性。本文提出了一种基于保形器的度量生成对抗网络（CMGAN），用于时频域语音增强。该发生器使用两级构象异构器模块对幅度和复频谱图信息进行编码，以对时间和频率依赖性进行建模。解码器然后将估计解耦到幅度掩码解码器分支和复数细化分支中，幅度掩码解码器分支用于滤除不想要的失真，复数细化分支用于进一步改进幅度估计并隐含地增强相位信息。另外，我们包括度量鉴别器，以通过相对于相应的评估分数优化生成器来减轻度量失配。客观和主观评价结果表明，CMGAN在3种语音增强任务（去噪、去混响和超分辨率）中均能表现出比现有方法更好的性能。例如，对Voice Bank+DEMAND数据集的定量去噪分析表明，CMGAN优于各种先前的模型，具有一定余量，即，PESQ为3.41，SSNR为11.10 dB。摘要：Convolution-augmented transformers (Conformers) are recently proposed in various speech-domain applications, such as automatic speech recognition (ASR) and speech separation, as they can capture both local and global dependencies. In this paper, we propose a conformer-based metric generative adversarial network (CMGAN) for speech enhancement (SE) in the time-frequency (TF) domain. The generator encodes the magnitude and complex spectrogram information using two-stage conformer blocks to model both time and frequency dependencies. The decoder then decouples the estimation into a magnitude mask decoder branch to filter out unwanted distortions and a complex refinement branch to further improve the magnitude estimation and implicitly enhance the phase information. Additionally, we include a metric discriminator to alleviate metric mismatch by optimizing the generator with respect to a corresponding evaluation score. Objective and subjective evaluations illustrate that CMGAN is able to show superior performance compared to state-of-the-art methods in three speech enhancement tasks (denoising, dereverberation and super-resolution). For instance, quantitative denoising analysis on Voice Bank+DEMAND dataset indicates that CMGAN outperforms various previous models with a margin, i.e., PESQ of 3.41 and SSNR of 11.10 dB.
【5】 Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networks
标题：使用并行评级数据和反对称双神经网络预测TTS音频刺激之间的成对偏好
链接：https://arxiv.org/abs/2209.11003
* 与cs.SD语音【2】为同一篇
作者：Cassia Valentini-Botinhao,Manuel Sam Ribeiro,Oliver Watts,Korin Richmond,Gustav Eje Henter机构：The Centre for Speech Technology Research, University of Edinburgh, UK, SpeakUnique Ltd., UK, Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden摘要：主观听力测试结果的自动预测是一项具有挑战性的任务。即使听众的偏好是一致的，评级也可能因人而异。虽然以前的工作集中在预测听众对单个刺激的评分（平均意见得分），但我们集中在预测同一文本的两个语音刺激的主观偏好这一更简单的任务上。我们提出了一个基于反对称孪生神经网络的模型，该模型在成对的波形和它们对应的偏好得分上进行训练。我们研究了注意神经网络和递归神经网络，以解释一对刺激中的刺激不是时间对齐的事实。为了获得一个大的训练集，我们将听众的评分从MUSHRA测试转换为反映成对刺激中一个刺激比另一个刺激评分高的频率的值。具体而言，我们评估了从五年来进行的12次MUSHRA评估中获得的数据的性能，这些评估包含不同的TTS系统，这些系统是根据不同说话者的数据构建的。我们的结果优于最先进的模型训练预测MOS评分。摘要：Automatically predicting the outcome of subjective listening tests is a challenging task. Ratings may vary from person to person even if preferences are consistent across listeners. While previous work has focused on predicting listeners' ratings (mean opinion scores) of individual stimuli, we focus on the simpler task of predicting subjective preference given two speech stimuli for the same text. We propose a model based on anti-symmetric twin neural networks, trained on pairs of waveforms and their corresponding preference scores. We explore both attention and recurrent neural nets to account for the fact that stimuli in a pair are not time aligned. To obtain a large training set we convert listeners' ratings from MUSHRA tests to values that reflect how often one stimulus in the pair was rated higher than the other. Specifically, we evaluate performance on data obtained from twelve MUSHRA evaluations conducted over five years, containing different TTS systems, built from data of different speakers. Our results compare favourably to a state-of-the-art model trained to predict MOS scores.
【6】 Maths, Computation and Flamenco: overview and challenges
标题：数学、计算和弗拉门戈：概述和挑战
链接：https://arxiv.org/abs/2209.10970
* 与cs.SD语音【3】为同一篇
作者：José-Miguel Díaz-Báñez,Nadine Kroher摘要：弗拉门戈是一种来自西班牙南部的以表演为导向的艺术音乐流派，吸引了全球越来越多的爱好者。由于其即兴和表现的性质，其独特的音乐特征，以及事实上，这一流派是很大程度上没有文件，弗拉门戈提出了一些有趣的数学和计算的挑战。音乐信息检索（MIR）中的大多数现有方法是在流行音乐或古典音乐的背景下开发的，并且通常不能很好地推广到非西方音乐传统，特别是当潜在的音乐理论假设不适用于这些流派时。在最近十年中，已经定义了与弗拉门戈音乐的自动分析相关的许多计算问题，并且已经提出了解决各种音乐方面的若干方法。本文概述了弗拉门戈音乐计算分析的挑战，并概述了现有的方法。摘要：Flamenco is a rich performance-oriented art music genre from Southern Spain which attracts a growing community of aficionados around the globe. Due to its improvisational and expressive nature, its unique musical characteristics, and the fact that the genre is largely undocumented, flamenco poses a number of interesting mathematical and computational challenges. Most existing approaches in Musical Information Retrieval (MIR) were developed in the context of popular or classical music and do often not generalize well to non-Western music traditions, in particular when the underlying music theoretical assumptions do not hold for these genres. Over the recent decade, a number of computational problems related to the automatic analysis of flamenco music have been defined and several methods addressing a variety of musical aspects have been proposed. This paper provides an overview of the challenges which arise in the context of computational analysis of flamenco music and outlines an overview of existing approaches.
【7】 A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS
标题：一种用于高性能神经TTS的多级多码本VQ-VAE方法
链接：https://arxiv.org/abs/2209.10887
* 与cs.SD语音【4】为同一篇
作者：Haohan Guo,Fenglong Xie,Frank K. Soong,Xixin Wu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, ‡ Xiaohongshu Inc., Shanghai, China, †Microsoft Research Asia, Beijing, China摘要：我们提出了一种多阶段、多码本（MSMC）的高性能神经TTS合成方法。基于矢量量化变分自动编码器（VQ-VAE）的特征分析器对语音训练数据的Mel谱图进行多级渐进下采样，得到不同时间分辨率的MSMC表示（MSMCR），并分别用多个VQ码本进行量化。通过最小化重建均方误差（MSE）和“三元组损失”的组合损失，训练多阶段预测器以将输入文本序列渐进地映射到MSMCR。在合成中，神经声码器将预测的MSMCR转换成最终的语音波形。在一个16小时的英语TTS语料库上，由一位女性说话人对该方法进行了训练和测试。申报TTS的MOS评分为4.41，优于基线的MOS评分3.62。所提出的TTS的具有少得多的参数的紧凑版本仍然可以保持高MOS分数。消融研究表明，多级和多个码本都能有效地实现高TTS性能。摘要：We propose a Multi-Stage, Multi-Codebook (MSMC) approach to high-performance neural TTS synthesis. A vector-quantized, variational autoencoder (VQ-VAE) based feature analyzer is used to encode Mel spectrograms of speech training data by down-sampling progressively in multiple stages into MSMC Representations (MSMCRs) with different time resolutions, and quantizing them with multiple VQ codebooks, respectively. Multi-stage predictors are trained to map the input text sequence to MSMCRs progressively by minimizing a combined loss of the reconstruction Mean Square Error (MSE) and "triplet loss". In synthesis, the neural vocoder converts the predicted MSMCRs into final speech waveforms. The proposed approach is trained and tested with an English TTS database of 16 hours by a female speaker. The proposed TTS achieves an MOS score of 4.41, which outperforms the baseline with an MOS of 3.62. Compact versions of the proposed TTS with much less parameters can still preserve high MOS scores. Ablation studies show that both multiple stages and multiple codebooks are effective for achieving high TTS performance.
【8】 MnTTS: An Open-Source Mongolian Text-to-Speech Synthesis Dataset and Accompanied Baseline
标题：MnTTS：一个开源的蒙古语文语合成数据集和伴随基线
链接：https://arxiv.org/abs/2209.10848
* 与cs.SD语音【5】为同一篇
作者：Yifan Hu,Pengkai Yin,Rui Liu,Feilong Bao,Guanglai Gao机构：College of Computer Science, Inner Mongolia University, Hohhot, China备注：Accepted at the 2022 International Conference on Asian Language Processing (IALP2022)摘要：本文介绍了一个高质量的蒙古语文语转换（TTS）合成数据集，该数据集是一个开源的高质量数据集。该数据集名为MnTTS，由一位22岁的蒙古族职业女播音员所讲的大约8小时的转录音频记录组成。它是第一个公开的数据集，旨在促进蒙古语TTS在学术界和工业界的应用。本文通过介绍数据集的开发过程和面临的挑战，分享了我们的经验。为了验证我们的数据集的可靠性，我们基于FastSpeech 2模型和HiFi-GAN声码器构建了一个功能强大的非自回归基线系统，并使用主观平均意见得分（MOS）和实时因子（RTF）度量对其进行了评估。评估结果表明，在我们的数据集上训练的强大的基线系统达到了4以上的MOS和约3.30 × 10 ^{-1}$的RTF，这使它适合于实际使用。数据集、训练方法和预训练的TTS模型可免费获得。摘要：This paper introduces a high-quality open-source text-to-speech (TTS) synthesis dataset for Mongolian, a low-resource language spoken by over 10 million people worldwide. The dataset, named MnTTS, consists of about 8 hours of transcribed audio recordings spoken by a 22-year-old professional female Mongolian announcer. It is the first publicly available dataset developed to promote Mongolian TTS applications in both academia and industry. In this paper, we share our experience by describing the dataset development procedures and faced challenges. To demonstrate the reliability of our dataset, we built a powerful non-autoregressive baseline system based on FastSpeech2 model and HiFi-GAN vocoder, and evaluated it using the subjective mean opinion score (MOS) and real time factor (RTF) metrics. Evaluation results show that the powerful baseline system trained on our dataset achieves MOS above 4 and RTF about $3.30\times10^{-1}$, which makes it applicable for practical use. The dataset, training recipe, and pretrained TTS models are freely available \footnote{\label{github}\url{https://github.com/walker-hyf/MnTTS}}.
【9】 The SpeakIn System Description for CNSRC2022
标题：CNSRC2022的扬声器系统描述
链接：https://arxiv.org/abs/2209.10846
* 与cs.SD语音【6】为同一篇
作者：Yu Zheng,Yihao Chen,Jinghan Peng,Yajun Zhang,Min Liu,Minqiang Xu机构：SpeakIn Technologies Co. Ltd.摘要：本报告描述了我们为2022年CN-Celeb说话人识别挑战赛（CNSRC 2022）任务而开发的说话人验证系统。该挑战包括两个任务，即说话人确认（SV）和说话人检索（SR）。SV任务涉及两个轨道：固定轨道和开放轨道。在固定轨道中，我们仅使用CN-Celeb.T作为训练集。对于SV任务和SR任务的开放音轨，我们添加了我们的开源音频数据。基于ResNet、基于RepVGG和基于TDNN的架构就是为应对这一挑战而开发的。采用全局统计池结构和MQMHA池结构对帧级特征进行跨时间聚合，得到话语级特征表示。采用AM-Softmax和AAM-Softmax结合Sub-Center方法对嵌入结果进行分类。我们还使用了大裕度微调策略来进一步提高模型性能。在后端，使用次平均值和AS-范数。在SV任务固定航迹中，系统融合了5个模型;在SV任务开放航迹中，融合了2个模型。我们在SR任务中使用了单个系统。我们的方法在SV任务的开放轨道中获得了第一名，在SV任务的固定轨道中获得了第二名，在SR任务中获得了第三名。摘要：This report describes our speaker verification systems for the tasks of the CN-Celeb Speaker Recognition Challenge 2022 (CNSRC 2022). This challenge includes two tasks, namely speaker verification(SV) and speaker retrieval(SR). The SV task involves two tracks: fixed track and open track. In the fixed track, we only used CN-Celeb.T as the training set. For the open track of the SV task and SR task, we added our open-source audio data. The ResNet-based, RepVGG-based, and TDNN-based architectures were developed for this challenge. Global statistic pooling structure and MQMHA pooling structure were used to aggregate the frame-level features across time to obtain utterance-level representation. We adopted AM-Softmax and AAM-Softmax combined with the Sub-Center method to classify the resulting embeddings. We also used the Large-Margin Fine-Tuning strategy to further improve the model performance. In the backend, Sub-Mean and AS-Norm were used. In the SV task fixed track, our system was a fusion of five models, and two models were fused in the SV task open track. And we used a single system in the SR task. Our approach leads to superior performance and comes the 1st place in the open track of the SV task, the 2nd place in the fixed track of the SV task, and the 3rd place in the SR task.
【10】 Controllable Accented Text-to-Speech Synthesis
标题：可控带重音的文语合成
链接：https://arxiv.org/abs/2209.10804
* 与cs.SD语音【7】为同一篇
作者：Rui Liu,Berrak Sisman,Guanglai Gao,Haizhou Li机构：Guanglai Gao is with the Department of Computer Science, Inner MongoliaUniversity备注：To be submitted for possible journal publication摘要：带重音的文本到语音（TTS）合成寻求生成带有重音（L2）的语音作为标准版本（L1）的变体。由于L2在语音渲染和韵律模式方面都不同于L1，因此重音TTS合成具有挑战性。此外，没有简单的解决方案来控制话语中的重音强度。在这项工作中，我们提出了一个神经TTS架构，它允许我们在推理过程中控制重音及其强度。这是通过三个新颖的机制实现的，1）重音变化适配器用三个韵律控制因子，即音高、能量和持续时间来模拟复杂的重音变化; 2）重音强度建模策略，用于量化重音强度; 3）一致性约束模块，用于鼓励TTS系统以精细水平呈现预期重音强度。实验结果表明，该系统在重音渲染和亮度控制方面的性能优于基线模型。据我们所知，这是第一次研究重音TTS合成与显式强度控制。摘要：Accented text-to-speech (TTS) synthesis seeks to generate speech with an accent (L2) as a variant of the standard version (L1). Accented TTS synthesis is challenging as L2 is different from L1 in both in terms of phonetic rendering and prosody pattern. Furthermore, there is no easy solution to the control of the accent intensity in an utterance. In this work, we propose a neural TTS architecture, that allows us to control the accent and its intensity during inference. This is achieved through three novel mechanisms, 1) an accent variance adaptor to model the complex accent variance with three prosody controlling factors, namely pitch, energy and duration; 2) an accent intensity modeling strategy to quantify the accent intensity; 3) a consistency constraint module to encourage the TTS system to render the expected accent intensity at a fine level. Experiments show that the proposed system attains superior performance to the baseline models in terms of accent rendering and intensity control. To our best knowledge, this is the first study of accented TTS synthesis with explicit intensity control.
【11】 Homophone Reveals the Truth: A Reality Check for Speech2Vec
标题：同音字揭示真相：Speech2Vec的现实检验
链接：https://arxiv.org/abs/2209.10791
* 与cs.SD语音【8】为同一篇
机构：Renmin University of China摘要：生成具有语义信息的口语单词嵌入是一个令人感兴趣的课题。与基于文本的嵌入方式相比，它同时覆盖了语音和语义特征，能够提供更丰富的信息，对提高ASR和语音翻译系统具有潜在的帮助。在本文中，我们回顾并检验了这一领域一项开创性工作的真实性：语音2矢量首先，提出了一种基于同音异义词的检测方法，对Speech 2 Vec作者发布的语音嵌入进行检测。没有迹象表明这些嵌入由Speech 2 Vec模型生成。此外，通过对词汇构成的进一步分析，我们怀疑一个基于文本的模型制造了这些嵌入。最后，我们参考官方代码和原论文中的优化设置，重新生成了Speech 2 Vec模型。实验结果表明，该模型不能有效地学习语义嵌入。在单词相似度基准测试中，它在MEN测试中获得了0.08的相关性得分，在WS-353-SIM测试中获得了0.15的相关性得分，这比原论文中描述的相关性得分低了0.5以上。我们的数据和代码是可用的。摘要：Generating spoken word embeddings that possess semantic information is a fascinating topic. Compared with text-based embeddings, they cover both phonetic and semantic characteristics, which can provide richer information and are potentially helpful for improving ASR and speech translation systems. In this paper, we review and examine the authenticity of a seminal work in this field: Speech2Vec. First, a homophone-based inspection method is proposed to check the speech embeddings released by the author of Speech2Vec. There is no indication that these embeddings are generated by the Speech2Vec model. Moreover, through further analysis of the vocabulary composition, we suspect that a text-based model fabricates these embeddings. Finally, we reproduce the Speech2Vec model, referring to the official code and optimal settings in the original paper. Experiments showed that this model failed to learn effective semantic embeddings. In word similarity benchmarks, it gets a correlation score of 0.08 in MEN and 0.15 in WS-353-SIM tests, which is over 0.5 lower than those described in the original paper. Our data and code are available.
【12】 Modeling Perceptual Loudness of Piano Tone: Theory and Applications
标题：钢琴声调知觉响度模型的理论与应用
链接：https://arxiv.org/abs/2209.10674
* 与cs.SD语音【9】为同一篇
作者：Yang Qu,Yutian Qin,Lecheng Chao,Hangkai Qian,Ziyu Wang,Gus Xia机构：Music X Lab, NYU Shanghai, City University of Hong Kong, New York University, Mohamed Bin Zayed University of Artificial Intelligence备注：Accepted to ISMIR 2022摘要：感知响度与声音物理属性之间的关系是计算机音乐和心理声学研究的重要课题。“等响线”的早期研究可以追溯到20世纪20年代，从那时起，测量的响度在强度和频率方面已经被多次修正。然而，大多数的研究仅仅集中在合成音上，而对于复杂音色的自然音调的归纳理论却很少得到证实。为此，本文通过对钢琴音色的建模，对自然音色响度感知的理论和应用进行了研究。理论部分包括：1）音高的钢琴音调等响轮廓的精确测量，以及2）能够纯粹基于在人类对象测量上训练的频谱特征来推断响度的机器学习模型。至于应用，我们将我们的理论应用到钢琴控制转移中，其中我们在两个不同的演奏者钢琴上（在不同的声学环境中）调整MIDI速度，以实现相同的感知效果。实验结果表明，理论响度模型和相应的性能控制传递算法均明显优于基线。摘要：The relationship between perceptual loudness and physical attributes of sound is an important subject in both computer music and psychoacoustics. Early studies of "equal-loudness contour" can trace back to the 1920s and the measured loudness with respect to intensity and frequency has been revised many times since then. However, most studies merely focus on synthesized sound, and the induced theories on natural tones with complex timbre have rarely been justified. To this end, we investigate both theory and applications of natural-tone loudness perception in this paper via modeling piano tone. The theory part contains: 1) an accurate measurement of piano-tone equal-loudness contour of pitches, and 2) a machine-learning model capable of inferring loudness purely based on spectral features trained on human subject measurements. As for the application, we apply our theory to piano control transfer, in which we adjust the MIDI velocities on two different player pianos (in different acoustic environments) to achieve the same perceptual effect. Experiments show that both our theoretical loudness modeling and the corresponding performance control transfer algorithm significantly outperform their baselines.
机器翻译，仅供参考
每日语音速递