【1】 Controllable speech synthesis by learning discrete phoneme-level prosodic representations
标题：基于离散音素级韵律表征学习的可控语音合成
链接：https://arxiv.org/abs/2211.16307
作者：Nikolaos Ellinas,Myrsini Christidou,Alexandra Vioni,June Sig Sung,Aimilios Chalamandaris,Pirros Tsiakoulis,Paris Mastorocostas备注：Final published version available at: Speech Communication. arXiv admin note: substantial text overlap with arXiv:2111.10168摘要：本文提出了一种基于离散标签的音素级韵律控制方法。提出了一种无监督韵律聚类方法，用于对多说话人语音数据集的音素级F0和时长特征进行离散化.这些特征作为韵律标记的输入序列被馈送到韵律编码器模块，该韵律编码器模块增强基于自回归注意力的文本到语音模型。为了提高韵律控制的范围和覆盖率，我们采用了多种方法，如增强、F0归一化、时长均衡聚类和说话人无关聚类。最终的模型能够对训练集中包含的所有说话人进行细粒度的音素级韵律控制，同时保持说话人身份。我们引入了一种先验的韵律编码器，它学习每个说话者的风格，并且能够在不需要参考音频的情况下进行语音合成，而不是依赖于用于推断的参考话语。我们还利用有限的数据量对多说话人模型进行了微调，作为一个实际的应用场景，并显示了韵律控制能力得到了保持，验证了说话人无关韵律聚类的有效性。实验结果表明，该模型具有很高的输出语音质量，并且尽管多说话人设置引入了可变性，但所提出的方法允许在每个说话人的范围内进行有效的韵律控制。摘要：In this paper, we present a novel method for phoneme-level prosody control of F0 and duration using intuitive discrete labels. We propose an unsupervised prosodic clustering process which is used to discretize phoneme-level F0 and duration features from a multispeaker speech dataset. These features are fed as an input sequence of prosodic labels to a prosody encoder module which augments an autoregressive attention-based text-to-speech model. We utilize various methods in order to improve prosodic control range and coverage, such as augmentation, F0 normalization, balanced clustering for duration and speaker-independent clustering. The final model enables fine-grained phoneme-level prosody control for all speakers contained in the training set, while maintaining the speaker identity. Instead of relying on reference utterances for inference, we introduce a prior prosody encoder which learns the style of each speaker and enables speech synthesis without the requirement of reference audio. We also fine-tune the multispeaker model to unseen speakers with limited amounts of data, as a realistic application scenario and show that the prosody control capabilities are maintained, verifying that the speaker-independent prosodic clustering is effective. Experimental results show that the model has high output speech quality and that the proposed method allows efficient prosody control within each speaker's range despite the variability that a multispeaker setting introduces.
【2】 Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation
标题：神经传感器训练：通过逐个样本计算减少内存消耗
链接：https://arxiv.org/abs/2211.16270
作者：Stefan Braun,Erik McDermott,Roger Hsiao备注：5 pages, 4 figures, 1 table, 1 algorithm摘要：神经传感器是用于自动语音识别（ASR）的端到端模型。虽然该模型非常适合流式ASR，但训练过程仍然具有挑战性。在训练过程中，内存需求可能会很快超过最先进GPU的容量，从而限制批处理大小和序列长度。在这项工作中，我们分析了一个典型的换能器训练设置的时间和空间复杂性。我们提出了一种记忆有效的训练方法，该方法逐个样本地计算换能器损耗和梯度。我们提出了优化以提高逐样本方法的效率和并行性。在一组完整的基准测试中，我们表明，我们的样本方式大大减少了内存使用，并且与默认的批处理计算相比，执行速度具有竞争力。值得一提的是，我们仅使用6 GB内存，就成功计算了批量大小为1024、音频长度为40秒的换能器损耗和梯度。摘要：The neural transducer is an end-to-end model for automatic speech recognition (ASR). While the model is well-suited for streaming ASR, the training process remains challenging. During training, the memory requirements may quickly exceed the capacity of state-of-the-art GPUs, limiting batch size and sequence lengths. In this work, we analyze the time and space complexity of a typical transducer training setup. We propose a memory-efficient training method that computes the transducer loss and gradients sample by sample. We present optimizations to increase the efficiency and parallelism of the sample-wise method. In a set of thorough benchmarks, we show that our sample-wise method significantly reduces memory usage, and performs at competitive speed when compared to the default batched computation. As a highlight, we manage to compute the transducer loss and gradients for a batch size of 1024, and audio length of 40 seconds, using only 6 GB of memory.
【3】 Model Extraction Attack against Self-supervised Speech Models
标题：针对自监督语音模型的模型提取攻击
链接：https://arxiv.org/abs/2211.16044
作者：Tsu-Yuan Hsu,Chen-An Li,Tung-Yu Wu,Hung-yi Lee备注：Submitted to ICASSP 2023摘要：自监督学习（SSL）语音模型生成给定剪辑的有意义表示，并在各种下游任务中实现令人难以置信的性能。模型抽取攻击（MEA）通常指攻击者仅通过查询访问来窃取受害者模型的功能。本文主要研究了基于SSL语音模型的具有少量查询的MEA问题。我们提出了一个两阶段的框架来提取模型。在第一阶段，在大规模未标注语料库上进行SSL预训练一个小的语音模型。其次，从未标注语料库中主动抽取一小部分片段，并利用这些片段对目标模型进行查询，获取它们的表示作为标注，用于小模型的第二阶段训练。实验结果表明，本文提出的采样方法能够在不了解模型结构的情况下有效地提取目标模型。摘要：Self-supervised learning (SSL) speech models generate meaningful representations of given clips and achieve incredible performance across various downstream tasks. Model extraction attack (MEA) often refers to an adversary stealing the functionality of the victim model with only query access. In this work, we study the MEA problem against SSL speech model with a small number of queries. We propose a two-stage framework to extract the model. In the first stage, SSL is conducted on the large-scale unlabeled corpus to pre-train a small speech model. Secondly, we actively sample a small portion of clips from the unlabeled corpus and query the target model with these clips to acquire their representations as labels for the small model's second-stage training. Experiment results show that our sampling methods can effectively extract the target model without knowing any information about its model architecture.
【4】 Neural Speech Phase Prediction based on Parallel Estimation Architecture and Anti-Wrapping Losses
标题：基于并行估计结构和抗包裹损耗的神经语音相位预测
链接：https://arxiv.org/abs/2211.15974
备注：Submitted to ICASSP 2023摘要：本文提出了一种新的语音相位预测模型，该模型利用神经网络直接从幅度谱预测包裹相位谱。提出的模型是残差卷积网络和并行估计结构的级联。该并行估计结构由两个并行的线性卷积层和一个相位计算公式组成，模拟了从复谱的实部和虚部计算相位谱的过程，并将预测的相位值严格限制在主值区间内。为了避免相位卷绕引起的误差扩大问题，设计了反卷绕训练损耗，该损耗定义在预测的卷绕相位谱与自然相位谱之间，通过反卷绕函数激活瞬时相位误差、群时延误差和瞬时角频率误差。实验结果表明，本文提出的神经语音相位预测模型在重建语音质量和生成速度方面均优于迭代Griffin-Lim算法和其他基于神经网络的方法。摘要：This paper presents a novel speech phase prediction model which predicts wrapped phase spectra directly from amplitude spectra by neural networks. The proposed model is a cascade of a residual convolutional network and a parallel estimation architecture. The parallel estimation architecture is composed of two parallel linear convolutional layers and a phase calculation formula, imitating the process of calculating the phase spectra from the real and imaginary parts of complex spectra and strictly restricting the predicted phase values to the principal value interval. To avoid the error expansion issue caused by phase wrapping, we design anti-wrapping training losses defined between the predicted wrapped phase spectra and natural ones by activating the instantaneous phase error, group delay error and instantaneous angular frequency error using an anti-wrapping function. Experimental results show that our proposed neural speech phase prediction model outperforms the iterative Griffin-Lim algorithm and other neural network-based method, in terms of both reconstructed speech quality and generation speed.
【5】 Neural Vocoder Feature Estimation for Dry Singing Voice Separation
标题：用于干歌声分离的神经声码器特征估计
链接：https://arxiv.org/abs/2211.15948
作者：Jaekwon Im,Soonbeom Choi,Sangeon Yong,Juhan Nam摘要：歌唱声分离（SVS）是将歌唱声音频从其与乐器音频的混合中分离出来的任务。以往的SVS研究主要采用谱图掩模法，该方法在预测二值掩模时需要很大的维数。此外，他们还专注于提取保留了带有混响效果的湿音的声干。这一结果可能妨碍孤立的歌唱声音的可重用性。本文从混合音频中预测出干唱嗓音的Mel谱图作为神经声码器的特征，并利用神经声码器合成出歌唱声波形。我们试验了两种分离方法。一种是在Mel谱图域中预测二元掩模，另一种是直接预测Mel谱图。此外，我们增加了一个歌声检测器，以更明确地识别随时间变化的歌声片段。我们从音频、去混响、分离和整体质量方面测量了模型性能。实验结果表明，该模型在主观和客观评价上均优于现有的歌唱声分离模型，但在音质方面的评价效果不佳。摘要：Singing voice separation (SVS) is a task that separates singing voice audio from its mixture with instrumental audio. Previous SVS studies have mainly employed the spectrogram masking method which requires a large dimensionality in predicting the binary masks. In addition, they focused on extracting a vocal stem that retains the wet sound with the reverberation effect. This result may hinder the reusability of the isolated singing voice. This paper addresses the issues by predicting mel-spectrogram of dry singing voices from the mixed audio as neural vocoder features and synthesizing the singing voice waveforms from the neural vocoder. We experimented with two separation methods. One is predicting binary masks in the mel-spectrogram domain and the other is directly predicting the mel-spectrogram. Furthermore, we add a singing voice detector to identify the singing voice segments over time more explicitly. We measured the model performance in terms of audio, dereverberation, separation, and overall quality. The results show that our proposed model outperforms state-of-the-art singing voice separation models in both objective and subjective evaluation except the audio quality.
【6】 OK Computer Analysis: An Audio Corpus Study of Radiohead
标题：OK计算机分析：《电台司令》音频语料库研究
链接：https://arxiv.org/abs/2211.15834
摘要：The application of music information retrieval techniques in popular music studies has great promise. In the present work, a corpus of Radiohead songs across their career from 1992 to 2017 are subjected to automated audio analysis. We examine findings from a number of granularities and perspectives, including within song and between song examination of both timbral-rhythmic and harmonic features. Chronological changes include possible career spanning effects for a band's releases such as slowing tempi and reduced brightness, and the timbral markers of Radiohead's expanding approach to instrumental resources most identified with the Kid A and Amnesiac era. We conclude with a discussion highlighting some challenges for this approach, and the potential for a field of audio file based career analysis.
【7】 MuSFA: Improving Music Structural Function Analysis with Partially Labeled Data
标题：MuSFA：利用部分标注数据改进音乐结构函数分析
链接：https://arxiv.org/abs/2211.15787
作者：Ju-Chiang Wang,Jordan B. L. Smith,Yun-Ning Hung摘要：音乐结构分析（MSA）系统旨在将歌曲录音分割成具有有用标签的非重叠部分。先前的MSA系统通常在后处理步骤中预测抽象标签，并且需要歌曲的全部上下文。相比之下，我们最近提出了一个监督框架，称为“音乐结构功能分析”（MuSFA），它直接从音频中建模和预测有意义的标签，如“主歌”和“副歌”，而不需要歌曲的完整上下文。然而，该系统的性能取决于训练数据的数量和质量。本文提出对公共数据集HookTheory Lead Sheet Dataset（HLSD）进行再利用，以提高其性能。HLSD包含了超过18K的音乐片段摘录，这些片段最初是为研究自动旋律和声而收集的。我们将每个摘录视为部分标记的歌曲并提供标记映射，以便HLSD可以与其他公共数据集（如SALAMI、RWC和Isophonics）一起使用。在跨数据集评估中，我们发现在训练中包括HLSD可以将最先进的边界检测和切片标记评分分别提高约3%和约1%。摘要：Music structure analysis (MSA) systems aim to segment a song recording into non-overlapping sections with useful labels. Previous MSA systems typically predict abstract labels in a post-processing step and require the full context of the song. By contrast, we recently proposed a supervised framework, called "Music Structural Function Analysis" (MuSFA), that models and predicts meaningful labels like 'verse' and 'chorus' directly from audio, without requiring the full context of a song. However, the performance of this system depends on the amount and quality of training data. In this paper, we propose to repurpose a public dataset, HookTheory Lead Sheet Dataset (HLSD), to improve the performance. HLSD contains over 18K excerpts of music sections originally collected for studying automatic melody harmonization. We treat each excerpt as a partially labeled song and provide a label mapping, so that HLSD can be used together with other public datasets, such as SALAMI, RWC, and Isophonics. In cross-dataset evaluations, we find that including HLSD in training can improve state-of-the-art boundary detection and section labeling scores by ~3% and ~1% respectively.
【8】 Analysis of constant-Q filterbank based representations for speech emotion recognition
标题：基于恒定Q滤波器组的语音情感识别表示法分析
链接：https://arxiv.org/abs/2211.16363
作者：Premjeet Singh,Shefali Waldekar,Md Sahidullah,Goutam Saha摘要：分析了基于恒Q滤波器组的语音情感识别时频表示方法。恒定Q滤波器组提供非线性频谱-时间表示，在低频具有更高的频率分辨率。通过对短时Mel频谱系数（MFSC）和基于恒Q滤波器组的特征（即恒Q变换（CQT）和连续小波变换（CWT））的时域比较分析，发现恒Q表示在低频具有更高的时间不变性.这提供了针对音调中的情感不相关的时间变化的增强的鲁棒性，特别是对于低唤醒情感。在不同情感类别上的相应频域分析显示，基于恒定Q的时频表示比MFSC更好地分辨基音谐波。在四个公开可用的数据库上对特征进行的广泛评估中，SER性能进一步巩固了恒定Q表示的这些优势，其中六个高级深度神经网络架构作为后端分类器。我们在本研究中的推论暗示了恒定Q特征对于SER的适用性和潜力。摘要：This work analyzes the constant-Q filterbank-based time-frequency representations for speech emotion recognition (SER). Constant-Q filterbank provides non-linear spectro-temporal representation with higher frequency resolution at low frequencies. Our investigation reveals how the increased low-frequency resolution benefits SER. The time-domain comparative analysis between short-term mel-frequency spectral coefficients (MFSCs) and constant-Q filterbank-based features, namely constant-Q transform (CQT) and continuous wavelet transform (CWT), reveals that constant-Q representations provide higher time-invariance at low-frequencies. This provides increased robustness against emotion irrelevant temporal variations in pitch, especially for low-arousal emotions. The corresponding frequency-domain analysis over different emotion classes shows better resolution of pitch harmonics in constant-Q-based time-frequency representations than MFSC. These advantages of constant-Q representations are further consolidated by SER performance in the extensive evaluation of features over four publicly available databases with six advanced deep neural network architectures as the back-end classifiers. Our inferences in this study hint toward the suitability and potentiality of constant-Q features for SER.
【9】 Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
标题：码型转换自动语音识别的基准评价指标
链接：https://arxiv.org/abs/2211.16319
作者：Injy Hamed,Amir Hussein,Oumnia Chellah,Shammur Chowdhury,Hamdy Mubarak,Sunayana Sitaram,Nizar Habash,Ahmed Ali摘要：语码转换给多语言自动语音识别带来了许多挑战和机遇。本文主要研究鲁棒性和公平性评价指标问题。为此，我们开发了一个参考基准数据集的代码转换语音识别假设与人类的判断。我们为自动假设的最小编辑定义了明确的指导方针。我们使用4路注释器间协议来验证指南。我们根据与人类判断的相关性来评估大量指标。我们考虑的度量标准在表征（拼写、语音、语义）、直接性（内在与外在）、粒度（例如单词、字符）和相似性计算方法方面有所不同。与人类判断的最高相关性是使用音译，然后是文本规范化。我们发布了第一个可供人类接受的方言阿拉伯语/英语会话语音中的语码转换语音识别结果的语料库。摘要：Code-switching poses a number of challenges and opportunities for multilingual automatic speech recognition. In this paper, we focus on the question of robust and fair evaluation metrics. To that end, we develop a reference benchmark data set of code-switching speech recognition hypotheses with human judgments. We define clear guidelines for minimal editing of automatic hypotheses. We validate the guidelines using 4-way inter-annotator agreement. We evaluate a large number of metrics in terms of correlation with human judgments. The metrics we consider vary in terms of representation (orthographic, phonological, semantic), directness (intrinsic vs extrinsic), granularity (e.g. word, character), and similarity computation method. The highest correlation to human judgment is achieved using transliteration followed by text normalization. We release the first corpus for human acceptance of code-switching speech recognition results in dialectal Arabic/English conversation speech.
【10】 On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems
标题：多说话人语音识别系统的误词率定义及其有效计算
链接：https://arxiv.org/abs/2211.16112
作者：Thilo von Neumann,Christoph Boeddeker,Keisuke Kinoshita,Marc Delcroix,Reinhold Haeb-Umbach备注：Submitted to ICASSP 2023摘要：本文提出了一个计算ASR系统误字率的通用框架，该系统处理输入包含多个说话者的录音并产生多个输出字序列（MIMO）。典型地需要这样的ASR系统，用于会议记录。我们提供了一种基于多维Levenshtein距离张量中的动态规划搜索的有效实现，其约束是参考话语必须与一个假设输出一致地匹配。这也导致了ORC WER的有效实现，该ORC WER先前遭受指数复杂性。我们给出了多说话者场景中常用的WER定义的概述，并表明它们是针对特定应用场景调整的上述MIMO WER的专门化。最后，我们讨论了各种WER定义的优缺点，并建议何时使用。摘要：We present a general framework to compute the word error rate (WER) of ASR systems that process recordings containing multiple speakers at their input and that produce multiple output word sequences (MIMO). Such ASR systems are typically required, e.g., for meeting transcription. We provide an efficient implementation based on a dynamic programming search in a multi-dimensional Levenshtein distance tensor under the constraint that a reference utterance must be matched consistently with one hypothesis output. This also results in an efficient implementation of the ORC WER which previously suffered from exponential complexity. We give an overview of commonly used WER definitions for multi-speaker scenarios and show that they are specializations of the above MIMO WER tuned to particular application scenarios. We conclude with a discussion of the pros and cons of the various WER definitions and a recommendation when to use which.
【11】 Hiding speaker's sex in speech using zero-evidence speaker representation in an analysis/synthesis pipeline
标题：在分析/合成流水线中使用零证据说话人表示来隐藏说话人性别
链接：https://arxiv.org/abs/2211.16065
作者：Paul-Gauthier Noé,Xiaoxiao Miao,Xin Wang,Junichi Yamagishi,Jean-François Bonastre,Driss Matrouf备注：Submitted to ICASSP 2023, not peer-reviewed yet摘要：在分析/合成流水线中使用现代声码器使我们能够研究可用于隐私目的的高质量语音转换。在此，我们提出变换说话人嵌入和音高，以隐藏说话人的性别。基于ECAPA-TDNN的说话人表示输入到HiFiGAN声码器中，使用神经判别分析方法进行保护，这与隐私的零证据概念一致。这种方法显著地减少了语音中与说话者性别相关的信息，同时保留了语音内容和所得受保护话音中的一些一致性。摘要：The use of modern vocoders in an analysis/synthesis pipeline allows us to investigate high-quality voice conversion that can be used for privacy purposes. Here, we propose to transform the speaker embedding and the pitch in order to hide the sex of the speaker. ECAPA-TDNN-based speaker representation fed into a HiFiGAN vocoder is protected using a neural-discriminant analysis approach, which is consistent with the zero-evidence concept of privacy. This approach significantly reduces the information in speech related to the speaker's sex while preserving speech content and some consistency in the resulting protected voices.
【12】 Evaluating and reducing the distance between synthetic and real speech distributions
标题：评估和缩小合成语音分布与真实语音分布之间的距离
链接：https://arxiv.org/abs/2211.16049
作者：Christoph Minixhofer,Ondřej Klejch,Peter Bell备注：Submitted to ICASSP 2023摘要：虽然现代的文本到语音（TTS）系统可以产生在主观评价方面评价很高的语音，但是真实和合成语音分布之间的距离仍然未被充分研究，其中我们使用术语\textit{distribution}来表示来自给定说话者集合的所有可能的真实语音记录的样本空间;或者可以为同一组说话者生成的合成样本的数量。我们使用一系列语音处理度量和它们的分布的相应Wasserstein距离来评估真实和合成语音分布沿着声学环境、说话人特征和韵律的维度的距离。我们通过向模型提供从测量导出的话语级别信息来减小沿着所述维度的这些分布距离，并且示出它们可以在推理时间生成。通过评估作为训练数据的合成数据的适合度，对维度的改进转化为使用自动语音识别（ASR）近似的总体分布距离减小。摘要：While modern Text-to-Speech (TTS) systems can produce speech rated highly in terms of subjective evaluation, the distance between real and synthetic speech distributions remains understudied, where we use the term \textit{distribution} to mean the sample space of all possible real speech recordings from a given set of speakers; or of the synthetic samples that could be generated for the same set of speakers. We evaluate the distance of real and synthetic speech distributions along the dimensions of the acoustic environment, speaker characteristics and prosody using a range of speech processing measures and the respective Wasserstein distances of their distributions. We reduce these distribution distances along said dimensions by providing utterance-level information derived from the measures to the model and show they can be generated at inference time. The improvements to the dimensions translate to overall distribution distance reduction approximated using Automatic Speech Recognition (ASR) by evaluating the fitness of the synthetic data as training data.
【13】 jaCappella Corpus: A Japanese a Cappella Vocal Ensemble Corpus
标题：无伴奏合唱语料库：日语无伴奏合唱语料库
链接：https://arxiv.org/abs/2211.16028
作者：Tomohiko Nakamura,Shinnosuke Takamichi,Naoko Tanji,Satoru Fukayama,Hiroshi Saruwatari备注：Submitted to ICASSP2023摘要：我们构建了一个日语无伴奏合唱语料库（jaCappella corpus），用于合唱分离和合成。它由35首版权许可的声乐合奏歌曲和它们的个别声部的录音组成。这些歌曲改编自版权过期的日本儿童歌曲，有六个声部（主唱、女高音、女中音、男高音、男低音和声乐打击乐）。它们被分为七个子集，每个子集都具有一种音乐流派的典型特征，如爵士乐和恩卡。尽管传统声乐合奏数据集的主要目标是由女高音、女低音、男高音和男低音组成的合唱，但流派和声部的多样性与最近在YouTube等社交媒体服务中广泛传播的声乐合奏相匹配。实验结果表明，该语料库是一种具有挑战性的语音分离资源。我们的语料库可以在我们的项目页面上找到（https：//tomohikonakamura.github.io/jaCappella_corpus/）。摘要：We construct a corpus of Japanese a cappella vocal ensembles (jaCappella corpus) for vocal ensemble separation and synthesis. It consists of 35 copyright-cleared vocal ensemble songs and their audio recordings of individual voice parts. These songs were arranged from out-of-copyright Japanese children's songs and have six voice parts (lead vocal, soprano, alto, tenor, bass, and vocal percussion). They are divided into seven subsets, each of which features typical characteristics of a music genre such as jazz and enka. The variety in genre and voice part match vocal ensembles recently widespread in social media services such as YouTube, although the main targets of conventional vocal ensemble datasets are choral singing made up of soprano, alto, tenor, and bass. Experimental evaluation demonstrates that our corpus is a challenging resource for vocal ensemble separation. Our corpus is available on our project page (https://tomohikonakamura.github.io/jaCappella_corpus/).
【1】 Analysis of constant-Q filterbank based representations for speech emotion recognition
标题：基于恒定Q滤波器组的语音情感识别表示法分析
链接：https://arxiv.org/abs/2211.16363
* 与cs.SD语音【8】为同一篇
作者：Premjeet Singh,Shefali Waldekar,Md Sahidullah,Goutam Saha摘要：分析了基于恒Q滤波器组的语音情感识别时频表示方法。恒定Q滤波器组提供非线性频谱-时间表示，在低频具有更高的频率分辨率。通过对短时Mel频谱系数（MFSC）和基于恒Q滤波器组的特征（即恒Q变换（CQT）和连续小波变换（CWT））的时域比较分析，发现恒Q表示在低频具有更高的时间不变性.这提供了针对音调中的情感不相关的时间变化的增强的鲁棒性，特别是对于低唤醒情感。在不同情感类别上的相应频域分析显示，基于恒定Q的时频表示比MFSC更好地分辨基音谐波。在四个公开可用的数据库上对特征进行的广泛评估中，SER性能进一步巩固了恒定Q表示的这些优势，其中六个高级深度神经网络架构作为后端分类器。我们在本研究中的推论暗示了恒定Q特征对于SER的适用性和潜力。摘要：This work analyzes the constant-Q filterbank-based time-frequency representations for speech emotion recognition (SER). Constant-Q filterbank provides non-linear spectro-temporal representation with higher frequency resolution at low frequencies. Our investigation reveals how the increased low-frequency resolution benefits SER. The time-domain comparative analysis between short-term mel-frequency spectral coefficients (MFSCs) and constant-Q filterbank-based features, namely constant-Q transform (CQT) and continuous wavelet transform (CWT), reveals that constant-Q representations provide higher time-invariance at low-frequencies. This provides increased robustness against emotion irrelevant temporal variations in pitch, especially for low-arousal emotions. The corresponding frequency-domain analysis over different emotion classes shows better resolution of pitch harmonics in constant-Q-based time-frequency representations than MFSC. These advantages of constant-Q representations are further consolidated by SER performance in the extensive evaluation of features over four publicly available databases with six advanced deep neural network architectures as the back-end classifiers. Our inferences in this study hint toward the suitability and potentiality of constant-Q features for SER.
【2】 Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
标题：码型转换自动语音识别的基准评价指标
链接：https://arxiv.org/abs/2211.16319
* 与cs.SD语音【9】为同一篇
作者：Injy Hamed,Amir Hussein,Oumnia Chellah,Shammur Chowdhury,Hamdy Mubarak,Sunayana Sitaram,Nizar Habash,Ahmed Ali摘要：语码转换给多语言自动语音识别带来了许多挑战和机遇。本文主要研究鲁棒性和公平性评价指标问题。为此，我们开发了一个参考基准数据集的代码转换语音识别假设与人类的判断。我们为自动假设的最小编辑定义了明确的指导方针。我们使用4路注释器间协议来验证指南。我们根据与人类判断的相关性来评估大量指标。我们考虑的度量标准在表征（拼写、语音、语义）、直接性（内在与外在）、粒度（如单词、字符）和相似性计算方法方面有所不同。与人类判断的最高相关性是使用音译，然后是文本规范化。我们发布了第一个可供人类接受的方言阿拉伯语/英语会话语音中的语码转换语音识别结果的语料库。摘要：Code-switching poses a number of challenges and opportunities for multilingual automatic speech recognition. In this paper, we focus on the question of robust and fair evaluation metrics. To that end, we develop a reference benchmark data set of code-switching speech recognition hypotheses with human judgments. We define clear guidelines for minimal editing of automatic hypotheses. We validate the guidelines using 4-way inter-annotator agreement. We evaluate a large number of metrics in terms of correlation with human judgments. The metrics we consider vary in terms of representation (orthographic, phonological, semantic), directness (intrinsic vs extrinsic), granularity (e.g. word, character), and similarity computation method. The highest correlation to human judgment is achieved using transliteration followed by text normalization. We release the first corpus for human acceptance of code-switching speech recognition results in dialectal Arabic/English conversation speech.
【3】 On Word Error Rate Definitions and their Efficient Computation for Multi-Speaker Speech Recognition Systems
标题：多说话人语音识别系统的误词率定义及其有效计算
链接：https://arxiv.org/abs/2211.16112
* 与cs.SD语音【10】为同一篇
作者：Thilo von Neumann,Christoph Boeddeker,Keisuke Kinoshita,Marc Delcroix,Reinhold Haeb-Umbach备注：Submitted to ICASSP 2023摘要：本文提出了一个计算ASR系统误字率的通用框架，该系统处理输入包含多个说话者的录音并产生多个输出字序列（MIMO）。典型地需要这样的ASR系统，用于会议记录。我们提供了一种基于多维Levenshtein距离张量中的动态规划搜索的有效实现，其约束是参考话语必须与一个假设输出一致地匹配。这也导致了ORC WER的有效实现，该ORC WER先前遭受指数复杂性。我们给出了多说话者场景中常用的WER定义的概述，并表明它们是针对特定应用场景调整的上述MIMO WER的专门化。最后，我们讨论了各种WER定义的优缺点，并建议何时使用。摘要：We present a general framework to compute the word error rate (WER) of ASR systems that process recordings containing multiple speakers at their input and that produce multiple output word sequences (MIMO). Such ASR systems are typically required, e.g., for meeting transcription. We provide an efficient implementation based on a dynamic programming search in a multi-dimensional Levenshtein distance tensor under the constraint that a reference utterance must be matched consistently with one hypothesis output. This also results in an efficient implementation of the ORC WER which previously suffered from exponential complexity. We give an overview of commonly used WER definitions for multi-speaker scenarios and show that they are specializations of the above MIMO WER tuned to particular application scenarios. We conclude with a discussion of the pros and cons of the various WER definitions and a recommendation when to use which.
【4】 Hiding speaker's sex in speech using zero-evidence speaker representation in an analysis/synthesis pipeline
标题：在分析/合成流水线中使用零证据说话人表示来隐藏说话人性别
链接：https://arxiv.org/abs/2211.16065
* 与cs.SD语音【11】为同一篇
作者：Paul-Gauthier Noé,Xiaoxiao Miao,Xin Wang,Junichi Yamagishi,Jean-François Bonastre,Driss Matrouf备注：Submitted to ICASSP 2023, not peer-reviewed yet摘要：在分析/合成流水线中使用现代声码器使我们能够研究可用于隐私目的的高质量语音转换。在此，我们提出变换说话人嵌入和音高，以隐藏说话人的性别。基于ECAPA-TDNN的说话人表示输入到HiFiGAN声码器中，使用神经判别分析方法进行保护，这与隐私的零证据概念一致。这种方法显著地减少了语音中与说话者性别相关的信息，同时保留了语音内容和所得受保护话音中的一些一致性。摘要：The use of modern vocoders in an analysis/synthesis pipeline allows us to investigate high-quality voice conversion that can be used for privacy purposes. Here, we propose to transform the speaker embedding and the pitch in order to hide the sex of the speaker. ECAPA-TDNN-based speaker representation fed into a HiFiGAN vocoder is protected using a neural-discriminant analysis approach, which is consistent with the zero-evidence concept of privacy. This approach significantly reduces the information in speech related to the speaker's sex while preserving speech content and some consistency in the resulting protected voices.
【5】 Evaluating and reducing the distance between synthetic and real speech distributions
标题：评估和缩小合成语音分布与真实语音分布之间的距离
链接：https://arxiv.org/abs/2211.16049
* 与cs.SD语音【12】为同一篇
作者：Christoph Minixhofer,Ondřej Klejch,Peter Bell备注：Submitted to ICASSP 2023摘要：虽然现代的文本到语音（TTS）系统可以产生在主观评价方面评价很高的语音，但是真实和合成语音分布之间的距离仍然未被充分研究，其中我们使用术语\textit{distribution}来表示来自给定说话者集合的所有可能的真实语音记录的样本空间;或者可以为同一组说话者生成的合成样本的数量。我们使用一系列语音处理度量和它们的分布的相应Wasserstein距离来评估真实和合成语音分布沿着声学环境、说话人特征和韵律的维度的距离。我们通过向模型提供从测量导出的话语级别信息来减小沿着所述维度的这些分布距离，并且示出它们可以在推理时间生成。通过评估作为训练数据的合成数据的适合度，对维度的改进转化为使用自动语音识别（ASR）近似的总体分布距离减小。摘要：While modern Text-to-Speech (TTS) systems can produce speech rated highly in terms of subjective evaluation, the distance between real and synthetic speech distributions remains understudied, where we use the term \textit{distribution} to mean the sample space of all possible real speech recordings from a given set of speakers; or of the synthetic samples that could be generated for the same set of speakers. We evaluate the distance of real and synthetic speech distributions along the dimensions of the acoustic environment, speaker characteristics and prosody using a range of speech processing measures and the respective Wasserstein distances of their distributions. We reduce these distribution distances along said dimensions by providing utterance-level information derived from the measures to the model and show they can be generated at inference time. The improvements to the dimensions translate to overall distribution distance reduction approximated using Automatic Speech Recognition (ASR) by evaluating the fitness of the synthetic data as training data.
【6】 jaCappella Corpus: A Japanese a Cappella Vocal Ensemble Corpus
标题：无伴奏合唱语料库：日语无伴奏合唱语料库
链接：https://arxiv.org/abs/2211.16028
* 与cs.SD语音【13】为同一篇
作者：Tomohiko Nakamura,Shinnosuke Takamichi,Naoko Tanji,Satoru Fukayama,Hiroshi Saruwatari备注：Submitted to ICASSP2023摘要：我们构建了一个日语无伴奏合唱语料库（jaCappella corpus），用于合唱分离和合成。它由35首版权许可的声乐合奏歌曲和它们的个别声部的录音组成。这些歌曲改编自版权过期的日本儿童歌曲，有六个声部（主唱、女高音、女中音、男高音、男低音和声乐打击乐）。它们被分为七个子集，每个子集都具有一种音乐流派的典型特征，如爵士乐和恩卡。尽管传统声乐合奏数据集的主要目标是由女高音、女低音、男高音和男低音组成的合唱，但流派和声部的多样性与最近在YouTube等社交媒体服务中广泛传播的声乐合奏相匹配。实验结果表明，该语料库是一种具有挑战性的语音分离资源。我们的语料库可以在我们的项目页面上找到（https：//tomohikonakamura.github.io/jaCappella_corpus/）。摘要：We construct a corpus of Japanese a cappella vocal ensembles (jaCappella corpus) for vocal ensemble separation and synthesis. It consists of 35 copyright-cleared vocal ensemble songs and their audio recordings of individual voice parts. These songs were arranged from out-of-copyright Japanese children's songs and have six voice parts (lead vocal, soprano, alto, tenor, bass, and vocal percussion). They are divided into seven subsets, each of which features typical characteristics of a music genre such as jazz and enka. The variety in genre and voice part match vocal ensembles recently widespread in social media services such as YouTube, although the main targets of conventional vocal ensemble datasets are choral singing made up of soprano, alto, tenor, and bass. Experimental evaluation demonstrates that our corpus is a challenging resource for vocal ensemble separation. Our corpus is available on our project page (https://tomohikonakamura.github.io/jaCappella_corpus/).
【7】 Controllable speech synthesis by learning discrete phoneme-level prosodic representations
标题：基于离散音素级韵律表征学习的可控语音合成
链接：https://arxiv.org/abs/2211.16307
* 与cs.SD语音【1】为同一篇
作者：Nikolaos Ellinas,Myrsini Christidou,Alexandra Vioni,June Sig Sung,Aimilios Chalamandaris,Pirros Tsiakoulis,Paris Mastorocostas备注：Final published version available at: Speech Communication. arXiv admin note: substantial text overlap with arXiv:2111.10168摘要：本文提出了一种基于离散标签的音素级韵律控制方法。提出了一种无监督韵律聚类方法，用于对多说话人语音数据集的音素级F0和时长特征进行离散化.这些特征作为韵律标记的输入序列被馈送到韵律编码器模块，该韵律编码器模块增强基于自回归注意力的文本到语音模型。为了提高韵律控制的范围和覆盖率，我们采用了多种方法，如增强、F0归一化、时长均衡聚类和说话人无关聚类。最终的模型能够对训练集中包含的所有说话人进行细粒度的音素级韵律控制，同时保持说话人身份。我们引入了一种先验的韵律编码器，它学习每个说话者的风格，并且能够在不需要参考音频的情况下进行语音合成，而不是依赖于用于推断的参考话语。我们还利用有限的数据量对多说话人模型进行了微调，作为一个实际的应用场景，并显示了韵律控制能力得到了保持，验证了说话人无关韵律聚类的有效性。实验结果表明，该模型具有很高的输出语音质量，并且尽管多说话人设置引入了可变性，但所提出的方法允许在每个说话人的范围内进行有效的韵律控制。摘要：In this paper, we present a novel method for phoneme-level prosody control of F0 and duration using intuitive discrete labels. We propose an unsupervised prosodic clustering process which is used to discretize phoneme-level F0 and duration features from a multispeaker speech dataset. These features are fed as an input sequence of prosodic labels to a prosody encoder module which augments an autoregressive attention-based text-to-speech model. We utilize various methods in order to improve prosodic control range and coverage, such as augmentation, F0 normalization, balanced clustering for duration and speaker-independent clustering. The final model enables fine-grained phoneme-level prosody control for all speakers contained in the training set, while maintaining the speaker identity. Instead of relying on reference utterances for inference, we introduce a prior prosody encoder which learns the style of each speaker and enables speech synthesis without the requirement of reference audio. We also fine-tune the multispeaker model to unseen speakers with limited amounts of data, as a realistic application scenario and show that the prosody control capabilities are maintained, verifying that the speaker-independent prosodic clustering is effective. Experimental results show that the model has high output speech quality and that the proposed method allows efficient prosody control within each speaker's range despite the variability that a multispeaker setting introduces.
【8】 Neural Transducer Training: Reduced Memory Consumption with Sample-wise Computation
标题：神经传感器训练：通过逐个样本计算减少内存消耗
链接：https://arxiv.org/abs/2211.16270
* 与cs.SD语音【2】为同一篇
作者：Stefan Braun,Erik McDermott,Roger Hsiao备注：5 pages, 4 figures, 1 table, 1 algorithm摘要：神经传感器是用于自动语音识别（ASR）的端到端模型。虽然该模型非常适合流式ASR，但训练过程仍然具有挑战性。在训练期间，内存需求可能会很快超过最先进GPU的容量，从而限制批处理大小和序列长度。在这项工作中，我们分析了一个典型的换能器训练设置的时间和空间复杂性。我们提出了一种记忆有效的训练方法，该方法逐个样本地计算换能器损耗和梯度。我们提出了优化以提高逐样本方法的效率和并行性。在一组全面的基准测试中，我们显示了我们的样本方式显著减少了内存使用，并且与默认的批处理计算相比，执行速度具有竞争力。值得一提的是，我们仅使用6 GB内存，就成功计算了批量大小为1024、音频长度为40秒的换能器损耗和梯度。摘要：The neural transducer is an end-to-end model for automatic speech recognition (ASR). While the model is well-suited for streaming ASR, the training process remains challenging. During training, the memory requirements may quickly exceed the capacity of state-of-the-art GPUs, limiting batch size and sequence lengths. In this work, we analyze the time and space complexity of a typical transducer training setup. We propose a memory-efficient training method that computes the transducer loss and gradients sample by sample. We present optimizations to increase the efficiency and parallelism of the sample-wise method. In a set of thorough benchmarks, we show that our sample-wise method significantly reduces memory usage, and performs at competitive speed when compared to the default batched computation. As a highlight, we manage to compute the transducer loss and gradients for a batch size of 1024, and audio length of 40 seconds, using only 6 GB of memory.
【9】 Model Extraction Attack against Self-supervised Speech Models
标题：针对自监督语音模型的模型提取攻击
链接：https://arxiv.org/abs/2211.16044
* 与cs.SD语音【3】为同一篇
作者：Tsu-Yuan Hsu,Chen-An Li,Tung-Yu Wu,Hung-yi Lee备注：Submitted to ICASSP 2023摘要：自监督学习（SSL）语音模型生成给定剪辑的有意义表示，并在各种下游任务中实现令人难以置信的性能。模型抽取攻击（MEA）通常指攻击者仅通过查询访问来窃取受害者模型的功能。本文主要研究了基于SSL语音模型的具有少量查询的MEA问题。我们提出了一个两阶段的框架来提取模型。在第一阶段，在大规模未标注语料库上进行SSL预训练一个小的语音模型。其次，从未标注语料库中主动抽取一小部分片段，并利用这些片段对目标模型进行查询，获取它们的表示作为标注，用于小模型的第二阶段训练。实验结果表明，本文提出的采样方法能够在不了解模型结构的情况下有效地提取目标模型。摘要：Self-supervised learning (SSL) speech models generate meaningful representations of given clips and achieve incredible performance across various downstream tasks. Model extraction attack (MEA) often refers to an adversary stealing the functionality of the victim model with only query access. In this work, we study the MEA problem against SSL speech model with a small number of queries. We propose a two-stage framework to extract the model. In the first stage, SSL is conducted on the large-scale unlabeled corpus to pre-train a small speech model. Secondly, we actively sample a small portion of clips from the unlabeled corpus and query the target model with these clips to acquire their representations as labels for the small model's second-stage training. Experiment results show that our sampling methods can effectively extract the target model without knowing any information about its model architecture.
【10】 Neural Speech Phase Prediction based on Parallel Estimation Architecture and Anti-Wrapping Losses
标题：基于并行估计结构和抗包裹损耗的神经语音相位预测
链接：https://arxiv.org/abs/2211.15974
* 与cs.SD语音【4】为同一篇
备注：Submitted to ICASSP 2023摘要：本文提出了一种新的语音相位预测模型，该模型利用神经网络直接从幅度谱预测包裹相位谱。提出的模型是残差卷积网络和并行估计结构的级联。该并行估计结构由两个并行的线性卷积层和一个相位计算公式组成，模拟了从复谱的实部和虚部计算相位谱的过程，并将预测的相位值严格限制在主值区间内。为了避免相位卷绕引起的误差扩大问题，设计了反卷绕训练损耗，该损耗定义在预测的卷绕相位谱与自然相位谱之间，通过反卷绕函数激活瞬时相位误差、群时延误差和瞬时角频率误差。实验结果表明，本文提出的神经语音相位预测模型在重建语音质量和生成速度方面均优于迭代Griffin-Lim算法和其他基于神经网络的方法。摘要：This paper presents a novel speech phase prediction model which predicts wrapped phase spectra directly from amplitude spectra by neural networks. The proposed model is a cascade of a residual convolutional network and a parallel estimation architecture. The parallel estimation architecture is composed of two parallel linear convolutional layers and a phase calculation formula, imitating the process of calculating the phase spectra from the real and imaginary parts of complex spectra and strictly restricting the predicted phase values to the principal value interval. To avoid the error expansion issue caused by phase wrapping, we design anti-wrapping training losses defined between the predicted wrapped phase spectra and natural ones by activating the instantaneous phase error, group delay error and instantaneous angular frequency error using an anti-wrapping function. Experimental results show that our proposed neural speech phase prediction model outperforms the iterative Griffin-Lim algorithm and other neural network-based method, in terms of both reconstructed speech quality and generation speed.
【11】 Neural Vocoder Feature Estimation for Dry Singing Voice Separation
标题：用于干歌声分离的神经声码器特征估计
链接：https://arxiv.org/abs/2211.15948
* 与cs.SD语音【5】为同一篇
作者：Jaekwon Im,Soonbeom Choi,Sangeon Yong,Juhan Nam摘要：歌唱声分离（SVS）是将歌唱声音频从其与乐器音频的混合中分离出来的任务。以往的SVS研究主要采用谱图掩模法，该方法在预测二值掩模时需要很大的维数。此外，他们还专注于提取保留了带有混响效果的湿音的声干。这一结果可能妨碍孤立的歌唱声音的可重用性。本文从混合音频中预测出干声的Mel谱图作为神经声码器的特征，并利用神经声码器合成出歌唱声波形。我们试验了两种分离方法。一种是在Mel谱图域中预测二元掩模，另一种是直接预测Mel谱图。此外，我们增加了一个歌声检测器，以更明确地识别随时间变化的歌声片段。我们从音频、去混响、分离和整体质量方面测量了模型性能。实验结果表明，该模型在主观和客观评价上均优于现有的歌唱声分离模型，但在音质方面的评价效果不佳。摘要：Singing voice separation (SVS) is a task that separates singing voice audio from its mixture with instrumental audio. Previous SVS studies have mainly employed the spectrogram masking method which requires a large dimensionality in predicting the binary masks. In addition, they focused on extracting a vocal stem that retains the wet sound with the reverberation effect. This result may hinder the reusability of the isolated singing voice. This paper addresses the issues by predicting mel-spectrogram of dry singing voices from the mixed audio as neural vocoder features and synthesizing the singing voice waveforms from the neural vocoder. We experimented with two separation methods. One is predicting binary masks in the mel-spectrogram domain and the other is directly predicting the mel-spectrogram. Furthermore, we add a singing voice detector to identify the singing voice segments over time more explicitly. We measured the model performance in terms of audio, dereverberation, separation, and overall quality. The results show that our proposed model outperforms state-of-the-art singing voice separation models in both objective and subjective evaluation except the audio quality.
【12】 OK Computer Analysis: An Audio Corpus Study of Radiohead
标题：OK计算机分析：《电台司令》音频语料库研究
链接：https://arxiv.org/abs/2211.15834
* 与cs.SD语音【6】为同一篇
摘要：音乐信息检索技术在流行音乐研究中的应用前景广阔。在目前的工作中，电台司令歌曲的语料库在他们的职业生涯从1992年到2017年进行自动音频分析。我们从多个粒度和角度检查结果，包括在歌曲内和歌曲之间检查音色节奏和和声特征。年代变化包括可能的职业生涯跨越影响乐队的释放，如放慢节奏和减少亮度，以及音色标记的电台司令的扩大方法，以乐器资源最确定与儿童A和失忆症时代。最后，我们讨论了这种方法面临的一些挑战，以及基于音频文件的职业分析领域的潜力。摘要：The application of music information retrieval techniques in popular music studies has great promise. In the present work, a corpus of Radiohead songs across their career from 1992 to 2017 are subjected to automated audio analysis. We examine findings from a number of granularities and perspectives, including within song and between song examination of both timbral-rhythmic and harmonic features. Chronological changes include possible career spanning effects for a band's releases such as slowing tempi and reduced brightness, and the timbral markers of Radiohead's expanding approach to instrumental resources most identified with the Kid A and Amnesiac era. We conclude with a discussion highlighting some challenges for this approach, and the potential for a field of audio file based career analysis.
【13】 MuSFA: Improving Music Structural Function Analysis with Partially Labeled Data
标题：MuSFA：利用部分标注数据改进音乐结构函数分析
链接：https://arxiv.org/abs/2211.15787
* 与cs.SD语音【7】为同一篇
作者：Ju-Chiang Wang,Jordan B. L. Smith,Yun-Ning Hung摘要：音乐结构分析（MSA）系统旨在将歌曲录音分割成具有有用标签的非重叠部分。先前的MSA系统通常在后处理步骤中预测抽象标签，并且需要歌曲的全部上下文。相比之下，我们最近提出了一个监督框架，称为“音乐结构功能分析”（MuSFA），它直接从音频中建模和预测有意义的标签，如“主歌”和“副歌”，而不需要歌曲的完整上下文。然而，该系统的性能取决于训练数据的数量和质量。本文提出对公共数据集HookTheory Lead Sheet Dataset（HLSD）进行再利用，以提高其性能。HLSD包含了超过18K的音乐片段摘录，这些片段最初是为研究自动旋律和声而收集的。我们将每个摘录视为部分标记的歌曲并提供标记映射，以便HLSD可以与其他公共数据集（如SALAMI、RWC和Isophonics）一起使用。在跨数据集评估中，我们发现在训练中包括HLSD可以将最先进的边界检测和切片标记评分分别提高约3%和约1%。摘要：Music structure analysis (MSA) systems aim to segment a song recording into non-overlapping sections with useful labels. Previous MSA systems typically predict abstract labels in a post-processing step and require the full context of the song. By contrast, we recently proposed a supervised framework, called "Music Structural Function Analysis" (MuSFA), that models and predicts meaningful labels like 'verse' and 'chorus' directly from audio, without requiring the full context of a song. However, the performance of this system depends on the amount and quality of training data. In this paper, we propose to repurpose a public dataset, HookTheory Lead Sheet Dataset (HLSD), to improve the performance. HLSD contains over 18K excerpts of music sections originally collected for studying automatic melody harmonization. We treat each excerpt as a partially labeled song and provide a label mapping, so that HLSD can be used together with other public datasets, such as SALAMI, RWC, and Isophonics. In cross-dataset evaluations, we find that including HLSD in training can improve state-of-the-art boundary detection and section labeling scores by ~3% and ~1% respectively.
机器翻译，仅供参考
每日语音速递