【1】 Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis
标题：基于AR和基于流的先验网络预测表现性语音合成的音素级韵律潜伏期
链接：https://arxiv.org/abs/2211.01327
作者：Konstantinos Klapsas,Karolos Nikitaras,Nikolaos Ellinas,June Sig Sung,Inchul Hwang,Spyros Raptis,Aimilios Chalamandaris,Pirros Tsiakoulis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：表达性语音合成文献的大部分集中于学习语音信号的韵律表示，然后在推理期间通过先验分布来建模该韵律表示。在这篇文章中，我们比较了不同的先验结构在预测音素级韵律表示的任务中提取一个无监督的FVAE模型。我们使用主观和客观的度量来表明，基于先验网络的归一化流可以以质量的轻微下降为代价产生更有表达力的语音。此外，我们还表明，由于归一化流的性质，对于给定的文本，合成语音具有更高的可变性。我们还提出了一种动态VAE模型，与基于流的模型相比，该模型可以产生更高质量的语音，尽管表达性和可变性有所下降。摘要：A large part of the expressive speech synthesis literature focuses on learning prosodic representations of the speech signal which are then modeled by a prior distribution during inference. In this paper, we compare different prior architectures at the task of predicting phoneme level prosodic representations extracted with an unsupervised FVAE model. We use both subjective and objective metrics to show that normalizing flow based prior networks can result in more expressive speech at the cost of a slight drop in quality. Furthermore, we show that the synthesized speech has higher variability, for a given text, due to the nature of normalizing flows. We also propose a Dynamical VAE model, that can generate higher quality speech although with decreased expressiveness and variability compared to the flow based models.
【2】 Low-Resource Music Genre Classification with Advanced Neural Model Reprogramming标题：基于高级神经模型重编程的低资源音乐流派分类链接：https://arxiv.org/abs/2211.01317
作者：Yun-Ning Hung,Chao-Han Huck Yang,Pin-Yu Chen,Alexander Lerch机构： Georgia Institute of Technology, Atlanta, GA, USA, IBM Research, Yorktown Heights, NY, USA备注：Submitted to ICASSP 2023. Some experimental results were reduced due to the space limit. The implementation will be available at this https URL摘要：迁移学习（TL）方法在处理训练数据有限的任务时表现出良好的效果。然而，利用目标域数据对预先训练的神经网络进行微调通常需要相当大的存储器和计算资源。在这项工作中，我们介绍了一种基于神经模型重编程（NMR）概念的利用预训练模型进行低资源（音乐）分类的新方法。NMR的目的在于通过修改冻结的预训练模型的输入来将预训练模型从源域重新用于目标域。除了已知的、输入无关的重编程方法之外，我们提出了一种高级重编程范例：输入依赖型NMR，可提高对音乐音频等复杂输入数据的适应性。实验结果表明，在大规模数据集上预训练的神经网络模型可以成功地利用这种重编程方法进行音乐流派分类。在小的体裁分类数据集上，两种基于输入的NMR TL方法的性能优于基于微调的TL方法.摘要：Transfer learning (TL) approaches have shown promising results when handling tasks with limited training data. However, considerable memory and computational resources are often required for fine-tuning pre-trained neural networks with target domain data. In this work, we introduce a novel method for leveraging pre-trained models for low-resource (music) classification based on the concept of Neural Model Reprogramming (NMR). NMR aims at re-purposing a pre-trained model from a source domain to a target domain by modifying the input of a frozen pre-trained model. In addition to the known, input-independent, reprogramming method, we propose an advanced reprogramming paradigm: Input-dependent NMR, to increase adaptability to complex input data such as musical audio. Experimental results suggest that a neural model pre-trained on large-scale datasets can successfully perform music genre classification by using this reprogramming method. The two proposed Input-dependent NMR TL methods outperform fine-tuning-based TL methods on a small genre classification dataset.
【3】 A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognition标题：用于口述命令识别的声学建模的量子核学习方法链接：https://arxiv.org/abs/2211.01263
作者：Chao-Han Huck Yang,Bo Li,Yu Zhang,Nanxin Chen,Tara N. Sainath,Sabato Marco Siniscalchi,Chin-Hui Lee机构： Georgia Institute of Technology, USA, Google, USA, Kore University of Enna, Italy, Department of Electronic Systems, NTNU, Trondheim, Norway备注：Submitted to ICASSP 2023摘要：提出一种量子核学习（quantum kernel learning，QKL）框架，以解决低资源环境下训练大规模声学模型时经常遇到的固有数据稀疏问题.我们基于经典到量子特征编码来投影声学特征。与现有的量子卷积技术不同，我们利用量子空间中的特征QKL来设计基于核的分类器。在阿拉伯语、格鲁吉亚语、楚瓦什语和立陶宛语等低资源语言的语音命令识别任务上的实验结果表明，所提出的基于QKL的混合方法较现有的经典方法和量子方法有很好的改进.摘要：We propose a quantum kernel learning (QKL) framework to address the inherent data sparsity issues often encountered in training large-scare acoustic models in low-resource scenarios. We project acoustic features based on classical-to-quantum feature encoding. Different from existing quantum convolution techniques, we utilize QKL with features in the quantum space to design kernel-based classifiers. Experimental results on challenging spoken command recognition tasks for a few low-resource languages, such as Arabic, Georgian, Chuvash, and Lithuanian, show that the proposed QKL-based hybrid approach attains good improvements over existing classical and quantum solutions.
【4】 Audio Language Modeling using Perceptually-Guided Discrete Representations标题：基于感知引导离散表示的音频语言建模链接：https://arxiv.org/abs/2211.01223
作者：Felix Kreuk,Yaniv Taigman,Adam Polyak,Jade Copet,Gabriel Synnaeve,Alexandre Défossez,Yossi Adi机构：Meta AI Research, The Hebrew University of Jerusalem, Israel摘要：在这项工作中，我们研究了音频语言建模的任务，其中我们的目标是学习音频的概率模型，可以用于生成和完成。我们使用最先进的感知引导音频压缩模型，将音频编码为离散表示。接下来，我们使用这些表示来训练基于转换器的因果语言模型。在推理时，我们通过将音频提示编码为离散序列、将其馈送到音频语言模型、从该模型采样并合成相应的时域信号来执行音频自动完成。我们在Audioset（迄今为止最大的通用音频数据集）上评估了我们的方法生成的样本质量，结果表明它优于所评估的基准音频编码器。此外，我们还提供了广泛的分析，以更好地理解音频质量和语言建模能力之间的权衡。样品：链接。摘要：In this work, we study the task of Audio Language Modeling, in which we aim at learning probabilistic models for audio that can be used for generation and completion. We use a state-of-the-art perceptually-guided audio compression model, to encode audio to discrete representations. Next, we train a transformer-based causal language model using these representations. At inference time, we perform audio auto-completion by encoding an audio prompt as a discrete sequence, feeding it to the audio language model, sampling from the model, and synthesizing the corresponding time-domain signal. We evaluate the quality of samples generated by our method on Audioset, the largest dataset for general audio to date, and show that it is superior to the evaluated baseline audio encoders. We additionally provide an extensive analysis to better understand the trade-off between audio-quality and language-modeling capabilities. Samples:link.
【5】 Transformer-based encoder-encoder architecture for Spoken Term Detection标题：一种基于Transformer的语音检测编码器体系结构链接：https://arxiv.org/abs/2211.01089
作者：Jan Švec,Luboš Šmídl,Jan Lehečka机构： Jan LeheˇckaDepartment of Cybernetics, University of West Bohemia备注：Submitted to ICASSP 2023摘要：提出了一种基于Transformer架构的口语术语检测方法。我们提出了编码器-编码器架构，采用两个类BERT编码器，并进行了额外的修改，包括卷积和上采样层、注意力掩蔽和共享参数。编码器将识别的假设和搜索的项投影到共享嵌入空间中，其中使用校准的点积来计算推定命中的得分。在实验中，我们使用了Wav 2 Vec 2.0语音识别器，在基于USC Shoah Foundation Visual History Archive（MALACH）的英语和捷克语STD数据集上，我们提出的系统优于基于深度LSTM的基线方法。摘要：The paper presents a method for spoken term detection based on the Transformer architecture. We propose the encoder-encoder architecture employing two BERT-like encoders with additional modifications, including convolutional and upsampling layers, attention masking, and shared parameters. The encoders project a recognized hypothesis and a searched term into a shared embedding space, where the score of the putative hit is computed using the calibrated dot product. In the experiments, we used the Wav2Vec 2.0 speech recognizer, and the proposed system outperformed a baseline method based on deep LSTMs on the English and Czech STD datasets based on USC Shoah Foundation Visual History Archive (MALACH).
【6】 DSPGAN: a GAN-based universal vocoder for high-fidelity TTS by time-frequency domain supervision from DSP标题：DSPGAN：一种基于GAN的时频域监控高保真TTS通用声码器链接：https://arxiv.org/abs/2211.01087
作者：Kun Song,Yongmao Zhang,Yi Lei,Jian Cong,Hanzhao Li,Lei Xie,Gang He,Jinfeng Bai机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, TAL Education Group, Beijing, China备注：Submitted to ICASSP 2023摘要：近年来，基于生成式对抗神经网络（GAN）的神经声码器的发展显示出其以Mel谱图为条件产生原始波形的优势，具有快速的推理速度和轻量级的网络结构。然而，训练一个通用的神经声码器仍然具有挑战性，该声码器可以合成来自各种场景的高保真语音，这些场景具有不可见的说话者、语言和说话风格。本文提出了一种基于GAN的高保真语音合成通用声码器DSPGAN，它利用数字信号处理（DSP）的时频域监督功能。为了消除训练阶段的真实谱图与推理阶段的预测谱图之间的不匹配问题，采用从DSP模块产生的波形中提取的Mel谱图，而不是从TTS声学模型中预测的Mel谱图，作为基于GAN的声码器的时频域监督。我们也利用正弦激励作为时域监督，以改善谐波建模，并消除各种人为因素的GAN声码器。实验结果表明，DSPGAN的性能明显优于对比方法，能够基于TTS中的各种数据生成高保真语音。摘要：Recent development of neural vocoders based on the generative adversarial neural network (GAN) has shown their advantages of generating raw waveform conditioned on mel-spectrogram with fast inference speed and lightweight networks. Whereas, it is still challenging to train a universal neural vocoder that can synthesize high-fidelity speech from various scenarios with unseen speakers, languages, and speaking styles. In this paper, we propose DSPGAN, a GAN-based universal vocoder for high-fidelity speech synthesis by applying the time-frequency domain supervision from digital signal processing (DSP). To eliminate the mismatch problem caused by the ground-truth spectrograms in training phase and the predicted spectrograms in inference phase, we leverage the mel-spectrogram extracted from the waveform generated by a DSP module, rather than the predicted mel-spectrogram from the Text-to-Speech (TTS) acoustic model, as the time-frequency domain supervision to the GAN-based vocoder. We also utilize sine excitation as the time-domain supervision to improve the harmonic modeling and eliminate various artifacts of the GAN-based vocoder. Experimental results show that DSPGAN significantly outperforms the compared approaches and can generate high-fidelity speech based on diverse data in TTS.
【7】 Intermediate Fine-Tuning Using Imperfect Synthetic Speech for Improving Electrolaryngeal Speech Recognition标题：改进喉电语音识别的不完全合成语音中级微调链接：https://arxiv.org/abs/2211.01079
作者：Lester Phillip Violeta,Ding Ma,Wen-Chin Huang,Tomoki Toda机构：Nagoya University, Japan备注：Submitted to ICASSP 2023摘要：由于数据集较小，用于电子喉式扬声器的自动语音识别（ASR）系统的研究相对未被探索。当ASR缺乏训练数据时，大规模的预训练和微调框架通常足以实现高识别率;然而在喉电语音中，预训练和微调数据之间的域移位太大而无法克服，从而限制了识别率的最大改进。为了解决这个问题，我们提出了一个中间微调步骤，它使用不完美的合成语音来关闭预训练和目标数据之间的域移位间隙。尽管有不完美的合成数据，但我们在喉电语音数据集上显示了这种方法的有效性，与未使用不完美合成语音的基线相比，改善了6.1%。结果显示，中间微调阶段如何集中于学习不完美的合成数据的高级固有特征，而不是诸如可懂度的低级特征。摘要：Research on automatic speech recognition (ASR) systems for electrolaryngeal speakers has been relatively unexplored due to small datasets. When training data is lacking in ASR, a large-scale pretraining and fine tuning framework is often sufficient to achieve high recognition rates; however, in electrolaryngeal speech, the domain shift between the pretraining and fine-tuning data is too large to overcome, limiting the maximum improvement of recognition rates. To resolve this, we propose an intermediate fine-tuning step that uses imperfect synthetic speech to close the domain shift gap between the pretraining and target data. Despite the imperfect synthetic data, we show the effectiveness of this on electrolaryngeal speech datasets, with improvements of 6.1% over the baseline that did not use imperfect synthetic speech. Results show how the intermediate fine-tuning stage focuses on learning the high-level inherent features of the imperfect synthetic data rather than the low-level features such as intelligibility.
【8】 Singing Voice Synthesis with Vibrato Modeling and Latent Energy Representation标题：基于颤音建模和势能表示的歌声合成链接：https://arxiv.org/abs/2211.00996
作者：Yingjie Song,Wei Song,Wei Zhang,Zhengchen Zhang,Dan Zeng,Zhi Liu,Yang Yu机构： School of Communication and Information Engineering, Shanghai University, Shanghai, China, JD AI Research, Beijing, China, Shanghai Conservatory of Music, Shanghai, China摘要：通过引入显式颤音建模和潜在能量表示，提出了一个富有表现力的歌唱声合成系统。由于人类歌唱的固有特性，颤音对于合成声音的自然性至关重要。为此，本文提出了一种基于深度学习的颤音模型，用于控制歌唱中颤音的似然度、速率、深度和相位，其中颤音似然度表示颤音存在的概率，有助于提高歌唱声音的自然度。事实上，现有的歌唱语料库中并没有关于颤音似然性的标注。采用一种新的颤音相似度标注方法对颤音相似度进行自动标注。同时，音频的功率谱图包含丰富的信息，可以提高歌唱的表现力。提出了一种基于自动编码器的潜在能量瓶颈特征，用于表达性歌唱语音合成。在开放数据集NUS48E上的实验结果表明，颤音建模和潜在能量表示都能显著提高歌唱声音的表现力。演示网站中显示了音频样本。摘要：This paper proposes an expressive singing voice synthesis system by introducing explicit vibrato modeling and latent energy representation. Vibrato is essential to the naturalness of synthesized sound, due to the inherent characteristics of human singing. Hence, a deep learning-based vibrato model is introduced in this paper to control the vibrato's likeliness, rate, depth and phase in singing, where the vibrato likeliness represents the existence probability of vibrato and it would help improve the singing voice's naturalness. Actually, there is no annotated label about vibrato likeliness in existing singing corpus. We adopt a novel vibrato likeliness labeling method to label the vibrato likeliness automatically. Meanwhile, the power spectrogram of audio contains rich information that can improve the expressiveness of singing. An autoencoder-based latent energy bottleneck feature is proposed for expressive singing voice synthesis. Experimental results on the open dataset NUS48E show that both the vibrato modeling and the latent energy representation could significantly improve the expressiveness of singing voice. The audio samples are shown in the demo website.
【9】 Weighted variance variational autoencoder for speech enhancement标题：用于语音增强的加权方差变分自动编码器链接：https://arxiv.org/abs/2211.00990
作者：Ali Golmakani,Mostafa Sadeghi,Xavier Alameda-Pineda,Romain Serizel机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, Inria Grenoble & Univ. Grenoble Alpes, France摘要：本文提出了一种基于变分自编码器的语音增强方法，它包括在时频域中学习语音先验分布。对于生成模型，通常假设零均值复值高斯分布，其中语音信息被编码在作为潜在变量的函数的方差中。虽然这是常用的方法，但本文提出了一种加权方差生成模型，其中每个TF点在参数学习中的贡献是加权的。我们在权重上施加Gamma先验分布，这将有效地导致Student的t分布而不是用于语音建模的高斯分布。我们基于所提出的生成模型开发了高效的训练和语音增强算法。在声谱图建模和语音增强实验中，与标准的未加权方差模型相比，该方法具有较好的鲁棒性和有效性。摘要：We address speech enhancement based on variational autoencoders, which involves learning a speech prior distribution in the time-frequency (TF) domain. A zero-mean complexvalued Gaussian distribution is usually assumed for the generative model, where the speech information is encoded in the variance as a function of a latent variable. While this is the commonly used approach, in this paper we propose a weighted variance generative model, where the contribution of each TF point in parameter learning is weighted. We impose a Gamma prior distribution on the weights, which would effectively lead to a Student's t-distribution instead of Gaussian for speech modeling. We develop efficient training and speech enhancement algorithms based on the proposed generative model. Our experimental results on spectrogram modeling and speech enhancement demonstrate the effectiveness and robustness of the proposed approach compared to the standard unweighted variance model.
【10】 Audio-visual speech enhancement with a deep Kalman filter generative model标题：基于深度卡尔曼滤波生成模型的视听语音增强链接：https://arxiv.org/abs/2211.00988
作者：Ali Golmakani,Mostafa Sadeghi,Romain Serizel机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：基于变分自动编码器（VAE）的深隐变量生成模型在视听语音增强（AVSE）中表现出良好的性能。其基本思想是学习干净语音数据的基于VAE的视听先验分布，然后将其与统计噪声模型相结合，以从目标说话人的噪声音频记录和视频（唇图像）中恢复语音信号。现有的针对AVSE开发的生成模型没有考虑语音数据的顺序特性，这阻止了它们完全结合视觉数据的能力。本文提出了一种视听深度卡尔曼滤波（AV-DKF）生成模型，该模型假设隐变量为一阶马尔可夫链模型，并有效地融合视听数据。此外，我们开发了一种有效的推理方法来估计测试时间的语音信号。我们进行了一组实验来比较用于语音增强的生成模型的不同变体。实验结果表明，AV-DKF模型与纯音频模型、非序列纯音频模型和基于视听VAE的模型相比具有更好的性能。摘要：Deep latent variable generative models based on variational autoencoder (VAE) have shown promising performance for audiovisual speech enhancement (AVSE). The underlying idea is to learn a VAEbased audiovisual prior distribution for clean speech data, and then combine it with a statistical noise model to recover a speech signal from a noisy audio recording and video (lip images) of the target speaker. Existing generative models developed for AVSE do not take into account the sequential nature of speech data, which prevents them from fully incorporating the power of visual data. In this paper, we present an audiovisual deep Kalman filter (AV-DKF) generative model which assumes a first-order Markov chain model for the latent variables and effectively fuses audiovisual data. Moreover, we develop an efficient inference methodology to estimate speech signals at test time. We conduct a set of experiments to compare different variants of generative models for speech enhancement. The results demonstrate the superiority of the AV-DKF model compared with both its audio-only version and the non-sequential audio-only and audiovisual VAE-based models.
【11】 SpectroMap: Peak detection algorithm for audio fingerprinting标题：SpectrMap：音频指纹的峰值检测算法链接：https://arxiv.org/abs/2211.00982
作者：Aarón López-García机构： Peak detection algorithm for audio fingerprintingAar´on L´opez-Garc´ıaDepartment of Computer ScienceUniversity of Valencialogara8备注：7 pages, 3 figures摘要：我们介绍了SpectroMap，一个用Python编程语言编写的开源GitHub音频指纹库。它由一个峰值搜索算法组成，该算法通过时-频带从谱图中提取拓扑突出。本文通过在高质量城市声音数据集和环境音频记录中的两个实验应用，介绍了该算法的工作原理以及它在处理输入数据方面的有效性。摘要：We present SpectroMap, an open source GitHub repository for audio fingerprinting written in Python programming language. It is composed of a peak search algorithm that extracts topological prominences from a spectrogram via time-frequency bands. In this paper, we introduce the algorithm functioning with two experimental applications in a high-quality urban sound dataset and environmental audio recordings to describe how it works and how effective it is in handling the input data.
【12】 Internal Language Model Estimation based Adaptive Language Model Fusion for Domain Adaptation标题：基于内部语言模型估计的领域自适应语言模型融合链接：https://arxiv.org/abs/2211.00968
作者：Rao Ma,Xiaobo Wu,Jin Qiu,Yanan Qin,Haihua Xu,Peihao Wu,Zejun Ma机构：ByteDance AI Lab备注：submitted to ICASSP 2023摘要：ASR模型的部署环境是不断变化的，并且在会话期间传入的语音可以跨不同的域切换。这给只有目标领域文本数据时的有效领域自适应带来了挑战，我们的目标是在目标领域上获得明显改进的性能，同时在一般领域上的性能受到较小的损害。本文提出了一种基于内部语言模型估计的自适应域自适应（ILME-ADA）LM融合方法。为了实现这样的ILME-ADA，基于分别来自内部LM和外部LM（ELM）的得分的最大值来计算插值的对数似然性得分。在两个特定领域（目标）测试集上，我们分别使用神经网络和n-gram LM作为ELM，使用RNN-T和LAS建模框架验证了所提出的ILME-ADA方法的有效性。与浅层LM融合方法和基于ILME的LM融合方法相比，该方法在目标测试集上的性能明显提高，而在一般测试集上的性能下降最小。摘要：ASR model deployment environment is ever-changing, and the incoming speech can be switched across different domains during a session. This brings a challenge for effective domain adaptation when only target domain text data is available, and our objective is to obtain obviously improved performance on the target domain while the performance on the general domain is less undermined. In this paper, we propose an adaptive LM fusion approach called internal language model estimation based adaptive domain adaptation (ILME-ADA). To realize such an ILME-ADA, an interpolated log-likelihood score is calculated based on the maximum of the scores from the internal LM and the external LM (ELM) respectively. We demonstrate the efficacy of the proposed ILME-ADA method with both RNN-T and LAS modeling frameworks employing neural network and n-gram LMs as ELMs respectively on two domain specific (target) test sets. The proposed method can achieve significantly better performance on the target test sets while it gets minimal performance degradation on the general test set, compared with both shallow and ILME-based LM fusion methods.
【13】 Multi-Speaker Multi-Style Speech Synthesis with Timbre and Style Disentanglement标题：具有音色和风格解缠的多说话人多风格语音合成链接：https://arxiv.org/abs/2211.00967
作者：Wei Song,Yanghao Yue,Ya-jie Zhang,Zhengchen Zhang,Youzheng Wu,Xiaodong He机构：JD Technology Group摘要：在多说话者多风格的文语转换（TTS）场景中，说话者音色和风格的解纠缠对于风格迁移非常重要。随着音色和风格的分离，TTS系统可以合成具有任何风格的特定说话人的表达性语音。然而，目前关于音色与风格解缠的研究还存在一些不足。目前的方法要么需要单个说话者的多风格录音，这是困难和昂贵的收集，或使用复杂的网络和复杂的训练方法，这是难以再现和控制风格转移行为。为了提高音色和风格解纠缠的有效性，消除对单说话人多风格语料库的依赖，提出了一种简单有效的音色和风格解纠缠方法。采用FastSpeech2网络作为主干网络，以明确的时长、音高和能量轨迹来表示风格。将每个说话人的数据视为一个独立的风格，然后在FastSpeech2网络中加入说话人嵌入和风格嵌入，以学习解纠缠的表示。利用语音层次基音和能量归一化来改善去耦合效果。实验结果表明，该模型能够合成具有任意风格的语音，且具有较高的风格相似度，同时保持了较高的说话人相似度。摘要：Disentanglement of a speaker's timbre and style is very important for style transfer in multi-speaker multi-style text-to-speech (TTS) scenarios. With the disentanglement of timbres and styles, TTS systems could synthesize expressive speech for a given speaker with any style which has been seen in the training corpus. However, there are still some shortcomings with the current research on timbre and style disentanglement. The current method either requires single-speaker multi-style recordings, which are difficult and expensive to collect, or uses a complex network and complicated training method, which is difficult to reproduce and control the style transfer behavior. To improve the disentanglement effectiveness of timbres and styles, and to remove the reliance on single-speaker multi-style corpus, a simple but effective timbre and style disentanglement method is proposed in this paper. The FastSpeech2 network is employed as the backbone network, with explicit duration, pitch, and energy trajectory to represent the style. Each speaker's data is considered as a separate and isolated style, then a speaker embedding and a style embedding are added to the FastSpeech2 network to learn disentangled representations. Utterance level pitch and energy normalization are utilized to improve the decoupling effect. Experimental results demonstrate that the proposed model could synthesize speech with any style seen during training with high style similarity while maintaining very high speaker similarity.
【14】 Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames标题：Fast-U2++：联合CTC/注意帧中快速准确的端到端语音识别链接：https://arxiv.org/abs/2211.00941
作者：Chengdong Liang,Xiao-Lei Zhang,BinBin Zhang,Di Wu,Shengqiang Li,Xingchen Song,Zhendong Peng,Fuping Pan机构：School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, China, Horizon Robotics, Beijing, China ,WeNet Open Source Community备注：5 pages, 3 figures摘要：近年来，统一的流式和非流式双通道（U2/U2++）端到端语音识别模型在流式能力、准确性和延迟方面表现出了很好的性能。本文提出了快速U2 ++，它是U2++的一个增强版本，进一步降低了部分延迟。fast-U2++的核心思想是在其编码器中用小块输出底层的部分结果，而在其编码器的顶层使用大块来补偿由小块引起的性能下降。此外，我们还使用知识提取的方法来减少令牌的发送延迟。我们在Aishell-1数据集上进行了大量的实验。实验和消融研究表明，与U2++相比，fast-U2++将模型延迟从320 ms减少到80 ms，并在流设置下实现了5.06%的字符错误率（CER）。摘要：Recently, the unified streaming and non-streaming two-pass (U2/U2++) end-to-end model for speech recognition has shown great performance in terms of streaming capability, accuracy and latency. In this paper, we present fast-U2++, an enhanced version of U2++ to further reduce partial latency. The core idea of fast-U2++ is to output partial results of the bottom layers in its encoder with a small chunk, while using a large chunk in the top layers of its encoder to compensate the performance degradation caused by the small chunk. Moreover, we use knowledge distillation method to reduce the token emission latency. We present extensive experiments on Aishell-1 dataset. Experiments and ablation studies show that compared to U2++, fast-U2++ reduces model latency from 320ms to 80ms, and achieves a character error rate (CER) of 5.06% with a streaming setup.
【15】 SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation标题：SpeechBlender：用于错误发音数据生成的语音增强框架链接：https://arxiv.org/abs/2211.00923
作者：Yassine El Kheir,Shammur Absar Chowdhury,Hamdy Mubarak,Shazia Afzal,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, submitted to ICASSP 2023摘要：设计错误发音检测模型的最大挑战之一是标记的L2语音数据的不可用性。为了克服这样的数据不足，我们引入了SpeechBlender --一个细粒度的数据扩充管道，用于生成错误发音错误。SpeechBlender利用各种遮罩来针对语音单元的不同区域，并使用混合因子来线性内插原始语音信号，同时产生错误的发音实例。屏蔽有利于信号的平滑混合，从而产生比"剪切/粘贴“方法更有效的样本。我们在音素级发音质量评估任务中展示了我们的增强技术的有效性，仅利用良好的发音数据集。对于Speechcean 762测试集，与没有语音增强和发音增强的情况相比，我们观察到SpeechBlender增强的Pearson相关系数（PCC）分别增加了3%和2%。此外，当将我们的单任务音素级发音错误检测模型与使用多粒度信息的多任务学习模型进行比较时，观察到PCC增加了2%。摘要：One of the biggest challenges in designing mispronunciation detection models is the unavailability of labeled L2 speech data. To overcome such data scarcity, we introduce SpeechBlender -- a fine-grained data augmentation pipeline for generating mispronunciation errors. The SpeechBlender utilizes varieties of masks to target different regions of a phonetic unit, and use the mixing factors to linearly interpolate raw speech signals while generating erroneous pronunciation instances. The masks facilitate smooth blending of the signals, thus generating more effective samples than the `Cut/Paste' method. We show the effectiveness of our augmentation technique in a phoneme-level pronunciation quality assessment task, leveraging only a good pronunciation dataset. With SpeechBlender augmentation, we observed a 3% and 2% increase in Pearson correlation coefficient (PCC) compared to no-augmentation and goodness of pronunciation augmentation scenarios respectively for Speechocean762 testset. Moreover, a 2% rise in PCC is observed when comparing our single-task phoneme-level mispronunciation detection model with a multi-task learning model using multiple-granularity information.
【16】 SIMD-size aware weight regularization for fast neural vocoding on CPU标题：基于SIMD大小的权值正则化在CPU上的快速神经声码链接：https://arxiv.org/abs/2211.00898
作者：Hiroki Kanagawa,Yusuke Ijima机构：NTT Corporation备注：Accepted to SLT 2022摘要：本文提出了一种快速神经声码器的权值正则化方法。修剪耗时的DNN模块是在CPU上实现实时声码器（例如WaveRNN、LPCNet）的一种有前途的方法。鼓励稀疏性的正则化在避免由修剪造成的质量降级方面也是有效的。然而，权重矩阵的阶数在SIMD大小上必须是连续的，以用于快速声码。为了保证这个顺序，我们提出了显式SIMD大小感知正则化。该方法首先将权值矩阵转化为张量，使得权值与群大小预先对齐，然后计算群Lasso类正则化损失.在70%稀疏子带WaveRNN上的实验表明，传统Lasso算法和列分组Lasso算法中的剪枝降低了合成语音的自然度。具有所提出的正则化的声码器1）实现了与没有修剪的声码器相当的自然度，并且2）比使用正则化的其他传统声码器执行得更快。摘要：This paper proposes weight regularization for a faster neural vocoder. Pruning time-consuming DNN modules is a promising way to realize a real-time vocoder on a CPU (e.g. WaveRNN, LPCNet). Regularization that encourages sparsity is also effective in avoiding the quality degradation created by pruning. However, the orders of weight matrices must be contiguous in SIMD size for fast vocoding. To ensure this order, we propose explicit SIMD size aware regularization. Our proposed method reshapes a weight matrix into a tensor so that the weights are aligned by group size in advance, and then computes the group Lasso-like regularization loss. Experiments on 70% sparse subband WaveRNN show that pruning in conventional Lasso and column-wise group Lasso degrades the synthetic speech's naturalness. The vocoder with proposed regularization 1) achieves comparable naturalness to that without pruning and 2) performs meaningfully faster than other conventional vocoders using regularization.
【17】 Pop2Piano : Pop Audio-based Piano Cover Generation标题：Pop2Piano：基于流行音频的钢琴封面生成链接：https://arxiv.org/abs/2211.00895
作者：Jongho Choi,Kyogu Lee机构：rebellions Inc., Music and Audio Research Group, Seoul National University摘要：流行音乐的钢琴封面被人们广泛喜爱。然而，流行钢琴封面的生成任务仍然是欠研究的。这部分是由于缺乏同步的{Pop，Piano Cover}数据对，这使得应用最新的数据密集型深度学习方法具有挑战性。为了充分利用数据驱动方法的强大功能，我们使用自动化管道制作了大量配对和同步的{pop，piano cover}数据。本文介绍了一种Transformer网络Pop 2 Piano，它可以根据流行音乐的波形生成钢琴曲。据我们所知，这是第一个直接从流行音乐生成钢琴封面的模型，没有旋律和和弦提取模块。我们证明了用我们的数据集训练的Pop 2 Piano可以生成可信的钢琴封面。摘要：The piano cover of pop music is widely enjoyed by people. However, the generation task of the pop piano cover is still understudied. This is partly due to the lack of synchronized {Pop, Piano Cover} data pairs, which made it challenging to apply the latest data-intensive deep learning-based methods. To leverage the power of the data-driven approach, we make a large amount of paired and synchronized {pop, piano cover} data using an automated pipeline. In this paper, we present Pop2Piano, a Transformer network that generates piano covers given waveforms of pop music. To the best of our knowledge, this is the first model to directly generate a piano cover from pop audio without melody and chord extraction modules. We show that Pop2Piano trained with our dataset can generate plausible piano covers.
【18】 Conversation-oriented ASR with multi-look-ahead CBS architecture标题：具有多前瞻CBS架构的面向会话的ASR链接：https://arxiv.org/abs/2211.00858
作者：Huaibo Zhao,Shinya Fujie,Tetsuji Ogawa,Jin Sakuma,Yusuke Kida,Tetsunori Kobayashi机构：Department of Communications and Computer Engineering, Waseda University, Tokyo Japan, Line corporation, Tokyo Japan备注：Submitted to ICASSP2023摘要：在谈话过程中，人类能够在讲话的任何一点上推断讲话者的意图，以便迅速准备接下来的行动。这种能力也是会话系统实现有节奏、自然会话的关键。为此，用于实时转录语音的自动语音识别（ASR）必须无延迟地实现高精度。在流式ASR中，通过关注超前帧来确保高精度，这导致延迟增量。为了解决这一折衷问题，我们提出了一种多延迟流式ASR，以实现零前瞻的高精度。所提出的系统包含并行操作的两个编码器，其中主编码器利用先行帧生成精确输出，而辅助编码器在没有先行的情况下识别主编码器的先行部分。该系统基于上下文块流（CBS）架构，充分利用了块处理能力，对多延迟架构具有很高的亲和力。还研究了用于构建该系统的各种方法，包括移动网络以作为不同的编码器来执行;以及在一个编码通道中生成两个编码器的输出。摘要：During conversations, humans are capable of inferring the intention of the speaker at any point of the speech to prepare the following action promptly. Such ability is also the key for conversational systems to achieve rhythmic and natural conversation. To perform this, the automatic speech recognition (ASR) used for transcribing the speech in real-time must achieve high accuracy without delay. In streaming ASR, high accuracy is assured by attending to look-ahead frames, which leads to delay increments. To tackle this trade-off issue, we propose a multiple latency streaming ASR to achieve high accuracy with zero look-ahead. The proposed system contains two encoders that operate in parallel, where a primary encoder generates accurate outputs utilizing look-ahead frames, and the auxiliary encoder recognizes the look-ahead portion of the primary encoder without look-ahead. The proposed system is constructed based on contextual block streaming (CBS) architecture, which leverages block processing and has a high affinity for the multiple latency architecture. Various methods are also studied for architecting the system, including shifting the network to perform as different encoders; as well as generating both encoders' outputs in one encoding pass.
【19】 More Speaking or More Speakers?标题：更多的演讲者还是更多的演讲者？链接：https://arxiv.org/abs/2211.00854
作者：Dan Berrebbi,Ronan Collobert,Navdeep Jaitly,Tatiana Likhomanenko机构：‡Carnegie Mellon University, †Apple摘要：自训练（ST）和自监督学习（SSL）方法在自动语音识别（ASR）中已经显示出很大的改进。尽管有这些进步，但就我们所知，没有分析这些方法中使用的标记和未标记数据集的组成如何影响结果。在这项工作中，我们的目标是分析训练数据中说话人数量对最近的SSL算法（wav2vec 2.0）和最近的ST算法（slimIPL）的影响。我们通过改变说话者的数量同时保持小时数固定，对有标记和无标记的数据进行系统分析，反之亦然。我们的研究结果表明，SSL需要大量的未标记数据才能产生高准确率的结果，而ST需要标记数据中足够数量的说话人，特别是在低音域环境中。以这种方式，这两种方法改进了在数据集构成的不同机制中的监督学习。摘要：Self-training (ST) and self-supervised learning (SSL) methods have demonstrated strong improvements in automatic speech recognition (ASR). In spite of these advances, to the best of our knowledge, there is no analysis of how the composition of the labelled and unlabelled datasets used in these methods affects the results. In this work we aim to analyse the effect of numbers of speakers in the training data on a recent SSL algorithm (wav2vec 2.0), and a recent ST algorithm (slimIPL). We perform a systematic analysis on both labeled and unlabeled data by varying the number of speakers while keeping the number of hours fixed and vice versa. Our findings suggest that SSL requires a large amount of unlabeled data to produce high accuracy results, while ST requires a sufficient number of speakers in the labelled data, especially in the low-regime setting. In this manner these two approaches improve supervised learning in different regimes of dataset composition.
【20】 Build a SRE Challenge System: Lessons from VoxSRC 2022 and CNSRC 2022标题：构建SRE挑战体系：VoxSRC 2022和CNSRC 2022的经验教训链接：https://arxiv.org/abs/2211.00815
作者：Zhengyang Chen,Bing Han,Xu Xiang,Houjun Huang,Bei Liu,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, AISpeech Ltd, Suzhou, China摘要：不同的说话人识别挑战赛已经举行，以评估说话人确认系统在野外和探索性能极限。Voxceleb说话人识别挑战赛（VoxSRC）是基于Voxceleb的，是最受欢迎的。此外，今年还举办了另一项挑战赛，名为“中国名人说话人识别挑战赛”（CNSRC），该挑战赛基于中国名人多体裁数据集CN-Celeb。今年，我们团队参加了CNSRC 2022和VoxSRC 2022的两个演讲人验证封闭赛道，分别取得了第1名和第3名的成绩。在大多数系统报告中，作者通常只提供他们的系统的描述，而缺乏对他们的方法的有效分析。本文将概述如何构建一个强有力的说话人确认挑战系统，并对每种方法与其他一些流行的技术手段进行了详细的分析比较。摘要：Different speaker recognition challenges have been held to assess the speaker verification system in the wild and probe the performance limit. Voxceleb Speaker Recognition Challenge (VoxSRC), based on the voxceleb, is the most popular. Besides, another challenge called CN-Celeb Speaker Recognition Challenge (CNSRC) is also held this year, which is based on the Chinese celebrity multi-genre dataset CN-Celeb. This year, our team participated in both speaker verification closed tracks in CNSRC 2022 and VoxSRC 2022, and achieved the 1st place and 3rd place respectively. In most system reports, the authors usually only provide a description of their systems but lack an effective analysis of their methods. In this paper, we will outline how to build a strong speaker verification challenge system and give a detailed analysis of each method compared with some other popular technical means.
【21】 Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild标题：在野外笑声自动评估中，注释通道对标签质量和模型性能的影响链接：https://arxiv.org/abs/2211.00794
作者：Jose Vargas-Quiros,Laura Cabrera-Quiros,Catharine Oertel,Hayley Hung机构： Hung work at the Department of IntelligentSystems at TU Delft摘要：笑被认为是最明显的快乐信号之一。笑是公认的多模态现象，但最常见的是通过感觉笑声来检测。目前还不清楚，当通过笑的身体运动对笑进行注释时，对笑的感知和注释与其他模态（如视频）有何不同。在本文中，我们在这个方向上迈出了第一步，通过询问当注释者仅可获得音频、仅可获得视频（包含全身运动信息）或视听模态时，是否可以注释笑声以及注释效果如何。我们询问笑的注释是否在模态之间一致，并比较标记模态对机器学习模型性能的影响。我们比较了笑声检测、强度估计和分割的注释和模型，这三个任务在以前的笑声研究中很常见。我们对48位注释者的4000多条注释进行了分析，发现了笑的感知及其强度在不同模态之间存在不一致的证据。进一步分析注释与合并视听参考注释，发现与音频条件相比，视频条件下的平均回忆率较低，但随着笑声样本的强度增加，回忆率趋于增加。我们的机器学习实验比较了最先进的单模态（基于音频、基于视频和基于加速度）和多模态模型在输入模态、训练标签模态和测试标签模态的不同组合下的性能。无论训练标签模态如何，具有视频和加速度输入的模型具有相似的性能，这表明使用视频获得的标签训练模型从身体运动中检测笑声可能是完全合适的，尽管它们的评分者间一致性较低。摘要：Laughter is considered one of the most overt signals of joy. Laughter is well-recognized as a multimodal phenomenon but is most commonly detected by sensing the sound of laughter. It is unclear how perception and annotation of laughter differ when annotated from other modalities like video, via the body movements of laughter. In this paper we take a first step in this direction by asking if and how well laughter can be annotated when only audio, only video (containing full body movement information) or audiovisual modalities are available to annotators. We ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, three tasks common in previous studies of laughter. Our analysis of more than 4000 annotations acquired from 48 annotators revealed evidence for incongruity in the perception of laughter, and its intensity between modalities. Further analysis of annotations against consolidated audiovisual reference annotations revealed that recall was lower on average for video when compared to the audio condition, but tended to increase with the intensity of the laughter samples. Our machine learning experiments compared the performance of state-of-the-art unimodal (audio-based, video-based and acceleration-based) and multi-modal models for different combinations of input modalities, training label modality, and testing label modality. Models with video and acceleration inputs had similar performance regardless of training label modality, suggesting that it may be entirely appropriate to train models for laughter detection from body movements using video-acquired labels, despite their lower inter-rater agreement.
【22】 Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems标题：用于快速高效语音系统的统一端到端语音识别和端点链接：https://arxiv.org/abs/2211.00786
作者：Shaan Bijwadia,Shuo-yiin Chang,Bo Li,Tara Sainath,Chao Zhang,Yanzhang He机构：Google, Inc.备注：To be published in Spoken Language Technology Workshop (SLT) 2022摘要：自动语音识别（ASR）系统典型地依赖于外部端点指示器（EP）模型来识别语音边界。在这项工作中，我们提出了一种方法，在一个单一的端到端（E2E）多任务模型中联合训练ASR和EP任务，通过选择性地利用来自ASR音频编码器的信息来提高EP质量。我们引入了一个“开关”连接，它训练EP直接消费音频帧或来自ASR模型的低级别潜在表示。这产生了单个E2E模型，该模型可以在推断期间用于以低成本执行帧过滤，并且还基于正在进行的ASR计算来进行高质量的查询结束（EOQ）预测。我们提供了语音搜索测试集的结果，结果显示，与单独的单任务模型相比，该方法将中位终点延迟缩短了120 ms（缩短了30.8%），将第90百分位延迟缩短了170 ms（缩短了23.0%），而不会使单词错误率回归。对于连续识别，WER提高了10.6%（相对）。摘要：Automatic speech recognition (ASR) systems typically rely on an external endpointer (EP) model to identify speech boundaries. In this work, we propose a method to jointly train the ASR and EP tasks in a single end-to-end (E2E) multitask model, improving EP quality by optionally leveraging information from the ASR audio encoder. We introduce a "switch" connection, which trains the EP to consume either the audio frames directly or low-level latent representations from the ASR model. This results in a single E2E model that can be used during inference to perform frame filtering at low cost, and also make high quality end-of-query (EOQ) predictions based on ongoing ASR computation. We present results on a voice search test set showing that, compared to separate single-task models, this approach reduces median endpoint latency by 120 ms (30.8% reduction), and 90th percentile latency by 170 ms (23.0% reduction), without regressing word error rate. For continuous recognition, WER improves by 10.6% (relative).
【23】 Comparision Of Adversarial And Non-Adversarial LSTM Music Generative Models标题：对抗性与非对抗性LSTM音乐生成模式的比较链接：https://arxiv.org/abs/2211.00731
作者：Moseli Mots'oehli,Anna Sergeevna Bosman,Johan Pieter De Villiers机构：Department of Computer Science, University Of Pretoria, Department of Electrical and Computer Engineering备注：Submitted to a 2023 conference, 20 pages, 13 figures摘要：算法音乐创作是一种在最少甚至没有人为干预的情况下创作音乐作品的方式。虽然递归神经网络传统上被应用于许多序列到序列的预测任务，包括音乐创作的成功实现，但是它们基于输入到输出映射的标准监督学习方法导致音符变化的缺乏。因此，这些模型可以被视为潜在地不适合于诸如音乐生成之类的任务。生成式对抗网络学习数据的生成分布，并导致不同的样本。本工作在MIDI数据上实现并比较了递归神经网络音乐作曲家的对抗性和非对抗性训练。所得到的音乐样本由人类听众评估，他们的偏好被记录下来。评价表明对抗性训练产生了更美观的音乐。摘要：Algorithmic music composition is a way of composing musical pieces with minimal to no human intervention. While recurrent neural networks are traditionally applied to many sequence-to-sequence prediction tasks, including successful implementations of music composition, their standard supervised learning approach based on input-to-output mapping leads to a lack of note variety. These models can therefore be seen as potentially unsuitable for tasks such as music generation. Generative adversarial networks learn the generative distribution of data and lead to varied samples. This work implements and compares adversarial and non-adversarial training of recurrent neural network music composers on MIDI data. The resulting music samples are evaluated by human listeners, their preferences recorded. The evaluation indicates that adversarial training produces more aesthetically pleasing music.
【24】 Technology Pipeline for Large Scale Cross-Lingual Dubbing of Lecture Videos into Multiple Indian Languages标题：用于将讲座视频大规模跨语言配音成多种印度语言的技术管道链接：https://arxiv.org/abs/2211.01338
作者：Anusha Prakash,Arun Kumar,Ashish Seth,Bhagyashree Mukherjee,Ishika Gupta,Jom Kuriakose,Jordan Fernandes,K V Vikram,Mano Ranjith Kumar M,Metilda Sagaya Mary,Mohammad Wajahat,Mohana N,Mudit Batra,Navina K,Nihal John George,Nithya Ravi,Pruthwik Mishra,Sudhanshu Srivastava,Vasista Sai Lodagala,Vandan Mujadia,Kada Sai Venkata Vineeth,Vrunda Sukhadia,Dipti Sharma,Hema Murthy,Pushpak Bhattacharya,S Umesh,Rajeev Sangal机构： SMT Lab, Department of Computer Science and Engineering, IIT Madras, India, Speech Lab, Department of Electrical Engineering, IIT Madras, India, IIIT Hyderabad, India, IIT Bombay, India摘要：演讲视频的跨语言配音需要原始音频的转录、不流利的校正和去除、领域术语发现、文本到文本到目标语言的翻译、使用目标语言节奏的文本组块、文本到语音合成以及随后的与原始视频的同步口型同步。当源语言和目标语言属于不同的语系时，这一任务变得具有挑战性，从而导致所生成的音频持续时间的差异。这是进一步复杂的原始发言者的节奏，特别是即兴演讲。本文描述了在半自动地以印度语言重新生成英语演讲视频方面所面临的挑战。开发了一个将讲座配音成9种印度语言的原型。在两个不同的课程上，获得了两种语言（印地语和泰米尔语）的平均意见得分（MOS）。输出视频与原始视频在MOS（1-5）和唇同步方面进行比较，得分分别为4.09和3.74。人力也减少了75%。摘要：Cross-lingual dubbing of lecture videos requires the transcription of the original audio, correction and removal of disfluencies, domain term discovery, text-to-text translation into the target language, chunking of text using target language rhythm, text-to-speech synthesis followed by isochronous lipsyncing to the original video. This task becomes challenging when the source and target languages belong to different language families, resulting in differences in generated audio duration. This is further compounded by the original speaker's rhythm, especially for extempore speech. This paper describes the challenges in regenerating English lecture videos in Indian languages semi-automatically. A prototype is developed for dubbing lectures into 9 Indian languages. A mean-opinion-score (MOS) is obtained for two languages, Hindi and Tamil, on two different courses. The output video is compared with the original video in terms of MOS (1-5) and lip synchronisation with scores of 4.09 and 3.74, respectively. The human effort also reduces by 75%.
【25】 Towards End-to-end Speaker Diarization in the Wild标题：在荒野中走向端到端的说话人二元化链接：https://arxiv.org/abs/2211.01299
作者：Zexu Pan,Gordon Wichern,François G. Germain,Aswin Subramanian,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, Institute of Data Science, National University of Singapore, Singapore备注：5 pages, 2 figures, 2 tables. Submitted to ICASSP 2023摘要：说话人日记化算法解决了音频记录中的“谁在何时说话”问题。经端到端训练的算法已被证明在具有少量扬声器的受限场景中优于经典的模块级联系统。然而，它们对于包含更多说话者和更短话语长度的野生录音的性能仍有待研究。在本文中，我们解决了这一差距，表明基于吸引子的端到端系统也可以在后一种情况下表现得非常好，当第一次在精心设计的模拟数据集上进行预训练时，该数据集与野生记录的分布相匹配。我们还提出了使用注意机制来增加解码更多说话人吸引子的网络容量，以及在说话人识别任务中联合训练吸引子以改善说话人吸引子表示。尽管我们提出的模型仅适用于音频，但我们发现它在AVA-AVD基准数据集上的性能明显优于仅适用于音频和视听的基线，实现了最先进的结果，二值化误差绝对减少了23.3%。摘要：Speaker diarization algorithms address the "who spoke when" problem in audio recordings. Algorithms trained end-to-end have proven superior to classical modular-cascaded systems in constrained scenarios with a small number of speakers. However, their performance for in-the-wild recordings containing more speakers with shorter utterance lengths remains to be investigated. In this paper, we address this gap, showing that an attractor-based end-to-end system can also perform remarkably well in the latter scenario when first pre-trained on a carefully-designed simulated dataset that matches the distribution of in-the-wild recordings. We also propose to use an attention mechanism to increase the network capacity in decoding more speaker attractors, and to jointly train the attractors on a speaker recognition task to improve the speaker attractor representation. Even though the model we propose is audio-only, we find it significantly outperforms both audio-only and audio-visual baselines on the AVA-AVD benchmark dataset, achieving state-of-the-art results with an absolute reduction in diarization error of 23.3%.
【26】 data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setup标题：Data2vec-AQC：在师生训练体系中寻找合适的助教链接：https://arxiv.org/abs/2211.01246
作者：Vasista Sai Lodagala,Sreyan Ghosh,S. Umesh机构：Indian Institute of Technology, Madras, India, University of Maryland, College Park, USA备注：Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2210.02592摘要：本文提出了一种新的自监督学习（SSL）算法data 2 vec-aqc，用于从无标记语音数据中学习语音表示。我们的目标是在未标记和标记数据都有限的域中改进用于语音的SSL。在最近引入的data 2 vec的基础上，我们向data 2 vec框架引入了额外的模块，这些模块利用了数据扩充、量化表示和聚类的优势。这些模块之间的相互作用有助于解决交叉对比损失作为一个额外的自我监督的目标。在不使用任何语言模型的情况下，data 2 vec-aqc在LibriSpeech的test-clean和test-other集上分别比现有的data 2 vec系统实现了14.1%和20.9%的相对WER改善。当在Switchboard数据上进行微调时，我们提出的模型也实现了比基线data 2 vec高出17.8%的相对WER改善。摘要：In this paper, we propose a new Self-Supervised Learning (SSL) algorithm called data2vec-aqc, for speech representation learning from unlabeled speech data. Our goal is to improve SSL for speech in domains where both unlabeled and labeled data are limited. Building on the recently introduced data2vec, we introduce additional modules to the data2vec framework that leverage the benefit of data augmentations, quantized representations, and clustering. The interaction between these modules helps solve the cross-contrastive loss as an additional self-supervised objective. data2vec-aqc achieves up to 14.1% and 20.9% relative WER improvement over the existing state-of-the-art data2vec system on the test-clean and test-other sets, respectively, of LibriSpeech, without the use of any language model. Our proposed model also achieves up to 17.8% relative WER improvement over the baseline data2vec when fine-tuned on Switchboard data.
【27】 Analysis of Noisy-target Training for DNN-based speech enhancement标题：基于DNN的语音增强中的噪声目标训练分析链接：https://arxiv.org/abs/2211.01198
作者：Takuya Fujimura,Tomoki Toda机构：Nagoya University, Japan备注：Submitted to ICASSP 2023摘要：基于深度神经网络（DNN）的语音增强通常以干净的语音作为训练目标。然而，很难收集大量干净的语音，因为录音是非常昂贵的。换句话说，当前语音增强的性能受到训练数据量的限制。为了缓和这种限制，已经提出了利用噪声语音作为训练目标的噪声目标训练（NyTT）。虽然实验表明NyTT可以在没有干净语音的情况下训练DNN，但是还没有进行详细的分析，并且还没有很好地理解其行为。本文对NyTT进行了多方面的分析，以加深对NyTT的理解。此外，基于NyTT的特性，我们提出了一种改进的方法，该方法可以与使用干净语音的方法相媲美。此外，我们还表明，我们可以通过使用大量的噪声语音与干净的语音改善性能。摘要：Deep neural network (DNN)-based speech enhancement usually uses a clean speech as a training target. However, it is hard to collect large amounts of clean speech because the recording is very costly. In other words, the performance of current speech enhancement has been limited by the amount of training data. To relax this limitation, Noisy-target Training (NyTT) that utilizes noisy speech as a training target has been proposed. Although it has been experimentally shown that NyTT can train a DNN without clean speech, a detailed analysis has not been conducted and its behavior has not been understood well. In this paper, we conduct various analyses to deepen our understanding of NyTT. In addition, based on the property of NyTT, we propose a refined method that is comparable to the method using clean speech. Furthermore, we show that we can improve the performance by using a huge amount of noisy speech with clean speech.
【28】 Inference and Denoise: Causal Inference-based Neural Speech Enhancement标题：推理与去噪：基于因果推理的神经语音增强链接：https://arxiv.org/abs/2211.01189
作者：Tsun-An Hsieh,Chao-Han Huck Yang,Pin-Yu Chen,Sabato Marco Siniscalchi,Yu Tsao机构：Research Center for Information Technology Innovation, Academia Sinica, Taiwan, Georgia Institute of Technology, GA, USA; ,IBM Research, NY, USA, Computer Engineering School, Norwegian University of Science and Technology, Norway备注：Submitted to ICASSP 2023摘要：本研究通过将噪声的存在作为一种干预来建模，在因果推理范式中解决了语音增强（SE）任务。基于潜在结果框架，提出了一种基于因果推理的语音增强算法（CISE），该算法利用噪声检测器从噪声语音中分离出干净帧和噪声帧，并将这两组帧分配给两个基于掩蔽的增强模块（EM）进行噪声条件SE。具体地，我们在训练期间使用噪声的存在作为EM选择的指导，并且噪声检测器根据对每一帧的噪声的存在的预测来选择增强模块。此外，我们推导了SE特异性平均治疗效应，以充分量化因果效应。实验结果表明，CISE在所研究的场景中优于基于非因果掩模的SE方法，并且比更复杂的SE模型具有更好的性能和效率。摘要：This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. Based on the potential outcome framework, the proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE. Specifically, we use the presence of noise as guidance for EM selection during training, and the noise detector selects the enhancement module according to the prediction of the presence of noise for each frame. Moreover, we derived a SE-specific average treatment effect to quantify the causal effect adequately. Experimental evidence demonstrates that CISE outperforms a non-causal mask-based SE approach in the studied settings and has better performance and efficiency than more complex SE models.
【29】 I4U System Description for NIST SRE'20 CTS Challenge标题：NIST SRE‘20 CTS挑战赛的I4U系统描述链接：https://arxiv.org/abs/2211.01091
作者：Kong Aik Lee,Tomi Kinnunen,Daniele Colibro,Claudio Vair,Andreas Nautsch,Hanwu Sun,Liang He,Tianyu Liang,Qiongqiong Wang,Mickael Rouvier,Pierre-Michel Bousquet,Rohan Kumar Das,Ignacio Viñals Bailo,Meng Liu,Héctor Deldago,Xuechen Liu,Md Sahidullah,Sandro Cumani,Boning Zhang,Koji Okabe,Hitoshi Yamamoto,Ruijie Tao,Haizhou Li,Alfonso Ortega Giménez,Longbiao Wang,Luis Buera机构：Institute for Infocomm Research, ASTAR, Singapore, School of Computing, University of Eastern Finland, Finland, Loquendo and Politecnico di Torino, Italy, EURECOM, France and vitas.ai, Germany, Department of Electronic Engineering, Tsinghua University, China备注：SRE 2021, NIST Speaker Recognition Evaluation Workshop, CTS Speaker Recognition Challenge, 14-12 December 2021摘要：本文描述了I4 U提交的2020年NIST说话人识别评估（SRE'20）会话电话语音（CTS）挑战。I4 U的提交是来自八个研究团队的研究人员之间积极合作的结果- I$^2$R（新加坡），UEF（芬兰），VALPT（意大利，西班牙），NEC（日本），THUEE（中国），LIA（法国），NUS（新加坡），INRIA（法国）和TJU（中国）。该提交是基于最佳性能子系统的融合和各团队贡献的子融合系统。已努力使用通用开发和验证集、提交时间表和里程碑，最大限度地减少各研究中心试验列表和评分文件格式的不一致性。摘要：This manuscript describes the I4U submission to the 2020 NIST Speaker Recognition Evaluation (SRE'20) Conversational Telephone Speech (CTS) Challenge. The I4U's submission was resulted from active collaboration among researchers across eight research teams - I$^2$R (Singapore), UEF (Finland), VALPT (Italy, Spain), NEC (Japan), THUEE (China), LIA (France), NUS (Singapore), INRIA (France) and TJU (China). The submission was based on the fusion of top performing sub-systems and sub-fusion systems contributed by individual teams. Efforts have been spent on the use of common development and validation sets, submission schedule and milestone, minimizing inconsistency in trial list and score file format across sites.
【30】 Monolingual Recognizers Fusion for Code-switching Speech Recognition标题：码型转换语音识别中的单语识别器融合链接：https://arxiv.org/abs/2211.01046
作者：Tongtong Song,Qiang Xu,Haoyu Lu,Longbiao Wang,Hao Shi,Yuqin Lin,Yanbing Yang,Jianwu Dang机构：Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin, China, Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan备注：Submitted to ICASSP2023摘要：双编码器结构在码切换（CS）自动语音识别（ASR）中得到了广泛研究。然而，现有的大多数方法要求两个单语ASR模型的结构相同，并且只使用两个单语ASR模型的编码器。这导致了预先培训的MAM不能及时和充分地用于CS ASR的问题。本文提出了一种面向CS ASR的单语识别器融合方法。它有两个阶段：言语意识阶段和语言融合阶段。在SA阶段，声学特征由两个独立的MAM映射到两个语言特定的预测。为了让MAM专注于他们自己的语言，我们进一步扩展了MAM的语言感知培训策略。在LF阶段，BELM融合两个语言特定的预测以获得最终预测。此外，我们提出了一种文本模拟策略，以简化BELM的训练过程，减少对CS数据的依赖。在汉英语料库上的实验表明了该方法的有效性。在使用开源预训练MAM后，测试集上的混合错误率显著降低。摘要：The bi-encoder structure has been intensively investigated in code-switching (CS) automatic speech recognition (ASR). However, most existing methods require the structures of two monolingual ASR models (MAMs) should be the same and only use the encoder of MAMs. This leads to the problem that pre-trained MAMs cannot be timely and fully used for CS ASR. In this paper, we propose a monolingual recognizers fusion method for CS ASR. It has two stages: the speech awareness (SA) stage and the language fusion (LF) stage. In the SA stage, acoustic features are mapped to two language-specific predictions by two independent MAMs. To keep the MAMs focused on their own language, we further extend the language-aware training strategy for the MAMs. In the LF stage, the BELM fuses two language-specific predictions to get the final prediction. Moreover, we propose a text simulation strategy to simplify the training process of the BELM and reduce reliance on CS data. Experiments on a Mandarin-English corpus show the efficiency of the proposed method. The mix error rate is significantly reduced on the test set after using open-source pre-trained MAMs.
【31】 Adversarial Guitar Amplifier Modelling With Unpaired Data标题：基于非配对数据的对抗性吉他放大器建模链接：https://arxiv.org/abs/2211.00943
作者：Alec Wright,Vesa Välimäki,Lauri Juvela机构：Acoustics Lab, Dept. of Signal Processing and Acoustics, Aalto University, Espoo, Finland, Neural DSP Technologies, Helsinki, Finland摘要：我们提出了一个音效处理框架，它学习模仿来自录音的目标电吉他音调。我们使用对抗性方法训练深度神经网络，目标是在例如通过吉他放大器应用音频效果处理之后将吉他的音色转换成另一吉他的音色。模型训练不需要配对数据，并且所得到的模型很好地仿真了目标音色，同时能够在现代个人计算机上进行实时处理。为了验证我们的方法，我们提供了两个实验，一个实验使用配对数据进行非配对训练，允许我们通过客观指标来监控训练，另一个实验使用完全非配对数据，对应于用户希望仅使用录音中的音频数据来模拟吉他音色的现实场景。我们的听力测试结果证实了模型在感知上是令人信服的。摘要：We propose an audio effects processing framework that learns to emulate a target electric guitar tone from a recording. We train a deep neural network using an adversarial approach, with the goal of transforming the timbre of a guitar, into the timbre of another guitar after audio effects processing has been applied, for example, by a guitar amplifier. The model training requires no paired data, and the resulting model emulates the target timbre well whilst being capable of real-time processing on a modern personal computer. To verify our approach we present two experiments, one which carries out unpaired training using paired data, allowing us to monitor training via objective metrics, and another that uses fully unpaired data, corresponding to a realistic scenario where a user wants to emulate a guitar timbre only using audio data from a recording. Our listening test results confirm that the models are perceptually convincing.
【32】 Factorized Blank Thresholding for Improved Runtime Efficiency of Neural Transducers标题：用于提高神经传感器运行效率的因式空白阈值链接：https://arxiv.org/abs/2211.00896
作者：Duc Le,Frank Seide,Yuhao Wang,Yang Li,Kjell Schubert,Ozlem Kalinli,Michael L. Seltzer机构：Meta, USA备注：Submitted to ICASSP 2023摘要：我们展示了分解RNN-T的输出分布如何能够显著降低设备上ASR推断的计算成本和功耗，而不损失准确性。随着用于设备上ASR的RNN-T等神经传感器类型模型的普及，优化RNN-T的运行时效率引起了极大的兴趣。以前的工作主要集中在RNN-T的声编码器和预测器的优化上，而本文将注意力集中在连接器上。我们表明，尽管连接器只是RNN-T的一小部分，但它对整个模型的运行时效率有很大的影响。我们建议将连接符分解为空白和非空白部分，以便在空白概率超过某个阈值时跳过更昂贵的非空白计算。由于可以非常有效地计算空白概率，并且RNN-T输出由空白占主导，因此我们提出的方法导致26-30%的解码速度提高和43-53%的设备上功耗降低，同时始终不会导致准确性降低并且实现相对简单。摘要：We show how factoring the RNN-T's output distribution can significantly reduce the computation cost and power consumption for on-device ASR inference with no loss in accuracy. With the rise in popularity of neural-transducer type models like the RNN-T for on-device ASR, optimizing RNN-T's runtime efficiency is of great interest. While previous work has primarily focused on the optimization of RNN-T's acoustic encoder and predictor, this paper focuses the attention on the joiner. We show that despite being only a small part of RNN-T, the joiner has a large impact on the overall model's runtime efficiency. We propose to factorize the joiner into blank and non-blank portions for the purpose of skipping the more expensive non-blank computation when the blank probability exceeds a certain threshold. Since the blank probability can be computed very efficiently and the RNN-T output is dominated by blanks, our proposed method leads to a 26-30% decoding speed-up and 43-53% reduction in on-device power consumption, all the while incurring no accuracy degradation and being relatively simple to implement.
【33】 Neural Fourier Shift for Binaural Speech Rendering标题：用于双耳语音再现的神经傅里叶变换链接：https://arxiv.org/abs/2211.00878
作者：Jin Woo Lee,Kyogu Lee机构：Department of Intelligence and Information, Seoul National University备注：Submitted to ICASSP 2023摘要：我们提出了一种神经网络，用于从给定的单耳音频、源的位置和方向渲染双耳语音。以往的研究主要集中在通过调节卷积神经网络在特征空间中的位置和方向来合成双耳语音。这些合成方法在估计目标双耳语音方面是强大的，即使对于野生数据也是如此，但是难以推广用于从分布域外呈现音频。为了缓解这一问题，我们提出了神经傅立叶变换（NFS），一种新颖的网络架构，能够在傅立叶空间中实现双耳语音渲染。具体地，利用基于源和接收器之间的距离的几何时间延迟，训练NFS以预测各种早期反射的延迟和尺度。NFS在内存和计算成本方面都是高效的，是可解释的，并且通过其设计独立于源域运行。实验结果表明，NFS在性能指标评测数据集上的表现优于之前的研究，其内存占用量减少了25倍，计算量减少了6倍。摘要：We present a neural network for rendering binaural speech from given monaural audio, position, and orientation of the source. Most of the previous works have focused on synthesizing binaural speeches by conditioning the positions and orientations in the feature space of convolutional neural networks. These synthesis approaches are powerful in estimating the target binaural speeches even for in-the-wild data but are difficult to generalize for rendering the audio from out-of-distribution domains. To alleviate this, we propose Neural Fourier Shift (NFS), a novel network architecture that enables binaural speech rendering in the Fourier space. Specifically, utilizing a geometric time delay based on the distance between the source and the receiver, NFS is trained to predict the delays and scales of various early reflections. NFS is efficient in both memory and computational cost, is interpretable, and operates independently of the source domain by its design. With up to 25 times lighter memory and 6 times fewer calculations, the experimental results show that NFS outperforms the previous studies on the benchmark dataset.
【34】 LMD: A Learnable Mask Network to Detect Adversarial Examples for Speaker Verification标题：LMD：一种用于说话人确认的可学习掩码网络链接：https://arxiv.org/abs/2211.00825
作者：Xing Chen,Jie Wang,Xiao-Lei Zhang,Wei-Qiang Zhang,Kunde Yang备注：13 pages, 9 figures摘要：近年来，对抗性攻击严重威胁着自动说话人确认（ASV）系统的安全性，但目前已经有了一些应对措施。然而，许多防御方法不仅需要攻击者的先验知识，而且可解释性较弱。针对这一问题，本文提出了一种与攻击者无关且可解释的方法--可学习掩码检测器（learnable mask detector，LMD），用于区分攻击者的攻击样本和真实样本。它利用分数变化作为指示器来检测对抗性示例，其中分数变化是原始音频记录的ASV分数与从其掩蔽的复杂谱图合成的其变换音频之间的绝对差异。乐谱变化检测器的核心部件是通过神经网络生成掩蔽谱图。神经网络只需要真实的例子来训练，这使它成为一种攻击者独立的方法。其可解释性在于，神经网络的训练目标是最小化目标ASV的分数变化，并最大化真实训练样本的掩蔽谱图bin数。它的基础是基于这样的观察，即用很少的说话者信息掩盖绝大多数的谱图箱将不可避免地向对抗性示例引入大的分数变化，而向真实示例引入小的分数变化。在12个攻击者和两个典型的ASV系统上的实验结果表明，该方法的性能优于5个现有的基准.大量的实验结果也可以作为基于检测的ASV防御的基准。摘要：Although the security of automatic speaker verification (ASV) is seriously threatened by recently emerged adversarial attacks, there have been some countermeasures to alleviate the threat. However, many defense approaches not only require the prior knowledge of the attackers but also possess weak interpretability. To address this issue, in this paper, we propose an attacker-independent and interpretable method, named learnable mask detector (LMD), to separate adversarial examples from the genuine ones. It utilizes score variation as an indicator to detect adversarial examples, where the score variation is the absolute discrepancy between the ASV scores of an original audio recording and its transformed audio synthesized from its masked complex spectrogram. A core component of the score variation detector is to generate the masked spectrogram by a neural network. The neural network needs only genuine examples for training, which makes it an attacker-independent approach. Its interpretability lies that the neural network is trained to minimize the score variation of the targeted ASV, and maximize the number of the masked spectrogram bins of the genuine training examples. Its foundation is based on the observation that, masking out the vast majority of the spectrogram bins with little speaker information will inevitably introduce a large score variation to the adversarial example, and a small score variation to the genuine example. Experimental results with 12 attackers and two representative ASV systems show that our proposed method outperforms five state-of-the-art baselines. The extensive experimental results can also be a benchmark for the detection-based ASV defenses.
【35】 InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss标题：InterMPL：具有中间CTC损失的动量伪标记链接：https://arxiv.org/abs/2211.00795
作者：Yosuke Higuchi,Tetsuji Ogawa,Tetsunori Kobayashi,Shinji Watanabe机构：Carnegie Mellon University, USA ,Waseda University, Japan备注：Submitted to ICASSP2023摘要：本文提出了一种半监督学习的端到端自动语音识别方法InterMPL，该方法在中间监督下进行伪标注。Momentum PL（MPL）通过动态生成伪标签并提高其质量，在未标记数据上训练基于连接主义时态分类（CTC）的模型。与诸如基于注意力的编码器-解码器和转换器的自回归公式相比，CTC非常适合于MPL，或通常基于PL的半监督ASR，这归因于其简单/快速的推理算法和对生成折叠标签的鲁棒性。然而，由于条件独立性假设，CTC通常产生比自回归模型差的性能，从而限制了MPL的性能。我们建议通过引入中间损耗来增强MPL，这是受基于CTC建模的最新进展的启发。特别地，我们关注自条件和分层条件CTC，它们将辅助CTC损失应用于中间层，使得条件独立性假设显式地放松。我们还探讨了如何生成伪标签，并将其用于监督中间损失。在不同半监督环境下的实验结果表明，与MPL算法相比，该算法提高了ASR模型12.1%的绝对性能增益.此外，我们的详细分析验证了中间损失的重要性。摘要：This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision. Momentum PL (MPL) trains a connectionist temporal classification (CTC)-based model on unlabeled data by continuously generating pseudo-labels on the fly and improving their quality. In contrast to autoregressive formulations, such as the attention-based encoder-decoder and transducer, CTC is well suited for MPL, or PL-based semi-supervised ASR in general, owing to its simple/fast inference algorithm and robustness against generating collapsed labels. However, CTC generally yields inferior performance than the autoregressive models due to the conditional independence assumption, thereby limiting the performance of MPL. We propose to enhance MPL by introducing intermediate loss, inspired by the recent advances in CTC-based modeling. Specifically, we focus on self-conditional and hierarchical conditional CTC, that apply auxiliary CTC losses to intermediate layers such that the conditional independence assumption is explicitly relaxed. We also explore how pseudo-labels should be generated and used as supervision for intermediate losses. Experimental results in different semi-supervised settings demonstrate that the proposed approach outperforms MPL and improves an ASR model by up to a 12.1% absolute performance gain. In addition, our detailed analysis validates the importance of the intermediate loss.
【36】 BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder标题：BECTRA：采用BERT增强编码器的基于传感器的端到端ASR链接：https://arxiv.org/abs/2211.00792
作者：Yosuke Higuchi,Tetsuji Ogawa,Tetsunori Kobayashi,Shinji Watanabe机构：Waseda University, Japan ,Carnegie Mellon University, USA备注：Submitted to ICASSP2023摘要：提出了一种基于BERT-CTC-Transducer（BECTRA）的端到端自动语音识别（E2 E-ASR）模型，该模型是由一个带有BERT增强编码器的换能器构成的。将大规模预训练语言模型（LM）集成到E2 E-ASR中是一个积极的研究方向，旨在利用通用的语言学知识生成准确的文本。使这种整合具有挑战性的一个关键因素在于词汇不匹配;为预先训练的LM构造的词汇表对于E2 E-ASR训练来说通常太大，并且很可能与目标ASR域不匹配。为了克服这样的问题，我们提出了BECTRA，它是我们以前的BERT-CTC的扩展版本，它使用感兴趣的词汇表实现基于BERT的E2 E-ASR。BECTRA是一个基于转换器的模型，它采用BERT-CTC作为编码器，并使用适合于目标任务的词汇表训练ASR专用解码器。通过换能器和BERT-CTC的结合，我们还提出了一种新的推理算法，以利用自回归和非自回归解码的优点。在不同数据量、说话风格和语言的多个ASR任务上的实验结果表明，BECTRA在利用BERT知识的同时有效地处理了词汇失配问题，比BERT-CTC具有更好的性能。摘要：We present BERT-CTC-Transducer (BECTRA), a novel end-to-end automatic speech recognition (E2E-ASR) model formulated by the transducer with a BERT-enhanced encoder. Integrating a large-scale pre-trained language model (LM) into E2E-ASR has been actively studied, aiming to utilize versatile linguistic knowledge for generating accurate text. One crucial factor that makes this integration challenging lies in the vocabulary mismatch; the vocabulary constructed for a pre-trained LM is generally too large for E2E-ASR training and is likely to have a mismatch against a target ASR domain. To overcome such an issue, we propose BECTRA, an extended version of our previous BERT-CTC, that realizes BERT-based E2E-ASR using a vocabulary of interest. BECTRA is a transducer-based model, which adopts BERT-CTC for its encoder and trains an ASR-specific decoder using a vocabulary suitable for a target task. With the combination of the transducer and BERT-CTC, we also propose a novel inference algorithm for taking advantage of both autoregressive and non-autoregressive decoding. Experimental results on several ASR tasks, varying in amounts of data, speaking styles, and languages, demonstrate that BECTRA outperforms BERT-CTC by effectively dealing with the vocabulary mismatch while exploiting BERT knowledge.
【1】 Technology Pipeline for Large Scale Cross-Lingual Dubbing of Lecture Videos into Multiple Indian Languages标题：用于将讲座视频大规模跨语言配音成多种印度语言的技术管道链接：https://arxiv.org/abs/2211.01338
* 与cs.SD语音【24】为同一篇
作者：Anusha Prakash,Arun Kumar,Ashish Seth,Bhagyashree Mukherjee,Ishika Gupta,Jom Kuriakose,Jordan Fernandes,K V Vikram,Mano Ranjith Kumar M,Metilda Sagaya Mary,Mohammad Wajahat,Mohana N,Mudit Batra,Navina K,Nihal John George,Nithya Ravi,Pruthwik Mishra,Sudhanshu Srivastava,Vasista Sai Lodagala,Vandan Mujadia,Kada Sai Venkata Vineeth,Vrunda Sukhadia,Dipti Sharma,Hema Murthy,Pushpak Bhattacharya,S Umesh,Rajeev Sangal机构： SMT Lab, Department of Computer Science and Engineering, IIT Madras, India, Speech Lab, Department of Electrical Engineering, IIT Madras, India, IIIT Hyderabad, India, IIT Bombay, India摘要：演讲视频的跨语言配音需要原始音频的转录、不流利的校正和去除、领域术语发现、文本到文本到目标语言的翻译、使用目标语言节奏的文本组块、文本到语音合成以及随后的与原始视频的同步口型同步。当源语言和目标语言属于不同的语系时，这一任务变得具有挑战性，从而导致所生成的音频持续时间的差异。这是进一步复杂的原始发言者的节奏，特别是即兴演讲。本文描述了在半自动地以印度语言重新生成英语演讲视频方面所面临的挑战。开发了一个将讲座配音成9种印度语言的原型。在两个不同的课程上，获得了两种语言（印地语和泰米尔语）的平均意见得分（MOS）。输出视频与原始视频在MOS（1-5）和唇同步方面进行比较，得分分别为4.09和3.74。人力也减少了75%。摘要：Cross-lingual dubbing of lecture videos requires the transcription of the original audio, correction and removal of disfluencies, domain term discovery, text-to-text translation into the target language, chunking of text using target language rhythm, text-to-speech synthesis followed by isochronous lipsyncing to the original video. This task becomes challenging when the source and target languages belong to different language families, resulting in differences in generated audio duration. This is further compounded by the original speaker's rhythm, especially for extempore speech. This paper describes the challenges in regenerating English lecture videos in Indian languages semi-automatically. A prototype is developed for dubbing lectures into 9 Indian languages. A mean-opinion-score (MOS) is obtained for two languages, Hindi and Tamil, on two different courses. The output video is compared with the original video in terms of MOS (1-5) and lip synchronisation with scores of 4.09 and 3.74, respectively. The human effort also reduces by 75%.
【2】 Towards End-to-end Speaker Diarization in the Wild标题：在荒野中走向端到端的说话人二元化链接：https://arxiv.org/abs/2211.01299
* 与cs.SD语音【25】为同一篇
作者：Zexu Pan,Gordon Wichern,François G. Germain,Aswin Subramanian,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, Institute of Data Science, National University of Singapore, Singapore备注：5 pages, 2 figures, 2 tables. Submitted to ICASSP 2023摘要：说话人日记化算法解决了音频记录中的“谁在何时说话”问题。经端到端训练的算法已被证明在具有少量扬声器的受限场景中优于经典的模块级联系统。然而，它们对于包含更多说话者和更短话语长度的野生录音的性能仍有待研究。在本文中，我们解决了这一差距，表明基于吸引子的端到端系统也可以在后一种情况下表现得非常好，当第一次在精心设计的模拟数据集上进行预训练时，该数据集与野生记录的分布相匹配。我们还提出了使用注意机制来增加解码更多说话人吸引子的网络容量，以及在说话人识别任务中联合训练吸引子以改善说话人吸引子表示。尽管我们提出的模型仅适用于音频，但我们发现它在AVA-AVD基准数据集上的性能明显优于仅适用于音频和视听的基线，实现了最先进的结果，二值化误差绝对减少了23.3%。摘要：Speaker diarization algorithms address the "who spoke when" problem in audio recordings. Algorithms trained end-to-end have proven superior to classical modular-cascaded systems in constrained scenarios with a small number of speakers. However, their performance for in-the-wild recordings containing more speakers with shorter utterance lengths remains to be investigated. In this paper, we address this gap, showing that an attractor-based end-to-end system can also perform remarkably well in the latter scenario when first pre-trained on a carefully-designed simulated dataset that matches the distribution of in-the-wild recordings. We also propose to use an attention mechanism to increase the network capacity in decoding more speaker attractors, and to jointly train the attractors on a speaker recognition task to improve the speaker attractor representation. Even though the model we propose is audio-only, we find it significantly outperforms both audio-only and audio-visual baselines on the AVA-AVD benchmark dataset, achieving state-of-the-art results with an absolute reduction in diarization error of 23.3%.
【3】 data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setup标题：Data2vec-AQC：在师生训练体系中寻找合适的助教链接：https://arxiv.org/abs/2211.01246
* 与cs.SD语音【26】为同一篇
作者：Vasista Sai Lodagala,Sreyan Ghosh,S. Umesh机构：Indian Institute of Technology, Madras, India, University of Maryland, College Park, USA备注：Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2210.02592摘要：本文提出了一种新的自监督学习（SSL）算法data 2 vec-aqc，用于从无标记语音数据中学习语音表示。我们的目标是在未标记和标记数据都有限的域中改进用于语音的SSL。在最近引入的data 2 vec的基础上，我们向data 2 vec框架引入了额外的模块，这些模块利用了数据扩充、量化表示和聚类的优势。这些模块之间的相互作用有助于解决交叉对比损失作为一个额外的自我监督的目标。在不使用任何语言模型的情况下，data 2 vec-aqc在LibriSpeech的test-clean和test-other集上分别比现有的data 2 vec系统实现了14.1%和20.9%的相对WER改善。当在Switchboard数据上进行微调时，我们提出的模型也实现了比基线data 2 vec高出17.8%的相对WER改善。摘要：In this paper, we propose a new Self-Supervised Learning (SSL) algorithm called data2vec-aqc, for speech representation learning from unlabeled speech data. Our goal is to improve SSL for speech in domains where both unlabeled and labeled data are limited. Building on the recently introduced data2vec, we introduce additional modules to the data2vec framework that leverage the benefit of data augmentations, quantized representations, and clustering. The interaction between these modules helps solve the cross-contrastive loss as an additional self-supervised objective. data2vec-aqc achieves up to 14.1% and 20.9% relative WER improvement over the existing state-of-the-art data2vec system on the test-clean and test-other sets, respectively, of LibriSpeech, without the use of any language model. Our proposed model also achieves up to 17.8% relative WER improvement over the baseline data2vec when fine-tuned on Switchboard data.
【4】 Analysis of Noisy-target Training for DNN-based speech enhancement标题：基于DNN的语音增强中的噪声目标训练分析链接：https://arxiv.org/abs/2211.01198
* 与cs.SD语音【27】为同一篇
作者：Takuya Fujimura,Tomoki Toda机构：Nagoya University, Japan备注：Submitted to ICASSP 2023摘要：基于深度神经网络（DNN）的语音增强通常以干净的语音作为训练目标。然而，很难收集大量干净的语音，因为录音是非常昂贵的。换句话说，当前语音增强的性能受到训练数据量的限制。为了缓和这种限制，已经提出了利用噪声语音作为训练目标的噪声目标训练（NyTT）。虽然实验表明NyTT可以在没有干净语音的情况下训练DNN，但是还没有进行详细的分析，并且还没有很好地理解其行为。本文对NyTT进行了多方面的分析，以加深对NyTT的理解。此外，基于NyTT的特性，我们提出了一种改进的方法，该方法可以与使用干净语音的方法相媲美。此外，我们还表明，我们可以通过使用大量的噪声语音与干净的语音改善性能。摘要：Deep neural network (DNN)-based speech enhancement usually uses a clean speech as a training target. However, it is hard to collect large amounts of clean speech because the recording is very costly. In other words, the performance of current speech enhancement has been limited by the amount of training data. To relax this limitation, Noisy-target Training (NyTT) that utilizes noisy speech as a training target has been proposed. Although it has been experimentally shown that NyTT can train a DNN without clean speech, a detailed analysis has not been conducted and its behavior has not been understood well. In this paper, we conduct various analyses to deepen our understanding of NyTT. In addition, based on the property of NyTT, we propose a refined method that is comparable to the method using clean speech. Furthermore, we show that we can improve the performance by using a huge amount of noisy speech with clean speech.
【5】 Inference and Denoise: Causal Inference-based Neural Speech Enhancement标题：推理与去噪：基于因果推理的神经语音增强链接：https://arxiv.org/abs/2211.01189
* 与cs.SD语音【28】为同一篇
作者：Tsun-An Hsieh,Chao-Han Huck Yang,Pin-Yu Chen,Sabato Marco Siniscalchi,Yu Tsao机构：Research Center for Information Technology Innovation, Academia Sinica, Taiwan, Georgia Institute of Technology, GA, USA; ,IBM Research, NY, USA, Computer Engineering School, Norwegian University of Science and Technology, Norway备注：Submitted to ICASSP 2023摘要：本研究通过将噪声的存在作为一种干预来建模，在因果推理范式中解决了语音增强（SE）任务。基于潜在结果框架，提出了一种基于因果推理的语音增强算法（CISE），该算法利用噪声检测器从噪声语音中分离出干净帧和噪声帧，并将这两组帧分配给两个基于掩蔽的增强模块（EM）进行噪声条件SE。具体地，我们在训练期间使用噪声的存在作为EM选择的指导，并且噪声检测器根据对每一帧的噪声的存在的预测来选择增强模块。此外，我们推导了SE特异性平均治疗效应，以充分量化因果效应。实验结果表明，CISE在所研究的场景中优于基于非因果掩模的SE方法，并且比更复杂的SE模型具有更好的性能和效率。摘要：This study addresses the speech enhancement (SE) task within the causal inference paradigm by modeling the noise presence as an intervention. Based on the potential outcome framework, the proposed causal inference-based speech enhancement (CISE) separates clean and noisy frames in an intervened noisy speech using a noise detector and assigns both sets of frames to two mask-based enhancement modules (EMs) to perform noise-conditional SE. Specifically, we use the presence of noise as guidance for EM selection during training, and the noise detector selects the enhancement module according to the prediction of the presence of noise for each frame. Moreover, we derived a SE-specific average treatment effect to quantify the causal effect adequately. Experimental evidence demonstrates that CISE outperforms a non-causal mask-based SE approach in the studied settings and has better performance and efficiency than more complex SE models.
【6】 I4U System Description for NIST SRE'20 CTS Challenge标题：NIST SRE‘20 CTS挑战赛的I4U系统描述链接：https://arxiv.org/abs/2211.01091
* 与cs.SD语音【29】为同一篇
作者：Kong Aik Lee,Tomi Kinnunen,Daniele Colibro,Claudio Vair,Andreas Nautsch,Hanwu Sun,Liang He,Tianyu Liang,Qiongqiong Wang,Mickael Rouvier,Pierre-Michel Bousquet,Rohan Kumar Das,Ignacio Viñals Bailo,Meng Liu,Héctor Deldago,Xuechen Liu,Md Sahidullah,Sandro Cumani,Boning Zhang,Koji Okabe,Hitoshi Yamamoto,Ruijie Tao,Haizhou Li,Alfonso Ortega Giménez,Longbiao Wang,Luis Buera机构：Institute for Infocomm Research, ASTAR, Singapore, School of Computing, University of Eastern Finland, Finland, Loquendo and Politecnico di Torino, Italy, EURECOM, France and vitas.ai, Germany, Department of Electronic Engineering, Tsinghua University, China备注：SRE 2021, NIST Speaker Recognition Evaluation Workshop, CTS Speaker Recognition Challenge, 14-12 December 2021摘要：本文描述了I4 U提交的2020年NIST说话人识别评估（SRE'20）会话电话语音（CTS）挑战。I4 U的提交是来自八个研究团队的研究人员之间积极合作的结果- I$^2$R（新加坡），UEF（芬兰），VALPT（意大利，西班牙），NEC（日本），THUEE（中国），LIA（法国），NUS（新加坡），INRIA（法国）和TJU（中国）。该提交是基于最佳性能子系统的融合和各团队贡献的子融合系统。已努力使用通用开发和验证集、提交时间表和里程碑，最大限度地减少各研究中心试验列表和评分文件格式的不一致性。摘要：This manuscript describes the I4U submission to the 2020 NIST Speaker Recognition Evaluation (SRE'20) Conversational Telephone Speech (CTS) Challenge. The I4U's submission was resulted from active collaboration among researchers across eight research teams - I$^2$R (Singapore), UEF (Finland), VALPT (Italy, Spain), NEC (Japan), THUEE (China), LIA (France), NUS (Singapore), INRIA (France) and TJU (China). The submission was based on the fusion of top performing sub-systems and sub-fusion systems contributed by individual teams. Efforts have been spent on the use of common development and validation sets, submission schedule and milestone, minimizing inconsistency in trial list and score file format across sites.
【7】 Monolingual Recognizers Fusion for Code-switching Speech Recognition标题：码型转换语音识别中的单语识别器融合链接：https://arxiv.org/abs/2211.01046
* 与cs.SD语音【30】为同一篇
作者：Tongtong Song,Qiang Xu,Haoyu Lu,Longbiao Wang,Hao Shi,Yuqin Lin,Yanbing Yang,Jianwu Dang机构：Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin, China, Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan备注：Submitted to ICASSP2023摘要：双编码器结构在码切换（CS）自动语音识别（ASR）中得到了广泛研究。然而，现有的大多数方法要求两个单语ASR模型的结构相同，并且只使用两个单语ASR模型的编码器。这导致了预先培训的MAM不能及时和充分地用于CS ASR的问题。本文提出了一种面向CS ASR的单语识别器融合方法。它有两个阶段：言语意识阶段和语言融合阶段。在SA阶段，声学特征由两个独立的MAM映射到两个语言特定的预测。为了让MAM专注于他们自己的语言，我们进一步扩展了MAM的语言感知培训策略。在LF阶段，BELM融合两个语言特定的预测以获得最终预测。此外，我们提出了一种文本模拟策略，以简化BELM的训练过程，减少对CS数据的依赖。在汉英语料库上的实验表明了该方法的有效性。在使用开源预训练MAM后，测试集上的混合错误率显著降低。摘要：The bi-encoder structure has been intensively investigated in code-switching (CS) automatic speech recognition (ASR). However, most existing methods require the structures of two monolingual ASR models (MAMs) should be the same and only use the encoder of MAMs. This leads to the problem that pre-trained MAMs cannot be timely and fully used for CS ASR. In this paper, we propose a monolingual recognizers fusion method for CS ASR. It has two stages: the speech awareness (SA) stage and the language fusion (LF) stage. In the SA stage, acoustic features are mapped to two language-specific predictions by two independent MAMs. To keep the MAMs focused on their own language, we further extend the language-aware training strategy for the MAMs. In the LF stage, the BELM fuses two language-specific predictions to get the final prediction. Moreover, we propose a text simulation strategy to simplify the training process of the BELM and reduce reliance on CS data. Experiments on a Mandarin-English corpus show the efficiency of the proposed method. The mix error rate is significantly reduced on the test set after using open-source pre-trained MAMs.
【8】 Adversarial Guitar Amplifier Modelling With Unpaired Data标题：基于非配对数据的对抗性吉他放大器建模链接：https://arxiv.org/abs/2211.00943
* 与cs.SD语音【31】为同一篇
作者：Alec Wright,Vesa Välimäki,Lauri Juvela机构：Acoustics Lab, Dept. of Signal Processing and Acoustics, Aalto University, Espoo, Finland, Neural DSP Technologies, Helsinki, Finland摘要：我们提出了一个音效处理框架，它学习模仿来自录音的目标电吉他音调。我们使用对抗性方法训练深度神经网络，目标是在例如通过吉他放大器应用音频效果处理之后将吉他的音色转换成另一吉他的音色。模型训练不需要配对数据，并且所得到的模型很好地仿真了目标音色，同时能够在现代个人计算机上进行实时处理。为了验证我们的方法，我们提供了两个实验，一个实验使用配对数据进行非配对训练，允许我们通过客观指标来监控训练，另一个实验使用完全非配对数据，对应于用户希望仅使用录音中的音频数据来模拟吉他音色的现实场景。我们的听力测试结果证实了模型在感知上是令人信服的。摘要：We propose an audio effects processing framework that learns to emulate a target electric guitar tone from a recording. We train a deep neural network using an adversarial approach, with the goal of transforming the timbre of a guitar, into the timbre of another guitar after audio effects processing has been applied, for example, by a guitar amplifier. The model training requires no paired data, and the resulting model emulates the target timbre well whilst being capable of real-time processing on a modern personal computer. To verify our approach we present two experiments, one which carries out unpaired training using paired data, allowing us to monitor training via objective metrics, and another that uses fully unpaired data, corresponding to a realistic scenario where a user wants to emulate a guitar timbre only using audio data from a recording. Our listening test results confirm that the models are perceptually convincing.
【9】 Factorized Blank Thresholding for Improved Runtime Efficiency of Neural Transducers标题：用于提高神经传感器运行效率的因式空白阈值链接：https://arxiv.org/abs/2211.00896
* 与cs.SD语音【32】为同一篇
作者：Duc Le,Frank Seide,Yuhao Wang,Yang Li,Kjell Schubert,Ozlem Kalinli,Michael L. Seltzer机构：Meta, USA备注：Submitted to ICASSP 2023摘要：我们展示了分解RNN-T的输出分布如何能够显著降低设备上ASR推断的计算成本和功耗，而不损失准确性。随着用于设备上ASR的RNN-T等神经传感器类型模型的普及，优化RNN-T的运行时效率引起了极大的兴趣。以前的工作主要集中在RNN-T的声编码器和预测器的优化上，而本文将注意力集中在连接器上。我们表明，尽管连接器只是RNN-T的一小部分，但它对整个模型的运行时效率有很大的影响。我们建议将连接符分解为空白和非空白部分，以便在空白概率超过某个阈值时跳过更昂贵的非空白计算。由于可以非常有效地计算空白概率，并且RNN-T输出由空白占主导，因此我们提出的方法导致26-30%的解码速度提高和43-53%的设备上功耗降低，同时始终不会导致准确性降低并且实现相对简单。摘要：We show how factoring the RNN-T's output distribution can significantly reduce the computation cost and power consumption for on-device ASR inference with no loss in accuracy. With the rise in popularity of neural-transducer type models like the RNN-T for on-device ASR, optimizing RNN-T's runtime efficiency is of great interest. While previous work has primarily focused on the optimization of RNN-T's acoustic encoder and predictor, this paper focuses the attention on the joiner. We show that despite being only a small part of RNN-T, the joiner has a large impact on the overall model's runtime efficiency. We propose to factorize the joiner into blank and non-blank portions for the purpose of skipping the more expensive non-blank computation when the blank probability exceeds a certain threshold. Since the blank probability can be computed very efficiently and the RNN-T output is dominated by blanks, our proposed method leads to a 26-30% decoding speed-up and 43-53% reduction in on-device power consumption, all the while incurring no accuracy degradation and being relatively simple to implement.
【10】 Neural Fourier Shift for Binaural Speech Rendering标题：用于双耳语音再现的神经傅里叶变换链接：https://arxiv.org/abs/2211.00878
* 与cs.SD语音【33】为同一篇
作者：Jin Woo Lee,Kyogu Lee机构：Department of Intelligence and Information, Seoul National University备注：Submitted to ICASSP 2023摘要：我们提出了一种神经网络，用于从给定的单耳音频、源的位置和方向渲染双耳语音。以往的研究主要集中在通过调节卷积神经网络在特征空间中的位置和方向来合成双耳语音。这些合成方法在估计目标双耳语音方面是强大的，即使对于野生数据也是如此，但是难以推广用于从分布域外呈现音频。为了缓解这一问题，我们提出了神经傅立叶变换（NFS），一种新颖的网络架构，能够在傅立叶空间中实现双耳语音渲染。具体地，利用基于源和接收器之间的距离的几何时间延迟，训练NFS以预测各种早期反射的延迟和尺度。NFS在内存和计算成本方面都是高效的，是可解释的，并且通过其设计独立于源域运行。实验结果表明，NFS在性能指标评测数据集上的表现优于之前的研究，其内存占用量减少了25倍，计算量减少了6倍。摘要：We present a neural network for rendering binaural speech from given monaural audio, position, and orientation of the source. Most of the previous works have focused on synthesizing binaural speeches by conditioning the positions and orientations in the feature space of convolutional neural networks. These synthesis approaches are powerful in estimating the target binaural speeches even for in-the-wild data but are difficult to generalize for rendering the audio from out-of-distribution domains. To alleviate this, we propose Neural Fourier Shift (NFS), a novel network architecture that enables binaural speech rendering in the Fourier space. Specifically, utilizing a geometric time delay based on the distance between the source and the receiver, NFS is trained to predict the delays and scales of various early reflections. NFS is efficient in both memory and computational cost, is interpretable, and operates independently of the source domain by its design. With up to 25 times lighter memory and 6 times fewer calculations, the experimental results show that NFS outperforms the previous studies on the benchmark dataset.
【11】 LMD: A Learnable Mask Network to Detect Adversarial Examples for Speaker Verification标题：LMD：一种用于说话人确认的可学习掩码网络链接：https://arxiv.org/abs/2211.00825
* 与cs.SD语音【34】为同一篇
作者：Xing Chen,Jie Wang,Xiao-Lei Zhang,Wei-Qiang Zhang,Kunde Yang备注：13 pages, 9 figures摘要：近年来，对抗性攻击严重威胁着自动说话人确认（ASV）系统的安全性，但目前已经有了一些应对措施。然而，许多防御方法不仅需要攻击者的先验知识，而且可解释性较弱。针对这一问题，本文提出了一种与攻击者无关且可解释的方法--可学习掩码检测器（learnable mask detector，LMD），用于区分攻击者的攻击样本和真实样本。它利用分数变化作为指示器来检测对抗性示例，其中分数变化是原始音频记录的ASV分数与从其掩蔽的复杂谱图合成的其变换音频之间的绝对差异。乐谱变化检测器的核心部件是通过神经网络生成掩蔽谱图。神经网络只需要真实的例子来训练，这使它成为一种攻击者独立的方法。其可解释性在于，神经网络的训练目标是最小化目标ASV的分数变化，并最大化真实训练样本的掩蔽谱图bin数。它的基础是基于这样的观察，即用很少的说话者信息掩盖绝大多数的谱图箱将不可避免地向对抗性示例引入大的分数变化，而向真实示例引入小的分数变化。在12个攻击者和两个典型的ASV系统上的实验结果表明，该方法的性能优于5个现有的基准.大量的实验结果也可以作为基于检测的ASV防御的基准。摘要：Although the security of automatic speaker verification (ASV) is seriously threatened by recently emerged adversarial attacks, there have been some countermeasures to alleviate the threat. However, many defense approaches not only require the prior knowledge of the attackers but also possess weak interpretability. To address this issue, in this paper, we propose an attacker-independent and interpretable method, named learnable mask detector (LMD), to separate adversarial examples from the genuine ones. It utilizes score variation as an indicator to detect adversarial examples, where the score variation is the absolute discrepancy between the ASV scores of an original audio recording and its transformed audio synthesized from its masked complex spectrogram. A core component of the score variation detector is to generate the masked spectrogram by a neural network. The neural network needs only genuine examples for training, which makes it an attacker-independent approach. Its interpretability lies that the neural network is trained to minimize the score variation of the targeted ASV, and maximize the number of the masked spectrogram bins of the genuine training examples. Its foundation is based on the observation that, masking out the vast majority of the spectrogram bins with little speaker information will inevitably introduce a large score variation to the adversarial example, and a small score variation to the genuine example. Experimental results with 12 attackers and two representative ASV systems show that our proposed method outperforms five state-of-the-art baselines. The extensive experimental results can also be a benchmark for the detection-based ASV defenses.
【12】 InterMPL: Momentum Pseudo-Labeling with Intermediate CTC Loss标题：InterMPL：具有中间CTC损失的动量伪标记链接：https://arxiv.org/abs/2211.00795
* 与cs.SD语音【35】为同一篇
作者：Yosuke Higuchi,Tetsuji Ogawa,Tetsunori Kobayashi,Shinji Watanabe机构：Carnegie Mellon University, USA ,Waseda University, Japan备注：Submitted to ICASSP2023摘要：本文提出了一种半监督学习的端到端自动语音识别方法InterMPL，该方法在中间监督下进行伪标注。Momentum PL（MPL）通过动态生成伪标签并提高其质量，在未标记数据上训练基于连接主义时态分类（CTC）的模型。与诸如基于注意力的编码器-解码器和转换器的自回归公式相比，CTC非常适合于MPL，或通常基于PL的半监督ASR，这归因于其简单/快速的推理算法和对生成折叠标签的鲁棒性。然而，由于条件独立性假设，CTC通常产生比自回归模型差的性能，从而限制了MPL的性能。我们建议通过引入中间损耗来增强MPL，这是受基于CTC建模的最新进展的启发。特别地，我们关注自条件和分层条件CTC，它们将辅助CTC损失应用于中间层，使得条件独立性假设显式地放松。我们还探讨了如何生成伪标签，并将其用于监督中间损失。在不同半监督环境下的实验结果表明，与MPL算法相比，该算法提高了ASR模型12.1%的绝对性能增益.此外，我们的详细分析验证了中间损失的重要性。摘要：This paper presents InterMPL, a semi-supervised learning method of end-to-end automatic speech recognition (ASR) that performs pseudo-labeling (PL) with intermediate supervision. Momentum PL (MPL) trains a connectionist temporal classification (CTC)-based model on unlabeled data by continuously generating pseudo-labels on the fly and improving their quality. In contrast to autoregressive formulations, such as the attention-based encoder-decoder and transducer, CTC is well suited for MPL, or PL-based semi-supervised ASR in general, owing to its simple/fast inference algorithm and robustness against generating collapsed labels. However, CTC generally yields inferior performance than the autoregressive models due to the conditional independence assumption, thereby limiting the performance of MPL. We propose to enhance MPL by introducing intermediate loss, inspired by the recent advances in CTC-based modeling. Specifically, we focus on self-conditional and hierarchical conditional CTC, that apply auxiliary CTC losses to intermediate layers such that the conditional independence assumption is explicitly relaxed. We also explore how pseudo-labels should be generated and used as supervision for intermediate losses. Experimental results in different semi-supervised settings demonstrate that the proposed approach outperforms MPL and improves an ASR model by up to a 12.1% absolute performance gain. In addition, our detailed analysis validates the importance of the intermediate loss.
【13】 BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder标题：BECTRA：采用BERT增强编码器的基于传感器的端到端ASR链接：https://arxiv.org/abs/2211.00792
* 与cs.SD语音【36】为同一篇
作者：Yosuke Higuchi,Tetsuji Ogawa,Tetsunori Kobayashi,Shinji Watanabe机构：Waseda University, Japan ,Carnegie Mellon University, USA备注：Submitted to ICASSP2023摘要：提出了一种基于BERT-CTC-Transducer（BECTRA）的端到端自动语音识别（E2 E-ASR）模型，该模型是由一个带有BERT增强编码器的换能器构成的。将大规模预训练语言模型（LM）集成到E2 E-ASR中是一个积极的研究方向，旨在利用通用的语言学知识生成准确的文本。使这种整合具有挑战性的一个关键因素在于词汇不匹配;为预先训练的LM构造的词汇表对于E2 E-ASR训练来说通常太大，并且很可能与目标ASR域不匹配。为了克服这样的问题，我们提出了BECTRA，它是我们以前的BERT-CTC的扩展版本，它使用感兴趣的词汇表实现基于BERT的E2 E-ASR。BECTRA是一个基于转换器的模型，它采用BERT-CTC作为编码器，并使用适合于目标任务的词汇表训练ASR专用解码器。通过换能器和BERT-CTC的结合，我们还提出了一种新的推理算法，以利用自回归和非自回归解码的优点。在不同数据量、说话风格和语言的多个ASR任务上的实验结果表明，BECTRA在利用BERT知识的同时有效地处理了词汇失配问题，比BERT-CTC具有更好的性能。摘要：We present BERT-CTC-Transducer (BECTRA), a novel end-to-end automatic speech recognition (E2E-ASR) model formulated by the transducer with a BERT-enhanced encoder. Integrating a large-scale pre-trained language model (LM) into E2E-ASR has been actively studied, aiming to utilize versatile linguistic knowledge for generating accurate text. One crucial factor that makes this integration challenging lies in the vocabulary mismatch; the vocabulary constructed for a pre-trained LM is generally too large for E2E-ASR training and is likely to have a mismatch against a target ASR domain. To overcome such an issue, we propose BECTRA, an extended version of our previous BERT-CTC, that realizes BERT-based E2E-ASR using a vocabulary of interest. BECTRA is a transducer-based model, which adopts BERT-CTC for its encoder and trains an ASR-specific decoder using a vocabulary suitable for a target task. With the combination of the transducer and BERT-CTC, we also propose a novel inference algorithm for taking advantage of both autoregressive and non-autoregressive decoding. Experimental results on several ASR tasks, varying in amounts of data, speaking styles, and languages, demonstrate that BECTRA outperforms BERT-CTC by effectively dealing with the vocabulary mismatch while exploiting BERT knowledge.
【14】 Predicting phoneme-level prosody latents using AR and flow-based Prior Networks for expressive speech synthesis标题：基于AR和基于流的先验网络预测表现性语音合成的音素级韵律潜伏期链接：https://arxiv.org/abs/2211.01327
* 与cs.SD语音【1】为同一篇
作者：Konstantinos Klapsas,Karolos Nikitaras,Nikolaos Ellinas,June Sig Sung,Inchul Hwang,Spyros Raptis,Aimilios Chalamandaris,Pirros Tsiakoulis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：表达性语音合成文献的大部分集中于学习语音信号的韵律表示，然后在推理期间通过先验分布来建模该韵律表示。在这篇文章中，我们比较了不同的先验结构在预测音素级韵律表示的任务中提取一个无监督的FVAE模型。我们使用主观和客观的度量来表明，基于先验网络的归一化流可以以质量的轻微下降为代价产生更有表达力的语音。此外，我们还表明，由于归一化流的性质，对于给定的文本，合成语音具有更高的可变性。我们还提出了一种动态VAE模型，与基于流的模型相比，该模型可以产生更高质量的语音，尽管表达性和可变性有所下降。摘要：A large part of the expressive speech synthesis literature focuses on learning prosodic representations of the speech signal which are then modeled by a prior distribution during inference. In this paper, we compare different prior architectures at the task of predicting phoneme level prosodic representations extracted with an unsupervised FVAE model. We use both subjective and objective metrics to show that normalizing flow based prior networks can result in more expressive speech at the cost of a slight drop in quality. Furthermore, we show that the synthesized speech has higher variability, for a given text, due to the nature of normalizing flows. We also propose a Dynamical VAE model, that can generate higher quality speech although with decreased expressiveness and variability compared to the flow based models.
【15】 Low-Resource Music Genre Classification with Advanced Neural Model Reprogramming标题：基于高级神经模型重编程的低资源音乐流派分类链接：https://arxiv.org/abs/2211.01317
* 与cs.SD语音【2】为同一篇
作者：Yun-Ning Hung,Chao-Han Huck Yang,Pin-Yu Chen,Alexander Lerch机构： Georgia Institute of Technology, Atlanta, GA, USA, IBM Research, Yorktown Heights, NY, USA备注：Submitted to ICASSP 2023. Some experimental results were reduced due to the space limit. The implementation will be available at this https URL摘要：迁移学习（TL）方法在处理训练数据有限的任务时表现出良好的效果。然而，利用目标域数据对预先训练的神经网络进行微调通常需要相当大的存储器和计算资源。在这项工作中，我们介绍了一种基于神经模型重编程（NMR）概念的利用预训练模型进行低资源（音乐）分类的新方法。NMR的目的在于通过修改冻结的预训练模型的输入来将预训练模型从源域重新用于目标域。除了已知的、输入无关的重编程方法之外，我们提出了一种高级重编程范例：输入依赖型NMR，可提高对音乐音频等复杂输入数据的适应性。实验结果表明，在大规模数据集上预训练的神经网络模型可以成功地利用这种重编程方法进行音乐流派分类。在小的体裁分类数据集上，两种基于输入的NMR TL方法的性能优于基于微调的TL方法.摘要：Transfer learning (TL) approaches have shown promising results when handling tasks with limited training data. However, considerable memory and computational resources are often required for fine-tuning pre-trained neural networks with target domain data. In this work, we introduce a novel method for leveraging pre-trained models for low-resource (music) classification based on the concept of Neural Model Reprogramming (NMR). NMR aims at re-purposing a pre-trained model from a source domain to a target domain by modifying the input of a frozen pre-trained model. In addition to the known, input-independent, reprogramming method, we propose an advanced reprogramming paradigm: Input-dependent NMR, to increase adaptability to complex input data such as musical audio. Experimental results suggest that a neural model pre-trained on large-scale datasets can successfully perform music genre classification by using this reprogramming method. The two proposed Input-dependent NMR TL methods outperform fine-tuning-based TL methods on a small genre classification dataset.
【16】 A Quantum Kernel Learning Approach to Acoustic Modeling for Spoken Command Recognition标题：用于口述命令识别的声学建模的量子核学习方法链接：https://arxiv.org/abs/2211.01263
* 与cs.SD语音【3】为同一篇
作者：Chao-Han Huck Yang,Bo Li,Yu Zhang,Nanxin Chen,Tara N. Sainath,Sabato Marco Siniscalchi,Chin-Hui Lee机构： Georgia Institute of Technology, USA, Google, USA, Kore University of Enna, Italy, Department of Electronic Systems, NTNU, Trondheim, Norway备注：Submitted to ICASSP 2023摘要：提出一种量子核学习（quantum kernel learning，QKL）框架，以解决低资源环境下训练大规模声学模型时经常遇到的固有数据稀疏问题.我们基于经典到量子特征编码来投影声学特征。与现有的量子卷积技术不同，我们利用量子空间中的特征QKL来设计基于核的分类器。在阿拉伯语、格鲁吉亚语、楚瓦什语和立陶宛语等低资源语言的语音命令识别任务上的实验结果表明，所提出的基于QKL的混合方法较现有的经典方法和量子方法有很好的改进.摘要：We propose a quantum kernel learning (QKL) framework to address the inherent data sparsity issues often encountered in training large-scare acoustic models in low-resource scenarios. We project acoustic features based on classical-to-quantum feature encoding. Different from existing quantum convolution techniques, we utilize QKL with features in the quantum space to design kernel-based classifiers. Experimental results on challenging spoken command recognition tasks for a few low-resource languages, such as Arabic, Georgian, Chuvash, and Lithuanian, show that the proposed QKL-based hybrid approach attains good improvements over existing classical and quantum solutions.
【17】 Audio Language Modeling using Perceptually-Guided Discrete Representations标题：基于感知引导离散表示的音频语言建模链接：https://arxiv.org/abs/2211.01223
* 与cs.SD语音【4】为同一篇
作者：Felix Kreuk,Yaniv Taigman,Adam Polyak,Jade Copet,Gabriel Synnaeve,Alexandre Défossez,Yossi Adi机构：Meta AI Research, The Hebrew University of Jerusalem, Israel摘要：在这项工作中，我们研究了音频语言建模的任务，其中我们的目标是学习音频的概率模型，可以用于生成和完成。我们使用最先进的感知引导音频压缩模型，将音频编码为离散表示。接下来，我们使用这些表示来训练基于转换器的因果语言模型。在推理时，我们通过将音频提示编码为离散序列、将其馈送到音频语言模型、从该模型采样并合成相应的时域信号来执行音频自动完成。我们在Audioset（迄今为止最大的通用音频数据集）上评估了我们的方法生成的样本质量，结果表明它优于所评估的基准音频编码器。此外，我们还提供了广泛的分析，以更好地理解音频质量和语言建模能力之间的权衡。样品：链接。摘要：In this work, we study the task of Audio Language Modeling, in which we aim at learning probabilistic models for audio that can be used for generation and completion. We use a state-of-the-art perceptually-guided audio compression model, to encode audio to discrete representations. Next, we train a transformer-based causal language model using these representations. At inference time, we perform audio auto-completion by encoding an audio prompt as a discrete sequence, feeding it to the audio language model, sampling from the model, and synthesizing the corresponding time-domain signal. We evaluate the quality of samples generated by our method on Audioset, the largest dataset for general audio to date, and show that it is superior to the evaluated baseline audio encoders. We additionally provide an extensive analysis to better understand the trade-off between audio-quality and language-modeling capabilities. Samples:link.
【18】 Transformer-based encoder-encoder architecture for Spoken Term Detection标题：一种基于Transformer的语音检测编码器体系结构链接：https://arxiv.org/abs/2211.01089
* 与cs.SD语音【5】为同一篇
作者：Jan Švec,Luboš Šmídl,Jan Lehečka机构： Jan LeheˇckaDepartment of Cybernetics, University of West Bohemia备注：Submitted to ICASSP 2023摘要：提出了一种基于Transformer架构的口语术语检测方法。我们提出了编码器-编码器架构，采用两个类BERT编码器，并进行了额外的修改，包括卷积和上采样层、注意力掩蔽和共享参数。编码器将识别的假设和搜索的项投影到共享嵌入空间中，其中使用校准的点积来计算推定命中的得分。在实验中，我们使用了Wav 2 Vec 2.0语音识别器，在基于USC Shoah Foundation Visual History Archive（MALACH）的英语和捷克语STD数据集上，我们提出的系统优于基于深度LSTM的基线方法。摘要：The paper presents a method for spoken term detection based on the Transformer architecture. We propose the encoder-encoder architecture employing two BERT-like encoders with additional modifications, including convolutional and upsampling layers, attention masking, and shared parameters. The encoders project a recognized hypothesis and a searched term into a shared embedding space, where the score of the putative hit is computed using the calibrated dot product. In the experiments, we used the Wav2Vec 2.0 speech recognizer, and the proposed system outperformed a baseline method based on deep LSTMs on the English and Czech STD datasets based on USC Shoah Foundation Visual History Archive (MALACH).
【19】 DSPGAN: a GAN-based universal vocoder for high-fidelity TTS by time-frequency domain supervision from DSP标题：DSPGAN：一种基于GAN的时频域监控高保真TTS通用声码器链接：https://arxiv.org/abs/2211.01087
* 与cs.SD语音【6】为同一篇
作者：Kun Song,Yongmao Zhang,Yi Lei,Jian Cong,Hanzhao Li,Lei Xie,Gang He,Jinfeng Bai机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, TAL Education Group, Beijing, China备注：Submitted to ICASSP 2023摘要：近年来，基于生成式对抗神经网络（GAN）的神经声码器的发展显示出其以Mel谱图为条件产生原始波形的优势，具有快速的推理速度和轻量级的网络结构。然而，训练一个通用的神经声码器仍然具有挑战性，该声码器可以合成来自各种场景的高保真语音，这些场景具有不可见的说话者、语言和说话风格。本文提出了一种基于GAN的高保真语音合成通用声码器DSPGAN，它利用数字信号处理（DSP）的时频域监督功能。为了消除训练阶段的真实谱图与推理阶段的预测谱图之间的不匹配问题，采用从DSP模块产生的波形中提取的Mel谱图，而不是从TTS声学模型中预测的Mel谱图，作为基于GAN的声码器的时频域监督。我们也利用正弦激励作为时域监督，以改善谐波建模，并消除各种人为因素的GAN声码器。实验结果表明，DSPGAN的性能明显优于对比方法，能够基于TTS中的各种数据生成高保真语音。摘要：Recent development of neural vocoders based on the generative adversarial neural network (GAN) has shown their advantages of generating raw waveform conditioned on mel-spectrogram with fast inference speed and lightweight networks. Whereas, it is still challenging to train a universal neural vocoder that can synthesize high-fidelity speech from various scenarios with unseen speakers, languages, and speaking styles. In this paper, we propose DSPGAN, a GAN-based universal vocoder for high-fidelity speech synthesis by applying the time-frequency domain supervision from digital signal processing (DSP). To eliminate the mismatch problem caused by the ground-truth spectrograms in training phase and the predicted spectrograms in inference phase, we leverage the mel-spectrogram extracted from the waveform generated by a DSP module, rather than the predicted mel-spectrogram from the Text-to-Speech (TTS) acoustic model, as the time-frequency domain supervision to the GAN-based vocoder. We also utilize sine excitation as the time-domain supervision to improve the harmonic modeling and eliminate various artifacts of the GAN-based vocoder. Experimental results show that DSPGAN significantly outperforms the compared approaches and can generate high-fidelity speech based on diverse data in TTS.
【20】 Intermediate Fine-Tuning Using Imperfect Synthetic Speech for Improving Electrolaryngeal Speech Recognition标题：改进喉电语音识别的不完全合成语音中级微调链接：https://arxiv.org/abs/2211.01079
* 与cs.SD语音【7】为同一篇
作者：Lester Phillip Violeta,Ding Ma,Wen-Chin Huang,Tomoki Toda机构：Nagoya University, Japan备注：Submitted to ICASSP 2023摘要：由于数据集较小，用于电子喉式扬声器的自动语音识别（ASR）系统的研究相对未被探索。当ASR缺乏训练数据时，大规模的预训练和微调框架通常足以实现高识别率;然而在喉电语音中，预训练和微调数据之间的域移位太大而无法克服，从而限制了识别率的最大改进。为了解决这个问题，我们提出了一个中间微调步骤，它使用不完美的合成语音来关闭预训练和目标数据之间的域移位间隙。尽管有不完美的合成数据，但我们在喉电语音数据集上显示了这种方法的有效性，与未使用不完美合成语音的基线相比，改善了6.1%。结果显示，中间微调阶段如何集中于学习不完美的合成数据的高级固有特征，而不是诸如可懂度的低级特征。摘要：Research on automatic speech recognition (ASR) systems for electrolaryngeal speakers has been relatively unexplored due to small datasets. When training data is lacking in ASR, a large-scale pretraining and fine tuning framework is often sufficient to achieve high recognition rates; however, in electrolaryngeal speech, the domain shift between the pretraining and fine-tuning data is too large to overcome, limiting the maximum improvement of recognition rates. To resolve this, we propose an intermediate fine-tuning step that uses imperfect synthetic speech to close the domain shift gap between the pretraining and target data. Despite the imperfect synthetic data, we show the effectiveness of this on electrolaryngeal speech datasets, with improvements of 6.1% over the baseline that did not use imperfect synthetic speech. Results show how the intermediate fine-tuning stage focuses on learning the high-level inherent features of the imperfect synthetic data rather than the low-level features such as intelligibility.
【21】 Singing Voice Synthesis with Vibrato Modeling and Latent Energy Representation标题：基于颤音建模和势能表示的歌声合成链接：https://arxiv.org/abs/2211.00996
* 与cs.SD语音【8】为同一篇
作者：Yingjie Song,Wei Song,Wei Zhang,Zhengchen Zhang,Dan Zeng,Zhi Liu,Yang Yu机构： School of Communication and Information Engineering, Shanghai University, Shanghai, China, JD AI Research, Beijing, China, Shanghai Conservatory of Music, Shanghai, China摘要：通过引入显式颤音建模和潜在能量表示，提出了一个富有表现力的歌唱声合成系统。由于人类歌唱的固有特性，颤音对于合成声音的自然性至关重要。为此，本文提出了一种基于深度学习的颤音模型，用于控制歌唱中颤音的似然度、速率、深度和相位，其中颤音似然度表示颤音存在的概率，有助于提高歌唱声音的自然度。事实上，现有的歌唱语料库中并没有关于颤音似然性的标注。采用一种新的颤音相似度标注方法对颤音相似度进行自动标注。同时，音频的功率谱图包含丰富的信息，可以提高歌唱的表现力。提出了一种基于自动编码器的潜在能量瓶颈特征，用于表达性歌唱语音合成。在开放数据集NUS48E上的实验结果表明，颤音建模和潜在能量表示都能显著提高歌唱声音的表现力。演示网站中显示了音频样本。摘要：This paper proposes an expressive singing voice synthesis system by introducing explicit vibrato modeling and latent energy representation. Vibrato is essential to the naturalness of synthesized sound, due to the inherent characteristics of human singing. Hence, a deep learning-based vibrato model is introduced in this paper to control the vibrato's likeliness, rate, depth and phase in singing, where the vibrato likeliness represents the existence probability of vibrato and it would help improve the singing voice's naturalness. Actually, there is no annotated label about vibrato likeliness in existing singing corpus. We adopt a novel vibrato likeliness labeling method to label the vibrato likeliness automatically. Meanwhile, the power spectrogram of audio contains rich information that can improve the expressiveness of singing. An autoencoder-based latent energy bottleneck feature is proposed for expressive singing voice synthesis. Experimental results on the open dataset NUS48E show that both the vibrato modeling and the latent energy representation could significantly improve the expressiveness of singing voice. The audio samples are shown in the demo website.
【22】 Weighted variance variational autoencoder for speech enhancement标题：用于语音增强的加权方差变分自动编码器链接：https://arxiv.org/abs/2211.00990
* 与cs.SD语音【9】为同一篇
作者：Ali Golmakani,Mostafa Sadeghi,Xavier Alameda-Pineda,Romain Serizel机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France, Inria Grenoble & Univ. Grenoble Alpes, France摘要：本文提出了一种基于变分自编码器的语音增强方法，它包括在时频域中学习语音先验分布。对于生成模型，通常假设零均值复值高斯分布，其中语音信息被编码在作为潜在变量的函数的方差中。虽然这是常用的方法，但本文提出了一种加权方差生成模型，其中每个TF点在参数学习中的贡献是加权的。我们在权重上施加Gamma先验分布，这将有效地导致Student的t分布而不是用于语音建模的高斯分布。我们基于所提出的生成模型开发了高效的训练和语音增强算法。在声谱图建模和语音增强实验中，与标准的未加权方差模型相比，该方法具有较好的鲁棒性和有效性。摘要：We address speech enhancement based on variational autoencoders, which involves learning a speech prior distribution in the time-frequency (TF) domain. A zero-mean complexvalued Gaussian distribution is usually assumed for the generative model, where the speech information is encoded in the variance as a function of a latent variable. While this is the commonly used approach, in this paper we propose a weighted variance generative model, where the contribution of each TF point in parameter learning is weighted. We impose a Gamma prior distribution on the weights, which would effectively lead to a Student's t-distribution instead of Gaussian for speech modeling. We develop efficient training and speech enhancement algorithms based on the proposed generative model. Our experimental results on spectrogram modeling and speech enhancement demonstrate the effectiveness and robustness of the proposed approach compared to the standard unweighted variance model.
【23】 Audio-visual speech enhancement with a deep Kalman filter generative model标题：基于深度卡尔曼滤波生成模型的视听语音增强链接：https://arxiv.org/abs/2211.00988
* 与cs.SD语音【10】为同一篇
作者：Ali Golmakani,Mostafa Sadeghi,Romain Serizel机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：基于变分自动编码器（VAE）的深隐变量生成模型在视听语音增强（AVSE）中表现出良好的性能。其基本思想是学习干净语音数据的基于VAE的视听先验分布，然后将其与统计噪声模型相结合，以从目标说话人的噪声音频记录和视频（唇图像）中恢复语音信号。现有的针对AVSE开发的生成模型没有考虑语音数据的顺序特性，这阻止了它们完全结合视觉数据的能力。本文提出了一种视听深度卡尔曼滤波（AV-DKF）生成模型，该模型假设隐变量为一阶马尔可夫链模型，并有效地融合视听数据。此外，我们开发了一种有效的推理方法来估计测试时间的语音信号。我们进行了一组实验来比较用于语音增强的生成模型的不同变体。实验结果表明，AV-DKF模型与纯音频模型、非序列纯音频模型和基于视听VAE的模型相比具有更好的性能。摘要：Deep latent variable generative models based on variational autoencoder (VAE) have shown promising performance for audiovisual speech enhancement (AVSE). The underlying idea is to learn a VAEbased audiovisual prior distribution for clean speech data, and then combine it with a statistical noise model to recover a speech signal from a noisy audio recording and video (lip images) of the target speaker. Existing generative models developed for AVSE do not take into account the sequential nature of speech data, which prevents them from fully incorporating the power of visual data. In this paper, we present an audiovisual deep Kalman filter (AV-DKF) generative model which assumes a first-order Markov chain model for the latent variables and effectively fuses audiovisual data. Moreover, we develop an efficient inference methodology to estimate speech signals at test time. We conduct a set of experiments to compare different variants of generative models for speech enhancement. The results demonstrate the superiority of the AV-DKF model compared with both its audio-only version and the non-sequential audio-only and audiovisual VAE-based models.
【24】 SpectroMap: Peak detection algorithm for audio fingerprinting标题：SpectrMap：音频指纹的峰值检测算法链接：https://arxiv.org/abs/2211.00982
* 与cs.SD语音【11】为同一篇
作者：Aarón López-García机构： Peak detection algorithm for audio fingerprintingAar´on L´opez-Garc´ıaDepartment of Computer ScienceUniversity of Valencialogara8备注：7 pages, 3 figures摘要：我们介绍了SpectroMap，一个用Python编程语言编写的开源GitHub音频指纹库。它由一个峰值搜索算法组成，该算法通过时-频带从谱图中提取拓扑突出。本文通过在高质量城市声音数据集和环境音频记录中的两个实验应用，介绍了该算法的工作原理以及它在处理输入数据方面的有效性。摘要：We present SpectroMap, an open source GitHub repository for audio fingerprinting written in Python programming language. It is composed of a peak search algorithm that extracts topological prominences from a spectrogram via time-frequency bands. In this paper, we introduce the algorithm functioning with two experimental applications in a high-quality urban sound dataset and environmental audio recordings to describe how it works and how effective it is in handling the input data.
【25】 Internal Language Model Estimation based Adaptive Language Model Fusion for Domain Adaptation标题：基于内部语言模型估计的领域自适应语言模型融合链接：https://arxiv.org/abs/2211.00968
* 与cs.SD语音【12】为同一篇
作者：Rao Ma,Xiaobo Wu,Jin Qiu,Yanan Qin,Haihua Xu,Peihao Wu,Zejun Ma机构：ByteDance AI Lab备注：submitted to ICASSP 2023摘要：ASR模型的部署环境是不断变化的，并且在会话期间传入的语音可以跨不同的域切换。这给只有目标领域文本数据时的有效领域自适应带来了挑战，我们的目标是在目标领域上获得明显改进的性能，同时在一般领域上的性能受到较小的损害。本文提出了一种基于内部语言模型估计的自适应域自适应（ILME-ADA）LM融合方法。为了实现这样的ILME-ADA，基于分别来自内部LM和外部LM（ELM）的得分的最大值来计算插值的对数似然性得分。在两个特定领域（目标）测试集上，我们分别使用神经网络和n-gram LM作为ELM，使用RNN-T和LAS建模框架验证了所提出的ILME-ADA方法的有效性。与浅层LM融合方法和基于ILME的LM融合方法相比，该方法在目标测试集上的性能明显提高，而在一般测试集上的性能下降最小。摘要：ASR model deployment environment is ever-changing, and the incoming speech can be switched across different domains during a session. This brings a challenge for effective domain adaptation when only target domain text data is available, and our objective is to obtain obviously improved performance on the target domain while the performance on the general domain is less undermined. In this paper, we propose an adaptive LM fusion approach called internal language model estimation based adaptive domain adaptation (ILME-ADA). To realize such an ILME-ADA, an interpolated log-likelihood score is calculated based on the maximum of the scores from the internal LM and the external LM (ELM) respectively. We demonstrate the efficacy of the proposed ILME-ADA method with both RNN-T and LAS modeling frameworks employing neural network and n-gram LMs as ELMs respectively on two domain specific (target) test sets. The proposed method can achieve significantly better performance on the target test sets while it gets minimal performance degradation on the general test set, compared with both shallow and ILME-based LM fusion methods.
【26】 Multi-Speaker Multi-Style Speech Synthesis with Timbre and Style Disentanglement标题：具有音色和风格解缠的多说话人多风格语音合成链接：https://arxiv.org/abs/2211.00967
* 与cs.SD语音【13】为同一篇
作者：Wei Song,Yanghao Yue,Ya-jie Zhang,Zhengchen Zhang,Youzheng Wu,Xiaodong He机构：JD Technology Group摘要：在多说话者多风格的文语转换（TTS）场景中，说话者音色和风格的解纠缠对于风格迁移非常重要。随着音色和风格的分离，TTS系统可以合成具有任何风格的特定说话人的表达性语音。然而，目前关于音色与风格解缠的研究还存在一些不足。目前的方法要么需要单个说话者的多风格录音，这是困难和昂贵的收集，或使用复杂的网络和复杂的训练方法，这是难以再现和控制风格转移行为。为了提高音色和风格解纠缠的有效性，消除对单说话人多风格语料库的依赖，提出了一种简单有效的音色和风格解纠缠方法。采用FastSpeech2网络作为主干网络，以明确的时长、音高和能量轨迹来表示风格。将每个说话人的数据视为一个独立的风格，然后在FastSpeech2网络中加入说话人嵌入和风格嵌入，以学习解纠缠的表示。利用语音层次基音和能量归一化来改善去耦合效果。实验结果表明，该模型能够合成具有任意风格的语音，且具有较高的风格相似度，同时保持了较高的说话人相似度。摘要：Disentanglement of a speaker's timbre and style is very important for style transfer in multi-speaker multi-style text-to-speech (TTS) scenarios. With the disentanglement of timbres and styles, TTS systems could synthesize expressive speech for a given speaker with any style which has been seen in the training corpus. However, there are still some shortcomings with the current research on timbre and style disentanglement. The current method either requires single-speaker multi-style recordings, which are difficult and expensive to collect, or uses a complex network and complicated training method, which is difficult to reproduce and control the style transfer behavior. To improve the disentanglement effectiveness of timbres and styles, and to remove the reliance on single-speaker multi-style corpus, a simple but effective timbre and style disentanglement method is proposed in this paper. The FastSpeech2 network is employed as the backbone network, with explicit duration, pitch, and energy trajectory to represent the style. Each speaker's data is considered as a separate and isolated style, then a speaker embedding and a style embedding are added to the FastSpeech2 network to learn disentangled representations. Utterance level pitch and energy normalization are utilized to improve the decoupling effect. Experimental results demonstrate that the proposed model could synthesize speech with any style seen during training with high style similarity while maintaining very high speaker similarity.
【27】 Fast-U2++: Fast and Accurate End-to-End Speech Recognition in Joint CTC/Attention Frames标题：Fast-U2++：联合CTC/注意帧中快速准确的端到端语音识别链接：https://arxiv.org/abs/2211.00941
* 与cs.SD语音【14】为同一篇
作者：Chengdong Liang,Xiao-Lei Zhang,BinBin Zhang,Di Wu,Shengqiang Li,Xingchen Song,Zhendong Peng,Fuping Pan机构：School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an, China, Horizon Robotics, Beijing, China ,WeNet Open Source Community备注：5 pages, 3 figures摘要：近年来，统一的流式和非流式双通道（U2/U2++）端到端语音识别模型在流式能力、准确性和延迟方面表现出了很好的性能。本文提出了快速U2 ++，它是U2++的一个增强版本，进一步降低了部分延迟。fast-U2++的核心思想是在其编码器中用小块输出底层的部分结果，而在其编码器的顶层使用大块来补偿由小块引起的性能下降。此外，我们还使用知识提取的方法来减少令牌的发送延迟。我们在Aishell-1数据集上进行了大量的实验。实验和消融研究表明，与U2++相比，fast-U2++将模型延迟从320 ms减少到80 ms，并在流设置下实现了5.06%的字符错误率（CER）。摘要：Recently, the unified streaming and non-streaming two-pass (U2/U2++) end-to-end model for speech recognition has shown great performance in terms of streaming capability, accuracy and latency. In this paper, we present fast-U2++, an enhanced version of U2++ to further reduce partial latency. The core idea of fast-U2++ is to output partial results of the bottom layers in its encoder with a small chunk, while using a large chunk in the top layers of its encoder to compensate the performance degradation caused by the small chunk. Moreover, we use knowledge distillation method to reduce the token emission latency. We present extensive experiments on Aishell-1 dataset. Experiments and ablation studies show that compared to U2++, fast-U2++ reduces model latency from 320ms to 80ms, and achieves a character error rate (CER) of 5.06% with a streaming setup.
【28】 SpeechBlender: Speech Augmentation Framework for Mispronunciation Data Generation标题：SpeechBlender：用于错误发音数据生成的语音增强框架链接：https://arxiv.org/abs/2211.00923
* 与cs.SD语音【15】为同一篇
作者：Yassine El Kheir,Shammur Absar Chowdhury,Hamdy Mubarak,Shazia Afzal,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, submitted to ICASSP 2023摘要：设计错误发音检测模型的最大挑战之一是标记的L2语音数据的不可用性。为了克服这样的数据不足，我们引入了SpeechBlender --一个细粒度的数据扩充管道，用于生成错误发音错误。SpeechBlender利用各种遮罩来针对语音单元的不同区域，并使用混合因子来线性内插原始语音信号，同时产生错误的发音实例。屏蔽有利于信号的平滑混合，从而产生比"剪切/粘贴“方法更有效的样本。我们在音素级发音质量评估任务中展示了我们的增强技术的有效性，仅利用良好的发音数据集。对于Speechcean 762测试集，与没有语音增强和发音增强的情况相比，我们观察到SpeechBlender增强的Pearson相关系数（PCC）分别增加了3%和2%。此外，当将我们的单任务音素级发音错误检测模型与使用多粒度信息的多任务学习模型进行比较时，观察到PCC增加了2%。摘要：One of the biggest challenges in designing mispronunciation detection models is the unavailability of labeled L2 speech data. To overcome such data scarcity, we introduce SpeechBlender -- a fine-grained data augmentation pipeline for generating mispronunciation errors. The SpeechBlender utilizes varieties of masks to target different regions of a phonetic unit, and use the mixing factors to linearly interpolate raw speech signals while generating erroneous pronunciation instances. The masks facilitate smooth blending of the signals, thus generating more effective samples than the `Cut/Paste' method. We show the effectiveness of our augmentation technique in a phoneme-level pronunciation quality assessment task, leveraging only a good pronunciation dataset. With SpeechBlender augmentation, we observed a 3% and 2% increase in Pearson correlation coefficient (PCC) compared to no-augmentation and goodness of pronunciation augmentation scenarios respectively for Speechocean762 testset. Moreover, a 2% rise in PCC is observed when comparing our single-task phoneme-level mispronunciation detection model with a multi-task learning model using multiple-granularity information.
【29】 SIMD-size aware weight regularization for fast neural vocoding on CPU标题：基于SIMD大小的权值正则化在CPU上的快速神经声码链接：https://arxiv.org/abs/2211.00898
* 与cs.SD语音【16】为同一篇
作者：Hiroki Kanagawa,Yusuke Ijima机构：NTT Corporation备注：Accepted to SLT 2022摘要：本文提出了一种快速神经声码器的权值正则化方法。修剪耗时的DNN模块是在CPU上实现实时声码器（例如WaveRNN、LPCNet）的一种有前途的方法。鼓励稀疏性的正则化在避免由修剪造成的质量降级方面也是有效的。然而，权重矩阵的阶数在SIMD大小上必须是连续的，以用于快速声码。为了保证这个顺序，我们提出了显式SIMD大小感知正则化。该方法首先将权值矩阵转化为张量，使得权值与群大小预先对齐，然后计算群Lasso类正则化损失.在70%稀疏子带WaveRNN上的实验表明，传统Lasso算法和列分组Lasso算法中的剪枝降低了合成语音的自然度。具有所提出的正则化的声码器1）实现了与没有修剪的声码器相当的自然度，并且2）比使用正则化的其他传统声码器执行得更快。摘要：This paper proposes weight regularization for a faster neural vocoder. Pruning time-consuming DNN modules is a promising way to realize a real-time vocoder on a CPU (e.g. WaveRNN, LPCNet). Regularization that encourages sparsity is also effective in avoiding the quality degradation created by pruning. However, the orders of weight matrices must be contiguous in SIMD size for fast vocoding. To ensure this order, we propose explicit SIMD size aware regularization. Our proposed method reshapes a weight matrix into a tensor so that the weights are aligned by group size in advance, and then computes the group Lasso-like regularization loss. Experiments on 70% sparse subband WaveRNN show that pruning in conventional Lasso and column-wise group Lasso degrades the synthetic speech's naturalness. The vocoder with proposed regularization 1) achieves comparable naturalness to that without pruning and 2) performs meaningfully faster than other conventional vocoders using regularization.
【30】 Pop2Piano : Pop Audio-based Piano Cover Generation标题：Pop2Piano：基于流行音频的钢琴封面生成链接：https://arxiv.org/abs/2211.00895
* 与cs.SD语音【17】为同一篇
作者：Jongho Choi,Kyogu Lee机构：rebellions Inc., Music and Audio Research Group, Seoul National University摘要：流行音乐的钢琴封面被人们广泛喜爱。然而，流行钢琴封面的生成任务仍然是欠研究的。这部分是由于缺乏同步的{Pop，Piano Cover}数据对，这使得应用最新的数据密集型深度学习方法具有挑战性。为了充分利用数据驱动方法的强大功能，我们使用自动化管道制作了大量配对和同步的{pop，piano cover}数据。本文介绍了一种Transformer网络Pop 2 Piano，它可以根据流行音乐的波形生成钢琴曲。据我们所知，这是第一个直接从流行音乐生成钢琴封面的模型，没有旋律和和弦提取模块。我们证明了用我们的数据集训练的Pop 2 Piano可以生成可信的钢琴封面。摘要：The piano cover of pop music is widely enjoyed by people. However, the generation task of the pop piano cover is still understudied. This is partly due to the lack of synchronized {Pop, Piano Cover} data pairs, which made it challenging to apply the latest data-intensive deep learning-based methods. To leverage the power of the data-driven approach, we make a large amount of paired and synchronized {pop, piano cover} data using an automated pipeline. In this paper, we present Pop2Piano, a Transformer network that generates piano covers given waveforms of pop music. To the best of our knowledge, this is the first model to directly generate a piano cover from pop audio without melody and chord extraction modules. We show that Pop2Piano trained with our dataset can generate plausible piano covers.
【31】 Conversation-oriented ASR with multi-look-ahead CBS architecture标题：具有多前瞻CBS架构的面向会话的ASR链接：https://arxiv.org/abs/2211.00858
* 与cs.SD语音【18】为同一篇
作者：Huaibo Zhao,Shinya Fujie,Tetsuji Ogawa,Jin Sakuma,Yusuke Kida,Tetsunori Kobayashi机构：Department of Communications and Computer Engineering, Waseda University, Tokyo Japan, Line corporation, Tokyo Japan备注：Submitted to ICASSP2023摘要：在谈话过程中，人类能够在讲话的任何一点上推断讲话者的意图，以便迅速准备接下来的行动。这种能力也是会话系统实现有节奏、自然会话的关键。为此，用于实时转录语音的自动语音识别（ASR）必须无延迟地实现高精度。在流式ASR中，通过关注超前帧来确保高精度，这导致延迟增量。为了解决这一折衷问题，我们提出了一种多延迟流式ASR，以实现零前瞻的高精度。所提出的系统包含并行操作的两个编码器，其中主编码器利用先行帧生成精确输出，而辅助编码器在没有先行的情况下识别主编码器的先行部分。该系统基于上下文块流（CBS）架构，充分利用了块处理能力，对多延迟架构具有很高的亲和力。还研究了用于构建该系统的各种方法，包括移动网络以作为不同的编码器来执行;以及在一个编码通道中生成两个编码器的输出。摘要：During conversations, humans are capable of inferring the intention of the speaker at any point of the speech to prepare the following action promptly. Such ability is also the key for conversational systems to achieve rhythmic and natural conversation. To perform this, the automatic speech recognition (ASR) used for transcribing the speech in real-time must achieve high accuracy without delay. In streaming ASR, high accuracy is assured by attending to look-ahead frames, which leads to delay increments. To tackle this trade-off issue, we propose a multiple latency streaming ASR to achieve high accuracy with zero look-ahead. The proposed system contains two encoders that operate in parallel, where a primary encoder generates accurate outputs utilizing look-ahead frames, and the auxiliary encoder recognizes the look-ahead portion of the primary encoder without look-ahead. The proposed system is constructed based on contextual block streaming (CBS) architecture, which leverages block processing and has a high affinity for the multiple latency architecture. Various methods are also studied for architecting the system, including shifting the network to perform as different encoders; as well as generating both encoders' outputs in one encoding pass.
【32】 More Speaking or More Speakers?标题：更多的演讲者还是更多的演讲者？链接：https://arxiv.org/abs/2211.00854
* 与cs.SD语音【19】为同一篇
作者：Dan Berrebbi,Ronan Collobert,Navdeep Jaitly,Tatiana Likhomanenko机构：‡Carnegie Mellon University, †Apple摘要：自训练（ST）和自监督学习（SSL）方法在自动语音识别（ASR）中已经显示出很大的改进。尽管有这些进步，但就我们所知，没有分析这些方法中使用的标记和未标记数据集的组成如何影响结果。在这项工作中，我们的目标是分析训练数据中说话人数量对最近的SSL算法（wav2vec 2.0）和最近的ST算法（slimIPL）的影响。我们通过改变说话者的数量同时保持小时数固定，对有标记和无标记的数据进行系统分析，反之亦然。我们的研究结果表明，SSL需要大量的未标记数据才能产生高准确率的结果，而ST需要标记数据中足够数量的说话人，特别是在低音域环境中。以这种方式，这两种方法改进了在数据集构成的不同机制中的监督学习。摘要：Self-training (ST) and self-supervised learning (SSL) methods have demonstrated strong improvements in automatic speech recognition (ASR). In spite of these advances, to the best of our knowledge, there is no analysis of how the composition of the labelled and unlabelled datasets used in these methods affects the results. In this work we aim to analyse the effect of numbers of speakers in the training data on a recent SSL algorithm (wav2vec 2.0), and a recent ST algorithm (slimIPL). We perform a systematic analysis on both labeled and unlabeled data by varying the number of speakers while keeping the number of hours fixed and vice versa. Our findings suggest that SSL requires a large amount of unlabeled data to produce high accuracy results, while ST requires a sufficient number of speakers in the labelled data, especially in the low-regime setting. In this manner these two approaches improve supervised learning in different regimes of dataset composition.
【33】 Build a SRE Challenge System: Lessons from VoxSRC 2022 and CNSRC 2022标题：构建SRE挑战体系：VoxSRC 2022和CNSRC 2022的经验教训链接：https://arxiv.org/abs/2211.00815
* 与cs.SD语音【20】为同一篇
作者：Zhengyang Chen,Bing Han,Xu Xiang,Houjun Huang,Bei Liu,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, AISpeech Ltd, Suzhou, China摘要：不同的说话人识别挑战赛已经举行，以评估说话人确认系统在野外和探索性能极限。Voxceleb说话人识别挑战赛（VoxSRC）是基于Voxceleb的，是最受欢迎的。此外，今年还举办了另一项挑战赛，名为“中国名人说话人识别挑战赛”（CNSRC），该挑战赛基于中国名人多体裁数据集CN-Celeb。今年，我们团队参加了CNSRC 2022和VoxSRC 2022的两个演讲人验证封闭赛道，分别取得了第1名和第3名的成绩。在大多数系统报告中，作者通常只提供他们的系统的描述，而缺乏对他们的方法的有效分析。本文将概述如何构建一个强有力的说话人确认挑战系统，并对每种方法与其他一些流行的技术手段进行了详细的分析比较。摘要：Different speaker recognition challenges have been held to assess the speaker verification system in the wild and probe the performance limit. Voxceleb Speaker Recognition Challenge (VoxSRC), based on the voxceleb, is the most popular. Besides, another challenge called CN-Celeb Speaker Recognition Challenge (CNSRC) is also held this year, which is based on the Chinese celebrity multi-genre dataset CN-Celeb. This year, our team participated in both speaker verification closed tracks in CNSRC 2022 and VoxSRC 2022, and achieved the 1st place and 3rd place respectively. In most system reports, the authors usually only provide a description of their systems but lack an effective analysis of their methods. In this paper, we will outline how to build a strong speaker verification challenge system and give a detailed analysis of each method compared with some other popular technical means.
【34】 Impact of annotation modality on label quality and model performance in the automatic assessment of laughter in-the-wild标题：在野外笑声自动评估中，注释通道对标签质量和模型性能的影响链接：https://arxiv.org/abs/2211.00794
* 与cs.SD语音【21】为同一篇
作者：Jose Vargas-Quiros,Laura Cabrera-Quiros,Catharine Oertel,Hayley Hung机构： Hung work at the Department of IntelligentSystems at TU Delft摘要：笑被认为是最明显的快乐信号之一。笑是公认的多模态现象，但最常见的是通过感觉笑声来检测。目前还不清楚，当通过笑的身体运动对笑进行注释时，对笑的感知和注释与其他模态（如视频）有何不同。在本文中，我们在这个方向上迈出了第一步，通过询问当注释者仅可获得音频、仅可获得视频（包含全身运动信息）或视听模态时，是否可以注释笑声以及注释效果如何。我们询问笑的注释是否在模态之间一致，并比较标记模态对机器学习模型性能的影响。我们比较了笑声检测、强度估计和分割的注释和模型，这三个任务在以前的笑声研究中很常见。我们对48位注释者的4000多条注释进行了分析，发现了笑的感知及其强度在不同模态之间存在不一致的证据。进一步分析注释与合并视听参考注释，发现与音频条件相比，视频条件下的平均回忆率较低，但随着笑声样本的强度增加，回忆率趋于增加。我们的机器学习实验比较了最先进的单模态（基于音频、基于视频和基于加速度）和多模态模型在输入模态、训练标签模态和测试标签模态的不同组合下的性能。无论训练标签模态如何，具有视频和加速度输入的模型具有相似的性能，这表明使用视频获得的标签训练模型从身体运动中检测笑声可能是完全合适的，尽管它们的评分者间一致性较低。摘要：Laughter is considered one of the most overt signals of joy. Laughter is well-recognized as a multimodal phenomenon but is most commonly detected by sensing the sound of laughter. It is unclear how perception and annotation of laughter differ when annotated from other modalities like video, via the body movements of laughter. In this paper we take a first step in this direction by asking if and how well laughter can be annotated when only audio, only video (containing full body movement information) or audiovisual modalities are available to annotators. We ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, three tasks common in previous studies of laughter. Our analysis of more than 4000 annotations acquired from 48 annotators revealed evidence for incongruity in the perception of laughter, and its intensity between modalities. Further analysis of annotations against consolidated audiovisual reference annotations revealed that recall was lower on average for video when compared to the audio condition, but tended to increase with the intensity of the laughter samples. Our machine learning experiments compared the performance of state-of-the-art unimodal (audio-based, video-based and acceleration-based) and multi-modal models for different combinations of input modalities, training label modality, and testing label modality. Models with video and acceleration inputs had similar performance regardless of training label modality, suggesting that it may be entirely appropriate to train models for laughter detection from body movements using video-acquired labels, despite their lower inter-rater agreement.
【35】 Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems标题：用于快速高效语音系统的统一端到端语音识别和端点链接：https://arxiv.org/abs/2211.00786
* 与cs.SD语音【22】为同一篇
作者：Shaan Bijwadia,Shuo-yiin Chang,Bo Li,Tara Sainath,Chao Zhang,Yanzhang He机构：Google, Inc.备注：To be published in Spoken Language Technology Workshop (SLT) 2022摘要：自动语音识别（ASR）系统典型地依赖于外部端点指示器（EP）模型来识别语音边界。在这项工作中，我们提出了一种方法，在一个单一的端到端（E2E）多任务模型中联合训练ASR和EP任务，通过选择性地利用来自ASR音频编码器的信息来提高EP质量。我们引入了一个“开关”连接，它训练EP直接消费音频帧或来自ASR模型的低级别潜在表示。这产生了单个E2E模型，该模型可以在推断期间用于以低成本执行帧过滤，并且还基于正在进行的ASR计算来进行高质量的查询结束（EOQ）预测。我们提供了语音搜索测试集的结果，结果显示，与单独的单任务模型相比，该方法将中位终点延迟缩短了120 ms（缩短了30.8%），将第90百分位延迟缩短了170 ms（缩短了23.0%），而不会使单词错误率回归。对于连续识别，WER提高了10.6%（相对）。摘要：Automatic speech recognition (ASR) systems typically rely on an external endpointer (EP) model to identify speech boundaries. In this work, we propose a method to jointly train the ASR and EP tasks in a single end-to-end (E2E) multitask model, improving EP quality by optionally leveraging information from the ASR audio encoder. We introduce a "switch" connection, which trains the EP to consume either the audio frames directly or low-level latent representations from the ASR model. This results in a single E2E model that can be used during inference to perform frame filtering at low cost, and also make high quality end-of-query (EOQ) predictions based on ongoing ASR computation. We present results on a voice search test set showing that, compared to separate single-task models, this approach reduces median endpoint latency by 120 ms (30.8% reduction), and 90th percentile latency by 170 ms (23.0% reduction), without regressing word error rate. For continuous recognition, WER improves by 10.6% (relative).
【36】 Comparision Of Adversarial And Non-Adversarial LSTM Music Generative Models标题：对抗性与非对抗性LSTM音乐生成模式的比较链接：https://arxiv.org/abs/2211.00731
* 与cs.SD语音【23】为同一篇
作者：Moseli Mots'oehli,Anna Sergeevna Bosman,Johan Pieter De Villiers机构：Department of Computer Science, University Of Pretoria, Department of Electrical and Computer Engineering备注：Submitted to a 2023 conference, 20 pages, 13 figures摘要：算法音乐创作是一种在最少甚至没有人为干预的情况下创作音乐作品的方式。虽然递归神经网络传统上被应用于许多序列到序列的预测任务，包括音乐创作的成功实现，但是它们基于输入到输出映射的标准监督学习方法导致音符变化的缺乏。因此，这些模型可以被视为潜在地不适合于诸如音乐生成之类的任务。生成式对抗网络学习数据的生成分布，并导致不同的样本。本工作在MIDI数据上实现并比较了递归神经网络音乐作曲家的对抗性和非对抗性训练。所得到的音乐样本由人类听众评估，他们的偏好被记录下来。评价表明对抗性训练产生了更美观的音乐。摘要：Algorithmic music composition is a way of composing musical pieces with minimal to no human intervention. While recurrent neural networks are traditionally applied to many sequence-to-sequence prediction tasks, including successful implementations of music composition, their standard supervised learning approach based on input-to-output mapping leads to a lack of note variety. These models can therefore be seen as potentially unsuitable for tasks such as music generation. Generative adversarial networks learn the generative distribution of data and lead to varied samples. This work implements and compares adversarial and non-adversarial training of recurrent neural network music composers on MIDI data. The resulting music samples are evaluated by human listeners, their preferences recorded. The evaluation indicates that adversarial training produces more aesthetically pleasing music.
机器翻译，仅供参考
每日语音速递