今日论文合集：cs.SD语音26篇，eess.AS音频处理26篇。
cs.SD语音
【1】 Magnitude or Phase? A Two Stage Algorithm for Dereverberation
标题：量级还是阶段性？一种两阶段的混响消除算法
链接：https://arxiv.org/abs/2211.00607
作者：Ayal Schwartz,Sharon Gannot,Shlomo E. Chazan机构： OriginAI, Ramat-Gan, Israel, Bar-Ilan University, Ramat-Gan, Israel摘要：本文提出了一种新的单麦克风语音去混响算法。首先，通过性能分析，说明了单纯提高幅度或相位的算法是不够好的。此外，我们证明了少数客观测量与清洁星等高度相关，而其他与清洁相位高度相关。因此，我们提出了一个新的架构，它由两个子模型组成，每个子模型负责不同的任务。第一模型估计给定噪声输入的干净幅度。然后，增强的幅度与噪声输入相位一起被用作第二模型的输入，以估计去混响信号的实部和虚部。本文提出了一种包括预训练和微调的训练方案。我们使用来自REVERB挑战的数据评估我们提出的方法，并将我们的结果与其他方法进行比较。我们证明了在所有测量中的一致改进，这可以归因于幅度和相位的改进估计。摘要：In this work we present a new single-microphone speech dereverberation algorithm. First, a performance analysis is presented to interpret that algorithms focused on improving solely magnitude or phase are not good enough. Furthermore, we demonstrate that few objective measurements have high correlation with the clean magnitude while others with the clean phase. Consequently ,we propose a new architecture which consists of two sub-models, each of which is responsible for a different task. The first model estimates the clean magnitude given the noisy input. The enhanced magnitude together with the noisy-input phase are then used as inputs to the second model to estimate the real and imaginary portions of the dereverberated signal. A training scheme including pre-training and fine-tuning is presented in the paper. We evaluate our proposed approach using data from the REVERB challenge and compare our results to other methods. We demonstrate consistent improvements in all measures, which can be attributed to the improved estimates of both the magnitude and the phase.
【2】 T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5
标题：T5lephone：通过音素级别T5为口语理解搭建语音和文本自我监督模型的桥梁
链接：https://arxiv.org/abs/2211.00586
作者：Chan-Jan Hsu,Ho-Lam Chung,Hung-yi Lee,Yu Tsao机构： Tsao 3 1MediaTek Research 2National Taiwan University摘要：在口语理解（SLU）中，自然的解决方案是将预先训练的语音模型（例如，HuBERT）和预先训练的语言模型（PLM，例如，T5）级联。大多数先前的工作使用预先训练的语言模型和基于子词的标记化。然而，输入单元的粒度影响语音模型输出和语言模型输入的对齐，并且基于字符的标记化的PLM未被充分开发。本研究旨在探讨不同标记化策略的PLM对口语理解任务（包括口语问答和语音翻译）的影响。我们进一步扩展了这个想法，创建了T5lephone（发音为telephone），它是T5的一个变体，使用音位化文本进行了预训练。我们使用现有PLM初始化T5lephone，以使用相对轻量级的计算资源对其进行预训练。我们在NMSQA上达到了最先进的水平，T5lephone型号在端到端SQA和ST上超过了T5和其他类型的单元。摘要：In Spoken language understanding (SLU), a natural solution is concatenating pre-trained speech models (e.g. HuBERT) and pretrained language models (PLM, e.g. T5). Most previous works use pretrained language models with subword-based tokenization. However, the granularity of input units affects the alignment of speech model outputs and language model inputs, and PLM with character-based tokenization is underexplored. In this work, we conduct extensive studies on how PLMs with different tokenization strategies affect spoken language understanding task including spoken question answering (SQA) and speech translation (ST). We further extend the idea to create T5lephone(pronounced as telephone), a variant of T5 that is pretrained using phonemicized text. We initialize T5lephone with existing PLMs to pretrain it using relatively lightweight computational resources. We reached state-of-the-art on NMSQA, and the T5lephone model exceeds T5 with other types of units on end-to-end SQA and ST.
【3】 No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration
标题：通过基于视觉姿势的过滤和可穿戴式加速在拥挤环境中检测无音频说话状态
链接：https://arxiv.org/abs/2211.00549
作者：Jose Vargas-Quiros,Laura Cabrera-Quiros,Hayley Hung机构： Hung were with the Department of Intelligent Systemsat TU Delft摘要：在拥挤的场景中识别谁在说话是理解其中正在进行的社会互动的一个关键挑战。仅从身体运动检测说话状态为分析其中无法获得个人音频的社交场景打开了大门。视频和可穿戴传感器使它能够以一种不引人注目的、保护隐私的方式识别说话。当考虑视频模态时，在动作识别问题中，传统上使用边界框来定位和分割出目标对象，然后识别在其中发生的动作。然而，交叉污染、遮挡和人体的关节特性使得该方法在拥挤场景中具有挑战性。这里，我们利用铰接的身体姿势来进行对象定位和随后的语音检测阶段。实验结果表明，选取姿态关键点周围的局部特征对泛化性能有积极的影响，同时也显著减少了所考虑的局部特征的数量，使得方法更加有效。利用两个不同视角的野外数据集，我们研究了交叉污染在这一效应中的作用。此外，我们还利用了通过可穿戴传感器测量的加速度来完成相同的任务，并提出了一种结合这两种方法的多模态方法。摘要：Recognizing who is speaking in a crowded scene is a key challenge towards the understanding of the social interactions going on within. Detecting speaking status from body movement alone opens the door for the analysis of social scenes in which personal audio is not obtainable. Video and wearable sensors make it possible recognize speaking in an unobtrusive, privacy-preserving way. When considering the video modality, in action recognition problems, a bounding box is traditionally used to localize and segment out the target subject, to then recognize the action taking place within it. However, cross-contamination, occlusion, and the articulated nature of the human body, make this approach challenging in a crowded scene. Here, we leverage articulated body poses for subject localization and in the subsequent speech detection stage. We show that the selection of local features around pose keypoints has a positive effect on generalization performance while also significantly reducing the number of local features considered, making for a more efficient method. Using two in-the-wild datasets with different viewpoints of subjects, we investigate the role of cross-contamination in this effect. We additionally make use of acceleration measured through wearable sensors for the same task, and present a multimodal approach combining both methods.
【4】 Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis
标题：表现性语音合成中基于令牌级声学潜伏期预测的语音级表示学习
链接：https://arxiv.org/abs/2211.00523
作者：Karolos Nikitaras,Konstantinos Klapsas,Nikolaos Ellinas,Georgia Maniati,June Sig Sung,Inchul Hwang,Spyros Raptis,Aimilios Chalamandaris,Pirros Tsiakoulis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：本文提出了一种表达性语音合成模型，该模型利用符号级潜在韵律变量来捕获和控制话语级属性，如角色的动作声音和说话风格。当前的工作旨在将这种细粒度和话语级的语音属性显式地分解成由在相应级别中操作的模块提取的不同表示。我们证明了细粒度的潜在空间也能捕获粗粒度的信息，随着潜在空间维数的增加，这种捕获更明显，以捕获不同的韵律表示。因此，在标记级和话语级表示的多样性和它们的解纠缠之间出现了折衷。我们通过首先将丰富的语音属性捕获到标记级潜在空间中，然后单独训练给定输入文本的先验网络，学习话语级表示，以便预测在前一步骤中提取的音素级后验潜在，来缓解该问题。通过定性和定量评价，验证了该方法的有效性。音频样本可在我们的演示页面。摘要：This paper proposes an Expressive Speech Synthesis model that utilizes token-level latent prosodic variables in order to capture and control utterance-level attributes, such as character acting voice and speaking style. Current works aim to explicitly factorize such fine-grained and utterance-level speech attributes into different representations extracted by modules that operate in the corresponding level. We show that the fine-grained latent space also captures coarse-grained information, which is more evident as the dimension of latent space increases in order to capture diverse prosodic representations. Therefore, a trade-off arises between the diversity of the token-level and utterance-level representations and their disentanglement. We alleviate this issue by first capturing rich speech attributes into a token-level latent space and then, separately train a prior network that given the input text, learns utterance-level representations in order to predict the phoneme-level, posterior latents extracted during the previous step. Both qualitative and quantitative evaluations are used to demonstrate the effectiveness of the proposed approach. Audio samples are available in our demo page.
【5】 TrimTail: Low-Latency Streaming ASR with Simple but Effective Spectrogram-Level Length Penalty
标题：TrimTail：低延迟流ASR，具有简单但有效的频谱图级长度惩罚
链接：https://arxiv.org/abs/2211.00522
作者：Xingchen Song,Di Wu,Zhiyong Wu,Binbin Zhang,Yuekai Zhang,Zhendong Peng,Wenpeng Li,Fuping Pan,Changbao Zhu机构：Tsinghua Univ., Beijing, China, Horizon Inc., Beijing, China, WeNet Open Source Community备注：submitted to ICASSP 2023摘要：本文提出了一种简单有效的发射正则化方法TrimTail，用于改善流式ASR模型的延迟。TrimTail的核心思想是应用长度惩罚（即，通过修剪尾随帧，参见图1-（b）），这不需要任何对准。我们证明了TrimTail在计算上是廉价的，并且可以在线应用，并且通过将其应用于各种端到端流式ASR网络（使用CTC损失[1]或换能器损失[2]进行训练），在任何数据集上使用任何训练损失或任何模型架构进行优化，而无需任何额外工作。我们在Aishell-1和Librispeech上实现了100 $\sim $200ms的延迟减少，同时具有相同甚至更高的准确性。此外，通过使用TrimTail，我们可以实现用户敏感延迟（USD）的400 ms算法改进，并且精度损失小于0.2。摘要：In this paper, we present TrimTail, a simple but effective emission regularization method to improve the latency of streaming ASR models. The core idea of TrimTail is to apply length penalty (i.e., by trimming trailing frames, see Fig. 1-(b)) directly on the spectrogram of input utterances, which does not require any alignment. We demonstrate that TrimTail is computationally cheap and can be applied online and optimized with any training loss or any model architecture on any dataset without any extra effort by applying it on various end-to-end streaming ASR networks either trained with CTC loss [1] or Transducer loss [2]. We achieve 100 $\sim$ 200ms latency reduction with equal or even better accuracy on both Aishell-1 and Librispeech. Moreover, by using TrimTail, we can achieve a 400ms algorithmic improvement of User Sensitive Delay (USD) with an accuracy loss of less than 0.2.
【6】 Modelling black-box audio effects with time-varying feature modulation
标题：用时变特征调制对黑盒音频效果进行建模
链接：https://arxiv.org/abs/2211.00497
作者：Marco Comunità,Christian J. Steinmetz,Huy Phan,Joshua D. Reiss机构：Centre for Digital Music, Queen Mary University of London, UK摘要：用于音频效果黑箱建模的深度学习方法已经显示出前景，然而，大多数现有工作集中在行为在相对较短的时间尺度上的非线性效果，如吉他放大器和失真。虽然循环和卷积架构在理论上可以扩展到在更长时间尺度上捕获行为，但我们表明，在对诸如模糊和动态范围压缩之类的音频效果进行建模时，简单地缩放现有架构的宽度、深度或膨胀因子并不能产生令人满意的性能。为了解决这一问题，我们提出将时变特征线性调制集成到现有的时域卷积骨干中，这是一种能够实现中间激活的可学习自适应的方法。我们证明，我们的方法更准确地捕捉了一系列模糊和压缩器实现在时域和频域指标上的长期依赖性。我们提供可靠的示例、源代码和预先训练的模型，以提高可重复性。摘要：Deep learning approaches for black-box modelling of audio effects have shown promise, however, the majority of existing work focuses on nonlinear effects with behaviour on relatively short time-scales, such as guitar amplifiers and distortion. While recurrent and convolutional architectures can theoretically be extended to capture behaviour at longer time scales, we show that simply scaling the width, depth, or dilation factor of existing architectures does not result in satisfactory performance when modelling audio effects such as fuzz and dynamic range compression. To address this, we propose the integration of time-varying feature-wise linear modulation into existing temporal convolutional backbones, an approach that enables learnable adaptation of the intermediate activations. We demonstrate that our approach more accurately captures long-range dependencies for a range of fuzz and compressor implementations across both time and frequency domain metrics. We provide sound examples, source code, and pretrained models to faciliate reproducibility.
【7】 Generating Gender-Ambiguous Text-to-Speech Voices
标题：生成性别模糊的文本到语音转换语音
链接：https://arxiv.org/abs/2211.00375
作者：Konstantinos Markopoulos,Georgia Maniati,Georgios Vamvoukakis,Nikolaos Ellinas,Karolos Nikitaras,Konstantinos Klapsas,Georgios Vardaxoglou,Panos Kakoulidis,June Sig Sung,Inchul Hwang,Aimilios Chalamandaris,Pirros Tsiakoulis,Spyros Raptis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：语音助手或任何语音用户界面的性别是其感知身份的中心元素。虽然女性的声音是一种常见的选择，但人们越来越感兴趣的是性别不明确而不是明确确定为女性或男性的替代方法。这项工作解决了生成不对应于任何现有人的性别模糊的文本到语音（TTS）声音的任务。这是通过从潜在说话人嵌入空间采样来实现的，该潜在说话人嵌入空间是在对来自多个男性和女性说话人的数据训练多语言、多说话人TTS系统时形成的。调查了有关取样过程的各种选项。在实验中，我们评估了不同的采样选择对语音的性别模糊度和自然度的影响。实验结果表明，该方法能够有效地生成比基线平均说话人嵌入更好的新说话人。据我们所知，这是第一个能够可靠地产生一系列性别不明确的声音以满足不同用户要求的系统方法。摘要：The gender of a voice assistant or any voice user interface is a central element of its perceived identity. While a female voice is a common choice, there is an increasing interest in alternative approaches where the gender is ambiguous rather than clearly identifying as female or male. This work addresses the task of generating gender-ambiguous text-to-speech (TTS) voices that do not correspond to any existing person. This is accomplished by sampling from a latent speaker embeddings' space that was formed while training a multilingual, multi-speaker TTS system on data from multiple male and female speakers. Various options are investigated regarding the sampling process. In our experiments, the effects of different sampling choices on the gender ambiguity and the naturalness of the resulting voices are evaluated. The proposed method is shown able to efficiently generate novel speakers that are superior to a baseline averaged speaker embedding. To our knowledge, this is the first systematic approach that can reliably generate a range of gender-ambiguous voices to meet diverse user requirements.
【8】 Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features
标题：基于韵律和语言特征的内容感知神经文语转换MOS预测研究
链接：https://arxiv.org/abs/2211.00342
作者：Alexandra Vioni,Georgia Maniati,Nikolaos Ellinas,June Sig Sung,Inchul Hwang,Aimilios Chalamandaris,Pirros Tsiakoulis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：当前用于自动合成语音评估的现有技术的方法是基于MOS预测神经模型的。这样的MOS预测模型包括使用频谱特征作为输入的MOSNet和LDNet，以及依赖于直接使用语音信号作为输入的预训练的自监督学习模型的SSL-MOS。在现代高质量的神经TTS系统中，韵律相对于口语内容的适当性是语音自然度的决定性因素。因此，我们建议将韵律和语言特征作为MOS预测系统的附加输入，并评估它们对预测结果的影响。我们将音素层F0和持续时间特征作为韵律输入，将Tacotron编码器输出、词性标签和BERT嵌入作为高级语言输入。所有MOS预测系统都在SOMOS上训练，SOMOS是一个具有众包自然度MOS评估的神经TTS数据集。结果表明，所提出的附加特征在MOS预测任务中是有益的，通过在话语级和系统级预测两者上改进预测的MOS得分与地面真实的相关性。摘要：Current state-of-the-art methods for automatic synthetic speech evaluation are based on MOS prediction neural models. Such MOS prediction models include MOSNet and LDNet that use spectral features as input, and SSL-MOS that relies on a pretrained self-supervised learning model that directly uses the speech signal as input. In modern high-quality neural TTS systems, prosodic appropriateness with regard to the spoken content is a decisive factor for speech naturalness. For this reason, we propose to include prosodic and linguistic features as additional inputs in MOS prediction systems, and evaluate their impact on the prediction outcome. We consider phoneme level F0 and duration features as prosodic inputs, as well as Tacotron encoder outputs, POS tags and BERT embeddings as higher-level linguistic inputs. All MOS prediction systems are trained on SOMOS, a neural TTS-only dataset with crowdsourced naturalness MOS evaluations. Results show that the proposed additional features are beneficial in the MOS prediction task, by improving the predicted MOS scores' correlation with the ground truths, both at utterance-level and system-level predictions.
【9】 SDMuse: Stochastic Differential Music Editing and Generation via Hybrid Representation
标题：SDMuse：基于混合表示的随机差分音乐编辑与生成
链接：https://arxiv.org/abs/2211.00222
作者：Chen Zhang,Yi Ren,Kejun Zhang,Shuicheng Yan机构：Zhejiang University, China, Sea AI Lab, Singapore摘要：虽然深度生成模型已经使音乐生成成为可能，但是在细粒度下编辑现有音乐作品仍然是一个具有挑战性和未被探索的问题。本文提出了一个统一的随机微分音乐编辑和生成框架SDMuse，它不仅可以从零开始合成一个完整的音乐作品，而且可以对现有的音乐作品进行多种修改，如组合、延续、修复和风格转换。SDMuse遵循两级流水线，在钢琴滚动和MIDI事件的混合表示之上实现音乐生成和编辑。具体地，SDMuse首先通过基于扩散模型生成先验的随机微分方程（SDE）迭代地去噪来生成/编辑钢琴滚动，然后精炼所生成的钢琴滚动并自回归地预测MIDI事件标记。我们在ailabs 1 k7流行音乐数据集上对我们的方法生成的音乐在各种音乐编辑和生成任务上的质量和可控性进行了评估。实验结果证明了我们提出的随机微分音乐编辑和生成过程以及混合表示的有效性。摘要：While deep generative models have empowered music generation, it remains a challenging and under-explored problem to edit an existing musical piece at fine granularity. In this paper, we propose SDMuse, a unified Stochastic Differential Music editing and generation framework, which can not only compose a whole musical piece from scratch, but also modify existing musical pieces in many ways, such as combination, continuation, inpainting, and style transferring. The proposed SDMuse follows a two-stage pipeline to achieve music generation and editing on top of a hybrid representation including pianoroll and MIDI-event. In particular, SDMuse first generates/edits pianoroll by iteratively denoising through a stochastic differential equation (SDE) based on a diffusion model generative prior, and then refines the generated pianoroll and predicts MIDI-event tokens auto-regressively. We evaluate the generated music of our method on ailabs1k7 pop music dataset in terms of quality and controllability on various music editing and generation tasks. Experimental results demonstrate the effectiveness of our proposed stochastic differential music editing and generation process, as well as the hybrid representations.
【10】 Active Learning of Non-semantic Speech Tasks with Pretrained Models
标题：基于预训练模型的非语义语音任务主动学习
链接：https://arxiv.org/abs/2211.00119
作者：Harlin Lee,Aaqib Saeed,Andrea L. Bertozzi机构：University of California Los Angeles, Los Angeles, CA, USA, Philips Research, Eindhoven, The Netherlands摘要：利用大量未标记数据集对神经网络进行预训练，可以为深度模型求解下游任务提供更好的先验知识，因而受到了广泛的关注。然而，这种方法通常假设对于下游任务，我们可以访问足够大小的注释数据。在本研究中，我们提出了一个新的系统ALOE，用于提高非语义语音任务的数据和标签效率与主动学习（AL）。ALOE结合主动学习使用预先训练的模型来递增地标记数据，并学习用于下游任务的分类器，从而减少预先获取标记数据的需要。我们证明了ALOE在各种任务、基于不确定性的获取函数和模型结构上的有效性。使用ALOE在冻结的编码器上训练线性分类器被示出为实现与利用整个标记数据的若干基线类似的性能。摘要：Pretraining neural networks with massive unlabeled datasets have become popular as it equips the deep models with a better prior to solve downstream tasks. However, this approach generally assumes that for downstream tasks, we have access to annotated data of sufficient size. In this work, we propose ALOE, a novel system for improving the data- and label-efficiency of non-semantic speech tasks with active learning (AL). ALOE uses pre-trained models in conjunction with active learning to label data incrementally and learns classifiers for downstream tasks, thereby mitigating the need to acquire labeled data beforehand. We demonstrate the effectiveness of ALOE on a wide range of tasks, uncertainty-based acquisition functions, and model architectures. Training a linear classifier on top of a frozen encoder with ALOE is shown to achieve performance similar to several baselines that utilize the entire labeled data.
【11】 SCA: Streaming Cross-attention Alignment for Echo Cancellation
标题：SCA：用于回声消除的流交叉注意对齐
链接：https://arxiv.org/abs/2211.00589
作者：Yang Liu,Yangyang Shi,Yun Li,Kaustubh Kalgaonkar,Sriram Srinivasan,Xin Lei摘要：端到端深度学习在语音增强任务中表现出了良好的效果，例如噪声抑制、去混响和语音分离。然而，大多数用于回声消除的现有技术方法是基于经典DSP的算法或混合DSP-ML算法。诸如延迟估计器和自适应线性滤波器之类的组件基于传统的信号处理概念，并且深度学习算法通常仅用于替换非线性残余回声抑制器。本文介绍了一种基于流媒体交叉注意对齐的端到端回声消除网络。我们提出的方法可以处理未对准的输入，而不需要外部对准，并产生高质量的语音没有回声。同时，端到端算法简化了用于时变回声路径情况的当前回声消除流水线。我们在ICASSP 2022和Interspeech 2021微软深度回声消除挑战评估数据集上测试了我们提出的方法，其中我们的方法优于其他一些混合和端到端方法。摘要：End-to-End deep learning has shown promising results for speech enhancement tasks, such as noise suppression, dereverberation, and speech separation. However, most state-of-the-art methods for echo cancellation are either classical DSP-based or hybrid DSP-ML algorithms. Components such as the delay estimator and adaptive linear filter are based on traditional signal processing concepts, and deep learning algorithms typically only serve to replace the non-linear residual echo suppressor. This paper introduces an end-to-end echo cancellation network with a streaming cross-attention alignment (SCA). Our proposed method can handle unaligned inputs without requiring external alignment and generate high-quality speech without echoes. At the same time, the end-to-end algorithm simplifies the current echo cancellation pipeline for time-variant echo path cases. We test our proposed method on the ICASSP2022 and Interspeech2021 Microsoft deep echo cancellation challenge evaluation dataset, where our method outperforms some of the other hybrid and end-to-end methods.
【12】 Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers标题：基于适配器的新说话人文语转换模型的扩展链接：https://arxiv.org/abs/2211.00585
作者：Cheng-Ping Hsieh,Subhankar Ghosh,Boris Ginsburg机构：University of California, San Diego, NVIDIA, Santa Clara备注：Submitted to ICASSP 2023摘要：微调是使文本到语音（TTS）模型适应新说话人的常用方法。然而，这种方法存在一些挑战。通常，微调需要每个扬声器几个小时的高质量语音。还存在微调将负面影响先前学习的说话者的语音合成的质量。在本文中，我们提出了一种基于使用参数高效适配器模块的TTS适配的替代方法。该方法在原有网络的基础上增加了少量的适配器模块。原始权重被冻结，并且只有适配器在新说话者的语音上被微调。参数有效的微调方法将产生与原始模型具有高水平参数共享的新模型。在LibriTTS、HiFi-TTS和VCTK数据集上的实验通过主客观指标验证了该方法的有效性。摘要：Fine-tuning is a popular method for adapting text-to-speech (TTS) models to new speakers. However this approach has some challenges. Usually fine-tuning requires several hours of high quality speech per speaker. There is also that fine-tuning will negatively affect the quality of speech synthesis for previously learnt speakers. In this paper we propose an alternative approach for TTS adaptation based on using parameter-efficient adapter modules. In the proposed approach, a few small adapter modules are added to the original network. The original weights are frozen, and only the adapters are fine-tuned on speech for new speaker. The parameter-efficient fine-tuning approach will produce a new model with high level of parameter sharing with original model. Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate the effectiveness of adapter-based method through objective and subjective metrics.
【13】 Ambisonic Encoding of Signals From Equatorial Microphone Arrays标题：赤道传声器阵列信号的双声编码链接：https://arxiv.org/abs/2211.00584
作者：Jens Ahrens机构：Technical note∗ v. , Chalmers University of Technology摘要：在（Ahrens等人，2021）基于沿着刚性球形挡板的赤道的压力传感器来计算声场的球面谐波（SH）表示。原始公式使用复值SH基函数。如果所捕获声场的SH表示意图借助于实数值音频信号而被存储在时域中，则这是不方便的，因为这在立体声的空间音频格式中是常见的。本文件总结了需要应用于来自（Ahrens等人，2021）以产生与诸如SPARTA和IEM插件套件之类的已建立的立体声软件工具兼容的所捕获声场的立体声表示。提供了实现该公式的示例MATLAB脚本。摘要：The equatorial microphone array presented in (Ahrens et al., 2021) computes a spherical harmonic (SH) representation of a sound field based on pressure sensors along the equator of a rigid spherical baffle. The original formulation uses complex-valued SH basis functions. This is inconvenient if the SH representation of the captured sound field is intended to be stored in time domain by means of real-valued audio signals as it is common in the spatial audio format of ambisonics. The present document summarizes the modifications that need to be applied to the mathematical formulation from (Ahrens et al., 2021) to produce an ambisonic representation of the captured sound field that is compatible with the established ambisonic software tools like SPARTA and the IEM Plugin Suite. An example MATLAB script that implements this formulation is provided.
【14】 Ambisonic Encoding of Signals From Spherical Microphone Arrays标题：球形传声器阵列信号的双声编码链接：https://arxiv.org/abs/2211.00583
作者：Jens Ahrens机构：Technical note∗ v. , Chalmers University of Technology摘要：本文档示出了如何处理来自刚性球高阶立体声麦克风阵列的麦克风的信号，使得它们利用N3D归一化和ACN通道阶数来编码，从而可以与诸如SPARTA和IEM插件套件之类的标准立体声软件工具一起使用。提供了MATLAB脚本。摘要：This document illustrates how to process the signals from the microphones of a rigid-sphere higher-order ambisonic microphone array so that they are encoded with N3D normalization and ACN channel order and thereby can be used with the standard ambisonic software tools such as SPARTA and the IEM Plugin Suite. A MATLAB script is provided.
【15】 Few-shot Bioacoustic Event Detection with Machine Learning Methods标题：基于机器学习方法的少发生物声事件检测链接：https://arxiv.org/abs/2211.00569
作者：Leah Chowenhill,Gaurav Satyanath,Shubhranshu Singh,Madhav Mahendra Wagh机构： biologists at the University of Puerto Rico installedsolar-powered iPods in the frog’s habitat and began recording the soundscape for 1 minute备注：7 pages, 6 tables, 1 figure摘要：Few-Shot学习是一种分类类型，通过它基于每个类的有限数量的样本进行预测。这种类型的分类有时被称为元学习问题，其中模型学习如何学习识别罕见的情况。我们试图从哺乳动物或鸟类的五种典型发声中提取信息，并在现场录音中检测和分类这些声音[2]。2021年的“声学场景和事件的检测和分类（DCASE）挑战”中提供了该任务。我们没有像大多数人那样利用深度学习，而是仅使用机器学习方法制定了一个新颖的解决方案。对各种模型进行了测试，发现逻辑回归优于线性回归和模板匹配。然而，所有这些方法都过度预测了现场记录中的事件数量。摘要：Few-shot learning is a type of classification through which predictions are made based on a limited number of samples for each class. This type of classification is sometimes referred to as a meta-learning problem, in which the model learns how to learn to identify rare cases. We seek to extract information from five exemplar vocalisations of mammals or birds and detect and classify these sounds in field recordings [2]. This task was provided in the Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge of 2021. Rather than utilize deep learning, as is most commonly done, we formulated a novel solution using only machine learning methods. Various models were tested, and it was found that logistic regression outperformed both linear regression and template matching. However, all of these methods over-predicted the number of events in the field recordings.
【16】 A Comparative Study on multichannel Speaker-attributed automatic speech recognition in Multi-party Meetings标题：多党会议中多通道说话人自动语音识别的比较研究链接：https://arxiv.org/abs/2211.00511
作者：Mohan Shi,Jie Zhang,Zhihao Du,Fan Yu,Shiliang Zhang,Li-Rong Dai机构：NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China, Speech Lab, Alibaba Group, China摘要：多方会议场景下的说话人归属自动语音识别（SA-ASR）是最有价值和最具挑战性的ASR任务之一。研究表明，单通道帧级Diarization与串行输出训练（SC-FD-SOT）、单通道词级Diarization与SOT（SC-WD-SOT）以及单通道目标语者分离与ASR联合训练（SC-TS-ASR）可以部分解决这一问题。SC-FD-SOT通过将说话人diarization结果与ASR假设对齐，得到说话人归属的录音; SC-WD-SOT采用词级diarization，消除对齐对时间戳的依赖; SC-TS-ASR联合训练目标说话人分离和ASR模块，取得了最佳性能。本文提出了三种相应的多通道（MC）SA-ASR方法，即MC-FD-SOT、MC-WD-SOT和MC-TS-ASR。针对不同的任务/模型，考虑了不同的多通道数据融合策略，包括针对MC-FD-SOT的通道级跨通道注意、针对MC-WD-SOT的帧级跨通道注意和针对MC-TS-ASR的神经波束形成。在AliMeeting语料库上的实验结果表明，本文提出的多通道SA-ASR模型在说话人相关字符错误率（SD-CER）方面始终优于相应的单通道模型.摘要：Speaker-attributed automatic speech recognition (SA-ASR) in multiparty meeting scenarios is one of the most valuable and challenging ASR task. It was shown that single-channel frame-level diarization with serialized output training (SC-FD-SOT), single-channel word-level diarization with SOT (SC-WD-SOT) and joint training of single-channel target-speaker separation and ASR (SC-TS-ASR) can be exploited to partially solve this problem. SC-FD-SOT obtains the speaker-attributed transcriptions by aligning the speaker diarization results with the ASR hypotheses, SC-WD-SOT uses word-level diarization to get rid of the alignment dependence on timestamps, and SC-TS-ASR jointly trains target-speaker separation and ASR modules, which achieves the best performance. In this paper, we propose three corresponding multichannel (MC) SA-ASR approaches, namely MC-FD-SOT, MC-WD-SOT and MC-TS-ASR. For different tasks/models, different multichannel data fusion strategies are considered, including channel-level cross-channel attention for MC-FD-SOT, frame-level cross-channel attention for MC-WD-SOT and neural beamforming for MC-TS-ASR. Experimental results on the AliMeeting corpus reveal that our proposed multichannel SA-ASR models can consistently outperform the corresponding single-channel counterparts in terms of the speaker-dependent character error rate (SD-CER).
【17】 Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation标题：用于知识提取的多码本矢量量化指标预测链接：https://arxiv.org/abs/2211.00508
作者：Liyong Guo,Xiaoyu Yang,Quandong Wang,Yuxiang Kong,Zengwei Yao,Fan Cui,Fangjun Kuang,Wei Kang,Long Lin,Mingshuang Luo,Piotr Zelasko,Daniel Povey机构： Xiaomi Corp., Beijing, China, Meaning.Team Inc, USA备注：Submitted to ICASSP 2022摘要：在自动语音识别（ASR）中，知识提取（KD）是提高模型性能的一种常用方法，学生模型被训练成模仿教师模型的输出行为。然而，传统的知识发现方法存在教师标签存储问题，尤其是在训练语料库规模较大的情况下。尽管动态教师标签生成解决了这个问题，但是训练速度明显较慢，因为教师模型必须在每个批次进行评估。本文将教师标签的生成问题转化为一个编解码问题。提出了一种新的多码本矢量量化（MVQ）方法，将教师嵌入压缩到码本索引（CI）中。在此基础上，提出了一个知识发现训练框架（MVQ-KD），其中学生模型预测由自监督的预训练教师模型嵌入生成的CI。在LibriSpeech clean-100 hour上的实验表明，MVQ-KD框架与传统的KD方法（l1，l2）相比具有相当的性能，而所需的存储空间却减少了256倍。当使用完整的LibriSpeech数据集时，MVQ-KD框架导致非流式换能器在测试-清洁和测试-其他时的相对单词错误率（WERR）降低13.8%和8.2%，流式换能器的相对单词错误率降低4.0%和4.9%。这项工作的实现已经作为开源项目icefall的一部分发布。摘要：Knowledge distillation(KD) is a common approach to improve model performance in automatic speech recognition (ASR), where a student model is trained to imitate the output behaviour of a teacher model. However, traditional KD methods suffer from teacher label storage issue, especially when the training corpora are large. Although on-the-fly teacher label generation tackles this issue, the training speed is significantly slower as the teacher model has to be evaluated every batch. In this paper, we reformulate the generation of teacher label as a codec problem. We propose a novel Multi-codebook Vector Quantization (MVQ) approach that compresses teacher embeddings to codebook indexes (CI). Based on this, a KD training framework (MVQ-KD) is proposed where a student model predicts the CI generated from the embeddings of a self-supervised pre-trained teacher model. Experiments on the LibriSpeech clean-100 hour show that MVQ-KD framework achieves comparable performance as traditional KD methods (l1, l2), while requiring 256 times less storage. When the full LibriSpeech dataset is used, MVQ-KD framework results in 13.8% and 8.2% relative word error rate reductions (WERRs) for non -streaming transducer on test-clean and test-other and 4.0% and 4.9% for streaming transducer. The implementation of this work is already released as a part of the open-source project icefall.
【18】 Delay-penalized transducer for low-latency streaming ASR标题：用于低延迟流ASR的延迟惩罚传感器链接：https://arxiv.org/abs/2211.00490
作者：Wei Kang,Zengwei Yao,Fangjun Kuang,Liyong Guo,Xiaoyu Yang,Long lin,Piotr Żelasko,Daniel Povey机构： Xiaomi Corp., Beijing, China, Meaning.Team Inc, USA备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：在流式自动语音识别（ASR）中，期望尽可能地减少等待时间，同时对识别准确度具有最小的影响。虽然一些现有的方法能够实现这一目标，但是由于它们依赖于外部对准，所以它们难以实现。本文提出了一种简单的方法来补偿换能器模型中的符号延迟，从而可以在没有外部校准的流式模型中平衡符号延迟和精度之间的折衷。具体地，我们的方法将小的常数乘以（T/2 - t），其中T是帧的数量，t是当前帧，添加到被馈送到二维换能器递归中的所有非空白对数概率（在归一化之后）。实验结果表明，在流Conformer模型和单向长短时记忆（LSTM）模型下，该算法能够在可接受的性能下降范围内显著降低符号延迟。我们的方法实现了与之前发布的FastEmit类似的延迟-精度平衡，但我们认为我们的方法更可取，因为它具有更好的合理性：这等效于惩罚平均符号延迟。我们的工作是开源的，可以公开获得（https：//github.com/k2-fsa/k2）。摘要：In streaming automatic speech recognition (ASR), it is desirable to reduce latency as much as possible while having minimum impact on recognition accuracy. Although a few existing methods are able to achieve this goal, they are difficult to implement due to their dependency on external alignments. In this paper, we propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Specifically, our method adds a small constant times (T/2 - t), where T is the number of frames and t is the current frame, to all the non-blank log-probabilities (after normalization) that are fed into the two dimensional transducer recursion. For both streaming Conformer models and unidirectional long short-term memory (LSTM) models, experimental results show that it can significantly reduce the symbol delay with an acceptable performance degradation. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay. Our work is open-sourced and publicly available (https://github.com/k2-fsa/k2).
【19】 Fast and parallel decoding for transducer标题：变送器的快速并行译码链接：https://arxiv.org/abs/2211.00484
作者：Wei Kang,Liyong Guo,Fangjun Kuang,Long Lin,Mingshuang Luo,Zengwei Yao,Xiaoyu Yang,Piotr Żelasko,Daniel Povey机构： Xiaomi Corp., Beijing, China, Meaning.Team Inc, USA备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：换能器结构在语音识别领域中变得越来越流行，因为它是自然的流以及高精度。换能器的缺点之一是由于每个时间步长可以发射的符号的数目不受限制，所以难以以快速和并行的方式解码。 在这项工作中，我们引入了一个约束版本的换能器损失，以学习序列之间的严格单调排列;我们还通过限制在换能器解码中每个时间步长可以发射的符号的数量来改进标准的贪婪搜索和波束搜索算法，从而使其更有效地以批量并行解码。在此基础上，提出了一种基于有限状态自动机（FSA）的并行波束搜索算法，该算法能够在GPU上高效地运行于图上.实验结果表明，该算法在提高解码速度的同时，误字率（WER）也略有改善。我们的工作是开源的，可以公开获得。摘要：The transducer architecture is becoming increasingly popular in the field of speech recognition, because it is naturally streaming as well as high in accuracy. One of the drawbacks of transducer is that it is difficult to decode in a fast and parallel way due to an unconstrained number of symbols that can be emitted per time step. In this work, we introduce a constrained version of transducer loss to learn strictly monotonic alignments between the sequences; we also improve the standard greedy search and beam search algorithms by limiting the number of symbols that can be emitted per time step in transducer decoding, making it more efficient to decode in parallel with batches. Furthermore, we propose an finite state automaton-based (FSA) parallel beam search algorithm that can run with graphs on GPU efficiently. The experiment results show that we achieve slight word error rate (WER) improvement as well as significant speedup in decoding. Our work is open-sourced and publicly available\footnote{https://github.com/k2-fsa/icefall}.
【20】 Adapting self-supervised models to multi-talker speech recognition using speaker embeddings标题：利用说话人嵌入使自监督模型适应多说话人语音识别链接：https://arxiv.org/abs/2211.00482
作者：Zili Huang,Desh Raj,Paola García,Sanjeev Khudanpur机构：Center for Language and Speech Processing and Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, USA备注：submitted to ICASSP 2023摘要：自监督学习（SSL）方法在语音处理任务中，特别是在单说话者应用中得到了普及，该方法在没有显式监督的情况下学习数据的表示。然而，这些模型对于多说话者情形通常具有降级的性能--可能是由于域不匹配--这严重地限制了它们在此类应用中的使用。本文研究了在两种情况下，上游SSL模型对多说话者自动语音识别（ASR）任务的适应性。首先，当给定分段话语时，我们表明添加基于注册嵌入的目标说话人提取（TSE）模块是对混合感知预训练的补充。其次，针对未分割的混合语音，提出了一种新的联合说话人建模（JSM）方法，该方法通过对混合语音中所有说话人的信息进行嵌入来聚合说话人信息。通过在Libri 2 Mix上的控制实验，我们表明使用说话人嵌入对于分割和未分割的情况分别提供了9.1%和42.1%的相对WER改善。我们还通过在AMI数据集上的实验证明了我们的模型对真实会话混合的有效性。摘要：Self-supervised learning (SSL) methods which learn representations of data without explicit supervision have gained popularity in speech-processing tasks, particularly for single-talker applications. However, these models often have degraded performance for multi-talker scenarios -- possibly due to the domain mismatch -- which severely limits their use for such applications. In this paper, we investigate the adaptation of upstream SSL models to the multi-talker automatic speech recognition (ASR) task under two conditions. First, when segmented utterances are given, we show that adding a target speaker extraction (TSE) module based on enrollment embeddings is complementary to mixture-aware pre-training. Second, for unsegmented mixtures, we propose a novel joint speaker modeling (JSM) approach, which aggregates information from all speakers in the mixture through their embeddings. With controlled experiments on Libri2Mix, we show that using speaker embeddings provides relative WER improvements of 9.1% and 42.1% over strong baselines for the segmented and unsegmented cases, respectively. We also demonstrate the effectiveness of our models for real conversational mixtures through experiments on the AMI dataset.
【21】 Metric Learning for User-defined Keyword Spotting标题：用于用户定义的关键字识别的度量学习链接：https://arxiv.org/abs/2211.00439
作者：Jaemin Jung,Youkyum Kim,Jihwan Park,Youshin Lim,Byeong-Yeol Kim,Youngjoon Jang,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea, Hyundai Motor Company,dot Inc., Seoul, Republic of Korea摘要：本工作的目标是检测用户定义的新口语术语。虽然大多数以前的工作解决关键字定位（KWS）作为一个封闭集分类问题，这限制了他们的可转移性，以看不见的术语。定义自定义关键字的能力在用户体验方面具有优势。 本文提出了一种基于度量学习的用户自定义关键词识别训练策略。特别是，我们做出了以下贡献：（1）利用已有的语音语料库构建大规模的关键词数据集，提出一种过滤方法去除影响模型训练的数据;（2）提出了一种基于度量学习的两阶段训练策略，并证明了该方法通过丰富关键词的表示形式，提高了自定义关键词识别任务的性能;（3）为了便于在用户自定义的知识服务领域进行公平比较，提出了统一的评价协议和度量标准。 我们提出的系统不需要对用户定义的关键词进行增量训练，并且在使用所提出的以及现有度量的Google Speech Commands数据集上显著优于先前的工作。摘要：The goal of this work is to detect new spoken terms defined by users. While most previous works address Keyword Spotting (KWS) as a closed-set classification problem, this limits their transferability to unseen terms. The ability to define custom keywords has advantages in terms of user experience. In this paper, we propose a metric learning-based training strategy for user-defined keyword spotting. In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training; (2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics. Our proposed system does not require an incremental training on the user-defined keywords, and outperforms previous works by a significant margin on the Google Speech Commands dataset using the proposed as well as the existing metrics.
【22】 Disentangled representation learning for multilingual speaker recognition标题：用于多语种说话人识别的解缠表征学习链接：https://arxiv.org/abs/2211.00437
作者：Kihyun Nam,Youkyum Kim,Hee Soo Heo,Jee-weon Jung,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, South Korea, Naver Corporation, South Korea摘要：本文的目标是训练对双语说话场景具有鲁棒性的说话人嵌入。世界上大多数人口至少讲两种语言;然而，当以不同语言说话时，大多数说话人识别系统不能识别同一说话人。 现有的说话人识别评价集没有考虑双语环境，难以分析双语说话人对说话人识别性能的影响。本文在VoxCeleb的基础上，提出了一种新的考虑双语场景的大规模评价集。我们还介绍了一种表征学习策略，该策略将语言信息从说话人表征中分离出来，以考虑双语场景。这种语言解纠缠的表征学习策略可以适应现有的模型，只需对训练管道进行小的改变。 实验结果表明，在双语测试集上，基线模型的性能明显下降.与此相反，使用该策略训练的模型在双语测试场景下表现出了显著的改进，同时在现有的单语测试集上保持了竞争性能。摘要：The goal of this paper is to train speaker embeddings that are robust to bilingual speaking scenario. The majority of the world's population speak at least two languages; however, most speaker recognition systems fail to recognise the same speaker when speaking in different languages. Popular speaker recognition evaluation sets do not consider the bilingual scenario, making it difficult to analyse the effect of bilingual speakers on speaker recognition performance. This paper proposes a new large-scale evaluation set derived from VoxCeleb that considers bilingual scenarios. We also introduce a representation learning strategy, which disentangles language information from speaker representation to account for the bilingual scenario. This language-disentangled representation learning strategy can be adapted to existing models with small changes to the training pipeline. Experimental results demonstrate that the baseline models suffer significant performance degradation when evaluated on the proposed bilingual test set. On the contrary, the model trained with the proposed disentanglement strategy shows significant improvement under the bilingual evaluation scenario while simultaneously retaining competitive performance on existing monolingual test sets.
【23】 Speech-text based multi-modal training with bidirectional attention for improved speech recognition标题：基于语音-文本的双向注意多模式训练改进语音识别链接：https://arxiv.org/abs/2211.00325
作者：Yuhang Yang,Haihua Xu,Hao Huang,Eng Siong Chng,Sheng Li机构：School of Information Science and Engineering, Xinjiang University, China, Bytedance AI Lab, Singapore, School of Computer Science and Engineering, Nanyang Technological University, Singapore备注：5 pages, 3 figures, 3 tables摘要：为了让最先进的端到端ASR模型享受数据效率，以及通过多模态训练获得更多的不成对文本数据，需要解决两个问题：1）语音和语言（又称文本数据）之间的特征采样率的同步性; 2）来自两个编码器的学习表示的同质性。本文提出了一种新的双向注意机制（BiAM），利用多模态学习方法对ASR编码器（底层）和文本编码器进行联合学习。BiAM是为了便于特征采样率的交换，在另一个空间中实现待测量的一种类型的变换特征的质量，具有多样化的目标函数。结果，语音表示被丰富了更多的语言信息，而由文本编码器生成的表示与对应的语音表示更相似，因此共享ASR模型更适合于不成对文本数据预训练。为了验证所提出方法的有效性，我们进行了两类实验，有和没有额外的不成对文本数据。在Librispeech语料库上的实验结果表明，仅使用成对数据学习时，该算法的误词率降低率最高可达6.15%，而使用更多非成对文本数据时，误词率降低率为9.23%。摘要：To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as well as much more unpaired text data by multi-modal training, one needs to address two problems: 1) the synchronicity of feature sampling rates between speech and language (aka text data); 2) the homogeneity of the learned representations from two encoders. In this paper we propose to employ a novel bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder (bottom layers) and text encoder with a multi-modal learning method. The BiAM is to facilitate feature sampling rate exchange, realizing the quality of the transformed features for the one kind to be measured in another space, with diversified objective functions. As a result, the speech representations are enriched with more linguistic information, while the representations generated by the text encoder are more similar to corresponding speech ones, and therefore the shared ASR models are more amenable for unpaired text data pretraining. To validate the efficacy of the proposed method, we perform two categories of experiments with or without extra unpaired text data. Experimental results on Librispeech corpus show it can achieve up to 6.15% word error rate reduction (WERR) with only paired data learning, while 9.23% WERR when more unpaired text data is employed.
【24】 Waveform Boundary Detection for Partially Spoofed Audio标题：部分欺骗音频的波形边界检测链接：https://arxiv.org/abs/2211.00226
作者：Zexin Cai,Weiqing Wang,Ming Li机构：Department of Electrical and Computer Engineering, Duke University, Durham, NC , USA, Data Science Research Center, Duke Kunshan University, Kunshan , PR China摘要：提出了一种针对音频欺骗攻击的波形边界检测系统。部分欺骗/伪造音频，其中部分话语被替换为合成或自然音频剪辑，最近已被报告为音频深度伪造的一种场景。由于deepfakes可能对社会安全构成威胁，因此检测此类欺骗音频至关重要。因此，我们提出了一种基于深度学习的帧级检测系统来解决这个问题，该系统可以检测部分欺骗音频并定位被操纵的片段。我们提出的方法在ADD 2022挑战提供的数据上进行了训练和评估。我们评估了我们的检测模型，涉及各种声学特征和网络配置。因此，我们的检测系统在ADD 2022挑战测试集上实现了6.58%的等错误率（EER），这是能够定位操纵片段的部分欺骗音频检测系统中的最佳性能。摘要：The present paper proposes a waveform boundary detection system for audio spoofing attacks containing partially manipulated segments. Partially spoofed/fake audio, where part of the utterance is replaced, either with synthetic or natural audio clips, has recently been reported as one scenario of audio deepfakes. As deepfakes can be a threat to social security, the detection of such spoofing audio is essential. Accordingly, we propose to address the problem with a deep learning-based frame-level detection system that can detect partially spoofed audio and locate the manipulated pieces. Our proposed method is trained and evaluated on data provided by the ADD2022 Challenge. We evaluate our detection model concerning various acoustic features and network configurations. As a result, our detection system achieves an equal error rate (EER) of 6.58% on the ADD2022 challenge test set, which is the best performance in partially spoofed audio detection systems that can locate manipulated clips.
【25】 ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting标题：ImagineNET：基于嵌入修复的间歇性视觉提示目标说话人提取链接：https://arxiv.org/abs/2211.00109
作者：Zexu Pan,Wupeng Wang,Marvin Borsdorf,Haizhou Li机构：Institute of Data Science, National University of Singapore (NUS), Singapore, Department of Electrical and Computer Engineering, NUS, Singapore, Machine Listening Lab, University of Bremen, Germany摘要：说话人提取技术寻求从语音混合中的干扰话音中挑选出目标说话人的话音。典型地，目标说话者的辅助参考被用于形成自主注意。视频剪辑中的预先记录的话语或同步的嘴唇运动可以用作辅助参考。视觉线索的使用不仅可行，而且由于其对噪声的鲁棒性而有效，并逐渐受到人们的青睐。然而，在可能发生视觉遮挡或间歇通信的真实世界应用中，难以保证这种并行视觉提示总是可用的。本文研究了间断视觉线索下的音视频说话人提取算法。提出了一种基于说话人提取和视觉嵌入的联合修复框架。为了促进两个任务之间的交互，它们以交错结构交替执行并联合优化。我们还提出了两种类型的视觉修复损失，并研究了我们提出的方法与两种类型的普遍使用的视觉嵌入。实验结果表明，该算法在信号质量、感知质量和可懂度方面均优于基线算法。摘要：The speaker extraction technique seeks to single out the voice of a target speaker from the interfering voices in a speech mixture. Typically an auxiliary reference of the target speaker is used to form voluntary attention. Either a pre-recorded utterance or a synchronized lip movement in a video clip can serve as the auxiliary reference. The use of visual cue is not only feasible, but also effective due to its noise robustness, and becoming popular. However, it is difficult to guarantee that such parallel visual cue is always available in real-world applications where visual occlusion or intermittent communication can occur. In this paper, we study the audio-visual speaker extraction algorithms with intermittent visual cue. We propose a joint speaker extraction and visual embedding inpainting framework to explore the mutual benefits. To encourage the interaction between the two tasks, they are performed alternately with an interlacing structure and optimized jointly. We also propose two types of visual inpainting losses and study our proposed method with two types of popularly used visual embeddings. The experimental results show that we outperform the baseline in terms of signal quality, perceptual quality, and intelligibility.
【26】 An analysis of degenerating speech due to progressive dysarthria on ASR performance标题：进行性构音障碍致退行性语音对ASR表现的影响分析链接：https://arxiv.org/abs/2211.00089
作者：Katrin Tomanek,Katie Seaver,Pan-Pan Jiang,Richard Cave,Lauren Harrel,Jordan R. Green机构：Google LLC,MGH Institute of Health Professions,Language and Cognition, UCL备注：Submitted to ICASSP 2023摘要：尽管个性化自动语音识别（ASR）模型最近已经被设计成识别甚至严重受损的语音，但是对于具有退化语音的人来说，模型性能可能随着时间而降低。本研究的目的是（1）分析语音退化个体的ASR性能随时间的变化，（2）探索缓解策略，以优化整个疾病进展过程中的识别。4名因肌萎缩性侧索硬化症（ALS）而言语退化的个体记录了言语。针对三种ASR模型计算了记录会话中的单词错误率（WER）：非自适应说话人独立（U-SI）、自适应说话人独立（A-SI）和自适应说话人相关（A-SD或个性化）。随着时间的推移，所有三种模型的性能都随着语音受损程度的加重而显著下降，但当使用语音进展严重阶段的记录更新A-SD模型时，其性能显著改善。在语言明显退化之前，在疾病早期记录额外的话语并没有改善A-SD模型的性能。总的来说，我们的研究结果强调了在为进行性言语障碍患者提供个性化模型时，连续记录（和模型再训练）的重要性。摘要：Although personalized automatic speech recognition (ASR) models have recently been designed to recognize even severely impaired speech, model performance may degrade over time for persons with degenerating speech. The aims of this study were to (1) analyze the change of performance of ASR over time in individuals with degrading speech, and (2) explore mitigation strategies to optimize recognition throughout disease progression. Speech was recorded by four individuals with degrading speech due to amyotrophic lateral sclerosis (ALS). Word error rates (WER) across recording sessions were computed for three ASR models: Unadapted Speaker Independent (U-SI), Adapted Speaker Independent (A-SI), and Adapted Speaker Dependent (A-SD or personalized). The performance of all three models degraded significantly over time as speech became more impaired, but the performance of the A-SD model improved markedly when it was updated with recordings from the severe stages of speech progression. Recording additional utterances early in the disease before speech degraded significantly did not improve the performance of A-SD models. Overall, our findings emphasize the importance of continuous recording (and model retraining) when providing personalized models for individuals with progressive speech impairments.
【1】 SCA: Streaming Cross-attention Alignment for Echo Cancellation标题：SCA：用于回声消除的流交叉注意对齐链接：https://arxiv.org/abs/2211.00589
* 与cs.SD语音【11】为同一篇
作者：Yang Liu,Yangyang Shi,Yun Li,Kaustubh Kalgaonkar,Sriram Srinivasan,Xin Lei机构：Meta, US摘要：端到端深度学习在语音增强任务中表现出了良好的效果，例如噪声抑制、去混响和语音分离。然而，大多数用于回声消除的现有技术方法是基于经典DSP的算法或混合DSP-ML算法。诸如延迟估计器和自适应线性滤波器之类的组件基于传统的信号处理概念，并且深度学习算法通常仅用于替换非线性残余回声抑制器。本文介绍了一种基于流媒体交叉注意对齐的端到端回声消除网络。我们提出的方法可以处理未对准的输入，而不需要外部对准，并产生高质量的语音没有回声。同时，端到端算法简化了用于时变回声路径情况的当前回声消除流水线。我们在ICASSP 2022和Interspeech 2021微软深度回声消除挑战评估数据集上测试了我们提出的方法，其中我们的方法优于其他一些混合和端到端方法。摘要：End-to-End deep learning has shown promising results for speech enhancement tasks, such as noise suppression, dereverberation, and speech separation. However, most state-of-the-art methods for echo cancellation are either classical DSP-based or hybrid DSP-ML algorithms. Components such as the delay estimator and adaptive linear filter are based on traditional signal processing concepts, and deep learning algorithms typically only serve to replace the non-linear residual echo suppressor. This paper introduces an end-to-end echo cancellation network with a streaming cross-attention alignment (SCA). Our proposed method can handle unaligned inputs without requiring external alignment and generate high-quality speech without echoes. At the same time, the end-to-end algorithm simplifies the current echo cancellation pipeline for time-variant echo path cases. We test our proposed method on the ICASSP2022 and Interspeech2021 Microsoft deep echo cancellation challenge evaluation dataset, where our method outperforms some of the other hybrid and end-to-end methods.
【2】 Adapter-Based Extension of Multi-Speaker Text-to-Speech Model for New Speakers标题：基于适配器的新说话人文语转换模型的扩展链接：https://arxiv.org/abs/2211.00585
* 与cs.SD语音【12】为同一篇
作者：Cheng-Ping Hsieh,Subhankar Ghosh,Boris Ginsburg机构：University of California, San Diego, NVIDIA, Santa Clara备注：Submitted to ICASSP 2023摘要：微调是使文本到语音（TTS）模型适应新说话人的常用方法。然而，这种方法存在一些挑战。通常，微调需要每个扬声器几个小时的高质量语音。还存在微调将负面影响先前学习的说话者的语音合成的质量。在本文中，我们提出了一种基于使用参数高效适配器模块的TTS适配的替代方法。该方法在原有网络的基础上增加了少量的适配器模块。原始权重被冻结，并且只有适配器在新说话者的语音上被微调。参数有效的微调方法将产生与原始模型具有高水平参数共享的新模型。在LibriTTS、HiFi-TTS和VCTK数据集上的实验通过主客观指标验证了该方法的有效性。摘要：Fine-tuning is a popular method for adapting text-to-speech (TTS) models to new speakers. However this approach has some challenges. Usually fine-tuning requires several hours of high quality speech per speaker. There is also that fine-tuning will negatively affect the quality of speech synthesis for previously learnt speakers. In this paper we propose an alternative approach for TTS adaptation based on using parameter-efficient adapter modules. In the proposed approach, a few small adapter modules are added to the original network. The original weights are frozen, and only the adapters are fine-tuned on speech for new speaker. The parameter-efficient fine-tuning approach will produce a new model with high level of parameter sharing with original model. Our experiments on LibriTTS, HiFi-TTS and VCTK datasets validate the effectiveness of adapter-based method through objective and subjective metrics.
【3】 Ambisonic Encoding of Signals From Equatorial Microphone Arrays标题：赤道传声器阵列信号的双声编码链接：https://arxiv.org/abs/2211.00584
* 与cs.SD语音【13】为同一篇
作者：Jens Ahrens机构：Technical note∗ v. , Chalmers University of Technology摘要：在（Ahrens等人，2021）基于沿着刚性球形挡板的赤道的压力传感器来计算声场的球面谐波（SH）表示。原始公式使用复值SH基函数。如果所捕获声场的SH表示意图借助于实数值音频信号而被存储在时域中，则这是不方便的，因为这在立体声的空间音频格式中是常见的。本文件总结了需要应用于来自（Ahrens等人，2021）以产生与诸如SPARTA和IEM插件套件之类的已建立的立体声软件工具兼容的所捕获声场的立体声表示。提供了实现该公式的示例MATLAB脚本。摘要：The equatorial microphone array presented in (Ahrens et al., 2021) computes a spherical harmonic (SH) representation of a sound field based on pressure sensors along the equator of a rigid spherical baffle. The original formulation uses complex-valued SH basis functions. This is inconvenient if the SH representation of the captured sound field is intended to be stored in time domain by means of real-valued audio signals as it is common in the spatial audio format of ambisonics. The present document summarizes the modifications that need to be applied to the mathematical formulation from (Ahrens et al., 2021) to produce an ambisonic representation of the captured sound field that is compatible with the established ambisonic software tools like SPARTA and the IEM Plugin Suite. An example MATLAB script that implements this formulation is provided.
【4】 Ambisonic Encoding of Signals From Spherical Microphone Arrays标题：球形传声器阵列信号的双声编码链接：https://arxiv.org/abs/2211.00583
* 与cs.SD语音【14】为同一篇
作者：Jens Ahrens机构：Technical note∗ v. , Chalmers University of Technology摘要：本文档示出了如何处理来自刚性球高阶立体声麦克风阵列的麦克风的信号，使得它们利用N3D归一化和ACN通道阶数来编码，从而可以与诸如SPARTA和IEM插件套件之类的标准立体声软件工具一起使用。提供了MATLAB脚本。摘要：This document illustrates how to process the signals from the microphones of a rigid-sphere higher-order ambisonic microphone array so that they are encoded with N3D normalization and ACN channel order and thereby can be used with the standard ambisonic software tools such as SPARTA and the IEM Plugin Suite. A MATLAB script is provided.
【5】 Few-shot Bioacoustic Event Detection with Machine Learning Methods标题：基于机器学习方法的少发生物声事件检测链接：https://arxiv.org/abs/2211.00569
* 与cs.SD语音【15】为同一篇
作者：Leah Chowenhill,Gaurav Satyanath,Shubhranshu Singh,Madhav Mahendra Wagh机构： biologists at the University of Puerto Rico installedsolar-powered iPods in the frog’s habitat and began recording the soundscape for 1 minute备注：7 pages, 6 tables, 1 figure摘要：少镜头学习是一种分类类型，通过它基于每个类的有限数量的样本进行预测。这种类型的分类有时被称为元学习问题，其中模型学习如何学习识别罕见的情况。我们试图从哺乳动物或鸟类的五种典型发声中提取信息，并在现场录音中检测和分类这些声音[2]。2021年的“声学场景和事件的检测和分类（DCASE）挑战”中提供了该任务。我们没有像大多数人那样利用深度学习，而是仅使用机器学习方法制定了一个新颖的解决方案。对各种模型进行了测试，发现逻辑回归优于线性回归和模板匹配。然而，所有这些方法都过度预测了现场记录中的事件数量。摘要：Few-shot learning is a type of classification through which predictions are made based on a limited number of samples for each class. This type of classification is sometimes referred to as a meta-learning problem, in which the model learns how to learn to identify rare cases. We seek to extract information from five exemplar vocalisations of mammals or birds and detect and classify these sounds in field recordings [2]. This task was provided in the Detection and Classification of Acoustic Scenes and Events (DCASE) Challenge of 2021. Rather than utilize deep learning, as is most commonly done, we formulated a novel solution using only machine learning methods. Various models were tested, and it was found that logistic regression outperformed both linear regression and template matching. However, all of these methods over-predicted the number of events in the field recordings.
【6】 A Comparative Study on multichannel Speaker-attributed automatic speech recognition in Multi-party Meetings标题：多党会议中多通道说话人自动语音识别的比较研究链接：https://arxiv.org/abs/2211.00511
* 与cs.SD语音【16】为同一篇
作者：Mohan Shi,Jie Zhang,Zhihao Du,Fan Yu,Shiliang Zhang,Li-Rong Dai机构：NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China, Speech Lab, Alibaba Group, China摘要：多方会议场景下的说话人归属自动语音识别（SA-ASR）是最有价值和最具挑战性的ASR任务之一。研究表明，单通道帧级Diarization与串行输出训练（SC-FD-SOT）、单通道词级Diarization与SOT（SC-WD-SOT）以及单通道目标语者分离与ASR联合训练（SC-TS-ASR）可以部分解决这一问题。SC-FD-SOT通过将说话人diarization结果与ASR假设对齐，得到说话人归属的录音; SC-WD-SOT采用词级diarization，消除对齐对时间戳的依赖; SC-TS-ASR联合训练目标说话人分离和ASR模块，取得了最佳性能。本文提出了三种相应的多通道（MC）SA-ASR方法，即MC-FD-SOT、MC-WD-SOT和MC-TS-ASR。针对不同的任务/模型，考虑了不同的多通道数据融合策略，包括针对MC-FD-SOT的通道级跨通道注意、针对MC-WD-SOT的帧级跨通道注意和针对MC-TS-ASR的神经波束形成。在AliMeeting语料库上的实验结果表明，本文提出的多通道SA-ASR模型在说话人相关字符错误率（SD-CER）方面始终优于相应的单通道模型.摘要：Speaker-attributed automatic speech recognition (SA-ASR) in multiparty meeting scenarios is one of the most valuable and challenging ASR task. It was shown that single-channel frame-level diarization with serialized output training (SC-FD-SOT), single-channel word-level diarization with SOT (SC-WD-SOT) and joint training of single-channel target-speaker separation and ASR (SC-TS-ASR) can be exploited to partially solve this problem. SC-FD-SOT obtains the speaker-attributed transcriptions by aligning the speaker diarization results with the ASR hypotheses, SC-WD-SOT uses word-level diarization to get rid of the alignment dependence on timestamps, and SC-TS-ASR jointly trains target-speaker separation and ASR modules, which achieves the best performance. In this paper, we propose three corresponding multichannel (MC) SA-ASR approaches, namely MC-FD-SOT, MC-WD-SOT and MC-TS-ASR. For different tasks/models, different multichannel data fusion strategies are considered, including channel-level cross-channel attention for MC-FD-SOT, frame-level cross-channel attention for MC-WD-SOT and neural beamforming for MC-TS-ASR. Experimental results on the AliMeeting corpus reveal that our proposed multichannel SA-ASR models can consistently outperform the corresponding single-channel counterparts in terms of the speaker-dependent character error rate (SD-CER).
【7】 Predicting Multi-Codebook Vector Quantization Indexes for Knowledge Distillation标题：用于知识提取的多码本矢量量化指标预测链接：https://arxiv.org/abs/2211.00508
* 与cs.SD语音【17】为同一篇
作者：Liyong Guo,Xiaoyu Yang,Quandong Wang,Yuxiang Kong,Zengwei Yao,Fan Cui,Fangjun Kuang,Wei Kang,Long Lin,Mingshuang Luo,Piotr Zelasko,Daniel Povey机构： Xiaomi Corp., Beijing, China, Meaning.Team Inc, USA备注：Submitted to ICASSP 2022摘要：在自动语音识别（ASR）中，知识提取（KD）是提高模型性能的一种常用方法，学生模型被训练成模仿教师模型的输出行为。然而，传统的知识发现方法存在教师标签存储问题，尤其是在训练语料库规模较大的情况下。尽管动态教师标签生成解决了这个问题，但是训练速度明显较慢，因为教师模型必须在每个批次进行评估。本文将教师标签的生成问题转化为一个编解码问题。提出了一种新的多码本矢量量化（MVQ）方法，将教师嵌入压缩到码本索引（CI）中。在此基础上，提出了一个知识发现训练框架（MVQ-KD），其中学生模型预测由自监督的预训练教师模型嵌入生成的CI。在LibriSpeech clean-100 hour上的实验表明，MVQ-KD框架与传统的KD方法（l1，l2）相比具有相当的性能，而所需的存储空间却减少了256倍。当使用完整的LibriSpeech数据集时，MVQ-KD框架导致非流式换能器在测试-清洁和测试-其他时的相对单词错误率（WERR）降低13.8%和8.2%，流式换能器的相对单词错误率降低4.0%和4.9%。这项工作的实现已经作为开源项目icefall的一部分发布。摘要：Knowledge distillation(KD) is a common approach to improve model performance in automatic speech recognition (ASR), where a student model is trained to imitate the output behaviour of a teacher model. However, traditional KD methods suffer from teacher label storage issue, especially when the training corpora are large. Although on-the-fly teacher label generation tackles this issue, the training speed is significantly slower as the teacher model has to be evaluated every batch. In this paper, we reformulate the generation of teacher label as a codec problem. We propose a novel Multi-codebook Vector Quantization (MVQ) approach that compresses teacher embeddings to codebook indexes (CI). Based on this, a KD training framework (MVQ-KD) is proposed where a student model predicts the CI generated from the embeddings of a self-supervised pre-trained teacher model. Experiments on the LibriSpeech clean-100 hour show that MVQ-KD framework achieves comparable performance as traditional KD methods (l1, l2), while requiring 256 times less storage. When the full LibriSpeech dataset is used, MVQ-KD framework results in 13.8% and 8.2% relative word error rate reductions (WERRs) for non -streaming transducer on test-clean and test-other and 4.0% and 4.9% for streaming transducer. The implementation of this work is already released as a part of the open-source project icefall.
【8】 Delay-penalized transducer for low-latency streaming ASR标题：用于低延迟流ASR的延迟惩罚传感器链接：https://arxiv.org/abs/2211.00490
* 与cs.SD语音【18】为同一篇
作者：Wei Kang,Zengwei Yao,Fangjun Kuang,Liyong Guo,Xiaoyu Yang,Long lin,Piotr Żelasko,Daniel Povey机构： Xiaomi Corp., Beijing, China, Meaning.Team Inc, USA备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：在流式自动语音识别（ASR）中，期望尽可能地减少等待时间，同时对识别准确度具有最小的影响。虽然一些现有的方法能够实现这一目标，但是由于它们依赖于外部对准，所以它们难以实现。本文提出了一种简单的方法来补偿换能器模型中的符号延迟，从而可以在没有外部校准的流式模型中平衡符号延迟和精度之间的折衷。具体地，我们的方法将小的常数乘以（T/2 - t），其中T是帧的数量，t是当前帧，添加到被馈送到二维换能器递归中的所有非空白对数概率（在归一化之后）。实验结果表明，在流Conformer模型和单向长短时记忆（LSTM）模型下，该算法能够在可接受的性能下降范围内显著降低符号延迟。我们的方法实现了与之前发布的FastEmit类似的延迟-精度平衡，但我们认为我们的方法更可取，因为它具有更好的合理性：这等效于惩罚平均符号延迟。我们的工作是开源的，可以公开获得（https：//github.com/k2-fsa/k2）。摘要：In streaming automatic speech recognition (ASR), it is desirable to reduce latency as much as possible while having minimum impact on recognition accuracy. Although a few existing methods are able to achieve this goal, they are difficult to implement due to their dependency on external alignments. In this paper, we propose a simple way to penalize symbol delay in transducer model, so that we can balance the trade-off between symbol delay and accuracy for streaming models without external alignments. Specifically, our method adds a small constant times (T/2 - t), where T is the number of frames and t is the current frame, to all the non-blank log-probabilities (after normalization) that are fed into the two dimensional transducer recursion. For both streaming Conformer models and unidirectional long short-term memory (LSTM) models, experimental results show that it can significantly reduce the symbol delay with an acceptable performance degradation. Our method achieves similar delay-accuracy trade-off to the previously published FastEmit, but we believe our method is preferable because it has a better justification: it is equivalent to penalizing the average symbol delay. Our work is open-sourced and publicly available (https://github.com/k2-fsa/k2).
【9】 Fast and parallel decoding for transducer标题：变送器的快速并行译码链接：https://arxiv.org/abs/2211.00484
* 与cs.SD语音【19】为同一篇
作者：Wei Kang,Liyong Guo,Fangjun Kuang,Long Lin,Mingshuang Luo,Zengwei Yao,Xiaoyu Yang,Piotr Żelasko,Daniel Povey机构： Xiaomi Corp., Beijing, China, Meaning.Team Inc, USA备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：换能器结构在语音识别领域中变得越来越流行，因为它是自然的流以及高精度。换能器的缺点之一是由于每个时间步长可以发射的符号的数目不受限制，所以难以以快速和并行的方式解码。 在这项工作中，我们引入了一个约束版本的换能器损失，以学习序列之间的严格单调排列;我们还通过限制在换能器解码中每个时间步长可以发射的符号的数量来改进标准的贪婪搜索和波束搜索算法，从而使其更有效地以批量并行解码。在此基础上，提出了一种基于有限状态自动机（FSA）的并行波束搜索算法，该算法能够在GPU上高效地运行于图上.实验结果表明，该算法在提高解码速度的同时，误字率（WER）也略有改善。我们的工作是开源的，可以公开获得。摘要：The transducer architecture is becoming increasingly popular in the field of speech recognition, because it is naturally streaming as well as high in accuracy. One of the drawbacks of transducer is that it is difficult to decode in a fast and parallel way due to an unconstrained number of symbols that can be emitted per time step. In this work, we introduce a constrained version of transducer loss to learn strictly monotonic alignments between the sequences; we also improve the standard greedy search and beam search algorithms by limiting the number of symbols that can be emitted per time step in transducer decoding, making it more efficient to decode in parallel with batches. Furthermore, we propose an finite state automaton-based (FSA) parallel beam search algorithm that can run with graphs on GPU efficiently. The experiment results show that we achieve slight word error rate (WER) improvement as well as significant speedup in decoding. Our work is open-sourced and publicly available\footnote{https://github.com/k2-fsa/icefall}.
【10】 Adapting self-supervised models to multi-talker speech recognition using speaker embeddings标题：利用说话人嵌入使自监督模型适应多说话人语音识别链接：https://arxiv.org/abs/2211.00482
* 与cs.SD语音【20】为同一篇
作者：Zili Huang,Desh Raj,Paola García,Sanjeev Khudanpur机构：Center for Language and Speech Processing and Human Language Technology Center of Excellence, Johns Hopkins University, Baltimore, USA备注：submitted to ICASSP 2023摘要：自监督学习（SSL）方法在语音处理任务中，特别是在单说话者应用中得到了普及，该方法在没有显式监督的情况下学习数据的表示。然而，这些模型对于多说话者情形通常具有降级的性能--可能是由于域不匹配--这严重地限制了它们在此类应用中的使用。本文研究了在两种情况下，上游SSL模型对多说话者自动语音识别（ASR）任务的适应性。首先，当给定分段话语时，我们表明添加基于注册嵌入的目标说话人提取（TSE）模块是对混合感知预训练的补充。其次，针对未分割的混合语音，提出了一种新的联合说话人建模（JSM）方法，该方法通过对混合语音中所有说话人的信息进行嵌入来聚合说话人信息。通过在Libri 2 Mix上的控制实验，我们表明使用说话人嵌入对于分割和未分割的情况分别提供了9.1%和42.1%的相对WER改善。我们还通过在AMI数据集上的实验证明了我们的模型对真实会话混合的有效性。摘要：Self-supervised learning (SSL) methods which learn representations of data without explicit supervision have gained popularity in speech-processing tasks, particularly for single-talker applications. However, these models often have degraded performance for multi-talker scenarios -- possibly due to the domain mismatch -- which severely limits their use for such applications. In this paper, we investigate the adaptation of upstream SSL models to the multi-talker automatic speech recognition (ASR) task under two conditions. First, when segmented utterances are given, we show that adding a target speaker extraction (TSE) module based on enrollment embeddings is complementary to mixture-aware pre-training. Second, for unsegmented mixtures, we propose a novel joint speaker modeling (JSM) approach, which aggregates information from all speakers in the mixture through their embeddings. With controlled experiments on Libri2Mix, we show that using speaker embeddings provides relative WER improvements of 9.1% and 42.1% over strong baselines for the segmented and unsegmented cases, respectively. We also demonstrate the effectiveness of our models for real conversational mixtures through experiments on the AMI dataset.
【11】 Metric Learning for User-defined Keyword Spotting标题：用于用户定义的关键字识别的度量学习链接：https://arxiv.org/abs/2211.00439
* 与cs.SD语音【21】为同一篇
作者：Jaemin Jung,Youkyum Kim,Jihwan Park,Youshin Lim,Byeong-Yeol Kim,Youngjoon Jang,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea, Hyundai Motor Company,dot Inc., Seoul, Republic of Korea摘要：本工作的目标是检测用户定义的新口语术语。虽然大多数以前的工作解决关键字定位（KWS）作为一个封闭集分类问题，这限制了他们的可转移性，以看不见的术语。定义自定义关键字的能力在用户体验方面具有优势。 本文提出了一种基于度量学习的用户自定义关键词识别训练策略。特别是，我们做出了以下贡献：（1）利用已有的语音语料库构建大规模的关键词数据集，提出一种过滤方法去除影响模型训练的数据;（2）提出了一种基于度量学习的两阶段训练策略，并证明了该方法通过丰富关键词的表示形式，提高了自定义关键词识别任务的性能;（3）为了便于在用户自定义的知识服务领域进行公平比较，提出了统一的评价协议和度量标准。 我们提出的系统不需要对用户定义的关键词进行增量训练，并且在使用所提出的以及现有度量的Google Speech Commands数据集上显著优于先前的工作。摘要：The goal of this work is to detect new spoken terms defined by users. While most previous works address Keyword Spotting (KWS) as a closed-set classification problem, this limits their transferability to unseen terms. The ability to define custom keywords has advantages in terms of user experience. In this paper, we propose a metric learning-based training strategy for user-defined keyword spotting. In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training; (2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics. Our proposed system does not require an incremental training on the user-defined keywords, and outperforms previous works by a significant margin on the Google Speech Commands dataset using the proposed as well as the existing metrics.
【12】 Disentangled representation learning for multilingual speaker recognition标题：用于多语种说话人识别的解缠表征学习链接：https://arxiv.org/abs/2211.00437
* 与cs.SD语音【22】为同一篇
作者：Kihyun Nam,Youkyum Kim,Hee Soo Heo,Jee-weon Jung,Joon Son Chung机构：Korea Advanced Institute of Science and Technology, South Korea, Naver Corporation, South Korea摘要：本文的目标是训练对双语说话场景具有鲁棒性的说话人嵌入。世界上大多数人口至少讲两种语言;然而，当以不同语言说话时，大多数说话人识别系统不能识别同一说话人。 现有的说话人识别评价集没有考虑双语环境，难以分析双语说话人对说话人识别性能的影响。本文在VoxCeleb的基础上，提出了一种新的考虑双语场景的大规模评价集。我们还介绍了一种表征学习策略，该策略将语言信息从说话人表征中分离出来，以考虑双语场景。这种语言解纠缠的表征学习策略可以适应现有的模型，只需对训练管道进行小的改变。 实验结果表明，在双语测试集上，基线模型的性能明显下降.与此相反，使用该策略训练的模型在双语测试场景下表现出了显著的改进，同时在现有的单语测试集上保持了竞争性能。摘要：The goal of this paper is to train speaker embeddings that are robust to bilingual speaking scenario. The majority of the world's population speak at least two languages; however, most speaker recognition systems fail to recognise the same speaker when speaking in different languages. Popular speaker recognition evaluation sets do not consider the bilingual scenario, making it difficult to analyse the effect of bilingual speakers on speaker recognition performance. This paper proposes a new large-scale evaluation set derived from VoxCeleb that considers bilingual scenarios. We also introduce a representation learning strategy, which disentangles language information from speaker representation to account for the bilingual scenario. This language-disentangled representation learning strategy can be adapted to existing models with small changes to the training pipeline. Experimental results demonstrate that the baseline models suffer significant performance degradation when evaluated on the proposed bilingual test set. On the contrary, the model trained with the proposed disentanglement strategy shows significant improvement under the bilingual evaluation scenario while simultaneously retaining competitive performance on existing monolingual test sets.
【13】 Speech-text based multi-modal training with bidirectional attention for improved speech recognition标题：基于语音-文本的双向注意多模式训练改进语音识别链接：https://arxiv.org/abs/2211.00325
* 与cs.SD语音【23】为同一篇
作者：Yuhang Yang,Haihua Xu,Hao Huang,Eng Siong Chng,Sheng Li机构：School of Information Science and Engineering, Xinjiang University, China, Bytedance AI Lab, Singapore, School of Computer Science and Engineering, Nanyang Technological University, Singapore备注：5 pages, 3 figures, 3 tables摘要：为了让最先进的端到端ASR模型享受数据效率，以及通过多模态训练获得更多的不成对文本数据，需要解决两个问题：1）语音和语言（又称文本数据）之间的特征采样率的同步性; 2）来自两个编码器的学习表示的同质性。本文提出了一种新的双向注意机制（BiAM），利用多模态学习方法对ASR编码器（底层）和文本编码器进行联合学习。BiAM是为了便于特征采样率的交换，在另一个空间中实现待测量的一种类型的变换特征的质量，具有多样化的目标函数。结果，语音表示被丰富了更多的语言信息，而由文本编码器生成的表示与对应的语音表示更相似，因此共享ASR模型更适合于不成对文本数据预训练。为了验证所提出方法的有效性，我们进行了两类实验，有和没有额外的不成对文本数据。在Librispeech语料库上的实验结果表明，仅使用成对数据学习时，该算法的误词率降低率最高可达6.15%，而使用更多非成对文本数据时，误词率降低率为9.23%。摘要：To let the state-of-the-art end-to-end ASR model enjoy data efficiency, as well as much more unpaired text data by multi-modal training, one needs to address two problems: 1) the synchronicity of feature sampling rates between speech and language (aka text data); 2) the homogeneity of the learned representations from two encoders. In this paper we propose to employ a novel bidirectional attention mechanism (BiAM) to jointly learn both ASR encoder (bottom layers) and text encoder with a multi-modal learning method. The BiAM is to facilitate feature sampling rate exchange, realizing the quality of the transformed features for the one kind to be measured in another space, with diversified objective functions. As a result, the speech representations are enriched with more linguistic information, while the representations generated by the text encoder are more similar to corresponding speech ones, and therefore the shared ASR models are more amenable for unpaired text data pretraining. To validate the efficacy of the proposed method, we perform two categories of experiments with or without extra unpaired text data. Experimental results on Librispeech corpus show it can achieve up to 6.15% word error rate reduction (WERR) with only paired data learning, while 9.23% WERR when more unpaired text data is employed.
【14】 Waveform Boundary Detection for Partially Spoofed Audio标题：部分欺骗音频的波形边界检测链接：https://arxiv.org/abs/2211.00226
* 与cs.SD语音【24】为同一篇
作者：Zexin Cai,Weiqing Wang,Ming Li机构：Department of Electrical and Computer Engineering, Duke University, Durham, NC , USA, Data Science Research Center, Duke Kunshan University, Kunshan , PR China摘要：提出了一种针对音频欺骗攻击的波形边界检测系统。部分欺骗/伪造音频，其中部分话语被替换为合成或自然音频剪辑，最近已被报告为音频深度伪造的一种场景。由于deepfakes可能对社会安全构成威胁，因此检测此类欺骗音频至关重要。因此，我们提出了一种基于深度学习的帧级检测系统来解决这个问题，该系统可以检测部分欺骗音频并定位被操纵的片段。我们提出的方法在ADD 2022挑战提供的数据上进行了训练和评估。我们评估了我们的检测模型，涉及各种声学特征和网络配置。因此，我们的检测系统在ADD 2022挑战测试集上实现了6.58%的等错误率（EER），这是能够定位操纵片段的部分欺骗音频检测系统中的最佳性能。摘要：The present paper proposes a waveform boundary detection system for audio spoofing attacks containing partially manipulated segments. Partially spoofed/fake audio, where part of the utterance is replaced, either with synthetic or natural audio clips, has recently been reported as one scenario of audio deepfakes. As deepfakes can be a threat to social security, the detection of such spoofing audio is essential. Accordingly, we propose to address the problem with a deep learning-based frame-level detection system that can detect partially spoofed audio and locate the manipulated pieces. Our proposed method is trained and evaluated on data provided by the ADD2022 Challenge. We evaluate our detection model concerning various acoustic features and network configurations. As a result, our detection system achieves an equal error rate (EER) of 6.58% on the ADD2022 challenge test set, which is the best performance in partially spoofed audio detection systems that can locate manipulated clips.
【15】 ImagineNET: Target Speaker Extraction with Intermittent Visual Cue through Embedding Inpainting标题：ImagineNET：基于嵌入修复的间歇性视觉提示目标说话人提取链接：https://arxiv.org/abs/2211.00109
* 与cs.SD语音【25】为同一篇
作者：Zexu Pan,Wupeng Wang,Marvin Borsdorf,Haizhou Li机构：Institute of Data Science, National University of Singapore (NUS), Singapore, Department of Electrical and Computer Engineering, NUS, Singapore, Machine Listening Lab, University of Bremen, Germany摘要：说话人提取技术寻求从语音混合中的干扰话音中挑选出目标说话人的话音。典型地，目标说话者的辅助参考被用于形成自主注意。视频剪辑中的预先记录的话语或同步的嘴唇运动可以用作辅助参考。视觉线索的使用不仅可行，而且由于其对噪声的鲁棒性而有效，并逐渐受到人们的青睐。然而，在可能发生视觉遮挡或间歇通信的真实世界应用中，难以保证这种并行视觉提示总是可用的。本文研究了间断视觉线索下的音视频说话人提取算法。提出了一种基于说话人提取和视觉嵌入的联合修复框架。为了促进两个任务之间的交互，它们以交错结构交替执行并联合优化。我们还提出了两种类型的视觉修复损失，并研究了我们提出的方法与两种类型的普遍使用的视觉嵌入。实验结果表明，该算法在信号质量、感知质量和可懂度方面均优于基线算法。摘要：The speaker extraction technique seeks to single out the voice of a target speaker from the interfering voices in a speech mixture. Typically an auxiliary reference of the target speaker is used to form voluntary attention. Either a pre-recorded utterance or a synchronized lip movement in a video clip can serve as the auxiliary reference. The use of visual cue is not only feasible, but also effective due to its noise robustness, and becoming popular. However, it is difficult to guarantee that such parallel visual cue is always available in real-world applications where visual occlusion or intermittent communication can occur. In this paper, we study the audio-visual speaker extraction algorithms with intermittent visual cue. We propose a joint speaker extraction and visual embedding inpainting framework to explore the mutual benefits. To encourage the interaction between the two tasks, they are performed alternately with an interlacing structure and optimized jointly. We also propose two types of visual inpainting losses and study our proposed method with two types of popularly used visual embeddings. The experimental results show that we outperform the baseline in terms of signal quality, perceptual quality, and intelligibility.
【16】 An analysis of degenerating speech due to progressive dysarthria on ASR performance标题：进行性构音障碍致退行性语音对ASR表现的影响分析链接：https://arxiv.org/abs/2211.00089
* 与cs.SD语音【26】为同一篇
作者：Katrin Tomanek,Katie Seaver,Pan-Pan Jiang,Richard Cave,Lauren Harrel,Jordan R. Green机构：Google LLC,MGH Institute of Health Professions,Language and Cognition, UCL备注：Submitted to ICASSP 2023摘要：尽管个性化自动语音识别（ASR）模型最近已经被设计成识别甚至严重受损的语音，但是对于具有退化语音的人来说，模型性能可能随着时间而降低。本研究的目的是（1）分析语音退化个体的ASR性能随时间的变化，（2）探索缓解策略，以优化整个疾病进展过程中的识别。4名因肌萎缩性侧索硬化症（ALS）而言语退化的个体记录了言语。针对三种ASR模型计算了记录会话中的单词错误率（WER）：非自适应说话人独立（U-SI）、自适应说话人独立（A-SI）和自适应说话人相关（A-SD或个性化）。随着时间的推移，所有三种模型的性能都随着语音受损程度的加重而显著下降，但当使用语音进展严重阶段的记录更新A-SD模型时，其性能显著改善。在语言明显退化之前，在疾病早期记录额外的话语并没有改善A-SD模型的性能。总的来说，我们的研究结果强调了在为进行性言语障碍患者提供个性化模型时，连续记录（和模型再训练）的重要性。摘要：Although personalized automatic speech recognition (ASR) models have recently been designed to recognize even severely impaired speech, model performance may degrade over time for persons with degenerating speech. The aims of this study were to (1) analyze the change of performance of ASR over time in individuals with degrading speech, and (2) explore mitigation strategies to optimize recognition throughout disease progression. Speech was recorded by four individuals with degrading speech due to amyotrophic lateral sclerosis (ALS). Word error rates (WER) across recording sessions were computed for three ASR models: Unadapted Speaker Independent (U-SI), Adapted Speaker Independent (A-SI), and Adapted Speaker Dependent (A-SD or personalized). The performance of all three models degraded significantly over time as speech became more impaired, but the performance of the A-SD model improved markedly when it was updated with recordings from the severe stages of speech progression. Recording additional utterances early in the disease before speech degraded significantly did not improve the performance of A-SD models. Overall, our findings emphasize the importance of continuous recording (and model retraining) when providing personalized models for individuals with progressive speech impairments.
【17】 Magnitude or Phase? A Two Stage Algorithm for Dereverberation标题：量级还是阶段性？一种两阶段的混响消除算法链接：https://arxiv.org/abs/2211.00607
* 与cs.SD语音【1】为同一篇
作者：Ayal Schwartz,Sharon Gannot,Shlomo E. Chazan机构： OriginAI, Ramat-Gan, Israel, Bar-Ilan University, Ramat-Gan, Israel摘要：本文提出了一种新的单麦克风语音去混响算法。首先，通过性能分析，说明了单纯提高幅度或相位的算法是不够好的。此外，我们证明了少数客观测量与清洁星等高度相关，而其他与清洁相位高度相关。因此，我们提出了一个新的架构，它由两个子模型组成，每个子模型负责不同的任务。第一模型估计给定噪声输入的干净幅度。然后，增强的幅度与噪声输入相位一起被用作第二模型的输入，以估计去混响信号的实部和虚部。本文提出了一种包括预训练和微调的训练方案。我们使用来自REVERB挑战的数据评估我们提出的方法，并将我们的结果与其他方法进行比较。我们证明了在所有测量中的一致改进，这可以归因于幅度和相位的改进估计。摘要：In this work we present a new single-microphone speech dereverberation algorithm. First, a performance analysis is presented to interpret that algorithms focused on improving solely magnitude or phase are not good enough. Furthermore, we demonstrate that few objective measurements have high correlation with the clean magnitude while others with the clean phase. Consequently ,we propose a new architecture which consists of two sub-models, each of which is responsible for a different task. The first model estimates the clean magnitude given the noisy input. The enhanced magnitude together with the noisy-input phase are then used as inputs to the second model to estimate the real and imaginary portions of the dereverberated signal. A training scheme including pre-training and fine-tuning is presented in the paper. We evaluate our proposed approach using data from the REVERB challenge and compare our results to other methods. We demonstrate consistent improvements in all measures, which can be attributed to the improved estimates of both the magnitude and the phase.
【18】 T5lephone: Bridging Speech and Text Self-supervised Models for Spoken Language Understanding via Phoneme level T5标题：T5lephone：通过音素级别T5为口语理解搭建语音和文本自我监督模型的桥梁链接：https://arxiv.org/abs/2211.00586
* 与cs.SD语音【2】为同一篇
作者：Chan-Jan Hsu,Ho-Lam Chung,Hung-yi Lee,Yu Tsao机构： Tsao 3 1MediaTek Research 2National Taiwan University摘要：在口语理解（SLU）中，自然的解决方案是将预先训练的语音模型（例如，HuBERT）和预先训练的语言模型（PLM，例如，T5）级联。大多数先前的工作使用预先训练的语言模型和基于子词的标记化。然而，输入单元的粒度影响语音模型输出和语言模型输入的对齐，并且基于字符的标记化的PLM未被充分开发。本研究旨在探讨不同标记化策略的PLM对口语理解任务（包括口语问答和语音翻译）的影响。我们进一步扩展了这个想法，创建了T5lephone（发音为telephone），它是T5的一个变体，使用音位化文本进行了预训练。我们使用现有PLM初始化T5lephone，以使用相对轻量级的计算资源对其进行预训练。我们在NMSQA上达到了最先进的水平，T5lephone型号在端到端SQA和ST上超过了T5和其他类型的单元。摘要：In Spoken language understanding (SLU), a natural solution is concatenating pre-trained speech models (e.g. HuBERT) and pretrained language models (PLM, e.g. T5). Most previous works use pretrained language models with subword-based tokenization. However, the granularity of input units affects the alignment of speech model outputs and language model inputs, and PLM with character-based tokenization is underexplored. In this work, we conduct extensive studies on how PLMs with different tokenization strategies affect spoken language understanding task including spoken question answering (SQA) and speech translation (ST). We further extend the idea to create T5lephone(pronounced as telephone), a variant of T5 that is pretrained using phonemicized text. We initialize T5lephone with existing PLMs to pretrain it using relatively lightweight computational resources. We reached state-of-the-art on NMSQA, and the T5lephone model exceeds T5 with other types of units on end-to-end SQA and ST.
【19】 No-audio speaking status detection in crowded settings via visual pose-based filtering and wearable acceleration标题：通过基于视觉姿势的过滤和可穿戴式加速在拥挤环境中检测无音频说话状态链接：https://arxiv.org/abs/2211.00549
* 与cs.SD语音【3】为同一篇
作者：Jose Vargas-Quiros,Laura Cabrera-Quiros,Hayley Hung机构： Hung were with the Department of Intelligent Systemsat TU Delft摘要：在拥挤的场景中识别谁在说话是理解其中正在进行的社会互动的一个关键挑战。仅从身体运动检测说话状态为分析其中无法获得个人音频的社交场景打开了大门。视频和可穿戴传感器使它能够以一种不引人注目的、保护隐私的方式识别说话。当考虑视频模态时，在动作识别问题中，传统上使用边界框来定位和分割出目标对象，然后识别在其中发生的动作。然而，交叉污染、遮挡和人体的关节特性使得该方法在拥挤场景中具有挑战性。这里，我们利用铰接的身体姿势来进行对象定位和随后的语音检测阶段。实验结果表明，选取姿态关键点周围的局部特征对泛化性能有积极的影响，同时也显著减少了所考虑的局部特征的数量，使得方法更加有效。利用两个不同视角的野外数据集，我们研究了交叉污染在这一效应中的作用。此外，我们还利用了通过可穿戴传感器测量的加速度来完成相同的任务，并提出了一种结合这两种方法的多模态方法。摘要：Recognizing who is speaking in a crowded scene is a key challenge towards the understanding of the social interactions going on within. Detecting speaking status from body movement alone opens the door for the analysis of social scenes in which personal audio is not obtainable. Video and wearable sensors make it possible recognize speaking in an unobtrusive, privacy-preserving way. When considering the video modality, in action recognition problems, a bounding box is traditionally used to localize and segment out the target subject, to then recognize the action taking place within it. However, cross-contamination, occlusion, and the articulated nature of the human body, make this approach challenging in a crowded scene. Here, we leverage articulated body poses for subject localization and in the subsequent speech detection stage. We show that the selection of local features around pose keypoints has a positive effect on generalization performance while also significantly reducing the number of local features considered, making for a more efficient method. Using two in-the-wild datasets with different viewpoints of subjects, we investigate the role of cross-contamination in this effect. We additionally make use of acceleration measured through wearable sensors for the same task, and present a multimodal approach combining both methods.
【20】 Learning utterance-level representations through token-level acoustic latents prediction for Expressive Speech Synthesis标题：表现性语音合成中基于令牌级声学潜伏期预测的语音级表示学习链接：https://arxiv.org/abs/2211.00523
* 与cs.SD语音【4】为同一篇
作者：Karolos Nikitaras,Konstantinos Klapsas,Nikolaos Ellinas,Georgia Maniati,June Sig Sung,Inchul Hwang,Spyros Raptis,Aimilios Chalamandaris,Pirros Tsiakoulis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：本文提出了一种表达性语音合成模型，该模型利用符号级潜在韵律变量来捕获和控制话语级属性，如角色的动作声音和说话风格。当前的工作旨在将这种细粒度和话语级的语音属性显式地分解成由在相应级别中操作的模块提取的不同表示。我们证明了细粒度的潜在空间也能捕获粗粒度的信息，随着潜在空间维数的增加，这种捕获更明显，以捕获不同的韵律表示。因此，在标记级和话语级表示的多样性和它们的解纠缠之间出现了折衷。我们通过首先将丰富的语音属性捕获到标记级潜在空间中，然后单独训练给定输入文本的先验网络，学习话语级表示，以便预测在前一步骤中提取的音素级后验潜在，来缓解该问题。通过定性和定量评价，验证了该方法的有效性。音频样本可在我们的演示页面。摘要：This paper proposes an Expressive Speech Synthesis model that utilizes token-level latent prosodic variables in order to capture and control utterance-level attributes, such as character acting voice and speaking style. Current works aim to explicitly factorize such fine-grained and utterance-level speech attributes into different representations extracted by modules that operate in the corresponding level. We show that the fine-grained latent space also captures coarse-grained information, which is more evident as the dimension of latent space increases in order to capture diverse prosodic representations. Therefore, a trade-off arises between the diversity of the token-level and utterance-level representations and their disentanglement. We alleviate this issue by first capturing rich speech attributes into a token-level latent space and then, separately train a prior network that given the input text, learns utterance-level representations in order to predict the phoneme-level, posterior latents extracted during the previous step. Both qualitative and quantitative evaluations are used to demonstrate the effectiveness of the proposed approach. Audio samples are available in our demo page.
【21】 TrimTail: Low-Latency Streaming ASR with Simple but Effective Spectrogram-Level Length Penalty标题：TrimTail：低延迟流ASR，具有简单但有效的频谱图级长度惩罚链接：https://arxiv.org/abs/2211.00522
* 与cs.SD语音【5】为同一篇
作者：Xingchen Song,Di Wu,Zhiyong Wu,Binbin Zhang,Yuekai Zhang,Zhendong Peng,Wenpeng Li,Fuping Pan,Changbao Zhu机构：Tsinghua Univ., Beijing, China, Horizon Inc., Beijing, China, WeNet Open Source Community备注：submitted to ICASSP 2023摘要：本文提出了一种简单有效的发射正则化方法TrimTail，用于改善流式ASR模型的延迟。TrimTail的核心思想是应用长度惩罚（即，通过修剪尾随帧，参见图1-（b）），这不需要任何对准。我们证明了TrimTail在计算上是廉价的，并且可以在线应用，并且通过将其应用于各种端到端流式ASR网络（使用CTC损失[1]或换能器损失[2]进行训练），在任何数据集上使用任何训练损失或任何模型架构进行优化，而无需任何额外工作。我们在Aishell-1和Librispeech上实现了100 $\sim $200ms的延迟减少，同时具有相同甚至更高的准确性。此外，通过使用TrimTail，我们可以实现用户敏感延迟（USD）的400 ms算法改进，并且精度损失小于0.2。摘要：In this paper, we present TrimTail, a simple but effective emission regularization method to improve the latency of streaming ASR models. The core idea of TrimTail is to apply length penalty (i.e., by trimming trailing frames, see Fig. 1-(b)) directly on the spectrogram of input utterances, which does not require any alignment. We demonstrate that TrimTail is computationally cheap and can be applied online and optimized with any training loss or any model architecture on any dataset without any extra effort by applying it on various end-to-end streaming ASR networks either trained with CTC loss [1] or Transducer loss [2]. We achieve 100 $\sim$ 200ms latency reduction with equal or even better accuracy on both Aishell-1 and Librispeech. Moreover, by using TrimTail, we can achieve a 400ms algorithmic improvement of User Sensitive Delay (USD) with an accuracy loss of less than 0.2.
【22】 Modelling black-box audio effects with time-varying feature modulation标题：用时变特征调制对黑盒音频效果进行建模链接：https://arxiv.org/abs/2211.00497
* 与cs.SD语音【6】为同一篇
作者：Marco Comunità,Christian J. Steinmetz,Huy Phan,Joshua D. Reiss机构：Centre for Digital Music, Queen Mary University of London, UK摘要：用于音频效果黑箱建模的深度学习方法已经显示出前景，然而，大多数现有工作集中在行为在相对较短的时间尺度上的非线性效果，如吉他放大器和失真。虽然循环和卷积架构在理论上可以扩展到在更长时间尺度上捕获行为，但我们表明，在对诸如模糊和动态范围压缩之类的音频效果进行建模时，简单地缩放现有架构的宽度、深度或膨胀因子并不能产生令人满意的性能。为了解决这一问题，我们提出将时变特征线性调制集成到现有的时域卷积骨干中，这是一种能够实现中间激活的可学习自适应的方法。我们证明，我们的方法更准确地捕捉了一系列模糊和压缩器实现在时域和频域指标上的长期依赖性。我们提供可靠的示例、源代码和预先训练的模型，以提高可重复性。摘要：Deep learning approaches for black-box modelling of audio effects have shown promise, however, the majority of existing work focuses on nonlinear effects with behaviour on relatively short time-scales, such as guitar amplifiers and distortion. While recurrent and convolutional architectures can theoretically be extended to capture behaviour at longer time scales, we show that simply scaling the width, depth, or dilation factor of existing architectures does not result in satisfactory performance when modelling audio effects such as fuzz and dynamic range compression. To address this, we propose the integration of time-varying feature-wise linear modulation into existing temporal convolutional backbones, an approach that enables learnable adaptation of the intermediate activations. We demonstrate that our approach more accurately captures long-range dependencies for a range of fuzz and compressor implementations across both time and frequency domain metrics. We provide sound examples, source code, and pretrained models to faciliate reproducibility.
【23】 Generating Gender-Ambiguous Text-to-Speech Voices标题：生成性别模糊的文本到语音转换语音链接：https://arxiv.org/abs/2211.00375
* 与cs.SD语音【7】为同一篇
作者：Konstantinos Markopoulos,Georgia Maniati,Georgios Vamvoukakis,Nikolaos Ellinas,Karolos Nikitaras,Konstantinos Klapsas,Georgios Vardaxoglou,Panos Kakoulidis,June Sig Sung,Inchul Hwang,Aimilios Chalamandaris,Pirros Tsiakoulis,Spyros Raptis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：语音助手或任何语音用户界面的性别是其感知身份的中心元素。虽然女性的声音是一种常见的选择，但人们越来越感兴趣的是性别不明确而不是明确确定为女性或男性的替代方法。这项工作解决了生成不对应于任何现有人的性别模糊的文本到语音（TTS）声音的任务。这是通过从潜在说话人嵌入空间采样来实现的，该潜在说话人嵌入空间是在对来自多个男性和女性说话人的数据训练多语言、多说话人TTS系统时形成的。调查了有关取样过程的各种选项。在实验中，我们评估了不同的采样选择对语音的性别模糊度和自然度的影响。实验结果表明，该方法能够有效地生成比基线平均说话人嵌入更好的新说话人。据我们所知，这是第一个能够可靠地产生一系列性别不明确的声音以满足不同用户要求的系统方法。摘要：The gender of a voice assistant or any voice user interface is a central element of its perceived identity. While a female voice is a common choice, there is an increasing interest in alternative approaches where the gender is ambiguous rather than clearly identifying as female or male. This work addresses the task of generating gender-ambiguous text-to-speech (TTS) voices that do not correspond to any existing person. This is accomplished by sampling from a latent speaker embeddings' space that was formed while training a multilingual, multi-speaker TTS system on data from multiple male and female speakers. Various options are investigated regarding the sampling process. In our experiments, the effects of different sampling choices on the gender ambiguity and the naturalness of the resulting voices are evaluated. The proposed method is shown able to efficiently generate novel speakers that are superior to a baseline averaged speaker embedding. To our knowledge, this is the first systematic approach that can reliably generate a range of gender-ambiguous voices to meet diverse user requirements.
【24】 Investigating Content-Aware Neural Text-To-Speech MOS Prediction Using Prosodic and Linguistic Features标题：基于韵律和语言特征的内容感知神经文语转换MOS预测研究链接：https://arxiv.org/abs/2211.00342
* 与cs.SD语音【8】为同一篇
作者：Alexandra Vioni,Georgia Maniati,Nikolaos Ellinas,June Sig Sung,Inchul Hwang,Aimilios Chalamandaris,Pirros Tsiakoulis机构：⋆ Innoetics, Samsung Electronics, Greece, † Mobile eXperience Business, Samsung Electronics, Republic of Korea备注：Submitted to ICASSP 2023摘要：当前用于自动合成语音评估的现有技术的方法是基于MOS预测神经模型的。这样的MOS预测模型包括使用频谱特征作为输入的MOSNet和LDNet，以及依赖于直接使用语音信号作为输入的预训练的自监督学习模型的SSL-MOS。在现代高质量的神经TTS系统中，韵律相对于口语内容的适当性是语音自然度的决定性因素。因此，我们建议将韵律和语言特征作为MOS预测系统的附加输入，并评估它们对预测结果的影响。我们将音素层F0和持续时间特征作为韵律输入，将Tacotron编码器输出、词性标签和BERT嵌入作为高级语言输入。所有MOS预测系统都在SOMOS上训练，SOMOS是一个具有众包自然度MOS评估的神经TTS数据集。结果表明，所提出的附加特征在MOS预测任务中是有益的，通过在话语级和系统级预测两者上改进预测的MOS得分与地面真实的相关性。摘要：Current state-of-the-art methods for automatic synthetic speech evaluation are based on MOS prediction neural models. Such MOS prediction models include MOSNet and LDNet that use spectral features as input, and SSL-MOS that relies on a pretrained self-supervised learning model that directly uses the speech signal as input. In modern high-quality neural TTS systems, prosodic appropriateness with regard to the spoken content is a decisive factor for speech naturalness. For this reason, we propose to include prosodic and linguistic features as additional inputs in MOS prediction systems, and evaluate their impact on the prediction outcome. We consider phoneme level F0 and duration features as prosodic inputs, as well as Tacotron encoder outputs, POS tags and BERT embeddings as higher-level linguistic inputs. All MOS prediction systems are trained on SOMOS, a neural TTS-only dataset with crowdsourced naturalness MOS evaluations. Results show that the proposed additional features are beneficial in the MOS prediction task, by improving the predicted MOS scores' correlation with the ground truths, both at utterance-level and system-level predictions.
【25】 SDMuse: Stochastic Differential Music Editing and Generation via Hybrid Representation标题：SDMuse：基于混合表示的随机差分音乐编辑与生成链接：https://arxiv.org/abs/2211.00222
* 与cs.SD语音【9】为同一篇
作者：Chen Zhang,Yi Ren,Kejun Zhang,Shuicheng Yan机构：Zhejiang University, China, Sea AI Lab, Singapore摘要：虽然深度生成模型已经使音乐生成成为可能，但是在细粒度下编辑现有音乐作品仍然是一个具有挑战性和未被探索的问题。本文提出了一个统一的随机微分音乐编辑和生成框架SDMuse，它不仅可以从零开始合成一个完整的音乐作品，而且可以对现有的音乐作品进行多种修改，如组合、延续、修复和风格转换。SDMuse遵循两级流水线，在钢琴滚动和MIDI事件的混合表示之上实现音乐生成和编辑。具体地，SDMuse首先通过基于扩散模型生成先验的随机微分方程（SDE）迭代地去噪来生成/编辑钢琴滚动，然后精炼所生成的钢琴滚动并自回归地预测MIDI事件标记。我们在ailabs 1 k7流行音乐数据集上对我们的方法生成的音乐在各种音乐编辑和生成任务上的质量和可控性进行了评估。实验结果证明了我们提出的随机微分音乐编辑和生成过程以及混合表示的有效性。摘要：While deep generative models have empowered music generation, it remains a challenging and under-explored problem to edit an existing musical piece at fine granularity. In this paper, we propose SDMuse, a unified Stochastic Differential Music editing and generation framework, which can not only compose a whole musical piece from scratch, but also modify existing musical pieces in many ways, such as combination, continuation, inpainting, and style transferring. The proposed SDMuse follows a two-stage pipeline to achieve music generation and editing on top of a hybrid representation including pianoroll and MIDI-event. In particular, SDMuse first generates/edits pianoroll by iteratively denoising through a stochastic differential equation (SDE) based on a diffusion model generative prior, and then refines the generated pianoroll and predicts MIDI-event tokens auto-regressively. We evaluate the generated music of our method on ailabs1k7 pop music dataset in terms of quality and controllability on various music editing and generation tasks. Experimental results demonstrate the effectiveness of our proposed stochastic differential music editing and generation process, as well as the hybrid representations.
【26】 Active Learning of Non-semantic Speech Tasks with Pretrained Models标题：基于预训练模型的非语义语音任务主动学习链接：https://arxiv.org/abs/2211.00119
* 与cs.SD语音【10】为同一篇
作者：Harlin Lee,Aaqib Saeed,Andrea L. Bertozzi机构：University of California Los Angeles, Los Angeles, CA, USA, Philips Research, Eindhoven, The Netherlands摘要：利用大量未标记数据集对神经网络进行预训练，可以为深度模型求解下游任务提供更好的先验知识，因而受到了广泛的关注。然而，这种方法通常假设对于下游任务，我们可以访问足够大小的注释数据。在本研究中，我们提出了一个新的系统ALOE，用于提高非语义语音任务的数据和标签效率与主动学习（AL）。ALOE结合主动学习使用预先训练的模型来递增地标记数据，并学习用于下游任务的分类器，从而减少预先获取标记数据的需要。我们证明了ALOE在各种任务、基于不确定性的获取函数和模型结构上的有效性。使用ALOE在冻结的编码器上训练线性分类器被示出为实现与利用整个标记数据的若干基线类似的性能。摘要：Pretraining neural networks with massive unlabeled datasets have become popular as it equips the deep models with a better prior to solve downstream tasks. However, this approach generally assumes that for downstream tasks, we have access to annotated data of sufficient size. In this work, we propose ALOE, a novel system for improving the data- and label-efficiency of non-semantic speech tasks with active learning (AL). ALOE uses pre-trained models in conjunction with active learning to label data incrementally and learns classifiers for downstream tasks, thereby mitigating the need to acquire labeled data beforehand. We demonstrate the effectiveness of ALOE on a wide range of tasks, uncertainty-based acquisition functions, and model architectures. Training a linear classifier on top of a frozen encoder with ALOE is shown to achieve performance similar to several baselines that utilize the entire labeled data.
机器翻译，仅供参考
每日语音速递