今日论文合集：cs.SD语音40篇，eess.AS音频处理40篇。
【1】 Efficient Speech Translation with Dynamic Latent Perceivers标题：基于动态潜在感知器的高效语音翻译链接：https://arxiv.org/abs/2210.16264
作者：Ioannis Tsiamas,Gerard I. Gállego,José A. R. Fonollosa,Marta R. Costa-jussá机构：Universitat Politecnica de Catalunya, Barcelona, Marta R. Costa-jussa, Meta AI, Paris摘要：近年来，Transformers已成为语音翻译的主流架构，在翻译质量方面取得了显著的改进。由于语音信号比其文本对应物长，并且由于Transformer的二次复杂性，下采样步骤对于其在语音翻译中的采用是必不可少的。相反，在本研究中，我们建议使用感知器编码器将语音输入映射为固定长度的潜在表示，以降低复杂度。此外，我们还引入了一种新的训练感知器的方法，即动态潜在访问（DLA），在不增加任何计算开销的情况下释放更大的潜在空间。采用DLA的语音到文本感知器可以在Must-C的三种语言对中与Transformer基线的性能相匹配。最后，经过DLA训练的模型在推理时容易适应DLA，并且可以灵活地部署在各种计算预算下，而不会显著降低翻译质量。摘要：Transformers have been the dominant architecture for Speech Translation in recent years, achieving significant improvements in translation quality. Since speech signals are longer than their textual counterparts, and due to the quadratic complexity of the Transformer, a down-sampling step is essential for its adoption in Speech Translation. Instead, in this research, we propose to ease the complexity by using a Perceiver encoder to map the speech inputs to a fixed-length latent representation. Furthermore, we introduce a novel way of training Perceivers, with Dynamic Latent Access (DLA), unlocking larger latent spaces without any additional computational overhead. Speech-to-Text Perceivers with DLA can match the performance of a Transformer baseline across three language pairs in MuST-C. Finally, a DLA-trained model is easily adaptable to DLA at inference, and can be flexibly deployed with various computational budgets, without significant drops in translation quality.
【2】 Universal speaker recognition encoders for different speech segments duration标题：适用于不同语音段时长的通用说话人识别编码器链接：https://arxiv.org/abs/2210.16231
作者：Sergey Novoselov,Vladimir Volokhov,Galina Lavrentyeva机构：ITMO University, St.Petersburg, Russia, STC Ltd., St.Petersburg, Russia备注：Submitted to ICASSP'23摘要：创建对不同声学和语音持续时间条件鲁棒的通用扬声器编码器是当今的一大挑战。根据我们的观察，在短语音片段上训练的系统对于短语说话人确认是最优的，而在长片段上训练的系统对于长片段确认是优越的。同时在汇集的短和长语音段上训练的系统不能给出最佳的验证结果，并且通常对于短和长段都降级。针对不同语音段时长，提出了一种通用的说话人编码器。我们描述了我们的简单配方训练通用说话人编码器的任何类型的选择神经网络架构。根据我们针对NIST SRE和VoxCeleb 1基准测试获得的基于wav 2 vec-TDNN的系统的评估结果，所提出的通用编码器在不同的登记和测试语音段持续时间的情况下提供说话人确认改进。所提出的编码器的关键特征是它具有与所选择的神经网络结构相同的推理时间。摘要：Creating universal speaker encoders which are robust for different acoustic and speech duration conditions is a big challenge today. According to our observations systems trained on short speech segments are optimal for short phrase speaker verification and systems trained on long segments are superior for long segments verification. A system trained simultaneously on pooled short and long speech segments does not give optimal verification results and usually degrades both for short and long segments. This paper addresses the problem of creating universal speaker encoders for different speech segments duration. We describe our simple recipe for training universal speaker encoder for any type of selected neural network architecture. According to our evaluation results of wav2vec-TDNN based systems obtained for NIST SRE and VoxCeleb1 benchmarks the proposed universal encoder provides speaker verification improvements in case of different enrollment and test speech segment duration. The key feature of the proposed encoder is that it has the same inference time as the selected neural network architecture.
【3】 Supervised Contrastive Learning for Respiratory Sound Classification标题：用于呼吸音分类的有监督对比学习链接：https://arxiv.org/abs/2210.16192
作者：Ilyass Moummad,Nicolas Farrugia机构：OSO-AI, Brest, France摘要：由于大的生物变异性、不平衡的数据集以及用于捕获呼吸信号的记录技术的多样性，使用机器学习的自动呼吸声分类是一项具有挑战性的任务。虽然已经提出了具有注释的呼吸循环的数据集，但是基于仅使用注释的监督学习的方法可能在它们的概括能力方面受到限制。在本研究中，我们使用监督对比学习来解决这一问题，依赖于呼吸循环注释和频谱图频率和时间掩蔽方法SpecAugment来生成增强样本，用于具有对比损失的表征学习。我们通过对一个从头训练的卷积神经网络进行实验，证明了这种方法优于监督学习，达到了新的技术水平。我们的工作显示了监督对比学习在不平衡和噪声环境中的潜力。我们的代码发布于https://github.com/ilyassmoummad/scl_icbhi2017摘要：Automatic respiratory sound classification using machine learning is a challenging task, due to large biological variability, imbalanced datasets, as well as a diversity in recording techniques used to capture the respiration signal. While datasets with annotated respiration cycles have been proposed, methods based on supervised learning using annotations only may be limited in their generalization capability. In this study, we address this issue using supervised contrastive learning, relying both on respiration cycle annotations and a spectrogram frequency and temporal masking method SpecAugment to generate augmented samples for representation learning with a contrastive loss. We demonstrate that such an approach can outperform supervised learning using experiments on a convolutional neural network trained from scratch, achieving the new state of the art. Our work shows the potential of supervised contrastive learning in imbalanced and noisy settings. Our code is released at https://github.com/ilyassmoummad/scl_icbhi2017
【4】 Deep Learning Object Detection Approaches to Source Identification标题：用于信源识别的深度学习对象检测方法链接：https://arxiv.org/abs/2210.16173
作者：Luke Wood,Kevin Anderson,Peter Gerstoft机构： University of California, San Diego, La Jolla, CA, USA, Google摘要：传统的源识别是使用基于阈值的能量检测算法来解决的。这些算法经常对区域中的活动进行求和，并将高于特定活动阈值的区域视为源。虽然这些算法适用于大多数情况，但是它们经常不能检测占用小频带的信号，不能区分具有重叠频带的源，并且不能检测在指定信噪比下的任何信号。通过将原始信号数据转换为频谱图，可以将源识别问题归结为目标检测问题。通过利用基于深度学习的对象检测的现代进步，我们提出了一种系统，该系统设法减轻在使用传统源识别算法时遇到的失败情况。我们的贡献包括将源识别作为一个对象检测问题进行框架化、发布谱图对象检测数据集以及评估在该数据集上训练的RetinaNet和YOLOv5对象检测模型。我们的最终模型达到了平均精度高达0.906。由于具有如此高的平均值精度，这些模型对于在现实世界应用中的使用是足够稳健的。摘要：Traditionally source identification is solved using threshold based energy detection algorithms. These algorithms frequently sum up the activity in regions, and consider regions above a specific activity threshold to be sources. While these algorithms work for the majority of cases, they often fail to detect signals that occupy small frequency bands, fail to distinguish sources with overlapping frequency bands, and cannot detect any signals under a specified signal to noise ratio. Through the conversion of raw signal data to spectrogram, source identification can be framed as an object detection problem. By leveraging modern advancements in deep learning based object detection, we propose a system that manages to alleviate the failure cases encountered when using traditional source identification algorithms. Our contributions include framing source identification as an object detection problem, the publication of a spectrogram object detection dataset, and evaluation of the RetinaNet and YOLOv5 object detection models trained on the dataset. Our final models achieve Mean Average Precisions of up to 0.906. With such a high Mean Average Precision, these models are sufficiently robust for use in real world applications.
【5】 Towards zero-shot Text-based voice editing using acoustic context conditioning, utterance embeddings, and reference encoders标题：使用声学上下文调节、话语嵌入和参考编码器的基于Zero-Shot文本的语音编辑链接：https://arxiv.org/abs/2210.16045
作者：Jason Fong,Yun Wang,Prabhav Agrawal,Vimal Manohar,Jilong Wu,Thilo Köhler,Qing He机构：University of Edinburgh,Meta AI备注：Submitted to ICASSP 2023摘要：基于文本的语音编辑（TBVE）使用文本到语音（TTS）系统的合成输出来替换原始录音中的单词。最近的工作已经使用神经模型来产生在清晰度、说话者身份和韵律方面类似于原始语音的编辑语音。然而，现有工作的一个限制是使用微调来优化性能：这需要对来自目标发音人的数据进行进一步的模型训练，这是一个可能将潜在的敏感数据合并到服务器端模型中的昂贵过程。相比之下，本研究的重点是zero-shot方法，该方法完全避免了微调，而是使用预先训练的说话人验证嵌入和联合训练的参考编码器来编码话语级信息，这有助于捕获说话人身份和韵律等方面。主观听力测试发现，在zero-shot环境下，话语嵌入和参考编码器都能改善编辑后的合成语音和未编辑原始录音之间的说话人身份和韵律的连续性。摘要：Text-based voice editing (TBVE) uses synthetic output from text-to-speech (TTS) systems to replace words in an original recording. Recent work has used neural models to produce edited speech that is similar to the original speech in terms of clarity, speaker identity, and prosody. However, one limitation of prior work is the usage of finetuning to optimise performance: this requires further model training on data from the target speaker, which is a costly process that may incorporate potentially sensitive data into server-side models. In contrast, this work focuses on the zero-shot approach which avoids finetuning altogether, and instead uses pretrained speaker verification embeddings together with a jointly trained reference encoder to encode utterance-level information that helps capture aspects such as speaker identity and prosody. Subjective listening tests find that both utterance embeddings and a reference encoder improve the continuity of speaker identity and prosody between the edited synthetic speech and unedited original recording in the zero-shot setting.
【6】 Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models标题：从预先训练的自监督语音模型分析声学单词嵌入链接：https://arxiv.org/abs/2210.16043
作者：Ramon Sanabria,Hao Tang,Sharon Goldwater机构：The University of Edinburgh备注：Submitted to IEEE ICASSP 2023摘要：鉴于自监督模型在各种任务上的强有力的结果，令人惊讶的是，很少有研究探索用于声学单词嵌入（AWE）的自监督表示，固定维向量表示可变长度的口语单词片段。在这项工作中，我们研究了几个预先训练的模型和池方法来构造具有自监督表示的AWE。由于自监督表示的上下文化性质，我们假设简单的池化方法，如平均，可能已经对构造AWE有用。在一个标准的单词辨别任务上，我们发现HuBERT表征与平均池可以与英语AWE的最新水平相媲美。更令人惊讶的是，尽管仅在英语上训练，但在Xitsonga、普通话和法语上评估的HuBERT表示始终优于多语言模型XLSR-53（以及在英语上训练的Wav 2 Vec 2.0）。摘要：Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings (AWE), fixed-dimensional vectors representing variable-length spoken word segments. In this work, we study several pre-trained models and pooling methods for constructing AWEs with self-supervised representations. Owing to the contextualized nature of self-supervised representations, we hypothesize that simple pooling methods, such as averaging, might already be useful for constructing AWEs. When evaluating on a standard word discrimination task, we find that HuBERT representations with mean-pooling rival the state of the art on English AWEs. More surprisingly, despite being trained only on English, HuBERT representations evaluated on Xitsonga, Mandarin, and French consistently outperform the multilingual model XLSR-53 (as well as Wav2Vec 2.0 trained on English).
【7】 Assessing Phrase Break of ESL speech with Pre-trained Language Models标题：基于预训练语言模型的ESL语音断词评估链接：https://arxiv.org/abs/2210.16029
作者：Zhiyi Wang,Shaoguang Mao,Wenshan Wu,Yan Xia机构：Microsoft Research Asia备注：Under Review, ICASSP 2023摘要：本文介绍了一种使用预先训练的语言模型（PLM）来评估ESL学习者语音中短语中断的方法。与传统的方法不同，该方法将语音转换为令牌序列，然后利用PLM的能力进行语音识别。有两个子任务：对语音剪辑短语中断的总体评估;对每个可能的短语中断位置进行精细的评估。语音输入首先与文本强制对齐，然后被预处理为包括单词和相关联的短语中断信息的令牌序列。然后，将令牌序列馈送到预训练和微调流水线中。在预训练中，用标记数据训练被替换的中断标记检测模块，其中每个标记具有一定百分比的被随机替换的机会。在细调阶段，分别利用文本分类和序列标注流水线优化总体评分和细粒度评分。随着PLM的引入，对已标记训练数据的依赖性已大大降低，并且性能已得到改善。摘要：This work introduces an approach to assessing phrase break in ESL learners' speech with pre-trained language models (PLMs). Different with traditional methods, this proposal converts speech to token sequences, and then leverages the power of PLMs. There are two sub-tasks: overall assessment of phrase break for a speech clip; fine-grained assessment of every possible phrase break position. Speech input is first force-aligned with texts, then pre-processed to a token sequence, including words and associated phrase break information. The token sequence is then fed into the pre-training and fine-tuning pipeline. In pre-training, a replaced break token detection module is trained with token data where each token has a certain percentage chance to be randomly replaced. In fine-tuning, overall and fine-grained scoring are optimized with text classification and sequence labeling pipeline, respectively. With the introduction of PLMs, the dependence on labeled training data has been greatly reduced, and performance has improved.
【8】 Spectrograms Are Sequences of Patches标题：光谱图是补丁的序列链接：https://arxiv.org/abs/2210.15988
作者：Leyi Zhao,Yi Li机构：Sichuan University摘要：自监督预训练模型已经成功地应用于多个机器学习领域。然而，只有极少量的作品与音乐有关。在我们的工作中，我们将音乐的声谱图视为一系列的补丁，并设计了一个自监督模型来捕获这些连续补丁的特征：Patchifier算法充分利用了NLP和CV域的自监督学习方法。我们不使用标记数据用于预训练过程，仅使用包含16k个音乐片段的MTAT数据集的子集。在预训练之后，我们将该模型应用于几个下游任务。与其他音频表示模型相比，我们的模型获得了相当可接受的结果。同时，我们的工作表明，将音频视为一系列补丁片段是有意义的。摘要：Self-supervised pre-training models have been used successfully in several machine learning domains. However, only a tiny amount of work is related to music. In our work, we treat a spectrogram of music as a series of patches and design a self-supervised model that captures the features of these sequential patches: Patchifier, which makes good use of self-supervised learning methods from both NLP and CV domains. We do not use labeled data for the pre-training process, only a subset of the MTAT dataset containing 16k music clips. After pre-training, we apply the model to several downstream tasks. Our model achieves a considerably acceptable result compared to other audio representation models. Meanwhile, our work demonstrates that it makes sense to consider audio as a series of patch segments.
【9】 Determining Ratio of Prunable Channels in MobileNet by Sparsity for Acoustic Scene Classification标题：利用稀疏度确定MobileNet中可剪缩通道的比例进行声场分类链接：https://arxiv.org/abs/2210.15960
作者：Yiqiang Cai,Shengchen Li机构：School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Ren’ai Road, Suzhou, China备注：5 pages, 2 figures, submitted to ICASSP 2023摘要：移动网络广泛应用于嵌入式系统中的声学场景分类。现有的ASC算法通过修剪部分分量来降低算法的复杂度，如修剪卷积层的信道。在实践中，被修剪的信道的最大比例（其被定义为可修剪信道的比率（$R_\textit{PC}$））通常根据经验来确定。本文提出了一种通过与卷积层中的信道稀疏度（$S_C$）相关的简单线性回归模型来确定$R_\textit{PC}$的方法。在实验中，通过移除不活动通道直到达到性能降低的拐点来检查$R_\textit{PC}$。提出了计算训练模型的$S_C$和结果$R_\textit{PC}$的简单方法。实验结果表明：（1）$R_\textit{PC}$的决策与$S_C$线性相关，超参数对这种关系的影响很小; 2）移动网络对该方法具有较高的敏感性和稳定性。摘要：MobileNet is widely used for Acoustic Scene Classification (ASC) in embedded systems. Existing works reduce the complexity of ASC algorithms by pruning some components, e.g. pruning channels in the convolutional layer. In practice, the maximum proportion of channels being pruned, which is defined as Ratio of Prunable Channels ($R_\textit{PC}$), is often decided empirically. This paper proposes a method that determines the $R_\textit{PC}$ by simple linear regression models related to the Sparsity of Channels ($S_C$) in the convolutional layers. In the experiment, $R_\textit{PC}$ is examined by removing inactive channels until reaching a knee point of performance decrease. Simple methods for calculating the $S_C$ of trained models and resulted $R_\textit{PC}$ are proposed. The experiment results demonstrate that 1) the decision of $R_\textit{PC}$ is linearly dependent on $S_C$ and the hyper-parameters have a little impact on the relationship; 2) MobileNet shows a high sensitivity and stability on proposed method.
【10】 On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis标题：关于使用特定于通道的大规模预训练编码器进行多通道情感分析链接：https://arxiv.org/abs/2210.15937
作者：Atsushi Ando,Ryo Masumura,Akihiko Takashima,Satoshi Suzuki,Naoki Makishima,Keita Suzuki,Takafumi Moriya,Takanori Ashihara,Hiroshi Sato机构：NTT Corporation备注：Accepted to SLT 2022摘要：本文研究了用于多模态情感分析（MSA）的特定模态的大规模预训练编码器的有效性和实现。虽然已经报道了预训练编码器在各个领域中的有效性，但是传统的MSA方法仅将它们用于语言模态，并且它们的应用还没有被研究。本文将大规模预训练编码器产生的特征与传统启发式特征进行了比较。每种模态均使用一个最大的预训练编码器; CLIP-ViT、WavLM和BERT分别用于视觉、听觉和语言模态。在两个数据集上的实验表明，在单峰和多峰场景下，使用特定领域的预训练编码器的方法都比使用传统特征的方法获得了更好的性能。我们还发现，使用编码器的中间层的输出比使用输出层的输出更好。这些代码可在www.example.com上找到https://github.com/ando-hub/MSA_Pretrain。摘要：This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis~(MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded by large-scale pre-trained encoders with conventional heuristic features. One each of the largest pre-trained encoders publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for visual, acoustic, and linguistic modalities, respectively. Experiments on two datasets reveal that methods with domain-specific pre-trained encoders attain better performance than those with conventional features in both unimodal and multimodal scenarios. We also find it better to use the outputs of the intermediate layers of the encoders than those of the output layer. The codes are available at https://github.com/ando-hub/MSA_Pretrain.
【11】 A comprehensive study on self-supervised distillation for speaker representation learning标题：说话人表征学习的自监督精馏综合研究链接：https://arxiv.org/abs/2210.15936
作者：Zhengyang Chen,Yao Qian,Bing Han,Yanmin Qian,Michael Zeng机构： MoE Key Lab of Artificial Intelligence, AI Institute , X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Microsoft Cognitive Services Research, USA备注：Accepted by SLT2022摘要：在实际应用场景中，由于说话人隐私问题，获取大量标记数据进行说话人表示学习往往具有挑战性。无标签自监督学习是解决这一问题的一种有效方法，与对比学习相比，自提取方法在损失函数中只使用正样本，因而更具吸引力。本文对自提取自监督说话人表征学习进行了全面的研究，特别是对关键数据扩充的研究。我们提出的音频扰动增强策略将说话人表示的性能推向了一个新的极限。实验结果表明，该模型能够在Voxceleb 1说话人确认评测基准上实现一种新的SoTA，对于试验Vox 1-O、Vox 1-E和Vox 1-H，等错误率（EER）分别为2.505%、2.473%和4.791%），丢弃训练阶段中的任何说话人标记。摘要：In real application scenarios, it is often challenging to obtain a large amount of labeled data for speaker representation learning due to speaker privacy concerns. Self-supervised learning with no labels has become a more and more promising way to solve it. Compared with contrastive learning, self-distilled approaches use only positive samples in the loss function and thus are more attractive. In this paper, we present a comprehensive study on self-distilled self-supervised speaker representation learning, especially on critical data augmentation. Our proposed strategy of audio perturbation augmentation has pushed the performance of the speaker representation to a new limit. The experimental results show that our model can achieve a new SoTA on Voxceleb1 speaker verification evaluation benchmark ( i.e., equal error rate (EER) 2.505%, 2.473%, and 4.791% for trial Vox1-O, Vox1-E and Vox1-H , respectively), discarding any speaker labels in the training phase.
【12】 Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation标题：用于Few-Shot文语转换说话人自适应的残差适配器链接：https://arxiv.org/abs/2210.15868
作者：Nobuyuki Morioka,Heiga Zen,Nanxin Chen,Yu Zhang,Yifan Ding机构： Google Research, Japan, Google Research, USA备注：Submitted to ICASSP 2023摘要：使神经文本到语音（TTS）模型适应于目标说话者通常涉及微调预训练的多说话者骨干模型的大多数（如果不是全部的话）参数。然而，服务数百个微调的神经TTS模型是昂贵的，因为它们中的每一个都需要显著的占用空间和单独的计算资源（例如，加速器、存储器）。为了将说话人自适应神经TTS语音扩展到数百个说话人，同时保持说话人的自然度和相似度，提出了一种参数有效的Few-Shot说话人自适应算法。该架构允许在不同的目标说话者之间共享骨干模型。实验结果表明，与完全微调的方法相比，该方法能够获得较好的自然度和说话人相似度，而对每个说话人的骨架模型参数的要求仅为0. 1%。摘要：Adapting a neural text-to-speech (TTS) model to a target speaker typically involves fine-tuning most if not all of the parameters of a pretrained multi-speaker backbone model. However, serving hundreds of fine-tuned neural TTS models is expensive as each of them requires significant footprint and separate computational resources (e.g., accelerators, memory). To scale speaker adapted neural TTS voices to hundreds of speakers while preserving the naturalness and speaker similarity, this paper proposes a parameter-efficient few-shot speaker adaptation, where the backbone model is augmented with trainable lightweight modules called residual adapters. This architecture allows the backbone model to be shared across different target speakers. Experimental results show that the proposed approach can achieve competitive naturalness and speaker similarity compared to the full fine-tuning approaches, while requiring only $\sim$0.1% of the backbone model parameters for each speaker.
【13】 Speech Enhancement with Intelligent Neural Homomorphic Synthesis标题：基于智能神经同态合成的语音增强链接：https://arxiv.org/abs/2210.15853
作者：Shulin He,Wei Rao,Jinjiang Liu,Jun Chen,Yukai Ju,Xueliang Zhang,Yannan Wang,Shidong Shang机构：College of Computer Science, Inner Mongolia University, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China备注：Submitted to ICASSP 2023摘要：大多数神经网络语音增强模型通过直接映射傅里叶变换频谱或波形而忽略语音产生数学模型。在本研究中，我们提出一种神经源滤波器网路来进行语音增强。具体地说，我们利用同态信号处理和倒谱分析来获得含噪语音的激励和声道。与传统的信号处理方法不同，我们使用注意递归网络（ARN）模型预测比率掩模来代替提升分离函数。然后，利用两个卷积注意递归网络（CARN）分别对干净语音的激励和声道进行预测。该系统的输出是从估计的激励和人声合成的。实验结果表明，该方法比FullSubNet算法的SI-SNR提高了1.363dB。摘要：Most neural network speech enhancement models ignore speech production mathematical models by directly mapping Fourier transform spectrums or waveforms. In this work, we propose a neural source filter network for speech enhancement. Specifically, we use homomorphic signal processing and cepstral analysis to obtain noisy speech's excitation and vocal tract. Unlike traditional signal processing, we use an attentive recurrent network (ARN) model predicted ratio mask to replace the liftering separation function. Then two convolutional attentive recurrent network (CARN) networks are used to predict the excitation and vocal tract of clean speech, respectively. The system's output is synthesized from the estimated excitation and vocal. Experiments prove that our proposed method performs better, with SI-SNR improving by 1.363dB compared to FullSubNet.
【14】 Local-global speaker representation for target speaker extraction标题：用于目标说话人提取的局部-全局说话人表示链接：https://arxiv.org/abs/2210.15849
作者：Shulin He,Wei Rao,Kanghao Zhang,Yukai Ju,Yang Yang,Xueliang Zhang,Yannan Wang,Shidong Shang机构：College of Computer Science, Inner Mongolia University, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China备注：Submitted to ICASSP 2023摘要：目标说话人提取是根据给定的注册话语从混合信号中提取目标说话人的语音。目标说话人的注册语音也被称为锚定语音。锚定语音的有效利用是说话人提取的关键。在本研究中，我们提出了一个新的系统，以充分利用说话人信息从锚定语音。与仅使用锚点的局部或全局特征的模型不同，所提出的方法在全局和局部水平上提取说话人信息，并将特征馈送到语音分离网络。该方法充分利用了全局特征和局部特征的优势互补，提高了说话人提取的性能。我们使用多个说话人提取模型验证了这种局部-全局表示（LGR）方法的可行性。在开源数据库Libri-2 talker上进行了系统实验，实验结果表明，该方法的性能明显优于基线模型.摘要：Target speaker extraction is to extract the target speaker's voice from a mixture of signals according to the given enrollment utterance. The target speaker's enrollment utterance is also called as anchor speech. The effective utilization of anchor speech is crucial for speaker extraction. In this study, we propose a new system to exploit speaker information from anchor speech fully. Unlike models that use only local or global features of the anchor, the proposed method extracts speaker information on global and local levels and feeds the features into a speech separation network. Our approach benefits from the complementary advantages of both global and local features, and the performance of speaker extraction is improved. We verified the feasibility of this local-global representation (LGR) method using multiple speaker extraction models. Systematic experiments were conducted on the open-source dataset Libri-2talker, and the results showed that the proposed method significantly outperformed the baseline models.
【15】 GM-TCNet: Gated Multi-scale Temporal Convolutional Network using Emotion Causality for Speech Emotion Recognition标题：GM-TCNet：基于情感因果关系的门控多尺度时间卷积网络语音情感识别链接：https://arxiv.org/abs/2210.15834
作者：Jia-Xin Ye,Xin-Cheng Wen,Xuan-Ze Wang,Yong Xu,Yan Luo,Chang-Li Wu,Li-Yan Chen,Kun-Hong Liu机构：School of Informatics, Xiamen University, Xiamen, China, Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China, Department of Computer Science, Harbin Institute of Technology (Shenzhen), Shenzhen, China备注：The source code is available at: this https URL摘要：在人机交互中，语音情感识别（SER）对于理解用户意图、改善交互体验起着至关重要的作用。相似的情感语音虽然具有不同的说话人特征，但却具有共同的前因和后果，如何通过语音情感之间的因果关系产生鲁棒的、有区别的表征是SER面临的一个重要挑战。本文提出了一种门控多尺度时间卷积网络（GM-TCNet），用于构造一种新的具有多尺度感受野的情绪因果关系表征学习组件。GM-TCNet通过扩展因果卷积层和门控机制构建了一个新颖的情感因果表示学习组件，用于捕捉情感在时域上的动态变化。此外，该算法还利用跳接融合不同选通卷积块的高级特征，捕捉语音中丰富而细微的情感变化。GM-TCNet首先使用单一类型的特征，即Mel频率倒谱系数作为输入，然后将其通过门控时间卷积模块以生成高级特征。最后，将这些特征送入情感分类器以完成SER任务。实验结果表明，与现有技术相比，该模型在大多数情况下保持了最高的性能。摘要：In human-computer interaction, Speech Emotion Recognition (SER) plays an essential role in understanding the user's intent and improving the interactive experience. While similar sentimental speeches own diverse speaker characteristics but share common antecedents and consequences, an essential challenge for SER is how to produce robust and discriminative representations through causality between speech emotions. In this paper, we propose a Gated Multi-scale Temporal Convolutional Network (GM-TCNet) to construct a novel emotional causality representation learning component with a multi-scale receptive field. GM-TCNet deploys a novel emotional causality representation learning component to capture the dynamics of emotion across the time domain, constructed with dilated causal convolution layer and gating mechanism. Besides, it utilizes skip connection fusing high-level features from different gated convolution blocks to capture abundant and subtle emotion changes in human speech. GM-TCNet first uses a single type of feature, mel-frequency cepstral coefficients, as inputs and then passes them through the gated temporal convolutional module to generate the high-level features. Finally, the features are fed to the emotion classifier to accomplish the SER task. The experimental results show that our model maintains the highest performance in most cases compared to state-of-the-art techniques.
【16】 On the Role of Visual Context in Enriching Music Representations标题：论视觉语境在丰富音乐表现形式中的作用链接：https://arxiv.org/abs/2210.15828
作者：Kleanthis Avramidis,Shanti Stewart,Shrikanth Narayanan机构：Signal Analysis and Interpretation Lab, University of Southern California, Los Angeles, CA备注：5 pages, 4 figures, 1 table摘要：人类对音乐的感知和体验是高度依赖于语境的。背景的变化导致了我们对音乐的理解和与音乐互动的方式的差异，这对信息检索的鲁棒模型的设计提出了挑战。结合来自不同来源的多模态背景提供了一种对这种可变性建模的有前途的方法。在诸如电影和音乐视频之类的媒体中呈现的音乐提供了丰富的多模态上下文，其调节潜在的人类体验。然而，这样的上下文建模未被充分开发，因为它需要大量的多模态数据以及相关注释。自监督学习可以通过自动提取不同模态之间的丰富、高级别对应关系来帮助解决这些挑战，从而减轻对大规模细粒度注释的需求。在本研究中，我们提出了VCMR --视频条件音乐表征，一个从音频和伴随的音乐视频中学习音乐表征的对比学习框架。上下文视觉信息增强了音乐音频的表示，如在音乐标记的下游任务上评估的。实验结果表明，提出的框架能够为音频表示提供附加的鲁棒性，并指示音乐元素在多大程度上受视觉上下文的影响或决定。摘要：Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the design of robust models for information retrieval. Incorporating multimodal context from diverse sources provides a promising approach toward modeling this variability. Music presented in media such as movies and music videos provide rich multimodal context that modulates underlying human experiences. However, such context modeling is underexplored, as it requires large amounts of multimodal data along with relevant annotations. Self-supervised learning can help address these challenges by automatically extracting rich, high-level correspondences between different modalities, hence alleviating the need for fine-grained annotations at scale. In this study, we propose VCMR -- Video-Conditioned Music Representations, a contrastive learning framework that learns music representations from audio and the accompanying music videos. The contextual visual information enhances representations of music audio, as evaluated on the downstream task of music tagging. Experimental results show that the proposed framework can contribute additive robustness to audio representations and indicates to what extent musical elements are affected or determined by visual context.
【17】 Evaluating context-invariance in unsupervised speech representations标题：评估无监督语音表示中的上下文不变性链接：https://arxiv.org/abs/2210.15775
作者：Mark Hallap,Emmanuel Dupoux,Ewan Dunbar机构： University of Toronto, CoML, ENSCNRSEHESSINRIAPSL Research University备注：Submitted to ICASSP 2023摘要：无监督语音表示已经开始起步，基准测试（SUPERB，ZeroSpeech）证明了在半监督语音识别、语音合成和纯语音语言建模方面的重大进展。灵感来自于“发现一种语言的音素”或类似的低比特率编码的承诺。然而，音素转录的关键性质之一是上下文不变性：在文本保持稳定的同时，语音的语音上下文可以对它的发音方式具有巨大的影响。这就是为什么同一个单词的标记可以有相同的转录--这是语言理解的关键。当前的基准测试无法衡量上下文不变性。我们开发了一个新版本的ZeroSpeech ABX基准测试来测量上下文不变性，并将其应用于最近的自监督表示。我们证明了表征的上下文无关性预示着词级表征的稳定性。我们建议研究集中在提高自监督和无监督表示的上下文无关性。摘要：Unsupervised speech representations have taken off, with benchmarks (SUPERB, ZeroSpeech) demonstrating major progress on semi-supervised speech recognition, speech synthesis, and speech-only language modelling. Inspiration comes from the promise of ``discovering the phonemes'' of a language or a similar low-bitrate encoding. However, one of the critical properties of phoneme transcriptions is context-invariance: the phonetic context of a speech sound can have massive influence on the way it is pronounced, while the text remains stable. This is what allows tokens of the same word to have the same transcriptions -- key to language understanding. Current benchmarks do not measure context-invariance. We develop a new version of the ZeroSpeech ABX benchmark that measures context-invariance, and apply it to recent self-supervised representations. We demonstrate that the context-independence of representations is predictive of the stability of word-level representations. We suggest research concentrate on improving context-independence of self-supervised and unsupervised representations.
【18】 Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge标题：从原始音频中进行自我监督的语言学习：来自零资源语音挑战的经验链接：https://arxiv.org/abs/2210.15759
作者：Ewan Dunbar,Nicolas Hamilakis,Emmanuel Dupoux备注：None摘要：自监督或无监督机器学习的最新进展开启了从原始音频构建完整语音处理系统而不使用任何文本表示或专家标签（例如音素、字典或解析树）的可能性。自2015年以来，零资源语音挑战系列的贡献是将这一长期目标分解为四个明确定义的任务--声学单元发现、口语术语发现、离散再合成和口语建模--并引入相关的指标和基准，实现模型比较和累积进展。我们将概述自2015年以来的六个版本的挑战系列，讨论所吸取的经验教训，并概述需要进一步工作或给出令人困惑的结果的领域。摘要：Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks -- Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling -- and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.
【19】 One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hall标题：音频信号的一次声学匹配--学会在任何房间/音乐厅听音乐链接：https://arxiv.org/abs/2210.15750
作者：Prateek Verma,Chris Chafe,Jonathan Berger机构：Stanford University备注：5 pages, 1 figure摘要：产生和听到声音的声学空间通过提供一种独特的\textit{存在}感，在声音的感知方式中发挥着重要作用。我们听到的每一个声音都是由声源固有的连续卷积运算和外部因素（如麦克风特性和房间脉冲响应）产生的。通常，研究人员使用诸如手枪射击或气球爆裂之类的激励作为脉冲信号，利用该脉冲信号可以创建可听化。与感兴趣的信号卷积的房间“脉冲”响应可以将输入声音变换成在感兴趣的声学空间中播放的声音。在此，我们提出了一种新颖的架构，该架构可以通过使用作为气球爆裂声的代理记录的任意音频来将感兴趣的声音变换到感兴趣的任何其他声学空间（房间或大厅）。该架构基于简单的信号处理思想，以从学习的声学签名和输入信号学习残差信号。我们的框架允许神经网络调整时频表示中每一点的增益，给出可靠的定性和定量结果。摘要：The acoustic space in which a sound is created and heard plays an essential role in how that sound is perceived by affording a unique sense of \textit{presence}. Every sound we hear results from successive convolution operations intrinsic to the sound source and external factors such as microphone characteristics and room impulse responses. Typically, researchers use an excitation such as a pistol shot or balloon pop as an impulse signal with which an auralization can be created. The room "impulse" responses convolved with the signal of interest can transform the input sound into the sound played in the acoustic space of interest. Here we propose a novel architecture that can transform a sound of interest into any other acoustic space(room or hall) of interest by using arbitrary audio recorded as a proxy for a balloon pop. The architecture is grounded in simple signal processing ideas to learn residual signals from a learned acoustic signature and the input signal. Our framework allows a neural network to adjust gains of every point in the time-frequency representation, giving sound qualitative and quantitative results.
【20】 Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models标题：使用组合端到端模型进行口语理解的令牌级序列标注链接：https://arxiv.org/abs/2210.15734
作者：Siddhant Arora,Siddharth Dalmia,Brian Yan,Florian Metze,Alan W Black,Shinji Watanabe机构：Language Technologies Institute, Carnegie Mellon University, USA备注：Accepted at EMNLP 2022 Findings. Our code and models will be publicly available as part of the ESPnet-SLU toolkit: this https URL and the release can be followed here: this https URL摘要：端到端口语理解（SLU）系统由于其简单性和避免错误传播的能力而比级联方法更受欢迎。然而，这些系统将序列标记建模为序列预测任务，导致与其良好建立的标记水平标记公式的分歧。我们构建了组合式端到端SLU系统，明确地将SLU中识别口语提及的复杂性与序列标记的NLU任务分离。通过依赖于为ASR训练的中间解码器，我们的端到端系统将输入模态从语音转换为可以在传统序列标记框架中使用的令牌级表示。在我们的端到端SLU系统中，ASR和NLU公式的这种组合提供了与预先训练的ASR和NLU系统的直接兼容性，允许对单个组件进行性能监测，并允许使用像CRF这样的全局归一化损耗，使其在实际场景中具有吸引力。在SLU基准测试中，我们的模型在命名实体识别的标注任务上优于级联模型和直接端到端模型。摘要：End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation. However, these systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation. We build compositional end-to-end SLU systems that explicitly separate the added complexity of recognizing spoken mentions in SLU from the NLU task of sequence labeling. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to token-level representations that can be used in the traditional sequence labeling framework. This composition of ASR and NLU formulations in our end-to-end SLU system offers direct compatibility with pre-trained ASR and NLU systems, allows performance monitoring of individual components and enables the use of globally normalized losses like CRF, making them attractive in practical scenarios. Our models outperform both cascaded and direct end-to-end models on a labeling task of named entity recognition across SLU benchmarks.
【21】 FedAudio: A Federated Learning Benchmark for Audio Tasks标题：FedAudio：一种用于音频任务的联合学习基准链接：https://arxiv.org/abs/2210.15707
作者：Tuo Zhang,Tiantian Feng,Samiul Alam,Sunwoo Lee,Mi Zhang,Shrikanth S. Narayanan,Salman Avestimehr机构：Department of Electrical and Computer Engineering, University of Southern California, Michigan State University, Department of Computer Engineering, Inha University, Department of Computer Science and Engineering, The Ohio State University摘要：近年来，由于与持续从用户收集数据的消费设备的普遍性相关的数据隐私问题，联合学习（FL）获得了大量关注。虽然已经开发了一些外语基准来促进外语研究，但没有一个包括音频数据和音频相关的任务。在本文中，我们通过引入一个新的音频任务FL基准来填补这一关键空白，我们称之为FedAudio。FedAudio包括来自三个重要音频任务的四个代表性和常用的音频数据集，这些音频任务与FL用例非常一致。特别是，FedAudio的一个独特贡献是在数据集中引入了数据噪声和标签错误，以模拟在现实环境中部署FL系统时的挑战。FedAudio还包括数据集的基准测试结果和PyTorch库，目的是方便研究人员公平地比较他们的算法。我们希望FedAudio能够作为一个催化剂，激发新的外语研究的音频任务，从而有利于声学和语音研究社区。数据集和性能指标评测结果可访问https://github.com/zhang-tuo pdf/FedAudio。摘要：Federated learning (FL) has gained substantial attention in recent years due to the data privacy concerns related to the pervasiveness of consumer devices that continuously collect data from users. While a number of FL benchmarks have been developed to facilitate FL research, none of them include audio data and audio-related tasks. In this paper, we fill this critical gap by introducing a new FL benchmark for audio tasks which we refer to as FedAudio. FedAudio includes four representative and commonly used audio datasets from three important audio tasks that are well aligned with FL use cases. In particular, a unique contribution of FedAudio is the introduction of data noises and label errors to the datasets to emulate challenges when deploying FL systems in real-world settings. FedAudio also includes the benchmark results of the datasets and a PyTorch library with the objective of facilitating researchers to fairly compare their algorithms. We hope FedAudio could act as a catalyst to inspire new FL research for audio tasks and thus benefit the acoustic and speech research community. The datasets and benchmark results can be accessed at https://github.com/zhang-tuo pdf/FedAudio.
【22】 Contextual-Utterance Training for Automatic Speech Recognition标题：用于自动语音识别的语境话语训练链接：https://arxiv.org/abs/2210.16238
作者：Alejandro Gomez-Alanis,Lukas Drude,Andreas Schwarz,Rupak Vignesh Swaminathan,Simon Wiesler机构：Wiesler, Amazon Alexa, Aachen, Germany; ,Amazon Alexa, Pittsburgh, USA摘要：基于递归神经网络转换器（RNN-T）的流式自动语音识别（ASR）系统的最新研究已经将过去的上下文信息馈送给编码器，以改善其误字率（WER）性能。本文首先提出了一种上下文话语训练技术，它利用先前和未来的上下文话语来对说话人、话题和声学环境进行隐式适应。此外，我们提出一种双模式的上下文-话语训练技术用于流式自动语音识别（ASR）系统。该方法允许通过将教师的知识“就地”提取到学生来更好地利用流模型中的可用声学上下文，其中教师能够看到过去和未来的上下文话语，而学生只能看到当前和过去的上下文话语。实验结果表明，使用所提方法训练的保形器-换能器系统的性能优于使用经典RNN-T损失训练的相同系统。具体地说，所提出的技术能够将WER和平均最后令牌发送延迟分别减少6%和40 ms以上。摘要：Recent studies of streaming automatic speech recognition (ASR) recurrent neural network transducer (RNN-T)-based systems have fed the encoder with past contextual information in order to improve its word error rate (WER) performance. In this paper, we first propose a contextual-utterance training technique which makes use of the previous and future contextual utterances in order to do an implicit adaptation to the speaker, topic and acoustic environment. Also, we propose a dual-mode contextual-utterance training technique for streaming automatic speech recognition (ASR) systems. This proposed approach allows to make a better use of the available acoustic context in streaming models by distilling "in-place" the knowledge of a teacher, which is able to see both past and future contextual utterances, to the student which can only see the current and past contextual utterances. The experimental results show that a conformer-transducer system trained with the proposed techniques outperforms the same system trained with the classical RNN-T loss. Specifically, the proposed technique is able to reduce both the WER and the average last token emission latency by more than 6% and 40ms relative, respectively.
【23】 Target-Speaker Voice Activity Detection via Sequence-to-Sequence Prediction标题：基于序列到序列预测的目标说话人语音活动检测链接：https://arxiv.org/abs/2210.16127
作者：Ming Cheng,Weiqing Wang,Yucong Zhang,Xiaoyi Qin,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China备注：submitted to ICASSP2023摘要：目标说话人语音活动检测是当前复杂声学环境下说话人识别的一种有效方法。提出了一种新的序列到序列目标说话人语音活动检测（Seq 2Seq-TSVAD）方法，能够有效解决大规模说话人的联合建模和高分辨率语音活动预测问题。实验结果表明，较大的扬声器容量和较高的输出分辨率可以显著降低二值化错误率（DER），在VoxConverse测试集和DIHARD-III评测集上分别达到了4.55%和10.77%的最佳性能。摘要：Target-speaker voice activity detection is currently a promising approach for speaker diarization in complex acoustic environments. This paper presents a novel Sequence-to-Sequence Target-Speaker Voice Activity Detection (Seq2Seq-TSVAD) method that can efficiently address the joint modeling of large-scale speakers and predict high-resolution voice activities. Experimental results show that larger speaker capacity and higher output resolution can significantly reduce the diarization error rate (DER), which achieves the new state-of-the-art performance of 4.55% on the VoxConverse test set and 10.77% on Track 1 of the DIHARD-III evaluation set under the widely-used evaluation metrics.
【24】 Parameter-efficient transfer learning of pre-trained Transformer models for speaker verification using adapters标题：使用适配器的用于说话人确认的预训练Transformer模型的参数高效转移学习链接：https://arxiv.org/abs/2210.16032
作者：Junyi Peng,Themos Stafylakis,Rongzhi Gu,Oldřich Plchot,Ladislav Mošner,Lukáš Burget,Jan Černocký机构：Luk´aˇs Burget, Jan ˇCernock´y, Omilia - Conversational Intelligence, Athens, Greece, Tencent AI Lab, Shenzhen, China备注：submitted to ICASSP2023摘要：近年来，预训练的Transformer模型由于在各种下游任务中取得的巨大成功，在语音处理领域受到了越来越多的关注。然而，大多数微调方法更新预训练模型的所有参数，这随着模型大小的增长而变得禁止，并且有时导致对小数据集的过拟合。本文对参数有效迁移学习（PETL）方法在说话人确认中的应用进行了全面分析，以减少所需的可学习参数，从而适应说话人确认任务。具体来说，在微调过程中，预先训练的模型被冻结，只有插入每个Transformer块中的轻量级模块是可训练的（一种称为适配器的方法）。此外，为了提高跨语言低资源场景中的性能，在直接在小数据集上进行微调之前，Transformer模型会在大的中间数据集上进行进一步的微调。通过更新不到4%的参数，（我们提出的）基于PETL的方法实现了与完全微调方法（Vox 1-O：0.55%，Vox1-E：0.82%，Vox 1-H为1.73%）。摘要：Recently, the pre-trained Transformer models have received a rising interest in the field of speech processing thanks to their great success in various downstream tasks. However, most fine-tuning approaches update all the parameters of the pre-trained model, which becomes prohibitive as the model size grows and sometimes results in overfitting on small datasets. In this paper, we conduct a comprehensive analysis of applying parameter-efficient transfer learning (PETL) methods to reduce the required learnable parameters for adapting to speaker verification tasks. Specifically, during the fine-tuning process, the pre-trained models are frozen, and only lightweight modules inserted in each Transformer block are trainable (a method known as adapters). Moreover, to boost the performance in a cross-language low-resource scenario, the Transformer model is further tuned on a large intermediate dataset before directly fine-tuning it on a small dataset. With updating fewer than 4% of parameters, (our proposed) PETL-based methods achieve comparable performances with full fine-tuning methods (Vox1-O: 0.55%, Vox1-E: 0.82%, Vox1-H:1.73%).
【25】 Laugh Betrays You? Learning Robust Speaker Representation From Speech Containing Non-Verbal Fragments标题：笑背叛了你？从含有非语言片段的语音中学习稳健的说话人表征链接：https://arxiv.org/abs/2210.16028
作者：Yuke Lin,Xiaoyi Qin,Huahua Cui,Zhenyi Zhu,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China, Advanced Computing East China Sub-Center, Kunshan, China摘要：自动说话人确认的成功表明，可以从中性语音中提取出有区别的说话人表示。然而，作为一种非语言的声音，笑声也应该直观地携带说话人信息。因此，本文主要研究了含有非言语笑声片段的说话人确认问题。我们通过在VoxCeleb和CN-Celeb数据集的一部分上执行笑声检测脚本，收集了一组包含笑声成分的剪辑。为了进一步过滤不可信的视频片段，我们使用了我们的二元笑声检测分类器来计算概率得分，该分类器是由纯笑声和中性语音预先训练的。然后，基于得分超过阈值的片段，我们构建了两种不同评估场景下的试验：笑-笑（LL）和说-笑（SL）。在此基础上，提出了一种新的基于笑点拼接的网络（Laughter-Splicing based Network，LSN）方法，该方法在两种情况下都能显著提高性能，并且在中性语音（如VoxCeleb 1测试集）上保持了较好的性能。具体来说，我们的系统在笑-笑和说-笑试验中分别实现了20%和22%的相对改善。元数据和示例剪辑已在www.example.com上发布https://github.com/nevermoreLin/Laugh_LSN。摘要：The success of automatic speaker verification shows that discriminative speaker representations can be extracted from neutral speech. However, as a kind of non-verbal voice, laughter should also carry speaker information intuitively. Thus, this paper focuses on exploring speaker verification about utterances containing non-verbal laughter segments. We collect a set of clips with laughter components by conducting a laughter detection script on VoxCeleb and part of the CN-Celeb dataset. To further filter untrusted clips, probability scores are calculated by our binary laughter detection classifier, which is pre-trained by pure laughter and neutral speech. After that, based on the clips whose scores are over the threshold, we construct trials under two different evaluation scenarios: Laughter-Laughter (LL) and Speech-Laughter (SL). Then a novel method called Laughter-Splicing based Network (LSN) is proposed, which can significantly boost performance in both scenarios and maintain the performance on the neutral speech, such as the VoxCeleb1 test set. Specifically, our system achieves relative 20% and 22% improvement on Laughter-Laughter and Speech-Laughter trials, respectively. The meta-data and sample clips have been released at https://github.com/nevermoreLin/Laugh_LSN.
【26】 SG-VAD: Stochastic Gates Based Speech Activity Detection标题：SG-VAD：基于随机门的语音活动检测链接：https://arxiv.org/abs/2210.16022
作者：Jonathan Svirsky,Ofir Lindenbaum机构：Bar-Ilan University, Ramat-Gan, Israel摘要：提出了一种低资源环境下的语音激活检测（VAD）模型.我们的核心思想是将VAD建模为一个去噪任务，并构造一个网络，该网络被设计用于识别语音分类任务的多余特征。我们训练该模型以在预测语音事件类型的同时识别不相关的特征。我们的模型仅包含7.8K个参数，在AVA-Speech评估集上优于先前提出的方法，并且在HAVIC数据集上提供了比较结果。本文介绍了该模型的结构、实验结果以及对模型各部件的烧蚀研究。我们在www.example.com上发布了代码和模型https://www.github.com/jsvir/vad。摘要：We propose a novel voice activity detection (VAD) model in a low-resource environment. Our key idea is to model VAD as a denoising task, and construct a network that is designed to identify nuisance features for a speech classification task. We train the model to simultaneously identify irrelevant features while predicting the type of speech event. Our model contains only 7.8K parameters, outperforms the previously proposed methods on the AVA-Speech evaluation set, and provides comparative results on the HAVIC dataset. We present its architecture, experimental results, and ablation study on the model's components. We publish the code and the models here https://www.github.com/jsvir/vad.
【27】 NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit标题：NNSVS：一个基于神经网络的歌唱语音合成工具包链接：https://arxiv.org/abs/2210.15987
作者：Ryuichi Yamamoto,Reo Yoneyama,Tomoki Toda机构：LINE Corp., Japan,Nagoya University, Japan.备注：Submitted to ICASSP 2023摘要：本文介绍了一个基于神经网络的歌唱语音合成研究的开源软件NNSVS的设计。NNSVS的灵感来自于歌唱声音合成研究领域的开源先驱Sinsy，它提供了许多附加功能，如多流模型、自回归基频模型和神经声码器。此外，NNSVS还提供了大量的文档和大量脚本来构建完整的歌唱声音合成系统。实验结果表明，我们的最佳系统明显优于我们的Sinsy和其他基线系统的复制。该工具包可从www.example.com获得https://github.com/nnsvs/nnsvs。摘要：This paper describes the design of NNSVS, an open-source software for neural network-based singing voice synthesis research. NNSVS is inspired by Sinsy, an open-source pioneer in singing voice synthesis research, and provides many additional features such as multi-stream models, autoregressive fundamental frequency models, and neural vocoders. Furthermore, NNSVS provides extensive documentation and numerous scripts to build complete singing voice synthesis systems. Experimental results demonstrate that our best system significantly outperforms our reproduction of Sinsy and other baseline systems. The toolkit is available at https://github.com/nnsvs/nnsvs.
【28】 Dysfluencies Seldom Come Alone -- Detection as a Multi-Label Problem标题：不流利很少单独出现--检测是一个多标签问题链接：https://arxiv.org/abs/2210.15982
作者：Sebastian P. Bayerl,Dominik Wagner,Florian Hönig,Tobias Bocklet,Elmar Nöth,Korbinian Riedhammer机构：⋆ Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, ‡KST Institut GmbH, ⋄Intel Labs, † Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Germany备注：Submitted to ICASSP 2023摘要：为了处理口吃的语音，需要特别适配的语音识别模型。为了以有针对性的方式使用这些，必须可靠地检测口吃语音。最近的研究将口吃视为多类分类问题，或者将检测每种不流畅类型视为一项孤立的任务;这没有抓住口吃的本质，口吃的本质是一种不流畅很少单独出现，即，与其他事件同时发生。本研究探讨一种以wav 2 vec 2.0系统为基础的多标签问题之端对端口吃侦测与分类方法。该方法在包含英语和德语口吃语音的三个数据集的组合上进行评估，在SEP-28 k-Extended数据集上产生了口吃检测的最新结果。实验结果证明了特征的可移植性以及该方法在数据集和语言间的通用性。摘要：Specially adapted speech recognition models are necessary to handle stuttered speech. For these to be used in a targeted manner, stuttered speech must be reliably detected. Recent works have treated stuttering as a multi-class classification problem or viewed detecting each dysfluency type as an isolated task; that does not capture the nature of stuttering, where one dysfluency seldom comes alone, i.e., co-occurs with others. This work explores an approach based on a modified wav2vec 2.0 system for end-to-end stuttering detection and classification as a multi-label problem. The method is evaluated on combinations of three datasets containing English and German stuttered speech, yielding state-of-the-art results for stuttering detection on the SEP-28k-Extended dataset. Experimental results provide evidence for the transferability of features and the generalizability of the method across datasets and languages.
【29】 End-to-end Ensemble-based Feature Selection for Paralinguistics Tasks标题：基于端到端集成的副语言学任务特征选择链接：https://arxiv.org/abs/2210.15978
作者：Tamás Grósz,Mittul Singh,Sudarsana Reddy Kadiri,Hemant Kathania,Mikko Kurimo机构：The authors would like to thank Antonia Hamilton and Alexis Macintyre for granting access to a subset of the UCL Speech, BreathMonitoring (UCL-SBM) database used in the Breathing Sub-Challenge. This work was supported by the Academy of Finland摘要：近年来发生的事件突出了远程医疗解决方案的重要性，这种解决方案有可能实现远程治疗和诊断。计算副语言学是语音处理领域的一个独特分支，其目标是提取说话人的信息，是远程医疗应用的重要组成部分。在本研究中，我们关注两个副语言问题：面罩检测和呼吸状态预测。为这些任务开发的解决方案可能是无价的，并有可能帮助监测和限制像COVID-19这样的病毒的传播。目前针对这些任务提出的最先进的方法是基于深度神经网络（如ResNets）结合特征工程的集成。虽然这些系综可以实现高精度，但是它们也具有大的覆盖区并且需要相当大的计算能力，从而降低了具有有限资源的设备的便携性。这些缺点还意味着先前提出的解决方案由于它们的尺寸和速度而不适用于远程医疗系统。另一方面，采用较轻的特征工程系统可能是费力的，并且增加了进一步的复杂性，使得它们难以快速地创建可部署的系统。本文提出了一种基于集成的自动特征选择方法，以实现快速、高效的系统开发。特别地，我们提出了一种基于输出梯度的方法，在训练一个较小的集合之前，使用大的、性能良好的集合来发现本质特征。在我们的实验中，我们观察到使用基于输出梯度的特征的神经网络集合在推理时间上有相当大的减少（25-32%）。我们的方法提供了一种简单的方法来提高系统的速度，并实现实时使用，同时保持与使用所有光谱特征的更大覆盖区集合的竞争结果。摘要：The events of recent years have highlighted the importance of telemedicine solutions which could potentially allow remote treatment and diagnosis. Relatedly, Computational Paralinguistics, a unique subfield of Speech Processing, aims to extract information about the speaker and form an important part of telemedicine applications. In this work, we focus on two paralinguistic problems: mask detection and breathing state prediction. Solutions developed for these tasks could be invaluable and have the potential to help monitor and limit the spread of a virus like COVID-19. The current state-of-the-art methods proposed for these tasks are ensembles based on deep neural networks like ResNets in conjunction with feature engineering. Although these ensembles can achieve high accuracy, they also have a large footprint and require substantial computational power reducing portability to devices with limited resources. These drawbacks also mean that the previously proposed solutions are infeasible to be used in a telemedicine system due to their size and speed. On the other hand, employing lighter feature-engineered systems can be laborious and add further complexity making them difficult to create a deployable system quickly. This work proposes an ensemble-based automatic feature selection method to enable the development of fast and memory-efficient systems. In particular, we propose an output-gradient-based method to discover essential features using large, well-performing ensembles before training a smaller one. In our experiments, we observed considerable (25-32%) reductions in inference times using neural network ensembles based on output-gradient-based features. Our method offers a simple way to increase the speed of the system and enable real-time usage while maintaining competitive results with larger-footprint ensemble using all spectral features.
【30】 Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform标题：采用多频带生成和短时傅里叶逆变换的轻量级高保真端到端文语转换链接：https://arxiv.org/abs/2210.15975
作者：Masaya Kawamura,Yuma Shirahata,Ryuichi Yamamoto,Kentaro Tachibana机构：The University of Tokyo, Japan,LINE Corp., Japan.备注：Submitted to ICASSP 2023摘要：提出了一种基于多频带生成和短时傅立叶逆变换的轻量级端到端语音合成模型。我们的模型基于高质量的端到端文本到语音模型--VITS，但是为了更有效的推理，我们采用了两个变化：1）用简单的逆短时傅立叶变换部分地替换计算上最昂贵的分量，以及2）使用具有固定或可训练合成滤波器的多频带生成来生成波形。与传统的轻量级模型不同，传统的轻量级模型分别采用优化或知识提取来训练两个级联的组件，我们的方法享有端到端优化的全部好处。实验结果表明，在Intel Core i7处理器上，我们的模型合成的语音与通过VITS合成的语音一样自然，同时实现了0.066的实时系数，比VITS快4.1倍。此外，该模型的较小版本在自然度和推理速度方面明显优于轻量级基线模型。代码和音频样本可从www.example.com获得https://github.com/MasayaKawamura/MB-iSTFT-VITS。摘要：We propose a lightweight end-to-end text-to-speech model using multi-band generation and inverse short-time Fourier transform. Our model is based on VITS, a high-quality end-to-end text-to-speech model, but adopts two changes for more efficient inference: 1) the most computationally expensive component is partially replaced with a simple inverse short-time Fourier transform, and 2) multi-band generation, with fixed or trainable synthesis filters, is used to generate waveforms. Unlike conventional lightweight models, which employ optimization or knowledge distillation separately to train two cascaded components, our method enjoys the full benefits of end-to-end optimization. Experimental results show that our model synthesized speech as natural as that synthesized by VITS, while achieving a real-time factor of 0.066 on an Intel Core i7 CPU, 4.1 times faster than VITS. Moreover, a smaller version of the model significantly outperformed a lightweight baseline model with respect to both naturalness and inference speed. Code and audio samples are available from https://github.com/MasayaKawamura/MB-iSTFT-VITS.
【31】 Period VITS: Variational Inference with Explicit Pitch Modeling for End-to-end Emotional Speech Synthesis标题：PERIOD VITS：端到端情感语音合成的变分推理和显式基音建模链接：https://arxiv.org/abs/2210.15964
作者：Yuma Shirahata,Ryuichi Yamamoto,Eunwoo Song,Ryo Terashima,Jae-Min Kim,Kentaro Tachibana机构：LINE Corp., Tokyo, Japan, NAVER Corp., Seongnam, Korea备注：Submitted to ICASSP 2023摘要：已经提出了几种完全端到端的文本到语音（TTS）模型，分别训练声学和声码器模型）。然而，当数据集包含情感属性时，它们经常生成具有可听伪像的不稳定音调轮廓，即，发音和韵律巨大差异。针对这一问题，本文提出了一种新的端到端TTS模型--Period VITS，该模型包含一个显式的周期发生器。在该方法中，我们引入了一个帧基音预测器，该预测器从输入文本中预测韵律特征，如基音和浊音标志。根据这些特征，所提出的周期性发生器产生样本级正弦源，该正弦源使得波形解码器能够准确地再现音调。最后，采用变分推理和对抗性目标，以端到端的方式对整个模型进行联合优化。结果，解码器变得能够产生更稳定、更有表现力和更自然的输出波形。实验结果表明，该模型在自然度方面明显优于基线模型，在生成的样本中，基音稳定性得到了改善。摘要：Several fully end-to-end text-to-speech (TTS) models have been proposed that have shown better performance compared to cascade models (i.e., training acoustic and vocoder models separately). However, they often generate unstable pitch contour with audible artifacts when the dataset contains emotional attributes, i.e., large diversity of pronunciation and prosody. To address this problem, we propose Period VITS, a novel end-to-end TTS model that incorporates an explicit periodicity generator. In the proposed method, we introduce a frame pitch predictor that predicts prosodic features, such as pitch and voicing flags, from the input text. From these features, the proposed periodicity generator produces a sample-level sinusoidal source that enables the waveform decoder to accurately reproduce the pitch. Finally, the entire model is jointly optimized in an end-to-end manner with variational inference and adversarial objectives. As a result, the decoder becomes capable of generating more stable, expressive, and natural output waveforms. The experimental results showed that the proposed model significantly outperforms baseline models in terms of naturalness, with improved pitch stability in the generated samples.
【32】 The Importance of Speech Stimuli for Pathologic Speech Classification标题：语音刺激对病理性语音分类的重要性链接：https://arxiv.org/abs/2210.15941
作者：Ilja Baumann,Dominik Wagner,Franziska Braun,Sebastian P. Bayerl,Elmar Nöth,Korbinian Riedhammer,Tobias Bocklet机构：†Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, ‡Friedrich-Alexander-University Erlangen-N¨urnberg, Pattern Recognition Lab, Erlangen, Germany, Intel Labs备注：Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2210.15336摘要：目前的研究结果表明，预先训练好的wav2vec 2.0模型可以成功地用作特征提取器来区分基于说话人的任务。我们证明了在预先训练的wav2vec 2.0系统的不同层提取的潜在表示可以有效地用于各种类型的病理语音的二进制分类。为此，我们检查了喉切除术、口腔鳞状细胞癌、帕金森病和唇腭裂的病理。结果表明，病理和健康嗓音之间的区分，特别是利用来自较低层的潜在表征，表现良好，具有从帕金森病的77.2%到喉切除术分类的100%的最低准确率。然而，交叉病理学和交叉健康测试表明，训练的分类器似乎有偏见。如果在训练和域外测试数据之间存在不匹配，在年龄、说话内容或声音条件方面。摘要：Current findings show that pre-trained wav2vec 2.0 models can be successfully used as feature extractors to discriminate on speaker-based tasks. We demonstrate that latent representations extracted at different layers of a pre-trained wav2vec 2.0 system can be effectively used for binary classification of various types of pathologic speech. We examine the pathologies laryngectomy, oral squamous cell carcinoma, parkinson's disease and cleft lip and palate for this purpose. The results show that a distinction between pathological and healthy voices, especially with latent representations from the lower layers, performs well with the lowest accuracy from 77.2% for parkinson's disease to 100% for laryngectomy classification. However, cross-pathology and cross-healthy tests show that the trained classifiers seem to be biased. The recognition rates vary considerably if there is a mismatch between training and out-of-domain test data, e.g., in age, spoken content or acoustic conditions.
【33】 Speaker recognition with two-step multi-modal deep cleansing标题：基于两步多模式深度清洗的说话人识别链接：https://arxiv.org/abs/2210.15903
作者：Ruijie Tao,Kong Aik Lee,Zhan Shi,Haizhou Li机构：College of Design and Engineering, National University of Singapore, Singapore, Institute for Infocomm Research, A⋆STAR, Singapore, The Chinese University of Hong Kong, Shenzhen, China备注：5 pages, 3 figures摘要：近年来，基于神经网络的说话人识别技术取得了显著的进展。鲁棒的说话人表示从训练集中的难样本和易样本学习有意义的知识以获得良好的性能。然而，噪声样本（即，具有错误的标签）引起混淆并使网络学习到不正确的表示。本文提出了一种两步视听深度清洗框架，以消除说话人表征学习中噪声标签的影响。该框架包含一个粗粒度的清洗步骤来搜索特殊的样本，然后是一个细粒度的清洗步骤来过滤掉有噪声的标签。我们的研究从一个高效的音视频说话人识别系统开始，该系统在Vox-O，E和H测试集上的等误率（EER）分别为0.01%，0.07%和0.13%.采用本文提出的多模态清洗机制，四种不同的说话人识别网络取得了平均5.9%的改进.代码可从以下网址获得：{1}，{2}{3}您可以在这里看到您的照片。{4}摘要：Neural network-based speaker recognition has achieved significant improvement in recent years. A robust speaker representation learns meaningful knowledge from both hard and easy samples in the training set to achieve good performance. However, noisy samples (i.e., with wrong labels) in the training set induce confusion and cause the network to learn the incorrect representation. In this paper, we propose a two-step audio-visual deep cleansing framework to eliminate the effect of noisy labels in speaker representation learning. This framework contains a coarse-grained cleansing step to search for the peculiar samples, followed by a fine-grained cleansing step to filter out the noisy labels. Our study starts from an efficient audio-visual speaker recognition system, which achieves a close to perfect equal-error-rate (EER) of 0.01\%, 0.07\% and 0.13\% on the Vox-O, E and H test sets. With the proposed multi-modal cleansing mechanism, four different speaker recognition networks achieve an average improvement of 5.9\%. Code has been made available at: \textcolor{magenta}{\url{https://github.com/TaoRuijie/AVCleanse}}.
【34】 Nonparallel High-Quality Audio Super Resolution with Domain Adaptation and Resampling CycleGANs标题：域自适应和重采样循环阵列的非并行高质量音频超分辨率链接：https://arxiv.org/abs/2210.15887
作者：Reo Yoneyama,Ryuichi Yamamoto,Kentaro Tachibana机构：Nagoya University, Japan,LINE Corp., Japan.备注：Submitted to ICASSP 2023摘要：神经音频超分辨率模型通常在低分辨率和高分辨率音频信号对上训练。尽管如果输入数据的声学特性与训练数据的声学特性相似，则这些方法实现了高度精确的超分辨率，但是挑战仍然存在：对于域外数据，模型遭受质量降级，并且需要成对的数据用于训练。针对这些问题，提出了一种基于两个连通的循环一致生成对抗网络（CycleGAN）的高质量音频超分辨率方法--Dual-CycleGAN。我们的方法将超分辨率方法分解为域自适应和重采样过程，以处理不成对的低分辨率和高分辨率信号中的声学失配。然后在CycleGAN框架内联合优化这两个过程。实验结果表明，在数据不成对的情况下，所提方法的性能明显优于传统方法。代码和音频样本可从www.example.com获得https://chomeyama.github.io/DualCycleGAN-Demo/。摘要：Neural audio super-resolution models are typically trained on low- and high-resolution audio signal pairs. Although these methods achieve highly accurate super-resolution if the acoustic characteristics of the input data are similar to those of the training data, challenges remain: the models suffer from quality degradation for out-of-domain data, and paired data are required for training. To address these problems, we propose Dual-CycleGAN, a high-quality audio super-resolution method that can utilize unpaired data based on two connected cycle consistent generative adversarial networks (CycleGAN). Our method decomposes the super-resolution method into domain adaptation and resampling processes to handle acoustic mismatch in the unpaired low- and high-resolution signals. The two processes are then jointly optimized within the CycleGAN framework. Experimental results verify that the proposed method significantly outperforms conventional methods when paired data are not available. Code and audio samples are available from https://chomeyama.github.io/DualCycleGAN-Demo/.
【35】 Improving short-video speech recognition using random utterance concatenation标题：利用随机语音级联改进短视频语音识别链接：https://arxiv.org/abs/2210.15876
作者：Haihua Xu,Van Tung Pham,Yerbolat Khassanov,Yist Lin,Tao Han,Tze Yuan Chong,Yi He,Zejun Ma机构：ByteDance AI Lab备注：5 pages, 2 figures, 4 tables摘要：端到端自动语音识别框架的局限性之一是如果训练-测试话语长度不匹配，其性能将受到影响。针对短视频语音识别任务中训练-测试语句长度不匹配的问题，提出一种随机语句拼接方法。具体而言，我们的动机是观察到我们的人工转录的训练话语往往短得多的短视频自发语音（平均约3秒），而我们的测试话语从语音活动检测前端生成的长得多（平均约10秒）。这种不匹配可能导致次优性能。实验结果表明，通过使用所提出的RUC方法，可以在训练数据量增加约3倍以及每个训练数据使用两个语音级联的情况下实现最佳的误字率降低（WERR）。在实践中，所提出的方法始终优于强基线模型，其中在14种语言上实现了3.64%的平均WERR。摘要：One of the limitations in end-to-end automatic speech recognition framework is its performance would be compromised if train-test utterance lengths are mismatched. In this paper, we propose a random utterance concatenation (RUC) method to alleviate train-test utterance length mismatch issue for short-video speech recognition task. Specifically, we are motivated by observations our human-transcribed training utterances tend to be much shorter for short-video spontaneous speech (~3 seconds on average), while our test utterance generated from voice activity detection front-end is much longer (~10 seconds on average). Such a mismatch can lead to sub-optimal performance. Experimentally, by using the proposed RUC method, the best word error rate reduction (WERR) can be achieved with around three fold training data size increase as well as two utterance concatenation for each. In practice, the proposed method consistently outperforms the strong baseline models, where 3.64% average WERR is achieved on 14 languages.
【36】 UX-NET: Filter-and-Process-based Improved U-Net for Real-time Time-domain Audio Separation标题：UX-Net：基于滤波和处理的改进U-Net实时时域音频分离链接：https://arxiv.org/abs/2210.15822
作者：Kashyap Patel,Anton Kovalyov,Issa Panahi机构：Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX, USA备注：Submitted to ICASSP 2023摘要：本文提出了一种基于改进的U-Net结构的时域音频分离网络（TasNet）,, UX-Net。所提出的UX-Net实时工作，可处理单个或多个麦克风输入。受基于滤波和处理的人类听觉行为的启发，所提出的系统引入了新颖的混合器和分离模块，这导致语音源的成本和存储器有效的建模。混合器模块在潜在特征空间中组合编码的输入，并输出期望数量的输出流。然后，在分离模块中，应用修改的U-Net（UX）块。UX块首先以各种分辨率对编码输入进行滤波，然后聚集滤波后的信息并应用递归处理来估计分离源的掩码。UX-Net中的字母“X”是UX块中使用的循环层类型的名称占位符。WSJ 0 - 2 mix基准测试数据集的实证研究结果表明，其中一种UX-Net配置的SI-SNR比最先进的Conv-TasNet系统高0.85 dB，同时仅使用16%的模型参数，计算量减少58%，并保持低延迟。摘要：This study presents UX-Net, a time-domain audio separation network (TasNet) based on a modified U-Net architecture. The proposed UX-Net works in real-time and handles either single or multi-microphone input. Inspired by the filter-and-process-based human auditory behavior, the proposed system introduces novel mixer and separation modules, which result in cost and memory efficient modeling of speech sources. The mixer module combines encoded input in a latent feature space and outputs a desired number of output streams. Then, in the separation module, a modified U-Net (UX) block is applied. The UX block first filters the encoded input at various resolutions followed by aggregating the filtered information and applying recurrent processing to estimate masks of separated sources. The letter 'X' in UX-Net is a name placeholder for the type of recurrent layer employed in the UX block. Empirical findings on the WSJ0-2mix benchmark dataset show that one of the UX-Net configurations outperforms the state-of-the-art Conv-TasNet system by 0.85 dB SI-SNR while using only 16% of the model parameters, 58% fewer computations, and maintaining low latency.
【37】 Conditioning and Sampling in Variational Diffusion Models for Speech Super-resolution标题：语音超分辨率变分扩散模型中的条件和采样链接：https://arxiv.org/abs/2210.15793
作者：Chin-Yun Yu,Sung-Lin Yeh,György Fazekas,Hao Tang机构：Centre for Digital Music, Queen Mary University of London, UK, Institute for Language, Cognition and Computation, University of Edinburgh, UK备注：Submitted to ICASSP 2023摘要：近年来，扩散模型（diffusion models，DM）被越来越多地应用于音频处理任务，包括语音超分辨率（speech super-resolution，SR），其目的是在给定低分辨率语音的情况下恢复高频内容。这通常通过用低分辨率音频调节噪声预测器的网络来实现。本文提出了一种新的采样算法，通过DM的逆采样过程来传递低分辨率音频的信息。该方法可以直接替代传统的香草抽样过程，并能显著提高现有工作的性能。此外，通过将所提出的采样方法与无条件DM相耦合，对于没有辅助输入到其噪声预测器的DM，我们可以将其推广到宽范围的SR设置。我们还利用这种新配方在VCTK多扬声器基准测试中获得了最先进的结果。摘要：Recently, diffusion models (DMs) have been increasingly used in audio processing tasks, including speech super-resolution (SR), which aims to restore high-frequency content given low-resolution speech utterances. This is commonly achieved by conditioning the network of noise predictor with low-resolution audio. In this paper, we propose a novel sampling algorithm that communicates the information of the low-resolution audio via the reverse sampling process of DMs. The proposed method can be a drop-in replacement for the vanilla sampling process and can significantly improve the performance of the existing works. Moreover, by coupling the proposed sampling method with an unconditional DM, i.e., a DM with no auxiliary inputs to its noise predictor, we can generalize it to a wide range of SR setups. We also attain state-of-the-art results on the VCTK Multi-Speaker benchmark with this novel formulation.
【38】 AmberNet: A Compact End-to-End Model for Spoken Language Identification标题：AmberNet：一种紧凑的端到端口语识别模型链接：https://arxiv.org/abs/2210.15781
作者：Fei Jia,Nithin Rao Koluguri,Jagadeesh Balam,Boris Ginsburg机构：NVIDIA Corporation备注：Submitted to ICASSP 2023摘要：本文提出了一种用于口语识别的端到端神经网络-, AmberNet。AmberNet由1D深度方向可分离卷积和具有全局上下文的挤压和激励层组成，随后是统计池和线性层。AmberNet在VoxLingua 107数据集上实现了与最先进的（SOTA）模型相似的性能，而体积却小了10倍。AmberNet可以通过简单的微调来适应看不见的语言和新的声学条件。在FLEURS基准测试中，SOTA精度达到75.8%。我们证明了该模型易于扩展，以实现准确性和速度之间的更好平衡。我们进一步检验了模型对输入长度的敏感性，并表明AmberNet即使在短话语上也表现良好。摘要：We present AmberNet, a compact end-to-end neural network for Spoken Language Identification. AmberNet consists of 1D depth-wise separable convolutions and Squeeze-and-Excitation layers with global context, followed by statistics pooling and linear layers. AmberNet achieves performance similar to state-of-the-art(SOTA) models on VoxLingua107 dataset, while being 10x smaller. AmberNet can be adapted to unseen languages and new acoustic conditions with simple finetuning. It attains SOTA accuracy of 75.8% on FLEURS benchmark. We show the model is easily scalable to achieve a better trade-off between accuracy and speed. We further inspect the model's sensitivity to input length and show that AmberNet performs well even on short utterances.
【39】 Proceedings of the ACII Affective Vocal Bursts Workshop and Competition 2022 (A-VB): Understanding a critically understudied modality of emotional expression标题：ACII情感爆发研讨会和竞赛论文集2022(A-VB)：理解一种未被充分研究的情绪表达方式链接：https://arxiv.org/abs/2210.15754
作者：Alice Baird,Panagiotis Tzirakis,Jeffrey A. Brooks,Christopher B. Gregory,Björn Schuller,Anton Batliner,Dacher Keltner,Alan Cowen摘要：这是ACII情感爆发式发声研讨会和比赛（A-VB）的会议记录。A-VB是一个基于研讨会的挑战，它引入了理解声音爆发中的情感表达的问题--一个广泛的非语言发声，包括笑、咕噜声、喘息声等等。随着情感状态通知心理和身体健康，A-VB研讨会的核心焦点是更广泛地讨论当前的情感计算策略，以建模声乐情感表达。在A-VB挑战的第一次迭代中，参与者面临着四个以情绪为中心的子挑战，这些子挑战利用了大规模和“野生”的休谟-VB数据集。数据集和四个子挑战吸引了人们对情感科学新创新的关注，因为它涉及声音表达，解决了情感表达的低维和高维理论，文化差异，和'呼叫类型'（笑，哭，叹息等）。摘要：This is the Proceedings of the ACII Affective Vocal Bursts Workshop and Competition (A-VB). A-VB was a workshop-based challenge that introduces the problem of understanding emotional expression in vocal bursts -- a wide range of non-verbal vocalizations that includes laughs, grunts, gasps, and much more. With affective states informing both mental and physical wellbeing, the core focus of the A-VB workshop was the broader discussion of current strategies in affective computing for modeling vocal emotional expression. Within this first iteration of the A-VB challenge, the participants were presented with four emotion-focused sub-challenges that utilize the large-scale and `in-the-wild' Hume-VB dataset. The dataset and the four sub-challenges draw attention to new innovations in emotion science as it pertains to vocal expression, addressing low- and high-dimensional theories of emotional expression, cultural variation, and `call types' (laugh, cry, sigh, etc.).
【40】 Simulating realistic speech overlaps improves multi-talker ASR标题：模拟真实语音重叠提高多人ASR链接：https://arxiv.org/abs/2210.15715
作者：Muqiao Yang,Naoyuki Kanda,Xiaofei Wang,Jian Wu,Sunit Sivasankaran,Zhuo Chen,Jinyu Li,Takuya Yoshioka机构： Carnegie Mellon University, Pittsburgh, PA, USA, Microsoft, Redmond, WA, USA备注：5 pages摘要：多说话者自动语音识别（ASR）已经被研究用于生成包括多个说话者的重叠语音的自然会话的转录。由于很难通过高质量的人工转录获得真实的会话数据，传统的模型训练方法是通过随机混合多个说话人的话语来进行朴素的多说话人语音模拟.在本文中，我们提出了一种改进的技术来模拟具有真实语音重叠的多说话者重叠语音，其中语音重叠的任意模式由离散令牌序列来表示。利用这种表示，可以基于诸如N元语法的统计语言模型从真实会话中学习语音重叠模式，然后可以使用该统计语言模型来生成用于训练的多说话者语音。在我们的实验中，使用所提出的方法训练的多说话者ASR模型在多个数据集上显示出一致的单词错误率改善。摘要：Multi-talker automatic speech recognition (ASR) has been studied to generate transcriptions of natural conversation including overlapping speech of multiple speakers. Due to the difficulty in acquiring real conversation data with high-quality human transcriptions, a na\"ive simulation of multi-talker speech by randomly mixing multiple utterances was conventionally used for model training. In this work, we propose an improved technique to simulate multi-talker overlapping speech with realistic speech overlaps, where an arbitrary pattern of speech overlaps is represented by a sequence of discrete tokens. With this representation, speech overlapping patterns can be learned from real conversations based on a statistical language model, such as N-gram, which can be then used to generate multi-talker speech for training. In our experiments, multi-talker ASR models trained with the proposed method show consistent improvement on the word error rates across multiple datasets.
【1】 Contextual-Utterance Training for Automatic Speech Recognition标题：用于自动语音识别的语境话语训练链接：https://arxiv.org/abs/2210.16238
* 与cs.SD语音【22】为同一篇
作者：Alejandro Gomez-Alanis,Lukas Drude,Andreas Schwarz,Rupak Vignesh Swaminathan,Simon Wiesler机构：Wiesler, Amazon Alexa, Aachen, Germany; ,Amazon Alexa, Pittsburgh, USA摘要：基于递归神经网络转换器（RNN-T）的流式自动语音识别（ASR）系统的最新研究已经将过去的上下文信息馈送给编码器，以改善其误字率（WER）性能。本文首先提出了一种上下文话语训练技术，它利用先前和未来的上下文话语来对说话人、话题和声学环境进行隐式适应。此外，我们提出一种双模式的上下文-话语训练技术用于流式自动语音识别（ASR）系统。该方法允许通过将教师的知识“就地”提取到学生来更好地利用流模型中的可用声学上下文，其中教师能够看到过去和未来的上下文话语，而学生只能看到当前和过去的上下文话语。实验结果表明，使用所提方法训练的保形器-换能器系统的性能优于使用经典RNN-T损失训练的相同系统。具体地说，所提出的技术能够将WER和平均最后令牌发送延迟分别减少6%和40 ms以上。摘要：Recent studies of streaming automatic speech recognition (ASR) recurrent neural network transducer (RNN-T)-based systems have fed the encoder with past contextual information in order to improve its word error rate (WER) performance. In this paper, we first propose a contextual-utterance training technique which makes use of the previous and future contextual utterances in order to do an implicit adaptation to the speaker, topic and acoustic environment. Also, we propose a dual-mode contextual-utterance training technique for streaming automatic speech recognition (ASR) systems. This proposed approach allows to make a better use of the available acoustic context in streaming models by distilling "in-place" the knowledge of a teacher, which is able to see both past and future contextual utterances, to the student which can only see the current and past contextual utterances. The experimental results show that a conformer-transducer system trained with the proposed techniques outperforms the same system trained with the classical RNN-T loss. Specifically, the proposed technique is able to reduce both the WER and the average last token emission latency by more than 6% and 40ms relative, respectively.
【2】 Target-Speaker Voice Activity Detection via Sequence-to-Sequence Prediction标题：基于序列到序列预测的目标说话人语音活动检测链接：https://arxiv.org/abs/2210.16127
* 与cs.SD语音【23】为同一篇
作者：Ming Cheng,Weiqing Wang,Yucong Zhang,Xiaoyi Qin,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China备注：submitted to ICASSP2023摘要：目标说话人语音活动检测是当前复杂声学环境下说话人识别的一种有效方法。提出了一种新的序列到序列目标说话人语音活动检测（Seq 2Seq-TSVAD）方法，能够有效解决大规模说话人的联合建模和高分辨率语音活动预测问题。实验结果表明，较大的扬声器容量和较高的输出分辨率可以显著降低二值化错误率（DER），在VoxConverse测试集和DIHARD-III评测集上分别达到了4.55%和10.77%的最佳性能。摘要：Target-speaker voice activity detection is currently a promising approach for speaker diarization in complex acoustic environments. This paper presents a novel Sequence-to-Sequence Target-Speaker Voice Activity Detection (Seq2Seq-TSVAD) method that can efficiently address the joint modeling of large-scale speakers and predict high-resolution voice activities. Experimental results show that larger speaker capacity and higher output resolution can significantly reduce the diarization error rate (DER), which achieves the new state-of-the-art performance of 4.55% on the VoxConverse test set and 10.77% on Track 1 of the DIHARD-III evaluation set under the widely-used evaluation metrics.
【3】 Parameter-efficient transfer learning of pre-trained Transformer models for speaker verification using adapters标题：使用适配器的用于说话人确认的预训练Transformer模型的参数高效转移学习链接：https://arxiv.org/abs/2210.16032
* 与cs.SD语音【24】为同一篇
作者：Junyi Peng,Themos Stafylakis,Rongzhi Gu,Oldřich Plchot,Ladislav Mošner,Lukáš Burget,Jan Černocký机构：Luk´aˇs Burget, Jan ˇCernock´y, Omilia - Conversational Intelligence, Athens, Greece, Tencent AI Lab, Shenzhen, China备注：submitted to ICASSP2023摘要：近年来，预训练的Transformer模型由于在各种下游任务中取得的巨大成功，在语音处理领域受到了越来越多的关注。然而，大多数微调方法更新预训练模型的所有参数，这随着模型大小的增长而变得禁止，并且有时导致对小数据集的过拟合。本文对参数有效迁移学习（PETL）方法在说话人确认中的应用进行了全面分析，以减少所需的可学习参数，从而适应说话人确认任务。具体来说，在微调过程中，预先训练的模型被冻结，只有插入每个Transformer块中的轻量级模块是可训练的（一种称为适配器的方法）。此外，为了提高跨语言低资源场景中的性能，在直接在小数据集上进行微调之前，Transformer模型会在大的中间数据集上进行进一步的微调。通过更新不到4%的参数，（我们提出的）基于PETL的方法实现了与完全微调方法（Vox 1-O：0.55%，Vox1-E：0.82%，Vox 1-H为1.73%）。摘要：Recently, the pre-trained Transformer models have received a rising interest in the field of speech processing thanks to their great success in various downstream tasks. However, most fine-tuning approaches update all the parameters of the pre-trained model, which becomes prohibitive as the model size grows and sometimes results in overfitting on small datasets. In this paper, we conduct a comprehensive analysis of applying parameter-efficient transfer learning (PETL) methods to reduce the required learnable parameters for adapting to speaker verification tasks. Specifically, during the fine-tuning process, the pre-trained models are frozen, and only lightweight modules inserted in each Transformer block are trainable (a method known as adapters). Moreover, to boost the performance in a cross-language low-resource scenario, the Transformer model is further tuned on a large intermediate dataset before directly fine-tuning it on a small dataset. With updating fewer than 4% of parameters, (our proposed) PETL-based methods achieve comparable performances with full fine-tuning methods (Vox1-O: 0.55%, Vox1-E: 0.82%, Vox1-H:1.73%).
【4】 Laugh Betrays You? Learning Robust Speaker Representation From Speech Containing Non-Verbal Fragments标题：笑背叛了你？从含有非语言片段的语音中学习稳健的说话人表征链接：https://arxiv.org/abs/2210.16028
* 与cs.SD语音【25】为同一篇
作者：Yuke Lin,Xiaoyi Qin,Huahua Cui,Zhenyi Zhu,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China, Advanced Computing East China Sub-Center, Kunshan, China摘要：自动说话人确认的成功表明，可以从中性语音中提取出有区别的说话人表示。然而，作为一种非语言的声音，笑声也应该直观地携带说话人信息。因此，本文主要研究了含有非言语笑声片段的说话人确认问题。我们通过在VoxCeleb和CN-Celeb数据集的一部分上执行笑声检测脚本，收集了一组包含笑声成分的剪辑。为了进一步过滤不可信的视频片段，我们使用了我们的二元笑声检测分类器来计算概率得分，该分类器是由纯笑声和中性语音预先训练的。然后，基于得分超过阈值的片段，我们构建了两种不同评估场景下的试验：笑-笑（LL）和说-笑（SL）。在此基础上，提出了一种新的基于笑点拼接的网络（Laughter-Splicing based Network，LSN）方法，该方法在两种情况下都能显著提高性能，并且在中性语音（如VoxCeleb 1测试集）上保持了较好的性能。具体来说，我们的系统在笑-笑和说-笑试验中分别实现了20%和22%的相对改善。元数据和示例剪辑已在www.example.com上发布https://github.com/nevermoreLin/Laugh_LSN。摘要：The success of automatic speaker verification shows that discriminative speaker representations can be extracted from neutral speech. However, as a kind of non-verbal voice, laughter should also carry speaker information intuitively. Thus, this paper focuses on exploring speaker verification about utterances containing non-verbal laughter segments. We collect a set of clips with laughter components by conducting a laughter detection script on VoxCeleb and part of the CN-Celeb dataset. To further filter untrusted clips, probability scores are calculated by our binary laughter detection classifier, which is pre-trained by pure laughter and neutral speech. After that, based on the clips whose scores are over the threshold, we construct trials under two different evaluation scenarios: Laughter-Laughter (LL) and Speech-Laughter (SL). Then a novel method called Laughter-Splicing based Network (LSN) is proposed, which can significantly boost performance in both scenarios and maintain the performance on the neutral speech, such as the VoxCeleb1 test set. Specifically, our system achieves relative 20% and 22% improvement on Laughter-Laughter and Speech-Laughter trials, respectively. The meta-data and sample clips have been released at https://github.com/nevermoreLin/Laugh_LSN.
【5】 SG-VAD: Stochastic Gates Based Speech Activity Detection标题：SG-VAD：基于随机门的语音活动检测链接：https://arxiv.org/abs/2210.16022
* 与cs.SD语音【26】为同一篇
作者：Jonathan Svirsky,Ofir Lindenbaum机构：Bar-Ilan University, Ramat-Gan, Israel摘要：提出了一种低资源环境下的语音激活检测（VAD）模型.我们的核心思想是将VAD建模为一个去噪任务，并构造一个网络，该网络被设计用于识别语音分类任务的多余特征。我们训练该模型以在预测语音事件类型的同时识别不相关的特征。我们的模型仅包含7.8K个参数，在AVA-Speech评估集上优于先前提出的方法，并且在HAVIC数据集上提供了比较结果。本文介绍了该模型的结构、实验结果以及对模型各部件的烧蚀研究。我们在www.example.com上发布了代码和模型https://www.github.com/jsvir/vad。摘要：We propose a novel voice activity detection (VAD) model in a low-resource environment. Our key idea is to model VAD as a denoising task, and construct a network that is designed to identify nuisance features for a speech classification task. We train the model to simultaneously identify irrelevant features while predicting the type of speech event. Our model contains only 7.8K parameters, outperforms the previously proposed methods on the AVA-Speech evaluation set, and provides comparative results on the HAVIC dataset. We present its architecture, experimental results, and ablation study on the model's components. We publish the code and the models here https://www.github.com/jsvir/vad.
【6】 NNSVS: A Neural Network-Based Singing Voice Synthesis Toolkit标题：NNSVS：一个基于神经网络的歌唱语音合成工具包链接：https://arxiv.org/abs/2210.15987
* 与cs.SD语音【27】为同一篇
作者：Ryuichi Yamamoto,Reo Yoneyama,Tomoki Toda机构：LINE Corp., Japan,Nagoya University, Japan.备注：Submitted to ICASSP 2023摘要：本文介绍了一个基于神经网络的歌唱语音合成研究的开源软件NNSVS的设计。NNSVS的灵感来自于歌唱声音合成研究领域的开源先驱Sinsy，它提供了许多附加功能，如多流模型、自回归基频模型和神经声码器。此外，NNSVS还提供了大量的文档和大量脚本来构建完整的歌唱声音合成系统。实验结果表明，我们的最佳系统明显优于我们的Sinsy和其他基线系统的复制。该工具包可从www.example.com获得https://github.com/nnsvs/nnsvs。摘要：This paper describes the design of NNSVS, an open-source software for neural network-based singing voice synthesis research. NNSVS is inspired by Sinsy, an open-source pioneer in singing voice synthesis research, and provides many additional features such as multi-stream models, autoregressive fundamental frequency models, and neural vocoders. Furthermore, NNSVS provides extensive documentation and numerous scripts to build complete singing voice synthesis systems. Experimental results demonstrate that our best system significantly outperforms our reproduction of Sinsy and other baseline systems. The toolkit is available at https://github.com/nnsvs/nnsvs.
【7】 Dysfluencies Seldom Come Alone -- Detection as a Multi-Label Problem标题：不流利很少单独出现--检测是一个多标签问题链接：https://arxiv.org/abs/2210.15982
* 与cs.SD语音【28】为同一篇
作者：Sebastian P. Bayerl,Dominik Wagner,Florian Hönig,Tobias Bocklet,Elmar Nöth,Korbinian Riedhammer机构：⋆ Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, ‡KST Institut GmbH, ⋄Intel Labs, † Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Germany备注：Submitted to ICASSP 2023摘要：为了处理口吃的语音，需要特别适配的语音识别模型。为了以有针对性的方式使用这些，必须可靠地检测口吃语音。最近的研究将口吃视为多类分类问题，或者将检测每种不流畅类型视为一项孤立的任务;这没有抓住口吃的本质，口吃的本质是一种不流畅很少单独出现，即，与其他事件同时发生。本研究探讨一种以wav 2 vec 2.0系统为基础的多标签问题之端对端口吃侦测与分类方法。该方法在包含英语和德语口吃语音的三个数据集的组合上进行评估，在SEP-28 k-Extended数据集上产生了口吃检测的最新结果。实验结果证明了特征的可移植性以及该方法在数据集和语言间的通用性。摘要：Specially adapted speech recognition models are necessary to handle stuttered speech. For these to be used in a targeted manner, stuttered speech must be reliably detected. Recent works have treated stuttering as a multi-class classification problem or viewed detecting each dysfluency type as an isolated task; that does not capture the nature of stuttering, where one dysfluency seldom comes alone, i.e., co-occurs with others. This work explores an approach based on a modified wav2vec 2.0 system for end-to-end stuttering detection and classification as a multi-label problem. The method is evaluated on combinations of three datasets containing English and German stuttered speech, yielding state-of-the-art results for stuttering detection on the SEP-28k-Extended dataset. Experimental results provide evidence for the transferability of features and the generalizability of the method across datasets and languages.
【8】 End-to-end Ensemble-based Feature Selection for Paralinguistics Tasks标题：基于端到端集成的副语言学任务特征选择链接：https://arxiv.org/abs/2210.15978
* 与cs.SD语音【29】为同一篇
作者：Tamás Grósz,Mittul Singh,Sudarsana Reddy Kadiri,Hemant Kathania,Mikko Kurimo机构：The authors would like to thank Antonia Hamilton and Alexis Macintyre for granting access to a subset of the UCL Speech, BreathMonitoring (UCL-SBM) database used in the Breathing Sub-Challenge. This work was supported by the Academy of Finland摘要：近年来发生的事件突出了远程医疗解决方案的重要性，这种解决方案有可能实现远程治疗和诊断。计算副语言学是语音处理领域的一个独特分支，其目标是提取说话人的信息，是远程医疗应用的重要组成部分。在本研究中，我们关注两个副语言问题：面罩检测和呼吸状态预测。为这些任务开发的解决方案可能是无价的，并有可能帮助监测和限制像COVID-19这样的病毒的传播。目前针对这些任务提出的最先进的方法是基于深度神经网络（如ResNets）结合特征工程的集成。虽然这些系综可以实现高精度，但是它们也具有大的覆盖区并且需要相当大的计算能力，从而降低了具有有限资源的设备的便携性。这些缺点还意味着先前提出的解决方案由于它们的尺寸和速度而不适用于远程医疗系统。另一方面，采用较轻的特征工程系统可能是费力的，并且增加了进一步的复杂性，使得它们难以快速地创建可部署的系统。本文提出了一种基于集成的自动特征选择方法，以实现快速、高效的系统开发。特别地，我们提出了一种基于输出梯度的方法，在训练一个较小的集合之前，使用大的、性能良好的集合来发现本质特征。在我们的实验中，我们观察到使用基于输出梯度的特征的神经网络集合在推理时间上有相当大的减少（25-32%）。我们的方法提供了一种简单的方法来提高系统的速度，并实现实时使用，同时保持与使用所有光谱特征的更大覆盖区集合的竞争结果。摘要：The events of recent years have highlighted the importance of telemedicine solutions which could potentially allow remote treatment and diagnosis. Relatedly, Computational Paralinguistics, a unique subfield of Speech Processing, aims to extract information about the speaker and form an important part of telemedicine applications. In this work, we focus on two paralinguistic problems: mask detection and breathing state prediction. Solutions developed for these tasks could be invaluable and have the potential to help monitor and limit the spread of a virus like COVID-19. The current state-of-the-art methods proposed for these tasks are ensembles based on deep neural networks like ResNets in conjunction with feature engineering. Although these ensembles can achieve high accuracy, they also have a large footprint and require substantial computational power reducing portability to devices with limited resources. These drawbacks also mean that the previously proposed solutions are infeasible to be used in a telemedicine system due to their size and speed. On the other hand, employing lighter feature-engineered systems can be laborious and add further complexity making them difficult to create a deployable system quickly. This work proposes an ensemble-based automatic feature selection method to enable the development of fast and memory-efficient systems. In particular, we propose an output-gradient-based method to discover essential features using large, well-performing ensembles before training a smaller one. In our experiments, we observed considerable (25-32%) reductions in inference times using neural network ensembles based on output-gradient-based features. Our method offers a simple way to increase the speed of the system and enable real-time usage while maintaining competitive results with larger-footprint ensemble using all spectral features.
【9】 Lightweight and High-Fidelity End-to-End Text-to-Speech with Multi-Band Generation and Inverse Short-Time Fourier Transform标题：采用多频带生成和短时傅里叶逆变换的轻量级高保真端到端文语转换链接：https://arxiv.org/abs/2210.15975
* 与cs.SD语音【30】为同一篇
作者：Masaya Kawamura,Yuma Shirahata,Ryuichi Yamamoto,Kentaro Tachibana机构：The University of Tokyo, Japan,LINE Corp., Japan.备注：Submitted to ICASSP 2023摘要：提出了一种基于多频带生成和短时傅立叶逆变换的轻量级端到端语音合成模型。我们的模型基于高质量的端到端文本到语音模型--VITS，但是为了更有效的推理，我们采用了两个变化：1）用简单的逆短时傅立叶变换部分地替换计算上最昂贵的分量，以及2）使用具有固定或可训练合成滤波器的多频带生成来生成波形。与传统的轻量级模型不同，传统的轻量级模型分别采用优化或知识提取来训练两个级联的组件，我们的方法享有端到端优化的全部好处。实验结果表明，在Intel Core i7处理器上，我们的模型合成的语音与通过VITS合成的语音一样自然，同时实现了0.066的实时系数，比VITS快4.1倍。此外，该模型的较小版本在自然度和推理速度方面明显优于轻量级基线模型。代码和音频样本可从www.example.com获得https://github.com/MasayaKawamura/MB-iSTFT-VITS。摘要：We propose a lightweight end-to-end text-to-speech model using multi-band generation and inverse short-time Fourier transform. Our model is based on VITS, a high-quality end-to-end text-to-speech model, but adopts two changes for more efficient inference: 1) the most computationally expensive component is partially replaced with a simple inverse short-time Fourier transform, and 2) multi-band generation, with fixed or trainable synthesis filters, is used to generate waveforms. Unlike conventional lightweight models, which employ optimization or knowledge distillation separately to train two cascaded components, our method enjoys the full benefits of end-to-end optimization. Experimental results show that our model synthesized speech as natural as that synthesized by VITS, while achieving a real-time factor of 0.066 on an Intel Core i7 CPU, 4.1 times faster than VITS. Moreover, a smaller version of the model significantly outperformed a lightweight baseline model with respect to both naturalness and inference speed. Code and audio samples are available from https://github.com/MasayaKawamura/MB-iSTFT-VITS.
【10】 Period VITS: Variational Inference with Explicit Pitch Modeling for End-to-end Emotional Speech Synthesis标题：PERIOD VITS：端到端情感语音合成的变分推理和显式基音建模链接：https://arxiv.org/abs/2210.15964
* 与cs.SD语音【31】为同一篇
作者：Yuma Shirahata,Ryuichi Yamamoto,Eunwoo Song,Ryo Terashima,Jae-Min Kim,Kentaro Tachibana机构：LINE Corp., Tokyo, Japan, NAVER Corp., Seongnam, Korea备注：Submitted to ICASSP 2023摘要：已经提出了几种完全端到端的文本到语音（TTS）模型，分别训练声学和声码器模型）。然而，当数据集包含情感属性时，它们经常生成具有可听伪像的不稳定音调轮廓，即，发音和韵律巨大差异。针对这一问题，本文提出了一种新的端到端TTS模型--Period VITS，该模型包含一个显式的周期发生器。在该方法中，我们引入了一个帧基音预测器，该预测器从输入文本中预测韵律特征，如基音和浊音标志。根据这些特征，所提出的周期性发生器产生样本级正弦源，该正弦源使得波形解码器能够准确地再现音调。最后，采用变分推理和对抗性目标，以端到端的方式对整个模型进行联合优化。结果，解码器变得能够产生更稳定、更有表现力和更自然的输出波形。实验结果表明，该模型在自然度方面明显优于基线模型，在生成的样本中，基音稳定性得到了改善。摘要：Several fully end-to-end text-to-speech (TTS) models have been proposed that have shown better performance compared to cascade models (i.e., training acoustic and vocoder models separately). However, they often generate unstable pitch contour with audible artifacts when the dataset contains emotional attributes, i.e., large diversity of pronunciation and prosody. To address this problem, we propose Period VITS, a novel end-to-end TTS model that incorporates an explicit periodicity generator. In the proposed method, we introduce a frame pitch predictor that predicts prosodic features, such as pitch and voicing flags, from the input text. From these features, the proposed periodicity generator produces a sample-level sinusoidal source that enables the waveform decoder to accurately reproduce the pitch. Finally, the entire model is jointly optimized in an end-to-end manner with variational inference and adversarial objectives. As a result, the decoder becomes capable of generating more stable, expressive, and natural output waveforms. The experimental results showed that the proposed model significantly outperforms baseline models in terms of naturalness, with improved pitch stability in the generated samples.
【11】 The Importance of Speech Stimuli for Pathologic Speech Classification标题：语音刺激对病理性语音分类的重要性链接：https://arxiv.org/abs/2210.15941
* 与cs.SD语音【32】为同一篇
作者：Ilja Baumann,Dominik Wagner,Franziska Braun,Sebastian P. Bayerl,Elmar Nöth,Korbinian Riedhammer,Tobias Bocklet机构：†Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, ‡Friedrich-Alexander-University Erlangen-N¨urnberg, Pattern Recognition Lab, Erlangen, Germany, Intel Labs备注：Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2210.15336摘要：目前的研究结果表明，预先训练好的wav2vec 2.0模型可以成功地用作特征提取器来区分基于说话人的任务。我们证明了在预先训练的wav2vec 2.0系统的不同层提取的潜在表示可以有效地用于各种类型的病理语音的二进制分类。为此，我们检查了喉切除术、口腔鳞状细胞癌、帕金森病和唇腭裂的病理。结果表明，病理和健康嗓音之间的区分，特别是利用来自较低层的潜在表征，表现良好，具有从帕金森病的77.2%到喉切除术分类的100%的最低准确率。然而，交叉病理学和交叉健康测试表明，训练的分类器似乎有偏见。如果在训练和域外测试数据之间存在不匹配，在年龄、说话内容或声音条件方面。摘要：Current findings show that pre-trained wav2vec 2.0 models can be successfully used as feature extractors to discriminate on speaker-based tasks. We demonstrate that latent representations extracted at different layers of a pre-trained wav2vec 2.0 system can be effectively used for binary classification of various types of pathologic speech. We examine the pathologies laryngectomy, oral squamous cell carcinoma, parkinson's disease and cleft lip and palate for this purpose. The results show that a distinction between pathological and healthy voices, especially with latent representations from the lower layers, performs well with the lowest accuracy from 77.2% for parkinson's disease to 100% for laryngectomy classification. However, cross-pathology and cross-healthy tests show that the trained classifiers seem to be biased. The recognition rates vary considerably if there is a mismatch between training and out-of-domain test data, e.g., in age, spoken content or acoustic conditions.
【12】 Speaker recognition with two-step multi-modal deep cleansing标题：基于两步多模式深度清洗的说话人识别链接：https://arxiv.org/abs/2210.15903
* 与cs.SD语音【33】为同一篇
作者：Ruijie Tao,Kong Aik Lee,Zhan Shi,Haizhou Li机构：College of Design and Engineering, National University of Singapore, Singapore, Institute for Infocomm Research, A⋆STAR, Singapore, The Chinese University of Hong Kong, Shenzhen, China备注：5 pages, 3 figures摘要：近年来，基于神经网络的说话人识别技术取得了显著的进展。鲁棒的说话人表示从训练集中的难样本和易样本学习有意义的知识以获得良好的性能。然而，噪声样本（即，具有错误的标签）引起混淆并使网络学习到不正确的表示。本文提出了一种两步视听深度清洗框架，以消除说话人表征学习中噪声标签的影响。该框架包含一个粗粒度的清洗步骤来搜索特殊的样本，然后是一个细粒度的清洗步骤来过滤掉有噪声的标签。我们的研究从一个高效的音视频说话人识别系统开始，该系统在Vox-O，E和H测试集上的等误率（EER）分别为0.01%，0.07%和0.13%.采用本文提出的多模态清洗机制，四种不同的说话人识别网络取得了平均5.9%的改进.代码可从以下网址获得：{1}，{2}{3}您可以在这里看到您的照片。{4}摘要：Neural network-based speaker recognition has achieved significant improvement in recent years. A robust speaker representation learns meaningful knowledge from both hard and easy samples in the training set to achieve good performance. However, noisy samples (i.e., with wrong labels) in the training set induce confusion and cause the network to learn the incorrect representation. In this paper, we propose a two-step audio-visual deep cleansing framework to eliminate the effect of noisy labels in speaker representation learning. This framework contains a coarse-grained cleansing step to search for the peculiar samples, followed by a fine-grained cleansing step to filter out the noisy labels. Our study starts from an efficient audio-visual speaker recognition system, which achieves a close to perfect equal-error-rate (EER) of 0.01\%, 0.07\% and 0.13\% on the Vox-O, E and H test sets. With the proposed multi-modal cleansing mechanism, four different speaker recognition networks achieve an average improvement of 5.9\%. Code has been made available at: \textcolor{magenta}{\url{https://github.com/TaoRuijie/AVCleanse}}.
【13】 Nonparallel High-Quality Audio Super Resolution with Domain Adaptation and Resampling CycleGANs标题：域自适应和重采样循环阵列的非并行高质量音频超分辨率链接：https://arxiv.org/abs/2210.15887
* 与cs.SD语音【34】为同一篇
作者：Reo Yoneyama,Ryuichi Yamamoto,Kentaro Tachibana机构：Nagoya University, Japan,LINE Corp., Japan.备注：Submitted to ICASSP 2023摘要：神经音频超分辨率模型通常在低分辨率和高分辨率音频信号对上训练。尽管如果输入数据的声学特性与训练数据的声学特性相似，则这些方法实现了高度精确的超分辨率，但是挑战仍然存在：对于域外数据，模型遭受质量降级，并且需要成对的数据用于训练。针对这些问题，提出了一种基于两个连通的循环一致生成对抗网络（CycleGAN）的高质量音频超分辨率方法--Dual-CycleGAN。我们的方法将超分辨率方法分解为域自适应和重采样过程，以处理不成对的低分辨率和高分辨率信号中的声学失配。然后在CycleGAN框架内联合优化这两个过程。实验结果表明，在数据不成对的情况下，所提方法的性能明显优于传统方法。代码和音频样本可从www.example.com获得https://chomeyama.github.io/DualCycleGAN-Demo/。摘要：Neural audio super-resolution models are typically trained on low- and high-resolution audio signal pairs. Although these methods achieve highly accurate super-resolution if the acoustic characteristics of the input data are similar to those of the training data, challenges remain: the models suffer from quality degradation for out-of-domain data, and paired data are required for training. To address these problems, we propose Dual-CycleGAN, a high-quality audio super-resolution method that can utilize unpaired data based on two connected cycle consistent generative adversarial networks (CycleGAN). Our method decomposes the super-resolution method into domain adaptation and resampling processes to handle acoustic mismatch in the unpaired low- and high-resolution signals. The two processes are then jointly optimized within the CycleGAN framework. Experimental results verify that the proposed method significantly outperforms conventional methods when paired data are not available. Code and audio samples are available from https://chomeyama.github.io/DualCycleGAN-Demo/.
【14】 Improving short-video speech recognition using random utterance concatenation标题：利用随机语音级联改进短视频语音识别链接：https://arxiv.org/abs/2210.15876
* 与cs.SD语音【35】为同一篇
作者：Haihua Xu,Van Tung Pham,Yerbolat Khassanov,Yist Lin,Tao Han,Tze Yuan Chong,Yi He,Zejun Ma机构：ByteDance AI Lab备注：5 pages, 2 figures, 4 tables摘要：端到端自动语音识别框架的局限性之一是如果训练-测试话语长度不匹配，其性能将受到影响。针对短视频语音识别任务中训练-测试语句长度不匹配的问题，提出一种随机语句拼接方法。具体而言，我们的动机是观察到我们的人工转录的训练话语往往短得多的短视频自发语音（平均约3秒），而我们的测试话语从语音活动检测前端生成的长得多（平均约10秒）。这种不匹配可能导致次优性能。实验结果表明，通过使用所提出的RUC方法，可以在训练数据量增加约3倍以及每个训练数据使用两个语音级联的情况下实现最佳的误字率降低（WERR）。在实践中，所提出的方法始终优于强基线模型，其中在14种语言上实现了3.64%的平均WERR。摘要：One of the limitations in end-to-end automatic speech recognition framework is its performance would be compromised if train-test utterance lengths are mismatched. In this paper, we propose a random utterance concatenation (RUC) method to alleviate train-test utterance length mismatch issue for short-video speech recognition task. Specifically, we are motivated by observations our human-transcribed training utterances tend to be much shorter for short-video spontaneous speech (~3 seconds on average), while our test utterance generated from voice activity detection front-end is much longer (~10 seconds on average). Such a mismatch can lead to sub-optimal performance. Experimentally, by using the proposed RUC method, the best word error rate reduction (WERR) can be achieved with around three fold training data size increase as well as two utterance concatenation for each. In practice, the proposed method consistently outperforms the strong baseline models, where 3.64% average WERR is achieved on 14 languages.
【15】 UX-NET: Filter-and-Process-based Improved U-Net for Real-time Time-domain Audio Separation标题：UX-Net：基于滤波和处理的改进U-Net实时时域音频分离链接：https://arxiv.org/abs/2210.15822
* 与cs.SD语音【36】为同一篇
作者：Kashyap Patel,Anton Kovalyov,Issa Panahi机构：Electrical and Computer Engineering, University of Texas at Dallas, Richardson, TX, USA备注：Submitted to ICASSP 2023摘要：本文提出了一种基于改进的U-Net结构的时域音频分离网络（TasNet）,, UX-Net。所提出的UX-Net实时工作，可处理单个或多个麦克风输入。受基于滤波和处理的人类听觉行为的启发，所提出的系统引入了新颖的混合器和分离模块，这导致语音源的成本和存储器有效的建模。混合器模块在潜在特征空间中组合编码的输入，并输出期望数量的输出流。然后，在分离模块中，应用修改的U-Net（UX）块。UX块首先以各种分辨率对编码输入进行滤波，然后聚集滤波后的信息并应用递归处理来估计分离源的掩码。UX-Net中的字母“X”是UX块中使用的循环层类型的名称占位符。WSJ 0 - 2 mix基准测试数据集的实证研究结果表明，其中一种UX-Net配置的SI-SNR比最先进的Conv-TasNet系统高0.85 dB，同时仅使用16%的模型参数，计算量减少58%，并保持低延迟。摘要：This study presents UX-Net, a time-domain audio separation network (TasNet) based on a modified U-Net architecture. The proposed UX-Net works in real-time and handles either single or multi-microphone input. Inspired by the filter-and-process-based human auditory behavior, the proposed system introduces novel mixer and separation modules, which result in cost and memory efficient modeling of speech sources. The mixer module combines encoded input in a latent feature space and outputs a desired number of output streams. Then, in the separation module, a modified U-Net (UX) block is applied. The UX block first filters the encoded input at various resolutions followed by aggregating the filtered information and applying recurrent processing to estimate masks of separated sources. The letter 'X' in UX-Net is a name placeholder for the type of recurrent layer employed in the UX block. Empirical findings on the WSJ0-2mix benchmark dataset show that one of the UX-Net configurations outperforms the state-of-the-art Conv-TasNet system by 0.85 dB SI-SNR while using only 16% of the model parameters, 58% fewer computations, and maintaining low latency.
【16】 Conditioning and Sampling in Variational Diffusion Models for Speech Super-resolution标题：语音超分辨率变分扩散模型中的条件和采样链接：https://arxiv.org/abs/2210.15793
* 与cs.SD语音【37】为同一篇
作者：Chin-Yun Yu,Sung-Lin Yeh,György Fazekas,Hao Tang机构：Centre for Digital Music, Queen Mary University of London, UK, Institute for Language, Cognition and Computation, University of Edinburgh, UK备注：Submitted to ICASSP 2023摘要：近年来，扩散模型（diffusion models，DM）被越来越多地应用于音频处理任务，包括语音超分辨率（speech super-resolution，SR），其目的是在给定低分辨率语音的情况下恢复高频内容。这通常通过用低分辨率音频调节噪声预测器的网络来实现。本文提出了一种新的采样算法，通过DM的逆采样过程来传递低分辨率音频的信息。该方法可以直接替代传统的香草抽样过程，并能显著提高现有工作的性能。此外，通过将所提出的采样方法与无条件DM相耦合，对于没有辅助输入到其噪声预测器的DM，我们可以将其推广到宽范围的SR设置。我们还利用这种新配方在VCTK多扬声器基准测试中获得了最先进的结果。摘要：Recently, diffusion models (DMs) have been increasingly used in audio processing tasks, including speech super-resolution (SR), which aims to restore high-frequency content given low-resolution speech utterances. This is commonly achieved by conditioning the network of noise predictor with low-resolution audio. In this paper, we propose a novel sampling algorithm that communicates the information of the low-resolution audio via the reverse sampling process of DMs. The proposed method can be a drop-in replacement for the vanilla sampling process and can significantly improve the performance of the existing works. Moreover, by coupling the proposed sampling method with an unconditional DM, i.e., a DM with no auxiliary inputs to its noise predictor, we can generalize it to a wide range of SR setups. We also attain state-of-the-art results on the VCTK Multi-Speaker benchmark with this novel formulation.
【17】 AmberNet: A Compact End-to-End Model for Spoken Language Identification标题：AmberNet：一种紧凑的端到端口语识别模型链接：https://arxiv.org/abs/2210.15781
* 与cs.SD语音【38】为同一篇
作者：Fei Jia,Nithin Rao Koluguri,Jagadeesh Balam,Boris Ginsburg机构：NVIDIA Corporation备注：Submitted to ICASSP 2023摘要：本文提出了一种用于口语识别的端到端神经网络-, AmberNet。AmberNet由1D深度方向可分离卷积和具有全局上下文的挤压和激励层组成，随后是统计池和线性层。AmberNet在VoxLingua 107数据集上实现了与最先进的（SOTA）模型相似的性能，而体积却小了10倍。AmberNet可以通过简单的微调来适应看不见的语言和新的声学条件。在FLEURS基准测试中，SOTA精度达到75.8%。我们证明了该模型易于扩展，以实现准确性和速度之间的更好平衡。我们进一步检验了模型对输入长度的敏感性，并表明AmberNet即使在短话语上也表现良好。摘要：We present AmberNet, a compact end-to-end neural network for Spoken Language Identification. AmberNet consists of 1D depth-wise separable convolutions and Squeeze-and-Excitation layers with global context, followed by statistics pooling and linear layers. AmberNet achieves performance similar to state-of-the-art(SOTA) models on VoxLingua107 dataset, while being 10x smaller. AmberNet can be adapted to unseen languages and new acoustic conditions with simple finetuning. It attains SOTA accuracy of 75.8% on FLEURS benchmark. We show the model is easily scalable to achieve a better trade-off between accuracy and speed. We further inspect the model's sensitivity to input length and show that AmberNet performs well even on short utterances.
【18】 Proceedings of the ACII Affective Vocal Bursts Workshop and Competition 2022 (A-VB): Understanding a critically understudied modality of emotional expression标题：ACII情感爆发研讨会和竞赛论文集2022(A-VB)：理解一种未被充分研究的情绪表达方式链接：https://arxiv.org/abs/2210.15754
* 与cs.SD语音【39】为同一篇
作者：Alice Baird,Panagiotis Tzirakis,Jeffrey A. Brooks,Christopher B. Gregory,Björn Schuller,Anton Batliner,Dacher Keltner,Alan Cowen摘要：这是ACII情感爆发式发声研讨会和比赛（A-VB）的会议记录。A-VB是一个基于研讨会的挑战，它引入了理解声音爆发中的情感表达的问题--一个广泛的非语言发声，包括笑、咕噜声、喘息声等等。随着情感状态通知心理和身体健康，A-VB研讨会的核心焦点是更广泛地讨论当前的情感计算策略，以建模声乐情感表达。在A-VB挑战的第一次迭代中，参与者面临着四个以情绪为中心的子挑战，这些子挑战利用了大规模和“野生”的休谟-VB数据集。数据集和四个子挑战吸引了人们对情感科学新创新的关注，因为它涉及声音表达，解决了情感表达的低维和高维理论，文化差异，和'呼叫类型'（笑，哭，叹息等）。摘要：This is the Proceedings of the ACII Affective Vocal Bursts Workshop and Competition (A-VB). A-VB was a workshop-based challenge that introduces the problem of understanding emotional expression in vocal bursts -- a wide range of non-verbal vocalizations that includes laughs, grunts, gasps, and much more. With affective states informing both mental and physical wellbeing, the core focus of the A-VB workshop was the broader discussion of current strategies in affective computing for modeling vocal emotional expression. Within this first iteration of the A-VB challenge, the participants were presented with four emotion-focused sub-challenges that utilize the large-scale and `in-the-wild' Hume-VB dataset. The dataset and the four sub-challenges draw attention to new innovations in emotion science as it pertains to vocal expression, addressing low- and high-dimensional theories of emotional expression, cultural variation, and `call types' (laugh, cry, sigh, etc.).
【19】 Simulating realistic speech overlaps improves multi-talker ASR标题：模拟真实语音重叠提高多人ASR链接：https://arxiv.org/abs/2210.15715
* 与cs.SD语音【40】为同一篇
作者：Muqiao Yang,Naoyuki Kanda,Xiaofei Wang,Jian Wu,Sunit Sivasankaran,Zhuo Chen,Jinyu Li,Takuya Yoshioka机构： Carnegie Mellon University, Pittsburgh, PA, USA, Microsoft, Redmond, WA, USA备注：5 pages摘要：多说话者自动语音识别（ASR）已经被研究用于生成包括多个说话者的重叠语音的自然会话的转录。由于很难通过高质量的人工转录获得真实的会话数据，传统的模型训练方法是通过随机混合多个说话人的话语来进行朴素的多说话人语音模拟.在本文中，我们提出了一种改进的技术来模拟具有真实语音重叠的多说话者重叠语音，其中语音重叠的任意模式由离散令牌序列来表示。利用这种表示，可以基于诸如N元语法的统计语言模型从真实会话中学习语音重叠模式，然后可以使用该统计语言模型来生成用于训练的多说话者语音。在我们的实验中，使用所提出的方法训练的多说话者ASR模型在多个数据集上显示出一致的单词错误率改善。摘要：Multi-talker automatic speech recognition (ASR) has been studied to generate transcriptions of natural conversation including overlapping speech of multiple speakers. Due to the difficulty in acquiring real conversation data with high-quality human transcriptions, a na\"ive simulation of multi-talker speech by randomly mixing multiple utterances was conventionally used for model training. In this work, we propose an improved technique to simulate multi-talker overlapping speech with realistic speech overlaps, where an arbitrary pattern of speech overlaps is represented by a sequence of discrete tokens. With this representation, speech overlapping patterns can be learned from real conversations based on a statistical language model, such as N-gram, which can be then used to generate multi-talker speech for training. In our experiments, multi-talker ASR models trained with the proposed method show consistent improvement on the word error rates across multiple datasets.
【20】 Efficient Speech Translation with Dynamic Latent Perceivers标题：基于动态潜在感知器的高效语音翻译链接：https://arxiv.org/abs/2210.16264
* 与cs.SD语音【1】为同一篇
作者：Ioannis Tsiamas,Gerard I. Gállego,José A. R. Fonollosa,Marta R. Costa-jussá机构：Universitat Politecnica de Catalunya, Barcelona, Marta R. Costa-jussa, Meta AI, Paris摘要：近年来，Transformers已成为语音翻译的主流架构，在翻译质量方面取得了显著的改进。由于语音信号比其文本对应物长，并且由于Transformer的二次复杂性，下采样步骤对于其在语音翻译中的采用是必不可少的。相反，在本研究中，我们建议使用感知器编码器将语音输入映射为固定长度的潜在表示，以降低复杂度。此外，我们还引入了一种新的训练感知器的方法，即动态潜在访问（DLA），在不增加任何计算开销的情况下释放更大的潜在空间。采用DLA的语音到文本感知器可以在Must-C的三种语言对中与Transformer基线的性能相匹配。最后，经过DLA训练的模型在推理时容易适应DLA，并且可以灵活地部署在各种计算预算下，而不会显著降低翻译质量。摘要：Transformers have been the dominant architecture for Speech Translation in recent years, achieving significant improvements in translation quality. Since speech signals are longer than their textual counterparts, and due to the quadratic complexity of the Transformer, a down-sampling step is essential for its adoption in Speech Translation. Instead, in this research, we propose to ease the complexity by using a Perceiver encoder to map the speech inputs to a fixed-length latent representation. Furthermore, we introduce a novel way of training Perceivers, with Dynamic Latent Access (DLA), unlocking larger latent spaces without any additional computational overhead. Speech-to-Text Perceivers with DLA can match the performance of a Transformer baseline across three language pairs in MuST-C. Finally, a DLA-trained model is easily adaptable to DLA at inference, and can be flexibly deployed with various computational budgets, without significant drops in translation quality.
【21】 Universal speaker recognition encoders for different speech segments duration标题：适用于不同语音段时长的通用说话人识别编码器链接：https://arxiv.org/abs/2210.16231
* 与cs.SD语音【2】为同一篇
作者：Sergey Novoselov,Vladimir Volokhov,Galina Lavrentyeva机构：ITMO University, St.Petersburg, Russia, STC Ltd., St.Petersburg, Russia备注：Submitted to ICASSP'23摘要：创建对不同声学和语音持续时间条件鲁棒的通用扬声器编码器是当今的一大挑战。根据我们的观察，在短语音片段上训练的系统对于短语说话人确认是最优的，而在长片段上训练的系统对于长片段确认是优越的。同时在汇集的短和长语音段上训练的系统不能给出最佳的验证结果，并且通常对于短和长段都降级。针对不同语音段时长，提出了一种通用的说话人编码器。我们描述了我们的简单配方训练通用说话人编码器的任何类型的选择神经网络架构。根据我们针对NIST SRE和VoxCeleb 1基准测试获得的基于wav 2 vec-TDNN的系统的评估结果，所提出的通用编码器在不同的登记和测试语音段持续时间的情况下提供说话人确认改进。所提出的编码器的关键特征是它具有与所选择的神经网络结构相同的推理时间。摘要：Creating universal speaker encoders which are robust for different acoustic and speech duration conditions is a big challenge today. According to our observations systems trained on short speech segments are optimal for short phrase speaker verification and systems trained on long segments are superior for long segments verification. A system trained simultaneously on pooled short and long speech segments does not give optimal verification results and usually degrades both for short and long segments. This paper addresses the problem of creating universal speaker encoders for different speech segments duration. We describe our simple recipe for training universal speaker encoder for any type of selected neural network architecture. According to our evaluation results of wav2vec-TDNN based systems obtained for NIST SRE and VoxCeleb1 benchmarks the proposed universal encoder provides speaker verification improvements in case of different enrollment and test speech segment duration. The key feature of the proposed encoder is that it has the same inference time as the selected neural network architecture.
【22】 Supervised Contrastive Learning for Respiratory Sound Classification标题：用于呼吸音分类的有监督对比学习链接：https://arxiv.org/abs/2210.16192
* 与cs.SD语音【3】为同一篇
作者：Ilyass Moummad,Nicolas Farrugia机构：OSO-AI, Brest, France摘要：由于大的生物变异性、不平衡的数据集以及用于捕获呼吸信号的记录技术的多样性，使用机器学习的自动呼吸声分类是一项具有挑战性的任务。虽然已经提出了具有注释的呼吸循环的数据集，但是基于仅使用注释的监督学习的方法可能在它们的概括能力方面受到限制。在本研究中，我们使用监督对比学习来解决这一问题，依赖于呼吸循环注释和频谱图频率和时间掩蔽方法SpecAugment来生成增强样本，用于具有对比损失的表征学习。我们通过对一个从头训练的卷积神经网络进行实验，证明了这种方法优于监督学习，达到了新的技术水平。我们的工作显示了监督对比学习在不平衡和噪声环境中的潜力。我们的代码发布于https://github.com/ilyassmoummad/scl_icbhi2017摘要：Automatic respiratory sound classification using machine learning is a challenging task, due to large biological variability, imbalanced datasets, as well as a diversity in recording techniques used to capture the respiration signal. While datasets with annotated respiration cycles have been proposed, methods based on supervised learning using annotations only may be limited in their generalization capability. In this study, we address this issue using supervised contrastive learning, relying both on respiration cycle annotations and a spectrogram frequency and temporal masking method SpecAugment to generate augmented samples for representation learning with a contrastive loss. We demonstrate that such an approach can outperform supervised learning using experiments on a convolutional neural network trained from scratch, achieving the new state of the art. Our work shows the potential of supervised contrastive learning in imbalanced and noisy settings. Our code is released at https://github.com/ilyassmoummad/scl_icbhi2017
【23】 Deep Learning Object Detection Approaches to Source Identification标题：用于信源识别的深度学习对象检测方法链接：https://arxiv.org/abs/2210.16173
* 与cs.SD语音【4】为同一篇
作者：Luke Wood,Kevin Anderson,Peter Gerstoft机构： University of California, San Diego, La Jolla, CA, USA, Google摘要：传统的源识别是使用基于阈值的能量检测算法来解决的。这些算法经常对区域中的活动进行求和，并将高于特定活动阈值的区域视为源。虽然这些算法适用于大多数情况，但是它们经常不能检测占用小频带的信号，不能区分具有重叠频带的源，并且不能检测在指定信噪比下的任何信号。通过将原始信号数据转换为频谱图，可以将源识别问题归结为目标检测问题。通过利用基于深度学习的对象检测的现代进步，我们提出了一种系统，该系统设法减轻在使用传统源识别算法时遇到的失败情况。我们的贡献包括将源识别作为一个对象检测问题进行框架化、发布谱图对象检测数据集以及评估在该数据集上训练的RetinaNet和YOLOv5对象检测模型。我们的最终模型达到了平均精度高达0.906。由于具有如此高的平均值精度，这些模型对于在现实世界应用中的使用是足够稳健的。摘要：Traditionally source identification is solved using threshold based energy detection algorithms. These algorithms frequently sum up the activity in regions, and consider regions above a specific activity threshold to be sources. While these algorithms work for the majority of cases, they often fail to detect signals that occupy small frequency bands, fail to distinguish sources with overlapping frequency bands, and cannot detect any signals under a specified signal to noise ratio. Through the conversion of raw signal data to spectrogram, source identification can be framed as an object detection problem. By leveraging modern advancements in deep learning based object detection, we propose a system that manages to alleviate the failure cases encountered when using traditional source identification algorithms. Our contributions include framing source identification as an object detection problem, the publication of a spectrogram object detection dataset, and evaluation of the RetinaNet and YOLOv5 object detection models trained on the dataset. Our final models achieve Mean Average Precisions of up to 0.906. With such a high Mean Average Precision, these models are sufficiently robust for use in real world applications.
【24】 Towards zero-shot Text-based voice editing using acoustic context conditioning, utterance embeddings, and reference encoders标题：使用声学上下文调节、话语嵌入和参考编码器的基于Zero-Shot文本的语音编辑链接：https://arxiv.org/abs/2210.16045
* 与cs.SD语音【5】为同一篇
作者：Jason Fong,Yun Wang,Prabhav Agrawal,Vimal Manohar,Jilong Wu,Thilo Köhler,Qing He机构：University of Edinburgh,Meta AI备注：Submitted to ICASSP 2023摘要：基于文本的语音编辑（TBVE）使用文本到语音（TTS）系统的合成输出来替换原始录音中的单词。最近的工作已经使用神经模型来产生在清晰度、说话者身份和韵律方面类似于原始语音的编辑语音。然而，现有工作的一个限制是使用微调来优化性能：这需要对来自目标发音人的数据进行进一步的模型训练，这是一个可能将潜在的敏感数据合并到服务器端模型中的昂贵过程。相比之下，本研究的重点是zero-shot方法，该方法完全避免了微调，而是使用预先训练的说话人验证嵌入和联合训练的参考编码器来编码话语级信息，这有助于捕获说话人身份和韵律等方面。主观听力测试发现，在zero-shot环境下，话语嵌入和参考编码器都能改善编辑后的合成语音和未编辑原始录音之间的说话人身份和韵律的连续性。摘要：Text-based voice editing (TBVE) uses synthetic output from text-to-speech (TTS) systems to replace words in an original recording. Recent work has used neural models to produce edited speech that is similar to the original speech in terms of clarity, speaker identity, and prosody. However, one limitation of prior work is the usage of finetuning to optimise performance: this requires further model training on data from the target speaker, which is a costly process that may incorporate potentially sensitive data into server-side models. In contrast, this work focuses on the zero-shot approach which avoids finetuning altogether, and instead uses pretrained speaker verification embeddings together with a jointly trained reference encoder to encode utterance-level information that helps capture aspects such as speaker identity and prosody. Subjective listening tests find that both utterance embeddings and a reference encoder improve the continuity of speaker identity and prosody between the edited synthetic speech and unedited original recording in the zero-shot setting.
【25】 Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Models标题：从预先训练的自监督语音模型分析声学单词嵌入链接：https://arxiv.org/abs/2210.16043
* 与cs.SD语音【6】为同一篇
作者：Ramon Sanabria,Hao Tang,Sharon Goldwater机构：The University of Edinburgh备注：Submitted to IEEE ICASSP 2023摘要：鉴于自监督模型在各种任务上的强有力的结果，令人惊讶的是，很少有研究探索用于声学单词嵌入（AWE）的自监督表示，固定维向量表示可变长度的口语单词片段。在这项工作中，我们研究了几个预先训练的模型和池方法来构造具有自监督表示的AWE。由于自监督表示的上下文化性质，我们假设简单的池化方法，如平均，可能已经对构造AWE有用。在一个标准的单词辨别任务上，我们发现HuBERT表征与平均池可以与英语AWE的最新水平相媲美。更令人惊讶的是，尽管仅在英语上训练，但在Xitsonga、普通话和法语上评估的HuBERT表示始终优于多语言模型XLSR-53（以及在英语上训练的Wav 2 Vec 2.0）。摘要：Given the strong results of self-supervised models on various tasks, there have been surprisingly few studies exploring self-supervised representations for acoustic word embeddings (AWE), fixed-dimensional vectors representing variable-length spoken word segments. In this work, we study several pre-trained models and pooling methods for constructing AWEs with self-supervised representations. Owing to the contextualized nature of self-supervised representations, we hypothesize that simple pooling methods, such as averaging, might already be useful for constructing AWEs. When evaluating on a standard word discrimination task, we find that HuBERT representations with mean-pooling rival the state of the art on English AWEs. More surprisingly, despite being trained only on English, HuBERT representations evaluated on Xitsonga, Mandarin, and French consistently outperform the multilingual model XLSR-53 (as well as Wav2Vec 2.0 trained on English).
【26】 Assessing Phrase Break of ESL speech with Pre-trained Language Models标题：基于预训练语言模型的ESL语音断词评估链接：https://arxiv.org/abs/2210.16029
* 与cs.SD语音【7】为同一篇
作者：Zhiyi Wang,Shaoguang Mao,Wenshan Wu,Yan Xia机构：Microsoft Research Asia备注：Under Review, ICASSP 2023摘要：本文介绍了一种使用预先训练的语言模型（PLM）来评估ESL学习者语音中短语中断的方法。与传统的方法不同，该方法将语音转换为令牌序列，然后利用PLM的能力进行语音识别。有两个子任务：对语音剪辑短语中断的总体评估;对每个可能的短语中断位置进行精细的评估。语音输入首先与文本强制对齐，然后被预处理为包括单词和相关联的短语中断信息的令牌序列。然后，将令牌序列馈送到预训练和微调流水线中。在预训练中，用标记数据训练被替换的中断标记检测模块，其中每个标记具有一定百分比的被随机替换的机会。在细调阶段，分别利用文本分类和序列标注流水线优化总体评分和细粒度评分。随着PLM的引入，对已标记训练数据的依赖性已大大降低，并且性能已得到改善。摘要：This work introduces an approach to assessing phrase break in ESL learners' speech with pre-trained language models (PLMs). Different with traditional methods, this proposal converts speech to token sequences, and then leverages the power of PLMs. There are two sub-tasks: overall assessment of phrase break for a speech clip; fine-grained assessment of every possible phrase break position. Speech input is first force-aligned with texts, then pre-processed to a token sequence, including words and associated phrase break information. The token sequence is then fed into the pre-training and fine-tuning pipeline. In pre-training, a replaced break token detection module is trained with token data where each token has a certain percentage chance to be randomly replaced. In fine-tuning, overall and fine-grained scoring are optimized with text classification and sequence labeling pipeline, respectively. With the introduction of PLMs, the dependence on labeled training data has been greatly reduced, and performance has improved.
【27】 Spectrograms Are Sequences of Patches标题：光谱图是补丁的序列链接：https://arxiv.org/abs/2210.15988
* 与cs.SD语音【8】为同一篇
作者：Leyi Zhao,Yi Li机构：Sichuan University摘要：自监督预训练模型已经成功地应用于多个机器学习领域。然而，只有极少量的作品与音乐有关。在我们的工作中，我们将音乐的声谱图视为一系列的补丁，并设计了一个自监督模型来捕获这些连续补丁的特征：Patchifier算法充分利用了NLP和CV域的自监督学习方法。我们不使用标记数据用于预训练过程，仅使用包含16k个音乐片段的MTAT数据集的子集。在预训练之后，我们将该模型应用于几个下游任务。与其他音频表示模型相比，我们的模型获得了相当可接受的结果。同时，我们的工作表明，将音频视为一系列补丁片段是有意义的。摘要：Self-supervised pre-training models have been used successfully in several machine learning domains. However, only a tiny amount of work is related to music. In our work, we treat a spectrogram of music as a series of patches and design a self-supervised model that captures the features of these sequential patches: Patchifier, which makes good use of self-supervised learning methods from both NLP and CV domains. We do not use labeled data for the pre-training process, only a subset of the MTAT dataset containing 16k music clips. After pre-training, we apply the model to several downstream tasks. Our model achieves a considerably acceptable result compared to other audio representation models. Meanwhile, our work demonstrates that it makes sense to consider audio as a series of patch segments.
【28】 Determining Ratio of Prunable Channels in MobileNet by Sparsity for Acoustic Scene Classification标题：利用稀疏度确定MobileNet中可剪缩通道的比例进行声场分类链接：https://arxiv.org/abs/2210.15960
* 与cs.SD语音【9】为同一篇
作者：Yiqiang Cai,Shengchen Li机构：School of Advanced Technology, Xi’an Jiaotong-Liverpool University, Ren’ai Road, Suzhou, China备注：5 pages, 2 figures, submitted to ICASSP 2023摘要：移动网络广泛应用于嵌入式系统中的声学场景分类。现有的ASC算法通过修剪部分分量来降低算法的复杂度，如修剪卷积层的信道。在实践中，被修剪的信道的最大比例（其被定义为可修剪信道的比率（$R_\textit{PC}$））通常根据经验来确定。本文提出了一种通过与卷积层中的信道稀疏度（$S_C$）相关的简单线性回归模型来确定$R_\textit{PC}$的方法。在实验中，通过移除不活动通道直到达到性能降低的拐点来检查$R_\textit{PC}$。提出了计算训练模型的$S_C$和结果$R_\textit{PC}$的简单方法。实验结果表明：（1）$R_\textit{PC}$的决策与$S_C$线性相关，超参数对这种关系的影响很小; 2）移动网络对该方法具有较高的敏感性和稳定性。摘要：MobileNet is widely used for Acoustic Scene Classification (ASC) in embedded systems. Existing works reduce the complexity of ASC algorithms by pruning some components, e.g. pruning channels in the convolutional layer. In practice, the maximum proportion of channels being pruned, which is defined as Ratio of Prunable Channels ($R_\textit{PC}$), is often decided empirically. This paper proposes a method that determines the $R_\textit{PC}$ by simple linear regression models related to the Sparsity of Channels ($S_C$) in the convolutional layers. In the experiment, $R_\textit{PC}$ is examined by removing inactive channels until reaching a knee point of performance decrease. Simple methods for calculating the $S_C$ of trained models and resulted $R_\textit{PC}$ are proposed. The experiment results demonstrate that 1) the decision of $R_\textit{PC}$ is linearly dependent on $S_C$ and the hyper-parameters have a little impact on the relationship; 2) MobileNet shows a high sensitivity and stability on proposed method.
【29】 On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis标题：关于使用特定于通道的大规模预训练编码器进行多通道情感分析链接：https://arxiv.org/abs/2210.15937
* 与cs.SD语音【10】为同一篇
作者：Atsushi Ando,Ryo Masumura,Akihiko Takashima,Satoshi Suzuki,Naoki Makishima,Keita Suzuki,Takafumi Moriya,Takanori Ashihara,Hiroshi Sato机构：NTT Corporation备注：Accepted to SLT 2022摘要：本文研究了用于多模态情感分析（MSA）的特定模态的大规模预训练编码器的有效性和实现。虽然已经报道了预训练编码器在各个领域中的有效性，但是传统的MSA方法仅将它们用于语言模态，并且它们的应用还没有被研究。本文将大规模预训练编码器产生的特征与传统启发式特征进行了比较。每种模态均使用一个最大的预训练编码器; CLIP-ViT、WavLM和BERT分别用于视觉、听觉和语言模态。在两个数据集上的实验表明，在单峰和多峰场景下，使用特定领域的预训练编码器的方法都比使用传统特征的方法获得了更好的性能。我们还发现，使用编码器的中间层的输出比使用输出层的输出更好。这些代码可在www.example.com上找到https://github.com/ando-hub/MSA_Pretrain。摘要：This paper investigates the effectiveness and implementation of modality-specific large-scale pre-trained encoders for multimodal sentiment analysis~(MSA). Although the effectiveness of pre-trained encoders in various fields has been reported, conventional MSA methods employ them for only linguistic modality, and their application has not been investigated. This paper compares the features yielded by large-scale pre-trained encoders with conventional heuristic features. One each of the largest pre-trained encoders publicly available for each modality are used; CLIP-ViT, WavLM, and BERT for visual, acoustic, and linguistic modalities, respectively. Experiments on two datasets reveal that methods with domain-specific pre-trained encoders attain better performance than those with conventional features in both unimodal and multimodal scenarios. We also find it better to use the outputs of the intermediate layers of the encoders than those of the output layer. The codes are available at https://github.com/ando-hub/MSA_Pretrain.
【30】 A comprehensive study on self-supervised distillation for speaker representation learning标题：说话人表征学习的自监督精馏综合研究链接：https://arxiv.org/abs/2210.15936
* 与cs.SD语音【11】为同一篇
作者：Zhengyang Chen,Yao Qian,Bing Han,Yanmin Qian,Michael Zeng机构： MoE Key Lab of Artificial Intelligence, AI Institute , X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Microsoft Cognitive Services Research, USA备注：Accepted by SLT2022摘要：在实际应用场景中，由于说话人隐私问题，获取大量标记数据进行说话人表示学习往往具有挑战性。无标签自监督学习是解决这一问题的一种有效方法，与对比学习相比，自提取方法在损失函数中只使用正样本，因而更具吸引力。本文对自提取自监督说话人表征学习进行了全面的研究，特别是对关键数据扩充的研究。我们提出的音频扰动增强策略将说话人表示的性能推向了一个新的极限。实验结果表明，该模型能够在Voxceleb 1说话人确认评测基准上实现一种新的SoTA，对于试验Vox 1-O、Vox 1-E和Vox 1-H，等错误率（EER）分别为2.505%、2.473%和4.791%），丢弃训练阶段中的任何说话人标记。摘要：In real application scenarios, it is often challenging to obtain a large amount of labeled data for speaker representation learning due to speaker privacy concerns. Self-supervised learning with no labels has become a more and more promising way to solve it. Compared with contrastive learning, self-distilled approaches use only positive samples in the loss function and thus are more attractive. In this paper, we present a comprehensive study on self-distilled self-supervised speaker representation learning, especially on critical data augmentation. Our proposed strategy of audio perturbation augmentation has pushed the performance of the speaker representation to a new limit. The experimental results show that our model can achieve a new SoTA on Voxceleb1 speaker verification evaluation benchmark ( i.e., equal error rate (EER) 2.505%, 2.473%, and 4.791% for trial Vox1-O, Vox1-E and Vox1-H , respectively), discarding any speaker labels in the training phase.
【31】 Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation标题：用于Few-Shot文语转换说话人自适应的残差适配器链接：https://arxiv.org/abs/2210.15868
* 与cs.SD语音【12】为同一篇
作者：Nobuyuki Morioka,Heiga Zen,Nanxin Chen,Yu Zhang,Yifan Ding机构： Google Research, Japan, Google Research, USA备注：Submitted to ICASSP 2023摘要：使神经文本到语音（TTS）模型适应于目标说话者通常涉及微调预训练的多说话者骨干模型的大多数（如果不是全部的话）参数。然而，服务数百个微调的神经TTS模型是昂贵的，因为它们中的每一个都需要显著的占用空间和单独的计算资源（例如，加速器、存储器）。为了将说话人自适应神经TTS语音扩展到数百个说话人，同时保持说话人的自然度和相似度，提出了一种参数有效的Few-Shot说话人自适应算法。该架构允许在不同的目标说话者之间共享骨干模型。实验结果表明，与完全微调的方法相比，该方法能够获得较好的自然度和说话人相似度，而对每个说话人的骨架模型参数的要求仅为0. 1%。摘要：Adapting a neural text-to-speech (TTS) model to a target speaker typically involves fine-tuning most if not all of the parameters of a pretrained multi-speaker backbone model. However, serving hundreds of fine-tuned neural TTS models is expensive as each of them requires significant footprint and separate computational resources (e.g., accelerators, memory). To scale speaker adapted neural TTS voices to hundreds of speakers while preserving the naturalness and speaker similarity, this paper proposes a parameter-efficient few-shot speaker adaptation, where the backbone model is augmented with trainable lightweight modules called residual adapters. This architecture allows the backbone model to be shared across different target speakers. Experimental results show that the proposed approach can achieve competitive naturalness and speaker similarity compared to the full fine-tuning approaches, while requiring only $\sim$0.1% of the backbone model parameters for each speaker.
【32】 Speech Enhancement with Intelligent Neural Homomorphic Synthesis标题：基于智能神经同态合成的语音增强链接：https://arxiv.org/abs/2210.15853
* 与cs.SD语音【13】为同一篇
作者：Shulin He,Wei Rao,Jinjiang Liu,Jun Chen,Yukai Ju,Xueliang Zhang,Yannan Wang,Shidong Shang机构：College of Computer Science, Inner Mongolia University, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China备注：Submitted to ICASSP 2023摘要：大多数神经网络语音增强模型通过直接映射傅里叶变换频谱或波形而忽略语音产生数学模型。在本研究中，我们提出一种神经源滤波器网路来进行语音增强。具体地说，我们利用同态信号处理和倒谱分析来获得含噪语音的激励和声道。与传统的信号处理方法不同，我们使用注意递归网络（ARN）模型预测比率掩模来代替提升分离函数。然后，利用两个卷积注意递归网络（CARN）分别对干净语音的激励和声道进行预测。该系统的输出是从估计的激励和人声合成的。实验结果表明，该方法比FullSubNet算法的SI-SNR提高了1.363dB。摘要：Most neural network speech enhancement models ignore speech production mathematical models by directly mapping Fourier transform spectrums or waveforms. In this work, we propose a neural source filter network for speech enhancement. Specifically, we use homomorphic signal processing and cepstral analysis to obtain noisy speech's excitation and vocal tract. Unlike traditional signal processing, we use an attentive recurrent network (ARN) model predicted ratio mask to replace the liftering separation function. Then two convolutional attentive recurrent network (CARN) networks are used to predict the excitation and vocal tract of clean speech, respectively. The system's output is synthesized from the estimated excitation and vocal. Experiments prove that our proposed method performs better, with SI-SNR improving by 1.363dB compared to FullSubNet.
【33】 Local-global speaker representation for target speaker extraction标题：用于目标说话人提取的局部-全局说话人表示链接：https://arxiv.org/abs/2210.15849
* 与cs.SD语音【14】为同一篇
作者：Shulin He,Wei Rao,Kanghao Zhang,Yukai Ju,Yang Yang,Xueliang Zhang,Yannan Wang,Shidong Shang机构：College of Computer Science, Inner Mongolia University, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China备注：Submitted to ICASSP 2023摘要：目标说话人提取是根据给定的注册话语从混合信号中提取目标说话人的语音。目标说话人的注册语音也被称为锚定语音。锚定语音的有效利用是说话人提取的关键。在本研究中，我们提出了一个新的系统，以充分利用说话人信息从锚定语音。与仅使用锚点的局部或全局特征的模型不同，所提出的方法在全局和局部水平上提取说话人信息，并将特征馈送到语音分离网络。该方法充分利用了全局特征和局部特征的优势互补，提高了说话人提取的性能。我们使用多个说话人提取模型验证了这种局部-全局表示（LGR）方法的可行性。在开源数据库Libri-2 talker上进行了系统实验，实验结果表明，该方法的性能明显优于基线模型.摘要：Target speaker extraction is to extract the target speaker's voice from a mixture of signals according to the given enrollment utterance. The target speaker's enrollment utterance is also called as anchor speech. The effective utilization of anchor speech is crucial for speaker extraction. In this study, we propose a new system to exploit speaker information from anchor speech fully. Unlike models that use only local or global features of the anchor, the proposed method extracts speaker information on global and local levels and feeds the features into a speech separation network. Our approach benefits from the complementary advantages of both global and local features, and the performance of speaker extraction is improved. We verified the feasibility of this local-global representation (LGR) method using multiple speaker extraction models. Systematic experiments were conducted on the open-source dataset Libri-2talker, and the results showed that the proposed method significantly outperformed the baseline models.
【34】 GM-TCNet: Gated Multi-scale Temporal Convolutional Network using Emotion Causality for Speech Emotion Recognition标题：GM-TCNet：基于情感因果关系的门控多尺度时间卷积网络语音情感识别链接：https://arxiv.org/abs/2210.15834
* 与cs.SD语音【15】为同一篇
作者：Jia-Xin Ye,Xin-Cheng Wen,Xuan-Ze Wang,Yong Xu,Yan Luo,Chang-Li Wu,Li-Yan Chen,Kun-Hong Liu机构：School of Informatics, Xiamen University, Xiamen, China, Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China, Department of Computer Science, Harbin Institute of Technology (Shenzhen), Shenzhen, China备注：The source code is available at: this https URL摘要：在人机交互中，语音情感识别（SER）对于理解用户意图、改善交互体验起着至关重要的作用。相似的情感语音虽然具有不同的说话人特征，但却具有共同的前因和后果，如何通过语音情感之间的因果关系产生鲁棒的、有区别的表征是SER面临的一个重要挑战。本文提出了一种门控多尺度时间卷积网络（GM-TCNet），用于构造一种新的具有多尺度感受野的情绪因果关系表征学习组件。GM-TCNet通过扩展因果卷积层和门控机制构建了一个新颖的情感因果表示学习组件，用于捕捉情感在时域上的动态变化。此外，该算法还利用跳接融合不同选通卷积块的高级特征，捕捉语音中丰富而细微的情感变化。GM-TCNet首先使用单一类型的特征，即Mel频率倒谱系数作为输入，然后将其通过门控时间卷积模块以生成高级特征。最后，将这些特征送入情感分类器以完成SER任务。实验结果表明，与现有技术相比，该模型在大多数情况下保持了最高的性能。摘要：In human-computer interaction, Speech Emotion Recognition (SER) plays an essential role in understanding the user's intent and improving the interactive experience. While similar sentimental speeches own diverse speaker characteristics but share common antecedents and consequences, an essential challenge for SER is how to produce robust and discriminative representations through causality between speech emotions. In this paper, we propose a Gated Multi-scale Temporal Convolutional Network (GM-TCNet) to construct a novel emotional causality representation learning component with a multi-scale receptive field. GM-TCNet deploys a novel emotional causality representation learning component to capture the dynamics of emotion across the time domain, constructed with dilated causal convolution layer and gating mechanism. Besides, it utilizes skip connection fusing high-level features from different gated convolution blocks to capture abundant and subtle emotion changes in human speech. GM-TCNet first uses a single type of feature, mel-frequency cepstral coefficients, as inputs and then passes them through the gated temporal convolutional module to generate the high-level features. Finally, the features are fed to the emotion classifier to accomplish the SER task. The experimental results show that our model maintains the highest performance in most cases compared to state-of-the-art techniques.
【35】 On the Role of Visual Context in Enriching Music Representations标题：论视觉语境在丰富音乐表现形式中的作用链接：https://arxiv.org/abs/2210.15828
* 与cs.SD语音【16】为同一篇
作者：Kleanthis Avramidis,Shanti Stewart,Shrikanth Narayanan机构：Signal Analysis and Interpretation Lab, University of Southern California, Los Angeles, CA备注：5 pages, 4 figures, 1 table摘要：人类 对 音乐 的 感知 和 体验 是 高度 依赖 于 语境 的 。背景 的 变化 导致 了 我们 对 音乐 的 理解 和 与 音乐 互动 的 方式 的 差异 ， 这 对 信息 检索 的 鲁 棒 模型 的 设计 提出 了 挑战 。结合 来自 不同 来源 的 多 模态 背景 提供 了 一 种 对 这种 可变 性 建模 的 有 前途 的 方法 。在 诸如 电影 和 音乐 视频 之类 的 媒体 中 呈现 的 音乐 提供 了 丰富 的 多 模态 上下文 ， 其 调节 潜在 的 人类 体验 。然而 ， 这样 的 上下文 建模 未 被 充分 开发 ， 因为 它 需要 大量 的 多 模态 数据 以及 相关 注释 。自 监督 学习 可以 通过 自动 提取 不同 模态 之间 的 丰富 、 高 级别 对应 关系 来 帮助 解决 这些 挑战 ， 从而 减轻 对 大 规模 细 粒度 注释 的 需求 。在 本 研究 中 ， 我们 提出 了 VCMR - - 视频 条件 音乐 表征 ， 一 个 从 音频 和 伴随 的 音乐 视频 中 学习 音乐 表征 的 对比 学习 框架 。上下文 视觉 信息 增强 了 音乐 音频 的 表示 ， 如 在 音乐 标记 的 下游 任务 上 评估 的 。实验 结果 表明 ， 提出 的 框架 能够 为 音频 表示 提供 附加 的 鲁 棒 性 ， 并 指示 音乐 元素 在 多大 程度 上 受 视觉 上下文 的 影响 或 决定 。摘要：Human perception and experience of music is highly context-dependent. Contextual variability contributes to differences in how we interpret and interact with music, challenging the design of robust models for information retrieval. Incorporating multimodal context from diverse sources provides a promising approach toward modeling this variability. Music presented in media such as movies and music videos provide rich multimodal context that modulates underlying human experiences. However, such context modeling is underexplored, as it requires large amounts of multimodal data along with relevant annotations. Self-supervised learning can help address these challenges by automatically extracting rich, high-level correspondences between different modalities, hence alleviating the need for fine-grained annotations at scale. In this study, we propose VCMR -- Video-Conditioned Music Representations, a contrastive learning framework that learns music representations from audio and the accompanying music videos. The contextual visual information enhances representations of music audio, as evaluated on the downstream task of music tagging. Experimental results show that the proposed framework can contribute additive robustness to audio representations and indicates to what extent musical elements are affected or determined by visual context.
【36】 Evaluating context-invariance in unsupervised speech representations标题：评估无监督语音表示中的上下文不变性链接：https://arxiv.org/abs/2210.15775
* 与cs.SD语音【17】为同一篇
作者：Mark Hallap,Emmanuel Dupoux,Ewan Dunbar机构： University of Toronto, CoML, ENSCNRSEHESSINRIAPSL Research University备注：Submitted to ICASSP 2023摘要：无监督语音表示已经开始起步，基准测试（SUPERB，ZeroSpeech）证明了在半监督语音识别、语音合成和纯语音语言建模方面的重大进展。灵感来自于“发现一种语言的音素”或类似的低比特率编码的承诺。然而，音素转录的关键性质之一是上下文不变性：在文本保持稳定的同时，语音的语音上下文可以对它的发音方式具有巨大的影响。这就是为什么同一个单词的标记可以有相同的转录--这是语言理解的关键。当前的基准测试无法衡量上下文不变性。我们开发了一个新版本的ZeroSpeech ABX基准测试来测量上下文不变性，并将其应用于最近的自监督表示。我们证明了表征的上下文无关性预示着词级表征的稳定性。我们建议研究集中在提高自监督和无监督表示的上下文无关性。摘要：Unsupervised speech representations have taken off, with benchmarks (SUPERB, ZeroSpeech) demonstrating major progress on semi-supervised speech recognition, speech synthesis, and speech-only language modelling. Inspiration comes from the promise of ``discovering the phonemes'' of a language or a similar low-bitrate encoding. However, one of the critical properties of phoneme transcriptions is context-invariance: the phonetic context of a speech sound can have massive influence on the way it is pronounced, while the text remains stable. This is what allows tokens of the same word to have the same transcriptions -- key to language understanding. Current benchmarks do not measure context-invariance. We develop a new version of the ZeroSpeech ABX benchmark that measures context-invariance, and apply it to recent self-supervised representations. We demonstrate that the context-independence of representations is predictive of the stability of word-level representations. We suggest research concentrate on improving context-independence of self-supervised and unsupervised representations.
【37】 Self-supervised language learning from raw audio: Lessons from the Zero Resource Speech Challenge标题：从原始音频中进行自我监督的语言学习：来自零资源语音挑战的经验链接：https://arxiv.org/abs/2210.15759
* 与cs.SD语音【18】为同一篇
作者：Ewan Dunbar,Nicolas Hamilakis,Emmanuel Dupoux备注：None摘要：自监督或无监督机器学习的最新进展开启了从原始音频构建完整语音处理系统而不使用任何文本表示或专家标签（例如音素、字典或解析树）的可能性。自2015年以来，零资源语音挑战系列的贡献是将这一长期目标分解为四个明确定义的任务--声学单元发现、口语术语发现、离散再合成和口语建模--并引入相关的指标和基准，实现模型比较和累积进展。我们将概述自2015年以来的六个版本的挑战系列，讨论所吸取的经验教训，并概述需要进一步工作或给出令人困惑的结果的领域。摘要：Recent progress in self-supervised or unsupervised machine learning has opened the possibility of building a full speech processing system from raw audio without using any textual representations or expert labels such as phonemes, dictionaries or parse trees. The contribution of the Zero Resource Speech Challenge series since 2015 has been to break down this long-term objective into four well-defined tasks -- Acoustic Unit Discovery, Spoken Term Discovery, Discrete Resynthesis, and Spoken Language Modeling -- and introduce associated metrics and benchmarks enabling model comparison and cumulative progress. We present an overview of the six editions of this challenge series since 2015, discuss the lessons learned, and outline the areas which need more work or give puzzling results.
【38】 One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hall标题：音频信号的一次声学匹配--学会在任何房间/音乐厅听音乐链接：https://arxiv.org/abs/2210.15750
* 与cs.SD语音【19】为同一篇
作者：Prateek Verma,Chris Chafe,Jonathan Berger机构：Stanford University备注：5 pages, 1 figure摘要：产生和听到声音的声学空间通过提供一种独特的\textit{存在}感，在声音的感知方式中发挥着重要作用。我们听到的每一个声音都是由声源固有的连续卷积运算和外部因素（如麦克风特性和房间脉冲响应）产生的。通常，研究人员使用诸如手枪射击或气球爆裂之类的激励作为脉冲信号，利用该脉冲信号可以创建可听化。与感兴趣的信号卷积的房间“脉冲”响应可以将输入声音变换成在感兴趣的声学空间中播放的声音。在此，我们提出了一种新颖的架构，该架构可以通过使用作为气球爆裂声的代理记录的任意音频来将感兴趣的声音变换到感兴趣的任何其他声学空间（房间或大厅）。该架构基于简单的信号处理思想，以从学习的声学签名和输入信号学习残差信号。我们的框架允许神经网络调整时频表示中每一点的增益，给出可靠的定性和定量结果。摘要：The acoustic space in which a sound is created and heard plays an essential role in how that sound is perceived by affording a unique sense of \textit{presence}. Every sound we hear results from successive convolution operations intrinsic to the sound source and external factors such as microphone characteristics and room impulse responses. Typically, researchers use an excitation such as a pistol shot or balloon pop as an impulse signal with which an auralization can be created. The room "impulse" responses convolved with the signal of interest can transform the input sound into the sound played in the acoustic space of interest. Here we propose a novel architecture that can transform a sound of interest into any other acoustic space(room or hall) of interest by using arbitrary audio recorded as a proxy for a balloon pop. The architecture is grounded in simple signal processing ideas to learn residual signals from a learned acoustic signature and the input signal. Our framework allows a neural network to adjust gains of every point in the time-frequency representation, giving sound qualitative and quantitative results.
【39】 Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models标题：使用组合端到端模型进行口语理解的令牌级序列标注链接：https://arxiv.org/abs/2210.15734
* 与cs.SD语音【20】为同一篇
作者：Siddhant Arora,Siddharth Dalmia,Brian Yan,Florian Metze,Alan W Black,Shinji Watanabe机构：Language Technologies Institute, Carnegie Mellon University, USA备注：Accepted at EMNLP 2022 Findings. Our code and models will be publicly available as part of the ESPnet-SLU toolkit: this https URL and the release can be followed here: this https URL摘要：端到端口语理解（SLU）系统由于其简单性和避免错误传播的能力而比级联方法更受欢迎。然而，这些系统将序列标记建模为序列预测任务，导致与其良好建立的标记水平标记公式的分歧。我们构建了组合式端到端SLU系统，明确地将SLU中识别口语提及的复杂性与序列标记的NLU任务分离。通过依赖于为ASR训练的中间解码器，我们的端到端系统将输入模态从语音转换为可以在传统序列标记框架中使用的令牌级表示。在我们的端到端SLU系统中，ASR和NLU公式的这种组合提供了与预先训练的ASR和NLU系统的直接兼容性，允许对单个组件进行性能监测，并允许使用像CRF这样的全局归一化损耗，使其在实际场景中具有吸引力。在SLU基准测试中，我们的模型在命名实体识别的标注任务上优于级联模型和直接端到端模型。摘要：End-to-end spoken language understanding (SLU) systems are gaining popularity over cascaded approaches due to their simplicity and ability to avoid error propagation. However, these systems model sequence labeling as a sequence prediction task causing a divergence from its well-established token-level tagging formulation. We build compositional end-to-end SLU systems that explicitly separate the added complexity of recognizing spoken mentions in SLU from the NLU task of sequence labeling. By relying on intermediate decoders trained for ASR, our end-to-end systems transform the input modality from speech to token-level representations that can be used in the traditional sequence labeling framework. This composition of ASR and NLU formulations in our end-to-end SLU system offers direct compatibility with pre-trained ASR and NLU systems, allows performance monitoring of individual components and enables the use of globally normalized losses like CRF, making them attractive in practical scenarios. Our models outperform both cascaded and direct end-to-end models on a labeling task of named entity recognition across SLU benchmarks.
【40】 FedAudio: A Federated Learning Benchmark for Audio Tasks标题：FedAudio：一种用于音频任务的联合学习基准链接：https://arxiv.org/abs/2210.15707
* 与cs.SD语音【21】为同一篇
作者：Tuo Zhang,Tiantian Feng,Samiul Alam,Sunwoo Lee,Mi Zhang,Shrikanth S. Narayanan,Salman Avestimehr机构：Department of Electrical and Computer Engineering, University of Southern California, Michigan State University, Department of Computer Engineering, Inha University, Department of Computer Science and Engineering, The Ohio State University摘要：近年来，由于与持续从用户收集数据的消费设备的普遍性相关的数据隐私问题，联合学习（FL）获得了大量关注。虽然已经开发了一些外语基准来促进外语研究，但没有一个包括音频数据和音频相关的任务。在本文中，我们通过引入一个新的音频任务FL基准来填补这一关键空白，我们称之为FedAudio。FedAudio包括来自三个重要音频任务的四个代表性和常用的音频数据集，这些音频任务与FL用例非常一致。特别是，FedAudio的一个独特贡献是在数据集中引入了数据噪声和标签错误，以模拟在现实环境中部署FL系统时的挑战。FedAudio还包括数据集的基准测试结果和PyTorch库，目的是方便研究人员公平地比较他们的算法。我们希望FedAudio能够作为一个催化剂，激发新的外语研究的音频任务，从而有利于声学和语音研究社区。数据集和性能指标评测结果可访问https://github.com/zhang-tuo pdf/FedAudio。摘要：Federated learning (FL) has gained substantial attention in recent years due to the data privacy concerns related to the pervasiveness of consumer devices that continuously collect data from users. While a number of FL benchmarks have been developed to facilitate FL research, none of them include audio data and audio-related tasks. In this paper, we fill this critical gap by introducing a new FL benchmark for audio tasks which we refer to as FedAudio. FedAudio includes four representative and commonly used audio datasets from three important audio tasks that are well aligned with FL use cases. In particular, a unique contribution of FedAudio is the introduction of data noises and label errors to the datasets to emulate challenges when deploying FL systems in real-world settings. FedAudio also includes the benchmark results of the datasets and a PyTorch library with the objective of facilitating researchers to fairly compare their algorithms. We hope FedAudio could act as a catalyst to inspire new FL research for audio tasks and thus benefit the acoustic and speech research community. The datasets and benchmark results can be accessed at https://github.com/zhang-tuo pdf/FedAudio.
机器翻译，仅供参考
每日语音速递