今日论文合集：cs.SD语音23篇，eess.AS音频处理25篇。
【1】 MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets标题：MT4SSL：通过集成多个目标来促进自监督语音表示学习链接：https://arxiv.org/abs/2211.07321
作者：Ziyang Ma,Zhisheng Zhen,Changli Tang,Yujin Wang,Xie Chen机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Department of Electronic Engineering, Tsinghua University备注：Submitted to ICASSP 2023摘要：本文从自训练目标的获取入手，为自监督语音模型的研究提供了一个新的视角。我们将目标提取器推广为离线目标提取器（Off-TE）和在线目标提取器（On-TE），而不考虑具体的借口任务。在此基础上，提出了一种新的多任务自监督学习框架MT4 SSL，即通过集成多个目标来提升自监督语音表示学习。MT4 SSL是指两个典型的模型，HuBERT和data 2 vec，它们分别使用K-means算法作为Off-TE和使用无梯度的教师网络作为On-TE。我们的模型在LibriSpeech基准测试中的性能超过了以前的SSL方法，并且在不需要那么多数据的情况下，与性能最好的模型相当，甚至更好。此外，我们发现使用Off-TE和On-TE两者导致在预训练阶段更好的收敛。从有效性和效率两方面考虑，我们认为在我们的视角下对自监督语音模型进行多任务学习是一个很有前途的趋势。摘要：In this paper, we provide a new perspective on self-supervised speech models from how the self-training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor (On-TE), without caring about specific pretext tasks. Based on this, we propose a new multi-tasking learning framework for self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets. MT4SSL refers to two typical models, HuBERT and data2vec, which use the K-means algorithm as an Off-TE and a teacher network without gradients as an On-TE, respectively. Our model outperforms previous SSL methods by nontrivial margins on the LibriSpeech benchmark, and is comparable to or even better than the best-performing models with no need for that much data. Furthermore, we find that using both Off-TE and On-TE results in better convergence in the pre-training phase. With both effectiveness and efficiency, we think that doing multi-task learning on self-supervised speech models from our perspective is a promising trend.
【2】 MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation标题：Medley Vox：一种多歌声分离评价数据集链接：https://arxiv.org/abs/2211.07302
作者：Chang-Bin Jeon,Hyeongi Moon,Keunwoo Choi,Ben Sangbae Chon,Kyogu Lee机构：Gaudio Lab, Inc., Seoul, South Korea, Department of Intelligence and Information, Music and Audio Research Group (MARG), Seoul National University, Seoul, South Korea备注：5 pages, 3 figures, 6 tables, submitted to ICASSP 2023摘要：在音乐源分离研究中，多个演唱声部的分离是一个研究较少的领域。缺乏基准数据集阻碍了其进展。在这篇论文中，我们提出了一个评估数据集，并提供了多个歌唱声音分离的基线研究。首先，我们介绍了一个用于多个歌唱声音分离的评估数据集MedleyVox，它对应于这些类别。我们通过将问题分类为i）二重唱、ii）和声、iii）主音与休止音以及iv）N-歌唱分离来指定该数据集中的问题定义。其次，我们提出了一个策略，用于构建多个歌唱混合使用各种单一的歌唱数据集。这可用于获得训练数据。第三，提出了改进的超分辨率网络（iSRNet）。通过与Conv-TasNet和多声部混合结构策略联合训练，iSRNet在MedleyVox的二重唱和合唱子集上取得了与理想时频模板相当的性能。音频样本、数据集和代码可以在我们的GitHub页面上找到（https：//github.com/jeonchangbin49/MedleyVox）。摘要：Separation of multiple singing voices into each voice is a rarely studied area in music source separation research. The absence of a benchmark dataset has hindered its progress. In this paper, we present an evaluation dataset and provide baseline studies for multiple singing voices separation. First, we introduce MedleyVox, an evaluation dataset for multiple singing voices separation that corresponds to such categories. We specify the problem definition in this dataset by categorizing the problem into i) duet, ii) unison, iii)main vs. rest, and iv) N-singing separation. Second, we present a strategy for construction of multiple singing mixtures using various single-singing datasets. This can be used to obtain training data. Third, we propose the improved super-resolution network (iSRNet). Jointly trained with the Conv-TasNet and the multi-singing mixture construction strategy, the proposed iSRNet achieved comparable performance to ideal time-frequency masks on duet and unison subsets of MedleyVox. Audio samples, the dataset, and codes are available on our GitHub page (https://github.com/jeonchangbin49/MedleyVox).
【3】 Exploiting Device and Audio Data to Tag Music with User-Aware Listening Contexts标题：利用设备和音频数据为音乐添加用户感知的收听环境链接：https://arxiv.org/abs/2211.07250
作者：Karim M. Ibrahim,Elena V. Epure,Geoffroy Peeters,Gaël Richard机构：LTCI, Télécom Paris, Institut Polytechnique de Paris, Deezer Research备注：Published in ISMIR摘要：随着音乐越来越多，特别是在音乐流媒体平台上，人们开始有不同的偏好，以适应他们不同的收听情况，也称为上下文。因此，在向用户推荐音乐时考虑用户的情况的兴趣日益增长。先前的工作已经提出了用户感知的自动标签器，以从音乐内容和用户的全局收听偏好推断与情境相关的标签。然而，在实际的音乐检索系统中，自动标记器只能通过假设上下文类由用户显式地提供来使用。在本文中，为了设计一个全自动的音乐检索系统，我们提出了从用户的流数据中消除用户的收听信息的歧义。即，我们提出了一种系统，该系统可以1）通过利用用户知晓的音乐自动标记器，以及2）通过从流数据（例如，设备、网络）和用户的一般简档信息（例如，年龄）自动推断用户的情况，来生成用户在某个时间的情况播放列表。实验表明，这种基于上下文感知的个性化音乐检索系统是可行的，但在有新用户、新曲目或上下文类数量增加的情况下，检索性能会有所下降。摘要：As music has become more available especially on music streaming platforms, people have started to have distinct preferences to fit to their varying listening situations, also known as context. Hence, there has been a growing interest in considering the user's situation when recommending music to users. Previous works have proposed user-aware autotaggers to infer situation-related tags from music content and user's global listening preferences. However, in a practical music retrieval system, the autotagger could be only used by assuming that the context class is explicitly provided by the user. In this work, for designing a fully automatised music retrieval system, we propose to disambiguate the user's listening information from their stream data. Namely, we propose a system which can generate a situational playlist for a user at a certain time 1) by leveraging user-aware music autotaggers, and 2) by automatically inferring the user's situation from stream data (e.g. device, network) and user's general profile information (e.g. age). Experiments show that such a context-aware personalized music retrieval system is feasible, but the performance decreases in the case of new users, new tracks or when the number of context classes increases.
【4】 YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations标题：YM2413-MDB：一个带情感标注的多乐器调频视频游戏音乐数据集链接：https://arxiv.org/abs/2211.07131
作者：Eunjin Choi,Yoonjin Chung,Seolhee Lee,JongIk Jeon,Taegyun Kwon,Juhan Nam机构：Graduate School of Culture Technology, KAIST, South Korea, Graduate School of AI, KAIST, South Korea, Department of Industrial Design, KAIST, South Korea备注：The paper has been accepted for publication at ISMIR 2022摘要：None摘要：Existing multi-instrumental datasets tend to be biased toward pop and classical music. In addition, they generally lack high-level annotations such as emotion tags. In this paper, we propose YM2413-MDB, an 80s FM video game music dataset with multi-label emotion annotations. It includes 669 audio and MIDI files of music from Sega and MSX PC games in the 80s using YM2413, a programmable sound generator based on FM. The collected game music is arranged with a subset of 15 monophonic instruments and one drum instrument. They were converted from binary commands of the YM2413 sound chip. Each song was labeled with 19 emotion tags by two annotators and validated by three verifiers to obtain refined tags. We provide the baseline models and results for emotion recognition and emotion-conditioned symbolic music generation using YM2413-MDB.
【5】 Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing标题：自动声码器：使用可差分数字信号处理从学习的语音表示快速产生波形链接：https://arxiv.org/abs/2211.06989
作者：Jacob J Webber,Cassia Valentini-Botinhao,Evelyn Williams,Gustav Eje Henter,Simon King机构：The Centre for Speech Technology Research, University of Edinburgh, United Kingdom, Division of Speech, Music, and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：Submitted to the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：大多数现有技术的文本到语音系统使用mel谱图作为中间表示，以将任务分解为声学建模和波形生成。 通过简单、快速的DSP操作从波形中提取Mel谱图，但是从Mel谱图生成高质量波形需要计算上昂贵的机器学习：神经声码器。我们提出的"自动声码器“正好颠倒了这种安排。我们使用机器学习来获得替代Mel谱图的表示，并且可以使用简单、快速的操作（包括逆STFT的可微分实现）将其反转回波形。 自动声码器生成波形的速度比基于DSP的Griffin-Lim算法快5倍，比神经声码器HiFi-GAN快14倍。我们提供了感知听觉测试结果，以确认在复制合成任务中，语音具有与HiFi-GAN相当的质量。摘要：Most state-of-the-art Text-to-Speech systems use the mel-spectrogram as an intermediate representation, to decompose the task into acoustic modelling and waveform generation. A mel-spectrogram is extracted from the waveform by a simple, fast DSP operation, but generating a high-quality waveform from a mel-spectrogram requires computationally expensive machine learning: a neural vocoder. Our proposed ``autovocoder'' reverses this arrangement. We use machine learning to obtain a representation that replaces the mel-spectrogram, and that can be inverted back to a waveform using simple, fast operations including a differentiable implementation of the inverse STFT. The autovocoder generates a waveform 5 times faster than the DSP-based Griffin-Lim algorithm, and 14 times faster than the neural vocoder HiFi-GAN. We provide perceptual listening test results to confirm that the speech is of comparable quality to HiFi-GAN in the copy synthesis task.
【6】 FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs标题：FullPack：在通用CPU上实现亚字节量化推理的全向量利用链接：https://arxiv.org/abs/2211.06982
作者：Hossein Katebi,Navidreza Asadi,Maziar Goudarzi机构：These schemes usually quantize weights 1Department of Computer Engineering, Sharif University ofTechnology, Technical University of Mu-nich摘要：尽管现有技术已经证明了在子字节量化中可忽略的精度下降-其中权重和/或激活由小于8位表示-CPU的流行SIMD指令本身不支持这些数据类型。虽然最近的方法，如ULPPACK，已经在具有矢量单元的通用CPU上使用子字节量化，但它们在存储器和矢量寄存器中的子字节值之间留出了几个空位，以避免在操作期间溢出到相邻的子字节值。这会导致内存占用量和带宽使用效率低下，性能达不到最佳。在本文中，我们介绍了用于存储的存储器布局，以及用于处理子字节（4位、2位或1位）模型的机制，这些模型利用存储器和矢量寄存器中的所有位来存储实际数据。我们为权重和不同数据类型（例如，8-位激活和4位权重）。为了进行评估，我们扩展了TFLite包并将我们的方法添加到其中，然后在周期精确的gem 5模拟器上运行模型，以比较每种方法的详细内存和CPU周期。我们与生产中常用的其他九种方法进行了比较，包括GEMLOWP、Ruy、XNNPack和ULPPACK。此外，我们还探讨了深度学习层的不同输入和输出大小对所提方法性能的影响。实验结果表明，对于小尺寸的加速比为0.96- 2.1倍，对于中到大尺寸的加速比为1.2- 6.7倍。将我们的建议应用于真实世界的语音识别模型Mozilla DeepSpeech，我们证明了我们的方法与现有技术相比实现了1.56- 2.11倍的端到端加速，具体取决于所采用的位宽。摘要：Although prior art has demonstrated negligible accuracy drop in sub-byte quantization -- where weights and/or activations are represented by less than 8 bits -- popular SIMD instructions of CPUs do not natively support these datatypes. While recent methods, such as ULPPACK, are already using sub-byte quantization on general-purpose CPUs with vector units, they leave out several empty bits between the sub-byte values in memory and in vector registers to avoid overflow to the neighbours during the operations. This results in memory footprint and bandwidth-usage inefficiencies and suboptimal performance. In this paper, we present memory layouts for storing, and mechanisms for processing sub-byte (4-, 2-, or 1-bit) models that utilize all the bits in the memory as well as in the vector registers for the actual data. We provide compute kernels for the proposed layout for the GEMV (GEneral Matrix-Vector multiplication) operations between weights and activations of different datatypes (e.g., 8-bit activations and 4-bit weights). For evaluation, we extended the TFLite package and added our methods to it, then ran the models on the cycle-accurate gem5 simulator to compare detailed memory and CPU cycles of each method. We compare against nine other methods that are actively used in production including GEMLOWP, Ruy, XNNPack, and ULPPACK. Furthermore, we explore the effect of different input and output sizes of deep learning layers on the performance of our proposed method. Experimental results show 0.96-2.1x speedup for small sizes and 1.2-6.7x speedup for mid to large sizes. Applying our proposal to a real-world speech recognition model, Mozilla DeepSpeech, we proved that our method achieves 1.56-2.11x end-to-end speedup compared to the state-of-the-art, depending on the bit-width employed.
【7】 Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation标题：基于特征融合和关键词到字幕增强的大规模对比性语言-音频预训练链接：https://arxiv.org/abs/2211.06687
作者：Yusong Wu,Ke Chen,Tianyu Zhang,Yuchen Hui,Taylor Berg-Kirkpatrick,Shlomo Dubnov机构：Mila, Quebec Artificial Intelligence Institute, Universit´e de Montr´eal, University of California San Diego, LAION摘要：对比学习在多模态表征学习领域取得了显著的成功。本文提出了一种对比语言-音频预训练的流水线，通过将音频数据与自然语言描述相结合来开发音频表示。为了实现这一目标，我们首先发布了LAION-Audio-630 K，这是一个来自不同数据源的633，526个音频-文本对的大型集合。其次，我们通过考虑不同的音频编码器和文本编码器，构建了一个对比的语言-音频预训练模型。我们将特征融合机制和关键词到字幕的增强引入模型设计中，以进一步使模型能够处理可变长度的音频输入并提高性能。第三，我们通过三个任务执行综合实验来评估我们的模型：文本到音频检索、zero-shot音频分类和监督音频分类。实验结果表明，该模型在文本到音频的检索任务中取得了较好的性能。在音频分类任务中，该模型在zero-shot设置中实现了最先进的性能，并且能够获得与非零炮设置中的模型结果相当的性能。LAION-Audio-630 K和建议的模型都可供公众使用。摘要：Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.
【8】 Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement标题：通过数据增强、课程学习和多任务增强提高DistilHuBERT对未知噪声条件的健壮性链接：https://arxiv.org/abs/2211.06562
作者：Heitor R. Guimarães,Arthur Pimentel,Anderson R. Avila,Mehdi Rezagholizadeh,Tiago H. Falk机构：Institut National de la Recherche Scientifique, Huawei Noah’s Ark Lab备注：ENLSP-II NeurIPS Workshop 2022, 6 pages摘要：自监督语音表示学习的目的是从语音信号中提取有意义的因子，这些因子稍后可以用于不同的下游任务，诸如语音和/或情感识别。然而，诸如HuBERT的现有模型可能相当大，因此可能不适合于边缘语音应用。此外，现实应用通常涉及被噪声和房间混响破坏的语音，因此模型需要提供对这样的环境因素鲁棒的表示。在本研究中，我们建立了所谓的DistilHuBERT模型，该模型将HuBERT蒸馏到其原始大小的一小部分，并进行了三处修改，即：（i）在学生模型需要从教师模型中提取清晰表示的同时，用噪声和混响来扩充训练数据;（ii）引入课程学习方法，其中随着模型训练引入增加的噪声水平，从而有助于收敛和创建更鲁棒的表示;以及（iii）引入多任务学习方法，其中模型还与蒸馏任务联合地重构纯净波形，因此还用作增强步骤以确保对表示的附加环境鲁棒性。在三个SUPERB任务上的实验表明，所提出的方法不仅相对于原始的DistilHuBERT，而且相对于原始的HuBERT都具有优势，从而显示了所提出的方法在“野生”边缘语音应用中的优势。摘要：Self-supervised speech representation learning aims to extract meaningful factors from the speech signal that can later be used across different downstream tasks, such as speech and/or emotion recognition. Existing models, such as HuBERT, however, can be fairly large thus may not be suitable for edge speech applications. Moreover, realistic applications typically involve speech corrupted by noise and room reverberation, hence models need to provide representations that are robust to such environmental factors. In this study, we build on the so-called DistilHuBERT model, which distils HuBERT to a fraction of its original size, with three modifications, namely: (i) augment the training data with noise and reverberation, while the student model needs to distill the clean representations from the teacher model; (ii) introduce a curriculum learning approach where increasing levels of noise are introduced as the model trains, thus helping with convergence and with the creation of more robust representations; and (iii) introduce a multi-task learning approach where the model also reconstructs the clean waveform jointly with the distillation task, thus also acting as an enhancement step to ensure additional environment robustness to the representation. Experiments on three SUPERB tasks show the advantages of the proposed method not only relative to the original DistilHuBERT, but also to the original HuBERT, thus showing the advantages of the proposed method for ``in the wild'' edge speech applications.
【9】 Low Pass Filtering and Bandwidth Extension for Robust Anti-spoofing Countermeasure Against Codec Variabilities标题：基于低通滤波和带宽扩展的抗编解码变化的稳健反欺骗对策链接：https://arxiv.org/abs/2211.06546
作者：Yikang Wang,Xingming Wang,Hiromitsu Nishizaki,Ming Li机构：Integrated Graduate School of Medicine, Engineering, and Agricultural Sciences, University of Yamanashi, Japan, Data Science Research Center, Duke Kunshan University, China, School of Computer Science, Wuhan University, China备注：5 pages, 3 figures, accepted by ISCSLP 2022摘要：一个可靠的语音反欺骗对抗系统需要在各种欺骗场景下鲁棒地保护自动说话人确认（ASV）系统。然而，信道效应和编解码器会降低对抗系统的性能。本文证明了利用信号的低频子带作为输入可以减轻编解码器对对抗系统带来的负面影响。为了验证这一点，将两种不同截止频率的低通滤波器应用于对抗系统，等误码率（EER）相对降低了25%。此外，我们提出了一种基于深度学习的带宽扩展方法，以进一步提高检测精度。近年来的研究表明，语音活动检测（VAD）去除静音部分后，对抗系统的误码率急剧增加。实验结果表明，在VAD应用的编解码条件下，滤波和带宽扩展方法也是有效的。摘要：A reliable voice anti-spoofing countermeasure system needs to robustly protect automatic speaker verification (ASV) systems in various kinds of spoofing scenarios. However, the performance of countermeasure systems could be degraded by channel effects and codecs. In this paper, we show that using the low-frequency subbands of signals as input can mitigate the negative impact introduced by codecs on the countermeasure systems. To validate this, two types of low-pass filters with different cut-off frequencies are applied to countermeasure systems, and the equal error rate (EER) is reduced by up to 25% relatively. In addition, we propose a deep learning based bandwidth extension approach to further improve the detection accuracy. Recent studies show that the error rate of countermeasure systems increase dramatically when the silence part is removed by Voice Activity Detection (VAD), our experimental results show that the filtering and bandwidth extension approaches are also effective under the codec condition when VAD is applied.
【10】 On the robustness of non-intrusive speech quality model by adversarial examples标题：从对抗性实例看非干扰语音质量模型的鲁棒性链接：https://arxiv.org/abs/2211.06508
作者：Hsin-Yi Lin,Huan-Hsin Tseng,Yu Tsao机构：Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan摘要：最近已经表明，基于深度学习的模型在语音质量预测方面是有效的，并且可以在各个方面优于传统的度量。虽然网络模型有潜力成为复杂人类听觉感知的替代品，但它们可能包含预测中的不稳定性。这项工作表明，深度语音质量预测器可能易受对抗性扰动的影响，其中预测可能被与语音输入相比小到~ 30 dB的不明显扰动剧烈改变。除了揭示深层语音质量预测器的脆弱性外，我们还进一步探索和证实了对抗性训练用于增强模型鲁棒性的可行性。摘要：It has been shown recently that deep learning based models are effective on speech quality prediction and could outperform traditional metrics in various perspectives. Although network models have potential to be a surrogate for complex human hearing perception, they may contain instabilities in predictions. This work shows that deep speech quality predictors can be vulnerable to adversarial perturbations, where the prediction can be changed drastically by unnoticeable perturbations as small as $-30$ dB compared with speech inputs. In addition to exposing the vulnerability of deep speech quality predictors, we further explore and confirm the viability of adversarial training for strengthening robustness of models.
【11】 The Potential of Neural Speech Synthesis-based Data Augmentation for Personalized Speech Enhancement标题：基于神经语音合成的数据增强在个性化语音增强中的潜力链接：https://arxiv.org/abs/2211.07493
作者：Anastasia Kuznetsova,Aswin Sivaraman,Minje Kim机构：Indiana University, Luddy School of Informatics, Computing, and Engineering, Bloomington, IN, USA摘要：随着深度学习的发展，语音增强系统受益于大型神经网络架构，并实现了最先进的质量。然而，当在资源受限的环境中使用说话者不可知的方法时，就质量和它们的复杂性而言，它们并不总是合乎需要的。个性化语音增强（PSE）是一种很有前途的方法，它是一个较小的、更容易解决的语音增强问题，因为它关注于特定的测试时间用户。为了实现个性化的目标，在处理典型的个人数据不足的情况下，研究了基于神经语音合成（NSS）的数据增强的效果。在所提出的方法中，我们表明NSS系统的合成数据的质量很重要，并且如果它们足够好，则扩充数据集可用于改进PSE系统，使其优于说话人不可知的基线。所提出的PSE系统在保持增强质量的同时显示出显著的复杂度降低。摘要：With the advances in deep learning, speech enhancement systems benefited from large neural network architectures and achieved state-of-the-art quality. However, speaker-agnostic methods are not always desirable, both in terms of quality and their complexity, when they are to be used in a resource-constrained environment. One promising way is personalized speech enhancement (PSE), which is a smaller and easier speech enhancement problem for small models to solve, because it focuses on a particular test-time user. To achieve the personalization goal, while dealing with the typical lack of personal data, we investigate the effect of data augmentation based on neural speech synthesis (NSS). In the proposed method, we show that the quality of the NSS system's synthetic data matters, and if they are good enough the augmented dataset can be used to improve the PSE system that outperforms the speaker-agnostic baseline. The proposed PSE systems show significant complexity reduction while preserving the enhancement quality.
【12】 Exploring the Impact of Noise and Degradations on Heart Sound Classification Models标题：噪声和降质对心音分类模型的影响探讨链接：https://arxiv.org/abs/2211.07445
作者：Davoud Shariat Panah,Andrew Hines,Susan McKeever机构：School of Computer Science, Technological University Dublin, Dublin, Ireland, School of Computer Science, University College Dublin, Dublin, Ireland, A R T I C L E I N F O备注：Submitted to Computers in Biology and Medicine Journal摘要：数据驱动的心音分类模型的开发是近年来的一个活跃研究领域。为了首先开发这种数据驱动模型，需要使用信号采集设备来捕获心音信号。然而，在大多数情况下，由于内部和外部噪声的存在，几乎不可能捕获无噪声的心音信号。心音信号中的这种噪声和退化可能潜在地降低数据驱动分类模型的准确性。尽管在文献中已经提出了不同的技术来解决噪声问题，但是心音信号中的不同噪声和退化如何以及在多大程度上影响数据驱动分类模型的准确性仍然没有被探索。为了回答这个问题，我们制作了一个合成心音数据集，包括被各种噪声和退化污染的正常和异常心音。我们使用该数据集来研究心音记录中的噪声和退化对不同分类模型性能的影响。结果表明，不同的噪声和退化对心音分类模型的性能有不同程度的影响;对于分类模型来说，一些是更有问题的，而其它的是破坏性较小的。将本研究的结果与我们之前对一组临床医生进行的调查结果进行比较，结果表明，对分类模型更有害的噪声和退化对准确听诊也更具破坏性。本研究的结果可用于开发有针对性的心音质量增强方法-根据心音信号中的噪声和退化特征调整质量增强的类型和积极性。摘要：The development of data-driven heart sound classification models has been an active area of research in recent years. To develop such data-driven models in the first place, heart sound signals need to be captured using a signal acquisition device. However, it is almost impossible to capture noise-free heart sound signals due to the presence of internal and external noises in most situations. Such noises and degradations in heart sound signals can potentially reduce the accuracy of data-driven classification models. Although different techniques have been proposed in the literature to address the noise issue, how and to what extent different noise and degradations in heart sound signals impact the accuracy of data-driven classification models remains unexplored. To answer this question, we produced a synthetic heart sound dataset including normal and abnormal heart sounds contaminated with a large variety of noise and degradations. We used this dataset to investigate the impact of noise and degradation in heart sound recordings on the performance of different classification models. The results show different noises and degradations affect the performance of heart sound classification models to a different extent; some are more problematic for classification models, and others are less destructive. Comparing the findings of this study with the results of a survey we previously carried out with a group of clinicians shows noise and degradations that are more detrimental to classification models are also more disruptive to accurate auscultation. The findings of this study can be leveraged to develop targeted heart sound quality enhancement approaches - which adapt the type and aggressiveness of quality enhancement based on the characteristics of noise and degradation in heart sound signals.
【13】 Multi-Label Training for Text-Independent Speaker Identification标题：文本无关说话人辨认的多标签训练链接：https://arxiv.org/abs/2211.07373
作者：Yuqi Xue机构：Department of Acoustical Science and Engineering, University of Nanjing, Nanjing, China摘要：本文提出了一种新的与文本无关的说话人辨认策略：多标签培训（MLT）。与通常使用的语音和说话人标签之间的一对一对应关系不同，我们将每个说话人的所有语音分成若干子组，每个子组分配一组不同的标签。在识别过程中，只要预测标签与特定说话人的对应标签之一相同，就识别出该特定说话人。实验结果表明，该方法能够使模型更准确地区分数据，并在一定程度上利用了集成学习的优势，同时避免了计算量和存储量的显著增加。在实验中发现，无论是在干净的环境下，还是在有噪声的环境下，经过语音增强后，多标签训练都能取得比普通方法更好的识别性能。值得注意的是，所提出的策略可以很容易地应用于几乎所有现有的文本无关说话人识别模型，以实现进一步的改进。摘要：In this paper, we propose a novel strategy for text-independent speaker identification system: Multi-Label Training (MLT). Instead of the commonly used one-to-one correspondence between the speech and the speaker label, we divide all the speeches of each speaker into several subgroups, with each subgroup assigned a different set of labels. During the identification process, a specific speaker is identified as long as the predicted label is the same as one of his/her corresponding labels. We found that this method can force the model to distinguish the data more accurately, and somehow takes advantages of ensemble learning, while avoiding the significant increase of computation and storage burden. In the experiments, we found that not only in clean conditions, but also in noisy conditions with speech enhancement, Multi-Label Training can still achieve better identification performance than commom methods. It should be noted that the proposed strategy can be easily applied to almost all current text-independent speaker identification models to achieve further improvements.
【14】 SNIPER Training: Variable Sparsity Rate Training For Text-To-Speech标题：狙击手训练：文本到语音的可变稀疏率训练链接：https://arxiv.org/abs/2211.07283
作者：Perry Lam,Huayun Zhang,Nancy F. Chen,Berrak Sisman,Dorien Herremans机构：Singapore University of Technology and Design, Singapore, Institute for Infocomm Research, ASTAR, Singapore, Speech & Machine Learning Lab, The University of Texas at Dallas, Texas, USA摘要：近年来，文本到语音（TTS）模型已经取得了显著的自然度，但像大多数深层神经模型一样，它们的参数比必要的要多。稀疏TTS模型可以通过修剪和额外的重新训练来改进密集模型，或者比密集模型收敛得更快，但会有一些性能损失。受这些结果的启发，我们提出使用衰减的稀疏率来训练TTS模型，即首先使用高初始稀疏率来加速训练，然后使用渐进的速率降低来获得更好的最终性能。这种递减方法不同于当前的将稀疏度递增到期望目标的方法，当前的方法比密集训练花费更多的时间。我们称我们的方法为SNIPER训练：单次初始化剪枝进化率训练。我们在FastSpeech2上的实验表明，虽然我们只能在被基线超越之前的前几个时段获得更好的损失，但最终SNIPER训练的模型在性能上优于恒定稀疏模型和点密集模型。摘要：Text-to-speech (TTS) models have achieved remarkable naturalness in recent years, yet like most deep neural models, they have more parameters than necessary. Sparse TTS models can improve on dense models via pruning and extra retraining, or converge faster than dense models with some performance loss. Inspired by these results, we propose training TTS models using a decaying sparsity rate, i.e. a high initial sparsity to accelerate training first, followed by a progressive rate reduction to obtain better eventual performance. This decremental approach differs from current methods of incrementing sparsity to a desired target, which costs significantly more time than dense training. We call our method SNIPER training: Single-shot Initialization Pruning Evolving-Rate training. Our experiments on FastSpeech2 show that although we were only able to obtain better losses in the first few epochs before being overtaken by the baseline, the final SNIPER-trained models beat constant-sparsity models and pip dense models in performance.
【15】 Towards A Unified Conformer Structure: from ASR to ASV Task标题：走向统一的构象结构：从ASR到ASV任务链接：https://arxiv.org/abs/2211.07201
作者：Dexin Liao,Tao Jiang,Feng Wang,Lin Li,Qingyang Hong机构：School of Informatics, Xiamen University, China, School of Electronic Science and Engineering, Xiamen University, China, Xiamen Talentedsoft Co., Ltd., China摘要：Transformer凭借其强大的自注意机制在自然语言处理和计算机视觉任务中取得了非凡的性能，其变体Conformer已经成为自动语音识别（ASR）领域中最先进的体系结构。然而，目前自动说话人确认的主流体系结构是卷积神经网络，基于Conformer的自动说话人确认仍有很大的研究空间。本文首先将Conformer的体系结构从ASR修改为ASV，改动很小。为了提高模型的泛化能力，采用了长度尺度注意（LSA）方法和清晰度感知最小化（SAM）方法。在VoxCeleb和CN-Celeb上的实验结果表明，与ECAPA-TDNN相比，基于Conformer的ASV具有较好的性能。其次，受迁移学习策略的启发，ASV Conformer自然地从预先训练好的ASR模型初始化。通过参数转换，自注意机制能够更好地关注序列特征之间的关系，在VoxCeleb和CN-Celeb测试集上的EER相对提高了11%，显示了Conformer统一ASV和ASR任务的潜力。最后，我们在ASV-Subtools中提供了一个运行时来评估其在生产场景中的推理速度。我们的代码在www.example.com上发布https://github.com/Snowdar/asv-subtools/tree/master/doc/papers/conformer.md。摘要：Transformer has achieved extraordinary performance in Natural Language Processing and Computer Vision tasks thanks to its powerful self-attention mechanism, and its variant Conformer has become a state-of-the-art architecture in the field of Automatic Speech Recognition (ASR). However, the main-stream architecture for Automatic Speaker Verification (ASV) is convolutional Neural Networks, and there is still much room for research on the Conformer based ASV. In this paper, firstly, we modify the Conformer architecture from ASR to ASV with very minor changes. Length-Scaled Attention (LSA) method and Sharpness-Aware Minimizationis (SAM) are adopted to improve model generalization. Experiments conducted on VoxCeleb and CN-Celeb show that our Conformer based ASV achieves competitive performance compared with the popular ECAPA-TDNN. Secondly, inspired by the transfer learning strategy, ASV Conformer is natural to be initialized from the pretrained ASR model. Via parameter transferring, self-attention mechanism could better focus on the relationship between sequence features, brings about 11% relative improvement in EER on test set of VoxCeleb and CN-Celeb, which reveals the potential of Conformer to unify ASV and ASR task. Finally, we provide a runtime in ASV-Subtools to evaluate its inference speed in production scenario. Our code is released at https://github.com/Snowdar/asv-subtools/tree/master/doc/papers/conformer.md.
【16】 OverFlow: Putting flows on top of neural transducers for better TTS标题：溢出：将流量置于神经传感器之上，以获得更好的TTS链接：https://arxiv.org/abs/2211.06892
作者：Shivam Mehta,Ambika Kirkland,Harm Lameris,Jonas Beskow,Éva Székely,Gustav Eje Henter机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 2 figures, submitted to ICASSP 2023摘要：神经HMM是最近提出的一种用于文本到语音的序列到序列建模的神经转换器。它们结合了经典统计语音合成和现代神经TTS的最佳特性，需要更少的数据和更少的训练更新，并且不易于由神经注意力故障引起的乱码输出。本文将神经HMM TTS与归一化流相结合来描述语音声学的高度非高斯分布。其结果是一个强大的，完全概率模型的持续时间和声学，可以训练使用精确的最大似然。与主流的基于流的声学模型相比，我们的方法集成了自回归，用于改进诸如话语级韵律的长距离依赖性的建模。实验结果表明，基于该方法的语音识别系统在保留神经HMM原有优点的同时，能够获得比同类方法更准确的发音和更好的主观语音质量。音频示例和代码可从www.example.com获得https://shivammehta25.github.io/OverFlow/摘要：Neural HMMs are a type of neural transducer recently proposed for sequence-to-sequence modelling in text-to-speech. They combine the best features of classic statistical speech synthesis and modern neural TTS, requiring less data and fewer training updates, and are less prone to gibberish output caused by neural attention failures. In this paper, we combine neural HMM TTS with normalising flows for describing the highly non-Gaussian distribution of speech acoustics. The result is a powerful, fully probabilistic model of durations and acoustics that can be trained using exact maximum likelihood. Compared to dominant flow-based acoustic models, our approach integrates autoregression for improved modelling of long-range dependences such as utterance-level prosody. Experiments show that a system based on our proposal gives more accurate pronunciations and better subjective speech quality than comparable methods, whilst retaining the original advantages of neural HMMs. Audio examples and code are available at https://shivammehta25.github.io/OverFlow/
【17】 Multi-Speaker and Wide-Band Simulated Conversations as Training Data for End-to-End Neural Diarization标题：多说话人和宽带模拟对话作为端到端神经网络二值化的训练数据链接：https://arxiv.org/abs/2211.06750
作者：Federico Landini,Mireia Diez,Alicia Lozano-Diez,Lukáš Burget机构：AUDIAS (Audio, Data Intelligence and Speech), Universidad Aut´onoma de Madrid, Spain摘要：端到端二进制化提供了标准级联二进制化系统的有吸引力的替代方案，因为单个系统可以一次处理任务的所有方面。已经提出了许多端到端模型的风格，但是它们都需要（到目前为止还不存在）大量的带注释的数据来训练。折衷的解决方案在于生成合成数据，最近提出的模拟会话（SC）显示出比原始模拟混合（SM）显著的改进。在这项工作中，我们创建了每个会话有多个发言者的SC，并表明它们允许比SM显著更好的性能，也减少了对微调阶段的依赖。我们还利用宽带公共音频源创建了SC，并对几个评估集进行了分析。与本出版物一起，我们发布了生成此类数据和在公共集上训练的模型的方法，以及高效处理每个会话的多个说话者和辅助语音活动检测丢失的实现。摘要：End-to-end diarization presents an attractive alternative to standard cascaded diarization systems because a single system can handle all aspects of the task at once. Many flavors of end-to-end models have been proposed but all of them require (so far non-existing) large amounts of annotated data for training. The compromise solution consists in generating synthetic data and the recently proposed simulated conversations (SC) have shown remarkable improvements over the original simulated mixtures (SM). In this work, we create SC with multiple speakers per conversation and show that they allow for substantially better performance than SM, also reducing the dependence on a fine-tuning stage. We also create SC with wide-band public audio sources and present an analysis on several evaluation sets. Together with this publication, we release the recipes for generating such data and models trained on public sets as well as the implementation to efficiently handle multiple speakers per conversation and an auxiliary voice activity detection loss.
【18】 Efficient Speech Quality Assessment using Self-supervised Framewise Embeddings标题：基于自监督分帧嵌入的高效语音质量评估链接：https://arxiv.org/abs/2211.06646
作者：Karl El Hajal,Zihan Wu,Neil Scheidwasser-Clow,Gasser Elbanna,Milos Cernak机构：´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Lausanne, Switzerland, Logitech Europe, Lausanne, Switzerland, University of Copenhagen, Copenhagen, Denmark, McGovern Institute for Brain Research MIT, Cambridge, MA, USA摘要：自动语音质量评估对于音频研究人员、开发人员、语音和语言病理学家以及系统质量工程师来说至关重要。当前最先进的系统是基于与时间相关性建模相结合的逐帧语音特征（手工设计的或可学习的）。本文提出了一种高效的系统，其结果可与2022年会议语音挑战赛中的最佳性能模型相媲美。我们提出的系统的特点是参数数量更少（40- 60倍）、FLOPS更少（100倍）、内存消耗更低（10- 15倍）和延迟更低（30倍）。因此，语音质量从业者可以更快地迭代，在资源有限的硬件上部署系统，并且总体上，所提议的系统有助于可持续的机器学习。本文还得出结论，帧嵌入优于话语级嵌入，并且具有声学条件建模的多任务训练在提供更好的解释的同时不会降低语音质量预测。摘要：Automatic speech quality assessment is essential for audio researchers, developers, speech and language pathologists, and system quality engineers. The current state-of-the-art systems are based on framewise speech features (hand-engineered or learnable) combined with time dependency modeling. This paper proposes an efficient system with results comparable to the best performing model in the ConferencingSpeech 2022 challenge. Our proposed system is characterized by a smaller number of parameters (40-60x), fewer FLOPS (100x), lower memory consumption (10-15x), and lower latency (30x). Speech quality practitioners can therefore iterate much faster, deploy the system on resource-limited hardware, and, overall, the proposed system contributes to sustainable machine learning. The paper also concludes that framewise embeddings outperform utterance-level embeddings and that multi-task training with acoustic conditions modeling does not degrade speech quality prediction while providing better interpretation.
【19】 Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics标题：音频字幕研究：解决词汇失衡和评估以语言为中心的绩效指标的适宜性链接：https://arxiv.org/abs/2211.06547
作者：Sandeep Kothinti,Dimitra Emmanouilidou机构：Laboratory for Computational Auditory Perception, Johns Hopkins University, Baltimore, MD, USA, Microsoft Research Labs, Redmond, WA, USA备注：Submitted to ICASSP 2023摘要：从我们周围的声音中分析、处理和提取有意义的信息是更广泛的音频分析领域的主题。音频字幕是音频分析领域的新成员，它是一种跨模态翻译任务，专注于从音频流中发生的声音事件生成自然描述。在这项工作中，我们确定并改进了自动音频字幕的三个主要挑战：i）数据缺乏，ii）音频字幕词汇表中的不平衡或限制，以及iii）能够最佳地捕获听觉和语义特征的适当的性能评估度量。我们发现，在模型训练过程中，通常采用的损失函数会导致不公平的词汇失衡。我们提出了两种音频字幕增强方法，以丰富训练数据集和词汇量。我们进一步强调了通过探索先前针对不同音频任务进行训练的音频编码器的适用性来进行域内预训练的必要性。最后，我们系统地探讨了从图像字幕领域借用的五个性能指标，并强调了它们在音频领域的局限性。摘要：The analysis, processing, and extraction of meaningful information from sounds all around us is the subject of the broader area of audio analytics. Audio captioning is a recent addition to the domain of audio analytics, a cross-modal translation task that focuses on generating natural descriptions from sound events occurring in an audio stream. In this work, we identify and improve on three main challenges in automated audio captioning: i) data scarcity, ii) imbalance or limitations in the audio captions vocabulary, and iii) the proper performance evaluation metric that can best capture both auditory and semantic characteristics. We find that generally adopted loss functions can result in an unfair vocabulary imbalance during model training. We propose two audio captioning augmentation methods that enrich the training dataset and the vocabulary size. We further underline the need for in-domain pretraining by exploring the suitability of audio encoders that were previously trained on different audio tasks. Finally, we systematically explore five performance metrics borrowed from the image captioning domain and highlight their limitations for the audio domain.
【20】 A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units标题：一种具有自监督离散语音单元的统一单次韵律和说话人转换系统链接：https://arxiv.org/abs/2211.06535
作者：Li-Wei Chen,Shinji Watanabe,Alexander Rudnicky机构：Language Technologies Institute, Carnegie Mellon University备注：Submitted to ICASSP 2023摘要：提出了一个统一的语音转换系统，实现了基于音高、节奏和说话人属性的一次性语音转换（VC）。现有研究普遍忽视了韵律与语言内容之间的相关性，导致转换后语音的自然度下降。此外，缺少适当的语言特征会妨碍这些系统在转换后准确地保留语言内容。为了解决这些问题，我们设计了一个级联模块化系统，利用自监督离散语音单元作为语言表示。这些离散单元提供节奏建模所必需的持续时间信息。我们的系统首先从原始波形中提取话语级韵律和说话人表示。给定韵律表示，韵律预测器估计话语中每个离散单元的音调、能量和持续时间。合成器还基于预测的韵律、说话者表示和离散单元来重构语音。实验结果表明，该系统在自然度、可懂度、说话人可迁移性和韵律可迁移性等方面均优于已有方法。代码和示例是公开的。摘要：We present a unified system to realize one-shot voice conversion (VC) on the pitch, rhythm, and speaker attributes. Existing works generally ignore the correlation between prosody and language content, leading to the degradation of naturalness in converted speech. Additionally, the lack of proper language features prevents these systems from accurately preserving language content after conversion. To address these issues, we devise a cascaded modular system leveraging self-supervised discrete speech units as language representation. These discrete units provide duration information essential for rhythm modeling. Our system first extracts utterance-level prosody and speaker representations from the raw waveform. Given the prosody representation, a prosody predictor estimates pitch, energy, and duration for each discrete unit in the utterance. A synthesizer further reconstructs speech based on the predicted prosody, speaker representation, and discrete units. Experiments show that our system outperforms previous approaches in naturalness, intelligibility, speaker transferability, and prosody transferability. Code and samples are publicly available.
【21】 Breaking trade-offs in speech separation with sparsely-gated mixture of experts标题：使用稀疏门控专家混合打破语音分离中的权衡链接：https://arxiv.org/abs/2211.06493
作者：Xiaofei Wang,Zhuo Chen,Yu Shi,Jian Wu,Naoyuki Kanda,Takuya Yoshioka机构：Microsoft, One Microsoft Way, Redmond, WA , USA备注：Submitted to ICASSP 2023摘要：在会话式自动语音识别系统中使用单耳语音分离模型时，需要权衡几个因素。较大的SS模型通常以较高的计算代价获得较好的输出质量，同时，用于重叠语音的较好的SS模型常常产生用于非重叠语音的失真输出。本文通过稀疏门控专家混合（MoE）解决了这些权衡问题。稀疏门控MoE架构允许在不牺牲运行时效率的情况下扩大分离模型，这也有助于实现更好的分离-失真折衷。为了进一步减少语音失真而不损害SS能力，还探索了多门MoE框架，其中不同的门不同地处理非重叠和重叠帧。利用仿真数据集进行了语音识别实验，测量了语音分离精度和语音失真度。两个先进的SS模型，Conformer和基于WavLM的模型，被用作基线。稀疏门控MoE模型表现出更好的SS性能和更小的语音失真，同时略微增加了运行时计算成本。实验结果表明，MoE在端到端评估环境中的有效性。摘要：Several trade-offs need to be balanced when employing monaural speech separation (SS) models in conversational automatic speech recognition (ASR) systems. A larger SS model generally achieves better output quality at an expense of higher computation, meanwhile, a better SS model for overlapping speech often produces distorted output for non-overlapping speech. This paper addresses these trade-offs with a sparsely-gated mixture-of-experts (MoE). The sparsely-gated MoE architecture allows the separation models to be enlarged without compromising the run-time efficiency, which also helps achieve a better separation-distortion trade-off. To further reduce the speech distortion without compromising the SS capability, a multi-gate MoE framework is also explored, where different gates handle non-overlapping and overlapping frames differently. ASR experiments are conducted by using a simulated dataset for measuring both the speech separation accuracy and the speech distortion. Two advanced SS models, Conformer and WavLM-based models, are used as baselines. The sparsely-gated MoE models show a superior SS capability with less speech distortion, meanwhile marginally increasing the run-time computational cost. Experimental results using real conversation recordings are also presented, showing MoE's effectiveness in an end-to-end evaluation setting.
【22】 Augmenting Transformer-Transducer Based Speaker Change Detection With Token-Level Training Loss标题：利用令牌级训练损失增强基于Transformer-换能器的说话人变化检测链接：https://arxiv.org/abs/2211.06482
作者：Guanlong Zhao,Quan Wang,Han Lu,Yiling Huang,Ignacio Lopez Moreno机构：Google LLC, USA摘要：本文提出了一种新的基于标记的训练策略，提高了基于T-T变换的说话人变化检测（SCD）性能。传统的基于T-T的SCD模型损失同等地优化所有输出令牌。由于训练数据中说话人变化的稀疏性，传统的基于T-T的SCD模型丢失导致次优的检测精度。为了缓解这一问题，我们使用定制的编辑距离算法在训练过程中估计令牌级SCD错误接受（FA）和错误拒绝（FR）率，并优化模型参数以最小化FA和FR的加权组合，使模型专注于准确预测说话人变化。我们还提出了一组评估指标，这些指标更好地与商业用例保持一致。在一组具有挑战性的真实数据集上的实验表明，在参数个数相同的情况下，所提出的训练方法能够显著提高SCD模型的整体性能.摘要：In this work we propose a novel token-based training strategy that improves Transformer-Transducer (T-T) based speaker change detection (SCD) performance. The conventional T-T based SCD model loss optimizes all output tokens equally. Due to the sparsity of the speaker changes in the training data, the conventional T-T based SCD model loss leads to sub-optimal detection accuracy. To mitigate this issue, we use a customized edit-distance algorithm to estimate the token-level SCD false accept (FA) and false reject (FR) rates during training and optimize model parameters to minimize a weighted combination of the FA and FR, focusing the model on accurately predicting speaker changes. We also propose a set of evaluation metrics that align better with commercial use cases. Experiments on a group of challenging real-world datasets show that the proposed training method can significantly improve the overall performance of the SCD model with the same number of parameters.
【23】 Exploring Sequence-to-Sequence Transformer-Transducer Models for Keyword Spotting标题：探索用于关键词检测的序列到序列转换器-传感器模型链接：https://arxiv.org/abs/2211.06478
作者：Beltrán Labrador,Guanlong Zhao,Ignacio López Moreno,Angelo Scorza Scarpati,Liam Fowl,Quan Wang机构：AUDIAS (Audio, Data Intelligence and Speech), Universidad Aut´onoma de Madrid, Spain, Google LLC, USA摘要：本文提出了一种新的方法来适应序列到序列转换器-转换器ASR系统的关键字定位（KWS）任务。我们通过将文本转录中的关键字替换为特殊标记并训练系统检测音频流中的标记来实现这一点。在推理阶段，我们借鉴传统的KWS方法，构造了一个决策函数，使我们的方法更适合于KWS任务。此外，我们采用序列判别最小贝叶斯风险训练技术引入了特定的关键词识别损失。我们发现我们的方法明显优于基于ASR的KWS系统。当与传统的关键词识别系统相比时，我们的建议具有类似的性能，同时带来了序列到序列训练的优点和灵活性。此外，当与传统的KWS系统结合时，我们的方法可以改善任何操作点的性能。摘要：In this paper, we present a novel approach to adapt a sequence-to-sequence Transformer-Transducer ASR system to the keyword spotting (KWS) task. We achieve this by replacing the keyword in the text transcription with a special tokenand training the system to detect thetoken in an audio stream. At inference time, we create a decision function inspired by conventional KWS approaches, to make our approach more suitable for the KWS task. Furthermore, we introduce a specific keyword spotting loss by adapting the sequence-discriminative Minimum Bayes-Risk training technique. We find that our approach significantly outperforms ASR based KWS systems. When compared with a conventional keyword spotting system, our proposal has similar performance while bringing the advantages and flexibility of sequence-to-sequence training. Additionally, when combined with the conventional KWS system, our approach can improve the performance at any operation point.
【1】 The Potential of Neural Speech Synthesis-based Data Augmentation for Personalized Speech Enhancement标题：基于神经语音合成的数据增强在个性化语音增强中的潜力链接：https://arxiv.org/abs/2211.07493
* 与cs.SD语音【11】为同一篇
作者：Anastasia Kuznetsova,Aswin Sivaraman,Minje Kim机构：Indiana University, Luddy School of Informatics, Computing, and Engineering, Bloomington, IN, USA摘要：随着深度学习的发展，语音增强系统受益于大型神经网络架构，并实现了最先进的质量。然而，当在资源受限的环境中使用说话者不可知的方法时，就质量和它们的复杂性而言，它们并不总是合乎需要的。个性化语音增强（PSE）是一种很有前途的方法，它是一个较小的、更容易解决的语音增强问题，因为它关注于特定的测试时间用户。为了实现个性化的目标，在处理典型的个人数据不足的情况下，研究了基于神经语音合成（NSS）的数据增强的效果。在所提出的方法中，我们表明NSS系统的合成数据的质量很重要，并且如果它们足够好，则扩充数据集可用于改进PSE系统，使其优于说话人不可知的基线。所提出的PSE系统在保持增强质量的同时显示出显著的复杂度降低。摘要：With the advances in deep learning, speech enhancement systems benefited from large neural network architectures and achieved state-of-the-art quality. However, speaker-agnostic methods are not always desirable, both in terms of quality and their complexity, when they are to be used in a resource-constrained environment. One promising way is personalized speech enhancement (PSE), which is a smaller and easier speech enhancement problem for small models to solve, because it focuses on a particular test-time user. To achieve the personalization goal, while dealing with the typical lack of personal data, we investigate the effect of data augmentation based on neural speech synthesis (NSS). In the proposed method, we show that the quality of the NSS system's synthetic data matters, and if they are good enough the augmented dataset can be used to improve the PSE system that outperforms the speaker-agnostic baseline. The proposed PSE systems show significant complexity reduction while preserving the enhancement quality.
【2】 Exploring the Impact of Noise and Degradations on Heart Sound Classification Models标题：噪声和降质对心音分类模型的影响探讨链接：https://arxiv.org/abs/2211.07445
* 与cs.SD语音【12】为同一篇
作者：Davoud Shariat Panah,Andrew Hines,Susan McKeever机构：School of Computer Science, Technological University Dublin, Dublin, Ireland, School of Computer Science, University College Dublin, Dublin, Ireland, A R T I C L E I N F O备注：Submitted to Computers in Biology and Medicine Journal摘要：数据驱动的心音分类模型的开发是近年来的一个活跃研究领域。为了首先开发这种数据驱动模型，需要使用信号采集设备来捕获心音信号。然而，在大多数情况下，由于内部和外部噪声的存在，几乎不可能捕获无噪声的心音信号。心音信号中的这种噪声和退化可能潜在地降低数据驱动分类模型的准确性。尽管在文献中已经提出了不同的技术来解决噪声问题，但是心音信号中的不同噪声和退化如何以及在多大程度上影响数据驱动分类模型的准确性仍然没有被探索。为了回答这个问题，我们制作了一个合成心音数据集，包括被各种噪声和退化污染的正常和异常心音。我们使用该数据集来研究心音记录中的噪声和退化对不同分类模型性能的影响。结果表明，不同的噪声和退化对心音分类模型的性能有不同程度的影响;对于分类模型来说，一些是更有问题的，而其它的是破坏性较小的。将本研究的结果与我们之前对一组临床医生进行的调查结果进行比较，结果表明，对分类模型更有害的噪声和退化对准确听诊也更具破坏性。本研究的结果可用于开发有针对性的心音质量增强方法--根据心音信号中的噪声和退化特征调整质量增强的类型和积极性。摘要：The development of data-driven heart sound classification models has been an active area of research in recent years. To develop such data-driven models in the first place, heart sound signals need to be captured using a signal acquisition device. However, it is almost impossible to capture noise-free heart sound signals due to the presence of internal and external noises in most situations. Such noises and degradations in heart sound signals can potentially reduce the accuracy of data-driven classification models. Although different techniques have been proposed in the literature to address the noise issue, how and to what extent different noise and degradations in heart sound signals impact the accuracy of data-driven classification models remains unexplored. To answer this question, we produced a synthetic heart sound dataset including normal and abnormal heart sounds contaminated with a large variety of noise and degradations. We used this dataset to investigate the impact of noise and degradation in heart sound recordings on the performance of different classification models. The results show different noises and degradations affect the performance of heart sound classification models to a different extent; some are more problematic for classification models, and others are less destructive. Comparing the findings of this study with the results of a survey we previously carried out with a group of clinicians shows noise and degradations that are more detrimental to classification models are also more disruptive to accurate auscultation. The findings of this study can be leveraged to develop targeted heart sound quality enhancement approaches - which adapt the type and aggressiveness of quality enhancement based on the characteristics of noise and degradation in heart sound signals.
【3】 The Far Side of Failure: Investigating the Impact of Speech Recognition Errors on Subsequent Dementia Classification标题：失败的另一面：调查语音识别错误对随后的痴呆症分类的影响链接：https://arxiv.org/abs/2211.07430
作者：Changye Li,Trevor Cohen,Serguei Pakhomov机构：Institute of Health Informatics, University of Minnesota, Biomedical Informatics and Medical Education, University of Washington, Department of Pharmaceutical Care & Health Systems, University of Minnesota备注：Accepted as extended abstract for ML4H 2022摘要：在自发言语中可检测到的语言异常已显示出用于各种临床应用的前景，包括筛查痴呆和其他形式的认知障碍。部署能够对从大规模临床环境中的语音获得的语言样本进行分类的自动化工具的可行性取决于捕获和自动转录语音以用于后续分析的能力。然而，具有精选语音数据的自监督学习（SSL）自动语音识别（ASR）模型的令人印象深刻的性能对于来自临床设置的具有挑战性的语音样本并不明显。成功地将ASR模型应用于临床应用的关键问题之一是它们产生的不完美转录物是否为下游任务提供足够的信息以在可接受的准确性水平上操作。在本研究中，我们检查了几个深度学习ASR系统产生的错误之间的关系及其对痴呆分类下游任务的影响。我们的一个关键发现是，矛盾的是，错误率相对较高的ASR系统可以产生比基于逐字记录的分类更好的下游分类准确性的转录本。摘要：Linguistic anomalies detectable in spontaneous speech have shown promise for various clinical applications including screening for dementia and other forms of cognitive impairment. The feasibility of deploying automated tools that can classify language samples obtained from speech in large-scale clinical settings depends on the ability to capture and automatically transcribe the speech for subsequent analysis. However, the impressive performance of self-supervised learning (SSL) automatic speech recognition (ASR) models with curated speech data is not apparent with challenging speech samples from clinical settings. One of the key questions for successfully applying ASR models for clinical applications is whether imperfect transcripts they generate provide sufficient information for downstream tasks to operate at an acceptable level of accuracy. In this study, we examine the relationship between the errors produced by several deep learning ASR systems and their impact on the downstream task of dementia classification. One of our key findings is that, paradoxically, ASR systems with relatively high error rates can produce transcripts that result in better downstream classification accuracy than classification based on verbatim transcripts.
【4】 Multi-Label Training for Text-Independent Speaker Identification标题：文本无关说话人辨认的多标签训练链接：https://arxiv.org/abs/2211.07373
* 与cs.SD语音【13】为同一篇
作者：Yuqi Xue机构：Department of Acoustical Science and Engineering, University of Nanjing, Nanjing, China摘要：本文提出了一种新的与文本无关的说话人辨认策略：多标签培训（MLT）。与通常使用的语音和说话人标签之间的一对一对应关系不同，我们将每个说话人的所有语音分成若干子组，每个子组分配一组不同的标签。在识别过程中，只要预测标签与特定说话人的对应标签之一相同，就识别出该特定说话人。实验结果表明，该方法能够使模型更准确地区分数据，并在一定程度上利用了集成学习的优势，同时避免了计算量和存储量的显著增加。在实验中发现，无论是在干净的环境下，还是在有噪声的环境下，经过语音增强后，多标签训练都能取得比普通方法更好的识别性能。值得注意的是，所提出的策略可以很容易地应用于几乎所有现有的文本无关说话人识别模型，以实现进一步的改进。摘要：In this paper, we propose a novel strategy for text-independent speaker identification system: Multi-Label Training (MLT). Instead of the commonly used one-to-one correspondence between the speech and the speaker label, we divide all the speeches of each speaker into several subgroups, with each subgroup assigned a different set of labels. During the identification process, a specific speaker is identified as long as the predicted label is the same as one of his/her corresponding labels. We found that this method can force the model to distinguish the data more accurately, and somehow takes advantages of ensemble learning, while avoiding the significant increase of computation and storage burden. In the experiments, we found that not only in clean conditions, but also in noisy conditions with speech enhancement, Multi-Label Training can still achieve better identification performance than commom methods. It should be noted that the proposed strategy can be easily applied to almost all current text-independent speaker identification models to achieve further improvements.
【5】 Sentiment recognition of Italian elderly through domain adaptation on cross-corpus speech dataset标题：跨语料库语音数据集上领域自适应的意大利老年人情感识别链接：https://arxiv.org/abs/2211.07307
作者：Francesca Gasparini,Alessandra Grossi机构：Department of Computer Science, Systems and Communications, University of Milano - Bicocca, Italy备注：15 pages, 3 figures, 5 tables摘要：本研究的目的是定义一个能够识别意大利老年人自然会话中的积极、中性和消极情绪的语音情绪识别（SER）模型。文献中提供了几个SER数据集。然而，大多数是在英语或汉语，已记录下来，而演员发音短短语，因此不涉及自然会话。此外，在所有数据库中，只有少数演讲与老年人有关。因此，在这项工作中，一个多语言和多年龄的语料库被认为是合并一个英语数据集，其中也包括老年人，与一个意大利语数据集。基于XGBoost，提出了一个针对英国青年和成年男女演员训练的通用模型。在此基础上，提出了两种领域自适应策略，分别用于老年人和意大利语使用者的领域自适应。结果表明，这种方法提高了分类性能，同时也强调了应该收集新的数据集。摘要：The aim of this work is to define a speech emotion recognition (SER) model able to recognize positive, neutral and negative emotions in natural conversations of Italian elderly people. Several datasets for SER are available in the literature. However most of them are in English or Chinese, have been recorded while actors and actresses pronounce short phrases and thus are not related to natural conversation. Moreover only few speeches among all the databases are related to elderly people. Therefore, in this work, a multi-language and multi-age corpus is considered merging a dataset in English, that includes also elderly people, with a dataset in Italian. A general model, trained on young and adult English actors and actresses is proposed, based on XGBoost. Then two strategies of domain adaptation are proposed to adapt the model either to elderly people and to Italian speakers. The results suggest that this approach increases the classification performance, underlining also that new datasets should be collected.
【6】 SNIPER Training: Variable Sparsity Rate Training For Text-To-Speech标题：狙击手训练：文本到语音的可变稀疏率训练链接：https://arxiv.org/abs/2211.07283
* 与cs.SD语音【14】为同一篇
作者：Perry Lam,Huayun Zhang,Nancy F. Chen,Berrak Sisman,Dorien Herremans机构：Singapore University of Technology and Design, Singapore, Institute for Infocomm Research, ASTAR, Singapore, Speech & Machine Learning Lab, The University of Texas at Dallas, Texas, USA摘要：近年来，文本到语音（TTS）模型已经取得了显著的自然度，但像大多数深层神经模型一样，它们的参数比必要的要多。稀疏TTS模型可以通过修剪和额外的重新训练来改进密集模型，或者比密集模型收敛得更快，但会有一些性能损失。受这些结果的启发，我们提出使用衰减的稀疏率来训练TTS模型，即首先使用高初始稀疏率来加速训练，然后使用渐进的速率降低来获得更好的最终性能。这种递减方法不同于当前的将稀疏度递增到期望目标的方法，当前的方法比密集训练花费更多的时间。我们称我们的方法为SNIPER训练：单次初始化剪枝进化率训练。我们在FastSpeech2上的实验表明，虽然我们只能在被基线超越之前的前几个时段获得更好的损失，但最终SNIPER训练的模型在性能上优于恒定稀疏模型和点密集模型。摘要：Text-to-speech (TTS) models have achieved remarkable naturalness in recent years, yet like most deep neural models, they have more parameters than necessary. Sparse TTS models can improve on dense models via pruning and extra retraining, or converge faster than dense models with some performance loss. Inspired by these results, we propose training TTS models using a decaying sparsity rate, i.e. a high initial sparsity to accelerate training first, followed by a progressive rate reduction to obtain better eventual performance. This decremental approach differs from current methods of incrementing sparsity to a desired target, which costs significantly more time than dense training. We call our method SNIPER training: Single-shot Initialization Pruning Evolving-Rate training. Our experiments on FastSpeech2 show that although we were only able to obtain better losses in the first few epochs before being overtaken by the baseline, the final SNIPER-trained models beat constant-sparsity models and pip dense models in performance.
【7】 Towards A Unified Conformer Structure: from ASR to ASV Task标题：走向统一的构象结构：从ASR到ASV任务链接：https://arxiv.org/abs/2211.07201
* 与cs.SD语音【15】为同一篇
作者：Dexin Liao,Tao Jiang,Feng Wang,Lin Li,Qingyang Hong机构：School of Informatics, Xiamen University, China, School of Electronic Science and Engineering, Xiamen University, China, Xiamen Talentedsoft Co., Ltd., China摘要：Transformer凭借其强大的自注意机制在自然语言处理和计算机视觉任务中取得了非凡的性能，其变体Conformer已经成为自动语音识别（ASR）领域中最先进的体系结构。然而，目前自动说话人确认的主流体系结构是卷积神经网络，基于Conformer的自动说话人确认仍有很大的研究空间。本文首先将Conformer的体系结构从ASR修改为ASV，改动很小。为了提高模型的泛化能力，采用了长度尺度注意（LSA）方法和清晰度感知最小化（SAM）方法。在VoxCeleb和CN-Celeb上的实验结果表明，与ECAPA-TDNN相比，基于Conformer的ASV具有较好的性能。其次，受迁移学习策略的启发，ASV Conformer自然地从预先训练好的ASR模型初始化。通过参数转换，自注意机制能够更好地关注序列特征之间的关系，在VoxCeleb和CN-Celeb测试集上的EER相对提高了11%，显示了Conformer统一ASV和ASR任务的潜力。最后，我们在ASV-Subtools中提供了一个运行时来评估其在生产场景中的推理速度。我们的代码在www.example.com上发布https://github.com/Snowdar/asv-subtools/tree/master/doc/papers/conformer.md。摘要：Transformer has achieved extraordinary performance in Natural Language Processing and Computer Vision tasks thanks to its powerful self-attention mechanism, and its variant Conformer has become a state-of-the-art architecture in the field of Automatic Speech Recognition (ASR). However, the main-stream architecture for Automatic Speaker Verification (ASV) is convolutional Neural Networks, and there is still much room for research on the Conformer based ASV. In this paper, firstly, we modify the Conformer architecture from ASR to ASV with very minor changes. Length-Scaled Attention (LSA) method and Sharpness-Aware Minimizationis (SAM) are adopted to improve model generalization. Experiments conducted on VoxCeleb and CN-Celeb show that our Conformer based ASV achieves competitive performance compared with the popular ECAPA-TDNN. Secondly, inspired by the transfer learning strategy, ASV Conformer is natural to be initialized from the pretrained ASR model. Via parameter transferring, self-attention mechanism could better focus on the relationship between sequence features, brings about 11% relative improvement in EER on test set of VoxCeleb and CN-Celeb, which reveals the potential of Conformer to unify ASV and ASR task. Finally, we provide a runtime in ASV-Subtools to evaluate its inference speed in production scenario. Our code is released at https://github.com/Snowdar/asv-subtools/tree/master/doc/papers/conformer.md.
【8】 OverFlow: Putting flows on top of neural transducers for better TTS标题：溢出：将流量置于神经传感器之上，以获得更好的TTS链接：https://arxiv.org/abs/2211.06892
* 与cs.SD语音【16】为同一篇
作者：Shivam Mehta,Ambika Kirkland,Harm Lameris,Jonas Beskow,Éva Székely,Gustav Eje Henter机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 2 figures, submitted to ICASSP 2023摘要：神经HMM是最近提出的一种用于文本到语音的序列到序列建模的神经转换器。它们结合了经典统计语音合成和现代神经TTS的最佳特性，需要更少的数据和更少的训练更新，并且不易于由神经注意力故障引起的乱码输出。本文将神经HMM TTS与归一化流相结合来描述语音声学的高度非高斯分布。其结果是一个强大的，完全概率模型的持续时间和声学，可以训练使用精确的最大似然。与主流的基于流的声学模型相比，我们的方法集成了自回归，用于改进诸如话语级韵律的长距离依赖性的建模。实验结果表明，基于该方法的语音识别系统在保留神经HMM原有优点的同时，能够获得比同类方法更准确的发音和更好的主观语音质量。音频示例和代码可从www.example.com获得https://shivammehta25.github.io/OverFlow/摘要：Neural HMMs are a type of neural transducer recently proposed for sequence-to-sequence modelling in text-to-speech. They combine the best features of classic statistical speech synthesis and modern neural TTS, requiring less data and fewer training updates, and are less prone to gibberish output caused by neural attention failures. In this paper, we combine neural HMM TTS with normalising flows for describing the highly non-Gaussian distribution of speech acoustics. The result is a powerful, fully probabilistic model of durations and acoustics that can be trained using exact maximum likelihood. Compared to dominant flow-based acoustic models, our approach integrates autoregression for improved modelling of long-range dependences such as utterance-level prosody. Experiments show that a system based on our proposal gives more accurate pronunciations and better subjective speech quality than comparable methods, whilst retaining the original advantages of neural HMMs. Audio examples and code are available at https://shivammehta25.github.io/OverFlow/
【9】 Multi-Speaker and Wide-Band Simulated Conversations as Training Data for End-to-End Neural Diarization标题：多说话人和宽带模拟对话作为端到端神经网络二值化的训练数据链接：https://arxiv.org/abs/2211.06750
* 与cs.SD语音【17】为同一篇
作者：Federico Landini,Mireia Diez,Alicia Lozano-Diez,Lukáš Burget机构：AUDIAS (Audio, Data Intelligence and Speech), Universidad Aut´onoma de Madrid, Spain摘要：端到端二进制化提供了标准级联二进制化系统的有吸引力的替代方案，因为单个系统可以一次处理任务的所有方面。已经提出了许多端到端模型的风格，但是它们都需要（到目前为止还不存在）大量的带注释的数据来训练。折衷的解决方案在于生成合成数据，最近提出的模拟会话（SC）显示出比原始模拟混合（SM）显著的改进。在这项工作中，我们创建了每个会话有多个发言者的SC，并表明它们允许比SM显著更好的性能，也减少了对微调阶段的依赖。我们还利用宽带公共音频源创建了SC，并对几个评估集进行了分析。与本出版物一起，我们发布了生成此类数据和在公共集上训练的模型的方法，以及高效处理每个会话的多个说话者和辅助语音活动检测丢失的实现。摘要：End-to-end diarization presents an attractive alternative to standard cascaded diarization systems because a single system can handle all aspects of the task at once. Many flavors of end-to-end models have been proposed but all of them require (so far non-existing) large amounts of annotated data for training. The compromise solution consists in generating synthetic data and the recently proposed simulated conversations (SC) have shown remarkable improvements over the original simulated mixtures (SM). In this work, we create SC with multiple speakers per conversation and show that they allow for substantially better performance than SM, also reducing the dependence on a fine-tuning stage. We also create SC with wide-band public audio sources and present an analysis on several evaluation sets. Together with this publication, we release the recipes for generating such data and models trained on public sets as well as the implementation to efficiently handle multiple speakers per conversation and an auxiliary voice activity detection loss.
【10】 Efficient Speech Quality Assessment using Self-supervised Framewise Embeddings标题：基于自监督分帧嵌入的高效语音质量评估链接：https://arxiv.org/abs/2211.06646
* 与cs.SD语音【18】为同一篇
作者：Karl El Hajal,Zihan Wu,Neil Scheidwasser-Clow,Gasser Elbanna,Milos Cernak机构：´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Lausanne, Switzerland, Logitech Europe, Lausanne, Switzerland, University of Copenhagen, Copenhagen, Denmark, McGovern Institute for Brain Research MIT, Cambridge, MA, USA摘要：自动语音质量评估对于音频研究人员、开发人员、语音和语言病理学家以及系统质量工程师来说至关重要。当前最先进的系统是基于与时间相关性建模相结合的逐帧语音特征（手工设计的或可学习的）。本文提出了一种高效的系统，其结果可与2022年会议语音挑战赛中的最佳性能模型相媲美。我们提出的系统的特点是参数数量更少（40- 60倍）、FLOPS更少（100倍）、内存消耗更低（10- 15倍）和延迟更低（30倍）。因此，语音质量从业者可以更快地迭代，在资源有限的硬件上部署系统，并且总体上，所提议的系统有助于可持续的机器学习。本文还得出结论，帧嵌入优于话语级嵌入，并且具有声学条件建模的多任务训练在提供更好的解释的同时不会降低语音质量预测。摘要：Automatic speech quality assessment is essential for audio researchers, developers, speech and language pathologists, and system quality engineers. The current state-of-the-art systems are based on framewise speech features (hand-engineered or learnable) combined with time dependency modeling. This paper proposes an efficient system with results comparable to the best performing model in the ConferencingSpeech 2022 challenge. Our proposed system is characterized by a smaller number of parameters (40-60x), fewer FLOPS (100x), lower memory consumption (10-15x), and lower latency (30x). Speech quality practitioners can therefore iterate much faster, deploy the system on resource-limited hardware, and, overall, the proposed system contributes to sustainable machine learning. The paper also concludes that framewise embeddings outperform utterance-level embeddings and that multi-task training with acoustic conditions modeling does not degrade speech quality prediction while providing better interpretation.
【11】 Investigations in Audio Captioning: Addressing Vocabulary Imbalance and Evaluating Suitability of Language-Centric Performance Metrics标题：音频字幕研究：解决词汇失衡和评估以语言为中心的绩效指标的适宜性链接：https://arxiv.org/abs/2211.06547
* 与cs.SD语音【19】为同一篇
作者：Sandeep Kothinti,Dimitra Emmanouilidou机构：Laboratory for Computational Auditory Perception, Johns Hopkins University, Baltimore, MD, USA, Microsoft Research Labs, Redmond, WA, USA备注：Submitted to ICASSP 2023摘要：从我们周围的声音中分析、处理和提取有意义的信息是更广泛的音频分析领域的主题。音频字幕是音频分析领域的新成员，它是一种跨模态翻译任务，专注于从音频流中发生的声音事件生成自然描述。在这项工作中，我们确定并改进了自动音频字幕的三个主要挑战：i）数据缺乏，ii）音频字幕词汇表中的不平衡或限制，以及iii）能够最佳地捕获听觉和语义特征的适当的性能评估度量。我们发现，在模型训练过程中，通常采用的损失函数会导致不公平的词汇失衡。我们提出了两种音频字幕增强方法，以丰富训练数据集和词汇量。我们进一步强调了通过探索先前针对不同音频任务进行训练的音频编码器的适用性来进行域内预训练的必要性。最后，我们系统地探讨了从图像字幕领域借用的五个性能指标，并强调了它们在音频领域的局限性。摘要：The analysis, processing, and extraction of meaningful information from sounds all around us is the subject of the broader area of audio analytics. Audio captioning is a recent addition to the domain of audio analytics, a cross-modal translation task that focuses on generating natural descriptions from sound events occurring in an audio stream. In this work, we identify and improve on three main challenges in automated audio captioning: i) data scarcity, ii) imbalance or limitations in the audio captions vocabulary, and iii) the proper performance evaluation metric that can best capture both auditory and semantic characteristics. We find that generally adopted loss functions can result in an unfair vocabulary imbalance during model training. We propose two audio captioning augmentation methods that enrich the training dataset and the vocabulary size. We further underline the need for in-domain pretraining by exploring the suitability of audio encoders that were previously trained on different audio tasks. Finally, we systematically explore five performance metrics borrowed from the image captioning domain and highlight their limitations for the audio domain.
【12】 A unified one-shot prosody and speaker conversion system with self-supervised discrete speech units标题：一种具有自监督离散语音单元的统一单次韵律和说话人转换系统链接：https://arxiv.org/abs/2211.06535
* 与cs.SD语音【20】为同一篇
作者：Li-Wei Chen,Shinji Watanabe,Alexander Rudnicky机构：Language Technologies Institute, Carnegie Mellon University备注：Submitted to ICASSP 2023摘要：提出了一个统一的语音转换系统，实现了基于音高、节奏和说话人属性的一次性语音转换（VC）。现有研究普遍忽视了韵律与语言内容之间的相关性，导致转换后语音的自然度下降。此外，缺少适当的语言特征会妨碍这些系统在转换后准确地保留语言内容。为了解决这些问题，我们设计了一个级联模块化系统，利用自监督离散语音单元作为语言表示。这些离散单元提供节奏建模所必需的持续时间信息。我们的系统首先从原始波形中提取话语级韵律和说话人表示。给定韵律表示，韵律预测器估计话语中每个离散单元的音调、能量和持续时间。合成器还基于预测的韵律、说话者表示和离散单元来重构语音。实验结果表明，该系统在自然度、可懂度、说话人可迁移性和韵律可迁移性等方面均优于已有方法。代码和示例是公开的。摘要：We present a unified system to realize one-shot voice conversion (VC) on the pitch, rhythm, and speaker attributes. Existing works generally ignore the correlation between prosody and language content, leading to the degradation of naturalness in converted speech. Additionally, the lack of proper language features prevents these systems from accurately preserving language content after conversion. To address these issues, we devise a cascaded modular system leveraging self-supervised discrete speech units as language representation. These discrete units provide duration information essential for rhythm modeling. Our system first extracts utterance-level prosody and speaker representations from the raw waveform. Given the prosody representation, a prosody predictor estimates pitch, energy, and duration for each discrete unit in the utterance. A synthesizer further reconstructs speech based on the predicted prosody, speaker representation, and discrete units. Experiments show that our system outperforms previous approaches in naturalness, intelligibility, speaker transferability, and prosody transferability. Code and samples are publicly available.
【13】 Breaking trade-offs in speech separation with sparsely-gated mixture of experts标题：使用稀疏门控专家混合打破语音分离中的权衡链接：https://arxiv.org/abs/2211.06493
* 与cs.SD语音【21】为同一篇
作者：Xiaofei Wang,Zhuo Chen,Yu Shi,Jian Wu,Naoyuki Kanda,Takuya Yoshioka机构：Microsoft, One Microsoft Way, Redmond, WA , USA备注：Submitted to ICASSP 2023摘要：在会话式自动语音识别系统中使用单耳语音分离模型时，需要权衡几个因素。较大的SS模型通常以较高的计算代价获得较好的输出质量，同时，用于重叠语音的较好的SS模型常常产生用于非重叠语音的失真输出。本文通过稀疏门控专家混合（MoE）解决了这些权衡问题。稀疏门控MoE架构允许在不牺牲运行时效率的情况下扩大分离模型，这也有助于实现更好的分离-失真折衷。为了进一步减少语音失真而不损害SS能力，还探索了多门MoE框架，其中不同的门不同地处理非重叠和重叠帧。利用仿真数据集进行了语音识别实验，测量了语音分离精度和语音失真度。两个先进的SS模型，Conformer和基于WavLM的模型，被用作基线。稀疏门控MoE模型表现出更好的SS性能和更小的语音失真，同时略微增加了运行时计算成本。实验结果表明，MoE在端到端评估环境中的有效性。摘要：Several trade-offs need to be balanced when employing monaural speech separation (SS) models in conversational automatic speech recognition (ASR) systems. A larger SS model generally achieves better output quality at an expense of higher computation, meanwhile, a better SS model for overlapping speech often produces distorted output for non-overlapping speech. This paper addresses these trade-offs with a sparsely-gated mixture-of-experts (MoE). The sparsely-gated MoE architecture allows the separation models to be enlarged without compromising the run-time efficiency, which also helps achieve a better separation-distortion trade-off. To further reduce the speech distortion without compromising the SS capability, a multi-gate MoE framework is also explored, where different gates handle non-overlapping and overlapping frames differently. ASR experiments are conducted by using a simulated dataset for measuring both the speech separation accuracy and the speech distortion. Two advanced SS models, Conformer and WavLM-based models, are used as baselines. The sparsely-gated MoE models show a superior SS capability with less speech distortion, meanwhile marginally increasing the run-time computational cost. Experimental results using real conversation recordings are also presented, showing MoE's effectiveness in an end-to-end evaluation setting.
【14】 Augmenting Transformer-Transducer Based Speaker Change Detection With Token-Level Training Loss标题：利用令牌级训练损失增强基于Transformer-换能器的说话人变化检测链接：https://arxiv.org/abs/2211.06482
* 与cs.SD语音【22】为同一篇
作者：Guanlong Zhao,Quan Wang,Han Lu,Yiling Huang,Ignacio Lopez Moreno机构：Google LLC, USA摘要：本文提出了一种新的基于标记的训练策略，提高了基于T-T变换的说话人变化检测（SCD）性能。传统的基于T-T的SCD模型损失同等地优化所有输出令牌。由于训练数据中说话人变化的稀疏性，传统的基于T-T的SCD模型丢失导致次优的检测精度。为了缓解这一问题，我们使用定制的编辑距离算法在训练过程中估计令牌级SCD错误接受（FA）和错误拒绝（FR）率，并优化模型参数以最小化FA和FR的加权组合，使模型专注于准确预测说话人变化。我们还提出了一组评估指标，这些指标更好地与商业用例保持一致。在一组具有挑战性的真实数据集上的实验表明，在参数个数相同的情况下，所提出的训练方法能够显著提高SCD模型的整体性能.摘要：In this work we propose a novel token-based training strategy that improves Transformer-Transducer (T-T) based speaker change detection (SCD) performance. The conventional T-T based SCD model loss optimizes all output tokens equally. Due to the sparsity of the speaker changes in the training data, the conventional T-T based SCD model loss leads to sub-optimal detection accuracy. To mitigate this issue, we use a customized edit-distance algorithm to estimate the token-level SCD false accept (FA) and false reject (FR) rates during training and optimize model parameters to minimize a weighted combination of the FA and FR, focusing the model on accurately predicting speaker changes. We also propose a set of evaluation metrics that align better with commercial use cases. Experiments on a group of challenging real-world datasets show that the proposed training method can significantly improve the overall performance of the SCD model with the same number of parameters.
【15】 Exploring Sequence-to-Sequence Transformer-Transducer Models for Keyword Spotting标题：探索用于关键词检测的序列到序列转换器-传感器模型链接：https://arxiv.org/abs/2211.06478
* 与cs.SD语音【23】为同一篇
作者：Beltrán Labrador,Guanlong Zhao,Ignacio López Moreno,Angelo Scorza Scarpati,Liam Fowl,Quan Wang机构：AUDIAS (Audio, Data Intelligence and Speech), Universidad Aut´onoma de Madrid, Spain, Google LLC, USA摘要：本文提出了一种新的方法来适应序列到序列转换器-转换器ASR系统的关键字定位（KWS）任务。我们通过将文本转录中的关键字替换为特殊标记并训练系统检测音频流中的标记来实现这一点。在推理阶段，我们借鉴传统的KWS方法，构造了一个决策函数，使我们的方法更适合于KWS任务。此外，我们采用序列判别最小贝叶斯风险训练技术引入了特定的关键词识别损失。我们发现我们的方法明显优于基于ASR的KWS系统。当与传统的关键词识别系统相比时，我们的建议具有类似的性能，同时带来了序列到序列训练的优点和灵活性。此外，当与传统的KWS系统结合时，我们的方法可以改善任何操作点的性能。摘要：In this paper, we present a novel approach to adapt a sequence-to-sequence Transformer-Transducer ASR system to the keyword spotting (KWS) task. We achieve this by replacing the keyword in the text transcription with a special tokenand training the system to detect thetoken in an audio stream. At inference time, we create a decision function inspired by conventional KWS approaches, to make our approach more suitable for the KWS task. Furthermore, we introduce a specific keyword spotting loss by adapting the sequence-discriminative Minimum Bayes-Risk training technique. We find that our approach significantly outperforms ASR based KWS systems. When compared with a conventional keyword spotting system, our proposal has similar performance while bringing the advantages and flexibility of sequence-to-sequence training. Additionally, when combined with the conventional KWS system, our approach can improve the performance at any operation point.
【16】 MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets标题：MT4SSL：通过集成多个目标来促进自监督语音表示学习链接：https://arxiv.org/abs/2211.07321
* 与cs.SD语音【1】为同一篇
作者：Ziyang Ma,Zhisheng Zhen,Changli Tang,Yujin Wang,Xie Chen机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China, Department of Electronic Engineering, Tsinghua University备注：Submitted to ICASSP 2023摘要：本文从自训练目标的获取入手，为自监督语音模型的研究提供了一个新的视角。我们将目标提取器推广为离线目标提取器（Off-TE）和在线目标提取器（On-TE），而不考虑具体的借口任务。在此基础上，提出了一种新的多任务自监督学习框架MT4 SSL，即通过集成多个目标来提升自监督语音表示学习。MT4 SSL是指两个典型的模型，HuBERT和data 2 vec，它们分别使用K-means算法作为Off-TE和使用无梯度的教师网络作为On-TE。我们的模型在LibriSpeech基准测试中的性能超过了以前的SSL方法，并且在不需要那么多数据的情况下，与性能最好的模型相当，甚至更好。此外，我们发现使用Off-TE和On-TE两者导致在预训练阶段更好的收敛。从有效性和效率两方面考虑，我们认为在我们的视角下对自监督语音模型进行多任务学习是一个很有前途的趋势。摘要：In this paper, we provide a new perspective on self-supervised speech models from how the self-training targets are obtained. We generalize the targets extractor into Offline Targets Extractor (Off-TE) and Online Targets Extractor (On-TE), without caring about specific pretext tasks. Based on this, we propose a new multi-tasking learning framework for self-supervised learning, MT4SSL, which stands for Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targets. MT4SSL refers to two typical models, HuBERT and data2vec, which use the K-means algorithm as an Off-TE and a teacher network without gradients as an On-TE, respectively. Our model outperforms previous SSL methods by nontrivial margins on the LibriSpeech benchmark, and is comparable to or even better than the best-performing models with no need for that much data. Furthermore, we find that using both Off-TE and On-TE results in better convergence in the pre-training phase. With both effectiveness and efficiency, we think that doing multi-task learning on self-supervised speech models from our perspective is a promising trend.
【17】 MedleyVox: An Evaluation Dataset for Multiple Singing Voices Separation标题：Medley Vox：一种多歌声分离评价数据集链接：https://arxiv.org/abs/2211.07302
* 与cs.SD语音【2】为同一篇
作者：Chang-Bin Jeon,Hyeongi Moon,Keunwoo Choi,Ben Sangbae Chon,Kyogu Lee机构：Gaudio Lab, Inc., Seoul, South Korea, Department of Intelligence and Information, Music and Audio Research Group (MARG), Seoul National University, Seoul, South Korea备注：5 pages, 3 figures, 6 tables, submitted to ICASSP 2023摘要：在音乐源分离研究中，多个演唱声部的分离是一个研究较少的领域。缺乏基准数据集阻碍了其进展。在这篇论文中，我们提出了一个评估数据集，并提供了多个歌唱声音分离的基线研究。首先，我们介绍了一个用于多个歌唱声音分离的评估数据集MedleyVox，它对应于这些类别。我们通过将问题分类为i）二重唱、ii）和声、iii）主音与休止音以及iv）N-歌唱分离来指定该数据集中的问题定义。其次，我们提出了一个策略，用于构建多个歌唱混合使用各种单一的歌唱数据集。这可用于获得训练数据。第三，提出了改进的超分辨率网络（iSRNet）。通过与Conv-TasNet和多声部混合结构策略联合训练，iSRNet在MedleyVox的二重唱和合唱子集上取得了与理想时频模板相当的性能。音频样本、数据集和代码可以在我们的GitHub页面上找到（https：//github.com/jeonchangbin49/MedleyVox）。摘要：Separation of multiple singing voices into each voice is a rarely studied area in music source separation research. The absence of a benchmark dataset has hindered its progress. In this paper, we present an evaluation dataset and provide baseline studies for multiple singing voices separation. First, we introduce MedleyVox, an evaluation dataset for multiple singing voices separation that corresponds to such categories. We specify the problem definition in this dataset by categorizing the problem into i) duet, ii) unison, iii)main vs. rest, and iv) N-singing separation. Second, we present a strategy for construction of multiple singing mixtures using various single-singing datasets. This can be used to obtain training data. Third, we propose the improved super-resolution network (iSRNet). Jointly trained with the Conv-TasNet and the multi-singing mixture construction strategy, the proposed iSRNet achieved comparable performance to ideal time-frequency masks on duet and unison subsets of MedleyVox. Audio samples, the dataset, and codes are available on our GitHub page (https://github.com/jeonchangbin49/MedleyVox).
【18】 Exploiting Device and Audio Data to Tag Music with User-Aware Listening Contexts标题：利用设备和音频数据为音乐添加用户感知的收听环境链接：https://arxiv.org/abs/2211.07250
* 与cs.SD语音【3】为同一篇
作者：Karim M. Ibrahim,Elena V. Epure,Geoffroy Peeters,Gaël Richard机构：LTCI, Télécom Paris, Institut Polytechnique de Paris, Deezer Research备注：Published in ISMIR摘要：随着音乐越来越多，特别是在音乐流媒体平台上，人们开始有不同的偏好，以适应他们不同的收听情况，也称为上下文。因此，在向用户推荐音乐时考虑用户的情况的兴趣日益增长。先前的工作已经提出了用户感知的自动标签器，以从音乐内容和用户的全局收听偏好推断与情境相关的标签。然而，在实际的音乐检索系统中，自动标记器只能通过假设上下文类由用户显式地提供来使用。在本文中，为了设计一个全自动的音乐检索系统，我们提出了从用户的流数据中消除用户的收听信息的歧义。即，我们提出了一种系统，该系统可以1）通过利用用户知晓的音乐自动标记器，以及2）通过从流数据（例如，设备、网络）和用户的一般简档信息（例如，年龄）自动推断用户的情况，来生成用户在某个时间的情况播放列表。实验表明，这种基于上下文感知的个性化音乐检索系统是可行的，但在有新用户、新曲目或上下文类数量增加的情况下，检索性能会有所下降。摘要：As music has become more available especially on music streaming platforms, people have started to have distinct preferences to fit to their varying listening situations, also known as context. Hence, there has been a growing interest in considering the user's situation when recommending music to users. Previous works have proposed user-aware autotaggers to infer situation-related tags from music content and user's global listening preferences. However, in a practical music retrieval system, the autotagger could be only used by assuming that the context class is explicitly provided by the user. In this work, for designing a fully automatised music retrieval system, we propose to disambiguate the user's listening information from their stream data. Namely, we propose a system which can generate a situational playlist for a user at a certain time 1) by leveraging user-aware music autotaggers, and 2) by automatically inferring the user's situation from stream data (e.g. device, network) and user's general profile information (e.g. age). Experiments show that such a context-aware personalized music retrieval system is feasible, but the performance decreases in the case of new users, new tracks or when the number of context classes increases.
【19】 YM2413-MDB: A Multi-Instrumental FM Video Game Music Dataset with Emotion Annotations标题：YM2413-MDB：一个带情感标注的多乐器调频视频游戏音乐数据集链接：https://arxiv.org/abs/2211.07131
* 与cs.SD语音【4】为同一篇
作者：Eunjin Choi,Yoonjin Chung,Seolhee Lee,JongIk Jeon,Taegyun Kwon,Juhan Nam机构：Graduate School of Culture Technology, KAIST, South Korea, Graduate School of AI, KAIST, South Korea, Department of Industrial Design, KAIST, South Korea备注：The paper has been accepted for publication at ISMIR 2022摘要：现有的多乐器数据集倾向于流行音乐和古典音乐。此外，它们通常缺少诸如情感标签之类的高级注释。本文提出了一个带有多标签情感标注的80年代FM视频游戏音乐数据集YM 2413-MDB。它包括669个音频和MIDI文件的音乐，从世嘉和MSX电脑游戏在80年代使用YM 2413，一个可编程的声音发生器的基础上调频。收集的游戏音乐安排了一个子集的15个单声道乐器和一个鼓乐器。它们是从YM 2413声音芯片的二进制命令转换而来的。每首歌曲由两名注释者标注19个情感标签，并由三名验证者验证以获得细化的标签。我们提供了基于YM 2413-MDB的情绪识别和情绪条件化符号音乐生成的基本模型和结果。摘要：Existing multi-instrumental datasets tend to be biased toward pop and classical music. In addition, they generally lack high-level annotations such as emotion tags. In this paper, we propose YM2413-MDB, an 80s FM video game music dataset with multi-label emotion annotations. It includes 669 audio and MIDI files of music from Sega and MSX PC games in the 80s using YM2413, a programmable sound generator based on FM. The collected game music is arranged with a subset of 15 monophonic instruments and one drum instrument. They were converted from binary commands of the YM2413 sound chip. Each song was labeled with 19 emotion tags by two annotators and validated by three verifiers to obtain refined tags. We provide the baseline models and results for emotion recognition and emotion-conditioned symbolic music generation using YM2413-MDB.
【20】 Autovocoder: Fast Waveform Generation from a Learned Speech Representation using Differentiable Digital Signal Processing标题：自动声码器：使用可差分数字信号处理从学习的语音表示快速产生波形链接：https://arxiv.org/abs/2211.06989
* 与cs.SD语音【5】为同一篇
作者：Jacob J Webber,Cassia Valentini-Botinhao,Evelyn Williams,Gustav Eje Henter,Simon King机构：The Centre for Speech Technology Research, University of Edinburgh, United Kingdom, Division of Speech, Music, and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：Submitted to the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：大多数现有技术的文本到语音系统使用mel谱图作为中间表示，以将任务分解为声学建模和波形生成。 通过简单、快速的DSP操作从波形中提取Mel谱图，但是从Mel谱图生成高质量波形需要计算上昂贵的机器学习：神经声码器。我们提出的"自动声码器“正好颠倒了这种安排。我们使用机器学习来获得替代Mel谱图的表示，并且可以使用简单、快速的操作（包括逆STFT的可微分实现）将其反转回波形。 自动声码器生成波形的速度比基于DSP的Griffin-Lim算法快5倍，比神经声码器HiFi-GAN快14倍。我们提供了感知听觉测试结果，以确认在复制合成任务中，语音具有与HiFi-GAN相当的质量。摘要：Most state-of-the-art Text-to-Speech systems use the mel-spectrogram as an intermediate representation, to decompose the task into acoustic modelling and waveform generation. A mel-spectrogram is extracted from the waveform by a simple, fast DSP operation, but generating a high-quality waveform from a mel-spectrogram requires computationally expensive machine learning: a neural vocoder. Our proposed ``autovocoder'' reverses this arrangement. We use machine learning to obtain a representation that replaces the mel-spectrogram, and that can be inverted back to a waveform using simple, fast operations including a differentiable implementation of the inverse STFT. The autovocoder generates a waveform 5 times faster than the DSP-based Griffin-Lim algorithm, and 14 times faster than the neural vocoder HiFi-GAN. We provide perceptual listening test results to confirm that the speech is of comparable quality to HiFi-GAN in the copy synthesis task.
【21】 FullPack: Full Vector Utilization for Sub-Byte Quantized Inference on General Purpose CPUs标题：FullPack：在通用CPU上实现亚字节量化推理的全向量利用链接：https://arxiv.org/abs/2211.06982
* 与cs.SD语音【6】为同一篇
作者：Hossein Katebi,Navidreza Asadi,Maziar Goudarzi机构：These schemes usually quantize weights 1Department of Computer Engineering, Sharif University ofTechnology, Technical University of Mu-nich摘要：尽管现有技术已经证明了在子字节量化中可忽略的精度下降-其中权重和/或激活由小于8位表示-CPU的流行SIMD指令本身不支持这些数据类型。虽然最近的方法，如ULPPACK，已经在具有矢量单元的通用CPU上使用子字节量化，但它们在存储器和矢量寄存器中的子字节值之间留出了几个空位，以避免在操作期间溢出到相邻的子字节值。这会导致内存占用量和带宽使用效率低下，性能达不到最佳。在本文中，我们介绍了用于存储的存储器布局，以及用于处理子字节（4位、2位或1位）模型的机制，这些模型利用存储器和矢量寄存器中的所有位来存储实际数据。我们为权重和不同数据类型（例如，8-位激活和4位权重）。为了进行评估，我们扩展了TFLite包并将我们的方法添加到其中，然后在周期精确的gem 5模拟器上运行模型，以比较每种方法的详细内存和CPU周期。我们与生产中常用的其他九种方法进行了比较，包括GEMLOWP、Ruy、XNNPack和ULPPACK。此外，我们还探讨了深度学习层的不同输入和输出大小对所提方法性能的影响。实验结果表明，对于小尺寸的加速比为0.96- 2.1倍，对于中到大尺寸的加速比为1.2- 6.7倍。将我们的建议应用于真实世界的语音识别模型Mozilla DeepSpeech，我们证明了我们的方法与现有技术相比实现了1.56- 2.11倍的端到端加速，具体取决于所采用的位宽。摘要：Although prior art has demonstrated negligible accuracy drop in sub-byte quantization -- where weights and/or activations are represented by less than 8 bits -- popular SIMD instructions of CPUs do not natively support these datatypes. While recent methods, such as ULPPACK, are already using sub-byte quantization on general-purpose CPUs with vector units, they leave out several empty bits between the sub-byte values in memory and in vector registers to avoid overflow to the neighbours during the operations. This results in memory footprint and bandwidth-usage inefficiencies and suboptimal performance. In this paper, we present memory layouts for storing, and mechanisms for processing sub-byte (4-, 2-, or 1-bit) models that utilize all the bits in the memory as well as in the vector registers for the actual data. We provide compute kernels for the proposed layout for the GEMV (GEneral Matrix-Vector multiplication) operations between weights and activations of different datatypes (e.g., 8-bit activations and 4-bit weights). For evaluation, we extended the TFLite package and added our methods to it, then ran the models on the cycle-accurate gem5 simulator to compare detailed memory and CPU cycles of each method. We compare against nine other methods that are actively used in production including GEMLOWP, Ruy, XNNPack, and ULPPACK. Furthermore, we explore the effect of different input and output sizes of deep learning layers on the performance of our proposed method. Experimental results show 0.96-2.1x speedup for small sizes and 1.2-6.7x speedup for mid to large sizes. Applying our proposal to a real-world speech recognition model, Mozilla DeepSpeech, we proved that our method achieves 1.56-2.11x end-to-end speedup compared to the state-of-the-art, depending on the bit-width employed.
【22】 Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation标题：基于特征融合和关键词到字幕增强的大规模对比性语言-音频预训练链接：https://arxiv.org/abs/2211.06687
* 与cs.SD语音【7】为同一篇
作者：Yusong Wu,Ke Chen,Tianyu Zhang,Yuchen Hui,Taylor Berg-Kirkpatrick,Shlomo Dubnov机构：Mila, Quebec Artificial Intelligence Institute, Universit´e de Montr´eal, University of California San Diego, LAION摘要：对比学习在多模态表征学习领域取得了显著的成功。本文提出了一种对比语言-音频预训练的流水线，通过将音频数据与自然语言描述相结合来开发音频表示。为了实现这一目标，我们首先发布了LAION-Audio-630 K，这是一个来自不同数据源的633，526个音频-文本对的大型集合。其次，我们通过考虑不同的音频编码器和文本编码器，构建了一个对比的语言-音频预训练模型。我们将特征融合机制和关键词到字幕的增强引入模型设计中，以进一步使模型能够处理可变长度的音频输入并提高性能。第三，我们通过三个任务执行综合实验来评估我们的模型：文本到音频检索、zero-shot音频分类和监督音频分类。实验结果表明，该模型在文本到音频的检索任务中取得了较好的性能。在音频分类任务中，该模型在zero-shot设置中实现了最先进的性能，并且能够获得与非零炮设置中的模型结果相当的性能。LAION-Audio-630 K和建议的模型都可供公众使用。摘要：Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.
【23】 Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement标题：通过数据增强、课程学习和多任务增强提高DistilHuBERT对未知噪声条件的健壮性链接：https://arxiv.org/abs/2211.06562
* 与cs.SD语音【8】为同一篇
作者：Heitor R. Guimarães,Arthur Pimentel,Anderson R. Avila,Mehdi Rezagholizadeh,Tiago H. Falk机构：Institut National de la Recherche Scientifique, Huawei Noah’s Ark Lab备注：ENLSP-II NeurIPS Workshop 2022, 6 pages摘要：自监督语音表示学习的目的是从语音信号中提取有意义的因子，这些因子稍后可以用于不同的下游任务，诸如语音和/或情感识别。然而，诸如HuBERT的现有模型可能相当大，因此可能不适合于边缘语音应用。此外，现实应用通常涉及被噪声和房间混响破坏的语音，因此模型需要提供对这样的环境因素鲁棒的表示。在本研究中，我们建立了所谓的DistilHuBERT模型，该模型将HuBERT蒸馏到其原始大小的一小部分，并进行了三处修改，即：（i）在学生模型需要从教师模型中提取清晰表示的同时，用噪声和混响来扩充训练数据;（ii）引入课程学习方法，其中随着模型训练引入增加的噪声水平，从而有助于收敛和创建更鲁棒的表示;以及（iii）引入多任务学习方法，其中模型还与蒸馏任务联合地重构纯净波形，因此还用作增强步骤以确保对表示的附加环境鲁棒性。在三个SUPERB任务上的实验表明，所提出的方法不仅相对于原始的DistilHuBERT，而且相对于原始的HuBERT都具有优势，从而显示了所提出的方法在“野生”边缘语音应用中的优势。摘要：Self-supervised speech representation learning aims to extract meaningful factors from the speech signal that can later be used across different downstream tasks, such as speech and/or emotion recognition. Existing models, such as HuBERT, however, can be fairly large thus may not be suitable for edge speech applications. Moreover, realistic applications typically involve speech corrupted by noise and room reverberation, hence models need to provide representations that are robust to such environmental factors. In this study, we build on the so-called DistilHuBERT model, which distils HuBERT to a fraction of its original size, with three modifications, namely: (i) augment the training data with noise and reverberation, while the student model needs to distill the clean representations from the teacher model; (ii) introduce a curriculum learning approach where increasing levels of noise are introduced as the model trains, thus helping with convergence and with the creation of more robust representations; and (iii) introduce a multi-task learning approach where the model also reconstructs the clean waveform jointly with the distillation task, thus also acting as an enhancement step to ensure additional environment robustness to the representation. Experiments on three SUPERB tasks show the advantages of the proposed method not only relative to the original DistilHuBERT, but also to the original HuBERT, thus showing the advantages of the proposed method for ``in the wild'' edge speech applications.
【24】 Low Pass Filtering and Bandwidth Extension for Robust Anti-spoofing Countermeasure Against Codec Variabilities标题：基于低通滤波和带宽扩展的抗编解码变化的稳健反欺骗对策链接：https://arxiv.org/abs/2211.06546
* 与cs.SD语音【9】为同一篇
作者：Yikang Wang,Xingming Wang,Hiromitsu Nishizaki,Ming Li机构：Integrated Graduate School of Medicine, Engineering, and Agricultural Sciences, University of Yamanashi, Japan, Data Science Research Center, Duke Kunshan University, China, School of Computer Science, Wuhan University, China备注：5 pages, 3 figures, accepted by ISCSLP 2022摘要：一个可靠的语音反欺骗对抗系统需要在各种欺骗场景下鲁棒地保护自动说话人确认（ASV）系统。然而，信道效应和编解码器会降低对抗系统的性能。本文证明了利用信号的低频子带作为输入可以减轻编解码器对对抗系统带来的负面影响。为了验证这一点，将两种不同截止频率的低通滤波器应用于对抗系统，等误码率（EER）相对降低了25%。此外，我们提出了一种基于深度学习的带宽扩展方法，以进一步提高检测精度。近年来的研究表明，语音活动检测（VAD）去除静音部分后，对抗系统的误码率急剧增加。实验结果表明，在VAD应用的编解码条件下，滤波和带宽扩展方法也是有效的。摘要：A reliable voice anti-spoofing countermeasure system needs to robustly protect automatic speaker verification (ASV) systems in various kinds of spoofing scenarios. However, the performance of countermeasure systems could be degraded by channel effects and codecs. In this paper, we show that using the low-frequency subbands of signals as input can mitigate the negative impact introduced by codecs on the countermeasure systems. To validate this, two types of low-pass filters with different cut-off frequencies are applied to countermeasure systems, and the equal error rate (EER) is reduced by up to 25% relatively. In addition, we propose a deep learning based bandwidth extension approach to further improve the detection accuracy. Recent studies show that the error rate of countermeasure systems increase dramatically when the silence part is removed by Voice Activity Detection (VAD), our experimental results show that the filtering and bandwidth extension approaches are also effective under the codec condition when VAD is applied.
【25】 On the robustness of non-intrusive speech quality model by adversarial examples标题：从对抗性实例看非干扰语音质量模型的鲁棒性链接：https://arxiv.org/abs/2211.06508
* 与cs.SD语音【10】为同一篇
作者：Hsin-Yi Lin,Huan-Hsin Tseng,Yu Tsao机构：Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan摘要：最近已经表明，基于深度学习的模型在语音质量预测方面是有效的，并且可以在各个方面优于传统的度量。虽然网络模型有潜力成为复杂人类听觉感知的替代品，但它们可能包含预测中的不稳定性。这项工作表明，深度语音质量预测器可能易受对抗性扰动的影响，其中预测可能被与语音输入相比小到~ 30 dB的不明显扰动剧烈改变。除了揭示深层语音质量预测器的脆弱性外，我们还进一步探索和证实了对抗性训练用于增强模型鲁棒性的可行性。摘要：It has been shown recently that deep learning based models are effective on speech quality prediction and could outperform traditional metrics in various perspectives. Although network models have potential to be a surrogate for complex human hearing perception, they may contain instabilities in predictions. This work shows that deep speech quality predictors can be vulnerable to adversarial perturbations, where the prediction can be changed drastically by unnoticeable perturbations as small as $-30$ dB compared with speech inputs. In addition to exposing the vulnerability of deep speech quality predictors, we further explore and confirm the viability of adversarial training for strengthening robustness of models.
机器翻译，仅供参考
每日语音速递