今日论文合集：cs.SD语音11篇，eess.AS音频处理11篇。
cs.SD语音
【1】 Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis
标题：多方会议分析中的说话人重叠感知神经二值化
链接：https://arxiv.org/abs/2211.10243
作者：Zhihao Du,Shiliang Zhang,Siqi Zheng,Zhijie Yan机构：Speech Lab, Alibaba Group, China备注：Accepted by EMNLP 2022摘要：近年来，聚类和神经网络模型的混合系统已经成功地应用于多方会议分析。然而，现有的重叠说话人识别模型往往将重叠说话人识别问题视为多标签分类问题，没有很好地考虑说话人的相关性和重叠性。为了克服这一缺点，提出了一种基于幂集编码（PSE）的重叠说话人识别问题，并将其转化为单标签预测问题。通过该公式化，说话者依赖性和重叠可以被显式地建模。为了充分利用这一公式，我们进一步提出了说话人重叠感知神经diarization（SOND）模型，该模型由上下文无关（CI）评分器、上下文相关（CD）评分器和说话人组合网络（SCN）组成，其中CI评分器用于建模说话人的全局可辨别性，CD评分器用于建模说话人的局部可辨别性，SCN用于组合和重新分配说话人活动。实验结果表明，与基于目标说话人语音活动检测的方法相比，该方法的性能得到了进一步的提高，相对二值化误差降低了6.30%。摘要：Recently, hybrid systems of clustering and neural diarization models have been successfully applied in multi-party meeting analysis. However, current models always treat overlapped speaker diarization as a multi-label classification problem, where speaker dependency and overlaps are not well considered. To overcome the disadvantages, we reformulate overlapped speaker diarization task as a single-label prediction problem via the proposed power set encoding (PSE). Through this formulation, speaker dependency and overlaps can be explicitly modeled. To fully leverage this formulation, we further propose the speaker overlap-aware neural diarization (SOND) model, which consists of a context-independent (CI) scorer to model global speaker discriminability, a context-dependent scorer (CD) to model local discriminability, and a speaker combining network (SCN) to combine and reassign speaker activities. Experimental results show that using the proposed formulation can outperform the state-of-the-art methods based on target speaker voice activity detection, and the performance can be further improved with SOND, resulting in a 6.30% relative diarization error reduction.
【2】 AVATAR submission to the Ego4D AV Transcription Challenge
标题：阿凡达提交给Ego4D AV转录挑战赛
链接：https://arxiv.org/abs/2211.09966
作者：Paul Hongsuck Seo,Arsha Nagrani,Cordelia Schmid摘要：In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022. Our pipeline is based on AVATAR, a state of the art encoder-decoder model for AV-ASR that performs early fusion of spectrograms and RGB images. We describe the datasets, experimental settings and ablations. Our final method achieves a WER of 68.40 on the challenge test set, outperforming the baseline by 43.7%, and winning the challenge.
【3】 Compressing Transformer-based self-supervised models for speech processing
标题：基于压缩Transformer的语音处理自监督模型
链接：https://arxiv.org/abs/2211.09949
作者：Tzu-Quan Lin,Tsung-Huan Yang,Chun-Yao Chang,Kuang-Ming Chen,Tzu-hsun Feng,Hung-yi Lee,Hao Tang机构：†National Taiwan University, ‡University of Edinburgh备注：Submitted to ICASSP 2023摘要：尽管Transformers在自监督学习中成功应用于各种下游任务，但训练和推理的计算成本仍然是将这些模型应用于广泛设备的主要挑战。在将Transformer应用于下游任务之前，已经进行了几次单独的尝试来压缩Transformer。在这项工作中，我们旨在为孤立的结果提供上下文，研究几种常用的压缩技术，包括权重修剪、头修剪、低秩近似和知识蒸馏。我们报告了这些技术的挂钟时间、参数数量和乘累加操作的数量，绘制了压缩基于Transformer的自监督模型的前景。摘要：Despite the success of Transformers in self-supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices. Several isolated attempts have been made to compress Transformers, prior to applying them to downstream tasks. In this work, we aim to provide context for the isolated results, studying several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report wall-clock time, the number of parameters, and the number of multiply-accumulate operations for these techniques, charting the landscape of compressing Transformer-based self-supervised models.
【4】 MelHuBERT: A simplified HuBERT on Mel spectrogram
标题：MelHuBERT：一种简化的Hubert on Mel谱图
链接：https://arxiv.org/abs/2211.09944
作者：Tzu-Quan Lin,Hung-yi Lee,Hao Tang机构：†National Taiwan University, ‡University of Edinburgh备注：Submitted to ICASSP 2023摘要：自监督模型在学习可以推广到各种下游任务的语音表示方面取得了巨大成功。特别是HuBERT，与其他机器人相比，它在训练相对简单的同时，也实现了强大的性能。最初的实验设置在计算上是广泛的，阻碍了模型的再现性。也不清楚为什么做出某些设计决策，例如ad-hoc损失函数，以及这些决策是否对学习的表示有影响。我们提出了MelHuBERT，它是HuBERT的一个简化版本，以Mel谱图作为输入，大大减少了计算和内存消耗。我们研究了训练的几个方面，包括损失函数、多阶段训练和流选项。我们的结果是一个高效而性能良好的模型，可以在单个GPU上训练。摘要：Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks. HuBERT, in particular, achieves strong performance while being relatively simple in training compared to others. The original experimental setting is computationally extensive, hindering the reproducibility of the models. It is also unclear why certain design decisions are made, such as the ad-hoc loss function, and whether these decisions have an impact on the learned representations. We propose MelHuBERT, a simplified version of HuBERT that takes Mel spectrograms as input, significantly reducing computation and memory consumption. We study several aspects of training, including the loss function, multi-stage training, and streaming options. Our result is a efficient yet performant model that can be trained on a single GPU.
【5】 Multi-source Domain Adaptation for Text-independent Forensic Speaker Recognition
标题：文本无关法医说话人识别的多源域自适应算法
链接：https://arxiv.org/abs/2211.09913
作者：Zhenyu Wang,John H. L. Hansen机构：This work was supportedby the University of Texas at Dallas from the Distinguished UniversityChair in Telecommunications Engineering held by John H, Erik JonssonSchool of Engineering University of Texas at Dallas备注：IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING摘要：使说话人识别系统适应新环境是一种广泛使用的技术，用于将从大规模数据中学习到的性能良好的模型改进为特定任务的小规模数据场景。然而，以往的研究主要集中在单域自适应上，忽略了一个更实际的场景，即在取证场景中需要从多个声学域收集训练数据。由于位置/场景的不确定性以及参考和自然现场录音之间的多样性不匹配，用于法庭说话人识别的音频分析在使用多域训练数据进行模型训练时提出了独特的挑战。由于领域失配和性能损失，直接使用小规模领域特定数据来训练复杂的神经网络结构也是困难的。微调是一种常用的自适应方法，以便用从良好训练的模型初始化的权重重新训练模型。另外，本研究提出了基于域对抗训练、差异最小化和矩匹配的三种新的自适应方法，以进一步提高跨多个声学域的自适应性能。进行了一组全面的实验以证明：1）不同的声学环境确实影响说话人识别性能，这可以推进音频取证中的研究，2）域对抗训练学习对于域之间的移位也不变的区别特征，3）差异最小化自适应在多个声学域上同时实现有效性能，以及4）矩匹配自适应与动态分布对准一起还显著地提高了每个域上的说话人识别性能，特别是对于具有噪声的LENA场域。摘要：Adapting speaker recognition systems to new environments is a widely-used technique to improve a well-performing model learned from large-scale data towards a task-specific small-scale data scenarios. However, previous studies focus on single domain adaptation, which neglects a more practical scenario where training data are collected from multiple acoustic domains needed in forensic scenarios. Audio analysis for forensic speaker recognition offers unique challenges in model training with multi-domain training data due to location/scenario uncertainty and diversity mismatch between reference and naturalistic field recordings. It is also difficult to directly employ small-scale domain-specific data to train complex neural network architectures due to domain mismatch and performance loss. Fine-tuning is a commonly-used method for adaptation in order to retrain the model with weights initialized from a well-trained model. Alternatively, in this study, three novel adaptation methods based on domain adversarial training, discrepancy minimization, and moment-matching approaches are proposed to further promote adaptation performance across multiple acoustic domains. A comprehensive set of experiments are conducted to demonstrate that: 1) diverse acoustic environments do impact speaker recognition performance, which could advance research in audio forensics, 2) domain adversarial training learns the discriminative features which are also invariant to shifts between domains, 3) discrepancy-minimizing adaptation achieves effective performance simultaneously across multiple acoustic domains, and 4) moment-matching adaptation along with dynamic distribution alignment also significantly promotes speaker recognition performance on each domain, especially for the LENA-field domain with noise compared to all other systems.
【6】 Audio Anti-spoofing Using a Simple Attention Module and Joint Optimization Based on Additive Angular Margin Loss and Meta-learning
标题：基于加性角度余量损失和元学习的简单注意模块音频反欺骗联合优化
链接：https://arxiv.org/abs/2211.09898
作者：Zhenyu Wang,John H. L. Hansen机构：Center for Robust Speech Systems (CRSS), University of Texas at Dallas, TX摘要：自动说话人确认系统易受各种访问威胁的影响，这促使研究制定有效的欺骗检测系统，以充当过滤此类欺骗攻击的闸门。该研究引入了一个简单的注意力模型来推断卷积层中特征映射的三维注意力权重，然后优化能量函数来确定每个神经元的重要性。随着语音转换和语音合成技术的发展，不可见的欺骗攻击不断出现，限制了欺骗检测系统的性能。在此，我们提出了一种基于加权加性角间隔损失的二值分类联合优化方法，并结合元学习训练框架，开发了一个对各种欺骗攻击具有鲁棒性的高效系统，以增强模型泛化能力.因此，与当前最先进的系统相比，我们提出的方法提供了具有竞争力的结果，合并EER为0.99%，最小t-DCF为0.0289。摘要：Automatic speaker verification systems are vulnerable to a variety of access threats, prompting research into the formulation of effective spoofing detection systems to act as a gate to filter out such spoofing attacks. This study introduces a simple attention module to infer 3-dim attention weights for the feature map in a convolutional layer, which then optimizes an energy function to determine each neuron's importance. With the advancement of both voice conversion and speech synthesis technologies, unseen spoofing attacks are constantly emerging to limit spoofing detection system performance. Here, we propose a joint optimization approach based on the weighted additive angular margin loss for binary classification, with a meta-learning training framework to develop an efficient system that is robust to a wide range of spoofing attacks for model generalization enhancement. As a result, when compared to current state-of-the-art systems, our proposed approach delivers a competitive result with a pooled EER of 0.99% and min t-DCF of 0.0289.
【7】 Robust Vocal Quality Feature Embeddings for Dysphonic Voice Detection
标题：用于发声障碍语音检测的稳健声质特征嵌入
链接：https://arxiv.org/abs/2211.09858
作者：Jianwei Zhang,Julie Liss,Suren Jayasuriya,Visar Berisha机构：Jianwei Zhang is with School of Electrical, Arizona State University备注：This manuscript is submitted on July 06, 2022 to IEEE/ACM Transactions on Audio, Speech, and Language Processing for peer-review摘要：全世界约有1.2%的人口有发声障碍。因此，自动发音困难语音检测已经吸引了相当大的学术和临床兴趣。然而，用于自动语音评估的现有方法通常不能推广到训练条件之外或其他相关应用。本文提出了一种深度学习框架，用于生成对音质敏感且跨语料库鲁棒的声学特征嵌入。对比损失与分类损失相结合，共同训练我们的深度学习模型。在输入语音样本上使用数据扭曲方法以提高我们的方法的鲁棒性。实验结果表明，该方法不仅能获得较高的语料内和跨语料分类准确率，而且能生成对语音质量敏感的良好嵌入，并在不同语料库间具有较好的鲁棒性。我们还将我们的结果与三种基线方法在干净和三种变体的退化语料库内和跨语料库数据集上进行了比较，并证明所提出的模型始终优于基线方法。摘要：Approximately 1.2% of the world's population has impaired voice production. As a result, automatic dysphonic voice detection has attracted considerable academic and clinical interest. However, existing methods for automated voice assessment often fail to generalize outside the training conditions or to other related applications. In this paper, we propose a deep learning framework for generating acoustic feature embeddings sensitive to vocal quality and robust across different corpora. A contrastive loss is combined with a classification loss to train our deep learning model jointly. Data warping methods are used on input voice samples to improve the robustness of our method. Empirical results demonstrate that our method not only achieves high in-corpus and cross-corpus classification accuracy but also generates good embeddings sensitive to voice quality and robust across different corpora. We also compare our results against three baseline methods on clean and three variations of deteriorated in-corpus and cross-corpus datasets and demonstrate that the proposed model consistently outperforms the baseline methods.
【8】 Self-Remixing: Unsupervised Speech Separation via Separation and Remixing
标题：自混合：基于分离和混合的无监督语音分离
链接：https://arxiv.org/abs/2211.10194
作者：Kohei Saijo,Tetsuji Ogawa机构：Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan备注：Submitted to ICASSP2023, 5pages, 2figures, 2tables摘要：本文提出了一种新的自监督语音分离方法--自 再 混合 法，它以无监督的方式对预先训练好的分离模型进行改进。该方法由混洗器模块和求解器模块组成，二者通过分离和再混合过程共同成长 。具体地，混洗器首先分离观察到的混合信号，并通过混合和重新混合分离的信号来产生 伪 混合 信号 。然后，解算器分离伪混合信号，并将分离的信号重新混合回观察到的混合信号。使用 观测 到 的 混合 作为监督来训练解算器，同时通过采用解算器的移动平均值来更新洗牌器的权重，从而生成失真较少的伪混合。实验结果表明，在无监督的情况下，自混合算法在相同或更少的训练代价下，比现有的基于混合的自监督算法具有更好的性能 .在半监督域自适应中，自混合也优于基线，显示了在多个设置中的有效性 。摘要：We present Self-Remixing, a novel self-supervised speech separation method, which refines a pre-trained separation model in an unsupervised manner. The proposed method consists of a shuffler module and a solver module, and they grow together through separation and remixing processes. Specifically, the shuffler first separates observed mixtures and makes pseudo-mixtures by shuffling and remixing the separated signals. The solver then separates the pseudo-mixtures and remixes the separated signals back to the observed mixtures. The solver is trained using the observed mixtures as supervision, while the shuffler's weights are updated by taking the moving average with the solver's, generating the pseudo-mixtures with fewer distortions. Our experiments demonstrate that Self-Remixing gives better performance over existing remixing-based self-supervised methods with the same or less training costs under unsupervised setup. Self-Remixing also outperforms baselines in semi-supervised domain adaptation, showing effectiveness in multiple setups.
【9】 Self-Transriber: Few-shot Lyrics Transcription with Self-training
标题：Self-Transriber：自我训练的Few-Shot歌词转录
链接：https://arxiv.org/abs/2211.10152
作者：Xiaoxue Gao,Xianghu Yue,Haizhou Li机构：Department of Electrical and Computer Engineering, National University of Singapore, Singapore, The Chinese University of Hong Kong, Shenzhen, China摘要：目前的歌词转录方法严重依赖于带有标记数据的监督学习，但是这样的数据是稀缺的，并且演唱的人工标记是昂贵的。如何利用未标注数据并缓解有限数据的问题，一直是歌词转录研究的热点问题。我们提出了第一个半监督歌词转录范例，自我转录器，通过利用无标签数据使用自训练与噪声学生扩增。我们试图证明歌词转录的可能性与少量的标记数据。Self-Transcriber利用教师模型对未标注的演唱数据生成伪标签，并将伪标签添加到已标注的演唱数据中，用于在自训练和有监督训练损失的情况下更新学生模型。这项工作填补了监督学习和半监督学习之间的空白，并为歌词转录的Few-Shot学习打开了大门。实验结果表明，与使用149.1小时的标注数据训练的有监督的歌词转录方法相比，我们的方法仅使用12.7小时的标注数据就获得了有竞争力的性能.摘要：The current lyrics transcription approaches heavily rely on supervised learning with labeled data, but such data are scarce and manual labeling of singing is expensive. How to benefit from unlabeled data and alleviate limited data problem have not been explored for lyrics transcription. We propose the first semi-supervised lyrics transcription paradigm, Self-Transcriber, by leveraging on unlabeled data using self-training with noisy student augmentation. We attempt to demonstrate the possibility of lyrics transcription with a few amount of labeled data. Self-Transcriber generates pseudo labels of the unlabeled singing using teacher model, and augments pseudo-labels to the labeled data for student model update with both self-training and supervised training losses. This work closes the gap between supervised and semi-supervised learning as well as opens doors for few-shot learning of lyrics transcription. Our experiments show that our approach using only 12.7 hours of labeled data achieves competitive performance compared with the supervised approaches trained on 149.1 hours of labeled data for lyrics transcription.
【10】 Exploring WavLM on Speech Enhancement
标题：WavLM在语音增强中的应用研究
链接：https://arxiv.org/abs/2211.09988
作者：Hyungchan Song,Sanyuan Chen,Zhuo Chen,Yu Wu,Takuya Yoshioka,Min Tang,Jong Won Shin,Shujie Liu机构：Gwanju Institute of Science and Technology, Republic of Korea, Microsoft, China, Microsoft, USA备注：Accepted by IEEE SLT 2022摘要：近年 来 ， 基于 自 监督 学习 的 端 到 端 语音 编码 方法 取得 了 巨大 的 成功 ， 引起 了 人们 的 极大 兴趣 。特别 是 ， WavLM 在 各种 语音 处理 任务 上 表现 出 了 最 先进 的 性能 。为了 更 好 地 理解 自 监督 学习 模型 在 语音 增强 中 的 有效 性 ， 本文 将 WavLM 与 两 个 高 质量 语音 增强 系统 相 结合 ， 在 三 种 资源 条件 下 设计 并 进行 了 一 系列 实验 。同时 ， 我们 提出 了 一 种 基于 回归 的 WavLM 训练 目标 和 一 种 混合 噪声 的 数据 配置 ， 以 进一步 提高 下游 增强 性能 。在 DNS 挑战 数据 集 和 仿真 数据 集 上 的 实验 结果 表明 ， WavLM 算法 在 语音 质量 和 语音 识别 准确 率 方面 均 优于 传统 的 语音 增强 算法 ， 尤其 是 在 低 微调 资源 的 情况 下 。对于 高 微调 资源 条件 ， 仅 字 错误 率 得到 实质 性 改善 。摘要：There is a surge in interest in self-supervised learning approaches for end-to-end speech encoding in recent years as they have achieved great success. Especially, WavLM showed state-of-the-art performance on various speech processing tasks. To better understand the efficacy of self-supervised learning models for speech enhancement, in this work, we design and conduct a series of experiments with three resource conditions by combining WavLM and two high-quality speech enhancement systems. Also, we propose a regression-based WavLM training objective and a noise-mixing data configuration to further boost the downstream enhancement performance. The experiments on the DNS challenge dataset and a simulation dataset show that the WavLM benefits the speech enhancement task in terms of both speech quality and speech recognition accuracy, especially for low fine-tuning resources. For the high fine-tuning resource condition, only the word error rate is substantially improved.
【11】 A Persian ASR-based SER: Modification of Sharif Emotional Speech Database and Investigation of Persian Text Corpora
标题：基于波斯语ASR的SER：Sharif情感语料库的修改和波斯语文本语料库的研究
链接：https://arxiv.org/abs/2211.09956
作者：Ali Yazdani,Yasser Shekofteh机构：Shahid Beheshti University, Tehran, Iran备注：7 pages, 4 figures, 8 tables摘要：语音情感识别（SER）是人类理解情境和与他人进行交互的基本感知方法之一，因此，近年来人们尝试在人机通信系统中加入情感识别的功能。由于SER过程依赖于标记数据，因此数据库对于SER过程至关重要。不完整、低质量或有缺陷的数据可能导致不准确的预测。本文利用自动语音识别系统（ASR）和从可访问的波斯语文本语料库中获取的波斯语语言模型，对Sharif情感语音数据库（ShEMO）中的不一致性进行了修正。我们还引入了一个基于波斯语/波斯语ASR的SER系统，该系统使用ASR输出的语言特征和基于深度学习的模型。摘要：Speech Emotion Recognition (SER) is one of the essential perceptual methods of humans in understanding the situation and how to interact with others, therefore, in recent years, it has been tried to add the ability to recognize emotions to human-machine communication systems. Since the SER process relies on labeled data, databases are essential for it. Incomplete, low-quality or defective data may lead to inaccurate predictions. In this paper, we fixed the inconsistencies in Sharif Emotional Speech Database (ShEMO), as a Persian database, by using an Automatic Speech Recognition (ASR) system and investigating the effect of Farsi language models obtained from accessible Persian text corpora. We also introduced a Persian/Farsi ASR-based SER system that uses linguistic features of the ASR outputs and Deep Learning-based models.
【1】 Self-Remixing: Unsupervised Speech Separation via Separation and Remixing
标题：自混合：基于分离和混合的无监督语音分离
链接：https://arxiv.org/abs/2211.10194
* 与cs.SD语音【8】为同一篇
作者：Kohei Saijo,Tetsuji Ogawa机构：Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan备注：Submitted to ICASSP2023, 5pages, 2figures, 2tables摘要：本文提出了一种新的自监督语音分离方法--自 再 混合 法，它以无监督的方式对预先训练好的分离模型进行改进。该方法由混洗器模块和求解器模块组成，二者通过分离和再混合过程共同成长 。具体地，混洗器首先分离观察到的混合信号，并通过混合和重新混合分离的信号来产生 伪 混合 信号 。然后，解算器分离伪混合信号，并将分离的信号重新混合回观察到的混合信号。使用 观测 到 的 混合 作为监督来训练解算器，同时通过采用解算器的移动平均值来更新洗牌器的权重，从而生成失真较少的伪混合。实验结果表明，在无监督的情况下，自混合算法在相同或更少的训练代价下，比现有的基于混合的自监督算法具有更好的性能 .在半监督域自适应中，自混合也优于基线，显示了在多个设置中的有效性 。摘要：We present Self-Remixing, a novel self-supervised speech separation method, which refines a pre-trained separation model in an unsupervised manner. The proposed method consists of a shuffler module and a solver module, and they grow together through separation and remixing processes. Specifically, the shuffler first separates observed mixtures and makes pseudo-mixtures by shuffling and remixing the separated signals. The solver then separates the pseudo-mixtures and remixes the separated signals back to the observed mixtures. The solver is trained using the observed mixtures as supervision, while the shuffler's weights are updated by taking the moving average with the solver's, generating the pseudo-mixtures with fewer distortions. Our experiments demonstrate that Self-Remixing gives better performance over existing remixing-based self-supervised methods with the same or less training costs under unsupervised setup. Self-Remixing also outperforms baselines in semi-supervised domain adaptation, showing effectiveness in multiple setups.
【2】 Self-Transriber: Few-shot Lyrics Transcription with Self-training
标题：Self-Transriber：自我训练的Few-Shot歌词转录
链接：https://arxiv.org/abs/2211.10152
* 与cs.SD语音【9】为同一篇
作者：Xiaoxue Gao,Xianghu Yue,Haizhou Li机构：Department of Electrical and Computer Engineering, National University of Singapore, Singapore, The Chinese University of Hong Kong, Shenzhen, China摘要：目前 的 歌词 转录 方法 严重 依赖 于 带有 标记 数据 的 监督 学习 ， 但是 这样 的 数据 是 稀缺 的 ， 并且 演唱 的 人工 标记 是 昂贵 的 。如何 利用 未 标注 数据 并 缓解 有限 数据 的 问题 ， 一直 是 歌词 转录 研究 的 热点 问题 。我们 提出 了 第 一 个 半 监督 歌词 转录 范例 ， 自我 转录 器 ， 通过 利用 无 标签 数据 使用 自 训练 与 噪声 学生 扩增 。我们 试图 证明 歌词 转录 的 可能 性 与 少量 的 标记 数据 。Self-Transcriber 利用 教师 模型 对 未 标注 的 演唱 数据 生成 伪 标签 ， 并 将 伪 标签 添加 到 已 标注 的 演唱 数据 中 ， 用于 在 自 训练 和 有 监督 训练 损失 的 情况 下 更新 学生 模型 。这项 工作 填补 了 监督 学习 和 半 监督 学习 之间 的 空白 ， 并 为 歌词 转录 的 少 镜头 学习 打开 了 大门 。实验 结果 表明 ， 与 使用 149.1 小时 的 标注 数据 训练 的 有 监督 的 歌词 转录 方法 相比 ， 我们 的 方法 仅 使用 12.7 小时 的 标注 数据 就 获得 了 有 竞争 力 的 性能 .摘要：The current lyrics transcription approaches heavily rely on supervised learning with labeled data, but such data are scarce and manual labeling of singing is expensive. How to benefit from unlabeled data and alleviate limited data problem have not been explored for lyrics transcription. We propose the first semi-supervised lyrics transcription paradigm, Self-Transcriber, by leveraging on unlabeled data using self-training with noisy student augmentation. We attempt to demonstrate the possibility of lyrics transcription with a few amount of labeled data. Self-Transcriber generates pseudo labels of the unlabeled singing using teacher model, and augments pseudo-labels to the labeled data for student model update with both self-training and supervised training losses. This work closes the gap between supervised and semi-supervised learning as well as opens doors for few-shot learning of lyrics transcription. Our experiments show that our approach using only 12.7 hours of labeled data achieves competitive performance compared with the supervised approaches trained on 149.1 hours of labeled data for lyrics transcription.
【3】 Exploring WavLM on Speech Enhancement
标题：WavLM在语音增强中的应用研究
链接：https://arxiv.org/abs/2211.09988
* 与cs.SD语音【10】为同一篇
作者：Hyungchan Song,Sanyuan Chen,Zhuo Chen,Yu Wu,Takuya Yoshioka,Min Tang,Jong Won Shin,Shujie Liu机构：Gwanju Institute of Science and Technology, Republic of Korea, Microsoft, China, Microsoft, USA备注：Accepted by IEEE SLT 2022摘要：近年来，基于自监督学习的端到端语音编码方法取得了巨大的成功，引起了人们的极大兴趣。特别是，WavLM在各种语音处理任务上表现出了最先进的性能。为了更好地理解自监督学习模型在语音增强中的有效性，本文将WavLM与两个高质量语音增强系统相结合，在三种资源条件下设计并进行了一系列实验。同时，我们提出了一种基于回归的WavLM训练目标和一种混合噪声的数据配置，以进一步提高下游增强性能。在DNS挑战数据集和仿真数据集上的实验结果表明，WavLM算法在语音质量和语音识别准确率方面均优于传统的语音增强算法，尤其是在低微调资源的情况下。对于高微调资源条件，仅字错误率得到实质性改善。摘要：There is a surge in interest in self-supervised learning approaches for end-to-end speech encoding in recent years as they have achieved great success. Especially, WavLM showed state-of-the-art performance on various speech processing tasks. To better understand the efficacy of self-supervised learning models for speech enhancement, in this work, we design and conduct a series of experiments with three resource conditions by combining WavLM and two high-quality speech enhancement systems. Also, we propose a regression-based WavLM training objective and a noise-mixing data configuration to further boost the downstream enhancement performance. The experiments on the DNS challenge dataset and a simulation dataset show that the WavLM benefits the speech enhancement task in terms of both speech quality and speech recognition accuracy, especially for low fine-tuning resources. For the high fine-tuning resource condition, only the word error rate is substantially improved.
【4】 A Persian ASR-based SER: Modification of Sharif Emotional Speech Database and Investigation of Persian Text Corpora
标题：基于波斯语ASR的SER：Sharif情感语料库的修改和波斯语文本语料库的研究
链接：https://arxiv.org/abs/2211.09956
* 与cs.SD语音【11】为同一篇
作者：Ali Yazdani,Yasser Shekofteh机构：Shahid Beheshti University, Tehran, Iran备注：7 pages, 4 figures, 8 tables摘要：语音情感识别（SER）是人类理解情境和与他人进行交互的基本感知方法之一，因此，近年来人们尝试在人机通信系统中加入情感识别的功能。由于SER过程依赖于标记数据，因此数据库对于SER过程至关重要。不完整、低质量或有缺陷的数据可能导致不准确的预测。本文利用自动语音识别系统（ASR）和从可访问的波斯语文本语料库中获取的波斯语语言模型，对Sharif情感语音数据库（ShEMO）中的不一致性进行了修正。我们还引入了一个基于波斯语/波斯语ASR的SER系统，该系统使用ASR输出的语言特征和基于深度学习的模型。摘要：Speech Emotion Recognition (SER) is one of the essential perceptual methods of humans in understanding the situation and how to interact with others, therefore, in recent years, it has been tried to add the ability to recognize emotions to human-machine communication systems. Since the SER process relies on labeled data, databases are essential for it. Incomplete, low-quality or defective data may lead to inaccurate predictions. In this paper, we fixed the inconsistencies in Sharif Emotional Speech Database (ShEMO), as a Persian database, by using an Automatic Speech Recognition (ASR) system and investigating the effect of Farsi language models obtained from accessible Persian text corpora. We also introduced a Persian/Farsi ASR-based SER system that uses linguistic features of the ASR outputs and Deep Learning-based models.
【5】 Speaker Overlap-aware Neural Diarization for Multi-party Meeting Analysis
标题：多方会议分析中的说话人重叠感知神经二值化
链接：https://arxiv.org/abs/2211.10243
* 与cs.SD语音【1】为同一篇
作者：Zhihao Du,Shiliang Zhang,Siqi Zheng,Zhijie Yan机构：Speech Lab, Alibaba Group, China备注：Accepted by EMNLP 2022摘要：近年来，聚类和神经网络模型的混合系统已经成功地应用于多方会议分析。然而，现有的重叠说话人识别模型往往将重叠说话人识别问题视为多标签分类问题，没有很好地考虑说话人的相关性和重叠性。为了克服这一缺点，提出了一种基于幂集编码（PSE）的重叠说话人识别问题，并将其转化为单标签预测问题。通过该公式化，说话者依赖性和重叠可以被显式地建模。为了充分利用这一公式，我们进一步提出了说话人重叠感知神经diarization（SOND）模型，该模型由上下文无关（CI）评分器、上下文相关（CD）评分器和说话人组合网络（SCN）组成，其中CI评分器用于建模说话人的全局可辨别性，CD评分器用于建模说话人的局部可辨别性，SCN用于组合和重新分配说话人活动。实验结果表明，与基于目标说话人语音活动检测的方法 相比，该方法的性能得到了进一步的提高，相对二值化误差降低了6.30%。摘要：Recently, hybrid systems of clustering and neural diarization models have been successfully applied in multi-party meeting analysis. However, current models always treat overlapped speaker diarization as a multi-label classification problem, where speaker dependency and overlaps are not well considered. To overcome the disadvantages, we reformulate overlapped speaker diarization task as a single-label prediction problem via the proposed power set encoding (PSE). Through this formulation, speaker dependency and overlaps can be explicitly modeled. To fully leverage this formulation, we further propose the speaker overlap-aware neural diarization (SOND) model, which consists of a context-independent (CI) scorer to model global speaker discriminability, a context-dependent scorer (CD) to model local discriminability, and a speaker combining network (SCN) to combine and reassign speaker activities. Experimental results show that using the proposed formulation can outperform the state-of-the-art methods based on target speaker voice activity detection, and the performance can be further improved with SOND, resulting in a 6.30% relative diarization error reduction.
【6】 AVATAR submission to the Ego4D AV Transcription Challenge
标题：阿凡达提交给Ego4D AV转录挑战赛
链接：https://arxiv.org/abs/2211.09966
* 与cs.SD语音【2】为同一篇
作者：Paul Hongsuck Seo,Arsha Nagrani,Cordelia Schmid摘要：在这份报告中，我们描述了我们提交的2022年Ego 4D视听（AV）语音转录挑战赛。我们的流水线基于AVATAR，这是一种用于AV-ASR的最先进的编码器-解码器模型，可执行光谱图和RGB图像的早期融合。我们描述了数据集、实验设置和消融。我们的最终方法在挑战测试集上获得了68.40的WER，比基线高出43.7%，并赢得了挑战。摘要：In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022. Our pipeline is based on AVATAR, a state of the art encoder-decoder model for AV-ASR that performs early fusion of spectrograms and RGB images. We describe the datasets, experimental settings and ablations. Our final method achieves a WER of 68.40 on the challenge test set, outperforming the baseline by 43.7%, and winning the challenge.
【7】 Compressing Transformer-based self-supervised models for speech processing
标题：基于压缩Transformer的语音处理自监督模型
链接：https://arxiv.org/abs/2211.09949
* 与cs.SD语音【3】为同一篇
作者：Tzu-Quan Lin,Tsung-Huan Yang,Chun-Yao Chang,Kuang-Ming Chen,Tzu-hsun Feng,Hung-yi Lee,Hao Tang机构：†National Taiwan University, ‡University of Edinburgh备注：Submitted to ICASSP 2023摘要：尽管Transformers在自监督学习中成功应用于各种下游任务，但训练和推理的计算成本仍然是将这些模型应用于广泛设备的主要挑战。在将Transformer应用于下游任务之前，已经进行了几次单独的尝试来压缩Transformer。在这项工作中，我们旨在为孤立的结果提供上下文，研究几种常用的压缩技术，包括权重修剪、头修剪、低秩近似和知识蒸馏。我们报告了这些技术的挂钟时间、参数数量和乘累加操作的数量，绘制了压缩基于Transformer的自监督模型的前景。摘要：Despite the success of Transformers in self-supervised learning with applications to various downstream tasks, the computational cost of training and inference remains a major challenge for applying these models to a wide spectrum of devices. Several isolated attempts have been made to compress Transformers, prior to applying them to downstream tasks. In this work, we aim to provide context for the isolated results, studying several commonly used compression techniques, including weight pruning, head pruning, low-rank approximation, and knowledge distillation. We report wall-clock time, the number of parameters, and the number of multiply-accumulate operations for these techniques, charting the landscape of compressing Transformer-based self-supervised models.
【8】 MelHuBERT: A simplified HuBERT on Mel spectrogram
标题：MelHuBERT：一种简化的Hubert on Mel谱图
链接：https://arxiv.org/abs/2211.09944
* 与cs.SD语音【4】为同一篇
作者：Tzu-Quan Lin,Hung-yi Lee,Hao Tang机构：†National Taiwan University, ‡University of Edinburgh备注：Submitted to ICASSP 2023摘要：自监督模型在学习可以推广到各种下游任务的语音表示方面取得了巨大成功。特别是HuBERT，与其他机器人相比，它在训练相对简单的同时，也实现了强大的性能。最初的实验设置在计算上是广泛的，阻碍了模型的再现性。也不清楚为什么做出某些设计决策，例如ad-hoc损失函数，以及这些决策是否对学习的表示有影响。我们提出了MelHuBERT，它是HuBERT的一个简化版本，以Mel谱图作为输入，大大减少了计算和内存消耗。我们研究了训练的几个方面，包括损失函数、多阶段训练和流选项。我们的结果是一个高效而性能良好的模型，可以在单个GPU上训练。摘要：Self-supervised models have had great success in learning speech representations that can generalize to various downstream tasks. HuBERT, in particular, achieves strong performance while being relatively simple in training compared to others. The original experimental setting is computationally extensive, hindering the reproducibility of the models. It is also unclear why certain design decisions are made, such as the ad-hoc loss function, and whether these decisions have an impact on the learned representations. We propose MelHuBERT, a simplified version of HuBERT that takes Mel spectrograms as input, significantly reducing computation and memory consumption. We study several aspects of training, including the loss function, multi-stage training, and streaming options. Our result is a efficient yet performant model that can be trained on a single GPU.
【9】 Multi-source Domain Adaptation for Text-independent Forensic Speaker Recognition
标题：文本无关法医说话人识别的多源域自适应算法
链接：https://arxiv.org/abs/2211.09913
* 与cs.SD语音【5】为同一篇
作者：Zhenyu Wang,John H. L. Hansen机构：This work was supportedby the University of Texas at Dallas from the Distinguished UniversityChair in Telecommunications Engineering held by John H, Erik JonssonSchool of Engineering University of Texas at Dallas备注：IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING摘要：使说话人识别系统适应新环境是一种广泛使用的技术，用于将从大规模数据中学习到的性能良好的模型改进为特定任务的小规模数据场景。然而，以往的研究主要集中在单域自适应上，忽略了一个更实际的场景，即在取证场景中需要从多个声学域收集训练数据。由于位置/场景的不确定性以及参考和自然现场录音之间的多样性不匹配，用于法庭说话人识别的音频分析在使用多域训练数据进行模型训练时提出了独特的挑战。由于领域失配和性能损失，直接使用小规模领域特定数据来训练复杂的神经网络结构也是困难的。微调是一种常用的自适应方法，以便用从良好训练的模型初始化的权重重新训练模型。另外，本研究提出了基于域对抗训练、差异最小化和矩匹配的三种新的自适应方法，以进一步提高跨多个声学域的自适应性能。进行了一组全面的实验以证明：1）不同的声学环境确实影响说话人识别性能，这可以推进音频取证中的研究，2）域对抗训练学习对于域之间的移位也不变的区别特征，3）差异最小化自适应在多个声学域上同时实现有效性能，以及4）矩匹配自适应与动态分布对准一起还显著地提高了每个域上的说话人识别性能，特别是对于具有噪声的LENA场域。摘要：Adapting speaker recognition systems to new environments is a widely-used technique to improve a well-performing model learned from large-scale data towards a task-specific small-scale data scenarios. However, previous studies focus on single domain adaptation, which neglects a more practical scenario where training data are collected from multiple acoustic domains needed in forensic scenarios. Audio analysis for forensic speaker recognition offers unique challenges in model training with multi-domain training data due to location/scenario uncertainty and diversity mismatch between reference and naturalistic field recordings. It is also difficult to directly employ small-scale domain-specific data to train complex neural network architectures due to domain mismatch and performance loss. Fine-tuning is a commonly-used method for adaptation in order to retrain the model with weights initialized from a well-trained model. Alternatively, in this study, three novel adaptation methods based on domain adversarial training, discrepancy minimization, and moment-matching approaches are proposed to further promote adaptation performance across multiple acoustic domains. A comprehensive set of experiments are conducted to demonstrate that: 1) diverse acoustic environments do impact speaker recognition performance, which could advance research in audio forensics, 2) domain adversarial training learns the discriminative features which are also invariant to shifts between domains, 3) discrepancy-minimizing adaptation achieves effective performance simultaneously across multiple acoustic domains, and 4) moment-matching adaptation along with dynamic distribution alignment also significantly promotes speaker recognition performance on each domain, especially for the LENA-field domain with noise compared to all other systems.
【10】 Audio Anti-spoofing Using a Simple Attention Module and Joint Optimization Based on Additive Angular Margin Loss and Meta-learning
标题：基于加性角度余量损失和元学习的简单注意模块音频反欺骗联合优化
链接：https://arxiv.org/abs/2211.09898
* 与cs.SD语音【6】为同一篇
作者：Zhenyu Wang,John H. L. Hansen机构：Center for Robust Speech Systems (CRSS), University of Texas at Dallas, TX摘要：自动说话人确认系统易受各种访问威胁的影响，这促使研究制定有效的欺骗检测系统，以充当过滤此类欺骗攻击的闸门。该研究引入了一个简单的注意力模型来推断卷积层中特征映射的三维注意力权重，然后优化能量函数来确定每个神经元的重要性。随着语音转换和语音合成技术的发展，不可见的欺骗攻击不断出现，限制了欺骗检测系统的性能。在此，我们提出了一种基于加权加性角间隔损失的二值分类联合优化方法，并结合元学习训练框架，开发了一个对各种欺骗攻击具有鲁棒性的高效系统，以增强模型泛化能力.因此，与当前最先进的系统相比，我们提出的方法提供了具有竞争力的结果，合并EER为0.99%，最小t-DCF为0.0289。摘要：Automatic speaker verification systems are vulnerable to a variety of access threats, prompting research into the formulation of effective spoofing detection systems to act as a gate to filter out such spoofing attacks. This study introduces a simple attention module to infer 3-dim attention weights for the feature map in a convolutional layer, which then optimizes an energy function to determine each neuron's importance. With the advancement of both voice conversion and speech synthesis technologies, unseen spoofing attacks are constantly emerging to limit spoofing detection system performance. Here, we propose a joint optimization approach based on the weighted additive angular margin loss for binary classification, with a meta-learning training framework to develop an efficient system that is robust to a wide range of spoofing attacks for model generalization enhancement. As a result, when compared to current state-of-the-art systems, our proposed approach delivers a competitive result with a pooled EER of 0.99% and min t-DCF of 0.0289.
【11】 Robust Vocal Quality Feature Embeddings for Dysphonic Voice Detection
标题：用于发声障碍语音检测的稳健声质特征嵌入
链接：https://arxiv.org/abs/2211.09858
* 与cs.SD语音【7】为同一篇
作者：Jianwei Zhang,Julie Liss,Suren Jayasuriya,Visar Berisha机构：Jianwei Zhang is with School of Electrical, Arizona State University备注：This manuscript is submitted on July 06, 2022 to IEEE/ACM Transactions on Audio, Speech, and Language Processing for peer-review摘要：全世界约有1.2%的人口有发声障碍。因此，自动发音困难语音检测已经吸引了相当大的学术和临床兴趣。然而，用于自动语音评估的现有方法通常不能推广到训练条件之外或其他相关应用。本文提出了一种深度学习框架，用于生成对音质敏感且跨语料库鲁棒的声学特征嵌入。对比损失与分类损失相结合，共同训练我们的深度学习模型。在输入语音样本上使用数据扭曲方法以提高我们的方法的鲁棒性。实验结果表明，该方法不仅能获得较高的语料内和跨语料分类准确率，而且能生成对语音质量敏感的良好嵌入，并在不同语料库间具有较好的鲁棒性。我们还将我们的结果与三种基线方法在干净和三种变体的退化语料库内和跨语料库数据集上进行了比较，并证明所提出的模型始终优于基线方法。摘要：Approximately 1.2% of the world's population has impaired voice production. As a result, automatic dysphonic voice detection has attracted considerable academic and clinical interest. However, existing methods for automated voice assessment often fail to generalize outside the training conditions or to other related applications. In this paper, we propose a deep learning framework for generating acoustic feature embeddings sensitive to vocal quality and robust across different corpora. A contrastive loss is combined with a classification loss to train our deep learning model jointly. Data warping methods are used on input voice samples to improve the robustness of our method. Empirical results demonstrate that our method not only achieves high in-corpus and cross-corpus classification accuracy but also generates good embeddings sensitive to voice quality and robust across different corpora. We also compare our results against three baseline methods on clean and three variations of deteriorated in-corpus and cross-corpus datasets and demonstrate that the proposed model consistently outperforms the baseline methods.
机器翻译，仅供参考
每日语音速递