今日论文合集：cs.SD语音8篇，eess.AS音频处理9篇。【1】 Improving Massively Multilingual ASR With Auxiliary CTC Objectives
标题：利用辅助CTC目标改进大规模多语言ASR
链接：https://arxiv.org/abs/2302.12829
作者：William Chen,Brian Yan,Jiatong Shi,Yifan Peng,Soumi Maiti,Shinji Watanabe机构：Language Technologies Institute, Carnegie Mellon University, USA备注：5 pages, 1 figure, accepted at ICASSP 2023摘要：多语言自动语音识别（ASR）模型已经将语音技术的可用性扩展到各种语言。然而，考虑到这些模型必须处理多少种语言，理解它们在不同语言之间的不平衡性能的关键是检查模型是否真的知道它应该转录哪种语言。本文介绍了我们在FLEURS（一个包含102种语言的开放ASR基准测试）上通过将整个模型置于语言身份（LID）的条件下来提高性能的工作。我们研究了最近的联结主义时间分类（CTC）研究所启发的技术，以帮助模型处理大量的语言，条件是辅助任务的LID预测。实验结果证明了该方法相对于标准CTC/Attention-based混合模型的有效性。此外，我们最先进的系统使用具有Conformer架构的自监督模型，相对于FLEURS的先前工作结果提高了28.4% CER。训练模型是可重现的配方，可在https://github.com/espnet/espnet/tree/master/egs 2/fleurs/asr 1上获得。摘要：Multilingual Automatic Speech Recognition (ASR) models have extended the usability of speech technologies to a wide variety of languages. With how many languages these models have to handle, however, a key to understanding their imbalanced performance across different languages is to examine if the model actually knows which language it should transcribe. In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID). We investigate techniques inspired from recent Connectionist Temporal Classification (CTC) studies to help the model handle the large number of languages, conditioning on the LID predictions of auxiliary tasks. Our experimental results demonstrate the effectiveness of our technique over standard CTC/Attention-based hybrid mod- els. Furthermore, our state-of-the-art systems using self-supervised models with the Conformer architecture improve over the results of prior work on FLEURS by a relative 28.4% CER. Trained models are reproducible recipes are available at https://github.com/ espnet/espnet/tree/master/egs2/fleurs/asr1.【2】 Multi-task learning of speech and speaker recognition
标题：语音多任务学习与说话人识别
链接：https://arxiv.org/abs/2302.12773
作者：Nik Vaessen,David A. van Leeuwen机构：Institute for Computing and Information Sciences, Radboud University, Nijmegen, The, Netherlands备注：submitted to ICML 2022摘要：我们研究两个正交语音技术任务的多任务学习：语音和说话人识别。我们使用wav2vec2作为基础架构，它有两个特定于任务的输出头。我们实验了不同的方法来混合说话人和语音信息在输出嵌入序列中，并提出了一个简单的动态方法来平衡语音和说话人识别损失函数。我们的多任务学习网络可以生成共享的说话人和语音嵌入，并在LibriSpeech和VoxCeleb测试集上进行评估，其性能可与单独的单任务模型相媲美。代码可从www.example.com获取。https://github.com/nikvaessen/2022-repo-mt-w2v2.摘要：We study multi-task learning for two orthogonal speech technology tasks: speech and speaker recognition. We use wav2vec2 as a base architecture with two task-specific output heads. We experiment with different methods to mix speaker and speech information in the output embedding sequence, and propose a simple dynamic approach to balance the speech and speaker recognition loss functions. Our multi-task learning networks can produce a shared speaker and speech embedding, which are evaluated on the LibriSpeech and VoxCeleb test sets, and achieve a performance comparable to separate single-task models. Code is available at https://github.com/nikvaessen/2022-repo-mt-w2v2.【3】 Supervised Hierarchical Clustering using Graph Neural Networks for Speaker Diarization
标题：基于图神经网络的有监督层次聚类说话人二值化
链接：https://arxiv.org/abs/2302.12716
作者：Prachi Singh,Amrit Kaul,Sriram Ganapathy机构：LEAP Lab, Electrical Engineering, Indian Institute of Science,Bangalore.备注：5 pages including references. Accepted in ICASSP 2023摘要：传统的说话人二值化方法包括将音频文件开窗为短片段以提取说话人嵌入，随后对嵌入进行无监督聚类。这种多步骤方法为每个片段生成发言者分配。本文提出了一种新的有监督的层次gRaph聚类算法（SHARC）用于说话人识别，该算法引入了一个层次结构，并利用图神经网络（GNN）进行有监督聚类。该监督允许模型更新表示并直接提高聚类性能，从而实现用于diarization的单步方法。在所提出的工作中，输入片段嵌入被视为图的节点，其边权重对应于节点之间的相似性得分。我们还提出了一种方法，联合更新嵌入提取器和GNN模型，以执行端到端说话人二值化（E2 E-SHARC）。在推理过程中，使用节点密度和边存在概率执行层次聚类以合并片段直到收敛。在二值化实验中，我们在AMI和Voxconverse等基准数据集上验证了E2 E-SHARC方法分别比基准系统提高了53%和44%。摘要：Conventional methods for speaker diarization involve windowing an audio file into short segments to extract speaker embeddings, followed by an unsupervised clustering of the embeddings. This multi-step approach generates speaker assignments for each segment. In this paper, we propose a novel Supervised HierArchical gRaph Clustering algorithm (SHARC) for speaker diarization where we introduce a hierarchical structure using Graph Neural Network (GNN) to perform supervised clustering. The supervision allows the model to update the representations and directly improve the clustering performance, thus enabling a single-step approach for diarization. In the proposed work, the input segment embeddings are treated as nodes of a graph with the edge weights corresponding to the similarity scores between the nodes. We also propose an approach to jointly update the embedding extractor and the GNN model to perform end-to-end speaker diarization (E2E-SHARC). During inference, the hierarchical clustering is performed using node densities and edge existence probabilities to merge the segments until convergence. In the diarization experiments, we illustrate that the proposed E2E-SHARC approach achieves 53% and 44% relative improvements over the baseline systems on benchmark datasets like AMI and Voxconverse, respectively.【4】 Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion
标题：捕捉你和我可以：根据声音转换揭示源声纹
链接：https://arxiv.org/abs/2302.12434
作者：Jiangyi Deng,Yanjiao Chen,Yinan Zhong,Qianhao Miao,Xueluan Gong,Wenyuan Xu机构：Zhejiang University,Wuhan University备注：Accepted by USENIX Security Symposium 2023. Please cite this paper as "Jiangyi Deng, Yanjiao Chen, Yinan Zhong, Qianhao Miao, Xueluan Gong, Wenyuan Xu. Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion. In 32nd USENIX Security Symposium (USENIX Security 23)."摘要：语音转换（VC）技术可能被恶意方滥用以将其音频转换成听起来像目标说话者，使得人类或说话者验证/识别系统难以追踪源说话者。本文首次尝试从高可信度的语音转换方法合成的音频中恢复源声纹。然而，从转换后的音频中揭示源说话者的特征是具有挑战性的，因为语音转换操作旨在解开原始特征并注入目标说话者的特征。为了实现我们的目标，我们开发了一个表示学习模型Revelio，它学习从转换的音频样本中有效地提取源说话人的声纹。我们为Revelio配备了精心设计的差分校正算法，通过去除与目标说话人声纹平行的表征分量来消除目标说话人的影响。我们已经进行了大量的实验来评估Revelio从VQVC、VQVC+、AGAIN和BNE转换的音频中恢复声纹的能力。实验结果表明，Revelio能够重建出说话人确认和识别系统能够追踪到源说话人的声纹。Revelio在性别转换、看不见的语言和电话网络方面也表现出强劲的表现。摘要：Voice conversion (VC) techniques can be abused by malicious parties to transform their audios to sound like a target speaker, making it hard for a human being or a speaker verification/identification system to trace the source speaker. In this paper, we make the first attempt to restore the source voiceprint from audios synthesized by voice conversion methods with high credit. However, unveiling the features of the source speaker from a converted audio is challenging since the voice conversion operation intends to disentangle the original features and infuse the features of the target speaker. To fulfill our goal, we develop Revelio, a representation learning model, which learns to effectively extract the voiceprint of the source speaker from converted audio samples. We equip Revelio with a carefully-designed differential rectification algorithm to eliminate the influence of the target speaker by removing the representation component that is parallel to the voiceprint of the target speaker. We have conducted extensive experiments to evaluate the capability of Revelio in restoring voiceprint from audios converted by VQVC, VQVC+, AGAIN, and BNE. The experiments verify that Revelio is able to rebuild voiceprints that can be traced to the source speaker by speaker verification and identification systems. Revelio also exhibits robust performance under inter-gender conversion, unseen languages, and telephony networks.【5】 Data leakage in cross-modal retrieval training: A case study
标题：跨通道检索训练中的数据泄露：案例研究
链接：https://arxiv.org/abs/2302.12258
作者：Benno Weck,Xavier Serra机构：Huawei Technologies, Munich Research Center, Germany, Universitat Pompeu Fabra, Music Technology Group, Spain备注：5 pages. Accepted at ICASSP2023摘要：近年来基于文本的音频检索技术的发展在很大程度上得益于合适数据集的发布。由于手动创建此类数据集是一项费力的任务，因此从在线资源获取数据可能是创建大规模数据集的廉价解决方案。我们研究了最近提出的SoundDesc基准数据集，它是从BBC声音效果网页自动获取的。在我们的分析中，我们发现SoundDesc包含几个重复项，导致训练数据泄漏到评估数据中。这种数据泄漏最终会导致以前的基准测试中对检索性能的估计过于乐观。我们为在线提供的数据集提出了新的训练、验证和测试拆分。为了避免测试数据的弱污染，我们将共享类似录制设置的音频文件汇集在一起。在我们的实验中，我们发现新的分割是一个更具挑战性的基准。摘要：The recent progress in text-based audio retrieval was largely propelled by the release of suitable datasets. Since the manual creation of such datasets is a laborious task, obtaining data from online resources can be a cheap solution to create large-scale datasets. We study the recently proposed SoundDesc benchmark dataset, which was automatically sourced from the BBC Sound Effects web page. In our analysis, we find that SoundDesc contains several duplicates that cause leakage of training data to the evaluation data. This data leakage ultimately leads to overly optimistic retrieval performance estimates in previous benchmarks. We propose new training, validation, and testing splits for the dataset that we make available online. To avoid weak contamination of the test data, we pool audio files that share similar recording setups. In our experiments, we find that the new splits serve as a more challenging benchmark.【6】 Ensemble knowledge distillation of self-supervised speech models
标题：自监督语音模型的集成知识提取
链接：https://arxiv.org/abs/2302.12757
作者：Kuan-Po Huang,Tzu-hsun Feng,Yu-Kuan Fu,Tsu-Yuan Hsu,Po-Chieh Yen,Wei-Cheng Tseng,Kai-Wei Chang,Hung-yi Lee机构：allCollege of Electrical Engineering and Computer Science, National Taiwan University, †ASUS Intelligent Cloud Services备注：Accepted by ICASSP 2023摘要：近年来，精炼自监督模型表现出了有竞争力的性能和效率。然而，联合提取多个自监督语音模型缺乏经验。在我们的工作中，我们对各种自监督语音模型如HuBERT、RobustHuBERT和WavLM进行了集成知识提取（EKD）。我们尝试了两种不同的聚合技术，分层平均和分层连接，以不同的教师模型的表示，发现前者更有效。在此基础上，提出了一种学生模型的多预测头方法，用于同时预测多个教师模型的不同层输出。实验结果表明，该方法在SUPERB基准测试的隐集跟踪中，提高了提取模型在音素识别、说话人识别、情感识别和自动语音识别4个下游语音处理任务中的性能。摘要：Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we performed Ensemble Knowledge Distillation (EKD) on various self-supervised speech models such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective. On top of that, we proposed a multiple prediction head method for student models to predict different layer outputs of multiple teacher models simultaneously. The experimental results show that our method improves the performance of the distilled models on four downstream speech processing tasks, Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic Speech Recognition in the hidden-set track of the SUPERB benchmark.【7】 PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS
标题：PITS：端到端音调可控TTS的无基频变音调推理
链接：https://arxiv.org/abs/2302.12391
作者：Junhyeok Lee,Wonbin Jung,Hyunjae Cho,Jaeyeon Kim机构：MINDsLab Inc., Republic of Korea, Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea, Seoul National University, Republic of Korea摘要：现有的音调可控的文语转换模型依赖于直接对基频进行建模，导致合成语音的方差较小。为了解决这个问题，我们提出了PITS，一个端到端的音调可控TTS模型，它利用变分推理来建模音调。PITS在VITS的基础上，结合了英拉姆编码器、英拉姆解码器和对抗式的移频合成训练来实现基音可控性。实验结果表明，PITS算法能够产生与真实语音难以区分的高质量语音，并且在不降低语音质量的前提下具有较高的基音可控性。代码和音频样本将在www.example.com上提供https://github.com/anonymous-pits/pits。摘要：Previous pitch-controllable text-to-speech (TTS) models rely on directly modeling fundamental frequency, leading to low variance in synthesized speech. To address this issue, we propose PITS, an end-to-end pitch-controllable TTS model that utilizes variational inference to model pitch. Based on VITS, PITS incorporates the Yingram encoder, the Yingram decoder, and adversarial training of pitch-shifted synthesis to achieve pitch-controllability. Experiments demonstrate that PITS generates high-quality speech that is indistinguishable from ground truth speech and has high pitch-controllability without quality degradation. Code and audio samples will be available at https://github.com/anonymous-pits/pits.【8】 Factual Consistency Oriented Speech Recognition
标题：面向事实一致性的语音识别
链接：https://arxiv.org/abs/2302.12369
作者：Naoyuki Kanda,Takuya Yoshioka,Yang Liu备注：5 pages, 1 figure, 3 tables摘要：提出了一种新的自动语音识别（ASR）优化框架，目的是减少ASR模型产生的幻觉。该框架优化了ASR模型，使ASR假设和地面实况转录之间的预期事实一致性得分最大化，其中事实一致性得分由单独训练的估计器计算。在AMI会议语料库和VoxPopuli语料库上的实验结果表明，使用该框架训练的ASR模型生成的ASR假设与地面事实转录具有显著更高的一致性得分，同时保持了接近交叉熵训练的ASR模型的单词错误率。此外，实验结果表明，使用所提出的框架训练ASR模型可以提高语音摘要的质量，这可以通过大型语言模型生成的会议会话摘要的事实一致性来衡量。摘要：This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.【1】 Ensemble knowledge distillation of self-supervised speech models
标题：自监督语音模型的集成知识提取
链接：https://arxiv.org/abs/2302.12757
*与cs.SD语音【6】为同一篇
作者：Kuan-Po Huang,Tzu-hsun Feng,Yu-Kuan Fu,Tsu-Yuan Hsu,Po-Chieh Yen,Wei-Cheng Tseng,Kai-Wei Chang,Hung-yi Lee机构：allCollege of Electrical Engineering and Computer Science, National Taiwan University, †ASUS Intelligent Cloud Services备注：Accepted by ICASSP 2023摘要：Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we performed Ensemble Knowledge Distillation (EKD) on various self-supervised speech models such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective. On top of that, we proposed a multiple prediction head method for student models to predict different layer outputs of multiple teacher models simultaneously. The experimental results show that our method improves the performance of the distilled models on four downstream speech processing tasks, Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic Speech Recognition in the hidden-set track of the SUPERB benchmark.【2】 PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS
标题：PITS：端到端音调可控TTS的无基频变音调推理
链接：https://arxiv.org/abs/2302.12391
*与cs.SD语音【7】为同一篇
作者：Junhyeok Lee,Wonbin Jung,Hyunjae Cho,Jaeyeon Kim机构：MINDsLab Inc., Republic of Korea, Korea Advanced Institute of Science and Technology (KAIST), Republic of Korea, Seoul National University, Republic of Korea摘要：现有的音调可控的文语转换模型依赖于直接对基频进行建模，导致合成语音的方差较小。为了解决这个问题，我们提出了PITS，一个端到端的音调可控TTS模型，它利用变分推理来建模音调。PITS在VITS的基础上，结合了英拉姆编码器、英拉姆解码器和对抗式的移频合成训练来实现基音可控性。实验结果表明，PITS算法能够产生与真实语音难以区分的高质量语音，并且在不降低语音质量的前提下具有较高的基音可控性。代码和音频样本将在www.example.com上提供https://github.com/anonymous-pits/pits。摘要：Previous pitch-controllable text-to-speech (TTS) models rely on directly modeling fundamental frequency, leading to low variance in synthesized speech. To address this issue, we propose PITS, an end-to-end pitch-controllable TTS model that utilizes variational inference to model pitch. Based on VITS, PITS incorporates the Yingram encoder, the Yingram decoder, and adversarial training of pitch-shifted synthesis to achieve pitch-controllability. Experiments demonstrate that PITS generates high-quality speech that is indistinguishable from ground truth speech and has high pitch-controllability without quality degradation. Code and audio samples will be available at https://github.com/anonymous-pits/pits.【3】 Factual Consistency Oriented Speech Recognition
标题：面向事实一致性的语音识别
链接：https://arxiv.org/abs/2302.12369
*与cs.SD语音【8】为同一篇
作者：Naoyuki Kanda,Takuya Yoshioka,Yang Liu备注：5 pages, 1 figure, 3 tables摘要：提出了一种新的自动语音识别（ASR）优化框架，目的是减少ASR模型产生的幻觉。该框架优化了ASR模型，使ASR假设和地面实况转录之间的预期事实一致性得分最大化，其中事实一致性得分由单独训练的估计器计算。在AMI会议语料库和VoxPopuli语料库上的实验结果表明，使用该框架训练的ASR模型生成的ASR假设与地面事实转录具有显著更高的一致性得分，同时保持了接近交叉熵训练的ASR模型的单词错误率。此外，实验结果表明，使用所提出的框架训练ASR模型可以提高语音摘要的质量，这可以通过大型语言模型生成的会议会话摘要的事实一致性来衡量。摘要：This paper presents a novel optimization framework for automatic speech recognition (ASR) with the aim of reducing hallucinations produced by an ASR model. The proposed framework optimizes the ASR model to maximize an expected factual consistency score between ASR hypotheses and ground-truth transcriptions, where the factual consistency score is computed by a separately trained estimator. Experimental results using the AMI meeting corpus and the VoxPopuli corpus show that the ASR model trained with the proposed framework generates ASR hypotheses that have significantly higher consistency scores with ground-truth transcriptions while maintaining the word error rates close to those of cross entropy-trained ASR models. Furthermore, it is shown that training the ASR models with the proposed framework improves the speech summarization quality as measured by the factual consistency of meeting conversation summaries generated by a large language model.【4】 Improving Massively Multilingual ASR With Auxiliary CTC Objectives
标题：利用辅助CTC目标改进大规模多语言ASR
链接：https://arxiv.org/abs/2302.12829
*与cs.SD语音【1】为同一篇
作者：William Chen,Brian Yan,Jiatong Shi,Yifan Peng,Soumi Maiti,Shinji Watanabe机构：Language Technologies Institute, Carnegie Mellon University, USA备注：5 pages, 1 figure, accepted at ICASSP 2023摘要：多语言自动语音识别（ASR）模型已经将语音技术的可用性扩展到各种语言。然而，考虑到这些模型必须处理多少种语言，理解它们在不同语言之间的不平衡性能的关键是检查模型是否真的知道它应该转录哪种语言。本文介绍了我们在FLEURS（一个包含102种语言的开放ASR基准测试）上通过将整个模型置于语言身份（LID）的条件下来提高性能的工作。我们研究了最近的联结主义时间分类（CTC）研究所启发的技术，以帮助模型处理大量的语言，条件是辅助任务的LID预测。实验结果证明了该方法相对于标准CTC/Attention-based混合模型的有效性。此外，我们最先进的系统使用具有Conformer架构的自监督模型，相对于FLEURS的先前工作结果提高了28.4% CER。训练模型是可重现的配方，可在https://github.com/espnet/espnet/tree/master/egs 2/fleurs/asr 1上获得。摘要：Multilingual Automatic Speech Recognition (ASR) models have extended the usability of speech technologies to a wide variety of languages. With how many languages these models have to handle, however, a key to understanding their imbalanced performance across different languages is to examine if the model actually knows which language it should transcribe. In this paper, we introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark, by conditioning the entire model on language identity (LID). We investigate techniques inspired from recent Connectionist Temporal Classification (CTC) studies to help the model handle the large number of languages, conditioning on the LID predictions of auxiliary tasks. Our experimental results demonstrate the effectiveness of our technique over standard CTC/Attention-based hybrid mod- els. Furthermore, our state-of-the-art systems using self-supervised models with the Conformer architecture improve over the results of prior work on FLEURS by a relative 28.4% CER. Trained models are reproducible recipes are available at https://github.com/ espnet/espnet/tree/master/egs2/fleurs/asr1.【5】 Multi-task learning of speech and speaker recognition
标题：语音多任务学习与说话人识别
链接：https://arxiv.org/abs/2302.12773
*与cs.SD语音【2】为同一篇
作者：Nik Vaessen,David A. van Leeuwen机构：Institute for Computing and Information Sciences, Radboud University, Nijmegen, The, Netherlands备注：submitted to ICML 2022摘要：我们研究两个正交语音技术任务的多任务学习：语音和说话人识别。我们使用wav2vec2作为基础架构，它有两个特定于任务的输出头。我们实验了不同的方法来混合说话人和语音信息在输出嵌入序列中，并提出了一个简单的动态方法来平衡语音和说话人识别损失函数。我们的多任务学习网络可以生成共享的说话人和语音嵌入，并在LibriSpeech和VoxCeleb测试集上进行评估，其性能可与单独的单任务模型相媲美。代码可从www.example.com获取。https://github.com/nikvaessen/2022-repo-mt-w2v2.摘要：We study multi-task learning for two orthogonal speech technology tasks: speech and speaker recognition. We use wav2vec2 as a base architecture with two task-specific output heads. We experiment with different methods to mix speaker and speech information in the output embedding sequence, and propose a simple dynamic approach to balance the speech and speaker recognition loss functions. Our multi-task learning networks can produce a shared speaker and speech embedding, which are evaluated on the LibriSpeech and VoxCeleb test sets, and achieve a performance comparable to separate single-task models. Code is available at https://github.com/nikvaessen/2022-repo-mt-w2v2.【6】 Supervised Hierarchical Clustering using Graph Neural Networks for Speaker Diarization
标题：基于图神经网络的有监督层次聚类说话人二值化
链接：https://arxiv.org/abs/2302.12716
*与cs.SD语音【3】为同一篇
作者：Prachi Singh,Amrit Kaul,Sriram Ganapathy机构：LEAP Lab, Electrical Engineering, Indian Institute of Science,Bangalore.备注：5 pages including references. Accepted in ICASSP 2023摘要：传统的说话人二值化方法包括将音频文件开窗为短片段以提取说话人嵌入，随后对嵌入进行无监督聚类。这种多步骤方法为每个片段生成发言者分配。本文提出了一种新的有监督的层次gRaph聚类算法（SHARC）用于说话人识别，该算法引入了一个层次结构，并利用图神经网络（GNN）进行有监督聚类。该监督允许模型更新表示并直接提高聚类性能，从而实现用于diarization的单步方法。在所提出的工作中，输入片段嵌入被视为图的节点，其边权重对应于节点之间的相似性得分。我们还提出了一种方法，联合更新嵌入提取器和GNN模型，以执行端到端说话人二值化（E2 E-SHARC）。在推理过程中，使用节点密度和边存在概率执行层次聚类以合并片段直到收敛。在二值化实验中，我们在AMI和Voxconverse等基准数据集上验证了E2 E-SHARC方法分别比基准系统提高了53%和44%。摘要：Conventional methods for speaker diarization involve windowing an audio file into short segments to extract speaker embeddings, followed by an unsupervised clustering of the embeddings. This multi-step approach generates speaker assignments for each segment. In this paper, we propose a novel Supervised HierArchical gRaph Clustering algorithm (SHARC) for speaker diarization where we introduce a hierarchical structure using Graph Neural Network (GNN) to perform supervised clustering. The supervision allows the model to update the representations and directly improve the clustering performance, thus enabling a single-step approach for diarization. In the proposed work, the input segment embeddings are treated as nodes of a graph with the edge weights corresponding to the similarity scores between the nodes. We also propose an approach to jointly update the embedding extractor and the GNN model to perform end-to-end speaker diarization (E2E-SHARC). During inference, the hierarchical clustering is performed using node densities and edge existence probabilities to merge the segments until convergence. In the diarization experiments, we illustrate that the proposed E2E-SHARC approach achieves 53% and 44% relative improvements over the baseline systems on benchmark datasets like AMI and Voxconverse, respectively.【7】 Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion
标题：捕捉你和我可以：根据声音转换揭示源声纹
链接：https://arxiv.org/abs/2302.12434
*与cs.SD语音【4】为同一篇
作者：Jiangyi Deng,Yanjiao Chen,Yinan Zhong,Qianhao Miao,Xueluan Gong,Wenyuan Xu机构：Zhejiang University,Wuhan University备注：Accepted by USENIX Security Symposium 2023. Please cite this paper as "Jiangyi Deng, Yanjiao Chen, Yinan Zhong, Qianhao Miao, Xueluan Gong, Wenyuan Xu. Catch You and I Can: Revealing Source Voiceprint Against Voice Conversion. In 32nd USENIX Security Symposium (USENIX Security 23)."摘要：语音转换（VC）技术可能被恶意方滥用以将其音频转换成听起来像目标说话者，使得人类或说话者验证/识别系统难以追踪源说话者。本文首次尝试从高可信度的语音转换方法合成的音频中恢复源声纹。然而，从转换后的音频中揭示源说话者的特征是具有挑战性的，因为语音转换操作旨在解开原始特征并注入目标说话者的特征。为了实现我们的目标，我们开发了一个表示学习模型Revelio，它学习从转换的音频样本中有效地提取源说话人的声纹。我们为Revelio配备了精心设计的差分校正算法，通过去除与目标说话人声纹平行的表征分量来消除目标说话人的影响。我们已经进行了大量的实验来评估Revelio从VQVC、VQVC+、AGAIN和BNE转换的音频中恢复声纹的能力。实验结果表明，Revelio能够重建出说话人确认和识别系统能够追踪到源说话人的声纹。Revelio在性别转换、看不见的语言和电话网络方面也表现出强劲的表现。摘要：Voice conversion (VC) techniques can be abused by malicious parties to transform their audios to sound like a target speaker, making it hard for a human being or a speaker verification/identification system to trace the source speaker. In this paper, we make the first attempt to restore the source voiceprint from audios synthesized by voice conversion methods with high credit. However, unveiling the features of the source speaker from a converted audio is challenging since the voice conversion operation intends to disentangle the original features and infuse the features of the target speaker. To fulfill our goal, we develop Revelio, a representation learning model, which learns to effectively extract the voiceprint of the source speaker from converted audio samples. We equip Revelio with a carefully-designed differential rectification algorithm to eliminate the influence of the target speaker by removing the representation component that is parallel to the voiceprint of the target speaker. We have conducted extensive experiments to evaluate the capability of Revelio in restoring voiceprint from audios converted by VQVC, VQVC+, AGAIN, and BNE. The experiments verify that Revelio is able to rebuild voiceprints that can be traced to the source speaker by speaker verification and identification systems. Revelio also exhibits robust performance under inter-gender conversion, unseen languages, and telephony networks.【8】 Data leakage in cross-modal retrieval training: A case study
标题：跨通道检索训练中的数据泄露：案例研究
链接：https://arxiv.org/abs/2302.12258
*与cs.SD语音【5】为同一篇
作者：Benno Weck,Xavier Serra机构：Huawei Technologies, Munich Research Center, Germany, Universitat Pompeu Fabra, Music Technology Group, Spain备注：5 pages. Accepted at ICASSP2023摘要：近年来基于文本的音频检索技术的发展在很大程度上得益于合适数据集的发布。由于手动创建此类数据集是一项费力的任务，因此从在线资源获取数据可能是创建大规模数据集的廉价解决方案。我们研究了最近提出的SoundDesc基准数据集，它是从BBC声音效果网页自动获取的。在我们的分析中，我们发现SoundDesc包含几个重复项，导致训练数据泄漏到评估数据中。这种数据泄漏最终会导致以前的基准测试中对检索性能的估计过于乐观。我们为在线提供的数据集提出了新的训练、验证和测试拆分。为了避免测试数据的弱污染，我们将共享类似录制设置的音频文件汇集在一起。在我们的实验中，我们发现新的分割是一个更具挑战性的基准。摘要：The recent progress in text-based audio retrieval was largely propelled by the release of suitable datasets. Since the manual creation of such datasets is a laborious task, obtaining data from online resources can be a cheap solution to create large-scale datasets. We study the recently proposed SoundDesc benchmark dataset, which was automatically sourced from the BBC Sound Effects web page. In our analysis, we find that SoundDesc contains several duplicates that cause leakage of training data to the evaluation data. This data leakage ultimately leads to overly optimistic retrieval performance estimates in previous benchmarks. We propose new training, validation, and testing splits for the dataset that we make available online. To avoid weak contamination of the test data, we pool audio files that share similar recording setups. In our experiments, we find that the new splits serve as a more challenging benchmark.【9】 Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook
标题：强化学习与语音和语言处理：教程、回顾与展望
链接：https://arxiv.org/abs/2210.13623
机构：Columbia University, New York, NY备注：INTERSPEECH 2022 Tutorial摘要：近年来，强化学习和bandps已经改变了现实世界中的广泛应用，包括医疗保健、金融、推荐系统、机器人技术，以及最后但并非最不重要的语音和自然语言处理。虽然强化学习算法的大多数语音和语言应用都集中在利用其灵活的优化特性来改进深度神经网络的训练，但仍然有许多理由来探索利用强化学习的好处，例如其奖励驱动的适应性、状态表示、时间结构和泛化能力。在这篇综述中，我们对强化学习和土匪的最新进展进行了概述，并讨论了如何利用它们来有效地解决语音和自然语言处理问题，模型是自适应的，交互式的和可扩展的。摘要：In recent years, reinforcement learning and bandits have transformed a wide range of real-world applications including healthcare, finance, recommendation systems, robotics, and last but not least, the speech and natural language processing. While most speech and language applications of reinforcement learning algorithms are centered around improving the training of deep neural networks with its flexible optimization properties, there are still many grounds to explore to utilize the benefits of reinforcement learning, such as its reward-driven adaptability, state representations, temporal structures and generalizability. In this survey, we present an overview of recent advancements of reinforcement learning and bandits, and discuss how they can be effectively employed to solve speech and natural language processing problems with models that are adaptive, interactive and scalable.机器翻译，仅供参考
每日语音速递