今日论文合集：cs.SD语音5篇，eess.AS音频处理4篇。
cs.SD语音
【1】 Roman Numeral Analysis with Graph Neural Networks: Onset-wise  Predictions from Note-wise Features标题：基于图形神经网络的罗马数字分析：基于音符特征的起始式预测链接：https://arxiv.org/abs/2307.03544
作者：Emmanouil Karystinaios,Gerhard Widmer机构：Institute of Computational Perception, Johannes Kepler University Linz, Austria,  LIT AI Lab, Linz Institute of Technology, Austria备注：submitted for ISMIR 2023 conference, 6 pages of content, 5 figures摘要：罗马数字分析是在调性音乐作品中识别和弦及其功能背景的重要任务。本文提出了一种新的方法来自动罗马数字分析的象征性音乐。虽然现有的技术依赖于中间有损表示的分数，我们提出了一种新的方法，基于图神经网络（GNN），使每个单独的注意在分数的直接描述和处理。所提出的架构可以利用noteswise的功能和笔记之间的相互依赖性，但产生的起始表示凭借我们的新的边缘收缩算法。我们的研究结果表明，ChordGNN优于现有的最先进的模型，在参考数据集上的罗马数字分析中实现了更高的准确性。此外，我们调查我们的模型的变体使用所提出的技术，如NADE，和后处理的和弦预测。这项工作的完整源代码可在www.example.com获得https://github.com/manoskary/chordgnn摘要：Roman Numeral analysis is the important task of identifying chords and their functional context in pieces of tonal music. This paper presents a new approach to automatic Roman Numeral analysis in symbolic music. While existing techniques rely on an intermediate lossy representation of the score, we propose a new method based on Graph Neural Networks (GNNs) that enable the direct description and processing of each individual note in the score. The proposed architecture can leverage notewise features and interdependencies between notes but yield onset-wise representation by virtue of our novel edge contraction algorithm. Our results demonstrate that ChordGNN outperforms existing state-of-the-art models, achieving higher accuracy in Roman Numeral analysis on the reference datasets. In addition, we investigate variants of our model using proposed techniques such as NADE, and post-processing of the chord predictions. The full source code for this work is available at https://github.com/manoskary/chordgnn
【2】 The CHiME-7 UDASE task: Unsupervised domain adaptation for  conversational speech enhancement标题：CHINE-7 UDASE任务：会话语音增强的无监督域自适应链接：https://arxiv.org/abs/2307.03533
作者：Simon Leglaive,Léonie Borne,Efthymios Tzinis,Mostafa Sadeghi,Matthieu Fraticelli,Scott Wisdom,Manuel Pariente,Daniel Pressnitzer,John R. Hershey机构：CentraleSup´elec, IETR UMR CNRS , France, Pulse Audition, France, University of, Illinois at Urbana-Champaign, USA, Inria, France,  ´Ecole Normale Sup´erieure, PSL, University, CNRS, France, Google, USA摘要：监督语音增强模型使用人工生成的干净语音和噪声信号的混合物进行训练，这可能与测试时的真实记录条件不匹配。如果测试域与合成训练域显著不同，则这种不匹配可能导致较差的性能。在本文中，我们介绍了第七届CHiME挑战赛的无监督域自适应会话语音增强（UDASE）任务。该任务旨在利用来自目标测试域的真实噪声语音记录进行语音增强模型的无监督域自适应。目标测试域对应于CHiME-5数据集的多说话者混响对话语音记录，对于该多说话者混响对话语音记录，地面实况干净语音参考不可用。给定CHiME-5录音，任务是估计干净的、潜在的多扬声器混响语音，去除加性背景噪声。我们讨论了CHiME-7 UDASE任务的动机，并描述了数据，任务和基线系统。摘要：Supervised speech enhancement models are trained using artificially generated mixtures of clean speech and noise signals, which may not match real-world recording conditions at test time. This mismatch can lead to poor performance if the test domain significantly differs from the synthetic training domain. In this paper, we introduce the unsupervised domain adaptation for conversational speech enhancement (UDASE) task of the 7th CHiME challenge. This task aims to leverage real-world noisy speech recordings from the target test domain for unsupervised domain adaptation of speech enhancement models. The target test domain corresponds to the multi-speaker reverberant conversational speech recordings of the CHiME-5 dataset, for which the ground-truth clean speech reference is not available. Given a CHiME-5 recording, the task is to estimate the clean, potentially multi-speaker, reverberant speech, removing the additive background noise. We discuss the motivation for the CHiME-7 UDASE task and describe the data, the task, and the baseline system.
【3】 Token-Level Serialized Output Training for Joint Streaming ASR and ST  Leveraging Textual Alignments标题：利用文本对齐的联合流ASR和ST的令牌级串行化输出训练链接：https://arxiv.org/abs/2307.03354
作者：Sara Papi,Peidong Wan,Junkun Chen,Jian Xue,Jinyu Li,Yashesh Gaur机构：†Microsoft, USA, ‡Fondazione Bruno Kessler and University of Trento, Italy摘要：在现实世界的应用中，用户通常需要语音的翻译和转录来增强他们的理解，特别是在需要增量生成的流媒体场景中。本文介绍了一种流式转换器换能器，联合产生自动语音识别（ASR）和语音翻译（ST）的输出，使用一个单一的解码器。为了以最小的延迟有效地生成ASR和ST内容，我们提出了一种联合令牌级序列化输出训练方法，该方法通过利用现成的文本对齐器来交错源和目标单词。单语（it-en）和多语言（de，es，it\}-en）设置的实验表明，我们的方法实现了最佳的质量延迟平衡。平均ASR延迟为1秒，ST延迟为1.3秒，与单独的ASR和ST模型相比，我们的模型没有显示出任何退化，甚至提高了输出质量，在多语言情况下平均提高了1.1 WER和0.4 BLEU。摘要：In real-world applications, users often require both translations and transcriptions of speech to enhance their comprehension, particularly in streaming scenarios where incremental generation is necessary. This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. To produce ASR and ST content effectively with minimal latency, we propose a joint token-level serialized output training method that interleaves source and target words by leveraging an off-the-shelf textual aligner. Experiments in monolingual (it-en) and multilingual (\{de,es,it\}-en) settings demonstrate that our approach achieves the best quality-latency balance. With an average ASR latency of 1s and ST latency of 1.3s, our model shows no degradation or even improves output quality compared to separate ASR and ST models, yielding an average improvement of 1.1 WER and 0.4 BLEU in the multilingual case.
【4】 A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony  in Talking Head Generation标题：一种多尺度语音与动态同步的综合方法链接：https://arxiv.org/abs/2307.03270
作者：Louis Airale,Dominique Vaufreydaz,Xavier Alameda-Pineda机构：Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, France, Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK摘要：使用语音输入信号用深度生成模型动画化静止面部图像是一个活跃的研究课题，并且最近已经取得了重要进展。然而，大部分的努力已经投入到嘴唇同步和渲染质量，而自然的头部运动的生成，更不用说头部运动和语音之间的视听相关性，往往被忽视。在这项工作中，我们提出了一个多尺度视听同步损失和多尺度自回归GAN，以更好地处理语音和头部和嘴唇的动态之间的短期和长期相关性。特别是，我们训练一堆的syncer模型的多模式输入金字塔，并使用这些模型作为指导，在多尺度的发电机网络产生音频对齐的运动展开在不同的时间尺度。我们的生成器在面部地标域中操作，这是一个标准的低维头部表示。实验表明，在头部运动的动态质量和多尺度的视听同步在地标域和图像域的最先进的显着改善。摘要：Animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress. However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected. In this work, we propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips. In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audio-aligned motion unfolding over diverse time scales. Our generator operates in the facial landmark domain, which is a standard low-dimensional head representation. The experiments show significant improvements over the state of the art in head motion dynamics quality and in multi-scale audio-visual synchrony both in the landmark domain and in the image domain.
【5】 Gammatonegram Representation for End-to-End Dysarthric Speech Processing  Tasks: Speech Recognition, Speaker Identification, and Intelligibility  Assessment标题：端到端动态语音处理任务的伽马图表示：语音识别、说话人识别和清晰度评估链接：https://arxiv.org/abs/2307.03296
作者：Aref Farhadipour,Hadi Veisi机构：Media Engineering, IRI Broadcasting University, Tehran, Iran., University of Tehran, Tehran, Iran.备注：12 pages, 8 figures摘要：构音障碍是一种导致人类语音系统干扰并降低人的语音质量和清晰度的残疾。由于这种效应，正常的语音处理系统不能对受损的语音正常工作。这种残疾通常与身体残疾有关。因此，设计一个可以通过在智能家居中接收语音命令来执行一些任务的系统可能是一项重大成就。在这项工作中，我们引入gammatonegram作为一种有效的方法来表示音频文件与歧视性的细节，这是用来作为输入的卷积神经网络。另一方面，我们将每一个语音文件转换成一个图像，并提出了图像识别系统来分类语音在不同的场景。提出的CNN基于预训练的Alexnet上的迁移学习方法。在这项研究中，所提出的系统的语音识别，说话人识别和可懂度评估的效率进行评估。根据UA数据集上的结果，所提出的语音识别系统在说话人相关模式下实现了91.29%的准确率，说话人识别系统在文本相关模式下获得了87.74%的准确率，可懂度评估系统在两类模式下实现了96.47%的准确率。最后，我们提出了一个多网络语音识别系统，工作完全自动化。该系统与两级可懂度评估系统级联布置，并且该系统的输出激活每个语音识别网络。该架构实现了92.3%WRR的准确性。本文的源代码是可用的。摘要：Dysarthria is a disability that causes a disturbance in the human speech system and reduces the quality and intelligibility of a person's speech. Because of this effect, the normal speech processing systems can not work properly on impaired speech. This disability is usually associated with physical disabilities. Therefore, designing a system that can perform some tasks by receiving voice commands in the smart home can be a significant achievement. In this work, we introduce gammatonegram as an effective method to represent audio files with discriminative details, which is used as input for the convolutional neural network. On the other word, we convert each speech file into an image and propose image recognition system to classify speech in different scenarios. Proposed CNN is based on the transfer learning method on the pre-trained Alexnet. In this research, the efficiency of the proposed system for speech recognition, speaker identification, and intelligibility assessment is evaluated. According to the results on the UA dataset, the proposed speech recognition system achieved 91.29% accuracy in speaker-dependent mode, the speaker identification system acquired 87.74% accuracy in text-dependent mode, and the intelligibility assessment system achieved 96.47% accuracy in two-class mode. Finally, we propose a multi-network speech recognition system that works fully automatically. This system is located in a cascade arrangement with the two-class intelligibility assessment system, and the output of this system activates each one of the speech recognition networks. This architecture achieves an accuracy of 92.3% WRR. The source code of this paper is available.
eess.AS音频处理
【1】 Gammatonegram Representation for End-to-End Dysarthric Speech Processing  Tasks: Speech Recognition, Speaker Identification, and Intelligibility  Assessment标题：端到端动态语音处理任务的伽马图表示：语音识别、说话人识别和清晰度评估链接：https://arxiv.org/abs/2307.03296
作者：Aref Farhadipour,Hadi Veisi机构：Media Engineering, IRI Broadcasting University, Tehran, Iran., University of Tehran, Tehran, Iran.备注：12 pages, 8 figures摘要：构音障碍是一种导致人类语音系统干扰并降低人的语音质量和清晰度的残疾。由于这种效应，正常的语音处理系统不能对受损的语音正常工作。这种残疾通常与身体残疾有关。因此，设计一个可以通过在智能家居中接收语音命令来执行一些任务的系统可能是一项重大成就。在这项工作中，我们引入gammatonegram作为一种有效的方法来表示音频文件与歧视性的细节，这是用来作为输入的卷积神经网络。另一方面，我们将每一个语音文件转换成一个图像，并提出了图像识别系统来分类语音在不同的场景。提出的CNN基于预训练的Alexnet上的迁移学习方法。在这项研究中，所提出的系统的语音识别，说话人识别和可懂度评估的效率进行评估。根据UA数据集上的结果，所提出的语音识别系统在说话人相关模式下实现了91.29%的准确率，说话人识别系统在文本相关模式下获得了87.74%的准确率，可懂度评估系统在两类模式下实现了96.47%的准确率。最后，我们提出了一个多网络语音识别系统，工作完全自动化。该系统与两级可懂度评估系统级联布置，并且该系统的输出激活每个语音识别网络。该架构实现了92.3%WRR的准确性。本文的源代码是可用的。摘要：Dysarthria is a disability that causes a disturbance in the human speech system and reduces the quality and intelligibility of a person's speech. Because of this effect, the normal speech processing systems can not work properly on impaired speech. This disability is usually associated with physical disabilities. Therefore, designing a system that can perform some tasks by receiving voice commands in the smart home can be a significant achievement. In this work, we introduce gammatonegram as an effective method to represent audio files with discriminative details, which is used as input for the convolutional neural network. On the other word, we convert each speech file into an image and propose image recognition system to classify speech in different scenarios. Proposed CNN is based on the transfer learning method on the pre-trained Alexnet. In this research, the efficiency of the proposed system for speech recognition, speaker identification, and intelligibility assessment is evaluated. According to the results on the UA dataset, the proposed speech recognition system achieved 91.29% accuracy in speaker-dependent mode, the speaker identification system acquired 87.74% accuracy in text-dependent mode, and the intelligibility assessment system achieved 96.47% accuracy in two-class mode. Finally, we propose a multi-network speech recognition system that works fully automatically. This system is located in a cascade arrangement with the two-class intelligibility assessment system, and the output of this system activates each one of the speech recognition networks. This architecture achieves an accuracy of 92.3% WRR. The source code of this paper is available.
【2】 The CHiME-7 UDASE task: Unsupervised domain adaptation for  conversational speech enhancement标题：CHINE-7 UDASE任务：会话语音增强的无监督域自适应链接：https://arxiv.org/abs/2307.03533
作者：Simon Leglaive,Léonie Borne,Efthymios Tzinis,Mostafa Sadeghi,Matthieu Fraticelli,Scott Wisdom,Manuel Pariente,Daniel Pressnitzer,John R. Hershey机构：CentraleSup´elec, IETR UMR CNRS , France, Pulse Audition, France, University of, Illinois at Urbana-Champaign, USA, Inria, France,  ´Ecole Normale Sup´erieure, PSL, University, CNRS, France, Google, USA摘要：监督语音增强模型使用人工生成的干净语音和噪声信号的混合物进行训练，这可能与测试时的真实记录条件不匹配。如果测试域与合成训练域显著不同，则这种不匹配可能导致较差的性能。在本文中，我们介绍了第七届CHiME挑战赛的无监督域自适应会话语音增强（UDASE）任务。该任务旨在利用来自目标测试域的真实噪声语音记录进行语音增强模型的无监督域自适应。目标测试域对应于CHiME-5数据集的多说话者混响对话语音记录，对于该多说话者混响对话语音记录，地面实况干净语音参考不可用。给定CHiME-5录音，任务是估计干净的、潜在的多扬声器混响语音，去除加性背景噪声。我们讨论了CHiME-7 UDASE任务的动机，并描述了数据，任务和基线系统。摘要：Supervised speech enhancement models are trained using artificially generated mixtures of clean speech and noise signals, which may not match real-world recording conditions at test time. This mismatch can lead to poor performance if the test domain significantly differs from the synthetic training domain. In this paper, we introduce the unsupervised domain adaptation for conversational speech enhancement (UDASE) task of the 7th CHiME challenge. This task aims to leverage real-world noisy speech recordings from the target test domain for unsupervised domain adaptation of speech enhancement models. The target test domain corresponds to the multi-speaker reverberant conversational speech recordings of the CHiME-5 dataset, for which the ground-truth clean speech reference is not available. Given a CHiME-5 recording, the task is to estimate the clean, potentially multi-speaker, reverberant speech, removing the additive background noise. We discuss the motivation for the CHiME-7 UDASE task and describe the data, the task, and the baseline system.
【3】 Token-Level Serialized Output Training for Joint Streaming ASR and ST  Leveraging Textual Alignments标题：利用文本对齐的联合流ASR和ST的令牌级串行化输出训练链接：https://arxiv.org/abs/2307.03354
作者：Sara Papi,Peidong Wan,Junkun Chen,Jian Xue,Jinyu Li,Yashesh Gaur机构：†Microsoft, USA, ‡Fondazione Bruno Kessler and University of Trento, Italy摘要：在现实世界的应用中，用户通常需要语音的翻译和转录来增强他们的理解，特别是在需要增量生成的流媒体场景中。本文介绍了一种流式转换器换能器，联合产生自动语音识别（ASR）和语音翻译（ST）的输出，使用一个单一的解码器。为了以最小的延迟有效地生成ASR和ST内容，我们提出了一种联合令牌级序列化输出训练方法，该方法通过利用现成的文本对齐器来交错源和目标单词。单语（it-en）和多语言（de，es，it\}-en）设置的实验表明，我们的方法实现了最佳的质量延迟平衡。平均ASR延迟为1秒，ST延迟为1.3秒，与单独的ASR和ST模型相比，我们的模型没有显示出任何退化，甚至提高了输出质量，在多语言情况下平均提高了1.1 WER和0.4 BLEU。摘要：In real-world applications, users often require both translations and transcriptions of speech to enhance their comprehension, particularly in streaming scenarios where incremental generation is necessary. This paper introduces a streaming Transformer-Transducer that jointly generates automatic speech recognition (ASR) and speech translation (ST) outputs using a single decoder. To produce ASR and ST content effectively with minimal latency, we propose a joint token-level serialized output training method that interleaves source and target words by leveraging an off-the-shelf textual aligner. Experiments in monolingual (it-en) and multilingual (\{de,es,it\}-en) settings demonstrate that our approach achieves the best quality-latency balance. With an average ASR latency of 1s and ST latency of 1.3s, our model shows no degradation or even improves output quality compared to separate ASR and ST models, yielding an average improvement of 1.1 WER and 0.4 BLEU in the multilingual case.
【4】 A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony  in Talking Head Generation标题：一种多尺度语音与动态同步的综合方法链接：https://arxiv.org/abs/2307.03270
作者：Louis Airale,Dominique Vaufreydaz,Xavier Alameda-Pineda机构：Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, France, Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK摘要：使用语音输入信号用深度生成模型动画化静止面部图像是一个活跃的研究课题，并且最近已经取得了重要进展。然而，大部分的努力已经投入到嘴唇同步和渲染质量，而自然的头部运动的生成，更不用说头部运动和语音之间的视听相关性，往往被忽视。在这项工作中，我们提出了一个多尺度视听同步损失和多尺度自回归GAN，以更好地处理语音和头部和嘴唇的动态之间的短期和长期相关性。特别是，我们训练一堆的syncer模型的多模式输入金字塔，并使用这些模型作为指导，在多尺度的发电机网络产生音频对齐的运动展开在不同的时间尺度。我们的生成器在面部地标域中操作，这是一个标准的低维头部表示。实验表明，在头部运动的动态质量和多尺度的视听同步在地标域和图像域的最先进的显着改善。摘要：Animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress. However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected. In this work, we propose a multi-scale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips. In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audio-aligned motion unfolding over diverse time scales. Our generator operates in the facial landmark domain, which is a standard low-dimensional head representation. The experiments show significant improvements over the state of the art in head motion dynamics quality and in multi-scale audio-visual synchrony both in the landmark domain and in the image domain.

机器翻译，仅供参考