今日论文合集：cs.SD语音9篇，eess.AS音频处理10篇。
cs.SD语音
【1】Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder
标题：基于唇子词相关性的视觉预训练和跨模式融合编码器改进视听语音识别
链接：http://arxiv.org/pdf/2308.08488v1
作者：Yusheng Dai,Hang Chen,Jun Du,Xiaofei Ding,Ning Ding,Feijun Jiang,Chin-Hui Lee备注：6 pages, 2 figures, published in ICME2023摘要：在最近的研究中，从自动语音识别系统到具有低质量视频的端到端框架中的视听语音识别系统，观察到轻微的性能改善。不匹配的收敛速度和专门的输入表示之间的音频和视觉模态被认为是导致问题。在本文中，我们提出了两种新的技术，以提高视听语音识别（AVSR）下的预训练和微调训练框架。首先，我们探讨唇形和音节水平的子词单位在普通话中建立良好的框架级音节边界从唇形之间的相关性。这使得能够在视觉模型预训练和跨模态融合期间精确对齐视频和音频流。接下来，我们提出了一个音频引导的跨模态融合编码器（CMFE）神经网络，利用多个跨模态注意层的主要训练参数，以充分利用模态互补性。在MISP 2021-AVSR数据集上的实验表明了这两种方法的有效性。总之，仅使用相对少量的训练数据，最终系统实现了比具有更复杂的前端和后端的最先进系统更好的性能。摘要：In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with low-quality videos. Unmatching convergence rates and specialized input representations between audio and visual modalities are considered to cause the problem. In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. This enables accurate alignment of video and audio streams during visual model pre-training and cross-modal fusion. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers to make full use of modality complementarity. Experiments on the MISP2021-AVSR data set show the effectiveness of the two proposed techniques. Together, using only a relatively small amount of training data, the final system achieves better performances than state-of-the-art systems with more complex front-ends and back-ends.
【2】Improving CTC-AED model with integrated-CTC and auxiliary loss regularization
标题：基于集成CTC和辅助损失正则化的CTC-AED模型改进
链接：http://arxiv.org/pdf/2308.08449v1
作者：Daobin Zhu,Xiangdong Su,Hongbin Zhang摘要：连接主义时间分类（CTC）和基于注意力的编码解码器（AED）联合训练在自动语音识别（ASR）中得到了广泛的应用。与大多数混合模型，分别计算的CTC和AED的损失，我们建议的集成CTC利用AED的注意力机制，以指导CTC的输出。在本文中，我们采用两种融合方法，即直接添加的logits（DAL）和保持最大概率（PMP）。我们通过自适应仿射变换的注意力的结果，以匹配CTC的尺寸，实现尺寸的一致性。为了加速模型收敛和提高精度，我们引入辅助损失正则化加速收敛。实验结果表明，DAL方法在注意力再评分方面表现更好，而PMP方法在CTC前缀波束搜索和贪婪搜索方面表现更好。摘要：Connectionist temporal classification (CTC) and attention-based encoder decoder (AED) joint training has been widely applied in automatic speech recognition (ASR). Unlike most hybrid models that separately calculate the CTC and AED losses, our proposed integrated-CTC utilizes the attention mechanism of AED to guide the output of CTC. In this paper, we employ two fusion methods, namely direct addition of logits (DAL) and preserving the maximum probability (PMP). We achieve dimensional consistency by adaptively affine transforming the attention results to match the dimensions of CTC. To accelerate model convergence and improve accuracy, we introduce auxiliary loss regularization for accelerated convergence. Experimental results demonstrate that the DAL method performs better in attention rescoring, while the PMP method excels in CTC prefix beam search and greedy search.
【3】Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction
标题：减轻句子级字素到音素(G2P)转换中的暴露偏差
链接：http://arxiv.org/pdf/2308.08442v1
作者：Eunseop Yoon,Hee Suk Yoon,Dhananjaya Gowda,SooHwan Eom,Daehyeok Kim,John Harvill,Heting Gao,Mark Hasegawa-Johnson,Chanwoo Kim,Chang D. Yoo备注：INTERSPEECH 2023摘要：文本到文本转换Transformer（T5）最近被考虑用于字形到音素（G2 P）转换。作为后续，一个基于T5的无令牌化器字节级模型被称为ByT 5，最近通过用其对应的UTF-8编码表示每个输入字符，在字级G2 P转换上给出了有希望的结果。虽然人们普遍认为，句子级或段落级的G2 P可以提高现实世界应用程序的可用性，因为它更适合于在异义词上执行和连接单词之间的声音，但我们发现，使用ByT 5这些场景是不平凡的。由于ByT 5在字符级别上操作，因此需要更长的解码步骤，这由于在自回归生成模型中常见的曝光偏差而劣化了性能。本文表明，句子级和段落级的G2 P的性能可以改善，减少这种曝光偏差使用我们提出的基于损失的采样方法。摘要：Text-to-Text Transfer Transformer (T5) has recently been considered for the Grapheme-to-Phoneme (G2P) transduction. As a follow-up, a tokenizer-free byte-level model based on T5 referred to as ByT5, recently gave promising results on word-level G2P conversion by representing each input character with its corresponding UTF-8 encoding. Although it is generally understood that sentence-level or paragraph-level G2P can improve usability in real-world applications as it is better suited to perform on heteronyms and linking sounds between words, we find that using ByT5 for these scenarios is nontrivial. Since ByT5 operates on the character level, it requires longer decoding steps, which deteriorates the performance due to the exposure bias commonly observed in auto-regressive generation models. This paper shows that the performance of sentence-level and paragraph-level G2P can be improved by mitigating such exposure bias using our proposed loss-based sampling method.
【4】Accurate synthesis of Dysarthric Speech for ASR data augmentation
标题：ASR数据增强中节律障碍语音的精确合成
链接：http://arxiv.org/pdf/2308.08438v1
作者：Mohammad Soleymanpour,Michael T. Johnson,Rahim Soleymanpour,Jeffrey Berry备注：arXiv admin note: text overlap with arXiv:2201.11571摘要：摘要：构音障碍是一种运动性言语障碍，其特征通常是通过缓慢、不协调地控制言语产生肌肉而降低言语清晰度。自动语音识别（ASR）系统可以帮助构音障碍的说话者更有效地沟通。然而，稳健的构音障碍特异性ASR需要大量的训练语音，这对于构音障碍的说话者来说是不容易获得的。本文提出了一种新的构音障碍语音合成方法的目的是增强ASR的训练数据。在不同的严重程度水平的构音障碍的自发语音的韵律和声学特征的差异是构音障碍的语音建模，合成和增强的重要组成部分。对于构音障碍语音合成，通过添加构音障碍严重程度系数和暂停插入模型来合成构音障碍语音以用于不同严重程度来实现修改的神经多说话者TTS。为了评估ASR训练数据合成的有效性，使用了构音障碍特异性语音识别。结果表明，与基线相比，在额外的合成构音障碍语音上训练的DNN-HMM模型实现了12.2%的WER改善，并且添加严重程度和暂停插入控制使WER降低了6.5%，显示了添加这些参数的有效性。TORGO数据库上的总体结果表明，使用构音障碍合成语音来增加用于训练的构音障碍模式语音的量对构音障碍ASR系统具有显著影响。此外，我们进行了主观评价，以评估合成语音的构音障碍和相似性。我们的主观评价显示，合成语音的构音障碍感与真正构音障碍语音的构音障碍感相似，特别是对于构音障碍程度较高的语音摘要：Dysarthria is a motor speech disorder often characterized by reduced speech intelligibility through slow, uncoordinated control of speech production muscles. Automatic Speech recognition (ASR) systems can help dysarthric talkers communicate more effectively. However, robust dysarthria-specific ASR requires a significant amount of training speech, which is not readily available for dysarthric talkers. This paper presents a new dysarthric speech synthesis method for the purpose of ASR training data augmentation. Differences in prosodic and acoustic characteristics of dysarthric spontaneous speech at varying severity levels are important components for dysarthric speech modeling, synthesis, and augmentation. For dysarthric speech synthesis, a modified neural multi-talker TTS is implemented by adding a dysarthria severity level coefficient and a pause insertion model to synthesize dysarthric speech for varying severity levels. To evaluate the effectiveness for synthesis of training data for ASR, dysarthria-specific speech recognition was used. Results show that a DNN-HMM model trained on additional synthetic dysarthric speech achieves WER improvement of 12.2% compared to the baseline, and that the addition of the severity level and pause insertion controls decrease WER by 6.5%, showing the effectiveness of adding these parameters. Overall results on the TORGO database demonstrate that using dysarthric synthetic speech to increase the amount of dysarthric-patterned speech for training has significant impact on the dysarthric ASR systems. In addition, we have conducted a subjective evaluation to evaluate the dysarthric-ness and similarity of synthesized speech. Our subjective evaluation shows that the perceived dysartrhic-ness of synthesized speech is similar to that of true dysarthric speech, especially for higher levels of dysarthria
【5】ChinaTelecom System Description to VoxCeleb Speaker Recognition Challenge 2023
标题：中国电信系统描述VoxCeleb说话人识别挑战赛2023
链接：http://arxiv.org/pdf/2308.08181v1
作者：Mengjie Du,Xiang Fang,Jie Li备注：System description of VoxSRC 2023摘要：本技术报告描述了VoxCeleb2023扬声器识别挑战赛（VoxSRC 2023）的第一轨道（已关闭）的中国电信系统。我们的系统由几个仅在VoxCeleb2上训练的ResNet变体组成，这些变体后来被融合以获得更好的性能。还对每个变体和融合系统应用评分校准。最终申报的minDCF为0.1066，EER为1.980%。摘要：This technical report describes ChinaTelecom system for Track 1 (closed) of the VoxCeleb2023 Speaker Recognition Challenge (VoxSRC 2023). Our system consists of several ResNet variants trained only on VoxCeleb2, which were fused for better performance later. Score calibration was also applied for each variant and the fused system. The final submission achieved minDCF of 0.1066 and EER of 1.980%.
【6】SCANet: A Self- and Cross-Attention Network for Audio-Visual Speech Separation
标题：SCANet：一种用于视听语音分离的自注意和交叉注意网络
链接：http://arxiv.org/pdf/2308.08143v1
作者：Kai Li,Runxuan Yang,Xiaolin Hu备注：14 pages, 3 figures摘要：不同模态（例如音频和视觉信息）的整合在人类对周围环境的感知中起着至关重要的作用。最近的研究取得了重大进展，在设计融合模块的视听语音分离。然而，他们主要集中在位于顶部或底部位置的多模态融合架构，而不是全面考虑在网络内的各个层次位置的多模态融合。在本文中，我们提出了一种新的模型称为自我和交叉注意力网络（SCANet），它利用了高效的视听功能融合的注意力机制。SCANet由两种类型的注意力块组成：自我注意（SA）和交叉注意（CA）块，其中CA块分布在SCANet的顶部（TCA）、中部（MCA）和底部（BCA）。这些块保持学习模态特定特征的能力，并且使得能够从视听特征提取不同的语义。三个标准的视听分离基准（LRS2，LRS3和VoxCeleb2）的综合实验证明了SCANet的有效性，优于现有的国家的最先进的（SOTA）的方法，同时保持可比的推理时间。摘要：The integration of different modalities, such as audio and visual information, plays a crucial role in human perception of the surrounding environment. Recent research has made significant progress in designing fusion modules for audio-visual speech separation. However, they predominantly focus on multi-modal fusion architectures situated either at the top or bottom positions, rather than comprehensively considering multi-modal fusion at various hierarchical positions within the network. In this paper, we propose a novel model called self- and cross-attention network (SCANet), which leverages the attention mechanism for efficient audio-visual feature fusion. SCANet consists of two types of attention blocks: self-attention (SA) and cross-attention (CA) blocks, where the CA blocks are distributed at the top (TCA), middle (MCA) and bottom (BCA) of SCANet. These blocks maintain the ability to learn modality-specific features and enable the extraction of different semantics from audio-visual features. Comprehensive experiments on three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2) demonstrate the effectiveness of SCANet, outperforming existing state-of-the-art (SOTA) methods while maintaining comparable inference time.
【7】Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals
标题：Radio2Text：使用毫米波无线电信号的流语音识别
链接：http://arxiv.org/pdf/2308.08125v1
作者：Running Zhao,Jiangtao Yu,Hang Zhao,Edith C. H. Ngai备注：Accepted by Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (ACM IMWUTUbiComp 2023)摘要：基于毫米波（mmWave）的语音识别为与音频相关的应用提供了更多的可能性，例如会议语音转录和窃听。然而，考虑到实际场景中的实用性，延迟和可识别词汇量是两个不可忽视的关键因素。在本文中，我们提出了Radio 2 Text，这是第一个基于mmWave的流式自动语音识别（ASR）系统，词汇量超过13，000个单词。Radio 2 Text基于量身定制的流媒体Transformer，能够有效地学习语音相关特征的表示，为具有大量词汇的流媒体ASR铺平了道路。为了缓解流媒体网络无法访问整个未来的输入的不足，我们提出了指导初始化，有利于转移的特征知识相关的全球背景下，从非流媒体Transformer量身定制的流媒体变压器通过权重继承。此外，我们提出了一种基于知识蒸馏（KD）的交叉模态结构，称为交叉模态KD，以减轻低质量毫米波信号对识别性能的负面影响。在跨模态KD中，音频流传输Transformer提供特征和响应指导，其继承丰富且准确的语音信息以监督定制的无线电流传输Transformer的训练。实验结果表明，我们的Radio 2 Text可以实现5.7%的字符错误率和9.4%的单词错误率识别的词汇组成的超过13,000个单词。摘要：Millimeter wave (mmWave) based speech recognition provides more possibility for audio-related applications, such as conference speech transcription and eavesdropping. However, considering the practicality in real scenarios, latency and recognizable vocabulary size are two critical factors that cannot be overlooked. In this paper, we propose Radio2Text, the first mmWave-based system for streaming automatic speech recognition (ASR) with a vocabulary size exceeding 13,000 words. Radio2Text is based on a tailored streaming Transformer that is capable of effectively learning representations of speech-related features, paving the way for streaming ASR with a large vocabulary. To alleviate the deficiency of streaming networks unable to access entire future inputs, we propose the Guidance Initialization that facilitates the transfer of feature knowledge related to the global context from the non-streaming Transformer to the tailored streaming Transformer through weight inheritance. Further, we propose a cross-modal structure based on knowledge distillation (KD), named cross-modal KD, to mitigate the negative effect of low quality mmWave signals on recognition performance. In the cross-modal KD, the audio streaming Transformer provides feature and response guidance that inherit fruitful and accurate speech information to supervise the training of the tailored radio streaming Transformer. The experimental results show that our Radio2Text can achieve a character error rate of 5.7% and a word error rate of 9.4% for the recognition of a vocabulary consisting of over 13,000 words.
【8】Classifying Dementia in the Presence of Depression: A Cross-Corpus Study
标题：抑郁症痴呆的分类：一项跨语料库的研究链接：http://arxiv.org/pdf/2308.08306v1
作者：Franziska Braun,Sebastian P. Bayerl,Paula A. Pérez-Toro,Florian Hönig,Hartmut Lehfeld,Thomas Hillemacher,Elmar Nöth,Tobias Bocklet,Korbinian Riedhammer备注：Accepted at INTERSPEECH 2023摘要：自动化痴呆症筛查可以早期发现和干预，降低医疗保健系统的成本，提高受影响者的生活质量。抑郁症与痴呆症有共同的症状，增加了诊断的复杂性。到目前为止，研究的重点一直是痴呆症（DEM）和健康对照（HC）使用语音从图片描述测试从一个单一的数据集的二进制分类。在这项工作中，我们应用已建立的基线系统来区分语音中的认知障碍，从语义言语流畅性测试和波士顿命名测试中使用文本，音频和情感嵌入在3类分类问题（HC与MCI对DEM）。我们在两个独立记录的德语数据集上进行跨语料库和混合语料库实验，以研究对更大人群和不同记录条件的泛化。在详细的错误分析中，我们将抑郁症视为次要诊断，以了解分类器实际上学到了什么。摘要：Automated dementia screening enables early detection and intervention, reducing costs to healthcare systems and increasing quality of life for those affected. Depression has shared symptoms with dementia, adding complexity to diagnoses. The research focus so far has been on binary classification of dementia (DEM) and healthy controls (HC) using speech from picture description tests from a single dataset. In this work, we apply established baseline systems to discriminate cognitive impairment in speech from the semantic Verbal Fluency Test and the Boston Naming Test using text, audio and emotion embeddings in a 3-class classification problem (HC vs. MCI vs. DEM). We perform cross-corpus and mixed-corpus experiments on two independently recorded German datasets to investigate generalization to larger populations and different recording conditions. In a detailed error analysis, we look at depression as a secondary diagnosis to understand what our classifiers actually learn.
【9】End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations标题：基于多语言神经表示的端到端开放式词汇关键词检索链接：http://arxiv.org/pdf/2308.08027v1
作者：Bolaji Yusuf,Jan Cernocky,Murat SaraclarJournal-ref：in IEEEACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3070-3080, 2023摘要：传统的关键词搜索系统在自动语音识别（ASR）输出上操作，这使得它们具有复杂的索引和搜索流水线。这引起了人们对简化搜索过程的无ASR方法的兴趣。我们最近提出了一种无神经ASR的关键字搜索模型，该模型在保持高效和简化的管道的同时实现了具有竞争力的性能，其中查询和文档使用一对递归神经网络编码器进行编码，并且编码与点积相结合。在本文中，我们通过多语言预训练和模型的详细分析来扩展这项工作。我们的实验表明，所提出的多语言训练显着提高了模型的性能，尽管没有匹配一个强大的基于ASR的传统关键字搜索系统的短查询和查询，包括词汇表中的单词，所提出的模型优于基于ASR的系统的长查询和查询，不出现在训练数据。摘要：Conventional keyword search systems operate on automatic speech recognition (ASR) outputs, which causes them to have a complex indexing and search pipeline. This has led to interest in ASR-free approaches to simplify the search procedure. We recently proposed a neural ASR-free keyword search model which achieves competitive performance while maintaining an efficient and simplified pipeline, where queries and documents are encoded with a pair of recurrent neural network encoders and the encodings are combined with a dot-product. In this article, we extend this work with multilingual pretraining and detailed analysis of the model. Our experiments show that the proposed multilingual training significantly improves the model performance and that despite not matching a strong ASR-based conventional keyword search system for short queries and queries comprising in-vocabulary words, the proposed model outperforms the ASR-based system for long queries and queries that do not appear in the training data.
eess.AS音频处理
【1】Classifying Dementia in the Presence of Depression: A Cross-Corpus Study
标题：抑郁症痴呆的分类：一项跨语料库的研究链接：http://arxiv.org/pdf/2308.08306v1
* 与cs.SD语音【8】为同一篇
作者：Franziska Braun,Sebastian P. Bayerl,Paula A. Pérez-Toro,Florian Hönig,Hartmut Lehfeld,Thomas Hillemacher,Elmar Nöth,Tobias Bocklet,Korbinian Riedhammer备注：Accepted at INTERSPEECH 2023摘要：自动化痴呆症筛查可以早期发现和干预，降低医疗保健系统的成本，提高受影响者的生活质量。抑郁症与痴呆症有共同的症状，增加了诊断的复杂性。到目前为止，研究的重点一直是痴呆症（DEM）和健康对照（HC）使用语音从图片描述测试从一个单一的数据集的二进制分类。在这项工作中，我们应用已建立的基线系统来区分语音中的认知障碍，从语义言语流畅性测试和波士顿命名测试中使用文本，音频和情感嵌入在3类分类问题（HC与MCI对DEM）。我们在两个独立记录的德语数据集上进行跨语料库和混合语料库实验，以研究对更大人群和不同记录条件的泛化。在详细的错误分析中，我们将抑郁症视为次要诊断，以了解分类器实际上学到了什么。摘要：Automated dementia screening enables early detection and intervention, reducing costs to healthcare systems and increasing quality of life for those affected. Depression has shared symptoms with dementia, adding complexity to diagnoses. The research focus so far has been on binary classification of dementia (DEM) and healthy controls (HC) using speech from picture description tests from a single dataset. In this work, we apply established baseline systems to discriminate cognitive impairment in speech from the semantic Verbal Fluency Test and the Boston Naming Test using text, audio and emotion embeddings in a 3-class classification problem (HC vs. MCI vs. DEM). We perform cross-corpus and mixed-corpus experiments on two independently recorded German datasets to investigate generalization to larger populations and different recording conditions. In a detailed error analysis, we look at depression as a secondary diagnosis to understand what our classifiers actually learn.
【2】The ID R&D VoxCeleb Speaker Recognition Challenge 2023 System Description
标题：ID研发VoxCeleb说话人识别挑战赛2023系统描述
链接：http://arxiv.org/pdf/2308.08294v1
作者：Nikita Torgashov,Rostislav Makarov,Ivan Yakovlev,Pavel Malov,Andrei Balykin,Anton Okhotnikov摘要：本报告描述了ID研发团队提交的第2轨道（开放）的VoxCeleb扬声器识别挑战赛2023（VoxSRC-23）。我们的解决方案是基于深度ResNets和基于自监督学习（SSL）的模型的融合，这些模型是在VoxCeleb 2数据集和大版本VoxTube数据集的混合物上训练的。第二阶段的最终提交作品在VoxSRC-23公开排行榜上获得第一名，minDCF（0.05）为0.0762，EER为1.30%。摘要：This report describes ID R&D team submissions for Track 2 (open) to the VoxCeleb Speaker Recognition Challenge 2023 (VoxSRC-23). Our solution is based on the fusion of deep ResNets and self-supervised learning (SSL) based models trained on a mixture of a VoxCeleb2 dataset and a large version of a VoxTube dataset. The final submission to the Track 2 achieved the first place on the VoxSRC-23 public leaderboard with a minDCF(0.05) of 0.0762 and EER of 1.30%.
【3】End-to-End Open Vocabulary Keyword Search With Multilingual Neural Representations标题：基于多语言神经表示的端到端开放式词汇关键词检索链接：http://arxiv.org/pdf/2308.08027v1
* 与cs.SD语音【9】为同一篇
作者：Bolaji Yusuf,Jan Cernocky,Murat SaraclarJournal-ref：in IEEEACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3070-3080, 2023摘要：传统的关键词搜索系统在自动语音识别（ASR）输出上操作，这使得它们具有复杂的索引和搜索流水线。这引起了人们对简化搜索过程的无ASR方法的兴趣。我们最近提出了一种无神经ASR的关键字搜索模型，该模型在保持高效和简化的管道的同时实现了具有竞争力的性能，其中查询和文档使用一对递归神经网络编码器进行编码，并且编码与点积相结合。在本文中，我们通过多语言预训练和模型的详细分析来扩展这项工作。我们的实验表明，所提出的多语言训练显着提高了模型的性能，尽管没有匹配一个强大的基于ASR的传统关键字搜索系统的短查询和查询，包括词汇表中的单词，所提出的模型优于基于ASR的系统的长查询和查询，不出现在训练数据。摘要：Conventional keyword search systems operate on automatic speech recognition (ASR) outputs, which causes them to have a complex indexing and search pipeline. This has led to interest in ASR-free approaches to simplify the search procedure. We recently proposed a neural ASR-free keyword search model which achieves competitive performance while maintaining an efficient and simplified pipeline, where queries and documents are encoded with a pair of recurrent neural network encoders and the encodings are combined with a dot-product. In this article, we extend this work with multilingual pretraining and detailed analysis of the model. Our experiments show that the proposed multilingual training significantly improves the model performance and that despite not matching a strong ASR-based conventional keyword search system for short queries and queries comprising in-vocabulary words, the proposed model outperforms the ASR-based system for long queries and queries that do not appear in the training data.
【4】Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder
标题：基于唇子词相关性的视觉预训练和跨模式融合编码器改进视听语音识别
链接：http://arxiv.org/pdf/2308.08488v1
* 与cs.SD语音【1】为同一篇
作者：Yusheng Dai,Hang Chen,Jun Du,Xiaofei Ding,Ning Ding,Feijun Jiang,Chin-Hui Lee备注：6 pages, 2 figures, published in ICME2023摘要：在最近的研究中，从自动语音识别系统到具有低质量视频的端到端框架中的视听语音识别系统，观察到轻微的性能改善。不匹配的收敛速度和专门的输入表示之间的音频和视觉模态被认为是导致问题。在本文中，我们提出了两种新的技术，以提高视听语音识别（AVSR）下的预训练和微调训练框架。首先，我们探讨唇形和音节水平的子词单位在普通话中建立良好的框架级音节边界从唇形之间的相关性。这使得能够在视觉模型预训练和跨模态融合期间精确对齐视频和音频流。接下来，我们提出了一个音频引导的跨模态融合编码器（CMFE）神经网络，利用多个跨模态注意层的主要训练参数，以充分利用模态互补性。在MISP 2021-AVSR数据集上的实验表明了这两种方法的有效性。总之，仅使用相对少量的训练数据，最终系统实现了比具有更复杂的前端和后端的最先进系统更好的性能。摘要：In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with low-quality videos. Unmatching convergence rates and specialized input representations between audio and visual modalities are considered to cause the problem. In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. This enables accurate alignment of video and audio streams during visual model pre-training and cross-modal fusion. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers to make full use of modality complementarity. Experiments on the MISP2021-AVSR data set show the effectiveness of the two proposed techniques. Together, using only a relatively small amount of training data, the final system achieves better performances than state-of-the-art systems with more complex front-ends and back-ends.
【5】Improving CTC-AED model with integrated-CTC and auxiliary loss regularization
标题：基于集成CTC和辅助损失正则化的CTC-AED模型改进
链接：http://arxiv.org/pdf/2308.08449v1
* 与cs.SD语音【2】为同一篇
作者：Daobin Zhu,Xiangdong Su,Hongbin Zhang摘要：连接主义时间分类（CTC）和基于注意力的编码解码器（AED）联合训练在自动语音识别（ASR）中得到了广泛的应用。与大多数混合模型，分别计算的CTC和AED的损失，我们建议的集成CTC利用AED的注意力机制，以指导CTC的输出。在本文中，我们采用两种融合方法，即直接添加的logits（DAL）和保持最大概率（PMP）。我们通过自适应仿射变换的注意力的结果，以匹配CTC的尺寸，实现尺寸的一致性。为了加速模型收敛和提高精度，我们引入辅助损失正则化加速收敛。实验结果表明，DAL方法在注意力再评分方面表现更好，而PMP方法在CTC前缀波束搜索和贪婪搜索方面表现更好。摘要：Connectionist temporal classification (CTC) and attention-based encoder decoder (AED) joint training has been widely applied in automatic speech recognition (ASR). Unlike most hybrid models that separately calculate the CTC and AED losses, our proposed integrated-CTC utilizes the attention mechanism of AED to guide the output of CTC. In this paper, we employ two fusion methods, namely direct addition of logits (DAL) and preserving the maximum probability (PMP). We achieve dimensional consistency by adaptively affine transforming the attention results to match the dimensions of CTC. To accelerate model convergence and improve accuracy, we introduce auxiliary loss regularization for accelerated convergence. Experimental results demonstrate that the DAL method performs better in attention rescoring, while the PMP method excels in CTC prefix beam search and greedy search.
【6】Mitigating the Exposure Bias in Sentence-Level Grapheme-to-Phoneme (G2P) Transduction
标题：减轻句子级字素到音素(G2P)转换中的暴露偏差
链接：http://arxiv.org/pdf/2308.08442v1
* 与cs.SD语音【3】为同一篇
作者：Eunseop Yoon,Hee Suk Yoon,Dhananjaya Gowda,SooHwan Eom,Daehyeok Kim,John Harvill,Heting Gao,Mark Hasegawa-Johnson,Chanwoo Kim,Chang D. Yoo备注：INTERSPEECH 2023摘要：文本到文本转换Transformer（T5）最近被考虑用于字形到音素（G2 P）转换。作为后续，一个基于T5的无令牌化器字节级模型被称为ByT 5，最近通过用其对应的UTF-8编码表示每个输入字符，在字级G2 P转换上给出了有希望的结果。虽然人们普遍认为，句子级或段落级的G2 P可以提高现实世界应用程序的可用性，因为它更适合于在异义词上执行和连接单词之间的声音，但我们发现，使用ByT 5这些场景是不平凡的。由于ByT 5在字符级别上操作，因此需要更长的解码步骤，这由于在自回归生成模型中常见的曝光偏差而劣化了性能。本文表明，句子级和段落级的G2 P的性能可以改善，减少这种曝光偏差使用我们提出的基于损失的采样方法。摘要：Text-to-Text Transfer Transformer (T5) has recently been considered for the Grapheme-to-Phoneme (G2P) transduction. As a follow-up, a tokenizer-free byte-level model based on T5 referred to as ByT5, recently gave promising results on word-level G2P conversion by representing each input character with its corresponding UTF-8 encoding. Although it is generally understood that sentence-level or paragraph-level G2P can improve usability in real-world applications as it is better suited to perform on heteronyms and linking sounds between words, we find that using ByT5 for these scenarios is nontrivial. Since ByT5 operates on the character level, it requires longer decoding steps, which deteriorates the performance due to the exposure bias commonly observed in auto-regressive generation models. This paper shows that the performance of sentence-level and paragraph-level G2P can be improved by mitigating such exposure bias using our proposed loss-based sampling method.
【7】Accurate synthesis of Dysarthric Speech for ASR data augmentation
标题：ASR数据增强中节律障碍语音的精确合成
链接：http://arxiv.org/pdf/2308.08438v1
* 与cs.SD语音【4】为同一篇
作者：Mohammad Soleymanpour,Michael T. Johnson,Rahim Soleymanpour,Jeffrey Berry备注：arXiv admin note: text overlap with arXiv:2201.11571摘要：摘要：构音障碍是一种运动性言语障碍，其特征通常是通过缓慢、不协调地控制言语产生肌肉而降低言语清晰度。自动语音识别（ASR）系统可以帮助构音障碍的说话者更有效地沟通。然而，稳健的构音障碍特异性ASR需要大量的训练语音，这对于构音障碍的说话者来说是不容易获得的。本文提出了一种新的构音障碍语音合成方法的目的是增强ASR的训练数据。在不同的严重程度水平的构音障碍的自发语音的韵律和声学特征的差异是构音障碍的语音建模，合成和增强的重要组成部分。对于构音障碍语音合成，通过添加构音障碍严重程度系数和暂停插入模型来合成构音障碍语音以用于不同严重程度来实现修改的神经多说话者TTS。为了评估ASR训练数据合成的有效性，使用了构音障碍特异性语音识别。结果表明，与基线相比，在额外的合成构音障碍语音上训练的DNN-HMM模型实现了12.2%的WER改善，并且添加严重程度和暂停插入控制使WER降低了6.5%，显示了添加这些参数的有效性。TORGO数据库上的总体结果表明，使用构音障碍合成语音来增加用于训练的构音障碍模式语音的量对构音障碍ASR系统具有显著影响。此外，我们进行了主观评价，以评估合成语音的构音障碍和相似性。我们的主观评价显示，合成语音的构音障碍感与真正构音障碍语音的构音障碍感相似，特别是对于构音障碍程度较高的语音摘要：Dysarthria is a motor speech disorder often characterized by reduced speech intelligibility through slow, uncoordinated control of speech production muscles. Automatic Speech recognition (ASR) systems can help dysarthric talkers communicate more effectively. However, robust dysarthria-specific ASR requires a significant amount of training speech, which is not readily available for dysarthric talkers. This paper presents a new dysarthric speech synthesis method for the purpose of ASR training data augmentation. Differences in prosodic and acoustic characteristics of dysarthric spontaneous speech at varying severity levels are important components for dysarthric speech modeling, synthesis, and augmentation. For dysarthric speech synthesis, a modified neural multi-talker TTS is implemented by adding a dysarthria severity level coefficient and a pause insertion model to synthesize dysarthric speech for varying severity levels. To evaluate the effectiveness for synthesis of training data for ASR, dysarthria-specific speech recognition was used. Results show that a DNN-HMM model trained on additional synthetic dysarthric speech achieves WER improvement of 12.2% compared to the baseline, and that the addition of the severity level and pause insertion controls decrease WER by 6.5%, showing the effectiveness of adding these parameters. Overall results on the TORGO database demonstrate that using dysarthric synthetic speech to increase the amount of dysarthric-patterned speech for training has significant impact on the dysarthric ASR systems. In addition, we have conducted a subjective evaluation to evaluate the dysarthric-ness and similarity of synthesized speech. Our subjective evaluation shows that the perceived dysartrhic-ness of synthesized speech is similar to that of true dysarthric speech, especially for higher levels of dysarthria
【8】ChinaTelecom System Description to VoxCeleb Speaker Recognition Challenge 2023
标题：中国电信系统描述VoxCeleb说话人识别挑战赛2023
链接：http://arxiv.org/pdf/2308.08181v1
* 与cs.SD语音【5】为同一篇
作者：Mengjie Du,Xiang Fang,Jie Li备注：System description of VoxSRC 2023摘要：本技术报告描述了VoxCeleb2023扬声器识别挑战赛（VoxSRC 2023）的第一轨道（已关闭）的中国电信系统。我们的系统由几个仅在VoxCeleb2上训练的ResNet变体组成，这些变体后来被融合以获得更好的性能。还对每个变体和融合系统应用评分校准。最终申报的minDCF为0.1066，EER为1.980%。摘要：This technical report describes ChinaTelecom system for Track 1 (closed) of the VoxCeleb2023 Speaker Recognition Challenge (VoxSRC 2023). Our system consists of several ResNet variants trained only on VoxCeleb2, which were fused for better performance later. Score calibration was also applied for each variant and the fused system. The final submission achieved minDCF of 0.1066 and EER of 1.980%.
【9】SCANet: A Self- and Cross-Attention Network for Audio-Visual Speech Separation
标题：SCANet：一种用于视听语音分离的自注意和交叉注意网络
链接：http://arxiv.org/pdf/2308.08143v1
* 与cs.SD语音【6】为同一篇
作者：Kai Li,Runxuan Yang,Xiaolin Hu备注：14 pages, 3 figures摘要：不同模态（例如音频和视觉信息）的整合在人类对周围环境的感知中起着至关重要的作用。最近的研究取得了重大进展，在设计融合模块的视听语音分离。然而，他们主要集中在位于顶部或底部位置的多模态融合架构，而不是全面考虑在网络内的各个层次位置的多模态融合。在本文中，我们提出了一种新的模型称为自我和交叉注意力网络（SCANet），它利用了高效的视听功能融合的注意力机制。SCANet由两种类型的注意力块组成：自我注意（SA）和交叉注意（CA）块，其中CA块分布在SCANet的顶部（TCA）、中部（MCA）和底部（BCA）。这些块保持学习模态特定特征的能力，并且使得能够从视听特征提取不同的语义。三个标准的视听分离基准（LRS2，LRS3和VoxCeleb2）的综合实验证明了SCANet的有效性，优于现有的国家的最先进的（SOTA）的方法，同时保持可比的推理时间。摘要：The integration of different modalities, such as audio and visual information, plays a crucial role in human perception of the surrounding environment. Recent research has made significant progress in designing fusion modules for audio-visual speech separation. However, they predominantly focus on multi-modal fusion architectures situated either at the top or bottom positions, rather than comprehensively considering multi-modal fusion at various hierarchical positions within the network. In this paper, we propose a novel model called self- and cross-attention network (SCANet), which leverages the attention mechanism for efficient audio-visual feature fusion. SCANet consists of two types of attention blocks: self-attention (SA) and cross-attention (CA) blocks, where the CA blocks are distributed at the top (TCA), middle (MCA) and bottom (BCA) of SCANet. These blocks maintain the ability to learn modality-specific features and enable the extraction of different semantics from audio-visual features. Comprehensive experiments on three standard audio-visual separation benchmarks (LRS2, LRS3, and VoxCeleb2) demonstrate the effectiveness of SCANet, outperforming existing state-of-the-art (SOTA) methods while maintaining comparable inference time.
【10】Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals
标题：Radio2Text：使用毫米波无线电信号的流语音识别
链接：http://arxiv.org/pdf/2308.08125v1
* 与cs.SD语音【7】为同一篇
作者：Running Zhao,Jiangtao Yu,Hang Zhao,Edith C. H. Ngai备注：Accepted by Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (ACM IMWUTUbiComp 2023)摘要：基于毫米波（mmWave）的语音识别为与音频相关的应用提供了更多的可能性，例如会议语音转录和窃听。然而，考虑到实际场景中的实用性，延迟和可识别词汇量是两个不可忽视的关键因素。在本文中，我们提出了Radio 2 Text，这是第一个基于mmWave的流式自动语音识别（ASR）系统，词汇量超过13，000个单词。Radio 2 Text基于量身定制的流媒体Transformer，能够有效地学习语音相关特征的表示，为具有大量词汇的流媒体ASR铺平了道路。为了缓解流媒体网络无法访问整个未来的输入的不足，我们提出了指导初始化，有利于转移的特征知识相关的全球背景下，从非流媒体Transformer量身定制的流媒体变压器通过权重继承。此外，我们提出了一种基于知识蒸馏（KD）的交叉模态结构，称为交叉模态KD，以减轻低质量毫米波信号对识别性能的负面影响。在跨模态KD中，音频流传输Transformer提供特征和响应指导，其继承丰富且准确的语音信息以监督定制的无线电流传输Transformer的训练。实验结果表明，我们的Radio 2 Text可以实现5.7%的字符错误率和9.4%的单词错误率识别的词汇组成的超过13,000个单词。摘要：Millimeter wave (mmWave) based speech recognition provides more possibility for audio-related applications, such as conference speech transcription and eavesdropping. However, considering the practicality in real scenarios, latency and recognizable vocabulary size are two critical factors that cannot be overlooked. In this paper, we propose Radio2Text, the first mmWave-based system for streaming automatic speech recognition (ASR) with a vocabulary size exceeding 13,000 words. Radio2Text is based on a tailored streaming Transformer that is capable of effectively learning representations of speech-related features, paving the way for streaming ASR with a large vocabulary. To alleviate the deficiency of streaming networks unable to access entire future inputs, we propose the Guidance Initialization that facilitates the transfer of feature knowledge related to the global context from the non-streaming Transformer to the tailored streaming Transformer through weight inheritance. Further, we propose a cross-modal structure based on knowledge distillation (KD), named cross-modal KD, to mitigate the negative effect of low quality mmWave signals on recognition performance. In the cross-modal KD, the audio streaming Transformer provides feature and response guidance that inherit fruitful and accurate speech information to supervise the training of the tailored radio streaming Transformer. The experimental results show that our Radio2Text can achieve a character error rate of 5.7% and a word error rate of 9.4% for the recognition of a vocabulary consisting of over 13,000 words.
机器翻译，仅供参考