今日论文合集：cs.SD语音11篇，eess.AS音频处理13篇。
cs.SD语音
【1】 CheapNET: Improving Light-weight speech enhancement network by projected  loss function标题：CheapNet：利用投影损失函数改进轻量级语音增强网络链接：https://arxiv.org/abs/2311.15959
作者：Kaijun Tan,Benzhe Dai,Jiakui Li,Wenyu Mao摘要：噪声抑制和回声消除在语音增强中至关重要，对于智能设备和实时通信至关重要。部署在语音处理前端和边缘设备中，这些算法必须确保高效的实时推理，同时计算需求较低。传统的基于边缘的噪声抑制通常使用基于MSE的幅度谱模板训练，但这种方法具有局限性。我们引入了一种新的投影损失函数，偏离MSE，以提高噪声抑制。该方法使用投影技术将关键音频分量与噪声隔离，从而显著提高模型性能。对于回声消除，该功能可以对LAEC预处理输出进行直接预测，从而大大提高性能。我们的噪声抑制模型仅用3.1M参数和0.4GFlops s的计算负载就达到了接近最先进的结果。此外，我们的回声消除模型优于复制的行业领先模型，为语音增强引入了新的视角。摘要：Noise suppression and echo cancellation are critical in speech enhancement and essential for smart devices and real-time communication. Deployed in voice processing front-ends and edge devices, these algorithms must ensure efficient real-time inference with low computational demands. Traditional edge-based noise suppression often uses MSE-based amplitude spectrum mask training, but this approach has limitations. We introduce a novel projection loss function, diverging from MSE, to enhance noise suppression. This method uses projection techniques to isolate key audio components from noise, significantly improving model performance. For echo cancellation, the function enables direct predictions on LAEC pre-processed outputs, substantially enhancing performance. Our noise suppression model achieves near state-of-the-art results with only 3.1M parameters and 0.4GFlops s computational load. Moreover, our echo cancellation model outperforms replicated industry-leading models, introducing a new perspective in speech enhancement.
【2】 A-JEPA: Joint-Embedding Predictive Architecture Can Listen标题：A-JEPA：可监听的联合嵌入预测体系结构链接：https://arxiv.org/abs/2311.15830
作者：Zhengcong Fei,Mingyuan Fan,Junshi Huang摘要：本文提出，通过在潜在空间中进行预测，可以将推动大型基础视觉模型成功的掩蔽建模原理有效地应用于音频。我们介绍了基于音频的联合嵌入预测架构（A-JEPA），这是一种简单的扩展方法，用于从音频频谱中进行自监督学习。在I-JPEA的设计之后，我们的A-JEPA通过上下文编码器使用课程掩蔽策略对可见音频频谱图补丁进行编码，并预测在精心设计的位置处采样的区域的表示。通过上下文编码器的指数移动平均来提取这些区域的目标表示，目标编码器，在整个频谱图上。我们发现这是有益的随机块掩蔽到时间-频率感知掩蔽课程的方式，考虑到在本地时间和频率的音频频谱图的高度相关的复杂性。为了增强上下文语义理解和鲁棒性，我们对编码器进行了微调，对目标数据集进行了正则化掩码，而不是输入丢弃或零。从经验上讲，当使用Vision Transformers结构构建时，我们发现A-JEPA具有高度可扩展性，并在多个音频和语音分类任务上设置了新的最先进性能，优于其他使用外部监督预训练的最新模型。摘要：This paper presents that the masked-modeling principle driving the success of large foundational vision models can be effectively applied to audio by making predictions in a latent space. We introduce Audio-based Joint-Embedding Predictive Architecture (A-JEPA), a simple extension method for self-supervised learning from the audio spectrum. Following the design of I-JPEA, our A-JEPA encodes visible audio spectrogram patches with a curriculum masking strategy via context encoder, and predicts the representations of regions sampled at well-designed locations. The target representations of those regions are extracted by the exponential moving average of context encoder, emph{i.e.}, target encoder, on the whole spectrogram. We find it beneficial to transfer random block masking into time-frequency aware masking in a curriculum manner, considering the complexity of highly correlated in local time and frequency in audio spectrograms. To enhance contextual semantic understanding and robustness, we fine-tune the encoder with a regularized masking on target datasets, instead of input dropping or zero. Empirically, when built with Vision Transformers structure, we find A-JEPA to be highly scalable and sets new state-of-the-art performance on multiple audio and speech classification tasks, outperforming other recent models that use externally supervised pre-training.
【3】 Phonetic-aware speaker embedding for far-field speaker verification标题：用于远场说话人确认的语音感知说话人嵌入链接：https://arxiv.org/abs/2311.15627
作者：Zezhong Jin,Youzhi Tu,Man-Wai Mak备注：submitted to ICASSP2024摘要：当说话人确认（SV）系统远离声源工作时，由于噪声和混响的干扰，会出现重大挑战。研究表明，将语音信息融入说话人嵌入可以提高文本无关SV的性能。受此启发，我们提出了一个联合训练的语音识别和说话人识别（JTSS）框架，利用语音内容的远场SV。该框架通过将说话人嵌入网络的基于帧的特征图与wav2vec的向量进行匹配，来鼓励说话人嵌入保留语音信息。直觉是，语音信息可以与说话者信息一起保持低水平的声学动态，从而部分地补偿由于噪声和混响引起的退化。结果表明，该框架在VOiCES Challenge 2019评估集和VoxCeleb1测试集上的性能优于标准说话人嵌入。这表明在远场条件下利用语音信息对于学习鲁棒的说话人表示是有效的。摘要：When a speaker verification (SV) system operates far from the sound sourced, significant challenges arise due to the interference of noise and reverberation. Studies have shown that incorporating phonetic information into speaker embedding can improve the performance of text-independent SV. Inspired by this observation, we propose a joint-training speech recognition and speaker recognition (JTSS) framework to exploit phonetic content for far-field SV. The framework encourages speaker embeddings to preserve phonetic information by matching the frame-based feature maps of a speaker embedding network with wav2vec's vectors. The intuition is that phonetic information can preserve low-level acoustic dynamics with speaker information and thus partly compensate for the degradation due to noise and reverberation. Results show that the proposed framework outperforms the standard speaker embedding on the VOiCES Challenge 2019 evaluation set and the VoxCeleb1 test set. This indicates that leveraging phonetic information under far-field conditions is effective for learning robust speaker representations.
【4】 Lightly Weighted Automatic Audio Parameter Extraction for the Quality  Assessment of Consensus Auditory-Perceptual Evaluation of Voice标题：基于轻量级自动音频参数提取的语音一致性听觉感知质量评价链接：https://arxiv.org/abs/2311.15582
作者：Yi-Heng Lin,Wen-Hsuan Tseng,Li-Chin Chen,Ching-Ting Tan,Yu Tsao备注：Published in IEEE 42th International Conference on Consumer Electronics (ICCE 2024)摘要：语音的一致性听觉感知评价是临床语音质量评估中广泛使用的工具，对于临床专业人员之间的流式通信和确定进一步治疗的基准具有重要意义。目前，由于评估依赖于经验丰富的临床医生，因此往往不一致，因此难以标准化。为了解决这个问题，我们建议利用轻加权自动音频参数提取，以增加临床相关性，降低复杂性，并提高语音质量评估的可解释性。所提出的方法利用年龄，性别和五个音频参数：抖动，绝对抖动，闪烁，谐波噪声比（HNR），和零交叉。采用经典的机器学习方法。结果表明，我们的方法与最先进的（SOTA）方法相似，并且优于使用流行的音频预训练模型获得的潜在表示。这种方法提供了不同的语音评价的特征提取方法的可行性的见解。音频参数，如抖动和HNR被证明是适合的语音质量属性，如粗糙度和应变的特点。相反，预先训练的模型在有效解决噪声相关评分方面存在局限性。本研究通过对多种语音质量评估方法的全面探索，为更全面、更准确地评估语音质量做出了贡献。摘要：The Consensus Auditory-Perceptual Evaluation of Voice is a widely employed tool in clinical voice quality assessment that is significant for streaming communication among clinical professionals and benchmarking for the determination of further treatment. Currently, because the assessment relies on experienced clinicians, it tends to be inconsistent, and thus, difficult to standardize. To address this problem, we propose to leverage lightly weighted automatic audio parameter extraction, to increase the clinical relevance, reduce the complexity, and enhance the interpretability of voice quality assessment. The proposed method utilizes age, sex, and five audio parameters: jitter, absolute jitter, shimmer, harmonic-to-noise ratio (HNR), and zero crossing. A classical machine learning approach is employed. The result reveals that our approach performs similar to state-of-the-art (SOTA) methods, and outperforms the latent representation obtained by using popular audio pre-trained models. This approach provide insights into the feasibility of different feature extraction approaches for voice evaluation. Audio parameters such as jitter and the HNR are proven to be suitable for characterizing voice quality attributes, such as roughness and strain. Conversely, pre-trained models exhibit limitations in effectively addressing noise-related scorings. This study contributes toward more comprehensive and precise voice quality evaluations, achieved by a comprehensively exploring diverse assessment methodologies.
【5】 Automatic Time Signature Determination for New Scores Using Lyrics for  Latent Rhythmic Structure标题：利用潜在节奏结构的歌词自动确定新乐谱的时间特征链接：https://arxiv.org/abs/2311.15480
作者：Callie C. Liao,Duoduo Liao,Jesse Guessford备注：Submitted to IEEE Big Data 2023 Conference摘要：最近，人们对人工智能生成内容（AIGC）的兴趣急剧增加。尽管如此，音乐成分，如时间签名还没有得到充分的研究，形成一个算法确定新的组成，特别是抒情歌曲的方法。这可能是因为忽略了音乐细节，这对于构建一个强大的框架至关重要。具体来说，拍号为歌曲的几乎所有方面建立了基本的节奏结构，包括乐句和音符。在本文中，我们提出了一种新的方法，只使用歌词作为输入，自动生成一个合适的时间签名的抒情歌曲，并利用可解释的机器学习模型揭示潜在的节奏结构。特别是，我们设计了多种方法，发现抒情模式和创建新的功能，同时包含抒情，节奏和统计信息。在这种方法中，我们最好的实验结果显示97.6%F1评分和0.996受试者操作特征（ROC）评分的曲线下面积（AUC）。总之，我们的研究利用机器学习直接从歌词中自动生成新乐谱的时间签名，这是一个创新的想法，接近音乐学的一个未充分研究的组成部分，因此对人工智能（AI）音乐生成的未来做出了重大贡献。摘要：There has recently been a sharp increase in interest in Artificial Intelligence-Generated Content (AIGC). Despite this, musical components such as time signatures have not been studied sufficiently to form an algorithmic determination approach for new compositions, especially lyrical songs. This is likely because of the neglect of musical details, which is critical for constructing a robust framework. Specifically, time signatures establish the fundamental rhythmic structure for almost all aspects of a song, including the phrases and notes. In this paper, we propose a novel approach that only uses lyrics as input to automatically generate a fitting time signature for lyrical songs and uncover the latent rhythmic structure utilizing explainable machine learning models. In particular, we devise multiple methods that are associated with discovering lyrical patterns and creating new features that simultaneously contain lyrical, rhythmic, and statistical information. In this approach, the best of our experimental results reveal a 97.6% F1 score and a 0.996 Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) score. In conclusion, our research directly generates time signatures from lyrics automatically for new scores utilizing machine learning, which is an innovative idea that approaches an understudied component of musicology and therefore contributes significantly to the future of Artificial Intelligence (AI) music generation.
【6】 Weakly-Supervised Audio-Visual Segmentation标题：弱监督视听分割链接：https://arxiv.org/abs/2311.15080
作者：Shentong Mo,Bhiksha Raj摘要：视听分割是一项具有挑战性的任务，旨在预测视频中声源的像素级掩模。以前的工作应用了一个全面的手动设计的架构与无数的像素明智的准确的面具作为监督。然而，这些像素级掩模是昂贵的，并且不是在所有情况下都可用。在这项工作中，我们的目标是将监督简化为实例级注释，即，弱监督视听分割我们提出了一种新的弱监督视听分割框架，即WS-AVS，可以学习多尺度视听对齐与多尺度多实例对比学习视听分割。在AVSBench上进行的大量实验证明了WS-AVS在单源和多源弱监督音视频分割中的有效性。摘要：Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e., weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.
【7】 Multi-Scale Sub-Band Constant-Q Transform Discriminator for  High-Fidelity Vocoder标题：高保真声码器的多尺度子带恒定Q变换鉴别器链接：https://arxiv.org/abs/2311.14957
作者：Yicheng Gu,Xueyao Zhang,Liumeng Xue,Zhizheng Wu摘要：当从声学表示重建可听波形时，基于生成对抗网络（GAN）的声码器在推理速度和合成质量方面优越。本研究的重点是改进的FPGA，以促进基于GAN的声码器。大多数现有的基于时频表示的鉴别器都植根于短时傅里叶变换（STFT），其在频谱图中的时频分辨率是固定的，这使得它与需要灵活注意不同频带的歌声等信号不兼容。基于此，我们的研究利用了恒Q变换（CQT），它具有频率之间的动态分辨率，有助于更好的建模能力，在基音精度和谐波跟踪。具体来说，我们提出了一种多尺度子带CQT（MS-SB-CQT）鉴别器，它在多个尺度上对CQT谱图进行操作，并根据不同的倍频程进行子带处理。对语音和歌声进行的实验证实了我们所提出的方法的有效性。此外，我们还验证了基于CQT和基于STFT的鉴别器可以在联合训练下互补。具体而言，通过所提出的MS-SB-CQT和现有的MS-STFT鉴别器的增强，HiFi-GAN的MOS可以从3.27提升到3.87，对于看到的歌手，从3.40提升到3.78。摘要：Generative Adversarial Network (GAN) based vocoders are superior in inference speed and synthesis quality when reconstructing an audible waveform from an acoustic representation. This study focuses on improving the discriminator to promote GAN-based vocoders. Most existing time-frequency-representation-based discriminators are rooted in Short-Time Fourier Transform (STFT), whose time-frequency resolution in a spectrogram is fixed, making it incompatible with signals like singing voices that require flexible attention for different frequency bands. Motivated by that, our study utilizes the Constant-Q Transform (CQT), which owns dynamic resolution among frequencies, contributing to a better modeling ability in pitch accuracy and harmonic tracking. Specifically, we propose a Multi-Scale Sub-Band CQT (MS-SB-CQT) Discriminator, which operates on the CQT spectrogram at multiple scales and performs sub-band processing according to different octaves. Experiments conducted on both speech and singing voices confirm the effectiveness of our proposed method. Moreover, we also verified that the CQT-based and the STFT-based discriminators could be complementary under joint training. Specifically, enhanced by the proposed MS-SB-CQT and the existing MS-STFT Discriminators, the MOS of HiFi-GAN can be boosted from 3.27 to 3.87 for seen singers and from 3.40 to 3.78 for unseen singers.
【8】 Custom Data Augmentation for low resource ASR using Bark and  Retrieval-Based Voice Conversion标题：使用Bark和基于检索的语音转换实现低资源ASR的定制数据增强链接：https://arxiv.org/abs/2311.14836
作者：Anand Kamble,Aniket Tathe,Suyash Kumbharkar,Atharva Bhandare,Anirban C. Mitra摘要：本文提出了两种创新的方法来构建定制的通用语音数据集的低资源的语言，如印地语。第一种方法利用Bark，一种由Suno开发的基于transformer的文本到音频模型，并结合Meta的enCodec和预训练的HuBert模型来增强Bark的性能。第二种方法采用基于检索的语音转换（RVC），并使用Ozen工具包进行数据准备。这两种方法都有助于ASR技术的进步，并为解决为资源不足的语言构建定制的通用语音数据集的挑战提供了宝贵的见解。此外，它们还为一系列应用提供了实现高质量、个性化语音生成的途径。摘要：This paper proposes two innovative methodologies to construct customized Common Voice datasets for low-resource languages like Hindi. The first methodology leverages Bark, a transformer-based text-to-audio model developed by Suno, and incorporates Meta's enCodec and a pre-trained HuBert model to enhance Bark's performance. The second methodology employs Retrieval-Based Voice Conversion (RVC) and uses the Ozen toolkit for data preparation. Both methodologies contribute to the advancement of ASR technology and offer valuable insights into addressing the challenges of constructing customized Common Voice datasets for under-resourced languages. Furthermore, they provide a pathway to achieving high-quality, personalized voice generation for a range of applications.
【9】 Weak Alignment Supervision from Hybrid Model Improves End-to-end ASR标题：来自混合模型的弱对齐监督改善了端到端ASR链接：https://arxiv.org/abs/2311.14835
作者：Jintao Jiang,Yingbo Gao,Zoltan Tuske备注：7 pages, 7 figures, and 5 tables摘要：在本文中，我们的目标是创建弱对齐监督，以帮助端到端建模。为此，我们使用现有的混合ASR系统来产生训练音频的三音素对齐。然后，我们使用导出的对齐在编码器的某个层创建交叉熵损失。与具有或不具有损失加权的一般one-hot交叉熵损失相比，这里我们使用具有标签平滑参数的交叉熵损失来正则化监督。作为比较，我们还进行了实验与一热交叉熵损失和CTC损失与损失加权。结果表明，将标签平滑参数为0.5的弱对齐监督放置在第三编码器层优于其他两种方法，并导致TED-LIUM 2数据集上的相对WER降低约5%。当我们将这种开箱即用的方法应用于TagServer端到端ASR系统时，我们看到了类似的改进。摘要：In this paper, we aim to create weak alignment supervision to aid the end-to-end modeling. Towards this end, we use the existing hybrid ASR system to produce triphone alignments of the training audios. We then create a cross-entropy loss at a certain layer of the encoder using the derived alignments. In contrast to the general one-hot cross-entropy losses with or without loss weighting, here we use a cross-entropy loss with a label smoothing parameter to regularize the supervision. As a comparison, we also conduct the experiments with one-hot cross-entropy losses and CTC losses with loss weighting. The results show that placing the weak alignment supervision with the label smoothing parameter of 0.5 at the third encoder layer outperforms the other two approaches and leads to about 5% relative WER reduction on the TED-LIUM 2 dataset over the baseline. We see similar improvements when applying the method out-of-the-box on a Tagalog end-to-end ASR system.
【10】 Ultrasensitive Textile Strain Sensors Redefine Wearable Silent Speech  Interfaces with High Machine Learning Efficiency标题：超灵敏纺织品应变传感器以高机器学习效率重新定义可穿戴无声语音接口链接：https://arxiv.org/abs/2311.15683
作者：Chenyu Tang,Muzi Xu,Wentian Yi,Zibo Zhang,Edoardo Occhipinti,Chaoqun Dong,Dafydd Ravenscroft,Sung-Min Jung,Sanghyo Lee,Shuo Gao,Jong Min Kim,Luigi G. Occhipinti备注：5 figures in the article; 11 figures and 4 tables in supplementary information摘要：我们的研究提出了一种可穿戴的静音语音接口（SSI）技术，该技术在设备舒适度，时间能源效率和语音解码准确性方面表现出色，可用于现实世界。我们开发了一种生物相容性，耐用的纺织品choker与嵌入式石墨烯为基础的应变传感器，能够准确地检测微妙的喉咙运动。与传统的语音识别方法相比，该传感器的灵敏度超过其他应变传感器420%，简化了信号处理。我们的系统使用计算效率高的神经网络，特别是具有残差结构的一维卷积神经网络来解码语音信号。该网络具有能源和时间效率，将计算负载减少了90%，同时对20个单词的词典实现了95.25%的准确率，并以最少的样本快速适应新用户和单词。这一创新展示了一种实用、灵敏、精确的可穿戴SSI，适用于日常通信应用。摘要：Our research presents a wearable Silent Speech Interface (SSI) technology that excels in device comfort, time-energy efficiency, and speech decoding accuracy for real-world use. We developed a biocompatible, durable textile choker with an embedded graphene-based strain sensor, capable of accurately detecting subtle throat movements. This sensor, surpassing other strain sensors in sensitivity by 420%, simplifies signal processing compared to traditional voice recognition methods. Our system uses a computationally efficient neural network, specifically a one-dimensional convolutional neural network with residual structures, to decode speech signals. This network is energy and time-efficient, reducing computational load by 90% while achieving 95.25% accuracy for a 20-word lexicon and swiftly adapting to new users and words with minimal samples. This innovation demonstrates a practical, sensitive, and precise wearable SSI suitable for daily communication applications.
【11】 Spatial Diarization for Meeting Transcription with Ad-Hoc Acoustic  Sensor Networks标题：用于满足自组织声传感器网络转录的空间二值化链接：https://arxiv.org/abs/2311.15597
作者：Tobias Gburrek,Joerg Schmalenstroeer,Reinhold Haeb-Umbach备注：Accepted at Asilomar Conference on Signals, Systems, and Computers 2023摘要：我们提出了一个日记系统，估计“谁发言时”的基础上的空间信息，作为一个前端的会议转录系统上收集的信号从一个声学传感器网络（ESTA）运行。虽然麦克风的空间分布是有利的，但是利用空间分集进行日志化和信号增强是具有挑战性的，因为麦克风的位置通常是未知的，并且记录的信号通常最初是不同步的。在这里，我们首先通过盲同步信号，然后估计到达时间差（TDOA）来解决这些问题。利用TDOA信息来估计说话者的活动，即使在存在多个说话者同时活动的情况下。该说话者活动信息用作空间混合模型的指导，在该空间混合模型的基础上，经由波束成形来提取各个说话者的信号。最后，将提取的信号转发到语音识别器。此外，提出了一种新的初始化方案的基础上的TDOA估计空间混合模型。从LibriWirth数据集的真实记录进行的实验表明，我们提出的系统是有利的相比，使用空间混合模型的系统，它不利用外部日志信息。摘要：We propose a diarization system, that estimates "who spoke when" based on spatial information, to be used as a front-end of a meeting transcription system running on the signals gathered from an acoustic sensor network (ASN). Although the spatial distribution of the microphones is advantageous, exploiting the spatial diversity for diarization and signal enhancement is challenging, because the microphones' positions are typically unknown, and the recorded signals are initially unsynchronized in general. Here, we approach these issues by first blindly synchronizing the signals and then estimating time differences of arrival (TDOAs). The TDOA information is exploited to estimate the speakers' activity, even in the presence of multiple speakers being simultaneously active. This speaker activity information serves as a guide for a spatial mixture model, on which basis the individual speaker's signals are extracted via beamforming. Finally, the extracted signals are forwarded to a speech recognizer. Additionally, a novel initialization scheme for spatial mixture models based on the TDOA estimates is proposed. Experiments conducted on real recordings from the LibriWASN data set have shown that our proposed system is advantageous compared to a system using a spatial mixture model, which does not make use of external diarization information.
eess.AS音频处理
【1】 Voice Anonymization for All -- Bias Evaluation of the Voice Privacy  Challenge Baseline System标题：面向所有人的语音匿名化--语音隐私挑战基线系统的偏差评估链接：https://arxiv.org/abs/2311.15804
作者：Anna Leschanowsky,Ünal Ege Gaznepoglu,Nils Peters备注：Submitted to ICASSP 2024摘要：在语音技术的时代，语音匿名化提供了一种保护人们隐私的解决方案，前提是这些系统在各个子群体中同样有效。本研究调查的语音隐私挑战的背景下，语音匿名系统的偏见。我们策划了一个新的基准数据集，以评估基于性别和方言的扬声器子组之间的性能差异。我们分析了三个匿名化系统和攻击模型对说话人分组偏见的影响，并揭示了显着的性能变化。值得注意的是，子组偏差随着高级攻击者能力的增强而加剧，强调了在所有子组中实现同等性能的挑战。我们的研究强调了对包容性基准数据集和全面评估策略的需求，这些策略可以解决语音匿名化中的亚组偏见。摘要：In an age of voice-enabled technology, voice anonymization offers a solution to protect people's privacy, provided these systems work equally well across subgroups. This study investigates bias in voice anonymization systems within the context of the Voice Privacy Challenge. We curate a novel benchmark dataset to assess performance disparities among speaker subgroups based on sex and dialect. We analyze the impact of three anonymization systems and attack models on speaker subgroup bias and reveal significant performance variations. Notably, subgroup bias intensifies with advanced attacker capabilities, emphasizing the challenge of achieving equal performance across all subgroups. Our study highlights the need for inclusive benchmark datasets and comprehensive evaluation strategies that address subgroup bias in voice anonymization.
【2】 Ultrasensitive Textile Strain Sensors Redefine Wearable Silent Speech  Interfaces with High Machine Learning Efficiency标题：超灵敏纺织品应变传感器以高机器学习效率重新定义可穿戴无声语音接口链接：https://arxiv.org/abs/2311.15683
作者：Chenyu Tang,Muzi Xu,Wentian Yi,Zibo Zhang,Edoardo Occhipinti,Chaoqun Dong,Dafydd Ravenscroft,Sung-Min Jung,Sanghyo Lee,Shuo Gao,Jong Min Kim,Luigi G. Occhipinti备注：5 figures in the article; 11 figures and 4 tables in supplementary information摘要：我们的研究提出了一种可穿戴的静音语音接口（SSI）技术，该技术在设备舒适度，时间能源效率和语音解码准确性方面表现出色，可用于现实世界。我们开发了一种生物相容性，耐用的纺织品choker与嵌入式石墨烯为基础的应变传感器，能够准确地检测微妙的喉咙运动。与传统的语音识别方法相比，该传感器的灵敏度超过其他应变传感器420%，简化了信号处理。我们的系统使用计算效率高的神经网络，特别是具有残差结构的一维卷积神经网络来解码语音信号。该网络具有能源和时间效率，将计算负载减少了90%，同时对20个单词的词典实现了95.25%的准确率，并以最少的样本快速适应新用户和单词。这一创新展示了一种实用、灵敏、精确的可穿戴SSI，适用于日常通信应用。摘要：Our research presents a wearable Silent Speech Interface (SSI) technology that excels in device comfort, time-energy efficiency, and speech decoding accuracy for real-world use. We developed a biocompatible, durable textile choker with an embedded graphene-based strain sensor, capable of accurately detecting subtle throat movements. This sensor, surpassing other strain sensors in sensitivity by 420%, simplifies signal processing compared to traditional voice recognition methods. Our system uses a computationally efficient neural network, specifically a one-dimensional convolutional neural network with residual structures, to decode speech signals. This network is energy and time-efficient, reducing computational load by 90% while achieving 95.25% accuracy for a 20-word lexicon and swiftly adapting to new users and words with minimal samples. This innovation demonstrates a practical, sensitive, and precise wearable SSI suitable for daily communication applications.
【3】 Spatial Diarization for Meeting Transcription with Ad-Hoc Acoustic  Sensor Networks标题：用于满足自组织声传感器网络转录的空间二值化链接：https://arxiv.org/abs/2311.15597
作者：Tobias Gburrek,Joerg Schmalenstroeer,Reinhold Haeb-Umbach备注：Accepted at Asilomar Conference on Signals, Systems, and Computers 2023摘要：我们提出了一个日记系统，估计“谁发言时”的基础上的空间信息，作为一个前端的会议转录系统上收集的信号从一个声学传感器网络（ESTA）运行。虽然麦克风的空间分布是有利的，但是利用空间分集进行日志化和信号增强是具有挑战性的，因为麦克风的位置通常是未知的，并且记录的信号通常最初是不同步的。在这里，我们首先通过盲同步信号，然后估计到达时间差（TDOA）来解决这些问题。利用TDOA信息来估计说话者的活动，即使在存在多个说话者同时活动的情况下。该说话者活动信息用作空间混合模型的指导，在该空间混合模型的基础上，经由波束成形来提取各个说话者的信号。最后，将提取的信号转发到语音识别器。此外，提出了一种新的初始化方案的基础上的TDOA估计空间混合模型。从LibriWirth数据集的真实记录进行的实验表明，我们提出的系统是有利的相比，使用空间混合模型的系统，它不利用外部日志信息。摘要：We propose a diarization system, that estimates "who spoke when" based on spatial information, to be used as a front-end of a meeting transcription system running on the signals gathered from an acoustic sensor network (ASN). Although the spatial distribution of the microphones is advantageous, exploiting the spatial diversity for diarization and signal enhancement is challenging, because the microphones' positions are typically unknown, and the recorded signals are initially unsynchronized in general. Here, we approach these issues by first blindly synchronizing the signals and then estimating time differences of arrival (TDOAs). The TDOA information is exploited to estimate the speakers' activity, even in the presence of multiple speakers being simultaneously active. This speaker activity information serves as a guide for a spatial mixture model, on which basis the individual speaker's signals are extracted via beamforming. Finally, the extracted signals are forwarded to a speech recognizer. Additionally, a novel initialization scheme for spatial mixture models based on the TDOA estimates is proposed. Experiments conducted on real recordings from the LibriWASN data set have shown that our proposed system is advantageous compared to a system using a spatial mixture model, which does not make use of external diarization information.
【4】 Learning Arousal-Valence Representation from Categorical Emotion Labels  of Speech标题：从语音的范畴情感标签学习唤醒-配价表征链接：https://arxiv.org/abs/2311.14816
作者：Enting Zhou,You Zhang,Zhiyao Duan摘要：语音情感的维度表示，如唤醒效价（AV）表示提供了一个连续的和细粒度的描述和控制比他们的分类同行。它们在动态情感理解和表达性文本到语音合成等任务中有着广泛的应用。现有的方法，预测的维度情感表示从语音铸造它作为一个监督回归任务。这些方法面临数据稀缺的问题，因为维度标注比分类标签更难获得。在这项工作中，我们建议从语音的分类情感标签中学习AV表示。我们首先使用自监督预训练和情感分类微调来学习丰富且与情感相关的高维语音特征表示。然后，根据心理学研究结果，通过锚定降维将该表示映射到2D AV空间。实验表明，我们的方法实现了一致性相关系数（CCC）性能，可与IEMOCAP上最先进的监督回归方法相媲美，而无需在训练期间利用地面实况AV注释。这验证了我们提出的AV预测方法。此外，在MEAD和MADB数据集上的AV预测的可视化显示了所学习的AV表示的可解释性。摘要：Dimensional representations of speech emotions such as the arousal-valence (AV) representation provide a continuous and fine-grained description and control than their categorical counterparts. They have wide applications in tasks such as dynamic emotion understanding and expressive text-to-speech synthesis. Existing methods that predict the dimensional emotion representation from speech cast it as a supervised regression task. These methods face data scarcity issues, as dimensional annotations are much harder to acquire than categorical labels. In this work, we propose to learn the AV representation from categorical emotion labels of speech. We start by learning a rich and emotion-relevant high-dimensional speech feature representation using self-supervised pre-training and emotion classification fine-tuning. This representation is then mapped to the 2D AV space according to psychological findings through anchored dimensionality reduction. Experiments show that our method achieves a Concordance Correlation Coefficient (CCC) performance comparable to state-of-the-art supervised regression methods on IEMOCAP without leveraging ground-truth AV annotations during training. This validates our proposed approach on AV prediction. Furthermore, visualization of AV predictions on MEAD and EmoDB datasets shows the interpretability of the learned AV representations.
【5】 CheapNET: Improving Light-weight speech enhancement network by projected  loss function标题：CheapNet：利用投影损失函数改进轻量级语音增强网络链接：https://arxiv.org/abs/2311.15959
作者：Kaijun Tan,Benzhe Dai,Jiakui Li,Wenyu Mao摘要：噪声抑制和回声消除在语音增强中至关重要，对于智能设备和实时通信至关重要。部署在语音处理前端和边缘设备中，这些算法必须确保高效的实时推理，同时计算需求较低。传统的基于边缘的噪声抑制通常使用基于MSE的幅度谱模板训练，但这种方法具有局限性。我们引入了一种新的投影损失函数，偏离MSE，以提高噪声抑制。该方法使用投影技术将关键音频分量与噪声隔离，从而显著提高模型性能。对于回声消除，该功能可以对LAEC预处理输出进行直接预测，从而大大提高性能。我们的噪声抑制模型仅用3.1M参数和0.4GFlops s的计算负载就达到了接近最先进的结果。此外，我们的回声消除模型优于复制的行业领先模型，为语音增强引入了新的视角。摘要：Noise suppression and echo cancellation are critical in speech enhancement and essential for smart devices and real-time communication. Deployed in voice processing front-ends and edge devices, these algorithms must ensure efficient real-time inference with low computational demands. Traditional edge-based noise suppression often uses MSE-based amplitude spectrum mask training, but this approach has limitations. We introduce a novel projection loss function, diverging from MSE, to enhance noise suppression. This method uses projection techniques to isolate key audio components from noise, significantly improving model performance. For echo cancellation, the function enables direct predictions on LAEC pre-processed outputs, substantially enhancing performance. Our noise suppression model achieves near state-of-the-art results with only 3.1M parameters and 0.4GFlops s computational load. Moreover, our echo cancellation model outperforms replicated industry-leading models, introducing a new perspective in speech enhancement.
【6】 A Quantitative Approach to Understand Self-Supervised Models as  Cross-lingual Feature Extractors标题：将自监督模型理解为跨语言特征提取的量化方法链接：https://arxiv.org/abs/2311.15954
作者：Shuyue Stella Li,Beining Xu,Xiangyu Zhang,Hexin Liu,Wenhan Chao,Leibny Paola Garcia备注：12 pages, 5 figures, 4 tables摘要：在这项工作中，我们研究了英语自监督学习（SSL）模型在跨语言背景下提取的特征，并提出了一个新的度量来预测特征表示的质量。使用自动语音识别（ASR）作为一个下游任务，我们分析了模型的大小，训练目标和模型架构对模型的性能作为一组拓扑多样的语料库的特征提取器的影响。我们开发了一种新的度量标准，语音比（PSR），使用深度广义典型相关分析来测量提取的表示中的语音和合成信息。结果表明，wav2vec2.0目标中的对比度损失有助于更有效的跨语言特征提取。PSR分数和ASR表现之间存在正相关，这表明单语SSL模型提取的语音信息可以用于跨语言环境中的下游任务。建议的度量是一个有效的指标的质量表示，可以是有用的模型选择。摘要：In this work, we study the features extracted by English self-supervised learning (SSL) models in cross-lingual contexts and propose a new metric to predict the quality of feature representations. Using automatic speech recognition (ASR) as a downstream task, we analyze the effect of model size, training objectives, and model architecture on the models' performance as a feature extractor for a set of topologically diverse corpora. We develop a novel metric, the Phonetic-Syntax Ratio (PSR), to measure the phonetic and synthetic information in the extracted representations using deep generalized canonical correlation analysis. Results show the contrastive loss in the wav2vec2.0 objective facilitates more effective cross-lingual feature extraction. There is a positive correlation between PSR scores and ASR performance, suggesting that phonetic information extracted by monolingual SSL models can be used for downstream tasks in cross-lingual settings. The proposed metric is an effective indicator of the quality of the representations and can be useful for model selection.
【7】 A-JEPA: Joint-Embedding Predictive Architecture Can Listen标题：A-JEPA：可监听的联合嵌入预测体系结构链接：https://arxiv.org/abs/2311.15830
作者：Zhengcong Fei,Mingyuan Fan,Junshi Huang摘要：本文提出，通过在潜在空间中进行预测，可以将推动大型基础视觉模型成功的掩蔽建模原理有效地应用于音频。我们介绍了基于音频的联合嵌入预测架构（A-JEPA），这是一种简单的扩展方法，用于从音频频谱中进行自监督学习。在I-JPEA的设计之后，我们的A-JEPA通过上下文编码器使用课程掩蔽策略对可见音频频谱图补丁进行编码，并预测在精心设计的位置处采样的区域的表示。通过上下文编码器的指数移动平均来提取这些区域的目标表示，目标编码器，在整个频谱图上。我们发现这是有益的随机块掩蔽到时间-频率感知掩蔽课程的方式，考虑到在本地时间和频率的音频频谱图的高度相关的复杂性。为了增强上下文语义理解和鲁棒性，我们对编码器进行了微调，对目标数据集进行了正则化掩码，而不是输入丢弃或零。从经验上讲，当使用Vision Transformers结构构建时，我们发现A-JEPA具有高度可扩展性，并在多个音频和语音分类任务上设置了新的最先进性能，优于其他使用外部监督预训练的最新模型。摘要：This paper presents that the masked-modeling principle driving the success of large foundational vision models can be effectively applied to audio by making predictions in a latent space. We introduce Audio-based Joint-Embedding Predictive Architecture (A-JEPA), a simple extension method for self-supervised learning from the audio spectrum. Following the design of I-JPEA, our A-JEPA encodes visible audio spectrogram patches with a curriculum masking strategy via context encoder, and predicts the representations of regions sampled at well-designed locations. The target representations of those regions are extracted by the exponential moving average of context encoder, emph{i.e.}, target encoder, on the whole spectrogram. We find it beneficial to transfer random block masking into time-frequency aware masking in a curriculum manner, considering the complexity of highly correlated in local time and frequency in audio spectrograms. To enhance contextual semantic understanding and robustness, we fine-tune the encoder with a regularized masking on target datasets, instead of input dropping or zero. Empirically, when built with Vision Transformers structure, we find A-JEPA to be highly scalable and sets new state-of-the-art performance on multiple audio and speech classification tasks, outperforming other recent models that use externally supervised pre-training.
【8】 Phonetic-aware speaker embedding for far-field speaker verification标题：语音感知的说话人嵌入远场说话人确认链接：https://arxiv.org/abs/2311.15627
作者：Zezhong Jin,Youzhi Tu,Man-Wai Mak备注：submitted to ICASSP2024摘要：当说话人确认（SV）系统远离声源工作时，由于噪声和混响的干扰，会出现重大挑战。研究表明，将语音信息融入说话人嵌入可以提高文本无关SV的性能。受此启发，我们提出了一个联合训练的语音识别和说话人识别（JTSS）框架，利用语音内容的远场SV。该框架通过将说话人嵌入网络的基于帧的特征图与wav2vec的向量进行匹配，来鼓励说话人嵌入保留语音信息。直觉是，语音信息可以与说话者信息一起保持低水平的声学动态，从而部分地补偿由于噪声和混响引起的退化。结果表明，该框架在VOiCES Challenge 2019评估集和VoxCeleb1测试集上的性能优于标准说话人嵌入。这表明在远场条件下利用语音信息对于学习鲁棒的说话人表示是有效的。摘要：When a speaker verification (SV) system operates far from the sound sourced, significant challenges arise due to the interference of noise and reverberation. Studies have shown that incorporating phonetic information into speaker embedding can improve the performance of text-independent SV. Inspired by this observation, we propose a joint-training speech recognition and speaker recognition (JTSS) framework to exploit phonetic content for far-field SV. The framework encourages speaker embeddings to preserve phonetic information by matching the frame-based feature maps of a speaker embedding network with wav2vec's vectors. The intuition is that phonetic information can preserve low-level acoustic dynamics with speaker information and thus partly compensate for the degradation due to noise and reverberation. Results show that the proposed framework outperforms the standard speaker embedding on the VOiCES Challenge 2019 evaluation set and the VoxCeleb1 test set. This indicates that leveraging phonetic information under far-field conditions is effective for learning robust speaker representations.
【9】 Lightly Weighted Automatic Audio Parameter Extraction for the Quality  Assessment of Consensus Auditory-Perceptual Evaluation of Voice标题：用于声音一致性听觉感知评价质量评估的轻加权自动音频参数提取链接：https://arxiv.org/abs/2311.15582
作者：Yi-Heng Lin,Wen-Hsuan Tseng,Li-Chin Chen,Ching-Ting Tan,Yu Tsao备注：Published in IEEE 42th International Conference on Consumer Electronics (ICCE 2024)摘要：语音的一致性听觉感知评价是临床语音质量评估中广泛使用的工具，对于临床专业人员之间的流式通信和确定进一步治疗的基准具有重要意义。目前，由于评估依赖于经验丰富的临床医生，因此往往不一致，因此难以标准化。为了解决这个问题，我们建议利用轻加权自动音频参数提取，以增加临床相关性，降低复杂性，并提高语音质量评估的可解释性。所提出的方法利用年龄，性别和五个音频参数：抖动，绝对抖动，闪烁，谐波噪声比（HNR），和零交叉。采用经典的机器学习方法。结果表明，我们的方法与最先进的（SOTA）方法相似，并且优于使用流行的音频预训练模型获得的潜在表示。这种方法提供了不同的语音评价的特征提取方法的可行性的见解。音频参数，如抖动和HNR被证明是适合的语音质量属性，如粗糙度和应变的特点。相反，预先训练的模型在有效解决噪声相关评分方面存在局限性。本研究通过对多种语音质量评估方法的全面探索，为更全面、更准确地评估语音质量做出了贡献。摘要：The Consensus Auditory-Perceptual Evaluation of Voice is a widely employed tool in clinical voice quality assessment that is significant for streaming communication among clinical professionals and benchmarking for the determination of further treatment. Currently, because the assessment relies on experienced clinicians, it tends to be inconsistent, and thus, difficult to standardize. To address this problem, we propose to leverage lightly weighted automatic audio parameter extraction, to increase the clinical relevance, reduce the complexity, and enhance the interpretability of voice quality assessment. The proposed method utilizes age, sex, and five audio parameters: jitter, absolute jitter, shimmer, harmonic-to-noise ratio (HNR), and zero crossing. A classical machine learning approach is employed. The result reveals that our approach performs similar to state-of-the-art (SOTA) methods, and outperforms the latent representation obtained by using popular audio pre-trained models. This approach provide insights into the feasibility of different feature extraction approaches for voice evaluation. Audio parameters such as jitter and the HNR are proven to be suitable for characterizing voice quality attributes, such as roughness and strain. Conversely, pre-trained models exhibit limitations in effectively addressing noise-related scorings. This study contributes toward more comprehensive and precise voice quality evaluations, achieved by a comprehensively exploring diverse assessment methodologies.
【10】 Weakly-Supervised Audio-Visual Segmentation标题：弱监督视听分割链接：https://arxiv.org/abs/2311.15080
作者：Shentong Mo,Bhiksha Raj摘要：视听分割是一项具有挑战性的任务，旨在预测视频中声源的像素级掩模。以前的工作应用了一个全面的手动设计的架构与无数的像素明智的准确的面具作为监督。然而，这些像素级掩模是昂贵的，并且不是在所有情况下都可用。在这项工作中，我们的目标是将监督简化为实例级注释，即，弱监督视听分割我们提出了一种新的弱监督视听分割框架，即WS-AVS，可以学习多尺度视听对齐与多尺度多实例对比学习视听分割。在AVSBench上进行的大量实验证明了WS-AVS在单源和多源弱监督音视频分割中的有效性。摘要：Audio-visual segmentation is a challenging task that aims to predict pixel-level masks for sound sources in a video. Previous work applied a comprehensive manually designed architecture with countless pixel-wise accurate masks as supervision. However, these pixel-level masks are expensive and not available in all cases. In this work, we aim to simplify the supervision as the instance-level annotation, i.e., weakly-supervised audio-visual segmentation. We present a novel Weakly-Supervised Audio-Visual Segmentation framework, namely WS-AVS, that can learn multi-scale audio-visual alignment with multi-scale multiple-instance contrastive learning for audio-visual segmentation. Extensive experiments on AVSBench demonstrate the effectiveness of our WS-AVS in the weakly-supervised audio-visual segmentation of single-source and multi-source scenarios.
【11】 Multi-Scale Sub-Band Constant-Q Transform Discriminator for  High-Fidelity Vocoder标题：高保真声码器的多尺度子带恒定Q变换鉴别器链接：https://arxiv.org/abs/2311.14957
作者：Yicheng Gu,Xueyao Zhang,Liumeng Xue,Zhizheng Wu摘要：当从声学表示重建可听波形时，基于生成对抗网络（GAN）的声码器在推理速度和合成质量方面优越。本研究的重点是改进的FPGA，以促进基于GAN的声码器。大多数现有的基于时频表示的鉴别器都植根于短时傅里叶变换（STFT），其在频谱图中的时频分辨率是固定的，这使得它与需要灵活注意不同频带的歌声等信号不兼容。基于此，我们的研究利用了恒Q变换（CQT），它具有频率之间的动态分辨率，有助于更好的建模能力，在基音精度和谐波跟踪。具体来说，我们提出了一种多尺度子带CQT（MS-SB-CQT）鉴别器，它在多个尺度上对CQT谱图进行操作，并根据不同的倍频程进行子带处理。对语音和歌声进行的实验证实了我们所提出的方法的有效性。此外，我们还验证了基于CQT和基于STFT的鉴别器可以在联合训练下互补。具体而言，通过所提出的MS-SB-CQT和现有的MS-STFT鉴别器的增强，HiFi-GAN的MOS可以从3.27提升到3.87，对于看到的歌手，从3.40提升到3.78。摘要：Generative Adversarial Network (GAN) based vocoders are superior in inference speed and synthesis quality when reconstructing an audible waveform from an acoustic representation. This study focuses on improving the discriminator to promote GAN-based vocoders. Most existing time-frequency-representation-based discriminators are rooted in Short-Time Fourier Transform (STFT), whose time-frequency resolution in a spectrogram is fixed, making it incompatible with signals like singing voices that require flexible attention for different frequency bands. Motivated by that, our study utilizes the Constant-Q Transform (CQT), which owns dynamic resolution among frequencies, contributing to a better modeling ability in pitch accuracy and harmonic tracking. Specifically, we propose a Multi-Scale Sub-Band CQT (MS-SB-CQT) Discriminator, which operates on the CQT spectrogram at multiple scales and performs sub-band processing according to different octaves. Experiments conducted on both speech and singing voices confirm the effectiveness of our proposed method. Moreover, we also verified that the CQT-based and the STFT-based discriminators could be complementary under joint training. Specifically, enhanced by the proposed MS-SB-CQT and the existing MS-STFT Discriminators, the MOS of HiFi-GAN can be boosted from 3.27 to 3.87 for seen singers and from 3.40 to 3.78 for unseen singers.
【12】 Custom Data Augmentation for low resource ASR using Bark and  Retrieval-Based Voice Conversion标题：使用Bark和基于检索的语音转换实现低资源ASR的定制数据增强链接：https://arxiv.org/abs/2311.14836
作者：Anand Kamble,Aniket Tathe,Suyash Kumbharkar,Atharva Bhandare,Anirban C. Mitra摘要：本文提出了两种创新的方法来构建定制的通用语音数据集的低资源的语言，如印地语。第一种方法利用Bark，一种由Suno开发的基于transformer的文本到音频模型，并结合Meta的enCodec和预训练的HuBert模型来增强Bark的性能。第二种方法采用基于检索的语音转换（RVC），并使用Ozen工具包进行数据准备。这两种方法都有助于ASR技术的进步，并为解决为资源不足的语言构建定制的通用语音数据集的挑战提供了宝贵的见解。此外，它们还为一系列应用提供了实现高质量、个性化语音生成的途径。摘要：This paper proposes two innovative methodologies to construct customized Common Voice datasets for low-resource languages like Hindi. The first methodology leverages Bark, a transformer-based text-to-audio model developed by Suno, and incorporates Meta's enCodec and a pre-trained HuBert model to enhance Bark's performance. The second methodology employs Retrieval-Based Voice Conversion (RVC) and uses the Ozen toolkit for data preparation. Both methodologies contribute to the advancement of ASR technology and offer valuable insights into addressing the challenges of constructing customized Common Voice datasets for under-resourced languages. Furthermore, they provide a pathway to achieving high-quality, personalized voice generation for a range of applications.
【13】 Weak Alignment Supervision from Hybrid Model Improves End-to-end ASR标题：来自混合模型的弱对齐监督改善了端到端ASR链接：https://arxiv.org/abs/2311.14835
作者：Jintao Jiang,Yingbo Gao,Zoltan Tuske备注：7 pages, 7 figures, and 5 tables摘要：在本文中，我们的目标是创建弱对齐监督，以帮助端到端建模。为此，我们使用现有的混合ASR系统来产生训练音频的三音素对齐。然后，我们使用导出的对齐在编码器的某个层创建交叉熵损失。与具有或不具有损失加权的一般one-hot交叉熵损失相比，这里我们使用具有标签平滑参数的交叉熵损失来正则化监督。作为比较，我们还进行了实验与一热交叉熵损失和CTC损失与损失加权。结果表明，将标签平滑参数为0.5的弱对齐监督放置在第三编码器层优于其他两种方法，并导致TED-LIUM 2数据集上的相对WER降低约5%。当我们将这种开箱即用的方法应用于TagServer端到端ASR系统时，我们看到了类似的改进。摘要：In this paper, we aim to create weak alignment supervision to aid the end-to-end modeling. Towards this end, we use the existing hybrid ASR system to produce triphone alignments of the training audios. We then create a cross-entropy loss at a certain layer of the encoder using the derived alignments. In contrast to the general one-hot cross-entropy losses with or without loss weighting, here we use a cross-entropy loss with a label smoothing parameter to regularize the supervision. As a comparison, we also conduct the experiments with one-hot cross-entropy losses and CTC losses with loss weighting. The results show that placing the weak alignment supervision with the label smoothing parameter of 0.5 at the third encoder layer outperforms the other two approaches and leads to about 5% relative WER reduction on the TED-LIUM 2 dataset over the baseline. We see similar improvements when applying the method out-of-the-box on a Tagalog end-to-end ASR system.
机器翻译，仅供参考