今日论文合集：cs.SD语音16篇，eess.AS音频处理19篇。
cs.SD语音

【1】 WhisperX: Time-Accurate Speech Transcription of Long-Form Audio标题：WhisperX：长格式音频的时间精确语音转录链接：http://arxiv.org/pdf/2303.00747v1

作者：Max Bain,Jaesung Huh,Tengda Han,Andrew Zisserman机构：Visual Geometry Group, University of Oxford, Voice Activity, Detection, Cut &, Merge, Phoneme, Model, Forced, Alignment, [,.,→,.,] That’s, [,.,→,.,] right., [,.,→,.,] Over, …, [,.,→,.,] Go!, Transcript &, word-level timestamps, Whisper, Batch摘要：大规模、弱监督的语音识别模型，如Whisper，在跨领域和跨语言的语音识别中取得了令人印象深刻的效果。然而，通过缓冲或滑动窗口方法将其应用于长音频转录容易出现漂移、幻觉和重复;并且由于它们的顺序特性而禁止批量转录。此外，对应于每一话语的时间戳易于不准确，且字级时间戳不可用。为了克服这些挑战，我们提出了WhisperX，一个时间精确的语音识别系统与词级时间戳利用语音活动检测和强制音素对齐。在这样做的过程中，我们展示了长格式转录和分词基准测试的最新性能。此外，我们还证明了使用我们提出的VAD剪切和合并策略对音频进行预分割可以提高转录质量，并通过批量推理使转录速度提高12倍。摘要：Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.

【2】 READ Avatars: Realistic Emotion-controllable Audio Driven Avatars标题：阅读头像：现实情感可控的音频驱动头像链接：http://arxiv.org/pdf/2303.00744v1

作者：Jack Saunders,Vinay Namboodiri机构：Our method, READ Avatars allows for the creation of photo-realistic and lip synchronized video from audio and, a reference video, with control over emotion. The same audio can be used to generate videos in multiple emotions.备注：13 Pages, 8 Figures For demo video see this https URL摘要：我们提出READ Avatars，一种基于3D的方法，用于生成由音频输入驱动的2D化身，具有对情感的直接和粒度控制。由于音频到表情映射的多对多性质，先前的方法不能实现逼真的动画。我们通过在音频到表达的生成过程中引入对抗损耗来缓解这个问题。这去除了基于回归的模型的平滑效果，并且有助于提高所生成的化身的真实感和表现力。此外，我们注意到，当生成嘴部内部时应当直接利用音频，并且其他基于3D的方法没有尝试这一点。我们用与分辨率无关的听觉神经纹理来解决这个问题。为了评估我们方法的性能，我们进行了定量和定性实验，包括用户研究。我们还提出了一种新的度量标准，用于比较演员的情感在生成的化身中重构得有多好。我们的结果表明，我们的方法优于最先进的音频驱动的化身生成方法在几个指标。演示视频可在 url{https：youtu.be QSyMl3vV0pA}上找到。摘要：We present READ Avatars, a 3D-based approach for generating 2D avatars that are driven by audio input with direct and granular control over the emotion. Previous methods are unable to achieve realistic animation due to the many-to-many nature of audio to expression mappings. We alleviate this issue by introducing an adversarial loss in the audio-to-expression generation process. This removes the smoothing effect of regression-based models and helps to improve the realism and expressiveness of the generated avatars. We note furthermore, that audio should be directly utilized when generating mouth interiors and that other 3D-based methods do not attempt this. We address this with audio-conditioned neural textures, which are resolution-independent. To evaluate the performance of our method, we perform quantitative and qualitative experiments, including a user study. We also propose a new metric for comparing how well an actor's emotion is reconstructed in the generated avatar. Our results show that our approach outperforms state of the art audio-driven avatar generation methods across several metrics. A demo video can be found at url{https: youtu.be QSyMl3vV0pA}.

【3】 A Comparison of Speech Data Augmentation Methods Using S3PRL Toolkit标题：基于S3PRL工具包的语音数据增强方法比较链接：http://arxiv.org/pdf/2303.00510v1

作者：Mina Huh,Ruchira Ray,Corey Karnei机构：Department of Computer Science, University of Texas at Austin, Department of Electrical and Computer Engineering摘要：已知数据扩充可以提高语音处理任务中的鲁棒性。在本研究中，我们总结并比较了使用S3PRL工具包的不同数据扩增策略。我们探索了HuBERT和wav2vec如何使用不同的增强技术（SpecAugment、高斯噪声、速度扰动）执行音素识别（PR）和自动语音识别（ASR）任务。我们用音素错误率（PER）和单词错误率（WER）来评估模型的性能。从实验中，我们观察到SpecAugment略微改善了HuBERT和wav2vec在原始数据集上的性能。此外，我们还表明，使用高斯噪声和速度扰动数据集训练的模型在使用扩充测试集进行测试时更加稳健。摘要：Data augmentations are known to improve robustness in speech-processing tasks. In this study, we summarize and compare different data augmentation strategies using S3PRL toolkit. We explore how HuBERT and wav2vec perform using different augmentation techniques (SpecAugment, Gaussian Noise, Speed Perturbation) for Phoneme Recognition (PR) and Automatic Speech Recognition (ASR) tasks. We evaluate model performance in terms of phoneme error rate (PER) and word error rate (WER). From the experiments, we observed that SpecAugment slightly improves the performance of HuBERT and wav2vec on the original dataset. Also, we show that models trained using the Gaussian Noise and Speed Perturbation dataset are more robust when tested with augmented test sets.

【4】 On the Audio-visual Synchronization for Lip-to-Speech Synthesis标题：唇语合成中的视听同步研究链接：http://arxiv.org/pdf/2303.00502v1

作者：Zhe Niu,Brian Mak机构：Department of Computer Science and Engineering, The Hong Kong University of Science and Technology摘要：大多数唇语合成模型是在假设数据集中的音视频对完全同步的情况下训练和评估的。在本文中，我们展示了常用的视听数据集，如GRID、TCD-TIMIT和Lip 2 Wav，可能存在数据异步问题。用这样的数据集训练唇到语音可能会进一步导致模型异步问题--也就是说，生成的语音和输入视频不同步。为了解决这些异步问题，我们提出了一个同步唇到语音（SLTS）模型与自动同步机制（ASM），以纠正数据异步和惩罚模型异步。进一步论证了常用的LTS异步测试数据评估指标的局限性，并在对时间对齐敏感的指标前引入音频对齐前端，以获得更好的评估效果。我们将我们的方法与传统和时间对齐度量上的最新方法进行比较，以显示同步训练的好处。摘要：Most lip-to-speech (LTS) synthesis models are trained and evaluated under the assumption that the audio-video pairs in the dataset are perfectly synchronized. In this work, we show that the commonly used audio-visual datasets, such as GRID, TCD-TIMIT, and Lip2Wav, can have data asynchrony issues. Training lip-to-speech with such datasets may further cause the model asynchrony issue -- that is, the generated speech and the input video are out of sync. To address these asynchrony issues, we propose a synchronized lip-to-speech (SLTS) model with an automatic synchronization mechanism (ASM) to correct data asynchrony and penalize model asynchrony. We further demonstrate the limitation of the commonly adopted evaluation metrics for LTS with asynchronous test data and introduce an audio alignment frontend before the metrics sensitive to time alignment for better evaluation. We compare our method with state-of-the-art approaches on conventional and time-aligned metrics to show the benefits of synchronization training.

【5】 N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space标题：N-Best T5：基于多输入假设和受限译码空间的稳健ASR纠错链接：http://arxiv.org/pdf/2303.00456v1

作者：Rao Ma,Mark J F Gales,Kate Knill,Mengjie Qian机构：ALTA Institute, Machine Intelligence Lab, Department of Engineering, Cambridge University, UK摘要：纠错模型是自动语音识别（ASR）后处理的重要组成部分，它可以提高语音的可读性和质量。大多数先前的工作使用1-最佳ASR假设作为输入，因此只能通过利用一个句子内的上下文来执行校正。在这项工作中，我们提出了一个新的N-最佳T5模型，它是从T5模型微调，并利用ASR N-最佳列表作为模型输入。通过从预先训练的语言模型传递知识并从ASR解码空间获得更丰富的信息，所提出的方法优于强Conformer-Transducer基线。标准纠错的另一个问题是生成过程没有得到很好的引导。为了解决这个问题，使用基于N最佳列表或ASR格的约束解码过程，其允许传播附加信息。摘要：Error correction models form an important part of Automatic Speech Recognition (ASR) post-processing to improve the readability and quality of transcriptions. Most prior works use the 1-best ASR hypothesis as input and therefore can only perform correction by leveraging the context within one sentence. In this work, we propose a novel N-best T5 model for this task, which is fine-tuned from a T5 model and utilizes ASR N-best lists as model input. By transferring knowledge from the pre-trained language model and obtaining richer information from the ASR decoding space, the proposed approach outperforms a strong Conformer-Transducer baseline. Another issue with standard error correction is that the generation process is not well-guided. To address this a constrained decoding process, either based on the N-best list or an ASR lattice, is used which allows additional information to be propagated.

【6】 CAM++: A Fast and Efficient Network For Speaker Verification Using Context-Aware Masking标题：CAM++：一种快速高效的上下文感知掩蔽说话人确认网络链接：http://arxiv.org/pdf/2303.00332v1

作者：Hui Wang,Siqi Zheng,Yafeng Chen,Luyao Cheng,Qian Chen机构：Speech Lab, Alibaba Group摘要：时延神经网络（TDNN）已被证明是学习区分性说话人嵌入的有效方法。ECAPA-TDNN是其成功的变体之一，它以更高的计算复杂度和更慢的推理速度为代价，在说话人确认任务中实现了最先进的性能。这使得它不足以满足具有苛刻推理速率和有限计算资源的场景。因此，我们希望找到一种既能达到ECAPA-TDNN的性能，又能达到普通TDNN的效率的体系结构。本文提出了一种基于上下文感知掩蔽的高效网络CAM++，该网络以密集连接时延神经网络（D-TDNN）为骨干，采用一种新的多粒度池技术从不同层次捕获上下文信息。在VoxCeleb和CN-Celeb两个公共基准测试上的实验表明，与其他主流说话人确认系统相比，该体系结构具有更低的计算开销和更快的推理速度。摘要：Time delay neural network (TDNN) has been proven to be efficient in learning discriminative speaker embeddings. One of its successful variant, ECAPA-TDNN, achieved state-of-the-art performance on speaker verification tasks at the cost of much higher computational complexity and slower inference speed. This makes it inadequate for scenarios with demanding inference rate and limited computational resources. We are thus interested in finding an architecture that can achieve the performance of ECAPA-TDNN and the efficiency of vanilla TDNN. In this paper, we propose an efficient network based on context-aware masking, namely CAM++, which uses densely connected time delay neural network (D-TDNN) as backbone and adopts a novel multi-granularity pooling to capture contextual information at different levels. Extensive experiments on two public benchmarks, VoxCeleb and CN-Celeb, demonstrate that the proposed architecture outperforms other mainstream speaker verification systems with lower computational cost and faster inference speed.

【7】 Distance-based Weight Transfer for Fine-tuning from Near-field to Far-field Speaker Verification标题：基于距离的近场到远场说话人精细调谐的权值传递链接：http://arxiv.org/pdf/2303.00264v1

作者：Li Zhang,Qing Wang,Hongji Wang,Yue Li,Wei Rao,Yannan Wang,Lei Xie机构：Northwestern Polytechnical University (NPU), Xi’an, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China摘要：标记远场语音的稀缺性是训练优良远场说话人确认系统的一个限制。微调在大规模近场语音上预训练的模型实质上优于从头训练。然而，微调方法有两个局限性--灾难性遗忘和过拟合。本文提出了一种权值转移正则化（WTR）损失来约束大规模近场语音的预训练模型和少量远场语音的微调模型之间的权值距离。在WTR损失的情况下，微调过程利用先前从大规模近场语音获得的区分能力，而不会发生灾难性遗忘。同时，利用PAC-Bayes推广理论分析了考虑WTR损失的模型的推广界。分析结果表明，WTR项的引入使得模型具有更严格的推广上界。此外，我们还探讨了三种用于权重转移的范数距离，即L1-范数距离、L2-范数距离和最大范数距离。最后，我们评价了VoxCeleb（预训练数据集）和FFSVC（微调数据集）数据集上WTR丢失的有效性。摘要：The scarcity of labeled far-field speech is a constraint for training superior far-field speaker verification systems. Fine-tuning the model pre-trained on large-scale near-field speech substantially outperforms training from scratch. However, the fine-tuning method suffers from two limitations--catastrophic forgetting and overfitting. In this paper, we propose a weight transfer regularization(WTR) loss to constrain the distance of the weights between the pre-trained model with large-scale near-field speech and the fine-tuned model through a small number of far-field speech. With the WTR loss, the fine-tuning process takes advantage of the previously acquired discriminative ability from the large-scale near-field speech without catastrophic forgetting. Meanwhile, we use the PAC-Bayes generalization theory to analyze the generalization bound of the fine-tuned model with the WTR loss. The analysis result indicates that the WTR term makes the fine-tuned model have a tighter generalization upper bound. Moreover, we explore three kinds of norm distance for weight transfer, which are L1-norm distance, L2-norm distance and Max-norm distance. Finally, we evaluate the effectiveness of the WTR loss on VoxCeleb (pre-trained dataset) and FFSVC (fine-tuned dataset) datasets.

【8】PCF: ECAPA-TDNN with Progressive Channel Fusion for Speaker Verification标题：用于说话人确认的渐进通道融合PCF：ECAPA-TDNN链接：http://arxiv.org/pdf/2303.00204v1

作者：Zhenduo Zhao,Zhuo Li,Wenchao Wang,Pengyuan Zhang机构：Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese, University of Chinese Academy of Sciences, Beijing, China备注：Accepted by ICASSP 2023摘要：ECAPA-TDNN是目前最流行的TDNN系列说话人确认模型，它刷新了TDNN模型的最先进（SOTA）性能。然而，一维卷积在特征通道上具有全局感受野。它破坏了频谱图的时频相关性。此外，由于ECAPA-TDNN只有五层，与ResNet相比，结构浅得多，限制了生成深度表示的能力。为了进一步改进ECAPA-TDNN，提出了一种渐进式通道融合策略，该策略将频谱图分割为多个特征通道，并通过网络逐步扩展感受野。其次，通过扩展深度和增加分支来扩大模型。该模型在vox 1 o上的EER为0.718，minDCF（0.01）为0.0858，与ECAPA-TDNN-large相比分别提高了16.1%和19.5%。摘要：ECAPA-TDNN is currently the most popular TDNN-series model for speaker verification, which refreshed the state-of-the-art(SOTA) performance of TDNN models. However, one-dimensional convolution has a global receptive field over the feature channel. It destroys the time-frequency relevance of the spectrogram. Besides, as ECAPA-TDNN only has five layers, a much shallower structure compared to ResNet restricts the capability to generate deep representations. To further improve ECAPA-TDNN, we propose a progressive channel fusion strategy that splits the spectrogram across the feature channel and gradually expands the receptive field through the network. Secondly, we enlarge the model by extending the depth and adding branches. Our proposed model achieves EER with 0.718 and minDCF(0.01) with 0.0858 on vox1o, relatively improved 16.1 % and 19.5 % compared with ECAPA-TDNN-large.

【9】 I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue标题：我比你先知道你的感受：在人机对话中预测未来的情感反应链接：http://arxiv.org/pdf/2303.00146v1

作者：Yuanchao Li,Koji Inoue,Leimin Tian,Changzeng Fu,Carlos Ishi,Hiroshi Ishiguro,Tatsuya Kawahara,Catherine Lai机构：University of Edinburgh, UK, Kyoto University, Japan, Monash University, Australia, u.ac.jp, Osaka University & RIKEN, ATR & RIKEN, Osaka University & ATR备注：Accepted to CHI2023 LBW. Part of this work was done while working on the JST ERATO ISHIGURO Symbiotic Human-Robot Interaction Project at Kawahara Lab at Kyoto University. {dag}Authors contributed equally and are listed in alphabetical order摘要：当前的口语对话系统（SDS）通常充当被动收听者，其仅在接收到用户语音之后才作出响应。为了实现类人对话，我们提出一种新的未来预测架构，允许SDS在用户说话之前根据其当前行为预测未来的情感反应。在本研究中，我们研究了两种情况：言语和笑声。在语音中，我们提出基于用户的未来情绪与系统的当前情绪的时间关系以及用户的未来情绪与系统的当前对话行为（DA）的因果关系来预测用户的未来情绪。在笑声方面，我们提出利用系统在当前回合中的笑声行为来预测用户笑声的发生和类型。对人机对话的初步分析表明，人与机器人在情感和笑声上具有同步性，对话中DA-情感具有因果关系。这证明了我们的体系结构可以为预期SDS的开发做出贡献。摘要：Current Spoken Dialogue Systems (SDSs) often serve as passive listeners that respond only after receiving user speech. To achieve human-like dialogue, we propose a novel future prediction architecture that allows an SDS to anticipate future affective reactions based on its current behaviors before the user speaks. In this work, we investigate two scenarios: speech and laughter. In speech, we propose to predict the user's future emotion based on its temporal relationship with the system's current emotion and its causal relationship with the system's current Dialogue Act (DA). In laughter, we propose to predict the occurrence and type of the user's laughter using the system's laughter behaviors in the current turn. Preliminary analysis of human-robot dialogue demonstrated synchronicity in the emotions and laughter displayed by the human and robot, as well as DA-emotion causality in their dialogue. This verifies that our architecture can contribute to the development of an anticipatory SDS.

【10】 ClArTTS: An Open-Source Classical Arabic Text-to-Speech Corpus标题：ClArTTS：一个开源的经典阿拉伯语文语转换语料库链接：http://arxiv.org/pdf/2303.00069v1

作者：Ajinkya Kulkarni,Atharva Kulkarni,Sara Abedalmonem Mohammad Shatnawi,Hanan Aldarmaki机构：Aldarmaki⋆, MBZUAI UAE⋆, Erisha Labs India摘要：目前，使用端到端神经模型用高质量转录语音数据训练的文本到语音（TTS）系统可以生成可理解的、自然的并且非常类似于人类语音的语音。这些模型用相对较大的单个扬声器专业记录的音频（通常从有声读物中提取）来训练。同时，由于此类免费语料库的稀缺，阿拉伯语TTS的研究与开发存在较大差距。大多数现有的可免费获得的阿拉伯语语音语料库不适合于TTS训练，因为它们包含在记录条件和质量方面变化的多说话者随意语音，而为语音合成而策划的语料库通常尺寸小并且不适合于训练最先进的端到端模型。为了填补这一资源缺口，我们提出了一个经典阿拉伯语文本到语音（ClArTTS）的语音语料库，以支持阿拉伯语端到端TTS系统的开发。语音是从LibriVox有声读物中提取出来的，然后经过处理、分段、人工转录和注释。最终的ClArTTS语料库包含以40100 kHz采样的来自单个男性说话者的大约12小时的语音。在本文中，我们描述了语料库的创建过程，并提供了详细的语料库统计和与现有资源的比较。此外，我们开发了基于Grad-TTS和Glow-TTS的两个TTS系统，并通过主观和客观评价说明了所得到的系统的性能。该语料库将在www.example.com上公开www.clartts.com以供研究之用，同时还将提供基本的TTS系统演示。摘要：At present, Text-to-speech (TTS) systems that are trained with high-quality transcribed speech data using end-to-end neural models can generate speech that is intelligible, natural, and closely resembles human speech. These models are trained with relatively large single-speaker professionally recorded audio, typically extracted from audiobooks. Meanwhile, due to the scarcity of freely available speech corpora of this kind, a larger gap exists in Arabic TTS research and development. Most of the existing freely available Arabic speech corpora are not suitable for TTS training as they contain multi-speaker casual speech with variations in recording conditions and quality, whereas the corpus curated for speech synthesis are generally small in size and not suitable for training state-of-the-art end-to-end models. In a move towards filling this gap in resources, we present a speech corpus for Classical Arabic Text-to-Speech (ClArTTS) to support the development of end-to-end TTS systems for Arabic. The speech is extracted from a LibriVox audiobook, which is then processed, segmented, and manually transcribed and annotated. The final ClArTTS corpus contains about 12 hours of speech from a single male speaker sampled at 40100 kHz. In this paper, we describe the process of corpus creation and provide details of corpus statistics and a comparison with existing resources. Furthermore, we develop two TTS systems based on Grad-TTS and Glow-TTS and illustrate the performance of the resulting systems via subjective and objective evaluations. The corpus will be made publicly available at www.clartts.com for research purposes, along with the baseline TTS systems demo.

【11】SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks标题：SpeechPrompt v2：语音分类任务的提示调整链接：http://arxiv.org/pdf/2303.00733v1

作者：Kai-Wei Chang,Yu-Kai Wang,Hua Shen,Iu-thing Kang,Wei-Cheng Tseng,Shang-Wen Li,Hung-yi Lee机构：Graduate Institute of Communication Engineering, National Taiwan University, Taiwan, National Taiwan University, Taiwan,Pennsylvania State University,Meta AI备注：Project website: this https URL摘要：即时调优是一种技术，它调优一小组参数来引导预先训练的语言模型（LM）直接生成下游任务的输出。近年来，即时调优在自然语言处理和语音处理领域都显示出了其存储和计算效率。这些优点还揭示了迅速调整作为候选方法，以统一的方式为多个任务服务预先训练的LM。对于语音处理，SpeechPrompt在一些语音分类任务上显示出其高参数效率和竞争性性能。但是，SpeechPrompt是否能够服务于大量的任务还没有答案。在本文中，我们提出了SpeechPrompt v2，这是一个能够执行多种语音分类任务的提示调优框架，涵盖了多种语言和韵律相关的任务。实验结果表明，SpeechPrompt v2在一个统一的框架下，使用不到0.15M的可训练参数，取得了与现有算法相当的性能。摘要：Prompt tuning is a technology that tunes a small set of parameters to steer a pre-trained language model (LM) to directly generate the output for downstream tasks. Recently, prompt tuning has demonstrated its storage and computation efficiency in both natural language processing (NLP) and speech processing fields. These advantages have also revealed prompt tuning as a candidate approach to serving pre-trained LM for multiple tasks in a unified manner. For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks. However, whether SpeechPrompt is capable of serving a large number of tasks is unanswered. In this work, we propose SpeechPrompt v2, a prompt tuning framework capable of performing a wide variety of speech classification tasks, covering multiple languages and prosody-related tasks. The experiment result shows that SpeechPrompt v2 achieves performance on par with prior works with less than 0.15M trainable parameters in a unified framework.

【12】 audb -- Sharing and Versioning of Audio and Annotation Data in Python标题：AUDB--在Python中实现音频和注释数据的共享和版本控制

链接：http://arxiv.org/pdf/2303.00645v1

作者：Hagen Wierstorf,Johannes Wagner,Florian Eyben,Felix Burkhardt,Björn W. Schuller机构：audEERING GmbH, Gilching, Germany, Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany, GLAM – Group on Language, Audio, & Music, Imperial College, UK摘要：由于需要更大、更多样化的数据集来预训练和微调日益复杂的机器学习模型，数据集的数量正在快速增长。audb是一个开源Python库，支持音频数据集的版本控制和文档。它旨在提供一个标准化的简单用户界面来发布、维护和访问数据集的注释和音频文件。为了有效地将数据存储在服务器上，audb会自动解析数据集版本之间的依赖关系，并且只在发布新版本时上载新添加或更改的文件。该库支持部分加载数据集和本地缓存，以便快速访问已下载的数据。AudB是轻量级库，并且可以与任何机器学习库接口。它支持在大学或公司内或整个研究社区内的单台PC上管理数据集。摘要：Driven by the need for larger and more diverse datasets to pre-train and fine-tune increasingly complex machine learning models, the number of datasets is rapidly growing. audb is an open-source Python library that supports versioning and documentation of audio datasets. It aims to provide a standardized and simple user-interface to publish, maintain, and access the annotations and audio files of a dataset. To efficiently store the data on a server, audb automatically resolves dependencies between versions of a dataset and only uploads newly added or altered files when a new version is published. The library supports partial loading of a dataset and local caching for fast access to already downloaded data. audb is a lightweight library and can be interfaced with any machine learning library. It supports the management of datasets on a single PC, within a university or company, or within a whole research community.

【13】 Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation标题：基于精英抽样和集成知识提炼的ASR领域泛化链接：http://arxiv.org/pdf/2303.00550v1

作者：Rehan Ahmad,Md Asif Jalal,Muhammad Umar Farooq,Anna Ollerenshaw,Thomas Hain机构：Speech and Hearing Group, The University of Sheffield, UK摘要：知识提取已经广泛地用于语音应用的模型压缩和域自适应。在多个教师在场的情况下，通过对模型输出求平均值，可以很容易地将知识传递给学生。然而，以往的研究表明，学生并不适应这种组合。本文提出在集成教师模型的输出端使用精英抽样策略，选择完全域外教师模型生成的最佳解码语音，用于泛化不可见域。教师模型在AMI、LibriSpeech和WSJ上训练，而学生模型适应于Switchboard数据。结果表明，使用基于个体模型后验的选择策略，学生模型与所有教师和基线相比实现了更好的WER，最小绝对改进约为8.4%。此外，通过相关性分析，我们还研究了域外数据的模型自适应问题。摘要：Knowledge distillation has widely been used for model compression and domain adaptation for speech applications. In the presence of multiple teachers, knowledge can easily be transferred to the student by averaging the models output. However, previous research shows that the student do not adapt well with such combination. This paper propose to use an elitist sampling strategy at the output of ensemble teacher models to select the best-decoded utterance generated by completely out-of-domain teacher models for generalizing unseen domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the student is adapted for the Switchboard data. The results show that with the selection strategy based on the individual models posteriors the student model achieves a better WER compared to all the teachers and baselines with a minimum absolute improvement of about 8.4 percent. Furthermore, an insights on the model adaptation with out-of-domain data has also been studied via correlation analysis.

【14】 Extending DNN-based Multiplicative Masking to Deep Subband Filtering for Improved Dereverberation标题：将基于DNN的乘性掩蔽扩展到深子带滤波以改进混响链接：http://arxiv.org/pdf/2303.00529v1

作者：Jean-Marie Lemercier,Julian Tobergte,Timo Gerkmann机构：Universit¨at Hamburg摘要：本文提出了一种将基于深度神经网络的乘性掩蔽器扩展到深度子带滤波器的方案，用于时频域语音恢复。所得到的方法可以一般地应用于在时间-频率域中提供掩码的任何深度神经网络，同时仅需要几个更多的可训练参数和对于现有技术的神经网络可忽略的计算开销。我们证明了所得到的深子带滤波方案在去混响方面优于乘性掩蔽，同时保持了几乎相同的去噪性能。我们认为这是因为时频域中的深子带滤波适合于去混响文献中经常假设的子带近似，而乘法掩蔽对应于去噪中通常采用的窄带近似。摘要：In this paper, we present a scheme for extending deep neural network-based multiplicative maskers to deep subband filters for speech restoration in the time-frequency domain. The resulting method can be generically applied to any deep neural network providing masks in the time-frequency domain, while requiring only few more trainable parameters and a computational overhead that is negligible for state-of-the-art neural networks. We demonstrate that the resulting deep subband filtering scheme outperforms multiplicative masking for dereverberation, while leaving the denoising performance virtually the same. We argue that this is because deep subband filtering in the time-frequency domain fits the subband approximation often assumed in the dereverberation literature, whereas multiplicative masking corresponds to the narrowband approximation generally employed in denoising.

【15】 First-shot anomaly sound detection for machine condition monitoring: A domain generalization baseline标题：用于机器状态监测的首炮异常声检测：一种域泛化基线链接：http://arxiv.org/pdf/2303.00455v1

作者：Noboru Harada,Daisuke Niizumi,Yasunori Ohishi,Daiki Takeuchi,Masahiro Yasuda机构：NTT Corporation, Japan备注：5 pages, 2 figures摘要：本文提出了一种基于First-shot的无监督异常检测（ASD）的基线系统。First-shot ASD不允许系统基于用大事实计算的性能度量来进行机器类型相关的超参数调整或刀具装配。为了展示First-shot ASD的基准性能，本文提出了一种异常声音检测系统，该系统在声学场景和事件的检测和分类（DCASE）2022挑战任务2中的域泛化任务上工作："应用域泛化技术进行机器状态监测的无监督异常声音检测"，同时符合DCASE 2023挑战任务2（DCASE2023T2）中引入的第一次要求。一个简单的基于自动编码器的实现结合选择性马氏度量被实现为基线系统。性能评估旨在为即将推出的DCASE2023T2设定目标基准。基线系统的源代码将在GitHub上提供：www.example.com网站。https: github.com nttcslab dcase2023_task2_baseline_ae 。摘要：This paper provides a baseline system for First-shot-compliant unsupervised anomaly detection (ASD) for machine condition monitoring. First-shot ASD does not allow systems to do machine-type dependent hyperparameter tuning or tool ensembling based on the performance metric calculated with the grand truth. To show benchmark performance for First-shot ASD, this paper proposes an anomaly sound detection system that works on the domain generalization task in the Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Challenge Task 2: "Unsupervised Anomalous Sound Detection for Machine Condition Monitoring Applying Domain Generalization Technique" while complying with the First-shot requirements introduced in the DCASE 2023 Challenge Task 2 (DCASE2023T2). A simple autoencoder based implementation combined with selective Mahalanobis metric is implemented as a baseline system. The performance evaluation is conducted to set the target benchmark for the forthcoming DCASE2023T2. Source code of the baseline system will be available on GitHub: https: github.com nttcslab dcase2023_task2_baseline_ae .

【16】 Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model标题：用视觉语言预训练模型提高医学语音到文本的准确率链接：http://arxiv.org/pdf/2303.00091v1

作者：Jaeyoung Huh,Sangjoon Park,Jeong Eun Lee,Jong Chul Ye机构：Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon , Republic of Korea摘要：自动语音识别（ASR）是一种将口语转换为文本的技术，有助于人机交互。ASR最常见的应用之一是语音到文本（STT）技术，该技术通过将口语转录为文本来简化用户工作流。在医疗领域，STT有可能显著减少临床医生的工作量，因为他们依赖打字员转录他们的语音记录。然而，由于缺乏足够的语音和文本数据集，开发用于医学领域的STT模型具有挑战性。为了解决这一问题，我们提出了一种医学领域文本校正方法，该方法使用视觉语言预训练（VLP）方法修改通用STT系统的输出文本。VLP结合文本和视觉信息来基于图像知识校正文本。我们广泛的实验证明，所提出的方法在医学领域中提供了STT性能的定量和临床显著改进。我们进一步证明了图像和文本信息的多模态理解优于仅使用文本信息的单模态理解。摘要：Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.
eess.AS音频处理

【1】SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks标题：SpeechPrompt v2：语音分类任务的提示调整链接：http://arxiv.org/pdf/2303.00733v1

*与cs.SD语音【11】为同一篇

作者：Kai-Wei Chang,Yu-Kai Wang,Hua Shen,Iu-thing Kang,Wei-Cheng Tseng,Shang-Wen Li,Hung-yi Lee机构：Graduate Institute of Communication Engineering, National Taiwan University, Taiwan, National Taiwan University, Taiwan,Pennsylvania State University,Meta AI备注：Project website: this https URL摘要：即时调优是一种技术，它调优一小组参数来引导预先训练的语言模型（LM）直接生成下游任务的输出。近年来，即时调优在自然语言处理和语音处理领域都显示出了其存储和计算效率。这些优点还揭示了迅速调整作为候选方法，以统一的方式为多个任务服务预先训练的LM。对于语音处理，SpeechPrompt在一些语音分类任务上显示出其高参数效率和竞争性性能。但是，SpeechPrompt是否能够服务于大量的任务还没有答案。在本文中，我们提出了SpeechPrompt v2，这是一个能够执行多种语音分类任务的提示调优框架，涵盖了多种语言和韵律相关的任务。实验结果表明，SpeechPrompt v2在一个统一的框架下，使用不到0.15M的可训练参数，取得了与现有算法相当的性能。摘要：Prompt tuning is a technology that tunes a small set of parameters to steer a pre-trained language model (LM) to directly generate the output for downstream tasks. Recently, prompt tuning has demonstrated its storage and computation efficiency in both natural language processing (NLP) and speech processing fields. These advantages have also revealed prompt tuning as a candidate approach to serving pre-trained LM for multiple tasks in a unified manner. For speech processing, SpeechPrompt shows its high parameter efficiency and competitive performance on a few speech classification tasks. However, whether SpeechPrompt is capable of serving a large number of tasks is unanswered. In this work, we propose SpeechPrompt v2, a prompt tuning framework capable of performing a wide variety of speech classification tasks, covering multiple languages and prosody-related tasks. The experiment result shows that SpeechPrompt v2 achieves performance on par with prior works with less than 0.15M trainable parameters in a unified framework.

【2】Leveraging Redundancy in Multiple Audio Signals for Far-Field Speech Recognition标题：利用多个音频信号中的冗余进行远场语音识别链接：http://arxiv.org/pdf/2303.00692v1

作者：Feng-Ju Chang,Anastasios Alexandridis,Rupak Vignesh Swaminathan,Martin Radfar,Harish Mallidi,Maurizio Omologo,Athanasios Mouchtaris,Brian King,Roland Maas机构：Alexa Speech, Amazon, USA摘要：为了实现鲁棒的远场自动语音识别（ASR），现有技术通常采用与神经换能器（NT）ASR模型级联的声学前端（AFE）。然而，AFE输出可能不可靠，因为AFE中的波束成形输出被引导到错误的方向。解决这一问题的一种有前途的方法是在AFE中的波束形成阶段之前和声学回声消除（后AEC）之后利用麦克风信号。我们认为，AEC后和AFE输出是互补的，可以利用这些信号之间的冗余来补偿潜在的AFE处理误差。我们提出两种融合网络来探索这种冗余并聚合这些多通道（MC）信号：（1）基于频率-LSTM，以及（2）基于卷积神经网络的融合网络。我们将MC融合网络扩展为构象传感器模型，并以端到端的方式对其进行训练。我们在商业虚拟助手任务上的实验结果表明，使用AFE输出和两个AEC后信号与融合网络相比，在参数增加〈= 2%的代价下，仅使用AFE输出的模型提供了高达25.9%的字错误率（WER）相对改善。摘要：To achieve robust far-field automatic speech recognition (ASR), existing techniques typically employ an acoustic front end (AFE) cascaded with a neural transducer (NT) ASR model. The AFE output, however, could be unreliable, as the beamforming output in AFE is steered to a wrong direction. A promising way to address this issue is to exploit the microphone signals before the beamforming stage and after the acoustic echo cancellation (post-AEC) in AFE. We argue that both, post-AEC and AFE outputs, are complementary and it is possible to leverage the redundancy between these signals to compensate for potential AFE processing errors. We present two fusion networks to explore this redundancy and aggregate these multi-channel (MC) signals: (1) Frequency-LSTM based, and (2) Convolutional Neural Network based fusion networks. We augment the MC fusion networks to a conformer transducer model and train it in an end-to-end fashion. Our experimental results on commercial virtual assistant tasks demonstrate that using the AFE output and two post-AEC signals with fusion networks offers up to 25.9% word error rate (WER) relative improvement over the model using the AFE output only, at the cost of <= 2% parameter increase.

【3】 audb -- Sharing and Versioning of Audio and Annotation Data in Python标题：AUDB--在Python中实现音频和注释数据的共享和版本控制

链接：http://arxiv.org/pdf/2303.00645v1

*与cs.SD语音【12】为同一篇

作者：Hagen Wierstorf,Johannes Wagner,Florian Eyben,Felix Burkhardt,Björn W. Schuller机构：audEERING GmbH, Gilching, Germany, Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Germany, GLAM – Group on Language, Audio, & Music, Imperial College, UK摘要：由于需要更大、更多样化的数据集来预训练和微调日益复杂的机器学习模型，数据集的数量正在快速增长。audb是一个开源Python库，支持音频数据集的版本控制和文档。它旨在提供一个标准化的简单用户界面来发布、维护和访问数据集的注释和音频文件。为了有效地将数据存储在服务器上，audb会自动解析数据集版本之间的依赖关系，并且只在发布新版本时上载新添加或更改的文件。该库支持部分加载数据集和本地缓存，以便快速访问已下载的数据。AudB是轻量级库，并且可以与任何机器学习库接口。它支持在大学或公司内或整个研究社区内的单台PC上管理数据集。摘要：Driven by the need for larger and more diverse datasets to pre-train and fine-tune increasingly complex machine learning models, the number of datasets is rapidly growing. audb is an open-source Python library that supports versioning and documentation of audio datasets. It aims to provide a standardized and simple user-interface to publish, maintain, and access the annotations and audio files of a dataset. To efficiently store the data on a server, audb automatically resolves dependencies between versions of a dataset and only uploads newly added or altered files when a new version is published. The library supports partial loading of a dataset and local caching for fast access to already downloaded data. audb is a lightweight library and can be interfaced with any machine learning library. It supports the management of datasets on a single PC, within a university or company, or within a whole research community.

【4】 Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation标题：基于精英抽样和集成知识提炼的ASR领域泛化链接：http://arxiv.org/pdf/2303.00550v1

*与cs.SD语音【13】为同一篇

作者：Rehan Ahmad,Md Asif Jalal,Muhammad Umar Farooq,Anna Ollerenshaw,Thomas Hain机构：Speech and Hearing Group, The University of Sheffield, UK摘要：知识提取已经广泛地用于语音应用的模型压缩和域自适应。在多个教师在场的情况下，通过对模型输出求平均值，可以很容易地将知识传递给学生。然而，以往的研究表明，学生并不适应这种组合。本文提出在集成教师模型的输出端使用精英抽样策略，选择完全域外教师模型生成的最佳解码语音，用于泛化不可见域。教师模型在AMI、LibriSpeech和WSJ上训练，而学生模型适应于Switchboard数据。结果表明，使用基于个体模型后验的选择策略，学生模型与所有教师和基线相比实现了更好的WER，最小绝对改进约为8.4%。此外，通过相关性分析，我们还研究了域外数据的模型自适应问题。摘要：Knowledge distillation has widely been used for model compression and domain adaptation for speech applications. In the presence of multiple teachers, knowledge can easily be transferred to the student by averaging the models output. However, previous research shows that the student do not adapt well with such combination. This paper propose to use an elitist sampling strategy at the output of ensemble teacher models to select the best-decoded utterance generated by completely out-of-domain teacher models for generalizing unseen domain. The teacher models are trained on AMI, LibriSpeech and WSJ while the student is adapted for the Switchboard data. The results show that with the selection strategy based on the individual models posteriors the student model achieves a better WER compared to all the teachers and baselines with a minimum absolute improvement of about 8.4 percent. Furthermore, an insights on the model adaptation with out-of-domain data has also been studied via correlation analysis.

【5】 Extending DNN-based Multiplicative Masking to Deep Subband Filtering for Improved Dereverberation标题：将基于DNN的乘性掩蔽扩展到深子带滤波以改进混响链接：http://arxiv.org/pdf/2303.00529v1

*与cs.SD语音【14】为同一篇

作者：Jean-Marie Lemercier,Julian Tobergte,Timo Gerkmann机构：Universit¨at Hamburg摘要：本文提出了一种将基于深度神经网络的乘性掩蔽器扩展到深度子带滤波器的方案，用于时频域语音恢复。所得到的方法可以一般地应用于在时间-频率域中提供掩码的任何深度神经网络，同时仅需要几个更多的可训练参数和对于现有技术的神经网络可忽略的计算开销。我们证明了所得到的深子带滤波方案在去混响方面优于乘性掩蔽，同时保持了几乎相同的去噪性能。我们认为这是因为时频域中的深子带滤波适合于去混响文献中经常假设的子带近似，而乘法掩蔽对应于去噪中通常采用的窄带近似。摘要：In this paper, we present a scheme for extending deep neural network-based multiplicative maskers to deep subband filters for speech restoration in the time-frequency domain. The resulting method can be generically applied to any deep neural network providing masks in the time-frequency domain, while requiring only few more trainable parameters and a computational overhead that is negligible for state-of-the-art neural networks. We demonstrate that the resulting deep subband filtering scheme outperforms multiplicative masking for dereverberation, while leaving the denoising performance virtually the same. We argue that this is because deep subband filtering in the time-frequency domain fits the subband approximation often assumed in the dereverberation literature, whereas multiplicative masking corresponds to the narrowband approximation generally employed in denoising.

【6】 First-shot anomaly sound detection for machine condition monitoring: A domain generalization baseline标题：用于机器状态监测的首炮异常声检测：一种域泛化基线链接：http://arxiv.org/pdf/2303.00455v1

*与cs.SD语音【15】为同一篇

作者：Noboru Harada,Daisuke Niizumi,Yasunori Ohishi,Daiki Takeuchi,Masahiro Yasuda机构：NTT Corporation, Japan备注：5 pages, 2 figures摘要：本文提出了一种基于First-shot的无监督异常检测（ASD）的基线系统。First-shot ASD不允许系统基于用大事实计算的性能度量来进行机器类型相关的超参数调整或刀具装配。为了展示First-shot ASD的基准性能，本文提出了一种异常声音检测系统，该系统在声学场景和事件的检测和分类（DCASE）2022挑战任务2中的域泛化任务上工作："应用域泛化技术进行机器状态监测的无监督异常声音检测"，同时符合DCASE 2023挑战任务2（DCASE2023T2）中引入的第一次要求。一个简单的基于自动编码器的实现结合选择性马氏度量被实现为基线系统。性能评估旨在为即将推出的DCASE2023T2设定目标基准。基线系统的源代码将在GitHub上提供：www.example.com网站。https: github.com nttcslab dcase2023_task2_baseline_ae 。摘要：This paper provides a baseline system for First-shot-compliant unsupervised anomaly detection (ASD) for machine condition monitoring. First-shot ASD does not allow systems to do machine-type dependent hyperparameter tuning or tool ensembling based on the performance metric calculated with the grand truth. To show benchmark performance for First-shot ASD, this paper proposes an anomaly sound detection system that works on the domain generalization task in the Detection and Classification of Acoustic Scenes and Events (DCASE) 2022 Challenge Task 2: "Unsupervised Anomalous Sound Detection for Machine Condition Monitoring Applying Domain Generalization Technique" while complying with the First-shot requirements introduced in the DCASE 2023 Challenge Task 2 (DCASE2023T2). A simple autoencoder based implementation combined with selective Mahalanobis metric is implemented as a baseline system. The performance evaluation is conducted to set the target benchmark for the forthcoming DCASE2023T2. Source code of the baseline system will be available on GitHub: https: github.com nttcslab dcase2023_task2_baseline_ae .

【7】 Improving Medical Speech-to-Text Accuracy with Vision-Language Pre-training Model标题：用视觉语言预训练模型提高医学语音到文本的准确率链接：http://arxiv.org/pdf/2303.00091v1

*与cs.SD语音【16】为同一篇

作者：Jaeyoung Huh,Sangjoon Park,Jeong Eun Lee,Jong Chul Ye机构：Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon , Republic of Korea摘要：自动语音识别（ASR）是一种将口语转换为文本的技术，有助于人机交互。ASR最常见的应用之一是语音到文本（STT）技术，该技术通过将口语转录为文本来简化用户工作流。在医疗领域，STT有可能显著减少临床医生的工作量，因为他们依赖打字员转录他们的语音记录。然而，由于缺乏足够的语音和文本数据集，开发用于医学领域的STT模型具有挑战性。为了解决这一问题，我们提出了一种医学领域文本校正方法，该方法使用视觉语言预训练（VLP）方法修改通用STT系统的输出文本。VLP结合文本和视觉信息来基于图像知识校正文本。我们广泛的实验证明，所提出的方法在医学领域中提供了STT性能的定量和临床显著改进。我们进一步证明了图像和文本信息的多模态理解优于仅使用文本信息的单模态理解。摘要：Automatic Speech Recognition (ASR) is a technology that converts spoken words into text, facilitating interaction between humans and machines. One of the most common applications of ASR is Speech-To-Text (STT) technology, which simplifies user workflows by transcribing spoken words into text. In the medical field, STT has the potential to significantly reduce the workload of clinicians who rely on typists to transcribe their voice recordings. However, developing an STT model for the medical domain is challenging due to the lack of sufficient speech and text datasets. To address this issue, we propose a medical-domain text correction method that modifies the output text of a general STT system using the Vision Language Pre-training (VLP) method. VLP combines textual and visual information to correct text based on image knowledge. Our extensive experiments demonstrate that the proposed method offers quantitatively and clinically significant improvements in STT performance in the medical field. We further show that multi-modal understanding of image and text information outperforms single-modal understanding using only text information.

【8】 WhisperX: Time-Accurate Speech Transcription of Long-Form Audio标题：WhisperX：长格式音频的时间精确语音转录链接：http://arxiv.org/pdf/2303.00747v1

*与cs.SD语音【1】为同一篇

作者：Max Bain,Jaesung Huh,Tengda Han,Andrew Zisserman机构：Visual Geometry Group, University of Oxford, Voice Activity, Detection, Cut &, Merge, Phoneme, Model, Forced, Alignment, [,.,→,.,] That’s, [,.,→,.,] right., [,.,→,.,] Over, …, [,.,→,.,] Go!, Transcript &, word-level timestamps, Whisper, Batch摘要：大规模、弱监督的语音识别模型，如Whisper，在跨领域和跨语言的语音识别中取得了令人印象深刻的效果。然而，通过缓冲或滑动窗口方法将其应用于长音频转录容易出现漂移、幻觉和重复;并且由于它们的顺序特性而禁止批量转录。此外，对应于每一话语的时间戳易于不准确，且字级时间戳不可用。为了克服这些挑战，我们提出了WhisperX，一个时间精确的语音识别系统与词级时间戳利用语音活动检测和强制音素对齐。在这样做的过程中，我们展示了长格式转录和分词基准测试的最新性能。此外，我们还证明了使用我们提出的VAD剪切和合并策略对音频进行预分割可以提高转录质量，并通过批量推理使转录速度提高12倍。摘要：Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelve-fold transcription speedup via batched inference.

【9】 READ Avatars: Realistic Emotion-controllable Audio Driven Avatars标题：阅读头像：现实情感可控的音频驱动头像链接：http://arxiv.org/pdf/2303.00744v1

*与cs.SD语音【2】为同一篇

作者：Jack Saunders,Vinay Namboodiri机构：Our method, READ Avatars allows for the creation of photo-realistic and lip synchronized video from audio and, a reference video, with control over emotion. The same audio can be used to generate videos in multiple emotions.备注：13 Pages, 8 Figures For demo video see this https URL摘要：我们提出READ Avatars，一种基于3D的方法，用于生成由音频输入驱动的2D化身，具有对情感的直接和粒度控制。由于音频到表情映射的多对多性质，先前的方法不能实现逼真的动画。我们通过在音频到表达的生成过程中引入对抗损耗来缓解这个问题。这去除了基于回归的模型的平滑效果，并且有助于提高所生成的化身的真实感和表现力。此外，我们注意到，当生成嘴部内部时应当直接利用音频，并且其他基于3D的方法没有尝试这一点。我们用与分辨率无关的听觉神经纹理来解决这个问题。为了评估我们方法的性能，我们进行了定量和定性实验，包括用户研究。我们还提出了一种新的度量标准，用于比较演员的情感在生成的化身中重构得有多好。我们的结果表明，我们的方法优于最先进的音频驱动的化身生成方法在几个指标。演示视频可在 url{https：youtu.be QSyMl3vV0pA}上找到。摘要：We present READ Avatars, a 3D-based approach for generating 2D avatars that are driven by audio input with direct and granular control over the emotion. Previous methods are unable to achieve realistic animation due to the many-to-many nature of audio to expression mappings. We alleviate this issue by introducing an adversarial loss in the audio-to-expression generation process. This removes the smoothing effect of regression-based models and helps to improve the realism and expressiveness of the generated avatars. We note furthermore, that audio should be directly utilized when generating mouth interiors and that other 3D-based methods do not attempt this. We address this with audio-conditioned neural textures, which are resolution-independent. To evaluate the performance of our method, we perform quantitative and qualitative experiments, including a user study. We also propose a new metric for comparing how well an actor's emotion is reconstructed in the generated avatar. Our results show that our approach outperforms state of the art audio-driven avatar generation methods across several metrics. A demo video can be found at url{https: youtu.be QSyMl3vV0pA}.

【10】 MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation标题：MuAViC：用于稳健语音识别和稳健语音到文本翻译的多语言视听语料库链接：http://arxiv.org/pdf/2303.00628v1

作者：Mohamed Anwar,Bowen Shi,Vedanuj Goswami,Wei-Ning Hsu,Juan Pino,Changhan Wang机构：Meta AI摘要：我们介绍MuAViC，一个用于鲁棒语音识别和鲁棒语音到文本翻译的多语言视听语料库，提供9种语言的1200小时视听语音。它是完全转录和涵盖6英语到X的翻译以及6 X到英语的翻译方向。据我们所知，这是首个面向音视频语音转文本翻译的开放基准，也是最大的面向多语种音视频语音识别的开放基准。实验结果表明，MuAViC能够有效地构建抗噪语音识别和翻译模型。我们在www.example.com上提供语料库https: github.com facebookresearch muavic。摘要：We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. It is fully transcribed and covers 6 English-to-X translation as well as 6 X-to-English translation directions. To the best of our knowledge, this is the first open benchmark for audio-visual speech-to-text translation and the largest open benchmark for multilingual audio-visual speech recognition. Our baseline results show that MuAViC is effective for building noise-robust speech recognition and translation models. We make the corpus available at https: github.com facebookresearch muavic.

【11】 A Comparison of Speech Data Augmentation Methods Using S3PRL Toolkit标题：基于S3PRL工具包的语音数据增强方法比较链接：http://arxiv.org/pdf/2303.00510v1

*与cs.SD语音【3】为同一篇

作者：Mina Huh,Ruchira Ray,Corey Karnei机构：Department of Computer Science, University of Texas at Austin, Department of Electrical and Computer Engineering摘要：已知数据扩充可以提高语音处理任务中的鲁棒性。在本研究中，我们总结并比较了使用S3PRL工具包的不同数据扩增策略。我们探索了HuBERT和wav2vec如何使用不同的增强技术（SpecAugment、高斯噪声、速度扰动）执行音素识别（PR）和自动语音识别（ASR）任务。我们用音素错误率（PER）和单词错误率（WER）来评估模型的性能。从实验中，我们观察到SpecAugment略微改善了HuBERT和wav2vec在原始数据集上的性能。此外，我们还表明，使用高斯噪声和速度扰动数据集训练的模型在使用扩充测试集进行测试时更加稳健。摘要：Data augmentations are known to improve robustness in speech-processing tasks. In this study, we summarize and compare different data augmentation strategies using S3PRL toolkit. We explore how HuBERT and wav2vec perform using different augmentation techniques (SpecAugment, Gaussian Noise, Speed Perturbation) for Phoneme Recognition (PR) and Automatic Speech Recognition (ASR) tasks. We evaluate model performance in terms of phoneme error rate (PER) and word error rate (WER). From the experiments, we observed that SpecAugment slightly improves the performance of HuBERT and wav2vec on the original dataset. Also, we show that models trained using the Gaussian Noise and Speed Perturbation dataset are more robust when tested with augmented test sets.

【12】 On the Audio-visual Synchronization for Lip-to-Speech Synthesis标题：唇语合成中的视听同步研究链接：http://arxiv.org/pdf/2303.00502v1

*与cs.SD语音【4】为同一篇

作者：Zhe Niu,Brian Mak机构：Department of Computer Science and Engineering, The Hong Kong University of Science and Technology摘要：大多数唇语合成模型是在假设数据集中的音视频对完全同步的情况下训练和评估的。在本文中，我们展示了常用的视听数据集，如GRID、TCD-TIMIT和Lip 2 Wav，可能存在数据异步问题。用这样的数据集训练唇到语音可能会进一步导致模型异步问题--也就是说，生成的语音和输入视频不同步。为了解决这些异步问题，我们提出了一个同步唇到语音（SLTS）模型与自动同步机制（ASM），以纠正数据异步和惩罚模型异步。进一步论证了常用的LTS异步测试数据评估指标的局限性，并在对时间对齐敏感的指标前引入音频对齐前端，以获得更好的评估效果。我们将我们的方法与传统和时间对齐度量上的最新方法进行比较，以显示同步训练的好处。摘要：Most lip-to-speech (LTS) synthesis models are trained and evaluated under the assumption that the audio-video pairs in the dataset are perfectly synchronized. In this work, we show that the commonly used audio-visual datasets, such as GRID, TCD-TIMIT, and Lip2Wav, can have data asynchrony issues. Training lip-to-speech with such datasets may further cause the model asynchrony issue -- that is, the generated speech and the input video are out of sync. To address these asynchrony issues, we propose a synchronized lip-to-speech (SLTS) model with an automatic synchronization mechanism (ASM) to correct data asynchrony and penalize model asynchrony. We further demonstrate the limitation of the commonly adopted evaluation metrics for LTS with asynchronous test data and introduce an audio alignment frontend before the metrics sensitive to time alignment for better evaluation. We compare our method with state-of-the-art approaches on conventional and time-aligned metrics to show the benefits of synchronization training.

【13】 N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space标题：N-Best T5：基于多输入假设和受限译码空间的稳健ASR纠错链接：http://arxiv.org/pdf/2303.00456v1

*与cs.SD语音【5】为同一篇

作者：Rao Ma,Mark J F Gales,Kate Knill,Mengjie Qian机构：ALTA Institute, Machine Intelligence Lab, Department of Engineering, Cambridge University, UK摘要：纠错模型是自动语音识别（ASR）后处理的重要组成部分，它可以提高语音的可读性和质量。大多数先前的工作使用1-最佳ASR假设作为输入，因此只能通过利用一个句子内的上下文来执行校正。在这项工作中，我们提出了一个新的N-最佳T5模型，它是从T5模型微调，并利用ASR N-最佳列表作为模型输入。通过从预先训练的语言模型传递知识并从ASR解码空间获得更丰富的信息，所提出的方法优于强Conformer-Transducer基线。标准纠错的另一个问题是生成过程没有得到很好的引导。为了解决这个问题，使用基于N最佳列表或ASR格的约束解码过程，其允许传播附加信息。摘要：Error correction models form an important part of Automatic Speech Recognition (ASR) post-processing to improve the readability and quality of transcriptions. Most prior works use the 1-best ASR hypothesis as input and therefore can only perform correction by leveraging the context within one sentence. In this work, we propose a novel N-best T5 model for this task, which is fine-tuned from a T5 model and utilizes ASR N-best lists as model input. By transferring knowledge from the pre-trained language model and obtaining richer information from the ASR decoding space, the proposed approach outperforms a strong Conformer-Transducer baseline. Another issue with standard error correction is that the generation process is not well-guided. To address this a constrained decoding process, either based on the N-best list or an ASR lattice, is used which allows additional information to be propagated.

【14】 CAM++: A Fast and Efficient Network For Speaker Verification Using Context-Aware Masking标题：CAM++：一种快速高效的上下文感知掩蔽说话人确认网络链接：http://arxiv.org/pdf/2303.00332v1

*与cs.SD语音【6】为同一篇

作者：Hui Wang,Siqi Zheng,Yafeng Chen,Luyao Cheng,Qian Chen机构：Speech Lab, Alibaba Group摘要：时延神经网络（TDNN）已被证明是学习区分性说话人嵌入的有效方法。ECAPA-TDNN是其成功的变体之一，它以更高的计算复杂度和更慢的推理速度为代价，在说话人确认任务中实现了最先进的性能。这使得它不足以满足具有苛刻推理速率和有限计算资源的场景。因此，我们希望找到一种既能达到ECAPA-TDNN的性能，又能达到普通TDNN的效率的体系结构。本文提出了一种基于上下文感知掩蔽的高效网络CAM++，该网络以密集连接时延神经网络（D-TDNN）为骨干，采用一种新的多粒度池技术从不同层次捕获上下文信息。在VoxCeleb和CN-Celeb两个公共基准测试上的实验表明，与其他主流说话人确认系统相比，该体系结构具有更低的计算开销和更快的推理速度。摘要：Time delay neural network (TDNN) has been proven to be efficient in learning discriminative speaker embeddings. One of its successful variant, ECAPA-TDNN, achieved state-of-the-art performance on speaker verification tasks at the cost of much higher computational complexity and slower inference speed. This makes it inadequate for scenarios with demanding inference rate and limited computational resources. We are thus interested in finding an architecture that can achieve the performance of ECAPA-TDNN and the efficiency of vanilla TDNN. In this paper, we propose an efficient network based on context-aware masking, namely CAM++, which uses densely connected time delay neural network (D-TDNN) as backbone and adopts a novel multi-granularity pooling to capture contextual information at different levels. Extensive experiments on two public benchmarks, VoxCeleb and CN-Celeb, demonstrate that the proposed architecture outperforms other mainstream speaker verification systems with lower computational cost and faster inference speed.

【15】 Distance-based Weight Transfer for Fine-tuning from Near-field to Far-field Speaker Verification标题：基于距离的近场到远场说话人精细调谐的权值传递链接：http://arxiv.org/pdf/2303.00264v1

*与cs.SD语音【7】为同一篇

作者：Li Zhang,Qing Wang,Hongji Wang,Yue Li,Wei Rao,Yannan Wang,Lei Xie机构：Northwestern Polytechnical University (NPU), Xi’an, China, Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China摘要：标记远场语音的稀缺性是训练优良远场说话人确认系统的一个限制。微调在大规模近场语音上预训练的模型实质上优于从头训练。然而，微调方法有两个局限性--灾难性遗忘和过拟合。本文提出了一种权值转移正则化（WTR）损失来约束大规模近场语音的预训练模型和少量远场语音的微调模型之间的权值距离。在WTR损失的情况下，微调过程利用先前从大规模近场语音获得的区分能力，而不会发生灾难性遗忘。同时，利用PAC-Bayes推广理论分析了考虑WTR损失的模型的推广界。分析结果表明，WTR项的引入使得模型具有更严格的推广上界。此外，我们还探讨了三种用于权重转移的范数距离，即L1-范数距离、L2-范数距离和最大范数距离。最后，我们评价了VoxCeleb（预训练数据集）和FFSVC（微调数据集）数据集上WTR丢失的有效性。摘要：The scarcity of labeled far-field speech is a constraint for training superior far-field speaker verification systems. Fine-tuning the model pre-trained on large-scale near-field speech substantially outperforms training from scratch. However, the fine-tuning method suffers from two limitations--catastrophic forgetting and overfitting. In this paper, we propose a weight transfer regularization(WTR) loss to constrain the distance of the weights between the pre-trained model with large-scale near-field speech and the fine-tuned model through a small number of far-field speech. With the WTR loss, the fine-tuning process takes advantage of the previously acquired discriminative ability from the large-scale near-field speech without catastrophic forgetting. Meanwhile, we use the PAC-Bayes generalization theory to analyze the generalization bound of the fine-tuned model with the WTR loss. The analysis result indicates that the WTR term makes the fine-tuned model have a tighter generalization upper bound. Moreover, we explore three kinds of norm distance for weight transfer, which are L1-norm distance, L2-norm distance and Max-norm distance. Finally, we evaluate the effectiveness of the WTR loss on VoxCeleb (pre-trained dataset) and FFSVC (fine-tuned dataset) datasets.

【16】PCF: ECAPA-TDNN with Progressive Channel Fusion for Speaker Verification标题：用于说话人确认的渐进通道融合PCF：ECAPA-TDNN链接：http://arxiv.org/pdf/2303.00204v1

*与cs.SD语音【8】为同一篇

作者：Zhenduo Zhao,Zhuo Li,Wenchao Wang,Pengyuan Zhang机构：Key Laboratory of Speech Acoustics and Content Understanding, Institute of Acoustics, Chinese, University of Chinese Academy of Sciences, Beijing, China备注：Accepted by ICASSP 2023摘要：ECAPA-TDNN是目前最流行的TDNN系列说话人确认模型，它刷新了TDNN模型的最先进（SOTA）性能。然而，一维卷积在特征通道上具有全局感受野。它破坏了频谱图的时频相关性。此外，由于ECAPA-TDNN只有五层，与ResNet相比，结构浅得多，限制了生成深度表示的能力。为了进一步改进ECAPA-TDNN，提出了一种渐进式通道融合策略，该策略将频谱图分割为多个特征通道，并通过网络逐步扩展感受野。其次，通过扩展深度和增加分支来扩大模型。该模型在vox 1 o上的EER为0.718，minDCF（0.01）为0.0858，与ECAPA-TDNN-large相比分别提高了16.1%和19.5%。摘要：ECAPA-TDNN is currently the most popular TDNN-series model for speaker verification, which refreshed the state-of-the-art(SOTA) performance of TDNN models. However, one-dimensional convolution has a global receptive field over the feature channel. It destroys the time-frequency relevance of the spectrogram. Besides, as ECAPA-TDNN only has five layers, a much shallower structure compared to ResNet restricts the capability to generate deep representations. To further improve ECAPA-TDNN, we propose a progressive channel fusion strategy that splits the spectrogram across the feature channel and gradually expands the receptive field through the network. Secondly, we enlarge the model by extending the depth and adding branches. Our proposed model achieves EER with 0.718 and minDCF(0.01) with 0.0858 on vox1o, relatively improved 16.1 % and 19.5 % compared with ECAPA-TDNN-large.

【17】DTW-SiameseNet: Dynamic Time Warped Siamese Network for Mispronunciation Detection and Correction标题：DTW-SiameseNet：用于错误发音检测和纠正的动态时间扭曲暹罗网络链接：http://arxiv.org/pdf/2303.00171v1

作者：Raviteja Anantha,Kriti Bhasin,Daniela de la Parra Aguilar,Prabal Vashisht,Becci Williamson,Srinivas Chappidi机构：Apple备注：Preprint version摘要：个人数字助理（PDA）--例如Siri、Alexa和Google Assistant等-,在跨多个域访问信息和完成任务方面发挥着越来越重要的作用，用户群体也各不相同。文本到语音（TTS）模块允许PDA以自然的、类似人类的方式进行交互，并且当交互涉及具有视觉障碍或其他残疾的人时，TTS模块发挥了至关重要的作用。为了满足不同用户的需求，包容性TTS对于识别不同语言和方言的文本并正确发音非常重要。尽管语音合成技术取得了很大的进步，但命名实体在多语言环境下的发音准确性仍有很大的提高空间。纠正命名实体（NE）错误发音的现有方法，如重新训练字形到音素（G2 P）模型，或维护TTS发音字典，需要对地面真实发音进行昂贵的注释，这也是耗时的。在这项工作中，我们提出了一个高精度的，PDA兼容的发音学习框架的TTS错误发音检测和纠正的任务。此外，我们还提出了一种新的错误发音检测模型DTW-SiameseNet，该模型采用Siamese结构的度量学习方法，用于具有三元组丢失的动态时间规整（DTW）。我们证明了一个区域无关的，隐私保护的方案来解决TTS错误发音检测问题是可行的。我们在真实世界的数据集上评估了我们的方法，以及来自10个不同地点的参与者记录的人名匿名音频数据集的NE发音语料库。人类评估表明，与基于音素和基于音频的基准相比，我们提出的方法平均提高了约6%的发音准确率。摘要：Personal Digital Assistants (PDAs) - such as Siri, Alexa and Google Assistant, to name a few - play an increasingly important role to access information and complete tasks spanning multiple domains, and by diverse groups of users. A text-to-speech (TTS) module allows PDAs to interact in a natural, human-like manner, and play a vital role when the interaction involves people with visual impairments or other disabilities. To cater to the needs of a diverse set of users, inclusive TTS is important to recognize and pronounce correctly text in different languages and dialects. Despite great progress in speech synthesis, the pronunciation accuracy of named entities in a multi-lingual setting still has a large room for improvement. Existing approaches to correct named entity (NE) mispronunciations, like retraining Grapheme-to-Phoneme (G2P) models, or maintaining a TTS pronunciation dictionary, require expensive annotation of the ground truth pronunciation, which is also time consuming. In this work, we present a highly-precise, PDA-compatible pronunciation learning framework for the task of TTS mispronunciation detection and correction. In addition, we also propose a novel mispronunciation detection model called DTW-SiameseNet, which employs metric learning with a Siamese architecture for Dynamic Time Warping (DTW) with triplet loss. We demonstrate that a locale-agnostic, privacy-preserving solution to the problem of TTS mispronunciation detection is feasible. We evaluate our approach on a real-world dataset, and a corpus of NE pronunciations of an anonymized audio dataset of person names recorded by participants from 10 different locales. Human evaluation shows our proposed approach improves pronunciation accuracy on average by ~6% compared to strong phoneme-based and audio-based baseline

【18】 I Know Your Feelings Before You Do: Predicting Future Affective Reactions in Human-Computer Dialogue标题：我比你先知道你的感受：在人机对话中预测未来的情感反应链接：http://arxiv.org/pdf/2303.00146v1

*与cs.SD语音【9】为同一篇

作者：Yuanchao Li,Koji Inoue,Leimin Tian,Changzeng Fu,Carlos Ishi,Hiroshi Ishiguro,Tatsuya Kawahara,Catherine Lai机构：University of Edinburgh, UK, Kyoto University, Japan, Monash University, Australia, u.ac.jp, Osaka University & RIKEN, ATR & RIKEN, Osaka University & ATR备注：Accepted to CHI2023 LBW. Part of this work was done while working on the JST ERATO ISHIGURO Symbiotic Human-Robot Interaction Project at Kawahara Lab at Kyoto University. {dag}Authors contributed equally and are listed in alphabetical order摘要：当前的口语对话系统（SDS）通常充当被动收听者，其仅在接收到用户语音之后才作出响应。为了实现类人对话，我们提出一种新的未来预测架构，允许SDS在用户说话之前根据其当前行为预测未来的情感反应。在本研究中，我们研究了两种情况：言语和笑声。在语音中，我们提出基于用户的未来情绪与系统的当前情绪的时间关系以及用户的未来情绪与系统的当前对话行为（DA）的因果关系来预测用户的未来情绪。在笑声方面，我们提出利用系统在当前回合中的笑声行为来预测用户笑声的发生和类型。对人机对话的初步分析表明，人与机器人在情感和笑声上具有同步性，对话中DA-情感具有因果关系。这证明了我们的体系结构可以为预期SDS的开发做出贡献。摘要：Current Spoken Dialogue Systems (SDSs) often serve as passive listeners that respond only after receiving user speech. To achieve human-like dialogue, we propose a novel future prediction architecture that allows an SDS to anticipate future affective reactions based on its current behaviors before the user speaks. In this work, we investigate two scenarios: speech and laughter. In speech, we propose to predict the user's future emotion based on its temporal relationship with the system's current emotion and its causal relationship with the system's current Dialogue Act (DA). In laughter, we propose to predict the occurrence and type of the user's laughter using the system's laughter behaviors in the current turn. Preliminary analysis of human-robot dialogue demonstrated synchronicity in the emotions and laughter displayed by the human and robot, as well as DA-emotion causality in their dialogue. This verifies that our architecture can contribute to the development of an anticipatory SDS.

【19】 ClArTTS: An Open-Source Classical Arabic Text-to-Speech Corpus标题：ClArTTS：一个开源的经典阿拉伯语文语转换语料库链接：http://arxiv.org/pdf/2303.00069v1

*与cs.SD语音【10】为同一篇

作者：Ajinkya Kulkarni,Atharva Kulkarni,Sara Abedalmonem Mohammad Shatnawi,Hanan Aldarmaki机构：Aldarmaki⋆, MBZUAI UAE⋆, Erisha Labs India摘要：目前，使用端到端神经模型用高质量转录语音数据训练的文本到语音（TTS）系统可以生成可理解的、自然的并且非常类似于人类语音的语音。这些模型用相对较大的单个扬声器专业记录的音频（通常从有声读物中提取）来训练。同时，由于此类免费语料库的稀缺，阿拉伯语TTS的研究与开发存在较大差距。大多数现有的可免费获得的阿拉伯语语音语料库不适合于TTS训练，因为它们包含在记录条件和质量方面变化的多说话者随意语音，而为语音合成而策划的语料库通常尺寸小并且不适合于训练最先进的端到端模型。为了填补这一资源缺口，我们提出了一个经典阿拉伯语文本到语音（ClArTTS）的语音语料库，以支持阿拉伯语端到端TTS系统的开发。语音是从LibriVox有声读物中提取出来的，然后经过处理、分段、人工转录和注释。最终的ClArTTS语料库包含以40100 kHz采样的来自单个男性说话者的大约12小时的语音。在本文中，我们描述了语料库的创建过程，并提供了详细的语料库统计和与现有资源的比较。此外，我们开发了基于Grad-TTS和Glow-TTS的两个TTS系统，并通过主观和客观评价说明了所得到的系统的性能。该语料库将在www.example.com上公开www.clartts.com以供研究之用，同时还将提供基本的TTS系统演示。摘要：At present, Text-to-speech (TTS) systems that are trained with high-quality transcribed speech data using end-to-end neural models can generate speech that is intelligible, natural, and closely resembles human speech. These models are trained with relatively large single-speaker professionally recorded audio, typically extracted from audiobooks. Meanwhile, due to the scarcity of freely available speech corpora of this kind, a larger gap exists in Arabic TTS research and development. Most of the existing freely available Arabic speech corpora are not suitable for TTS training as they contain multi-speaker casual speech with variations in recording conditions and quality, whereas the corpus curated for speech synthesis are generally small in size and not suitable for training state-of-the-art end-to-end models. In a move towards filling this gap in resources, we present a speech corpus for Classical Arabic Text-to-Speech (ClArTTS) to support the development of end-to-end TTS systems for Arabic. The speech is extracted from a LibriVox audiobook, which is then processed, segmented, and manually transcribed and annotated. The final ClArTTS corpus contains about 12 hours of speech from a single male speaker sampled at 40100 kHz. In this paper, we describe the process of corpus creation and provide details of corpus statistics and a comparison with existing resources. Furthermore, we develop two TTS systems based on Grad-TTS and Glow-TTS and illustrate the performance of the resulting systems via subjective and objective evaluations. The corpus will be made publicly available at www.clartts.com for research purposes, along with the baseline TTS systems demo.

机器翻译，仅供参考