今日论文合集：cs.SD语音9篇，eess.AS音频处理11篇。
【1】 Speaking Style Conversion With Discrete Self-Supervised Units
标题：用离散的自我监督单元进行讲话风格转换
链接：https://arxiv.org/abs/2212.09730
作者：Gallil Maimon,Yossi Adi机构：School of Computer Science and Engineering, The Hebrew University of Jerusalem摘要：语音转换（VC）是使一个说话者说出的话语听起来像是由另一个说话者说出的，同时保持其他方面如内容不变的任务。目前的VC方法主要关注音色等频谱特征，而忽略了人们独特的说话风格对韵律的影响。在本研究中，我们介绍了一种不仅将音色，而且将韵律信息（即，节奏和音调变化）到目标说话人的那些。所提出的方法基于一个预先训练的、自监督的、用于将语音编码为离散单元的模型，这使得它简单、有效并且易于优化。我们考虑没有配对数据的多对多设置。我们为该设置引入了一套定量和定性的评估度量，并通过实证证明了所提出的方法显著优于所评估的基线。代码和示例可在www.example.com下找到https://pages.cs.huji.ac.il/adiyoss-lab/dissc/。摘要：Voice Conversion (VC) is the task of making a spoken utterance by one speaker sound as if uttered by a different speaker, while keeping other aspects like content unchanged. Current VC methods, focus primarily on spectral features like timbre, while ignoring the unique speaking style of people which often impacts prosody. In this study, we introduce a method for converting not only the timbre, but also prosodic information (i.e., rhythm and pitch changes) to those of the target speaker. The proposed approach is based on a pretrained, self-supervised, model for encoding speech to discrete units, which make it simple, effective, and easy to optimise. We consider the many-to-many setting with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate the proposed approach is significantly superior to the evaluated baselines. Code and samples can be found under https://pages.cs.huji.ac.il/adiyoss-lab/dissc/ .
【2】 SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations
标题：分段增强：通过基于分段的扩充最大化语音翻译数据的效用
链接：https://arxiv.org/abs/2212.09699
作者：Ioannis Tsiamas,José A. R. Fonollosa,Marta R. Costa-jussà机构：Universitat Politècnica de Catalunya, Barcelona, Meta AI, Paris备注：Work in progress, 10 pages + appendix摘要：与级联方法相比，数据稀缺是端到端语音翻译方法的主要问题之一。尽管大多数语音翻译的数据资源最初都是文档级的，但它们提供了句子级的视图，可以在训练期间直接使用。但是这种句子层次的观点是单一的和静态的，潜在地限制了数据的效用。我们提出的数据扩充方法SegAugment挑战了这一想法，旨在通过提供数据集的多个可供选择的句子级视图来提高数据可用性。该方法主要依靠音频分割系统对每个文档的语音进行重新分割，然后通过对齐方法得到目标文本。音频分段系统可以用不同的长度约束进行参数化，从而使我们能够访问每个文档的多个不同的句子级视图。在Must-C中的实验表明，在8种语言对中的增益是一致的，平均增加2.2个BLEU点，在mTEDx中的低资源场景中最高增加4.7个BLEU点。此外，我们发现SegAugment也适用于纯句子级别的数据，如CoVoST，并且它使语音翻译模型能够在推理时完全消除黄金分割和自动分割之间的差距。摘要：Data scarcity is one of the main issues with the end-to-end approach for Speech Translation, as compared to the cascaded one. Although most data resources for Speech Translation are originally document-level, they offer a sentence-level view, which can be directly used during training. But this sentence-level view is single and static, potentially limiting the utility of the data. Our proposed data augmentation method SegAugment challenges this idea and aims to increase data availability by providing multiple alternative sentence-level views of a dataset. Our method heavily relies on an Audio Segmentation system to re-segment the speech of each document, after which we obtain the target text with alignment methods. The Audio Segmentation system can be parameterized with different length constraints, thus giving us access to multiple and diverse sentence-level views for each document. Experiments in MuST-C show consistent gains across 8 language pairs, with an average increase of 2.2 BLEU points, and up to 4.7 BLEU for lower-resource scenarios in mTEDx. Additionally, we find that SegAugment is also applicable to purely sentence-level data, as in CoVoST, and that it enables Speech Translation models to completely close the gap between the gold and automatic segmentation at inference time.
【3】 Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models
标题：MU$^{2}$SLAM：多任务、多语言语音和语言模型
链接：https://arxiv.org/abs/2212.09553
作者：Yong Cheng,Yu Zhang,Melvin Johnson,Wolfgang Macherey,Ankur Bapna摘要：我们介绍Mu$^{2}$SLAM，这是一种多语言序列到序列模型，针对未标记语音、未标记文本和监督数据进行了联合预训练，涵盖了100多种语言的自动语音识别（ASR）、自动语音翻译（AST）和机器翻译（MT）。通过利用语音的量化表示作为目标，Mu$^{2}$SLAM针对未标记语音和文本，利用类似于解码器上的T5的序列到序列掩蔽去噪目标和编码器上的掩蔽语言建模（MLM）目标来训练语音-文本模型，同时利用监督任务来改善模型内的跨语言和跨模态表示对齐。在CoVoST AST上，Mu$^{2}$SLAM为在公共数据集上训练的模型建立了一个新的最先进的模型，与以前的最佳模型相比，xx-en转换提高了1.9 BLEU点，en-xx转换提高了1.1 BLEU点。在Voxpopuli ASR上，我们的模型与使用RNN-T解码器微调的mSLAM模型的性能相匹配，尽管使用了相对较弱的序列到序列架构。在文本理解任务上，我们的模型比mSLAM在XNLI上的性能提高了6%以上，更接近于mT5模型在XNLI和TydiQA上的性能，为实现一个单一的模型来处理所有的语音和文本理解任务铺平了道路.摘要：We present Mu$^{2}$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AST) and Machine Translation (MT), in over 100 languages. By leveraging a quantized representation of speech as a target, Mu$^{2}$SLAM trains the speech-text models with a sequence-to-sequence masked denoising objective similar to T5 on the decoder and a masked language modeling (MLM) objective on the encoder, for both unlabeled speech and text, while utilizing the supervised tasks to improve cross-lingual and cross-modal representation alignment within the model. On CoVoST AST, Mu$^{2}$SLAM establishes a new state-of-the-art for models trained on public datasets, improving on xx-en translation over the previous best by 1.9 BLEU points and on en-xx translation by 1.1 BLEU points. On Voxpopuli ASR, our model matches the performance of an mSLAM model fine-tuned with an RNN-T decoder, despite using a relatively weaker sequence-to-sequence architecture. On text understanding tasks, our model improves by more than 6\% over mSLAM on XNLI, getting closer to the performance of mT5 models of comparable capacity on XNLI and TydiQA, paving the way towards a single model for all speech and text understanding tasks.
【4】 WACO: Word-Aligned Contrastive Learning for Speech Translation
标题：WACO：语音翻译中的词对齐对比学习
链接：https://arxiv.org/abs/2212.09359
作者：Siqi Ouyang,Rong Ye,Lei Li机构：University of California, Santa Barbara, CA, USA, ByteDance AI Lab, Shanghai, China摘要：端到端语音翻译（E2E ST）的目的是在不生成中间译文的情况下将源语音翻译成目标译文。然而，当仅有限的ST数据可用时，用于E2E ST的现有方法显著降级。我们观察到ST模型的性能与其嵌入语音和转录文本的相似度密切相关。本文提出了一种新的少镜头语音到文本翻译方法--词对齐反译学习（WACO）。我们的关键思想是通过对比学习来桥接两种模态的词级表征。我们在被广泛使用的ST基准数据集Must-C上评估WACO和其他方法。我们的实验表明，WACO比最好的基线方法性能好0.7 - 8.5个BLEU点，只有1小时的平行数据。Code is available at https://anonymous.4open.science/r/WACO .摘要：End-to-end Speech Translation (E2E ST) aims to translate source speech into target translation without generating the intermediate transcript. However, existing approaches for E2E ST degrade considerably when only limited ST data are available. We observe that an ST model's performance strongly correlates with its embedding similarity from speech and transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a novel method for few-shot speech-to-text translation. Our key idea is bridging word-level representations for both modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark. Our experiments demonstrate that WACO outperforms the best baseline methods by 0.7-8.5 BLEU points with only 1-hour parallel data. Code is available at https://anonymous.4open.science/r/WACO .
【5】 Exploring Workplace Behaviors through Speaking Patterns using Large-scale Multimodal Wearable Recordings: A Study of Healthcare Providers
标题：使用大规模多模式可穿戴录音通过说话模式探索工作场所行为：一项针对医疗保健提供者的研究
链接：https://arxiv.org/abs/2212.09090
作者：Tiantian Feng,Shrikanth Narayanan机构：University of Southern California, Department of Electrical and Computer Engineering, Los Angeles, CA, USC摘要：人际口语交际是人际交往和信息交换的核心。这种互动过程不仅涉及言语和口语，而且涉及非言语线索，如手势、面部表情和非言语发声，这些线索用于表达感情和提供反馈。这些多模式通信信号携带关于人的各种信息：性别、年龄等特征以及身体、心理状态和行为。这项工作使用可穿戴多模式传感器来调查人际沟通行为，重点是医疗保健提供者之间的说话模式，重点是护士。我们分析了从一家大型医院的99美元护士那里收集的10周的纵向数据。结果表明，不同轮班制度和不同工作单位的员工说话模式存在差异。此外，研究结果表明，结合生理测量的说话模式可以用于预测情感测量和生活满意度得分。这项工作的执行情况可在www.example.com上查阅https://github.com/usc-sail/tiles-audio-arousal。摘要：Interpersonal spoken communication is central to human interaction and the exchange of information. Such interactive processes involve not only speech and spoken language but also non-verbal cues such as hand gestures, facial expressions, and nonverbal vocalization, that are used to express feelings and provide feedback. These multimodal communication signals carry a variety of information about the people: traits like gender and age as well as about physical and psychological states and behavior. This work uses wearable multimodal sensors to investigate interpersonal communication behaviors focusing on speaking patterns among healthcare providers with a focus on nurses. We analyze longitudinal data collected from $99$ nurses in a large hospital setting over ten weeks. The results indicate that speaking pattern differences across shift schedules and working units. Moreover, results show that speaking patterns combined with physiological measures can be used to predict affect measures and life satisfaction scores. The implementation of this work can be accessed at https://github.com/usc-sail/tiles-audio-arousal.
【6】 A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness
标题：以语音为中心的可信机器学习：隐私、安全和公平
链接：https://arxiv.org/abs/2212.09006
作者：Tiantian Feng,Rajat Hebbar,Nicholas Mehlman,Xuan Shi,Aditya Kommineni,and Shrikanth Narayanan机构：Learning: Privacy, Safety, and Fairness”.,†: Contribute equally in terms of text., teaching, andor private study. Commercial use or systematic downloading, (by robots or other automatic processes) is prohibited without ex-, plicit Publisher approval.摘要：以语音为中心的机器学习系统已经彻底改变了许多领先的领域，从交通和医疗保健到教育和国防，深刻地改变了人们的生活、工作和互动方式。然而，最近的研究已经证明，许多以语音为中心的ML系统可能需要被认为是更值得信赖的，以用于更广泛的部署。具体而言，隐私泄露、区分性能和对抗攻击的脆弱性等问题都已在ML研究领域中被发现。为了应对上述挑战和风险，已经做出了大量努力，以确保这些反洗钱系统值得信赖，特别是保密、安全和公平。在这篇文章中，我们对以语音为中心的可信机器学习主题进行了第一次全面的调查，这些主题涉及隐私、安全和公平。除了作为研究团体的总结报告外，我们还指出了几个有前途的未来研究方向，以激励希望在这一领域进一步探索的研究人员。摘要：Speech-centric machine learning systems have revolutionized many leading domains ranging from transportation and healthcare to education and defense, profoundly changing how people live, work, and interact with each other. However, recent studies have demonstrated that many speech-centric ML systems may need to be considered more trustworthy for broader deployment. Specifically, concerns over privacy breaches, discriminating performance, and vulnerability to adversarial attacks have all been discovered in ML research fields. In order to address the above challenges and risks, a significant number of efforts have been made to ensure these ML systems are trustworthy, especially private, safe, and fair. In this paper, we conduct the first comprehensive survey on speech-centric trustworthy ML topics related to privacy, safety, and fairness. In addition to serving as a summary report for the research community, we point out several promising future research directions to inspire the researchers who wish to explore further in this area.
【7】 Learning from Taxonomy: Multi-label Few-Shot Classification for Everyday Sound Recognition
标题：向分类学学习：用于日常声音识别的多标签少发声分类
链接：https://arxiv.org/abs/2212.08952
作者：Jinhua Liang,Huy Phan,Emmanouil Benetos机构：⋆ Centre for Digital Music, Queen Mary University of London, UK, † The Alan Turing Institute, UK备注：submitted to ICASSP2023摘要：日常声音识别的目的是推断音频流中声音事件的类型。虽然许多工作成功地以全监督方式训练出高性能的模型，但它们仍然受限于对大量标记数据和预定义类别范围的需求。为了克服这些缺点，本文首先建立了一个新的多标记Few-Shot音频分类数据库FSD-FS。然后探讨了如何将音频分类学融入到Few-Shot学习中。具体而言，本文提出了标签依赖原型网络（LaD-prototypical networks，LaD-protonet）来挖掘标签之间的父子关系.此外，它还应用了分类感知标签平滑技术来提高模型性能。实验结果表明，LaD-protonet的性能优于原始原型网络和其他现有方法。此外，当与分类感知标签平滑相结合时，它的性能可以进一步提升。摘要：Everyday sound recognition aims to infer types of sound events in audio streams. While many works succeeded in training models with high performance in a fully-supervised manner, they are still restricted to the demand of large quantities of labelled data and the range of predefined classes. To overcome these drawbacks, this work firstly curates a new database named FSD-FS for multi-label few-shot audio classification. It then explores how to incorporate audio taxonomy in few-shot learning. Specifically, this work proposes label-dependent prototypical networks (LaD-protonet) to exploit parent-children relationships between labels. Plus, it applies taxonomy-aware label smoothing techniques to boost model performance. Experiments demonstrate that LaD-protonet outperforms original prototypical networks as well as other state-of-the-art methods. Moreover, its performance can be further boosted when combined with taxonomy-aware label smoothing.
【8】 AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
标题：AdaTranS：适应端到端语音翻译的基于边界的收缩
链接：https://arxiv.org/abs/2212.08911
作者：Xingshan Zeng,Liangyou Li,Qun Liu摘要：为了解决端到端语音翻译中数据不足的问题，语音识别和机器翻译中的数据预训练被认为是一项重要技术。然而，语音和文本之间的模态间隙阻碍了ST模型从预先训练的模型中有效地继承知识。在本文中，我们提出了AdaTranS用于端到端的语义转换，它通过一种新的收缩机制来适应语音特征，通过预测词边界来减轻语音和文本特征之间的长度失配。在MUST-C数据集上的实验结果表明，AdaTranS具有更高的推理速度和更低的内存占用，比其他基于收缩的方法具有更好的性能。进一步的实验还表明，AdaTranS可以配备额外的对准损耗，以进一步提高性能。摘要：To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.
【9】 BEATs: Audio Pre-Training with Acoustic Tokenizers
标题：Beats：使用声学令牌器进行音频预训练
链接：https://arxiv.org/abs/2212.09058
作者：Sanyuan Chen,Yu Wu,Chengyi Wang,Shujie Liu,Daniel Tompkins,Zhuo Chen,Furu Wei摘要：在过去的几年里，自监督学习（SSL）在语言、视觉、语音和音频领域都有了长足的发展。虽然离散标记预测被广泛用于其他模态，但是现有技术的音频SSL模型仍然采用重构损失用于预训练。与重构损失相比，语义丰富的离散标签预测鼓励SSL模型抽象高级音频语义并丢弃冗余细节，就像人类感知一样。然而，由于音频的连续性和语音等音素序列的不可用性，用于一般音频预训练的语义丰富的声学表征器通常不是直接获得的。为了解决这一难题，我们提出了BEAT，这是一个迭代音频预训练框架，用于从音频Transformers学习双向编码器表示，其中声学标记器和音频SSL模型通过迭代进行优化。在第一次迭代中，我们使用随机投影作为声学标记器，以掩码和标签预测的方式训练音频SSL模型。然后，我们通过从预先训练或微调的音频SSL模型中提取语义知识，为下一次迭代训练声学标记器。反复进行迭代，希望声学令牌化器和音频SSL模型能够相互促进。实验结果表明，我们的声学标记器能够生成具有丰富音频语义的离散标签，我们的音频SSL模型在各种音频分类基准测试中取得了最好的结果，甚至显著优于使用更多训练数据和模型参数的先前模型。具体而言，我们在AudioSet-2 M上为纯音频型号设置了最新的mAP 50.6%，而无需使用任何外部数据，并在ESC-50上设置了98.1%的准确度。代码和预训练模型可在www.example.com上获得https://aka.ms/beats。摘要：The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.
【1】 BEATs: Audio Pre-Training with Acoustic Tokenizers
标题：Beats：使用声学令牌器进行音频预训练
链接：https://arxiv.org/abs/2212.09058
*与cs.SD语音【9】为同一篇
作者：Sanyuan Chen,Yu Wu,Chengyi Wang,Shujie Liu,Daniel Tompkins,Zhuo Chen,Furu Wei摘要：The massive growth of self-supervised learning (SSL) has been witnessed in language, vision, speech, and audio domains over the past few years. While discrete label prediction is widely adopted for other modalities, the state-of-the-art audio SSL models still employ reconstruction loss for pre-training. Compared with reconstruction loss, semantic-rich discrete label prediction encourages the SSL model to abstract the high-level audio semantics and discard the redundant details as in human perception. However, a semantic-rich acoustic tokenizer for general audio pre-training is usually not straightforward to obtain, due to the continuous property of audio and unavailable phoneme sequences like speech. To tackle this challenge, we propose BEATs, an iterative audio pre-training framework to learn Bidirectional Encoder representation from Audio Transformers, where an acoustic tokenizer and an audio SSL model are optimized by iterations. In the first iteration, we use random projection as the acoustic tokenizer to train an audio SSL model in a mask and label prediction manner. Then, we train an acoustic tokenizer for the next iteration by distilling the semantic knowledge from the pre-trained or fine-tuned audio SSL model. The iteration is repeated with the hope of mutual promotion of the acoustic tokenizer and audio SSL model. The experimental results demonstrate our acoustic tokenizers can generate discrete labels with rich audio semantics and our audio SSL models achieve state-of-the-art results across various audio classification benchmarks, even outperforming previous models that use more training data and model parameters significantly. Specifically, we set a new state-of-the-art mAP 50.6% on AudioSet-2M for audio-only models without using any external data, and 98.1% accuracy on ESC-50. The code and pre-trained models are available at https://aka.ms/beats.
【2】 Fast FullSubNet: Accelerate Full-band and Sub-band Fusion Model for Single-channel Speech Enhancement
标题：Fast FullSubNet：单通道语音增强的加速全带和子带融合模型
链接：https://arxiv.org/abs/2212.09019
机构：Westlake University & Westlake Institute for Advanced Study, Hangzhou, China, The Chinese University of Hong Kong, Hong Kong SAR, China备注：submitted to 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：FullSubNet是我们最近提出的实时单通道语音增强网络，在深度噪声抑制（DNS）挑战数据集上实现了出色的性能。最近提出了许多FullSubNet的变体，但是它们都集中于结构设计以获得更好的性能，而很少关注计算效率。本文提出了一种新的快速全子网体系结构，致力于加速全子网的计算。具体地说，Fast FullSubNet通过使用级联的线性到mel全频带、子频带和mel到线性全频带模型来处理mel频域中的子频带语音频谱，使得极大地减少了子频带计算中涉及的频率。在此基础上，提出了对子带输入序列进行下采样操作，进一步降低时间轴上的计算复杂度。实验结果表明，与FullSubNet相比，Fast FullSubNet的计算复杂度仅为FullSubNet的13%，处理时间仅为FullSubNet的16%，性能相当甚至更优。摘要：FullSubNet is our recently proposed real-time single-channel speech enhancement network that achieves outstanding performance on the Deep Noise Suppression (DNS) Challenge dataset. A number of variants of FullSubNet have been proposed recently, but they all focus on the structure design towards better performance and are rarely concerned with computational efficiency. This work proposes a new architecture named Fast FullSubNet dedicated to accelerating the computation of FullSubNet. Specifically, Fast FullSubNet processes sub-band speech spectra in the mel-frequency domain by using cascaded linear-to-mel full-band, sub-band, and mel-to-linear full-band models such that frequencies involved in the sub-band computation are vastly reduced. After that, a down-sampling operation is proposed for the sub-band input sequence to further reduce the computational complexity along the time axis. Experimental results show that, compared to FullSubNet, Fast FullSubNet has only 13% computational complexity and 16% processing time, and achieves comparable or even better performance.
【3】 Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition
标题：端到端自动语音识别中基于信息熵的词级置信度快速估计方法
链接：https://arxiv.org/abs/2212.08703
作者：Aleksandr Laptev,Boris Ginsburg机构：NVIDIA, USA, ITMO University, St. Petersburg, Russia备注：To appear in Proc. SLT 2022, Jan 09-12, 2023, Doha, Qatar. 8 pages, 4 figures, 4 tables摘要：本文提出了一类新的用于自动语音识别的快速不可训练的基于熵的置信度估计方法。我们展示了每帧的熵值如何被归一化和聚合，以获得连接主义时间分类（CTC）和递归神经网络转换器（RNN-T）模型的每单元和每单词的置信度。所提方法与传统的基于最大每帧概率的方法计算复杂度相似，但具有更强的可调性，有效阈值范围更宽，更好地区分了正确词和错误词的置信度分布。我们在LibriSpeech测试集上对所提出的置信度进行了评估，并表明对于Conformer-CTC和Conformer-RNN-T模型，它们在检测错误单词时分别比基于最大每帧概率的置信度估计好2倍和4倍。摘要：This paper presents a class of new fast non-trainable entropy-based confidence estimation methods for automatic speech recognition. We show how per-frame entropy values can be normalized and aggregated to obtain a confidence measure per unit and per word for Connectionist Temporal Classification (CTC) and Recurrent Neural Network Transducer (RNN-T) models. Proposed methods have similar computational complexity to the traditional method based on the maximum per-frame probability, but they are more adjustable, have a wider effective threshold range, and better push apart the confidence distributions of correct and incorrect words. We evaluate the proposed confidence measures on LibriSpeech test sets, and show that they are up to 2 and 4 times better than confidence estimation based on the maximum per-frame probability at detecting incorrect words for Conformer-CTC and Conformer-RNN-T models, respectively.
【4】 Speaking Style Conversion With Discrete Self-Supervised Units
标题：用离散的自我监督单元进行讲话风格转换
链接：https://arxiv.org/abs/2212.09730
*与cs.SD语音【1】为同一篇
作者：Gallil Maimon,Yossi Adi机构：School of Computer Science and Engineering, The Hebrew University of Jerusalem摘要：语音转换（VC）是使一个说话者说出的话语听起来像是由另一个说话者说出的，同时保持其他方面如内容不变的任务。目前的VC方法主要关注音色等频谱特征，而忽略了人们独特的说话风格对韵律的影响。在本研究中，我们介绍了一种不仅将音色，而且将韵律信息（即，节奏和音调变化）到目标说话人的那些。所提出的方法基于一个预先训练的、自监督的、用于将语音编码为离散单元的模型，这使得它简单、有效并且易于优化。我们考虑没有配对数据的多对多设置。我们为该设置引入了一套定量和定性的评估度量，并通过实证证明了所提出的方法显著优于所评估的基线。代码和示例可在www.example.com下找到https://pages.cs.huji.ac.il/adiyoss-lab/dissc/。摘要：Voice Conversion (VC) is the task of making a spoken utterance by one speaker sound as if uttered by a different speaker, while keeping other aspects like content unchanged. Current VC methods, focus primarily on spectral features like timbre, while ignoring the unique speaking style of people which often impacts prosody. In this study, we introduce a method for converting not only the timbre, but also prosodic information (i.e., rhythm and pitch changes) to those of the target speaker. The proposed approach is based on a pretrained, self-supervised, model for encoding speech to discrete units, which make it simple, effective, and easy to optimise. We consider the many-to-many setting with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate the proposed approach is significantly superior to the evaluated baselines. Code and samples can be found under https://pages.cs.huji.ac.il/adiyoss-lab/dissc/ .
【5】 SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations
标题：分段增强：通过基于分段的扩充最大化语音翻译数据的效用
链接：https://arxiv.org/abs/2212.09699
*与cs.SD语音【2】为同一篇
作者：Ioannis Tsiamas,José A. R. Fonollosa,Marta R. Costa-jussà机构：Universitat Politècnica de Catalunya, Barcelona, Meta AI, Paris备注：Work in progress, 10 pages + appendix摘要：与级联方法相比，数据稀缺是端到端语音翻译方法的主要问题之一。尽管大多数语音翻译的数据资源最初都是文档级的，但它们提供了句子级的视图，可以在训练期间直接使用。但是这种句子层次的观点是单一的和静态的，潜在地限制了数据的效用。我们提出的数据扩充方法SegAugment挑战了这一想法，旨在通过提供数据集的多个可供选择的句子级视图来提高数据可用性。该方法主要依靠音频分割系统对每个文档的语音进行重新分割，然后通过对齐方法得到目标文本。音频分段系统可以用不同的长度约束进行参数化，从而使我们能够访问每个文档的多个不同的句子级视图。在Must-C中的实验表明，在8种语言对中的增益是一致的，平均增加2.2个BLEU点，在mTEDx中的低资源场景中最高增加4.7个BLEU点。此外，我们发现SegAugment也适用于纯句子级别的数据，如CoVoST，并且它使语音翻译模型能够在推理时完全消除黄金分割和自动分割之间的差距。摘要：Data scarcity is one of the main issues with the end-to-end approach for Speech Translation, as compared to the cascaded one. Although most data resources for Speech Translation are originally document-level, they offer a sentence-level view, which can be directly used during training. But this sentence-level view is single and static, potentially limiting the utility of the data. Our proposed data augmentation method SegAugment challenges this idea and aims to increase data availability by providing multiple alternative sentence-level views of a dataset. Our method heavily relies on an Audio Segmentation system to re-segment the speech of each document, after which we obtain the target text with alignment methods. The Audio Segmentation system can be parameterized with different length constraints, thus giving us access to multiple and diverse sentence-level views for each document. Experiments in MuST-C show consistent gains across 8 language pairs, with an average increase of 2.2 BLEU points, and up to 4.7 BLEU for lower-resource scenarios in mTEDx. Additionally, we find that SegAugment is also applicable to purely sentence-level data, as in CoVoST, and that it enables Speech Translation models to completely close the gap between the gold and automatic segmentation at inference time.
【6】 Mu$^{2}$SLAM: Multitask, Multilingual Speech and Language Models
标题：MU$^{2}$SLAM：多任务、多语言语音和语言模型
链接：https://arxiv.org/abs/2212.09553
*与cs.SD语音【3】为同一篇
作者：Yong Cheng,Yu Zhang,Melvin Johnson,Wolfgang Macherey,Ankur Bapna摘要：We present Mu$^{2}$SLAM, a multilingual sequence-to-sequence model pre-trained jointly on unlabeled speech, unlabeled text and supervised data spanning Automatic Speech Recognition (ASR), Automatic Speech Translation (AST) and Machine Translation (MT), in over 100 languages. By leveraging a quantized representation of speech as a target, Mu$^{2}$SLAM trains the speech-text models with a sequence-to-sequence masked denoising objective similar to T5 on the decoder and a masked language modeling (MLM) objective on the encoder, for both unlabeled speech and text, while utilizing the supervised tasks to improve cross-lingual and cross-modal representation alignment within the model. On CoVoST AST, Mu$^{2}$SLAM establishes a new state-of-the-art for models trained on public datasets, improving on xx-en translation over the previous best by 1.9 BLEU points and on en-xx translation by 1.1 BLEU points. On Voxpopuli ASR, our model matches the performance of an mSLAM model fine-tuned with an RNN-T decoder, despite using a relatively weaker sequence-to-sequence architecture. On text understanding tasks, our model improves by more than 6\% over mSLAM on XNLI, getting closer to the performance of mT5 models of comparable capacity on XNLI and TydiQA, paving the way towards a single model for all speech and text understanding tasks.
【7】 WACO: Word-Aligned Contrastive Learning for Speech Translation
标题：WACO：语音翻译中的词对齐对比学习
链接：https://arxiv.org/abs/2212.09359
*与cs.SD语音【4】为同一篇
作者：Siqi Ouyang,Rong Ye,Lei Li机构：University of California, Santa Barbara, CA, USA, ByteDance AI Lab, Shanghai, China摘要：端到端语音翻译（E2E ST）的目的是在不生成中间译文的情况下将源语音翻译成目标译文。然而，当仅有限的ST数据可用时，用于E2E ST的现有方法显著降级。我们观察到ST模型的性能与其嵌入语音和转录文本的相似度密切相关。本文提出了一种新的少镜头语音到文本翻译方法--词对齐反译学习（WACO）。我们的关键思想是通过对比学习来桥接两种模态的词级表征。我们在被广泛使用的ST基准数据集Must-C上评估WACO和其他方法。我们的实验表明，WACO比最好的基线方法性能好0.7 - 8.5个BLEU点，只有1小时的平行数据。Code is available at https://anonymous.4open.science/r/WACO .摘要：End-to-end Speech Translation (E2E ST) aims to translate source speech into target translation without generating the intermediate transcript. However, existing approaches for E2E ST degrade considerably when only limited ST data are available. We observe that an ST model's performance strongly correlates with its embedding similarity from speech and transcript. In this paper, we propose Word-Aligned COntrastive learning (WACO), a novel method for few-shot speech-to-text translation. Our key idea is bridging word-level representations for both modalities via contrastive learning. We evaluate WACO and other methods on the MuST-C dataset, a widely used ST benchmark. Our experiments demonstrate that WACO outperforms the best baseline methods by 0.7-8.5 BLEU points with only 1-hour parallel data. Code is available at https://anonymous.4open.science/r/WACO .
【8】 Exploring Workplace Behaviors through Speaking Patterns using Large-scale Multimodal Wearable Recordings: A Study of Healthcare Providers
标题：使用大规模多模式可穿戴录音通过说话模式探索工作场所行为：一项针对医疗保健提供者的研究
链接：https://arxiv.org/abs/2212.09090
*与cs.SD语音【5】为同一篇
作者：Tiantian Feng,Shrikanth Narayanan机构：University of Southern California, Department of Electrical and Computer Engineering, Los Angeles, CA, USC摘要：人际口语交际是人际交往和信息交换的核心。这种互动过程不仅涉及言语和口语，而且涉及非言语线索，如手势、面部表情和非言语发声，这些线索用于表达感情和提供反馈。这些多模式通信信号携带关于人的各种信息：性别、年龄等特征以及身体、心理状态和行为。这项工作使用可穿戴多模式传感器来调查人际沟通行为，重点是医疗保健提供者之间的说话模式，重点是护士。我们分析了从一家大型医院的99美元护士那里收集的10周的纵向数据。结果表明，不同轮班制度和不同工作单位的员工说话模式存在差异。此外，研究结果表明，结合生理测量的说话模式可以用于预测情感测量和生活满意度得分。这项工作的执行情况可在www.example.com上查阅https://github.com/usc-sail/tiles-audio-arousal。摘要：Interpersonal spoken communication is central to human interaction and the exchange of information. Such interactive processes involve not only speech and spoken language but also non-verbal cues such as hand gestures, facial expressions, and nonverbal vocalization, that are used to express feelings and provide feedback. These multimodal communication signals carry a variety of information about the people: traits like gender and age as well as about physical and psychological states and behavior. This work uses wearable multimodal sensors to investigate interpersonal communication behaviors focusing on speaking patterns among healthcare providers with a focus on nurses. We analyze longitudinal data collected from $99$ nurses in a large hospital setting over ten weeks. The results indicate that speaking pattern differences across shift schedules and working units. Moreover, results show that speaking patterns combined with physiological measures can be used to predict affect measures and life satisfaction scores. The implementation of this work can be accessed at https://github.com/usc-sail/tiles-audio-arousal.
【9】 A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness
标题：以语音为中心的可信机器学习：隐私、安全和公平
链接：https://arxiv.org/abs/2212.09006
*与cs.SD语音【6】为同一篇
作者：Tiantian Feng,Rajat Hebbar,Nicholas Mehlman,Xuan Shi,Aditya Kommineni,and Shrikanth Narayanan机构：Learning: Privacy, Safety, and Fairness”.,†: Contribute equally in terms of text., teaching, andor private study. Commercial use or systematic downloading, (by robots or other automatic processes) is prohibited without ex-, plicit Publisher approval.摘要：以语音为中心的机器学习系统已经彻底改变了许多领先的领域，从交通和医疗保健到教育和国防，深刻地改变了人们的生活、工作和互动方式。然而，最近的研究已经证明，许多以语音为中心的ML系统可能需要被认为是更值得信赖的，以用于更广泛的部署。具体而言，隐私泄露、区分性能和对抗攻击的脆弱性等问题都已在ML研究领域中被发现。为了应对上述挑战和风险，已经做出了大量努力，以确保这些反洗钱系统值得信赖，特别是保密、安全和公平。在这篇文章中，我们对以语音为中心的可信机器学习主题进行了第一次全面的调查，这些主题涉及隐私、安全和公平。除了作为研究团体的总结报告外，我们还指出了几个有前途的未来研究方向，以激励希望在这一领域进一步探索的研究人员。摘要：Speech-centric machine learning systems have revolutionized many leading domains ranging from transportation and healthcare to education and defense, profoundly changing how people live, work, and interact with each other. However, recent studies have demonstrated that many speech-centric ML systems may need to be considered more trustworthy for broader deployment. Specifically, concerns over privacy breaches, discriminating performance, and vulnerability to adversarial attacks have all been discovered in ML research fields. In order to address the above challenges and risks, a significant number of efforts have been made to ensure these ML systems are trustworthy, especially private, safe, and fair. In this paper, we conduct the first comprehensive survey on speech-centric trustworthy ML topics related to privacy, safety, and fairness. In addition to serving as a summary report for the research community, we point out several promising future research directions to inspire the researchers who wish to explore further in this area.
【10】 Learning from Taxonomy: Multi-label Few-Shot Classification for Everyday Sound Recognition
标题：向分类学学习：用于日常声音识别的多标签少发声分类
链接：https://arxiv.org/abs/2212.08952
*与cs.SD语音【7】为同一篇
作者：Jinhua Liang,Huy Phan,Emmanouil Benetos机构：⋆ Centre for Digital Music, Queen Mary University of London, UK, † The Alan Turing Institute, UK备注：submitted to ICASSP2023摘要：日常声音识别的目的是推断音频流中声音事件的类型。虽然许多工作成功地以全监督方式训练出高性能的模型，但它们仍然受限于对大量标记数据和预定义类别范围的需求。为了克服这些缺点，本文首先建立了一个新的多标记Few-Shot音频分类数据库FSD-FS。然后探讨了如何将音频分类学融入到Few-Shot学习中。具体而言，本文提出了标签依赖原型网络（LaD-prototypical networks，LaD-protonet）来挖掘标签之间的父子关系.此外，它还应用了分类感知标签平滑技术来提高模型性能。实验结果表明，LaD-protonet的性能优于原始原型网络和其他现有方法。此外，当与分类感知标签平滑相结合时，它的性能可以进一步提升。摘要：Everyday sound recognition aims to infer types of sound events in audio streams. While many works succeeded in training models with high performance in a fully-supervised manner, they are still restricted to the demand of large quantities of labelled data and the range of predefined classes. To overcome these drawbacks, this work firstly curates a new database named FSD-FS for multi-label few-shot audio classification. It then explores how to incorporate audio taxonomy in few-shot learning. Specifically, this work proposes label-dependent prototypical networks (LaD-protonet) to exploit parent-children relationships between labels. Plus, it applies taxonomy-aware label smoothing techniques to boost model performance. Experiments demonstrate that LaD-protonet outperforms original prototypical networks as well as other state-of-the-art methods. Moreover, its performance can be further boosted when combined with taxonomy-aware label smoothing.
【11】 AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation
标题：AdaTranS：适应端到端语音翻译的基于边界的收缩
链接：https://arxiv.org/abs/2212.08911
*与cs.SD语音【8】为同一篇
作者：Xingshan Zeng,Liangyou Li,Qun Liu摘要：为了解决端到端语音翻译中数据不足的问题，语音识别和机器翻译中的数据预训练被认为是一项重要技术。然而，语音和文本之间的模态间隙阻碍了ST模型从预先训练的模型中有效地继承知识。在本文中，我们提出了AdaTranS用于端到端的语义转换，它通过一种新的收缩机制来适应语音特征，通过预测词边界来减轻语音和文本特征之间的长度失配。在MUST-C数据集上的实验结果表明，AdaTranS具有更高的推理速度和更低的内存占用，比其他基于收缩的方法具有更好的性能。进一步的实验还表明，AdaTranS可以配备额外的对准损耗，以进一步提高性能。摘要：To alleviate the data scarcity problem in End-to-end speech translation (ST), pre-training on data for speech recognition and machine translation is considered as an important technique. However, the modality gap between speech and text prevents the ST model from efficiently inheriting knowledge from the pre-trained models. In this work, we propose AdaTranS for end-to-end ST. It adapts the speech features with a new shrinking mechanism to mitigate the length mismatch between speech and text features by predicting word boundaries. Experiments on the MUST-C dataset demonstrate that AdaTranS achieves better performance than the other shrinking-based methods, with higher inference speed and lower memory usage. Further experiments also show that AdaTranS can be equipped with additional alignment losses to further improve performance.
机器翻译，仅供参考
每日语音速递