【1】 Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities
标题：70种语言的大规模多语言ASR：标记化、体系结构和泛化功能
链接：https://arxiv.org/abs/2211.05756
作者：Andros Tjandra,Nayan Singhal,David Zhang,Ozlem Kalinli,Abdelrahman Mohamed,Duc Le,Michael L. Seltzer备注：Submitted to ICASSP 2023摘要：端到端多语言ASR已变得越来越有吸引力，这是因为它简化了培训和部署流程，并从高资源语言向低资源语言实现了良好的性能转换。但是，扩展语言数量、总小时数和唯一令牌数量并不是一项简单的任务。本文研究了70种语言的大规模多语言ASR模型。我们考察两种架构：（1）共享嵌入和输出模型;（2）多嵌入和输出模型。在共享模型的实验中，我们展示了标记化策略在不同语言中的重要性。之后，我们使用我们最佳的记号化策略来训练多重嵌入与输出模型，以进一步改善我们的结果。与单语模型相比，我们的多语言ASR实现了13.9%-15.6%的平均WER相对改善。实验结果表明，该算法在未知数据集和未知领域上具有良好的泛化能力，在zero-shot和微调的多语种库语音（MLS）上分别获得了9.5%和7.5%的WER.摘要：End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high-resource to low-resource languages. However, scaling up the number of languages, total hours, and number of unique tokens is not a trivial task. This paper explores large-scale multilingual ASR models on 70 languages. We inspect two architectures: (1) Shared embedding and output and (2) Multiple embedding and output model. In the shared model experiments, we show the importance of tokenization strategy across different languages. Later, we use our optimal tokenization strategy to train multiple embedding and output model to further improve our result. Our multilingual ASR achieves 13.9%-15.6% average WER relative improvement compared to monolingual models. We show that our multilingual ASR generalizes well on an unseen dataset and domain, achieving 9.5% and 7.5% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively.
【2】 Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation
标题：Vis2Mus：探索可控音乐生成的多通道表示映射
链接：https://arxiv.org/abs/2211.05543
作者：Runbang Zhang,Yixiao Zhang,Kai Shao,Ying Shan,Gus Xia机构：Music X lab, NYU Shanghai, Centre for Digital Music, Queen Mary University of London, Tencent Inc. , MBZUAI备注：Submitted to ICASSP 2023. GitHub repo: this https URL摘要：本研究探讨视觉艺术领域到音乐领域的表征映射，以视觉艺术作为控制音乐生成的有效抓手。与大多数纯数据驱动的多模态表征学习研究不同，我们采用了一种综合分析的方法，将深度音乐表征学习与用户研究相结合。这种方法使我们能够发现\textit{interpretable}表示映射，而无需大量的成对数据。特别地，我们发现视觉到音乐的映射具有类似于等变的良好性质。换句话说，我们可以使用各种图像变换，例如，改变亮度、改变对比度、风格转移，来控制音乐领域中的相应变换。此外，我们还发布了Vis 2 Mus系统，作为符号音乐生成的可控界面。摘要：In this study, we explore the representation mapping from the domain of visual arts to the domain of music, with which we can use visual arts as an effective handle to control music generation. Unlike most studies in multimodal representation learning that are purely data-driven, we adopt an analysis-by-synthesis approach that combines deep music representation learning with user studies. Such an approach enables us to discover \textit{interpretable} representation mapping without a huge amount of paired data. In particular, we discover that visual-to-music mapping has a nice property similar to equivariant. In other words, we can use various image transformations, say, changing brightness, changing contrast, style transfer, to control the corresponding transformations in the music domain. In addition, we released the Vis2Mus system as a controllable interface for symbolic music generation.
【3】 Privacy-Utility Balanced Voice De-Identification Using Adversarial Examples
标题：隐私-效用平衡语音识别--使用对抗性实例
链接：https://arxiv.org/abs/2211.05446
作者：Meng Chen,Li Lu,Jiadi Yu,Yingying Chen,Zhongjie Ba,Feng Lin,Kui Ren机构：∗Zhejiang University †Shanghai Jiao Tong University ‡Rutgers University摘要：面对语音数据发布过程中身份泄露的威胁，用户在享受便捷的语音服务时陷入了隐私与效用的两难境地。现有的研究采用直接修改或基于文本的再合成来去识别用户的声音，但在人类参与者在场的情况下导致不一致的可听性。提出了一种语音去身份识别系统，该系统使用对抗性实例来平衡语音服务的隐私性和实用性。我们设计了一个新颖的卷积对抗性例子，将扰动调制到真实世界的房间脉冲响应中，而不是引入可感知失真的典型加性例子。该系统通过自动说话人识别（ASI）来保护用户身份，同时保持语音感知质量，实现非侵入式去识别。此外，我们的系统通过一个条件变分自动编码器学习一个紧凑的说话人分布，以根据需要采样不同的目标嵌入。结合不同的目标生成和输入特定的扰动构造，我们的系统能够实现任意到任意的身份转换，以实现自适应去身份化。实验结果表明，该系统在主流ASI和商用系统上的去身份识别成功率分别为98%和79%，客观Mel倒谱失真为4.31dB，主观平均评价得分为4.48。摘要：Faced with the threat of identity leakage during voice data publishing, users are engaged in a privacy-utility dilemma when enjoying convenient voice services. Existing studies employ direct modification or text-based re-synthesis to de-identify users' voices, but resulting in inconsistent audibility in the presence of human participants. In this paper, we propose a voice de-identification system, which uses adversarial examples to balance the privacy and utility of voice services. Instead of typical additive examples inducing perceivable distortions, we design a novel convolutional adversarial example that modulates perturbations into real-world room impulse responses. Benefit from this, our system could preserve user identity from exposure by Automatic Speaker Identification (ASI) while remaining the voice perceptual quality for non-intrusive de-identification. Moreover, our system learns a compact speaker distribution through a conditional variational auto-encoder to sample diverse target embeddings on demand. Combining diverse target generation and input-specific perturbation construction, our system enables any-to-any identify transformation for adaptive de-identification. Experimental results show that our system could achieve 98% and 79% successful de-identification on mainstream ASIs and commercial systems with an objective Mel cepstral distortion of 4.31dB and a subjective mean opinion score of 4.48.
【4】 Speech Enhancement with Fullband-Subband Cross-Attention Network
标题：基于全带-子带交叉注意网络的语音增强
链接：https://arxiv.org/abs/2211.05432
作者：Jun Chen,Wei Rao,Zilin Wang,Zhiyong Wu,Yannan Wang,Tao Yu,Shidong Shang,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Tencent Ethereal Audio Lab, Shenzhen, China, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by InterSpeech 2022. arXiv admin note: text overlap with arXiv:2203.12188摘要：FullSubNet在语音增强中充分利用了全带和子带信息，表现出了良好的性能。然而，在FullSubNet中，全带和子带之间的关系是通过简单地级联全带模型和子带单元的输出来实现的。该算法仅对子带单元补充少量的全局信息，没有考虑全带与子带之间的相互影响。提出了一种全频带-子频带交叉注意（FSCA）模型，用于交互式融合全局和局部信息，并将其应用于FullSubNet。这个新的框架被称为FS-CANet。此外，与FullSubNet不同，FS-CANet通过时间卷积网络（TCN）块对全带提取器进行优化，进一步减小了模型规模。在DNS Challenge - Interspeech 2021数据集上的实验结果表明，FS-CANet的语音增强性能优于其他现有的语音增强方法，并验证了全频段-子频段交叉注意的有效性.摘要：FullSubNet has shown its promising performance on speech enhancement by utilizing both fullband and subband information. However, the relationship between fullband and subband in FullSubNet is achieved by simply concatenating the output of fullband model and subband units. It only supplements the subband units with a small quantity of global information and has not considered the interaction between fullband and subband. This paper proposes a fullband-subband cross-attention (FSCA) module to interactively fuse the global and local information and applies it to FullSubNet. This new framework is called as FS-CANet. Moreover, different from FullSubNet, the proposed FS-CANet optimize the fullband extractor by temporal convolutional network (TCN) blocks to further reduce the model size. Experimental results on DNS Challenge - Interspeech 2021 dataset show that the proposed FS-CANet outperforms other state-of-the-art speech enhancement approaches, and demonstrate the effectiveness of fullband-subband cross-attention.
【5】 GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioning
标题：GanStrument：具有基音不变实例条件的对抗性乐器声音合成
链接：https://arxiv.org/abs/2211.05385
作者：Gaku Narita,Junichi Shimizu,Taketo Akama机构：Sony Computer Science Laboratories, Tokyo, Japan备注：5 pages, 4 figures, Audio examples: this https URL摘要：本文提出了一种生成式对抗乐器声音合成模型GANStrument。给定一个单拍声音作为输入，它能够在交互时间内生成反映输入音色的音高乐器声音。GANStrument通过实例条件化，提高了合成语音的保真度和多样性，并对不同的输入具有较好的泛化能力。此外，我们还引入了一种用于基音不变特征提取器的对抗训练方案，该方案显著地提高了基音精度和音色一致性。实验结果表明，GANStrument在生成质量和输入可编辑性方面优于未使用实例条件的强基线。定性示例可在线获取。摘要：We propose GANStrument, a generative adversarial model for instrument sound synthesis. Given a one-shot sound as input, it is able to generate pitched instrument sounds that reflect the timbre of the input within an interactive time. By exploiting instance conditioning, GANStrument achieves better fidelity and diversity of synthesized sounds and generalization ability to various inputs. In addition, we introduce an adversarial training scheme for a pitch-invariant feature extractor that significantly improves the pitch accuracy and timbre consistency. Experimental results show that GANStrument outperforms strong baselines that do not use instance conditioning in terms of generation quality and input editability. Qualitative examples are available online.
【6】 EMOFAKE: An Initial Dataset For Emotion Fake Audio Detection
标题：EMOFAKE：一种情感伪音频检测的初始数据集
链接：https://arxiv.org/abs/2211.05363
作者：Yan Zhao,Jiangyan Yi,Jianhua Tao,Chenglong Wang,Chu Yuan Zhang,Tao Wang,Yongfeng Dong机构：School of Artifcial Intelligence, Hebei University of Technology, China, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China摘要：已经有一些数据集用于伪音频检测，例如ASVspoof和ADD数据集。然而，这些数据库没有考虑音频的情感已经从一个改变到另一个，而其它信息（例如，说话者身份和内容）保持相同的情况。情感的变化往往会导致语义的变化。这可能对社会稳定构成极大威胁。因此，本文报告了我们在开发这样一个涉及原始音频的变化情感状态的情感假音频检测数据集方面的进展。数据集命名为EmoFake。EmoFake中的假音频是使用最先进的情感语音转换模型生成的。在此数据集上进行了一些基准实验。结果表明，该数据集对ASVspoof 2021的LCNN和RawNet2基线模型提出了挑战.摘要：There are already some datasets used for fake audio detection, such as the ASVspoof and ADD datasets. However, these databases do not consider a situation that the emotion of the audio has been changed from one to another, while other information (e.g. speaker identity and content) remains the same. Changing emotions often leads to semantic changes. This may be a great threat to social stability. Therefore, this paper reports our progress in developing such an emotion fake audio detection dataset involving changing emotion state of the original audio. The dataset is named EmoFake. The fake audio in EmoFake is generated using the state-of-the-art emotion voice conversion models. Some benchmark experiments are conducted on this dataset. The results show that our designed dataset poses a challenge to the LCNN and RawNet2 baseline models of ASVspoof 2021.
【7】 Multimodal Dyadic Impression Recognition via Listener Adaptive Cross-Domain Fusion
标题：基于听者自适应跨域融合的多模式二元印象识别
链接：https://arxiv.org/abs/2211.05163
作者：Yuanchao Li,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, UK备注：submitted to ICASSP2023. arXiv admin note: substantial text overlap with arXiv:2203.13932摘要：作为情感计算的一个分支，印象识别，对说话者特征（例如热情或能力）的感知潜在地是人与人的对话和口语对话系统的关键部分。大多数研究只从说话人的行为或听话人的反应来研究印象，而忽略了它们之间的潜在联系。本文提出了一种听者自适应跨域结构，该结构由听者自适应函数和跨域融合函数组成，听者自适应函数用于建模说话人和听者行为之间的因果关系，跨域融合函数用于加强说话人和听者行为之间的联系。在二元IMPRESSION数据集上的实验验证了该方法的有效性，在能力和热情维度上的一致性相关系数分别为78.8%和77.5%，优于以往的研究。所提出的方法有望推广到类似的二元交互场景。摘要：As a sub-branch of affective computing, impression recognition, e.g., perception of speaker characteristics such as warmth or competence, is potentially a critical part of both human-human conversations and spoken dialogue systems. Most research has studied impressions only from the behaviors expressed by the speaker or the response from the listener, yet ignored their latent connection. In this paper, we perform impression recognition using a proposed listener adaptive cross-domain architecture, which consists of a listener adaptation function to model the causality between speaker and listener behaviors and a cross-domain fusion function to strengthen their connection. The experimental evaluation on the dyadic IMPRESSION dataset verified the efficacy of our method, producing concordance correlation coefficients of 78.8% and 77.5% in the competence and warmth dimensions, outperforming previous studies. The proposed method is expected to be generalized to similar dyadic interaction scenarios.
【8】 Self-supervised learning with bi-label masked speech prediction for streaming multi-talker speech recognition
标题：用于流式多说话人语音识别的双标签掩蔽语音预测自监督学习
链接：https://arxiv.org/abs/2211.05564
作者：Zili Huang,Zhuo Chen,Naoyuki Kanda,Jian Wu,Yiming Wang,Jinyu Li,Takuya Yoshioka,Xiaofei Wang,Peidong Wang机构：Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA, Microsoft, One Microsoft Way, Redmond, WA, USA备注：submitted to ICASSP 2023摘要：自监督学习（Self-supervised learning，SSL）利用输入数据本身进行表征学习，已经在各种下游语音任务中取得了最好的结果。然而，以往的研究大多集中于离线单说话者应用，而对多说话者应用的研究较少，尤其是流媒体应用.本文研究了用于流式多说话者语音识别的SSL，它以流式方式生成重叠说话者的转录。我们首先观察到，由于重叠语音的较差表示，常规SSL技术在此任务上不能很好地工作。然后，我们提出了一种新的SSL训练目标，称为双标记掩蔽语音预测，它显式地保留了重叠语音中所有说话者的表示。我们研究了所提出的系统的各个方面，包括数据配置和量化器选择。所提出的SSL设置在LibriSpeechMix数据集上实现了显著更好的单词错误率。摘要：Self-supervised learning (SSL), which utilizes the input data itself for representation learning, has achieved state-of-the-art results for various downstream speech tasks. However, most of the previous studies focused on offline single-talker applications, with limited investigations in multi-talker cases, especially for streaming scenarios. In this paper, we investigate SSL for streaming multi-talker speech recognition, which generates transcriptions of overlapping speakers in a streaming fashion. We first observe that conventional SSL techniques do not work well on this task due to the poor representation of overlapping speech. We then propose a novel SSL training objective, referred to as bi-label masked speech prediction, which explicitly preserves representations of all speakers in overlapping speech. We investigate various aspects of the proposed system including data configuration and quantizer selection. The proposed SSL setup achieves substantially better word error rates on the LibriSpeechMix dataset.
【9】 Self-supervised learning of audio representations using angular contrastive loss
标题：使用角度对比损失的音频表示的自监督学习
链接：https://arxiv.org/abs/2211.05442
作者：Shanshan Wang,Soumya Tripathy,Annamaria Mesaros机构：Computing Sciences, Tampere University, Tampere, Finland摘要：在自监督学习（Self-Supervised Learning，SSL）中，设计了多种借口任务，通过对比损失学习特征表示。然而，已有的研究表明，由于实例判别目标的固有缺陷，这种损失对语义相似的样本容忍度较低，可能会影响下游任务中学习到的特征嵌入的质量。为了提高SSL中特征嵌入的区分能力，提出了一种新的损失函数--角度对比损失（ACL），它是角度裕度和对比损失的线性组合。ACL通过在SSL中的正和负增广对之间显式地添加角边缘来改进对比学习。实验结果表明，在有监督学习和无监督学习中使用ACL都能显著提高学习性能。我们使用FSDnoisy 18 k数据集验证了我们的新损失函数，其中我们使用监督和自监督学习在声音事件分类中分别实现了73.6%和77.1%的准确率。摘要：In Self-Supervised Learning (SSL), various pretext tasks are designed for learning feature representations through contrastive loss. However, previous studies have shown that this loss is less tolerant to semantically similar samples due to the inherent defect of instance discrimination objectives, which may harm the quality of learned feature embeddings used in downstream tasks. To improve the discriminative ability of feature embeddings in SSL, we propose a new loss function called Angular Contrastive Loss (ACL), a linear combination of angular margin and contrastive loss. ACL improves contrastive learning by explicitly adding an angular margin between positive and negative augmented pairs in SSL. Experimental results show that using ACL for both supervised and unsupervised learning significantly improves performance. We validated our new loss function using the FSDnoisy18k dataset, where we achieved 73.6% and 77.1% accuracy in sound event classification using supervised and self-supervised learning, respectively.
【10】 Speech separation with large-scale self-supervised learning
标题：大规模自监督学习的语音分离
链接：https://arxiv.org/abs/2211.05172
作者：Zhuo Chen,Naoyuki Kanda,Jian Wu,Yu Wu,Xiaofei Wang,Takuya Yoshioka,Jinyu Li,Sunit Sivasankaran,Sefik Emre Eskimez机构：Microsoft, One Microsoft Way, Redmond, USA摘要：自监督学习（SSL）方法如WavLM在小规模仿真实验中显示出良好的语音分离（SS）效果。在这项工作中，我们通过大规模扩展预训练数据（超过30万小时）和微调数据（1万小时），扩展了对基于SSL的SS的探索。我们还研究了在有限的计算预算下有效地将预训练模型与SS网络集成的各种技术，包括低帧速率SSL模型训练设置和仅使用预训练模型的一部分的微调方案。在一个模拟的远场语音混合测试集上，与有监督的基线模型和基于WavLM的SS模型相比，我们的模型分别获得了15.9%和11.2%的相对词错误率（WER）降低.对于使用连续语音分离的真实会议记录上的会话转录，所提出的模型在AMI和ICSI评估集上分别实现了相对于纯监督基线的6.8%和10.6%的相对WER减少，同时减少了38%的计算成本。摘要：Self-supervised learning (SSL) methods such as WavLM have shown promising speech separation (SS) results in small-scale simulation-based experiments. In this work, we extend the exploration of the SSL-based SS by massively scaling up both the pre-training data (more than 300K hours) and fine-tuning data (10K hours). We also investigate various techniques to efficiently integrate the pre-trained model with the SS network under a limited computation budget, including a low frame rate SSL model training setup and a fine-tuning scheme using only the part of the pre-trained model. Compared with a supervised baseline and the WavLM-based SS model using feature embeddings obtained with the previously released 94K hours trained WavLM, our proposed model obtains 15.9% and 11.2% of relative word error rate (WER) reductions, respectively, for a simulated far-field speech mixture test set. For conversation transcription on real meeting recordings using continuous speech separation, the proposed model achieves 6.8% and 10.6% of relative WER reductions over the purely supervised baseline on AMI and ICSI evaluation sets, respectively, while reducing the computational cost by 38%.
【11】 Fusing ASR Outputs in Joint Training for Speech Emotion Recognition
标题：语音情感识别联合训练中ASR输出的融合
链接：https://arxiv.org/abs/2110.15684
作者：Yuanchao Li,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, Scotland, UK备注：Accepted for ICASSP 2022摘要：除了声学信息外，基于语音记录的语言特征在语音情感识别中也被证明是有用的。然而，由于情感标注数据的稀缺性和情感语音识别的困难性，使得该领域的研究难以获得可靠的语言学特征和模型。本文提出了一种融合自动语音识别的方法（ASR）输出进入联合训练SER的流水线，ASR和SER之间的关系还没有得到充分的研究，ASR的哪些特征以及如何使SER受益还不清楚.通过考察不同的ASR输出和融合方法，我们的实验表明，在联合训练ASR和SER时，使用分级共同注意力融合方法合并ASR隐藏和文本输出两者最大程度地改善了SER性能。在IEMOCAP语料库上，我们的方法获得了63.4%的加权准确率，这接近于通过结合地面实况转录本获得的基线结果。此外，本文还提出了IEMOCAP的误词率分析和Wav2vec 2.0模型的层差分析，以更好地理解ASR和SER之间的关系。摘要：Alongside acoustic information, linguistic features based on speech transcripts have been proven useful in Speech Emotion Recognition (SER). However, due to the scarcity of emotion labelled data and the difficulty of recognizing emotional speech, it is hard to obtain reliable linguistic features and models in this research area. In this paper, we propose to fuse Automatic Speech Recognition (ASR) outputs into the pipeline for joint training SER. The relationship between ASR and SER is understudied, and it is unclear what and how ASR features benefit SER. By examining various ASR outputs and fusion methods, our experiments show that in joint ASR-SER training, incorporating both ASR hidden and text output using a hierarchical co-attention fusion approach improves the SER performance the most. On the IEMOCAP corpus, our approach achieves 63.4% weighted accuracy, which is close to the baseline results achieved by combining ground-truth transcripts. In addition, we also present novel word error rate analysis on IEMOCAP and layer-difference analysis of the Wav2vec 2.0 model to better understand the relationship between ASR and SER.
【1】 Self-supervised learning with bi-label masked speech prediction for streaming multi-talker speech recognition
标题：用于流式多说话人语音识别的双标签掩蔽语音预测自监督学习
链接：https://arxiv.org/abs/2211.05564
* 与cs.SD语音【8】为同一篇
作者：Zili Huang,Zhuo Chen,Naoyuki Kanda,Jian Wu,Yiming Wang,Jinyu Li,Takuya Yoshioka,Xiaofei Wang,Peidong Wang机构：Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA, Microsoft, One Microsoft Way, Redmond, WA, USA备注：submitted to ICASSP 2023摘要：自监督学习（Self-supervised learning，SSL）利用输入数据本身进行表征学习，已经在各种下游语音任务中取得了最好的结果。然而，以往的研究大多集中于离线单说话者应用，而对多说话者应用的研究较少，尤其是流媒体应用.本文研究了用于流式多说话者语音识别的SSL，它以流式方式生成重叠说话者的转录。我们首先观察到，由于重叠语音的较差表示，常规SSL技术在此任务上不能很好地工作。然后，我们提出了一种新的SSL训练目标，称为双标记掩蔽语音预测，它显式地保留了重叠语音中所有说话者的表示。我们研究了所提出的系统的各个方面，包括数据配置和量化器选择。所提出的SSL设置在LibriSpeechMix数据集上实现了显著更好的单词错误率。摘要：Self-supervised learning (SSL), which utilizes the input data itself for representation learning, has achieved state-of-the-art results for various downstream speech tasks. However, most of the previous studies focused on offline single-talker applications, with limited investigations in multi-talker cases, especially for streaming scenarios. In this paper, we investigate SSL for streaming multi-talker speech recognition, which generates transcriptions of overlapping speakers in a streaming fashion. We first observe that conventional SSL techniques do not work well on this task due to the poor representation of overlapping speech. We then propose a novel SSL training objective, referred to as bi-label masked speech prediction, which explicitly preserves representations of all speakers in overlapping speech. We investigate various aspects of the proposed system including data configuration and quantizer selection. The proposed SSL setup achieves substantially better word error rates on the LibriSpeechMix dataset.
【2】 Self-supervised learning of audio representations using angular contrastive loss
标题：使用角度对比损失的音频表示的自监督学习
链接：https://arxiv.org/abs/2211.05442
* 与cs.SD语音【9】为同一篇
作者：Shanshan Wang,Soumya Tripathy,Annamaria Mesaros机构：Computing Sciences, Tampere University, Tampere, Finland摘要：在自监督学习（Self-Supervised Learning，SSL）中，设计了多种借口任务，通过对比损失学习特征表示。然而，已有的研究表明，由于实例判别目标的固有缺陷，这种损失对语义相似的样本容忍度较低，可能会影响下游任务中学习到的特征嵌入的质量。为了提高SSL中特征嵌入的区分能力，提出了一种新的损失函数--角度对比损失（ACL），它是角度裕度和对比损失的线性组合。ACL通过在SSL中的正和负增广对之间显式地添加角边缘来改进对比学习。实验结果表明，在有监督学习和无监督学习中使用ACL都能显著提高学习性能。我们使用FSDnoisy 18 k数据集验证了我们的新损失函数，其中我们使用监督和自监督学习在声音事件分类中分别实现了73.6%和77.1%的准确率。摘要：In Self-Supervised Learning (SSL), various pretext tasks are designed for learning feature representations through contrastive loss. However, previous studies have shown that this loss is less tolerant to semantically similar samples due to the inherent defect of instance discrimination objectives, which may harm the quality of learned feature embeddings used in downstream tasks. To improve the discriminative ability of feature embeddings in SSL, we propose a new loss function called Angular Contrastive Loss (ACL), a linear combination of angular margin and contrastive loss. ACL improves contrastive learning by explicitly adding an angular margin between positive and negative augmented pairs in SSL. Experimental results show that using ACL for both supervised and unsupervised learning significantly improves performance. We validated our new loss function using the FSDnoisy18k dataset, where we achieved 73.6% and 77.1% accuracy in sound event classification using supervised and self-supervised learning, respectively.
【3】 Speech separation with large-scale self-supervised learning
标题：大规模自监督学习的语音分离
链接：https://arxiv.org/abs/2211.05172
* 与cs.SD语音【10】为同一篇
作者：Zhuo Chen,Naoyuki Kanda,Jian Wu,Yu Wu,Xiaofei Wang,Takuya Yoshioka,Jinyu Li,Sunit Sivasankaran,Sefik Emre Eskimez机构：Microsoft, One Microsoft Way, Redmond, USA摘要：自监督学习（SSL）方法如WavLM在小规模仿真实验中显示出良好的语音分离（SS）效果。在这项工作中，我们通过大规模扩展预训练数据（超过30万小时）和微调数据（1万小时），扩展了对基于SSL的SS的探索。我们还研究了在有限的计算预算下有效地将预训练模型与SS网络集成的各种技术，包括低帧速率SSL模型训练设置和仅使用预训练模型的一部分的微调方案。在一个模拟的远场语音混合测试集上，与有监督的基线模型和基于WavLM的SS模型相比，我们的模型分别获得了15.9%和11.2%的相对词错误率（WER）降低.对于使用连续语音分离的真实会议记录上的会话转录，所提出的模型在AMI和ICSI评估集上分别实现了相对于纯监督基线的6.8%和10.6%的相对WER减少，同时减少了38%的计算成本。摘要：Self-supervised learning (SSL) methods such as WavLM have shown promising speech separation (SS) results in small-scale simulation-based experiments. In this work, we extend the exploration of the SSL-based SS by massively scaling up both the pre-training data (more than 300K hours) and fine-tuning data (10K hours). We also investigate various techniques to efficiently integrate the pre-trained model with the SS network under a limited computation budget, including a low frame rate SSL model training setup and a fine-tuning scheme using only the part of the pre-trained model. Compared with a supervised baseline and the WavLM-based SS model using feature embeddings obtained with the previously released 94K hours trained WavLM, our proposed model obtains 15.9% and 11.2% of relative word error rate (WER) reductions, respectively, for a simulated far-field speech mixture test set. For conversation transcription on real meeting recordings using continuous speech separation, the proposed model achieves 6.8% and 10.6% of relative WER reductions over the purely supervised baseline on AMI and ICSI evaluation sets, respectively, while reducing the computational cost by 38%.
【4】 Adaptive Multi-Corpora Language Model Training for Speech Recognition
标题：用于语音识别的自适应多语料库语言模型训练
链接：https://arxiv.org/abs/2211.05121
作者：Yingyi Ma,Zhe Liu,Xuedong Zhang机构：Meta AI, Menlo Park, CA, USA摘要：神经网络语言模型（NNLM）在自动语音识别（ASR）系统中起着至关重要的作用，特别是在只有文本数据时的自适应任务中。在实践中，NNLM通常在从多个语料库采样的数据的组合上训练。因此，数据采样策略对于自适应性能是重要的。现有的研究大多集中在静态采样策略的设计上。然而，每个语料库可能在不同的NNLM训练阶段显示出不同的影响。本文提出了一种新的自适应多语料库训练算法，该算法在训练过程中动态学习和调整每个语料库的采样概率。该算法对语料库规模和领域相关性具有鲁棒性。与静态采样策略基线相比，该方法在域内和域外自适应任务上的误字率分别降低了7%和9%。摘要：Neural network language model (NNLM) plays an essential role in automatic speech recognition (ASR) systems, especially in adaptation tasks when text-only data is available. In practice, an NNLM is typically trained on a combination of data sampled from multiple corpora. Thus, the data sampling strategy is important to the adaptation performance. Most existing works focus on designing static sampling strategies. However, each corpus may show varying impacts at different NNLM training stages. In this paper, we introduce a novel adaptive multi-corpora training algorithm that dynamically learns and adjusts the sampling probability of each corpus along the training process. The algorithm is robust to corpora sizes and domain relevance. Compared with static sampling strategy baselines, the proposed approach yields remarkable improvement by achieving up to relative 7% and 9% word error rate (WER) reductions on in-domain and out-of-domain adaptation tasks, respectively.
【5】 Massively Multilingual ASR on 70 Languages: Tokenization, Architecture, and Generalization Capabilities
标题：70种语言的大规模多语言ASR：标记化、体系结构和泛化功能
链接：https://arxiv.org/abs/2211.05756
* 与cs.SD语音【1】为同一篇
作者：Andros Tjandra,Nayan Singhal,David Zhang,Ozlem Kalinli,Abdelrahman Mohamed,Duc Le,Michael L. Seltzer备注：Submitted to ICASSP 2023摘要：端到端多语言ASR已变得越来越有吸引力，这是因为它简化了培训和部署流程，并从高资源语言向低资源语言实现了良好的性能转换。但是，扩展语言数量、总小时数和唯一令牌数量并不是一项简单的任务。本文研究了70种语言的大规模多语言ASR模型。我们考察两种架构：（1）共享嵌入和输出模型;（2）多嵌入和输出模型。在共享模型的实验中，我们展示了标记化策略在不同语言中的重要性。之后，我们使用我们最佳的记号化策略来训练多重嵌入与输出模型，以进一步改善我们的结果。与单语模型相比，我们的多语言ASR实现了13.9%-15.6%的平均WER相对改善。实验结果表明，该算法在未知数据集和未知领域上具有良好的泛化能力，在zero-shot和微调的多语种库语音（MLS）上分别获得了9.5%和7.5%的WER.摘要：End-to-end multilingual ASR has become more appealing because of several reasons such as simplifying the training and deployment process and positive performance transfer from high-resource to low-resource languages. However, scaling up the number of languages, total hours, and number of unique tokens is not a trivial task. This paper explores large-scale multilingual ASR models on 70 languages. We inspect two architectures: (1) Shared embedding and output and (2) Multiple embedding and output model. In the shared model experiments, we show the importance of tokenization strategy across different languages. Later, we use our optimal tokenization strategy to train multiple embedding and output model to further improve our result. Our multilingual ASR achieves 13.9%-15.6% average WER relative improvement compared to monolingual models. We show that our multilingual ASR generalizes well on an unseen dataset and domain, achieving 9.5% and 7.5% WER on Multilingual Librispeech (MLS) with zero-shot and finetuning, respectively.
【6】 Vis2Mus: Exploring Multimodal Representation Mapping for Controllable Music Generation
标题：Vis2Mus：探索可控音乐生成的多通道表示映射
链接：https://arxiv.org/abs/2211.05543
* 与cs.SD语音【2】为同一篇
作者：Runbang Zhang,Yixiao Zhang,Kai Shao,Ying Shan,Gus Xia机构：Music X lab, NYU Shanghai, Centre for Digital Music, Queen Mary University of London, Tencent Inc. , MBZUAI备注：Submitted to ICASSP 2023. GitHub repo: this https URL摘要：本研究探讨视觉艺术领域到音乐领域的表征映射，以视觉艺术作为控制音乐生成的有效抓手。与大多数纯数据驱动的多模态表征学习研究不同，我们采用了一种综合分析的方法，将深度音乐表征学习与用户研究相结合。这种方法使我们能够发现\textit{interpretable}表示映射，而无需大量的成对数据。特别地，我们发现视觉到音乐的映射具有类似于等变的良好性质。换句话说，我们可以使用各种图像变换，例如，改变亮度、改变对比度、风格转移，来控制音乐领域中的相应变换。此外，我们还发布了Vis 2 Mus系统，作为符号音乐生成的可控界面。摘要：In this study, we explore the representation mapping from the domain of visual arts to the domain of music, with which we can use visual arts as an effective handle to control music generation. Unlike most studies in multimodal representation learning that are purely data-driven, we adopt an analysis-by-synthesis approach that combines deep music representation learning with user studies. Such an approach enables us to discover \textit{interpretable} representation mapping without a huge amount of paired data. In particular, we discover that visual-to-music mapping has a nice property similar to equivariant. In other words, we can use various image transformations, say, changing brightness, changing contrast, style transfer, to control the corresponding transformations in the music domain. In addition, we released the Vis2Mus system as a controllable interface for symbolic music generation.
【7】 Privacy-Utility Balanced Voice De-Identification Using Adversarial Examples
标题：隐私-效用平衡语音识别--使用对抗性实例
链接：https://arxiv.org/abs/2211.05446
* 与cs.SD语音【3】为同一篇
作者：Meng Chen,Li Lu,Jiadi Yu,Yingying Chen,Zhongjie Ba,Feng Lin,Kui Ren机构：∗Zhejiang University †Shanghai Jiao Tong University ‡Rutgers University摘要：面对语音数据发布过程中身份泄露的威胁，用户在享受便捷的语音服务时陷入了隐私与效用的两难境地。现有的研究采用直接修改或基于文本的再合成来去识别用户的声音，但在人类参与者在场的情况下导致不一致的可听性。提出了一种语音去身份识别系统，该系统使用对抗性实例来平衡语音服务的隐私性和实用性。我们设计了一个新颖的卷积对抗性例子，将扰动调制到真实世界的房间脉冲响应中，而不是引入可感知失真的典型加性例子。该系统通过自动说话人识别（ASI）来保护用户身份，同时保持语音感知质量，实现非侵入式去识别。此外，我们的系统通过一个条件变分自动编码器学习一个紧凑的说话人分布，以根据需要采样不同的目标嵌入。结合不同的目标生成和输入特定的扰动构造，我们的系统能够实现任意到任意的身份转换，以实现自适应去身份化。实验结果表明，该系统在主流ASI和商用系统上的去身份识别成功率分别为98%和79%，客观Mel倒谱失真为4.31dB，主观平均评价得分为4.48。摘要：Faced with the threat of identity leakage during voice data publishing, users are engaged in a privacy-utility dilemma when enjoying convenient voice services. Existing studies employ direct modification or text-based re-synthesis to de-identify users' voices, but resulting in inconsistent audibility in the presence of human participants. In this paper, we propose a voice de-identification system, which uses adversarial examples to balance the privacy and utility of voice services. Instead of typical additive examples inducing perceivable distortions, we design a novel convolutional adversarial example that modulates perturbations into real-world room impulse responses. Benefit from this, our system could preserve user identity from exposure by Automatic Speaker Identification (ASI) while remaining the voice perceptual quality for non-intrusive de-identification. Moreover, our system learns a compact speaker distribution through a conditional variational auto-encoder to sample diverse target embeddings on demand. Combining diverse target generation and input-specific perturbation construction, our system enables any-to-any identify transformation for adaptive de-identification. Experimental results show that our system could achieve 98% and 79% successful de-identification on mainstream ASIs and commercial systems with an objective Mel cepstral distortion of 4.31dB and a subjective mean opinion score of 4.48.
【8】 Speech Enhancement with Fullband-Subband Cross-Attention Network
标题：基于全带-子带交叉注意网络的语音增强
链接：https://arxiv.org/abs/2211.05432
* 与cs.SD语音【4】为同一篇
作者：Jun Chen,Wei Rao,Zilin Wang,Zhiyong Wu,Yannan Wang,Tao Yu,Shidong Shang,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Tencent Ethereal Audio Lab, Shenzhen, China, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by InterSpeech 2022. arXiv admin note: text overlap with arXiv:2203.12188摘要：FullSubNet在语音增强中充分利用了全带和子带信息，表现出了良好的性能。然而，在FullSubNet中，全带和子带之间的关系是通过简单地级联全带模型和子带单元的输出来实现的。该算法仅对子带单元补充少量的全局信息，没有考虑全带与子带之间的相互影响。提出了一种全频带-子频带交叉注意（FSCA）模型，用于交互式融合全局和局部信息，并将其应用于FullSubNet。这个新的框架被称为FS-CANet。此外，与FullSubNet不同，FS-CANet通过时间卷积网络（TCN）块对全带提取器进行优化，进一步减小了模型规模。在DNS Challenge - Interspeech 2021数据集上的实验结果表明，FS-CANet的语音增强性能优于其他现有的语音增强方法，并验证了全频段-子频段交叉注意的有效性.摘要：FullSubNet has shown its promising performance on speech enhancement by utilizing both fullband and subband information. However, the relationship between fullband and subband in FullSubNet is achieved by simply concatenating the output of fullband model and subband units. It only supplements the subband units with a small quantity of global information and has not considered the interaction between fullband and subband. This paper proposes a fullband-subband cross-attention (FSCA) module to interactively fuse the global and local information and applies it to FullSubNet. This new framework is called as FS-CANet. Moreover, different from FullSubNet, the proposed FS-CANet optimize the fullband extractor by temporal convolutional network (TCN) blocks to further reduce the model size. Experimental results on DNS Challenge - Interspeech 2021 dataset show that the proposed FS-CANet outperforms other state-of-the-art speech enhancement approaches, and demonstrate the effectiveness of fullband-subband cross-attention.
【9】 GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioning
标题：GanStrument：具有基音不变实例条件的对抗性乐器声音合成
链接：https://arxiv.org/abs/2211.05385
* 与cs.SD语音【5】为同一篇
作者：Gaku Narita,Junichi Shimizu,Taketo Akama机构：Sony Computer Science Laboratories, Tokyo, Japan备注：5 pages, 4 figures, Audio examples: this https URL摘要：本文提出了一种生成式对抗乐器声音合成模型GANStrument。给定一个单拍声音作为输入，它能够在交互时间内生成反映输入音色的音高乐器声音。GANStrument通过实例条件化，提高了合成语音的保真度和多样性，并对不同的输入具有较好的泛化能力。此外，我们还引入了一种用于基音不变特征提取器的对抗训练方案，该方案显著地提高了基音精度和音色一致性。实验结果表明，GANStrument在生成质量和输入可编辑性方面优于未使用实例条件的强基线。定性示例可在线获取。摘要：We propose GANStrument, a generative adversarial model for instrument sound synthesis. Given a one-shot sound as input, it is able to generate pitched instrument sounds that reflect the timbre of the input within an interactive time. By exploiting instance conditioning, GANStrument achieves better fidelity and diversity of synthesized sounds and generalization ability to various inputs. In addition, we introduce an adversarial training scheme for a pitch-invariant feature extractor that significantly improves the pitch accuracy and timbre consistency. Experimental results show that GANStrument outperforms strong baselines that do not use instance conditioning in terms of generation quality and input editability. Qualitative examples are available online.
【10】 EMOFAKE: An Initial Dataset For Emotion Fake Audio Detection
标题：EMOFAKE：一种情感伪音频检测的初始数据集
链接：https://arxiv.org/abs/2211.05363
* 与cs.SD语音【6】为同一篇
作者：Yan Zhao,Jiangyan Yi,Jianhua Tao,Chenglong Wang,Chu Yuan Zhang,Tao Wang,Yongfeng Dong机构：School of Artifcial Intelligence, Hebei University of Technology, China, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China摘要：已经有一些数据集用于伪音频检测，例如ASVspoof和ADD数据集。然而，这些数据库没有考虑音频的情感已经从一个改变到另一个，而其它信息（例如，说话者身份和内容）保持相同的情况。情感的变化往往会导致语义的变化。这可能对社会稳定构成极大威胁。因此，本文报告了我们在开发这样一个涉及原始音频的变化情感状态的情感假音频检测数据集方面的进展。数据集命名为EmoFake。EmoFake中的假音频是使用最先进的情感语音转换模型生成的。在此数据集上进行了一些基准实验。结果表明，该数据集对ASVspoof 2021的LCNN和RawNet2基线模型提出了挑战.摘要：There are already some datasets used for fake audio detection, such as the ASVspoof and ADD datasets. However, these databases do not consider a situation that the emotion of the audio has been changed from one to another, while other information (e.g. speaker identity and content) remains the same. Changing emotions often leads to semantic changes. This may be a great threat to social stability. Therefore, this paper reports our progress in developing such an emotion fake audio detection dataset involving changing emotion state of the original audio. The dataset is named EmoFake. The fake audio in EmoFake is generated using the state-of-the-art emotion voice conversion models. Some benchmark experiments are conducted on this dataset. The results show that our designed dataset poses a challenge to the LCNN and RawNet2 baseline models of ASVspoof 2021.
【11】 Multimodal Dyadic Impression Recognition via Listener Adaptive Cross-Domain Fusion
标题：基于听者自适应跨域融合的多模式二元印象识别
链接：https://arxiv.org/abs/2211.05163
* 与cs.SD语音【7】为同一篇
作者：Yuanchao Li,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, UK备注：submitted to ICASSP2023. arXiv admin note: substantial text overlap with arXiv:2203.13932摘要：作为情感计算的一个分支，印象识别，对说话者特征（例如热情或能力）的感知潜在地是人与人的对话和口语对话系统的关键部分。大多数研究只从说话人的行为或听话人的反应来研究印象，而忽略了它们之间的潜在联系。本文提出了一种听者自适应跨域结构，该结构由听者自适应函数和跨域融合函数组成，听者自适应函数用于建模说话人和听者行为之间的因果关系，跨域融合函数用于加强说话人和听者行为之间的联系。在二元IMPRESSION数据集上的实验验证了该方法的有效性，在能力和热情维度上的一致性相关系数分别为78.8%和77.5%，优于以往的研究。所提出的方法有望推广到类似的二元交互场景。摘要：As a sub-branch of affective computing, impression recognition, e.g., perception of speaker characteristics such as warmth or competence, is potentially a critical part of both human-human conversations and spoken dialogue systems. Most research has studied impressions only from the behaviors expressed by the speaker or the response from the listener, yet ignored their latent connection. In this paper, we perform impression recognition using a proposed listener adaptive cross-domain architecture, which consists of a listener adaptation function to model the causality between speaker and listener behaviors and a cross-domain fusion function to strengthen their connection. The experimental evaluation on the dyadic IMPRESSION dataset verified the efficacy of our method, producing concordance correlation coefficients of 78.8% and 77.5% in the competence and warmth dimensions, outperforming previous studies. The proposed method is expected to be generalized to similar dyadic interaction scenarios.
机器翻译，仅供参考
每日语音速递