今日论文合集：cs.SD语音13篇，eess.AS音频处理14篇。
【1】 iQuery: Instruments as Queries for Audio-Visual Sound Separation
标题：IQuery：将乐器作为视听声音分离的查询
链接：https://arxiv.org/abs/2212.03814
作者：Jiaben Chen,Renrui Zhang,Dongze Lian,Jiaqi Yang,Ziyao Zeng,Jianbo Shi机构：UC San Diego, Shanghai AI Laboratory, The Chinese University of Hong Kong, National University of Singapore, ShanghaiTech University, University of Pennsylvania摘要：当前的视听分离方法共享标准架构设计，其中音频编码器-解码器网络在编码器瓶颈处与视觉编码特征融合。该设计将多模态特征编码的学习与用于音频分离的鲁棒声音解码相混淆。推广到新乐器：必须为所有乐器微调整个视觉和音频网络。我们重新定义了视听分离任务，并提出了具有灵活查询扩展机制的仪器即查询（iQuery）。我们的方法确保了跨模态一致性和跨仪器解纠缠。我们利用“视觉命名”查询来启动音频查询的学习，并使用跨模态注意力来去除估计波形处的潜在声源干扰。为了推广到一个新的仪器或事件类，我们从文本提示设计中汲取灵感，插入一个额外的查询作为音频提示，同时冻结注意机制。在3个基准测试上的实验结果表明，iQuery提高了音视频源分离性能。摘要：Current audio-visual separation methods share a standard architecture design where an audio encoder-decoder network is fused with visual encoding features at the encoder bottleneck. This design confounds the learning of multi-modal feature encoding with robust sound decoding for audio separation. To generalize to a new instrument: one must finetune the entire visual and audio network for all musical instruments. We re-formulate visual-sound separation task and propose Instrument as Query (iQuery) with a flexible query expansion mechanism. Our approach ensures cross-modal consistency and cross-instrument disentanglement. We utilize "visually named" queries to initiate the learning of audio queries and use cross-modal attention to remove potential sound source interference at the estimated waveforms. To generalize to a new instrument or event class, drawing inspiration from the text-prompt design, we insert an additional query as an audio prompt while freezing the attention mechanism. Experimental results on three benchmarks demonstrate that our iQuery improves audio-visual sound source separation performance.
【2】 An Overview of Indian Spoken Language Recognition from Machine Learning Perspective
标题：机器学习视角下的印度口语识别研究综述
链接：https://arxiv.org/abs/2212.03812
作者：Spandan Dey,Md Sahidullah,Goutam Saha机构：Dept. of E&ECE, Indian Institute of Technology Kharagpur, Kharagpur, West Bengal, India, Université de Lorraine, CNRS, Inria, LORIA, A R T I C L E I N F O摘要：在基于语音命令的多语言人机交互时代，口语自动识别是一个非常重要的研究领域。前端LID模块有助于提高多语言场景中许多基于语音的应用程序的性能。印度是一个人口众多的国家，拥有多种文化和语言。大多数印度人需要使用各自的母语与机器进行口头交互。因此，开发高效的印度口语识别系统对于在印度社会的各个部门采用智能技术是有用的。在过去的二十年里，印度LID领域开始获得动力，主要是由于几个标准的印度语言多语言语音语料库的开发。尽管在这一领域已经取得了重大的研究进展，但就我们所知，对它们进行集体分析性审查的尝试并不多。在这项工作中，我们进行了一个非常第一次尝试，提出了一个全面的审查印度口语识别研究领域。深入的分析强调了在印度背景下发展LID系统所面临的资源不足和相互影响的独特挑战。本文对印度LID研究的几个重要方面进行了讨论，如对现有语音语料库的详细描述，主要研究成果，包括早期基于统计建模的尝试到最近基于不同神经网络结构的方法，以及未来的研究趋势。这一回顾工作将有助于评估目前印度LID研究的状态，由任何积极的研究人员或任何研究爱好者从相关领域。摘要：Automatic spoken language identification (LID) is a very important research field in the era of multilingual voice-command-based human-computer interaction (HCI). A front-end LID module helps to improve the performance of many speech-based applications in the multilingual scenario. India is a populous country with diverse cultures and languages. The majority of the Indian population needs to use their respective native languages for verbal interaction with machines. Therefore, the development of efficient Indian spoken language recognition systems is useful for adapting smart technologies in every section of Indian society. The field of Indian LID has started gaining momentum in the last two decades, mainly due to the development of several standard multilingual speech corpora for the Indian languages. Even though significant research progress has already been made in this field, to the best of our knowledge, there are not many attempts to analytically review them collectively. In this work, we have conducted one of the very first attempts to present a comprehensive review of the Indian spoken language recognition research field. In-depth analysis has been presented to emphasize the unique challenges of low-resource and mutual influences for developing LID systems in the Indian contexts. Several essential aspects of the Indian LID research, such as the detailed description of the available speech corpora, the major research contributions, including the earlier attempts based on statistical modeling to the recent approaches based on different neural network architectures, and the future research trends are discussed. This review work will help assess the state of the present Indian LID research by any active researcher or any research enthusiasts from related fields.
【3】 Magic: Multi Art Genre Intelligent Choreography Dataset and Network for 3D Dance Generation
标题：Magic：面向3D舞蹈生成的多艺术流派智能编排数据集和网络
链接：https://arxiv.org/abs/2212.03741
作者：Ronghui Li,Junfan Zhao,Yachao Zhang,Mingyang Su,Zeping Ren,Han Zhang,Xiu Li机构：Shenzhen International Graduate School, Tsinghua University, School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, ..., Audio data, ["Name": "zentan", “Class": "Classical", “Genre": "ShenYun", "Frames": ,], TAGS:摘要：Achieving multiple genres and long-term choreography sequences from given music is a challenging task, due to the lack of a multi-genre dataset. To tackle this problem,we propose a Multi Art Genre Intelligent Choreography Dataset (MagicDance). The data of MagicDance is captured from professional dancers assisted by motion capture technicians. It has a total of 8 hours 3D motioncapture human dances with paired music, and 16 different dance genres. To the best of our knowledge, MagicDance is the 3D dance dataset with the most genres. In addition, we find that the existing two types of methods (generation-based method and synthesis-based method) can only satisfy one of the diversity and duration, but they can complement to some extent. Based on this observation, we also propose a generation-synthesis choreography network (MagicNet), which cascades a Diffusion-based 3D Diverse Dance fragments Generation Network (3DGNet) and a Genre&Coherent aware Retrieval Module (GCRM). The former can generate various dance fragments from only one music clip. The latter is utilized to select the best dance fragment generated by 3DGNet and switch them into a complete dance according to the genre and coherent matching score. Quantitative and qualitative experiments demonstrate the quality of MagicDance, and the state-of-the-art performance of MagicNet.
【4】 M3ST: Mix at Three Levels for Speech Translation
标题：M3ST：语音翻译的三个层次混合
链接：https://arxiv.org/abs/2212.03657
作者：Xuxin Cheng,Qianqian Dong,Fengpeng Yue,Tom Ko,Mingxuan Wang,Yuexian Zou机构：ADSPLAB, School of ECE, Peking University,ByteDance AI Lab,Peng Cheng Laboratory备注：Submitted to ICASSP 2023摘要：How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
【5】 Low-Resource End-to-end Sanskrit TTS using Tacotron2, WaveGlow and Transfer Learning
标题：使用Taco tron2、WaveGlow和迁移学习的低资源端到端梵文TTS
链接：https://arxiv.org/abs/2212.03558
作者：Ankur Debnath,Shridevi S Patil,Gangotri Nadiger,Ramakrishnan Angarai Ganesan机构：Department of Electrical Engineering, Indian Institute of Science, Bengaluru, Dept of Information Science and Engineering, SDM College of Engineering and Technology, Dharwad摘要：端到端文本到语音（TTS）系统已经被开发用于欧洲语言，如英语和西班牙语，具有最先进的语音质量、韵律和自然度。然而，印度语端到端TTS的开发在质量方面落后。这项任务所涉及的挑战是：1）缺乏高质量的培训数据; 2）训练和推理效率低; 3）在词汇量大的情况下收敛慢。在本文报告的工作中，我们研究了使用有限的梵语数据微调英语预训练的Tacotron 2模型，以在低资源设置下合成梵语的自然发音语音。我们的实验显示了令人鼓舞的结果，从37个具有良好梵语口语知识的评估者获得了3.38的总体MOS。考虑到我们所使用的语音数据的持续时间仅为2.5小时的事实，这确实是一个非常好的结果。摘要：End-to-end text-to-speech (TTS) systems have been developed for European languages like English and Spanish with state-of-the-art speech quality, prosody, and naturalness. However, development of end-to-end TTS for Indian languages is lagging behind in terms of quality. The challenges involved in such a task are: 1) scarcity of quality training data; 2) low efficiency during training and inference; 3) slow convergence in the case of large vocabulary size. In our work reported in this paper, we have investigated the use of fine-tuning the English-pretrained Tacotron2 model with limited Sanskrit data to synthesize natural sounding speech in Sanskrit in low resource settings. Our experiments show encouraging results, achieving an overall MOS of 3.38 from 37 evaluators with good Sanskrit spoken knowledge. This is really a very good result, considering the fact that the speech data we have used is of duration 2.5 hours only.
【6】 Improve Bilingual TTS Using Dynamic Language and Phonology Embedding
标题：利用动态语言和音位嵌入改进双语文语转换
链接：https://arxiv.org/abs/2212.03435
作者：Fengyu Yang,Jian Luan,Yujun Wang机构：Xiaomi AI Lab, Beijing, China备注：Submitted to ICASSP2023摘要：在大多数情况下，双语TTS需要处理三种类型的输入脚本：仅第一语言、仅第二语言和嵌入第一语言中的第二语言。在后两种情况下，由于第一种语言的影响，第二种语言的语音和语调通常会有很大的不同。因此，如何在不同语境下准确地对第二语言的语音和语调进行建模，而又不相互干扰，是一个很大的挑战。本文构建了一个汉英TTS系统，以获取更标准的汉语单语者的英语口语语音。我们引入语音嵌入来捕捉不同语音之间的英语差异。嵌入掩码应用于语言嵌入以区分不同语言之间的信息，应用于语音嵌入以关注英语表达。我们特别设计了一个嵌入式强度调制器来捕捉语言和语音的动态强度。实验结果表明，该方法能显著提高汉语单语者的英语口语表达的自然度和标准度。通过分析，我们发现在不同的情景下，适当的语音控制有助于更好的表现。摘要：In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, the pronunciation and intonation of the second language are usually quite different due to the influence of the first language. Therefore, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingual Chinese speaker. We introduce phonology embedding to capture the English differences between different phonology. Embedding mask is applied to language embedding for distinguishing information between different languages and to phonology embedding for focusing on English expression. We specially design an embedding strength modulator to capture the dynamic strength of language and phonology. Experiments show that our approach can produce significantly more natural and standard spoken English speech of the monolingual Chinese speaker. From analysis, we find that suitable phonology control contributes to better performance in different scenarios.
【7】 Improved Speech Pre-Training with Supervision-Enhanced Acoustic Unit
标题：使用监听增强型声学单元改进语音预训练
链接：https://arxiv.org/abs/2212.03482
作者：Pengcheng Li,Genshun Wan,Fenglin Ding,Hang Chen,Jianqing Gao,Jia Pan,Cong Liu机构：Shanghai AI Laboratory, China, FLYTEK Research, iFLYTEK Co. Ltd., China, University of Science and Technology of China, China备注：Submitted to ICASSP 2023摘要：语音预训练在从大规模未标记数据中学习有用的和一般的潜在表示方面取得了很大的成功。基于设计良好的自监督学习模式，预先训练好的模型可用于许多下游语音任务，如自动语音识别。为了充分利用低资源任务中的标记数据，提出了一种改进的预训练方法，通过引入监督增强声学单元（SEAU）模式来强化文本信息的表达，降低训练成本。从SEAU模式中提取的编码器表示被用于为HuBERT预训练过程生成更有代表性的目标单元。在500小时和100小时微调数据的土库曼语语音识别任务上，与标准HuBERT相比，SeHuBERT的相对误词率分别降低了10.5%和4.9%.扩展到更多语言和更多数据，SeHuBERT还可以实现大约10%的相对单词错误率降低，而训练成本是HuBERT的一半。摘要：Speech pre-training has shown great success in learning useful and general latent representations from large-scale unlabeled data. Based on a well-designed self-supervised learning pattern, pre-trained models can be used to serve lots of downstream speech tasks such as automatic speech recognition. In order to take full advantage of the labed data in low resource task, we present an improved pre-training method by introducing a supervision-enhanced acoustic unit (SEAU) pattern to intensify the expression of comtext information and ruduce the training cost. Encoder representations extracted from the SEAU pattern are used to generate more representative target units for HuBERT pre-training process. The proposed method, named SeHuBERT, achieves a relative word error rate reductions of 10.5% and 4.9% comared with the standard HuBERT on Turkmen speech recognition task with 500 hours and 100 hours fine-tuning data respectively. Extended to more languages and more data, SeHuBERT can aslo achieve a relative word error rate reductions of approximately 10% at half of the training cost compared with HuBERT.
【8】 Progressive Multi-Scale Self-Supervised Learning for Speech Recognition
标题：用于语音识别的渐进式多尺度自监督学习
链接：https://arxiv.org/abs/2212.03480
作者：Genshun Wan,Tan Liu,Hang Chen,Jia Pan,Cong Liu,Zhongfu Ye机构：University of Science and Technology of China, China, FLYTEK Research, iFLYTEK Co. Ltd., China备注：Submitted to ICASSP 2023摘要：自监督学习（SSL）模型在自动语音识别（ASR）中取得了显著的改进。此外，如果该模型专用于音频内容信息学习，理论上可以进一步提高ASR性能。为此，提出了一种渐进式多尺度自监督学习（PMS-SSL）方法，该方法在上层使用细粒度目标集计算SSL损失，在中间层使用粗粒度目标集计算SSL损失.此外，PMS-SSL将多尺度结构引入到多头部自注意中，在高层将注意区域限制在较大的范围内，而在低层将注意区域限制在较小的范围内，以提高语音的表示能力。在Librispeech数据集上的实验表明了该方法的有效性。与HuBERT相比，PMS-SSL在10 hours/100 hours测试子集上的相对WER降低分别为13.7% / 12.7%。摘要：Self-supervised learning (SSL) models have achieved considerable improvements in automatic speech recognition (ASR). In addition, ASR performance could be further improved if the model is dedicated to audio content information learning theoretically. To this end, we propose a progressive multi-scale self-supervised learning (PMS-SSL) method, which uses fine-grained target sets to compute SSL loss at top layer while uses coarse-grained target sets at intermediate layers. Furthermore, PMS-SSL introduces multi-scale structure into multi-head self-attention for better speech representation, which restricts the attention area into a large scope at higher layers while restricts the attention area into a small scope at lower layers. Experiments on Librispeech dataset indicate the effectiveness of our proposed method. Compared with HuBERT, PMS-SSL achieves 13.7% / 12.7% relative WER reduction on test other evaluation subsets respectively when fine-tuned on 10hours / 100hours subsets.
【9】 Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information
标题：结合辅助语言信息的改进的自监督多语言语音表征学习
链接：https://arxiv.org/abs/2212.03476
作者：Fenglin Ding,Genshun Wan,Pengcheng Li,Jia Pan,Cong Liu机构：Shanghai AI Laboratory, China, FLYTEK Research, iFLYTEK Co. Ltd., China, University of Science and Technology of China, China备注：Subimitted to ICASSP 2023摘要：多语言端到端模型已经显示出比单语言系统有很大的改进。随着语音预训练方法的发展，自监督多语言语音表示学习（XLSR）在提高多语言自动语音识别（ASR）性能方面取得了成功。然而，与有监督学习类似，多语种预训练也可能受到语言干扰，进而影响多语种系统的应用。本文介绍了利用辅助语言信息改进自监督多语言预训练的几种技术，包括预训练阶段的语言对抗训练、语言嵌入和语言自适应训练。我们在一个由16种语言组成的多语言ASR任务上进行了实验。实验结果表明，与标准XLSR模型相比，该模型的相对增益为14.3%，与未进行预训练的多语言模型相比，该模型的相对增益为19.8%。摘要：Multilingual end-to-end models have shown great improvement over monolingual systems. With the development of pre-training methods on speech, self-supervised multilingual speech representation learning like XLSR has shown success in improving the performance of multilingual automatic speech recognition (ASR). However, similar to the supervised learning, multilingual pre-training may also suffer from language interference and further affect the application of multilingual system. In this paper, we introduce several techniques for improving self-supervised multilingual pre-training by leveraging auxiliary language information, including the language adversarial training, language embedding and language adaptive training during the pre-training stage. We conduct experiments on a multilingual ASR task consisting of 16 languages. Our experimental results demonstrate 14.3% relative gain over the standard XLSR model, and 19.8% relative gain over the no pre-training multilingual model.
【10】 Improving trajectory localization accuracy via direction-of-arrival estimation
标题：利用波达方向估计提高弹道定位精度
链接：https://arxiv.org/abs/2212.03470
作者：Ruchi Pandey,Shreyas Jaiswal,Huy Phan,Santosh Nannuru机构：† Signal Processing and Communication Research Centre, IIIT Hyderabad, India, ∗ School of Electronic Engineering and Computer Science, Queen Mary University of London, UK摘要：声源定位在声传感和监测相关应用中至关重要。本文综合分析了波达方向（DOA）及其导数对声源定位性能的改善作用。本研究使用SALSA-Lite特征和卷积递归神经网络（CRNN）模型来预测DOA及其一阶导数。通过引入一个修正规则，将预测的DOA与估计的导数相结合，得到最终的DOA。使用TAU-NIGENS空间声音事件（TNSSE）2021数据集进行了实验验证。我们比较了在低信噪比水平下，使用导数预测DOA的网络与仅预测DOA的网络的性能。结果表明，将导数与DOA相结合可以提高运动源的定位精度。摘要：Sound source localization is crucial in acoustic sensing and monitoring-related applications. In this paper, we do a comprehensive analysis of improvement in sound source localization by combining the direction of arrivals (DOAs) with their derivatives which quantify the changes in the positions of sources over time. This study uses the SALSA-Lite feature with a convolutional recurrent neural network (CRNN) model for predicting DOAs and their first-order derivatives. An update rule is introduced to combine the predicted DOAs with the estimated derivatives to obtain the final DOAs. The experimental validation is done using TAU-NIGENS Spatial Sound Events (TNSSE) 2021 dataset. We compare the performance of the networks predicting DOAs with derivative vs. the one predicting only the DOAs at low SNR levels. The results show that combining the derivatives with the DOAs improves the localization accuracy of moving sources.
【11】 Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement
标题：选择器-增强器：语音增强中局部和非局部注意操作的动态选择
链接：https://arxiv.org/abs/2212.03408
作者：Xinmeng Xu,Weiping Tu,Yuhong Yang机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China, Hubei Luojia Laboratory, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China摘要：注意力机制，如局部和非局部注意力，在最近的基于深度学习的语音增强（SE）系统中起着重要作用。然而，自然语音包含许多快速变化且相对短暂的声学事件，因此，通过不加区别地使用局部和非局部注意力来捕获最具信息量的语音特征是一个挑战。我们观察到噪声类型和语音特征在语音序列中是变化的，并且局部和非局部操作可以分别从受损语音中提取不同的特征。为了利用这一点，我们提出了选择器增强器，一个基于双注意的卷积神经网络（CNN）与特征过滤器，可以动态地选择区域从低分辨率语音特征，并把他们馈送到本地或非本地注意操作。特别地，所提出的特征过滤器是通过使用强化学习（RL）来训练的，其中开发的难度调节奖励与网络性能、模型复杂度和“SE任务的难度”有关。实验结果表明，该方法具有与现有方法相当或更好的性能.特别地，Selector-Enhancer对于真实世界的去噪是潜在有效的，其中噪声的数量和类型在单个噪声混合上变化。摘要：Attention mechanisms, such as local and non-local attention, play a fundamental role in recent deep learning based speech enhancement (SE) systems. However, natural speech contains many fast-changing and relatively brief acoustic events, therefore, capturing the most informative speech features by indiscriminately using local and non-local attention is challenged. We observe that the noise type and speech feature vary within a sequence of speech and the local and non-local operations can respectively extract different features from corrupted speech. To leverage this, we propose Selector-Enhancer, a dual-attention based convolution neural network (CNN) with a feature-filter that can dynamically select regions from low-resolution speech features and feed them to local or non-local attention operations. In particular, the proposed feature-filter is trained by using reinforcement learning (RL) with a developed difficulty-regulated reward that is related to network performance, model complexity, and "the difficulty of the SE task". The results show that our method achieves comparable or superior performance to existing approaches. In particular, Selector-Enhancer is potentially effective for real-world denoising, where the number and types of noise are varies on a single noisy mixture.
【12】 MIMO-DBnet: Multi-channel Input and Multiple Outputs DOA-aware Beamforming Network for Speech Separation
标题：MIMO-DBnet：多通道输入多输出DOA感知的语音分离波束形成网络
链接：https://arxiv.org/abs/2212.03401
作者：Yanjie Fu,Haoran Yin,Meng Ge,Longbiao Wang,Gaoyan Zhang,Jianwu Dang,Chengyun Deng,Fei Wang机构：College of Intelligence and Computing, Tianjin University, Tianjin, China, The Chinese University of Hong Kong, Shenzhen, China, Beijing Xiaoju Technology Co., Ltd., Beijing, China备注：Submitted to ICASSP 2023摘要：最近，已经提出了许多基于深度学习的波束形成器用于多通道语音分离。然而，它们大多依赖于预先已知的额外线索，如说话者特征、面部图像或方向信息。本文提出了一种基于混合信号的端到端波束形成网络，即MIMO-DBnet。具体地说，我们设计了一个多通道输入和多输出结构来预测每个源的基于到达方向的嵌入和波束形成权值。精确估计的方向嵌入为神经波束形成器提供了相当有效的空间辨别引导，以抵消相位缠绕的影响，从而允许两个源的语音信号的更精确的重构。实验结果表明，该MIMO-DBnet不仅在相位重叠的情况下，在高频段的性能得到了很大的改善，而且在低频段的性能也得到了很大的提高。摘要：Recently, many deep learning based beamformers have been proposed for multi-channel speech separation. Nevertheless, most of them rely on extra cues known in advance, such as speaker feature, face image or directional information. In this paper, we propose an end-to-end beamforming network for direction guided speech separation given merely the mixture signal, namely MIMO-DBnet. Specifically, we design a multi-channel input and multiple outputs architecture to predict the direction-of-arrival based embeddings and beamforming weights for each source. The precisely estimated directional embedding provides quite effective spatial discrimination guidance for the neural beamformer to offset the effect of phase wrapping, thus allowing more accurate reconstruction of two sources' speech signals. Experiments show that our proposed MIMO-DBnet not only achieves a comprehensive decent improvement compared to baseline systems, but also maintain the performance on high frequency bands when phase wrapping occurs.
【13】 Analysis and Utilization of Entrainment on Acoustic and Emotion Features in User-agent Dialogue
标题：用户-代理对话中声学特征和情感特征的训练分析与利用
链接：https://arxiv.org/abs/2212.03398
作者：Daxin Tan,Nikos Kargas,David McHardy,Constantinos Papayiannis,Antonio Bonafonte,Marek Strelec,Jonas Rohnke,Agis Oikonomou Filandras,Trevor Wood机构：Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Amazon Alexa, Cambridge, United Kingdom摘要：夹带是对话者在对话中调整自己的说话风格以与对方保持一致的现象。它存在于声学、韵律、词汇和句法等多个层面。在本研究中，我们探讨并利用夹带现象来改善语音助理的口语对话系统。本文首先从声学特征的角度分析了人与人对话中夹带现象的存在，然后将分析扩展到情感特征。分析结果表明，在声学和情感特征方面都有强有力的证据表明夹带。基于这一发现，我们实现了两种夹带策略，并评估了将夹带原理集成到文本到语音（TTS）系统中是否改善了合成性能和用户体验。实验结果表明，在考虑声学特征时，将夹带原理引入到TTS系统中可以提高系统的性能，而在考虑情感特征时，其性能没有明显的提高。摘要：Entrainment is the phenomenon by which an interlocutor adapts their speaking style to align with their partner in conversations. It has been found in different dimensions as acoustic, prosodic, lexical or syntactic. In this work, we explore and utilize the entrainment phenomenon to improve spoken dialogue systems for voice assistants. We first examine the existence of the entrainment phenomenon in human-to-human dialogues in respect to acoustic feature and then extend the analysis to emotion features. The analysis results show strong evidence of entrainment in terms of both acoustic and emotion features. Based on this findings, we implement two entrainment policies and assess if the integration of entrainment principle into a Text-to-Speech (TTS) system improves the synthesis performance and the user experience. It is found that the integration of the entrainment principle into a TTS system brings performance improvement when considering acoustic features, while no obvious improvement is observed when considering emotion features.
【1】 Improved Speech Pre-Training with Supervision-Enhanced Acoustic Unit
标题：使用监听增强型声学单元改进语音预训练
链接：https://arxiv.org/abs/2212.03482
*与cs.SD语音【7】为同一篇
作者：Pengcheng Li,Genshun Wan,Fenglin Ding,Hang Chen,Jianqing Gao,Jia Pan,Cong Liu机构：Shanghai AI Laboratory, China, FLYTEK Research, iFLYTEK Co. Ltd., China, University of Science and Technology of China, China备注：Submitted to ICASSP 2023摘要：语音预训练在从大规模未标记数据中学习有用的和一般的潜在表示方面取得了很大的成功。基于设计良好的自监督学习模式，预先训练好的模型可用于许多下游语音任务，如自动语音识别。为了充分利用低资源任务中的标记数据，提出了一种改进的预训练方法，通过引入监督增强声学单元（SEAU）模式来强化文本信息的表达，降低训练成本。从SEAU模式中提取的编码器表示被用于为HuBERT预训练过程生成更有代表性的目标单元。在500小时和100小时微调数据的土库曼语语音识别任务上，与标准HuBERT相比，SeHuBERT的相对误词率分别降低了10.5%和4.9%.扩展到更多语言和更多数据，SeHuBERT还可以实现大约10%的相对单词错误率降低，而训练成本是HuBERT的一半。摘要：Speech pre-training has shown great success in learning useful and general latent representations from large-scale unlabeled data. Based on a well-designed self-supervised learning pattern, pre-trained models can be used to serve lots of downstream speech tasks such as automatic speech recognition. In order to take full advantage of the labed data in low resource task, we present an improved pre-training method by introducing a supervision-enhanced acoustic unit (SEAU) pattern to intensify the expression of comtext information and ruduce the training cost. Encoder representations extracted from the SEAU pattern are used to generate more representative target units for HuBERT pre-training process. The proposed method, named SeHuBERT, achieves a relative word error rate reductions of 10.5% and 4.9% comared with the standard HuBERT on Turkmen speech recognition task with 500 hours and 100 hours fine-tuning data respectively. Extended to more languages and more data, SeHuBERT can aslo achieve a relative word error rate reductions of approximately 10% at half of the training cost compared with HuBERT.
【2】 Progressive Multi-Scale Self-Supervised Learning for Speech Recognition
标题：用于语音识别的渐进式多尺度自监督学习
链接：https://arxiv.org/abs/2212.03480
*与cs.SD语音【8】为同一篇
作者：Genshun Wan,Tan Liu,Hang Chen,Jia Pan,Cong Liu,Zhongfu Ye机构：University of Science and Technology of China, China, FLYTEK Research, iFLYTEK Co. Ltd., China备注：Submitted to ICASSP 2023摘要：Self-supervised learning (SSL) models have achieved considerable improvements in automatic speech recognition (ASR). In addition, ASR performance could be further improved if the model is dedicated to audio content information learning theoretically. To this end, we propose a progressive multi-scale self-supervised learning (PMS-SSL) method, which uses fine-grained target sets to compute SSL loss at top layer while uses coarse-grained target sets at intermediate layers. Furthermore, PMS-SSL introduces multi-scale structure into multi-head self-attention for better speech representation, which restricts the attention area into a large scope at higher layers while restricts the attention area into a small scope at lower layers. Experiments on Librispeech dataset indicate the effectiveness of our proposed method. Compared with HuBERT, PMS-SSL achieves 13.7% / 12.7% relative WER reduction on test other evaluation subsets respectively when fine-tuned on 10hours / 100hours subsets.
【3】 Improved Self-Supervised Multilingual Speech Representation Learning Combined with Auxiliary Language Information
标题：结合辅助语言信息的改进的自监督多语言语音表征学习
链接：https://arxiv.org/abs/2212.03476
*与cs.SD语音【9】为同一篇
作者：Fenglin Ding,Genshun Wan,Pengcheng Li,Jia Pan,Cong Liu机构：Shanghai AI Laboratory, China, FLYTEK Research, iFLYTEK Co. Ltd., China, University of Science and Technology of China, China备注：Subimitted to ICASSP 2023摘要：Multilingual end-to-end models have shown great improvement over monolingual systems. With the development of pre-training methods on speech, self-supervised multilingual speech representation learning like XLSR has shown success in improving the performance of multilingual automatic speech recognition (ASR). However, similar to the supervised learning, multilingual pre-training may also suffer from language interference and further affect the application of multilingual system. In this paper, we introduce several techniques for improving self-supervised multilingual pre-training by leveraging auxiliary language information, including the language adversarial training, language embedding and language adaptive training during the pre-training stage. We conduct experiments on a multilingual ASR task consisting of 16 languages. Our experimental results demonstrate 14.3% relative gain over the standard XLSR model, and 19.8% relative gain over the no pre-training multilingual model.
【4】 Improving trajectory localization accuracy via direction-of-arrival estimation
标题：利用波达方向估计提高弹道定位精度
链接：https://arxiv.org/abs/2212.03470
*与cs.SD语音【10】为同一篇
作者：Ruchi Pandey,Shreyas Jaiswal,Huy Phan,Santosh Nannuru机构：† Signal Processing and Communication Research Centre, IIIT Hyderabad, India, ∗ School of Electronic Engineering and Computer Science, Queen Mary University of London, UK摘要：声源定位在声传感和监测相关应用中至关重要。本文综合分析了波达方向（DOA）及其导数对声源定位性能的改善作用。本研究使用SALSA-Lite特征和卷积递归神经网络（CRNN）模型来预测DOA及其一阶导数。通过引入一个修正规则，将预测的DOA与估计的导数相结合，得到最终的DOA。使用TAU-NIGENS空间声音事件（TNSSE）2021数据集进行了实验验证。我们比较了在低信噪比水平下，使用导数预测DOA的网络与仅预测DOA的网络的性能。结果表明，将导数与波达方向相结合，提高了运动源的定位精度。摘要：Sound source localization is crucial in acoustic sensing and monitoring-related applications. In this paper, we do a comprehensive analysis of improvement in sound source localization by combining the direction of arrivals (DOAs) with their derivatives which quantify the changes in the positions of sources over time. This study uses the SALSA-Lite feature with a convolutional recurrent neural network (CRNN) model for predicting DOAs and their first-order derivatives. An update rule is introduced to combine the predicted DOAs with the estimated derivatives to obtain the final DOAs. The experimental validation is done using TAU-NIGENS Spatial Sound Events (TNSSE) 2021 dataset. We compare the performance of the networks predicting DOAs with derivative vs. the one predicting only the DOAs at low SNR levels. The results show that combining the derivatives with the DOAs improves the localization accuracy of moving sources.
【5】 Selector-Enhancer: Learning Dynamic Selection of Local and Non-local Attention Operation for Speech Enhancement
标题：选择器-增强器：语音增强中局部和非局部注意操作的动态选择
链接：https://arxiv.org/abs/2212.03408
*与cs.SD语音【11】为同一篇
作者：Xinmeng Xu,Weiping Tu,Yuhong Yang机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China, Hubei Luojia Laboratory, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China摘要：注意力机制，如局部和非局部注意力，在最近的基于深度学习的语音增强（SE）系统中起着重要作用。然而，自然语音包含许多快速变化且相对短暂的声学事件，因此，通过不加区别地使用局部和非局部注意力来捕获最具信息量的语音特征是一个挑战。我们观察到噪声类型和语音特征在语音序列中是变化的，并且局部和非局部操作可以分别从受损语音中提取不同的特征。为了利用这一点，我们提出了选择器增强器，一个基于双注意的卷积神经网络（CNN）与特征过滤器，可以动态地选择区域从低分辨率语音特征，并把他们馈送到本地或非本地注意操作。特别地，所提出的特征过滤器是通过使用强化学习（RL）来训练的，其中开发的难度调节奖励与网络性能、模型复杂度和“SE任务的难度”有关。实验结果表明，该方法具有与现有方法相当或更好的性能.特别地，Selector-Enhancer对于真实世界的去噪是潜在有效的，其中噪声的数量和类型在单个噪声混合上变化。摘要：Attention mechanisms, such as local and non-local attention, play a fundamental role in recent deep learning based speech enhancement (SE) systems. However, natural speech contains many fast-changing and relatively brief acoustic events, therefore, capturing the most informative speech features by indiscriminately using local and non-local attention is challenged. We observe that the noise type and speech feature vary within a sequence of speech and the local and non-local operations can respectively extract different features from corrupted speech. To leverage this, we propose Selector-Enhancer, a dual-attention based convolution neural network (CNN) with a feature-filter that can dynamically select regions from low-resolution speech features and feed them to local or non-local attention operations. In particular, the proposed feature-filter is trained by using reinforcement learning (RL) with a developed difficulty-regulated reward that is related to network performance, model complexity, and "the difficulty of the SE task". The results show that our method achieves comparable or superior performance to existing approaches. In particular, Selector-Enhancer is potentially effective for real-world denoising, where the number and types of noise are varies on a single noisy mixture.
【6】 MIMO-DBnet: Multi-channel Input and Multiple Outputs DOA-aware Beamforming Network for Speech Separation
标题：MIMO-DBnet：多通道输入多输出DOA感知的语音分离波束形成网络
链接：https://arxiv.org/abs/2212.03401
*与cs.SD语音【12】为同一篇
作者：Yanjie Fu,Haoran Yin,Meng Ge,Longbiao Wang,Gaoyan Zhang,Jianwu Dang,Chengyun Deng,Fei Wang机构：College of Intelligence and Computing, Tianjin University, Tianjin, China, The Chinese University of Hong Kong, Shenzhen, China, Beijing Xiaoju Technology Co., Ltd., Beijing, China备注：Submitted to ICASSP 2023摘要：最近，已经提出了许多基于深度学习的波束形成器用于多通道语音分离。然而，它们大多依赖于预先已知的额外线索，如说话人特征、面部图像或方向信息。本文提出了一种基于混合信号的端到端波束形成网络，即MIMO-DBnet。具体地说，我们设计了一个多通道输入和多输出结构来预测每个源的基于到达方向的嵌入和波束形成权值。精确估计的方向嵌入为神经波束形成器提供了相当有效的空间辨别引导，以抵消相位缠绕的影响，从而允许两个源的语音信号的更精确的重构。实验结果表明，该MIMO-DBnet不仅在相位重叠的情况下，在高频段的性能得到了很大的改善，而且在低频段的性能也得到了很大的提高。摘要：Recently, many deep learning based beamformers have been proposed for multi-channel speech separation. Nevertheless, most of them rely on extra cues known in advance, such as speaker feature, face image or directional information. In this paper, we propose an end-to-end beamforming network for direction guided speech separation given merely the mixture signal, namely MIMO-DBnet. Specifically, we design a multi-channel input and multiple outputs architecture to predict the direction-of-arrival based embeddings and beamforming weights for each source. The precisely estimated directional embedding provides quite effective spatial discrimination guidance for the neural beamformer to offset the effect of phase wrapping, thus allowing more accurate reconstruction of two sources' speech signals. Experiments show that our proposed MIMO-DBnet not only achieves a comprehensive decent improvement compared to baseline systems, but also maintain the performance on high frequency bands when phase wrapping occurs.
【7】 Analysis and Utilization of Entrainment on Acoustic and Emotion Features in User-agent Dialogue
标题：用户-代理对话中声学特征和情感特征的训练分析与利用
链接：https://arxiv.org/abs/2212.03398
*与cs.SD语音【13】为同一篇
作者：Daxin Tan,Nikos Kargas,David McHardy,Constantinos Papayiannis,Antonio Bonafonte,Marek Strelec,Jonas Rohnke,Agis Oikonomou Filandras,Trevor Wood机构：Department of Electronic Engineering, The Chinese University of Hong Kong, Hong Kong, Amazon Alexa, Cambridge, United Kingdom摘要：夹带是对话者在对话中调整自己的说话风格以与对方保持一致的现象。它存在于声学、韵律、词汇和句法等多个层面。在本研究中，我们探讨并利用夹带现象来改善语音助理的口语对话系统。本文首先从声学特征的角度分析了人与人对话中夹带现象的存在，然后将分析扩展到情感特征。分析结果表明，在声学和情感特征方面都有强有力的证据表明夹带。基于这一发现，我们实现了两种夹带策略，并评估了将夹带原理集成到文本到语音（TTS）系统中是否改善了合成性能和用户体验。实验结果表明，在考虑声学特征时，将夹带原理引入到TTS系统中可以提高系统的性能，而在考虑情感特征时，其性能没有明显的提高。摘要：Entrainment is the phenomenon by which an interlocutor adapts their speaking style to align with their partner in conversations. It has been found in different dimensions as acoustic, prosodic, lexical or syntactic. In this work, we explore and utilize the entrainment phenomenon to improve spoken dialogue systems for voice assistants. We first examine the existence of the entrainment phenomenon in human-to-human dialogues in respect to acoustic feature and then extend the analysis to emotion features. The analysis results show strong evidence of entrainment in terms of both acoustic and emotion features. Based on this findings, we implement two entrainment policies and assess if the integration of entrainment principle into a Text-to-Speech (TTS) system improves the synthesis performance and the user experience. It is found that the integration of the entrainment principle into a TTS system brings performance improvement when considering acoustic features, while no obvious improvement is observed when considering emotion features.
【8】 A Transformer-Based User Satisfaction Prediction for Proactive Interaction Mechanism in DuerOS
标题：DuerOS主动交互机制中基于转换器的用户满意度预测
链接：https://arxiv.org/abs/2212.03817
作者：Wei Shen,Xiaonan He,Chuheng Zhang,Xuyun Zhang,Jian XIe机构：Baidu Inc., Beijing, China, IIIS, Tsinghua University, Macquarie University, Sydney, Australia摘要：近来，口语对话系统已经广泛地部署在各种应用中，服务于大量的终端用户。一个常见的问题是由嘈杂的话语、语义误解或缺乏知识导致的错误使得真实系统难以正确地响应，可能导致不满意的用户体验。为了避免这种情况，我们考虑主动交互机制，其中系统在将候选响应提供给用户之前预测用户对候选响应的满意度。如果用户根据预测不可能满意，则系统将询问用户合适的问题以确定用户的真实意图，而不是直接提供响应。通过与用户的这种交互，系统可以给用户更好的响应。DuerOS是一个大规模的商业对话系统，以往的用户满意度预测模型不适用于DuerOS。它们是基于手工制作的特征，因此很难学习数百万会话背后的复杂模式和多轮会话中的时间依赖性。此外，它们在具有适当标签的基准数据集上训练和评估，这在商业对话系统中是昂贵的。为了应对这些挑战，我们提出了一个管道来预测用户满意度，以帮助DuerOS决定是否在每个回合中要求澄清。具体来说，我们提出首先生成大量弱标签，然后训练基于变换器的模型来预测用户对这些弱标签的满意度。通过在DuerOS上的部署和评估，我们观察到用户满意度预测的准确性相对提高了19%，用户体验相对提高了2.3%。摘要：Recently, spoken dialogue systems have been widely deployed in a variety of applications, serving a huge number of end-users. A common issue is that the errors resulting from noisy utterances, semantic misunderstandings, or lack of knowledge make it hard for a real system to respond properly, possibly leading to an unsatisfactory user experience. To avoid such a case, we consider a proactive interaction mechanism where the system predicts the user satisfaction with the candidate response before giving it to the user. If the user is not likely to be satisfied according to the prediction, the system will ask the user a suitable question to determine the real intent of the user instead of providing the response directly. With such an interaction with the user, the system can give a better response to the user. Previous models that predict the user satisfaction are not applicable to DuerOS which is a large-scale commercial dialogue system. They are based on hand-crafted features and thus can hardly learn the complex patterns lying behind millions of conversations and temporal dependency in multiple turns of the conversation. Moreover, they are trained and evaluated on the benchmark datasets with adequate labels, which are expensive to obtain in a commercial dialogue system. To face these challenges, we propose a pipeline to predict the user satisfaction to help DuerOS decide whether to ask for clarification in each turn. Specifically, we propose to first generate a large number of weak labels and then train a transformer-based model to predict the user satisfaction with these weak labels. Empirically, we deploy and evaluate our model on DuerOS, and observe a 19% relative improvement on the accuracy of user satisfaction prediction and 2.3% relative improvement on user experience.
【9】 iQuery: Instruments as Queries for Audio-Visual Sound Separation
标题：IQuery：将乐器作为视听声音分离的查询
链接：https://arxiv.org/abs/2212.03814
*与cs.SD语音【1】为同一篇
作者：Jiaben Chen,Renrui Zhang,Dongze Lian,Jiaqi Yang,Ziyao Zeng,Jianbo Shi机构：UC San Diego, Shanghai AI Laboratory, The Chinese University of Hong Kong, National University of Singapore, ShanghaiTech University, University of Pennsylvania摘要：当前的视听分离方法共享标准架构设计，其中音频编码器-解码器网络在编码器瓶颈处与视觉编码特征融合。该设计将多模态特征编码的学习与用于音频分离的鲁棒声音解码相混淆。推广到新乐器：必须为所有乐器微调整个视觉和音频网络。我们重新定义了视听分离任务，并提出了具有灵活查询扩展机制的仪器即查询（iQuery）。我们的方法确保了跨模态一致性和跨仪器解纠缠。我们利用“视觉命名”查询来启动音频查询的学习，并使用跨模态注意力来去除估计波形处的潜在声源干扰。为了推广到一个新的仪器或事件类，我们从文本提示设计中汲取灵感，插入一个额外的查询作为音频提示，同时冻结注意机制。在3个基准测试上的实验结果表明，iQuery提高了音视频源分离性能。摘要：Current audio-visual separation methods share a standard architecture design where an audio encoder-decoder network is fused with visual encoding features at the encoder bottleneck. This design confounds the learning of multi-modal feature encoding with robust sound decoding for audio separation. To generalize to a new instrument: one must finetune the entire visual and audio network for all musical instruments. We re-formulate visual-sound separation task and propose Instrument as Query (iQuery) with a flexible query expansion mechanism. Our approach ensures cross-modal consistency and cross-instrument disentanglement. We utilize "visually named" queries to initiate the learning of audio queries and use cross-modal attention to remove potential sound source interference at the estimated waveforms. To generalize to a new instrument or event class, drawing inspiration from the text-prompt design, we insert an additional query as an audio prompt while freezing the attention mechanism. Experimental results on three benchmarks demonstrate that our iQuery improves audio-visual sound source separation performance.
【10】 An Overview of Indian Spoken Language Recognition from Machine Learning Perspective
标题：机器学习视角下的印度口语识别研究综述
链接：https://arxiv.org/abs/2212.03812
*与cs.SD语音【2】为同一篇
作者：Spandan Dey,Md Sahidullah,Goutam Saha机构：Dept. of E&ECE, Indian Institute of Technology Kharagpur, Kharagpur, West Bengal, India, Université de Lorraine, CNRS, Inria, LORIA, A R T I C L E I N F O摘要：Automatic spoken language identification (LID) is a very important research field in the era of multilingual voice-command-based human-computer interaction (HCI). A front-end LID module helps to improve the performance of many speech-based applications in the multilingual scenario. India is a populous country with diverse cultures and languages. The majority of the Indian population needs to use their respective native languages for verbal interaction with machines. Therefore, the development of efficient Indian spoken language recognition systems is useful for adapting smart technologies in every section of Indian society. The field of Indian LID has started gaining momentum in the last two decades, mainly due to the development of several standard multilingual speech corpora for the Indian languages. Even though significant research progress has already been made in this field, to the best of our knowledge, there are not many attempts to analytically review them collectively. In this work, we have conducted one of the very first attempts to present a comprehensive review of the Indian spoken language recognition research field. In-depth analysis has been presented to emphasize the unique challenges of low-resource and mutual influences for developing LID systems in the Indian contexts. Several essential aspects of the Indian LID research, such as the detailed description of the available speech corpora, the major research contributions, including the earlier attempts based on statistical modeling to the recent approaches based on different neural network architectures, and the future research trends are discussed. This review work will help assess the state of the present Indian LID research by any active researcher or any research enthusiasts from related fields.
【11】 Magic: Multi Art Genre Intelligent Choreography Dataset and Network for 3D Dance Generation
标题：Magic：面向3D舞蹈生成的多艺术流派智能编排数据集和网络
链接：https://arxiv.org/abs/2212.03741
*与cs.SD语音【3】为同一篇
作者：Ronghui Li,Junfan Zhao,Yachao Zhang,Mingyang Su,Zeping Ren,Han Zhang,Xiu Li机构：Shenzhen International Graduate School, Tsinghua University, School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University, ..., Audio data, ["Name": "zentan", “Class": "Classical", “Genre": "ShenYun", "Frames": ,], TAGS:摘要：由于缺乏多流派数据集，从给定音乐中获得多流派和长期编排序列是一项具有挑战性的任务。为了解决这个问题，我们提出了一个多艺术流派智能编排数据集（MagicDance）。MagicDance的数据是在动作捕捉技术人员的协助下从专业舞者那里捕捉到的。它总共有8小时的3D动作捕捉人类舞蹈与配对音乐，和16个不同的舞蹈流派。据我们所知，MagicDance是拥有最多流派的3D舞蹈数据集。此外，我们发现现有的两类方法（基于生成的方法和基于合成的方法）只能满足多样性和持续性中的一个，但它们在一定程度上可以互补。在此基础上，提出了一种生成-合成编舞网络（MagicNet），该网络级联了基于扩散的3D多样舞蹈片段生成网络（3DGNet）和体裁连贯感知检索模块（GCRM）。前者可以从一个音乐片段中生成各种舞蹈片段。后者用于选择3DGNet生成的最佳舞蹈片段，并根据舞蹈类型和连贯匹配分数将其转换为完整的舞蹈。定量和定性实验证明了MagicDance的质量和MagicNet的最新性能。摘要：Achieving multiple genres and long-term choreography sequences from given music is a challenging task, due to the lack of a multi-genre dataset. To tackle this problem,we propose a Multi Art Genre Intelligent Choreography Dataset (MagicDance). The data of MagicDance is captured from professional dancers assisted by motion capture technicians. It has a total of 8 hours 3D motioncapture human dances with paired music, and 16 different dance genres. To the best of our knowledge, MagicDance is the 3D dance dataset with the most genres. In addition, we find that the existing two types of methods (generation-based method and synthesis-based method) can only satisfy one of the diversity and duration, but they can complement to some extent. Based on this observation, we also propose a generation-synthesis choreography network (MagicNet), which cascades a Diffusion-based 3D Diverse Dance fragments Generation Network (3DGNet) and a Genre&Coherent aware Retrieval Module (GCRM). The former can generate various dance fragments from only one music clip. The latter is utilized to select the best dance fragment generated by 3DGNet and switch them into a complete dance according to the genre and coherent matching score. Quantitative and qualitative experiments demonstrate the quality of MagicDance, and the state-of-the-art performance of MagicNet.
【12】 M3ST: Mix at Three Levels for Speech Translation
标题：M3ST：语音翻译的三个层次混合
链接：https://arxiv.org/abs/2212.03657
*与cs.SD语音【4】为同一篇
作者：Xuxin Cheng,Qianqian Dong,Fengpeng Yue,Tom Ko,Mingxuan Wang,Yuexian Zou机构：ADSPLAB, School of ECE, Peking University,ByteDance AI Lab,Peng Cheng Laboratory备注：Submitted to ICASSP 2023摘要：如何解决端到端的语篇翻译中的数据稀缺问题？众所周知，数据扩充是一种通过扩大数据集来提高许多任务性能的有效方法。本文提出了三层混合语音翻译（M^3ST）方法，以增加扩充训练语料库的多样性。具体而言，我们使用外部机器翻译（MT）数据，基于预先训练的模型进行两个阶段的微调。在第一阶段的微调中，我们将训练语料在词层、句子层和框架层三个层次上进行混合，并使用混合数据对整个模型进行微调。在第二阶段的微调中，我们将原始语音序列和原始文本序列并行地引入到模型中对网络进行微调，并利用Jensen-Shannon散度对它们的输出进行正则化。在Must-C语音翻译基准上的实验和分析表明，M^3ST在8个方向上的平均BLEU为29.9，优于现有的强基准，达到了最佳的翻译效果.摘要：How to solve the data scarcity problem for end-to-end speech-to-text translation (ST)? It's well known that data augmentation is an efficient method to improve performance for many tasks by enlarging the dataset. In this paper, we propose Mix at three levels for Speech Translation (M^3ST) method to increase the diversity of the augmented training corpus. Specifically, we conduct two phases of fine-tuning based on a pre-trained model using external machine translation (MT) data. In the first stage of fine-tuning, we mix the training corpus at three levels, including word level, sentence level and frame level, and fine-tune the entire model with mixed data. At the second stage of fine-tuning, we take both original speech sequences and original text sequences in parallel into the model to fine-tune the network, and use Jensen-Shannon divergence to regularize their outputs. Experiments on MuST-C speech translation benchmark and analysis show that M^3ST outperforms current strong baselines and achieves state-of-the-art results on eight directions with an average BLEU of 29.9.
【13】 Low-Resource End-to-end Sanskrit TTS using Tacotron2, WaveGlow and Transfer Learning
标题：使用Taco tron2、WaveGlow和迁移学习的低资源端到端梵文TTS
链接：https://arxiv.org/abs/2212.03558
*与cs.SD语音【5】为同一篇
作者：Ankur Debnath,Shridevi S Patil,Gangotri Nadiger,Ramakrishnan Angarai Ganesan机构：Department of Electrical Engineering, Indian Institute of Science, Bengaluru, Dept of Information Science and Engineering, SDM College of Engineering and Technology, Dharwad摘要：端到端文本到语音（TTS）系统已经被开发用于欧洲语言，如英语和西班牙语，具有最先进的语音质量、韵律和自然度。然而，印度语端到端TTS的开发在质量方面落后。这项任务所涉及的挑战是：1）缺乏高质量的培训数据; 2）训练和推理效率低; 3）在词汇量大的情况下收敛慢。在本文报告的工作中，我们研究了使用有限的梵语数据微调英语预训练的Tacotron 2模型，以在低资源设置下合成梵语的自然发音语音。我们的实验显示了令人鼓舞的结果，从37个具有良好梵语口语知识的评估者获得了3.38的总体MOS。考虑到我们所使用的语音数据的持续时间仅为2.5小时的事实，这确实是一个非常好的结果。摘要：End-to-end text-to-speech (TTS) systems have been developed for European languages like English and Spanish with state-of-the-art speech quality, prosody, and naturalness. However, development of end-to-end TTS for Indian languages is lagging behind in terms of quality. The challenges involved in such a task are: 1) scarcity of quality training data; 2) low efficiency during training and inference; 3) slow convergence in the case of large vocabulary size. In our work reported in this paper, we have investigated the use of fine-tuning the English-pretrained Tacotron2 model with limited Sanskrit data to synthesize natural sounding speech in Sanskrit in low resource settings. Our experiments show encouraging results, achieving an overall MOS of 3.38 from 37 evaluators with good Sanskrit spoken knowledge. This is really a very good result, considering the fact that the speech data we have used is of duration 2.5 hours only.
【14】 Improve Bilingual TTS Using Dynamic Language and Phonology Embedding
标题：利用动态语言和音位嵌入改进双语文语转换
链接：https://arxiv.org/abs/2212.03435
*与cs.SD语音【6】为同一篇
作者：Fengyu Yang,Jian Luan,Yujun Wang机构：Xiaomi AI Lab, Beijing, China备注：Submitted to ICASSP2023摘要：在大多数情况下，双语TTS需要处理三种类型的输入脚本：仅第一语言、仅第二语言和嵌入第一语言中的第二语言。在后两种情况下，由于第一种语言的影响，第二种语言的语音和语调通常会有很大的不同。因此，如何在不同语境下准确地对第二语言的语音和语调进行建模，而又不相互干扰，是一个很大的挑战。本文构建了一个汉英TTS系统，以获取更标准的汉语单语者的英语口语语音。我们引入语音嵌入来捕捉不同语音之间的英语差异。嵌入掩码应用于语言嵌入以区分不同语言之间的信息，应用于语音嵌入以关注英语表达。我们特别设计了一个嵌入式强度调制器来捕捉语言和语音的动态强度。实验结果表明，该方法能显著提高汉语单语者的英语口语表达的自然度和标准度。通过分析，我们发现在不同的情景下，适当的语音控制有助于更好的表现。摘要：In most cases, bilingual TTS needs to handle three types of input scripts: first language only, second language only, and second language embedded in the first language. In the latter two situations, the pronunciation and intonation of the second language are usually quite different due to the influence of the first language. Therefore, it is a big challenge to accurately model the pronunciation and intonation of the second language in different contexts without mutual interference. This paper builds a Mandarin-English TTS system to acquire more standard spoken English speech from a monolingual Chinese speaker. We introduce phonology embedding to capture the English differences between different phonology. Embedding mask is applied to language embedding for distinguishing information between different languages and to phonology embedding for focusing on English expression. We specially design an embedding strength modulator to capture the dynamic strength of language and phonology. Experiments show that our approach can produce significantly more natural and standard spoken English speech of the monolingual Chinese speaker. From analysis, we find that suitable phonology control contributes to better performance in different scenarios.
机器翻译，仅供参考
每日语音速递