今日论文合集：cs.SD语音17篇，eess.AS音频处理19篇。
cs.SD语音
【1】 AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation标题：AV-TranSpeech：视听稳健的语音到语音翻译链接：https://arxiv.org/abs/2305.15403
作者：Rongjie Huang,Huadai Liu,Xize Cheng,Yi Ren,Linjun Li,Zhenhui Ye,Jinzheng He,Lichao Zhang,Jinglin Liu,Xiang Yin,Zhou Zhao机构：Zhejiang University, ByteDance备注：Accepted to ACL 2023摘要：直接语音到语音翻译（S2 ST）旨在将语音从一种语言转换成另一种语言，并且迄今为止已经取得了显著的进展。尽管最近取得了成功，但是当前的S2 ST模型在噪声环境中仍然遭受明显的退化，并且不能翻译视觉语音（即，嘴唇和牙齿的运动）。在这项工作中，我们提出了AV-TranSpeech，第一个视听语音到语音（AV-S2 ST）的翻译模型，而不依赖于中间文本。AV-TranSpeech为音频流补充了视觉信息，以提高系统的鲁棒性，并开辟了许多实际应用：口述或配音档案电影。为了缓解有限的并行AV-S2 ST数据的数据稀缺性，我们1）探索使用未标记的视听数据进行自监督预训练以学习上下文表示，以及2）使用在仅音频语料库上训练的S2 ST模型引入跨模态蒸馏以进一步降低对视觉数据的要求。两种语言对的实验结果表明，AV-TranSpeech优于音频模型在所有设置下，无论噪声的类型。对于低资源视听数据（10 h，30 h），与基线相比，交叉模态蒸馏平均产生7.6 BLEU的改善。音频样本可在www.example.com上获得https://AV-TranSpeech.github.io摘要：Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, current S2ST models still suffer from distinct degradation in noisy environments and fail to translate visual speech (i.e., the movement of lips and teeth). In this work, we present AV-TranSpeech, the first audio-visual speech-to-speech (AV-S2ST) translation model without relying on intermediate text. AV-TranSpeech complements the audio stream with visual information to promote system robustness and opens up a host of practical applications: dictation or dubbing archival films. To mitigate the data scarcity with limited parallel AV-S2ST data, we 1) explore self-supervised pre-training with unlabeled audio-visual data to learn contextual representation, and 2) introduce cross-modal distillation with S2ST models trained on the audio-only corpus to further reduce the requirements of visual data. Experimental results on two language pairs demonstrate that AV-TranSpeech outperforms audio-only models under all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with baselines. Audio samples are available at https://AV-TranSpeech.github.io
【2】 Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR标题：Vistaar：印度语言ASR的不同基准和训练集链接：https://arxiv.org/abs/2305.15386
作者：Kaushal Santosh Bhogale,Sai Sundaresan,Abhigyan Raman,Tahir Javed,Mitesh M. Khapra,Pratyush Kumar机构：Indian Institute of Technology Madras, India, AI,Bharat, India, Microsoft, India备注：Accepted in INTERSPEECH 2023摘要：改进ASR系统是必要的，使新的基于LLM的用例可供全球各地的人们访问。在本文中，我们专注于印度语言，并提出了不同的基准需要评估和改善印度语言的ASR系统的情况。为了解决这个问题，我们整理了Vistaar作为一组59个基准，跨越各种语言和领域的组合，我们评估3个公开可用的ASR系统和2个商业系统。我们还通过在12种印度语言的公开训练数据集上微调Whisper模型来训练IndicWhisper模型，总计10.7K小时。我们表明，IndicWhisper显着改善考虑ASR系统的Vistaar基准。事实上，IndicWhisper在59个基准中的39个中具有最低的WER，平均减少4.1 WER。我们开源所有数据集、代码和模型。摘要：Improving ASR systems is necessary to make new LLM-based use-cases accessible to people across the globe. In this paper, we focus on Indian languages, and make the case that diverse benchmarks are required to evaluate and improve ASR systems for Indian languages. To address this, we collate Vistaar as a set of 59 benchmarks across various language and domain combinations, on which we evaluate 3 publicly available ASR systems and 2 commercial systems. We also train IndicWhisper models by fine-tuning the Whisper models on publicly available training datasets across 12 Indian languages totalling to 10.7K hours. We show that IndicWhisper significantly improves on considered ASR systems on the Vistaar benchmark. Indeed, IndicWhisper has the lowest WER in 39 out of the 59 benchmarks, with an average reduction of 4.1 WER. We open-source all datasets, code and models.
【3】 LMs with a Voice: Spoken Language Modeling beyond Speech Tokens标题：有声音的LMS：超越言语符号的口语建模链接：https://arxiv.org/abs/2305.15255
作者：Eliya Nachmani,Alon Levkovitch,Julian Salazar,Chulayutsh Asawaroengchai,Soroosh Mariooryad,RJ Skerry-Ryan,Michelle Tadmor Ramanovich机构：Google Research摘要：我们提出了SPECTRON，一种新的方法来适应预训练的语言模型（LM）执行语音连续。通过利用预先训练的语音编码器，我们的模型生成文本和语音输出，整个系统直接在频谱图上进行端到端的训练。在频谱图域中训练整个模型简化了我们的语音连续系统，而现有的级联方法使用离散语音表示。我们进一步表明，我们的方法超越现有的口语模型在语义内容和扬声器保存，同时也受益于从预先存在的模型转移的知识。音频样本可以在我们的网站www.example.com上找到https://michelleramanovich.github.io/spectron/spectron摘要：We present SPECTRON, a novel approach to adapting pre-trained language models (LMs) to perform speech continuation. By leveraging pre-trained speech encoders, our model generates both text and speech outputs with the entire system being trained end-to-end operating directly on spectrograms. Training the entire model in the spectrogram domain simplifies our speech continuation system versus existing cascade methods which use discrete speech representations. We further show our method surpasses existing spoken language models both in semantic content and speaker preservation while also benefiting from the knowledge transferred from pre-existing models. Audio samples can be found in our website https://michelleramanovich.github.io/spectron/spectron
【4】 PLCMOS -- a data-driven non-intrusive metric for the evaluation of  packet loss concealment algorithms标题：PLCMOS--一种用于评估分组丢失隐藏算法的数据驱动的非侵入式度量链接：https://arxiv.org/abs/2305.15127
作者：Lorenz Diener,Marju Purin,Sten Sootla,Ando Saabas,Robert Aichner,Ross Cutler机构：Microsoft Corporation备注：to appear: INTERSPEECH 2023, associated model release: this https URL摘要：语音质量评估是每一个研究产生或处理语音的模型的研究人员所面临的问题。人的主观评级，在语音质量评估的黄金标准，是昂贵的和耗时的，以获得足够的数量，以获得可靠的数据，而自动化的客观指标显示出低相关性与黄金标准评级。本文介绍了PLCMOS，一个非侵入性的数据驱动的工具，用于生成一个强大的，准确的估计的平均意见评分的人类评分员会分配一个音频文件，已被处理，通过传输在一个退化的分组交换网络与丢失的数据包被愈合的数据包丢失隐藏算法。我们的新模型显示了模型明智的皮尔逊相关性为~0.97和等级相关性为~0.95与人类评级，基本上高于所有其他可用的侵入性和非侵入性指标。该模型作为ONNX模型发布，供其他研究人员在构建PLC系统时使用。摘要：Speech quality assessment is a problem for every researcher working on models that produce or process speech. Human subjective ratings, the gold standard in speech quality assessment, are expensive and time-consuming to acquire in a quantity that is sufficient to get reliable data, while automated objective metrics show a low correlation with gold standard ratings. This paper presents PLCMOS, a non-intrusive data-driven tool for generating a robust, accurate estimate of the mean opinion score a human rater would assign an audio file that has been processed by being transmitted over a degraded packet-switched network with missing packets being healed by a packet loss concealment algorithm. Our new model shows a model-wise Pearson's correlation of ~0.97 and rank correlation of ~0.95 with human ratings, substantially above all other available intrusive and non-intrusive metrics. The model is released as an ONNX model for other researchers to use when building PLC systems.
【5】 Iteratively Improving Speech Recognition and Voice Conversion标题：反复改进语音识别和语音转换链接：https://arxiv.org/abs/2305.15055
作者：Mayank Kumar Singh,Naoya Takahashi,Onoe Naoyuki机构：Sony Research India, India,Sony Group Corporation, Japan摘要：语音转换（VC）任务的许多现有工作使用自动语音识别（ASR）模型来确保源样本和转换样本之间的语言一致性。然而，对于低数据资源域，训练高质量的ASR仍然是一项具有挑战性的任务。在这项工作中，我们提出了一种新的迭代的方式来改善ASR和VC模型。我们首先训练一个ASR模型，该模型用于在训练VC模型时确保内容保留。在下一次迭代中，VC模型被用作数据增强方法，以进一步微调ASR模型并将其推广到不同的扬声器。通过迭代地利用改进的ASR模型来训练VC模型，反之亦然，我们在实验中显示了这两个模型的改进。我们提出的框架优于ASR和一杆VC基线模型在英语唱歌和印地语语音域的主观和客观评价，在低数据资源设置。摘要：Many existing works on voice conversion (VC) tasks use automatic speech recognition (ASR) models for ensuring linguistic consistency between source and converted samples. However, for the low-data resource domains, training a high-quality ASR remains to be a challenging task. In this work, we propose a novel iterative way of improving both the ASR and VC models. We first train an ASR model which is used to ensure content preservation while training a VC model. In the next iteration, the VC model is used as a data augmentation method to further fine-tune the ASR model and generalize it to diverse speakers. By iteratively leveraging the improved ASR model to train VC model and vice-versa, we experimentally show improvement in both the models. Our proposed framework outperforms the ASR and one-shot VC baseline models on English singing and Hindi speech domains in subjective and objective evaluations in low-data resource settings.
【6】 LoopBoxes -- Evaluation of a Collaborative Accessible Digital Musical  Instrument标题：LoopBox--协同可访问数字乐器的评测链接：https://arxiv.org/abs/2305.14875
作者：Andreas Förster,Alarith Uhde,Mathias Komesker,Christina Komesker,Irina Schmidt机构：Technische Universität Berlin, University of Furtwangen, University of Siegen, Siegen, Germany, imui e.V., Cologne, Germany备注：10 pages, 9 figures, to be published in the Proceedings of the International Conference on New Interfaces for Musical Expression (NIME'23)摘要：LoopBoxes是一种无障碍数字乐器，旨在为有特殊教育需求（SEN）的儿童创建直观的循环音乐制作。本文介绍了在德国柏林的音乐节期间的试点研究，以及与儿童和音乐教师在特殊教育学校设置的案例研究的形式的仪器的评价。我们创建了一个模块化的系统，由三个模块组成，提供单个用户以及协作音乐制作。试点研究进行了评估，使用非正式的观察和问卷调查（n = 39），并表示，该仪器提供音乐制作的人，没有事先的音乐知识，在所有年龄组和促进合作的音乐过程。个案研究是基于观察和定性访谈。它确认，该文书满足了学校环境的需要，并指出未来版本如何扩大所有学生，特别是那些患有复杂残疾的学生。此外，开箱即用的功能似乎是至关重要的长期实施的文书在学校设置。摘要：LoopBoxes is an accessible digital musical instrument designed to create an intuitive access to loop based music making for children with special educational needs (SEN). This paper describes the evaluation of the instrument in the form of a pilot study during a music festival in Berlin, Germany, as well as a case study with children and music teachers in a SEN school setting. We created a modular system composed of three modules that afford single user as well as collaborative music making. The pilot study was evaluated using informal observation and questionnaires (n = 39), and indicated that the instrument affords music making for people with and without prior musical knowledge across all age groups and fosters collaborative musical processes. The case study was based on observation and a qualitative interview. It confirmed that the instrument meets the needs of the school settings and indicated how future versions could expand access to all students, especially those experiencing complex disabilities. In addition, out-of-the-box functionality seems to be crucial for the long-term implementation of the instrument in a school setting.
【7】 Interactive Neural Resonators标题：交互式神经谐振器链接：https://arxiv.org/abs/2305.14867
作者：Rodrigo Diaz,Charalampos Saitis,Mark Sandler机构：Centre for Digital Music, Queen Mary University of, London摘要：在这项工作中，我们提出了一种使用神经共振器的实时接触声音的可控合成方法。以前的工作已经使用物理启发的统计方法和物理建模的对象材料和激励信号。我们的方法采用了可微二阶谐振器，并估计其系数使用的神经网络，是条件的物理参数。这允许交互式动态控制和以直观的方式生成新颖的声音。我们展示了我们的方法的实际实施，并探索其潜在的创造性应用。摘要：In this work, we propose a method for the controllable synthesis of real-time contact sounds using neural resonators. Previous works have used physically inspired statistical methods and physical modelling for object materials and excitation signals. Our method incorporates differentiable second-order resonators and estimates their coefficients using a neural network that is conditioned on physical parameters. This allows for interactive dynamic control and the generation of novel sounds in an intuitive manner. We demonstrate the practical implementation of our method and explore its potential creative applications.
【8】 ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text  Translation标题：ComSL：一种端到端语音到文本翻译的复合语音语言模型链接：https://arxiv.org/abs/2305.14838
作者：Chenyang Le,Yao Qian,Long Zhou,Shujie Liu,Michael Zeng,Xuedong Huang机构：Shanghai Jiao Tong University, Shanghai, China, Microsoft Cloud and AI, Redmond, WA, US, Microsoft Research Asia, Beijing, China, Yanmin Qian摘要：由于对训练数据和GPU消耗的大量需求，以及语音和语言之间的模态差距，联合语音-语言训练具有挑战性。我们提出了ComSL，一个语音语言模型建立在公共预训练的语音和语言模型的复合架构之上，并优化数据效率的口语任务。特别是，我们建议将跨模态学习纳入迁移学习，并同时进行下游任务的多任务学习方式。我们的方法已经证明了在端到端语音到文本翻译任务的有效性，实现了一个新的国家的最先进的平均BLEU分数31.5的多语言语音到英语文本翻译任务的21种语言，作为衡量公共CoVoST 2评估集。摘要：Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models and optimized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks, achieving a new state-of-the-art average BLEU score of 31.5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set.
【9】 CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation标题：CMOT：语音翻译中基于最优传输的跨模式混合链接：https://arxiv.org/abs/2305.14635
作者：Yan Zhou,Qingkai Fang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：ACL 2023 main conference摘要：端到端语音翻译是将源语言的语音信号翻译成目标语言的文本的任务。作为一个跨模态的任务，端到端的ST是难以训练有限的数据。现有的方法往往试图从机器翻译（MT）的知识转移，但他们的性能受到限制的语音和文本之间的模态差距。在本文中，我们提出了跨模态混合通过最佳运输CMOT，以克服模态差距。我们发现语音和文本序列之间的对齐，通过最佳的运输，然后混合的序列从不同的方式在令牌级别使用的对齐。在MuST-C ST基准测试上的实验表明，CMOT在8个翻译方向上的平均BLEU为30.0，优于以前的方法。进一步的分析表明，CMOT可以自适应地找到模态之间的对齐，这有助于缓解语音和文本之间的模态差距。代码可在www.example.com上公开获取https://github.com/ictnlp/CMOT。摘要：End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text. Code is publicly available at https://github.com/ictnlp/CMOT.
【10】 Improving Speech Emotion Recognition Performance using Differentiable  Architecture Search标题：利用可区分结构搜索提高语音情感识别性能链接：https://arxiv.org/abs/2305.14402
作者：Thejan Rajapakshe,Rajib Rana,Sara Khalifa,Berrak Sisman,Björn Schuller机构：University of Southern Queensland, Australia, Distributed Sensing Systems Group, Data, - CSIRO, The University of Texas at Dallas, USA, GLAM, Imperial College London, UK摘要：语音情感识别（SER）是人机交互中情感感知通信的关键使能器。深度学习（DL）通过提高模型复杂度来提高SER模型的性能。然而，设计DL架构需要先前的经验和实验评估。令人鼓舞的是，神经架构搜索（NAS）允许自动搜索最佳DL模型。特别地，差分架构搜索（DARTS）是使用NAS来搜索优化模型的有效方法。在本文中，我们提出了DARTS用于联合CNN和LSTM架构，以提高SER性能。我们选择CNN LSTM耦合的灵感来自于显示类似模型提供改进性能的结果。虽然SER的研究人员已经分别考虑了CNN和RNN，但将DART联合用于CNN和LSTM的可行性仍然需要探索。IEMOCAP数据集的实验，我们证明了我们的方法优于最好的报告结果使用DARTS SER。摘要：Speech Emotion Recognition (SER) is a critical enabler of emotion-aware communication in human-computer interactions. Deep Learning (DL) has improved the performance of SER models by improving model complexity. However, designing DL architectures requires prior experience and experimental evaluations. Encouragingly, Neural Architecture Search (NAS) allows automatic search for an optimum DL model. In particular, Differentiable Architecture Search (DARTS) is an efficient method of using NAS to search for optimised models. In this paper, we propose DARTS for a joint CNN and LSTM architecture for improving SER performance. Our choice of the CNN LSTM coupling is inspired by results showing that similar models offer improved performance. While SER researchers have considered CNNs and RNNs separately, the viability of using DARTs jointly for CNN and LSTM still needs exploration. Experimenting with the IEMOCAP dataset, we demonstrate that our approach outperforms best-reported results using DARTS for SER.
【11】 Connecting Multi-modal Contrastive Representations标题：连接多模式对比表征链接：https://arxiv.org/abs/2305.14381
作者：Zehan Wang,Yang Zhao,Xize Cheng,Haifeng Huang,Jiageng Liu,Li Tang,Linjun Li,Yongqi Wang,Aoxiong Yin,Ziang Zhang,Zhou Zhao机构：Zhejiang University备注：Demos are available at \url{this https URL}摘要：多模态对比表征（MCR）学习旨在将不同的模态编码到语义对齐的共享空间中。该范例在各种模态的许多下游任务上显示出显着的泛化能力。然而，对大量高质量数据对的依赖限制了其在更多模式上的进一步发展。本文提出了一种新的训练有效的方法来学习MCR没有配对的数据称为连接多模态对比表示（C-MCR）。具体来说，给定在（A，B）和（B，C）模态对上预训练的两个现有MCR，我们将它们投影到新空间，并使用来自重叠模态B的数据来在新空间中对齐两个MCR。同时，由于模态对（A，B）和（B，C）已经在每个MCR内对齐，所以通过重叠模态学习的连接也可以被转移到非重叠模态对（A，C）。为了释放C-MCR的潜力，我们进一步引入了一个语义增强的内部和内部MCR连接方法。我们首先提高了语义的一致性和完成不同的方式更强大的对齐嵌入。然后，我们利用MCR间比对来建立连接，并采用MCR内比对来更好地维持来自非重叠模态的输入的连接。我们以视听对比学习领域为例，证明了C-MCR的有效性。我们通过文本连接预训练的CLIP和CLAP模型，以获得视听对比表示。值得注意的是，在不使用任何配对的视听数据和进一步调整的情况下，C-MCR在三个视听下游任务的六个数据集上实现了最先进的性能。摘要：Multi-modal Contrastive Representation (MCR) learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them to a new space and use the data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A, C). To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. We take the field of audio-visual contrastive learning as an example to demonstrate the effectiveness of C-MCR. We connect pre-trained CLIP and CLAP models via texts to derive audio-visual contrastive representations. Remarkably, without using any paired audio-visual data and further tuning, C-MCR achieves state-of-the-art performance on six datasets across three audio-visual downstream tasks.
【12】 Zero-shot personalized lip-to-speech synthesis with face image based  voice control标题：基于人脸图像语音控制的Zero-Shot个性化唇语合成链接：https://arxiv.org/abs/2305.14359
作者：Zheng-Yan Sheng,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, P.R.China备注：ICASSP 2023摘要：唇语音合成（Lip2Speech），预测相应的语音从说话的人脸图像，已经见证了显着的进展，各种模型和训练策略在一系列的独立研究。然而，现有的研究不能实现语音控制下的zero-shot条件下，因为额外的发言人嵌入需要提取的自然参考语音和不可用时，只有一个看不见的发言人的无声视频。在本文中，我们提出了一个zero-shot个性化Lip2Speech合成方法，在人脸图像控制扬声器的身份。采用变分自动编码器来解开说话人身份和语言内容的表示，这使得说话人嵌入，以控制合成语音的语音特性，为看不见的说话人。此外，我们提出了相关的跨模态表示学习，以提高基于人脸的说话人嵌入（FSE）的语音控制能力。大量的实验验证了该方法的有效性，其合成的话语是更自然和匹配的个性输入视频比比较方法。据我们所知，本文首次尝试zero-shot个性化Lip2Speech合成与人脸图像，而不是参考音频来控制语音特征。摘要：Lip-to-Speech (Lip2Speech) synthesis, which predicts corresponding speech from talking face images, has witnessed significant progress with various models and training strategies in a series of independent studies. However, existing studies can not achieve voice control under zero-shot condition, because extra speaker embeddings need to be extracted from natural reference speech and are unavailable when only the silent video of an unseen speaker is given. In this paper, we propose a zero-shot personalized Lip2Speech synthesis method, in which face images control speaker identities. A variational autoencoder is adopted to disentangle the speaker identity and linguistic content representations, which enables speaker embeddings to control the voice characteristics of synthetic speech for unseen speakers. Furthermore, we propose associated cross-modal representation learning to promote the ability of face-based speaker embeddings (FSE) on voice control. Extensive experiments verify the effectiveness of the proposed method whose synthetic utterances are more natural and matching with the personality of input video than the compared methods. To our best knowledge, this paper makes the first attempt on zero-shot personalized Lip2Speech synthesis with a face image rather than reference audio to control voice characteristics.
【13】 Diffusion-Based Audio Inpainting标题：基于扩散的音频修复链接：https://arxiv.org/abs/2305.15266
作者：Eloi Moliner,Vesa Välimäki机构：Acoustics Lab, Department of Information and Communications Engineering, Aalto University, Espoo, Finland备注：Submitted for publication to the Journal of Audio Engineering Society on January 30th, 2023摘要：音频修复的目的是重建损坏的录音中丢失的片段。以前的方法产生合理的重建时，间隙长度短于约100\; ms，但是对于更长的间隙，质量降低。本文探讨了深度学习的最新进展，特别是音频修复任务的扩散模型。所提出的方法使用无条件训练的生成模型，其可以以zero-shot方式进行音频修复，为再生任意长度的间隙提供高度灵活性。还提出了基于恒定Q变换的改进的深度神经网络架构，其允许模型利用音频中的音高等变对称性。所提出的算法的性能进行评估，通过客观和主观的指标重建短到中等大小的差距的任务。一个正式的听力测试的结果表明，所提出的方法提供了一个相当的性能对国家的最先进的短间隙，同时保持了良好的音频质量和优于基线的最长间隙长度测试，150\; ms和200这项工作有助于改善具有相当长的局部干扰或脱落的录音的恢复，这些录音必须重建。摘要：Audio inpainting aims to reconstruct missing segments in corrupted recordings. Previous methods produce plausible reconstructions when the gap length is shorter than about 100\;ms, but the quality decreases for longer gaps. This paper explores recent advancements in deep learning and, particularly, diffusion models, for the task of audio inpainting. The proposed method uses an unconditionally trained generative model, which can be conditioned in a zero-shot fashion for audio inpainting, offering high flexibility to regenerate gaps of arbitrary length. An improved deep neural network architecture based on the constant-Q transform, which allows the model to exploit pitch-equivariant symmetries in audio, is also presented. The performance of the proposed algorithm is evaluated through objective and subjective metrics for the task of reconstructing short to mid-sized gaps. The results of a formal listening test show that the proposed method delivers a comparable performance against state-of-the-art for short gaps, while retaining a good audio quality and outperforming the baselines for the longest gap lengths tested, 150\;ms and 200\;ms. This work helps improve the restoration of sound recordings having fairly long local disturbances or dropouts, which must be reconstructed.
【14】 Incorporating Ultrasound Tongue Images for Audio-Visual Speech  Enhancement through Knowledge Distillation标题：利用知识提炼结合超声舌象增强视听语音链接：https://arxiv.org/abs/2305.14933
作者：Rui-Chen Zheng,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, P. R. China备注：To be published in InterSpeech 2023摘要：视听语音增强（AV-SE）旨在增强退化的语音以及额外的视觉信息，如嘴唇视频，并已被证明是比音频语音增强更有效。本文提出进一步结合超声舌图像，以提高基于嘴唇的AV-SE系统的性能。在训练阶段采用知识蒸馏来解决在推断期间获取超声舌图像的挑战，使得音频-唇语音增强学生模型能够从预训练的音频-唇-舌语音增强教师模型学习。实验结果表明，显着改善的质量和清晰度的语音增强所提出的方法相比，传统的音频唇语音增强基线。使用自动语音识别（ASR）的电话错误率（PER）的进一步分析表明，腭和软腭辅音从超声舌图像的引入中受益最大。摘要：Audio-visual speech enhancement (AV-SE) aims to enhance degraded speech along with extra visual information such as lip videos, and has been shown to be more effective than audio-only speech enhancement. This paper proposes further incorporating ultrasound tongue images to improve lip-based AV-SE systems' performance. Knowledge distillation is employed at the training stage to address the challenge of acquiring ultrasound tongue images during inference, enabling an audio-lip speech enhancement student model to learn from a pre-trained audio-lip-tongue speech enhancement teacher model. Experimental results demonstrate significant improvements in the quality and intelligibility of the speech enhanced by the proposed method compared to the traditional audio-lip speech enhancement baselines. Further analysis using phone error rates (PER) of automatic speech recognition (ASR) shows that palatal and velar consonants benefit most from the introduction of ultrasound tongue images.
【15】 P-vectors: A Parallel-Coupled TDNN/Transformer Network for Speaker  Verification标题：P-向量：一种用于说话人确认的并行耦合TDNN/变换网络链接：https://arxiv.org/abs/2305.14778
作者：Xiyuan wang,Fangyuan Wang,Bo Xu,Liang Xu,Jing Xiao机构：Institute of Automation, Chinese Academy of Sciences, School of Future Technology,University of Chinese Academy of Sciences, GammaLab, PingAn OneConnect,PingAn Insurance (Group) Company of China备注：Accepted by INTERSPEECH 2023摘要：通常，时间延迟神经网络（TDNN）和Transformer可以用作说话人验证（SV）的骨干。从全局和局部特征造型的角度看，两种方法各有优缺点。如何将这两种风格特征有效地融合在一起，仍然是一个悬而未决的问题。在本文中，我们探讨了一个并行耦合的TDNN/Transformer网络（p-向量），以取代串行混合网络。p向量允许TDNN和Transformer在保留局部和全局特征的前提下，通过软特征对齐交互（SFAI）来学习彼此的互补信息。此外，p-向量使用空间频率通道注意力（SFA）来增强输入特征的空间相互依赖性建模。最后，通过嵌入聚合层（EAL）对p向量的双分支输出进行组合。实验表明，p-vectors优于MACCIF-TDNN和MFA-Conformer，在VoxCeleb 1-O上的EER相对提高了11.5%和13.9%。摘要：Typically, the Time-Delay Neural Network (TDNN) and Transformer can serve as a backbone for Speaker Verification (SV). Both of them have advantages and disadvantages from the perspective of global and local feature modeling. How to effectively integrate these two style features is still an open issue. In this paper, we explore a Parallel-coupled TDNN/Transformer Network (p-vectors) to replace the serial hybrid networks. The p-vectors allows TDNN and Transformer to learn the complementary information from each other through Soft Feature Alignment Interaction (SFAI) under the premise of preserving local and global features. Also, p-vectors uses the Spatial Frequency-channel Attention (SFA) to enhance the spatial interdependence modeling for input features. Finally, the outputs of dual branches of p-vectors are combined by Embedding Aggregation Layer (EAL). Experiments show that p-vectors outperforms MACCIF-TDNN and MFA-Conformer with relative improvements of 11.5% and 13.9% in EER on VoxCeleb1-O.
【16】 Downstream Task Agnostic Speech Enhancement with Self-Supervised  Representation Loss标题：具有自监督表示丢失的下行任务不可知语音增强链接：https://arxiv.org/abs/2305.14723
作者：Hiroshi Sato,Ryo Masumura,Tsubasa Ochiai,Marc Delcroix,Takafumi Moriya,Takanori Ashihara,Kentaro Shinayama,Saki Mizuno,Mana Ihori,Tomohiro Tanaka,Nobukatsu Hojo机构：NTT Corporation, Japan备注：4 pages , 2 figures, Accepted to Interspeech 2023摘要：自监督学习（SSL）是语音处理领域的最新突破，特别是通过利用大量未标记的音频数据来完成标签稀缺的下游任务。SSL的噪声鲁棒性是扩大其应用的重要挑战之一。我们可以使用语音增强（SE）来解决这个问题。然而，SE模型和SSL模型之间的不匹配潜在地限制了其效果。在这项工作中，我们提出了一个新的SE训练标准，最大限度地减少清洁和增强信号之间的距离，在SSL模型的特征表示，以减轻不匹配。我们期望SSL域中的损失可以引导SE训练以保留或增强高级下游任务可能需要的语音信号的各种级别的特征。实验表明，我们的建议提高了性能的SE和SSL流水线上的5个下游任务的噪声输入，同时保持SE的性能。摘要：Self-supervised learning (SSL) is the latest breakthrough in speech processing, especially for label-scarce downstream tasks by leveraging massive unlabeled audio data. The noise robustness of the SSL is one of the important challenges to expanding its application. We can use speech enhancement (SE) to tackle this issue. However, the mismatch between the SE model and SSL models potentially limits its effect. In this work, we propose a new SE training criterion that minimizes the distance between clean and enhanced signals in the feature representation of the SSL model to alleviate the mismatch. We expect that the loss in the SSL domain could guide SE training to preserve or enhance various levels of characteristics of the speech signals that may be required for high-level downstream tasks. Experiments show that our proposal improves the performance of an SE and SSL pipeline on five downstream tasks with noisy input while maintaining the SE performance.
【17】 On the Transferability of Whisper-based Representations for  "In-the-Wild" Cross-Task Downstream Speech Applications链接：https://arxiv.org/abs/2305.14546
作者：Vamsikrishna Chemudupati,Marzieh Tahaei,Heitor Guimaraes,Arthur Pimentel,Anderson Avila,Mehdi Rezagholizadeh,Boxing Chen,Tiago Falk机构：Huawei Noah’s Ark Lab, Canada, Institut National de la Recherche Scientifique, Canada摘要：大型自监督预训练语音模型在各种语音处理任务中取得了显着的成功。这些模型的自我监督训练产生了通用的语音表示，可用于不同的下游任务，从自动语音识别（ASR）到说话人识别。最近，Whisper提出了一种基于变换器的ASR模型，并在大量弱监督数据上进行了训练;它的表现超过了几个最先进的自我监督模型。鉴于耳语ASR的优越性，在本文中，我们探讨了SUPERB基准测试中的其他四个语音任务的表示的可移植性。此外，我们探讨了“在野外”的任务，语音被破坏的环境噪声和室内混响的耳语表示的鲁棒性。实验结果表明，Whisper在跨任务和环境条件下取得了令人鼓舞的结果，从而显示出跨任务现实世界部署的潜力。摘要：Large self-supervised pre-trained speech models have achieved remarkable success across various speech-processing tasks. The self-supervised training of these models leads to universal speech representations that can be used for different downstream tasks, ranging from automatic speech recognition (ASR) to speaker identification. Recently, Whisper, a transformer-based model was proposed and trained on large amount of weakly supervised data for ASR; it outperformed several state-of-the-art self-supervised models. Given the superiority of Whisper for ASR, in this paper we explore the transferability of the representation for four other speech tasks in SUPERB benchmark. Moreover, we explore the robustness of Whisper representation for ``in the wild'' tasks where speech is corrupted by environment noise and room reverberation. Experimental results show Whisper achieves promising results across tasks and environmental conditions, thus showing potential for cross-task real-world deployment.
eess.AS音频处理
【1】 Diffusion-Based Audio Inpainting标题：基于扩散的音频修复链接：https://arxiv.org/abs/2305.15266
*与cs.SD语音【13】为同一篇
作者：Eloi Moliner,Vesa Välimäki机构：Acoustics Lab, Department of Information and Communications Engineering, Aalto University, Espoo, Finland备注：Submitted for publication to the Journal of Audio Engineering Society on January 30th, 2023摘要：音频修复的目的是重建损坏的录音中丢失的片段。以前的方法产生合理的重建时，间隙长度短于约100\; ms，但是对于更长的间隙，质量降低。本文探讨了深度学习的最新进展，特别是音频修复任务的扩散模型。所提出的方法使用无条件训练的生成模型，其可以以zero-shot方式进行音频修复，为再生任意长度的间隙提供高度灵活性。还提出了基于恒定Q变换的改进的深度神经网络架构，其允许模型利用音频中的音高等变对称性。所提出的算法的性能进行评估，通过客观和主观的指标重建短到中等大小的差距的任务。一个正式的听力测试的结果表明，所提出的方法提供了一个相当的性能对国家的最先进的短间隙，同时保持了良好的音频质量和优于基线的最长间隙长度测试，150\; ms和200这项工作有助于改善具有相当长的局部干扰或脱落的录音的恢复，这些录音必须重建。摘要：Audio inpainting aims to reconstruct missing segments in corrupted recordings. Previous methods produce plausible reconstructions when the gap length is shorter than about 100\;ms, but the quality decreases for longer gaps. This paper explores recent advancements in deep learning and, particularly, diffusion models, for the task of audio inpainting. The proposed method uses an unconditionally trained generative model, which can be conditioned in a zero-shot fashion for audio inpainting, offering high flexibility to regenerate gaps of arbitrary length. An improved deep neural network architecture based on the constant-Q transform, which allows the model to exploit pitch-equivariant symmetries in audio, is also presented. The performance of the proposed algorithm is evaluated through objective and subjective metrics for the task of reconstructing short to mid-sized gaps. The results of a formal listening test show that the proposed method delivers a comparable performance against state-of-the-art for short gaps, while retaining a good audio quality and outperforming the baselines for the longest gap lengths tested, 150\;ms and 200\;ms. This work helps improve the restoration of sound recordings having fairly long local disturbances or dropouts, which must be reconstructed.
【2】 Music Representing Corpus Virtual: An Open Sourced Library for  Explorative Music Generation, Sound Design, and Instrument Creation with  Artificial Intelligence and Machine Learning标题：代表虚拟语料库的音乐：一个开源的库，用于探索性音乐生成、声音设计和使用人工智能和机器学习的乐器创作链接：https://arxiv.org/abs/2305.14948
作者：Christopher Johann Clarke机构：In Fulfillment of The National Arts Council Creation Grant, National Arts Council, Singapore备注：16 pages摘要：Music Representing Corpus Virtual（MRCV）是一个开源软件套件，旨在探索人工智能（AI）和机器学习（ML）在音乐生成，声音设计和虚拟乐器创作（MGSDIC）中的功能。该软件可供不同经验水平的用户使用，重点是为MGSDIC提供探索性方法。MRCV的主要目的是促进创造力，允许用户自定义用于训练神经网络的输入数据集，并为每个神经网络提供一系列选项（在Github Wiki中有详细说明）。该软件套件旨在供音乐家，音频专业人士，声音设计师和作曲家访问，无论他们之前在AI或ML方面的经验如何。本文档以抽象技术细节的方式编写，从而使其易于理解。该软件是开源的，这意味着用户可以为其开发做出贡献，社区可以从其他用户的见解和经验中共同受益。摘要：Music Representing Corpus Virtual (MRCV) is an open source software suite designed to explore the capabilities of Artificial Intelligence (AI) and Machine Learning (ML) in Music Generation, Sound Design, and Virtual Instrument Creation (MGSDIC). The software is accessible to users of varying levels of experience, with an emphasis on providing an explorative approach to MGSDIC. The main aim of MRCV is to facilitate creativity, allowing users to customize input datasets for training the neural networks, and offering a range of options for each neural network (thoroughly documented in the Github Wiki). The software suite is designed to be accessible to musicians, audio professionals, sound designers, and composers, regardless of their prior experience in AI or ML. The documentation is prepared in such a way as to abstract technical details, thereby making it easy to understand. The software is open source, meaning users can contribute to its development, and the community can collectively benefit from the insights and experience of other users.
【3】 Incorporating Ultrasound Tongue Images for Audio-Visual Speech  Enhancement through Knowledge Distillation标题：利用知识提炼结合超声舌象增强视听语音链接：https://arxiv.org/abs/2305.14933
*与cs.SD语音【14】为同一篇
作者：Rui-Chen Zheng,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, P. R. China备注：To be published in InterSpeech 2023摘要：视听语音增强（AV-SE）旨在增强退化的语音以及额外的视觉信息，如嘴唇视频，并已被证明是比音频语音增强更有效。本文提出进一步结合超声舌图像，以提高基于嘴唇的AV-SE系统的性能。在训练阶段采用知识蒸馏来解决在推断期间获取超声舌图像的挑战，使得音频-唇语音增强学生模型能够从预训练的音频-唇-舌语音增强教师模型学习。实验结果表明，显着改善的质量和清晰度的语音增强所提出的方法相比，传统的音频唇语音增强基线。使用自动语音识别（ASR）的电话错误率（PER）的进一步分析表明，腭和软腭辅音从超声舌图像的引入中受益最大。摘要：Audio-visual speech enhancement (AV-SE) aims to enhance degraded speech along with extra visual information such as lip videos, and has been shown to be more effective than audio-only speech enhancement. This paper proposes further incorporating ultrasound tongue images to improve lip-based AV-SE systems' performance. Knowledge distillation is employed at the training stage to address the challenge of acquiring ultrasound tongue images during inference, enabling an audio-lip speech enhancement student model to learn from a pre-trained audio-lip-tongue speech enhancement teacher model. Experimental results demonstrate significant improvements in the quality and intelligibility of the speech enhanced by the proposed method compared to the traditional audio-lip speech enhancement baselines. Further analysis using phone error rates (PER) of automatic speech recognition (ASR) shows that palatal and velar consonants benefit most from the introduction of ultrasound tongue images.
【4】 P-vectors: A Parallel-Coupled TDNN/Transformer Network for Speaker  Verification标题：P-向量：一种用于说话人确认的并行耦合TDNN/变换网络链接：https://arxiv.org/abs/2305.14778
*与cs.SD语音【15】为同一篇
作者：Xiyuan wang,Fangyuan Wang,Bo Xu,Liang Xu,Jing Xiao机构：Institute of Automation, Chinese Academy of Sciences, School of Future Technology,University of Chinese Academy of Sciences, GammaLab, PingAn OneConnect,PingAn Insurance (Group) Company of China备注：Accepted by INTERSPEECH 2023摘要：通常，时间延迟神经网络（TDNN）和Transformer可以用作说话人验证（SV）的骨干。从全局和局部特征造型的角度看，两种方法各有优缺点。如何将这两种风格特征有效地融合在一起，仍然是一个悬而未决的问题。在本文中，我们探讨了一个并行耦合的TDNN/Transformer网络（p-向量），以取代串行混合网络。p向量允许TDNN和Transformer在保留局部和全局特征的前提下，通过软特征对齐交互（SFAI）来学习彼此的互补信息。此外，p-向量使用空间频率通道注意力（SFA）来增强输入特征的空间相互依赖性建模。最后，通过嵌入聚合层（EAL）对p向量的双分支输出进行组合。实验表明，p-vectors优于MACCIF-TDNN和MFA-Conformer，在VoxCeleb 1-O上的EER相对提高了11.5%和13.9%。摘要：Typically, the Time-Delay Neural Network (TDNN) and Transformer can serve as a backbone for Speaker Verification (SV). Both of them have advantages and disadvantages from the perspective of global and local feature modeling. How to effectively integrate these two style features is still an open issue. In this paper, we explore a Parallel-coupled TDNN/Transformer Network (p-vectors) to replace the serial hybrid networks. The p-vectors allows TDNN and Transformer to learn the complementary information from each other through Soft Feature Alignment Interaction (SFAI) under the premise of preserving local and global features. Also, p-vectors uses the Spatial Frequency-channel Attention (SFA) to enhance the spatial interdependence modeling for input features. Finally, the outputs of dual branches of p-vectors are combined by Embedding Aggregation Layer (EAL). Experiments show that p-vectors outperforms MACCIF-TDNN and MFA-Conformer with relative improvements of 11.5% and 13.9% in EER on VoxCeleb1-O.
【5】 Downstream Task Agnostic Speech Enhancement with Self-Supervised  Representation Loss标题：具有自监督表示丢失的下行任务不可知语音增强链接：https://arxiv.org/abs/2305.14723
*与cs.SD语音【16】为同一篇
作者：Hiroshi Sato,Ryo Masumura,Tsubasa Ochiai,Marc Delcroix,Takafumi Moriya,Takanori Ashihara,Kentaro Shinayama,Saki Mizuno,Mana Ihori,Tomohiro Tanaka,Nobukatsu Hojo机构：NTT Corporation, Japan备注：4 pages , 2 figures, Accepted to Interspeech 2023摘要：自监督学习（SSL）是语音处理领域的最新突破，特别是通过利用大量未标记的音频数据来完成标签稀缺的下游任务。SSL的噪声鲁棒性是扩大其应用的重要挑战之一。我们可以使用语音增强（SE）来解决这个问题。然而，SE模型和SSL模型之间的不匹配潜在地限制了其效果。在这项工作中，我们提出了一个新的SE训练标准，最大限度地减少清洁和增强信号之间的距离，在SSL模型的特征表示，以减轻不匹配。我们期望SSL域中的损失可以引导SE训练以保留或增强高级下游任务可能需要的语音信号的各种级别的特征。实验表明，我们的建议提高了性能的SE和SSL流水线上的5个下游任务的噪声输入，同时保持SE的性能。摘要：Self-supervised learning (SSL) is the latest breakthrough in speech processing, especially for label-scarce downstream tasks by leveraging massive unlabeled audio data. The noise robustness of the SSL is one of the important challenges to expanding its application. We can use speech enhancement (SE) to tackle this issue. However, the mismatch between the SE model and SSL models potentially limits its effect. In this work, we propose a new SE training criterion that minimizes the distance between clean and enhanced signals in the feature representation of the SSL model to alleviate the mismatch. We expect that the loss in the SSL domain could guide SE training to preserve or enhance various levels of characteristics of the speech signals that may be required for high-level downstream tasks. Experiments show that our proposal improves the performance of an SE and SSL pipeline on five downstream tasks with noisy input while maintaining the SE performance.
【6】 On the Transferability of Whisper-based Representations for  "In-the-Wild" Cross-Task Downstream Speech Applications链接：https://arxiv.org/abs/2305.14546
*与cs.SD语音【17】为同一篇
作者：Vamsikrishna Chemudupati,Marzieh Tahaei,Heitor Guimaraes,Arthur Pimentel,Anderson Avila,Mehdi Rezagholizadeh,Boxing Chen,Tiago Falk机构：Huawei Noah’s Ark Lab, Canada, Institut National de la Recherche Scientifique, Canada摘要：大型自监督预训练语音模型在各种语音处理任务中取得了显着的成功。这些模型的自我监督训练产生了通用的语音表示，可用于不同的下游任务，从自动语音识别（ASR）到说话人识别。最近，Whisper提出了一种基于变换器的ASR模型，并在大量弱监督数据上进行了训练;它的表现超过了几个最先进的自我监督模型。鉴于耳语ASR的优越性，在本文中，我们探讨了SUPERB基准测试中的其他四个语音任务的表示的可移植性。此外，我们探讨了“在野外”的任务，语音被破坏的环境噪声和室内混响的耳语表示的鲁棒性。实验结果表明，Whisper在跨任务和环境条件下取得了令人鼓舞的结果，从而显示出跨任务现实世界部署的潜力。摘要：Large self-supervised pre-trained speech models have achieved remarkable success across various speech-processing tasks. The self-supervised training of these models leads to universal speech representations that can be used for different downstream tasks, ranging from automatic speech recognition (ASR) to speaker identification. Recently, Whisper, a transformer-based model was proposed and trained on large amount of weakly supervised data for ASR; it outperformed several state-of-the-art self-supervised models. Given the superiority of Whisper for ASR, in this paper we explore the transferability of the representation for four other speech tasks in SUPERB benchmark. Moreover, we explore the robustness of Whisper representation for ``in the wild'' tasks where speech is corrupted by environment noise and room reverberation. Experimental results show Whisper achieves promising results across tasks and environmental conditions, thus showing potential for cross-task real-world deployment.
【7】 Unit-based Speech-to-Speech Translation Without Parallel Data标题：无并行数据的基于单元的语音到语音翻译链接：https://arxiv.org/abs/2305.15405
作者：Anuj Diwan,Anirudh Srinivasan,David Harwath,Eunsol Choi机构：Department of Computer Science, The University of Texas at Austin备注：17 pages, 3 figures摘要：我们提出了一个无监督的语音到语音翻译（S2 ST）系统，不依赖于源语言和目标语言之间的并行数据。我们的方法将源语言和目标语言的语音信号映射到自动发现的，离散的单位和重新制定的问题，作为无监督的单位到单位的机器翻译。我们开发了一个三步训练过程，包括：（a）预训练基于单元的编码器-解码器语言模型，具有去噪目标;（b）用通过对齐单语文本嵌入空间创建的逐词翻译话语对训练它;（c）运行无监督的反向翻译引导脱离初始翻译模型。我们的方法避免映射到文本的语音信号，并使用语音到单元和单元到语音模型，而不是自动语音识别和文本到语音模型。我们评估我们的模型上的综合发言人Europarl-ST英语-德语和德语-英语评估集，发现单位为基础的翻译是可行的，在这种约束的情况下，实现9.29 ASR-BLEU在德语到英语和8.07在英语到德语。摘要：We propose an unsupervised speech-to-speech translation (S2ST) system that does not rely on parallel data between the source and target languages. Our approach maps source and target language speech signals into automatically discovered, discrete units and reformulates the problem as unsupervised unit-to-unit machine translation. We develop a three-step training procedure that involves (a) pre-training an unit-based encoder-decoder language model with a denoising objective (b) training it with word-by-word translated utterance pairs created by aligning monolingual text embedding spaces and (c) running unsupervised backtranslation bootstrapping off of the initial translation model. Our approach avoids mapping the speech signal into text and uses speech-to-unit and unit-to-speech models instead of automatic speech recognition and text to speech models. We evaluate our model on synthetic-speaker Europarl-ST English-German and German-English evaluation sets, finding that unit-based translation is feasible under this constrained scenario, achieving 9.29 ASR-BLEU in German to English and 8.07 in English to German.
【8】 AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation标题：AV-TranSpeech：视听稳健的语音到语音翻译链接：https://arxiv.org/abs/2305.15403
*与cs.SD语音【1】为同一篇
作者：Rongjie Huang,Huadai Liu,Xize Cheng,Yi Ren,Linjun Li,Zhenhui Ye,Jinzheng He,Lichao Zhang,Jinglin Liu,Xiang Yin,Zhou Zhao机构：Zhejiang University, ByteDance备注：Accepted to ACL 2023摘要：直接语音到语音翻译（S2 ST）旨在将语音从一种语言转换成另一种语言，并且迄今为止已经取得了显著的进展。尽管最近取得了成功，但是当前的S2 ST模型在噪声环境中仍然遭受明显的退化，并且不能翻译视觉语音（即，嘴唇和牙齿的运动）。在这项工作中，我们提出了AV-TranSpeech，第一个视听语音到语音（AV-S2 ST）的翻译模型，而不依赖于中间文本。AV-TranSpeech为音频流补充了视觉信息，以提高系统的鲁棒性，并开辟了许多实际应用：口述或配音档案电影。为了缓解有限的并行AV-S2 ST数据的数据稀缺性，我们1）探索使用未标记的视听数据进行自监督预训练以学习上下文表示，以及2）使用在仅音频语料库上训练的S2 ST模型引入跨模态蒸馏以进一步降低对视觉数据的要求。两种语言对的实验结果表明，AV-TranSpeech优于音频模型在所有设置下，无论噪声的类型。对于低资源视听数据（10 h，30 h），与基线相比，交叉模态蒸馏平均产生7.6 BLEU的改善。音频样本可在www.example.com上获得https://AV-TranSpeech.github.io摘要：Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, current S2ST models still suffer from distinct degradation in noisy environments and fail to translate visual speech (i.e., the movement of lips and teeth). In this work, we present AV-TranSpeech, the first audio-visual speech-to-speech (AV-S2ST) translation model without relying on intermediate text. AV-TranSpeech complements the audio stream with visual information to promote system robustness and opens up a host of practical applications: dictation or dubbing archival films. To mitigate the data scarcity with limited parallel AV-S2ST data, we 1) explore self-supervised pre-training with unlabeled audio-visual data to learn contextual representation, and 2) introduce cross-modal distillation with S2ST models trained on the audio-only corpus to further reduce the requirements of visual data. Experimental results on two language pairs demonstrate that AV-TranSpeech outperforms audio-only models under all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with baselines. Audio samples are available at https://AV-TranSpeech.github.io
【9】 Vistaar: Diverse Benchmarks and Training Sets for Indian Language ASR标题：Vistaar：印度语言ASR的不同基准和训练集链接：https://arxiv.org/abs/2305.15386
*与cs.SD语音【2】为同一篇
作者：Kaushal Santosh Bhogale,Sai Sundaresan,Abhigyan Raman,Tahir Javed,Mitesh M. Khapra,Pratyush Kumar机构：Indian Institute of Technology Madras, India, AI,Bharat, India, Microsoft, India备注：Accepted in INTERSPEECH 2023摘要：改进ASR系统是必要的，使新的基于LLM的用例可供全球各地的人们访问。在本文中，我们专注于印度语言，并提出了不同的基准需要评估和改善印度语言的ASR系统的情况。为了解决这个问题，我们整理了Vistaar作为一组59个基准，跨越各种语言和领域的组合，我们评估3个公开可用的ASR系统和2个商业系统。我们还通过在12种印度语言的公开训练数据集上微调Whisper模型来训练IndicWhisper模型，总计10.7K小时。我们表明，IndicWhisper显着改善考虑ASR系统的Vistaar基准。事实上，IndicWhisper在59个基准中的39个中具有最低的WER，平均减少4.1 WER。我们开源所有数据集、代码和模型。摘要：Improving ASR systems is necessary to make new LLM-based use-cases accessible to people across the globe. In this paper, we focus on Indian languages, and make the case that diverse benchmarks are required to evaluate and improve ASR systems for Indian languages. To address this, we collate Vistaar as a set of 59 benchmarks across various language and domain combinations, on which we evaluate 3 publicly available ASR systems and 2 commercial systems. We also train IndicWhisper models by fine-tuning the Whisper models on publicly available training datasets across 12 Indian languages totalling to 10.7K hours. We show that IndicWhisper significantly improves on considered ASR systems on the Vistaar benchmark. Indeed, IndicWhisper has the lowest WER in 39 out of the 59 benchmarks, with an average reduction of 4.1 WER. We open-source all datasets, code and models.
【10】 LMs with a Voice: Spoken Language Modeling beyond Speech Tokens标题：有声音的LMS：超越言语符号的口语建模链接：https://arxiv.org/abs/2305.15255
*与cs.SD语音【3】为同一篇
作者：Eliya Nachmani,Alon Levkovitch,Julian Salazar,Chulayutsh Asawaroengchai,Soroosh Mariooryad,RJ Skerry-Ryan,Michelle Tadmor Ramanovich机构：Google Research摘要：我们提出了SPECTRON，一种新的方法来适应预训练的语言模型（LM）执行语音连续。通过利用预先训练的语音编码器，我们的模型生成文本和语音输出，整个系统直接在频谱图上进行端到端的训练。在频谱图域中训练整个模型简化了我们的语音连续系统，而现有的级联方法使用离散语音表示。我们进一步表明，我们的方法超越现有的口语模型在语义内容和扬声器保存，同时也受益于从预先存在的模型转移的知识。音频样本可以在我们的网站www.example.com上找到https://michelleramanovich.github.io/spectron/spectron摘要：We present SPECTRON, a novel approach to adapting pre-trained language models (LMs) to perform speech continuation. By leveraging pre-trained speech encoders, our model generates both text and speech outputs with the entire system being trained end-to-end operating directly on spectrograms. Training the entire model in the spectrogram domain simplifies our speech continuation system versus existing cascade methods which use discrete speech representations. We further show our method surpasses existing spoken language models both in semantic content and speaker preservation while also benefiting from the knowledge transferred from pre-existing models. Audio samples can be found in our website https://michelleramanovich.github.io/spectron/spectron
【11】 PLCMOS -- a data-driven non-intrusive metric for the evaluation of  packet loss concealment algorithms标题：PLCMOS--一种用于评估分组丢失隐藏算法的数据驱动的非侵入式度量链接：https://arxiv.org/abs/2305.15127
*与cs.SD语音【4】为同一篇
作者：Lorenz Diener,Marju Purin,Sten Sootla,Ando Saabas,Robert Aichner,Ross Cutler机构：Microsoft Corporation备注：to appear: INTERSPEECH 2023, associated model release: this https URL摘要：语音质量评估是每一个研究产生或处理语音的模型的研究人员所面临的问题。人的主观评级，在语音质量评估的黄金标准，是昂贵的和耗时的，以获得足够的数量，以获得可靠的数据，而自动化的客观指标显示出低相关性与黄金标准评级。本文介绍了PLCMOS，一个非侵入性的数据驱动的工具，用于生成一个强大的，准确的估计的平均意见评分的人类评分员会分配一个音频文件，已被处理，通过传输在一个退化的分组交换网络与丢失的数据包被愈合的数据包丢失隐藏算法。我们的新模型显示了模型明智的皮尔逊相关性为~0.97和等级相关性为~0.95与人类评级，基本上高于所有其他可用的侵入性和非侵入性指标。该模型作为ONNX模型发布，供其他研究人员在构建PLC系统时使用。摘要：Speech quality assessment is a problem for every researcher working on models that produce or process speech. Human subjective ratings, the gold standard in speech quality assessment, are expensive and time-consuming to acquire in a quantity that is sufficient to get reliable data, while automated objective metrics show a low correlation with gold standard ratings. This paper presents PLCMOS, a non-intrusive data-driven tool for generating a robust, accurate estimate of the mean opinion score a human rater would assign an audio file that has been processed by being transmitted over a degraded packet-switched network with missing packets being healed by a packet loss concealment algorithm. Our new model shows a model-wise Pearson's correlation of ~0.97 and rank correlation of ~0.95 with human ratings, substantially above all other available intrusive and non-intrusive metrics. The model is released as an ONNX model for other researchers to use when building PLC systems.
【12】 Iteratively Improving Speech Recognition and Voice Conversion标题：反复改进语音识别和语音转换链接：https://arxiv.org/abs/2305.15055
*与cs.SD语音【5】为同一篇
作者：Mayank Kumar Singh,Naoya Takahashi,Onoe Naoyuki机构：Sony Research India, India,Sony Group Corporation, Japan摘要：语音转换（VC）任务的许多现有工作使用自动语音识别（ASR）模型来确保源样本和转换样本之间的语言一致性。然而，对于低数据资源域，训练高质量的ASR仍然是一项具有挑战性的任务。在这项工作中，我们提出了一种新的迭代的方式来改善ASR和VC模型。我们首先训练一个ASR模型，该模型用于在训练VC模型时确保内容保留。在下一次迭代中，VC模型被用作数据增强方法，以进一步微调ASR模型并将其推广到不同的扬声器。通过迭代地利用改进的ASR模型来训练VC模型，反之亦然，我们在实验中显示了这两个模型的改进。我们提出的框架优于ASR和一杆VC基线模型在英语唱歌和印地语语音域的主观和客观评价，在低数据资源设置。摘要：Many existing works on voice conversion (VC) tasks use automatic speech recognition (ASR) models for ensuring linguistic consistency between source and converted samples. However, for the low-data resource domains, training a high-quality ASR remains to be a challenging task. In this work, we propose a novel iterative way of improving both the ASR and VC models. We first train an ASR model which is used to ensure content preservation while training a VC model. In the next iteration, the VC model is used as a data augmentation method to further fine-tune the ASR model and generalize it to diverse speakers. By iteratively leveraging the improved ASR model to train VC model and vice-versa, we experimentally show improvement in both the models. Our proposed framework outperforms the ASR and one-shot VC baseline models on English singing and Hindi speech domains in subjective and objective evaluations in low-data resource settings.
【13】 LoopBoxes -- Evaluation of a Collaborative Accessible Digital Musical  Instrument标题：LoopBox--协同可访问数字乐器的评测链接：https://arxiv.org/abs/2305.14875
*与cs.SD语音【6】为同一篇
作者：Andreas Förster,Alarith Uhde,Mathias Komesker,Christina Komesker,Irina Schmidt机构：Technische Universität Berlin, University of Furtwangen, University of Siegen, Siegen, Germany, imui e.V., Cologne, Germany备注：10 pages, 9 figures, to be published in the Proceedings of the International Conference on New Interfaces for Musical Expression (NIME'23)摘要：LoopBoxes是一种无障碍数字乐器，旨在为有特殊教育需求（SEN）的儿童创建直观的循环音乐制作。本文介绍了在德国柏林的音乐节期间的试点研究，以及与儿童和音乐教师在特殊教育学校设置的案例研究的形式的仪器的评价。我们创建了一个模块化的系统，由三个模块组成，提供单个用户以及协作音乐制作。试点研究进行了评估，使用非正式的观察和问卷调查（n = 39），并表示，该仪器提供音乐制作的人，没有事先的音乐知识，在所有年龄组和促进合作的音乐过程。个案研究是基于观察和定性访谈。它确认，该文书满足了学校环境的需要，并指出未来版本如何扩大所有学生，特别是那些患有复杂残疾的学生。此外，开箱即用的功能似乎是至关重要的长期实施的文书在学校设置。摘要：LoopBoxes is an accessible digital musical instrument designed to create an intuitive access to loop based music making for children with special educational needs (SEN). This paper describes the evaluation of the instrument in the form of a pilot study during a music festival in Berlin, Germany, as well as a case study with children and music teachers in a SEN school setting. We created a modular system composed of three modules that afford single user as well as collaborative music making. The pilot study was evaluated using informal observation and questionnaires (n = 39), and indicated that the instrument affords music making for people with and without prior musical knowledge across all age groups and fosters collaborative musical processes. The case study was based on observation and a qualitative interview. It confirmed that the instrument meets the needs of the school settings and indicated how future versions could expand access to all students, especially those experiencing complex disabilities. In addition, out-of-the-box functionality seems to be crucial for the long-term implementation of the instrument in a school setting.
【14】 Interactive Neural Resonators标题：交互式神经谐振器链接：https://arxiv.org/abs/2305.14867
*与cs.SD语音【7】为同一篇
作者：Rodrigo Diaz,Charalampos Saitis,Mark Sandler机构：Centre for Digital Music, Queen Mary University of, London摘要：在这项工作中，我们提出了一种使用神经共振器的实时接触声音的可控合成方法。以前的工作已经使用物理启发的统计方法和物理建模的对象材料和激励信号。我们的方法采用了可微二阶谐振器，并估计其系数使用的神经网络，是条件的物理参数。这允许交互式动态控制和以直观的方式生成新颖的声音。我们展示了我们的方法的实际实施，并探索其潜在的创造性应用。摘要：In this work, we propose a method for the controllable synthesis of real-time contact sounds using neural resonators. Previous works have used physically inspired statistical methods and physical modelling for object materials and excitation signals. Our method incorporates differentiable second-order resonators and estimates their coefficients using a neural network that is conditioned on physical parameters. This allows for interactive dynamic control and the generation of novel sounds in an intuitive manner. We demonstrate the practical implementation of our method and explore its potential creative applications.
【15】 ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text  Translation标题：ComSL：一种端到端语音到文本翻译的复合语音语言模型链接：https://arxiv.org/abs/2305.14838
*与cs.SD语音【8】为同一篇
作者：Chenyang Le,Yao Qian,Long Zhou,Shujie Liu,Michael Zeng,Xuedong Huang机构：Shanghai Jiao Tong University, Shanghai, China, Microsoft Cloud and AI, Redmond, WA, US, Microsoft Research Asia, Beijing, China, Yanmin Qian摘要：由于对训练数据和GPU消耗的大量需求，以及语音和语言之间的模态差距，联合语音-语言训练具有挑战性。我们提出了ComSL，一个语音语言模型建立在公共预训练的语音和语言模型的复合架构之上，并优化数据效率的口语任务。特别是，我们建议将跨模态学习纳入迁移学习，并同时进行下游任务的多任务学习方式。我们的方法已经证明了在端到端语音到文本翻译任务的有效性，实现了一个新的国家的最先进的平均BLEU分数31.5的多语言语音到英语文本翻译任务的21种语言，作为衡量公共CoVoST 2评估集。摘要：Joint speech-language training is challenging due to the large demand for training data and GPU consumption, as well as the modality gap between speech and language. We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models and optimized data-efficiently for spoken language tasks. Particularly, we propose to incorporate cross-modality learning into transfer learning and conduct them simultaneously for downstream tasks in a multi-task learning manner. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks, achieving a new state-of-the-art average BLEU score of 31.5 on the multilingual speech to English text translation task for 21 languages, as measured on the public CoVoST2 evaluation set.
【16】 CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation标题：CMOT：语音翻译中基于最优传输的跨模式混合链接：https://arxiv.org/abs/2305.14635
*与cs.SD语音【9】为同一篇
作者：Yan Zhou,Qingkai Fang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：ACL 2023 main conference摘要：端到端语音翻译是将源语言的语音信号翻译成目标语言的文本的任务。作为一个跨模态的任务，端到端的ST是难以训练有限的数据。现有的方法往往试图从机器翻译（MT）的知识转移，但他们的性能受到限制的语音和文本之间的模态差距。在本文中，我们提出了跨模态混合通过最佳运输CMOT，以克服模态差距。我们发现语音和文本序列之间的对齐，通过最佳的运输，然后混合的序列从不同的方式在令牌级别使用的对齐。在MuST-C ST基准测试上的实验表明，CMOT在8个翻译方向上的平均BLEU为30.0，优于以前的方法。进一步的分析表明，CMOT可以自适应地找到模态之间的对齐，这有助于缓解语音和文本之间的模态差距。代码可在www.example.com上公开获取https://github.com/ictnlp/CMOT。摘要：End-to-end speech translation (ST) is the task of translating speech signals in the source language into text in the target language. As a cross-modal task, end-to-end ST is difficult to train with limited data. Existing methods often try to transfer knowledge from machine translation (MT), but their performances are restricted by the modality gap between speech and text. In this paper, we propose Cross-modal Mixup via Optimal Transport CMOT to overcome the modality gap. We find the alignment between speech and text sequences via optimal transport and then mix up the sequences from different modalities at a token level using the alignment. Experiments on the MuST-C ST benchmark demonstrate that CMOT achieves an average BLEU of 30.0 in 8 translation directions, outperforming previous methods. Further analysis shows CMOT can adaptively find the alignment between modalities, which helps alleviate the modality gap between speech and text. Code is publicly available at https://github.com/ictnlp/CMOT.
【17】 Improving Speech Emotion Recognition Performance using Differentiable  Architecture Search标题：利用可区分结构搜索提高语音情感识别性能链接：https://arxiv.org/abs/2305.14402
*与cs.SD语音【10】为同一篇
作者：Thejan Rajapakshe,Rajib Rana,Sara Khalifa,Berrak Sisman,Björn Schuller机构：University of Southern Queensland, Australia, Distributed Sensing Systems Group, Data, - CSIRO, The University of Texas at Dallas, USA, GLAM, Imperial College London, UK摘要：语音情感识别（SER）是人机交互中情感感知通信的关键使能器。深度学习（DL）通过提高模型复杂度来提高SER模型的性能。然而，设计DL架构需要先前的经验和实验评估。令人鼓舞的是，神经架构搜索（NAS）允许自动搜索最佳DL模型。特别地，差分架构搜索（DARTS）是使用NAS来搜索优化模型的有效方法。在本文中，我们提出了DARTS用于联合CNN和LSTM架构，以提高SER性能。我们选择CNN LSTM耦合的灵感来自于显示类似模型提供改进性能的结果。虽然SER的研究人员已经分别考虑了CNN和RNN，但将DART联合用于CNN和LSTM的可行性仍然需要探索。IEMOCAP数据集的实验，我们证明了我们的方法优于最好的报告结果使用DARTS SER。摘要：Speech Emotion Recognition (SER) is a critical enabler of emotion-aware communication in human-computer interactions. Deep Learning (DL) has improved the performance of SER models by improving model complexity. However, designing DL architectures requires prior experience and experimental evaluations. Encouragingly, Neural Architecture Search (NAS) allows automatic search for an optimum DL model. In particular, Differentiable Architecture Search (DARTS) is an efficient method of using NAS to search for optimised models. In this paper, we propose DARTS for a joint CNN and LSTM architecture for improving SER performance. Our choice of the CNN LSTM coupling is inspired by results showing that similar models offer improved performance. While SER researchers have considered CNNs and RNNs separately, the viability of using DARTs jointly for CNN and LSTM still needs exploration. Experimenting with the IEMOCAP dataset, we demonstrate that our approach outperforms best-reported results using DARTS for SER.
【18】 Connecting Multi-modal Contrastive Representations标题：连接多模式对比表征链接：https://arxiv.org/abs/2305.14381
*与cs.SD语音【11】为同一篇
作者：Zehan Wang,Yang Zhao,Xize Cheng,Haifeng Huang,Jiageng Liu,Li Tang,Linjun Li,Yongqi Wang,Aoxiong Yin,Ziang Zhang,Zhou Zhao机构：Zhejiang University备注：Demos are available at \url{this https URL}摘要：多模态对比表征（MCR）学习旨在将不同的模态编码到语义对齐的共享空间中。该范例在各种模态的许多下游任务上显示出显着的泛化能力。然而，对大量高质量数据对的依赖限制了其在更多模式上的进一步发展。本文提出了一种新的训练有效的方法来学习MCR没有配对的数据称为连接多模态对比表示（C-MCR）。具体来说，给定在（A，B）和（B，C）模态对上预训练的两个现有MCR，我们将它们投影到新空间，并使用来自重叠模态B的数据来在新空间中对齐两个MCR。同时，由于模态对（A，B）和（B，C）已经在每个MCR内对齐，所以通过重叠模态学习的连接也可以被转移到非重叠模态对（A，C）。为了释放C-MCR的潜力，我们进一步引入了一个语义增强的内部和内部MCR连接方法。我们首先提高了语义的一致性和完成不同的方式更强大的对齐嵌入。然后，我们利用MCR间比对来建立连接，并采用MCR内比对来更好地维持来自非重叠模态的输入的连接。我们以视听对比学习领域为例，证明了C-MCR的有效性。我们通过文本连接预训练的CLIP和CLAP模型，以获得视听对比表示。值得注意的是，在不使用任何配对的视听数据和进一步调整的情况下，C-MCR在三个视听下游任务的六个数据集上实现了最先进的性能。摘要：Multi-modal Contrastive Representation (MCR) learning aims to encode different modalities into a semantically aligned shared space. This paradigm shows remarkable generalization ability on numerous downstream tasks across various modalities. However, the reliance on massive high-quality data pairs limits its further development on more modalities. This paper proposes a novel training-efficient method for learning MCR without paired data called Connecting Multi-modal Contrastive Representations (C-MCR). Specifically, given two existing MCRs pre-trained on (A, B) and (B, C) modality pairs, we project them to a new space and use the data from the overlapping modality B to aligning the two MCRs in the new space. Meanwhile, since the modality pairs (A, B) and (B, C) are already aligned within each MCR, the connection learned by overlapping modality can also be transferred to non-overlapping modality pair (A, C). To unleash the potential of C-MCR, we further introduce a semantic-enhanced inter- and intra-MCR connection method. We first enhance the semantic consistency and completion of embeddings across different modalities for more robust alignment. Then we utilize the inter-MCR alignment to establish the connection, and employ the intra-MCR alignment to better maintain the connection for inputs from non-overlapping modalities. We take the field of audio-visual contrastive learning as an example to demonstrate the effectiveness of C-MCR. We connect pre-trained CLIP and CLAP models via texts to derive audio-visual contrastive representations. Remarkably, without using any paired audio-visual data and further tuning, C-MCR achieves state-of-the-art performance on six datasets across three audio-visual downstream tasks.
【19】 Zero-shot personalized lip-to-speech synthesis with face image based  voice control标题：基于人脸图像语音控制的Zero-Shot个性化唇语合成链接：https://arxiv.org/abs/2305.14359
*与cs.SD语音【12】为同一篇
作者：Zheng-Yan Sheng,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, P.R.China备注：ICASSP 2023摘要：唇语音合成（Lip2Speech），预测相应的语音从说话的人脸图像，已经见证了显着的进展，各种模型和训练策略在一系列的独立研究。然而，现有的研究不能实现语音控制下的zero-shot条件下，因为额外的发言人嵌入需要提取的自然参考语音和不可用时，只有一个看不见的发言人的无声视频。在本文中，我们提出了一个zero-shot个性化Lip2Speech合成方法，在人脸图像控制扬声器的身份。采用变分自动编码器来解开说话人身份和语言内容的表示，这使得说话人嵌入，以控制合成语音的语音特性，为看不见的说话人。此外，我们提出了相关的跨模态表示学习，以提高基于人脸的说话人嵌入（FSE）的语音控制能力。大量的实验验证了该方法的有效性，其合成的话语是更自然和匹配的个性输入视频比比较方法。据我们所知，本文首次尝试zero-shot个性化Lip2Speech合成与人脸图像，而不是参考音频来控制语音特征。摘要：Lip-to-Speech (Lip2Speech) synthesis, which predicts corresponding speech from talking face images, has witnessed significant progress with various models and training strategies in a series of independent studies. However, existing studies can not achieve voice control under zero-shot condition, because extra speaker embeddings need to be extracted from natural reference speech and are unavailable when only the silent video of an unseen speaker is given. In this paper, we propose a zero-shot personalized Lip2Speech synthesis method, in which face images control speaker identities. A variational autoencoder is adopted to disentangle the speaker identity and linguistic content representations, which enables speaker embeddings to control the voice characteristics of synthetic speech for unseen speakers. Furthermore, we propose associated cross-modal representation learning to promote the ability of face-based speaker embeddings (FSE) on voice control. Extensive experiments verify the effectiveness of the proposed method whose synthetic utterances are more natural and matching with the personality of input video than the compared methods. To our best knowledge, this paper makes the first attempt on zero-shot personalized Lip2Speech synthesis with a face image rather than reference audio to control voice characteristics.
机器翻译，仅供参考