语音/音频处理学术速递[2.15]

今日论文合集：cs.SD语音3篇，eess.AS音频处理3篇。

cs.SD语音

【1】 Synthesizing audio from tongue motion during speech using tagged MRI via transformer

标题：基于标记磁共振的语音语音合成技术

链接：https://arxiv.org/abs/2302.07203

作者：Xiaofeng Liu,Fangxu Xing,Jerry L. Prince,Maureen Stone,Georges El Fakhri,Jonghye Woo

机构：Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical, School, Boston, MA , USA, Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore

备注：SPIE Medical Imaging: Deep Dive Oral

摘要：研究标记MRI测量的舌内部组织点运动和口咽肌肉变形与可理解语音之间的关系有助于推进语音运动控制理论和开发语音相关障碍的新治疗方法。然而，阐明这两个信息源之间的关系是具有挑战性的，部分地由于时空运动场之间的数据结构的差异（即，4D运动场）和一维音频波形。在这项工作中，我们提出了一个高效的编码器-解码器转换网络，用于通过2D频谱图作为音频数据的替代来探索4D运动场中固有的预测信息。具体地，我们的编码器基于3D卷积空间建模和基于变换器的时间建模。所提取的特征由非对称2D卷积解码器处理以生成对应于4D运动场的谱图。此外，我们将生成对抗训练方法纳入我们的框架中，以进一步提高我们生成的谱图的合成质量。我们在63个成对的运动场序列和语音波形上进行了实验，证明了我们的框架能够从运动场序列生成清晰的音频波形。因此，我们的框架有可能提高我们对这两种模式之间关系的理解，并为言语障碍治疗的发展提供信息。

摘要：Investigating the relationship between internal tissue point motion of the tongue and oropharyngeal muscle deformation measured from tagged MRI and intelligible speech can aid in advancing speech motor control theories and developing novel treatment methods for speech related-disorders. However, elucidating the relationship between these two sources of information is challenging, due in part to the disparity in data structure between spatiotemporal motion fields (i.e., 4D motion fields) and one-dimensional audio waveforms. In this work, we present an efficient encoder-decoder translation network for exploring the predictive information inherent in 4D motion fields via 2D spectrograms as a surrogate of the audio data. Specifically, our encoder is based on 3D convolutional spatial modeling and transformer-based temporal modeling. The extracted features are processed by an asymmetric 2D convolution decoder to generate spectrograms that correspond to 4D motion fields. Furthermore, we incorporate a generative adversarial training approach into our framework to further improve synthesis quality on our generated spectrograms. We experiment on 63 paired motion field sequences and speech waveforms, demonstrating that our framework enables the generation of clear audio waveforms from a sequence of motion fields. Thus, our framework has the potential to improve our understanding of the relationship between these two modalities and inform the development of treatments for speech disorders.

【2】 Multi-Source Contrastive Learning from Musical Audio

标题：从音乐音频中进行多源对比学习

链接：https://arxiv.org/abs/2302.07077

作者：Christos Garoufis,Athanasia Zlatintsi,Petros Maragos

机构：School of ECE, National Technical Univ. of Athens, Greece, Institute of Language and Speech Proc., Athena Research and Innovation Center, Maroussi, Greece, Robotics Unit, Athena Research and Innovation Center, Maroussi, Greece

备注：8 pages, 5 figures, 3 tables. (Slightly edited) submission at SMC23

摘要：对比学习是自监督学习的一个新兴分支，它通过学习潜在空间来利用大量未标记数据，在潜在空间中，同一样本的不同视图对是关联的。在本文中，我们提出了音乐源联想作为对比音乐表征学习背景下的一种配对生成策略。为此，我们修改了COLA，一个广泛使用的对比学习音频框架，以学习将歌曲摘录与随机选择和自动提取的声乐或乐器源相关联。我们进一步引入了一个新的修改对比损失纳入信息的存在或不存在的具体来源。我们使用公开的Magna-Tag-A-Tune（MTAT）作为源数据集，在三个不同的下游任务（音乐自动标注、乐器分类和音乐流派分类）中进行了实验评估，产生了与现有文献方法竞争的结果，以及更快的网络收敛。该结果还示出了该预训练方法可以根据所选择的音乐源而被导向特定特征，同时还取决于所分离的源的质量。

摘要：Contrastive learning constitutes an emerging branch of self-supervised learning that leverages large amounts of unlabeled data, by learning a latent space, where pairs of different views of the same sample are associated. In this paper, we propose musical source association as a pair generation strategy in the context of contrastive music representation learning. To this end, we modify COLA, a widely used contrastive learning audio framework, to learn to associate a song excerpt with a stochastically selected and automatically extracted vocal or instrumental source. We further introduce a novel modification to the contrastive loss to incorporate information about the existence or absence of specific sources. Our experimental evaluation in three different downstream tasks (music auto-tagging, instrument classification and music genre classification) using the publicly available Magna-Tag-A-Tune (MTAT) as a source dataset yields competitive results to existing literature methods, as well as faster network convergence. The results also show that this pre-training method can be steered towards specific features, according to the selected musical source, while also being dependent on the quality of the separated sources.

【3】 Speaker-Independent Acoustic-to-Articulatory Speech Inversion

标题：不依赖说话人的声学到节律语音倒置

链接：https://arxiv.org/abs/2302.06774

作者：Peter Wu,Li-Wei Chen,Cheol Jun Cho,Shinji Watanabe,Louis Goldstein,Alan W Black,Gopala K. Anumanchipalli

机构：University of California, Berkeley,Carnegie Mellon University,University of Southern California

摘要：为了建立像人类一样自然地处理语音的语音处理方法，研究人员探索了多种方法来建立从语音到可解释空间的可逆映射。发音空间是一个很有希望的反转目标，因为这个空间捕捉到了语音产生的机制。为此，我们建立了一个声学到发音倒置（AAI）模型，利用自回归、对抗训练和自我监督来推广到看不见的说话人。我们的方法在电磁关节造影（EMA）数据集上获得了0.784的相关性，将最新技术水平提高了12.5%。此外，我们通过直接比较估计的表征与言语产生行为来显示这些表征的可解释性。最后，我们提出了一个基于再合成的不依赖发音标签的AAI评价指标，并在18人的数据集上验证了其有效性。

摘要：To build speech processing methods that can handle speech as naturally as humans, researchers have explored multiple ways of building an invertible mapping from speech to an interpretable space. The articulatory space is a promising inversion target, since this space captures the mechanics of speech production. To this end, we build an acoustic-to-articulatory inversion (AAI) model that leverages autoregression, adversarial training, and self supervision to generalize to unseen speakers. Our approach obtains 0.784 correlation on an electromagnetic articulography (EMA) dataset, improving the state-of-the-art by 12.5%. Additionally, we show the interpretability of these representations through directly comparing the behavior of estimated representations with speech production behavior. Finally, we propose a resynthesis-based AAI evaluation metric that does not rely on articulatory labels, demonstrating its efficacy with an 18-speaker dataset.

eess.AS音频处理