语音/音频处理学术速递[3.9]

今日论文合集：cs.SD语音5篇，eess.AS音频处理5篇。

cs.SD语音

【1】 New Audio Representations Image Gan Generation from BriVL

标题：从BriVL生成新的音频表示图像GAN

链接：http://arxiv.org/pdf/2303.03267v1

作者：Sen Fang,Yangjian Wu,Bowen Gao,Teik Toe Teoh

机构：Victoria University,Hainan University,Nanyang Technological University

备注：ICASSP Workshop submission

摘要：最近，研究人员逐渐意识到，在某些情况下，对大规模互联网数据的自监督预训练优于高质量人工标注的数据集，多模态大模型优于单一或双模态小模型。本文提出了一种基于视觉与语言桥梁（Bridging-Vision-and-Language，BriVL）的鲁棒音频表示学习方法WavBriVL。WavBriVL将音频、图像和文本投影到共享的嵌入式空间中，从而实现多模态应用。我们演示了从WavBriVL作为共享嵌入空间生成的图像的定性评估，本文的主要目的是：（1）学习音频和图像之间的相关性;（2）探索新的图像生成方式，即利用音频生成图片。实验结果表明，该方法能有效地从音频中生成合适的图像。

摘要：Recently, researchers have gradually realized that in some cases, the self-supervised pre-training on large-scale Internet data is better than that of high-quality manually labeled data sets, and multimodal large models are better than single or bimodal small models. In this paper, we propose a robust audio representation learning method WavBriVL based on Bridging-Vision-and-Language (BriVL). WavBriVL projects audio, image and text into a shared embedded space, so that multi-modal applications can be realized. We demonstrate the qualitative evaluation of the image generated from WavBriVL as a shared embedded space, with the main purposes of this paper: (1) Learning the correlation between audio and image; (2) Explore a new way of image generation, that is, use audio to generate pictures. Experimental results show that this method can effectively generate appropriate images from audio.

【2】 Onsets and Velocities: Affordable Real-Time Piano Transcription Using Convolutional Neural Networks

标题：音程和速度：使用卷积神经网络进行负担得起的实时钢琴转录

链接：http://arxiv.org/pdf/2303.03183v1

作者：Andres Fernandez

机构：University of T¨ubingen and IMPRS-IS, T¨ubingen, Germany

备注：Work supported by the Institut d'Estudis Balearics, Balearic Islands

摘要：复调钢琴转录最近经历了巨大的进步，这得益于复杂的深度学习设置的应用以及音符开始、偏移、速度和踏板检测等新子任务的引入。在这项工作中，我们集中在开始和速度检测，提出了一个卷积神经网络，大大减少了规模（3.1M个参数）和简单的推理方案，可在MAESTRO数据集上实现最先进的发作检测性能（F1=96.78%），并为开始+速度（F1=94.50%）设定了良好的新基线，同时在适度的商用硬件上保持了实时能力。此外，我们提出的ONSETS&VELOCITIES（O&V）模型表明，24ms的时间分辨率具有竞争力，与最近的趋势相反。我们提供了开源软件来重现我们的结果和一个预先训练好的模型的实时演示。

摘要：Polyphonic Piano Transcription has recently experienced substantial progress driven by the application of sophisticated Deep Learning setups and the introduction of new subtasks such as note onset, offset, velocity and pedal detection. In this work, we focus on onset and velocity detection, presenting a convolutional neural network with substantially reduced size (3.1M parameters) and a simple inference scheme that achieves state-of-the-art performance on the MAESTRO dataset for onset detection (F1=96.78%) and sets a good novel baseline for onset+velocity (F1=94.50%), while maintaining real-time capabilities on modest commodity hardware. Furthermore, our proposed ONSETS&VELOCITIES (O&V) model shows that a time resolution of 24ms is competitive, countering recent trends. We provide open-source software to reproduce our results and a real-time demo with a pretrained model.

【3】 A Light Weight Model for Active Speaker Detection

标题：一种用于主动说话人检测的轻量级模型

链接：http://arxiv.org/pdf/2303.03144v1

作者：Junhua Liao,Haihan Duan,Kanghui Feng,Wanbing Zhao,Yanbing Yang,Liangyin Chen

机构：College of Computer Science, Sichuan University, Chengdu, China., The Chinese University of Hong Kong, Shenzhen, China., The Institute for Industrial Internet Research, Sichuan University, Chengdu, China.

备注：Accepted by CVPR 2023

摘要：主动说话人检测是音视频场景理解中的一项挑战性任务，其目的是检测出一个或多个说话人场景中的说话人。由于它在说话人识别、说话人跟踪、视频自动编辑等应用中的重要性，因此受到了广泛的关注。已有的研究试图通过输入多个候选信息和设计复杂的模型来提高绩效。虽然这些方法取得了很好的性能，但是它们的高内存消耗和计算能力使得它们难以应用于资源受限的场景。因此，我们通过减少输入候选项、拆分2D和3D卷积进行音视频特征提取以及使用计算复杂度低的门控递归单元（GRU）进行跨模态建模，构建了一种轻量级的主动说话人检测架构。在AVA-ActiveSpeaker数据集上的实验结果表明，该框架的mAP性能（94.1% vs. 94.2%）具有竞争力，而资源开销显著低于现有方法，尤其是在模型参数（1.0M vs. 22.5M，约23倍）和FLOPs（0.6G vs. 2.6G，约4倍）方面。此外，该框架在Columbia数据集上也表现出良好的鲁棒性.代码和型号重量可从www.example.com获取。https: github.com Junhua-Liao Light-ASD.

摘要：Active speaker detection is a challenging task in audio-visual scenario understanding, which aims to detect who is speaking in one or more speakers scenarios. This task has received extensive attention as it is crucial in applications such as speaker diarization, speaker tracking, and automatic video editing. The existing studies try to improve performance by inputting multiple candidate information and designing complex models. Although these methods achieved outstanding performance, their high consumption of memory and computational power make them difficult to be applied in resource-limited scenarios. Therefore, we construct a lightweight active speaker detection architecture by reducing input candidates, splitting 2D and 3D convolutions for audio-visual feature extraction, and applying gated recurrent unit (GRU) with low computational complexity for cross-modal modeling. Experimental results on the AVA-ActiveSpeaker dataset show that our framework achieves competitive mAP performance (94.1% vs. 94.2%), while the resource costs are significantly lower than the state-of-the-art method, especially in model parameters (1.0M vs. 22.5M, about 23x) and FLOPs (0.6G vs. 2.6G, about 4x). In addition, our framework also performs well on the Columbia dataset showing good robustness. The code and model weights are available at https: github.com Junhua-Liao Light-ASD.

【4】 Do Prosody Transfer Models Transfer Prosody?

标题：韵律迁移模型会迁移韵律吗？

链接：http://arxiv.org/pdf/2303.03005v1

作者：Atli Thor Sigurgeirsson,Simon King

机构：The Centre for Speech Technology Research, University of Edinburgh, UK

备注：Accepted in ICASSP 2023, 5 pages, 2 figures, 3 tables

摘要：近年来的一些文本语音合成模型旨在将参考语音的韵律转换为生成的目标合成语音。这是通过使用参考话语的学习嵌入来完成的，参考话语用于调节语音生成。在训练期间，参考发音与目标发音相同。然而，在合成期间，这些模型经常被用于从与被合成的文本或说话者不同的参考传递韵律。为了解决这种不一致性，我们建议在训练过程中也使用不同的，但韵律相关的话语。我们相信这会鼓励模型学会只传递引用和目标共有的那些特征。如果韵律迁移方法确实能够迁移韵律，那么它们应该能够按照我们提出的方法进行训练。然而，结果表明，在这些条件下训练的模型比使用目标发音作为参考训练的模型表现得差得多。为了解释这一点，我们假设韵律迁移模型并不学习韵律的可迁移表征，而是学习高度依赖于参考说话人和参考文本的话语级表征。

摘要：Some recent models for Text-to-Speech synthesis aim to transfer the prosody of a reference utterance to the generated target synthetic speech. This is done by using a learned embedding of the reference utterance, which is used to condition speech generation. During training, the reference utterance is identical to the target utterance. Yet, during synthesis, these models are often used to transfer prosody from a reference that differs from the text or speaker being synthesized. To address this inconsistency, we propose to use a different, but prosodically-related, utterance during training too. We believe this should encourage the model to learn to transfer only those characteristics that the reference and target have in common. If prosody transfer methods do indeed transfer prosody they should be able to be trained in the way we propose. However, results show that a model trained under these conditions performs significantly worse than one trained using the target utterance as a reference. To explain this, we hypothesize that prosody transfer models do not learn a transferable representation of prosody, but rather an utterance-level representation which is highly dependent on both the reference speaker and reference text.

【5】 Self-supervised speech representation learning for keyword-spotting with light-weight transformers

标题：用于轻型Transformer关键词识别的自监督语音表征学习

链接：http://arxiv.org/pdf/2303.02673v1

作者：Chenyang Gao,Yue Gu,Francesco Caliva,Yuzong Liu

机构：⋆Rutgers, The State University of New Jersey, †Alexa Perceptual Technologies, Amazon

摘要：自监督语音表征学习（S3RL）正在彻底改变我们利用不断增长的可用数据的方式。虽然S3RL相关研究通常使用大型模型，但我们采用轻量级网络来满足计算受限设备的内存紧张。通过使用具有330k个参数的Transformers，我们证明了S3RL在关键词识别（KS）问题上的有效性，并提出了一种增强话语区分的机制，这被证明是提高分类任务性能的关键。在Google语音命令v2数据集上，与从头开始训练相比，应用于自回归预测编码S3RL的所提出的方法导致1.2%的准确度提高。在具有四个不同关键词的内部KS数据集上，它在固定错误拒绝率下提供了6%至23.7%的相对错误接受改善。我们认为这证明了S3RL方法对KS的轻量级模型的适用性，并证实了S3RL是资源受限应用中传统监督学习的有力替代。

摘要：Self-supervised speech representation learning (S3RL) is revolutionizing the way we leverage the ever-growing availability of data. While S3RL related studies typically use large models, we employ light-weight networks to comply with tight memory of compute-constrained devices. We demonstrate the effectiveness of S3RL on a keyword-spotting (KS) problem by using transformers with 330k parameters and propose a mechanism to enhance utterance-wise distinction, which proves crucial for improving performance on classification tasks. On the Google speech commands v2 dataset, the proposed method applied to the Auto-Regressive Predictive Coding S3RL led to a 1.2% accuracy improvement compared to training from scratch. On an in-house KS dataset with four different keywords, it provided 6% to 23.7% relative false accept improvement at fixed false reject rate. We argue this demonstrates the applicability of S3RL approaches to light-weight models for KS and confirms S3RL is a powerful alternative to traditional supervised learning for resource-constrained applications.

eess.AS音频处理