【1】 SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale
标题：SpeechNet：工业规模的弱监督端到端语音识别
链接：https://arxiv.org/abs/2211.11740
作者：Raphael Tang,Karun Kumar,Gefei Yang,Akshat Pandey,Yajie Mao,Vladislav Belyaev,Madhuri Emmadi,Craig Murray,Ferhan Ture,Jimmy Lin机构：Comcast Applied AI, University of Waterloo备注：Accepted to EMNLP 2022 Industry Track; 9 pages, 7 figures摘要：端到端的自动语音识别系统代表了现有技术的状态，但是它们依赖于数千小时的人工注释语音来进行训练，以及用于推理的重量级计算。当然，这阻碍了商业化，因为大多数公司缺乏大量的人力和计算资源。在这篇文章中，我们探讨了在标签稀缺、计算受限的环境中训练和部署ASR系统。为了减少人工劳动，我们使用第三方ASR系统作为弱监督源，并辅以从隐式用户反馈中导出的标记功能。为了加速推理，我们建议在一个不同输入长度的CUDA图池中路由生产时查询，这些CUDA图的分布与流量的分布最匹配。与我们的第三方ASR相比，我们实现了8%的字错误率和600%的加速比。我们的系统，称为SpeechNet，目前每天在我们的语音智能电视上服务1200万个查询。据我们所知，这是学术文献中首次描述基于Wav2vec的大规模部署。摘要：End-to-end automatic speech recognition systems represent the state of the art, but they rely on thousands of hours of manually annotated speech for training, as well as heavyweight computation for inference. Of course, this impedes commercialization since most companies lack vast human and computational resources. In this paper, we explore training and deploying an ASR system in the label-scarce, compute-limited setting. To reduce human labor, we use a third-party ASR system as a weak supervision source, supplemented with labeling functions derived from implicit user feedback. To accelerate inference, we propose to route production-time queries across a pool of CUDA graphs of varying input lengths, the distribution of which best matches the traffic's. Compared to our third-party ASR, we achieve a relative improvement in word-error rate of 8% and a speedup of 600%. Our system, called SpeechNet, currently serves 12 million queries per day on our voice-enabled smart television. To our knowledge, this is the first time a large-scale, Wav2vec-based deployment has been described in the academic literature.
【2】 Continually learning new languages
标题：持续学习新语言
链接：https://arxiv.org/abs/2211.11703
作者：Ngoc-Quan Pham,Jan Niehues,Alexander Waibel机构：Interactive Systems Lab, Karlsruhe Institute of Technology, Karlsruhe, Germany, Carnegie Mellon University, Pittsburgh PA, USA备注：Submitted to ICCASP 2023摘要：多语言语音识别中，神经网络通常采用批学习的方法，即在训练前所有语言都是可用的。在之前的训练课程之后增加新语言的能力在经济上是有益的，但主要的挑战是灾难性的遗忘。在本文中，我们将权重分解、迁移学习和弹性权重合并的特性结合起来，以对抗灾难性遗忘，促进新语言的快速学习。在从最初的10种语言学习到27种语言的实验中，这种组合允许我们消除灾难性遗忘，同时仍然实现与同时学习所有语言相当的新语言的性能摘要：Multilingual speech recognition with neural networks is often implemented with batch-learning, when all of the languages are available before training. An ability to add new languages after the prior training sessions can be economically beneficial, but the main challenge is catastrophic forgetting. In this work, we combine the qualities of weight factorization, transfer learning and Elastic Weight Consolidation in order to counter catastrophic forgetting and facilitate learning new languages quickly. Such combination allowed us to eliminate catastrophic forgetting while still achieving performance for the new languages comparable with having all languages at once, in experiments of learning from an initial 10 languages to achieve 27 languages
【3】 Dialogs Re-enacted Across Languages
标题：跨语言重新上演对话
链接：https://arxiv.org/abs/2211.11584
作者：Nigel G. Ward,Jonathan E. Avila,Emilia Rivas机构：Department of Computer Science, University of Texas at El Paso, Summary摘要：为了支持跨语言韵律映射的机器学习和其他改进语音到语音翻译的方法，我们提出了一种用于收集跨语言的紧密匹配的话语对的协议，对所得到的数据收集进行了描述，并提出了一些观察和思考。这份报告的目的是1）使用语料库的人，2）扩展语料库的人，3）设计类似的双语对话数据集的人。摘要：To support machine learning of cross-language prosodic mappings and other ways to improve speech-to-speech translation, we present a protocol for collecting closely matched pairs of utterances across languages, a description of the resulting data collection, and some observations and musings. This report is intended for 1) people using the corpus, 2) people extending the corpus, and 3) people designing similar collections of bilingual dialog data.
【4】 A Dataset for Greek Traditional and Folk Music: Lyra
标题：希腊传统和民间音乐数据集：Lyra
链接：https://arxiv.org/abs/2211.11479
作者：Charilaos Papaioannou,Ioannis Valiantzas,Theodoros Giannakopoulos,Maximos Kaliakatsos-Papakostas,Alexandros Potamianos机构：School of ECE, National Technical University of Athens, Greece, Department of Music Studies, National and Kapodistrian University Of Athens, Greece, National Center for Scientific Research - Demokritos, Greece, Athena RC, Greece摘要：在MIR的范围内研究未被充分代表的音乐传统是至关重要的，这不仅是为了开发新颖的分析工具，而且是为了揭示音乐功能，这可能会证明对研究世界音乐有用。本文介绍了一个希腊传统和民间音乐的数据集，其中包括1570件作品，总计约80小时的数据。该数据集包含用于检索音频和视频的YouTube时间戳链接，以及有关仪器、地理和流派等方面的丰富元数据信息。这些内容是从一个希腊纪录片系列中收集的，该系列可以在网上获得，学者们在节目中通过现场音乐和舞蹈表演展示希腊的音乐传统，并讨论所展示音乐的社会、文化和音乐学方面。因此，这一过程产生了大量关于各种方面的描述，如音乐流派、起源地和乐器。此外，音频录制在录制设备方面是在严格的生产级规格下进行的，从而产生了非常干净和均匀的音频内容。在这项工作中，除了详细介绍数据集，我们提出了一个基线深度学习分类方法来识别所涉及的音乐属性。数据集、基线分类方法和模型在公共储存库中提供。还讨论了进一步完善数据集的未来方向。摘要：Studying under-represented music traditions under the MIR scope is crucial, not only for developing novel analysis tools, but also for unveiling musical functions that might prove useful in studying world musics. This paper presents a dataset for Greek Traditional and Folk music that includes 1570 pieces, summing in around 80 hours of data. The dataset incorporates YouTube timestamped links for retrieving audio and video, along with rich metadata information with regards to instrumentation, geography and genre, among others. The content has been collected from a Greek documentary series that is available online, where academics present music traditions of Greece with live music and dance performance during the show, along with discussions about social, cultural and musicological aspects of the presented music. Therefore, this procedure has resulted in a significant wealth of descriptions regarding a variety of aspects, such as musical genre, places of origin and musical instruments. In addition, the audio recordings were performed under strict production-level specifications, in terms of recording equipment, leading to very clean and homogeneous audio content. In this work, apart from presenting the dataset in detail, we propose a baseline deep-learning classification approach to recognize the involved musicological attributes. The dataset, the baseline classification methods and the models are provided in public repositories. Future directions for further refining the dataset are also discussed.
【5】 Sequentially Sampled Chunk Conformer or Streaming End-to-End ASR
标题：顺序采样的区块整合器或流的端到端ASR
链接：https://arxiv.org/abs/2211.11419
作者：Fangyuan Wang,Xiyuan Wang,Bo Xu机构：Institute of Automation, Chinese Academy of Sciences, School of Future Technology,School of Artificial Intelligence, University of Chinese Academy of Sciences备注：This paper has been submitted to ICASSP 2023摘要：本文深入研究了一种适用于流媒体端到端ASR的顺序采样块一致性算法SSC-Conformer。SSC-Conformer首先证明了在Conformer编码器中使用顺序采样的块式多头自注意（SSC-MHSA）的显著性能增益，通过允许高效的跨块交互同时保持线性复杂度。此外，本文还探索了利用块卷积来利用块未来上下文，并在卷积层中结合偶然卷积来进一步降低CER。我们在AISHELL-1基准测试上验证了所提出的SSC-Conformer，并且实验结果表明，在没有LM重评分的情况下，流传输E2 E ASR达到了最先进的性能，CER为5.33%。而且，由于其线性复杂度，SSC-Conformer可以用大批量大小训练，并且更有效地推断。摘要：This paper presents an in-depth study on a Sequentially Sampled Chunk Conformer, SSC-Conformer, for streaming End-to-End (E2E) ASR. The SSC-Conformer first demonstrates the significant performance gains from using the sequentially sampled chunk-wise multi-head self-attention (SSC-MHSA) in the Conformer encoder by allowing efficient cross-chunk interactions while keeping linear complexities. Furthermore, it explores taking advantage of chunked convolution to make use of the chunk-wise future context and integrates with casual convolution in the convolution layers to further reduce CER. We verify the proposed SSC-Conformer on the AISHELL-1 benchmark and experimental results show that a state-of-the-art performance for streaming E2E ASR is achieved with CER 5.33% without LM rescoring. And, owing to its linear complexity, the SSC-Conformer can train with large batch sizes and infer more efficiently.
【6】 LISA: Localized Image Stylization with Audio via Implicit Neural Representation
标题：LISA：基于隐式神经表示的局部音频图像风格化
链接：https://arxiv.org/abs/2211.11381
作者：Seung Hyun Lee,Chanyoung Kim,Wonmin Byeon,Sang Ho Yoon,Jinkyu Kim,Sangpil Kim机构：Department of Artificial Intelligence and ,CSE, Korea University, NVIDIA Research, NVIDIA Corporation, Graduate School of Culture Technology, KAIST, Source Image, Audio Source, Localization Map, Fire Crackling, Stylized Image, Splashing Water, Underwater Bubbling摘要：提出了一种基于音频的局部化图像风格化（Localized Image Stylization with Audio，LISA）框架，实现了音频驱动的局部化图像风格化。声音通常提供关于场景的特定上下文的信息，并且与场景或对象的特定部分密切相关。然而，现有的图像风格化工作集中于使用图像或文本输入来风格化整个图像。基于音频输入风格化图像的特定部分是自然的，但具有挑战性。在这项工作中，我们提出了一个框架，用户提供音频输入来定位输入图像中的声源，另一个用于局部风格化目标对象或场景。LISA首先利用CLIP嵌入空间生成具有视听定位网络的精细定位地图。然后，我们利用隐式神经表示（INR）以及预测的定位图来基于声音信息风格化目标对象或场景。所提出的INR可以操纵局部化的像素值以与所提供的音频输入在语义上一致。通过一系列的实验，我们证明了所提出的框架优于其他音频引导风格化方法。此外，LISA构建简洁的定位图，并根据给定的音频输入自然地操纵目标对象或场景。摘要：We present a novel framework, Localized Image Stylization with Audio (LISA) which performs audio-driven localized image stylization. Sound often provides information about the specific context of the scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. In this work, we propose a framework that a user provides an audio input to localize the sound source in the input image and another for locally stylizing the target object or scene. LISA first produces a delicate localization map with an audio-visual localization network by leveraging CLIP embedding space. We then utilize implicit neural representation (INR) along with the predicted localization map to stylize the target object or scene based on sound information. The proposed INR can manipulate the localized pixel values to be semantically consistent with the provided audio input. Through a series of experiments, we show that the proposed framework outperforms the other audio-guided stylization methods. Moreover, LISA constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.
【7】 Video Background Music Generation: Dataset, Method and Evaluation
标题：视频背景音乐生成：数据集、方法与评价
链接：https://arxiv.org/abs/2211.11248
作者：Le Zhuo,Zhaokai Wang,Baisen Wang,Yue Liao,Stanley Peng,Chenxi Bao,Miao Lu,Xiaobo Li,Si Liu机构：Beihang University, Edinburgh College of Art, The University of Edinburgh, Alibaba Group摘要：编辑视频时，音乐是必不可少的，但手动选择音乐既困难又耗时。因此，我们寻求在给定视频输入的情况下自动生成背景音乐轨道。这是一个具有挑战性的任务，因为它需要大量的配对视频和音乐来学习他们的对应关系。遗憾的是，没有这样的数据集。为了弥补这一差距，我们引入了一个数据集，基准模型，和评估指标的视频背景音乐生成。我们介绍SymMV，一个视频和符号音乐数据集，以及和弦、节奏、旋律和伴奏注释。据我们所知，它是第一个包含高质量符号音乐和详细注释的视频音乐数据集。本文还提出了一个视频背景音乐生成框架V-MusProd，该框架利用了和弦、旋律和伴奏的音乐先验知识，以及语义、颜色和运动特征的视频-音乐关系。为了解决视频-音乐对应性缺乏客观度量的问题，我们提出了一种基于检索的度量VMCP，该度量建立在一个强大的视频-音乐表示学习模型之上。实验结果表明，V-MusProd在音乐质量和视频一致性方面均优于现有的方法.我们相信我们的数据集、基准模型和评估指标将推动视频背景音乐生成的发展。摘要：Music is essential when editing videos, but selecting music manually is difficult and time-consuming. Thus, we seek to automatically generate background music tracks given video input. This is a challenging task since it requires plenty of paired videos and music to learn their correspondence. Unfortunately, there exist no such datasets. To close this gap, we introduce a dataset, benchmark model, and evaluation metric for video background music generation. We introduce SymMV, a video and symbolic music dataset, along with chord, rhythm, melody, and accompaniment annotations. To the best of our knowledge, it is the first video-music dataset with high-quality symbolic music and detailed annotations. We also propose a benchmark video background music generation framework named V-MusProd, which utilizes music priors of chords, melody, and accompaniment along with video-music relations of semantic, color, and motion features. To address the lack of objective metrics for video-music correspondence, we propose a retrieval-based metric VMCP built upon a powerful video-music representation learning model. Experiments show that with our dataset, V-MusProd outperforms the state-of-the-art method in both music quality and correspondence with videos. We believe our dataset, benchmark model, and evaluation metric will boost the development of video background music generation.
【8】 TimbreCLIP: Connecting Timbre to Text and Images
标题：TimbreCLIP：将音色连接到文本和图像
链接：https://arxiv.org/abs/2211.11225
作者：Nicolas Jonason,Bob L. T. Sturm机构：KTH Royal Institute of Technology备注：Submitted to AAAI workshop on creative AI across modalities摘要：我们介绍了正在进行的TimbreCLIP工作，这是一个在单乐器音符上训练的音频-文本跨模态嵌入。我们在synth补丁上用跨模态检索任务评估模型。最后，我们演示了TimbreCLIP在两个任务中的应用：文本驱动音频均衡和音色到图像生成。摘要：We present work in progress on TimbreCLIP, an audio-text cross modal embedding trained on single instrument notes. We evaluate the models with a cross-modal retrieval task on synth patches. Finally, we demonstrate the application of TimbreCLIP on two tasks: text-driven audio equalization and timbre to image generation.
【9】 Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task
标题：探索预先训练的检查点在文本到音乐生成任务中的有效性
链接：https://arxiv.org/abs/2211.11216
作者：Shangda Wu,Maosong Sun机构：† Department of Music AI and Information Technology, Central Conservatory of Music, ‡ Department of Computer Science and Technology, Tsinghua University备注：5 pages, 2 figures, 2 tables摘要：得益于大规模数据集和预先训练的模型，生成模型领域最近获得了巨大的发展势头。然而，大多数符号音乐的数据集都非常小，这潜在地限制了数据驱动的多模态模型的性能。该问题的直观解决方案是利用来自其他模态的预训练模型（例如，自然语言）以提高与符号音乐相关的多模态任务的性能。本文首先对从文本描述中生成完整的、语义一致的符号乐谱进行了研究，并探讨了使用公开可用的检查点（即，BERT、GPT-2和BART），用于文本到音乐生成任务中的自然语言处理。实验结果表明，使用预先训练好的检查点，在BLEU评分和编辑距离相似度方面有统计学意义的改善。我们分析了我们的模型的能力和局限性，以更好地理解语言-音乐模型的潜力。摘要：Benefiting from large-scale datasets and pre-trained models, the field of generative models has recently gained significant momentum. However, most datasets for symbolic music are very small, which potentially limits the performance of data-driven multimodal models. An intuitive solution to this problem is to leverage pre-trained models from other modalities (e.g., natural language) to improve the performance of symbolic music-related multimodal tasks. In this paper, we carry out the first study of generating complete and semantically consistent symbolic music scores from text descriptions, and explore the efficacy of using publicly available checkpoints (i.e., BERT, GPT-2, and BART) for natural language processing in the task of text-to-music generation. Our experimental results show that the improvement from using pre-trained checkpoints is statistically significant in terms of BLEU score and edit distance similarity. We analyse the capabilities and limitations of our model to better understand the potential of language-music models.
【10】 LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders
标题：La-Voce：使用神经声码器的低信噪比视听语音增强
链接：https://arxiv.org/abs/2211.10999
作者：Rodrigo Mira,Buye Xu,Jacob Donley,Anurag Kumar,Stavros Petridis,Vamsi Krishna Ithapu,Maja Pantic机构：BUG, Imperial College London, UK, Meta Reality Labs Research, USA, Meta, UK备注：Submitted to ICASSP 2023摘要：视听语音增强的目的是通过不仅利用音频本身，而且利用目标说话人的嘴唇运动，从嘈杂的环境中提取干净的语音。该方法已经被示出为产生优于仅音频语音增强的改进，特别是对于干扰语音的去除。尽管在语音合成方面有了最新的进展，但是大多数视听方法继续使用频谱映射/掩蔽来再现干净的音频，这经常导致将视觉骨干添加到现有语音增强架构。在本文中，我们提出了一种新的两阶段方法LA-VocE，该方法通过基于变换器的结构从含噪视听语音中预测Mel谱图，然后使用神经声码器（HiFi-GAN）将其转换为波形音频。我们在数千个说话者和11种以上的不同语言上训练和评估我们的框架，并研究我们的模型适应不同水平的背景噪声和语音干扰的能力。实验结果表明，LA-VocE算法在多个指标上均优于现有算法，尤其是在噪声环境下.摘要：Audio-visual speech enhancement aims to extract clean speech from a noisy environment by leveraging not only the audio itself but also the target speaker's lip movements. This approach has been shown to yield improvements over audio-only speech enhancement, particularly for the removal of interfering speech. Despite recent advances in speech synthesis, most audio-visual approaches continue to use spectral mapping/masking to reproduce the clean audio, often resulting in visual backbones added to existing speech enhancement architectures. In this work, we propose LA-VocE, a new two-stage approach that predicts mel-spectrograms from noisy audio-visual speech via a transformer-based architecture, and then converts them into waveform audio using a neural vocoder (HiFi-GAN). We train and evaluate our framework on thousands of speakers and 11+ different languages, and study our model's ability to adapt to different levels of background noise and speech interference. Our experiments show that LA-VocE outperforms existing methods according to multiple metrics, particularly under very noisy scenarios.
【11】 Contrastive Regularization for Multimodal Emotion Recognition Using Audio and Text
标题：基于对比度正则化的音频和文本多模式情感识别
链接：https://arxiv.org/abs/2211.10885
机构：School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China备注：Completed in October 2020 and submitted to ICASSP2021摘要：语音情感识别是一个挑战，也是实现更自然的人机交互的重要一步。目前流行的方法是基于模型级融合的多模态情感识别，即对多模态信号进行编码得到嵌入，然后将嵌入连接在一起进行最终的分类。但是，由于噪声或其他因素的影响，每种模态并不总是倾向于同一情感类别，这影响了模型的泛化。本文提出了一种基于对比学习的多模态情感识别方法。通过引入区分器来区分相同和不同情感对之间的差异，显式地限制每个模态的潜代码包含相同的情感信息，以减少噪声干扰，获得更有区别的表示。在标准IEMOCAP数据集上进行了4类情感识别实验。结果表明，与基线系统相比，加权精度（WA）和未加权精度（UA）分别提高了1.44%和1.53%.摘要：Speech emotion recognition is a challenge and an important step towards more natural human-computer interaction (HCI). The popular approach is multimodal emotion recognition based on model-level fusion, which means that the multimodal signals can be encoded to acquire embeddings, and then the embeddings are concatenated together for the final classification. However, due to the influence of noise or other factors, each modality does not always tend to the same emotional category, which affects the generalization of a model. In this paper, we propose a novel regularization method via contrastive learning for multimodal emotion recognition using audio and text. By introducing a discriminator to distinguish the difference between the same and different emotional pairs, we explicitly restrict the latent code of each modality to contain the same emotional information, so as to reduce the noise interference and get more discriminative representation. Experiments are performed on the standard IEMOCAP dataset for 4-class emotion recognition. The results show a significant improvement of 1.44\% and 1.53\% in terms of weighted accuracy (WA) and unweighted accuracy (UA) compared to the baseline system.
【12】 VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement
标题：VarietySound：通过非监督信息解缠生成音色可控的视频到声音
链接：https://arxiv.org/abs/2211.10666
作者：Chenye Cui,Yi Ren,Jinglin Liu,Rongjie Huang,Zhou Zhao摘要：视频到声音生成的目的是在给定视频输入的情况下生成逼真且自然的声音。然而，以往的影音产生方法仅能产生随机或平均的音色，而无法对所产生的音色进行任何控制或特殊化，导致人们有时无法在这些方法下获得所需的音色的问题。在本文中，我们提出了在给定视频输入和参考音频样本的情况下生成具有特定音色的声音的任务。为了解决这个任务，我们将每个目标声音音频分解为三个分量：时间信息、声学信息和背景信息。我们首先使用三个编码器分别对这些分量进行编码：1）时间编码器，用于对时间信息进行编码，由于输入视频与原始音频共享相同的时间信息，所以向时间编码器馈送视频帧; 2）声音编码器，用于编码音色信息，其将原始音频作为输入，并通过时间破坏操作丢弃其时间信息;以及3）背景编码器，用于编码残留或背景声音，其使用原始音频的背景部分作为输入。为了使生成的结果达到更好的质量和时间对齐，我们还采用了一个梅尔鉴别器和一个时间鉴别器进行对抗训练。在VAS数据集上的实验结果表明，该方法能够生成高质量的音频样本，与视频中的事件具有良好的同步性，并且与参考音频具有较高的音质相似度.摘要：Video to sound generation aims to generate realistic and natural sound given a video input. However, previous video-to-sound generation methods can only generate a random or average timbre without any controls or specializations of the generated sound timbre, leading to the problem that people cannot obtain the desired timbre under these methods sometimes. In this paper, we pose the task of generating sound with a specific timbre given a video input and a reference audio sample. To solve this task, we disentangle each target sound audio into three components: temporal information, acoustic information, and background information. We first use three encoders to encode these components respectively: 1) a temporal encoder to encode temporal information, which is fed with video frames since the input video shares the same temporal information as the original audio; 2) an acoustic encoder to encode timbre information, which takes the original audio as input and discards its temporal information by a temporal-corrupting operation; and 3) a background encoder to encode the residual or background sound, which uses the background part of the original audio as input. To make the generated result achieve better quality and temporal alignment, we also adopt a mel discriminator and a temporal discriminator for the adversarial training. Our experimental results on the VAS dataset demonstrate that our method can generate high-quality audio samples with good synchronization with events in video and high timbre similarity with the reference audio.
【13】 Phonemic Adversarial Attack against Audio Recognition in Real World
标题：现实世界中针对音频识别的音素敌意攻击
链接：https://arxiv.org/abs/2211.10661
作者：Jiakai Wang,Zhendong Chen,Zixin Yin,Qinghong Yang,Xianglong Liu机构：Zhongguancun Laboratory, College of Software, Beihang University, State Key Lab of Software Development Environment, Beihang University摘要：近年来，针对音频识别的对抗性攻击引起了广泛的关注。然而，现有的研究大多依赖于实例级的粗粒度音频特征来生成对抗性噪声，导致生成时间开销大、通用攻击能力弱。基于所有音频语音都是由基本音素组成的这一事实，提出了一种音素对抗性攻击（PAT）范式，该范式在音素级攻击音频实例中共有的细粒度音频特征，以生成音素对抗性噪声，具有更通用的攻击能力和更快的生成速度。为了加快训练样本的生成速度，引入了音素密度均衡采样策略，通过估计音素密度，将数量较少但音素特征丰富的音频样本作为训练样本，大大减轻了对大训练样本的依赖.此外，为了提高通用攻击能力，采用滑动窗口异步优化音素噪声，增强音素多样性，从而更好地捕获关键的基本音素模式。通过进行大量的实验，我们全面地研究了所提出的PAT框架，并证明了它比SOTA基线有很大的优势（即，至少11倍的速度和78%的攻击能力提高）。摘要：Recently, adversarial attacks for audio recognition have attracted much attention. However, most of the existing studies mainly rely on the coarse-grain audio features at the instance level to generate adversarial noises, which leads to expensive generation time costs and weak universal attacking ability. Motivated by the observations that all audio speech consists of fundamental phonemes, this paper proposes a phonemic adversarial tack (PAT) paradigm, which attacks the fine-grain audio features at the phoneme level commonly shared across audio instances, to generate phonemic adversarial noises, enjoying the more general attacking ability with fast generation speed. Specifically, for accelerating the generation, a phoneme density balanced sampling strategy is introduced to sample quantity less but phonemic features abundant audio instances as the training data via estimating the phoneme density, which substantially alleviates the heavy dependency on the large training dataset. Moreover, for promoting universal attacking ability, the phonemic noise is optimized in an asynchronous way with a sliding window, which enhances the phoneme diversity and thus well captures the critical fundamental phonemic patterns. By conducting extensive experiments, we comprehensively investigate the proposed PAT framework and demonstrate that it outperforms the SOTA baselines by large margins (i.e., at least 11X speed up and 78% attacking ability improvement).
【14】 EDGE: Editable Dance Generation From Music
标题：EDGE：来自音乐的可编辑舞蹈世代
链接：https://arxiv.org/abs/2211.10658
作者：Jonathan Tseng,Rodrigo Castellon,C. Karen Liu备注：Project website: this https URL摘要：舞蹈是一种重要的人类艺术形式，但创造新的舞蹈可能是困难和耗时的。本文介绍了可编辑舞蹈生成（EDGE），它是一种先进的可编辑舞蹈生成方法，能够生成逼真的、物理上看似真实的舞蹈，同时保持对输入音乐的忠实。EDGE使用基于转换器的扩散模型与Jukebox（一个强大的音乐特征提取器）配合使用，并提供了非常适合舞蹈的强大编辑功能，包括关节调节和中间。我们引入了一种新的物理似然性度量，并通过（1）物理似然性、节拍对齐和多样性基准的多个定量度量，以及更重要的是，（2）大规模用户研究，广泛评估了由我们的方法生成的舞蹈质量，证明了相对于先前最先进的方法的显著改进。我们的模型的定性样本可以在我们的网站上找到。摘要：Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, and evaluate dance quality generated by our method extensively through (1) multiple quantitative metrics on physical plausibility, beat alignment, and diversity benchmarks, and more importantly, (2) a large-scale user study, demonstrating a significant improvement over previous state-of-the-art methods. Qualitative samples from our model can be found at our website.
【15】 VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
标题：VATLM：用于语音表征学习的统一掩蔽预测的视听文本预训练
链接：https://arxiv.org/abs/2211.11275
作者：Qiushi Zhu,Long Zhou,Ziqiang Zhang,Shujie Liu,Binxing Jiao,Jie Zhang,Lirong Dai,Daxin Jiang,Jinyu Li,Furu Wei机构：Fundamental Research Fundsfor the Central Universities and the Leading Plan of CAS (XDC080 10 200), University of Science and Technology of China (USTC)摘要：尽管语音是人类与外部世界通信的简单且有效的方式，但是更真实的语音交互包含多模态信息，视觉和文本。如何设计一个统一的框架来集成不同的模态信息并利用不同的资源（例如，视觉-音频对、音频-文本对、未标记语音和未标记文本）来促进语音表示学习的方法没有被很好地探索。本文提出了一个统一的跨模态表征学习框架VATLM（Visual-Audio-Text Language Model）。提出的VATLM使用统一的主干网络来建模与模态无关的信息，并使用三个简单的与模态相关的模块来预处理视觉、语音和文本输入。为了将这三种模态集成到一个共享的语义空间中，我们提出了统一标记器，通过统一标记的掩蔽预测任务对VATLM进行了优化。我们将预训练的VATLM用于视听相关的下游任务，包括视听语音识别（AVSR）、视觉语音识别（VSR）任务。实验结果表明，VATLM模型的性能优于现有的AV-HuBERT模型，并且能够将不同的模态对齐到同一空间。为了便于将来的研究，我们在www.example.com上发布了代码和预先训练的模型https://aka.ms/vatlm。摘要：Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e.g., vision, text. How to design a unified framework to integrate different modal information and leverage different resources (e.g., visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to facilitate speech representation learning was not well explored. In this paper, we propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model). The proposed VATLM employs a unified backbone network to model the modality-independent information and utilizes three simple modality-dependent modules to preprocess visual, speech, and text inputs. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens, given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on audio-visual related downstream tasks, including audio-visual speech recognition (AVSR), visual speech recognition (VSR) tasks. Results show that the proposed VATLM outperforms previous the state-of-the-art models, such as audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that VATLM is capable of aligning different modalities into the same space. To facilitate future research, we release the code and pre-trained models at https://aka.ms/vatlm.
【16】 Embedding a Differentiable Mel-cepstral Synthesis Filter to a Neural Speech Synthesis System
标题：神经语音合成系统中嵌入可微Mel-倒谱合成滤波器
链接：https://arxiv.org/abs/2211.11222
作者：Takenori Yoshimura,Shinji Takaki,Kazuhiro Nakamura,Keiichiro Oura,Yukiya Hono,Kei Hashimoto,Yoshihiko Nankaku,Keiichi Tokuda机构：Nagoya Institute of Technology, Department of Computer Science, Nagoya, Japan, Techno-Speech, Inc., Department of Research and Development, Nagoya, Japan备注：Submitted to ICASSP 2023摘要：本文将经典的Mel倒谱合成滤波器与现代神经语音合成系统相结合，实现端到端的可控语音合成。由于Mel倒谱合成滤波器被显式地嵌入在所提出的系统中的神经波形模型中，所以语音特征和合成语音的基音分别通过频率弯曲参数和基频被高度控制。我们将梅尔倒谱合成滤波器实现为可微分且GPU友好的模块，以使所提出的系统中的声学和波形模型能够以端到端的方式同时优化。实验结果表明，该系统在保持系统可控性的基础上，提高了语音质量。实验中使用的核心PyTorch模块将在GitHub上公开。摘要：This paper integrates a classic mel-cepstral synthesis filter into a modern neural speech synthesis system towards end-to-end controllable speech synthesis. Since the mel-cepstral synthesis filter is explicitly embedded in neural waveform models in the proposed system, both voice characteristics and the pitch of synthesized speech are highly controlled via a frequency warping parameter and fundamental frequency, respectively. We implement the mel-cepstral synthesis filter as a differentiable and GPU-friendly module to enable the acoustic and waveform models in the proposed system to be simultaneously optimized in an end-to-end manner. Experiments show that the proposed system improves speech quality from a baseline system maintaining controllability. The core PyTorch modules used in the experiments will be publicly available on GitHub.
【17】 Simultaneously Learning Robust Audio Embeddings and balanced Hash codes for Query-by-Example
标题：同时学习稳健的音频嵌入和均衡哈希码进行逐例查询
链接：https://arxiv.org/abs/2211.11060
作者：Anup Singh,Kris Demuynck,Vipul Arora机构：⋆ Department of Electrical Engineering, Indian Institute of Technology Kanpur, India, † IDLab, Department of Electronics and Information Systems, imec - Ghent University, Belgium备注：Submitted to ICASSP2023摘要：音频指纹识别系统必须有效地和鲁棒地识别大规模数据库中的查询片段。为此，最先进的系统使用深度学习来生成紧凑的音频指纹。这些系统部署索引方法，其以无人监督的方式将指纹量化为散列码以加速搜索。然而，这些方法会产生不平衡的杂凑程式码，导致其效能达不到最佳。因此，我们提出一个自我监督学习架构，以端对端的方式计算指纹和平衡杂凑码，以达到快速和准确的检索性能。我们将散列码建模为一个平衡的聚类过程，我们将其视为最优传输问题的一个实例。实验结果表明，与同类方法相比，该方法在保持较高准确率的同时提高了检索效率，尤其是在高失真度下。此外，我们的系统在计算负载和存储器存储方面是高效和可扩展的。摘要：Audio fingerprinting systems must efficiently and robustly identify query snippets in an extensive database. To this end, state-of-the-art systems use deep learning to generate compact audio fingerprints. These systems deploy indexing methods, which quantize fingerprints to hash codes in an unsupervised manner to expedite the search. However, these methods generate imbalanced hash codes, leading to their suboptimal performance. Therefore, we propose a self-supervised learning framework to compute fingerprints and balanced hash codes in an end-to-end manner to achieve both fast and accurate retrieval performance. We model hash codes as a balanced clustering process, which we regard as an instance of the optimal transport problem. Experimental results indicate that the proposed approach improves retrieval efficiency while preserving high accuracy, particularly at high distortion levels, compared to the competing methods. Moreover, our system is efficient and scalable in computational load and memory storage.
【18】 Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling
标题：基于多因素解耦的多说话人表达语音合成
链接：https://arxiv.org/abs/2211.10568
作者：Xinfa Zhu,Yi Lei,Kun Song,Yongmao Zhang,Tao Li,Lei Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China备注：Submitted to ICASSP2023摘要：本文的目的是通过从其他说话人录制的参考语音中转移风格和情感来合成目标说话人具有期望的说话风格和情感的语音。具体而言，我们通过一个两阶段的框架来解决这个具有挑战性的问题，该框架由文本到风格和情感（Text2SE）模块和风格和情感到波形（SE2Wave）模块组成，并通过神经瓶颈（BN）特征进行桥接。为了进一步解决多因素（说话人音色、说话风格和情感）的解耦问题，采用多标签二进制向量（MBV）和互信息（MI）最小化分别对提取的嵌入进行离散化，并在Text2SE和SE2Wave模块中对这些高度纠缠的因素进行解纠缠。此外，我们还引入了一种半监督的训练策略来利用来自多个说话人的数据，包括情感标记数据、风格标记数据和未标记数据。为了在非并行迁移中更好地将参考语的细粒度表达传递给目标说话人，引入了参考语候选库，提出了一种基于注意力的参考语选择方法。大量的实验证明了模型的良好设计。摘要：This paper aims to synthesize target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers. Specifically, we address this challenging problem with a two-stage framework composed of a text-to-style-and-emotion (Text2SE) module and a style-and-emotion-to-wave (SE2Wave) module, bridging by neural bottleneck (BN) features. To further solve the multi-factor (speaker timbre, speaking style and emotion) decoupling problem, we adopt the multi-label binary vector (MBV) and mutual information (MI) minimization to respectively discretize the extracted embeddings and disentangle these highly entangled factors in both Text2SE and SE2Wave modules. Moreover, we introduce a semi-supervised training strategy to leverage data from multiple speakers, including emotion-labelled data, style-labelled data, and unlabeled data. To better transfer the fine-grained expressiveness from references to the target speaker in the non-parallel transfer, we introduce a reference-candidate pool and propose an attention based reference selection approach. Extensive experiments demonstrate the good design of our model.
【19】 Filterbank Learning for Small-Footprint Keyword Spotting Robust to Noise
标题：基于滤波器组学习的小足迹关键词检测算法
链接：https://arxiv.org/abs/2211.10565
作者：Iván López-Espejo,Ram C. M. C. Shekar,Zheng-Hua Tan,Jesper Jensen,John H. L. Hansen机构：Department of Electronic Systems, Aalborg University, Denmark, Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, USA, Oticon AS, Denmark摘要：在关键词识别（KWS）的背景下，用可学习的特征代替手工制作的语音特征并没有产生优越的KWS性能。在本研究中，我们证明了当滤波器组通道的数量严重减少时，滤波器组学习优于KWS的手工语音特征。减少通道数量可能会导致某些KWS性能下降，但也会显著降低能耗，这在低资源设备上部署常见的始终在线KWS时是关键的。在Google Speech Commands Dataset的噪声版本上的实验结果表明，滤波器组学习适应于噪声特性，以提供对噪声的更高程度的鲁棒性，特别是当集成了丢失时。因此，从通常使用的40通道log-Mel特征切换到8通道学习特征导致仅3.5%的相对KWS精度损失，同时实现6.3倍的能耗降低。摘要：In the context of keyword spotting (KWS), the replacement of handcrafted speech features by learnable features has not yielded superior KWS performance. In this study, we demonstrate that filterbank learning outperforms handcrafted speech features for KWS whenever the number of filterbank channels is severely decreased. Reducing the number of channels might yield certain KWS performance drop, but also a substantial energy consumption reduction, which is key when deploying common always-on KWS on low-resource devices. Experimental results on a noisy version of the Google Speech Commands Dataset show that filterbank learning adapts to noise characteristics to provide a higher degree of robustness to noise, especially when dropout is integrated. Thus, switching from typically used 40-channel log-Mel features to 8-channel learned features leads to a relative KWS accuracy loss of only 3.5% while simultaneously achieving a 6.3x energy consumption reduction.
【20】 Impact of visual assistance for automated audio captioning
标题：视觉辅助对自动音频字幕的影响
链接：https://arxiv.org/abs/2211.10539
作者：Wim Boes,Hugo Van hamme机构：ESAT, KU Leuven, Belgium备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：我们研究视觉辅助对自动音频字幕的影响。利用多编码器Transformer架构，其先前已被用于在声音事件检测的上下文中引入视觉相关信息，我们分析了并入各种预训练特征的有用性。 我们在基于YouTube的视听数据集上进行了实验，并根据各种字幕指标研究了应用所考虑的迁移学习技术的效果。 我们发现，所考虑的预训练特征中只有一种提供了一致的改进，而其他特征根本没有提供任何值得注意的增益。有趣的是，先前的研究成果表明，在声音事件检测的情况下，情况正好相反，这使我们得出结论，视觉嵌入的最佳选择强烈依赖于手头的任务。 更具体地，集中于语义的视觉特征在自动音频字幕的上下文中显得适当，而对于声音事件检测，时间信息似乎更重要。摘要：We study the impact of visual assistance for automated audio captioning. Utilizing multi-encoder transformer architectures, which have previously been employed to introduce vision-related information in the context of sound event detection, we analyze the usefulness of incorporating a variety of pretrained features. We perform experiments on a YouTube-based audiovisual data set and investigate the effect of applying the considered transfer learning technique in terms of a variety of captioning metrics. We find that only one of the considered kinds of pretrained features provides consistent improvements, while the others do not provide any noteworthy gains at all. Interestingly, the outcomes of prior research efforts indicate that the exact opposite is true in the case of sound event detection, leading us to conclude that the optimal choice of visual embeddings is strongly dependent on the task at hand. More specifically, visual features focusing on semantics appear appropriate in the context of automated audio captioning, while for sound event detection, time information seems to be more important.
【1】 VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
标题：VATLM：用于语音表征学习的统一掩蔽预测的视听文本预训练
链接：https://arxiv.org/abs/2211.11275
* 与cs.SD语音【15】为同一篇
作者：Qiushi Zhu,Long Zhou,Ziqiang Zhang,Shujie Liu,Binxing Jiao,Jie Zhang,Lirong Dai,Daxin Jiang,Jinyu Li,Furu Wei机构：Fundamental Research Fundsfor the Central Universities and the Leading Plan of CAS (XDC080 10 200), University of Science and Technology of China (USTC)摘要：尽管语音是人类与外部世界通信的简单且有效的方式，但是更真实的语音交互包含多模态信息，视觉和文本。如何设计一个统一的框架来集成不同的模态信息并利用不同的资源（例如，视觉-音频对、音频-文本对、未标记语音和未标记文本）来促进语音表示学习的方法没有被很好地探索。本文提出了一个统一的跨模态表征学习框架VATLM（Visual-Audio-Text Language Model）。提出的VATLM使用统一的主干网络来建模与模态无关的信息，并使用三个简单的与模态相关的模块来预处理视觉、语音和文本输入。为了将这三种模态集成到一个共享的语义空间中，我们提出了统一标记器，通过统一标记的掩蔽预测任务对VATLM进行了优化。我们将预训练的VATLM用于视听相关的下游任务，包括视听语音识别（AVSR）、视觉语音识别（VSR）任务。实验结果表明，VATLM模型的性能优于现有的AV-HuBERT模型，并且能够将不同的模态对齐到同一空间。为了便于将来的研究，我们在www.example.com上发布了代码和预先训练的模型https://aka.ms/vatlm。摘要：Although speech is a simple and effective way for humans to communicate with the outside world, a more realistic speech interaction contains multimodal information, e.g., vision, text. How to design a unified framework to integrate different modal information and leverage different resources (e.g., visual-audio pairs, audio-text pairs, unlabeled speech, and unlabeled text) to facilitate speech representation learning was not well explored. In this paper, we propose a unified cross-modal representation learning framework VATLM (Visual-Audio-Text Language Model). The proposed VATLM employs a unified backbone network to model the modality-independent information and utilizes three simple modality-dependent modules to preprocess visual, speech, and text inputs. In order to integrate these three modalities into one shared semantic space, VATLM is optimized with a masked prediction task of unified tokens, given by our proposed unified tokenizer. We evaluate the pre-trained VATLM on audio-visual related downstream tasks, including audio-visual speech recognition (AVSR), visual speech recognition (VSR) tasks. Results show that the proposed VATLM outperforms previous the state-of-the-art models, such as audio-visual pre-trained AV-HuBERT model, and analysis also demonstrates that VATLM is capable of aligning different modalities into the same space. To facilitate future research, we release the code and pre-trained models at https://aka.ms/vatlm.
【2】 Embedding a Differentiable Mel-cepstral Synthesis Filter to a Neural Speech Synthesis System
标题：神经语音合成系统中嵌入可微Mel-倒谱合成滤波器
链接：https://arxiv.org/abs/2211.11222
* 与cs.SD语音【16】为同一篇
作者：Takenori Yoshimura,Shinji Takaki,Kazuhiro Nakamura,Keiichiro Oura,Yukiya Hono,Kei Hashimoto,Yoshihiko Nankaku,Keiichi Tokuda机构：Nagoya Institute of Technology, Department of Computer Science, Nagoya, Japan, Techno-Speech, Inc., Department of Research and Development, Nagoya, Japan备注：Submitted to ICASSP 2023摘要：本文将经典的Mel倒谱合成滤波器与现代神经语音合成系统相结合，实现端到端的可控语音合成。由于Mel倒谱合成滤波器被显式地嵌入在所提出的系统中的神经波形模型中，所以语音特征和合成语音的基音分别通过频率弯曲参数和基频被高度控制。我们将梅尔倒谱合成滤波器实现为可微分且GPU友好的模块，以使所提出的系统中的声学和波形模型能够以端到端的方式同时优化。实验结果表明，该系统在保持系统可控性的基础上，提高了语音质量。实验中使用的核心PyTorch模块将在GitHub上公开。摘要：This paper integrates a classic mel-cepstral synthesis filter into a modern neural speech synthesis system towards end-to-end controllable speech synthesis. Since the mel-cepstral synthesis filter is explicitly embedded in neural waveform models in the proposed system, both voice characteristics and the pitch of synthesized speech are highly controlled via a frequency warping parameter and fundamental frequency, respectively. We implement the mel-cepstral synthesis filter as a differentiable and GPU-friendly module to enable the acoustic and waveform models in the proposed system to be simultaneously optimized in an end-to-end manner. Experiments show that the proposed system improves speech quality from a baseline system maintaining controllability. The core PyTorch modules used in the experiments will be publicly available on GitHub.
【3】 Simultaneously Learning Robust Audio Embeddings and balanced Hash codes for Query-by-Example
标题：同时学习稳健的音频嵌入和均衡哈希码进行逐例查询
链接：https://arxiv.org/abs/2211.11060
* 与cs.SD语音【17】为同一篇
作者：Anup Singh,Kris Demuynck,Vipul Arora机构：⋆ Department of Electrical Engineering, Indian Institute of Technology Kanpur, India, † IDLab, Department of Electronics and Information Systems, imec - Ghent University, Belgium备注：Submitted to ICASSP2023摘要：音频指纹识别系统必须有效地和鲁棒地识别大规模数据库中的查询片段。为此，最先进的系统使用深度学习来生成紧凑的音频指纹。这些系统部署索引方法，其以无人监督的方式将指纹量化为散列码以加速搜索。然而，这些方法会产生不平衡的杂凑程式码，导致其效能达不到最佳。因此，我们提出一个自我监督学习架构，以端对端的方式计算指纹和平衡杂凑码，以达到快速和准确的检索性能。我们将散列码建模为一个平衡的聚类过程，我们将其视为最优传输问题的一个实例。实验结果表明，与同类方法相比，该方法在保持较高准确率的同时提高了检索效率，尤其是在高失真度下。此外，我们的系统在计算负载和存储器存储方面是高效和可扩展的。摘要：Audio fingerprinting systems must efficiently and robustly identify query snippets in an extensive database. To this end, state-of-the-art systems use deep learning to generate compact audio fingerprints. These systems deploy indexing methods, which quantize fingerprints to hash codes in an unsupervised manner to expedite the search. However, these methods generate imbalanced hash codes, leading to their suboptimal performance. Therefore, we propose a self-supervised learning framework to compute fingerprints and balanced hash codes in an end-to-end manner to achieve both fast and accurate retrieval performance. We model hash codes as a balanced clustering process, which we regard as an instance of the optimal transport problem. Experimental results indicate that the proposed approach improves retrieval efficiency while preserving high accuracy, particularly at high distortion levels, compared to the competing methods. Moreover, our system is efficient and scalable in computational load and memory storage.
【4】 Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling
标题：基于多因素解耦的多说话人表达语音合成
链接：https://arxiv.org/abs/2211.10568
* 与cs.SD语音【18】为同一篇
作者：Xinfa Zhu,Yi Lei,Kun Song,Yongmao Zhang,Tao Li,Lei Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China备注：Submitted to ICASSP2023摘要：本文的目的是通过从其他说话人录制的参考语音中转移风格和情感来合成目标说话人具有期望的说话风格和情感的语音。具体而言，我们通过一个两阶段的框架来解决这个具有挑战性的问题，该框架由文本到风格和情感（Text2SE）模块和风格和情感到波形（SE2Wave）模块组成，并通过神经瓶颈（BN）特征进行桥接。为了进一步解决多因素（说话人音色、说话风格和情感）的解耦问题，采用多标签二进制向量（MBV）和互信息（MI）最小化分别对提取的嵌入进行离散化，并在Text2SE和SE2Wave模块中对这些高度纠缠的因素进行解纠缠。此外，我们还引入了一种半监督的训练策略来利用来自多个说话人的数据，包括情感标记数据、风格标记数据和未标记数据。为了在非并行迁移中更好地将参考语的细粒度表达传递给目标说话人，引入了参考语候选库，提出了一种基于注意力的参考语选择方法。大量的实验证明了模型的良好设计。摘要：This paper aims to synthesize target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers. Specifically, we address this challenging problem with a two-stage framework composed of a text-to-style-and-emotion (Text2SE) module and a style-and-emotion-to-wave (SE2Wave) module, bridging by neural bottleneck (BN) features. To further solve the multi-factor (speaker timbre, speaking style and emotion) decoupling problem, we adopt the multi-label binary vector (MBV) and mutual information (MI) minimization to respectively discretize the extracted embeddings and disentangle these highly entangled factors in both Text2SE and SE2Wave modules. Moreover, we introduce a semi-supervised training strategy to leverage data from multiple speakers, including emotion-labelled data, style-labelled data, and unlabeled data. To better transfer the fine-grained expressiveness from references to the target speaker in the non-parallel transfer, we introduce a reference-candidate pool and propose an attention based reference selection approach. Extensive experiments demonstrate the good design of our model.
【5】 Filterbank Learning for Small-Footprint Keyword Spotting Robust to Noise
标题：基于滤波器组学习的小足迹关键词检测算法
链接：https://arxiv.org/abs/2211.10565
* 与cs.SD语音【19】为同一篇
作者：Iván López-Espejo,Ram C. M. C. Shekar,Zheng-Hua Tan,Jesper Jensen,John H. L. Hansen机构：Department of Electronic Systems, Aalborg University, Denmark, Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, USA, Oticon AS, Denmark摘要：在关键词识别（KWS）的背景下，用可学习的特征代替手工制作的语音特征并没有产生优越的KWS性能。在本研究中，我们证明了当滤波器组通道的数量严重减少时，滤波器组学习优于KWS的手工语音特征。减少通道数量可能会导致某些KWS性能下降，但也会显著降低能耗，这在低资源设备上部署常见的始终在线KWS时是关键的。在Google Speech Commands Dataset的噪声版本上的实验结果表明，滤波器组学习适应于噪声特性，以提供对噪声的更高程度的鲁棒性，特别是当集成了丢失时。因此，从通常使用的40通道log-Mel特征切换到8通道学习特征导致仅3.5%的相对KWS精度损失，同时实现6.3倍的能耗降低。摘要：In the context of keyword spotting (KWS), the replacement of handcrafted speech features by learnable features has not yielded superior KWS performance. In this study, we demonstrate that filterbank learning outperforms handcrafted speech features for KWS whenever the number of filterbank channels is severely decreased. Reducing the number of channels might yield certain KWS performance drop, but also a substantial energy consumption reduction, which is key when deploying common always-on KWS on low-resource devices. Experimental results on a noisy version of the Google Speech Commands Dataset show that filterbank learning adapts to noise characteristics to provide a higher degree of robustness to noise, especially when dropout is integrated. Thus, switching from typically used 40-channel log-Mel features to 8-channel learned features leads to a relative KWS accuracy loss of only 3.5% while simultaneously achieving a 6.3x energy consumption reduction.
【6】 Impact of visual assistance for automated audio captioning
标题：视觉辅助对自动音频字幕的影响
链接：https://arxiv.org/abs/2211.10539
* 与cs.SD语音【20】为同一篇
作者：Wim Boes,Hugo Van hamme机构：ESAT, KU Leuven, Belgium备注：Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：我们研究视觉辅助对自动音频字幕的影响。利用多编码器Transformer架构，其先前已被用于在声音事件检测的上下文中引入视觉相关信息，我们分析了并入各种预训练特征的有用性。 我们在基于YouTube的视听数据集上进行了实验，并根据各种字幕指标研究了应用所考虑的迁移学习技术的效果。 我们发现，所考虑的预训练特征中只有一种提供了一致的改进，而其他特征根本没有提供任何值得注意的增益。有趣的是，先前的研究成果表明，在声音事件检测的情况下，情况正好相反，这使我们得出结论，视觉嵌入的最佳选择强烈依赖于手头的任务。 更具体地，集中于语义的视觉特征在自动音频字幕的上下文中显得适当，而对于声音事件检测，时间信息似乎更重要。摘要：We study the impact of visual assistance for automated audio captioning. Utilizing multi-encoder transformer architectures, which have previously been employed to introduce vision-related information in the context of sound event detection, we analyze the usefulness of incorporating a variety of pretrained features. We perform experiments on a YouTube-based audiovisual data set and investigate the effect of applying the considered transfer learning technique in terms of a variety of captioning metrics. We find that only one of the considered kinds of pretrained features provides consistent improvements, while the others do not provide any noteworthy gains at all. Interestingly, the outcomes of prior research efforts indicate that the exact opposite is true in the case of sound event detection, leading us to conclude that the optimal choice of visual embeddings is strongly dependent on the task at hand. More specifically, visual features focusing on semantics appear appropriate in the context of automated audio captioning, while for sound event detection, time information seems to be more important.
【7】 SpeechNet: Weakly Supervised, End-to-End Speech Recognition at Industrial Scale
标题：SpeechNet：工业规模的弱监督端到端语音识别
链接：https://arxiv.org/abs/2211.11740
* 与cs.SD语音【1】为同一篇
作者：Raphael Tang,Karun Kumar,Gefei Yang,Akshat Pandey,Yajie Mao,Vladislav Belyaev,Madhuri Emmadi,Craig Murray,Ferhan Ture,Jimmy Lin机构：Comcast Applied AI, University of Waterloo备注：Accepted to EMNLP 2022 Industry Track; 9 pages, 7 figures摘要：端到端的自动语音识别系统代表了现有技术的状态，但是它们依赖于数千小时的人工注释语音来进行训练，以及用于推理的重量级计算。当然，这阻碍了商业化，因为大多数公司缺乏大量的人力和计算资源。在这篇文章中，我们探讨了在标签稀缺、计算受限的环境中训练和部署ASR系统。为了减少人工劳动，我们使用第三方ASR系统作为弱监督源，并辅以从隐式用户反馈中导出的标记功能。为了加速推理，我们建议在一个不同输入长度的CUDA图池中路由生产时查询，这些CUDA图的分布与流量的分布最匹配。与我们的第三方ASR相比，我们实现了8%的字错误率和600%的加速比。我们的系统，称为SpeechNet，目前每天在我们的语音智能电视上服务1200万个查询。据我们所知，这是学术文献中首次描述基于Wav2vec的大规模部署。摘要：End-to-end automatic speech recognition systems represent the state of the art, but they rely on thousands of hours of manually annotated speech for training, as well as heavyweight computation for inference. Of course, this impedes commercialization since most companies lack vast human and computational resources. In this paper, we explore training and deploying an ASR system in the label-scarce, compute-limited setting. To reduce human labor, we use a third-party ASR system as a weak supervision source, supplemented with labeling functions derived from implicit user feedback. To accelerate inference, we propose to route production-time queries across a pool of CUDA graphs of varying input lengths, the distribution of which best matches the traffic's. Compared to our third-party ASR, we achieve a relative improvement in word-error rate of 8% and a speedup of 600%. Our system, called SpeechNet, currently serves 12 million queries per day on our voice-enabled smart television. To our knowledge, this is the first time a large-scale, Wav2vec-based deployment has been described in the academic literature.
【8】 Continually learning new languages
标题：持续学习新语言
链接：https://arxiv.org/abs/2211.11703
* 与cs.SD语音【2】为同一篇
作者：Ngoc-Quan Pham,Jan Niehues,Alexander Waibel机构：Interactive Systems Lab, Karlsruhe Institute of Technology, Karlsruhe, Germany, Carnegie Mellon University, Pittsburgh PA, USA备注：Submitted to ICCASP 2023摘要：多语言语音识别中，神经网络通常采用批学习的方法，即在训练前所有语言都是可用的。在之前的训练课程之后增加新语言的能力在经济上是有益的，但主要的挑战是灾难性的遗忘。在本文中，我们将权重分解、迁移学习和弹性权重合并的特性结合起来，以对抗灾难性遗忘，促进新语言的快速学习。在从最初的10种语言学习到27种语言的实验中，这种组合允许我们消除灾难性遗忘，同时仍然实现与同时学习所有语言相当的新语言的性能摘要：Multilingual speech recognition with neural networks is often implemented with batch-learning, when all of the languages are available before training. An ability to add new languages after the prior training sessions can be economically beneficial, but the main challenge is catastrophic forgetting. In this work, we combine the qualities of weight factorization, transfer learning and Elastic Weight Consolidation in order to counter catastrophic forgetting and facilitate learning new languages quickly. Such combination allowed us to eliminate catastrophic forgetting while still achieving performance for the new languages comparable with having all languages at once, in experiments of learning from an initial 10 languages to achieve 27 languages
【9】 Dialogs Re-enacted Across Languages
标题：跨语言重新上演对话
链接：https://arxiv.org/abs/2211.11584
* 与cs.SD语音【3】为同一篇
作者：Nigel G. Ward,Jonathan E. Avila,Emilia Rivas机构：Department of Computer Science, University of Texas at El Paso, Summary摘要：为了支持跨语言韵律映射的机器学习和其他改进语音到语音翻译的方法，我们提出了一种用于收集跨语言的紧密匹配的话语对的协议，对所得到的数据收集进行了描述，并提出了一些观察和思考。这份报告的目的是1）使用语料库的人，2）扩展语料库的人，3）设计类似的双语对话数据集的人。摘要：To support machine learning of cross-language prosodic mappings and other ways to improve speech-to-speech translation, we present a protocol for collecting closely matched pairs of utterances across languages, a description of the resulting data collection, and some observations and musings. This report is intended for 1) people using the corpus, 2) people extending the corpus, and 3) people designing similar collections of bilingual dialog data.
【10】 A Dataset for Greek Traditional and Folk Music: Lyra
标题：希腊传统和民间音乐数据集：Lyra
链接：https://arxiv.org/abs/2211.11479
* 与cs.SD语音【4】为同一篇
作者：Charilaos Papaioannou,Ioannis Valiantzas,Theodoros Giannakopoulos,Maximos Kaliakatsos-Papakostas,Alexandros Potamianos机构：School of ECE, National Technical University of Athens, Greece, Department of Music Studies, National and Kapodistrian University Of Athens, Greece, National Center for Scientific Research - Demokritos, Greece, Athena RC, Greece摘要：在MIR的范围内研究未被充分代表的音乐传统是至关重要的，这不仅是为了开发新颖的分析工具，而且是为了揭示音乐功能，这可能会证明对研究世界音乐有用。本文介绍了一个希腊传统和民间音乐的数据集，其中包括1570件作品，总计约80小时的数据。该数据集包含用于检索音频和视频的YouTube时间戳链接，以及有关仪器、地理和流派等方面的丰富元数据信息。这些内容是从一个希腊纪录片系列中收集的，该系列可以在网上获得，学者们在节目中通过现场音乐和舞蹈表演展示希腊的音乐传统，并讨论所展示音乐的社会、文化和音乐学方面。因此，这一过程产生了大量关于各种方面的描述，如音乐流派、起源地和乐器。此外，音频录制在录制设备方面是在严格的生产级规格下进行的，从而产生了非常干净和均匀的音频内容。在这项工作中，除了详细介绍数据集，我们提出了一个基线深度学习分类方法来识别所涉及的音乐属性。数据集、基线分类方法和模型在公共储存库中提供。还讨论了进一步完善数据集的未来方向。摘要：Studying under-represented music traditions under the MIR scope is crucial, not only for developing novel analysis tools, but also for unveiling musical functions that might prove useful in studying world musics. This paper presents a dataset for Greek Traditional and Folk music that includes 1570 pieces, summing in around 80 hours of data. The dataset incorporates YouTube timestamped links for retrieving audio and video, along with rich metadata information with regards to instrumentation, geography and genre, among others. The content has been collected from a Greek documentary series that is available online, where academics present music traditions of Greece with live music and dance performance during the show, along with discussions about social, cultural and musicological aspects of the presented music. Therefore, this procedure has resulted in a significant wealth of descriptions regarding a variety of aspects, such as musical genre, places of origin and musical instruments. In addition, the audio recordings were performed under strict production-level specifications, in terms of recording equipment, leading to very clean and homogeneous audio content. In this work, apart from presenting the dataset in detail, we propose a baseline deep-learning classification approach to recognize the involved musicological attributes. The dataset, the baseline classification methods and the models are provided in public repositories. Future directions for further refining the dataset are also discussed.
【11】 Sequentially Sampled Chunk Conformer or Streaming End-to-End ASR
标题：顺序采样的区块整合器或流的端到端ASR
链接：https://arxiv.org/abs/2211.11419
* 与cs.SD语音【5】为同一篇
作者：Fangyuan Wang,Xiyuan Wang,Bo Xu机构：Institute of Automation, Chinese Academy of Sciences, School of Future Technology,School of Artificial Intelligence, University of Chinese Academy of Sciences备注：This paper has been submitted to ICASSP 2023摘要：本文深入研究了一种适用于流媒体端到端ASR的顺序采样块一致性算法SSC-Conformer。SSC-Conformer首先证明了在Conformer编码器中使用顺序采样的块式多头自注意（SSC-MHSA）的显著性能增益，通过允许高效的跨块交互同时保持线性复杂度。此外，本文还探索了利用块卷积来利用块未来上下文，并在卷积层中结合偶然卷积来进一步降低CER。我们在AISHELL-1基准测试上验证了所提出的SSC-Conformer，并且实验结果表明，在没有LM重评分的情况下，流传输E2EASR达到了最先进的性能，CER为5.33%。而且，由于其线性复杂度，SSC-Conformer可以用大批量大小训练，并且更有效地推断。摘要：This paper presents an in-depth study on a Sequentially Sampled Chunk Conformer, SSC-Conformer, for streaming End-to-End (E2E) ASR. The SSC-Conformer first demonstrates the significant performance gains from using the sequentially sampled chunk-wise multi-head self-attention (SSC-MHSA) in the Conformer encoder by allowing efficient cross-chunk interactions while keeping linear complexities. Furthermore, it explores taking advantage of chunked convolution to make use of the chunk-wise future context and integrates with casual convolution in the convolution layers to further reduce CER. We verify the proposed SSC-Conformer on the AISHELL-1 benchmark and experimental results show that a state-of-the-art performance for streaming E2E ASR is achieved with CER 5.33% without LM rescoring. And, owing to its linear complexity, the SSC-Conformer can train with large batch sizes and infer more efficiently.
【12】 LISA: Localized Image Stylization with Audio via Implicit Neural Representation
标题：LISA：基于隐式神经表示的局部音频图像风格化
链接：https://arxiv.org/abs/2211.11381
* 与cs.SD语音【6】为同一篇
作者：Seung Hyun Lee,Chanyoung Kim,Wonmin Byeon,Sang Ho Yoon,Jinkyu Kim,Sangpil Kim机构：Department of Artificial Intelligence and ,CSE, Korea University, NVIDIA Research, NVIDIA Corporation, Graduate School of Culture Technology, KAIST, Source Image, Audio Source, Localization Map, Fire Crackling, Stylized Image, Splashing Water, Underwater Bubbling摘要：提出了一种基于音频的局部化图像风格化（Localized Image Stylization with Audio，LISA）框架，实现了音频驱动的局部化图像风格化。声音通常提供关于场景的特定上下文的信息，并且与场景或对象的特定部分密切相关。然而，现有的图像风格化工作集中于使用图像或文本输入来风格化整个图像。基于音频输入风格化图像的特定部分是自然的，但具有挑战性。在这项工作中，我们提出了一个框架，用户提供音频输入来定位输入图像中的声源，另一个用于局部风格化目标对象或场景。LISA首先利用CLIP嵌入空间生成具有视听定位网络的精细定位地图。然后，我们利用隐式神经表示（INR）以及预测的定位图来基于声音信息风格化目标对象或场景。所提出的INR可以操纵局部化的像素值以与所提供的音频输入在语义上一致。通过一系列的实验，我们证明了所提出的框架优于其他音频引导风格化方法。此外，LISA构建简洁的定位图，并根据给定的音频输入自然地操纵目标对象或场景。摘要：We present a novel framework, Localized Image Stylization with Audio (LISA) which performs audio-driven localized image stylization. Sound often provides information about the specific context of the scene and is closely related to a certain part of the scene or object. However, existing image stylization works have focused on stylizing the entire image using an image or text input. Stylizing a particular part of the image based on audio input is natural but challenging. In this work, we propose a framework that a user provides an audio input to localize the sound source in the input image and another for locally stylizing the target object or scene. LISA first produces a delicate localization map with an audio-visual localization network by leveraging CLIP embedding space. We then utilize implicit neural representation (INR) along with the predicted localization map to stylize the target object or scene based on sound information. The proposed INR can manipulate the localized pixel values to be semantically consistent with the provided audio input. Through a series of experiments, we show that the proposed framework outperforms the other audio-guided stylization methods. Moreover, LISA constructs concise localization maps and naturally manipulates the target object or scene in accordance with the given audio input.
【13】 Video Background Music Generation: Dataset, Method and Evaluation
标题：视频背景音乐生成：数据集、方法与评价
链接：https://arxiv.org/abs/2211.11248
* 与cs.SD语音【7】为同一篇
作者：Le Zhuo,Zhaokai Wang,Baisen Wang,Yue Liao,Stanley Peng,Chenxi Bao,Miao Lu,Xiaobo Li,Si Liu机构：Beihang University, Edinburgh College of Art, The University of Edinburgh, Alibaba Group摘要：编辑视频时，音乐是必不可少的，但手动选择音乐既困难又耗时。因此，我们寻求在给定视频输入的情况下自动生成背景音乐轨道。这是一个具有挑战性的任务，因为它需要大量的配对视频和音乐来学习他们的对应关系。遗憾的是，没有这样的数据集。为了弥补这一差距，我们引入了一个数据集，基准模型，和评估指标的视频背景音乐生成。我们介绍SymMV，一个视频和符号音乐数据集，以及和弦、节奏、旋律和伴奏注释。据我们所知，它是第一个包含高质量符号音乐和详细注释的视频音乐数据集。本文还提出了一个视频背景音乐生成框架V-MusProd，该框架利用了和弦、旋律和伴奏的音乐先验知识，以及语义、颜色和运动特征的视频-音乐关系。为了解决视频-音乐对应性缺乏客观度量的问题，我们提出了一种基于检索的度量VMCP，该度量建立在一个强大的视频-音乐表示学习模型之上。实验结果表明，V-MusProd在音乐质量和视频一致性方面均优于现有的方法.我们相信我们的数据集、基准模型和评估指标将推动视频背景音乐生成的发展。摘要：Music is essential when editing videos, but selecting music manually is difficult and time-consuming. Thus, we seek to automatically generate background music tracks given video input. This is a challenging task since it requires plenty of paired videos and music to learn their correspondence. Unfortunately, there exist no such datasets. To close this gap, we introduce a dataset, benchmark model, and evaluation metric for video background music generation. We introduce SymMV, a video and symbolic music dataset, along with chord, rhythm, melody, and accompaniment annotations. To the best of our knowledge, it is the first video-music dataset with high-quality symbolic music and detailed annotations. We also propose a benchmark video background music generation framework named V-MusProd, which utilizes music priors of chords, melody, and accompaniment along with video-music relations of semantic, color, and motion features. To address the lack of objective metrics for video-music correspondence, we propose a retrieval-based metric VMCP built upon a powerful video-music representation learning model. Experiments show that with our dataset, V-MusProd outperforms the state-of-the-art method in both music quality and correspondence with videos. We believe our dataset, benchmark model, and evaluation metric will boost the development of video background music generation.
【14】 TimbreCLIP: Connecting Timbre to Text and Images
标题：TimbreCLIP：将音色连接到文本和图像
链接：https://arxiv.org/abs/2211.11225
* 与cs.SD语音【8】为同一篇
作者：Nicolas Jonason,Bob L. T. Sturm机构：KTH Royal Institute of Technology备注：Submitted to AAAI workshop on creative AI across modalities摘要：我们介绍了正在进行的TimbreCLIP工作，这是一个在单乐器音符上训练的音频-文本跨模态嵌入。我们在synth补丁上用跨模态检索任务评估模型。最后，我们演示了TimbreCLIP在两个任务中的应用：文本驱动音频均衡和音色到图像生成。摘要：We present work in progress on TimbreCLIP, an audio-text cross modal embedding trained on single instrument notes. We evaluate the models with a cross-modal retrieval task on synth patches. Finally, we demonstrate the application of TimbreCLIP on two tasks: text-driven audio equalization and timbre to image generation.
【15】 Exploring the Efficacy of Pre-trained Checkpoints in Text-to-Music Generation Task
标题：探索预先训练的检查点在文本到音乐生成任务中的有效性
链接：https://arxiv.org/abs/2211.11216
* 与cs.SD语音【9】为同一篇
作者：Shangda Wu,Maosong Sun机构：† Department of Music AI and Information Technology, Central Conservatory of Music, ‡ Department of Computer Science and Technology, Tsinghua University备注：5 pages, 2 figures, 2 tables摘要：得益于大规模数据集和预先训练的模型，生成模型领域最近获得了巨大的发展势头。然而，大多数符号音乐的数据集都非常小，这潜在地限制了数据驱动的多模态模型的性能。该问题的直观解决方案是利用来自其他模态的预训练模型（例如，自然语言）以提高与符号音乐相关的多模态任务的性能。本文首先对从文本描述中生成完整的、语义一致的符号乐谱进行了研究，并探讨了使用公开可用的检查点（即，BERT、GPT-2和BART），用于文本到音乐生成任务中的自然语言处理。实验结果表明，使用预先训练好的检查点，在BLEU评分和编辑距离相似度方面有统计学意义的改善。我们分析了我们的模型的能力和局限性，以更好地理解语言-音乐模型的潜力。摘要：Benefiting from large-scale datasets and pre-trained models, the field of generative models has recently gained significant momentum. However, most datasets for symbolic music are very small, which potentially limits the performance of data-driven multimodal models. An intuitive solution to this problem is to leverage pre-trained models from other modalities (e.g., natural language) to improve the performance of symbolic music-related multimodal tasks. In this paper, we carry out the first study of generating complete and semantically consistent symbolic music scores from text descriptions, and explore the efficacy of using publicly available checkpoints (i.e., BERT, GPT-2, and BART) for natural language processing in the task of text-to-music generation. Our experimental results show that the improvement from using pre-trained checkpoints is statistically significant in terms of BLEU score and edit distance similarity. We analyse the capabilities and limitations of our model to better understand the potential of language-music models.
【16】 LA-VocE: Low-SNR Audio-visual Speech Enhancement using Neural Vocoders
标题：La-Voce：使用神经声码器的低信噪比视听语音增强
链接：https://arxiv.org/abs/2211.10999
* 与cs.SD语音【10】为同一篇
作者：Rodrigo Mira,Buye Xu,Jacob Donley,Anurag Kumar,Stavros Petridis,Vamsi Krishna Ithapu,Maja Pantic机构：BUG, Imperial College London, UK, Meta Reality Labs Research, USA, Meta, UK备注：Submitted to ICASSP 2023摘要：视听语音增强的目的是通过不仅利用音频本身，而且利用目标说话人的嘴唇运动，从嘈杂的环境中提取干净的语音。该方法已经被示出为产生优于仅音频语音增强的改进，特别是对于干扰语音的去除。尽管在语音合成方面有了最新的进展，但是大多数视听方法继续使用频谱映射/掩蔽来再现干净的音频，这经常导致将视觉骨干添加到现有语音增强架构。在本文中，我们提出了一种新的两阶段方法LA-VocE，该方法通过基于变换器的结构从含噪视听语音中预测Mel谱图，然后使用神经声码器（HiFi-GAN）将其转换为波形音频。我们在数千个说话者和11种以上的不同语言上训练和评估我们的框架，并研究我们的模型适应不同水平的背景噪声和语音干扰的能力。实验结果表明，LA-VocE算法在多个指标上均优于现有算法，尤其是在噪声环境下.摘要：Audio-visual speech enhancement aims to extract clean speech from a noisy environment by leveraging not only the audio itself but also the target speaker's lip movements. This approach has been shown to yield improvements over audio-only speech enhancement, particularly for the removal of interfering speech. Despite recent advances in speech synthesis, most audio-visual approaches continue to use spectral mapping/masking to reproduce the clean audio, often resulting in visual backbones added to existing speech enhancement architectures. In this work, we propose LA-VocE, a new two-stage approach that predicts mel-spectrograms from noisy audio-visual speech via a transformer-based architecture, and then converts them into waveform audio using a neural vocoder (HiFi-GAN). We train and evaluate our framework on thousands of speakers and 11+ different languages, and study our model's ability to adapt to different levels of background noise and speech interference. Our experiments show that LA-VocE outperforms existing methods according to multiple metrics, particularly under very noisy scenarios.
【17】 Contrastive Regularization for Multimodal Emotion Recognition Using Audio and Text
标题：基于对比度正则化的音频和文本多模式情感识别
链接：https://arxiv.org/abs/2211.10885
* 与cs.SD语音【11】为同一篇
机构：School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China备注：Completed in October 2020 and submitted to ICASSP2021摘要：语音情感识别是一个挑战，也是实现更自然的人机交互的重要一步。目前流行的方法是基于模型级融合的多模态情感识别，即对多模态信号进行编码得到嵌入，然后将嵌入连接在一起进行最终的分类。但是，由于噪声或其他因素的影响，每种模态并不总是倾向于同一情感类别，这影响了模型的泛化。本文提出了一种基于对比学习的多模态情感识别方法。通过引入区分器来区分相同和不同情感对之间的差异，显式地限制每个模态的潜代码包含相同的情感信息，以减少噪声干扰，获得更有区别的表示。在标准IEMOCAP数据集上进行了4类情感识别实验。结果表明，与基线系统相比，加权精度（WA）和未加权精度（UA）分别提高了1.44%和1.53%.摘要：Speech emotion recognition is a challenge and an important step towards more natural human-computer interaction (HCI). The popular approach is multimodal emotion recognition based on model-level fusion, which means that the multimodal signals can be encoded to acquire embeddings, and then the embeddings are concatenated together for the final classification. However, due to the influence of noise or other factors, each modality does not always tend to the same emotional category, which affects the generalization of a model. In this paper, we propose a novel regularization method via contrastive learning for multimodal emotion recognition using audio and text. By introducing a discriminator to distinguish the difference between the same and different emotional pairs, we explicitly restrict the latent code of each modality to contain the same emotional information, so as to reduce the noise interference and get more discriminative representation. Experiments are performed on the standard IEMOCAP dataset for 4-class emotion recognition. The results show a significant improvement of 1.44\% and 1.53\% in terms of weighted accuracy (WA) and unweighted accuracy (UA) compared to the baseline system.
【18】 VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement
标题：VarietySound：通过非监督信息解缠生成音色可控的视频到声音
链接：https://arxiv.org/abs/2211.10666
* 与cs.SD语音【12】为同一篇
作者：Chenye Cui,Yi Ren,Jinglin Liu,Rongjie Huang,Zhou Zhao摘要：视频到声音生成的目的是在给定视频输入的情况下生成逼真且自然的声音。然而，以往的影音产生方法仅能产生随机或平均的音色，而无法对所产生的音色进行任何控制或特殊化，导致人们有时无法在这些方法下获得所需的音色的问题。在本文中，我们提出了在给定视频输入和参考音频样本的情况下生成具有特定音色的声音的任务。为了解决这个任务，我们将每个目标声音音频分解为三个分量：时间信息、声学信息和背景信息。我们首先使用三个编码器分别对这些分量进行编码：1）时间编码器，用于对时间信息进行编码，由于输入视频与原始音频共享相同的时间信息，所以向时间编码器馈送视频帧; 2）声音编码器，用于编码音色信息，其将原始音频作为输入，并通过时间破坏操作丢弃其时间信息;以及3）背景编码器，用于编码残留或背景声音，其使用原始音频的背景部分作为输入。为了使生成的结果达到更好的质量和时间对齐，我们还采用了一个梅尔鉴别器和一个时间鉴别器进行对抗训练。在VAS数据集上的实验结果表明，该方法能够生成高质量的音频样本，与视频中的事件具有良好的同步性，并且与参考音频具有较高的音质相似度.摘要：Video to sound generation aims to generate realistic and natural sound given a video input. However, previous video-to-sound generation methods can only generate a random or average timbre without any controls or specializations of the generated sound timbre, leading to the problem that people cannot obtain the desired timbre under these methods sometimes. In this paper, we pose the task of generating sound with a specific timbre given a video input and a reference audio sample. To solve this task, we disentangle each target sound audio into three components: temporal information, acoustic information, and background information. We first use three encoders to encode these components respectively: 1) a temporal encoder to encode temporal information, which is fed with video frames since the input video shares the same temporal information as the original audio; 2) an acoustic encoder to encode timbre information, which takes the original audio as input and discards its temporal information by a temporal-corrupting operation; and 3) a background encoder to encode the residual or background sound, which uses the background part of the original audio as input. To make the generated result achieve better quality and temporal alignment, we also adopt a mel discriminator and a temporal discriminator for the adversarial training. Our experimental results on the VAS dataset demonstrate that our method can generate high-quality audio samples with good synchronization with events in video and high timbre similarity with the reference audio.
【19】 Phonemic Adversarial Attack against Audio Recognition in Real World
标题：现实世界中针对音频识别的音素敌意攻击
链接：https://arxiv.org/abs/2211.10661
* 与cs.SD语音【13】为同一篇
作者：Jiakai Wang,Zhendong Chen,Zixin Yin,Qinghong Yang,Xianglong Liu机构：Zhongguancun Laboratory, College of Software, Beihang University, State Key Lab of Software Development Environment, Beihang University摘要：近年来，针对音频识别的对抗性攻击引起了广泛的关注。然而，现有的研究大多依赖于实例级的粗粒度音频特征来生成对抗性噪声，导致生成时间开销大、通用攻击能力弱。基于所有音频语音都是由基本音素组成的这一事实，提出了一种音素对抗性攻击（PAT）范式，该范式在音素级攻击音频实例中共有的细粒度音频特征，以生成音素对抗性噪声，具有更通用的攻击能力和更快的生成速度。为了加快训练样本的生成速度，引入了音素密度均衡采样策略，通过估计音素密度，将数量较少但音素特征丰富的音频样本作为训练样本，大大减轻了对大训练样本的依赖.此外，为了提高通用攻击能力，采用滑动窗口异步优化音素噪声，增强音素多样性，从而更好地捕获关键的基本音素模式。通过进行大量的实验，我们全面地研究了所提出的PAT框架，并证明了它比SOTA基线有很大的优势（即，至少11倍的速度和78%的攻击能力提高）。摘要：Recently, adversarial attacks for audio recognition have attracted much attention. However, most of the existing studies mainly rely on the coarse-grain audio features at the instance level to generate adversarial noises, which leads to expensive generation time costs and weak universal attacking ability. Motivated by the observations that all audio speech consists of fundamental phonemes, this paper proposes a phonemic adversarial tack (PAT) paradigm, which attacks the fine-grain audio features at the phoneme level commonly shared across audio instances, to generate phonemic adversarial noises, enjoying the more general attacking ability with fast generation speed. Specifically, for accelerating the generation, a phoneme density balanced sampling strategy is introduced to sample quantity less but phonemic features abundant audio instances as the training data via estimating the phoneme density, which substantially alleviates the heavy dependency on the large training dataset. Moreover, for promoting universal attacking ability, the phonemic noise is optimized in an asynchronous way with a sliding window, which enhances the phoneme diversity and thus well captures the critical fundamental phonemic patterns. By conducting extensive experiments, we comprehensively investigate the proposed PAT framework and demonstrate that it outperforms the SOTA baselines by large margins (i.e., at least 11X speed up and 78% attacking ability improvement).
【20】 EDGE: Editable Dance Generation From Music
标题：EDGE：来自音乐的可编辑舞蹈世代
链接：https://arxiv.org/abs/2211.10658
* 与cs.SD语音【14】为同一篇
作者：Jonathan Tseng,Rodrigo Castellon,C. Karen Liu备注：Project website: this https URL摘要：舞蹈是一种重要的人类艺术形式，但创造新的舞蹈可能是困难和耗时的。本文介绍了可编辑舞蹈生成（EDGE），它是一种先进的可编辑舞蹈生成方法，能够生成逼真的、物理上看似真实的舞蹈，同时保持对输入音乐的忠实。EDGE使用基于转换器的扩散模型与Jukebox（一个强大的音乐特征提取器）配合使用，并提供了非常适合舞蹈的强大编辑功能，包括关节调节和中间。我们引入了一种新的物理似然性度量，并通过（1）物理似然性、节拍对齐和多样性基准的多个定量度量，以及更重要的是，（2）大规模用户研究，广泛评估了由我们的方法生成的舞蹈质量，证明了相对于先前最先进的方法的显著改进。我们的模型的定性样本可以在我们的网站上找到。摘要：Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, and evaluate dance quality generated by our method extensively through (1) multiple quantitative metrics on physical plausibility, beat alignment, and diversity benchmarks, and more importantly, (2) a large-scale user study, demonstrating a significant improvement over previous state-of-the-art methods. Qualitative samples from our model can be found at our website.
机器翻译，仅供参考
每日语音速递