今日论文合集：cs.SD语音6篇，eess.AS音频处理7篇。

cs.SD语音

【1】 MAQA: A Multimodal QA Benchmark for Negation

标题：MAQA：一种用于否定的多模式问答基准

链接：https://arxiv.org/abs/2301.03238

作者：Judith Yue Li,Aren Jansen,Qingqing Huang,Joonseok Lee,Ravi Ganti,Dima Kuzmin
机构：Google Research, & Seoul National University
备注：NeurIPS 2022 SyntheticData4ML Workshop
摘要：多模态学习可以从预先训练的大语言模型（LLM）的表示能力中获益。然而，现有的基于Transformer的LLM常常忽略自然语言中的否定，并且没有现有的基准来定量评估多模态Transformers是否继承了这一弱点。在这项研究中，我们提出了一个新的多模式问答（QA）基准，改编自AudioSet中的标记音乐视频（Gemmeke等人，2017），其目标是系统地评估多模态Transformers是否可以执行复杂推理以将新概念识别为先前学习的概念的否定。我们表明，使用标准的微调方法，无论模型大小如何，多模态Transformers仍然不能正确地解释否定。然而，我们的实验表明，用否定的QA例子扩充原始训练任务分布允许模型可靠地用否定推理。为此，我们描述了一种新颖的数据生成过程，该过程提示540B参数PaLM模型自动生成否定的QA示例作为容易访问的视频标签的组成。生成的实例包含了更多的自然语言模式，与基于模板的任务扩充方法相比有显著的提高。
摘要：Multimodal learning can benefit from the representation power of pretrained Large Language Models (LLMs). However, state-of-the-art transformer based LLMs often ignore negations in natural language and there is no existing benchmark to quantitatively evaluate whether multimodal transformers inherit this weakness. In this study, we present a new multimodal question answering (QA) benchmark adapted from labeled music videos in AudioSet (Gemmeke et al., 2017) with the goal of systematically evaluating if multimodal transformers can perform complex reasoning to recognize new concepts as negation of previously learned concepts. We show that with standard fine-tuning approach multimodal transformers are still incapable of correctly interpreting negation irrespective of model size. However, our experiments demonstrate that augmenting the original training task distributions with negated QA examples allow the model to reliably reason with negation. To do this, we describe a novel data generation procedure that prompts the 540B-parameter PaLM model to automatically generate negated QA examples as compositions of easily accessible video tags. The generated examples contain more natural linguistic patterns and the gains compared to template-based task augmentation approach are significant.


【2】 Introducing Model Inversion Attacks on Automatic Speaker Recognition

标题：在说话人自动识别中引入模型反转攻击

链接：https://arxiv.org/abs/2301.03206

作者：Karla Pizzi,Franziska Boenisch,Ugur Sahin,Konstantin Böttinger
机构：Fraunhofer AISEC, Germany; ,Technical University Munich, Germany
备注：None
摘要：模型反演（MI）攻击允许重构机器学习（ML）模型的训练数据的平均每类表示。已经表明，在每个类别对应于不同个体的场景中，例如面部分类器，这代表了严重的隐私风险。在这项工作中，我们探索了MI的一种新应用：从说话人识别系统中提取说话人的声音。我们提出了一种方法，（1）从训练好的ML模型中重构音频样本，以及（2）提取中间语音特征表示，这为说话人的生物统计学提供了有价值的见解。  因此，我们提出了一种扩展的MI攻击，我们称之为滑动模型反演。我们的滑动MI通过迭代地反转音频样本的重叠块来扩展标准MI，从而利用音频数据的顺序属性来增强反转性能。我们展示了可以使用反转的音频数据来生成欺骗的音频样本，以模仿说话者，并代表他们为高度安全的系统执行语音保护命令。据我们所知，我们的工作是第一个将MI攻击扩展到音频数据的工作，我们的结果突出了在该设置中提取生物特征数据所导致的安全风险。
摘要：Model inversion (MI) attacks allow to reconstruct average per-class representations of a machine learning (ML) model's training data. It has been shown that in scenarios where each class corresponds to a different individual, such as face classifiers, this represents a severe privacy risk. In this work, we explore a new application for MI: the extraction of speakers' voices from a speaker recognition system. We present an approach to (1) reconstruct audio samples from a trained ML model and (2) extract intermediate voice feature representations which provide valuable insights into the speakers' biometrics.  Therefore, we propose an extension of MI attacks which we call sliding model inversion. Our sliding MI extends standard MI by iteratively inverting overlapping chunks of the audio samples and thereby leveraging the sequential properties of audio data for enhanced inversion performance. We show that one can use the inverted audio data to generate spoofed audio samples to impersonate a speaker, and execute voice-protected commands for highly secured systems on their behalf. To the best of our knowledge, our work is the first one extending MI attacks to audio data, and our results highlight the security risks resulting from the extraction of the biometric data in that setup.


【3】 Perceptual-Neural-Physical Sound Matching

标题：知觉-神经-物理声音匹配

链接：https://arxiv.org/abs/2301.02886

作者：Han Han,Vincent Lostanlen,Mathieu Lagrange
机构：Nantes Universit´e, ´Ecole Centrale Nantes, CNRS, LS,N, UMR , F-, Nantes, France
摘要：声音匹配算法寻求通过参数化音频合成来近似目标波形。深度神经网络在匹配持续谐波音调方面取得了很好的效果。然而，当目标不稳定和不和谐时，任务就更具挑战性，例如，打击乐。我们把这个问题归结为损失函数的不足。一方面，被称为“P损失”的参数域中的均方误差简单且快速，但是不能适应每个参数的不同感知意义。另一方面，频谱时域中的均方误差，即所谓的“频谱损失”，是由感知引起的，并且在可微分数字信号处理（DDSP）中起作用。然而，谱损失比P损失具有更多的局部最小值，并且其梯度可能在计算上是昂贵的;因此收敛较慢。针对这一难题，我们提出了知觉-神经-身体损失（PNP）。PNP是频谱损失的最佳二次近似，同时在训练期间与P损失一样快。我们以物理模型合成作为解码器，以联合时频散射变换（JTFS）作为谱表示来实现PNP。与其他损失函数相比，我们证明了它在匹配合成鼓声方面的潜力。
摘要：Sound matching algorithms seek to approximate a target waveform by parametric audio synthesis. Deep neural networks have achieved promising results in matching sustained harmonic tones. However, the task is more challenging when targets are nonstationary and inharmonic, e.g., percussion. We attribute this problem to the inadequacy of loss function. On one hand, mean square error in the parametric domain, known as "P-loss", is simple and fast but fails to accommodate the differing perceptual significance of each parameter. On the other hand, mean square error in the spectrotemporal domain, known as "spectral loss", is perceptually motivated and serves in differentiable digital signal processing (DDSP). Yet, spectral loss has more local minima than P-loss and its gradient may be computationally expensive; hence a slow convergence. Against this conundrum, we present Perceptual-Neural-Physical loss (PNP). PNP is the optimal quadratic approximation of spectral loss while being as fast as P-loss during training. We instantiate PNP with physical modeling synthesis as decoder and joint time-frequency scattering transform (JTFS) as spectral representation. We demonstrate its potential on matching synthetic drum sounds in comparison with other loss functions.


【4】 TunesFormer: Forming Tunes with Control Codes

标题：TunesFormer：使用控制代码形成Tunes

链接：https://arxiv.org/abs/2301.02884

作者：Shangda Wu,Maosong Sun
机构：Music AI and Information Technology, Central Conservatory of Music, Beijing, China, Computer Science and Technology, Tsinghua University
备注：9 pages, 11 figures
摘要：近年来，深度学习技术已经被应用于音乐生成系统，并取得了有前景的结果。然而，该领域的主要挑战之一是缺乏注释数据集，使得模型难以学习乐曲中的音乐形式。为了解决这个问题，我们提出了TunesFormer，一个基于Transformer的旋律生成系统，它是在285，449个ABC曲调的大数据集上训练的。通过使用ABC符号中常见的特定符号来指示小节边界，TunesFormer可以基于控制代码理解并生成具有给定音乐形式的旋律。我们的客观评价证明了控制代码在实现受控音乐形式方面的有效性，主观实验表明生成的旋律具有与人类作品相当的质量。我们的结果也提供了对控制代码的最佳放置及其对生成旋律的影响的见解。TunesFormer提出了一种通过使用深度学习技术生成具有所需音乐形式的旋律的有前途的方法。
摘要：In recent years, deep learning techniques have been applied to music generation systems with promising results. However, one of the main challenges in this field has been the lack of annotated datasets, making it difficult for models to learn musical forms in compositions. To address this issue, we present TunesFormer, a Transformer-based melody generation system that is trained on a large dataset of 285,449 ABC tunes. By utilizing specific symbols commonly found in ABC notation to indicate section boundaries, TunesFormer can understand and generate melodies with given musical forms based on control codes. Our objective evaluations demonstrate the effectiveness of the control codes in achieving controlled musical forms, and subjective experiments show that the generated melodies are of comparable quality to human compositions. Our results also provide insights into the optimal placement of control codes and their impact on the generated melodies. TunesFormer presents a promising approach for generating melodies with desired musical forms through the use of deep learning techniques.


【5】 Multimodal Lyrics-Rhythm Matching

标题：多模式歌词-节奏匹配

链接：https://arxiv.org/abs/2301.02732

作者：Callie C. Liao,Duoduo Liao,Jesse Guessford
机构：McLean High School, McLean, USA, School of Computing, George Mason University, Fairfax, USA, School of Music
摘要：尽管近来对音乐人工智能的研究有所增加，但歌词的关键组成部分与节奏（如关键词、重音音节和强节拍）之间的显著相关性并不经常被研究。这很可能是由于音频错位、音节识别不准确等挑战，最重要的是，需要跨学科知识。针对这一研究不足，本文提出了一种新的多模态歌词-节奏匹配方法，该方法可以在不受语言限制的情况下，将歌词和音乐的关键成分进行匹配。我们使用音频而不是乐谱，它具有容易获得的元数据，这带来了更多的挑战，但增加了我们方法的应用灵活性。此外，我们的方法创造性地生成了涉及各种多模态的几种模式，包括音乐强节拍、抒情音节、歌手发音的听觉变化，特别是抒情关键词，这些关键词用于将关键抒情元素与关键节奏元素进行匹配。这种有利的方法不仅提供了一种独特的方式来研究听觉歌词-节奏相关性，包括有效的基于节奏的音频对准算法，而且还将计算语言学与音乐以及音乐认知联系起来。我们的实验结果揭示了平均0.81的匹配概率，并且大约30%的歌曲具有0.9或更高的关键词落在强节拍上的概率，包括12%的歌曲具有完美的落着。此外，相似性度量被用来评估歌词和节奏之间的相关性。结果表明，近50%的歌曲具有0.70或更高的相似度。总之，我们的方法通过计算揭示了深刻的相关性，对歌词-节奏关系做出了重大贡献。
摘要：Despite the recent increase in research on artificial intelligence for music, prominent correlations between key components of lyrics and rhythm such as keywords, stressed syllables, and strong beats are not frequently studied. Ths is likely due to challenges such as audio misalignment, inaccuracies in syllabic identification, and most importantly, the need for cross-disciplinary knowledge. To address this lack of research, we propose a novel multimodal lyrics-rhythm matching approach in this paper that specifically matches key components of lyrics and music with each other without any language limitations. We use audio instead of sheet music with readily available metadata, which creates more challenges yet increases the application flexibility of our method. Furthermore, our approach creatively generates several patterns involving various multimodalities, including music strong beats, lyrical syllables, auditory changes in a singer's pronunciation, and especially lyrical keywords, which are utilized for matching key lyrical elements with key rhythmic elements. This advantageous approach not only provides a unique way to study auditory lyrics-rhythm correlations including efficient rhythm-based audio alignment algorithms, but also bridges computational linguistics with music as well as music cognition. Our experimental results reveal an 0.81 probability of matching on average, and around 30% of the songs have a probability of 0.9 or higher of keywords landing on strong beats, including 12% of the songs with a perfect landing. Also, the similarity metrics are used to evaluate the correlation between lyrics and rhythm. It shows that nearly 50% of the songs have 0.70 similarity or higher. In conclusion, our approach contributes significantly to the lyrics-rhythm relationship by computationally unveiling insightful correlations.


【6】 Using External Off-Policy Speech-To-Text Mappings in Contextual  End-To-End Automated Speech Recognition

标题：在上下文端到端自动语音识别中使用外部非策略语音到文本映射

链接：https://arxiv.org/abs/2301.02736

作者：David M. Chan,Shalini Ghosh,Ariya Rastrow,Björn Hoffmeister
机构：Bj¨orn Hoffmeister†, ⋆ University of California, Berkeley, † Amazon Alexa AI
摘要：尽管自动语音识别（ASR）模型的泛化性能有所改进，但针对下游任务专门化ASR模型仍然是一项具有挑战性的任务，主要是由于数据可用性降低（需要增加数据收集）和数据分布快速变化（需要更频繁的模型微调）。在这项工作中，我们研究了利用外部知识的潜力，特别是通过文本到语音方法生成的非策略键值存储，以允许灵活的训练后适应新的数据分布。在我们的方法中，从文本到语音捕获的音频嵌入，连同语义文本嵌入一起，通过基于近似k最近邻（KNN）的注意力融合步骤来偏置ASR。我们在LibiriSpeech和内部语音助手/搜索数据集上的实验表明，与微调基线相比，该方法可以将域自适应时间减少多达1 K GPU小时，同时提供多达3%的WER改善，这表明该方法在零次和Few-Shot的挑战场景下适用于生产ASR系统。
摘要：Despite improvements to the generalization performance of automated speech recognition (ASR) models, specializing ASR models for downstream tasks remains a challenging task, primarily due to reduced data availability (necessitating increased data collection), and rapidly shifting data distributions (requiring more frequent model fine-tuning). In this work, we investigate the potential of leveraging external knowledge, particularly through off-policy key-value stores generated with text-to-speech methods, to allow for flexible post-training adaptation to new data distributions. In our approach, audio embeddings captured from text-to-speech, along with semantic text embeddings, are used to bias ASR via an approximate k-nearest-neighbor (KNN) based attentive fusion step. Our experiments on LibiriSpeech and in-house voice assistant/search datasets show that the proposed approach can reduce domain adaptation time by up to 1K GPU-hours while providing up to 3% WER improvement compared to a fine-tuning baseline, suggesting a promising approach for adapting production ASR systems in challenging zero and few-shot scenarios.
eess.AS音频处理

【1】 Using External Off-Policy Speech-To-Text Mappings in Contextual  End-To-End Automated Speech Recognition

标题：在上下文端到端自动语音识别中使用外部非策略语音到文本映射

链接：https://arxiv.org/abs/2301.02736

*与cs.SD语音【6】为同一篇

作者：David M. Chan,Shalini Ghosh,Ariya Rastrow,Björn Hoffmeister
机构：Bj¨orn Hoffmeister†, ⋆ University of California, Berkeley, † Amazon Alexa AI
摘要：尽管自动语音识别（ASR）模型的泛化性能有所改进，但针对下游任务专门化ASR模型仍然是一项具有挑战性的任务，主要是由于数据可用性降低（需要增加数据收集）和数据分布快速变化（需要更频繁的模型微调）。在这项工作中，我们研究了利用外部知识的潜力，特别是通过文本到语音方法生成的非策略键值存储，以允许灵活的训练后适应新的数据分布。在我们的方法中，从文本到语音捕获的音频嵌入，连同语义文本嵌入一起，通过基于近似k最近邻（KNN）的注意力融合步骤来偏置ASR。我们在LibiriSpeech和内部语音助手/搜索数据集上的实验表明，与微调基线相比，该方法可以将域自适应时间减少多达1 K GPU小时，同时提供多达3%的WER改善，这表明该方法在零次和Few-Shot的挑战场景下适用于生产ASR系统。
摘要：Despite improvements to the generalization performance of automated speech recognition (ASR) models, specializing ASR models for downstream tasks remains a challenging task, primarily due to reduced data availability (necessitating increased data collection), and rapidly shifting data distributions (requiring more frequent model fine-tuning). In this work, we investigate the potential of leveraging external knowledge, particularly through off-policy key-value stores generated with text-to-speech methods, to allow for flexible post-training adaptation to new data distributions. In our approach, audio embeddings captured from text-to-speech, along with semantic text embeddings, are used to bias ASR via an approximate k-nearest-neighbor (KNN) based attentive fusion step. Our experiments on LibiriSpeech and in-house voice assistant/search datasets show that the proposed approach can reduce domain adaptation time by up to 1K GPU-hours while providing up to 3% WER improvement compared to a fine-tuning baseline, suggesting a promising approach for adapting production ASR systems in challenging zero and few-shot scenarios.


【2】 MAQA: A Multimodal QA Benchmark for Negation

标题：MAQA：一种用于否定的多模式问答基准

链接：https://arxiv.org/abs/2301.03238

*与cs.SD语音【1】为同一篇

作者：Judith Yue Li,Aren Jansen,Qingqing Huang,Joonseok Lee,Ravi Ganti,Dima Kuzmin
机构：Google Research, & Seoul National University
备注：NeurIPS 2022 SyntheticData4ML Workshop
摘要：多模态学习可以从预先训练的大语言模型（LLM）的表示能力中获益。然而，现有的基于Transformer的LLM常常忽略自然语言中的否定，并且没有现有的基准来定量评估多模态Transformers是否继承了这一弱点。在这项研究中，我们提出了一个新的多模式问答（QA）基准，改编自AudioSet中的标记音乐视频（Gemmeke等人，2017），其目标是系统地评估多模态Transformers是否可以执行复杂推理以将新概念识别为先前学习的概念的否定。我们表明，使用标准的微调方法，无论模型大小如何，多模态Transformers仍然不能正确地解释否定。然而，我们的实验表明，用否定的QA例子扩充原始训练任务分布允许模型可靠地用否定推理。为此，我们描述了一种新颖的数据生成过程，该过程提示540B参数PaLM模型自动生成否定的QA示例作为容易访问的视频标签的组成。生成的实例包含了更多的自然语言模式，与基于模板的任务扩充方法相比有显著的提高。
摘要：Multimodal learning can benefit from the representation power of pretrained Large Language Models (LLMs). However, state-of-the-art transformer based LLMs often ignore negations in natural language and there is no existing benchmark to quantitatively evaluate whether multimodal transformers inherit this weakness. In this study, we present a new multimodal question answering (QA) benchmark adapted from labeled music videos in AudioSet (Gemmeke et al., 2017) with the goal of systematically evaluating if multimodal transformers can perform complex reasoning to recognize new concepts as negation of previously learned concepts. We show that with standard fine-tuning approach multimodal transformers are still incapable of correctly interpreting negation irrespective of model size. However, our experiments demonstrate that augmenting the original training task distributions with negated QA examples allow the model to reliably reason with negation. To do this, we describe a novel data generation procedure that prompts the 540B-parameter PaLM model to automatically generate negated QA examples as compositions of easily accessible video tags. The generated examples contain more natural linguistic patterns and the gains compared to template-based task augmentation approach are significant.


【3】 Introducing Model Inversion Attacks on Automatic Speaker Recognition

标题：在说话人自动识别中引入模型反转攻击

链接：https://arxiv.org/abs/2301.03206

*与cs.SD语音【2】为同一篇

作者：Karla Pizzi,Franziska Boenisch,Ugur Sahin,Konstantin Böttinger
机构：Fraunhofer AISEC, Germany; ,Technical University Munich, Germany
备注：None
摘要：模型反演（MI）攻击允许重构机器学习（ML）模型的训练数据的平均每类表示。已经表明，在每个类别对应于不同个体的场景中，例如面部分类器，这代表了严重的隐私风险。在这项工作中，我们探索了MI的一种新应用：从说话人识别系统中提取说话人的声音。我们提出了一种方法，（1）从训练好的ML模型中重构音频样本，以及（2）提取中间语音特征表示，这为说话人的生物统计学提供了有价值的见解。  因此，我们提出了一种扩展的MI攻击，我们称之为滑动模型反演。我们的滑动MI通过迭代地反转音频样本的重叠块来扩展标准MI，从而利用音频数据的顺序属性来增强反转性能。我们展示了可以使用反转的音频数据来生成欺骗的音频样本，以模仿说话者，并代表他们为高度安全的系统执行语音保护命令。据我们所知，我们的工作是第一个将MI攻击扩展到音频数据的工作，我们的结果突出了在该设置中提取生物特征数据所导致的安全风险。
摘要：Model inversion (MI) attacks allow to reconstruct average per-class representations of a machine learning (ML) model's training data. It has been shown that in scenarios where each class corresponds to a different individual, such as face classifiers, this represents a severe privacy risk. In this work, we explore a new application for MI: the extraction of speakers' voices from a speaker recognition system. We present an approach to (1) reconstruct audio samples from a trained ML model and (2) extract intermediate voice feature representations which provide valuable insights into the speakers' biometrics.  Therefore, we propose an extension of MI attacks which we call sliding model inversion. Our sliding MI extends standard MI by iteratively inverting overlapping chunks of the audio samples and thereby leveraging the sequential properties of audio data for enhanced inversion performance. We show that one can use the inverted audio data to generate spoofed audio samples to impersonate a speaker, and execute voice-protected commands for highly secured systems on their behalf. To the best of our knowledge, our work is the first one extending MI attacks to audio data, and our results highlight the security risks resulting from the extraction of the biometric data in that setup.


【4】 SpeeChain: A Speech Toolkit for Large-Scale Machine Speech Chain

标题：SpeeChain：面向大规模机器语音链的语音工具包

链接：https://arxiv.org/abs/2301.02966

作者：Heli Qi,Sashi Novitasari,Andros Tjandra,Sakriani Sakti,Satoshi Nakamura
机构：Nara Institute of Science and Technology, Japan, Japan Advanced Institute of Science and Technology, Japan
备注：Submitted to ICASSP 2023
摘要：本文介绍了一个基于Pytorch的开源工具包SpeeChain，该工具包旨在开发大规模使用的机器语音链。这个第一个版本的重点是TTS到ASR链，这是机器语音链的核心组件，指的是通过ASR的未说出文本来增加TTS数据。为了构建大规模TTS到ASR链的高效流水线，我们实现了易于使用的多GPU批处理级模型推理、多数据加载器批处理生成和动态数据选择技术。在本文中，我们首先解释了TTS到ASR链的整体过程和每个步骤的难点。然后，我们对不同类型的未标记数据、数据过滤阈值、批次组成和真实-合成数据比进行了详细的消融研究。在LibriSpeech的train_clean_460上的实验结果表明，该TTS-to-ASR链能够显著提高半监督环境下的WER。
摘要：This paper introduces SpeeChain, an open-source Pytorch-based toolkit designed to develop the machine speech chain for large-scale use. This first release focuses on the TTS-to-ASR chain, a core component of the machine speech chain, that refers to the TTS data augmentation by unspoken text for ASR. To build an efficient pipeline for the large-scale TTS-to-ASR chain, we implement easy-to-use multi-GPU batch-level model inference, multi-dataloader batch generation, and on-the-fly data selection techniques. In this paper, we first explain the overall procedure of the TTS-to-ASR chain and the difficulties of each step. Then, we present a detailed ablation study on different types of unlabeled data, data filtering thresholds, batch composition, and real-synthetic data ratios. Our experimental results on train_clean_460 of LibriSpeech demonstrate that our TTS-to-ASR chain can significantly improve WER in a semi-supervised setting.


【5】 Perceptual-Neural-Physical Sound Matching

标题：知觉-神经-物理声音匹配

链接：https://arxiv.org/abs/2301.02886

*与cs.SD语音【3】为同一篇

作者：Han Han,Vincent Lostanlen,Mathieu Lagrange
机构：Nantes Universit´e, ´Ecole Centrale Nantes, CNRS, LS,N, UMR , F-, Nantes, France
摘要：声音匹配算法寻求通过参数化音频合成来近似目标波形。深度神经网络在匹配持续谐波音调方面取得了很好的效果。然而，当目标不稳定和不和谐时，任务就更具挑战性，例如，打击乐。我们把这个问题归结为损失函数的不足。一方面，被称为“P损失”的参数域中的均方误差简单且快速，但是不能适应每个参数的不同感知意义。另一方面，频谱时域中的均方误差，即所谓的“频谱损失”，是由感知引起的，并且在可微分数字信号处理（DDSP）中起作用。然而，谱损失比P损失具有更多的局部最小值，并且其梯度可能在计算上是昂贵的;因此收敛较慢。针对这一难题，我们提出了知觉-神经-身体损失（PNP）。PNP是频谱损失的最佳二次近似，同时在训练期间与P损失一样快。我们以物理模型合成作为解码器，以联合时频散射变换（JTFS）作为谱表示来实现PNP。与其他损失函数相比，我们证明了它在匹配合成鼓声方面的潜力。
摘要：Sound matching algorithms seek to approximate a target waveform by parametric audio synthesis. Deep neural networks have achieved promising results in matching sustained harmonic tones. However, the task is more challenging when targets are nonstationary and inharmonic, e.g., percussion. We attribute this problem to the inadequacy of loss function. On one hand, mean square error in the parametric domain, known as "P-loss", is simple and fast but fails to accommodate the differing perceptual significance of each parameter. On the other hand, mean square error in the spectrotemporal domain, known as "spectral loss", is perceptually motivated and serves in differentiable digital signal processing (DDSP). Yet, spectral loss has more local minima than P-loss and its gradient may be computationally expensive; hence a slow convergence. Against this conundrum, we present Perceptual-Neural-Physical loss (PNP). PNP is the optimal quadratic approximation of spectral loss while being as fast as P-loss during training. We instantiate PNP with physical modeling synthesis as decoder and joint time-frequency scattering transform (JTFS) as spectral representation. We demonstrate its potential on matching synthetic drum sounds in comparison with other loss functions.


【6】 TunesFormer: Forming Tunes with Control Codes

标题：TunesFormer：使用控制代码形成Tunes

链接：https://arxiv.org/abs/2301.02884

*与cs.SD语音【4】为同一篇

作者：Shangda Wu,Maosong Sun
机构：Music AI and Information Technology, Central Conservatory of Music, Beijing, China, Computer Science and Technology, Tsinghua University
备注：9 pages, 11 figures
摘要：近年来，深度学习技术已经被应用于音乐生成系统，并取得了有前景的结果。然而，该领域的主要挑战之一是缺乏注释数据集，使得模型难以学习乐曲中的音乐形式。为了解决这个问题，我们提出了TunesFormer，一个基于Transformer的旋律生成系统，它是在285，449个ABC曲调的大数据集上训练的。通过使用ABC符号中常见的特定符号来指示小节边界，TunesFormer可以基于控制代码理解并生成具有给定音乐形式的旋律。我们的客观评价证明了控制代码在实现受控音乐形式方面的有效性，主观实验表明生成的旋律具有与人类作品相当的质量。我们的结果也提供了对控制代码的最佳放置及其对生成旋律的影响的见解。TunesFormer提出了一种通过使用深度学习技术生成具有所需音乐形式的旋律的有前途的方法。
摘要：In recent years, deep learning techniques have been applied to music generation systems with promising results. However, one of the main challenges in this field has been the lack of annotated datasets, making it difficult for models to learn musical forms in compositions. To address this issue, we present TunesFormer, a Transformer-based melody generation system that is trained on a large dataset of 285,449 ABC tunes. By utilizing specific symbols commonly found in ABC notation to indicate section boundaries, TunesFormer can understand and generate melodies with given musical forms based on control codes. Our objective evaluations demonstrate the effectiveness of the control codes in achieving controlled musical forms, and subjective experiments show that the generated melodies are of comparable quality to human compositions. Our results also provide insights into the optimal placement of control codes and their impact on the generated melodies. TunesFormer presents a promising approach for generating melodies with desired musical forms through the use of deep learning techniques.


【7】 Multimodal Lyrics-Rhythm Matching

标题：多模式歌词-节奏匹配

链接：https://arxiv.org/abs/2301.02732

*与cs.SD语音【5】为同一篇

作者：Callie C. Liao,Duoduo Liao,Jesse Guessford
机构：McLean High School, McLean, USA, School of Computing, George Mason University, Fairfax, USA, School of Music
摘要：尽管近来对音乐人工智能的研究有所增加，但歌词的关键组成部分与节奏（如关键词、重音音节和强节拍）之间的显著相关性并不经常被研究。这很可能是由于音频错位、音节识别不准确等挑战，最重要的是，需要跨学科知识。针对这一研究不足，本文提出了一种新的多模态歌词-节奏匹配方法，该方法可以在不受语言限制的情况下，将歌词和音乐的关键成分进行匹配。我们使用音频而不是乐谱，它具有容易获得的元数据，这带来了更多的挑战，但增加了我们方法的应用灵活性。此外，我们的方法创造性地生成了涉及各种多模态的几种模式，包括音乐强节拍、抒情音节、歌手发音的听觉变化，特别是抒情关键词，这些关键词用于将关键抒情元素与关键节奏元素进行匹配。这种有利的方法不仅提供了一种独特的方式来研究听觉歌词-节奏相关性，包括有效的基于节奏的音频对准算法，而且还将计算语言学与音乐以及音乐认知联系起来。我们的实验结果揭示了平均0.81的匹配概率，并且大约30%的歌曲具有0.9或更高的关键词落在强节拍上的概率，包括12%的歌曲具有完美的落着。此外，相似性度量被用来评估歌词和节奏之间的相关性。结果表明，近50%的歌曲具有0.70或更高的相似度。总之，我们的方法通过计算揭示了深刻的相关性，对歌词-节奏关系做出了重大贡献。
摘要：Despite the recent increase in research on artificial intelligence for music, prominent correlations between key components of lyrics and rhythm such as keywords, stressed syllables, and strong beats are not frequently studied. Ths is likely due to challenges such as audio misalignment, inaccuracies in syllabic identification, and most importantly, the need for cross-disciplinary knowledge. To address this lack of research, we propose a novel multimodal lyrics-rhythm matching approach in this paper that specifically matches key components of lyrics and music with each other without any language limitations. We use audio instead of sheet music with readily available metadata, which creates more challenges yet increases the application flexibility of our method. Furthermore, our approach creatively generates several patterns involving various multimodalities, including music strong beats, lyrical syllables, auditory changes in a singer's pronunciation, and especially lyrical keywords, which are utilized for matching key lyrical elements with key rhythmic elements. This advantageous approach not only provides a unique way to study auditory lyrics-rhythm correlations including efficient rhythm-based audio alignment algorithms, but also bridges computational linguistics with music as well as music cognition. Our experimental results reveal an 0.81 probability of matching on average, and around 30% of the songs have a probability of 0.9 or higher of keywords landing on strong beats, including 12% of the songs with a perfect landing. Also, the similarity metrics are used to evaluate the correlation between lyrics and rhythm. It shows that nearly 50% of the songs have 0.70 similarity or higher. In conclusion, our approach contributes significantly to the lyrics-rhythm relationship by computationally unveiling insightful correlations.

机器翻译，仅供参考