【1】The Arrow of Time in Music -- Revisiting the Temporal Structure of Music with Distinguishability and Unique Orientability as the Anchor Point标题：音乐中的时间之箭--以可区分性和独特指向性为基点重新审视音乐的时间结构链接：https://arxiv.org/abs/2312.17633作者：Qi Xu摘要：本文以“时间之箭”为主题，从其词源：哲学（认识论）与物理学（热力学）展开音乐讨论。特别是，本文探讨了两个具体的条件：可兼容性和独特的方向性，从这两个条件出发，本文得出了各自的音乐命题和案例研究。在可重复性条件下，本文以音乐中的“重现”为切入点，试图从“出生/复活”的角度解读巴赫的《圣诞清唱剧》。针对这种独特的定向性条件，本文讨论了音乐发展的高潮延迟过程，提出了“AB-AAB左复制”模式，将音乐的时间结构（如曲式）视为一个动态过程的产物：有机生长，从而体现了有机主义的观点。摘要：Driven by the term "the arrow of time" as a general topic, the article develops a musical discussion by referring to the etymological origin of the term: philosophy (epistemology) and physics (thermodynamics). In particular, the article explores two specific conditions: distinguishability and unique orientability, from which the article derives respective musical propositions and case studies. For the distinguishability condition, the article focuses on the "recurrence" in music and tries to interpret Bach's Christmas Oratorio from the perspective of "birth/resurrection". For the unique orientability condition, the article discusses the process of delaying the climax, thereby proposing "AB-AAB left-replication" model, implying an organicist view by treating the temporal structure of music (e.g. form) as the product of a dynamic process: organic growth.【2】 AQUALLM: Audio Question Answering Data Generation Using Large Language Models标题：AQUALLM：使用大型语言模型生成音频问答数据链接：https://arxiv.org/abs/2312.17343作者：Swarup Ranjan Behera,Krishna Mohan Injeti,Jaya Sai Kiran Patibandla,Praveen Kumar Pokala,Balakrishna Reddy Pailla摘要：音频问答（AQA）是机器分析音频信号和自然语言问题以产生精确的自然语言答案的关键任务。当目标是AQA系统的精度时，拥有高质量，多样化和广泛的AQA数据集的重要性不能被夸大。虽然开发准确和有效的AQA模型一直受到关注，但为手头的特定任务创建高质量，多样化和广泛的数据集并没有引起相当大的关注。为了应对这一挑战，这项工作作出了几项贡献。我们引入了一个可扩展的AQA数据生成管道，表示为AQA LM框架，它依赖于大型语言模型（LLM）。该框架利用现有的音频字幕注释并结合最先进的LLM来生成扩展的高质量AQA数据集。此外，我们还为AQA提供了三个广泛而高质量的基准数据集，为AQA研究的进展做出了重大贡献。与现有的最新技术相比，在所提出的数据集上训练的AQA模型设置了更好的基准。此外，与使用人类注释的AQA数据训练的模型相比，在我们的数据集上训练的模型表现出更强的泛化能力。代码和数据集可以在GitHub上访问~\footnote{\url{https：//github.com/swarupbehera/AuberLM}}。摘要：Audio Question Answering (AQA) constitutes a pivotal task in which machines analyze both audio signals and natural language questions to produce precise natural language answers. The significance of possessing high-quality, diverse, and extensive AQA datasets cannot be overstated when aiming for the precision of an AQA system. While there has been notable focus on developing accurate and efficient AQA models, the creation of high-quality, diverse, and extensive datasets for the specific task at hand has not garnered considerable attention. To address this challenge, this work makes several contributions. We introduce a scalable AQA data generation pipeline, denoted as the AQUALLM framework, which relies on Large Language Models (LLMs). This framework utilizes existing audio-caption annotations and incorporates state-of-the-art LLMs to generate expansive, high-quality AQA datasets. Additionally, we present three extensive and high-quality benchmark datasets for AQA, contributing significantly to the progression of AQA research. AQA models trained on the proposed datasets set superior benchmarks compared to the existing state-of-the-art. Moreover, models trained on our datasets demonstrate enhanced generalizability when compared to models trained using human-annotated AQA data. Code and datasets will be accessible on GitHub~\footnote{\url{https://github.com/swarupbehera/AQUALLM}}.【3】 Revolutionizing Personalized Voice Synthesis: The Journey towards Emotional and Individual Authenticity with DIVSE (Dynamic Individual Voice Synthesis Engine)标题：革命性的个性化语音合成：使用DIVSE(动态个人语音合成引擎)迈向情感和个人真实性的旅程链接：https://arxiv.org/abs/2312.17281作者：Fan Shi备注：8 pages摘要：这篇全面的论文深入研究了人工智能（AI）中个性化语音合成的最前沿，重点介绍了动态个人语音合成引擎（DIVSE）。DIVSE代表了文本到语音（TTS）技术的突破性飞跃，独特地专注于适应和个性化语音输出，以匹配个人的声音特征。这项研究强调了当前人工智能生成的声音的差距，虽然技术先进，但在复制人类语音固有的独特个性和表达能力方面存在不足。它概述了个性化语音合成的挑战和进步，强调情感表现力，口音和方言变化的重要性，并捕捉个人的语音特征。DIVSE的架构非常详细，展示了它的三个核心组件：语音特征学习模块（VCLM），情绪音调和口音适应模块（ETAAM）和动态语音合成引擎（DSSE）。DIVSE的创新方法在于其自适应学习能力，该能力随着时间的推移而发展，以根据特定的用户特征定制语音输出。本文提出了一个严格的实验设置，利用公认的数据集和个性化指标，如平均意见分数（MOS）和情感对齐分数，以验证DIVSE的优越性超过主流模型。结果描绘了在人工智能生成的声音中实现更高个性化和情感共鸣方面的明显进步。摘要：This comprehensive paper delves into the forefront of personalized voice synthesis within artificial intelligence (AI), spotlighting the Dynamic Individual Voice Synthesis Engine (DIVSE). DIVSE represents a groundbreaking leap in text-to-voice (TTS) technology, uniquely focusing on adapting and personalizing voice outputs to match individual vocal characteristics. The research underlines the gap in current AI-generated voices, which, while technically advanced, fall short in replicating the unique individuality and expressiveness intrinsic to human speech. It outlines the challenges and advancements in personalized voice synthesis, emphasizing the importance of emotional expressiveness, accent and dialect variability, and capturing individual voice traits. The architecture of DIVSE is meticulously detailed, showcasing its three core components: Voice Characteristic Learning Module (VCLM), Emotional Tone and Accent Adaptation Module (ETAAM), and Dynamic Speech Synthesis Engine (DSSE). The innovative approach of DIVSE lies in its adaptive learning capability, which evolves over time to tailor voice outputs to specific user traits. The paper presents a rigorous experimental setup, utilizing accepted datasets and personalization metrics like Mean Opinion Score (MOS) and Emotional Alignment Score, to validate DIVSE's superiority over mainstream models. The results depict a clear advancement in achieving higher personalization and emotional resonance in AI-generated voices.【4】 Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion标题：基于注意力的交互式解缠网络实例级情感语音转换链接：https://arxiv.org/abs/2312.17508作者：Yun Chen,Lingxiao Yang,Qi Chen,Jian-Huang Lai,Xiaohua Xie备注：Accepted by INTERSPEECH 2023摘要：情感语音转换的目的是根据给定的情感来处理语音，同时保留非情感成分。现有的方法不能很好地表达细粒度的情感属性。在本文中，我们提出了一个基于注意力的交互式diseNtangling网络（AINN），利用实例明智的情感知识的语音转换。我们引入了一个两阶段的管道来有效地训练我们的网络：第一阶段利用语音间对比学习来建模细粒度的情感和语音内分离学习来更好地分离情感和内容。在第二阶段，我们建议使用多视图一致性机制来规范转换。这种技术可以帮助我们传递精细的情感并保持语音内容。大量的实验表明，我们的AINN优于国家的最先进的客观和主观的指标。摘要：Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. Existing approaches cannot well express fine-grained emotional attributes. In this paper, we propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion. We introduce a two-stage pipeline to effectively train our network: Stage I utilizes inter-speech contrastive learning to model fine-grained emotion and intra-speech disentanglement learning to better separate emotion and content. In Stage II, we propose to regularize the conversion with a multi-view consistency mechanism. This technique helps us transfer fine-grained emotion and maintain speech content. Extensive experiments show that our AINN outperforms state-of-the-arts in both objective and subjective metrics.【5】 Single-channel speech enhancement using learnable loss mixup标题：基于可学习损失混合的单通道语音增强链接：https://arxiv.org/abs/2312.17255作者：Oscar Chang,Dung N. Tran,Kazuhito Koishida摘要：泛化是单通道语音增强有监督学习中的一个主要问题。在这项工作中，我们提出了可学习的损失混淆（LLM），这是一种简单而轻松的训练图，可以提高基于深度学习的语音增强模型的泛化能力。损失混淆是损失混淆的一个特殊变体，它优化随机样本对的损失函数的混合，以在由这些样本对构造的虚拟训练数据上训练模型。在可学习的损失混合中，通过对混合数据进行调节，使用通过神经参数化自动学习的非线性混合函数来混合损失函数。我们在VCTK基准测试上的实验结果表明，可学习的损失混淆达到3.26 PESQ，优于最先进的。摘要：Generalization remains a major problem in supervised learning of single-channel speech enhancement. In this work, we propose learnable loss mixup (LLM), a simple and effortless training diagram, to improve the generalization of deep learning-based speech enhancement models. Loss mixup, of which learnable loss mixup is a special variant, optimizes a mixture of the loss functions of random sample pairs to train a model on virtual training data constructed from these pairs of samples. In learnable loss mixup, by conditioning on the mixed data, the loss functions are mixed using a non-linear mixing function automatically learned via neural parameterization. Our experimental results on the VCTK benchmark show that learnable loss mixup achieves 3.26 PESQ, outperforming the state-of-the-art.【6】 RCT: Random Consistency Training for Semi-supervised Sound Event Detection标题：RCT：用于半监督声音事件检测的随机一致性训练链接：https://arxiv.org/abs/2110.11144作者：Nian Shao,Erfan Loweimi,Xiaofei Li备注：Preprint for interspeech 2022摘要：声事件检测作为声环境分析的核心模块，存在数据不足的问题。半监督学习（SSL）的集成在很大程度上缓解了这个问题，同时没有带来额外的注释预算。本文研究了SSL的几个核心模块，介绍了一种随机一致性训练策略。首先，提出了一种自我一致性损失与师生模型融合，以稳定训练。第二，提出了一种硬混合数据增强方法来考虑声音的附加属性。第三，应用随机增强方案来灵活地组合不同类型的数据增强。实验结果表明，该策略优于其他广泛使用的策略。摘要：Sound event detection (SED), as a core module of acoustic environmental analysis, suffers from the problem of data deficiency. The integration of semi-supervised learning (SSL) largely mitigates such problem while bringing no extra annotation budget. This paper researches on several core modules of SSL, and introduces a random consistency training (RCT) strategy. First, a self-consistency loss is proposed to fuse with the teacher-student model to stabilize the training. Second, a hard mixup data augmentation is proposed to account for the additive property of sounds. Third, a random augmentation scheme is applied to flexibly combine different types of data augmentations. Experiments show that the proposed strategy outperform other widely-used strategies.eess.AS音频处理【1】 Attention-based Interactive Disentangling Network for Instance-level Emotional Voice Conversion标题：基于注意力的交互式解缠网络实例级情感语音转换链接：https://arxiv.org/abs/2312.17508作者：Yun Chen,Lingxiao Yang,Qi Chen,Jian-Huang Lai,Xiaohua Xie备注：Accepted by INTERSPEECH 2023摘要：情感语音转换的目的是根据给定的情感来处理语音，同时保留非情感成分。现有的方法不能很好地表达细粒度的情感属性。在本文中，我们提出了一个基于注意力的交互式diseNtangling网络（AINN），利用实例明智的情感知识的语音转换。我们引入了一个两阶段的管道来有效地训练我们的网络：第一阶段利用语音间对比学习来建模细粒度的情感和语音内分离学习来更好地分离情感和内容。在第二阶段，我们建议使用多视图一致性机制来规范转换。这种技术可以帮助我们传递精细的情感并保持语音内容。大量的实验表明，我们的AINN优于国家的最先进的客观和主观的指标。摘要：Emotional Voice Conversion aims to manipulate a speech according to a given emotion while preserving non-emotion components. Existing approaches cannot well express fine-grained emotional attributes. In this paper, we propose an Attention-based Interactive diseNtangling Network (AINN) that leverages instance-wise emotional knowledge for voice conversion. We introduce a two-stage pipeline to effectively train our network: Stage I utilizes inter-speech contrastive learning to model fine-grained emotion and intra-speech disentanglement learning to better separate emotion and content. In Stage II, we propose to regularize the conversion with a multi-view consistency mechanism. This technique helps us transfer fine-grained emotion and maintain speech content. Extensive experiments show that our AINN outperforms state-of-the-arts in both objective and subjective metrics.【2】 Single-channel speech enhancement using learnable loss mixup标题：基于可学习损失混合的单通道语音增强链接：https://arxiv.org/abs/2312.17255作者：Oscar Chang,Dung N. Tran,Kazuhito Koishida摘要：泛化是单通道语音增强有监督学习中的一个主要问题。在这项工作中，我们提出了可学习的损失混淆（LLM），这是一种简单而轻松的训练图，可以提高基于深度学习的语音增强模型的泛化能力。损失混淆是损失混淆的一个特殊变体，它优化随机样本对的损失函数的混合，以在由这些样本对构造的虚拟训练数据上训练模型。在可学习的损失混合中，通过对混合数据进行调节，使用通过神经参数化自动学习的非线性混合函数来混合损失函数。我们在VCTK基准测试上的实验结果表明，可学习的损失混淆达到3.26 PESQ，优于最先进的。摘要：Generalization remains a major problem in supervised learning of single-channel speech enhancement. In this work, we propose learnable loss mixup (LLM), a simple and effortless training diagram, to improve the generalization of deep learning-based speech enhancement models. Loss mixup, of which learnable loss mixup is a special variant, optimizes a mixture of the loss functions of random sample pairs to train a model on virtual training data constructed from these pairs of samples. In learnable loss mixup, by conditioning on the mixed data, the loss functions are mixed using a non-linear mixing function automatically learned via neural parameterization. Our experimental results on the VCTK benchmark show that learnable loss mixup achieves 3.26 PESQ, outperforming the state-of-the-art.【3】 The Arrow of Time in Music -- Revisiting the Temporal Structure of Music with Distinguishability and Unique Orientability as the Anchor Point标题：音乐中的时间之箭--以可区分性和独特定向性为落脚点重温音乐的时间结构链接：https://arxiv.org/abs/2312.17633作者：Qi Xu摘要：本文以“时间之箭”为主题，从其词源：哲学（认识论）与物理学（热力学）展开音乐讨论。特别是，本文探讨了两个具体的条件：可兼容性和独特的方向性，从这两个条件出发，本文得出了各自的音乐命题和案例研究。在可重复性条件下，本文以音乐中的“重现”为切入点，试图从“出生/复活”的角度解读巴赫的《圣诞清唱剧》。针对这种独特的定向性条件，本文讨论了音乐发展的高潮延迟过程，提出了“AB-AAB左复制”模式，将音乐的时间结构（如曲式）视为一个动态过程的产物：有机生长，从而体现了有机主义的观点。摘要：Driven by the term "the arrow of time" as a general topic, the article develops a musical discussion by referring to the etymological origin of the term: philosophy (epistemology) and physics (thermodynamics). In particular, the article explores two specific conditions: distinguishability and unique orientability, from which the article derives respective musical propositions and case studies. For the distinguishability condition, the article focuses on the "recurrence" in music and tries to interpret Bach's Christmas Oratorio from the perspective of "birth/resurrection". For the unique orientability condition, the article discusses the process of delaying the climax, thereby proposing "AB-AAB left-replication" model, implying an organicist view by treating the temporal structure of music (e.g. form) as the product of a dynamic process: organic growth.【4】 AQUALLM: Audio Question Answering Data Generation Using Large Language Models标题：AQUALLM：使用大型语言模型生成音频问答数据链接：https://arxiv.org/abs/2312.17343作者：Swarup Ranjan Behera,Krishna Mohan Injeti,Jaya Sai Kiran Patibandla,Praveen Kumar Pokala,Balakrishna Reddy Pailla摘要：音频问答（AQA）是机器分析音频信号和自然语言问题以产生精确的自然语言答案的关键任务。当目标是AQA系统的精度时，拥有高质量，多样化和广泛的AQA数据集的重要性不能被夸大。虽然开发准确和有效的AQA模型一直受到关注，但为手头的特定任务创建高质量，多样化和广泛的数据集并没有引起相当大的关注。为了应对这一挑战，这项工作作出了几项贡献。我们引入了一个可扩展的AQA数据生成管道，表示为AQA LM框架，它依赖于大型语言模型（LLM）。该框架利用现有的音频字幕注释并结合最先进的LLM来生成扩展的高质量AQA数据集。此外，我们还为AQA提供了三个广泛而高质量的基准数据集，为AQA研究的进展做出了重大贡献。与现有的最新技术相比，在所提出的数据集上训练的AQA模型设置了更好的基准。此外，与使用人类注释的AQA数据训练的模型相比，在我们的数据集上训练的模型表现出更强的泛化能力。代码和数据集可以在GitHub上访问~\footnote{\url{https：//github.com/swarupbehera/AuberLM}}。摘要：Audio Question Answering (AQA) constitutes a pivotal task in which machines analyze both audio signals and natural language questions to produce precise natural language answers. The significance of possessing high-quality, diverse, and extensive AQA datasets cannot be overstated when aiming for the precision of an AQA system. While there has been notable focus on developing accurate and efficient AQA models, the creation of high-quality, diverse, and extensive datasets for the specific task at hand has not garnered considerable attention. To address this challenge, this work makes several contributions. We introduce a scalable AQA data generation pipeline, denoted as the AQUALLM framework, which relies on Large Language Models (LLMs). This framework utilizes existing audio-caption annotations and incorporates state-of-the-art LLMs to generate expansive, high-quality AQA datasets. Additionally, we present three extensive and high-quality benchmark datasets for AQA, contributing significantly to the progression of AQA research. AQA models trained on the proposed datasets set superior benchmarks compared to the existing state-of-the-art. Moreover, models trained on our datasets demonstrate enhanced generalizability when compared to models trained using human-annotated AQA data. Code and datasets will be accessible on GitHub~\footnote{\url{https://github.com/swarupbehera/AQUALLM}}.【5】 Revolutionizing Personalized Voice Synthesis: The Journey towards Emotional and Individual Authenticity with DIVSE (Dynamic Individual Voice Synthesis Engine)标题：革命性的个性化语音合成：使用DIVSE(动态个人语音合成引擎)迈向情感和个人真实性的旅程链接：https://arxiv.org/abs/2312.17281作者：Fan Shi备注：8 pages摘要：这篇全面的论文深入研究了人工智能（AI）中个性化语音合成的最前沿，重点介绍了动态个人语音合成引擎（DIVSE）。DIVSE代表了文本到语音（TTS）技术的突破性飞跃，独特地专注于适应和个性化语音输出，以匹配个人的声音特征。这项研究强调了当前人工智能生成的声音的差距，虽然技术先进，但在复制人类语音固有的独特个性和表达能力方面存在不足。它概述了个性化语音合成的挑战和进步，强调情感表现力，口音和方言变化的重要性，并捕捉个人的语音特征。DIVSE的架构非常详细，展示了它的三个核心组件：语音特征学习模块（VCLM），情绪音调和口音适应模块（ETAAM）和动态语音合成引擎（DSSE）。DIVSE的创新方法在于其自适应学习能力，该能力随着时间的推移而发展，以根据特定的用户特征定制语音输出。本文提出了一个严格的实验设置，利用公认的数据集和个性化指标，如平均意见分数（MOS）和情感对齐分数，以验证DIVSE的优越性超过主流模型。结果描绘了在人工智能生成的声音中实现更高个性化和情感共鸣方面的明显进步。摘要：This comprehensive paper delves into the forefront of personalized voice synthesis within artificial intelligence (AI), spotlighting the Dynamic Individual Voice Synthesis Engine (DIVSE). DIVSE represents a groundbreaking leap in text-to-voice (TTS) technology, uniquely focusing on adapting and personalizing voice outputs to match individual vocal characteristics. The research underlines the gap in current AI-generated voices, which, while technically advanced, fall short in replicating the unique individuality and expressiveness intrinsic to human speech. It outlines the challenges and advancements in personalized voice synthesis, emphasizing the importance of emotional expressiveness, accent and dialect variability, and capturing individual voice traits. The architecture of DIVSE is meticulously detailed, showcasing its three core components: Voice Characteristic Learning Module (VCLM), Emotional Tone and Accent Adaptation Module (ETAAM), and Dynamic Speech Synthesis Engine (DSSE). The innovative approach of DIVSE lies in its adaptive learning capability, which evolves over time to tailor voice outputs to specific user traits. The paper presents a rigorous experimental setup, utilizing accepted datasets and personalization metrics like Mean Opinion Score (MOS) and Emotional Alignment Score, to validate DIVSE's superiority over mainstream models. The results depict a clear advancement in achieving higher personalization and emotional resonance in AI-generated voices.【6】 Stateful FastConformer with Cache-based Inference for Streaming Automatic Speech Recognition标题：基于Cache推理的有状态FastConformer流媒体自动语音识别链接：https://arxiv.org/abs/2312.17279作者：Vahid Noroozi,Somshubra Majumdar,Ankur Kumar,Jagadeesh Balam,Boris Ginsburg备注：Shorter version accepted to ICASSP 2024摘要：在本文中，我们提出了一个高效，准确的流式语音识别模型的基础上FastConformer架构。我们通过以下方式调整FastConformer架构用于流应用：（1）约束编码器中的前瞻和过去上下文，以及（2）引入激活缓存机制，使非自回归编码器能够在推理期间进行自回归操作。所提出的模型是经过深思熟虑的设计，以消除训练和推理时间之间的准确性差距，这是常见的许多流式模型。此外，我们提出的编码器与各种解码器配置，包括连接时间分类（CTC）和RNN传感器（RNNT）解码器。此外，我们引入了一种混合CTC/RNNT架构，该架构利用CTC和RNNT解码器共享编码器来提高准确性并节省计算。我们在LibriSpeech数据集和多域大规模数据集上评估了所提出的模型，并证明与传统的缓冲流模型基线相比，它可以以更低的延迟和推理时间实现更好的准确性。我们还表明，用多个延迟训练模型可以比单延迟模型实现更好的准确性，同时它使我们能够用单个模型支持多个延迟。我们的实验还表明，混合架构不仅加快了CTC解码器的收敛速度，而且还提高了流模型的准确性相比，单一的解码器模型。摘要：In this paper, we propose an efficient and accurate streaming speech recognition model based on the FastConformer architecture. We adapted the FastConformer architecture for streaming applications through: (1) constraining both the look-ahead and past contexts in the encoder, and (2) introducing an activation caching mechanism to enable the non-autoregressive encoder to operate autoregressively during inference. The proposed model is thoughtfully designed in a way to eliminate the accuracy disparity between the train and inference time which is common for many streaming models. Furthermore, our proposed encoder works with various decoder configurations including Connectionist Temporal Classification (CTC) and RNN-Transducer (RNNT) decoders. Additionally, we introduced a hybrid CTC/RNNT architecture which utilizes a shared encoder with both a CTC and RNNT decoder to boost the accuracy and save computation. We evaluate the proposed model on LibriSpeech dataset and a multi-domain large scale dataset and demonstrate that it can achieve better accuracy with lower latency and inference time compared to a conventional buffered streaming model baseline. We also showed that training a model with multiple latencies can achieve better accuracy than single latency models while it enables us to support multiple latencies with a single model. Our experiments also showed the hybrid architecture would not only speedup the convergence of the CTC decoder but also improves the accuracy of streaming models compared to single decoder models.机器翻译由腾讯交互翻译提供，仅供参考
每日语音速递
