今日论文合集：cs.SD语音20篇，eess.AS音频处理28篇。cs.SD语音【1】 Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator标题：基于Sidecar分离器的多人重叠语音识别与二值化统一建模链接：https://arxiv.org/abs/2305.16263作者：Lingwei Meng,Jiawen Kang,Mingyu Cui,Haibin Wu,Xixin Wu,Helen Meng机构：Dept. of Systems Engineering & Engineering Management, The Chinese University of Hong Kong, Graduate Institute of Communication Engineering, National Taiwan University备注：Accepted to INTERSPEECH 2023摘要：多说话人重叠语音对语音识别和语音日记化提出了重大挑战。最近的研究表明，这两个任务是相互依赖和互补的，激励我们探索一个统一的建模方法，以解决他们的上下文中的重叠语音。最近的一项研究提出了一种具有成本效益的方法，通过将Sidecar分离器插入到冻结的经过良好训练的ASR模型中，将单说话者自动语音识别（ASR）系统转换为多说话者系统。在此基础上扩展，我们将日志化分支纳入Sidecar，允许ASR和日志化的统一建模，开销可以忽略不计，只有768个参数。与LibriMix和LibriSpeechMix数据集上的基线相比，所提出的方法产生更好的ASR结果。此外，没有复杂的定制的日记任务，我们的方法实现了可接受的日记结果的两个扬声器子集的CALLHOME只有几个适应步骤。摘要：Multi-talker overlapped speech poses a significant challenge for speech recognition and diarization. Recent research indicated that these two tasks are inter-dependent and complementary, motivating us to explore a unified modeling method to address them in the context of overlapped speech. A recent study proposed a cost-effective method to convert a single-talker automatic speech recognition (ASR) system into a multi-talker one, by inserting a Sidecar separator into the frozen well-trained ASR model. Extending on this, we incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters. The proposed method yields better ASR results compared to the baseline on LibriMix and LibriSpeechMix datasets. Moreover, without sophisticated customization on the diarization task, our method achieves acceptable diarization results on the two-speaker subset of CALLHOME with only a few adaptation steps.【2】 VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation标题：Viola：用于语音识别、合成和翻译的统一编解码器语言模型链接：https://arxiv.org/abs/2305.16107作者：Tianrui Wang,Long Zhou,Ziqiang Zhang,Yu Wu,Shujie Liu,Yashesh Gaur,Zhuo Chen,Jinyu Li,Furu Wei机构：Microsoft备注：Working in progress摘要：最近的研究表明，在不同模态的各种任务中，模型架构、训练目标和推理方法都有很大的收敛。在本文中，我们提出了VioLA，一个单一的自回归Transformer解码器的网络，统一了各种跨模态的任务，涉及语音和文本，如语音到文本，文本到文本，文本到语音，语音到语音的任务，作为一个条件的编解码器语言模型任务，通过多任务学习框架。为了实现这一点，我们首先使用离线神经编解码器编码器将所有语音话语转换为离散令牌（类似于文本数据）。通过这种方式，所有这些任务都被转换为基于令牌的序列转换问题，这可以自然地用一个条件语言模型来处理。我们进一步整合任务ID（TID）和语言ID（LID）到所提出的模型，以提高处理不同的语言和任务的建模能力。实验结果表明，所提出的VioLA模型可以很好地支持单模态和跨模态任务，而仅解码器模型的性能与强基线模型相当，甚至更好。摘要：Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.【3】 End-to-End Simultaneous Speech Translation with Differentiable Segmentation标题：端到端可区分分词的同声翻译链接：https://arxiv.org/abs/2305.16093作者：Shaolei Zhang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：Accepted at ACL 2023 findings摘要：端到端同步语音翻译（SimulST）在接收流式语音输入（也称为“语音输入”）的同时输出翻译。流式语音翻译），因此需要对语音输入进行分段，然后基于当前接收的语音进行翻译。然而，在不利时刻分割语音输入可能破坏声学完整性并且不利地影响翻译模型的性能。因此，学习在那些有利于翻译模型产生高质量翻译的时刻分割语音输入是SimulST的关键。现有的SimulST方法，无论是使用固定长度的分割或外部分割模型，总是分离分割从底层的翻译模型，其中的差距导致分割结果，不一定有利于翻译过程。在本文中，我们提出了可区分分割（DiSeg）的SimulST直接学习分割从底层的翻译模型。DiSeg通过提出的期望训练将硬分割变成可区分的，使其能够与翻译模型联合训练，从而学习翻译有益的分割。实验结果表明，DiSeg达到国家的最先进的性能，并表现出优越的分割能力。摘要：End-to-end simultaneous speech translation (SimulST) outputs translation while receiving the streaming speech inputs (a.k.a. streaming speech translation), and hence needs to segment the speech inputs and then translate based on the current received speech. However, segmenting the speech inputs at unfavorable moments can disrupt the acoustic integrity and adversely affect the performance of the translation model. Therefore, learning to segment the speech inputs at those moments that are beneficial for the translation model to produce high-quality translation is the key to SimulST. Existing SimulST methods, either using the fixed-length segmentation or external segmentation model, always separate segmentation from the underlying translation model, where the gap results in segmentation outcomes that are not necessarily beneficial for the translation process. In this paper, we propose Differentiable Segmentation (DiSeg) for SimulST to directly learn segmentation from the underlying translation model. DiSeg turns hard segmentation into differentiable through the proposed expectation training, enabling it to be jointly trained with the translation model and thereby learn translation-beneficial segmentation. Experimental results demonstrate that DiSeg achieves state-of-the-art performance and exhibits superior segmentation capability.【4】 Visualizing data augmentation in deep speaker recognition标题：深度说话人识别中的可视化数据增强链接：https://arxiv.org/abs/2305.16070作者：Pengqi Li,Lantian Li,Askar Hamdulla,Dong Wang机构：School of Information Science and Engineering, Xinjiang University, China, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China, Center for Speech and Language Technologies, Tsinghua University, China备注：to be published in INTERSPEECH 2023摘要：可视化对于理解神经网络的内部机制具有重要价值。以前的工作发现，LayerCAM是一个可靠的可视化工具，用于深度扬声器模型。在本文中，我们使用LayerCAM来分析广泛采用的数据增强（DA）方法，以了解它如何导致模型的鲁棒性。我们在VoxCeleb1说话人识别数据集上进行了实验，结果表明，香草和基于激活的（Act）DA方法都增强了对干扰的鲁棒性，其中Act DA始终优于其他方法。LayerCAM的可视化表明，DA可以帮助模型学习删除被干扰破坏的时间频率（TF）频段。“学会删除”的行为解释了为什么DA模型比干净的模型更强大，以及为什么当干扰是非目标语音时，行为DA优于香草DA。然而，LayerCAM仍然不能清楚地解释ACT DA在其他情况下的优越性，建议进一步研究。摘要：Visualization is of great value in understanding the internal mechanisms of neural networks. Previous work found that LayerCAM is a reliable visualization tool for deep speaker models. In this paper, we use LayerCAM to analyze the widely-adopted data augmentation (DA) approach, to understand how it leads to model robustness. We conduct experiments on the VoxCeleb1 dataset for speaker identification, which shows that both vanilla and activation-based (Act) DA approaches enhance robustness against interference, with Act DA being consistently superior. Visualization with LayerCAM suggests DA helps models learn to delete temporal-frequency (TF) bins that are corrupted by interference. The `learn to delete' behavior explained why DA models are more robust than clean models, and why the Act DA is superior over the vanilla DA when the interference is nontarget speech. However, LayerCAM still cannot clearly explain the superiority of Act DA in other situations, suggesting further research.【5】 CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition标题：CN-Celeb-AV：一种用于人物识别的多类型视听数据集链接：https://arxiv.org/abs/2305.16049作者：Lantian Li,Xiaolou Li,Haoyu Jiang,Chen Chen,Ruihai Hou,Dong Wang机构：Center for Speech and Language Technologies, BNRist, Tsinghua University, China, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China, ∗Corresponding authors:备注：to be published in INTERSPEECH 2023摘要：视听人识别（AVPR）受到了广泛的关注。然而，大多数数据集用于AVPR的研究，到目前为止，收集在受约束的环境中，因此不能反映真实世界的场景中的AVPR系统的真实性能。为了满足在无约束条件下研究AVPR的要求，本文提出了一个多流派AVPR数据集收集“在野外”，命名为CN-Celeb-AV。该数据集包含来自公共媒体的1，136人的超过42万个视频片段。特别是，我们更加强调两个现实世界的复杂性：（1）多种类型的数据;（2）部分信息段。将CN-Celeb-AV与两个流行的公共AVPR基准数据集进行了全面的研究，结果表明CN-Celeb-AV更符合真实世界的场景，可以被视为AVPR研究的新基准数据集。该数据集还涉及一个开发集，可用于提高AVPR系统在现实生活中的性能。该数据集对研究人员是免费的，可以从www.example.com下载http://cnceleb.org/。摘要：Audio-visual person recognition (AVPR) has received extensive attention. However, most datasets used for AVPR research so far are collected in constrained environments, and thus cannot reflect the true performance of AVPR systems in real-world scenarios. To meet the request for research on AVPR in unconstrained conditions, this paper presents a multi-genre AVPR dataset collected `in the wild', named CN-Celeb-AV. This dataset contains more than 420k video segments from 1,136 persons from public media. In particular, we put more emphasis on two real-world complexities: (1) data in multiple genres; (2) segments with partial information. A comprehensive study was conducted to compare CN-Celeb-AV with two popular public AVPR benchmark datasets, and the results demonstrated that CN-Celeb-AV is more in line with real-world scenarios and can be regarded as a new benchmark dataset for AVPR research. The dataset also involves a development set that can be used to boost the performance of AVPR systems in real-life situations. The dataset is free for researchers and can be downloaded from http://cnceleb.org/.【6】 Ordered and Binary Speaker Embedding标题：有序和二进制说话人嵌入链接：https://arxiv.org/abs/2305.16043作者：Jiaying Wang,Xianglong Wang,Namin Wang,Lantian Li,Dong Wang机构：Center for Speech and Language Technologies, BNRist, Tsinghua University, China, Huawei Cloud, China, Beijing University of Posts and Telecommunications, China备注：to be published in INTERSPEECH 2023摘要：现代说话人识别系统通过嵌入向量来表示话语。传统的嵌入向量是密集的和非结构性的。在本文中，我们提出了一种有序的二进制嵌入方法，通过嵌套的丢弃的嵌入向量的尺寸排序，并通过伯努利采样的二进制码转换排序的向量。由此产生的有序二进制代码提供了一些重要的优点，如层次聚类，减少内存的使用，和快速检索。这些优点进行了实证验证的说话人识别任务与VoxCeleb和CN-Celeb数据集的综合实验。摘要：Modern speaker recognition systems represent utterances by embedding vectors. Conventional embedding vectors are dense and non-structural. In this paper, we propose an ordered binary embedding approach that sorts the dimensions of the embedding vector via a nested dropout and converts the sorted vectors to binary codes via Bernoulli sampling. The resultant ordered binary codes offer some important merits such as hierarchical clustering, reduced memory usage, and fast retrieval. These merits were empirically verified by comprehensive experiments on a speaker identification task with the VoxCeleb and CN-Celeb datasets.【7】 Latent Diffusion Model Based Foley Sound Generation System For DCASE Challenge 2023 Task 7标题：基于潜在扩散模型的DCASE挑战赛2023任务7的Foley声音生成系统链接：https://arxiv.org/abs/2305.15905作者：Yi Yuan,Haohe Liu,Xubo Liu,Xiyuan Kang,Mark D. Plumbley,Wenwu Wang机构：University of Surrey, Guildford, United Kingdom备注：DCASE 2023 task 7 technical report摘要：Foley声音生成旨在合成多媒体内容的背景声音，其涉及使用专门技术对声音效果进行计算建模。在这项工作中，我们为DCASE 2023挑战任务7提出了一个基于扩散的生成模型：Foley声音合成所提出的系统是基于AudioLDM，这是一个基于扩散的文本到音频生成模型。为了缓解任务7训练集的数据稀缺性，我们的模型最初使用大规模数据集进行训练，并通过迁移学习将其下游到这个DCASE任务中。我们已经观察到，由编码器提取的特征可以显著地影响生成模型的性能。因此，我们通过利用具有由大型语言模型获得的相关文本嵌入特征的输入标签来改进结果，即，对比语言-音频预训练（CLAP）。此外，我们利用过滤策略来进一步细化输出，即。通过从根据声音和目标标签之间的相似性得分生成的候选剪辑中选择最佳结果。整个系统在所有七个不同类别中平均达到4.765的Frechet音频距离（FAD）分数，显著优于达到9.7的FAD分数的基线系统。摘要：Foley sound generation aims to synthesise the background sound for multimedia content, which involves computationally modelling sound effects with specialized techniques. In this work, we proposed a diffusion based generative model for DCASE 2023 challenge task 7: Foley Sound Synthesis. The proposed system is based on AudioLDM, which is a diffusion-based text-to-audio generation model. To alleviate the data scarcity of the task 7 training set, our model is initially trained with large-scale datasets and downstream into this DCASE task via transfer learning. We have observed that the feature extracted by the encoder can significantly affect the performance of the generation model. Hence, we improve the results by leveraging the input label with related text embedding features obtained by a large language model, i.e., contrastive language-audio pretraining (CLAP). In addition, we utilize a filtering strategy to further refine the output, i.e. by selecting the best results from the candidate clips generated in terms of the similarity score between the sound and target labels. The overall system achieves a Frechet audio distance (FAD) score of 4.765 on average among all seven different classes, substantially outperforming the baseline system which achieves a FAD score of 9.7.【8】 Room Impulse Response Estimation in a Multiple Source Environment标题：多源环境下的房间脉冲响应估计链接：https://arxiv.org/abs/2305.15898作者：Kyungyun Lee,Jeonghun Seo,Keunwoo Choi,Sangmoon Lee,Ben Sangbae Chon机构：Gaudio Lab, Inc., Seoul, South Korea备注：2023 AES International Conference on Spatial and Immersive Audio摘要：在现实世界的声学场景中，房间中通常存在多个声源。这些声源位于不同的位置，并产生从多个方向到达收听者的声音。房间中多个源的存在在估计房间脉冲响应（RIR）中产生了新的挑战，因为每个源具有取决于其位置和取向的唯一RIR。因此，当输入信号是多个混响源的混合时，出现了确定应该预测哪个RIR以及如何预测它的问题。为了解决这些问题，我们提出了一个新的任务，预测一个房间在多源环境中的“代表性”RIR，并提出了一种训练方法来实现这一目标。与在单源环境中训练的模型相比，无论环境中的源数量如何，我们的方法都显示出强大的性能。摘要：In real-world acoustic scenarios, there often are multiple sound sources present in a room. These sources are situated in various locations and produce sounds that reach the listener from multiple directions. The presence of multiple sources in a room creates new challenges in estimating the room impulse response (RIR) as each source has a unique RIR, dependent on its location and orientation. Therefore, issues of determining which RIR should be predicted and how to predict it arise, when the input signal is a mixture of multiple reverberated sources. To address these, we propose a new task of predicting a "representative" RIR for a room in a multiple source environment and present a training method to achieve this goal. In contrast to the model trained in a single source environment, our method shows robust performance, regardless of the number of sources in the environment.【9】 Anomalous Sound Detection Based on Sound Separation标题：基于声音分离的异常声音检测链接：https://arxiv.org/abs/2305.15859作者：Kanta Shimonishi,Kota Dohi,Yohei Kawaguchi机构：Ritsumeikan University, Japan, Hitachi, Ltd., Japan备注：Accepted to INTERSPEECH2023摘要：提出了一种基于声音分离的无监督异常声音检测方法。在工厂环境中，背景噪声和非客观声音掩盖了所需的机器声音，使得检测异常声音具有挑战性。因此，期望在检测系统中使用不与背景噪声或非目的声音混合的声音。我们比较了我们提出的方法的两个版本，一个使用声音分离作为预处理步骤，另一个使用基于分离的离群值暴露，使用两个分离的声音之间的误差。基于正常声音和异常声音之间的分离性能差异影响检测结果的假设，在两个版本中使用特定于特定产品类型的声音分离模型。实验结果表明，该方法提高了异常声音检测性能的所有机器ID，实现了39%的最大改善。摘要：This paper proposes an unsupervised anomalous sound detection method using sound separation. In factory environments, background noise and non-objective sounds obscure desired machine sounds, making it challenging to detect anomalous sounds. Therefore, using sounds not mixed with background noise or non-purpose sounds in the detection system is desirable. We compared two versions of our proposed method, one using sound separation as a pre-processing step and the other using separation-based outlier exposure that uses the error between two separated sounds. Based on the assumption that differences in separation performance between normal and anomalous sounds affect detection results, a sound separation model specific to a particular product type was used in both versions. Experimental results indicate that the proposed method improved anomalous sound detection performance for all Machine IDs, achieving a maximum improvement of 39%.【10】 Svarah: Evaluating English ASR Systems on Indian Accents标题：Svarah：根据印度口音评估英语ASR系统链接：https://arxiv.org/abs/2305.15760作者：Tahir Javed,Sakshi Joshi,Vignesh Nagarajan,Sai Sundaresan,Janki Nawale,Abhigyan Raman,Kaushal Bhogale,Pratyush Kumar,Mitesh M. Khapra机构：Indian Institute of Technology Madras, India, AI,Bharat, India, Microsoft, India摘要：印度是世界上第二大英语国家，拥有约1.3亿人口。因此，它是必要的，英语的自动语音识别（ASR）系统应评估印度口音。不幸的是，印度人在现有的英语ASR基准中发现非常差的表示，例如LibriSpeech，Switchboard，Speech Accent Archive等。在这项工作中，我们通过创建Svarah来解决这一差距，这是一个基准，其中包含来自印度65个地理位置的117名发言者的9.6小时转录的英语音频，从而产生了各种各样的口音。Svarah包括阅读语音和自发会话数据两者，覆盖各种领域，诸如历史、文化、旅游等。确保词汇的多样性。我们评估了6个开源ASR模型和2个商业ASR系统在Svarah上，并表明印度口音有明显的改进空间。Svarah以及我们所有的代码都将公开。摘要：India is the second largest English-speaking country in the world with a speaker base of roughly 130 million. Thus, it is imperative that automatic speech recognition (ASR) systems for English should be evaluated on Indian accents. Unfortunately, Indian speakers find a very poor representation in existing English ASR benchmarks such as LibriSpeech, Switchboard, Speech Accent Archive, etc. In this work, we address this gap by creating Svarah, a benchmark that contains 9.6 hours of transcribed English audio from 117 speakers across 65 geographic locations throughout India, resulting in a diverse range of accents. Svarah comprises both read speech and spontaneous conversational data, covering various domains, such as history, culture, tourism, etc., ensuring a diverse vocabulary. We evaluate 6 open source ASR models and 2 commercial ASR systems on Svarah and show that there is clear scope for improvement on Indian accents. Svarah as well as all our code will be publicly available.【11】 Towards Solving Cocktail-Party: The First Method to Build a Realistic Dataset with Ground Truths for Speech Separation标题：走向解决鸡尾酒会：利用基本事实构建语音分离的真实数据集的第一种方法链接：https://arxiv.org/abs/2305.15758作者：Rawad Melhem,Assef Jafar,Oumayma Al Dakkak机构：Towards Solving Cocktail-Party: The First Method to Build a, Realistic Dataset with Ground Truths for Speech Separation, ), Institute for Applied Sciences and Technology, Damascus, Syria, Higher, (摘要：语音分离在诸如人机交互、助听器设备和自动会议转录等实际应用中非常重要。近年来，基于深度学习的解决方案出现了重大改进。事实上，人们已经注意到使用合成混合数据集的监督学习方法，尽管它们并不代表真实世界的混合物。建立真实数据集的困难导致研究人员使用无监督学习方法，因为它们能够直接处理真实的混合物。无监督学习方法的结果仍然不令人信服。在本文中，介绍了一种方法来创建一个真实的数据集与地面真值源的语音分离。设计现实数据集的主要挑战是扬声器信号的地面实况不可用。为了解决这个问题，我们提出了一种方法，同时记录两个扬声器，并获得每个地面真相。我们提出了一种使用基于双向门控递归单元（BGRU）和聚类算法的深度学习模型对现实数据集进行基准测试的方法。实验表明，我们提出的数据集改进SI-SDR（标度不变信号失真比）由1.65 dB和PESQ（语音质量的感知评价）由约0.5。我们还评估了我们的方法在麦克风和扬声器之间不同距离下的有效性，发现它提高了学习模型的稳定性。摘要：Speech separation is very important in real-world applications such as human-machine interaction, hearing aids devices, and automatic meeting transcription. In recent years, a significant improvement occurred towards the solution based on deep learning. In fact, much attention has been drawn to supervised learning methods using synthetic mixtures datasets despite their being not representative of real-world mixtures. The difficulty in building a realistic dataset led researchers to use unsupervised learning methods, because of their ability to handle realistic mixtures directly. The results of unsupervised learning methods are still unconvincing. In this paper, a method is introduced to create a realistic dataset with ground truth sources for speech separation. The main challenge in designing a realistic dataset is the unavailability of ground truths for speakers signals. To address this, we propose a method for simultaneously recording two speakers and obtaining the ground truth for each. We present a methodology for benchmarking our realistic dataset using a deep learning model based on Bidirectional Gated Recurrent Units (BGRU) and clustering algorithm. The experiments show that our proposed dataset improved SI-SDR (Scale Invariant Signal to Distortion Ratio) by 1.65 dB and PESQ (Perceptual Evaluation of Speech Quality) by approximately 0.5. We also evaluated the effectiveness of our method at different distances between the microphone and the speakers and found that it improved the stability of the learned model.【12】 Efficient Neural Music Generation标题：高效的神经音乐生成链接：https://arxiv.org/abs/2305.15719作者：Max W. Y. Lam,Qiao Tian,Tang Li,Zongyu Yin,Siyuan Feng,Ming Tu,Yuliang Ji,Rui Xia,Mingbo Ma,Xuchen Song,Jitong Chen,Yuping Wang,Yuxuan Wang机构：Speech, Audio & Music Intelligence (SAMI), ByteDance摘要：音乐生成的最新进展已经通过最先进的MusicLM显著地推进，其包括三个LM的层次结构，分别用于语义、粗糙声学和精细声学建模。然而，用MusicLM进行采样需要通过这些LM逐个处理以获得细粒度的声学令牌，这使得它在计算上是昂贵的并且对于实时生成是禁止的。高效的音乐生成与MusicLM的质量相当仍然是一个重大的挑战。在本文中，我们提出了MeLoDy（M用于音乐; L表示LM; D表示扩散），这是一种LM引导的扩散模型，可以生成最先进质量的音乐音频，同时在MusicLM中分别减少95.7%或99.6%的前向传递，用于采样10秒或30秒的音乐。MeLoDy继承了MusicLM的最高级别LM进行语义建模，并应用了一种新的双路径扩散（DPD）模型和音频VAE-GAN来有效地将条件语义令牌解码为波形。DPD提出了同时建模的粗和细的声学通过将语义信息有效地通过交叉注意在每个去噪步骤的潜在段。我们的实验结果表明，MeLoDy的优越性，不仅在其实际的采样速度和无限连续生成的优势，而且在其国家的最先进的音乐性，音频质量和文本相关性。 我们的样品可在www.example.com获得https://Efficient-MeLoDy.github.io/。摘要：Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the MusicLM requires processing through these LMs one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for a real-time generation. Efficient music generation with a quality on par with MusicLM remains a significant challenge. In this paper, we present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music audios of state-of-the-art quality meanwhile reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling 10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for semantic modeling, and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD is proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely continuable generation, but also in its state-of-the-art musicality, audio quality, and text correlation. Our samples are available at https://Efficient-MeLoDy.github.io/.【13】 Mixture-of-Expert Conformer for Streaming Multilingual ASR标题：用于流媒体多语言ASR的混合专家整形器链接：https://arxiv.org/abs/2305.15663作者：Ke Hu,Bo Li,Tara N. Sainath,Yu Zhang,Francoise Beaufays机构：Google LLC, USA备注：Accepted to Interspeech 2023摘要：具有大容量的端到端模型显著改善了多语言自动语音识别，但其计算成本对设备上的应用提出了挑战。我们提出了一个真正的多语言流整合混合专家（MoE）层，学习只激活一个子集的参数在训练和推理。MoE层由一个softmax门组成，它在前向传播中从许多专家中选择最好的两个专家。建议的MoE层提供了有效的推理，通过激活一个固定数量的参数，专家的数量增加。我们评估了一组12种语言的模型，并实现了平均11.9%的相对改善WER的基线。与使用地面实况信息的适配器模型相比，我们的MoE模型实现了类似的WER，并激活了类似数量的参数，但没有任何语言信息。我们进一步表明，通过多语言浅融合，WER相对改善约3%。摘要：End-to-end models with large capacity have significantly improved multilingual automatic speech recognition, but their computation cost poses challenges for on-device applications. We propose a streaming truly multilingual Conformer incorporating mixture-of-expert (MoE) layers that learn to only activate a subset of parameters in training and inference. The MoE layer consists of a softmax gate which chooses the best two experts among many in forward propagation. The proposed MoE layer offers efficient inference by activating a fixed number of parameters as the number of experts increases. We evaluate the proposed model on a set of 12 languages, and achieve an average 11.9% relative improvement in WER over the baseline. Compared to an adapter model using ground truth information, our MoE model achieves similar WER and activates similar number of parameters but without any language information. We further show around 3% relative WER improvement by multilingual shallow fusion.【14】 Metamathematics of Algorithmic Composition标题：算法合成的元数学链接：https://arxiv.org/abs/2305.15601作者：Michael Gogins机构：when I was a returning undergraduate majoring in comparative religion atthe University of Washington备注：15 pages, 0 figures. Comments are very welcome摘要：这篇文章讲述了我个人对算法音乐作曲的数学基础有更深入理解的旅程。我没有花太多时间在作曲家使用的特定数学算法上;相反，我专注于一般性的问题，如基本的限制和可能性，通过类比与元逻辑，元数学和可计算性理论。我讨论了这些基础的算法组成的未来的影响。摘要：This essay recounts my personal journey towards a deeper understanding of the mathematical foundations of algorithmic music composition. I do not spend much time on specific mathematical algorithms used by composers; rather, I focus on general issues such as fundamental limits and possibilities, by analogy with metalogic, metamathematics, and computability theory. I discuss implications from these foundations for the future of algorithmic composition.【15】 Sound Design Strategies for Latent Audio Space Explorations Using Deep Learning Architectures标题：利用深度学习结构进行潜在音频空间探索的声音设计策略链接：https://arxiv.org/abs/2305.15571作者：Kıvanç Tatar,Kelsey Cotton,Daniel Bisig机构：Chalmers University of Technology, Sweden, Zurich University of the Arts, Switzerland备注：In Proceedings of Sound and Music Computing 2023摘要：近年来，深度学习在声音和音乐计算中的应用研究引起了人们的兴趣;然而，在这些新技术和如何将它们纳入现实世界的艺术实践之间仍然缺少一个环节。在这项工作中，我们探索了一种着名的深度学习架构，称为变分自动编码器（VAE）。这些架构已经在许多领域中用于生成潜在空间，其中数据点被组织成使得相似的数据点彼此更靠近。以前，VAE已经用于生成潜在音色空间或符号音乐例外的潜在空间。将VAE应用于音色的音频特征需要声码器将由网络生成的音色变换为音频信号，这在计算上是昂贵的。在这项工作中，我们直接将VAE应用于原始音频数据，同时绕过音频特征提取。这种方法允许从业者使用任何音频记录，同时通过数据集策展提供灵活性和对美学的控制。音频信号生成中的较低计算时间允许将原始音频方法并入实时应用中。在这项工作中，我们提出了三种策略，探索潜在的空间的音频和音色的声音设计应用。通过这样做，我们的目标是发起一场关于艺术方法和策略的对话，以利用声音和音乐实践中的潜在音频空间。摘要：The research in Deep Learning applications in sound and music computing have gathered an interest in the recent years; however, there is still a missing link between these new technologies and on how they can be incorporated into real-world artistic practices. In this work, we explore a well-known Deep Learning architecture called Variational Autoencoders (VAEs). These architectures have been used in many areas for generating latent spaces where data points are organized so that similar data points locate closer to each other. Previously, VAEs have been used for generating latent timbre spaces or latent spaces of symbolic music excepts. Applying VAE to audio features of timbre requires a vocoder to transform the timbre generated by the network to an audio signal, which is computationally expensive. In this work, we apply VAEs to raw audio data directly while bypassing audio feature extraction. This approach allows the practitioners to use any audio recording while giving flexibility and control over the aesthetics through dataset curation. The lower computation time in audio signal generation allows the raw audio approach to be incorporated into real-time applications. In this work, we propose three strategies to explore latent spaces of audio and timbre for sound design applications. By doing so, our aim is to initiate a conversation on artistic approaches and strategies to utilize latent audio spaces in sound and music practices.【16】 Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognition标题：弱监督语音预训练：以目标语音识别为例链接：https://arxiv.org/abs/2305.16286作者：Wangyou Zhang,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, China备注：Accepted by Interspeech; 5 pages, 1 figure, 3 tables摘要：基于自监督学习（Self-supervised learning，SSL）的语音预训练因其能够从大量未标记数据中提取丰富的表征而备受关注。另一方面，弱监督数据的使用较少用于语音预训练。为了填补这一空白，我们提出了一种弱监督语音预训练方法的基础上说话人感知的语音数据。它采用了类似的训练过程，广泛使用的掩蔽语音预测的SSL框架，同时将额外的目标说话人注册信息作为辅助输入。以这种方式，即使在存在高度重叠的干扰的情况下，学习的表示也被引导向目标说话者，从而允许对诸如目标语音识别之类的任务的潜在应用。我们在Libri 2 Mix和WSJ 0 - 2 mix数据集上的实验表明，与具有去噪能力的最先进SSL模型WavLM相比，该模型实现了更好的ASR性能。摘要：Self-supervised learning (SSL) based speech pre-training has attracted much attention for its capability of extracting rich representations learned from massive unlabeled data. On the other hand, the use of weakly-supervised data is less explored for speech pre-training. To fill this gap, we propose a weakly-supervised speech pre-training method based on speaker-aware speech data. It adopts a similar training procedure to the widely-used masked speech prediction based SSL framework, while incorporating additional target-speaker enrollment information as an auxiliary input. In this way, the learned representation is steered towards the target speaker even in the presence of highly overlapping interference, allowing potential applications to tasks such as target speech recognition. Our experiments on Libri2Mix and WSJ0-2mix datasets show that the proposed model achieves significantly better ASR performance compared to WavLM, the state-of-the-art SSL model with denoising capability.【17】 Transfer Learning for Personality Perception via Speech Emotion Recognition标题：基于语音情感识别的人格感知迁移学习链接：https://arxiv.org/abs/2305.16076作者：Yuanchao Li,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, UK备注：Accepted to INTERSPEECH 2023摘要：情感属性的整体感知是人类一种重要的感知能力。然而，这种能力是远远没有实现在当前的情感计算，因为不是所有的属性都得到很好的研究和他们的相互关系知之甚少。在这项工作中，我们研究了两个情感属性之间的关系：个性和情感，从迁移学习的角度来看。具体来说，我们转移基于Transformer和基于wav2vec的情感识别模型来感知跨语料库的语音个性。与以往的研究相比，我们的研究结果表明，迁移情绪识别对人格知觉是有效的。更重要的是，这允许更好地使用和探索小的个性语料库。我们还提供了新的发现，个性和情绪之间的关系，将有助于未来的研究整体影响识别。摘要：Holistic perception of affective attributes is an important human perceptual ability. However, this ability is far from being realized in current affective computing, as not all of the attributes are well studied and their interrelationships are poorly understood. In this work, we investigate the relationship between two affective attributes: personality and emotion, from a transfer learning perspective. Specifically, we transfer Transformer-based and wav2vec-based emotion recognition models to perceive personality from speech across corpora. Compared with previous studies, our results show that transferring emotion recognition is effective for personality perception. Moreoever, this allows for better use and exploration of small personality corpora. We also provide novel findings on the relationship between personality and emotion that will aid future research on holistic affect recognition.【18】 ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition标题：ASR与情绪语音：语音与情绪识别相互影响的词汇层面研究链接：https://arxiv.org/abs/2305.16065作者：Yuanchao Li,Zeyu Zhao,Ondrej Klejch,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, UK备注：Accepted to INTERSPEECH 2023摘要：在语音情感识别（SER）中，文本数据通常与音频信号一起使用以解决其固有的可变性。然而，在大多数研究中对人类注释文本的依赖阻碍了实用SER系统的发展。为了克服这一挑战，我们研究了自动语音识别（ASR）如何执行情感语音，通过分析情感语料库上的ASR性能，并检查ASR成绩单中的单词错误和置信度分数的分布，以深入了解情感如何影响ASR。我们使用四个ASR系统，即Kaldi ASR，wav2vec，Conformer和Whisper，以及三个语料库：IEMOCAP、MOSI和MELD以确保普遍性。此外，我们进行基于文本的SER ASR成绩单增加单词错误率调查ASR如何影响SER。本研究的目的是揭示ASR和SER之间的关系和相互影响，以促进ASR对情感言语的适应和SER在现实世界中的使用。摘要：In Speech Emotion Recognition (SER), textual data is often used alongside audio signals to address their inherent variability. However, the reliance on human annotated text in most research hinders the development of practical SER systems. To overcome this challenge, we investigate how Automatic Speech Recognition (ASR) performs on emotional speech by analyzing the ASR performance on emotion corpora and examining the distribution of word errors and confidence scores in ASR transcripts to gain insight into how emotion affects ASR. We utilize four ASR systems, namely Kaldi ASR, wav2vec, Conformer, and Whisper, and three corpora: IEMOCAP, MOSI, and MELD to ensure generalizability. Additionally, we conduct text-based SER on ASR transcripts with increasing word error rates to investigate how ASR affects SER. The objective of this study is to uncover the relationship and mutual impact of ASR and SER, in order to facilitate ASR adaptation to emotional speech and the use of SER in real world.【19】 DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion标题：DDDM-VC：解缠表示和先验混合的去耦合去噪扩散模型用于验证稳健的语音转换链接：https://arxiv.org/abs/2305.15816作者：Ha-Yeong Choi,Sang-Hoon Lee,Seong-Whan Lee机构：Department of Artificial Intelligence, Korea University, Seoul, Korea备注：23 pages, 10 figures, 17 tables, under review摘要：近年来，基于扩散的生成模型表现出强大的生成性能。然而，由于数据分布中存在许多属性，并且由于在生成过程的所有级别之间共享模型参数的若干限制，因此控制每个属性的特定样式仍然具有挑战性。为了解决上述问题，本文提出了解耦去噪扩散模型（DDDM）与解纠缠表示，它可以控制生成模型中每个属性的风格。我们将DDDM应用于语音转换（VC）任务以解决解开和控制每个语音属性（例如，语言信息、语调和音色）。首先，我们使用一个自我监督的表示来解开的语音表示。随后，DDDM被应用于从解纠缠的表示重新合成语音，以相对于每个属性进行去噪。此外，我们还提出了鲁棒的语音风格转移的先验混合，它使用的混合风格的转换表示作为扩散模型的先验分布。实验结果表明，我们的方法优于公开可用的VC模型。此外，我们表明，我们的方法提供了强大的生成性能，无论模型大小。音频样本可从www.example.com获得https://hayeong0.github.io/DDDM-VC-demo/。摘要：Diffusion-based generative models have exhibited powerful generative performance in recent years. However, as many attributes exist in the data distribution and owing to several limitations of sharing the model parameters across all levels of the generation process, it remains challenging to control specific styles for each attribute. To address the above problem, this paper presents decoupled denoising diffusion models (DDDMs) with disentangled representations, which can control the style for each attribute in generative models. We apply DDDMs to voice conversion (VC) tasks to address the challenges of disentangling and controlling each speech attribute (e.g., linguistic information, intonation, and timbre). First, we use a self-supervised representation to disentangle the speech representation. Subsequently, the DDDMs are applied to resynthesize the speech from the disentangled representations for denoising with respect to each attribute. Moreover, we also propose the prior mixup for robust voice style transfer, which uses the converted representation of the mixed style as a prior distribution for the diffusion models. The experimental results reveal that our method outperforms publicly available VC models. Furthermore, we show that our method provides robust generative performance regardless of the model size. Audio samples are available https://hayeong0.github.io/DDDM-VC-demo/.【20】 Spoofing Attacker Also Benefits from Self-Supervised Pretrained Model标题：欺骗攻击者还受益于自我监督的预训练模型链接：https://arxiv.org/abs/2305.15518作者：Aoi Ito,Shota Horiguchi机构：Hosei University, Hitachi, Ltd.备注：Accepted to INTERSPEECH 2023摘要：据报道，使用自监督学习的大规模预训练模型提高了语音反欺骗的性能。然而，攻击者侧也可以利用这样的模型。此外，由于从头开始训练这样的模型非常昂贵，因此通常使用互联网上的预训练模型，但是攻击者和防御者可能使用相同的预训练模型。本文研究了在攻击者可获得模型的条件下，预训练模型对反欺骗的改进是否成立。作为攻击者，我们训练了一个模型，增强了欺骗性的话语，使说话人嵌入提取器的基础上的预训练模型不能区分真正的和欺骗性的话语。实验结果表明，如果攻击者也使用预训练模型，则使用预训练模型获得的反欺骗模型的增益几乎消失。摘要：Large-scale pretrained models using self-supervised learning have reportedly improved the performance of speech anti-spoofing. However, the attacker side may also make use of such models. Also, since it is very expensive to train such models from scratch, pretrained models on the Internet are often used, but the attacker and defender may possibly use the same pretrained model. This paper investigates whether the improvement in anti-spoofing with pretrained models holds under the condition that the models are available to attackers. As the attacker, we train a model that enhances spoofed utterances so that the speaker embedding extractor based on the pretrained models cannot distinguish between bona fide and spoofed utterances. Experimental results show that the gains the anti-spoofing models obtained by using the pretrained models almost disappear if the attacker also makes use of the pretrained models.
eess.AS音频处理
【1】 Weakly-Supervised Speech Pre-training: A Case Study on Target Speech Recognition标题：弱监督语音预训练：以目标语音识别为例链接：https://arxiv.org/abs/2305.16286作者：Wangyou Zhang,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, China备注：Accepted by Interspeech; 5 pages, 1 figure, 3 tables摘要：基于自监督学习（Self-supervised learning，SSL）的语音预训练因其能够从大量未标记数据中提取丰富的表征而备受关注。另一方面，弱监督数据的使用较少用于语音预训练。为了填补这一空白，我们提出了一种弱监督语音预训练方法的基础上说话人感知的语音数据。它采用了类似的训练过程，广泛使用的掩蔽语音预测的SSL框架，同时将额外的目标说话人注册信息作为辅助输入。以这种方式，即使在存在高度重叠的干扰的情况下，学习的表示也被引导向目标说话者，从而允许对诸如目标语音识别之类的任务的潜在应用。我们在Libri 2 Mix和WSJ 0 - 2 mix数据集上的实验表明，与具有去噪能力的最先进SSL模型WavLM相比，该模型实现了更好的ASR性能。摘要：Self-supervised learning (SSL) based speech pre-training has attracted much attention for its capability of extracting rich representations learned from massive unlabeled data. On the other hand, the use of weakly-supervised data is less explored for speech pre-training. To fill this gap, we propose a weakly-supervised speech pre-training method based on speaker-aware speech data. It adopts a similar training procedure to the widely-used masked speech prediction based SSL framework, while incorporating additional target-speaker enrollment information as an auxiliary input. In this way, the learned representation is steered towards the target speaker even in the presence of highly overlapping interference, allowing potential applications to tasks such as target speech recognition. Our experiments on Libri2Mix and WSJ0-2mix datasets show that the proposed model achieves significantly better ASR performance compared to WavLM, the state-of-the-art SSL model with denoising capability.【2】 Classifying Rhoticity of /r/ in Speech Sound Disorder using Age-and-Sex Normalized Formants标题：用年龄和性别归一化格式对语音障碍中/r/的音高进行分类链接：https://arxiv.org/abs/2305.16111作者：Nina R Benway,Jonathan L Preston,Asif Salekin,Yi Xiao,Harshit Sharma,Tara McAllister机构：Communication Sciences and Disorders, Syracuse University, New York, USA, Haskins Laboratories, New Haven, Connecticut, USA, Electrical Engineering and Computer Science, Syracuse University, New York, USA备注：To appear in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2023摘要：发音错误检测工具可以增加对影响例如语音障碍的语音障碍的治疗途径。/r/。我们发现，年龄和性别归一化共振峰估计优于倒谱表示检测完全旋转与。derhotic /r/在PERCEPT-R语料库。在此特征集上训练的门控递归神经网络实现了平均测试参与者特定的F1分数=.81（{\sigma}x=.10，med = .83，n = 48），事后建模显示儿童年龄或性别没有显着影响。摘要：Mispronunciation detection tools could increase treatment access for speech sound disorders impacting, e.g., /r/. We show age-and-sex normalized formant estimation outperforms cepstral representation for detection of fully rhotic vs. derhotic /r/ in the PERCEPT-R Corpus. Gated recurrent neural networks trained on this feature set achieve a mean test participant-specific F1-score =.81 ({\sigma}x=.10, med = .83, n = 48), with post hoc modeling showing no significant effect of child age or sex.【3】 Acoustic-to-Articulatory Speech Inversion Features for Mispronunciation Detection of /r/ in Child Speech Sound Disorders标题：声学-节律语音倒置特征在儿童语音障碍/r/发音错误检测中的应用链接：https://arxiv.org/abs/2305.16085作者：Nina R Benway,Yashish M Siriwardena,Jonathan L Preston,Elaine Hitchcock,Tara McAllister,Carol Espy-Wilson机构：Communication Sciences and Disorders, Syracuse University, New York, USA, Electrical and Computer Engineering, University of Maryland College Park, Maryland, USA, Haskins Laboratories, New Haven, Connecticut, USA备注：*denotes equal contribution. To appear in Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH 2023摘要：声学到发音语音反转可以增强自动临床发音错误检测，以提供基于共振峰的发音错误检测算法无法获得的详细发音反馈;然而，在（1）儿童和（2）临床语音的情况下，在成人语音上训练的语音倒置系统的执行程度是不清楚的。在没有发音数据集的儿童与发音语音障碍，我们表明，分类器训练的道变量从声学发音语音反转达到或超过性能的国家的最先进的功能时，预测临床医生的判断旋转。索引术语：耳鸣，语音障碍，发音错误检测摘要：Acoustic-to-articulatory speech inversion could enhance automated clinical mispronunciation detection to provide detailed articulatory feedback unattainable by formant-based mispronunciation detection algorithms; however, it is unclear the extent to which a speech inversion system trained on adult speech performs in the context of (1) child and (2) clinical speech. In the absence of an articulatory dataset in children with rhotic speech sound disorders, we show that classifiers trained on tract variables from acoustic-to-articulatory speech inversion meet or exceed the performance of state-of-the-art features when predicting clinician judgment of rhoticity. Index Terms: rhotic, speech sound disorder, mispronunciation detection【4】 Transfer Learning for Personality Perception via Speech Emotion Recognition标题：基于语音情感识别的人格感知迁移学习链接：https://arxiv.org/abs/2305.16076作者：Yuanchao Li,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, UK备注：Accepted to INTERSPEECH 2023摘要：情感属性的整体感知是人类一种重要的感知能力。然而，这种能力是远远没有实现在当前的情感计算，因为不是所有的属性都得到很好的研究和他们的相互关系知之甚少。在这项工作中，我们研究了两个情感属性之间的关系：个性和情感，从迁移学习的角度来看。具体来说，我们转移基于Transformer和基于wav2vec的情感识别模型来感知跨语料库的语音个性。与以往的研究相比，我们的研究结果表明，迁移情绪识别对人格知觉是有效的。更重要的是，这允许更好地使用和探索小的个性语料库。我们还提供了新的发现，个性和情绪之间的关系，将有助于未来的研究整体影响识别。摘要：Holistic perception of affective attributes is an important human perceptual ability. However, this ability is far from being realized in current affective computing, as not all of the attributes are well studied and their interrelationships are poorly understood. In this work, we investigate the relationship between two affective attributes: personality and emotion, from a transfer learning perspective. Specifically, we transfer Transformer-based and wav2vec-based emotion recognition models to perceive personality from speech across corpora. Compared with previous studies, our results show that transferring emotion recognition is effective for personality perception. Moreoever, this allows for better use and exploration of small personality corpora. We also provide novel findings on the relationship between personality and emotion that will aid future research on holistic affect recognition.【5】 ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognition标题：ASR与情绪语音：语音与情绪识别相互影响的词汇层面研究链接：https://arxiv.org/abs/2305.16065作者：Yuanchao Li,Zeyu Zhao,Ondrej Klejch,Peter Bell,Catherine Lai机构：Centre for Speech Technology Research, University of Edinburgh, UK备注：Accepted to INTERSPEECH 2023摘要：在语音情感识别（SER）中，文本数据通常与音频信号一起使用以解决其固有的可变性。然而，在大多数研究中对人类注释文本的依赖阻碍了实用SER系统的发展。为了克服这一挑战，我们研究了自动语音识别（ASR）如何执行情感语音，通过分析情感语料库上的ASR性能，并检查ASR成绩单中的单词错误和置信度分数的分布，以深入了解情感如何影响ASR。我们使用四个ASR系统，即Kaldi ASR，wav2vec，Conformer和Whisper，以及三个语料库：IEMOCAP、MOSI和MELD以确保普遍性。此外，我们进行基于文本的SER ASR成绩单增加单词错误率调查ASR如何影响SER。本研究的目的是揭示ASR和SER之间的关系和相互影响，以促进ASR对情感言语的适应和SER在现实世界中的使用。摘要：In Speech Emotion Recognition (SER), textual data is often used alongside audio signals to address their inherent variability. However, the reliance on human annotated text in most research hinders the development of practical SER systems. To overcome this challenge, we investigate how Automatic Speech Recognition (ASR) performs on emotional speech by analyzing the ASR performance on emotion corpora and examining the distribution of word errors and confidence scores in ASR transcripts to gain insight into how emotion affects ASR. We utilize four ASR systems, namely Kaldi ASR, wav2vec, Conformer, and Whisper, and three corpora: IEMOCAP, MOSI, and MELD to ensure generalizability. Additionally, we conduct text-based SER on ASR transcripts with increasing word error rates to investigate how ASR affects SER. The objective of this study is to uncover the relationship and mutual impact of ASR and SER, in order to facilitate ASR adaptation to emotional speech and the use of SER in real world.【6】 The Power of Prosody and Prosody of Power: An Acoustic Analysis of Finnish Parliamentary Speech标题：韵律的力量和韵律的力量--芬兰议会演讲的声学分析链接：https://arxiv.org/abs/2305.16040作者：Martti Vainio,Antti Suni,Juraj Šimko,Sofoklis Kakouros机构：University of Helsinki, Finland摘要：议会录音为研究政治家如何利用演讲来传达信息和影响听众提供了丰富的数据来源。这为研究政治家如何使用演讲，特别是韵律，以实现他们的目标提供了一个独特的背景。在这里，我们分析了议会演讲在芬兰议会之间的2008年至2020年的语料库，并强调与基于信号的功能相对于不同的录音条件和语料库设计的鲁棒性的方法考虑。我们还提出了长期变化的结果，有关发言人的地位，就其政党是在政府或反对派。看着大规模的基本频率的平均值-一个强大的韵律功能-我们发现系统的语音韵律的变化与尊重反对党的地位和选举期间。议会成员在任期开始时或处于反对党地位时的f0较高，反映了不同程度的紧迫性。摘要：Parliamentary recordings provide a rich source of data for studying how politicians use speech to convey their messages and influence their audience. This provides a unique context for studying how politicians use speech, especially prosody, to achieve their goals. Here we analyzed a corpus of parliamentary speeches in the Finnish parliament between the years 2008-2020 and highlight methodological considerations related to the robustness of signal based features with respect to varying recording conditions and corpus design. We also present results of long term changes pertaining to speakers' status with respect to their party being in government or in opposition. Looking at large scale averages of fundamental frequency - a robust prosodic feature - we found systematic changes in speech prosody with respect opposition status and the election term. Reflecting a different level of urgency, members of the parliament have higher f0 at the beginning of the term or when they are in opposition.【7】 Knowledge Distillation for Neural Transducer-based Target-Speaker ASR: Exploiting Parallel Mixture/Single-Talker Speech Data标题：基于神经传感器的目标说话人ASR知识提取：利用并行混合/单人语音数据链接：https://arxiv.org/abs/2305.15971作者：Takafumi Moriya,Hiroshi Sato,Tsubasa Ochiai,Marc Delcroix,Takanori Ashihara,Kohei Matsuura,Tomohiro Tanaka,Ryo Masumura,Atsunori Ogawa,Taichi Asami机构：NTT Corporation, Japan备注：Accepted to Interspeech 2023摘要：基于神经换能器（RNNT）的目标说话人语音识别（TS-RNNT）直接从多说话人混合物中转录目标说话人的语音。这是一种很有前途的方法，因为它不会产生额外的计算成本的目标语音提取前端，这是一个关键的障碍，快速响应。在给定输入语音的情况下，端到端地训练TS-RNNT（即，混合物和注册演讲）和参考转录。训练混合通常通过混合单个说话者信号来模拟，但是常规TS-RNNT训练不利用单个说话者信号。本文提出了利用知识蒸馏（KD）的并行混合/单说话人的语音数据。我们提出的KD方案使用了一个RNNT系统预训练的目标单说话者的语音输入生成伪标签的TS-RNNT训练。实验结果表明，TS-RNNT系统训练与建议的KD计划优于基线TS-RNNT。摘要：Neural transducer (RNNT)-based target-speaker speech recognition (TS-RNNT) directly transcribes a target speaker's voice from a multi-talker mixture. It is a promising approach for streaming applications because it does not incur the extra computation costs of a target speech extraction frontend, which is a critical barrier to quick response. TS-RNNT is trained end-to-end given the input speech (i.e., mixtures and enrollment speech) and reference transcriptions. The training mixtures are generally simulated by mixing single-talker signals, but conventional TS-RNNT training does not utilize single-speaker signals. This paper proposes using knowledge distillation (KD) to exploit the parallel mixture/single-talker speech data. Our proposed KD scheme uses an RNNT system pretrained with the target single-talker speech input to generate pseudo labels for the TS-RNNT training. Experimental results show that TS-RNNT systems trained with the proposed KD scheme outperform a baseline TS-RNNT.【8】 Improving Scheduled Sampling for Neural Transducer-based ASR标题：基于神经传感器的ASR定时采样的改进链接：https://arxiv.org/abs/2305.15958作者：Takafumi Moriya,Takanori Ashihara,Hiroshi Sato,Kohei Matsuura,Tomohiro Tanaka,Ryo Masumura机构：NTT Corporation, Japan备注：Accepted to ICASSP 2023摘要：递归神经网络换能器（RNNT）是一种很有前途的方法，自动语音识别（ASR）的预测网络，自回归考虑语言方面的介绍。为了训练自回归部分，地面实况令牌被用作先前输出令牌的替代，这导致对不正确的过去令牌的鲁棒性不足;解码中的识别错误导致进一步的错误。调度采样（SS）是一种通过用模型生成的实际输出随机替换一些地面真值令牌来训练自回归模型对过去误差鲁棒的技术。SS减轻了训练和解码步骤之间的间隙，称为暴露偏差，并且它通常用于注意编码器-解码器训练。然而，SS还没有得到充分的审查RNNT，因为在应用SS RNNT由于复杂的RNNT输出形式的困难。在本文中，我们提出了适合RNNT的SS方法。我们的SS方法对从RNNT本身的分布生成的令牌进行采样，即内部语言模型或RNNT输出。在三个数据集上的实验证实，用我们的SS方法训练的RNNT达到了最佳的ASR性能。特别是，在日本ASR任务，我们最好的系统优于以前的国家的最先进的替代品。摘要：The recurrent neural network-transducer (RNNT) is a promising approach for automatic speech recognition (ASR) with the introduction of a prediction network that autoregressively considers linguistic aspects. To train the autoregressive part, the ground-truth tokens are used as substitutions for the previous output token, which leads to insufficient robustness to incorrect past tokens; a recognition error in the decoding leads to further errors. Scheduled sampling (SS) is a technique to train autoregressive model robustly to past errors by randomly replacing some ground-truth tokens with actual outputs generated from a model. SS mitigates the gaps between training and decoding steps, known as exposure bias, and it is often used for attentional encoder-decoder training. However SS has not been fully examined for RNNT because of the difficulty in applying SS to RNNT due to the complicated RNNT output form. In this paper we propose SS approaches suited for RNNT. Our SS approaches sample the tokens generated from the distiribution of RNNT itself, i.e. internal language model or RNNT outputs. Experiments in three datasets confirm that RNNT trained with our SS approach achieves the best ASR performance. In particular, on a Japanese ASR task, our best system outperforms the previous state-of-the-art alternative.【9】 DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion标题：DDDM-VC：解缠表示和先验混合的去耦合去噪扩散模型用于验证稳健的语音转换链接：https://arxiv.org/abs/2305.15816作者：Ha-Yeong Choi,Sang-Hoon Lee,Seong-Whan Lee机构：Department of Artificial Intelligence, Korea University, Seoul, Korea备注：23 pages, 10 figures, 17 tables, under review摘要：近年来，基于扩散的生成模型表现出强大的生成性能。然而，由于数据分布中存在许多属性，并且由于在生成过程的所有级别之间共享模型参数的若干限制，因此控制每个属性的特定样式仍然具有挑战性。为了解决上述问题，本文提出了解耦去噪扩散模型（DDDM）与解纠缠表示，它可以控制生成模型中每个属性的风格。我们将DDDM应用于语音转换（VC）任务以解决解开和控制每个语音属性（例如，语言信息、语调和音色）。首先，我们使用一个自我监督的表示来解开的语音表示。随后，DDDM被应用于从解纠缠的表示重新合成语音，以相对于每个属性进行去噪。此外，我们还提出了鲁棒的语音风格转移的先验混合，它使用的混合风格的转换表示作为扩散模型的先验分布。实验结果表明，我们的方法优于公开可用的VC模型。此外，我们表明，我们的方法提供了强大的生成性能，无论模型大小。音频样本可从www.example.com获得https://hayeong0.github.io/DDDM-VC-demo/。摘要：Diffusion-based generative models have exhibited powerful generative performance in recent years. However, as many attributes exist in the data distribution and owing to several limitations of sharing the model parameters across all levels of the generation process, it remains challenging to control specific styles for each attribute. To address the above problem, this paper presents decoupled denoising diffusion models (DDDMs) with disentangled representations, which can control the style for each attribute in generative models. We apply DDDMs to voice conversion (VC) tasks to address the challenges of disentangling and controlling each speech attribute (e.g., linguistic information, intonation, and timbre). First, we use a self-supervised representation to disentangle the speech representation. Subsequently, the DDDMs are applied to resynthesize the speech from the disentangled representations for denoising with respect to each attribute. Moreover, we also propose the prior mixup for robust voice style transfer, which uses the converted representation of the mixed style as a prior distribution for the diffusion models. The experimental results reveal that our method outperforms publicly available VC models. Furthermore, we show that our method provides robust generative performance regardless of the model size. Audio samples are available https://hayeong0.github.io/DDDM-VC-demo/.【10】 Multilingual Text-to-Speech Synthesis for Turkic Languages Using Transliteration标题：基于音译的突厥语多语种文语合成链接：https://arxiv.org/abs/2305.15749作者：Rustem Yeshpanov,Saida Mussakhojayeva,Yerbolat Khassanov机构：Institute of Smart Systems and Artificial Intelligence, Nazarbayev University, Kazakhstan备注：5 pages, 1 figure, 3 tables, accepted to Interspeech摘要：这项工作旨在为十种资源较低的突厥语建立一个多语言文本到语音（TTS）合成系统：阿塞拜疆语、巴什基尔语、哈萨克语、吉尔吉斯语、萨哈语、鞑靼语、土耳其语、土库曼语、维吾尔语和乌兹别克语。我们专门针对zero-shot学习场景，其中使用一种语言的数据训练的TTS模型被应用于合成其他看不见的语言的语音。基于Tacotron 2架构的端到端TTS系统仅使用哈萨克语的可用数据进行训练。为了生成其他突厥语言的语音，我们首先将突厥字母表的字母映射到国际音标（IPA）的符号上，然后将其转换为哈萨克字母表的字母。为了证明所提出的方法的可行性，我们评估了多语种突厥文TTS模型的主观，并取得了可喜的成果。为了实现实验的复制，我们在GitHub存储库中公开了我们的代码和数据集。摘要：This work aims to build a multilingual text-to-speech (TTS) synthesis system for ten lower-resourced Turkic languages: Azerbaijani, Bashkir, Kazakh, Kyrgyz, Sakha, Tatar, Turkish, Turkmen, Uyghur, and Uzbek. We specifically target the zero-shot learning scenario, where a TTS model trained using the data of one language is applied to synthesise speech for other, unseen languages. An end-to-end TTS system based on the Tacotron 2 architecture was trained using only the available data of the Kazakh language. To generate speech for the other Turkic languages, we first mapped the letters of the Turkic alphabets onto the symbols of the International Phonetic Alphabet (IPA), which were then converted to the Kazakh alphabet letters. To demonstrate the feasibility of the proposed approach, we evaluated the multilingual Turkic TTS model subjectively and obtained promising results. To enable replication of the experiments, we make our code and dataset publicly available in our GitHub repository.【11】 Generalized domain adaptation framework for parametric back-end in speaker recognition标题：说话人识别中参数后端的广义域自适应框架链接：https://arxiv.org/abs/2305.15567作者：Qiongqiong Wang,Koji Okabe,Kong Aik Lee,Takafumi Koshinaka机构：Institute for Infocomm Research (I,R), A⋆STAR, Singapore, Data Science Laboratories, NEC Corporation, Japan, School of Data Science, Yokohama City University, Japan摘要：现有技术的说话人识别系统包括说话人嵌入前端，随后是概率线性判别分析（PLDA）后端。这些组件的有效性依赖于大量标记训练数据的可用性。在实践中，对于域（例如，语言、渠道、人口统计学），其中系统被部署为不同于系统被训练的系统。为了缩小由此产生的差距，领域适应往往是必不可少的PLDA模型。其中两种变体是重尾PLDA（HT-PLDA）和高斯PLDA（G-PLDA）。虽然前者比后者更适合真实的特征空间，但它的普及程度受到其计算复杂性的严重限制，特别是在域自适应方面的困难，这是由于其非高斯特性。已经针对G-PLDA提出了各种域自适应方法。本文提出了一种通用的框架域适应，可以应用到上述两个变种的PLDA说话人识别。它不仅包括几个现有的监督和无监督的领域适应方法，但也使得可能更灵活地使用不同领域的可用数据。特别是，我们在这里介绍两种新技术：（1）模型级的相关性对齐，以及（2）协方差正则化。据我们所知，这是第一次提出的应用程序的域适应w.r. t这样的技术。HT-PLDA。所提出的技术的有效性已经在NIST 2016，2018和2019扬声器识别评估（SRE'16，SRE'18和SRE'19）数据集上进行了实验验证。摘要：State-of-the-art speaker recognition systems comprise a speaker embedding front-end followed by a probabilistic linear discriminant analysis (PLDA) back-end. The effectiveness of these components relies on the availability of a large amount of labeled training data. In practice, it is common for domains (e.g., language, channel, demographic) in which a system is deployed to differ from that in which a system has been trained. To close the resulting gap, domain adaptation is often essential for PLDA models. Among two of its variants are Heavy-tailed PLDA (HT-PLDA) and Gaussian PLDA (G-PLDA). Though the former better fits real feature spaces than does the latter, its popularity has been severely limited by its computational complexity and, especially, by the difficulty, it presents in domain adaptation, which results from its non-Gaussian property. Various domain adaptation methods have been proposed for G-PLDA. This paper proposes a generalized framework for domain adaptation that can be applied to both of the above variants of PLDA for speaker recognition. It not only includes several existing supervised and unsupervised domain adaptation methods but also makes possible more flexible usage of available data in different domains. In particular, we introduce here two new techniques: (1) correlation-alignment in the model level, and (2) covariance regularization. To the best of our knowledge, this is the first proposed application of such techniques for domain adaptation w.r.t. HT-PLDA. The efficacy of the proposed techniques has been experimentally validated on NIST 2016, 2018, and 2019 Speaker Recognition Evaluation (SRE'16, SRE'18, and SRE'19) datasets.【12】 RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models标题：Rand：量化序列模型的稳健性感知范数衰减链接：https://arxiv.org/abs/2305.15536作者：David Qiu,David Rim,Shaojin Ding,Oleg Rybakov,Yanzhang He机构：Google, LLC摘要：随着神经网络规模的快速增长，模型压缩已成为一个重要的研究领域。量化是减少大型模型的模型大小、存储器访问和计算负载的有效技术。尽管最近在量化感知训练（QAT）技术方面取得了进展，但大多数论文的评估都集中在计算机视觉任务上，与序列任务相比，这些任务具有不同的训练动态。在本文中，我们首先基准的影响，流行的技术，如直通估计，伪量化噪声，可学习的尺度参数，裁剪等。在一套语音识别数据集上的4位seq 2seq模型上，范围从1,000小时到100万小时，以及一个机器翻译数据集，以说明其在语音之外的适用性。 通过实验，我们报告，基于噪声的QAT遭受时，有不足的正则化信号流回量化尺度。我们建议对QAT过程进行低复杂度更改，以提高模型准确性（优于流行的可学习尺度和裁剪方法）。随着精度的提高，它开辟了利用基于噪声的QAT的一些其他好处的可能性：1）训练在混合精度模式中表现良好的单个模型，以及2）改进长形式语音识别的泛化。摘要：With the rapid increase in the size of neural networks, model compression has become an important area of research. Quantization is an effective technique at decreasing the model size, memory access, and compute load of large models. Despite recent advances in quantization aware training (QAT) technique, most papers present evaluations that are focused on computer vision tasks, which have different training dynamics compared to sequence tasks. In this paper, we first benchmark the impact of popular techniques such as straight through estimator, pseudo-quantization noise, learnable scale parameter, clipping, etc. on 4-bit seq2seq models across a suite of speech recognition datasets ranging from 1,000 hours to 1 million hours, as well as one machine translation dataset to illustrate its applicability outside of speech. Through the experiments, we report that noise based QAT suffers when there is insufficient regularization signal flowing back to the quantization scale. We propose low complexity changes to the QAT process to improve model accuracy (outperforming popular learnable scale and clipping methods). With the improved accuracy, it opens up the possibility to exploit some of the other benefits of noise based QAT: 1) training a single model that performs well in mixed precision mode and 2) improved generalization on long form speech recognition.【13】 Spoofing Attacker Also Benefits from Self-Supervised Pretrained Model标题：欺骗攻击者还受益于自我监督的预训练模型链接：https://arxiv.org/abs/2305.15518作者：Aoi Ito,Shota Horiguchi机构：Hosei University, Hitachi, Ltd.备注：Accepted to INTERSPEECH 2023摘要：据报道，使用自监督学习的大规模预训练模型提高了语音反欺骗的性能。然而，攻击者侧也可以利用这样的模型。此外，由于从头开始训练这样的模型非常昂贵，因此通常使用互联网上的预训练模型，但是攻击者和防御者可能使用相同的预训练模型。本文研究了在攻击者可获得模型的条件下，预训练模型对反欺骗的改进是否成立。作为攻击者，我们训练了一个模型，增强了欺骗性的话语，使说话人嵌入提取器的基础上的预训练模型不能区分真正的和欺骗性的话语。实验结果表明，如果攻击者也使用预训练模型，则使用预训练模型获得的反欺骗模型的增益几乎消失。摘要：Large-scale pretrained models using self-supervised learning have reportedly improved the performance of speech anti-spoofing. However, the attacker side may also make use of such models. Also, since it is very expensive to train such models from scratch, pretrained models on the Internet are often used, but the attacker and defender may possibly use the same pretrained model. This paper investigates whether the improvement in anti-spoofing with pretrained models holds under the condition that the models are available to attackers. As the attacker, we train a model that enhances spoofed utterances so that the speaker embedding extractor based on the pretrained models cannot distinguish between bona fide and spoofed utterances. Experimental results show that the gains the anti-spoofing models obtained by using the pretrained models almost disappear if the attacker also makes use of the pretrained models.【14】 Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator标题：基于Sidecar分离器的多人重叠语音识别与二值化统一建模链接：https://arxiv.org/abs/2305.16263作者：Lingwei Meng,Jiawen Kang,Mingyu Cui,Haibin Wu,Xixin Wu,Helen Meng机构：Dept. of Systems Engineering & Engineering Management, The Chinese University of Hong Kong, Graduate Institute of Communication Engineering, National Taiwan University备注：Accepted to INTERSPEECH 2023摘要：多说话人重叠语音对语音识别和语音日记化提出了重大挑战。最近的研究表明，这两个任务是相互依赖和互补的，激励我们探索一个统一的建模方法，以解决他们的上下文中的重叠语音。最近的一项研究提出了一种具有成本效益的方法，通过将Sidecar分离器插入到冻结的经过良好训练的ASR模型中，将单说话者自动语音识别（ASR）系统转换为多说话者系统。在此基础上扩展，我们将日志化分支纳入Sidecar，允许ASR和日志化的统一建模，开销可以忽略不计，只有768个参数。与LibriMix和LibriSpeechMix数据集上的基线相比，所提出的方法产生更好的ASR结果。此外，没有复杂的定制的日记任务，我们的方法实现了可接受的日记结果的两个扬声器子集的CALLHOME只有几个适应步骤。摘要：Multi-talker overlapped speech poses a significant challenge for speech recognition and diarization. Recent research indicated that these two tasks are inter-dependent and complementary, motivating us to explore a unified modeling method to address them in the context of overlapped speech. A recent study proposed a cost-effective method to convert a single-talker automatic speech recognition (ASR) system into a multi-talker one, by inserting a Sidecar separator into the frozen well-trained ASR model. Extending on this, we incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters. The proposed method yields better ASR results compared to the baseline on LibriMix and LibriSpeechMix datasets. Moreover, without sophisticated customization on the diarization task, our method achieves acceptable diarization results on the two-speaker subset of CALLHOME with only a few adaptation steps.【15】 VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation标题：Viola：用于语音识别、合成和翻译的统一编解码器语言模型链接：https://arxiv.org/abs/2305.16107作者：Tianrui Wang,Long Zhou,Ziqiang Zhang,Yu Wu,Shujie Liu,Yashesh Gaur,Zhuo Chen,Jinyu Li,Furu Wei机构：Microsoft备注：Working in progress摘要：最近的研究表明，在不同模态的各种任务中，模型架构、训练目标和推理方法都有很大的收敛。在本文中，我们提出了VioLA，一个单一的自回归Transformer解码器的网络，统一了各种跨模态的任务，涉及语音和文本，如语音到文本，文本到文本，文本到语音，语音到语音的任务，作为一个条件的编解码器语言模型任务，通过多任务学习框架。为了实现这一点，我们首先使用离线神经编解码器编码器将所有语音话语转换为离散令牌（类似于文本数据）。通过这种方式，所有这些任务都被转换为基于令牌的序列转换问题，这可以自然地用一个条件语言模型来处理。我们进一步整合任务ID（TID）和语言ID（LID）到所提出的模型，以提高处理不同的语言和任务的建模能力。实验结果表明，所提出的VioLA模型可以很好地支持单模态和跨模态任务，而仅解码器模型的性能与强基线模型相当，甚至更好。摘要：Recent research shows a big convergence in model architecture, training objectives, and inference methods across various tasks for different modalities. In this paper, we propose VioLA, a single auto-regressive Transformer decoder-only network that unifies various cross-modal tasks involving speech and text, such as speech-to-text, text-to-text, text-to-speech, and speech-to-speech tasks, as a conditional codec language model task via multi-task learning framework. To accomplish this, we first convert all the speech utterances to discrete tokens (similar to the textual data) using an offline neural codec encoder. In such a way, all these tasks are converted to token-based sequence conversion problems, which can be naturally handled with one conditional language model. We further integrate task IDs (TID) and language IDs (LID) into the proposed model to enhance the modeling capability of handling different languages and tasks. Experimental results demonstrate that the proposed VioLA model can support both single-modal and cross-modal tasks well, and the decoder-only model achieves a comparable and even better performance than the strong baselines.【16】 End-to-End Simultaneous Speech Translation with Differentiable Segmentation标题：端到端可区分分词的同声翻译链接：https://arxiv.org/abs/2305.16093作者：Shaolei Zhang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：Accepted at ACL 2023 findings摘要：端到端同步语音翻译（SimulST）在接收流式语音输入（也称为“语音输入”）的同时输出翻译。流式语音翻译），因此需要对语音输入进行分段，然后基于当前接收的语音进行翻译。然而，在不利时刻分割语音输入可能破坏声学完整性并且不利地影响翻译模型的性能。因此，学习在那些有利于翻译模型产生高质量翻译的时刻分割语音输入是SimulST的关键。现有的SimulST方法，无论是使用固定长度的分割或外部分割模型，总是分离分割从底层的翻译模型，其中的差距导致分割结果，不一定有利于翻译过程。在本文中，我们提出了可区分分割（DiSeg）的SimulST直接学习分割从底层的翻译模型。DiSeg通过提出的期望训练将硬分割变成可区分的，使其能够与翻译模型联合训练，从而学习翻译有益的分割。实验结果表明，DiSeg达到国家的最先进的性能，并表现出优越的分割能力。摘要：End-to-end simultaneous speech translation (SimulST) outputs translation while receiving the streaming speech inputs (a.k.a. streaming speech translation), and hence needs to segment the speech inputs and then translate based on the current received speech. However, segmenting the speech inputs at unfavorable moments can disrupt the acoustic integrity and adversely affect the performance of the translation model. Therefore, learning to segment the speech inputs at those moments that are beneficial for the translation model to produce high-quality translation is the key to SimulST. Existing SimulST methods, either using the fixed-length segmentation or external segmentation model, always separate segmentation from the underlying translation model, where the gap results in segmentation outcomes that are not necessarily beneficial for the translation process. In this paper, we propose Differentiable Segmentation (DiSeg) for SimulST to directly learn segmentation from the underlying translation model. DiSeg turns hard segmentation into differentiable through the proposed expectation training, enabling it to be jointly trained with the translation model and thereby learn translation-beneficial segmentation. Experimental results demonstrate that DiSeg achieves state-of-the-art performance and exhibits superior segmentation capability.【17】 Visualizing data augmentation in deep speaker recognition标题：深度说话人识别中的可视化数据增强链接：https://arxiv.org/abs/2305.16070作者：Pengqi Li,Lantian Li,Askar Hamdulla,Dong Wang机构：School of Information Science and Engineering, Xinjiang University, China, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China, Center for Speech and Language Technologies, Tsinghua University, China备注：to be published in INTERSPEECH 2023摘要：可视化对于理解神经网络的内部机制具有重要价值。以前的工作发现，LayerCAM是一个可靠的可视化工具，用于深度扬声器模型。在本文中，我们使用LayerCAM来分析广泛采用的数据增强（DA）方法，以了解它如何导致模型的鲁棒性。我们在VoxCeleb1说话人识别数据集上进行了实验，结果表明，香草和基于激活的（Act）DA方法都增强了对干扰的鲁棒性，其中Act DA始终优于其他方法。LayerCAM的可视化表明，DA可以帮助模型学习删除被干扰破坏的时间频率（TF）频段。“学会删除”的行为解释了为什么DA模型比干净的模型更强大，以及为什么当干扰是非目标语音时，行为DA优于香草DA。然而，LayerCAM仍然不能清楚地解释ACT DA在其他情况下的优越性，建议进一步研究。摘要：Visualization is of great value in understanding the internal mechanisms of neural networks. Previous work found that LayerCAM is a reliable visualization tool for deep speaker models. In this paper, we use LayerCAM to analyze the widely-adopted data augmentation (DA) approach, to understand how it leads to model robustness. We conduct experiments on the VoxCeleb1 dataset for speaker identification, which shows that both vanilla and activation-based (Act) DA approaches enhance robustness against interference, with Act DA being consistently superior. Visualization with LayerCAM suggests DA helps models learn to delete temporal-frequency (TF) bins that are corrupted by interference. The `learn to delete' behavior explained why DA models are more robust than clean models, and why the Act DA is superior over the vanilla DA when the interference is nontarget speech. However, LayerCAM still cannot clearly explain the superiority of Act DA in other situations, suggesting further research.【18】 CN-Celeb-AV: A Multi-Genre Audio-Visual Dataset for Person Recognition标题：CN-Celeb-AV：一种用于人物识别的多类型视听数据集链接：https://arxiv.org/abs/2305.16049作者：Lantian Li,Xiaolou Li,Haoyu Jiang,Chen Chen,Ruihai Hou,Dong Wang机构：Center for Speech and Language Technologies, BNRist, Tsinghua University, China, School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China, ∗Corresponding authors:备注：to be published in INTERSPEECH 2023摘要：视听人识别（AVPR）受到了广泛的关注。然而，大多数数据集用于AVPR的研究，到目前为止，收集在受约束的环境中，因此不能反映真实世界的场景中的AVPR系统的真实性能。为了满足在无约束条件下研究AVPR的要求，本文提出了一个多流派AVPR数据集收集“在野外”，命名为CN-Celeb-AV。该数据集包含来自公共媒体的1，136人的超过42万个视频片段。特别是，我们更加强调两个现实世界的复杂性：（1）多种类型的数据;（2）部分信息段。将CN-Celeb-AV与两个流行的公共AVPR基准数据集进行了全面的研究，结果表明CN-Celeb-AV更符合真实世界的场景，可以被视为AVPR研究的新基准数据集。该数据集还涉及一个开发集，可用于提高AVPR系统在现实生活中的性能。该数据集对研究人员是免费的，可以从www.example.com下载http://cnceleb.org/。摘要：Audio-visual person recognition (AVPR) has received extensive attention. However, most datasets used for AVPR research so far are collected in constrained environments, and thus cannot reflect the true performance of AVPR systems in real-world scenarios. To meet the request for research on AVPR in unconstrained conditions, this paper presents a multi-genre AVPR dataset collected `in the wild', named CN-Celeb-AV. This dataset contains more than 420k video segments from 1,136 persons from public media. In particular, we put more emphasis on two real-world complexities: (1) data in multiple genres; (2) segments with partial information. A comprehensive study was conducted to compare CN-Celeb-AV with two popular public AVPR benchmark datasets, and the results demonstrated that CN-Celeb-AV is more in line with real-world scenarios and can be regarded as a new benchmark dataset for AVPR research. The dataset also involves a development set that can be used to boost the performance of AVPR systems in real-life situations. The dataset is free for researchers and can be downloaded from http://cnceleb.org/.【19】 Ordered and Binary Speaker Embedding标题：有序和二进制说话人嵌入链接：https://arxiv.org/abs/2305.16043作者：Jiaying Wang,Xianglong Wang,Namin Wang,Lantian Li,Dong Wang机构：Center for Speech and Language Technologies, BNRist, Tsinghua University, China, Huawei Cloud, China, Beijing University of Posts and Telecommunications, China备注：to be published in INTERSPEECH 2023摘要：现代说话人识别系统通过嵌入向量来表示话语。传统的嵌入向量是密集的和非结构性的。在本文中，我们提出了一种有序的二进制嵌入方法，通过嵌套的丢弃的嵌入向量的尺寸排序，并通过伯努利采样的二进制码转换排序的向量。由此产生的有序二进制代码提供了一些重要的优点，如层次聚类，减少内存的使用，和快速检索。这些优点进行了实证验证的说话人识别任务与VoxCeleb和CN-Celeb数据集的综合实验。摘要：Modern speaker recognition systems represent utterances by embedding vectors. Conventional embedding vectors are dense and non-structural. In this paper, we propose an ordered binary embedding approach that sorts the dimensions of the embedding vector via a nested dropout and converts the sorted vectors to binary codes via Bernoulli sampling. The resultant ordered binary codes offer some important merits such as hierarchical clustering, reduced memory usage, and fast retrieval. These merits were empirically verified by comprehensive experiments on a speaker identification task with the VoxCeleb and CN-Celeb datasets.【20】 Latent Diffusion Model Based Foley Sound Generation System For DCASE Challenge 2023 Task 7标题：基于潜在扩散模型的DCASE挑战赛2023任务7的Foley声音生成系统链接：https://arxiv.org/abs/2305.15905作者：Yi Yuan,Haohe Liu,Xubo Liu,Xiyuan Kang,Mark D. Plumbley,Wenwu Wang机构：University of Surrey, Guildford, United Kingdom备注：DCASE 2023 task 7 technical report摘要：Foley声音生成旨在合成多媒体内容的背景声音，其涉及使用专门技术对声音效果进行计算建模。在这项工作中，我们为DCASE 2023挑战任务7提出了一个基于扩散的生成模型：Foley声音合成所提出的系统是基于AudioLDM，这是一个基于扩散的文本到音频生成模型。为了缓解任务7训练集的数据稀缺性，我们的模型最初使用大规模数据集进行训练，并通过迁移学习将其下游到这个DCASE任务中。我们已经观察到，由编码器提取的特征可以显著地影响生成模型的性能。因此，我们通过利用具有由大型语言模型获得的相关文本嵌入特征的输入标签来改进结果，即，对比语言-音频预训练（CLAP）。此外，我们利用过滤策略来进一步细化输出，即。通过从根据声音和目标标签之间的相似性得分生成的候选剪辑中选择最佳结果。整个系统在所有七个不同类别中平均达到4.765的Frechet音频距离（FAD）分数，显著优于达到9.7的FAD分数的基线系统。摘要：Foley sound generation aims to synthesise the background sound for multimedia content, which involves computationally modelling sound effects with specialized techniques. In this work, we proposed a diffusion based generative model for DCASE 2023 challenge task 7: Foley Sound Synthesis. The proposed system is based on AudioLDM, which is a diffusion-based text-to-audio generation model. To alleviate the data scarcity of the task 7 training set, our model is initially trained with large-scale datasets and downstream into this DCASE task via transfer learning. We have observed that the feature extracted by the encoder can significantly affect the performance of the generation model. Hence, we improve the results by leveraging the input label with related text embedding features obtained by a large language model, i.e., contrastive language-audio pretraining (CLAP). In addition, we utilize a filtering strategy to further refine the output, i.e. by selecting the best results from the candidate clips generated in terms of the similarity score between the sound and target labels. The overall system achieves a Frechet audio distance (FAD) score of 4.765 on average among all seven different classes, substantially outperforming the baseline system which achieves a FAD score of 9.7.【21】 Room Impulse Response Estimation in a Multiple Source Environment标题：多源环境下的房间脉冲响应估计链接：https://arxiv.org/abs/2305.15898作者：Kyungyun Lee,Jeonghun Seo,Keunwoo Choi,Sangmoon Lee,Ben Sangbae Chon机构：Gaudio Lab, Inc., Seoul, South Korea备注：2023 AES International Conference on Spatial and Immersive Audio摘要：在现实世界的声学场景中，房间中通常存在多个声源。这些声源位于不同的位置，并产生从多个方向到达收听者的声音。房间中多个源的存在在估计房间脉冲响应（RIR）中产生了新的挑战，因为每个源具有取决于其位置和取向的唯一RIR。因此，当输入信号是多个混响源的混合时，出现了确定应该预测哪个RIR以及如何预测它的问题。为了解决这些问题，我们提出了一个新的任务，预测一个房间在多源环境中的“代表性”RIR，并提出了一种训练方法来实现这一目标。与在单源环境中训练的模型相比，无论环境中的源数量如何，我们的方法都显示出强大的性能。摘要：In real-world acoustic scenarios, there often are multiple sound sources present in a room. These sources are situated in various locations and produce sounds that reach the listener from multiple directions. The presence of multiple sources in a room creates new challenges in estimating the room impulse response (RIR) as each source has a unique RIR, dependent on its location and orientation. Therefore, issues of determining which RIR should be predicted and how to predict it arise, when the input signal is a mixture of multiple reverberated sources. To address these, we propose a new task of predicting a "representative" RIR for a room in a multiple source environment and present a training method to achieve this goal. In contrast to the model trained in a single source environment, our method shows robust performance, regardless of the number of sources in the environment.【22】 Anomalous Sound Detection Based on Sound Separation标题：基于声音分离的异常声音检测链接：https://arxiv.org/abs/2305.15859作者：Kanta Shimonishi,Kota Dohi,Yohei Kawaguchi机构：Ritsumeikan University, Japan, Hitachi, Ltd., Japan备注：Accepted to INTERSPEECH2023摘要：提出了一种基于声音分离的无监督异常声音检测方法。在工厂环境中，背景噪声和非客观声音掩盖了所需的机器声音，使得检测异常声音具有挑战性。因此，期望在检测系统中使用不与背景噪声或非目的声音混合的声音。我们比较了我们提出的方法的两个版本，一个使用声音分离作为预处理步骤，另一个使用基于分离的离群值暴露，使用两个分离的声音之间的误差。基于正常声音和异常声音之间的分离性能差异影响检测结果的假设，在两个版本中使用特定于特定产品类型的声音分离模型。实验结果表明，该方法提高了异常声音检测性能的所有机器ID，实现了39%的最大改善。摘要：This paper proposes an unsupervised anomalous sound detection method using sound separation. In factory environments, background noise and non-objective sounds obscure desired machine sounds, making it challenging to detect anomalous sounds. Therefore, using sounds not mixed with background noise or non-purpose sounds in the detection system is desirable. We compared two versions of our proposed method, one using sound separation as a pre-processing step and the other using separation-based outlier exposure that uses the error between two separated sounds. Based on the assumption that differences in separation performance between normal and anomalous sounds affect detection results, a sound separation model specific to a particular product type was used in both versions. Experimental results indicate that the proposed method improved anomalous sound detection performance for all Machine IDs, achieving a maximum improvement of 39%.【23】 Svarah: Evaluating English ASR Systems on Indian Accents标题：Svarah：根据印度口音评估英语ASR系统链接：https://arxiv.org/abs/2305.15760作者：Tahir Javed,Sakshi Joshi,Vignesh Nagarajan,Sai Sundaresan,Janki Nawale,Abhigyan Raman,Kaushal Bhogale,Pratyush Kumar,Mitesh M. Khapra机构：Indian Institute of Technology Madras, India, AI,Bharat, India, Microsoft, India摘要：印度是世界上第二大英语国家，拥有约1.3亿人口。因此，它是必要的，英语的自动语音识别（ASR）系统应评估印度口音。不幸的是，印度人在现有的英语ASR基准中发现非常差的表示，例如LibriSpeech，Switchboard，Speech Accent Archive等。在这项工作中，我们通过创建Svarah来解决这一差距，这是一个基准，其中包含来自印度65个地理位置的117名发言者的9.6小时转录的英语音频，从而产生了各种各样的口音。Svarah包括阅读语音和自发会话数据两者，覆盖各种领域，诸如历史、文化、旅游等。确保词汇的多样性。我们评估了6个开源ASR模型和2个商业ASR系统在Svarah上，并表明印度口音有明显的改进空间。Svarah以及我们所有的代码都将公开。摘要：India is the second largest English-speaking country in the world with a speaker base of roughly 130 million. Thus, it is imperative that automatic speech recognition (ASR) systems for English should be evaluated on Indian accents. Unfortunately, Indian speakers find a very poor representation in existing English ASR benchmarks such as LibriSpeech, Switchboard, Speech Accent Archive, etc. In this work, we address this gap by creating Svarah, a benchmark that contains 9.6 hours of transcribed English audio from 117 speakers across 65 geographic locations throughout India, resulting in a diverse range of accents. Svarah comprises both read speech and spontaneous conversational data, covering various domains, such as history, culture, tourism, etc., ensuring a diverse vocabulary. We evaluate 6 open source ASR models and 2 commercial ASR systems on Svarah and show that there is clear scope for improvement on Indian accents. Svarah as well as all our code will be publicly available.【24】 Towards Solving Cocktail-Party: The First Method to Build a Realistic Dataset with Ground Truths for Speech Separation标题：走向解决鸡尾酒会：利用基本事实构建语音分离的真实数据集的第一种方法链接：https://arxiv.org/abs/2305.15758作者：Rawad Melhem,Assef Jafar,Oumayma Al Dakkak机构：Towards Solving Cocktail-Party: The First Method to Build a, Realistic Dataset with Ground Truths for Speech Separation, ), Institute for Applied Sciences and Technology, Damascus, Syria, Higher, (摘要：语音分离在诸如人机交互、助听器设备和自动会议转录等实际应用中非常重要。近年来，基于深度学习的解决方案出现了重大改进。事实上，人们已经注意到使用合成混合数据集的监督学习方法，尽管它们并不代表真实世界的混合物。建立真实数据集的困难导致研究人员使用无监督学习方法，因为它们能够直接处理真实的混合物。无监督学习方法的结果仍然不令人信服。在本文中，介绍了一种方法来创建一个真实的数据集与地面真值源的语音分离。设计现实数据集的主要挑战是扬声器信号的地面实况不可用。为了解决这个问题，我们提出了一种方法，同时记录两个扬声器，并获得每个地面真相。我们提出了一种使用基于双向门控递归单元（BGRU）和聚类算法的深度学习模型对现实数据集进行基准测试的方法。实验表明，我们提出的数据集改进SI-SDR（标度不变信号失真比）由1.65 dB和PESQ（语音质量的感知评价）由约0.5。我们还评估了我们的方法在麦克风和扬声器之间不同距离下的有效性，发现它提高了学习模型的稳定性。摘要：Speech separation is very important in real-world applications such as human-machine interaction, hearing aids devices, and automatic meeting transcription. In recent years, a significant improvement occurred towards the solution based on deep learning. In fact, much attention has been drawn to supervised learning methods using synthetic mixtures datasets despite their being not representative of real-world mixtures. The difficulty in building a realistic dataset led researchers to use unsupervised learning methods, because of their ability to handle realistic mixtures directly. The results of unsupervised learning methods are still unconvincing. In this paper, a method is introduced to create a realistic dataset with ground truth sources for speech separation. The main challenge in designing a realistic dataset is the unavailability of ground truths for speakers signals. To address this, we propose a method for simultaneously recording two speakers and obtaining the ground truth for each. We present a methodology for benchmarking our realistic dataset using a deep learning model based on Bidirectional Gated Recurrent Units (BGRU) and clustering algorithm. The experiments show that our proposed dataset improved SI-SDR (Scale Invariant Signal to Distortion Ratio) by 1.65 dB and PESQ (Perceptual Evaluation of Speech Quality) by approximately 0.5. We also evaluated the effectiveness of our method at different distances between the microphone and the speakers and found that it improved the stability of the learned model.【25】 Efficient Neural Music Generation标题：高效的神经音乐生成链接：https://arxiv.org/abs/2305.15719作者：Max W. Y. Lam,Qiao Tian,Tang Li,Zongyu Yin,Siyuan Feng,Ming Tu,Yuliang Ji,Rui Xia,Mingbo Ma,Xuchen Song,Jitong Chen,Yuping Wang,Yuxuan Wang机构：Speech, Audio & Music Intelligence (SAMI), ByteDance摘要：音乐生成的最新进展已经通过最先进的MusicLM显著地推进，其包括三个LM的层次结构，分别用于语义、粗糙声学和精细声学建模。然而，用MusicLM进行采样需要通过这些LM逐个处理以获得细粒度的声学令牌，这使得它在计算上是昂贵的并且对于实时生成是禁止的。高效的音乐生成与MusicLM的质量相当仍然是一个重大的挑战。在本文中，我们提出了MeLoDy（M用于音乐; L表示LM; D表示扩散），这是一种LM引导的扩散模型，可以生成最先进质量的音乐音频，同时在MusicLM中分别减少95.7%或99.6%的前向传递，用于采样10秒或30秒的音乐。MeLoDy继承了MusicLM的最高级别LM进行语义建模，并应用了一种新的双路径扩散（DPD）模型和音频VAE-GAN来有效地将条件语义令牌解码为波形。DPD提出了同时建模的粗和细的声学通过将语义信息有效地通过交叉注意在每个去噪步骤的潜在段。我们的实验结果表明，MeLoDy的优越性，不仅在其实际的采样速度和无限连续生成的优势，而且在其国家的最先进的音乐性，音频质量和文本相关性。 我们的样品可在www.example.com获得https://Efficient-MeLoDy.github.io/。摘要：Recent progress in music generation has been remarkably advanced by the state-of-the-art MusicLM, which comprises a hierarchy of three LMs, respectively, for semantic, coarse acoustic, and fine acoustic modelings. Yet, sampling with the MusicLM requires processing through these LMs one by one to obtain the fine-grained acoustic tokens, making it computationally expensive and prohibitive for a real-time generation. Efficient music generation with a quality on par with MusicLM remains a significant challenge. In this paper, we present MeLoDy (M for music; L for LM; D for diffusion), an LM-guided diffusion model that generates music audios of state-of-the-art quality meanwhile reducing 95.7% or 99.6% forward passes in MusicLM, respectively, for sampling 10s or 30s music. MeLoDy inherits the highest-level LM from MusicLM for semantic modeling, and applies a novel dual-path diffusion (DPD) model and an audio VAE-GAN to efficiently decode the conditioning semantic tokens into waveform. DPD is proposed to simultaneously model the coarse and fine acoustics by incorporating the semantic information into segments of latents effectively via cross-attention at each denoising step. Our experimental results suggest the superiority of MeLoDy, not only in its practical advantages on sampling speed and infinitely continuable generation, but also in its state-of-the-art musicality, audio quality, and text correlation. Our samples are available at https://Efficient-MeLoDy.github.io/.【26】 Mixture-of-Expert Conformer for Streaming Multilingual ASR标题：用于流媒体多语言ASR的混合专家整形器链接：https://arxiv.org/abs/2305.15663作者：Ke Hu,Bo Li,Tara N. Sainath,Yu Zhang,Francoise Beaufays机构：Google LLC, USA备注：Accepted to Interspeech 2023摘要：具有大容量的端到端模型显著改善了多语言自动语音识别，但其计算成本对设备上的应用提出了挑战。我们提出了一个真正的多语言流整合混合专家（MoE）层，学习只激活一个子集的参数在训练和推理。MoE层由一个softmax门组成，它在前向传播中从许多专家中选择最好的两个专家。建议的MoE层提供了有效的推理，通过激活一个固定数量的参数，专家的数量增加。我们评估了一组12种语言的模型，并实现了平均11.9%的相对改善WER的基线。与使用地面实况信息的适配器模型相比，我们的MoE模型实现了类似的WER，并激活了类似数量的参数，但没有任何语言信息。我们进一步表明，通过多语言浅融合，WER相对改善约3%。摘要：End-to-end models with large capacity have significantly improved multilingual automatic speech recognition, but their computation cost poses challenges for on-device applications. We propose a streaming truly multilingual Conformer incorporating mixture-of-expert (MoE) layers that learn to only activate a subset of parameters in training and inference. The MoE layer consists of a softmax gate which chooses the best two experts among many in forward propagation. The proposed MoE layer offers efficient inference by activating a fixed number of parameters as the number of experts increases. We evaluate the proposed model on a set of 12 languages, and achieve an average 11.9% relative improvement in WER over the baseline. Compared to an adapter model using ground truth information, our MoE model achieves similar WER and activates similar number of parameters but without any language information. We further show around 3% relative WER improvement by multilingual shallow fusion.【27】 Metamathematics of Algorithmic Composition标题：算法合成的元数学链接：https://arxiv.org/abs/2305.15601作者：Michael Gogins机构：when I was a returning undergraduate majoring in comparative religion atthe University of Washington备注：15 pages, 0 figures. Comments are very welcome摘要：这篇文章讲述了我个人对算法音乐作曲的数学基础有更深入理解的旅程。我没有花太多时间在作曲家使用的特定数学算法上;相反，我专注于一般性的问题，如基本的限制和可能性，通过类比与元逻辑，元数学和可计算性理论。我讨论了这些基础的算法组成的未来的影响。摘要：This essay recounts my personal journey towards a deeper understanding of the mathematical foundations of algorithmic music composition. I do not spend much time on specific mathematical algorithms used by composers; rather, I focus on general issues such as fundamental limits and possibilities, by analogy with metalogic, metamathematics, and computability theory. I discuss implications from these foundations for the future of algorithmic composition.【28】 Sound Design Strategies for Latent Audio Space Explorations Using Deep Learning Architectures标题：利用深度学习结构进行潜在音频空间探索的声音设计策略链接：https://arxiv.org/abs/2305.15571作者：Kıvanç Tatar,Kelsey Cotton,Daniel Bisig机构：Chalmers University of Technology, Sweden, Zurich University of the Arts, Switzerland备注：In Proceedings of Sound and Music Computing 2023摘要：近年来，深度学习在声音和音乐计算中的应用研究引起了人们的兴趣;然而，在这些新技术和如何将它们纳入现实世界的艺术实践之间仍然缺少一个环节。在这项工作中，我们探索了一种着名的深度学习架构，称为变分自动编码器（VAE）。这些架构已经在许多领域中用于生成潜在空间，其中数据点被组织成使得相似的数据点彼此更靠近。以前，VAE已经用于生成潜在音色空间或符号音乐例外的潜在空间。将VAE应用于音色的音频特征需要声码器将由网络生成的音色变换为音频信号，这在计算上是昂贵的。在这项工作中，我们直接将VAE应用于原始音频数据，同时绕过音频特征提取。这种方法允许从业者使用任何音频记录，同时通过数据集策展提供灵活性和对美学的控制。音频信号生成中的较低计算时间允许将原始音频方法并入实时应用中。在这项工作中，我们提出了三种策略，探索潜在的空间的音频和音色的声音设计应用。通过这样做，我们的目标是发起一场关于艺术方法和策略的对话，以利用声音和音乐实践中的潜在音频空间。摘要：The research in Deep Learning applications in sound and music computing have gathered an interest in the recent years; however, there is still a missing link between these new technologies and on how they can be incorporated into real-world artistic practices. In this work, we explore a well-known Deep Learning architecture called Variational Autoencoders (VAEs). These architectures have been used in many areas for generating latent spaces where data points are organized so that similar data points locate closer to each other. Previously, VAEs have been used for generating latent timbre spaces or latent spaces of symbolic music excepts. Applying VAE to audio features of timbre requires a vocoder to transform the timbre generated by the network to an audio signal, which is computationally expensive. In this work, we apply VAEs to raw audio data directly while bypassing audio feature extraction. This approach allows the practitioners to use any audio recording while giving flexibility and control over the aesthetics through dataset curation. The lower computation time in audio signal generation allows the raw audio approach to be incorporated into real-time applications. In this work, we propose three strategies to explore latent spaces of audio and timbre for sound design applications. By doing so, our aim is to initiate a conversation on artistic approaches and strategies to utilize latent audio spaces in sound and music practices.机器翻译，仅供参考
每日语音速递
