今日论文合集：cs.SD语音17篇，eess.AS音频处理18篇。
cs.SD语音
【1】 An Efficient Temporary Deepfake Location Approach Based Embeddings for  Partially Spoofed Audio Detection标题：一种有效的基于嵌入的临时深伪定位方法用于部分欺骗音频检测链接：https://arxiv.org/abs/2309.03036
作者：Yuankun Xie,Haonan Cheng,Yutian Wang,Long Ye机构：State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing , China备注：Submitted to ICASSP 2024摘要：部分欺骗音频检测是一项具有挑战性的任务，在于需要在帧级别准确地定位音频的真实性。为了解决这个问题，我们提出了一种细粒度的部分欺骗音频检测方法，即时间深度假定位（TDL），它可以有效地捕获特征和位置的信息。具体来说，我们的方法涉及两个新颖的部分：嵌入相似度模块和时间卷积运算。为了增强真实和假特征的识别，嵌入相似度模块被设计为生成一个嵌入空间，将真实帧和假帧分离开来。为了有效地集中在位置信息，时间卷积运算被提出来计算相邻帧之间的帧特定的相似性，并动态地选择有信息的邻居进行卷积。大量实验表明，我们的方法在ASVspoof 2019部分欺骗数据集中优于基线模型，即使在交叉数据集场景中也表现出优异的性能。代码在网上发布。摘要：Partially spoofed audio detection is a challenging task, lying in the need to accurately locate the authenticity of audio at the frame level. To address this issue, we propose a fine-grained partially spoofed audio detection method, namely Temporal Deepfake Location (TDL), which can effectively capture information of both features and locations. Specifically, our approach involves two novel parts: embedding similarity module and temporal convolution operation. To enhance the identification between the real and fake features, the embedding similarity module is designed to generate an embedding space that can separate the real frames from fake frames. To effectively concentrate on the position information, temporal convolution operation is proposed to calculate the frame-specific similarities among neighboring frames, and dynamically select informative neighbors to convolution. Extensive experiments show that our method outperform baseline models in ASVspoof2019 Partial Spoof dataset and demonstrate superior performance even in the crossdataset scenario. The code is released online.
【2】 BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial  Network标题：BigVSAN：用分片对抗网络增强基于GAN的神经声码器链接：https://arxiv.org/abs/2309.02836
作者：Takashi Shibuya,Yuhta Takida,Yuki Mitsufuji机构：Sony AI, Tokyo, Japan, Sony Group Corporation, Tokyo, Japan备注：Submitted to ICASSP 2024摘要：基于生成对抗网络（GAN）的声码器已经被广泛研究，因为它们可以比实时更快地合成高保真音频波形。然而，据报道，大多数GANs未能获得用于在特征空间中区分真实和虚假数据的最佳投影。在文献中，它已经被证明，切片对抗网络（SAN），一个改进的GAN训练框架，可以找到最佳投影，在图像生成任务是有效的。在本文中，我们调查的有效性SAN的语音编码任务。为此，我们提出了一个方案来修改最小二乘GAN，其中大多数基于GAN的声码器采用，使其损失函数满足SAN的要求。通过我们的实验，我们证明SAN可以提高性能的基于GAN的声码器，包括BigVGAN，与小的修改。我们的代码可在https://github.com/sony/bigvsan上获得。摘要：Generative adversarial network (GAN)-based vocoders have been intensively studied because they can synthesize high-fidelity audio waveforms faster than real-time. However, it has been reported that most GANs fail to obtain the optimal projection for discriminating between real and fake data in the feature space. In the literature, it has been demonstrated that slicing adversarial network (SAN), an improved GAN training framework that can find the optimal projection, is effective in the image generation task. In this paper, we investigate the effectiveness of SAN in the vocoding task. For this purpose, we propose a scheme to modify least-squares GAN, which most GAN-based vocoders adopt, so that their loss functions satisfy the requirements of SAN. Through our experiments, we demonstrate that SAN can improve the performance of GAN-based vocoders, including BigVGAN, with small modifications. Our code is available at https://github.com/sony/bigvsan.
【3】 Self-Supervised Disentanglement of Harmonic and Rhythmic Features in  Music Audio Signals标题：音乐音频信号中和声和节奏特征的自监督解缠链接：https://arxiv.org/abs/2309.02796
作者：Yiming Wu机构：AlphaTheta Corporation, Yokohama, Japan备注：Accepted to DAFx 2023摘要：潜在变量解纠缠的目的是推断出数据生成过程背后的多个信息潜在表示，并且是可控数据生成的关键因素。在本文中，我们提出了一种基于深度神经网络的自监督学习方法来推断音乐音频生成背后的解开的节奏和和声表示。我们训练一个变分自动编码器，生成一个音频梅尔频谱图，从两个潜在的特征代表的节奏和谐波的内容。在训练阶段，训练变分自动编码器以在给定其音高移位版本的情况下重建输入梅尔频谱图。在训练阶段中的每个前向计算处，假设特征向量的维度与音调间隔相关，向量旋转操作被应用于潜在特征中的一个。因此，在训练的变分自动编码器中，旋转的潜在特征表示梅尔谱图的音高相关信息，而未旋转的潜在特征表示音高不变信息，即节奏的内容。所提出的方法进行了评估，使用基于预测器的解纠缠度量的学习特征。此外，我们展示了它的应用程序自动生成的音乐混音。摘要：The aim of latent variable disentanglement is to infer the multiple informative latent representations that lie behind a data generation process and is a key factor in controllable data generation. In this paper, we propose a deep neural network-based self-supervised learning method to infer the disentangled rhythmic and harmonic representations behind music audio generation. We train a variational autoencoder that generates an audio mel-spectrogram from two latent features representing the rhythmic and harmonic content. In the training phase, the variational autoencoder is trained to reconstruct the input mel-spectrogram given its pitch-shifted version. At each forward computation in the training phase, a vector rotation operation is applied to one of the latent features, assuming that the dimensions of the feature vectors are related to pitch intervals. Therefore, in the trained variational autoencoder, the rotated latent feature represents the pitch-related information of the mel-spectrogram, and the unrotated latent feature represents the pitch-invariant information, i.e., the rhythmic content. The proposed method was evaluated using a predictor-based disentanglement metric on the learned features. Furthermore, we demonstrate its application to the automatic generation of music remixes.
【4】 GRASS: Unified Generation Model for Speech Semantic Understanding标题：GRASS：语音语义理解的统一生成模型链接：https://arxiv.org/abs/2309.02780
作者：Aobo Xia,Shuyu Lei,Yushu Yang,Xiang Guo,Hua Chai机构：⋆ Beijing University of Posts and Telecommunications, Beijing, China, † Didi Chuxing, Beijing, China摘要：本文探讨了语音语义理解的指令微调技术，通过引入一个统一的端到端（E2E）的框架，生成语义标签的条件下，任务相关的提示音频数据。我们预训练模型使用大量和不同的数据，其中指令语音对构建通过一个文本到语音（TTS）系统。大量的实验表明，我们提出的模型显着优于国家的最先进的（SOTA）模型后微调下游任务。此外，所提出的模型实现了有竞争力的性能在zero-shot和Few-Shot的情况下。为了便于未来的语音到语义任务的指令微调工作，我们发布了我们的指令数据集和代码。摘要：This paper explores the instruction fine-tuning technique for speech semantic understanding by introducing a unified end-to-end (E2E) framework that generates semantic labels conditioned on a task-related prompt for audio data. We pre-train the model using large and diverse data, where instruction-speech pairs are constructed via a text-to-speech (TTS) system. Extensive experiments demonstrate that our proposed model significantly outperforms state-of-the-art (SOTA) models after fine-tuning downstream tasks. Furthermore, the proposed model achieves competitive performance in zero-shot and few-shot scenarios. To facilitate future work on instruction fine-tuning for speech-to-semantic tasks, we release our instruction dataset and code.
【5】 Simultaneous Measurement of Multiple Acoustic Attributes Using  Structured Periodic Test Signals Including Music and Other Sound Materials标题：使用包括音乐和其他声学材料的结构化周期性测试信号同时测量多个声学属性链接：https://arxiv.org/abs/2309.02767
作者：Hideki Kawahara,Kohei Yatabe,Ken-Ichi Sakakibara,Mitsunori Mizumachi,Tatsuya Kitamura机构：∗ Wakayama University, Japan, † Tokyo University of Agriculture and Technology, Japan, ‡ Health Sciences University of Hokkaido, Japan, § Kyushu Institute of Technology, Japan, ¶ Konan University, Japan备注：8 pages, 17 figures, accepted for APSIPA ASC 2023摘要：我们介绍了一个通用的框架，用于测量声学特性，如线性时不变（LTI）的响应，信号相关的时不变（SDTI）的组件，随机和时变（RTV）的组件，同时使用结构化的周期性测试信号。该框架还通过添加轻微的确定性噪声来“保护”音乐作品和其他声音材料作为测试信号。“使用扫频正弦、MLS（Maxim Length Sequence）及其变体的测量是所提出框架的特殊情况。我们实现了基于该框架的交互式和实时测量工具，并使其开源。此外，我们应用这个框架来客观地评估沥青提取器。摘要：We introduce a general framework for measuring acoustic properties such as liner time-invariant (LTI) response, signal-dependent time-invariant (SDTI) component, and random and time-varying (RTV) component simultaneously using structured periodic test signals. The framework also enables music pieces and other sound materials as test signals by "safeguarding" them by adding slight deterministic "noise." Measurement using swept-sin, MLS (Maxim Length Sequence), and their variants are special cases of the proposed framework. We implemented interactive and real-time measuring tools based on this framework and made them open-source. Furthermore, we applied this framework to assess pitch extractors objectively.
【6】 Music Source Separation with Band-Split RoPE Transformer标题：基于分带式绳索Transformer的音乐源分离链接：https://arxiv.org/abs/2309.02612
作者：Wei-Tsung Lu,Ju-Chiang Wang,Qiuqiang Kong,Yun-Ning Hung机构：SAMI, ByteDance备注：This paper explains the SAMI-ByteDance MSS system submitted to Sound Demixing Challenge (SDX23) Music Separation Track摘要：音乐源分离（MSS）旨在将音乐录音分离成多个音乐上不同的主干，例如人声、低音、鼓等。最近，已经使用了卷积神经网络（CNN）和递归神经网络（RNN）等深度学习方法，但改进仍然有限。在本文中，我们提出了一种新的频域方法的基础上的带分裂RoPE Transformer（称为BS-RoFormer）。BS-RoFormer在频带分割模块上回复，将输入的复杂频谱图投影到子频带级表示中，然后安排一堆分层Transformers来建模带内以及带间序列，用于多频带掩码估计。为了便于训练MSS模型，我们建议使用旋转位置嵌入（RoPE）。在MUSDB 18 HQ和500首额外歌曲训练的BS-RoFormer系统在Sound Demixing Challenge（SDX 23）MSS曲目中排名第一。在MUSDB 18 HQ上对BS-RoFormer的小型版本进行基准测试，我们在没有额外训练数据的情况下实现了最先进的结果，平均SDR为9.80 dB。摘要：Music source separation (MSS) aims to separate a music recording into multiple musically distinct stems, such as vocals, bass, drums, and more. Recently, deep learning approaches such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been used, but the improvement is still limited. In this paper, we propose a novel frequency-domain approach based on a Band-Split RoPE Transformer (called BS-RoFormer). BS-RoFormer replies on a band-split module to project the input complex spectrogram into subband-level representations, and then arranges a stack of hierarchical Transformers to model the inner-band as well as inter-band sequences for multi-band mask estimation. To facilitate training the model for MSS, we propose to use the Rotary Position Embedding (RoPE). The BS-RoFormer system trained on MUSDB18HQ and 500 extra songs ranked the first place in the MSS track of Sound Demixing Challenge (SDX23). Benchmarking a smaller version of BS-RoFormer on MUSDB18HQ, we achieve state-of-the-art result without extra training data, with 9.80 dB of average SDR.
【7】 Text-Only Domain Adaptation for End-to-End Speech Recognition through  Down-Sampling Acoustic Representation标题：通过下采样声学表示实现端到端语音识别的纯文本域自适应链接：https://arxiv.org/abs/2309.02459
作者：Jiaxu Zhu,Weinan Tong,Yaoxun Xu,Changhe Song,Zhiyong Wu,Zhao You,Dan Su,Dong Yu,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Peng Cheng Lab, Shenzhen, China, Tencent AI Lab, Shenzhen, China, Tencent AI Lab, Bellevue, WA, USA, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by INTERSPEECH 2023. arXiv admin note: text overlap with arXiv:2309.01437摘要：将语音和文本两种模态映射到共享的表示空间中，是使用纯文本数据来提高新领域中的端到端自动语音识别（ASR）性能的研究课题。然而，语音表示和文本表示的长度是不一致的。尽管先前的方法对文本表示进行上采样以与声学模态对齐，但其可能不匹配预期的实际持续时间。在本文中，我们提出了新的表示匹配策略，通过下采样声学表示对齐文本模态。通过引入一个连续的集成和消防（CIF）模块生成与令牌长度一致的声学表示，我们的ASR模型可以更好地从这两种模式学习统一的表示，允许使用目标域的纯文本数据的域自适应。新领域数据的实验结果表明了该方法的有效性。摘要：Mapping two modalities, speech and text, into a shared representation space, is a research topic of using text-only data to improve end-to-end automatic speech recognition (ASR) performance in new domains. However, the length of speech representation and text representation is inconsistent. Although the previous method up-samples the text representation to align with acoustic modality, it may not match the expected actual duration. In this paper, we proposed novel representations match strategy through down-sampling acoustic representation to align with text modality. By introducing a continuous integrate-and-fire (CIF) module generating acoustic representations consistent with token length, our ASR model can learn unified representations from both modalities better, allowing for domain adaptation using text-only data of the target domain. Experiment results of new domain data demonstrate the effectiveness of the proposed method.
【8】 Matcha-TTS: A fast TTS architecture with conditional flow matching标题：Matcha-TTS：一种支持条件流匹配的快速TTS体系结构链接：https://arxiv.org/abs/2309.03199
作者：Shivam Mehta,Ruibo Tu,Jonas Beskow,Éva Székely,Gustav Eje Henter机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 3 figures. Submitted to ICASSP 2024摘要：我们介绍Matcha-TTS，一种新的编码器-解码器架构，用于快速TTS声学建模，使用最优传输条件流匹配（OT-CFM）进行训练。这产生了基于ODE的解码器，其能够在比使用分数匹配训练的模型更少的合成步骤中实现高输出质量。仔细的设计选择还确保每个合成步骤快速运行。该方法是概率性的，非自回归的，并且在没有外部对齐的情况下从头开始学习说话。与强大的预训练基线模型相比，Matcha-TTS系统具有最小的内存占用，可与最快的模型在长话语上的速度相媲美，并在听力测试中获得最高的平均意见得分。请访问https://shivammehta25.github.io/Matcha-TTS/获取音频示例、代码和预训练模型。摘要：We introduce Matcha-TTS, a new encoder-decoder architecture for speedy TTS acoustic modelling, trained using optimal-transport conditional flow matching (OT-CFM). This yields an ODE-based decoder capable of high output quality in fewer synthesis steps than models trained using score matching. Careful design choices additionally ensure each synthesis step is fast to run. The method is probabilistic, non-autoregressive, and learns to speak from scratch without external alignments. Compared to strong pre-trained baseline models, the Matcha-TTS system has the smallest memory footprint, rivals the speed of the fastest models on long utterances, and attains the highest mean opinion score in a listening test. Please see https://shivammehta25.github.io/Matcha-TTS/ for audio examples, code, and pre-trained models.
【9】 Real-time auralization for performers on virtual stages标题：虚拟舞台上表演者的实时可听化链接：https://arxiv.org/abs/2309.03149
作者：Ernesto Accolti,Lukas Aspöck,Manuj Yadav,Michael Vorländer机构：Instituto de Autom´atica (Institute of Automation), National University of San Juan, (UNSJ) and the National Scientific and Technical Research Council (CONICET), Argentina, Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany摘要：本文介绍了一个交互式舞台声学实验系统，包括考虑听到自己和他人的仪器。用于音乐表演的心理物理学实验的实时可听化系统的质量取决于系统的校准和延迟，以及其他因素（例如：视觉、模拟方法、触觉等）。所提出的系统侧重于实验室实现的声学考虑。校准被实施为一组滤波器占麦克风-仪器的距离和方向性因素，以及换能器的频率响应。此外，使用最新信息和来自校准滤波器的数学定义的推导来表征误差源。为了补偿硬件延迟而不裁剪模拟脉冲响应的部分，从模拟中跳过音乐家听到他们自己的虚拟直接声音，并且通过让实际直接声音通过开放式耳机到达收听者来解决。交互部分的所需延迟补偿（即，听其他人）满足音乐家之间的最小距离要求，对于所实现的系统，该最小距离为2 m。最后，一个概念的证明，包括客观和主观的实验，这给予支持所提出的设置的可行性。摘要：This article presents an interactive system for stage acoustics experimentation including considerations for hearing one's own and others' instruments. The quality of real-time auralization systems for psychophysical experiments on music performance depends on the system's calibration and latency, among other factors (e.g. visuals, simulation methods, haptics, etc). The presented system focuses on the acoustic considerations for laboratory implementations. The calibration is implemented as a set of filters accounting for the microphone-instrument distances and the directivity factors, as well as the transducers' frequency responses. Moreover, sources of errors are characterized using both state-of-the-art information and derivations from the mathematical definition of the calibration filter. In order to compensate for hardware latency without cropping parts of the simulated impulse responses, the virtual direct sound of musicians hearing themselves is skipped from the simulation and addressed by letting the actual direct sound reach the listener through open headphones. The required latency compensation of the interactive part (i.e. hearing others) meets the minimum distance requirement between musicians, which is 2 m for the implemented system. Finally, a proof of concept is provided that includes objective and subjective experiments, which give support to the feasibility of the proposed setup.
【10】 Quid Manumit -- Freeing the Qubit for Art标题：QUID MANUMY--为艺术释放量子比特链接：https://arxiv.org/abs/2309.03104
作者：Mark Carney机构：Quantum Village Inc., Delaware, USALondon, UK备注：8 pages, 6 figures, to appear at ISQCMC in Berlin, Oct 5-6th 2023摘要：本文描述了如何通过创建独立的量子音乐效果和乐器来为艺术“释放量子比特”。这里使用了之前发布的基于ARM的Raspberry Pi Pico嵌入式微控制器的量子模拟器代码，并构建了几个示例，展示了利用嵌入式资源的不同方法：第一个是Quantum MIDI处理器，它可以根据输入的音符生成伴奏和独特的量子生成乐器，解码并通过嵌入式模拟器中的量子电路。第二个是量子失真模块，根据量子电路改变乐器的原始声音，它以两种形式呈现;一个独立的量子笔琴，和一个名为'QubitCrusher'的效果模块插件Korg Nu：Tekt NTS-1。本文还讨论了量子仪器的未来工作和方向，并提供了所有开源示例。据作者所知，这是第一个嵌入式量子模拟器音乐仪器（另一个QSIM）的例子。摘要：This paper describes how to `Free the Qubit' for art, by creating standalone quantum musical effects and instruments. Previously released quantum simulator code for an ARM-based Raspberry Pi Pico embedded microcontroller is utilised here, and several examples are built demonstrating different methods of utilising embedded resources: The first is a Quantum MIDI processor that generates additional notes for accompaniment and unique quantum generated instruments based on the input notes, decoded and passed through a quantum circuit in an embedded simulator. The second is a Quantum Distortion module that changes an instrument's raw sound according to a quantum circuit, which is presented in two forms; a self-contained Quantum Stylophone, and an effect module plugin called 'QubitCrusher' for the Korg Nu:Tekt NTS-1. This paper also discusses future work and directions for quantum instruments, and provides all examples as open source. This is, to the author's knowledge, the first example of embedded Quantum Simulators for Instruments of Music (another QSIM).
【11】 Indoor Localization Using Radio, Vision and Audio Sensors: Real-Life  Data Validation and Discussion标题：使用无线电、视觉和音频传感器的室内定位：真实数据验证和讨论链接：https://arxiv.org/abs/2309.02961
作者：Ilayda Yaman,Guoda Tian,Erik Tegler,Patrik Persson,Nikhil Challa,Fredrik Tufvesson,Ove Edfors,Kalle Astrom,Steffen Malkowsky,Liang Liu机构：∗Dept. of Electrical and Information Technology, Lund University, Sweden, †Dept. of Centre for Mathematical Sciences, Lund University, Sweden备注：6 pages, 6 figures摘要：本文研究了室内定位方法，使用无线电，视觉，和音频传感器，分别在相同的环境。该评估基于最先进的算法，并使用真实数据集。更具体地说，我们评估了一个基于无线电的定位与大规模MIMO技术的机器学习算法，基于视觉的定位与RGB-D相机的ORB-SLAM 3算法，和基于音频的定位与麦克风阵列的SFS 2算法。包括定位精度，可靠性，校准要求，和潜在的系统复杂性方面进行了讨论，以分析使用不同的传感器的室内定位任务的优势和局限性。该结果可作为进一步开发鲁棒和高精度多传感器定位系统的指导和基础，例如，通过传感器融合和环境感知适应。摘要：This paper investigates indoor localization methods using radio, vision, and audio sensors, respectively, in the same environment. The evaluation is based on state-of-the-art algorithms and uses a real-life dataset. More specifically, we evaluate a machine learning algorithm for radio-based localization with massive MIMO technology, an ORB-SLAM3 algorithm for vision-based localization with an RGB-D camera, and an SFS2 algorithm for audio-based localization with microphone arrays. Aspects including localization accuracy, reliability, calibration requirements, and potential system complexity are discussed to analyze the advantages and limitations of using different sensors for indoor localization tasks. The results can serve as a guideline and basis for further development of robust and high-precision multi-sensory localization systems, e.g., through sensor fusion and context and environment-aware adaptation.
【12】 MuLanTTS The Microsoft Speech Synthesis System for Blizzard Challenge  2023标题：微软2023年暴雪挑战赛语音合成系统MuLanTTS链接：https://arxiv.org/abs/2309.02743
作者：Zhihang Xu,Shaofei Zhang,Xi Wang,Jiajun Zhang,Wenning Wei,Lei He,Sheng Zhao机构：Microsoft Azure Speech, China备注：6 pages摘要：在本文中，我们介绍了MuLanTTS，微软为暴雪挑战赛2023年设计的端到端神经文本转语音（TTS）系统。大约50小时的法语TTS有声读物语料库作为枢纽任务和另外2小时的说话人适应作为发言任务被释放，以构建用于不同测试目的的合成语音，包括句子，段落，同形词，列表等。在DelightfulTTS的基础上，我们采用上下文和情感编码器来适应有声读物数据，以丰富超越句子的长形式韵律和对话表现力。关于录音质量，我们也应用去噪算法和长音频处理两个语料库。对于集线器任务，仅使用50小时的单扬声器数据来构建TTS系统，而对于辐条任务，多扬声器源模型用于目标扬声器微调。MuLanTTS在各自任务中的质量评估平均得分为4.3和4.5，在统计上与自然语音相当，同时根据相似性评估保持良好的相似性。今年新增密集统计评比中的优秀品质和相似性。摘要：In this paper, we present MuLanTTS, the Microsoft end-to-end neural text-to-speech (TTS) system designed for the Blizzard Challenge 2023. About 50 hours of audiobook corpus for French TTS as hub task and another 2 hours of speaker adaptation as spoke task are released to build synthesized voices for different test purposes including sentences, paragraphs, homographs, lists, etc. Building upon DelightfulTTS, we adopt contextual and emotion encoders to adapt the audiobook data to enrich beyond sentences for long-form prosody and dialogue expressiveness. Regarding the recording quality, we also apply denoise algorithms and long audio processing for both corpora. For the hub task, only the 50-hour single speaker data is used for building the TTS system, while for the spoke task, a multi-speaker source model is used for target speaker fine tuning. MuLanTTS achieves mean scores of quality assessment 4.3 and 4.5 in the respective tasks, statistically comparable with natural speech while keeping good similarity according to similarity assessment. The excellent quality and similarity in this year's new and dense statistical evaluation.
【13】 Stylebook: Content-Dependent Speaking Style Modeling for Any-to-Any  Voice Conversion using Only Speech Data标题：Stylebook：仅使用语音数据进行Any-to-Any Voice转换的内容相关讲话风格建模链接：https://arxiv.org/abs/2309.02730
作者：Hyungseob Lim,Kyungguen Byun,Sunkuk Moon,Erik Visser机构：Department of Electrical and Electronic Engineering, Yonsei University, Seoul, South Korea,  Qualcomm Technologies, Inc., San Diego, California备注：5 pages, 2 figures, 2 tables, submitted to ICASSP 2024摘要：虽然许多最近的任意到任意语音转换模型成功地将一些目标语音的风格信息转移到转换后的语音，但它们仍然缺乏忠实地再现目标说话者的说话风格的能力。在这项工作中，我们提出了一种新的方法来提取丰富的风格信息从目标话语，并有效地将其转移到源语音内容，而不需要文本转录或扬声器标记。我们所提出的方法引入了一个注意力机制，利用自监督学习（SSL）模型来收集目标扬声器的说话风格，每个对应于不同的语音内容。这些样式用一组称为stylebook的嵌入表示。在下一步骤中，样本本与源语音的语音内容一起被加入，以确定每个源内容的最终目标风格。最后，从源语音和内容相关的目标风格嵌入提取的内容信息被馈送到基于扩散的解码器以生成转换的语音梅尔频谱图。实验结果表明，我们所提出的方法结合基于扩散的生成模型，可以实现更好的说话人相似性在任何到任何语音转换任务时，基线模型相比，而计算复杂度的增加与较长的话语被抑制。摘要：While many recent any-to-any voice conversion models succeed in transferring some target speech's style information to the converted speech, they still lack the ability to faithfully reproduce the speaking style of the target speaker. In this work, we propose a novel method to extract rich style information from target utterances and to efficiently transfer it to source speech content without requiring text transcriptions or speaker labeling. Our proposed approach introduces an attention mechanism utilizing a self-supervised learning (SSL) model to collect the speaking styles of a target speaker each corresponding to the different phonetic content. The styles are represented with a set of embeddings called stylebook. In the next step, the stylebook is attended with the source speech's phonetic content to determine the final target style for each source content. Finally, content information extracted from the source speech and content-dependent target style embeddings are fed into a diffusion-based decoder to generate the converted speech mel-spectrogram. Experiment results show that our proposed method combined with a diffusion-based generative model can achieve better speaker similarity in any-to-any voice conversion tasks when compared to baseline models, while the increase in computational complexity with longer utterances is suppressed.
【14】 BWSNet: Automatic Perceptual Assessment of Audio Signals标题：BWSNet：音频信号的自动感知评估链接：https://arxiv.org/abs/2309.02592
作者：Clément Le Moine Veillon,Victor Rosi,Pablo Arias Sarah,Léane Salais,Nicolas Obin机构：STMS Lab - IRCAM, CNRS, Sorbonne Universit´e, Paris, France, Department of Speech Hearing and Phonetic Sciences, University College London, London, UK, School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK摘要：本文介绍了BWSNet，一个模型，可以从原始的人类判断通过最佳最差缩放（BWS）实验进行训练。它将声音样本映射到一个嵌入的空间中，该空间代表了所研究的属性的感知。为此，我们提出了一组成本函数和约束，解释为距离比较在度量学习任务的审判顺序关系。我们测试了我们的建议从两个BWS研究调查的言语社会态度和音色质量的感知数据。对于这两个数据集，我们的研究结果表明，潜在空间的结构是忠实于人类的判断。摘要：This paper introduces BWSNet, a model that can be trained from raw human judgements obtained through a Best-Worst scaling (BWS) experiment. It maps sound samples into an embedded space that represents the perception of a studied attribute. To this end, we propose a set of cost functions and constraints, interpreting trial-wise ordinal relations as distance comparisons in a metric learning task. We tested our proposal on data from two BWS studies investigating the perception of speech social attitudes and timbral qualities. For both datasets, our results show that the structure of the latent space is faithful to human judgements.
【15】 Symbolic Music Representations for Classification Tasks: A Systematic  Evaluation标题：符号音乐表征在分类任务中的系统评价链接：https://arxiv.org/abs/2309.02567
作者：Huan Zhang,Emmanouil Karystinaios,Simon Dixon,Gerhard Widmer,Carlos Eduardo Cancino-Chacón机构：Queen Mary University of London, United Kingdom,  Johannes Kepler University, Austra备注：None摘要：音乐信息检索（MIR）已经见证了基于深度学习的方法的最近激增，其通常涉及对符号音乐进行编码（即，以离散音符事件表示的音乐）。然而，符号音乐既不是一个图像，也不是一个句子，在符号领域的研究缺乏一个全面的概述不同可用的表示。在本文中，我们研究矩阵（钢琴卷），序列和图形表示及其相应的神经架构，结合符号分数和性能的三个分片级别的分类任务。我们还介绍了一种新的图形表示的符号性能和探索能力的图形表示在全球分类任务。我们的系统评估显示了每种输入表示的优点和局限性。我们的研究结果表明，图形表示，作为最新的和最少的探索三种方法中，表现出良好的性能，同时更轻的重量在训练中。摘要：Music Information Retrieval (MIR) has seen a recent surge in deep learning-based approaches, which often involve encoding symbolic music (i.e., music represented in terms of discrete note events) in an image-like or language like fashion. However, symbolic music is neither an image nor a sentence, and research in the symbolic domain lacks a comprehensive overview of the different available representations. In this paper, we investigate matrix (piano roll), sequence, and graph representations and their corresponding neural architectures, in combination with symbolic scores and performances on three piece-level classification tasks. We also introduce a novel graph representation for symbolic performances and explore the capability of graph representations in global classification tasks. Our systematic evaluation shows advantages and limitations of each input representation. Our results suggest that the graph representation, as the newest and least explored among the three approaches, exhibits promising performance, while being more light-weight in training.
【16】 A Generalized Bandsplit Neural Network for Cinematic Audio Source  Separation标题：一种用于电影音频源分离的广义带状分裂神经网络链接：https://arxiv.org/abs/2309.02539
作者：Karn N. Watcharasupat,Chih-Wei Wu,Yiwei Ding,Iroro Orife,Aaron J. Hipple. Phillip A. Williams,Scott Kramer,Alexander Lerch,William Wolcott机构：Netflix Inc., Los Gatos, CA, USA ( indicates work done during an internship), Music Informatics Group, Georgia Institute of Technology, Atlanta, GA, USA备注：Submitted to ICASSP-OJSP 2024摘要：电影音频源分离是音频源分离的相对较新的子任务，其目的是从它们的混合物中提取对话干、音乐干和效果干。在这项工作中，我们开发了一个模型推广的频带分裂RNN的频率轴的任何完整或过完整分区。心理声学激励的频率尺度被用来通知频带定义，其现在被定义为具有更可靠的特征提取的冗余。提出了一种由信噪比和1-范数的稀疏性促进特性激励的损失函数。我们还利用一个共同的编码器设置的信息共享属性，以降低计算复杂性，在训练和推理过程中，提高分离性能，难以概括类的声音，并允许在推理时间的灵活性与容易拆卸的解码器。我们的最佳模型在Divide和Remaster数据集上设置了最先进的技术水平，其性能高于对话词干的理想比例掩码。摘要：Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue stem, the music stem, and the effects stem from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psycho-acoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with easily detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.
【17】 Minimal Effective Theory for Phonotactic Memory: Capturing Local  Correlations due to Errors in Speech标题：语音定向记忆的最小有效理论：捕捉语音错误引起的局部关联链接：https://arxiv.org/abs/2309.02466
作者：Paul Myles Eugenio机构：Sept备注：16 pages; 7 figs摘要：口语的发展受到言语经济性的制约，而言语经济性取决于诸如人类口腔结构等因素。这导致口语中的局部语音相关性。在这里，我们证明，这些本地的相关性促进口语的学习，减少他们的信息内容。我们这样做，通过构建一个局部连接的张量网络模型，启发类似的变分模型用于多体物理学，利用这些本地语音相关性，以促进口语学习。因此，该模型是语音记忆的最小模型，其中“学习发音”和“学习单词”是同一个。其结果是习得了产生与目标语言发音合理的新词的能力;以及提供在语音动作期间可能产生的最可能错误的层次结构。我们用拉丁语和土耳其语来测试我们的模型。(The代码在GitHub上可用。）摘要：Spoken language evolves constrained by the economy of speech, which depends on factors such as the structure of the human mouth. This gives rise to local phonetic correlations in spoken words. Here we demonstrate that these local correlations facilitate the learning of spoken words by reducing their information content. We do this by constructing a locally-connected tensor-network model, inspired by similar variational models used for many-body physics, which exploits these local phonetic correlations to facilitate the learning of spoken words. The model is therefore a minimal model of phonetic memory, where "learning to pronounce" and "learning a word" are one and the same. A consequence of which is the learned ability to produce new words which are phonetically reasonable for the target language; as well as providing a hierarchy of the most likely errors that could be produced during the action of speech. We test our model against Latin and Turkish words. (The code is available on GitHub.)
eess.AS音频处理
【1】 Matcha-TTS: A fast TTS architecture with conditional flow matching标题：Matcha-TTS：一种支持条件流匹配的快速TTS体系结构链接：https://arxiv.org/abs/2309.03199
作者：Shivam Mehta,Ruibo Tu,Jonas Beskow,Éva Székely,Gustav Eje Henter机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 3 figures. Submitted to ICASSP 2024摘要：我们介绍Matcha-TTS，一种新的编码器-解码器架构，用于快速TTS声学建模，使用最优传输条件流匹配（OT-CFM）进行训练。这产生了基于ODE的解码器，其能够在比使用分数匹配训练的模型更少的合成步骤中实现高输出质量。仔细的设计选择还确保每个合成步骤快速运行。该方法是概率性的，非自回归的，并且在没有外部对齐的情况下从头开始学习说话。与强大的预训练基线模型相比，Matcha-TTS系统具有最小的内存占用，可与最快的模型在长话语上的速度相媲美，并在听力测试中获得最高的平均意见得分。请访问https://shivammehta25.github.io/Matcha-TTS/获取音频示例、代码和预训练模型。摘要：We introduce Matcha-TTS, a new encoder-decoder architecture for speedy TTS acoustic modelling, trained using optimal-transport conditional flow matching (OT-CFM). This yields an ODE-based decoder capable of high output quality in fewer synthesis steps than models trained using score matching. Careful design choices additionally ensure each synthesis step is fast to run. The method is probabilistic, non-autoregressive, and learns to speak from scratch without external alignments. Compared to strong pre-trained baseline models, the Matcha-TTS system has the smallest memory footprint, rivals the speed of the fastest models on long utterances, and attains the highest mean opinion score in a listening test. Please see https://shivammehta25.github.io/Matcha-TTS/ for audio examples, code, and pre-trained models.
【2】 Real-time auralization for performers on virtual stages标题：虚拟舞台上表演者的实时可听化链接：https://arxiv.org/abs/2309.03149
作者：Ernesto Accolti,Lukas Aspöck,Manuj Yadav,Michael Vorländer机构：Instituto de Autom´atica (Institute of Automation), National University of San Juan, (UNSJ) and the National Scientific and Technical Research Council (CONICET), Argentina, Institute for Hearing Technology and Acoustics, RWTH Aachen University, Aachen, Germany摘要：本文介绍了一个交互式舞台声学实验系统，包括考虑听到自己和他人的仪器。用于音乐表演的心理物理学实验的实时可听化系统的质量取决于系统的校准和延迟，以及其他因素（例如：视觉、模拟方法、触觉等）。所提出的系统侧重于实验室实现的声学考虑。校准被实施为一组滤波器占麦克风-仪器的距离和方向性因素，以及换能器的频率响应。此外，使用最新信息和来自校准滤波器的数学定义的推导来表征误差源。为了补偿硬件延迟而不裁剪模拟脉冲响应的部分，从模拟中跳过音乐家听到他们自己的虚拟直接声音，并且通过让实际直接声音通过开放式耳机到达收听者来解决。交互部分的所需延迟补偿（即，听其他人）满足音乐家之间的最小距离要求，对于所实现的系统，该最小距离为2 m。最后，一个概念的证明，包括客观和主观的实验，这给予支持所提出的设置的可行性。摘要：This article presents an interactive system for stage acoustics experimentation including considerations for hearing one's own and others' instruments. The quality of real-time auralization systems for psychophysical experiments on music performance depends on the system's calibration and latency, among other factors (e.g. visuals, simulation methods, haptics, etc). The presented system focuses on the acoustic considerations for laboratory implementations. The calibration is implemented as a set of filters accounting for the microphone-instrument distances and the directivity factors, as well as the transducers' frequency responses. Moreover, sources of errors are characterized using both state-of-the-art information and derivations from the mathematical definition of the calibration filter. In order to compensate for hardware latency without cropping parts of the simulated impulse responses, the virtual direct sound of musicians hearing themselves is skipped from the simulation and addressed by letting the actual direct sound reach the listener through open headphones. The required latency compensation of the interactive part (i.e. hearing others) meets the minimum distance requirement between musicians, which is 2 m for the implemented system. Finally, a proof of concept is provided that includes objective and subjective experiments, which give support to the feasibility of the proposed setup.
【3】 Leveraging ASR Pretrained Conformers for Speaker Verification through  Transfer Learning and Knowledge Distillation标题：通过迁移学习和知识提取利用ASR预先训练的符号器进行说话人确认链接：https://arxiv.org/abs/2309.03019
作者：Danwei Cai,Ming Li摘要：本文探讨了使用ASR预训练的Conformers说话人验证，利用他们的优势建模语音信号。我们介绍三种策略：（1）迁移学习初始化说话人嵌入网络，提高泛化能力，减少过拟合。(2)知识蒸馏以训练更灵活的说话人确认模型，将帧级ASR损失作为辅助任务。(3)一款轻量级的扬声器适配器，可在不改变原始ASR Conformer的情况下实现高效的功能转换，允许并行ASR和扬声器验证。在VoxCeleb上的实验显示出显著的改进：迁移学习产生0.48%的EER，知识提炼产生0.43%的EER，和扬声器适配器方法，仅添加4.92M参数到130.94M参数模型，实现0.57%的EER。总的来说，我们的方法有效地转移ASR能力说话人验证任务。摘要：This paper explores the use of ASR-pretrained Conformers for speaker verification, leveraging their strengths in modeling speech signals. We introduce three strategies: (1) Transfer learning to initialize the speaker embedding network, improving generalization and reducing overfitting. (2) Knowledge distillation to train a more flexible speaker verification model, incorporating frame-level ASR loss as an auxiliary task. (3) A lightweight speaker adaptor for efficient feature conversion without altering the original ASR Conformer, allowing parallel ASR and speaker verification. Experiments on VoxCeleb show significant improvements: transfer learning yields a 0.48% EER, knowledge distillation results in a 0.43% EER, and the speaker adaptor approach, with just an added 4.92M parameters to a 130.94M-parameter model, achieves a 0.57% EER. Overall, our methods effectively transfer ASR capabilities to speaker verification tasks.
【4】 Indoor Localization Using Radio, Vision and Audio Sensors: Real-Life  Data Validation and Discussion标题：使用无线电、视觉和音频传感器的室内定位：真实数据验证和讨论链接：https://arxiv.org/abs/2309.02961
作者：Ilayda Yaman,Guoda Tian,Erik Tegler,Patrik Persson,Nikhil Challa,Fredrik Tufvesson,Ove Edfors,Kalle Astrom,Steffen Malkowsky,Liang Liu机构：∗Dept. of Electrical and Information Technology, Lund University, Sweden, †Dept. of Centre for Mathematical Sciences, Lund University, Sweden备注：6 pages, 6 figures摘要：本文研究了室内定位方法，使用无线电，视觉，和音频传感器，分别在相同的环境。该评估基于最先进的算法，并使用真实数据集。更具体地说，我们评估了一个基于无线电的定位与大规模MIMO技术的机器学习算法，基于视觉的定位与RGB-D相机的ORB-SLAM 3算法，和基于音频的定位与麦克风阵列的SFS 2算法。包括定位精度，可靠性，校准要求，和潜在的系统复杂性方面进行了讨论，以分析使用不同的传感器的室内定位任务的优势和局限性。该结果可作为进一步开发鲁棒和高精度多传感器定位系统的指导和基础，例如，通过传感器融合和环境感知适应。摘要：This paper investigates indoor localization methods using radio, vision, and audio sensors, respectively, in the same environment. The evaluation is based on state-of-the-art algorithms and uses a real-life dataset. More specifically, we evaluate a machine learning algorithm for radio-based localization with massive MIMO technology, an ORB-SLAM3 algorithm for vision-based localization with an RGB-D camera, and an SFS2 algorithm for audio-based localization with microphone arrays. Aspects including localization accuracy, reliability, calibration requirements, and potential system complexity are discussed to analyze the advantages and limitations of using different sensors for indoor localization tasks. The results can serve as a guideline and basis for further development of robust and high-precision multi-sensory localization systems, e.g., through sensor fusion and context and environment-aware adaptation.
【5】 MuLanTTS The Microsoft Speech Synthesis System for Blizzard Challenge  2023标题：微软2023年暴雪挑战赛语音合成系统MuLanTTS链接：https://arxiv.org/abs/2309.02743
作者：Zhihang Xu,Shaofei Zhang,Xi Wang,Jiajun Zhang,Wenning Wei,Lei He,Sheng Zhao机构：Microsoft Azure Speech, China备注：6 pages摘要：在本文中，我们介绍了MuLanTTS，微软为暴雪挑战赛2023年设计的端到端神经文本转语音（TTS）系统。大约50小时的法语TTS有声读物语料库作为枢纽任务和另外2小时的说话人适应作为发言任务被释放，以构建用于不同测试目的的合成语音，包括句子，段落，同形词，列表等。在DelightfulTTS的基础上，我们采用上下文和情感编码器来适应有声读物数据，以丰富超越句子的长形式韵律和对话表现力。关于录音质量，我们也应用去噪算法和长音频处理两个语料库。对于集线器任务，仅使用50小时的单扬声器数据来构建TTS系统，而对于辐条任务，多扬声器源模型用于目标扬声器微调。MuLanTTS在各自任务中的质量评估平均得分为4.3和4.5，在统计上与自然语音相当，同时根据相似性评估保持良好的相似性。今年新增密集统计评比中的优秀品质和相似性。摘要：In this paper, we present MuLanTTS, the Microsoft end-to-end neural text-to-speech (TTS) system designed for the Blizzard Challenge 2023. About 50 hours of audiobook corpus for French TTS as hub task and another 2 hours of speaker adaptation as spoke task are released to build synthesized voices for different test purposes including sentences, paragraphs, homographs, lists, etc. Building upon DelightfulTTS, we adopt contextual and emotion encoders to adapt the audiobook data to enrich beyond sentences for long-form prosody and dialogue expressiveness. Regarding the recording quality, we also apply denoise algorithms and long audio processing for both corpora. For the hub task, only the 50-hour single speaker data is used for building the TTS system, while for the spoke task, a multi-speaker source model is used for target speaker fine tuning. MuLanTTS achieves mean scores of quality assessment 4.3 and 4.5 in the respective tasks, statistically comparable with natural speech while keeping good similarity according to similarity assessment. The excellent quality and similarity in this year's new and dense statistical evaluation.
【6】 Stylebook: Content-Dependent Speaking Style Modeling for Any-to-Any  Voice Conversion using Only Speech Data标题：Stylebook：仅使用语音数据进行Any-to-Any Voice转换的内容相关讲话风格建模链接：https://arxiv.org/abs/2309.02730
作者：Hyungseob Lim,Kyungguen Byun,Sunkuk Moon,Erik Visser机构：Department of Electrical and Electronic Engineering, Yonsei University, Seoul, South Korea,  Qualcomm Technologies, Inc., San Diego, California备注：5 pages, 2 figures, 2 tables, submitted to ICASSP 2024摘要：虽然许多最近的任意到任意语音转换模型成功地将一些目标语音的风格信息转移到转换后的语音，但它们仍然缺乏忠实地再现目标说话者的说话风格的能力。在这项工作中，我们提出了一种新的方法来提取丰富的风格信息从目标话语，并有效地将其转移到源语音内容，而不需要文本转录或扬声器标记。我们所提出的方法引入了一个注意力机制，利用自监督学习（SSL）模型来收集目标扬声器的说话风格，每个对应于不同的语音内容。这些样式用一组称为stylebook的嵌入表示。在下一步骤中，样本本与源语音的语音内容一起被加入，以确定每个源内容的最终目标风格。最后，从源语音和内容相关的目标风格嵌入提取的内容信息被馈送到基于扩散的解码器以生成转换的语音梅尔频谱图。实验结果表明，我们所提出的方法结合基于扩散的生成模型，可以实现更好的说话人相似性在任何到任何语音转换任务时，基线模型相比，而计算复杂度的增加与较长的话语被抑制。摘要：While many recent any-to-any voice conversion models succeed in transferring some target speech's style information to the converted speech, they still lack the ability to faithfully reproduce the speaking style of the target speaker. In this work, we propose a novel method to extract rich style information from target utterances and to efficiently transfer it to source speech content without requiring text transcriptions or speaker labeling. Our proposed approach introduces an attention mechanism utilizing a self-supervised learning (SSL) model to collect the speaking styles of a target speaker each corresponding to the different phonetic content. The styles are represented with a set of embeddings called stylebook. In the next step, the stylebook is attended with the source speech's phonetic content to determine the final target style for each source content. Finally, content information extracted from the source speech and content-dependent target style embeddings are fed into a diffusion-based decoder to generate the converted speech mel-spectrogram. Experiment results show that our proposed method combined with a diffusion-based generative model can achieve better speaker similarity in any-to-any voice conversion tasks when compared to baseline models, while the increase in computational complexity with longer utterances is suppressed.
【7】 BWSNet: Automatic Perceptual Assessment of Audio Signals标题：BWSNet：音频信号的自动感知评估链接：https://arxiv.org/abs/2309.02592
作者：Clément Le Moine Veillon,Victor Rosi,Pablo Arias Sarah,Léane Salais,Nicolas Obin机构：STMS Lab - IRCAM, CNRS, Sorbonne Universit´e, Paris, France, Department of Speech Hearing and Phonetic Sciences, University College London, London, UK, School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK摘要：本文介绍了BWSNet，一个模型，可以从原始的人类判断通过最佳最差缩放（BWS）实验进行训练。它将声音样本映射到一个嵌入的空间中，该空间代表了所研究的属性的感知。为此，我们提出了一组成本函数和约束，解释为距离比较在度量学习任务的审判顺序关系。我们测试了我们的建议从两个BWS研究调查的言语社会态度和音色质量的感知数据。对于这两个数据集，我们的研究结果表明，潜在空间的结构是忠实于人类的判断。摘要：This paper introduces BWSNet, a model that can be trained from raw human judgements obtained through a Best-Worst scaling (BWS) experiment. It maps sound samples into an embedded space that represents the perception of a studied attribute. To this end, we propose a set of cost functions and constraints, interpreting trial-wise ordinal relations as distance comparisons in a metric learning task. We tested our proposal on data from two BWS studies investigating the perception of speech social attitudes and timbral qualities. For both datasets, our results show that the structure of the latent space is faithful to human judgements.
【8】 Symbolic Music Representations for Classification Tasks: A Systematic  Evaluation标题：符号音乐表征在分类任务中的系统评价链接：https://arxiv.org/abs/2309.02567
作者：Huan Zhang,Emmanouil Karystinaios,Simon Dixon,Gerhard Widmer,Carlos Eduardo Cancino-Chacón机构：Queen Mary University of London, United Kingdom,  Johannes Kepler University, Austra备注：None摘要：音乐信息检索（MIR）已经见证了基于深度学习的方法的最近激增，其通常涉及对符号音乐进行编码（即，以离散音符事件表示的音乐）。然而，符号音乐既不是一个图像，也不是一个句子，在符号领域的研究缺乏一个全面的概述不同可用的表示。在本文中，我们研究矩阵（钢琴卷），序列和图形表示及其相应的神经架构，结合符号分数和性能的三个分片级别的分类任务。我们还介绍了一种新的图形表示的符号性能和探索能力的图形表示在全球分类任务。我们的系统评估显示了每种输入表示的优点和局限性。我们的研究结果表明，图形表示，作为最新的和最少的探索三种方法中，表现出良好的性能，同时更轻的重量在训练中。摘要：Music Information Retrieval (MIR) has seen a recent surge in deep learning-based approaches, which often involve encoding symbolic music (i.e., music represented in terms of discrete note events) in an image-like or language like fashion. However, symbolic music is neither an image nor a sentence, and research in the symbolic domain lacks a comprehensive overview of the different available representations. In this paper, we investigate matrix (piano roll), sequence, and graph representations and their corresponding neural architectures, in combination with symbolic scores and performances on three piece-level classification tasks. We also introduce a novel graph representation for symbolic performances and explore the capability of graph representations in global classification tasks. Our systematic evaluation shows advantages and limitations of each input representation. Our results suggest that the graph representation, as the newest and least explored among the three approaches, exhibits promising performance, while being more light-weight in training.
【9】 A Generalized Bandsplit Neural Network for Cinematic Audio Source  Separation标题：一种用于电影音频源分离的广义带状分裂神经网络链接：https://arxiv.org/abs/2309.02539
作者：Karn N. Watcharasupat,Chih-Wei Wu,Yiwei Ding,Iroro Orife,Aaron J. Hipple. Phillip A. Williams,Scott Kramer,Alexander Lerch,William Wolcott机构：Netflix Inc., Los Gatos, CA, USA ( indicates work done during an internship), Music Informatics Group, Georgia Institute of Technology, Atlanta, GA, USA备注：Submitted to ICASSP-OJSP 2024摘要：电影音频源分离是音频源分离的相对较新的子任务，其目的是从它们的混合物中提取对话干、音乐干和效果干。在这项工作中，我们开发了一个模型推广的频带分裂RNN的频率轴的任何完整或过完整分区。心理声学激励的频率尺度被用来通知频带定义，其现在被定义为具有更可靠的特征提取的冗余。提出了一种由信噪比和1-范数的稀疏性促进特性激励的损失函数。我们还利用一个共同的编码器设置的信息共享属性，以降低计算复杂性，在训练和推理过程中，提高分离性能，难以概括类的声音，并允许在推理时间的灵活性与容易拆卸的解码器。我们的最佳模型在Divide和Remaster数据集上设置了最先进的技术水平，其性能高于对话词干的理想比例掩码。摘要：Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue stem, the music stem, and the effects stem from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psycho-acoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with easily detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.
【10】 Minimal Effective Theory for Phonotactic Memory: Capturing Local  Correlations due to Errors in Speech标题：语音定向记忆的最小有效理论：捕捉语音错误引起的局部关联链接：https://arxiv.org/abs/2309.02466
作者：Paul Myles Eugenio机构：Sept备注：16 pages; 7 figs摘要：口语的发展受到言语经济性的制约，而言语经济性取决于诸如人类口腔结构等因素。这导致口语中的局部语音相关性。在这里，我们证明，这些本地的相关性促进口语的学习，减少他们的信息内容。我们这样做，通过构建一个局部连接的张量网络模型，启发类似的变分模型用于多体物理学，利用这些本地语音相关性，以促进口语学习。因此，该模型是语音记忆的最小模型，其中“学习发音”和“学习单词”是同一个。其结果是习得了产生与目标语言发音合理的新词的能力;以及提供在语音动作期间可能产生的最可能错误的层次结构。我们用拉丁语和土耳其语来测试我们的模型。(The代码在GitHub上可用。）摘要：Spoken language evolves constrained by the economy of speech, which depends on factors such as the structure of the human mouth. This gives rise to local phonetic correlations in spoken words. Here we demonstrate that these local correlations facilitate the learning of spoken words by reducing their information content. We do this by constructing a locally-connected tensor-network model, inspired by similar variational models used for many-body physics, which exploits these local phonetic correlations to facilitate the learning of spoken words. The model is therefore a minimal model of phonetic memory, where "learning to pronounce" and "learning a word" are one and the same. A consequence of which is the learned ability to produce new words which are phonetically reasonable for the target language; as well as providing a hierarchy of the most likely errors that could be produced during the action of speech. We test our model against Latin and Turkish words. (The code is available on GitHub.)
【11】 Quid Manumit -- Freeing the Qubit for Art标题：QUID MANUMY--为艺术释放量子比特链接：https://arxiv.org/abs/2309.03104
作者：Mark Carney机构：Quantum Village Inc., Delaware, USALondon, UK备注：8 pages, 6 figures, to appear at ISQCMC in Berlin, Oct 5-6th 2023摘要：本文描述了如何通过创建独立的量子音乐效果和乐器来为艺术“释放量子比特”。这里使用了之前发布的基于ARM的Raspberry Pi Pico嵌入式微控制器的量子模拟器代码，并构建了几个示例，展示了利用嵌入式资源的不同方法：第一个是Quantum MIDI处理器，它可以根据输入的音符生成伴奏和独特的量子生成乐器，解码并通过嵌入式模拟器中的量子电路。第二个是量子失真模块，根据量子电路改变乐器的原始声音，它以两种形式呈现;一个独立的量子笔琴，和一个名为'QubitCrusher'的效果模块插件Korg Nu：Tekt NTS-1。本文还讨论了量子仪器的未来工作和方向，并提供了所有开源示例。据作者所知，这是第一个嵌入式量子模拟器音乐仪器（另一个QSIM）的例子。摘要：This paper describes how to `Free the Qubit' for art, by creating standalone quantum musical effects and instruments. Previously released quantum simulator code for an ARM-based Raspberry Pi Pico embedded microcontroller is utilised here, and several examples are built demonstrating different methods of utilising embedded resources: The first is a Quantum MIDI processor that generates additional notes for accompaniment and unique quantum generated instruments based on the input notes, decoded and passed through a quantum circuit in an embedded simulator. The second is a Quantum Distortion module that changes an instrument's raw sound according to a quantum circuit, which is presented in two forms; a self-contained Quantum Stylophone, and an effect module plugin called 'QubitCrusher' for the Korg Nu:Tekt NTS-1. This paper also discusses future work and directions for quantum instruments, and provides all examples as open source. This is, to the author's knowledge, the first example of embedded Quantum Simulators for Instruments of Music (another QSIM).
【12】 An Efficient Temporary Deepfake Location Approach Based Embeddings for  Partially Spoofed Audio Detection标题：一种有效的基于嵌入的临时深伪定位方法用于部分欺骗音频检测链接：https://arxiv.org/abs/2309.03036
作者：Yuankun Xie,Haonan Cheng,Yutian Wang,Long Ye机构：State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing , China备注：Submitted to ICASSP 2024摘要：部分欺骗音频检测是一项具有挑战性的任务，在于需要在帧级别准确地定位音频的真实性。为了解决这个问题，我们提出了一种细粒度的部分欺骗音频检测方法，即时间深度假定位（TDL），它可以有效地捕获特征和位置的信息。具体来说，我们的方法涉及两个新颖的部分：嵌入相似度模块和时间卷积运算。为了增强真实和假特征的识别，嵌入相似度模块被设计为生成一个嵌入空间，将真实帧和假帧分离开来。为了有效地集中在位置信息，时间卷积运算被提出来计算相邻帧之间的帧特定的相似性，并动态地选择有信息的邻居进行卷积。大量实验表明，我们的方法在ASVspoof 2019部分欺骗数据集中优于基线模型，即使在交叉数据集场景中也表现出优异的性能。代码在网上发布。摘要：Partially spoofed audio detection is a challenging task, lying in the need to accurately locate the authenticity of audio at the frame level. To address this issue, we propose a fine-grained partially spoofed audio detection method, namely Temporal Deepfake Location (TDL), which can effectively capture information of both features and locations. Specifically, our approach involves two novel parts: embedding similarity module and temporal convolution operation. To enhance the identification between the real and fake features, the embedding similarity module is designed to generate an embedding space that can separate the real frames from fake frames. To effectively concentrate on the position information, temporal convolution operation is proposed to calculate the frame-specific similarities among neighboring frames, and dynamically select informative neighbors to convolution. Extensive experiments show that our method outperform baseline models in ASVspoof2019 Partial Spoof dataset and demonstrate superior performance even in the crossdataset scenario. The code is released online.
【13】 BigVSAN: Enhancing GAN-based Neural Vocoders with Slicing Adversarial  Network标题：BigVSAN：用分片对抗网络增强基于GAN的神经声码器链接：https://arxiv.org/abs/2309.02836
作者：Takashi Shibuya,Yuhta Takida,Yuki Mitsufuji机构：Sony AI, Tokyo, Japan, Sony Group Corporation, Tokyo, Japan备注：Submitted to ICASSP 2024摘要：基于生成对抗网络（GAN）的声码器已经被广泛研究，因为它们可以比实时更快地合成高保真音频波形。然而，据报道，大多数GANs未能获得用于在特征空间中区分真实和虚假数据的最佳投影。在文献中，它已经被证明，切片对抗网络（SAN），一个改进的GAN训练框架，可以找到最佳投影，在图像生成任务是有效的。在本文中，我们调查的有效性SAN的语音编码任务。为此，我们提出了一个方案来修改最小二乘GAN，其中大多数基于GAN的声码器采用，使其损失函数满足SAN的要求。通过我们的实验，我们证明SAN可以提高性能的基于GAN的声码器，包括BigVGAN，与小的修改。我们的代码可在https://github.com/sony/bigvsan上获得。摘要：Generative adversarial network (GAN)-based vocoders have been intensively studied because they can synthesize high-fidelity audio waveforms faster than real-time. However, it has been reported that most GANs fail to obtain the optimal projection for discriminating between real and fake data in the feature space. In the literature, it has been demonstrated that slicing adversarial network (SAN), an improved GAN training framework that can find the optimal projection, is effective in the image generation task. In this paper, we investigate the effectiveness of SAN in the vocoding task. For this purpose, we propose a scheme to modify least-squares GAN, which most GAN-based vocoders adopt, so that their loss functions satisfy the requirements of SAN. Through our experiments, we demonstrate that SAN can improve the performance of GAN-based vocoders, including BigVGAN, with small modifications. Our code is available at https://github.com/sony/bigvsan.
【14】 Self-Supervised Disentanglement of Harmonic and Rhythmic Features in  Music Audio Signals标题：音乐音频信号中和声和节奏特征的自监督解缠链接：https://arxiv.org/abs/2309.02796
作者：Yiming Wu机构：AlphaTheta Corporation, Yokohama, Japan备注：Accepted to DAFx 2023摘要：潜在变量解纠缠的目的是推断出数据生成过程背后的多个信息潜在表示，并且是可控数据生成的关键因素。在本文中，我们提出了一种基于深度神经网络的自监督学习方法来推断音乐音频生成背后的解开的节奏和和声表示。我们训练一个变分自动编码器，生成一个音频梅尔频谱图，从两个潜在的特征代表的节奏和谐波的内容。在训练阶段，训练变分自动编码器以在给定其音高移位版本的情况下重建输入梅尔频谱图。在训练阶段中的每个前向计算处，假设特征向量的维度与音调间隔相关，向量旋转操作被应用于潜在特征中的一个。因此，在训练的变分自动编码器中，旋转的潜在特征表示梅尔谱图的音高相关信息，而未旋转的潜在特征表示音高不变信息，即节奏的内容。所提出的方法进行了评估，使用基于预测器的解纠缠度量的学习特征。此外，我们展示了它的应用程序自动生成的音乐混音。摘要：The aim of latent variable disentanglement is to infer the multiple informative latent representations that lie behind a data generation process and is a key factor in controllable data generation. In this paper, we propose a deep neural network-based self-supervised learning method to infer the disentangled rhythmic and harmonic representations behind music audio generation. We train a variational autoencoder that generates an audio mel-spectrogram from two latent features representing the rhythmic and harmonic content. In the training phase, the variational autoencoder is trained to reconstruct the input mel-spectrogram given its pitch-shifted version. At each forward computation in the training phase, a vector rotation operation is applied to one of the latent features, assuming that the dimensions of the feature vectors are related to pitch intervals. Therefore, in the trained variational autoencoder, the rotated latent feature represents the pitch-related information of the mel-spectrogram, and the unrotated latent feature represents the pitch-invariant information, i.e., the rhythmic content. The proposed method was evaluated using a predictor-based disentanglement metric on the learned features. Furthermore, we demonstrate its application to the automatic generation of music remixes.
【15】 GRASS: Unified Generation Model for Speech Semantic Understanding标题：GRASS：语音语义理解的统一生成模型链接：https://arxiv.org/abs/2309.02780
作者：Aobo Xia,Shuyu Lei,Yushu Yang,Xiang Guo,Hua Chai机构：⋆ Beijing University of Posts and Telecommunications, Beijing, China, † Didi Chuxing, Beijing, China摘要：本文探讨了语音语义理解的指令微调技术，通过引入一个统一的端到端（E2E）的框架，生成语义标签的条件下，任务相关的提示音频数据。我们预训练模型使用大量和不同的数据，其中指令语音对构建通过一个文本到语音（TTS）系统。大量的实验表明，我们提出的模型显着优于国家的最先进的（SOTA）模型后微调下游任务。此外，所提出的模型实现了有竞争力的性能在zero-shot和Few-Shot的情况下。为了便于未来的语音到语义任务的指令微调工作，我们发布了我们的指令数据集和代码。摘要：This paper explores the instruction fine-tuning technique for speech semantic understanding by introducing a unified end-to-end (E2E) framework that generates semantic labels conditioned on a task-related prompt for audio data. We pre-train the model using large and diverse data, where instruction-speech pairs are constructed via a text-to-speech (TTS) system. Extensive experiments demonstrate that our proposed model significantly outperforms state-of-the-art (SOTA) models after fine-tuning downstream tasks. Furthermore, the proposed model achieves competitive performance in zero-shot and few-shot scenarios. To facilitate future work on instruction fine-tuning for speech-to-semantic tasks, we release our instruction dataset and code.
【16】 Simultaneous Measurement of Multiple Acoustic Attributes Using  Structured Periodic Test Signals Including Music and Other Sound Materials标题：使用包括音乐和其他声学材料的结构化周期性测试信号同时测量多个声学属性链接：https://arxiv.org/abs/2309.02767
作者：Hideki Kawahara,Kohei Yatabe,Ken-Ichi Sakakibara,Mitsunori Mizumachi,Tatsuya Kitamura机构：∗ Wakayama University, Japan, † Tokyo University of Agriculture and Technology, Japan, ‡ Health Sciences University of Hokkaido, Japan, § Kyushu Institute of Technology, Japan, ¶ Konan University, Japan备注：8 pages, 17 figures, accepted for APSIPA ASC 2023摘要：我们介绍了一个通用的框架，用于测量声学特性，如线性时不变（LTI）的响应，信号相关的时不变（SDTI）的组件，随机和时变（RTV）的组件，同时使用结构化的周期性测试信号。该框架还通过添加轻微的确定性噪声来“保护”音乐作品和其他声音材料作为测试信号。“使用扫频正弦、MLS（Maxim Length Sequence）及其变体的测量是所提出框架的特殊情况。我们实现了基于该框架的交互式和实时测量工具，并使其开源。此外，我们应用这个框架来客观地评估沥青提取器。摘要：We introduce a general framework for measuring acoustic properties such as liner time-invariant (LTI) response, signal-dependent time-invariant (SDTI) component, and random and time-varying (RTV) component simultaneously using structured periodic test signals. The framework also enables music pieces and other sound materials as test signals by "safeguarding" them by adding slight deterministic "noise." Measurement using swept-sin, MLS (Maxim Length Sequence), and their variants are special cases of the proposed framework. We implemented interactive and real-time measuring tools based on this framework and made them open-source. Furthermore, we applied this framework to assess pitch extractors objectively.
【17】 Music Source Separation with Band-Split RoPE Transformer标题：基于分带式绳索Transformer的音乐源分离链接：https://arxiv.org/abs/2309.02612
作者：Wei-Tsung Lu,Ju-Chiang Wang,Qiuqiang Kong,Yun-Ning Hung机构：SAMI, ByteDance备注：This paper explains the SAMI-ByteDance MSS system submitted to Sound Demixing Challenge (SDX23) Music Separation Track摘要：音乐源分离（MSS）旨在将音乐录音分离成多个音乐上不同的主干，例如人声、低音、鼓等。最近，已经使用了卷积神经网络（CNN）和递归神经网络（RNN）等深度学习方法，但改进仍然有限。在本文中，我们提出了一种新的频域方法的基础上的带分裂RoPE Transformer（称为BS-RoFormer）。BS-RoFormer在频带分割模块上回复，将输入的复杂频谱图投影到子频带级表示中，然后安排一堆分层Transformers来建模带内以及带间序列，用于多频带掩码估计。为了便于训练MSS模型，我们建议使用旋转位置嵌入（RoPE）。在MUSDB 18 HQ和500首额外歌曲训练的BS-RoFormer系统在Sound Demixing Challenge（SDX 23）MSS曲目中排名第一。在MUSDB 18 HQ上对BS-RoFormer的小型版本进行基准测试，我们在没有额外训练数据的情况下实现了最先进的结果，平均SDR为9.80 dB。摘要：Music source separation (MSS) aims to separate a music recording into multiple musically distinct stems, such as vocals, bass, drums, and more. Recently, deep learning approaches such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been used, but the improvement is still limited. In this paper, we propose a novel frequency-domain approach based on a Band-Split RoPE Transformer (called BS-RoFormer). BS-RoFormer replies on a band-split module to project the input complex spectrogram into subband-level representations, and then arranges a stack of hierarchical Transformers to model the inner-band as well as inter-band sequences for multi-band mask estimation. To facilitate training the model for MSS, we propose to use the Rotary Position Embedding (RoPE). The BS-RoFormer system trained on MUSDB18HQ and 500 extra songs ranked the first place in the MSS track of Sound Demixing Challenge (SDX23). Benchmarking a smaller version of BS-RoFormer on MUSDB18HQ, we achieve state-of-the-art result without extra training data, with 9.80 dB of average SDR.
【18】 Text-Only Domain Adaptation for End-to-End Speech Recognition through  Down-Sampling Acoustic Representation标题：通过下采样声学表示实现端到端语音识别的纯文本域自适应链接：https://arxiv.org/abs/2309.02459
作者：Jiaxu Zhu,Weinan Tong,Yaoxun Xu,Changhe Song,Zhiyong Wu,Zhao You,Dan Su,Dong Yu,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Peng Cheng Lab, Shenzhen, China, Tencent AI Lab, Shenzhen, China, Tencent AI Lab, Bellevue, WA, USA, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by INTERSPEECH 2023. arXiv admin note: text overlap with arXiv:2309.01437摘要：将语音和文本两种模态映射到共享的表示空间中，是使用纯文本数据来提高新领域中的端到端自动语音识别（ASR）性能的研究课题。然而，语音表示和文本表示的长度是不一致的。尽管先前的方法对文本表示进行上采样以与声学模态对齐，但其可能不匹配预期的实际持续时间。在本文中，我们提出了新的表示匹配策略，通过下采样声学表示对齐文本模态。通过引入一个连续的集成和消防（CIF）模块生成与令牌长度一致的声学表示，我们的ASR模型可以更好地从这两种模式学习统一的表示，允许使用目标域的纯文本数据的域自适应。新领域数据的实验结果表明了该方法的有效性。摘要：Mapping two modalities, speech and text, into a shared representation space, is a research topic of using text-only data to improve end-to-end automatic speech recognition (ASR) performance in new domains. However, the length of speech representation and text representation is inconsistent. Although the previous method up-samples the text representation to align with acoustic modality, it may not match the expected actual duration. In this paper, we proposed novel representations match strategy through down-sampling acoustic representation to align with text modality. By introducing a continuous integrate-and-fire (CIF) module generating acoustic representations consistent with token length, our ASR model can learn unified representations from both modalities better, allowing for domain adaptation using text-only data of the target domain. Experiment results of new domain data demonstrate the effectiveness of the proposed method.
机器翻译，仅供参考