今日论文合集：cs.SD语音5篇，eess.AS音频处理5篇。
cs.SD语音
【1】 Seamless: Multilingual Expressive and Streaming Speech Translation标题：无缝：多语言表现力和流媒体语音翻译链接：https://arxiv.org/abs/2312.05187
作者：Seamless Communication,Loïc Barrault,Yu-An Chung,Mariano Coria Meglioli,David Dale,Ning Dong,Mark Duppenthaler,Paul-Ambroise Duquenne,Brian Ellis,Hady Elsahar,Justin Haaheim,John Hoffman,Min-Jae Hwang,Hirofumi Inaguma,Christopher Klaiber,Ilia Kulikov,Pengwei Li,Daniel Licht,Jean Maillard,Ruslan Mavlyutov,Alice Rakotoarison,Kaushik Ram Sadagopan,Abinesh Ramakrishnan,Tuan Tran,Guillaume Wenzek,Yilin Yang,Ethan Ye,Ivan Evtimov,Pierre Fernandez,Cynthia Gao,Prangthip Hansanti,Elahe Kalbassi,Amanda Kallet,Artyom Kozhevnikov,Gabriel Mejia Gonzalez,Robin San Roman,Christophe Touret,Corinne Wong,Carleigh Wood,Bokai Yu,Pierre Andrews,Can Balioglu,Peng-Jen Chen,Marta R. Costa-jussà,Maha Elbayad,Hongyu Gong,Francisco Guzmán,Kevin Heffernan,Somya Jain,Justine Kao,Ann Lee,Xutai Ma,Alex Mourachko,Benjamin Peloquin,Juan Pino,Sravya Popuri,Christophe Ropers,Safiyyah Saleem,Holger Schwenk,Anna Sun,Paden Tomasello,Changhan Wang,Jeff Wang,Skyler Wang,Mary Williamson摘要：与人与人之间的对话相比，今天的大规模自动语音翻译系统缺乏帮助机器介导的通信感觉无缝的关键功能。在这项工作中，我们介绍了一系列的模型，使端到端的表达和多语言的翻译在流的方式。首先，我们贡献了一个改进版本的大规模多语言和多模式的multicellessM 4 T模型multicellessM 4 T v2。这个更新的模型，结合了更新的UnitY 2框架，在更多的低资源语言数据上进行了训练。无障碍M4 T v2为我们接下来的两个模型提供了基础。无歧义表达使翻译能够保留声乐风格和韵律。与以前的表达性语音研究相比，我们的工作解决了韵律的某些未充分探索的方面，如语速和停顿，同时也保留了一个人的声音的风格。至于无障碍流，我们的模型利用高效单调多头注意力机制来生成低延迟的目标翻译，而无需等待完整的源话语。作为同类产品中的第一个，无障碍流能够为多种源语言和目标语言同时进行语音到语音/文本翻译。为了确保我们的模型可以安全和负责任地使用，我们实施了第一个已知的多模式机器翻译的红队工作，一个检测和减轻附加毒性的系统，一个系统的性别偏见评估，以及一个旨在抑制deepfakes影响的听不见的本地化水印机制。因此，我们把主要的组件从millessExpressive和millessStreaming一起形成无缝，第一个公开可用的系统，实时解锁表达跨语言通信。对这项工作的贡献是公开发布的，可在https://github.com/facebookresearch/seamless_communication上访问摘要：Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication
【2】 neural concatenative singing voice conversion: rethinking  concatenation-based approach for one-shot singing voice conversion标题：神经元拼接歌声转换：基于拼接的单次歌声转换方法的再思考链接：https://arxiv.org/abs/2312.04919
作者：Binzhu Sha,Xu Li,Zhiyong Wu,Ying Shan,Helen Meng摘要：任意到任意的歌唱声音转换面临着由内容和扬声器音色之间的不充分分离引起的“音色泄漏”问题的重大挑战。为了解决这个问题，本研究介绍了一种新的神经级联歌唱语音转换（NeuCoSVC）框架。NeuCoSVC框架包括自监督学习（SSL）表示提取器、神经谐波信号发生器和波形合成器。具体地，SSL提取器将音频压缩成固定维度的SSL特征的序列。谐波信号产生器通过利用线性时变（LTV）滤波器产生原始谐波信号及经滤波谐波信号两者作为音调信息。最后，音频发生器基于SSL特征以及谐波信号和响度信息重构音频波形。在推断期间，系统通过用来自匹配池的源SSL特征的最接近的对应物替换源SSL特征来执行语音转换，匹配池包括从目标音频提取的SSL表示，而原始谐波信号和响度从源音频提取并且保持不变。由于在转换阶段中利用的SSL特征直接来自目标音频，因此所提出的框架具有很大的潜力来解决由先前的基于解纠缠的方法引起的“音色泄漏”问题。实验结果证实，所提出的系统提供了更好的性能比说话人嵌入方法（基于解纠缠）的上下文中的一杆SVC跨语言内，跨语言，跨域评估。摘要：Any-to-any singing voice conversion is confronted with a significant challenge of ``timbre leakage'' issue caused by inadequate disentanglement between the content and the speaker timbre. To address this issue, this study introduces a novel neural concatenative singing voice conversion (NeuCoSVC) framework. The NeuCoSVC framework comprises a self-supervised learning (SSL) representation extractor, a neural harmonic signal generator, and a waveform synthesizer. Specifically, the SSL extractor condenses the audio into a sequence of fixed-dimensional SSL features. The harmonic signal generator produces both raw and filtered harmonic signals as the pitch information by leveraging a linear time-varying (LTV) filter. Finally, the audio generator reconstructs the audio waveform based on the SSL features, as well as the harmonic signals and the loudness information. During inference, the system performs voice conversion by substituting source SSL features with their nearest counterparts from a matching pool, which comprises SSL representations extracted from the target audio, while the raw harmonic signals and the loudness are extracted from the source audio and are kept unchanged. Since the utilized SSL features in the conversion stage are directly from the target audio, the proposed framework has great potential to address the ``timbre leakage'' issue caused by previous disentanglement-based approaches. Experimental results confirm that the proposed system delivers much better performance than the speaker embedding approach (disentanglement-based) in the context of one-shot SVC across intra-language, cross-language, and cross-domain evaluations.
【3】 Sound Source Localization for a Source inside a Structure using  Ac-CycleGAN标题：基于AC-CycleGAN的结构内部声源定位链接：https://arxiv.org/abs/2312.04846
作者：Shunsuke Kita,Choong Sik Park,Yoshinobu Kajikawa摘要：我们提出了一种在不成对数据条件下使用Ac-CycleGAN对结构内部的源进行声源定位（SSL）的方法。该方法利用大量的模拟数据和少量的实际实验数据来定位真实环境中的结构内部的声源。Ac-CycleGAN生成器有助于将模拟数据转换为真实数据，反之亦然，使用来自两个域的未配对数据。Ac-CycleGAN模型的转换旨在区分由发生器生成的转换数据和真实数据，同时还预测声源的位置。代表在结构外三个点测量的加速度计（FSA）的频谱的矢量被用作输入数据，结构内的源区域被用作标签。输入数据向量垂直连接以形成图像。标签是通过将结构的内部划分为八个区域来定义的，每个区域都有一个热编码。因此，SSL问题被重新定义为图像分类问题，以随机估计声源的位置。我们表明，它是可能的，以估计声源的位置，使用Ac-CycleGAN跨域的未配对数据的估计。此外，我们分析了区分数据的判别因素。在80%的实际数据（12.5%的模拟数据）上训练后，该模型的准确率超过90%。尽管Ac-CycleGAN生成器执行的域转换过程中存在潜在的缺陷，但通过选择性地仅利用那些生成相对较小的转换误差的特征，该方法可以有效地区分传输数据和真实数据。摘要：We propose a method for sound source localization (SSL) for a source inside a structure using Ac-CycleGAN under unpaired data conditions. The proposed method utilizes a large amount of simulated data and a small amount of actual experimental data to locate a sound source inside a structure in a real environment. An Ac-CycleGAN generator contributes to the transformation of simulated data into real data, or vice versa, using unpaired data from both domains. The discriminator of an Ac-CycleGAN model is designed to differentiate between the transformed data generated by the generator and real data, while also predicting the location of the sound source. Vectors representing the frequency spectrum of the accelerometers (FSAs) measured at three points outside the structure are used as input data and the source areas inside the structure are used as labels. The input data vectors are concatenated vertically to form an image. Labels are defined by dividing the interior of the structure into eight areas with one-hot encoding for each area. Thus, the SSL problem is redefined as an image-classification problem to stochastically estimate the location of the sound source. We show that it is possible to estimate the sound source location using the Ac-CycleGAN discriminator for unpaired data across domains. Furthermore, we analyze the discriminative factors for distinguishing the data. The proposed model exhibited an accuracy exceeding 90\% when trained on 80\% of actual data (12.5\% of simulated data). Despite potential imperfections in the domain transformation process carried out by the Ac-CycleGAN generator, the discriminator can effectively distinguish between transferred and real data by selectively utilizing only those features that generate a relatively small transformation error.
【4】 SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and  Exploration标题：SynthScribe：用于合成器声音检索和探索的深度多通道工具链接：https://arxiv.org/abs/2312.04690
作者：Stephen Brade,Bryan Wang,Mauricio Sousa,Gregory Lee Newsome,Sageev Oore,Tovi Grossman摘要：合成器是强大的工具，让音乐家创造动态和原始的声音。现有的合成器商业界面通常需要音乐家与复杂的低级参数交互，或管理大型预制声音库。为了应对这些挑战，我们实现了SynthScribe --一个全栈系统，它使用多模态深度学习让用户在更高的层次上表达他们的意图。我们实现了一些功能，解决了一些困难，即1）搜索现有的声音，2）创建全新的声音，3）对给定的声音进行有意义的修改。这是实现了三个主要特点：一个多模态搜索引擎的一个大型图书馆的合成器的声音;一个用户为中心的遗传算法，通过它可以创建和选择全新的声音给用户的喜好;一个声音编辑支持功能，突出显示，并给出了关键控制参数的例子，相对于一个文本或音频为基础的查询。我们的用户研究结果表明，SynthScribe能够可靠地检索和修改声音，同时还能够创造全新的声音，扩大音乐家的创造视野。摘要：Synthesizers are powerful tools that allow musicians to create dynamic and original sounds. Existing commercial interfaces for synthesizers typically require musicians to interact with complex low-level parameters or to manage large libraries of premade sounds. To address these challenges, we implement SynthScribe -- a fullstack system that uses multimodal deep learning to let users express their intentions at a much higher level. We implement features which address a number of difficulties, namely 1) searching through existing sounds, 2) creating completely new sounds, 3) making meaningful modifications to a given sound. This is achieved with three main features: a multimodal search engine for a large library of synthesizer sounds; a user centered genetic algorithm by which completely new sounds can be created and selected given the users preferences; a sound editing support feature which highlights and gives examples for key control parameters with respect to a text or audio based query. The results of our user studies show SynthScribe is capable of reliably retrieving and modifying sounds while also affording the ability to create completely new sounds that expand a musicians creative horizon.
【5】 Binaural multichannel blind speaker separation with a causal low-latency  and low-complexity approach标题：基于因果低延迟低复杂度的双耳多声道盲人分离链接：https://arxiv.org/abs/2312.05173
作者：Nils L. Westhausen,Bernd T. Meyer备注：Accepted for publication at IEEE ICASSP 2024 OJSP track摘要：在本文中，我们介绍了一个因果的低延迟低复杂度的方法，双耳多通道盲说话人分离在嘈杂的混响条件。该模型被称为组通信双耳滤波器和求和网络（GCBFSnet），预测用于时频域中的滤波器和求和波束成形的复滤波器。我们采用组通信（GC），   也就是说，潜在模型变量被分成多个组，并利用共享序列模型进行处理，目的是降低仅包含一个卷积模块和一个递归模块的简单模型的复杂性。与没有GC的模型相比，使用GC我们能够将模型的大小减少高达83%，复杂性减少高达73%，同时大部分保持性能。即使对于最小的模型配置，GCBFSnet在大多数指标中也与低复杂度TasNet基线的性能相匹配，尽管基线的大小和所需操作的数量更大。摘要：In this paper, we introduce a causal low-latency low-complexity approach for binaural multichannel blind speaker separation in noisy reverberant conditions.  The model, referred to as Group Communication Binaural Filter and Sum Network (GCBFSnet) predicts complex filters for filter-and-sum beamforming in the time-frequency domain.  We apply Group Communication (GC),  i.e., latent model variables are split into groups and processed with a shared sequence model with the aim of reducing the complexity of a simple model only containing one convolutional and one recurrent module.  With GC we are able to reduce the size of the model by up to 83 % and the complexity up to 73 % compared to the model without GC, while mostly retaining performance.  Even for the smallest model configuration, GCBFSnet matches the performance of a low-complexity TasNet baseline in most metrics despite the larger size and higher number of required operations of the baseline.
eess.AS音频处理
【1】 Binaural multichannel blind speaker separation with a causal low-latency  and low-complexity approach标题：基于因果低延迟低复杂度的双耳多声道盲人分离链接：https://arxiv.org/abs/2312.05173
作者：Nils L. Westhausen,Bernd T. Meyer备注：Accepted for publication at IEEE ICASSP 2024 OJSP track摘要：在本文中，我们介绍了一个因果的低延迟低复杂度的方法，双耳多通道盲说话人分离在嘈杂的混响条件。该模型被称为组通信双耳滤波器和求和网络（GCBFSnet），预测用于时频域中的滤波器和求和波束成形的复滤波器。我们采用组通信（GC），   也就是说，潜在模型变量被分成多个组，并利用共享序列模型进行处理，目的是降低仅包含一个卷积模块和一个递归模块的简单模型的复杂性。与没有GC的模型相比，使用GC我们能够将模型的大小减少高达83%，复杂性减少高达73%，同时大部分保持性能。即使对于最小的模型配置，GCBFSnet在大多数指标中也与低复杂度TasNet基线的性能相匹配，尽管基线的大小和所需操作的数量更大。摘要：In this paper, we introduce a causal low-latency low-complexity approach for binaural multichannel blind speaker separation in noisy reverberant conditions.  The model, referred to as Group Communication Binaural Filter and Sum Network (GCBFSnet) predicts complex filters for filter-and-sum beamforming in the time-frequency domain.  We apply Group Communication (GC),  i.e., latent model variables are split into groups and processed with a shared sequence model with the aim of reducing the complexity of a simple model only containing one convolutional and one recurrent module.  With GC we are able to reduce the size of the model by up to 83 % and the complexity up to 73 % compared to the model without GC, while mostly retaining performance.  Even for the smallest model configuration, GCBFSnet matches the performance of a low-complexity TasNet baseline in most metrics despite the larger size and higher number of required operations of the baseline.
【2】 Seamless: Multilingual Expressive and Streaming Speech Translation标题：无缝：多语言表现力和流媒体语音翻译链接：https://arxiv.org/abs/2312.05187
作者：Seamless Communication,Loïc Barrault,Yu-An Chung,Mariano Coria Meglioli,David Dale,Ning Dong,Mark Duppenthaler,Paul-Ambroise Duquenne,Brian Ellis,Hady Elsahar,Justin Haaheim,John Hoffman,Min-Jae Hwang,Hirofumi Inaguma,Christopher Klaiber,Ilia Kulikov,Pengwei Li,Daniel Licht,Jean Maillard,Ruslan Mavlyutov,Alice Rakotoarison,Kaushik Ram Sadagopan,Abinesh Ramakrishnan,Tuan Tran,Guillaume Wenzek,Yilin Yang,Ethan Ye,Ivan Evtimov,Pierre Fernandez,Cynthia Gao,Prangthip Hansanti,Elahe Kalbassi,Amanda Kallet,Artyom Kozhevnikov,Gabriel Mejia Gonzalez,Robin San Roman,Christophe Touret,Corinne Wong,Carleigh Wood,Bokai Yu,Pierre Andrews,Can Balioglu,Peng-Jen Chen,Marta R. Costa-jussà,Maha Elbayad,Hongyu Gong,Francisco Guzmán,Kevin Heffernan,Somya Jain,Justine Kao,Ann Lee,Xutai Ma,Alex Mourachko,Benjamin Peloquin,Juan Pino,Sravya Popuri,Christophe Ropers,Safiyyah Saleem,Holger Schwenk,Anna Sun,Paden Tomasello,Changhan Wang,Jeff Wang,Skyler Wang,Mary Williamson摘要：与人与人之间的对话相比，今天的大规模自动语音翻译系统缺乏帮助机器介导的通信感觉无缝的关键功能。在这项工作中，我们介绍了一系列的模型，使端到端的表达和多语言的翻译在流的方式。首先，我们贡献了一个改进版本的大规模多语言和多模式的multicellessM 4 T模型multicellessM 4 T v2。这个更新的模型，结合了更新的UnitY 2框架，在更多的低资源语言数据上进行了训练。无障碍M4 T v2为我们接下来的两个模型提供了基础。无歧义表达使翻译能够保留声乐风格和韵律。与以前的表达性语音研究相比，我们的工作解决了韵律的某些未充分探索的方面，如语速和停顿，同时也保留了一个人的声音的风格。至于无障碍流，我们的模型利用高效单调多头注意力机制来生成低延迟的目标翻译，而无需等待完整的源话语。作为同类产品中的第一个，无障碍流能够为多种源语言和目标语言同时进行语音到语音/文本翻译。为了确保我们的模型可以安全和负责任地使用，我们实施了第一个已知的多模式机器翻译的红队工作，一个检测和减轻附加毒性的系统，一个系统的性别偏见评估，以及一个旨在抑制deepfakes影响的听不见的本地化水印机制。因此，我们把主要的组件从millessExpressive和millessStreaming一起形成无缝，第一个公开可用的系统，实时解锁表达跨语言通信。对这项工作的贡献是公开发布的，可在https://github.com/facebookresearch/seamless_communication上访问摘要：Large-scale automatic speech translation systems today lack key features that help machine-mediated communication feel seamless when compared to human-to-human dialogue. In this work, we introduce a family of models that enable end-to-end expressive and multilingual translations in a streaming fashion. First, we contribute an improved version of the massively multilingual and multimodal SeamlessM4T model-SeamlessM4T v2. This newer model, incorporating an updated UnitY2 framework, was trained on more low-resource language data. SeamlessM4T v2 provides the foundation on which our next two models are initiated. SeamlessExpressive enables translation that preserves vocal styles and prosody. Compared to previous efforts in expressive speech research, our work addresses certain underexplored aspects of prosody, such as speech rate and pauses, while also preserving the style of one's voice. As for SeamlessStreaming, our model leverages the Efficient Monotonic Multihead Attention mechanism to generate low-latency target translations without waiting for complete source utterances. As the first of its kind, SeamlessStreaming enables simultaneous speech-to-speech/text translation for multiple source and target languages. To ensure that our models can be used safely and responsibly, we implemented the first known red-teaming effort for multimodal machine translation, a system for the detection and mitigation of added toxicity, a systematic evaluation of gender bias, and an inaudible localized watermarking mechanism designed to dampen the impact of deepfakes. Consequently, we bring major components from SeamlessExpressive and SeamlessStreaming together to form Seamless, the first publicly available system that unlocks expressive cross-lingual communication in real-time. The contributions to this work are publicly released and accessible at https://github.com/facebookresearch/seamless_communication
【3】 neural concatenative singing voice conversion: rethinking  concatenation-based approach for one-shot singing voice conversion标题：神经连接的歌唱声转换：重新思考基于连接的单次歌唱声转换方法链接：https://arxiv.org/abs/2312.04919
作者：Binzhu Sha,Xu Li,Zhiyong Wu,Ying Shan,Helen Meng摘要：任意到任意的歌唱声音转换面临着由内容和扬声器音色之间的不充分分离引起的“音色泄漏”问题的重大挑战。为了解决这个问题，本研究介绍了一种新的神经级联歌唱语音转换（NeuCoSVC）框架。NeuCoSVC框架包括自监督学习（SSL）表示提取器、神经谐波信号发生器和波形合成器。具体地，SSL提取器将音频压缩成固定维度的SSL特征的序列。谐波信号产生器通过利用线性时变（LTV）滤波器产生原始谐波信号及经滤波谐波信号两者作为音调信息。最后，音频发生器基于SSL特征以及谐波信号和响度信息重构音频波形。在推断期间，系统通过用来自匹配池的源SSL特征的最接近的对应物替换源SSL特征来执行语音转换，匹配池包括从目标音频提取的SSL表示，而原始谐波信号和响度从源音频提取并且保持不变。由于在转换阶段中利用的SSL特征直接来自目标音频，因此所提出的框架具有很大的潜力来解决由先前的基于解纠缠的方法引起的“音色泄漏”问题。实验结果证实，所提出的系统提供了更好的性能比说话人嵌入方法（基于解纠缠）的上下文中的一杆SVC跨语言内，跨语言，跨域评估。摘要：Any-to-any singing voice conversion is confronted with a significant challenge of ``timbre leakage'' issue caused by inadequate disentanglement between the content and the speaker timbre. To address this issue, this study introduces a novel neural concatenative singing voice conversion (NeuCoSVC) framework. The NeuCoSVC framework comprises a self-supervised learning (SSL) representation extractor, a neural harmonic signal generator, and a waveform synthesizer. Specifically, the SSL extractor condenses the audio into a sequence of fixed-dimensional SSL features. The harmonic signal generator produces both raw and filtered harmonic signals as the pitch information by leveraging a linear time-varying (LTV) filter. Finally, the audio generator reconstructs the audio waveform based on the SSL features, as well as the harmonic signals and the loudness information. During inference, the system performs voice conversion by substituting source SSL features with their nearest counterparts from a matching pool, which comprises SSL representations extracted from the target audio, while the raw harmonic signals and the loudness are extracted from the source audio and are kept unchanged. Since the utilized SSL features in the conversion stage are directly from the target audio, the proposed framework has great potential to address the ``timbre leakage'' issue caused by previous disentanglement-based approaches. Experimental results confirm that the proposed system delivers much better performance than the speaker embedding approach (disentanglement-based) in the context of one-shot SVC across intra-language, cross-language, and cross-domain evaluations.
【4】 Sound Source Localization for a Source inside a Structure using  Ac-CycleGAN标题：基于AC-CycleGAN的结构内部声源定位链接：https://arxiv.org/abs/2312.04846
作者：Shunsuke Kita,Choong Sik Park,Yoshinobu Kajikawa摘要：我们提出了一种在不成对数据条件下使用Ac-CycleGAN对结构内部的源进行声源定位（SSL）的方法。该方法利用大量的模拟数据和少量的实际实验数据来定位真实环境中的结构内部的声源。Ac-CycleGAN生成器有助于将模拟数据转换为真实数据，反之亦然，使用来自两个域的未配对数据。Ac-CycleGAN模型的转换旨在区分由发生器生成的转换数据和真实数据，同时还预测声源的位置。代表在结构外三个点测量的加速度计（FSA）的频谱的矢量被用作输入数据，结构内的源区域被用作标签。输入数据向量垂直连接以形成图像。标签是通过将结构的内部划分为八个区域来定义的，每个区域都有一个热编码。因此，SSL问题被重新定义为图像分类问题，以随机估计声源的位置。我们表明，它是可能的，以估计声源的位置，使用Ac-CycleGAN跨域的未配对数据的估计。此外，我们分析了区分数据的判别因素。在80%的实际数据（12.5%的模拟数据）上训练后，该模型的准确率超过90%。尽管Ac-CycleGAN生成器执行的域转换过程中存在潜在的缺陷，但通过选择性地仅利用那些生成相对较小的转换误差的特征，该方法可以有效地区分传输数据和真实数据。摘要：We propose a method for sound source localization (SSL) for a source inside a structure using Ac-CycleGAN under unpaired data conditions. The proposed method utilizes a large amount of simulated data and a small amount of actual experimental data to locate a sound source inside a structure in a real environment. An Ac-CycleGAN generator contributes to the transformation of simulated data into real data, or vice versa, using unpaired data from both domains. The discriminator of an Ac-CycleGAN model is designed to differentiate between the transformed data generated by the generator and real data, while also predicting the location of the sound source. Vectors representing the frequency spectrum of the accelerometers (FSAs) measured at three points outside the structure are used as input data and the source areas inside the structure are used as labels. The input data vectors are concatenated vertically to form an image. Labels are defined by dividing the interior of the structure into eight areas with one-hot encoding for each area. Thus, the SSL problem is redefined as an image-classification problem to stochastically estimate the location of the sound source. We show that it is possible to estimate the sound source location using the Ac-CycleGAN discriminator for unpaired data across domains. Furthermore, we analyze the discriminative factors for distinguishing the data. The proposed model exhibited an accuracy exceeding 90\% when trained on 80\% of actual data (12.5\% of simulated data). Despite potential imperfections in the domain transformation process carried out by the Ac-CycleGAN generator, the discriminator can effectively distinguish between transferred and real data by selectively utilizing only those features that generate a relatively small transformation error.
【5】 SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and  Exploration标题：SynthScribe：用于合成器声音检索和探索的深度多通道工具链接：https://arxiv.org/abs/2312.04690
作者：Stephen Brade,Bryan Wang,Mauricio Sousa,Gregory Lee Newsome,Sageev Oore,Tovi Grossman摘要：合成器是强大的工具，让音乐家创造动态和原始的声音。现有的合成器商业界面通常需要音乐家与复杂的低级参数交互，或管理大型预制声音库。为了应对这些挑战，我们实现了SynthScribe --一个全栈系统，它使用多模态深度学习让用户在更高的层次上表达他们的意图。我们实现了一些功能，解决了一些困难，即1）搜索现有的声音，2）创建全新的声音，3）对给定的声音进行有意义的修改。这是实现了三个主要特点：一个多模态搜索引擎的一个大型图书馆的合成器的声音;一个用户为中心的遗传算法，通过它可以创建和选择全新的声音给用户的喜好;一个声音编辑支持功能，突出显示，并给出了关键控制参数的例子，相对于一个文本或音频为基础的查询。我们的用户研究结果表明，SynthScribe能够可靠地检索和修改声音，同时还能够创造全新的声音，扩大音乐家的创造视野。摘要：Synthesizers are powerful tools that allow musicians to create dynamic and original sounds. Existing commercial interfaces for synthesizers typically require musicians to interact with complex low-level parameters or to manage large libraries of premade sounds. To address these challenges, we implement SynthScribe -- a fullstack system that uses multimodal deep learning to let users express their intentions at a much higher level. We implement features which address a number of difficulties, namely 1) searching through existing sounds, 2) creating completely new sounds, 3) making meaningful modifications to a given sound. This is achieved with three main features: a multimodal search engine for a large library of synthesizer sounds; a user centered genetic algorithm by which completely new sounds can be created and selected given the users preferences; a sound editing support feature which highlights and gives examples for key control parameters with respect to a text or audio based query. The results of our user studies show SynthScribe is capable of reliably retrieving and modifying sounds while also affording the ability to create completely new sounds that expand a musicians creative horizon.
机器翻译由腾讯交互翻译提供，仅供参考