今日论文合集：cs.SD语音3篇，eess.AS音频处理3篇。
【1】 Novel-View Acoustic Synthesis
标题：声学合成的新视角
链接：https://arxiv.org/abs/2301.08730
作者：Changan Chen,Alexander Richard,Roman Shapovalov,Vamsi Krishna Ithapu,Natalia Neverova,Kristen Grauman,Andrea Vedaldi机构：University of Texas at Austin, Reality Labs Research at Meta, FAIR备注：Project page: this https URL摘要：我们介绍新视角声学合成（NVAS）任务：给定在源视点观察到的景象和声音，我们能否从一个看不见的目标视点合成该场景的声音？提出了一种神经绘制方法：视觉引导声学合成（ViGAS）网络，通过分析输入的视听提示来学习合成空间中任意点的声音。为了对这项任务进行基准测试，我们收集了两个第一次的大规模多视角视听数据集，一个是合成的，一个是真实的。我们证明了我们的模型成功地推理了空间线索，并在两个数据集上合成了忠实的音频。据我们所知，这项工作代表了解决新视角声学合成任务的第一个公式化、数据集和方法，其具有令人兴奋的潜在应用，范围从AR/VR到艺术和设计。通过这项工作，我们相信新视角合成的未来是从视频中多模态学习。摘要：We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the \emph{sound} of that scene from an unseen target viewpoint? We propose a neural rendering approach: Visually-Guided Acoustic Synthesis (ViGAS) network that learns to synthesize the sound of an arbitrary point in space by analyzing the input audio-visual cues. To benchmark this task, we collect two first-of-their-kind large-scale multi-view audio-visual datasets, one synthetic and one real. We show that our model successfully reasons about the spatial cues and synthesizes faithful audio on both datasets. To our knowledge, this work represents the very first formulation, dataset, and approach to solve the novel-view acoustic synthesis task, which has exciting potential applications ranging from AR/VR to art and design. Unlocked by this work, we believe that the future of novel-view synthesis is in multi-modal learning from videos.
【2】 Adjoint-Based Identification of Sound Sources for Sound Reinforcement and Source Localization标题：基于伴随的扩声声源识别与声源定位
链接：https://arxiv.org/abs/2301.08620
作者：Mathias Lemke,Lewin Stein机构：Institut f¨ur Str¨omungsmechanik und Technische Akustik, Technische Universit¨at Berlin, Germany摘要：声源识别是声学中的一个常见问题。寻求不同的参数，其中包括信号和源的位置。提出了一种基于伴随函数的声源识别方法，该方法采用了计算航空声学技术。本文介绍了两种不同的应用作为概念验证：声音增强设置的优化和（移动）声源的定位。摘要：The identification of sound sources is a common problem in acoustics. Different parameters are sought, among these are signal and position of the sources. We present an adjoint-based approach for sound source identification, which employs computational aeroacoustic techniques. Two different applications are presented as a proof-of-concept: optimization of a sound reinforcement setup and the localization of (moving) sound sources.
【3】 Latent Autoregressive Source Separation
标题：潜在自回归信源分离
链接：https://arxiv.org/abs/2301.08562
作者：Emilian Postolache,Giorgio Mariani,Michele Mancusi,Andrea Santilli,Luca Cosmo,Emanuele Rodolà机构：Sapienza University of Rome, Italy, Ca’ Foscari University of Venice, Italy摘要：自回归模型在发电质量和下游任务性能方面在广泛的领域取得了令人印象深刻的结果。在连续域中，这种成功背后的关键因素是量化的潜在空间的使用（例如，通过VQ-VAE自动编码器获得），其允许维度减少和更快的推断时间。然而，使用现有的预训练模型来执行新的非平凡任务是困难的，因为它需要额外的微调或广泛的训练来引出提示。本文介绍LASS作为执行矢量量化潜在自回归源分离（即，将输入信号去混频成其组成源），而不需要对现有模型进行额外的基于梯度的优化或修改。我们的分离方法依赖于贝叶斯公式，其中自回归模型是先验，并且通过对加数令牌的潜在和执行频率计数来构造离散（非参数）似然函数。我们用几种采样策略在图像和音频上测试我们的方法（例如，祖先、波束搜索）在分离质量方面显示出与现有方法有竞争力的结果，同时在推断时间和对更高维数据的可缩放性方面提供显著的加速。摘要：Autoregressive models have achieved impressive results over a wide range of domains in terms of generation quality and downstream task performance. In the continuous domain, a key factor behind this success is the usage of quantized latent spaces (e.g., obtained via VQ-VAE autoencoders), which allow for dimensionality reduction and faster inference times. However, using existing pre-trained models to perform new non-trivial tasks is difficult since it requires additional fine-tuning or extensive training to elicit prompting. This paper introduces LASS as a way to perform vector-quantized Latent Autoregressive Source Separation (i.e., de-mixing an input signal into its constituent sources) without requiring additional gradient-based optimization or modifications of existing models. Our separation method relies on the Bayesian formulation in which the autoregressive models are the priors, and a discrete (non-parametric) likelihood function is constructed by performing frequency counts over latent sums of addend tokens. We test our method on images and audio with several sampling strategies (e.g., ancestral, beam search) showing competitive results with existing approaches in terms of separation quality while offering at the same time significant speedups in terms of inference time and scalability to higher dimensional data.
【1】 Novel-View Acoustic Synthesis
标题：声学合成的新视角
链接：https://arxiv.org/abs/2301.08730
*与cs.SD语音【1】为同一篇
作者：Changan Chen,Alexander Richard,Roman Shapovalov,Vamsi Krishna Ithapu,Natalia Neverova,Kristen Grauman,Andrea Vedaldi机构：University of Texas at Austin, Reality Labs Research at Meta, FAIR备注：Project page: this https URL摘要：我们介绍新视角声学合成（NVAS）任务：给定在源视点观察到的景象和声音，我们能否从一个看不见的目标视点合成该场景的声音？提出了一种神经绘制方法：视觉引导声学合成（ViGAS）网络，通过分析输入的视听提示来学习合成空间中任意点的声音。为了对这项任务进行基准测试，我们收集了两个第一次的大规模多视角视听数据集，一个是合成的，一个是真实的。我们证明了我们的模型成功地推理了空间线索，并在两个数据集上合成了忠实的音频。据我们所知，这项工作代表了解决新视角声学合成任务的第一个公式化、数据集和方法，其具有令人兴奋的潜在应用，范围从AR/VR到艺术和设计。通过这项工作，我们相信新视角合成的未来是从视频中多模态学习。摘要：We introduce the novel-view acoustic synthesis (NVAS) task: given the sight and sound observed at a source viewpoint, can we synthesize the \emph{sound} of that scene from an unseen target viewpoint? We propose a neural rendering approach: Visually-Guided Acoustic Synthesis (ViGAS) network that learns to synthesize the sound of an arbitrary point in space by analyzing the input audio-visual cues. To benchmark this task, we collect two first-of-their-kind large-scale multi-view audio-visual datasets, one synthetic and one real. We show that our model successfully reasons about the spatial cues and synthesizes faithful audio on both datasets. To our knowledge, this work represents the very first formulation, dataset, and approach to solve the novel-view acoustic synthesis task, which has exciting potential applications ranging from AR/VR to art and design. Unlocked by this work, we believe that the future of novel-view synthesis is in multi-modal learning from videos.
【2】 Adjoint-Based Identification of Sound Sources for Sound Reinforcement and Source Localization
标题：基于伴随的扩声声源识别与声源定位
链接：https://arxiv.org/abs/2301.08620
*与cs.SD语音【2】为同一篇
作者：Mathias Lemke,Lewin Stein机构：Institut f¨ur Str¨omungsmechanik und Technische Akustik, Technische Universit¨at Berlin, Germany摘要：声源识别是声学中的一个常见问题。寻求不同的参数，其中包括信号和源的位置。提出了一种基于伴随函数的声源识别方法，该方法采用了计算航空声学技术。本文介绍了两种不同的应用作为概念验证：声音增强设置的优化和（移动）声源的定位。摘要：The identification of sound sources is a common problem in acoustics. Different parameters are sought, among these are signal and position of the sources. We present an adjoint-based approach for sound source identification, which employs computational aeroacoustic techniques. Two different applications are presented as a proof-of-concept: optimization of a sound reinforcement setup and the localization of (moving) sound sources.
【3】 Latent Autoregressive Source Separation
标题：潜在自回归信源分离
链接：https://arxiv.org/abs/2301.08562
*与cs.SD语音【3】为同一篇
作者：Emilian Postolache,Giorgio Mariani,Michele Mancusi,Andrea Santilli,Luca Cosmo,Emanuele Rodolà机构：Sapienza University of Rome, Italy, Ca’ Foscari University of Venice, Italy摘要：自回归模型在发电质量和下游任务性能方面在广泛的领域取得了令人印象深刻的结果。在连续域中，这种成功背后的关键因素是量化的潜在空间的使用（例如，通过VQ-VAE自动编码器获得），其允许维度减少和更快的推断时间。然而，使用现有的预训练模型来执行新的非平凡任务是困难的，因为它需要额外的微调或广泛的训练来引出提示。本文介绍LASS作为执行矢量量化潜在自回归源分离（即，将输入信号去混频成其组成源），而不需要对现有模型进行额外的基于梯度的优化或修改。我们的分离方法依赖于贝叶斯公式，其中自回归模型是先验，并且通过对加数令牌的潜在和执行频率计数来构造离散（非参数）似然函数。我们用几种采样策略在图像和音频上测试我们的方法（例如，祖先、波束搜索）在分离质量方面显示出与现有方法有竞争力的结果，同时在推断时间和对更高维数据的可缩放性方面提供显著的加速。摘要：Autoregressive models have achieved impressive results over a wide range of domains in terms of generation quality and downstream task performance. In the continuous domain, a key factor behind this success is the usage of quantized latent spaces (e.g., obtained via VQ-VAE autoencoders), which allow for dimensionality reduction and faster inference times. However, using existing pre-trained models to perform new non-trivial tasks is difficult since it requires additional fine-tuning or extensive training to elicit prompting. This paper introduces LASS as a way to perform vector-quantized Latent Autoregressive Source Separation (i.e., de-mixing an input signal into its constituent sources) without requiring additional gradient-based optimization or modifications of existing models. Our separation method relies on the Bayesian formulation in which the autoregressive models are the priors, and a discrete (non-parametric) likelihood function is constructed by performing frequency counts over latent sums of addend tokens. We test our method on images and audio with several sampling strategies (e.g., ancestral, beam search) showing competitive results with existing approaches in terms of separation quality while offering at the same time significant speedups in terms of inference time and scalability to higher dimensional data.
机器翻译，仅供参考
每日语音速递