今日论文合集：cs.SD语音4篇，eess.AS音频处理7篇。【1】Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
标题：传播：利用传播的力量和大型语言模型生成文本到音频链接：https://arxiv.org/abs/2401.01044
作者：Jinlong Xue,Yayue Deng,Yingming Gao,Ya Li备注：Demo and implementation at this https URL摘要：扩散模型和大型语言模型（LLM）的最新进展极大地推动了AIGC领域的发展。文本到音频（TTA），一个新兴的AIGC应用程序，旨在从自然语言提示生成音频，正在吸引越来越多的关注。然而，现有的TTA研究往往与生成质量和文本音频对齐，特别是复杂的文本输入斗争。从最先进的文本到图像（T2 I）的扩散模型的灵感，我们介绍了Aufflement，TTA系统T2 I模型框架适应TTA任务，有效地利用其固有的生成优势和精确的跨模态对齐。我们的客观和主观的评价表明，Auffelt超越以前的TTA方法，使用有限的数据和计算资源。此外，T2 I中的先前研究认识到编码器选择对跨模态对齐的重大影响，如细粒度细节和对象绑定，而在先前的TTA工作中缺乏类似的评估。通过全面的消融研究和创新的交叉注意力图可视化，我们提供了TTA中文本-音频对齐的深入评估。我们的研究结果揭示了Auffelt在生成与文本描述准确匹配的音频方面的卓越能力，这在几个相关任务中得到了进一步的证明，例如音频风格转换，修复和其他操作。我们的实现和演示可以在https://auffusion.github.io上找到。摘要：Recent advancements in diffusion models and large language models (LLMs) have significantly propelled the field of AIGC. Text-to-Audio (TTA), a burgeoning AIGC application designed to generate audio from natural language prompts, is attracting increasing attention. However, existing TTA studies often struggle with generation quality and text-audio alignment, especially for complex textual inputs. Drawing inspiration from state-of-the-art Text-to-Image (T2I) diffusion models, we introduce Auffusion, a TTA system adapting T2I model frameworks to TTA task, by effectively leveraging their inherent generative strengths and precise cross-modal alignment. Our objective and subjective evaluations demonstrate that Auffusion surpasses previous TTA approaches using limited data and computational resource. Furthermore, previous studies in T2I recognizes the significant impact of encoder choice on cross-modal alignment, like fine-grained details and object bindings, while similar evaluation is lacking in prior TTA works. Through comprehensive ablation studies and innovative cross-attention map visualizations, we provide insightful assessments of text-audio alignment in TTA. Our findings reveal Auffusion's superior capability in generating audios that accurately match textual descriptions, which further demonstrated in several related tasks, such as audio style transfer, inpainting and other manipulations. Our implementation and demos are available at https://auffusion.github.io.
【2】 HAAQI-Net: A non-intrusive neural music quality assessment model for hearing aids标题：HAAQI-Net：一种非侵入式的助听器神经音乐质量评价模型链接：https://arxiv.org/abs/2401.01145作者：Dyah A. M. G. Wisnu,Epri Pratiwi,Stefano Rini,Ryandhimas E. Zezario,Hsin-Min Wang,Yu Tsao摘要：本文介绍了HAAQI-Net，这是一种非侵入式深度学习模型，用于为助听器用户量身定制的音乐质量评估。与助听器音频质量指数（HAAQI）等传统方法相比，HAAQI-Net利用了具有注意力的双向长短期记忆（BLSTM）。它将评估的音乐样本和听力损失模式作为输入，生成预测的HAAQI分数。该模型采用来自音频Transformers（BEAT）的预训练的双向编码器表示进行声学特征提取。HAAQI-Net将预测分数与地面真实值进行比较，纵向一致性相关系数（LCC）为0.9257，斯皮尔曼等级相关系数（SRCC）为0.9394，均方误差（MSE）为0.0080。值得注意的是，这种高性能伴随着推理时间的大幅减少：从62.52秒（HAAQI）到2.71秒（HAAQI-Net），作为助听器用户的有效音乐质量评估模型。摘要：This paper introduces HAAQI-Net, a non-intrusive deep learning model for music quality assessment tailored to hearing aid users. In contrast to traditional methods like the Hearing Aid Audio Quality Index (HAAQI), HAAQI-Net utilizes a Bidirectional Long Short-Term Memory (BLSTM) with attention. It takes an assessed music sample and a hearing loss pattern as input, generating a predicted HAAQI score. The model employs the pre-trained Bidirectional Encoder representation from Audio Transformers (BEATs) for acoustic feature extraction. Comparing predicted scores with ground truth, HAAQI-Net achieves a Longitudinal Concordance Correlation (LCC) of 0.9257, Spearman's Rank Correlation Coefficient (SRCC) of 0.9394, and Mean Squared Error (MSE) of 0.0080. Notably, this high performance comes with a substantial reduction in inference time: from 62.52 seconds (by HAAQI) to 2.71 seconds (by HAAQI-Net), serving as an efficient music quality assessment model for hearing aid users.【3】 The role of direct sound spherical harmonics representation in externalization using binaural reproduction标题：直达声球谐表示在双耳再现外化中的作用链接：https://arxiv.org/abs/2401.00936作者：Eran Miller,Boaz Rafaely备注：None摘要：直达声中的信息对人类感知空间声源的重要性是一个正在进行的研究课题。直接声音和扩散或混响声音之间的分类形成了空间音频领域中众多研究的基础。具体地，参数空间音频表示方法使用该分类并采用信号处理以便增强再现时的音频质量。然而，目前的文献并没有提供信息的影响，理想的直接声音表示的外部化，在环境中的高保真度立体声。本文的目的是评估的重要性，直接声音的空间信息在外部的声场时，使用双耳复制。这是在球面谐波（SH）域中完成的，其中模拟另外的高保真度立体声响复制信号内的理想直接声音表示，并且在正式的收听测试中评估其感知的外部化。该研究得出的结论是，一阶高保真度立体声响复制信号的外部化可以通过增强直达声分量而被显著地改善，直到类似于三阶高保真度立体声响复制信号的水平。摘要：The importance of the information in the direct sound to human perception of spatial sound sources is an ongoing research topic. The classification between direct sound and diffuse or reverberant sound forms the basis of numerous studies in the field of spatial audio. In particular, parametric spatial audio representation methods use this classification and employ signal processing in order to enhance the audio quality at reproduction. However, current literature does not provide information concerning the impact of ideal direct sound representation on externalization, in the context of Ambisonics. This paper aims to assess the importance of the spatial information in the direct sound in the externalization of a sound field when using binaural reproduction. This is done in the spherical harmonics (SH) domain, where an ideal direct sound representation within an otherwise Ambisonics signal is simulated, and its perceived externalization is evaluated in a formal listening test. This investigation leads to the conclusion that externalization of a first order Ambisonics signal may be significantly improved by enhancing the direct sound component, up to a level similar to a third order Ambisonics signal.【4】 Detecting the presence of sperm whales echolocation clicks in noisy environments标题：检测噪声环境中抹香鲸回声定位滴答声的存在链接：https://arxiv.org/abs/2401.00900作者：Guy Gubnitsky,Roee Diamant备注：10 pages and 10 figures摘要：抹香鲸（Physeter macrocephalus）在水下航行时会发出一系列类似咔哒声的冲动声音，称为回声定位咔哒声。这些喀哒声的特征在于作为独特模式的多脉冲结构（MPS）。在这项工作中，我们使用的MPS的稳定性作为一个检测指标识别和分类的存在下，在嘈杂的环境中点击。为了区分噪声瞬变和处理多个抹香鲸的同时排放，我们的方法将MPS测量的时间序列聚类，同时去除不满足点击间隔，持续时间和频谱限制的潜在点击。因此，我们的方法可以处理高噪声瞬态和低信噪比。使用三个数据集检查我们的检测方法的性能：七个月的记录从地中海包含手动验证的环境噪声;几天的手动标记的数据收集从多米尼加岛包含约40,000点击从多个抹香鲸;和数据集从巴哈马包含1,203标记点击从一个抹香鲸。与两个基准检测器的结果相比，观察到更好的精确度和召回率之间的权衡以及显着降低误检率，特别是在噪声环境中。为了确保可重复性，我们提供了标记点击的数据库以及我们的实现代码。摘要：Sperm whales (Physeter macrocephalus) navigate underwater with a series of impulsive, click-like sounds known as echolocation clicks. These clicks are characterized by a multipulse structure (MPS) that serves as a distinctive pattern. In this work, we use the stability of the MPS as a detection metric for recognizing and classifying the presence of clicks in noisy environments. To distinguish between noise transients and to handle simultaneous emissions from multiple sperm whales, our approach clusters a time series of MPS measures while removing potential clicks that do not fulfil the limits of inter-click interval, duration and spectrum. As a result, our approach can handle high noise transients and low signal-to-noise ratio. The performance of our detection approach is examined using three datasets: seven months of recordings from the Mediterranean Sea containing manually verified ambient noise; several days of manually labelled data collected from the Dominica Island containing approximately 40,000 clicks from multiple sperm whales; and a dataset from the Bahamas containing 1,203 labelled clicks from a single sperm whale. Comparing with the results of two benchmark detectors, a better trade-off between precision and recall is observed as well as a significant reduction in false detection rates, especially in noisy environments. To ensure reproducibility, we provide our database of labelled clicks along with our implementation code.【1】 On the Parameter Estimation of Sinusoidal Models for Speech and Audio Signals标题：语音和音频信号正弦模型的参数估计链接：https://arxiv.org/abs/2401.01255作者：George P. Kafentzis摘要：在本文中，我们研究了三个著名的正弦模型的语音和音频的参数估计性能。第一种是基于快速傅立叶变换（FFT）的标准正弦模型（SM）。第二种是最近十年提出的指数阻尼正弦模型（EDSM），并利用子空间方法进行参数估计，最后是最近提出的用于AM-FM分解的扩展自适应准谐波模型（eaQHM），并在一组自适应于信号局部特性的基函数上使用最小二乘估计信号参数。每个模型的参数估计进行了简要描述，其性能进行了比较，在信号重建精度与窗口大小的各种合成信号和真实信号的正弦曲线的数量。后者包括高度非平稳的信号，如歌声和吉他独奏。通过合成信号分析了各模型的优缺点，并讨论了其在实际信号中的应用。总之，eaQHM优于EDS在中等到大的窗口大小的分析，而EDSM产生更高的重建值较小的分析窗口大小。因此，未来的研究方向似乎是合并的eaQHM和参数估计的EDSM的鲁棒性的自适应性在一个新的范例，用于高质量的分析和再合成的一般音频信号。摘要：In this paper, we examine the parameter estimation performance of three well-known sinusoidal models for speech and audio. The first one is the standard Sinusoidal Model (SM), which is based on the Fast Fourier Transform (FFT). The second is the Exponentially Damped Sinusoidal Model (EDSM) which has been proposed in the last decade, and utilizes a subspace method for parameter estimation, and finally the extended adaptive Quasi-Harmonic Model (eaQHM), which has been recently proposed for AM-FM decomposition, and estimates the signal parameters using Least Squares on a set of basis function that are adaptive to the local characteristics of the signal. The parameter estimation of each model is briefly described and its performance is compared to the others in terms of signal reconstruction accuracy versus window size on a variety of synthetic signals and versus the number of sinusoids on real signals. The latter include highly non stationary signals, such as singing voices and guitar solos. The advantages and disadvantages of each model are presented via synthetic signals and then the application on real signals is discussed. Conclusively, eaQHM outperforms EDS in medium-to-large window size analysis, whereas EDSM yields higher reconstruction values for smaller analysis window sizes. Thus, a future research direction appears to be the merge of adaptivity of the eaQHM and parameter estimation robustness of the EDSM in a new paradigm for high-quality analysis and resynthesis of general audio signals.【2】 Room impulse response reconstruction with physics-informed deep learning标题：基于物理信息深度学习的房间脉冲响应重建链接：https://arxiv.org/abs/2401.01206作者：Xenofon Karakonstantis,Diego Caviedes-Nozal,Antoine Richard,Efren Fernandez-Grande备注：Submitted to Journal of Acoustical Society of America (JASA)摘要：提出了一种使用物理信息神经网络来估计和重建房间内的声场的方法。通过将一组有限的实验室脉冲响应作为训练数据，这种方法将神经网络处理能力与声音传播的基本物理学相结合，如波动方程所示。除了声压之外，该网络还能够估计粒子速度和强度，这表明它能够代表声能的流动，并且只需几次测量就可以完全模拟声场。此外，这个网络作为一种工具，用于改善声学模拟的潜力进行调查。这是由于它在提供无网格声场映射方面具有最小的推理时间。此外，进行了一项研究，其中包括对目前的声场重建方法的比较分析。具体而言，所提出的方法进行评估，对数据驱动的技术和基本波为基础的回归方法。结果表明，物理信息神经网络在重建房间脉冲响应的早期部分时表现突出，同时允许在时域中进行完整的声场表征。摘要：A method is presented for estimating and reconstructing the sound field within a room using physics-informed neural networks. By incorporating a limited set of experimental room impulse responses as training data, this approach combines neural network processing capabilities with the underlying physics of sound propagation, as articulated by the wave equation. The network's ability to estimate particle velocity and intensity, in addition to sound pressure, demonstrates its capacity to represent the flow of acoustic energy and completely characterise the sound field with only a few measurements. Additionally, an investigation into the potential of this network as a tool for improving acoustic simulations is conducted. This is due to its profficiency in offering grid-free sound field mappings with minimal inference time. Furthermore, a study is carried out which encompasses comparative analyses against current approaches for sound field reconstruction. Specifically, the proposed approach is evaluated against both data-driven techniques and elementary wave-based regression methods. The results demonstrate that the physics-informed neural network stands out when reconstructing the early part of the room impulse response, while simultaneously allowing for complete sound field characterisation in the time domain.【3】 HAAQI-Net: A non-intrusive neural music quality assessment model for hearing aids标题：HAAQI-Net：一种非侵入式的助听器神经音乐质量评价模型链接：https://arxiv.org/abs/2401.01145作者：Dyah A. M. G. Wisnu,Epri Pratiwi,Stefano Rini,Ryandhimas E. Zezario,Hsin-Min Wang,Yu Tsao摘要：本文介绍了HAAQI-Net，这是一种非侵入式深度学习模型，用于为助听器用户量身定制的音乐质量评估。与助听器音频质量指数（HAAQI）等传统方法相比，HAAQI-Net利用了具有注意力的双向长短期记忆（BLSTM）。它将评估的音乐样本和听力损失模式作为输入，生成预测的HAAQI分数。该模型采用来自音频Transformers（BEAT）的预训练的双向编码器表示进行声学特征提取。HAAQI-Net将预测分数与地面真实值进行比较，纵向一致性相关系数（LCC）为0.9257，斯皮尔曼等级相关系数（SRCC）为0.9394，均方误差（MSE）为0.0080。值得注意的是，这种高性能伴随着推理时间的大幅减少：从62.52秒（HAAQI）到2.71秒（HAAQI-Net），作为助听器用户的有效音乐质量评估模型。摘要：This paper introduces HAAQI-Net, a non-intrusive deep learning model for music quality assessment tailored to hearing aid users. In contrast to traditional methods like the Hearing Aid Audio Quality Index (HAAQI), HAAQI-Net utilizes a Bidirectional Long Short-Term Memory (BLSTM) with attention. It takes an assessed music sample and a hearing loss pattern as input, generating a predicted HAAQI score. The model employs the pre-trained Bidirectional Encoder representation from Audio Transformers (BEATs) for acoustic feature extraction. Comparing predicted scores with ground truth, HAAQI-Net achieves a Longitudinal Concordance Correlation (LCC) of 0.9257, Spearman's Rank Correlation Coefficient (SRCC) of 0.9394, and Mean Squared Error (MSE) of 0.0080. Notably, this high performance comes with a substantial reduction in inference time: from 62.52 seconds (by HAAQI) to 2.71 seconds (by HAAQI-Net), serving as an efficient music quality assessment model for hearing aid users.【4】 Efficient Parallel Audio Generation using Group Masked Language Modeling标题：基于组屏蔽语言建模的高效并行音频生成链接：https://arxiv.org/abs/2401.01099作者：Myeonghun Jeong,Minchan Kim,Joun Yeop Lee,Nam Soo Kim备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：我们提出了一个快速和高质量的编解码器语言模型的并行音频生成。虽然SoundStorm是一种最先进的并行音频生成模型，与自回归模型相比加快了推理速度，但由于迭代采样，它仍然存在推理速度慢的问题。为了解决这个问题，我们提出了组掩码语言建模~（G-MLM）和组迭代并行解码~（G-IPD），以有效的并行音频生成。训练和采样方案都使模型能够通过有效地对逐组条件依赖性进行建模来合成具有少量迭代的高质量音频。此外，我们的模型采用了基于交叉注意的架构来捕捉提示语音的说话人风格，并提高了计算效率。实验结果表明，我们提出的模型优于基线在基于音频生成。摘要：We present a fast and high-quality codec language model for parallel audio generation. While SoundStorm, a state-of-the-art parallel audio generation model, accelerates inference speed compared to autoregressive models, it still suffers from slow inference due to iterative sampling. To resolve this problem, we propose Group-Masked Language Modeling~(G-MLM) and Group Iterative Parallel Decoding~(G-IPD) for efficient parallel audio generation. Both the training and sampling schemes enable the model to synthesize high-quality audio with a small number of iterations by effectively modeling the group-wise conditional dependencies. In addition, our model employs a cross-attention-based architecture to capture the speaker style of the prompt voice and improves computational efficiency. Experimental results demonstrate that our proposed model outperforms the baselines in prompt-based audio generation.【5】 The role of direct sound spherical harmonics representation in externalization using binaural reproduction标题：直达声球谐表示在双耳再现外化中的作用链接：https://arxiv.org/abs/2401.00936作者：Eran Miller,Boaz Rafaely备注：None摘要：直达声中的信息对人类感知空间声源的重要性是一个正在进行的研究课题。直接声音和扩散或混响声音之间的分类形成了空间音频领域中众多研究的基础。具体地，参数空间音频表示方法使用该分类并采用信号处理以便增强再现时的音频质量。然而，目前的文献并没有提供信息的影响，理想的直接声音表示的外部化，在环境中的高保真度立体声。本文的目的是评估的重要性，直接声音的空间信息在外部的声场时，使用双耳复制。这是在球面谐波（SH）域中完成的，其中模拟另外的高保真度立体声响复制信号内的理想直接声音表示，并且在正式的收听测试中评估其感知的外部化。该研究得出的结论是，一阶高保真度立体声响复制信号的外部化可以通过增强直达声分量而被显著地改善，直到类似于三阶高保真度立体声响复制信号的水平。摘要：The importance of the information in the direct sound to human perception of spatial sound sources is an ongoing research topic. The classification between direct sound and diffuse or reverberant sound forms the basis of numerous studies in the field of spatial audio. In particular, parametric spatial audio representation methods use this classification and employ signal processing in order to enhance the audio quality at reproduction. However, current literature does not provide information concerning the impact of ideal direct sound representation on externalization, in the context of Ambisonics. This paper aims to assess the importance of the spatial information in the direct sound in the externalization of a sound field when using binaural reproduction. This is done in the spherical harmonics (SH) domain, where an ideal direct sound representation within an otherwise Ambisonics signal is simulated, and its perceived externalization is evaluated in a formal listening test. This investigation leads to the conclusion that externalization of a first order Ambisonics signal may be significantly improved by enhancing the direct sound component, up to a level similar to a third order Ambisonics signal.【6】 Detecting the presence of sperm whales echolocation clicks in noisy environments标题：检测噪声环境中抹香鲸回声定位滴答声的存在链接：https://arxiv.org/abs/2401.00900作者：Guy Gubnitsky,Roee Diamant备注：10 pages and 10 figures摘要：抹香鲸（Physeter macrocephalus）在水下航行时会发出一系列类似咔哒声的冲动声音，称为回声定位咔哒声。这些喀哒声的特征在于作为独特模式的多脉冲结构（MPS）。在这项工作中，我们使用的MPS的稳定性作为一个检测指标识别和分类的存在下，在嘈杂的环境中点击。为了区分噪声瞬变和处理多个抹香鲸的同时排放，我们的方法将MPS测量的时间序列聚类，同时去除不满足点击间隔，持续时间和频谱限制的潜在点击。因此，我们的方法可以处理高噪声瞬态和低信噪比。使用三个数据集检查我们的检测方法的性能：七个月的记录从地中海包含手动验证的环境噪声;几天的手动标记的数据收集从多米尼加岛包含约40,000点击从多个抹香鲸;和数据集从巴哈马包含1,203标记点击从一个抹香鲸。与两个基准检测器的结果相比，观察到更好的精确度和召回率之间的权衡以及显着降低误检率，特别是在噪声环境中。为了确保可重复性，我们提供了标记点击的数据库以及我们的实现代码。摘要：Sperm whales (Physeter macrocephalus) navigate underwater with a series of impulsive, click-like sounds known as echolocation clicks. These clicks are characterized by a multipulse structure (MPS) that serves as a distinctive pattern. In this work, we use the stability of the MPS as a detection metric for recognizing and classifying the presence of clicks in noisy environments. To distinguish between noise transients and to handle simultaneous emissions from multiple sperm whales, our approach clusters a time series of MPS measures while removing potential clicks that do not fulfil the limits of inter-click interval, duration and spectrum. As a result, our approach can handle high noise transients and low signal-to-noise ratio. The performance of our detection approach is examined using three datasets: seven months of recordings from the Mediterranean Sea containing manually verified ambient noise; several days of manually labelled data collected from the Dominica Island containing approximately 40,000 clicks from multiple sperm whales; and a dataset from the Bahamas containing 1,203 labelled clicks from a single sperm whale. Comparing with the results of two benchmark detectors, a better trade-off between precision and recall is observed as well as a significant reduction in false detection rates, especially in noisy environments. To ensure reproducibility, we provide our database of labelled clicks along with our implementation code.【7】 Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation标题：传播：利用传播的力量和大型语言模型生成文本到音频链接：https://arxiv.org/abs/2401.01044作者：Jinlong Xue,Yayue Deng,Yingming Gao,Ya Li备注：Demo and implementation at this https URL摘要：扩散模型和大型语言模型（LLM）的最新进展极大地推动了AIGC领域的发展。文本到音频（TTA），一个新兴的AIGC应用程序，旨在从自然语言提示生成音频，正在吸引越来越多的关注。然而，现有的TTA研究往往与生成质量和文本音频对齐，特别是复杂的文本输入斗争。从最先进的文本到图像（T2 I）的扩散模型的灵感，我们介绍了Aufflement，TTA系统T2 I模型框架适应TTA任务，有效地利用其固有的生成优势和精确的跨模态对齐。我们的客观和主观的评价表明，Auffelt超越以前的TTA方法，使用有限的数据和计算资源。此外，T2 I中的先前研究认识到编码器选择对跨模态对齐的重大影响，如细粒度细节和对象绑定，而在先前的TTA工作中缺乏类似的评估。通过全面的消融研究和创新的交叉注意力图可视化，我们提供了TTA中文本-音频对齐的深入评估。我们的研究结果揭示了Auffelt在生成与文本描述准确匹配的音频方面的卓越能力，这在几个相关任务中得到了进一步的证明，例如音频风格转换，修复和其他操作。我们的实现和演示可以在https://auffusion.github.io上找到。摘要：Recent advancements in diffusion models and large language models (LLMs) have significantly propelled the field of AIGC. Text-to-Audio (TTA), a burgeoning AIGC application designed to generate audio from natural language prompts, is attracting increasing attention. However, existing TTA studies often struggle with generation quality and text-audio alignment, especially for complex textual inputs. Drawing inspiration from state-of-the-art Text-to-Image (T2I) diffusion models, we introduce Auffusion, a TTA system adapting T2I model frameworks to TTA task, by effectively leveraging their inherent generative strengths and precise cross-modal alignment. Our objective and subjective evaluations demonstrate that Auffusion surpasses previous TTA approaches using limited data and computational resource. Furthermore, previous studies in T2I recognizes the significant impact of encoder choice on cross-modal alignment, like fine-grained details and object bindings, while similar evaluation is lacking in prior TTA works. Through comprehensive ablation studies and innovative cross-attention map visualizations, we provide insightful assessments of text-audio alignment in TTA. Our findings reveal Auffusion's superior capability in generating audios that accurately match textual descriptions, which further demonstrated in several related tasks, such as audio style transfer, inpainting and other manipulations. Our implementation and demos are available at https://auffusion.github.io. 
每日语音速递