今日论文合集：cs.SD语音9篇，eess.AS音频处理11篇。
cs.SD语音
【1】 WavJourney: Compositional Audio Creation with Large Language Models标题：WavJourney：使用大型语言模型进行作曲音频创作链接：https://arxiv.org/abs/2307.14335
作者：Xubo Liu,Zhongkai Zhu,Haohe Liu,Yi Yuan,Meng Cui,Qiushi Huang,Jinhua Liang,Yin Cao,Qiuqiang Kong,Mark D. Plumbley,Wenwu Wang备注：Project Page: this https URL摘要：大型语言模型（LLM）在集成各种专家模型以解决复杂的语言和视觉任务方面表现出了巨大的潜力。尽管它们在推进人工智能生成内容（AIGC）领域中具有重要意义，但它们在智能音频内容创作中的潜力仍未被开发。在这项工作中，我们解决了创建音频内容的问题，故事情节包括语音，音乐和声音效果，由文本说明指导。我们提出了WavJourney，一个系统，利用LLM连接各种音频模型的音频内容生成。给定听觉场景的文本描述，WavJourney首先提示LLM生成专用于音频故事讲述的结构化脚本。音频脚本包含基于其时空关系组织的各种音频元素。作为音频的概念表示，音频脚本为人类参与提供了交互式和可解释的基本原理。之后，音频脚本被输入脚本编译器，将其转换为计算机程序。程序的每一行调用任务特定的音频生成模型或计算操作函数（例如，连接、混合）。然后执行计算机程序以获得用于音频生成的可解释的解决方案。我们展示了WavJourney在各种现实世界场景中的实用性，包括科幻小说，教育和广播剧。WavJourney的可解释性和交互式设计促进了多轮对话中的人机共同创作，增强了音频制作中的创造性控制和适应性。WavJourney将人类的想象力听觉化，为多媒体内容创作开辟了新的途径。摘要：Large Language Models (LLMs) have shown great promise in integrating diverse expert models to tackle intricate language and vision tasks. Despite their significance in advancing the field of Artificial Intelligence Generated Content (AIGC), their potential in intelligent audio content creation remains unexplored. In this work, we tackle the problem of creating audio content with storylines encompassing speech, music, and sound effects, guided by text instructions. We present WavJourney, a system that leverages LLMs to connect various audio models for audio content generation. Given a text description of an auditory scene, WavJourney first prompts LLMs to generate a structured script dedicated to audio storytelling. The audio script incorporates diverse audio elements, organized based on their spatio-temporal relationships. As a conceptual representation of audio, the audio script provides an interactive and interpretable rationale for human engagement. Afterward, the audio script is fed into a script compiler, converting it into a computer program. Each line of the program calls a task-specific audio generation model or computational operation function (e.g., concatenate, mix). The computer program is then executed to obtain an explainable solution for audio generation. We demonstrate the practicality of WavJourney across diverse real-world scenarios, including science fiction, education, and radio play. The explainable and interactive design of WavJourney fosters human-machine co-creation in multi-round dialogues, enhancing creative control and adaptability in audio production. WavJourney audiolizes the human imagination, opening up new avenues for creativity in multimedia content creation.
【2】 Say Goodbye to RNN-T Loss: A Novel CIF-based Transducer Architecture for  Automatic Speech Recognition标题：告别RNN-T Loss：一种基于CIF的新型自动语音识别换能器结构链接：https://arxiv.org/abs/2307.14132
作者：Tian-Hao Zhang,Dinghao Zhou,Guiping Zhon,Baoxiang Li摘要：RNN-T模型广泛用于ASR中，其依赖于RNN-T损失来实现输入音频和目标序列之间的长度对齐。然而，RNN-T损失的实现复杂性和基于对齐的优化目标分别导致计算冗余和预测器网络的作用降低。在本文中，我们提出了一种新的模型命名为CIF-传感器（CIF-T），它结合了连续积分和发射（CIF）机制与RNN-T模型，以实现高效的对齐。通过这种方式，RNN-T损失被放弃，从而带来了计算量的减少，并允许预测器网络发挥更重要的作用。我们还引入了漏斗CIF，上下文块，统一门控和双线性池联合网络，辅助训练策略，以进一步提高性能。在178小时的AISHELL-1和10000小时的WenetSpeech数据集上的实验表明，与RNN-T模型相比，CIF-T以更低的计算开销实现了最先进的结果。摘要：RNN-T models are widely used in ASR, which rely on the RNN-T loss to achieve length alignment between input audio and target sequence. However, the implementation complexity and the alignment-based optimization target of RNN-T loss lead to computational redundancy and a reduced role for predictor network, respectively. In this paper, we propose a novel model named CIF-Transducer (CIF-T) which incorporates the Continuous Integrate-and-Fire (CIF) mechanism with the RNN-T model to achieve efficient alignment. In this way, the RNN-T loss is abandoned, thus bringing a computational reduction and allowing the predictor network a more significant role. We also introduce Funnel-CIF, Context Blocks, Unified Gating and Bilinear Pooling joint network, and auxiliary training strategy to further improve performance. Experiments on the 178-hour AISHELL-1 and 10000-hour WenetSpeech datasets show that CIF-T achieves state-of-the-art results with lower computational overhead compared to RNN-T models.
【3】 BovineTalk: Machine Learning for Vocalization Analysis of Dairy Cattle  under Negative Affective States标题：BovineTalk：用于负面情绪状态下奶牛发声分析的机器学习链接：https://arxiv.org/abs/2307.13994
作者：Dinu Gavojdian,Teddy Lazebnik,Madalina Mincu,Ariel Oren,Ioana Nicolae,Anna Zamansky摘要：迫切需要开发和验证牲畜物种情感状态的非侵入性动物指标，以便将其整合到农场评估协议中，可能通过使用精确畜牧业（PLF）工具。其中一个有希望的方法是使用声音指标。发声的声学结构和它们的功能在重要的家畜物种中被广泛研究，例如猪、马、家禽和山羊，但是迄今为止在这方面对牛的研究仍然不足。奶牛被证明产生两种类型的发声：低频呼叫（LF），其在嘴闭合或部分闭合的情况下产生，用于近距离接触，以及张开嘴发出的高频呼叫（HF），其产生用于长距离通信，后者被认为在很大程度上与负面情感状态相关联。此外，牛的发声被证明包含在广泛的背景下，无论是积极的还是消极的个性信息。如今，奶牛在一个典型的生产周期中面临着一系列负面的挑战和压力源，在负面情感状态下发声的研究特别感兴趣。本研究的一个贡献是提供了迄今为止最大的预处理（从噪音中清除）数据集的哺乳期成年经产奶牛在视觉隔离挑战引起的负面情感状态。在这里，我们提出了两个计算框架-基于深度学习和基于可解释机器学习，对高频和低频牛叫声进行分类，并对个体牛的声音进行识别。在这两个框架中，我们的模型对LF和HF分类的准确率分别达到87.2%和89.4%，对奶牛个体识别的准确率分别为68.9%和72.5%。摘要：There is a critical need to develop and validate non-invasive animal-based indicators of affective states in livestock species, in order to integrate them into on-farm assessment protocols, potentially via the use of precision livestock farming (PLF) tools. One such promising approach is the use of vocal indicators. The acoustic structure of vocalizations and their functions were extensively studied in important livestock species, such as pigs, horses, poultry and goats, yet cattle remain understudied in this context to date. Cows were shown to produce two types vocalizations: low-frequency calls (LF), produced with the mouth closed, or partially closed, for close distance contacts and open mouth emitted high-frequency calls (HF), produced for long distance communication, with the latter considered to be largely associated with negative affective states. Moreover, cattle vocalizations were shown to contain information on individuality across a wide range of contexts, both negative and positive. Nowadays, dairy cows are facing a series of negative challenges and stressors in a typical production cycle, making vocalizations during negative affective states of special interest for research. One contribution of this study is providing the largest to date pre-processed (clean from noises) dataset of lactating adult multiparous dairy cows during negative affective states induced by visual isolation challenges. Here we present two computational frameworks - deep learning based and explainable machine learning based, to classify high and low-frequency cattle calls, and individual cow voice recognition. Our models in these two frameworks reached 87.2% and 89.4% accuracy for LF and HF classification, with 68.9% and 72.5% accuracy rates for the cow individual identification, respectively.
【4】 The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link  between Phonemes and Facial Features标题：音素与表情的隐藏之舞：揭开音素与面部特征之间的神秘联系链接：https://arxiv.org/abs/2307.13953
作者：Liao Qu,Xianwei Zou,Xiang Li,Yandong Wen,Rita Singh,Bhiksha Raj备注：Interspeech 2023摘要：这项工作揭示了音素和面部特征之间的神秘联系。传统的语音-人脸相关性研究通常涉及使用长时间的语音输入，包括从语音生成人脸图像和从语音重建3D人脸网格。然而，在基于语音的犯罪等情况下，可用的语音证据可能很短且有限。此外，从生理学的角度来看，语音的每个片段-音素-对应于面部不同类型的气流和运动。因此，发现音素和面部属性之间的隐藏联系是有利的。在本文中，我们提出了一个分析管道来帮助我们以细粒度的方式探索语音-面部关系，即，音素与面部人体测量（AM）。我们建立了一个估计每个音素AM对，并通过假设检验的相关性进行评估。我们的研究结果表明，AM是更可预测的元音相比，辅音，特别是与爆破音。此外，我们观察到，如果一个特定的AM在音素发音过程中表现出更多的运动，它是更可预测的。我们的研究结果支持了生理学中的相关性，并为未来的语音-人脸多模态学习研究奠定了基础。摘要：This work unveils the enigmatic link between phonemes and facial features. Traditional studies on voice-face correlations typically involve using a long period of voice input, including generating face images from voices and reconstructing 3D face meshes from voices. However, in situations like voice-based crimes, the available voice evidence may be short and limited. Additionally, from a physiological perspective, each segment of speech -- phoneme -- corresponds to different types of airflow and movements in the face. Therefore, it is advantageous to discover the hidden link between phonemes and face attributes. In this paper, we propose an analysis pipeline to help us explore the voice-face relationship in a fine-grained manner, i.e., phonemes v.s. facial anthropometric measurements (AM). We build an estimator for each phoneme-AM pair and evaluate the correlation through hypothesis testing. Our results indicate that AMs are more predictable from vowels compared to consonants, particularly with plosives. Additionally, we observe that if a specific AM exhibits more movement during phoneme pronunciation, it is more predictable. Our findings support those in physiology regarding correlation and lay the groundwork for future research on speech-face multimodal learning.
【5】 Rethinking Voice-Face Correlation: A Geometry View标题：从几何学角度重新思考声音与脸部的相关性链接：https://arxiv.org/abs/2307.13948
作者：Xiang Li,Yandong Wen,Muqiao Yang,Jinglu Wang,Rita Singh,Bhiksha Raj备注：ACM Multimedia 2023摘要：以前的工作语音-人脸匹配和语音引导的人脸合成表现出很强的语音和面部之间的相关性，但主要依赖于粗糙的语义线索，如性别，年龄和情感。在本文中，我们的目标是调查的能力重建的三维人脸形状从语音从几何角度没有任何语义信息。我们提出了一个语音人体测量（AM）的脸范式，识别可预测的面部AM的声音，并使用它们来指导三维人脸重建。通过利用AM作为代理来链接语音和面部几何，我们可以消除不可预测的AM的影响，并使面部几何易于处理。我们的方法在我们提出的数据集上进行了评估，其中包含地面真实3D人脸扫描和相应的语音记录，我们发现语音与面部几何形状的特定部分（如鼻腔和颅骨）之间存在显着相关性。我们的工作提供了一个新的视角，声音，人脸相关性，可以作为一个很好的人体测量科学的实证研究。摘要：Previous works on voice-face matching and voice-guided face synthesis demonstrate strong correlations between voice and face, but mainly rely on coarse semantic cues such as gender, age, and emotion. In this paper, we aim to investigate the capability of reconstructing the 3D facial shape from voice from a geometry perspective without any semantic information. We propose a voice-anthropometric measurement (AM)-face paradigm, which identifies predictable facial AMs from the voice and uses them to guide 3D face reconstruction. By leveraging AMs as a proxy to link the voice and face geometry, we can eliminate the influence of unpredictable AMs and make the face geometry tractable. Our approach is evaluated on our proposed dataset with ground-truth 3D face scans and corresponding voice recordings, and we find significant correlations between voice and specific parts of the face geometry, such as the nasal cavity and cranium. Our work offers a new perspective on voice-face correlation and can serve as a good empirical study for anthropometry science.
【6】 Fitting Auditory Filterbanks with Multiresolution Neural Networks标题：用多分辨神经网络拟合听觉滤波器组链接：https://arxiv.org/abs/2307.13821
作者：Vincent Lostanlen,Daniel Haider,Han Han,Mathieu Lagrange,Peter Balazs,Martin Ehler备注：4 pages, 4 figures, 1 table, conference摘要：基于波形的深度学习面临着非参数和参数方法之间的困境。一方面，卷积神经网络（convnets）可以逼近任何线性时不变系统;然而在实践中，它们的频率响应随着它们的感受野的增长而变得更加不规则。另一方面，一个参数模型，如LEAF，保证产生Gabor滤波器，从而最佳的时频局部化;然而，这种强烈的归纳偏见损害了代表能力。在本文中，我们的目标是克服这种困境，通过引入一个神经音频模型，命名为多分辨率神经网络（MuReNN）。MuReNN背后的关键思想是在离散小波变换（DWT）的倍频程子带上训练单独的卷积算子。由于DWT原子的规模在倍频程之间呈指数增长，因此MuReNN中后续可学习卷积的感受野相应地扩大。对于给定的真实世界数据集，我们将MuReNN的幅度响应拟合到完善的听觉滤波器组的幅度响应：伽马音用于语音，CQT用于音乐，第三个八度音用于城市声音。这是知识蒸馏（KD）的一种形式，其中过滤器组“教师”由领域知识设计，而神经网络“学生”由数据优化。我们将MuReNN与最先进的技术在KD后的拟合优度和海森堡时频局部化方面进行比较。与convnets和Gabor卷积相比，我们发现MuReNN在所有三个优化问题上都达到了最先进的性能。摘要：Waveform-based deep learning faces a dilemma between nonparametric and parametric approaches. On one hand, convolutional neural networks (convnets) may approximate any linear time-invariant system; yet, in practice, their frequency responses become more irregular as their receptive fields grow. On the other hand, a parametric model such as LEAF is guaranteed to yield Gabor filters, hence an optimal time-frequency localization; yet, this strong inductive bias comes at the detriment of representational capacity. In this paper, we aim to overcome this dilemma by introducing a neural audio model, named multiresolution neural network (MuReNN). The key idea behind MuReNN is to train separate convolutional operators over the octave subbands of a discrete wavelet transform (DWT). Since the scale of DWT atoms grows exponentially between octaves, the receptive fields of the subsequent learnable convolutions in MuReNN are dilated accordingly. For a given real-world dataset, we fit the magnitude response of MuReNN to that of a well-established auditory filterbank: Gammatone for speech, CQT for music, and third-octave for urban sounds, respectively. This is a form of knowledge distillation (KD), in which the filterbank ''teacher'' is engineered by domain knowledge while the neural network ''student'' is optimized from data. We compare MuReNN to the state of the art in terms of goodness of fit after KD on a hold-out set and in terms of Heisenberg time-frequency localization. Compared to convnets and Gabor convolutions, we find that MuReNN reaches state-of-the-art performance on all three optimization problems.
【7】 Histogram Layer Time Delay Neural Networks for Passive Sonar  Classification标题：基于直方图层时延神经网络的被动声纳分类链接：https://arxiv.org/abs/2307.13788
作者：Jarin Ritu,Ethan Barnes,Riley Martell,Alexandra Van Dine,Joshua Peeples备注：5 pages, 5 Figures, Accepted to 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics摘要：由于复杂的声波传播特性，海洋遥感中的水声目标检测具有挑战性。尽管可靠的声纳系统的可用性，目标识别仍然是一个困难的问题。各种方法致力于改进的目标识别。然而，大多数的斗争，以解开所观察到的目标记录的高维，非线性模式。在这项工作中，一种新的方法相结合的时间延迟神经网络和直方图层，将统计的背景下，改进的特征学习和水声目标分类。所提出的方法优于基线模型，展示了实用程序，将统计背景被动声纳目标识别。这项工作的代码是公开的。摘要：Underwater acoustic target detection in remote marine sensing operations is challenging due to complex sound wave propagation. Despite the availability of reliable sonar systems, target recognition remains a difficult problem. Various methods address improved target recognition. However, most struggle to disentangle the high-dimensional, non-linear patterns in the observed target recordings. In this work, a novel method combines a time delay neural network and histogram layer to incorporate statistical contexts for improved feature learning and underwater acoustic target classification. The proposed method outperforms the baseline model, demonstrating the utility in incorporating statistical contexts for passive sonar target recognition. The code for this work is publicly available.
【8】 Perceptual Quality Enhancement of Sound Field Synthesis Based on  Combination of Pressure and Amplitude Matching标题：基于声压和幅度匹配相结合的声场合成感知质量增强链接：https://arxiv.org/abs/2307.13941
作者：Keisuke Kimura,Shoichi Koyama,Hiroshi Saruwatari备注：Accepted to IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) 2023摘要：提出了一种增强感知质量的声场合成方法。使用多个扬声器的声场合成实现了具有广阔收听区域的空间音频再现;然而，在高频处被称为空间混叠伪像的合成误差是不可避免的。为了最大限度地减少这些文物，我们提出了一种方法的基础上相结合的压力和幅度匹配。在人的听觉特性的基础上，合成幅度分布将足以用于水平声音定位。此外，应尽可能合成平坦的振幅响应以避免着色。因此，我们应用振幅匹配，这是一种方法来合成所需的振幅分布与任意相位分布，为高频率和常规的压力匹配为低频率。数值模拟和实际系统的实验结果表明，该方法合成的声场的感知质量得到改善，从压力匹配合成。摘要：A sound field synthesis method enhancing perceptual quality is proposed. Sound field synthesis using multiple loudspeakers enables spatial audio reproduction with a broad listening area; however, synthesis errors at high frequencies called spatial aliasing artifacts are unavoidable. To minimize these artifacts, we propose a method based on the combination of pressure and amplitude matching. On the basis of the human's auditory properties, synthesizing the amplitude distribution will be sufficient for horizontal sound localization. Furthermore, a flat amplitude response should be synthesized as much as possible to avoid coloration. Therefore, we apply amplitude matching, which is a method to synthesize the desired amplitude distribution with arbitrary phase distribution, for high frequencies and conventional pressure matching for low frequencies. Experimental results of numerical simulations and listening tests using a practical system indicated that the perceptual quality of the sound field synthesized by the proposed method was improved from that synthesized by pressure matching.
【9】 Exploring the Interactions between Target Positive and Negative  Information for Acoustic Echo Cancellation标题：声学回波抵消中目标正负信息相互作用的研究链接：https://arxiv.org/abs/2307.13888
作者：Chang Han,Xinmeng Xu,Weiping Tu,Yuhong Yang,Yajie Liu备注：Accepted at INTERSPEECH 2023摘要：声学回声消除（AEC）的目的是去除干扰信号，同时使近端语音失真最小。由于近端语音和干扰信号的模式难以区分，近端语音不能完全分离，造成语音失真和干扰信号残留。我们观察到，除了目标积极信息，例如，参考地面实况语音和特征，目标负面信息，例如干扰信号和特征，有助于使目标语音和干扰信号的模式更具区分性。因此，我们提出了一种新的AEC模型编码器-解码器架构的指导下，负信息称为CMNet。协作模块（CM）被设计为经由三个块以可学习的方式建立目标正面信息和负面信息之间的相关性：目标正、目标负和交互块。实验结果表明，我们的CMNet实现了优越的性能比最近的方法。摘要：Acoustic echo cancellation (AEC) aims to remove interference signals while leaving near-end speech least distorted. As the indistinguishable patterns between near-end speech and interference signals, near-end speech can't be separated completely, causing speech distortion and interference signals residual. We observe that besides target positive information, e.g., ground-truth speech and features, the target negative information, such as interference signals and features, helps make pattern of target speech and interference signals more discriminative. Therefore, we present a novel AEC model encoder-decoder architecture with the guidance of negative information termed as CMNet. A collaboration module (CM) is designed to establish the correlation between the target positive and negative information in a learnable manner via three blocks: target positive, target negative, and interactive block. Experimental results demonstrate our CMNet achieves superior performance than recent methods.
eess.AS音频处理
【1】 NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals标题：NeuroHeed：基于脑电信号的神经控制说话人提取链接：https://arxiv.org/abs/2307.14303
作者：Zexu Pan,Marvin Borsdorf,Siqi Cai,Tanja Schultz,Haizhou Li摘要：人类拥有在竞争的声音和背景噪音中选择性地注意单个说话者的非凡能力，称为选择性听觉注意。听觉神经科学的最新研究表明，出席的语音信号和相应的大脑引起的神经元活动，后者可以测量使用负担得起的和非侵入性的脑电图（EEG）设备之间的强相关性。在这项研究中，我们提出了NeuroHeed，扬声器提取模型，利用EEG信号建立一个神经元吸引子，这是暂时与语音刺激，促进出席的语音信号的提取在鸡尾酒会的情况下。我们提出了一个离线和在线NeuroHeed，后者设计用于实时推理。在在线NeuroHeed中，我们还提出了一个自回归扬声器编码器，它积累了过去提取的语音信号，用于将参加的扬声器信息自注册到听觉吸引子中，该听觉吸引子随着时间的推移保留了注意力动量。在线NeuroHeed在来自两个吸引子的指导下提取语音信号的当前窗口。实验结果表明，NeuroHeed有效地提取大脑参加的语音信号，实现高信号质量，优秀的感知质量，在两个扬声器的情况下的可理解性。摘要：Humans possess the remarkable ability to selectively attend to a single speaker amidst competing voices and background noise, known as selective auditory attention. Recent studies in auditory neuroscience indicate a strong correlation between the attended speech signal and the corresponding brain's elicited neuronal activities, which the latter can be measured using affordable and non-intrusive electroencephalography (EEG) devices. In this study, we present NeuroHeed, a speaker extraction model that leverages EEG signals to establish a neuronal attractor which is temporally associated with the speech stimulus, facilitating the extraction of the attended speech signal in a cocktail party scenario. We propose both an offline and an online NeuroHeed, with the latter designed for real-time inference. In the online NeuroHeed, we additionally propose an autoregressive speaker encoder, which accumulates past extracted speech signals for self-enrollment of the attended speaker information into an auditory attractor, that retains the attentional momentum over time. Online NeuroHeed extracts the current window of the speech signals with guidance from both attractors. Experimental results demonstrate that NeuroHeed effectively extracts brain-attended speech signals, achieving high signal quality, excellent perceptual quality, and intelligibility in a two-speaker scenario.
【2】 Sound Field Estimation around a Rigid Sphere with Physics-informed  Neural Network标题：基于物理信息神经网络的刚性球体周围声场估计链接：https://arxiv.org/abs/2307.14013
作者：Xingyu Chen,Fei Ma,Amy Bastine,Prasanga Samarasinghe,Huiyuan Sun摘要：精确估计刚性球体周围的声场需要对球体进行充分采样，这可能并不总是可能的。为了克服这一挑战，本文提出了一种基于物理信息神经网络的声场估计方法。这种方法将物理知识集成到网络的架构和训练过程中。与其他基于学习的方法相比，所提出的方法采用了来自亥姆霍兹方程和零径向速度条件的刚性球的额外的约束。因此，它可以生成物理上可行的估计，而不需要大的数据集。与基于球谐函数的方法相比，该方法具有更好的拟合能力，并避免了截断引起的病态。仿真结果表明，该方法在实现准确的声场估计从有限的测量，优于球谐方法和平面波分解方法的有效性。摘要：Accurate estimation of the sound field around a rigid sphere necessitates adequate sampling on the sphere, which may not always be possible. To overcome this challenge, this paper proposes a method for sound field estimation based on a physics-informed neural network. This approach integrates physical knowledge into the architecture and training process of the network. In contrast to other learning-based methods, the proposed method incorporates additional constraints derived from the Helmholtz equation and the zero radial velocity condition on the rigid sphere. Consequently, it can generate physically feasible estimations without requiring a large dataset. In contrast to the spherical harmonic-based method, the proposed approach has better fitting abilities and circumvents the ill condition caused by truncation. Simulation results demonstrate the effectiveness of the proposed method in achieving accurate sound field estimations from limited measurements, outperforming the spherical harmonic method and plane-wave decomposition method.
【3】 Perceptual Quality Enhancement of Sound Field Synthesis Based on  Combination of Pressure and Amplitude Matching标题：基于声压和幅度匹配相结合的声场合成感知质量增强链接：https://arxiv.org/abs/2307.13941
作者：Keisuke Kimura,Shoichi Koyama,Hiroshi Saruwatari备注：Accepted to IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) 2023摘要：提出了一种增强感知质量的声场合成方法。使用多个扬声器的声场合成实现了具有广阔收听区域的空间音频再现;然而，在高频处被称为空间混叠伪像的合成误差是不可避免的。为了最大限度地减少这些文物，我们提出了一种方法的基础上相结合的压力和幅度匹配。在人的听觉特性的基础上，合成幅度分布将足以用于水平声音定位。此外，应尽可能合成平坦的振幅响应以避免着色。因此，我们应用振幅匹配，这是一种方法来合成所需的振幅分布与任意相位分布，为高频率和常规的压力匹配为低频率。数值模拟和实际系统的实验结果表明，该方法合成的声场的感知质量得到改善，从压力匹配合成。摘要：A sound field synthesis method enhancing perceptual quality is proposed. Sound field synthesis using multiple loudspeakers enables spatial audio reproduction with a broad listening area; however, synthesis errors at high frequencies called spatial aliasing artifacts are unavoidable. To minimize these artifacts, we propose a method based on the combination of pressure and amplitude matching. On the basis of the human's auditory properties, synthesizing the amplitude distribution will be sufficient for horizontal sound localization. Furthermore, a flat amplitude response should be synthesized as much as possible to avoid coloration. Therefore, we apply amplitude matching, which is a method to synthesize the desired amplitude distribution with arbitrary phase distribution, for high frequencies and conventional pressure matching for low frequencies. Experimental results of numerical simulations and listening tests using a practical system indicated that the perceptual quality of the sound field synthesized by the proposed method was improved from that synthesized by pressure matching.
【4】 Exploring the Interactions between Target Positive and Negative  Information for Acoustic Echo Cancellation标题：声学回波抵消中目标正负信息相互作用的研究链接：https://arxiv.org/abs/2307.13888
作者：Chang Han,Xinmeng Xu,Weiping Tu,Yuhong Yang,Yajie Liu备注：Accepted at INTERSPEECH 2023摘要：声学回声消除（AEC）的目的是去除干扰信号，同时使近端语音失真最小。由于近端语音和干扰信号的模式难以区分，近端语音不能完全分离，造成语音失真和干扰信号残留。我们观察到，除了目标积极信息，例如，参考地面实况语音和特征，目标负面信息，例如干扰信号和特征，有助于使目标语音和干扰信号的模式更具区分性。因此，我们提出了一种新的AEC模型编码器-解码器架构的指导下，负信息称为CMNet。协作模块（CM）被设计为经由三个块以可学习的方式建立目标正面信息和负面信息之间的相关性：目标正、目标负和交互块。实验结果表明，我们的CMNet实现了优越的性能比最近的方法。摘要：Acoustic echo cancellation (AEC) aims to remove interference signals while leaving near-end speech least distorted. As the indistinguishable patterns between near-end speech and interference signals, near-end speech can't be separated completely, causing speech distortion and interference signals residual. We observe that besides target positive information, e.g., ground-truth speech and features, the target negative information, such as interference signals and features, helps make pattern of target speech and interference signals more discriminative. Therefore, we present a novel AEC model encoder-decoder architecture with the guidance of negative information termed as CMNet. A collaboration module (CM) is designed to establish the correlation between the target positive and negative information in a learnable manner via three blocks: target positive, target negative, and interactive block. Experimental results demonstrate our CMNet achieves superior performance than recent methods.
【5】 WavJourney: Compositional Audio Creation with Large Language Models标题：WavJourney：使用大型语言模型进行作曲音频创作链接：https://arxiv.org/abs/2307.14335
作者：Xubo Liu,Zhongkai Zhu,Haohe Liu,Yi Yuan,Meng Cui,Qiushi Huang,Jinhua Liang,Yin Cao,Qiuqiang Kong,Mark D. Plumbley,Wenwu Wang备注：Project Page: this https URL摘要：大型语言模型（LLM）在集成各种专家模型以解决复杂的语言和视觉任务方面表现出了巨大的潜力。尽管它们在推进人工智能生成内容（AIGC）领域中具有重要意义，但它们在智能音频内容创作中的潜力仍未被开发。在这项工作中，我们解决了创建音频内容的问题，故事情节包括语音，音乐和声音效果，由文本说明指导。我们提出了WavJourney，一个系统，利用LLM连接各种音频模型的音频内容生成。给定听觉场景的文本描述，WavJourney首先提示LLM生成专用于音频故事讲述的结构化脚本。音频脚本包含基于其时空关系组织的各种音频元素。作为音频的概念表示，音频脚本为人类参与提供了交互式和可解释的基本原理。之后，音频脚本被输入脚本编译器，将其转换为计算机程序。程序的每一行调用任务特定的音频生成模型或计算操作函数（例如，连接、混合）。然后执行计算机程序以获得用于音频生成的可解释的解决方案。我们展示了WavJourney在各种现实世界场景中的实用性，包括科幻小说，教育和广播剧。WavJourney的可解释性和交互式设计促进了多轮对话中的人机共同创作，增强了音频制作中的创造性控制和适应性。WavJourney将人类的想象力听觉化，为多媒体内容创作开辟了新的途径。摘要：Large Language Models (LLMs) have shown great promise in integrating diverse expert models to tackle intricate language and vision tasks. Despite their significance in advancing the field of Artificial Intelligence Generated Content (AIGC), their potential in intelligent audio content creation remains unexplored. In this work, we tackle the problem of creating audio content with storylines encompassing speech, music, and sound effects, guided by text instructions. We present WavJourney, a system that leverages LLMs to connect various audio models for audio content generation. Given a text description of an auditory scene, WavJourney first prompts LLMs to generate a structured script dedicated to audio storytelling. The audio script incorporates diverse audio elements, organized based on their spatio-temporal relationships. As a conceptual representation of audio, the audio script provides an interactive and interpretable rationale for human engagement. Afterward, the audio script is fed into a script compiler, converting it into a computer program. Each line of the program calls a task-specific audio generation model or computational operation function (e.g., concatenate, mix). The computer program is then executed to obtain an explainable solution for audio generation. We demonstrate the practicality of WavJourney across diverse real-world scenarios, including science fiction, education, and radio play. The explainable and interactive design of WavJourney fosters human-machine co-creation in multi-round dialogues, enhancing creative control and adaptability in audio production. WavJourney audiolizes the human imagination, opening up new avenues for creativity in multimedia content creation.
【6】 Say Goodbye to RNN-T Loss: A Novel CIF-based Transducer Architecture for  Automatic Speech Recognition标题：告别RNN-T Loss：一种基于CIF的新型自动语音识别换能器结构链接：https://arxiv.org/abs/2307.14132
作者：Tian-Hao Zhang,Dinghao Zhou,Guiping Zhon,Baoxiang Li摘要：RNN-T模型广泛用于ASR中，其依赖于RNN-T损失来实现输入音频和目标序列之间的长度对齐。然而，RNN-T损失的实现复杂性和基于对齐的优化目标分别导致计算冗余和预测器网络的作用降低。在本文中，我们提出了一种新的模型命名为CIF-传感器（CIF-T），它结合了连续积分和发射（CIF）机制与RNN-T模型，以实现高效的对齐。通过这种方式，RNN-T损失被放弃，从而带来了计算量的减少，并允许预测器网络发挥更重要的作用。我们还引入了漏斗CIF，上下文块，统一门控和双线性池联合网络，辅助训练策略，以进一步提高性能。在178小时的AISHELL-1和10000小时的WenetSpeech数据集上的实验表明，与RNN-T模型相比，CIF-T以更低的计算开销实现了最先进的结果。摘要：RNN-T models are widely used in ASR, which rely on the RNN-T loss to achieve length alignment between input audio and target sequence. However, the implementation complexity and the alignment-based optimization target of RNN-T loss lead to computational redundancy and a reduced role for predictor network, respectively. In this paper, we propose a novel model named CIF-Transducer (CIF-T) which incorporates the Continuous Integrate-and-Fire (CIF) mechanism with the RNN-T model to achieve efficient alignment. In this way, the RNN-T loss is abandoned, thus bringing a computational reduction and allowing the predictor network a more significant role. We also introduce Funnel-CIF, Context Blocks, Unified Gating and Bilinear Pooling joint network, and auxiliary training strategy to further improve performance. Experiments on the 178-hour AISHELL-1 and 10000-hour WenetSpeech datasets show that CIF-T achieves state-of-the-art results with lower computational overhead compared to RNN-T models.
【7】 BovineTalk: Machine Learning for Vocalization Analysis of Dairy Cattle  under Negative Affective States标题：BovineTalk：用于负面情绪状态下奶牛发声分析的机器学习链接：https://arxiv.org/abs/2307.13994
作者：Dinu Gavojdian,Teddy Lazebnik,Madalina Mincu,Ariel Oren,Ioana Nicolae,Anna Zamansky摘要：迫切需要开发和验证牲畜物种情感状态的非侵入性动物指标，以便将其整合到农场评估协议中，可能通过使用精确畜牧业（PLF）工具。其中一个有希望的方法是使用声音指标。发声的声学结构和它们的功能在重要的家畜物种中被广泛研究，例如猪、马、家禽和山羊，但是迄今为止在这方面对牛的研究仍然不足。奶牛被证明产生两种类型的发声：低频呼叫（LF），其在嘴闭合或部分闭合的情况下产生，用于近距离接触，以及张开嘴发出的高频呼叫（HF），其产生用于长距离通信，后者被认为在很大程度上与负面情感状态相关联。此外，牛的发声被证明包含在广泛的背景下，无论是积极的还是消极的个性信息。如今，奶牛在一个典型的生产周期中面临着一系列负面的挑战和压力源，在负面情感状态下发声的研究特别感兴趣。本研究的一个贡献是提供了迄今为止最大的预处理（从噪音中清除）数据集的哺乳期成年经产奶牛在视觉隔离挑战引起的负面情感状态。在这里，我们提出了两个计算框架-基于深度学习和基于可解释机器学习，对高频和低频牛叫声进行分类，并对个体牛的声音进行识别。在这两个框架中，我们的模型对LF和HF分类的准确率分别达到87.2%和89.4%，对奶牛个体识别的准确率分别为68.9%和72.5%。摘要：There is a critical need to develop and validate non-invasive animal-based indicators of affective states in livestock species, in order to integrate them into on-farm assessment protocols, potentially via the use of precision livestock farming (PLF) tools. One such promising approach is the use of vocal indicators. The acoustic structure of vocalizations and their functions were extensively studied in important livestock species, such as pigs, horses, poultry and goats, yet cattle remain understudied in this context to date. Cows were shown to produce two types vocalizations: low-frequency calls (LF), produced with the mouth closed, or partially closed, for close distance contacts and open mouth emitted high-frequency calls (HF), produced for long distance communication, with the latter considered to be largely associated with negative affective states. Moreover, cattle vocalizations were shown to contain information on individuality across a wide range of contexts, both negative and positive. Nowadays, dairy cows are facing a series of negative challenges and stressors in a typical production cycle, making vocalizations during negative affective states of special interest for research. One contribution of this study is providing the largest to date pre-processed (clean from noises) dataset of lactating adult multiparous dairy cows during negative affective states induced by visual isolation challenges. Here we present two computational frameworks - deep learning based and explainable machine learning based, to classify high and low-frequency cattle calls, and individual cow voice recognition. Our models in these two frameworks reached 87.2% and 89.4% accuracy for LF and HF classification, with 68.9% and 72.5% accuracy rates for the cow individual identification, respectively.
【8】 The Hidden Dance of Phonemes and Visage: Unveiling the Enigmatic Link  between Phonemes and Facial Features标题：音素与表情的隐藏之舞：揭开音素与面部特征之间的神秘联系链接：https://arxiv.org/abs/2307.13953
作者：Liao Qu,Xianwei Zou,Xiang Li,Yandong Wen,Rita Singh,Bhiksha Raj备注：Interspeech 2023摘要：这项工作揭示了音素和面部特征之间的神秘联系。传统的语音-人脸相关性研究通常涉及使用长时间的语音输入，包括从语音生成人脸图像和从语音重建3D人脸网格。然而，在基于语音的犯罪等情况下，可用的语音证据可能很短且有限。此外，从生理学的角度来看，语音的每个片段-音素-对应于面部不同类型的气流和运动。因此，发现音素和面部属性之间的隐藏联系是有利的。在本文中，我们提出了一个分析管道来帮助我们以细粒度的方式探索语音-面部关系，即，音素与面部人体测量（AM）。我们建立了一个估计每个音素AM对，并通过假设检验的相关性进行评估。我们的研究结果表明，AM是更可预测的元音相比，辅音，特别是与爆破音。此外，我们观察到，如果一个特定的AM在音素发音过程中表现出更多的运动，它是更可预测的。我们的研究结果支持了生理学中的相关性，并为未来的语音-人脸多模态学习研究奠定了基础。摘要：This work unveils the enigmatic link between phonemes and facial features. Traditional studies on voice-face correlations typically involve using a long period of voice input, including generating face images from voices and reconstructing 3D face meshes from voices. However, in situations like voice-based crimes, the available voice evidence may be short and limited. Additionally, from a physiological perspective, each segment of speech -- phoneme -- corresponds to different types of airflow and movements in the face. Therefore, it is advantageous to discover the hidden link between phonemes and face attributes. In this paper, we propose an analysis pipeline to help us explore the voice-face relationship in a fine-grained manner, i.e., phonemes v.s. facial anthropometric measurements (AM). We build an estimator for each phoneme-AM pair and evaluate the correlation through hypothesis testing. Our results indicate that AMs are more predictable from vowels compared to consonants, particularly with plosives. Additionally, we observe that if a specific AM exhibits more movement during phoneme pronunciation, it is more predictable. Our findings support those in physiology regarding correlation and lay the groundwork for future research on speech-face multimodal learning.
【9】 Rethinking Voice-Face Correlation: A Geometry View标题：从几何学角度重新思考声音与脸部的相关性链接：https://arxiv.org/abs/2307.13948
作者：Xiang Li,Yandong Wen,Muqiao Yang,Jinglu Wang,Rita Singh,Bhiksha Raj备注：ACM Multimedia 2023摘要：以前的工作语音-人脸匹配和语音引导的人脸合成表现出很强的语音和面部之间的相关性，但主要依赖于粗糙的语义线索，如性别，年龄和情感。在本文中，我们的目标是调查的能力重建的三维人脸形状从语音从几何角度没有任何语义信息。我们提出了一个语音人体测量（AM）的脸范式，识别可预测的面部AM的声音，并使用它们来指导三维人脸重建。通过利用AM作为代理来链接语音和面部几何，我们可以消除不可预测的AM的影响，并使面部几何易于处理。我们的方法在我们提出的数据集上进行了评估，其中包含地面真实3D人脸扫描和相应的语音记录，我们发现语音与面部几何形状的特定部分（如鼻腔和颅骨）之间存在显着相关性。我们的工作提供了一个新的视角，声音，人脸相关性，可以作为一个很好的人体测量科学的实证研究。摘要：Previous works on voice-face matching and voice-guided face synthesis demonstrate strong correlations between voice and face, but mainly rely on coarse semantic cues such as gender, age, and emotion. In this paper, we aim to investigate the capability of reconstructing the 3D facial shape from voice from a geometry perspective without any semantic information. We propose a voice-anthropometric measurement (AM)-face paradigm, which identifies predictable facial AMs from the voice and uses them to guide 3D face reconstruction. By leveraging AMs as a proxy to link the voice and face geometry, we can eliminate the influence of unpredictable AMs and make the face geometry tractable. Our approach is evaluated on our proposed dataset with ground-truth 3D face scans and corresponding voice recordings, and we find significant correlations between voice and specific parts of the face geometry, such as the nasal cavity and cranium. Our work offers a new perspective on voice-face correlation and can serve as a good empirical study for anthropometry science.
【10】 Fitting Auditory Filterbanks with Multiresolution Neural Networks标题：用多分辨神经网络拟合听觉滤波器组链接：https://arxiv.org/abs/2307.13821
作者：Vincent Lostanlen,Daniel Haider,Han Han,Mathieu Lagrange,Peter Balazs,Martin Ehler备注：4 pages, 4 figures, 1 table, conference摘要：基于波形的深度学习面临着非参数和参数方法之间的困境。一方面，卷积神经网络（convnets）可以逼近任何线性时不变系统;然而在实践中，它们的频率响应随着它们的感受野的增长而变得更加不规则。另一方面，一个参数模型，如LEAF，保证产生Gabor滤波器，从而最佳的时频局部化;然而，这种强烈的归纳偏见损害了代表能力。在本文中，我们的目标是克服这种困境，通过引入一个神经音频模型，命名为多分辨率神经网络（MuReNN）。MuReNN背后的关键思想是在离散小波变换（DWT）的倍频程子带上训练单独的卷积算子。由于DWT原子的规模在倍频程之间呈指数增长，因此MuReNN中后续可学习卷积的感受野相应地扩大。对于给定的真实世界数据集，我们将MuReNN的幅度响应拟合到完善的听觉滤波器组的幅度响应：伽马音用于语音，CQT用于音乐，第三个八度音用于城市声音。这是知识蒸馏（KD）的一种形式，其中过滤器组“教师”由领域知识设计，而神经网络“学生”由数据优化。我们将MuReNN与最先进的技术在KD后的拟合优度和海森堡时频局部化方面进行比较。与convnets和Gabor卷积相比，我们发现MuReNN在所有三个优化问题上都达到了最先进的性能。摘要：Waveform-based deep learning faces a dilemma between nonparametric and parametric approaches. On one hand, convolutional neural networks (convnets) may approximate any linear time-invariant system; yet, in practice, their frequency responses become more irregular as their receptive fields grow. On the other hand, a parametric model such as LEAF is guaranteed to yield Gabor filters, hence an optimal time-frequency localization; yet, this strong inductive bias comes at the detriment of representational capacity. In this paper, we aim to overcome this dilemma by introducing a neural audio model, named multiresolution neural network (MuReNN). The key idea behind MuReNN is to train separate convolutional operators over the octave subbands of a discrete wavelet transform (DWT). Since the scale of DWT atoms grows exponentially between octaves, the receptive fields of the subsequent learnable convolutions in MuReNN are dilated accordingly. For a given real-world dataset, we fit the magnitude response of MuReNN to that of a well-established auditory filterbank: Gammatone for speech, CQT for music, and third-octave for urban sounds, respectively. This is a form of knowledge distillation (KD), in which the filterbank ''teacher'' is engineered by domain knowledge while the neural network ''student'' is optimized from data. We compare MuReNN to the state of the art in terms of goodness of fit after KD on a hold-out set and in terms of Heisenberg time-frequency localization. Compared to convnets and Gabor convolutions, we find that MuReNN reaches state-of-the-art performance on all three optimization problems.
【11】 Histogram Layer Time Delay Neural Networks for Passive Sonar  Classification标题：基于直方图层时延神经网络的被动声纳分类链接：https://arxiv.org/abs/2307.13788
作者：Jarin Ritu,Ethan Barnes,Riley Martell,Alexandra Van Dine,Joshua Peeples备注：5 pages, 5 Figures, Accepted to 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics摘要：由于复杂的声波传播特性，海洋遥感中的水声目标检测具有挑战性。尽管可靠的声纳系统的可用性，目标识别仍然是一个困难的问题。各种方法致力于改进的目标识别。然而，大多数的斗争，以解开所观察到的目标记录的高维，非线性模式。在这项工作中，一种新的方法相结合的时间延迟神经网络和直方图层，将统计的背景下，改进的特征学习和水声目标分类。所提出的方法优于基线模型，展示了实用程序，将统计背景被动声纳目标识别。这项工作的代码是公开的。摘要：Underwater acoustic target detection in remote marine sensing operations is challenging due to complex sound wave propagation. Despite the availability of reliable sonar systems, target recognition remains a difficult problem. Various methods address improved target recognition. However, most struggle to disentangle the high-dimensional, non-linear patterns in the observed target recordings. In this work, a novel method combines a time delay neural network and histogram layer to incorporate statistical contexts for improved feature learning and underwater acoustic target classification. The proposed method outperforms the baseline model, demonstrating the utility in incorporating statistical contexts for passive sonar target recognition. The code for this work is publicly available.
机器翻译，仅供参考