今日论文合集:cs.SD语音4篇,eess.AS音频处理7篇。cs.SD语音【1】On the Out-Of-Distribution Robustness of Self-Supervised Representation Learning for Phonocardiogram Signals标题:心音信号自监督表示学习的分布外稳健性链接:https://arxiv.org/abs/2312.00502作者:Aristotelis Ballas,Vasileios Papapanagiotou,Christos Diou备注:PREPRINT Manuscript under review摘要:尽管最近研究活动有所增加,但深度学习模型尚未在医学中被广泛接受。高质量的注释数据的短缺往往阻碍了强大的和可推广的模型的开发,这些模型在使用新收集的非分布(OOD)数据集时不会降低有效性。研究方法:对比自监督学习(SSL)提供了一个潜在的解决方案,因为它利用未标记的数据来提高模型的有效性和鲁棒性。在这项研究中,我们提出应用对比SSL检测异常的心音图(PCG)样本通过学习信号的广义表示。具体来说,我们对各种基于音频的增强进行了广泛的比较评估,并在不同的下游任务中对多个数据集上的训练分类器进行了评估。结果如下:我们通过实验证明,根据其训练分布,在对看不见的数据进行评估时,完全监督模型的有效性可能会降低高达32%,而SSL模型仅损失高达10%,甚至在某些情况下有所改善。结论:对比SSL预训练可以帮助提供强大的分类器,这些分类器可以推广到看不见的OOD数据,而不依赖于医学专家的时间和劳动密集型注释过程。此外,所提出的广泛的评估协议揭示了最有前途的和适当的增强强大的PCG信号处理。重要性:我们为研究人员和从业人员提供了一个路线图,以产生强大的模型PCG分类,除了开发新方法的开源代码库。摘要:Objective: Despite the recent increase in research activity, deep-learning models have not yet been widely accepted in medicine. The shortage of high-quality annotated data often hinders the development of robust and generalizable models, which do not suffer from degraded effectiveness when presented with newly-collected, out-of-distribution (OOD) datasets. Methods: Contrastive Self-Supervised Learning (SSL) offers a potential solution to the scarcity of labeled data as it takes advantage of unlabeled data to increase model effectiveness and robustness. In this research, we propose applying contrastive SSL for detecting abnormalities in phonocardiogram (PCG) samples by learning a generalized representation of the signal. Specifically, we perform an extensive comparative evaluation of a wide range of audio-based augmentations and evaluate trained classifiers on multiple datasets across different downstream tasks. Results: We experimentally demonstrate that, depending on its training distribution, the effectiveness of a fully-supervised model can degrade up to 32% when evaluated on unseen data, while SSL models only lose up to 10% or even improve in some cases. Conclusions: Contrastive SSL pretraining can assist in providing robust classifiers which can generalize to unseen, OOD data, without relying on time- and labor-intensive annotation processes by medical experts. Furthermore, the proposed extensive evaluation protocol sheds light on the most promising and appropriate augmentations for robust PCG signal processing. Significance: We provide researchers and practitioners with a roadmap towards producing robust models for PCG classification, in addition to an open-source codebase for developing novel approaches.【2】 Self-Supervised Learning of Spatial Acoustic Representation with Cross-Channel Signal Reconstruction and Multi-Channel Conformer标题:基于跨声道信号重构和多声道构形的空间声学表示的自监督学习链接:https://arxiv.org/abs/2312.00476作者:Bing Yang,Xiaofei Li摘要:监督学习方法在估计空间声学参数如到达时间差、直达混响比和混响时间方面显示出有效性。然而,他们仍然遭受从模拟到现实的推广问题,由于模拟和现实世界的声学特性之间的不匹配和注释的现实世界的数据的不足。为此,本文提出了一种自监督的方法,充分利用未标记的数据进行空间声学参数估计。首先,一个新的借口任务,即跨通道信号重建(CCSR),被设计为学习一个通用的空间声学表示从未标记的多通道麦克风信号。我们屏蔽一个通道的部分信号,并要求模型重建它们,这使得有可能从未屏蔽的信号中学习空间声学信息,并从另一个麦克风通道中提取源信息。编码器-解码器结构被用来解开这两种信息。通过用小的注释数据集微调预训练的空间编码器,该编码器可以用于估计空间声学参数。其次,采用了一种新的多声道音频一致性(MC-一致性)作为编码器模型架构,这是适合的前提和下游的任务。它经过精心设计,能够捕捉时频域中空间声学的局部和全局特性。五个声学参数估计任务的仿真和真实数据的实验结果表明,所提出的方法的有效性。据我们所知,这是空间声学表示学习和多通道音频信号处理领域的第一个自监督学习方法。摘要:Supervised learning methods have shown effectiveness in estimating spatial acoustic parameters such as time difference of arrival, direct-to-reverberant ratio and reverberation time. However, they still suffer from the simulation-to-reality generalization problem due to the mismatch between simulated and real-world acoustic characteristics and the deficiency of annotated real-world data. To this end, this work proposes a self-supervised method that takes full advantage of unlabeled data for spatial acoustic parameter estimation. First, a new pretext task, i.e. cross-channel signal reconstruction (CCSR), is designed to learn a universal spatial acoustic representation from unlabeled multi-channel microphone signals. We mask partial signals of one channel and ask the model to reconstruct them, which makes it possible to learn spatial acoustic information from unmasked signals and extract source information from the other microphone channel. An encoder-decoder structure is used to disentangle the two kinds of information. By fine-tuning the pre-trained spatial encoder with a small annotated dataset, this encoder can be used to estimate spatial acoustic parameters. Second, a novel multi-channel audio Conformer (MC-Conformer) is adopted as the encoder model architecture, which is suitable for both the pretext and downstream tasks. It is carefully designed to be able to capture the local and global characteristics of spatial acoustics exhibited in the time-frequency domain. Experimental results of five acoustic parameter estimation tasks on both simulated and real-world data show the effectiveness of the proposed method. To the best of our knowledge, this is the first self-supervised learning method in the field of spatial acoustic representation learning and multi-channel audio signal processing.【3】 Sound Terminology Describing Production and Perception of Sonification标题:描述发声的产生和感知的声音术语链接:https://arxiv.org/abs/2312.00091作者:Tim Ziemer备注:16 pages, 0 figures摘要:发音研究本质上是跨学科的。因此,一个适当的文件,和跨学科的话语发音往往是阻碍了所涉及的学科之间的术语差异,即,在发音研究中缺乏一个通用的发音术语。如果没有共同点,来自一个学科的研究人员可能会在理解实现和想象发声的声音感知方面遇到困难,如果发声是由另一个学科的研究人员描述的。为了找到一个共同点,我查阅了跨学科研究和话语的文献,确定了发音中出现的问题,并应用了推荐的解决方案。因此,我建议单独考虑发音的三个方面,即1。声音设计概念,2.)目标和3.)方法,明确哪个学科涉及哪个方面,坚持本学科术语。由于发音的两个要求是a)可再现和b)可解释,我建议使用音频工程术语和心理声学术语记录和讨论每一个发音设计。附录提供了来自这两个学科的健全术语的全面列表,以及相关文献和经常被误解和误用的术语的澄清。摘要:Sonification research is intrinsically interdisciplinary. Consequently, a proper documentation of, and interdisciplinary discourse about a sonification is often hindered by terminology discrepancies between involved disciplines, i.e., the lack of a common sound terminology in sonification research. Without a common ground, a researcher from one discipline may have troubles understanding the implementation and imagining the resulting sound perception of a sonification, if the sonification is described by a researcher from another discipline. To find a common ground, I consulted literature on interdisciplinary research and discourse, identified problems that occur in sonification, and applied the recommended solutions. As a result, I recommend considering three aspects of sonification individually, namely 1.) Sound Design Concept, 2.) Objective and 3.) Method, clarifying which discipline is involved in which aspect, and sticking to this discipline's terminology. As two requirements of sonifications are that they are a) reproducible and b) interpretable, I recommend documenting and discussing every sonification design once using audio engineering terminology, and once using psychoacoustic terminology. The appendix provides comprehensive lists of sound terms from both disciplines, together with relevant literature and a clarification of often misunderstood and misused terms.【4】 Acoustic Cybersecurity: Exploiting Voice-Activated Systems标题:声学网络安全:利用声控系统链接:https://arxiv.org/abs/2312.00039作者:Forrest McKee,David Noever摘要:在这项研究中,我们调查了针对数字语音助理的无声声学攻击的新威胁,这是一个严重的问题,因为到2024年,数字语音助理的流行率将超过全球人口。我们的研究扩展了这些攻击在亚马逊Alexa、Android、iOS和Cortana等各种平台上的可行性,揭示了智能设备中的重大漏洞。确定的12种攻击媒介包括成功操纵智能家居设备和汽车系统、军事通信中的潜在漏洞以及关键基础设施安全方面的挑战。我们定量地表明,攻击成功率徘徊在60%左右,能够从100英尺远的地方远程激活设备。此外,这些攻击威胁到关键基础设施,强调需要多方面的防御策略,结合声学屏蔽,先进的信号处理,机器学习和强大的用户身份验证来减轻这些风险。摘要:In this study, we investigate the emerging threat of inaudible acoustic attacks targeting digital voice assistants, a critical concern given their projected prevalence to exceed the global population by 2024. Our research extends the feasibility of these attacks across various platforms like Amazon's Alexa, Android, iOS, and Cortana, revealing significant vulnerabilities in smart devices. The twelve attack vectors identified include successful manipulation of smart home devices and automotive systems, potential breaches in military communication, and challenges in critical infrastructure security. We quantitatively show that attack success rates hover around 60%, with the ability to activate devices remotely from over 100 feet away. Additionally, these attacks threaten critical infrastructure, emphasizing the need for multifaceted defensive strategies combining acoustic shielding, advanced signal processing, machine learning, and robust user authentication to mitigate these risks.eess.AS音频处理【1】 SPIRE-SIES: A Spontaneous Indian English Speech Corpus标题:SPIRE-SIES:一个自发的印度英语语音语料库链接:https://arxiv.org/abs/2312.00698作者:Abhayjeet Singh,Charu Shah,Rajashri Varadaraj,Sonakshi Chauhan,Prasanta Kumar Ghosh备注:6 pages, 7 plots, 3 tables, Accepted at O-COCOSDA 2023摘要:在本文中,我们提出了一个170.83小时的印度英语自发语音数据集。缺乏印度英语语音数据是开发适合印度语音风格的强大语音系统的主要障碍之一。此外,这种稀缺性甚至更多地表现在自发的言语上。这个语料库是在不同的印度出生地,性别和年龄组的人群来源。传统的自发语音收集策略涉及在采访或对话期间捕获语音。在这项研究中,我们使用图像作为刺激,诱导自发性的讲话。生成并验证23小时的成绩单,其可以用作自发语音ASR基准。通过基于语音活动检测的分割、性别验证和图像语义相关等方法对语料库的质量进行了验证。其使用从Image2Text模型导出的字幕关键词和从耳语ASR生成的转录物导出的高出现词来确定图像刺激和记录的语音之间的关系。摘要:In this paper, we present a 170.83 hour Indian English spontaneous speech dataset. Lack of Indian English speech data is one of the major hindrances in developing robust speech systems which are adapted to the Indian speech style. Moreover this scarcity is even more for spontaneous speech. This corpus is crowd sourced over varied Indian nativities, genders and age groups. Traditional spontaneous speech collection strategies involve capturing of speech during interviewing or conversations. In this study, we use images as stimuli to induce spontaneity in speech. Transcripts for 23 hours is generated and validated which can serve as a spontaneous speech ASR benchmark. Quality of the corpus is validated with voice activity detection based segmentation, gender verification and image semantic correlation. Which determines a relationship between image stimulus and recorded speech using caption keywords derived from Image2Text model and high occurring words derived from whisper ASR generated transcripts.【2】 Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilities标题:声学提示调优:为大型语言模型提供试听功能链接:https://arxiv.org/abs/2312.00249作者:Jinhua Liang,Xubo Liu,Wenwu Wang,Mark D. Plumbley,Huy Phan,Emmanouil Benetos摘要:听觉系统在塑造整个人类感知体验方面起着重要作用。虽然流行的大型语言模型(LLM)和视觉语言模型(VLM)在解决各种视觉和语言理解任务方面表现出了希望,但只有少数可以推广到音频领域,而不会影响其特定领域的能力。在这项工作中,我们介绍了声学提示转向(APT),一个新的适配器扩展LLM和VLM的音频域的软提示。具体而言,APT应用了一个提示感知的音频对齐器来生成软提示,以输入文本和声音为条件,作为语言模型输入。为了缓解音频领域的数据稀缺性,提出了一种多任务学习策略,通过以序列到序列的方式制定不同的音频任务。此外,我们改进了音频语言模型的框架,使用交错音频文本嵌入作为输入序列。这种改进的框架对输入格式施加零约束,因此能够处理更多的理解任务,例如Few-Shot音频分类和音频推理。为了进一步评估音频网络的推理能力,我们提出了自然语言音频推理(NLAR),这是一个新的任务,通过比较和总结来分析两个音频片段。实验表明,APT增强的LLM(即APT-LLM)与专家模型(即,在目标数据集上训练的网络)跨各种任务。最后,我们证明了APT的能力,在扩展冻结VLMs的音频域没有微调,在视听问答任务中取得了可喜的成果。我们的代码和模型权重在https://github.com/JinhuaLiang/APT上发布。摘要:The auditory system plays a substantial role in shaping the overall human perceptual experience. While prevailing large language models (LLMs) and visual language models (VLMs) have shown their promise in solving a wide variety of vision and language understanding tasks, only a few of them can be generalised to the audio domain without compromising their domain-specific capacity. In this work, we introduce Acoustic Prompt Turning (APT), a new adapter extending LLMs and VLMs to the audio domain by soft prompting only. Specifically, APT applies an instruction-aware audio aligner to generate soft prompts, conditioned on both input text and sounds, as language model inputs. To mitigate the data scarcity in the audio domain, a multi-task learning strategy is proposed by formulating diverse audio tasks in a sequence-to-sequence manner. Moreover, we improve the framework of audio language model by using interleaved audio-text embeddings as the input sequence. This improved framework imposes zero constraints on the input format and thus is capable of tackling more understanding tasks, such as few-shot audio classification and audio reasoning. To further evaluate the reasoning ability of audio networks, we propose natural language audio reasoning (NLAR), a new task that analyses across two audio clips by comparison and summarization. Experiments show that APT-enhanced LLMs (namely APT-LLMs) achieve competitive results compared to the expert models (i.e., the networks trained on the targeted datasets) across various tasks. We finally demonstrate the APT's ability in extending frozen VLMs to the audio domain without finetuning, achieving promising results in the audio-visual question and answering task. Our code and model weights are released at https://github.com/JinhuaLiang/APT.【3】 Learning domain-invariant classifiers for infant cry sounds标题:婴儿哭声的学习域不变分类器链接:https://arxiv.org/abs/2312.00231作者:Charles C. Onu,Hemanth K. Sheetha,Arsenii Gorin,Doina Precup摘要:在大多数真实世界的数据集中,域偏移问题仍然是一个有问题的现象,临床音频也不例外。在这项工作中,我们研究的性质域转移的临床数据库中获得的婴儿哭声在不同的地理位置。我们发现,尽管婴儿哭声的音高分布相似,无论出生地如何,但其他特征会给数据带来特殊的偏差。我们探索的方法,以减轻域转移的影响,在一个模型中识别神经损伤的哭声。我们采用计算机视觉的无监督域自适应方法,该方法学习对医院来说是域不变的并且是任务区分的音频表示。我们还提出了一种新的方法,目标噪声注入(TNI),无监督域自适应,既不需要标签,也不需要从目标域的训练数据。我们表现最好的模型将目标准确率显著提高了7.2%,而不会对源域产生负面影响。摘要:The issue of domain shift remains a problematic phenomenon in most real-world datasets and clinical audio is no exception. In this work, we study the nature of domain shift in a clinical database of infant cry sounds acquired across different geographies. We find that though the pitches of infant cries are similarly distributed regardless of the place of birth, other characteristics introduce peculiar biases into the data. We explore methodologies for mitigating the impact of domain shift in a model for identifying neurological injury from cry sounds. We adapt unsupervised domain adaptation methods from computer vision which learn an audio representation that is domain-invariant to hospitals and is task discriminative. We also propose a new approach, target noise injection (TNI), for unsupervised domain adaptation which requires neither labels nor training data from the target domain. Our best-performing model significantly improves target accuracy by 7.2%, without negatively affecting the source domain.【4】 Compression of end-to-end non-autoregressive image-to-speech system for low-resourced devices标题:低资源设备端到端非自回归图像到语音系统的压缩链接:https://arxiv.org/abs/2312.00174作者:Gokul Srinivasagan,Michael Deisher,Munir Georges备注:5 pages, 2 figures, 2 tables, presented at the 15th ITG Conference on Speech Communications, September 2023, Aachen摘要:有视觉障碍的人很难访问支持触摸屏的个人计算设备,如移动电话和笔记本电脑。图像到语音(ITS)系统可以帮助他们缓解这个问题,但其庞大的模型尺寸使得其很难部署在低资源的嵌入式设备上。在本文中,我们的目标是通过开发一种高效的端到端神经架构来克服这一挑战,该架构用于在低资源设备上从微小的显示内容片段生成音频。我们引入了基于视觉变换器的图像编码器,并利用知识蒸馏将模型从610万个参数压缩到246万个参数。人类和自动评估结果表明,我们的方法导致性能下降非常小,可以加快推理时间22%。摘要:People with visual impairments have difficulty accessing touchscreen-enabled personal computing devices like mobile phones and laptops. The image-to-speech (ITS) systems can assist them in mitigating this problem, but their huge model size makes it extremely hard to be deployed on low-resourced embedded devices. In this paper, we aim to overcome this challenge by developing an efficient endto-end neural architecture for generating audio from tiny segments of display content on low-resource devices. We introduced a vision transformers-based image encoder and utilized knowledge distillation to compress the model from 6.1 million to 2.46 million parameters. Human and automatic evaluation results show that our approach leads to a very minimal drop in performance and can speed up the inference time by 22%.【5】 Self-Supervised Learning of Spatial Acoustic Representation with Cross-Channel Signal Reconstruction and Multi-Channel Conformer标题:基于跨声道信号重构和多声道构形的空间声学表示的自监督学习链接:https://arxiv.org/abs/2312.00476作者:Bing Yang,Xiaofei Li摘要:监督学习方法在估计空间声学参数如到达时间差、直达混响比和混响时间方面显示出有效性。然而,他们仍然遭受从模拟到现实的推广问题,由于模拟和现实世界的声学特性之间的不匹配和注释的现实世界的数据的不足。为此,本文提出了一种自监督的方法,充分利用未标记的数据进行空间声学参数估计。首先,一个新的借口任务,即跨通道信号重建(CCSR),被设计为学习一个通用的空间声学表示从未标记的多通道麦克风信号。我们屏蔽一个通道的部分信号,并要求模型重建它们,这使得有可能从未屏蔽的信号中学习空间声学信息,并从另一个麦克风通道中提取源信息。编码器-解码器结构被用来解开这两种信息。通过用小的注释数据集微调预训练的空间编码器,该编码器可以用于估计空间声学参数。其次,采用了一种新的多声道音频一致性(MC-一致性)作为编码器模型架构,这是适合的前提和下游的任务。它经过精心设计,能够捕捉时频域中空间声学的局部和全局特性。五个声学参数估计任务的仿真和真实数据的实验结果表明,所提出的方法的有效性。据我们所知,这是空间声学表示学习和多通道音频信号处理领域的第一个自监督学习方法。摘要:Supervised learning methods have shown effectiveness in estimating spatial acoustic parameters such as time difference of arrival, direct-to-reverberant ratio and reverberation time. However, they still suffer from the simulation-to-reality generalization problem due to the mismatch between simulated and real-world acoustic characteristics and the deficiency of annotated real-world data. To this end, this work proposes a self-supervised method that takes full advantage of unlabeled data for spatial acoustic parameter estimation. First, a new pretext task, i.e. cross-channel signal reconstruction (CCSR), is designed to learn a universal spatial acoustic representation from unlabeled multi-channel microphone signals. We mask partial signals of one channel and ask the model to reconstruct them, which makes it possible to learn spatial acoustic information from unmasked signals and extract source information from the other microphone channel. An encoder-decoder structure is used to disentangle the two kinds of information. By fine-tuning the pre-trained spatial encoder with a small annotated dataset, this encoder can be used to estimate spatial acoustic parameters. Second, a novel multi-channel audio Conformer (MC-Conformer) is adopted as the encoder model architecture, which is suitable for both the pretext and downstream tasks. It is carefully designed to be able to capture the local and global characteristics of spatial acoustics exhibited in the time-frequency domain. Experimental results of five acoustic parameter estimation tasks on both simulated and real-world data show the effectiveness of the proposed method. To the best of our knowledge, this is the first self-supervised learning method in the field of spatial acoustic representation learning and multi-channel audio signal processing.【6】 Sound Terminology Describing Production and Perception of Sonification标题:描述发声的产生和感知的声音术语链接:https://arxiv.org/abs/2312.00091作者:Tim Ziemer备注:16 pages, 0 figures摘要:发音研究本质上是跨学科的。因此,一个适当的文件,和跨学科的话语发音往往是阻碍了所涉及的学科之间的术语差异,即,在发音研究中缺乏一个通用的发音术语。如果没有共同点,来自一个学科的研究人员可能会在理解实现和想象发声的声音感知方面遇到困难,如果发声是由另一个学科的研究人员描述的。为了找到一个共同点,我查阅了跨学科研究和话语的文献,确定了发音中出现的问题,并应用了推荐的解决方案。因此,我建议单独考虑发音的三个方面,即1。声音设计概念,2.)目标和3.)方法,明确哪个学科涉及哪个方面,坚持本学科术语。由于发音的两个要求是a)可再现和b)可解释,我建议使用音频工程术语和心理声学术语记录和讨论每一个发音设计。附录提供了来自这两个学科的健全术语的全面列表,以及相关文献和经常被误解和误用的术语的澄清。摘要:Sonification research is intrinsically interdisciplinary. Consequently, a proper documentation of, and interdisciplinary discourse about a sonification is often hindered by terminology discrepancies between involved disciplines, i.e., the lack of a common sound terminology in sonification research. Without a common ground, a researcher from one discipline may have troubles understanding the implementation and imagining the resulting sound perception of a sonification, if the sonification is described by a researcher from another discipline. To find a common ground, I consulted literature on interdisciplinary research and discourse, identified problems that occur in sonification, and applied the recommended solutions. As a result, I recommend considering three aspects of sonification individually, namely 1.) Sound Design Concept, 2.) Objective and 3.) Method, clarifying which discipline is involved in which aspect, and sticking to this discipline's terminology. As two requirements of sonifications are that they are a) reproducible and b) interpretable, I recommend documenting and discussing every sonification design once using audio engineering terminology, and once using psychoacoustic terminology. The appendix provides comprehensive lists of sound terms from both disciplines, together with relevant literature and a clarification of often misunderstood and misused terms.【7】 Acoustic Cybersecurity: Exploiting Voice-Activated Systems标题:声学网络安全:利用声控系统链接:https://arxiv.org/abs/2312.00039作者:Forrest McKee,David Noever摘要:在这项研究中,我们调查了针对数字语音助理的无声声学攻击的新威胁,这是一个严重的问题,因为到2024年,数字语音助理的流行率将超过全球人口。我们的研究扩展了这些攻击在亚马逊Alexa、Android、iOS和Cortana等各种平台上的可行性,揭示了智能设备中的重大漏洞。确定的12种攻击媒介包括成功操纵智能家居设备和汽车系统、军事通信中的潜在漏洞以及关键基础设施安全方面的挑战。我们定量地表明,攻击成功率徘徊在60%左右,能够从100英尺远的地方远程激活设备。此外,这些攻击威胁到关键基础设施,强调需要多方面的防御策略,结合声学屏蔽,先进的信号处理,机器学习和强大的用户身份验证来减轻这些风险。摘要:In this study, we investigate the emerging threat of inaudible acoustic attacks targeting digital voice assistants, a critical concern given their projected prevalence to exceed the global population by 2024. Our research extends the feasibility of these attacks across various platforms like Amazon's Alexa, Android, iOS, and Cortana, revealing significant vulnerabilities in smart devices. The twelve attack vectors identified include successful manipulation of smart home devices and automotive systems, potential breaches in military communication, and challenges in critical infrastructure security. We quantitatively show that attack success rates hover around 60%, with the ability to activate devices remotely from over 100 feet away. Additionally, these attacks threaten critical infrastructure, emphasizing the need for multifaceted defensive strategies combining acoustic shielding, advanced signal processing, machine learning, and robust user authentication to mitigate these risks.机器翻译由腾讯交互翻译提供,仅供参考