今日论文合集：cs.SD语音14篇，eess.AS音频处理16篇。
cs.SD语音
【1】 Temporally Aligning Long Audio Interviews with Questions: A Case Study  in Multimodal Data Integration标题：在时间上使长时间的音频访谈与提问保持一致：多模式数据整合的案例研究链接：https://arxiv.org/abs/2310.06702
作者：Piyush Singh Pasi,Karthikeya Battepati,Preethi Jyothi,Ganesh Ramakrishnan,Tanmay Mahapatra,Manoj Singh备注：Work Accepted in IJCAI-23- AI and Social Good Track摘要：音频到文本对齐的问题已经在训练期间使用完全监督进行了大量的研究。然而，这通常不是在长音频记录的上下文中，其中被查询的文本在音频文件内不逐字出现。这项工作是与一个名为CARE India的非政府组织合作进行的，该组织收集了居住在印度比哈尔邦农村地区的年轻母亲的长音频健康调查。给定一个从用于指导这些调查的问卷中提取的问题，我们的目标是在长录音中找到问题的位置。这对非洲和亚洲的组织来说非常有价值，否则它们就不得不费力地通过长时间和嘈杂的录音来找到感兴趣的问题（和答案）。我们提出的框架，INDENT，使用一个基于交叉注意力的模型和先验信息的时间排序的句子学习语音嵌入，捕捉底层口语文本的语义。这些学习的嵌入用于在推理时基于文本查询检索相应的音频片段。我们的经验证明了显着的有效性（R-avg约3%的改善），我们的模型比那些使用基于文本的语法。我们还展示了如何嘈杂的ASR，使用国家的最先进的ASR模型为印度语言，产生更好的结果时，用于语音的地方。仅在印地语数据上训练的INDENT能够满足（语义上）共享文本空间支持的所有语言。我们以11种印度语言为例说明了这一点。摘要：The problem of audio-to-text alignment has seen significant amount of research using complete supervision during training. However, this is typically not in the context of long audio recordings wherein the text being queried does not appear verbatim within the audio file. This work is a collaboration with a non-governmental organization called CARE India that collects long audio health surveys from young mothers residing in rural parts of Bihar, India. Given a question drawn from a questionnaire that is used to guide these surveys, we aim to locate where the question is asked within a long audio recording. This is of great value to African and Asian organizations that would otherwise have to painstakingly go through long and noisy audio recordings to locate questions (and answers) of interest. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on the temporal ordering of sentences to learn speech embeddings that capture the semantics of the underlying spoken text. These learnt embeddings are used to retrieve the corresponding audio segment based on text queries at inference time. We empirically demonstrate the significant effectiveness (improvement in R-avg of about 3%) of our model over those obtained using text-based heuristics. We also show how noisy ASR, generated using state-of-the-art ASR models for Indian languages, yields better results when used in place of speech. INDENT, trained only on Hindi data is able to cater to all languages supported by the (semantically) shared text space. We illustrate this empirically on 11 Indic languages.
【2】 AutoCycle-VC: Towards Bottleneck-Independent Zero-Shot Cross-Lingual  Voice Conversion标题：AutoCycle-VC：迈向瓶颈独立的零点跨语言语音转换链接：https://arxiv.org/abs/2310.06546
作者：Haeyun Choi,Jio Gim,Yuho Lee,Youngin Kim,Young-Joo Suh摘要：本文提出了一种简单而鲁棒的具有循环结构和梅尔语谱预处理的zero-shot语音转换系统。以前的作品遭受信息丢失和合成质量差，由于他们依赖于一个精心设计的瓶颈结构。此外，仅依赖于自我重建损失的模型难以再现不同说话者的声音。为了解决这些问题，我们提出了一个考虑目标和源扬声器之间来回转换的周期一致性损失。此外，在说话人编码器训练期间利用堆叠的随机混洗梅尔频谱图和标签平滑方法从语音中提取与时间无关的全局说话人表示，这是zero-shot转换的关键。我们的模型在主观和客观评价方面都优于现有的最先进的结果。此外，它有助于跨语言的语音转换，提高合成语音的质量。摘要：This paper proposes a simple and robust zero-shot voice conversion system with a cycle structure and mel-spectrogram pre-processing. Previous works suffer from information loss and poor synthesis quality due to their reliance on a carefully designed bottleneck structure. Moreover, models relying solely on self-reconstruction loss struggled with reproducing different speakers' voices. To address these issues, we suggested a cycle-consistency loss that considers conversion back and forth between target and source speakers. Additionally, stacked random-shuffled mel-spectrograms and a label smoothing method are utilized during speaker encoder training to extract a time-independent global speaker representation from speech, which is the key to a zero-shot conversion. Our model outperforms existing state-of-the-art results in both subjective and objective evaluations. Furthermore, it facilitates cross-lingual voice conversions and enhances the quality of synthesized speech.
【3】 Topological data analysis of human vowels: Persistent homologies across  representation spaces标题：人类元音的拓扑数据分析：跨表示空间的持久同源链接：https://arxiv.org/abs/2310.06508
作者：Guillem Bonafos,Jean-Marc Freyermuth,Pierre Pudlo,Samuel Tronçon,Arnaud Rey摘要：拓扑数据分析（TDA）已成功地用于信号/图像处理中的各种任务，从可视化到监督/无监督分类。通常，拓扑特征是从持久的同调理论。标准TDA流水线从原始信号数据或其表示开始。然后，它包括使用预先指定的过滤在数据的顶部上建立多尺度拓扑结构，最后计算要进一步利用的拓扑签名。通常使用的拓扑签名是持久图（或其转换）。目前的研究讨论了利用拓扑签名的许多方法的后果，很少选择过滤，但据我们所知，信号表示的选择尚未成为任何研究的主题。本文试图对后一个问题提供一些答案。为此，我们收集了真实的音频数据，并建立了一个比较研究，以评估从三个不同的表示空间提取的拓扑签名的判别信息的质量。每个音频信号被表示为i）使用Taken表示在更高维空间中的观察数据的嵌入，ii）被视为3D环境空间中的表面的频谱图，iii）频谱图的零的集合。从元音音频记录，我们使用拓扑签名用于三个预测问题：说话者性别、元音类型和个体。我们发现，拓扑增强的随机森林提高了袋外误差（OOB），完全基于梅尔频率倒谱系数（MFCC）的最后两个问题。我们的研究结果还表明，从不同的信号表示提取的拓扑信息是互补的，频谱图的零提供了最好的改善性别预测。摘要：Topological Data Analysis (TDA) has been successfully used for various tasks in signal/image processing, from visualization to supervised/unsupervised classification. Often, topological characteristics are obtained from persistent homology theory. The standard TDA pipeline starts from the raw signal data or a representation of it. Then, it consists in building a multiscale topological structure on the top of the data using a pre-specified filtration, and finally to compute the topological signature to be further exploited. The commonly used topological signature is a persistent diagram (or transformations of it). Current research discusses the consequences of the many ways to exploit topological signatures, much less often the choice of the filtration, but to the best of our knowledge, the choice of the representation of a signal has not been the subject of any study yet. This paper attempts to provide some answers on the latter problem. To this end, we collected real audio data and built a comparative study to assess the quality of the discriminant information of the topological signatures extracted from three different representation spaces. Each audio signal is represented as i) an embedding of observed data in a higher dimensional space using Taken's representation, ii) a spectrogram viewed as a surface in a 3D ambient space, iii) the set of spectrogram's zeroes. From vowel audio recordings, we use topological signature for three prediction problems: speaker gender, vowel type, and individual. We show that topologically-augmented random forest improves the Out-of-Bag Error (OOB) over solely based Mel-Frequency Cepstral Coefficients (MFCC) for the last two problems. Our results also suggest that the topological information extracted from different signal representations is complementary, and that spectrogram's zeros offers the best improvement for gender prediction.
【4】 Whispering LLaMA: A Cross-Modal Generative Error Correction Framework  for Speech Recognition标题：Whispering Allama：一种用于语音识别的跨模式生成性纠错框架链接：https://arxiv.org/abs/2310.06434
作者：Srijith Radhakrishnan,Chao-Han Huck Yang,Sumeer Ahmad Khan,Rohit Kumar,Narsis A. Kiani,David Gomez-Cabrero,Jesper N. Tegner备注：Accepted to EMNLP 2023. 10 pages. This work has been done in October 2022 and was submitted to EMNLP 23 once the draft was finalized. GitHub: this https URL摘要：我们介绍了一种新的跨模态融合技术设计的自动语音识别（ASR）中的生成错误校正。我们的方法利用声学信息和外部语言表示来生成准确的语音转录上下文。这标志着一个新的范式生成错误校正的n-最好的假设范围内的一步。与现有的基于排名的重新评分方法不同，我们的方法熟练地使用不同的初始化技术和参数高效算法来提高从预训练的语音和文本模型中获得的ASR性能。通过对不同ASR数据集的评估，我们评估了我们的融合技术的稳定性和可重复性，证明了与n-最佳假设相比，其相对字错误率（WERR）性能提高了37.66%。为了鼓励未来的研究，我们在https://github.com/Srijith-rkr/Whispering-LLaMA上开源了我们的代码和预训练模型。摘要：We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we evaluate the stability and reproducibility of our fusion technique, demonstrating its improved word error rate relative (WERR) performance in comparison to n-best hypotheses by relatively 37.66%. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.
【5】 Proceedings of The first international workshop on eXplainable AI for  the Arts (XAIxArts)标题：首届可解释人工智能艺术国际研讨会(XAIxArt)论文集链接：https://arxiv.org/abs/2310.06428
作者：Nick Bryan-Kinns,Corey Ford,Alan Chamberlain,Steven David Benford,Helen Kennedy,Zijin Li,Wu Qiong,Gus G. Xia,Jeba Rezwana摘要：这是第一个关于可解释人工智能艺术（XAIxArts）的国际研讨会，汇集了HCI，交互设计，人工智能，可解释人工智能（XAI）和数字艺术的研究人员社区，以探索XAI对艺术的作用。  第15届ACM创意与认知会议（C&C 2023）摘要：This first international workshop on explainable AI for the Arts (XAIxArts) brought together a community of researchers in HCI, Interaction Design, AI, explainable AI (XAI), and digital arts to explore the role of XAI for the Arts.  Workshop held at the 15th ACM Conference on Creativity and Cognition (C&C 2023).
【6】 Noisy-ArcMix: Additive Noisy Angular Margin Loss Combined With Mixup  Anomalous Sound Detection标题：Noise-ArcMix：加性有噪角余量损失与混合异常声音检测相结合链接：https://arxiv.org/abs/2310.06364
作者：Soonhyeon Choi,Jung-Woo Choi机构：School of Electrical Engineering, KAIST, Daejeon, South Korea备注：Submitted to ICASSP 2024摘要：无监督异常声音检测（ASD）旨在通过学习正常操作声音的特征并感知其偏差来识别异常声音。最近的方法主要集中在利用正常数据分类的自监督任务上，先进的模型表明，通过表示学习产生紧凑的类内和分离良好的类内分布，确保异常数据的表示空间是很重要的。然而，我们发现，传统的方法往往无法确保足够的类内紧凑性，并表现出样本和它们相应的中心之间的角度差异。在本文中，我们提出了一种训练技术，旨在确保类内紧凑性和增加正常和异常样本之间的角度差距。此外，我们提出了一个架构，提取重要的时间区域的功能，使模型能够学习哪些时间帧应该被强调或抑制。实验结果表明，与DCASE 2020 Challenge Task2数据集上的最新方法相比，该方法在AUC，pAUC和mAUC方面分别提高了0.90%，0.83%和2.16%。摘要：Unsupervised anomalous sound detection (ASD) aims to identify anomalous sounds by learning the features of normal operational sounds and sensing their deviations. Recent approaches have focused on the self-supervised task utilizing the classification of normal data, and advanced models have shown that securing representation space for anomalous data is important through representation learning yielding compact intra-class and well-separated intra-class distributions. However, we show that conventional approaches often fail to ensure sufficient intra-class compactness and exhibit angular disparity between samples and their corresponding centers. In this paper, we propose a training technique aimed at ensuring intra-class compactness and increasing the angle gap between normal and abnormal samples. Furthermore, we present an architecture that extracts features for important temporal regions, enabling the model to learn which time frames should be emphasized or suppressed. Experimental results demonstrate that the proposed method achieves the best performance giving 0.90%, 0.83%, and 2.16% improvement in terms of AUC, pAUC, and mAUC, respectively, compared to the state-of-the-art method on DCASE 2020 Challenge Task2 dataset.
【7】 An experiment on an automated literature survey of data-driven speech  enhancement methods标题：数据驱动语音增强方法的自动化文献综述实验链接：https://arxiv.org/abs/2310.06260
作者：Arthur dos Santos,Jayr Pereira,Rodrigo Nogueira,Bruno Masiero,Shiva Sander-Tavallaey,Elias Zea机构：Universidade Estadual de Campinas, Campinas, Brazil, NeuralMind, ABB Corporate Research, Västerås, Sweden, KTH Royal Institute of Technology, Stockholm, Sweden摘要：一般来说，声学科学出版物的数量不断增加，给传统的文献调查带来了困难。这项工作探讨了使用生成式预训练Transformer（GPT）模型来自动化对116篇关于数据驱动语音增强方法的文章进行的文献调查。其主要目的是评估该模型在提供准确的响应，从参考的基于人类的调查中选择的文件的具体查询的能力和局限性。虽然我们看到了声学自动化文献调查的巨大潜力，但需要改进以更清晰准确地解决技术问题。摘要：The increasing number of scientific publications in acoustics, in general, presents difficulties in conducting traditional literature surveys. This work explores the use of a generative pre-trained transformer (GPT) model to automate a literature survey of 116 articles on data-driven speech enhancement methods. The main objective is to evaluate the capabilities and limitations of the model in providing accurate responses to specific queries about the papers selected from a reference human-based survey. While we see great potential to automate literature surveys in acoustics, improvements are needed to address technical questions more clearly and accurately.
【8】 Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for  Unbiased Question-Answering标题：解决音乐中的数据偏差-AVQA：为无偏见的问题回答精心设计一个平衡的数据集链接：https://arxiv.org/abs/2310.06238
作者：Xiulong Liu,Zhikang Dong,Peng Zhang机构：University of Washington, Stony Brook University摘要：近年来，人们越来越重视音频、视觉和文本模态的交叉，推动了多模态研究的进步。然而，存在于任何模态中的强偏差都可能导致模型忽略其他模态。因此，该模型的能力，有效地理由，这些不同的方式是妥协，阻碍进一步的进步。在本文中，我们仔细审查了原始数据集中的每种问题类型，选择了那些具有明显答案偏差的问题。为了克服这些偏见，我们收集了补充视频和问题，确保没有答案具有突出的偏态分布。特别是，对于二元问题，我们努力确保两个答案几乎均匀地分布在每个问题类别中。因此，我们构建了一个新的数据集，命名为MUSIC-AVQA v2. 0，这是更具挑战性的，我们相信可以更好地促进AVQA任务的进展。此外，我们提出了一个新的基线模型，深入研究视听文本的相互关系。在MUSIC-AVQA v2.0上，该模型超越了所有现有的基准，将准确度提高了2%，创造了新的最先进的性能。摘要：In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, strong bias that exists in any modality can lead to the model neglecting the others. Consequently, the model's ability to effectively reason across these diverse modalities is compromised, impeding further advancement. In this paper, we meticulously review each question type from the original dataset, selecting those with pronounced answer biases. To counter these biases, we gather complementary videos and questions, ensuring that no answers have outstanding skewed distribution. In particular, for binary questions, we strive to ensure that both answers are almost uniformly spread within each question category. As a result, we construct a new dataset, named MUSIC-AVQA v2.0, which is more challenging and we believe could better foster the progress of AVQA task. Furthermore, we present a novel baseline model that delves deeper into the audio-visual-text interrelation. On MUSIC-AVQA v2.0, this model surpasses all the existing benchmarks, improving accuracy by 2% on MUSIC-AVQA v2.0, setting a new state-of-the-art performance.
【9】 On Time Domain Conformer Models for Monaural Speech Separation in Noisy  Reverberant Acoustic Environments标题：混响噪声环境下单声道语音分离的时域构象模型链接：https://arxiv.org/abs/2310.06125
作者：William Ravenscroft,Stefan Goetze,Thomas Hain机构：Speech and Hearing Group, Dept. of Computer Science, The University of Sheffield, Sheffield, UK备注：Accepted at ASRU Workshop 2023摘要：语音分离一直是多说话人技术研究的重要课题。卷积增强Transformers（conformer）在许多语音处理任务中表现良好，但在语音分离中研究不足。最新的最先进的（SOTA）分离模型是时域音频分离网络（TasNets）。一些成功的模型已经利用了双路径（DP）网络，顺序处理本地和全球的信息。时域构造器（TD-构造器）是DP方法的类似物，因为它们也顺序地处理局部和全局上下文，但具有不同的时间复杂度函数。它示出，对于现实的较短的信号长度，构象更有效的特征尺寸控制时。提出了子采样层以进一步提高计算效率。最好的TD-Conformer在WHAMR和WSJ 0 - 2 Mix基准测试中分别实现了14.6 dB和21.2 dB的SISDR改善。摘要：Speech separation remains an important topic for multi-speaker technology researchers. Convolution augmented transformers (conformers) have performed well for many speech processing tasks but have been under-researched for speech separation. Most recent state-of-the-art (SOTA) separation models have been time-domain audio separation networks (TasNets). A number of successful models have made use of dual-path (DP) networks which sequentially process local and global information. Time domain conformers (TD-Conformers) are an analogue of the DP approach in that they also process local and global context sequentially but have a different time complexity function. It is shown that for realistic shorter signal lengths, conformers are more efficient when controlling for feature dimension. Subsampling layers are proposed to further improve computational efficiency. The best TD-Conformer achieves 14.6 dB and 21.2 dB SISDR improvement on the WHAMR and WSJ0-2Mix benchmarks, respectively.
【10】 Leveraging Multilingual Self-Supervised Pretrained Models for  Sequence-to-Sequence End-to-End Spoken Language Understanding标题：利用多语言自我监督预训练模型实现序列到序列端到端的口语理解链接：https://arxiv.org/abs/2310.06103
作者：Pavel Denisov,Ngoc Thang Vu机构：Institute for Natural Language Processing (IMS), University of Stuttgart, Germany备注：IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) 2023摘要：已经提出了许多使用预训练模型的端到端口语理解（E2 E-SLU）方法，但是它们的评估通常缺乏需要预测词汇填充的多语言设置和任务，例如槽填充。在这项工作中，我们提出了一种统一的方法，该方法集成了多语言预训练的语音和文本模型，并以生成方式在四种语言的六个数据集上执行E2 E-SLU，包括词汇填充的预测。我们研究如何改进所提出的方法可以通过预训练广泛使用的语音识别数据，使用几个训练目标。对7000小时的多语言数据进行预训练，使我们最终在两个SLU数据集上以及部分在两个SLU数据集上的表现优于最先进的技术。最后，我们检查了所提出的模型的跨语言能力，并将PortMEDIA语言数据集上最知名的结果提高了近一半，实现了23.65%的概念/值错误率。摘要：A number of methods have been proposed for End-to-End Spoken Language Understanding (E2E-SLU) using pretrained models, however their evaluation often lacks multilingual setup and tasks that require prediction of lexical fillers, such as slot filling. In this work, we propose a unified method that integrates multilingual pretrained speech and text models and performs E2E-SLU on six datasets in four languages in a generative manner, including the prediction of lexical fillers. We investigate how the proposed method can be improved by pretraining on widely available speech recognition data using several training objectives. Pretraining on 7000 hours of multilingual data allows us to outperform the state-of-the-art ultimately on two SLU datasets and partly on two more SLU datasets. Finally, we examine the cross-lingual capabilities of the proposed model and improve on the best known result on the PortMEDIA-Language dataset by almost half, achieving a Concept/Value Error Rate of 23.65%.
【11】 JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and  Nonverbal Expressions标题：JVNV：包含言语内容和非言语表达的日语情感言语语料库链接：https://arxiv.org/abs/2310.06072
作者：Detai Xin,Junfeng Jiang,Shinnosuke Takamichi,Yuki Saito,Akiko Aizawa,Hiroshi Saruwatari机构：Graduate School of Information Science and Technology, The University of Tokyo, Japan, National Institute of Informatics, Japan摘要：我们提出的JVNV，日本的情感语音语料库的口头内容和非口头发声的脚本是由一个大规模的语言模型。现有的情感言语语料库不仅缺乏合适的情感脚本，而且缺乏口语中表达情感的基本表达方式--非言语发声。我们提出了一种自动脚本生成方法，通过使用提示工程向ChatGPT提供具有情感极性的种子词和非言语发声短语来生成情感脚本。我们选择了514个脚本与平衡的音素覆盖从生成的候选脚本的情感信心分数和语言流畅性分数的帮助。我们证明了JVNV的有效性，显示JVNV有更好的音素覆盖率和情感识别比以前的日本情感语音语料库。然后，我们基准JVNV的情感文本到语音合成使用离散代码来表示NV。我们发现，在合成朗读语音和情感语音的表现之间仍然存在差距，在语音中添加NVs会使任务更加困难，这给这项任务带来了新的挑战，并使JVNV成为未来相关工作的宝贵资源。据我们所知，JVNV是第一个使用大型语言模型自动生成脚本的语音语料库。摘要：We present the JVNV, a Japanese emotional speech corpus with verbal content and nonverbal vocalizations whose scripts are generated by a large-scale language model. Existing emotional speech corpora lack not only proper emotional scripts but also nonverbal vocalizations (NVs) that are essential expressions in spoken language to express emotions. We propose an automatic script generation method to produce emotional scripts by providing seed words with sentiment polarity and phrases of nonverbal vocalizations to ChatGPT using prompt engineering. We select 514 scripts with balanced phoneme coverage from the generated candidate scripts with the assistance of emotion confidence scores and language fluency scores. We demonstrate the effectiveness of JVNV by showing that JVNV has better phoneme coverage and emotion recognizability than previous Japanese emotional speech corpora. We then benchmark JVNV on emotional text-to-speech synthesis using discrete codes to represent NVs. We show that there still exists a gap between the performance of synthesizing read-aloud speech and emotional speech, and adding NVs in the speech makes the task even harder, which brings new challenges for this task and makes JVNV a valuable resource for relevant works in the future. To our best knowledge, JVNV is the first speech corpus that generates scripts automatically using large language models.
【12】 Component attention network for multimodal dance improvisation  recognition标题：用于多通道舞蹈即兴表演识别的成分注意网络链接：https://arxiv.org/abs/2310.05938
作者：Jia Fu,Jiarui Tan,Wenjie Yin,Sepideh Pashami,Mårten Björkman机构：RISE Research Institutes of Sweden, KTH Royal Institute of Technology, Stockholm, Sweden备注：Accepted to 25th ACM International Conference on Multimodal Interaction (ICMI 2023)摘要：舞蹈即兴创作是一个活跃的艺术研究课题。即兴舞蹈的运动分析可以是具有挑战性的，由于其独特的动力学。数据驱动的舞蹈动作分析，包括识别和生成，通常限于骨骼数据。然而，可以记录其他模态的数据，例如音频，并使下游任务受益。本文探讨了多模态融合方法的应用和性能的人体运动识别的背景下，舞蹈即兴。我们提出了一个基于注意力的模型，组件注意力网络（CANet），用于三个层面的多模态融合：1）与CANet的特征融合，2）与CANet和图卷积网络（GCN）的模型融合，以及3）与投票策略的后期融合。我们进行了深入的实验，分析在不同的融合方法，并区分关键的时间或组件功能的每一个模态的影响。我们表明，我们提出的模型优于两个基线方法，展示了其潜力，分析即兴舞蹈。摘要：Dance improvisation is an active research topic in the arts. Motion analysis of improvised dance can be challenging due to its unique dynamics. Data-driven dance motion analysis, including recognition and generation, is often limited to skeletal data. However, data of other modalities, such as audio, can be recorded and benefit downstream tasks. This paper explores the application and performance of multimodal fusion methods for human motion recognition in the context of dance improvisation. We propose an attention-based model, component attention network (CANet), for multimodal fusion on three levels: 1) feature fusion with CANet, 2) model fusion with CANet and graph convolutional network (GCN), and 3) late fusion with a voting strategy. We conduct thorough experiments to analyze the impact of each modality in different fusion methods and distinguish critical temporal or component features. We show that our proposed model outperforms the two baseline methods, demonstrating its potential for analyzing improvisation in dance.
【13】 Modeling of Speech-dependent Own Voice Transfer Characteristics for  Hearables with In-ear Microphones标题：具有入耳式麦克风的受听者与语音相关的自身声音传输特性的建模链接：https://arxiv.org/abs/2310.06554
作者：Mattes Ohlenbusch,Christian Rollwage,Simon Doclo备注：18 pages, 11 figures; Extended version of arXiv:2309.08294 (more detailed description of the problem, additional models considered, more systematic evaluation conducted on a different, larger dataset)摘要：可听设备通常包含入耳式麦克风，其可以用于捕获其用户自己的语音。然而，由于耳道阻塞，入耳式麦克风主要记录身体传导的语音，其遭受频带限制效应并且经受低频内容的放大。这些传输特性被假定为基于语音内容和在个体说话者之间变化。期望具有可听麦克风之间的自身语音传输特性的精确模型。这样的模型可以被使用，例如，模拟大量的耳内录音，以训练基于监督学习的算法，旨在补偿自己的语音传输特性。本文提出了一种基于音素识别的语音相关系统辨识模型。使用录音从原型听，建模精度进行评估的技术措施。我们研究的鲁棒性转移特性模型的话语或说话人不匹配。仿真结果表明，使用所提出的语音相关的模型是更好的模拟入耳式录音相比，语音独立的模型。所提出的模型是能够更好地推广到新的话语比自适应过滤为基础的模型。此外，我们发现，谈话者平均模型比个人模型更好地推广到不同的谈话者。摘要：Hearables often contain an in-ear microphone, which may be used to capture the own voice of its user. However, due to ear canal occlusion the in-ear microphone mostly records body-conducted speech, which suffers from band-limitation effects and is subject to amplification of low frequency content. These transfer characteristics are assumed to vary both based on speech content and between individual talkers. It is desirable to have an accurate model of the own voice transfer characteristics between hearable microphones. Such a model can be used, e.g., to simulate a large amount of in-ear recordings to train supervised learning-based algorithms aiming at compensating own voice transfer characteristics. In this paper we propose a speech-dependent system identification model based on phoneme recognition. Using recordings from a prototype hearable, the modeling accuracy is evaluated in terms of technical measures. We investigate robustness of transfer characteristic models to utterance or talker mismatch. Simulation results show that using the proposed speech-dependent model is preferable for simulating in-ear recordings compared to a speech-independent model. The proposed model is able to generalize better to new utterances than an adaptive filtering-based model. Additionally, we find that talker-averaged models generalize better to different talkers than individual models.
【14】 Cross-modal Cognitive Consensus guided Audio-Visual Segmentation标题：跨模式认知共识引导的视听分割链接：https://arxiv.org/abs/2310.06259
作者：Zhaofeng Shi,Qingbo Wu,Hongliang Li,Fanman Meng,Linfeng Xu备注：10 pages摘要：视听分割（AVS）的目的是从视频帧中提取声音对象，这是由一个逐像素的分割掩模表示。这项开创性的工作通过密集的特征级视听交互来执行这项任务，忽略了不同模态之间的维度差距。更具体地说，音频片段只能在每个序列中提供一个\textit{Global}语义标签，但视频帧覆盖了跨不同\textit{Local}区域的多个语义对象。在本文中，我们提出了一个跨模态认知共识引导网络（C3 N），从全球范围内调整视听语义，并逐步注入到局部区域通过注意力机制。首先，跨模态认知共识推理模块（C3 IM）的开发，以提取一个统一的模态标签，通过整合音频/视觉分类的信心和相似性的特定模态的标签嵌入。然后，我们通过认知共识引导注意力模块（CCAM），突出了本地功能对应于感兴趣的对象反馈到视觉骨干作为显式语义级的指导。在AVSBench数据集的单声源分割（S4）设置和多声源分割（MS 3）设置上的大量实验证明了所提出的方法的有效性，其实现了最先进的性能。摘要：Audio-Visual Segmentation (AVS) aims to extract the sounding object from a video frame, which is represented by a pixel-wise segmentation mask. The pioneering work conducts this task through dense feature-level audio-visual interaction, which ignores the dimension gap between different modalities. More specifically, the audio clip could only provide a \textit{Global} semantic label in each sequence, but the video frame covers multiple semantic objects across different \textit{Local} regions. In this paper, we propose a Cross-modal Cognitive Consensus guided Network (C3N) to align the audio-visual semantics from the global dimension and progressively inject them into the local regions via an attention mechanism. Firstly, a Cross-modal Cognitive Consensus Inference Module (C3IM) is developed to extract a unified-modal label by integrating audio/visual classification confidence and similarities of modality-specific label embeddings. Then, we feed the unified-modal label back to the visual backbone as the explicit semantic-level guidance via a Cognitive Consensus guided Attention Module (CCAM), which highlights the local features corresponding to the interested object. Extensive experiments on the Single Sound Source Segmentation (S4) setting and Multiple Sound Source Segmentation (MS3) setting of the AVSBench dataset demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance.
eess.AS音频处理
【1】 Privacy-oriented manipulation of speaker representations标题：面向隐私的说话人表征操作链接：https://arxiv.org/abs/2310.06652
作者：Francisco Teixeira,Alberto Abad,Bhiksha Raj,Isabel Trancoso摘要：说话人嵌入无处不在，其应用范围从说话人识别和日记到语音合成和语音匿名化。这些嵌入的信息量使它们具有多功能性，但也引起了隐私问题。发言人嵌入已被证明包含年龄，性别，健康等信息，发言人可能希望保持隐私，特别是当这些信息不需要的目标任务。在这项工作中，我们提出了一种从扬声器嵌入中移除和操纵私有属性的方法，该方法利用矢量量化变分自动编码器架构，结合对抗分类器和新的互信息损失。我们验证了我们的模型的两个属性，性别和年龄，并与无知和完全知情的攻击者，并与域内和域外数据进行实验。摘要：Speaker embeddings are ubiquitous, with applications ranging from speaker recognition and diarization to speech synthesis and voice anonymisation. The amount of information held by these embeddings lends them versatility, but also raises privacy concerns. Speaker embeddings have been shown to contain information on age, sex, health and more, which speakers may want to keep private, especially when this information is not required for the target task. In this work, we propose a method for removing and manipulating private attributes from speaker embeddings that leverages a Vector-Quantized Variational Autoencoder architecture, combined with an adversarial classifier and a novel mutual information loss. We validate our model on two attributes, sex and age, and perform experiments with ignorant and fully-informed attackers, and with in-domain and out-of-domain data.
【2】 Modeling of Speech-dependent Own Voice Transfer Characteristics for  Hearables with In-ear Microphones标题：具有入耳式麦克风的受听者与语音相关的自身声音传输特性的建模链接：https://arxiv.org/abs/2310.06554
作者：Mattes Ohlenbusch,Christian Rollwage,Simon Doclo备注：18 pages, 11 figures; Extended version of arXiv:2309.08294 (more detailed description of the problem, additional models considered, more systematic evaluation conducted on a different, larger dataset)摘要：可听设备通常包含入耳式麦克风，其可以用于捕获其用户自己的语音。然而，由于耳道阻塞，入耳式麦克风主要记录身体传导的语音，其遭受频带限制效应并且经受低频内容的放大。这些传输特性被假定为基于语音内容和在个体说话者之间变化。期望具有可听麦克风之间的自身语音传输特性的精确模型。这样的模型可以被使用，例如，模拟大量的耳内录音，以训练基于监督学习的算法，旨在补偿自己的语音传输特性。本文提出了一种基于音素识别的语音相关系统辨识模型。使用录音从原型听，建模精度进行评估的技术措施。我们研究的鲁棒性转移特性模型的话语或说话人不匹配。仿真结果表明，使用所提出的语音相关的模型是更好的模拟入耳式录音相比，语音独立的模型。所提出的模型是能够更好地推广到新的话语比自适应过滤为基础的模型。此外，我们发现，谈话者平均模型比个人模型更好地推广到不同的谈话者。摘要：Hearables often contain an in-ear microphone, which may be used to capture the own voice of its user. However, due to ear canal occlusion the in-ear microphone mostly records body-conducted speech, which suffers from band-limitation effects and is subject to amplification of low frequency content. These transfer characteristics are assumed to vary both based on speech content and between individual talkers. It is desirable to have an accurate model of the own voice transfer characteristics between hearable microphones. Such a model can be used, e.g., to simulate a large amount of in-ear recordings to train supervised learning-based algorithms aiming at compensating own voice transfer characteristics. In this paper we propose a speech-dependent system identification model based on phoneme recognition. Using recordings from a prototype hearable, the modeling accuracy is evaluated in terms of technical measures. We investigate robustness of transfer characteristic models to utterance or talker mismatch. Simulation results show that using the proposed speech-dependent model is preferable for simulating in-ear recordings compared to a speech-independent model. The proposed model is able to generalize better to new utterances than an adaptive filtering-based model. Additionally, we find that talker-averaged models generalize better to different talkers than individual models.
【3】 Cross-modal Cognitive Consensus guided Audio-Visual Segmentation标题：跨模式认知共识引导的视听分割链接：https://arxiv.org/abs/2310.06259
作者：Zhaofeng Shi,Qingbo Wu,Hongliang Li,Fanman Meng,Linfeng Xu备注：10 pages摘要：视听分割（AVS）的目的是从视频帧中提取声音对象，这是由一个逐像素的分割掩模表示。这项开创性的工作通过密集的特征级视听交互来执行这项任务，忽略了不同模态之间的维度差距。更具体地说，音频片段只能在每个序列中提供一个\textit{Global}语义标签，但视频帧覆盖了跨不同\textit{Local}区域的多个语义对象。在本文中，我们提出了一个跨模态认知共识引导网络（C3 N），从全球范围内调整视听语义，并逐步注入到局部区域通过注意力机制。首先，跨模态认知共识推理模块（C3 IM）的开发，以提取一个统一的模态标签，通过整合音频/视觉分类的信心和相似性的特定模态的标签嵌入。然后，我们通过认知共识引导注意力模块（CCAM），突出了本地功能对应于感兴趣的对象反馈到视觉骨干作为显式语义级的指导。在AVSBench数据集的单声源分割（S4）设置和多声源分割（MS 3）设置上的大量实验证明了所提出的方法的有效性，其实现了最先进的性能。摘要：Audio-Visual Segmentation (AVS) aims to extract the sounding object from a video frame, which is represented by a pixel-wise segmentation mask. The pioneering work conducts this task through dense feature-level audio-visual interaction, which ignores the dimension gap between different modalities. More specifically, the audio clip could only provide a \textit{Global} semantic label in each sequence, but the video frame covers multiple semantic objects across different \textit{Local} regions. In this paper, we propose a Cross-modal Cognitive Consensus guided Network (C3N) to align the audio-visual semantics from the global dimension and progressively inject them into the local regions via an attention mechanism. Firstly, a Cross-modal Cognitive Consensus Inference Module (C3IM) is developed to extract a unified-modal label by integrating audio/visual classification confidence and similarities of modality-specific label embeddings. Then, we feed the unified-modal label back to the visual backbone as the explicit semantic-level guidance via a Cognitive Consensus guided Attention Module (CCAM), which highlights the local features corresponding to the interested object. Extensive experiments on the Single Sound Source Segmentation (S4) setting and Multiple Sound Source Segmentation (MS3) setting of the AVSBench dataset demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance.
【4】 Discriminative Speech Recognition Rescoring with Pre-trained Language  Models标题：基于预训练语言模型的区分性语音识别研究链接：https://arxiv.org/abs/2310.06248
作者：Prashanth Gurunath Shivakumar,Jari Kolehmainen,Yile Gu,Ankur Gandhe,Ariya Rastrow,Ivan Bulyko备注：ASRU 2023摘要：第二遍重评分是竞争式自动语音识别（ASR）系统的关键组成部分。大型语言模型已经证明了它们能够使用预先训练的信息来更好地对ASR假设进行重新评分。区别性训练，直接优化最小单词错误率（MWER）标准通常可以提高重新评分。在这项研究中，我们提出并探索了几个判别微调方案的预训练LM。我们提出了两种架构的基础上不同的池化策略的输出嵌入和比较基于概率的MWER。我们在歧视性设置中对预先训练的因果和双向LM进行了详细的比较。LibriSpeech上的实验表明，所有MWER训练方案都是有益的，额外的增益高达8.5\% WER。提出的池化变体实现了更低的延迟，同时保留了大多数改进。最后，我们的研究得出结论，双向性更好地利用判别训练。摘要：Second pass rescoring is a critical component of competitive automatic speech recognition (ASR) systems. Large language models have demonstrated their ability in using pre-trained information for better rescoring of ASR hypothesis. Discriminative training, directly optimizing the minimum word-error-rate (MWER) criterion typically improves rescoring. In this study, we propose and explore several discriminative fine-tuning schemes for pre-trained LMs. We propose two architectures based on different pooling strategies of output embeddings and compare with probability based MWER. We conduct detailed comparisons between pre-trained causal and bidirectional LMs in discriminative settings. Experiments on LibriSpeech demonstrate that all MWER training schemes are beneficial, giving additional gains upto 8.5\% WER. Proposed pooling variants achieve lower latency while retaining most improvements. Finally, our study concludes that bidirectionality is better utilized with discriminative training.
【5】 Temporally Aligning Long Audio Interviews with Questions: A Case Study  in Multimodal Data Integration标题：在时间上使长时间的音频访谈与提问保持一致：多模式数据整合的案例研究链接：https://arxiv.org/abs/2310.06702
作者：Piyush Singh Pasi,Karthikeya Battepati,Preethi Jyothi,Ganesh Ramakrishnan,Tanmay Mahapatra,Manoj Singh备注：Work Accepted in IJCAI-23- AI and Social Good Track摘要：音频到文本对齐的问题已经在训练期间使用完全监督进行了大量的研究。然而，这通常不是在长音频记录的上下文中，其中被查询的文本在音频文件内不逐字出现。这项工作是与一个名为CARE India的非政府组织合作进行的，该组织收集了居住在印度比哈尔邦农村地区的年轻母亲的长音频健康调查。给定一个从用于指导这些调查的问卷中提取的问题，我们的目标是在长录音中找到问题的位置。这对非洲和亚洲的组织来说非常有价值，否则它们就不得不费力地通过长时间和嘈杂的录音来找到感兴趣的问题（和答案）。我们提出的框架，INDENT，使用一个基于交叉注意力的模型和先验信息的时间排序的句子学习语音嵌入，捕捉底层口语文本的语义。这些学习的嵌入用于在推理时基于文本查询检索相应的音频片段。我们的经验证明了显着的有效性（R-avg约3%的改善），我们的模型比那些使用基于文本的语法。我们还展示了如何嘈杂的ASR，使用国家的最先进的ASR模型为印度语言，产生更好的结果时，用于语音的地方。仅在印地语数据上训练的INDENT能够满足（语义上）共享文本空间支持的所有语言。我们以11种印度语言为例说明了这一点。摘要：The problem of audio-to-text alignment has seen significant amount of research using complete supervision during training. However, this is typically not in the context of long audio recordings wherein the text being queried does not appear verbatim within the audio file. This work is a collaboration with a non-governmental organization called CARE India that collects long audio health surveys from young mothers residing in rural parts of Bihar, India. Given a question drawn from a questionnaire that is used to guide these surveys, we aim to locate where the question is asked within a long audio recording. This is of great value to African and Asian organizations that would otherwise have to painstakingly go through long and noisy audio recordings to locate questions (and answers) of interest. Our proposed framework, INDENT, uses a cross-attention-based model and prior information on the temporal ordering of sentences to learn speech embeddings that capture the semantics of the underlying spoken text. These learnt embeddings are used to retrieve the corresponding audio segment based on text queries at inference time. We empirically demonstrate the significant effectiveness (improvement in R-avg of about 3%) of our model over those obtained using text-based heuristics. We also show how noisy ASR, generated using state-of-the-art ASR models for Indian languages, yields better results when used in place of speech. INDENT, trained only on Hindi data is able to cater to all languages supported by the (semantically) shared text space. We illustrate this empirically on 11 Indic languages.
【6】 AutoCycle-VC: Towards Bottleneck-Independent Zero-Shot Cross-Lingual  Voice Conversion标题：AutoCycle-VC：迈向瓶颈独立的零点跨语言语音转换链接：https://arxiv.org/abs/2310.06546
作者：Haeyun Choi,Jio Gim,Yuho Lee,Youngin Kim,Young-Joo Suh摘要：本文提出了一种简单而鲁棒的具有循环结构和梅尔语谱预处理的zero-shot语音转换系统。以前的作品遭受信息丢失和合成质量差，由于他们依赖于一个精心设计的瓶颈结构。此外，仅依赖于自我重建损失的模型难以再现不同说话者的声音。为了解决这些问题，我们提出了一个考虑目标和源扬声器之间来回转换的周期一致性损失。此外，在说话人编码器训练期间利用堆叠的随机混洗梅尔频谱图和标签平滑方法从语音中提取与时间无关的全局说话人表示，这是zero-shot转换的关键。我们的模型在主观和客观评价方面都优于现有的最先进的结果。此外，它有助于跨语言的语音转换，提高合成语音的质量。摘要：This paper proposes a simple and robust zero-shot voice conversion system with a cycle structure and mel-spectrogram pre-processing. Previous works suffer from information loss and poor synthesis quality due to their reliance on a carefully designed bottleneck structure. Moreover, models relying solely on self-reconstruction loss struggled with reproducing different speakers' voices. To address these issues, we suggested a cycle-consistency loss that considers conversion back and forth between target and source speakers. Additionally, stacked random-shuffled mel-spectrograms and a label smoothing method are utilized during speaker encoder training to extract a time-independent global speaker representation from speech, which is the key to a zero-shot conversion. Our model outperforms existing state-of-the-art results in both subjective and objective evaluations. Furthermore, it facilitates cross-lingual voice conversions and enhances the quality of synthesized speech.
【7】 Topological data analysis of human vowels: Persistent homologies across  representation spaces标题：人类元音的拓扑数据分析：跨表示空间的持久同源链接：https://arxiv.org/abs/2310.06508
作者：Guillem Bonafos,Jean-Marc Freyermuth,Pierre Pudlo,Samuel Tronçon,Arnaud Rey摘要：拓扑数据分析（TDA）已成功地用于信号/图像处理中的各种任务，从可视化到监督/无监督分类。通常，拓扑特征是从持久的同调理论。标准TDA流水线从原始信号数据或其表示开始。然后，它包括使用预先指定的过滤在数据的顶部上建立多尺度拓扑结构，最后计算要进一步利用的拓扑签名。通常使用的拓扑签名是持久图（或其转换）。目前的研究讨论了利用拓扑签名的许多方法的后果，很少选择过滤，但据我们所知，信号表示的选择尚未成为任何研究的主题。本文试图对后一个问题提供一些答案。为此，我们收集了真实的音频数据，并建立了一个比较研究，以评估从三个不同的表示空间提取的拓扑签名的判别信息的质量。每个音频信号被表示为i）使用Taken表示在更高维空间中的观察数据的嵌入，ii）被视为3D环境空间中的表面的频谱图，iii）频谱图的零的集合。从元音音频记录，我们使用拓扑签名用于三个预测问题：说话者性别、元音类型和个体。我们发现，拓扑增强的随机森林提高了袋外误差（OOB），完全基于梅尔频率倒谱系数（MFCC）的最后两个问题。我们的研究结果还表明，从不同的信号表示提取的拓扑信息是互补的，频谱图的零提供了最好的改善性别预测。摘要：Topological Data Analysis (TDA) has been successfully used for various tasks in signal/image processing, from visualization to supervised/unsupervised classification. Often, topological characteristics are obtained from persistent homology theory. The standard TDA pipeline starts from the raw signal data or a representation of it. Then, it consists in building a multiscale topological structure on the top of the data using a pre-specified filtration, and finally to compute the topological signature to be further exploited. The commonly used topological signature is a persistent diagram (or transformations of it). Current research discusses the consequences of the many ways to exploit topological signatures, much less often the choice of the filtration, but to the best of our knowledge, the choice of the representation of a signal has not been the subject of any study yet. This paper attempts to provide some answers on the latter problem. To this end, we collected real audio data and built a comparative study to assess the quality of the discriminant information of the topological signatures extracted from three different representation spaces. Each audio signal is represented as i) an embedding of observed data in a higher dimensional space using Taken's representation, ii) a spectrogram viewed as a surface in a 3D ambient space, iii) the set of spectrogram's zeroes. From vowel audio recordings, we use topological signature for three prediction problems: speaker gender, vowel type, and individual. We show that topologically-augmented random forest improves the Out-of-Bag Error (OOB) over solely based Mel-Frequency Cepstral Coefficients (MFCC) for the last two problems. Our results also suggest that the topological information extracted from different signal representations is complementary, and that spectrogram's zeros offers the best improvement for gender prediction.
【8】 Whispering LLaMA: A Cross-Modal Generative Error Correction Framework  for Speech Recognition标题：Whispering Allama：一种用于语音识别的跨模式生成性纠错框架链接：https://arxiv.org/abs/2310.06434
作者：Srijith Radhakrishnan,Chao-Han Huck Yang,Sumeer Ahmad Khan,Rohit Kumar,Narsis A. Kiani,David Gomez-Cabrero,Jesper N. Tegner备注：Accepted to EMNLP 2023. 10 pages. This work has been done in October 2022 and was submitted to EMNLP 23 once the draft was finalized. GitHub: this https URL摘要：我们介绍了一种新的跨模态融合技术设计的自动语音识别（ASR）中的生成错误校正。我们的方法利用声学信息和外部语言表示来生成准确的语音转录上下文。这标志着一个新的范式生成错误校正的n-最好的假设范围内的一步。与现有的基于排名的重新评分方法不同，我们的方法熟练地使用不同的初始化技术和参数高效算法来提高从预训练的语音和文本模型中获得的ASR性能。通过对不同ASR数据集的评估，我们评估了我们的融合技术的稳定性和可重复性，证明了与n-最佳假设相比，其相对字错误率（WERR）性能提高了37.66%。为了鼓励未来的研究，我们在https://github.com/Srijith-rkr/Whispering-LLaMA上开源了我们的代码和预训练模型。摘要：We introduce a new cross-modal fusion technique designed for generative error correction in automatic speech recognition (ASR). Our methodology leverages both acoustic information and external linguistic representations to generate accurate speech transcription contexts. This marks a step towards a fresh paradigm in generative error correction within the realm of n-best hypotheses. Unlike the existing ranking-based rescoring methods, our approach adeptly uses distinct initialization techniques and parameter-efficient algorithms to boost ASR performance derived from pre-trained speech and text models. Through evaluation across diverse ASR datasets, we evaluate the stability and reproducibility of our fusion technique, demonstrating its improved word error rate relative (WERR) performance in comparison to n-best hypotheses by relatively 37.66%. To encourage future research, we have made our code and pre-trained models open source at https://github.com/Srijith-rkr/Whispering-LLaMA.
【9】 Proceedings of The first international workshop on eXplainable AI for  the Arts (XAIxArts)标题：首届可解释人工智能艺术国际研讨会(XAIxArt)论文集链接：https://arxiv.org/abs/2310.06428
作者：Nick Bryan-Kinns,Corey Ford,Alan Chamberlain,Steven David Benford,Helen Kennedy,Zijin Li,Wu Qiong,Gus G. Xia,Jeba Rezwana摘要：这是第一个关于可解释人工智能艺术（XAIxArts）的国际研讨会，汇集了HCI，交互设计，人工智能，可解释人工智能（XAI）和数字艺术的研究人员社区，以探索XAI对艺术的作用。  第15届ACM创意与认知会议（C&C 2023）摘要：This first international workshop on explainable AI for the Arts (XAIxArts) brought together a community of researchers in HCI, Interaction Design, AI, explainable AI (XAI), and digital arts to explore the role of XAI for the Arts.  Workshop held at the 15th ACM Conference on Creativity and Cognition (C&C 2023).
【10】 Noisy-ArcMix: Additive Noisy Angular Margin Loss Combined With Mixup  Anomalous Sound Detection标题：Noise-ArcMix：加性有噪角余量损失与混合异常声音检测相结合链接：https://arxiv.org/abs/2310.06364
作者：Soonhyeon Choi,Jung-Woo Choi机构：School of Electrical Engineering, KAIST, Daejeon, South Korea备注：Submitted to ICASSP 2024摘要：无监督异常声音检测（ASD）旨在通过学习正常操作声音的特征并感知其偏差来识别异常声音。最近的方法主要集中在利用正常数据分类的自监督任务上，先进的模型表明，通过表示学习产生紧凑的类内和分离良好的类内分布，确保异常数据的表示空间是很重要的。然而，我们发现，传统的方法往往无法确保足够的类内紧凑性，并表现出样本和它们相应的中心之间的角度差异。在本文中，我们提出了一种训练技术，旨在确保类内紧凑性和增加正常和异常样本之间的角度差距。此外，我们提出了一个架构，提取重要的时间区域的功能，使模型能够学习哪些时间帧应该被强调或抑制。实验结果表明，与DCASE 2020 Challenge Task2数据集上的最新方法相比，该方法在AUC，pAUC和mAUC方面分别提高了0.90%，0.83%和2.16%。摘要：Unsupervised anomalous sound detection (ASD) aims to identify anomalous sounds by learning the features of normal operational sounds and sensing their deviations. Recent approaches have focused on the self-supervised task utilizing the classification of normal data, and advanced models have shown that securing representation space for anomalous data is important through representation learning yielding compact intra-class and well-separated intra-class distributions. However, we show that conventional approaches often fail to ensure sufficient intra-class compactness and exhibit angular disparity between samples and their corresponding centers. In this paper, we propose a training technique aimed at ensuring intra-class compactness and increasing the angle gap between normal and abnormal samples. Furthermore, we present an architecture that extracts features for important temporal regions, enabling the model to learn which time frames should be emphasized or suppressed. Experimental results demonstrate that the proposed method achieves the best performance giving 0.90%, 0.83%, and 2.16% improvement in terms of AUC, pAUC, and mAUC, respectively, compared to the state-of-the-art method on DCASE 2020 Challenge Task2 dataset.
【11】 An experiment on an automated literature survey of data-driven speech  enhancement methods标题：数据驱动语音增强方法的自动化文献综述实验链接：https://arxiv.org/abs/2310.06260
作者：Arthur dos Santos,Jayr Pereira,Rodrigo Nogueira,Bruno Masiero,Shiva Sander-Tavallaey,Elias Zea机构：Universidade Estadual de Campinas, Campinas, Brazil, NeuralMind, ABB Corporate Research, Västerås, Sweden, KTH Royal Institute of Technology, Stockholm, Sweden摘要：一般来说，声学科学出版物的数量不断增加，给传统的文献调查带来了困难。这项工作探讨了使用生成式预训练Transformer（GPT）模型来自动化对116篇关于数据驱动语音增强方法的文章进行的文献调查。其主要目的是评估该模型在提供准确的响应，从参考的基于人类的调查中选择的文件的具体查询的能力和局限性。虽然我们看到了声学自动化文献调查的巨大潜力，但需要改进以更清晰准确地解决技术问题。摘要：The increasing number of scientific publications in acoustics, in general, presents difficulties in conducting traditional literature surveys. This work explores the use of a generative pre-trained transformer (GPT) model to automate a literature survey of 116 articles on data-driven speech enhancement methods. The main objective is to evaluate the capabilities and limitations of the model in providing accurate responses to specific queries about the papers selected from a reference human-based survey. While we see great potential to automate literature surveys in acoustics, improvements are needed to address technical questions more clearly and accurately.
【12】 Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for  Unbiased Question-Answering标题：解决音乐中的数据偏差-AVQA：为无偏见的问题回答精心设计一个平衡的数据集链接：https://arxiv.org/abs/2310.06238
作者：Xiulong Liu,Zhikang Dong,Peng Zhang机构：University of Washington, Stony Brook University摘要：近年来，人们越来越重视音频、视觉和文本模态的交叉，推动了多模态研究的进步。然而，存在于任何模态中的强偏差都可能导致模型忽略其他模态。因此，该模型的能力，有效地理由，这些不同的方式是妥协，阻碍进一步的进步。在本文中，我们仔细审查了原始数据集中的每种问题类型，选择了那些具有明显答案偏差的问题。为了克服这些偏见，我们收集了补充视频和问题，确保没有答案具有突出的偏态分布。特别是，对于二元问题，我们努力确保两个答案几乎均匀地分布在每个问题类别中。因此，我们构建了一个新的数据集，命名为MUSIC-AVQA v2. 0，这是更具挑战性的，我们相信可以更好地促进AVQA任务的进展。此外，我们提出了一个新的基线模型，深入研究视听文本的相互关系。在MUSIC-AVQA v2.0上，该模型超越了所有现有的基准，将准确度提高了2%，创造了新的最先进的性能。摘要：In recent years, there has been a growing emphasis on the intersection of audio, vision, and text modalities, driving forward the advancements in multimodal research. However, strong bias that exists in any modality can lead to the model neglecting the others. Consequently, the model's ability to effectively reason across these diverse modalities is compromised, impeding further advancement. In this paper, we meticulously review each question type from the original dataset, selecting those with pronounced answer biases. To counter these biases, we gather complementary videos and questions, ensuring that no answers have outstanding skewed distribution. In particular, for binary questions, we strive to ensure that both answers are almost uniformly spread within each question category. As a result, we construct a new dataset, named MUSIC-AVQA v2.0, which is more challenging and we believe could better foster the progress of AVQA task. Furthermore, we present a novel baseline model that delves deeper into the audio-visual-text interrelation. On MUSIC-AVQA v2.0, this model surpasses all the existing benchmarks, improving accuracy by 2% on MUSIC-AVQA v2.0, setting a new state-of-the-art performance.
【13】 On Time Domain Conformer Models for Monaural Speech Separation in Noisy  Reverberant Acoustic Environments标题：混响噪声环境下单声道语音分离的时域构象模型链接：https://arxiv.org/abs/2310.06125
作者：William Ravenscroft,Stefan Goetze,Thomas Hain机构：Speech and Hearing Group, Dept. of Computer Science, The University of Sheffield, Sheffield, UK备注：Accepted at ASRU Workshop 2023摘要：语音分离一直是多说话人技术研究的重要课题。卷积增强Transformers（conformer）在许多语音处理任务中表现良好，但在语音分离中研究不足。最新的最先进的（SOTA）分离模型是时域音频分离网络（TasNets）。一些成功的模型已经利用了双路径（DP）网络，顺序处理本地和全球的信息。时域构造器（TD-构造器）是DP方法的类似物，因为它们也顺序地处理局部和全局上下文，但具有不同的时间复杂度函数。它示出，对于现实的较短的信号长度，构象更有效的特征尺寸控制时。提出了子采样层以进一步提高计算效率。最好的TD-Conformer在WHAMR和WSJ 0 - 2 Mix基准测试中分别实现了14.6 dB和21.2 dB的SISDR改善。摘要：Speech separation remains an important topic for multi-speaker technology researchers. Convolution augmented transformers (conformers) have performed well for many speech processing tasks but have been under-researched for speech separation. Most recent state-of-the-art (SOTA) separation models have been time-domain audio separation networks (TasNets). A number of successful models have made use of dual-path (DP) networks which sequentially process local and global information. Time domain conformers (TD-Conformers) are an analogue of the DP approach in that they also process local and global context sequentially but have a different time complexity function. It is shown that for realistic shorter signal lengths, conformers are more efficient when controlling for feature dimension. Subsampling layers are proposed to further improve computational efficiency. The best TD-Conformer achieves 14.6 dB and 21.2 dB SISDR improvement on the WHAMR and WSJ0-2Mix benchmarks, respectively.
【14】 Leveraging Multilingual Self-Supervised Pretrained Models for  Sequence-to-Sequence End-to-End Spoken Language Understanding标题：利用多语言自我监督预训练模型实现序列到序列端到端的口语理解链接：https://arxiv.org/abs/2310.06103
作者：Pavel Denisov,Ngoc Thang Vu机构：Institute for Natural Language Processing (IMS), University of Stuttgart, Germany备注：IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) 2023摘要：已经提出了许多使用预训练模型的端到端口语理解（E2 E-SLU）方法，但是它们的评估通常缺乏需要预测词汇填充的多语言设置和任务，例如槽填充。在这项工作中，我们提出了一种统一的方法，该方法集成了多语言预训练的语音和文本模型，并以生成方式在四种语言的六个数据集上执行E2 E-SLU，包括词汇填充的预测。我们研究如何改进所提出的方法可以通过预训练广泛使用的语音识别数据，使用几个训练目标。对7000小时的多语言数据进行预训练，使我们最终在两个SLU数据集上以及部分在两个SLU数据集上的表现优于最先进的技术。最后，我们检查了所提出的模型的跨语言能力，并将PortMEDIA语言数据集上最知名的结果提高了近一半，实现了23.65%的概念/值错误率。摘要：A number of methods have been proposed for End-to-End Spoken Language Understanding (E2E-SLU) using pretrained models, however their evaluation often lacks multilingual setup and tasks that require prediction of lexical fillers, such as slot filling. In this work, we propose a unified method that integrates multilingual pretrained speech and text models and performs E2E-SLU on six datasets in four languages in a generative manner, including the prediction of lexical fillers. We investigate how the proposed method can be improved by pretraining on widely available speech recognition data using several training objectives. Pretraining on 7000 hours of multilingual data allows us to outperform the state-of-the-art ultimately on two SLU datasets and partly on two more SLU datasets. Finally, we examine the cross-lingual capabilities of the proposed model and improve on the best known result on the PortMEDIA-Language dataset by almost half, achieving a Concept/Value Error Rate of 23.65%.
【15】 JVNV: A Corpus of Japanese Emotional Speech with Verbal Content and  Nonverbal Expressions标题：JVNV：包含言语内容和非言语表达的日语情感言语语料库链接：https://arxiv.org/abs/2310.06072
作者：Detai Xin,Junfeng Jiang,Shinnosuke Takamichi,Yuki Saito,Akiko Aizawa,Hiroshi Saruwatari机构：Graduate School of Information Science and Technology, The University of Tokyo, Japan, National Institute of Informatics, Japan摘要：我们提出的JVNV，日本的情感语音语料库的口头内容和非口头发声的脚本是由一个大规模的语言模型。现有的情感言语语料库不仅缺乏合适的情感脚本，而且缺乏口语中表达情感的基本表达方式--非言语发声。我们提出了一种自动脚本生成方法，通过使用提示工程向ChatGPT提供具有情感极性的种子词和非言语发声短语来生成情感脚本。我们选择了514个脚本与平衡的音素覆盖从生成的候选脚本的情感信心分数和语言流畅性分数的帮助。我们证明了JVNV的有效性，显示JVNV有更好的音素覆盖率和情感识别比以前的日本情感语音语料库。然后，我们基准JVNV的情感文本到语音合成使用离散代码来表示NV。我们发现，在合成朗读语音和情感语音的表现之间仍然存在差距，在语音中添加NVs会使任务更加困难，这给这项任务带来了新的挑战，并使JVNV成为未来相关工作的宝贵资源。据我们所知，JVNV是第一个使用大型语言模型自动生成脚本的语音语料库。摘要：We present the JVNV, a Japanese emotional speech corpus with verbal content and nonverbal vocalizations whose scripts are generated by a large-scale language model. Existing emotional speech corpora lack not only proper emotional scripts but also nonverbal vocalizations (NVs) that are essential expressions in spoken language to express emotions. We propose an automatic script generation method to produce emotional scripts by providing seed words with sentiment polarity and phrases of nonverbal vocalizations to ChatGPT using prompt engineering. We select 514 scripts with balanced phoneme coverage from the generated candidate scripts with the assistance of emotion confidence scores and language fluency scores. We demonstrate the effectiveness of JVNV by showing that JVNV has better phoneme coverage and emotion recognizability than previous Japanese emotional speech corpora. We then benchmark JVNV on emotional text-to-speech synthesis using discrete codes to represent NVs. We show that there still exists a gap between the performance of synthesizing read-aloud speech and emotional speech, and adding NVs in the speech makes the task even harder, which brings new challenges for this task and makes JVNV a valuable resource for relevant works in the future. To our best knowledge, JVNV is the first speech corpus that generates scripts automatically using large language models.
【16】 Component attention network for multimodal dance improvisation  recognition标题：用于多通道舞蹈即兴表演识别的成分注意网络链接：https://arxiv.org/abs/2310.05938
作者：Jia Fu,Jiarui Tan,Wenjie Yin,Sepideh Pashami,Mårten Björkman机构：RISE Research Institutes of Sweden, KTH Royal Institute of Technology, Stockholm, Sweden备注：Accepted to 25th ACM International Conference on Multimodal Interaction (ICMI 2023)摘要：舞蹈即兴创作是一个活跃的艺术研究课题。即兴舞蹈的运动分析可以是具有挑战性的，由于其独特的动力学。数据驱动的舞蹈动作分析，包括识别和生成，通常限于骨骼数据。然而，可以记录其他模态的数据，例如音频，并使下游任务受益。本文探讨了多模态融合方法的应用和性能的人体运动识别的背景下，舞蹈即兴。我们提出了一个基于注意力的模型，组件注意力网络（CANet），用于三个层面的多模态融合：1）与CANet的特征融合，2）与CANet和图卷积网络（GCN）的模型融合，以及3）与投票策略的后期融合。我们进行了深入的实验，分析在不同的融合方法，并区分关键的时间或组件功能的每一个模态的影响。我们表明，我们提出的模型优于两个基线方法，展示了其潜力，分析即兴舞蹈。摘要：Dance improvisation is an active research topic in the arts. Motion analysis of improvised dance can be challenging due to its unique dynamics. Data-driven dance motion analysis, including recognition and generation, is often limited to skeletal data. However, data of other modalities, such as audio, can be recorded and benefit downstream tasks. This paper explores the application and performance of multimodal fusion methods for human motion recognition in the context of dance improvisation. We propose an attention-based model, component attention network (CANet), for multimodal fusion on three levels: 1) feature fusion with CANet, 2) model fusion with CANet and graph convolutional network (GCN), and 3) late fusion with a voting strategy. We conduct thorough experiments to analyze the impact of each modality in different fusion methods and distinguish critical temporal or component features. We show that our proposed model outperforms the two baseline methods, demonstrating its potential for analyzing improvisation in dance.
机器翻译，仅供参考