今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。
cs.SD语音

【1】A vector quantized masked autoencoder for speech emotion recognition标题：一种用于语音情感识别的矢量量化掩码自动编码器链接：http://arxiv.org/pdf/2304.11117v1

作者：Samir Sadok,Simon Leglaive,Renaud Séguier机构：CentraleSup´elec, IETR UMR CNRS , France备注：this https URL摘要：近年来，由于深度学习技术的进步，语音情感识别（SER）取得了显着进展。然而，标记数据的有限可用性仍然是该领域的重大挑战。自我监督学习最近成为解决这一挑战的一个有前途的解决方案。在本文中，我们提出了矢量量化掩蔽自动编码器的语音（VQ-MAE-S），一个自我监督的模型，是微调，以识别语音信号的情感。VQ-MAE-S模型基于在矢量量化变分自动编码器的离散潜在空间中操作的掩蔽自动编码器（MAE）。实验结果表明，所提出的VQ-MAE-S模型，在VoxCeleb 2数据集上进行预训练并在情感语音数据上进行微调，优于在原始频谱图表示和SER中的其他最先进方法上工作的MAE。摘要：Recent years have seen remarkable progress in speech emotion recognition (SER), thanks to advances in deep learning techniques. However, the limited availability of labeled data remains a significant challenge in the field. Self-supervised learning has recently emerged as a promising solution to address this challenge. In this paper, we propose the vector quantized masked autoencoder for speech (VQ-MAE-S), a self-supervised model that is fine-tuned to recognize emotions from speech signals. The VQ-MAE-S model is based on a masked autoencoder (MAE) that operates in the discrete latent space of a vector-quantized variational autoencoder. Experimental results show that the proposed VQ-MAE-S model, pre-trained on the VoxCeleb2 dataset and fine-tuned on emotional speech data, outperforms an MAE working on the raw spectrogram representation and other state-of-the-art methods in SER.

【2】Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialects标题：Spaiche：将最先进的ASR模型扩展到瑞士德语方言链接：http://arxiv.org/pdf/2304.11075v1

作者：Clément Sicard,Kajetan Pyszkowski,Victor Gillioz机构：ETH Zurich, Switzerland备注：5 pages, SwissText conference摘要：NLP最近的突破大大增加了ASR系统在我们日常生活中的存在。然而，对于许多低资源语言，ASR模型仍然需要改进，部分原因是难以获取相关数据。该项目旨在帮助推进瑞士德语方言ASR模型的研究，通过提供关于最先进的ASR模型在最近发布的瑞士德语语音数据集上的性能的见解。我们提出了一种新的损失，考虑到预测和地面实况标签之间的语义距离。通过在瑞士-德国数据集上微调OpenAI的Whisper模型，我们的表现优于当前最先进的结果。摘要：Recent breakthroughs in NLP largely increased the presence of ASR systems in our daily lives. However, for many low-resource languages, ASR models still need to be improved due in part to the difficulty of acquiring pertinent data. This project aims to help advance research in ASR models for Swiss German dialects, by providing insights about the performance of state-of-the-art ASR models on recently published Swiss German speech datasets. We propose a novel loss that takes into account the semantic distance between the predicted and the ground-truth labels. We outperform current state-of-the-art results by fine-tuning OpenAI's Whisper model on Swiss-German datasets.

【3】A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale标题：大规模流媒体ASR的半监督学习技术比较链接：http://arxiv.org/pdf/2304.11053v1

作者：Cal Peyser,Michael Picheny,Kyunghyun Cho,Rohit Prabhavalkar,Ronny Huang,Tara Sainath机构：New York University, Center for Data Science,Google Inc.Journal-ref：2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：不成对的文本和音频注入已经成为在没有大型标记语料库的情况下提高ASR性能的主要方法。然而，关于部署这些方法以改进生产ASR系统的指导很少，这些生产ASR系统在非常大的监督语料库上进行训练，并且具有现实的要求，如受约束的模型大小和CPU预算，流式传输能力以及用于重新评分和下游NLU任务的丰富网格。在这项工作中，我们比较了三种最先进的半监督方法，包括未配对的文本和音频，以及它们在使用联合训练的受控设置中的几种组合。我们发现，在我们的设置，这些方法提供了许多改进，超越了原始WER，包括大量的收益尾字WER，解码器计算在推理过程中，和晶格密度。摘要：Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.

【4】Using Mobile Data and Deep Models to Assess Auditory Verbal Hallucinations标题：使用移动数据和深度模型评估听觉语言幻觉链接：http://arxiv.org/pdf/2304.11049v1

作者：Shayan Mirjafari,Subigya Nepal,Weichen Wang,Andrew T. Campbell摘要：幻觉是在没有真实的外部感官刺激的情况下的明显感知。幻听是一种听到不真实声音的感觉。幻听的一种常见形式是在没有任何说话者的情况下听到声音，这被称为听觉语言幻觉（AVH）。AVH是大脑创造的片段，主要发生在被诊断患有精神疾病的人身上，如双相情感障碍和精神分裂症。评估幻觉声音的效价（即，消极或积极的声音）可以帮助衡量精神疾病的严重程度。我们研究了N=435个人，谁的经验听到的声音，以评估听觉言语幻觉。参与者通过生态瞬时评估报告他们一个月内每天听到四次的声音的效价，问题有四个回答尺度，从“根本没有”到“非常”。我们通过移动应用程序收集这些自我报告作为AVH事件的效价监督。使用该应用程序，参与者还记录音频日记，以口头描述幻觉声音的内容。此外，我们被动地收集移动传感数据作为上下文信号。然后，我们实验如何预测这些语言和上下文线索的音频日记和移动传感数据的听觉口头幻觉事件。最后，使用迁移学习和数据融合技术，我们训练了一个神经网络模型，预测AVH的效价，其性能为54%的top-1和72%的top-2 F1得分。摘要：Hallucination is an apparent perception in the absence of real external sensory stimuli. An auditory hallucination is a perception of hearing sounds that are not real. A common form of auditory hallucination is hearing voices in the absence of any speakers which is known as Auditory Verbal Hallucination (AVH). AVH is fragments of the mind's creation that mostly occur in people diagnosed with mental illnesses such as bipolar disorder and schizophrenia. Assessing the valence of hallucinated voices (i.e., how negative or positive voices are) can help measure the severity of a mental illness. We study N=435 individuals, who experience hearing voices, to assess auditory verbal hallucination. Participants report the valence of voices they hear four times a day for a month through ecological momentary assessments with questions that have four answering scales from not at all'' to extremely''. We collect these self-reports as the valence supervision of AVH events via a mobile application. Using the application, participants also record audio diaries to describe the content of hallucinated voices verbally. In addition, we passively collect mobile sensing data as contextual signals. We then experiment with how predictive these linguistic and contextual cues from the audio diary and mobile sensing data are of an auditory verbal hallucination event. Finally, using transfer learning and data fusion techniques, we train a neural net model that predicts the valance of AVH with a performance of 54 % top-1 and 72 % top-2 F1 score.

【5】Affective social anthropomorphic intelligent system标题：情感社会拟人化智能系统链接：http://arxiv.org/pdf/2304.11046v1

作者：Md. Adyelullahil Mamun,Hasnat Md. Abdullah,Md. Golam Rabiul Alam,Muhammad Mehedi Hassan,Md. Zia Uddin机构：© The Author(s)备注：Multimedia Tools and Applications (2023)摘要：人类的会话风格是由幽默感、个性和语调来衡量的。这些特征对于会话智能虚拟助理来说已经变得至关重要。然而，大多数最先进的智能虚拟助理（IVA）未能解释人类声音的情感语义。本研究提出一个拟人化的智能系统，可以举行适当的人类一样的谈话与情感和个性。还提出了一种语音风格转移方法来映射特定情感的属性。最初，通过转换时间音频波数据来创建频域数据（Mel频谱图），该时间音频波数据包括用于诸如音符、音高、节奏和旋律的音频特征的离散模式。一个并行的CNN变压器编码器被用来预测七种不同的情感状态的声音。语音也被并行地馈送到深度语音，这是一种从声谱图生成文本转录的RNN模型。然后，转录的文本被转移到多域会话代理使用混合技能谈话，基于变换器的检索和生成生成策略，和波束搜索解码，并生成适当的文本响应。该系统学习数据到可以被操纵的潜在空间的可逆映射，并基于先前的Mel频谱图帧生成Mel频谱图帧以进行语音合成和风格转移。最后，使用WaveGlow从频谱图生成波形。我们对个别模型进行的研究结果是吉祥的。此外，与该系统互动的用户提供了积极的反馈，证明了该系统的有效性。摘要：Human conversational styles are measured by the sense of humor, personality, and tone of voice. These characteristics have become essential for conversational intelligent virtual assistants. However, most of the state-of-the-art intelligent virtual assistants (IVAs) are failed to interpret the affective semantics of human voices. This research proposes an anthropomorphic intelligent system that can hold a proper human-like conversation with emotion and personality. A voice style transfer method is also proposed to map the attributes of a specific emotion. Initially, the frequency domain data (Mel-Spectrogram) is created by converting the temporal audio wave data, which comprises discrete patterns for audio features such as notes, pitch, rhythm, and melody. A collateral CNN-Transformer-Encoder is used to predict seven different affective states from voice. The voice is also fed parallelly to the deep-speech, an RNN model that generates the text transcription from the spectrogram. Then the transcripted text is transferred to the multi-domain conversation agent using blended skill talk, transformer-based retrieve-and-generate generation strategy, and beam-search decoding, and an appropriate textual response is generated. The system learns an invertible mapping of data to a latent space that can be manipulated and generates a Mel-spectrogram frame based on previous Mel-spectrogram frames to voice synthesize and style transfer. Finally, the waveform is generated using WaveGlow from the spectrogram. The outcomes of the studies we conducted on individual models were auspicious. Furthermore, users who interacted with the system provided positive feedback, demonstrating the system's effectiveness.

【6】Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms标题：基于机器学习算法的口语情感表达检测链接：http://arxiv.org/pdf/2304.11040v1

作者：Mehrab Hosain,Most. Yeasmin Arafat,Gazi Zahirul Islam,Jia Uddin,Md. Mobarak Hossain,Fatema Alam机构：Department of Computer Science and Engineering, University of Information Technology & Sciences, Dhaka-, Department of Computer Science and Engineering, Southeast University, Dhaka-, Bangladesh备注：Journal Pre-print (15 Pages, 9 Figures, 3 Tables)摘要：人类声音有多种特征，可分为音高、音色、响度和音调。在许多事件中观察到，人类在说话时使用不同的声音品质来表达他们的感情。本研究的主要目的是通过使用几个MATLAB函数，即频谱描述符，周期性和和谐性，识别人类的不同情绪，如愤怒，悲伤，恐惧，中立，厌恶，惊喜和幸福。为了完成这项工作，我们分析了CREMA-D（众包情感多模态演员数据）和TESS（多伦多情感语音集）人类语音数据集。音频文件包含具有各种特性（例如，噪声、快速、缓慢），从而ML（机器学习）模型的效率显著提高。在信号分解过程中采用了EMD（Empirical Mode Decomposition）。然后，通过使用诸如MFCC、GTCC、频谱质心、滚降点、熵、扩展、通量、谐波比、能量、偏斜度、平坦度和音频增量的几种技术来提取特征。数据使用一些著名的ML模型进行训练，即支持向量机，神经网络，集成和KNN。该算法的准确率分别为67.7%，63.3%，61.6%和59.0%的测试数据和77.7%，76.1%，99.1%和61.2%的训练数据。我们已经使用Matlab进行了实验，结果表明，我们的模型是非常突出的和灵活的比现有的类似作品。摘要：There are a variety of features of the human voice that can be classified as pitch, timbre, loudness, and vocal tone. It is observed in numerous incidents that human expresses their feelings using different vocal qualities when they are speaking. The primary objective of this research is to recognize different emotions of human beings such as anger, sadness, fear, neutrality, disgust, pleasant surprise, and happiness by using several MATLAB functions namely, spectral descriptors, periodicity, and harmonicity. To accomplish the work, we analyze the CREMA-D (Crowd-sourced Emotional Multimodal Actors Data) & TESS (Toronto Emotional Speech Set) datasets of human speech. The audio file contains data that have various characteristics (e.g., noisy, speedy, slow) thereby the efficiency of the ML (Machine Learning) models increases significantly. The EMD (Empirical Mode Decomposition) is utilized for the process of signal decomposition. Then, the features are extracted through the use of several techniques such as the MFCC, GTCC, spectral centroid, roll-off point, entropy, spread, flux, harmonic ratio, energy, skewness, flatness, and audio delta. The data is trained using some renowned ML models namely, Support Vector Machine, Neural Network, Ensemble, and KNN. The algorithms show an accuracy of 67.7%, 63.3%, 61.6%, and 59.0% respectively for the test data and 77.7%, 76.1%, 99.1%, and 61.2% for the training data. We have conducted experiments using Matlab and the result shows that our model is very prominent and flexible than existing similar works.

【7】CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval标题：CLAMP：对比语言--跨通道符号音乐信息检索的音乐预习链接：http://arxiv.org/pdf/2304.11029v1

作者：Shangda Wu,Dingyao Yu,Xu Tan,Maosong Sun机构：Central Conservatory of Music, China, Microsoft Research Asia, Tsinghua University, China备注：11 pages, 5 figures, 5 tables摘要：我们介绍CLaMP：对比语言-音乐预训练，它使用音乐编码器和文本编码器学习自然语言和符号音乐之间的跨模态表示，并使用对比损失进行联合训练。为了预训练CLaMP，我们收集了140万个音乐文本对的大型数据集。它采用文本丢弃作为数据增强技术和条形修补来有效地表示音乐数据，从而将序列长度减少到不到10%。此外，我们开发了一个掩蔽的音乐模型预训练目标，以提高音乐编码器的理解音乐的上下文和结构。CLaMP集成了文本信息，可以对符号音乐进行语义搜索和zero-shot分类，超越了以前模型的能力。为了支持语义搜索和音乐分类的评估，我们公开发布了WikiMusicText（WikiMT），这是一个由1010个ABC符号的铅表组成的数据集，每个铅表都附有标题，艺术家，流派和描述。与需要微调的最先进的模型相比，zero-shot CLaMP在面向分数的数据集上表现出相当或更好的性能。摘要：We introduce CLaMP: Contrastive Language-Music Pre-training, which learns cross-modal representations between natural language and symbolic music using a music encoder and a text encoder trained jointly with a contrastive loss. To pre-train CLaMP, we collected a large dataset of 1.4 million music-text pairs. It employed text dropout as a data augmentation technique and bar patching to efficiently represent music data which reduces sequence length to less than 10%. In addition, we developed a masked music model pre-training objective to enhance the music encoder's comprehension of musical context and structure. CLaMP integrates textual information to enable semantic search and zero-shot classification for symbolic music, surpassing the capabilities of previous models. To support the evaluation of semantic search and music classification, we publicly release WikiMusicText (WikiMT), a dataset of 1010 lead sheets in ABC notation, each accompanied by a title, artist, genre, and description. In comparison to state-of-the-art models that require fine-tuning, zero-shot CLaMP demonstrated comparable or superior performance on score-oriented datasets.

【8】OLISIA: a Cascade System for Spoken Dialogue State Tracking标题：OLISIA：一种级联语音对话状态跟踪系统链接：http://arxiv.org/pdf/2304.11073v1

作者：Léo Jacqmin,Lucas Druart,Valentin Vielzeuf,Lina Maria Rojas-Barahona,Yannick Estève,Benoît Favre机构：Lina M. Rojas-Barahona, Benoit Favre, Orange Innovation, France, LIS - Aix-Marseille Université, France, LIA - Avignon Université, France摘要：虽然对话状态跟踪（DST）是一个核心组成部分，口语对话系统，最近的工作主要是处理聊天语料库，忽略口语和书面语之间的差异，在本文中，我们提出了OLISIA，级联系统，集成了自动语音识别（ASR）模型和DST模型。我们在ASR和DST模块中引入了一些调整，以提高口语对话的集成性和鲁棒性。通过这些调整，我们的系统在DSTC11 Track 3中排名第一，这是一个评估口语DST的基准。我们对结果进行了深入分析，发现通过数据增强来规范ASR输出和调整DST输入，以及增加预训练模型的大小，都在减少书面和口头对话之间的性能差异方面发挥了重要作用。摘要：Though Dialogue State Tracking (DST) is a core component of spoken dialogue systems, recent work on this task mostly deals with chat corpora, disregarding the discrepancies between spoken and written language.In this paper, we propose OLISIA, a cascade system which integrates an Automatic Speech Recognition (ASR) model and a DST model. We introduce several adaptations in the ASR and DST modules to improve integration and robustness to spoken conversations.With these adaptations, our system ranked first in DSTC11 Track 3, a benchmark to evaluate spoken DST. We conduct an in-depth analysis of the results and find that normalizing the ASR outputs and adapting the DST inputs through data augmentation, along with increasing the pre-trained models size all play an important role in reducing the performance discrepancy between written and spoken conversations.

【9】Heart Rate Extraction from Abdominal Audio Signals标题：从腹部音频信号中提取心率链接：http://arxiv.org/pdf/2304.11020v1

作者：Jake Stuchbury-Wass,Erika Bondareva,Kayla-Jade Butkow,Sanja Scepanovic,Zoran Radivojevic,Cecilia Mascolo机构：University of Cambridge, UK;, Nokia Bell Labs, Cambridge, UK备注：ICASSP 2023摘要：腹音（ABS）传统上用于评估胃肠道（GI）疾病。然而，该评估需要训练有素的医疗专业人员执行多次腹部听诊会话，这是资源密集型的，并且可能无法提供患者的连续GI健康的准确画面。这已经产生了开发用于连续捕获ABS的可穿戴设备的技术兴趣，这使得能够以降低的成本获得患者的GI状态的更全面的图片。本文旨在评估从ABS监测设备中提取心率（HR）的可行性。直接从这些设备收集HR将使得能够在不需要额外的可穿戴设备的情况下收集生命体征以及GI数据，从而提供进一步的成本效益并提高一般可用性。我们使用了包含104小时ABS音频的数据集，使用电子听诊器从腹部收集，并将心电图作为基础事实。我们的评估首次表明，我们可以成功地从腹部可穿戴设备收集的音频中提取HR。由于从腹部收集的心音受到来自胃肠道和呼吸道的显著噪声的影响，我们利用小波去噪来改进心跳检测。平均HR算法的平均绝对误差为3.4BPM，整个数据集的平均方向误差为-1.2BPM。与基于光电体积描记术的可穿戴HR传感器的比较表明，我们的方法在平均和瞬时心率方面表现出与消费者腕戴式可穿戴设备相当的准确性。摘要：Abdominal sounds (ABS) have been traditionally used for assessing gastrointestinal (GI) disorders. However, the assessment requires a trained medical professional to perform multiple abdominal auscultation sessions, which is resource-intense and may fail to provide an accurate picture of patients' continuous GI wellbeing. This has generated a technological interest in developing wearables for continuous capture of ABS, which enables a fuller picture of patient's GI status to be obtained at reduced cost. This paper seeks to evaluate the feasibility of extracting heart rate (HR) from such ABS monitoring devices. The collection of HR directly from these devices would enable gathering vital signs alongside GI data without the need for additional wearable devices, providing further cost benefits and improving general usability. We utilised a dataset containing 104 hours of ABS audio, collected from the abdomen using an e-stethoscope, and electrocardiogram as ground truth. Our evaluation shows for the first time that we can successfully extract HR from audio collected from a wearable on the abdomen. As heart sounds collected from the abdomen suffer from significant noise from GI and respiratory tracts, we leverage wavelet denoising for improved heart beat detection. The mean absolute error of the algorithm for average HR is 3.4 BPM with mean directional error of -1.2 BPM over the whole dataset. A comparison to photoplethysmography-based wearable HR sensors shows that our approach exhibits comparable accuracy to consumer wrist-worn wearables for average and instantaneous heart rate.

【10】Non-autoregressive End-to-end Approaches for Joint Automatic Speech Recognition and Spoken Language Understanding标题：非自回归端到端联合自动语音识别和口语理解方法链接：http://arxiv.org/pdf/2304.10869v1

作者：Mohan Li,Rama Doddipatla机构：Toshiba Europe Ltd, Cambridge Research Laboratory备注：8 pages, 1 figure, accepted at IEEE SLT2023摘要：本文介绍了使用非自回归（NAR）的方法，联合自动语音识别（ASR）和口语理解（SLU）的任务。建议的NAR系统采用一个构象编码器，应用连接时间分类（CTC）转录的语音话语到原始ASR假设，这是进一步细化与双向编码器表示从Transformers（BERT）的解码器。同时，使用相同的解码器同时预测话语的意图和时隙标签。本研究探索了Mask-CTC和自调节CTC（SC-CTC）方法。SLURP数据集上进行的实验表明，所提出的SC-Mask-CTC NAR系统实现了3.7%和3.2%的绝对增益SLU指标和竞争水平的ASR准确性相比，一致性变压器为基础的自回归（AR）模型。此外，NAR系统实现了比AR基线快6倍的解码速度。摘要：This paper presents the use of non-autoregressive (NAR) approaches for joint automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. The proposed NAR systems employ a Conformer encoder that applies connectionist temporal classification (CTC) to transcribe the speech utterance into raw ASR hypotheses, which are further refined with a bidirectional encoder representations from Transformers (BERT)-like decoder. In the meantime, the intent and slot labels of the utterance are predicted simultaneously using the same decoder. Both Mask-CTC and self-conditioned CTC (SC-CTC) approaches are explored for this study. Experiments conducted on the SLURP dataset show that the proposed SC-Mask-CTC NAR system achieves 3.7% and 3.2% absolute gains in SLU metrics and a competitive level of ASR accuracy, when compared to a Conformer-Transformer based autoregressive (AR) model. Additionally, the NAR systems achieve 6x faster decoding speed than the AR baseline.
eess.AS音频处理

【1】OLISIA: a Cascade System for Spoken Dialogue State Tracking标题：OLISIA：一种级联语音对话状态跟踪系统链接：http://arxiv.org/pdf/2304.11073v1

*与cs.SD语音【8】为同一篇

作者：Léo Jacqmin,Lucas Druart,Valentin Vielzeuf,Lina Maria Rojas-Barahona,Yannick Estève,Benoît Favre机构：Lina M. Rojas-Barahona, Benoit Favre, Orange Innovation, France, LIS - Aix-Marseille Université, France, LIA - Avignon Université, France摘要：虽然对话状态跟踪（DST）是一个核心组成部分，口语对话系统，最近的工作主要是处理聊天语料库，忽略口语和书面语之间的差异，在本文中，我们提出了OLISIA，级联系统，集成了自动语音识别（ASR）模型和DST模型。我们在ASR和DST模块中引入了一些调整，以提高口语对话的集成性和鲁棒性。通过这些调整，我们的系统在DSTC11 Track 3中排名第一，这是一个评估口语DST的基准。我们对结果进行了深入分析，发现通过数据增强来规范ASR输出和调整DST输入，以及增加预训练模型的大小，都在减少书面和口头对话之间的性能差异方面发挥了重要作用。摘要：Though Dialogue State Tracking (DST) is a core component of spoken dialogue systems, recent work on this task mostly deals with chat corpora, disregarding the discrepancies between spoken and written language.In this paper, we propose OLISIA, a cascade system which integrates an Automatic Speech Recognition (ASR) model and a DST model. We introduce several adaptations in the ASR and DST modules to improve integration and robustness to spoken conversations.With these adaptations, our system ranked first in DSTC11 Track 3, a benchmark to evaluate spoken DST. We conduct an in-depth analysis of the results and find that normalizing the ASR outputs and adapting the DST inputs through data augmentation, along with increasing the pre-trained models size all play an important role in reducing the performance discrepancy between written and spoken conversations.

【2】Heart Rate Extraction from Abdominal Audio Signals标题：从腹部音频信号中提取心率链接：http://arxiv.org/pdf/2304.11020v1

*与cs.SD语音【9】为同一篇

作者：Jake Stuchbury-Wass,Erika Bondareva,Kayla-Jade Butkow,Sanja Scepanovic,Zoran Radivojevic,Cecilia Mascolo机构：University of Cambridge, UK;, Nokia Bell Labs, Cambridge, UK备注：ICASSP 2023摘要：腹音（ABS）传统上用于评估胃肠道（GI）疾病。然而，该评估需要训练有素的医疗专业人员执行多次腹部听诊会话，这是资源密集型的，并且可能无法提供患者的连续GI健康的准确画面。这已经产生了开发用于连续捕获ABS的可穿戴设备的技术兴趣，这使得能够以降低的成本获得患者的GI状态的更全面的图片。本文旨在评估从ABS监测设备中提取心率（HR）的可行性。直接从这些设备收集HR将使得能够在不需要额外的可穿戴设备的情况下收集生命体征以及GI数据，从而提供进一步的成本效益并提高一般可用性。我们使用了包含104小时ABS音频的数据集，使用电子听诊器从腹部收集，并将心电图作为基础事实。我们的评估首次表明，我们可以成功地从腹部可穿戴设备收集的音频中提取HR。由于从腹部收集的心音受到来自胃肠道和呼吸道的显著噪声的影响，我们利用小波去噪来改进心跳检测。平均HR算法的平均绝对误差为3.4BPM，整个数据集的平均方向误差为-1.2BPM。与基于光电体积描记术的可穿戴HR传感器的比较表明，我们的方法在平均和瞬时心率方面表现出与消费者腕戴式可穿戴设备相当的准确性。摘要：Abdominal sounds (ABS) have been traditionally used for assessing gastrointestinal (GI) disorders. However, the assessment requires a trained medical professional to perform multiple abdominal auscultation sessions, which is resource-intense and may fail to provide an accurate picture of patients' continuous GI wellbeing. This has generated a technological interest in developing wearables for continuous capture of ABS, which enables a fuller picture of patient's GI status to be obtained at reduced cost. This paper seeks to evaluate the feasibility of extracting heart rate (HR) from such ABS monitoring devices. The collection of HR directly from these devices would enable gathering vital signs alongside GI data without the need for additional wearable devices, providing further cost benefits and improving general usability. We utilised a dataset containing 104 hours of ABS audio, collected from the abdomen using an e-stethoscope, and electrocardiogram as ground truth. Our evaluation shows for the first time that we can successfully extract HR from audio collected from a wearable on the abdomen. As heart sounds collected from the abdomen suffer from significant noise from GI and respiratory tracts, we leverage wavelet denoising for improved heart beat detection. The mean absolute error of the algorithm for average HR is 3.4 BPM with mean directional error of -1.2 BPM over the whole dataset. A comparison to photoplethysmography-based wearable HR sensors shows that our approach exhibits comparable accuracy to consumer wrist-worn wearables for average and instantaneous heart rate.

【3】Non-autoregressive End-to-end Approaches for Joint Automatic Speech Recognition and Spoken Language Understanding标题：非自回归端到端联合自动语音识别和口语理解方法链接：http://arxiv.org/pdf/2304.10869v1

*与cs.SD语音【10】为同一篇

作者：Mohan Li,Rama Doddipatla机构：Toshiba Europe Ltd, Cambridge Research Laboratory备注：8 pages, 1 figure, accepted at IEEE SLT2023摘要：本文介绍了使用非自回归（NAR）的方法，联合自动语音识别（ASR）和口语理解（SLU）的任务。建议的NAR系统采用一个构象编码器，应用连接时间分类（CTC）转录的语音话语到原始ASR假设，这是进一步细化与双向编码器表示从Transformers（BERT）的解码器。同时，使用相同的解码器同时预测话语的意图和时隙标签。本研究探索了Mask-CTC和自调节CTC（SC-CTC）方法。SLURP数据集上进行的实验表明，所提出的SC-Mask-CTC NAR系统实现了3.7%和3.2%的绝对增益SLU指标和竞争水平的ASR准确性相比，一致性变压器为基础的自回归（AR）模型。此外，NAR系统实现了比AR基线快6倍的解码速度。摘要：This paper presents the use of non-autoregressive (NAR) approaches for joint automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. The proposed NAR systems employ a Conformer encoder that applies connectionist temporal classification (CTC) to transcribe the speech utterance into raw ASR hypotheses, which are further refined with a bidirectional encoder representations from Transformers (BERT)-like decoder. In the meantime, the intent and slot labels of the utterance are predicted simultaneously using the same decoder. Both Mask-CTC and self-conditioned CTC (SC-CTC) approaches are explored for this study. Experiments conducted on the SLURP dataset show that the proposed SC-Mask-CTC NAR system achieves 3.7% and 3.2% absolute gains in SLU metrics and a competitive level of ASR accuracy, when compared to a Conformer-Transformer based autoregressive (AR) model. Additionally, the NAR systems achieve 6x faster decoding speed than the AR baseline.

【4】A vector quantized masked autoencoder for speech emotion recognition标题：一种用于语音情感识别的矢量量化掩码自动编码器链接：http://arxiv.org/pdf/2304.11117v1

*与cs.SD语音【1】为同一篇

作者：Samir Sadok,Simon Leglaive,Renaud Séguier机构：CentraleSup´elec, IETR UMR CNRS , France备注：this https URL摘要：近年来，由于深度学习技术的进步，语音情感识别（SER）取得了显着进展。然而，标记数据的有限可用性仍然是该领域的重大挑战。自我监督学习最近成为解决这一挑战的一个有前途的解决方案。在本文中，我们提出了矢量量化掩蔽自动编码器的语音（VQ-MAE-S），一个自我监督的模型，是微调，以识别语音信号的情感。VQ-MAE-S模型基于在矢量量化变分自动编码器的离散潜在空间中操作的掩蔽自动编码器（MAE）。实验结果表明，所提出的VQ-MAE-S模型，在VoxCeleb 2数据集上进行预训练并在情感语音数据上进行微调，优于在原始频谱图表示和SER中的其他最先进方法上工作的MAE。摘要：Recent years have seen remarkable progress in speech emotion recognition (SER), thanks to advances in deep learning techniques. However, the limited availability of labeled data remains a significant challenge in the field. Self-supervised learning has recently emerged as a promising solution to address this challenge. In this paper, we propose the vector quantized masked autoencoder for speech (VQ-MAE-S), a self-supervised model that is fine-tuned to recognize emotions from speech signals. The VQ-MAE-S model is based on a masked autoencoder (MAE) that operates in the discrete latent space of a vector-quantized variational autoencoder. Experimental results show that the proposed VQ-MAE-S model, pre-trained on the VoxCeleb2 dataset and fine-tuned on emotional speech data, outperforms an MAE working on the raw spectrogram representation and other state-of-the-art methods in SER.

【5】Spaiche: Extending State-of-the-Art ASR Models to Swiss German Dialects标题：Spaiche：将最先进的ASR模型扩展到瑞士德语方言链接：http://arxiv.org/pdf/2304.11075v1

*与cs.SD语音【2】为同一篇

作者：Clément Sicard,Kajetan Pyszkowski,Victor Gillioz机构：ETH Zurich, Switzerland备注：5 pages, SwissText conference摘要：NLP最近的突破大大增加了ASR系统在我们日常生活中的存在。然而，对于许多低资源语言，ASR模型仍然需要改进，部分原因是难以获取相关数据。该项目旨在帮助推进瑞士德语方言ASR模型的研究，通过提供关于最先进的ASR模型在最近发布的瑞士德语语音数据集上的性能的见解。我们提出了一种新的损失，考虑到预测和地面实况标签之间的语义距离。通过在瑞士-德国数据集上微调OpenAI的Whisper模型，我们的表现优于当前最先进的结果。摘要：Recent breakthroughs in NLP largely increased the presence of ASR systems in our daily lives. However, for many low-resource languages, ASR models still need to be improved due in part to the difficulty of acquiring pertinent data. This project aims to help advance research in ASR models for Swiss German dialects, by providing insights about the performance of state-of-the-art ASR models on recently published Swiss German speech datasets. We propose a novel loss that takes into account the semantic distance between the predicted and the ground-truth labels. We outperform current state-of-the-art results by fine-tuning OpenAI's Whisper model on Swiss-German datasets.

【6】A Comparison of Semi-Supervised Learning Techniques for Streaming ASR at Scale标题：大规模流媒体ASR的半监督学习技术比较链接：http://arxiv.org/pdf/2304.11053v1

*与cs.SD语音【3】为同一篇

作者：Cal Peyser,Michael Picheny,Kyunghyun Cho,Rohit Prabhavalkar,Ronny Huang,Tara Sainath机构：New York University, Center for Data Science,Google Inc.Journal-ref：2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：不成对的文本和音频注入已经成为在没有大型标记语料库的情况下提高ASR性能的主要方法。然而，关于部署这些方法以改进生产ASR系统的指导很少，这些生产ASR系统在非常大的监督语料库上进行训练，并且具有现实的要求，如受约束的模型大小和CPU预算，流式传输能力以及用于重新评分和下游NLU任务的丰富网格。在这项工作中，我们比较了三种最先进的半监督方法，包括未配对的文本和音频，以及它们在使用联合训练的受控设置中的几种组合。我们发现，在我们的设置，这些方法提供了许多改进，超越了原始WER，包括大量的收益尾字WER，解码器计算在推理过程中，和晶格密度。摘要：Unpaired text and audio injection have emerged as dominant methods for improving ASR performance in the absence of a large labeled corpus. However, little guidance exists on deploying these methods to improve production ASR systems that are trained on very large supervised corpora and with realistic requirements like a constrained model size and CPU budget, streaming capability, and a rich lattice for rescoring and for downstream NLU tasks. In this work, we compare three state-of-the-art semi-supervised methods encompassing both unpaired text and audio as well as several of their combinations in a controlled setting using joint training. We find that in our setting these methods offer many improvements beyond raw WER, including substantial gains in tail-word WER, decoder computation during inference, and lattice density.

【7】Using Mobile Data and Deep Models to Assess Auditory Verbal Hallucinations标题：使用移动数据和深度模型评估听觉语言幻觉链接：http://arxiv.org/pdf/2304.11049v1

*与cs.SD语音【4】为同一篇

作者：Shayan Mirjafari,Subigya Nepal,Weichen Wang,Andrew T. Campbell摘要：幻觉是在没有真实的外部感官刺激的情况下的明显感知。幻听是一种听到不真实声音的感觉。幻听的一种常见形式是在没有任何说话者的情况下听到声音，这被称为听觉语言幻觉（AVH）。AVH是大脑创造的片段，主要发生在被诊断患有精神疾病的人身上，如双相情感障碍和精神分裂症。评估幻觉声音的效价（即，消极或积极的声音）可以帮助衡量精神疾病的严重程度。我们研究了N=435个人，谁的经验听到的声音，以评估听觉言语幻觉。参与者通过生态瞬时评估报告他们一个月内每天听到四次的声音的效价，问题有四个回答尺度，从“根本没有”到“非常”。我们通过移动应用程序收集这些自我报告作为AVH事件的效价监督。使用该应用程序，参与者还记录音频日记，以口头描述幻觉声音的内容。此外，我们被动地收集移动传感数据作为上下文信号。然后，我们实验如何预测这些语言和上下文线索的音频日记和移动传感数据的听觉口头幻觉事件。最后，使用迁移学习和数据融合技术，我们训练了一个神经网络模型，预测AVH的效价，其性能为54%的top-1和72%的top-2 F1得分。摘要：Hallucination is an apparent perception in the absence of real external sensory stimuli. An auditory hallucination is a perception of hearing sounds that are not real. A common form of auditory hallucination is hearing voices in the absence of any speakers which is known as Auditory Verbal Hallucination (AVH). AVH is fragments of the mind's creation that mostly occur in people diagnosed with mental illnesses such as bipolar disorder and schizophrenia. Assessing the valence of hallucinated voices (i.e., how negative or positive voices are) can help measure the severity of a mental illness. We study N=435 individuals, who experience hearing voices, to assess auditory verbal hallucination. Participants report the valence of voices they hear four times a day for a month through ecological momentary assessments with questions that have four answering scales from not at all'' to extremely''. We collect these self-reports as the valence supervision of AVH events via a mobile application. Using the application, participants also record audio diaries to describe the content of hallucinated voices verbally. In addition, we passively collect mobile sensing data as contextual signals. We then experiment with how predictive these linguistic and contextual cues from the audio diary and mobile sensing data are of an auditory verbal hallucination event. Finally, using transfer learning and data fusion techniques, we train a neural net model that predicts the valance of AVH with a performance of 54 % top-1 and 72 % top-2 F1 score.

【8】Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms标题：基于机器学习算法的口语情感表达检测链接：http://arxiv.org/pdf/2304.11040v1

*与cs.SD语音【6】为同一篇

作者：Mehrab Hosain,Most. Yeasmin Arafat,Gazi Zahirul Islam,Jia Uddin,Md. Mobarak Hossain,Fatema Alam机构：Department of Computer Science and Engineering, University of Information Technology & Sciences, Dhaka-, Department of Computer Science and Engineering, Southeast University, Dhaka-, Bangladesh备注：Journal Pre-print (15 Pages, 9 Figures, 3 Tables)摘要：人类声音有多种特征，可分为音高、音色、响度和音调。在许多事件中观察到，人类在说话时使用不同的声音品质来表达他们的感情。本研究的主要目的是通过使用几个MATLAB函数，即频谱描述符，周期性和和谐性，识别人类的不同情绪，如愤怒，悲伤，恐惧，中立，厌恶，惊喜和幸福。为了完成这项工作，我们分析了CREMA-D（众包情感多模态演员数据）和TESS（多伦多情感语音集）人类语音数据集。音频文件包含具有各种特性（例如，噪声、快速、缓慢），从而ML（机器学习）模型的效率显著提高。在信号分解过程中采用了EMD（Empirical Mode Decomposition）。然后，通过使用诸如MFCC、GTCC、频谱质心、滚降点、熵、扩展、通量、谐波比、能量、偏斜度、平坦度和音频增量的几种技术来提取特征。数据使用一些著名的ML模型进行训练，即支持向量机，神经网络，集成和KNN。该算法的准确率分别为67.7%，63.3%，61.6%和59.0%的测试数据和77.7%，76.1%，99.1%和61.2%的训练数据。我们已经使用Matlab进行了实验，结果表明，我们的模型是非常突出的和灵活的比现有的类似作品。摘要：There are a variety of features of the human voice that can be classified as pitch, timbre, loudness, and vocal tone. It is observed in numerous incidents that human expresses their feelings using different vocal qualities when they are speaking. The primary objective of this research is to recognize different emotions of human beings such as anger, sadness, fear, neutrality, disgust, pleasant surprise, and happiness by using several MATLAB functions namely, spectral descriptors, periodicity, and harmonicity. To accomplish the work, we analyze the CREMA-D (Crowd-sourced Emotional Multimodal Actors Data) & TESS (Toronto Emotional Speech Set) datasets of human speech. The audio file contains data that have various characteristics (e.g., noisy, speedy, slow) thereby the efficiency of the ML (Machine Learning) models increases significantly. The EMD (Empirical Mode Decomposition) is utilized for the process of signal decomposition. Then, the features are extracted through the use of several techniques such as the MFCC, GTCC, spectral centroid, roll-off point, entropy, spread, flux, harmonic ratio, energy, skewness, flatness, and audio delta. The data is trained using some renowned ML models namely, Support Vector Machine, Neural Network, Ensemble, and KNN. The algorithms show an accuracy of 67.7%, 63.3%, 61.6%, and 59.0% respectively for the test data and 77.7%, 76.1%, 99.1%, and 61.2% for the training data. We have conducted experiments using Matlab and the result shows that our model is very prominent and flexible than existing similar works.

【9】CLaMP: Contrastive Language-Music Pre-training for Cross-Modal Symbolic Music Information Retrieval标题：CLAMP：对比语言--跨通道符号音乐信息检索的音乐预习链接：http://arxiv.org/pdf/2304.11029v1

*与cs.SD语音【7】为同一篇

作者：Shangda Wu,Dingyao Yu,Xu Tan,Maosong Sun机构：Central Conservatory of Music, China, Microsoft Research Asia, Tsinghua University, China备注：11 pages, 5 figures, 5 tables摘要：我们介绍CLaMP：对比语言-音乐预训练，它使用音乐编码器和文本编码器学习自然语言和符号音乐之间的跨模态表示，并使用对比损失进行联合训练。为了预训练CLaMP，我们收集了140万个音乐文本对的大型数据集。它采用文本丢弃作为数据增强技术和条形修补来有效地表示音乐数据，从而将序列长度减少到不到10%。此外，我们开发了一个掩蔽的音乐模型预训练目标，以提高音乐编码器的理解音乐的上下文和结构。CLaMP集成了文本信息，可以对符号音乐进行语义搜索和zero-shot分类，超越了以前模型的能力。为了支持语义搜索和音乐分类的评估，我们公开发布了WikiMusicText（WikiMT），这是一个由1010个ABC符号的铅表组成的数据集，每个铅表都附有标题，艺术家，流派和描述。与需要微调的最先进的模型相比，zero-shot CLaMP在面向分数的数据集上表现出相当或更好的性能。摘要：We introduce CLaMP: Contrastive Language-Music Pre-training, which learns cross-modal representations between natural language and symbolic music using a music encoder and a text encoder trained jointly with a contrastive loss. To pre-train CLaMP, we collected a large dataset of 1.4 million music-text pairs. It employed text dropout as a data augmentation technique and bar patching to efficiently represent music data which reduces sequence length to less than 10%. In addition, we developed a masked music model pre-training objective to enhance the music encoder's comprehension of musical context and structure. CLaMP integrates textual information to enable semantic search and zero-shot classification for symbolic music, surpassing the capabilities of previous models. To support the evaluation of semantic search and music classification, we publicly release WikiMusicText (WikiMT), a dataset of 1010 lead sheets in ABC notation, each accompanied by a title, artist, genre, and description. In comparison to state-of-the-art models that require fine-tuning, zero-shot CLaMP demonstrated comparable or superior performance on score-oriented datasets.

【10】Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations标题：Chat2Map：多自我对话的高效场景映射链接：http://arxiv.org/pdf/2301.02184v2

作者：Sagnik Majumder,Hao Jiang,Pierre Moulon,Ethan Henderson,Paul Calamia,Kristen Grauman,Vamsi Krishna Ithapu机构：UT Austin, Reality Labs Research, Meta, FAIR备注：Accepted to CVPR 2023摘要：None摘要：Can conversational videos captured from multiple egocentric viewpoints reveal the map of a scene in a cost-efficient way? We seek to answer this question by proposing a new problem: efficiently building the map of a previously unseen 3D environment by exploiting shared information in the egocentric audio-visual observations of participants in a natural conversation. Our hypothesis is that as multiple people ("egos") move in a scene and talk among themselves, they receive rich audio-visual cues that can help uncover the unseen areas of the scene. Given the high cost of continuously processing egocentric visual streams, we further explore how to actively coordinate the sampling of visual information, so as to minimize redundancy and reduce power use. To that end, we present an audio-visual deep reinforcement learning approach that works with our shared scene mapper to selectively turn on the camera to efficiently chart out the space. We evaluate the approach using a state-of-the-art audio-visual simulator for 3D scenes as well as real-world video. Our model outperforms previous state-of-the-art mapping methods, and achieves an excellent cost-accuracy tradeoff. Project: http: vision.cs.utexas.edu projects chat2map.

机器翻译，仅供参考