今日论文合集：cs.SD语音9篇，eess.AS音频处理10篇。
cs.SD语音
【1】 Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong  General Audio Event Taggers标题：Whisper-AT：噪声稳健的自动语音识别器也是强大的通用音频事件标记器链接：https://arxiv.org/abs/2307.03183
作者：Yuan Gong,Sameer Khurana,Leonid Karlinsky,James Glass机构：MIT CSAIL, USA, MIT-IBM Watson AI Lab, USA备注：Accepted at Interspeech 2023. Code at this https URL摘要：在本文中，我们专注于Whisper，一个最近的自动语音识别模型训练了大量的680 k小时标记语音语料库在不同的条件下记录。我们首先展示了一个有趣的发现，虽然Whisper对现实世界的背景声音非常稳健（例如，音乐），其音频表示实际上不是噪声不变的，而是与非语音声音高度相关，这表明Whisper识别以噪声类型为条件的语音。有了这个发现，我们建立了一个统一的音频标记和语音识别模型Whisper-AT冻结的骨干Whisper，并在它上面训练一个轻量级的音频标记模型。只需<1%的额外计算成本，Whisper-AT就可以在单次向前传递中识别音频事件以及语音文本。摘要：In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.
【2】 Track Mix Generation on Music Streaming Services using Transformers标题：使用Transformers在音乐流媒体服务上生成曲目混合链接：https://arxiv.org/abs/2307.03045
作者：Walid Bendada,Théo Bontempelli,Mathieu Morlon,Benjamin Chapus,Thibault Cador,Thomas Bouabça,Guillaume Salha-Galvan备注：RecSys 2023 - Industry track with oral presentation摘要：本文介绍了2022年在音乐流媒体服务Deezer上发布的个性化播放列表生成系统Track Mix。Track Mix自动生成受初始音乐曲目启发的“混合”播放列表，允许用户发现与他们喜爱的内容相似的音乐。为了生成这些混音，我们考虑在来自用户播放列表的数百万个曲目序列上训练的Transformer模型。鉴于近年来越来越受欢迎的Transformers，我们分析的优点，缺点和技术挑战，使用这样一个模型的混合生成的服务，相比，一个更传统的协同过滤方法。自发布以来，Track Mix每天为数百万用户生成播放列表，增强了他们在Deezer上的音乐发现体验。摘要：This paper introduces Track Mix, a personalized playlist generation system released in 2022 on the music streaming service Deezer. Track Mix automatically generates "mix" playlists inspired by initial music tracks, allowing users to discover music similar to their favorite content. To generate these mixes, we consider a Transformer model trained on millions of track sequences from user playlists. In light of the growing popularity of Transformers in recent years, we analyze the advantages, drawbacks, and technical challenges of using such a model for mix generation on the service, compared to a more traditional collaborative filtering approach. Since its release, Track Mix has been generating playlists for millions of users daily, enhancing their music discovery experience on Deezer.
【3】 Transfer Learning for the Efficient Detection of COVID-19 from  Smartphone Audio Data标题：转移学习在智能手机音频数据新冠肺炎检测中的应用链接：https://arxiv.org/abs/2307.02975
作者：Mattia Giovanni Campana,Franca Delmastro,Elena Pagani机构：Institute for Informatics and Telematics of the National Research Council of Italy, (IIT-CNR), Pisa, Italy, Computer Science Department, University of Milano, Milan, Italy备注：None摘要：从智能手机数据的疾病检测代表了移动健康（m-health）系统中的开放式研究挑战。COVID-19及其呼吸道症状是该领域的重要个案研究，其早期检测是对抗疫情的潜在实际工具。该解决方案的有效性主要取决于应用于收集到的数据的AI算法的性能以及它们直接在用户的移动设备上的可能实现。考虑到这些问题，以及有限的可用数据，在本文中，我们提出了3种不同深度学习模型的实验评估，并与手工制作的特征进行了比较，以及在所考虑的场景中迁移学习的两种主要方法：特征提取和微调。具体而言，我们考虑了VGGish、YAMNET和L\text上标{3}-Net（包括12种不同的配置），这些配置是通过在4个不同数据集（总共13，447个样本）上进行的用户独立实验进行评估的。实验结果表明，L3-Net在所有实验条件下都具有明显的优势，在特征提取的精度-召回AUC方面，L3-Net比其他方法提高了12.3%，在模型微调时，L3-Net比其他方法提高了10%.此外，我们注意到，仅微调预训练模型的全连接层通常会导致性能更差，在特征提取方面平均下降6.6%。%，强调需要进一步调查。最后，我们评估的内存足迹的不同模型，其可能的应用在商业移动设备上。摘要：Disease detection from smartphone data represents an open research challenge in mobile health (m-health) systems. COVID-19 and its respiratory symptoms are an important case study in this area and their early detection is a potential real instrument to counteract the pandemic situation. The efficacy of this solution mainly depends on the performances of AI algorithms applied to the collected data and their possible implementation directly on the users' mobile devices. Considering these issues, and the limited amount of available data, in this paper we present the experimental evaluation of 3 different deep learning models, compared also with hand-crafted features, and of two main approaches of transfer learning in the considered scenario: both feature extraction and fine-tuning. Specifically, we considered VGGish, YAMNET, and L\textsuperscript{3}-Net (including 12 different configurations) evaluated through user-independent experiments on 4 different datasets (13,447 samples in total). Results clearly show the advantages of L\textsuperscript{3}-Net in all the experimental settings as it overcomes the other solutions by 12.3\% in terms of Precision-Recall AUC as features extractor, and by 10\% when the model is fine-tuned. Moreover, we note that to fine-tune only the fully-connected layers of the pre-trained models generally leads to worse performances, with an average drop of 6.6\% with respect to feature extraction. %highlighting the need for further investigations. Finally, we evaluate the memory footprints of the different models for their possible applications on commercial mobile devices.
【4】 The Relationship Between Speech Features Changes When You Get Depressed:  Feature Correlations for Improving Speed and Performance of Depression  Detection标题：抑郁时语音特征变化之间的关系：提高抑郁检测速度和性能的特征相关性链接：https://arxiv.org/abs/2307.02892
作者：Fuxiang Tao,Wei Ma,Xuri Ge,Anna Esposito,Alessandro Vinciarelli机构：University of Glasgow, Glasgow G,QQ, UK,  Università degli Studi della Campania “Luigi Vanvitelli”, Caserta, Italy摘要：这项工作表明，抑郁症改变了从语音中提取的特征之间的相关性。此外，它表明，使用这样的见解可以提高抑郁检测器的训练速度和性能的基础上支持向量机和LSTM。这些实验是在Androids Corpus上进行的，这是一个公开的数据集，涉及112名发言者，其中包括58名被专业精神病医生诊断为抑郁症的人。结果表明，在实验中使用的模型，提高了训练速度和性能时，美联储与特征相关矩阵，而不是与特征向量。误差率的相对降低范围在23.1%和26.6%之间，取决于模型。可能的解释是，功能相关矩阵出现在抑郁症的扬声器的情况下，是更多的变量。相应地，这种现象可以被认为是抑郁症的标志。摘要：This work shows that depression changes the correlation between features extracted from speech. Furthermore, it shows that using such an insight can improve the training speed and performance of depression detectors based on SVMs and LSTMs. The experiments were performed over the Androids Corpus, a publicly available dataset involving 112 speakers, including 58 people diagnosed with depression by professional psychiatrists. The results show that the models used in the experiments improve in terms of training speed and performance when fed with feature correlation matrices rather than with feature vectors. The relative reduction of the error rate ranges between 23.1% and 26.6% depending on the model. The probable explanation is that feature correlation matrices appear to be more variable in the case of depressed speakers. Correspondingly, such a phenomenon can be thought of as a depression marker.
【5】 Evaluating raw waveforms with deep learning frameworks for speech  emotion recognition标题：用于语音情感识别的深度学习框架评估原始波形链接：https://arxiv.org/abs/2307.02820
作者：Zeynep Hilal Kilimci,Ulku Bayraktar,Ayhan Kucukmanisa机构：Department of Information Systems Engineering, Kocaeli University, Kocaeli, Turkey, Department of Electronics and Communication Engineering, Kocaeli University, Kocaeli, Turkey, A R T I C L E I N F O备注：14 pages, 6 Figures, 8 Tables摘要：语音情感识别是语音处理领域的一个具有挑战性的课题。因此，特征提取过程对于语音信号的显示和处理具有至关重要的意义。在这项工作中，我们表示了一个模型，该模型将原始音频文件直接馈送到深度神经网络中，而无需任何特征提取阶段，利用六个不同的数据集EMO-DB，RAVDESS，TESS，CREMA，SAVEE和TESS+RAVDESS进行情感识别。为了证明所提出的模型的贡献，传统的特征提取技术，即梅尔尺度谱图，梅尔频率倒谱系系数的性能，与机器学习算法，集成学习方法，深度和混合深度学习技术混合。支持向量机，决策树，朴素贝叶斯，随机森林模型作为机器学习算法进行评估，而多数投票和堆叠方法作为集成学习技术进行评估。此外，卷积神经网络、长短期记忆网络和混合CNN模型被评估为深度学习技术，并与机器学习和集成学习方法进行比较。为了证明所提出的模型的有效性，与国家的最先进的研究进行了比较。基于实验结果，CNN模型优于现有的方法，使用原始音频文件的TESS+RAVDESS数据集的准确率为95.86%，从而确定了新的最先进的。该模型在说话人无关的音频分类问题中，EMO-DB与CNN模型的准确率为90.34%，RAVDESS与CNN模型的准确率为90.42%，TESS与LSTM模型的准确率为99.48%，CREMA与CNN模型的准确率为69.72%，SAVEE与CNN模型的准确率为85.76%。摘要：Speech emotion recognition is a challenging task in speech processing field. For this reason, feature extraction process has a crucial importance to demonstrate and process the speech signals. In this work, we represent a model, which feeds raw audio files directly into the deep neural networks without any feature extraction stage for the recognition of emotions utilizing six different data sets, EMO-DB, RAVDESS, TESS, CREMA, SAVEE, and TESS+RAVDESS. To demonstrate the contribution of proposed model, the performance of traditional feature extraction techniques namely, mel-scale spectogram, mel-frequency cepstral coefficients, are blended with machine learning algorithms, ensemble learning methods, deep and hybrid deep learning techniques. Support vector machine, decision tree, naive Bayes, random forests models are evaluated as machine learning algorithms while majority voting and stacking methods are assessed as ensemble learning techniques. Moreover, convolutional neural networks, long short-term memory networks, and hybrid CNN- LSTM model are evaluated as deep learning techniques and compared with machine learning and ensemble learning methods. To demonstrate the effectiveness of proposed model, the comparison with state-of-the-art studies are carried out. Based on the experiment results, CNN model excels existent approaches with 95.86% of accuracy for TESS+RAVDESS data set using raw audio files, thence determining the new state-of-the-art. The proposed model performs 90.34% of accuracy for EMO-DB with CNN model, 90.42% of accuracy for RAVDESS with CNN model, 99.48% of accuracy for TESS with LSTM model, 69.72% of accuracy for CREMA with CNN model, 85.76% of accuracy for SAVEE with CNN model in speaker-independent audio categorization problems.
【6】 DSARSR: Deep Stacked Auto-encoders Enhanced Robust Speaker Recognition标题：DSARSR：深层堆叠自动编码器增强的稳健说话人识别链接：https://arxiv.org/abs/2307.02751
作者：Zhifeng Wang,Chunyan Zeng,Surong Duan,Hongjie Ouyang,Hongmin Xu机构：and Hongmin XuDepartment of Digital Media Technology,  Central China Normal University,  Hubei University of Technology备注：12 pages, 3 figures摘要：说话者识别是利用说话者的语音段来识别身份的生物计量模态，确定测试说话者是否属于登记的说话者之一。为了提高i-vector框架在跨信道条件下的鲁棒性，并探索将深度学习应用于说话人识别的nova方法，使用Stacked Auto-encoder来获得i-vector的抽象提取，而不是应用PLDA。在预处理和特征提取之后，使用说话人和通道无关的语音进行UBM训练。然后使用UBM来提取登记和测试语音的i向量。与传统的i向量框架不同，该框架使用线性判别分析（LDA）来降低维度并增加说话人子空间之间的区分度，该研究使用堆叠式自动编码器来重建具有较低维度的i向量，并且可以选择不同的分类器来实现最终分类。实验结果表明，该方法取得了更好的性能比国家的最先进的方法。摘要：Speaker recognition is a biometric modality that utilizes the speaker's speech segments to recognize the identity, determining whether the test speaker belongs to one of the enrolled speakers. In order to improve the robustness of the i-vector framework on cross-channel conditions and explore the nova method for applying deep learning to speaker recognition, the Stacked Auto-encoders are used to get the abstract extraction of the i-vector instead of applying PLDA. After pre-processing and feature extraction, the speaker and channel-independent speeches are employed for UBM training. The UBM is then used to extract the i-vector of the enrollment and test speech. Unlike the traditional i-vector framework, which uses linear discriminant analysis (LDA) to reduce dimension and increase the discrimination between speaker subspaces, this research use stacked auto-encoders to reconstruct the i-vector with lower dimension and different classifiers can be chosen to achieve final classification. The experimental results show that the proposed method achieves better performance than the state-of-the-art method.
【7】 On-Device Constrained Self-Supervised Speech Representation Learning for  Keyword Spotting via Knowledge Distillation标题：基于知识提取的设备约束自监督语音表征学习关键字检测链接：https://arxiv.org/abs/2307.02720
作者：Gene-Ping Yang,Yue Gu,Qingming Tang,Dongsu Du,Yuzong Liu机构：Centre for Speech Technology Research, University of Edinburgh, Alexa Perceptual Technologies, Amazon, Zoom Video Communications, Inc.备注：Accepted to Interspeech 2023摘要：大型自监督模型是有效的特征提取器，但在设备预算限制和有偏见的数据集收集下，它们的应用具有挑战性，特别是在关键字识别方面。为了解决这个问题，我们提出了一种基于知识蒸馏的自监督语音表示学习（S3 RL）架构，用于设备上的关键字识别。我们的方法使用了一个教师-学生的框架，从一个更大的，更复杂的模型，以一个较小的，重量轻的模型，使用双视图互相关蒸馏和教师的学习目标的码本知识转移。我们使用16.6k小时的内部数据集评估了我们的模型在Alexa关键字定位检测任务上的性能。我们的技术在正常和嘈杂的条件下表现出卓越的性能，证明了知识蒸馏方法在构建关键字定位任务的自监督模型时的有效性，同时在设备上的资源限制下工作。摘要：Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we proposed a knowledge distillation-based self-supervised speech representation learning (S3RL) architecture for on-device keyword spotting. Our approach used a teacher-student framework to transfer knowledge from a larger, more complex model to a smaller, light-weight model using dual-view cross-correlation distillation and the teacher's codebook as learning objectives. We evaluated our model's performance on an Alexa keyword spotting detection task using a 16.6k-hour in-house dataset. Our technique showed exceptional performance in normal and noisy conditions, demonstrating the efficacy of knowledge distillation methods in constructing self-supervised models for keyword spotting tasks while working within on-device resource constraints.
【8】 Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation  and Recognition标题：视听端到端多声道语音分离、去混响和识别链接：https://arxiv.org/abs/2307.02909
作者：Guinan Li,Jiajun Deng,Mengzhe Geng,Zengrui Jin,Tianzi Wang,Shujie Hu,Mingyu Cui,Helen Meng,Xunying Liu备注：IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：鸡尾酒会语音包含重叠的扬声器，噪声和混响的准确识别仍然是一个极具挑战性的任务。基于视觉模态对声信号失真的不变性，提出了一种将视觉信息充分融合到系统各部分的视听多通道语音分离、去混响和识别方法。视频输入的有效性在基于掩码的MVDR语音分离、基于DNN-WPE或频谱映射（SpecM）的语音去混响前端和Conformer ASR后端中得到一致证明。视听集成前端架构执行语音分离和去混响在一个流水线或联合的方式，通过基于掩码的WPD的研究。语音增强前端和ASR后端组件之间的误差成本失配通过单独使用ASR成本函数或其具有语音增强损失的内插的端到端联合微调来最小化。实验上进行的混合重叠和混响的语音数据构建使用模拟或重放的牛津LRS 2数据集。建议的视听多通道语音分离，去混响和识别系统始终优于9.1%和6.2%的绝对（41.7%和36.0%的相对）字错误率（WER）减少可比的音频基线。在PESQ、STOI和SRMR评分上也获得了一致的语音增强改善。摘要：Accurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in mask-based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improvements were also obtained on PESQ, STOI and SRMR scores.
【9】 Exploring Multimodal Approaches for Alzheimer's Disease Detection Using  Patient Speech Transcript and Audio Data标题：利用患者语音记录和音频数据探索检测阿尔茨海默病的多模式方法链接：https://arxiv.org/abs/2307.02514
作者：Hongmin Cai,Xiaoke Huang,Zhengliang Liu,Wenxiong Liao,Haixing Dai,Zihao Wu,Dajiang Zhu,Hui Ren,Quanzheng Li,Tianming Liu,Xiang Li机构：South China University of Technology, Guangzhou, Guangdong, China, University of Georgia, Athens, Georgia, USA, University of Texas at Arlington, Arlington, Texas, USA, Massachusetts General Hospital, Boston, Massachusetts, USA摘要：阿尔茨海默病（AD）是严重影响患者健康的常见形式的痴呆。由于AD损害患者的语言理解和表达能力，因此AD患者的言语可以作为这种疾病的指标。本研究调查了各种方法检测AD患者的语音和成绩单数据从痴呆银行皮特数据库。所提出的方法涉及预训练的语言模型和图形神经网络（GNN），该网络从语音转录构建图形，并使用GNN提取特征用于AD检测。数据增强技术，包括同义词替换，基于GPT的增强器，等等，被用来解决小的数据集大小。同时介绍了音频数据，采用WavLM模型提取音频特征。然后使用各种方法将这些特征与文本特征融合。最后，一个对比学习的方法，试图通过转换语音转录回音频，并使用它与原始音频的对比学习。我们对上述方法进行了深入的实验和分析。我们的研究结果揭示了使用语音和音频数据进行AD检测的挑战和潜在解决方案。摘要：Alzheimer's disease (AD) is a common form of dementia that severely impacts patient health. As AD impairs the patient's language understanding and expression ability, the speech of AD patients can serve as an indicator of this disease. This study investigates various methods for detecting AD using patients' speech and transcripts data from the DementiaBank Pitt database. The proposed approach involves pre-trained language models and Graph Neural Network (GNN) that constructs a graph from the speech transcript, and extracts features using GNN for AD detection. Data augmentation techniques, including synonym replacement, GPT-based augmenter, and so on, were used to address the small dataset size. Audio data was also introduced, and WavLM model was used to extract audio features. These features were then fused with text features using various methods. Finally, a contrastive learning approach was attempted by converting speech transcripts back to audio and using it for contrastive learning with the original audio. We conducted intensive experiments and analysis on the above methods. Our findings shed light on the challenges and potential solutions in AD detection using speech and audio data.
eess.AS音频处理
【1】 Recovering implicit pitch contours from formants in whispered speech标题：从耳语中的共振峰恢复隐式基音轮廓链接：https://arxiv.org/abs/2307.03168
作者：Pablo Pérez Zarazaga,Zofia Malisz机构：TMH, KTH Royal Institute of Technology, Stockholm备注：5 pages, 3 figures, 2 tables, Accepted at ICPhS 2023摘要：耳语语音的特征在于导致基频缺失的类噪声激励。考虑到诸如语调的韵律现象通过f0变化被感知，耳语韵律的感知相对困难。与此同时，研究表明，说话者在耳语时确实试图产生语调，并且韵律变化正在被传递，这表明语调在耳语共振峰结构中“存活”。在本文中，我们的目标是估计共振峰轮廓与耳语中的“隐式”音高轮廓相关的方式，使用机器学习模型。我们提出了一个两步法：使用并行语料库，我们首先使用去噪自动编码器将低声共振峰转换成它们的发声等价物。然后，我们分析共振峰轮廓预测发声音高轮廓的变化。我们观察到，我们的方法是有效的，在建立低声和发声共振峰之间的关系，并在揭示隐含的音高轮廓耳语。摘要：Whispered speech is characterised by a noise-like excitation that results in the lack of fundamental frequency. Considering that prosodic phenomena such as intonation are perceived through f0 variation, the perception of whispered prosody is relatively difficult. At the same time, studies have shown that speakers do attempt to produce intonation when whispering and that prosodic variability is being transmitted, suggesting that intonation "survives" in whispered formant structure. In this paper, we aim to estimate the way in which formant contours correlate with an "implicit" pitch contour in whisper, using a machine learning model. We propose a two-step method: using a parallel corpus, we first transform the whispered formants into their phonated equivalents using a denoising autoencoder. We then analyse the formant contours to predict phonated pitch contour variation. We observe that our method is effective in establishing a relationship between whispered and phonated formants and in uncovering implicit pitch contours in whisper.
【2】 Label-Synchronous Neural Transducer for End-to-End ASR标题：用于端到端ASR的标签同步神经传感器链接：https://arxiv.org/abs/2307.03088
作者：Keqi Deng,Philip C. Woodland机构：Department of Engineering, University of Cambridge, Trumpington St., Cambridge, UK.摘要：神经传感器提供了一种自然的方法来流式传输ASR。然而，他们增加输出序列与空白令牌，导致域适应使用文本数据的挑战。本文提出了一种标签同步神经传感器（LS传感器），它提取一个标签级编码器表示，然后将其与预测网络输出相结合。因此，不再需要空白令牌，并且可以使用文本数据容易地调整预测网络。自回归积分和消防（AIF）机制，提出了生成标签级编码器表示，同时保留流属性。此外，设计了一种流联合解码方法，以提高ASR的准确性。实验表明，与标准神经传感器相比，所提出的LS-传感器对于域内Librispeech-100 h数据给出了10%的相对WER减少（WERR），并且对于跨域TED-LIUM 2和AESRC 2020数据，采用自适应预测网络给出了17%和19%的相对WERR。摘要：Neural transducers provide a natural approach to streaming ASR. However, they augment output sequences with blank tokens which leads to challenges for domain adaptation using text data. This paper proposes a label-synchronous neural transducer (LS-Transducer), which extracts a label-level encoder representation before combining it with the prediction network output. Hence blank tokens are no longer needed and the prediction network can be easily adapted using text data. An Auto-regressive Integrate-and-Fire (AIF) mechanism is proposed to generate the label-level encoder representation while retaining the streaming property. In addition, a streaming joint decoding method is designed to improve ASR accuracy. Experiments show that compared to standard neural transducers, the proposed LS-Transducer gave a 10% relative WER reduction (WERR) for intra-domain Librispeech-100h data, as well as 17% and 19% relative WERRs on cross-domain TED-LIUM 2 and AESRC2020 data with an adapted prediction network.
【3】 Audio-visual End-to-end Multi-channel Speech Separation, Dereverberation  and Recognition标题：视听端到端多声道语音分离、去混响和识别链接：https://arxiv.org/abs/2307.02909
作者：Guinan Li,Jiajun Deng,Mengzhe Geng,Zengrui Jin,Tianzi Wang,Shujie Hu,Mingyu Cui,Helen Meng,Xunying Liu备注：IEEE/ACM Transactions on Audio, Speech, and Language Processing摘要：鸡尾酒会语音包含重叠的扬声器，噪声和混响的准确识别仍然是一个极具挑战性的任务。基于视觉模态对声信号失真的不变性，提出了一种将视觉信息充分融合到系统各部分的视听多通道语音分离、去混响和识别方法。视频输入的有效性在基于掩码的MVDR语音分离、基于DNN-WPE或频谱映射（SpecM）的语音去混响前端和Conformer ASR后端中得到一致证明。视听集成前端架构执行语音分离和去混响在一个流水线或联合的方式，通过基于掩码的WPD的研究。语音增强前端和ASR后端组件之间的误差成本失配通过单独使用ASR成本函数或其具有语音增强损失的内插的端到端联合微调来最小化。实验上进行的混合重叠和混响的语音数据构建使用模拟或重放的牛津LRS 2数据集。建议的视听多通道语音分离，去混响和识别系统始终优于9.1%和6.2%的绝对（41.7%和36.0%的相对）字错误率（WER）减少可比的音频基线。在PESQ、STOI和SRMR评分上也获得了一致的语音增强改善。摘要：Accurate recognition of cocktail party speech containing overlapping speakers, noise and reverberation remains a highly challenging task to date. Motivated by the invariance of visual modality to acoustic signal corruption, an audio-visual multi-channel speech separation, dereverberation and recognition approach featuring a full incorporation of visual information into all system components is proposed in this paper. The efficacy of the video input is consistently demonstrated in mask-based MVDR speech separation, DNN-WPE or spectral mapping (SpecM) based speech dereverberation front-end and Conformer ASR back-end. Audio-visual integrated front-end architectures performing speech separation and dereverberation in a pipelined or joint fashion via mask-based WPD are investigated. The error cost mismatch between the speech enhancement front-end and ASR back-end components is minimized by end-to-end jointly fine-tuning using either the ASR cost function alone, or its interpolation with the speech enhancement loss. Experiments were conducted on the mixture overlapped and reverberant speech data constructed using simulation or replay of the Oxford LRS2 dataset. The proposed audio-visual multi-channel speech separation, dereverberation and recognition systems consistently outperformed the comparable audio-only baseline by 9.1% and 6.2% absolute (41.7% and 36.0% relative) word error rate (WER) reductions. Consistent speech enhancement improvements were also obtained on PESQ, STOI and SRMR scores.
【4】 Exploring Multimodal Approaches for Alzheimer's Disease Detection Using  Patient Speech Transcript and Audio Data标题：利用患者语音记录和音频数据探索检测阿尔茨海默病的多模式方法链接：https://arxiv.org/abs/2307.02514
作者：Hongmin Cai,Xiaoke Huang,Zhengliang Liu,Wenxiong Liao,Haixing Dai,Zihao Wu,Dajiang Zhu,Hui Ren,Quanzheng Li,Tianming Liu,Xiang Li机构：South China University of Technology, Guangzhou, Guangdong, China, University of Georgia, Athens, Georgia, USA, University of Texas at Arlington, Arlington, Texas, USA, Massachusetts General Hospital, Boston, Massachusetts, USA摘要：阿尔茨海默病（AD）是严重影响患者健康的常见形式的痴呆。由于AD损害患者的语言理解和表达能力，因此AD患者的言语可以作为这种疾病的指标。本研究调查了各种方法检测AD患者的语音和成绩单数据从痴呆银行皮特数据库。所提出的方法涉及预训练的语言模型和图形神经网络（GNN），该网络从语音转录构建图形，并使用GNN提取特征用于AD检测。数据增强技术，包括同义词替换，基于GPT的增强器，等等，被用来解决小的数据集大小。同时介绍了音频数据，采用WavLM模型提取音频特征。然后使用各种方法将这些特征与文本特征融合。最后，一个对比学习的方法，试图通过转换语音转录回音频，并使用它与原始音频的对比学习。我们对上述方法进行了深入的实验和分析。我们的研究结果揭示了使用语音和音频数据进行AD检测的挑战和潜在解决方案。摘要：Alzheimer's disease (AD) is a common form of dementia that severely impacts patient health. As AD impairs the patient's language understanding and expression ability, the speech of AD patients can serve as an indicator of this disease. This study investigates various methods for detecting AD using patients' speech and transcripts data from the DementiaBank Pitt database. The proposed approach involves pre-trained language models and Graph Neural Network (GNN) that constructs a graph from the speech transcript, and extracts features using GNN for AD detection. Data augmentation techniques, including synonym replacement, GPT-based augmenter, and so on, were used to address the small dataset size. Audio data was also introduced, and WavLM model was used to extract audio features. These features were then fused with text features using various methods. Finally, a contrastive learning approach was attempted by converting speech transcripts back to audio and using it for contrastive learning with the original audio. We conducted intensive experiments and analysis on the above methods. Our findings shed light on the challenges and potential solutions in AD detection using speech and audio data.
【5】 Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong  General Audio Event Taggers标题：Whisper-AT：噪声稳健的自动语音识别器也是强大的通用音频事件标记器链接：https://arxiv.org/abs/2307.03183
作者：Yuan Gong,Sameer Khurana,Leonid Karlinsky,James Glass机构：MIT CSAIL, USA, MIT-IBM Watson AI Lab, USA备注：Accepted at Interspeech 2023. Code at this https URL摘要：在本文中，我们专注于Whisper，一个最近的自动语音识别模型训练了大量的680 k小时标记语音语料库在不同的条件下记录。我们首先展示了一个有趣的发现，虽然Whisper对现实世界的背景声音非常稳健（例如，音乐），其音频表示实际上不是噪声不变的，而是与非语音声音高度相关，这表明Whisper识别以噪声类型为条件的语音。有了这个发现，我们建立了一个统一的音频标记和语音识别模型Whisper-AT冻结的骨干Whisper，并在它上面训练一个轻量级的音频标记模型。只需<1%的额外计算成本，Whisper-AT就可以在单次向前传递中识别音频事件以及语音文本。摘要：In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.
【6】 Track Mix Generation on Music Streaming Services using Transformers标题：使用Transformers在音乐流媒体服务上生成曲目混合链接：https://arxiv.org/abs/2307.03045
作者：Walid Bendada,Théo Bontempelli,Mathieu Morlon,Benjamin Chapus,Thibault Cador,Thomas Bouabça,Guillaume Salha-Galvan备注：RecSys 2023 - Industry track with oral presentation摘要：本文介绍了2022年在音乐流媒体服务Deezer上发布的个性化播放列表生成系统Track Mix。Track Mix自动生成受初始音乐曲目启发的“混合”播放列表，允许用户发现与他们喜爱的内容相似的音乐。为了生成这些混音，我们考虑在来自用户播放列表的数百万个曲目序列上训练的Transformer模型。鉴于近年来越来越受欢迎的Transformers，我们分析的优点，缺点和技术挑战，使用这样一个模型的混合生成的服务，相比，一个更传统的协同过滤方法。自发布以来，Track Mix每天为数百万用户生成播放列表，增强了他们在Deezer上的音乐发现体验。摘要：This paper introduces Track Mix, a personalized playlist generation system released in 2022 on the music streaming service Deezer. Track Mix automatically generates "mix" playlists inspired by initial music tracks, allowing users to discover music similar to their favorite content. To generate these mixes, we consider a Transformer model trained on millions of track sequences from user playlists. In light of the growing popularity of Transformers in recent years, we analyze the advantages, drawbacks, and technical challenges of using such a model for mix generation on the service, compared to a more traditional collaborative filtering approach. Since its release, Track Mix has been generating playlists for millions of users daily, enhancing their music discovery experience on Deezer.
【7】 The Relationship Between Speech Features Changes When You Get Depressed:  Feature Correlations for Improving Speed and Performance of Depression  Detection标题：抑郁时语音特征变化之间的关系：提高抑郁检测速度和性能的特征相关性链接：https://arxiv.org/abs/2307.02892
作者：Fuxiang Tao,Wei Ma,Xuri Ge,Anna Esposito,Alessandro Vinciarelli机构：University of Glasgow, Glasgow G,QQ, UK,  Università degli Studi della Campania “Luigi Vanvitelli”, Caserta, Italy摘要：这项工作表明，抑郁症改变了从语音中提取的特征之间的相关性。此外，它表明，使用这样的见解可以提高抑郁检测器的训练速度和性能的基础上支持向量机和LSTM。这些实验是在Androids Corpus上进行的，这是一个公开的数据集，涉及112名发言者，其中包括58名被专业精神病医生诊断为抑郁症的人。结果表明，在实验中使用的模型，提高了训练速度和性能时，美联储与特征相关矩阵，而不是与特征向量。误差率的相对降低范围在23.1%和26.6%之间，取决于模型。可能的解释是，功能相关矩阵出现在抑郁症的扬声器的情况下，是更多的变量。相应地，这种现象可以被认为是抑郁症的标志。摘要：This work shows that depression changes the correlation between features extracted from speech. Furthermore, it shows that using such an insight can improve the training speed and performance of depression detectors based on SVMs and LSTMs. The experiments were performed over the Androids Corpus, a publicly available dataset involving 112 speakers, including 58 people diagnosed with depression by professional psychiatrists. The results show that the models used in the experiments improve in terms of training speed and performance when fed with feature correlation matrices rather than with feature vectors. The relative reduction of the error rate ranges between 23.1% and 26.6% depending on the model. The probable explanation is that feature correlation matrices appear to be more variable in the case of depressed speakers. Correspondingly, such a phenomenon can be thought of as a depression marker.
【8】 Evaluating raw waveforms with deep learning frameworks for speech  emotion recognition标题：用于语音情感识别的深度学习框架评估原始波形链接：https://arxiv.org/abs/2307.02820
作者：Zeynep Hilal Kilimci,Ulku Bayraktar,Ayhan Kucukmanisa机构：Department of Information Systems Engineering, Kocaeli University, Kocaeli, Turkey, Department of Electronics and Communication Engineering, Kocaeli University, Kocaeli, Turkey, A R T I C L E I N F O备注：14 pages, 6 Figures, 8 Tables摘要：语音情感识别是语音处理领域的一个具有挑战性的课题。因此，特征提取过程对于语音信号的显示和处理具有至关重要的意义。在这项工作中，我们表示了一个模型，该模型将原始音频文件直接馈送到深度神经网络中，而无需任何特征提取阶段，利用六个不同的数据集EMO-DB，RAVDESS，TESS，CREMA，SAVEE和TESS+RAVDESS进行情感识别。为了证明所提出的模型的贡献，传统的特征提取技术，即梅尔尺度谱图，梅尔频率倒谱系系数的性能，与机器学习算法，集成学习方法，深度和混合深度学习技术混合。支持向量机，决策树，朴素贝叶斯，随机森林模型作为机器学习算法进行评估，而多数投票和堆叠方法作为集成学习技术进行评估。此外，卷积神经网络、长短期记忆网络和混合CNN模型被评估为深度学习技术，并与机器学习和集成学习方法进行比较。为了证明所提出的模型的有效性，与国家的最先进的研究进行了比较。基于实验结果，CNN模型优于现有的方法，使用原始音频文件的TESS+RAVDESS数据集的准确率为95.86%，从而确定了新的最先进的。该模型在说话人无关的音频分类问题中，EMO-DB与CNN模型的准确率为90.34%，RAVDESS与CNN模型的准确率为90.42%，TESS与LSTM模型的准确率为99.48%，CREMA与CNN模型的准确率为69.72%，SAVEE与CNN模型的准确率为85.76%。摘要：Speech emotion recognition is a challenging task in speech processing field. For this reason, feature extraction process has a crucial importance to demonstrate and process the speech signals. In this work, we represent a model, which feeds raw audio files directly into the deep neural networks without any feature extraction stage for the recognition of emotions utilizing six different data sets, EMO-DB, RAVDESS, TESS, CREMA, SAVEE, and TESS+RAVDESS. To demonstrate the contribution of proposed model, the performance of traditional feature extraction techniques namely, mel-scale spectogram, mel-frequency cepstral coefficients, are blended with machine learning algorithms, ensemble learning methods, deep and hybrid deep learning techniques. Support vector machine, decision tree, naive Bayes, random forests models are evaluated as machine learning algorithms while majority voting and stacking methods are assessed as ensemble learning techniques. Moreover, convolutional neural networks, long short-term memory networks, and hybrid CNN- LSTM model are evaluated as deep learning techniques and compared with machine learning and ensemble learning methods. To demonstrate the effectiveness of proposed model, the comparison with state-of-the-art studies are carried out. Based on the experiment results, CNN model excels existent approaches with 95.86% of accuracy for TESS+RAVDESS data set using raw audio files, thence determining the new state-of-the-art. The proposed model performs 90.34% of accuracy for EMO-DB with CNN model, 90.42% of accuracy for RAVDESS with CNN model, 99.48% of accuracy for TESS with LSTM model, 69.72% of accuracy for CREMA with CNN model, 85.76% of accuracy for SAVEE with CNN model in speaker-independent audio categorization problems.
【9】 DSARSR: Deep Stacked Auto-encoders Enhanced Robust Speaker Recognition标题：DSARSR：深层堆叠自动编码器增强的稳健说话人识别链接：https://arxiv.org/abs/2307.02751
作者：Zhifeng Wang,Chunyan Zeng,Surong Duan,Hongjie Ouyang,Hongmin Xu机构：and Hongmin XuDepartment of Digital Media Technology,  Central China Normal University,  Hubei University of Technology备注：12 pages, 3 figures摘要：说话者识别是利用说话者的语音段来识别身份的生物计量模态，确定测试说话者是否属于登记的说话者之一。为了提高i-vector框架在跨信道条件下的鲁棒性，并探索将深度学习应用于说话人识别的nova方法，使用Stacked Auto-encoder来获得i-vector的抽象提取，而不是应用PLDA。在预处理和特征提取之后，使用说话人和通道无关的语音进行UBM训练。然后使用UBM来提取登记和测试语音的i向量。与传统的i向量框架不同，该框架使用线性判别分析（LDA）来降低维度并增加说话人子空间之间的区分度，该研究使用堆叠式自动编码器来重建具有较低维度的i向量，并且可以选择不同的分类器来实现最终分类。实验结果表明，该方法取得了更好的性能比国家的最先进的方法。摘要：Speaker recognition is a biometric modality that utilizes the speaker's speech segments to recognize the identity, determining whether the test speaker belongs to one of the enrolled speakers. In order to improve the robustness of the i-vector framework on cross-channel conditions and explore the nova method for applying deep learning to speaker recognition, the Stacked Auto-encoders are used to get the abstract extraction of the i-vector instead of applying PLDA. After pre-processing and feature extraction, the speaker and channel-independent speeches are employed for UBM training. The UBM is then used to extract the i-vector of the enrollment and test speech. Unlike the traditional i-vector framework, which uses linear discriminant analysis (LDA) to reduce dimension and increase the discrimination between speaker subspaces, this research use stacked auto-encoders to reconstruct the i-vector with lower dimension and different classifiers can be chosen to achieve final classification. The experimental results show that the proposed method achieves better performance than the state-of-the-art method.
【10】 On-Device Constrained Self-Supervised Speech Representation Learning for  Keyword Spotting via Knowledge Distillation标题：基于知识提取的设备约束自监督语音表征学习关键字检测链接：https://arxiv.org/abs/2307.02720
作者：Gene-Ping Yang,Yue Gu,Qingming Tang,Dongsu Du,Yuzong Liu机构：Centre for Speech Technology Research, University of Edinburgh, Alexa Perceptual Technologies, Amazon, Zoom Video Communications, Inc.备注：Accepted to Interspeech 2023摘要：大型自监督模型是有效的特征提取器，但在设备预算限制和有偏见的数据集收集下，它们的应用具有挑战性，特别是在关键字识别方面。为了解决这个问题，我们提出了一种基于知识蒸馏的自监督语音表示学习（S3 RL）架构，用于设备上的关键字识别。我们的方法使用了一个教师-学生的框架，从一个更大的，更复杂的模型，以一个较小的，重量轻的模型，使用双视图互相关蒸馏和教师的学习目标的码本知识转移。我们使用16.6k小时的内部数据集评估了我们的模型在Alexa关键字定位检测任务上的性能。我们的技术在正常和嘈杂的条件下表现出卓越的性能，证明了知识蒸馏方法在构建关键字定位任务的自监督模型时的有效性，同时在设备上的资源限制下工作。摘要：Large self-supervised models are effective feature extractors, but their application is challenging under on-device budget constraints and biased dataset collection, especially in keyword spotting. To address this, we proposed a knowledge distillation-based self-supervised speech representation learning (S3RL) architecture for on-device keyword spotting. Our approach used a teacher-student framework to transfer knowledge from a larger, more complex model to a smaller, light-weight model using dual-view cross-correlation distillation and the teacher's codebook as learning objectives. We evaluated our model's performance on an Alexa keyword spotting detection task using a 16.6k-hour in-house dataset. Our technique showed exceptional performance in normal and noisy conditions, demonstrating the efficacy of knowledge distillation methods in constructing self-supervised models for keyword spotting tasks while working within on-device resource constraints.
机器翻译，仅供参考