今日论文合集：cs.SD语音15篇，eess.AS音频处理15篇。
cs.SD语音
【1】EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in  Speech-to-Speech Models
标题：Emphassess：评估语音到语音模型中重音转移的韵律基准链接：https://arxiv.org/abs/2312.14069
作者：Maureen de Seyssel,Antony D'Avirro,Adina Williams,Emmanuel Dupoux摘要：我们介绍EmphAssess，韵律基准设计来评估语音到语音模型的编码和再现韵律强调的能力。我们将其应用于两个任务：语音再合成和语音到语音翻译。在这两种情况下，基准测试评估模型在语音输入中编码强调并在输出中准确再现强调的能力，可能跨越说话者和语言的变化。作为评估管道的一部分，我们引入了EmphasClass，这是一种新的模型，可以在帧或单词级别对强调进行分类。摘要：We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech translation. In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language. As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.
【2】 Total variation in popular rap vocals from 2009-2023: extension of the  analysis by Georgieva, Ripolles & McFee标题：2009-2023年流行说唱歌手的总变异：Georgieva，Ripolles&McFee分析的扩展链接：https://arxiv.org/abs/2312.14036
作者：Kelvin L Walls,Iran R Roman,Bea Steers,Elena Georgieva备注：None摘要：说唱声乐中的音高变化被忽视，而倾向于这种类型独特的动态节奏特性。我们提出了一个基本频率（F0）的变化在过去14年的说唱声乐分析，重点是歌曲的例子，代表了现代说唱音乐的状态。我们的分析旨在确定随着时间的推移有意义的趋势，这反过来又是Georgieva，Ripolles & McFee 2023年分析的延续。他们发现说唱是一个离群值，与其他流派相比，F0变化更大，但自该流派成立以来一直呈下降趋势。然而，他们只分析了2010年的数据。我们的分析着眼于2010年以后。我们再次观察到说唱的大F0变化，但近年来下降速度减慢。摘要：Pitch variability in rap vocals is overlooked in favor of the genre's uniquely dynamic rhythmic properties. We present an analysis of fundamental frequency (F0) variation in rap vocals over the past 14 years, focusing on song examples that represent the state of modern rap music. Our analysis aims at identifying meaningful trends over time, and is in turn a continuation of the 2023 analysis by Georgieva, Ripolles & McFee. They found rap to be an outlier with larger F0 variation compared to other genres, but with a declining trend since the genre's inception. However, they only analyzed data through 2010. Our analysis looks beyond 2010. We once again observe rap's large F0 variation, but with a decelerated decline in recent years.
【3】 BANSpEmo: A Bangla Emotional Speech Recognition Dataset标题：BANSpEmo：孟加拉语情感语音识别数据集链接：https://arxiv.org/abs/2312.14020
作者：Md Gulzar Hussain,Mahmuda Rahman,Babe Sultana,Ye Shiren摘要：在音频和语音分析领域，从声学信号中识别情感的能力至关重要。人机交互（HCI）和行为分析只是区分情感和语音信号的能力具有广泛应用的许多领域中的一小部分。在这里，我们将介绍BanSpEmo，这是一个情感语音语料库，仅由音频记录组成，专为孟加拉语创建。该语料库包含超过1小时23分钟的792段录音。22名母语人士参与了两组代表六种期望情绪的句子的录音。数据集由12个孟加拉语句子组成，这些句子以6种情绪表达，如厌恶，快乐，悲伤，惊讶，愤怒和恐惧。这一语料库也没有实现性别平衡。十位有相关领域经验或表演经验的人参加了这个语料库的评估。它在每个情感类中有平衡数量的音频记录。BanSpEmo可以被认为是一个有用的资源，以促进情感和语音识别的研究和相关的孟加拉语的应用。该数据集可以在这里找到：https://data.mendeley.com/datasets/rdwn4bs5ky，并可能用于学术研究。摘要：In the field of audio and speech analysis, the ability to identify emotions from acoustic signals is essential. Human-computer interaction (HCI) and behavioural analysis are only a few of the many areas where the capacity to distinguish emotions from speech signals has an extensive range of applications. Here, we are introducing BanSpEmo, a corpus of emotional speech that only consists of audio recordings and has been created specifically for the Bangla language. This corpus contains 792 audio recordings over a duration of more than 1 hour and 23 minutes. 22 native speakers took part in the recording of two sets of sentences that represent the six desired emotions. The data set consists of 12 Bangla sentences which are uttered in 6 emotions as Disgust, Happy, Sad, Surprised, Anger, and Fear. This corpus is not also gender balanced. Ten individuals who either have experience in related field or have acting experience took part in the assessment of this corpus. It has a balanced number of audio recordings in each emotion class. BanSpEmo can be considered as a useful resource to promote emotion and speech recognition research and related applications in the Bangla language. The dataset can be found here: https://data.mendeley.com/datasets/rdwn4bs5ky and might be employed for academic research.
【4】 On the choice of the optimal temporal support for audio classification  with Pre-trained embeddings标题：基于预训练嵌入的音频分类最优时间支持的选择链接：https://arxiv.org/abs/2312.14005
作者：Aurian Quelennec,Michel Olvera,Geoffroy Peeters,Slim Essid备注：Copyright 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：目前最先进的音频分析系统依赖于预先训练的嵌入模型，通常用作（冻结）特征提取器。为一组任务选择最佳的一个是最近许多出版物的主题。然而，在这些作品中经常被忽视的一个方面是音频输入的持续时间对提取嵌入的影响，我们称之为时间支持（TS）。在这项工作中，我们研究了TS对成熟或新兴的预训练嵌入的影响，这些嵌入被选择来代表不同类型的架构和学习范式。我们使用乐器和环境声音数据集进行评估，即OpenMIC，TAU Urban Acoustic Scenes 2020 Mobile和ESC-50。我们特别强调，基于音频频谱图变换器的系统（PaSST和BEAT）在较小的TS下仍然有效，因此可以大幅降低内存和计算成本。此外，我们表明，通过选择最佳的TS，我们在所有任务中达到竞争力的结果。特别是，我们改进了OpenMIC上的最先进的结果，使用BEAT和PaSST，无需任何微调。摘要：Current state-of-the-art audio analysis systems rely on pre-trained embedding models, often used off-the-shelf as (frozen) feature extractors. Choosing the best one for a set of tasks is the subject of many recent publications. However, one aspect often overlooked in these works is the influence of the duration of audio input considered to extract an embedding, which we refer to as Temporal Support (TS). In this work, we study the influence of the TS for well-established or emerging pre-trained embeddings, chosen to represent different types of architectures and learning paradigms. We conduct this evaluation using both musical instrument and environmental sound datasets, namely OpenMIC, TAU Urban Acoustic Scenes 2020 Mobile, and ESC-50. We especially highlight that Audio Spectrogram Transformer-based systems (PaSST and BEATs) remain effective with smaller TS, which therefore allows for a drastic reduction in memory and computational cost. Moreover, we show that by choosing the optimal TS we reach competitive results across all tasks. In particular, we improve the state-of-the-art results on OpenMIC, using BEATs and PaSST without any fine-tuning.
【5】 Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models标题：用于预训练ASR模型的自监督自适应AV融合模块链接：https://arxiv.org/abs/2312.13873
作者：Christopher Simic,Tobias Bocklet备注：Accepted at ICASSP 2024摘要：近年来，自动语音识别（ASR）已经达到了一定的准确度，甚至在将语音转录为文本方面超过了人类。然而，所有当前的ASR方法显示出对环境噪声的一定弱点。为了减少这种弱点，视听语音识别（AVSR）方法还考虑了来自嘴唇运动的视觉信息用于转录。这种额外的模态增加了从头开始训练模型的计算成本。我们提出了一种方法，该方法建立在预先训练的ASR模型上，并使用自适应上游模块对其进行扩展，该模块融合了音频和视觉信息。由于我们不需要从头开始训练Transformer结构，因此与传统AVSR模型相比，我们的方法需要一小部分计算资源。与目前的SOTA系统（如AV-HuBERT）相比，我们的方法在不同的模型大小、噪声类别和宽SNR范围内的字错误率平均提高了8.3%。与常见的AVSR方法相比，该方法允许高达21%的较小模型，并且仅需要一小部分计算资源用于训练和推理。摘要：Automatic speech recognition (ASR) has reached a level of accuracy in recent years, that even outperforms humans in transcribing speech to text. Nevertheless, all current ASR approaches show a certain weakness against ambient noise. To reduce this weakness, audio-visual speech recognition (AVSR) approaches additionally consider visual information from lip movements for transcription. This additional modality increases the computational cost for training models from scratch. We propose an approach, that builds on a pre-trained ASR model and extends it with an adaptive upstream module, that fuses audio and visual information. Since we do not need to train the transformer structure from scratch, our approach requires a fraction of the computational resources compared to traditional AVSR models. Compared to current SOTA systems like AV-HuBERT, our approach achieves an average improvement of 8.3% in word error rate across different model sizes, noise categories and broad SNR range. The approach allows up to 21% smaller models and requires only a fraction of the computational resources for training and inference compared to common AVSR approaches.
【6】 BAE-Net: A Low complexity and high fidelity Bandwidth-Adaptive neural  network for speech super-resolution标题：BAE-Net：一种低复杂度高保真的语音超分辨带宽自适应神经网络链接：https://arxiv.org/abs/2312.13722
作者：Guochen Yu,Xiguang Zheng,Nan Li,Runqiang Han,Chengshi Zheng,Chen Zhang,Chao Zhou,Qi Huang,Bing Yu备注：Accepted to ICASSP 2024摘要：语音带宽扩展（BWE）在实际通信系统中表现出良好的性能，提高语音的感知质量。大多数现有的BWE研究主要集中在固定的上采样率，忽略了捕获的音频的有效带宽可能由于各种捕获设备和传输条件而频繁波动的事实。在本文中，我们提出了一种新的流自适应带宽扩展解决方案称为BAE-Net，这是适合于处理未知的和变化的有效带宽的低分辨率语音。为了解决盲目恢复高频幅度和相位语音内容的挑战，我们设计了一个双流架构，结合幅度修复和相位细化。对于边缘设备上的潜在应用，本文还介绍了BAE-NET-lite，这是一个轻量级，流式和高效的框架。定量结果表明，BAE网络的性能和计算效率方面的优越性相比，现有的国家的最先进的斗轮挖掘机方法。摘要：Speech bandwidth extension (BWE) has demonstrated promising performance in enhancing the perceptual speech quality in real communication systems. Most existing BWE researches primarily focus on fixed upsampling ratios, disregarding the fact that the effective bandwidth of captured audio may fluctuate frequently due to various capturing devices and transmission conditions. In this paper, we propose a novel streaming adaptive bandwidth extension solution dubbed BAE-Net, which is suitable to handle the low-resolution speech with unknown and varying effective bandwidth. To address the challenges of recovering both the high-frequency magnitude and phase speech content blindly, we devise a dual-stream architecture that incorporates the magnitude inpainting and phase refinement. For potential applications on edge devices, this paper also introduces BAE-NET-lite, which is a lightweight, streaming and efficient framework. Quantitative results demonstrate the superiority of BAE-Net in terms of both performance and computational efficiency when compared with existing state-of-the-art BWE methods.
【7】 Speech Translation with Large Language Models: An Industrial Practice标题：大型语言模型的语音翻译：一种工业实践链接：https://arxiv.org/abs/2312.13585
作者：Zhichao Huang,Rong Ye,Tom Ko,Qianqian Dong,Shanbo Cheng,Mingxuan Wang,Hang Li备注：Technical report. 13 pages. Demo: this https URL摘要：鉴于大型语言模型（LLM）在各种任务中取得的巨大成功，本文介绍了LLM-ST，这是一种基于预训练LLM构建的新颖有效的语音翻译模型。通过将大型语言模型（LLM）与语音编码器集成并采用多任务指令调优，LLM-ST可以生成精确的时间戳transmitting和translation，即使是长音频输入。此外，我们的研究结果表明，实施的思想链（CoT）提示可以在LLM-ST的上下文中产生的优势。通过严格的英语和汉语数据集上的实验，我们展示了LLM-ST的卓越性能，在语音翻译领域建立了一个新的基准。演示：https://speechtranslation.github.io/llm-st/.摘要：Given the great success of large language models (LLMs) across various tasks, in this paper, we introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained LLM. By integrating the large language model (LLM) with a speech encoder and employing multi-task instruction tuning, LLM-ST can produce accurate timestamped transcriptions and translations, even from long audio inputs. Furthermore, our findings indicate that the implementation of Chain-of-Thought (CoT) prompting can yield advantages in the context of LLM-ST. Through rigorous experimentation on English and Chinese datasets, we showcase the exceptional performance of LLM-ST, establishing a new benchmark in the field of speech translation. Demo: https://speechtranslation.github.io/llm-st/.
【8】 Fine-grained Disentangled Representation Learning for Multimodal Emotion  Recognition标题：用于多通道情感识别的细粒度解缠表征学习链接：https://arxiv.org/abs/2312.13567
作者：Haoqin Sun,Shiwan Zhao,Xuechen Wang,Wenjia Zeng,Yong Chen,Yong Qin备注：Accepted by ICASSP 2024摘要：多模态情感识别（MMER）是一个活跃的研究领域，旨在通过融合多种感知模态来准确地识别人类情感。然而，固有的异质性跨模态引入分布差距和信息冗余，构成重大挑战的MMER。在本文中，我们提出了一种新的细粒度解纠缠表示学习（FDRL）框架来解决这些挑战。具体来说，我们设计了模态共享和模态私有编码器，分别将每个模态投影到模态共享和模态私有子空间中。在共享子空间中，我们引入了一个细粒度的对齐组件来学习模态共享表示，从而捕获模态一致性。随后，我们定制了一个细粒度的差异组件来约束私有子空间，从而学习模态私有表示并增强其多样性。最后，我们引入了一个细粒度的预测器组件，以确保来自编码器的输出表示的标签保持不变。在IEMOCAP数据集上的实验结果表明，FDRL优于现有的方法，在WAR和UAR上分别达到78.34%和79.44%。摘要：Multimodal emotion recognition (MMER) is an active research field that aims to accurately recognize human emotions by fusing multiple perceptual modalities. However, inherent heterogeneity across modalities introduces distribution gaps and information redundancy, posing significant challenges for MMER. In this paper, we propose a novel fine-grained disentangled representation learning (FDRL) framework to address these challenges. Specifically, we design modality-shared and modality-private encoders to project each modality into modality-shared and modality-private subspaces, respectively. In the shared subspace, we introduce a fine-grained alignment component to learn modality-shared representations, thus capturing modal consistency. Subsequently, we tailor a fine-grained disparity component to constrain the private subspaces, thereby learning modality-private representations and enhancing their diversity. Lastly, we introduce a fine-grained predictor component to ensure that the labels of the output representations from the encoders remain unchanged. Experimental results on the IEMOCAP dataset show that FDRL outperforms the state-of-the-art methods, achieving 78.34% and 79.44% on WAR and UAR, respectively.
【9】 kNN-CTC: Enhancing ASR via Retrieval of CTC Pseudo Labels标题：KNN-CTC：通过检索CTC伪标签来增强ASR链接：https://arxiv.org/abs/2312.13560
作者：Jiaming Zhou,Shiwan Zhao,Yaqi Liu,Wenjia Zeng,Yong Chen,Yong Qin备注：Accepted by ICASSP 2024摘要：检索增强语言模型在各种自然语言处理（NLP）任务中的成功在自动语音识别（ASR）应用中受到限制，这是由于构建细粒度音频文本数据存储的挑战。本文提出了kNN-CTC，一种新的方法，克服了这些挑战，利用连接时间分类（CTC）伪标签建立帧级音频-文本键值对，规避了精确的地面实况对齐的需要。我们还引入了一个跳过空白的策略，战略性地忽略CTC空白帧，以减少存储器的大小。kNN-CTC将k-最近邻检索机制集成到预先训练的CTC ASR系统中，从而显著提高了性能。通过将k-最近邻检索机制整合到预先训练的CTC ASR系统中，并利用细粒度、修剪过的搜索引擎，kNN-CTC在各种实验设置下始终实现了性能的大幅提升。我们的代码可以在https://github.com/NKU-HLT/KNN-CTC上找到。摘要：The success of retrieval-augmented language models in various natural language processing (NLP) tasks has been constrained in automatic speech recognition (ASR) applications due to challenges in constructing fine-grained audio-text datastores. This paper presents kNN-CTC, a novel approach that overcomes these challenges by leveraging Connectionist Temporal Classification (CTC) pseudo labels to establish frame-level audio-text key-value pairs, circumventing the need for precise ground truth alignments. We further introduce a skip-blank strategy, which strategically ignores CTC blank frames, to reduce datastore size. kNN-CTC incorporates a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems, achieving significant improvements in performance. By incorporating a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems and leveraging a fine-grained, pruned datastore, kNN-CTC consistently achieves substantial improvements in performance under various experimental settings. Our code is available at https://github.com/NKU-HLT/KNN-CTC.
【10】 Multi-Level Knowledge Distillation for Speech Emotion Recognition in  Noisy Conditions标题：噪声环境下语音情感识别的多层次知识提取链接：https://arxiv.org/abs/2312.13556
作者：Yang Liu,Haoqin Sun,Geng Chen,Qingyue Wang,Zhen Zhao,Xugang Lu,Longbiao Wang备注：Accepted by INTERSPEECH 2023摘要：语音情感识别（SER）的性能显着恶化，在噪声的存在下，使其具有挑战性，以实现有竞争力的性能在嘈杂的条件下。为此，我们提出了一种多级知识蒸馏（MLKD）方法，其目的是将知识从干净语音训练的教师模型转移到噪声语音训练的更简单的学生模型。具体来说，我们使用干净的语音特征提取的wav 2 vec-2.0作为学习目标和训练的wav 2 vec-2.0近似的原始wav 2 vec-2.0在噪声条件下的特征提取能力。此外，我们利用原始wav 2 vec-2.0的多级知识来监督wav 2 vec-2.0的单级输出。我们通过使用IEMOCAP数据集上的五种类型的噪声污染语音进行广泛的实验来评估我们所提出的方法的有效性，与最先进的模型相比，这些实验显示出有希望的结果。摘要：Speech emotion recognition (SER) performance deteriorates significantly in the presence of noise, making it challenging to achieve competitive performance in noisy conditions. To this end, we propose a multi-level knowledge distillation (MLKD) method, which aims to transfer the knowledge from a teacher model trained on clean speech to a simpler student model trained on noisy speech. Specifically, we use clean speech features extracted by the wav2vec-2.0 as the learning goal and train the distil wav2vec-2.0 to approximate the feature extraction ability of the original wav2vec-2.0 under noisy conditions. Furthermore, we leverage the multi-level knowledge of the original wav2vec-2.0 to supervise the single-level output of the distil wav2vec-2.0. We evaluate the effectiveness of our proposed method by conducting extensive experiments using five types of noise-contaminated speech on the IEMOCAP dataset, which show promising results compared to state-of-the-art models.
【11】 Leveraging Visual Supervision for Array-based Active Speaker Detection  and Localization标题：利用视觉监督进行基于阵列的有源说话人检测和定位链接：https://arxiv.org/abs/2312.14021
作者：Davide Berghi,Philip J. B. Jackson摘要：用于主动说话者检测（ASD）的传统视听方法通常依赖于视觉上预提取的面部轨迹和对应的单通道音频来找到视频中的说话者。因此，每当说话者的脸不可见时，它们往往会失败。我们证明了一个简单的音频卷积递归神经网络（CRNN）训练的空间输入特征提取多通道音频可以执行同时水平主动扬声器检测和定位（ASDL），独立于视觉模态。为了解决生成地面实况标签来训练这样一个系统的时间和成本，我们提出了一个新的自我监督的训练管道，包括“学生-教师”的学习方法。传统的预先训练的主动说话者检测器被采用作为“教师”网络，以提供说话者的位置作为伪标签。训练多通道音频“学生”网络以生成相同的结果。在推理时，学生网络还可以概括和定位教师网络无法视觉检测到的被遮挡的说话者，从而大大提高了召回率。在TragicTalkers数据集上的实验表明，用所提出的自监督学习方法训练的音频网络可以超过典型的视听方法的性能，并产生与昂贵的传统监督训练相竞争的结果。我们证明，可以实现最小的人工监督时，在学习管道的改进。可以利用更大的训练集和将视觉与多通道音频系统集成来寻求进一步的增益。摘要：Conventional audio-visual approaches for active speaker detection (ASD) typically rely on visually pre-extracted face tracks and the corresponding single-channel audio to find the speaker in a video. Therefore, they tend to fail every time the face of the speaker is not visible. We demonstrate that a simple audio convolutional recurrent neural network (CRNN) trained with spatial input features extracted from multichannel audio can perform simultaneous horizontal active speaker detection and localization (ASDL), independently of the visual modality. To address the time and cost of generating ground truth labels to train such a system, we propose a new self-supervised training pipeline that embraces a ``student-teacher'' learning approach. A conventional pre-trained active speaker detector is adopted as a ``teacher'' network to provide the position of the speakers as pseudo-labels. The multichannel audio ``student'' network is trained to generate the same results. At inference, the student network can generalize and locate also the occluded speakers that the teacher network is not able to detect visually, yielding considerable improvements in recall rate. Experiments on the TragicTalkers dataset show that an audio network trained with the proposed self-supervised learning approach can exceed the performance of the typical audio-visual methods and produce results competitive with the costly conventional supervised training. We demonstrate that improvements can be achieved when minimal manual supervision is introduced in the learning pipeline. Further gains may be sought with larger training sets and integrating vision with the multichannel audio system.
【12】 Blind Localization of Room Reflections with Application to Spatial Audio标题：房间反射的盲定位及其在空间音频中的应用链接：https://arxiv.org/abs/2312.13707
作者：Yogev Hadadi,Vladimir Tourbabin,Paul Calamia,Boaz Rafaely备注：None摘要：在不知道房间脉冲响应的情况下，早期房间反射的盲估计具有很大的价值。FF-PHALCOR（频率聚焦相位对准相关），方法最近开发了这个目标，扩展了原始的PHALCOR方法从球面到任意阵列。然而，以前的研究只是在有限的条件下比较了这两种方法，没有提出一个全面的性能分析。这项研究提出了一个进步，通过评估在更广泛的条件下的算法的性能。此外，通过听力测试调查感知方面的表现。该测试涉及从已知的房间声学参数合成房间脉冲响应，并用估计的反射代替早期反射。通过这个测试证明了估计的反射对空间感知的重要性。摘要：Blind estimation of early room reflections, without knowledge of the room impulse response, holds substantial value. The FF-PHALCOR (Frequency Focusing PHase ALigned CORrelation), method was recently developed for this objective, extending the original PHALCOR method from spherical to arbitrary arrays. However, previous studies only compared the two methods under limited conditions without presenting a comprehensive performance analysis. This study presents an advance by evaluating the performance of the algorithm in a wider range of conditions. Additionally, performance in terms of perception is investigated through a listening test. This test involves synthesizing room impulse responses from known room acoustics parameters and replacing the early reflections with the estimated ones. The importance of the estimated reflections for spatial perception is demonstrated through this test.
【13】 Self-supervised Complex Network for Machine Sound Anomaly Detection标题：用于机器声音异常检测的自监督复杂网络链接：https://arxiv.org/abs/2312.13615
作者：Miseul Kim,Minh Tri Ho,Hong-Goo Kang备注：Published in EUSIPCO 2021摘要：在本文中，我们提出了一个机器声音异常检测算法与深度复杂网络训练的自我监督。利用相位连续性信息对于检测时序信号中的异常是至关重要的这一事实，我们所提出的算法利用复频谱作为输入，并在整个过程中执行复数运算。由于相位信息的有用性可以根据机器声音的类型而变化，因此我们还应用注意力机制来根据机器类型控制复杂和幅度谱瓶颈特征的权重。我们训练我们的网络来执行一个自我监督的任务，该任务将正常输入声音的机器标识符（id）分为多个类别。在测试时，如果训练模型无法正确分类id，则输入信号被检测为异常。换句话说，当多类识别任务的输出交叉熵得分低于预定义阈值时，我们确定存在异常。MIMII数据集的实验表明，该算法具有更高的曲线下面积（AUC）得分比传统的幅度谱为基础的算法。摘要：In this paper, we propose an anomaly detection algorithm for machine sounds with a deep complex network trained by self-supervision. Using the fact that phase continuity information is crucial for detecting abnormalities in time-series signals, our proposed algorithm utilizes the complex spectrum as an input and performs complex number arithmetic throughout the entire process. Since the usefulness of phase information can vary depending on the type of machine sound, we also apply an attention mechanism to control the weights of the complex and magnitude spectrum bottleneck features depending on the machine type. We train our network to perform a self-supervised task that classifies the machine identifier (id) of normal input sounds among multiple classes. At test time, an input signal is detected as anomalous if the trained model is unable to correctly classify the id. In other words, we determine the presence of an anomality when the output cross-entropy score of the multiclass identification task is lower than a pre-defined threshold. Experiments with the MIMII dataset show that the proposed algorithm has a much higher area under the curve (AUC) score than conventional magnitude spectrum-based algorithms.
【14】 Style Modeling for Multi-Speaker Articulation-to-Speech标题：多说话人发音转换的风格建模链接：https://arxiv.org/abs/2312.13603
作者：Miseul Kim,Zhenyu Piao,Jihyun Lee,Hong-Goo Kang备注：5 pages, Accepted to ICASSP 2023摘要：在本文中，我们提出了一个神经发音到语音（ATS）的框架，在多说话人的情况下，从发音信号合成高质量的语音。大多数传统的ATS方法只关注从单个说话人的发音特征建模语音的上下文信息。为了明确地表示每个扬声器的说话风格以及上下文信息，我们提出的模型估计风格嵌入，从基本的语音风格属性，如音高和能量的指导。我们的模型采用卷积层和基于transformer的注意力层，以充分利用由电磁关节成像（EMA）测量的发音信号的局部和全局信息。我们的模型显着提高了合成语音的质量相比，在Haskins数据集的客观和主观测量的基线。摘要：In this paper, we propose a neural articulation-to-speech (ATS) framework that synthesizes high-quality speech from articulatory signal in a multi-speaker situation. Most conventional ATS approaches only focus on modeling contextual information of speech from a single speaker's articulatory features. To explicitly represent each speaker's speaking style as well as the contextual information, our proposed model estimates style embeddings, guided from the essential speech style attributes such as pitch and energy. We adopt convolutional layers and transformer-based attention layers for our model to fully utilize both local and global information of articulatory signals, measured by electromagnetic articulography (EMA). Our model significantly improves the quality of synthesized speech compared to the baseline in terms of objective and subjective measurements in the Haskins dataset.
【15】 BrainTalker: Low-Resource Brain-to-Speech Synthesis with Transfer  Learning using Wav2Vec 2.0标题：BrainTalker：使用Wav2Vec 2.0进行迁移学习的低资源脑到语音合成链接：https://arxiv.org/abs/2312.13600
作者：Miseul Kim,Zhenyu Piao,Jihyun Lee,Hong-Goo Kang备注：5 pages. Accepted to BHI 2023摘要：从大脑的神经活动中解码口语是一个快速发展的研究课题，因为它可以为那些难以产生可听语音的人提供交流。对于这项任务，皮层电图（ECoG）是一种用于以高时间分辨率和高空间精度记录大脑活动的常用方法。然而，由于获得ECoG记录所需的风险外科手术，已经收集的这种数据相对较少，并且该量不足以训练基于神经网络的脑到语音（BTS）系统。为了解决这个问题，我们提出了BrainTalker-一种新颖的BTS框架，可以在极低资源的情况下从ECoG信号生成可理解的口语。我们应用了一种迁移学习方法，该方法利用了预训练的自监督模型Wav 2 Vec 2.0。具体而言，我们训练编码器模块，以将ECoG信号映射到与相应口语的Wav 2 Vec 2.0表示相匹配的潜在嵌入。然后，这些嵌入使用堆叠的卷积层和基于变换器的层转换为梅尔频谱图，这些频谱图被馈送到神经声码器中以合成语音波形。实验结果表明，我们提出的框架在主观和客观指标方面实现了出色的性能，包括生成的和地面真实梅尔频谱图之间的Pearson相关系数为0.9。我们共享公开的演示和代码。摘要：Decoding spoken speech from neural activity in the brain is a fast-emerging research topic, as it could enable communication for people who have difficulties with producing audible speech. For this task, electrocorticography (ECoG) is a common method for recording brain activity with high temporal resolution and high spatial precision. However, due to the risky surgical procedure required for obtaining ECoG recordings, relatively little of this data has been collected, and the amount is insufficient to train a neural network-based Brain-to-Speech (BTS) system. To address this problem, we propose BrainTalker-a novel BTS framework that generates intelligible spoken speech from ECoG signals under extremely low-resource scenarios. We apply a transfer learning approach utilizing a pre-trained self supervised model, Wav2Vec 2.0. Specifically, we train an encoder module to map ECoG signals to latent embeddings that match Wav2Vec 2.0 representations of the corresponding spoken speech. These embeddings are then transformed into mel-spectrograms using stacked convolutional and transformer-based layers, which are fed into a neural vocoder to synthesize speech waveform. Experimental results demonstrate our proposed framework achieves outstanding performance in terms of subjective and objective metrics, including a Pearson correlation coefficient of 0.9 between generated and ground truth mel spectrograms. We share publicly available Demos and Code.
eess.AS音频处理
【1】 Leveraging Visual Supervision for Array-based Active Speaker Detection  and Localization标题：利用视觉监督进行基于阵列的有源说话人检测和定位链接：https://arxiv.org/abs/2312.14021
作者：Davide Berghi,Philip J. B. Jackson摘要：用于主动说话者检测（ASD）的传统视听方法通常依赖于视觉上预提取的面部轨迹和对应的单通道音频来找到视频中的说话者。因此，每当说话者的脸不可见时，它们往往会失败。我们证明了一个简单的音频卷积递归神经网络（CRNN）训练的空间输入特征提取多通道音频可以执行同时水平主动扬声器检测和定位（ASDL），独立于视觉模态。为了解决生成地面实况标签来训练这样一个系统的时间和成本，我们提出了一个新的自我监督的训练管道，包括“学生-教师”的学习方法。传统的预先训练的主动说话者检测器被采用作为“教师”网络，以提供说话者的位置作为伪标签。训练多通道音频“学生”网络以生成相同的结果。在推理时，学生网络还可以概括和定位教师网络无法视觉检测到的被遮挡的说话者，从而大大提高了召回率。在TragicTalkers数据集上的实验表明，用所提出的自监督学习方法训练的音频网络可以超过典型的视听方法的性能，并产生与昂贵的传统监督训练相竞争的结果。我们证明，可以实现最小的人工监督时，在学习管道的改进。可以利用更大的训练集和将视觉与多通道音频系统集成来寻求进一步的增益。摘要：Conventional audio-visual approaches for active speaker detection (ASD) typically rely on visually pre-extracted face tracks and the corresponding single-channel audio to find the speaker in a video. Therefore, they tend to fail every time the face of the speaker is not visible. We demonstrate that a simple audio convolutional recurrent neural network (CRNN) trained with spatial input features extracted from multichannel audio can perform simultaneous horizontal active speaker detection and localization (ASDL), independently of the visual modality. To address the time and cost of generating ground truth labels to train such a system, we propose a new self-supervised training pipeline that embraces a ``student-teacher'' learning approach. A conventional pre-trained active speaker detector is adopted as a ``teacher'' network to provide the position of the speakers as pseudo-labels. The multichannel audio ``student'' network is trained to generate the same results. At inference, the student network can generalize and locate also the occluded speakers that the teacher network is not able to detect visually, yielding considerable improvements in recall rate. Experiments on the TragicTalkers dataset show that an audio network trained with the proposed self-supervised learning approach can exceed the performance of the typical audio-visual methods and produce results competitive with the costly conventional supervised training. We demonstrate that improvements can be achieved when minimal manual supervision is introduced in the learning pipeline. Further gains may be sought with larger training sets and integrating vision with the multichannel audio system.
【2】 Blind Localization of Room Reflections with Application to Spatial Audio标题：房间反射的盲定位及其在空间音频中的应用链接：https://arxiv.org/abs/2312.13707
作者：Yogev Hadadi,Vladimir Tourbabin,Paul Calamia,Boaz Rafaely备注：None摘要：None摘要：Blind estimation of early room reflections, without knowledge of the room impulse response, holds substantial value. The FF-PHALCOR (Frequency Focusing PHase ALigned CORrelation), method was recently developed for this objective, extending the original PHALCOR method from spherical to arbitrary arrays. However, previous studies only compared the two methods under limited conditions without presenting a comprehensive performance analysis. This study presents an advance by evaluating the performance of the algorithm in a wider range of conditions. Additionally, performance in terms of perception is investigated through a listening test. This test involves synthesizing room impulse responses from known room acoustics parameters and replacing the early reflections with the estimated ones. The importance of the estimated reflections for spatial perception is demonstrated through this test.
【3】 Self-supervised Complex Network for Machine Sound Anomaly Detection标题：机器声音异常检测的自监督复杂网络链接：https://arxiv.org/abs/2312.13615
作者：Miseul Kim,Minh Tri Ho,Hong-Goo Kang备注：Published in EUSIPCO 2021摘要：在本文中，我们提出了一个机器声音异常检测算法与深度复杂网络训练的自我监督。利用相位连续性信息对于检测时序信号中的异常是至关重要的这一事实，我们所提出的算法利用复频谱作为输入，并在整个过程中执行复数运算。由于相位信息的有用性可以根据机器声音的类型而变化，因此我们还应用注意力机制来根据机器类型控制复杂和幅度谱瓶颈特征的权重。我们训练我们的网络来执行一个自我监督的任务，该任务将正常输入声音的机器标识符（id）分为多个类别。在测试时，如果训练模型无法正确分类id，则输入信号被检测为异常。换句话说，当多类识别任务的输出交叉熵得分低于预定义阈值时，我们确定存在异常。MIMII数据集的实验表明，该算法具有更高的曲线下面积（AUC）得分比传统的幅度谱为基础的算法。摘要：In this paper, we propose an anomaly detection algorithm for machine sounds with a deep complex network trained by self-supervision. Using the fact that phase continuity information is crucial for detecting abnormalities in time-series signals, our proposed algorithm utilizes the complex spectrum as an input and performs complex number arithmetic throughout the entire process. Since the usefulness of phase information can vary depending on the type of machine sound, we also apply an attention mechanism to control the weights of the complex and magnitude spectrum bottleneck features depending on the machine type. We train our network to perform a self-supervised task that classifies the machine identifier (id) of normal input sounds among multiple classes. At test time, an input signal is detected as anomalous if the trained model is unable to correctly classify the id. In other words, we determine the presence of an anomality when the output cross-entropy score of the multiclass identification task is lower than a pre-defined threshold. Experiments with the MIMII dataset show that the proposed algorithm has a much higher area under the curve (AUC) score than conventional magnitude spectrum-based algorithms.
【4】 Style Modeling for Multi-Speaker Articulation-to-Speech标题：多说话人发音转换的风格建模链接：https://arxiv.org/abs/2312.13603
作者：Miseul Kim,Zhenyu Piao,Jihyun Lee,Hong-Goo Kang备注：5 pages, Accepted to ICASSP 2023摘要：在本文中，我们提出了一个神经发音到语音（ATS）的框架，在多说话人的情况下，从发音信号合成高质量的语音。大多数传统的ATS方法只关注从单个说话人的发音特征建模语音的上下文信息。为了明确地表示每个扬声器的说话风格以及上下文信息，我们提出的模型估计风格嵌入，从基本的语音风格属性，如音高和能量的指导。我们的模型采用卷积层和基于transformer的注意力层，以充分利用由电磁关节成像（EMA）测量的发音信号的局部和全局信息。我们的模型显着提高了合成语音的质量相比，在Haskins数据集的客观和主观测量的基线。摘要：In this paper, we propose a neural articulation-to-speech (ATS) framework that synthesizes high-quality speech from articulatory signal in a multi-speaker situation. Most conventional ATS approaches only focus on modeling contextual information of speech from a single speaker's articulatory features. To explicitly represent each speaker's speaking style as well as the contextual information, our proposed model estimates style embeddings, guided from the essential speech style attributes such as pitch and energy. We adopt convolutional layers and transformer-based attention layers for our model to fully utilize both local and global information of articulatory signals, measured by electromagnetic articulography (EMA). Our model significantly improves the quality of synthesized speech compared to the baseline in terms of objective and subjective measurements in the Haskins dataset.
【5】 BrainTalker: Low-Resource Brain-to-Speech Synthesis with Transfer  Learning using Wav2Vec 2.0标题：BrainTalker：使用Wav2Vec 2.0的低资源大脑到语音合成与迁移学习链接：https://arxiv.org/abs/2312.13600
作者：Miseul Kim,Zhenyu Piao,Jihyun Lee,Hong-Goo Kang备注：5 pages. Accepted to BHI 2023摘要：从大脑的神经活动中解码口语是一个快速发展的研究课题，因为它可以为那些难以产生可听语音的人提供交流。对于这项任务，皮层电图（ECoG）是一种用于以高时间分辨率和高空间精度记录大脑活动的常用方法。然而，由于获得ECoG记录所需的风险外科手术，已经收集的这种数据相对较少，并且该量不足以训练基于神经网络的脑到语音（BTS）系统。为了解决这个问题，我们提出了BrainTalker-一种新颖的BTS框架，可以在极低资源的情况下从ECoG信号生成可理解的口语。我们应用了一种迁移学习方法，该方法利用了预训练的自监督模型Wav 2 Vec 2.0。具体而言，我们训练编码器模块，以将ECoG信号映射到与相应口语的Wav 2 Vec 2.0表示相匹配的潜在嵌入。然后，这些嵌入使用堆叠的卷积层和基于变换器的层转换为梅尔频谱图，这些频谱图被馈送到神经声码器中以合成语音波形。实验结果表明，我们提出的框架在主观和客观指标方面实现了出色的性能，包括生成的和地面真实梅尔频谱图之间的Pearson相关系数为0.9。我们共享公开的演示和代码。摘要：Decoding spoken speech from neural activity in the brain is a fast-emerging research topic, as it could enable communication for people who have difficulties with producing audible speech. For this task, electrocorticography (ECoG) is a common method for recording brain activity with high temporal resolution and high spatial precision. However, due to the risky surgical procedure required for obtaining ECoG recordings, relatively little of this data has been collected, and the amount is insufficient to train a neural network-based Brain-to-Speech (BTS) system. To address this problem, we propose BrainTalker-a novel BTS framework that generates intelligible spoken speech from ECoG signals under extremely low-resource scenarios. We apply a transfer learning approach utilizing a pre-trained self supervised model, Wav2Vec 2.0. Specifically, we train an encoder module to map ECoG signals to latent embeddings that match Wav2Vec 2.0 representations of the corresponding spoken speech. These embeddings are then transformed into mel-spectrograms using stacked convolutional and transformer-based layers, which are fed into a neural vocoder to synthesize speech waveform. Experimental results demonstrate our proposed framework achieves outstanding performance in terms of subjective and objective metrics, including a Pearson correlation coefficient of 0.9 between generated and ground truth mel spectrograms. We share publicly available Demos and Code.
【6】 EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in  Speech-to-Speech Models标题：Emphassess：评估语音到语音模型中重音转移的韵律基准链接：https://arxiv.org/abs/2312.14069
作者：Maureen de Seyssel,Antony D'Avirro,Adina Williams,Emmanuel Dupoux摘要：我们介绍EmphAssess，韵律基准设计来评估语音到语音模型的编码和再现韵律强调的能力。我们将其应用于两个任务：语音再合成和语音到语音翻译。在这两种情况下，基准测试评估模型在语音输入中编码强调并在输出中准确再现强调的能力，可能跨越说话者和语言的变化。作为评估管道的一部分，我们引入了EmphasClass，这是一种新的模型，可以在帧或单词级别对强调进行分类。摘要：We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech translation. In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language. As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.
【7】 Total variation in popular rap vocals from 2009-2023: extension of the  analysis by Georgieva, Ripolles & McFee标题：2009-2023年流行说唱歌手的总变异：Georgieva，Ripolles&McFee分析的扩展链接：https://arxiv.org/abs/2312.14036
作者：Kelvin L Walls,Iran R Roman,Bea Steers,Elena Georgieva备注：None摘要：说唱声乐中的音高变化被忽视，而倾向于这种类型独特的动态节奏特性。我们提出了一个基本频率（F0）的变化在过去14年的说唱声乐分析，重点是歌曲的例子，代表了现代说唱音乐的状态。我们的分析旨在确定随着时间的推移有意义的趋势，这反过来又是Georgieva，Ripolles & McFee 2023年分析的延续。他们发现说唱是一个离群值，与其他流派相比，F0变化更大，但自该流派成立以来一直呈下降趋势。然而，他们只分析了2010年的数据。我们的分析着眼于2010年以后。我们再次观察到说唱的大F0变化，但近年来下降速度减慢。摘要：Pitch variability in rap vocals is overlooked in favor of the genre's uniquely dynamic rhythmic properties. We present an analysis of fundamental frequency (F0) variation in rap vocals over the past 14 years, focusing on song examples that represent the state of modern rap music. Our analysis aims at identifying meaningful trends over time, and is in turn a continuation of the 2023 analysis by Georgieva, Ripolles & McFee. They found rap to be an outlier with larger F0 variation compared to other genres, but with a declining trend since the genre's inception. However, they only analyzed data through 2010. Our analysis looks beyond 2010. We once again observe rap's large F0 variation, but with a decelerated decline in recent years.
【8】 BANSpEmo: A Bangla Emotional Speech Recognition Dataset标题：BANSpEmo：孟加拉情感语音识别数据集链接：https://arxiv.org/abs/2312.14020
作者：Md Gulzar Hussain,Mahmuda Rahman,Babe Sultana,Ye Shiren摘要：在音频和语音分析领域，从声学信号中识别情感的能力至关重要。人机交互（HCI）和行为分析只是区分情感和语音信号的能力具有广泛应用的许多领域中的一小部分。在这里，我们将介绍BanSpEmo，这是一个情感语音语料库，仅由音频记录组成，专为孟加拉语创建。该语料库包含超过1小时23分钟的792段录音。22名母语人士参与了两组代表六种期望情绪的句子的录音。数据集由12个孟加拉语句子组成，这些句子以6种情绪表达，如厌恶，快乐，悲伤，惊讶，愤怒和恐惧。这一语料库也没有实现性别平衡。十位有相关领域经验或表演经验的人参加了这个语料库的评估。它在每个情感类中有平衡数量的音频记录。BanSpEmo可以被认为是一个有用的资源，以促进情感和语音识别的研究和相关的孟加拉语的应用。该数据集可以在这里找到：https://data.mendeley.com/datasets/rdwn4bs5ky，并可能用于学术研究。摘要：In the field of audio and speech analysis, the ability to identify emotions from acoustic signals is essential. Human-computer interaction (HCI) and behavioural analysis are only a few of the many areas where the capacity to distinguish emotions from speech signals has an extensive range of applications. Here, we are introducing BanSpEmo, a corpus of emotional speech that only consists of audio recordings and has been created specifically for the Bangla language. This corpus contains 792 audio recordings over a duration of more than 1 hour and 23 minutes. 22 native speakers took part in the recording of two sets of sentences that represent the six desired emotions. The data set consists of 12 Bangla sentences which are uttered in 6 emotions as Disgust, Happy, Sad, Surprised, Anger, and Fear. This corpus is not also gender balanced. Ten individuals who either have experience in related field or have acting experience took part in the assessment of this corpus. It has a balanced number of audio recordings in each emotion class. BanSpEmo can be considered as a useful resource to promote emotion and speech recognition research and related applications in the Bangla language. The dataset can be found here: https://data.mendeley.com/datasets/rdwn4bs5ky and might be employed for academic research.
【9】 On the choice of the optimal temporal support for audio classification  with Pre-trained embeddings标题：基于预训练嵌入的音频分类中最佳时间支持度的选择链接：https://arxiv.org/abs/2312.14005
作者：Aurian Quelennec,Michel Olvera,Geoffroy Peeters,Slim Essid备注：Copyright 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：目前最先进的音频分析系统依赖于预先训练的嵌入模型，通常用作（冻结）特征提取器。为一组任务选择最佳的一个是最近许多出版物的主题。然而，在这些作品中经常被忽视的一个方面是音频输入的持续时间对提取嵌入的影响，我们称之为时间支持（TS）。在这项工作中，我们研究了TS对成熟或新兴的预训练嵌入的影响，这些嵌入被选择来代表不同类型的架构和学习范式。我们使用乐器和环境声音数据集进行评估，即OpenMIC，TAU Urban Acoustic Scenes 2020 Mobile和ESC-50。我们特别强调，基于音频频谱图变换器的系统（PaSST和BEAT）在较小的TS下仍然有效，因此可以大幅降低内存和计算成本。此外，我们表明，通过选择最佳的TS，我们在所有任务中达到竞争力的结果。特别是，我们改进了OpenMIC上的最先进的结果，使用BEAT和PaSST，无需任何微调。摘要：Current state-of-the-art audio analysis systems rely on pre-trained embedding models, often used off-the-shelf as (frozen) feature extractors. Choosing the best one for a set of tasks is the subject of many recent publications. However, one aspect often overlooked in these works is the influence of the duration of audio input considered to extract an embedding, which we refer to as Temporal Support (TS). In this work, we study the influence of the TS for well-established or emerging pre-trained embeddings, chosen to represent different types of architectures and learning paradigms. We conduct this evaluation using both musical instrument and environmental sound datasets, namely OpenMIC, TAU Urban Acoustic Scenes 2020 Mobile, and ESC-50. We especially highlight that Audio Spectrogram Transformer-based systems (PaSST and BEATs) remain effective with smaller TS, which therefore allows for a drastic reduction in memory and computational cost. Moreover, we show that by choosing the optimal TS we reach competitive results across all tasks. In particular, we improve the state-of-the-art results on OpenMIC, using BEATs and PaSST without any fine-tuning.
【10】 Self-Supervised Adaptive AV Fusion Module for Pre-Trained ASR Models标题：用于预训练ASR模型的自监督自适应AV融合模块链接：https://arxiv.org/abs/2312.13873
作者：Christopher Simic,Tobias Bocklet备注：Accepted at ICASSP 2024摘要：None摘要：Automatic speech recognition (ASR) has reached a level of accuracy in recent years, that even outperforms humans in transcribing speech to text. Nevertheless, all current ASR approaches show a certain weakness against ambient noise. To reduce this weakness, audio-visual speech recognition (AVSR) approaches additionally consider visual information from lip movements for transcription. This additional modality increases the computational cost for training models from scratch. We propose an approach, that builds on a pre-trained ASR model and extends it with an adaptive upstream module, that fuses audio and visual information. Since we do not need to train the transformer structure from scratch, our approach requires a fraction of the computational resources compared to traditional AVSR models. Compared to current SOTA systems like AV-HuBERT, our approach achieves an average improvement of 8.3% in word error rate across different model sizes, noise categories and broad SNR range. The approach allows up to 21% smaller models and requires only a fraction of the computational resources for training and inference compared to common AVSR approaches.
【11】 BAE-Net: A Low complexity and high fidelity Bandwidth-Adaptive neural  network for speech super-resolution标题：BAE-Net：一种低复杂度高保真的语音超分辨带宽自适应神经网络链接：https://arxiv.org/abs/2312.13722
作者：Guochen Yu,Xiguang Zheng,Nan Li,Runqiang Han,Chengshi Zheng,Chen Zhang,Chao Zhou,Qi Huang,Bing Yu备注：Accepted to ICASSP 2024摘要：None摘要：Speech bandwidth extension (BWE) has demonstrated promising performance in enhancing the perceptual speech quality in real communication systems. Most existing BWE researches primarily focus on fixed upsampling ratios, disregarding the fact that the effective bandwidth of captured audio may fluctuate frequently due to various capturing devices and transmission conditions. In this paper, we propose a novel streaming adaptive bandwidth extension solution dubbed BAE-Net, which is suitable to handle the low-resolution speech with unknown and varying effective bandwidth. To address the challenges of recovering both the high-frequency magnitude and phase speech content blindly, we devise a dual-stream architecture that incorporates the magnitude inpainting and phase refinement. For potential applications on edge devices, this paper also introduces BAE-NET-lite, which is a lightweight, streaming and efficient framework. Quantitative results demonstrate the superiority of BAE-Net in terms of both performance and computational efficiency when compared with existing state-of-the-art BWE methods.
【12】 Speech Translation with Large Language Models: An Industrial Practice标题：使用大语言模型的语音翻译：一种实业实践链接：https://arxiv.org/abs/2312.13585
作者：Zhichao Huang,Rong Ye,Tom Ko,Qianqian Dong,Shanbo Cheng,Mingxuan Wang,Hang Li备注：Technical report. 13 pages. Demo: this https URL摘要：鉴于大型语言模型（LLM）在各种任务中取得的巨大成功，本文介绍了LLM-ST，这是一种基于预训练LLM构建的新颖有效的语音翻译模型。通过将大型语言模型（LLM）与语音编码器集成并采用多任务指令调优，LLM-ST可以生成精确的时间戳transmitting和translation，即使是长音频输入。此外，我们的研究结果表明，实施的思想链（CoT）提示可以在LLM-ST的上下文中产生的优势。通过严格的英语和汉语数据集上的实验，我们展示了LLM-ST的卓越性能，在语音翻译领域建立了一个新的基准。演示：https://speechtranslation.github.io/llm-st/.摘要：Given the great success of large language models (LLMs) across various tasks, in this paper, we introduce LLM-ST, a novel and effective speech translation model constructed upon a pre-trained LLM. By integrating the large language model (LLM) with a speech encoder and employing multi-task instruction tuning, LLM-ST can produce accurate timestamped transcriptions and translations, even from long audio inputs. Furthermore, our findings indicate that the implementation of Chain-of-Thought (CoT) prompting can yield advantages in the context of LLM-ST. Through rigorous experimentation on English and Chinese datasets, we showcase the exceptional performance of LLM-ST, establishing a new benchmark in the field of speech translation. Demo: https://speechtranslation.github.io/llm-st/.
【13】 Fine-grained Disentangled Representation Learning for Multimodal Emotion  Recognition标题：用于多通道情感识别的细粒度解缠表征学习链接：https://arxiv.org/abs/2312.13567
作者：Haoqin Sun,Shiwan Zhao,Xuechen Wang,Wenjia Zeng,Yong Chen,Yong Qin备注：Accepted by ICASSP 2024摘要：多模态情感识别（MMER）是一个活跃的研究领域，旨在通过融合多种感知模态来准确地识别人类情感。然而，固有的异质性跨模态引入分布差距和信息冗余，构成重大挑战的MMER。在本文中，我们提出了一种新的细粒度解纠缠表示学习（FDRL）框架来解决这些挑战。具体来说，我们设计了模态共享和模态私有编码器，分别将每个模态投影到模态共享和模态私有子空间中。在共享子空间中，我们引入了一个细粒度的对齐组件来学习模态共享表示，从而捕获模态一致性。随后，我们定制了一个细粒度的差异组件来约束私有子空间，从而学习模态私有表示并增强其多样性。最后，我们引入了一个细粒度的预测器组件，以确保来自编码器的输出表示的标签保持不变。在IEMOCAP数据集上的实验结果表明，FDRL优于现有的方法，在WAR和UAR上分别达到78.34%和79.44%。摘要：Multimodal emotion recognition (MMER) is an active research field that aims to accurately recognize human emotions by fusing multiple perceptual modalities. However, inherent heterogeneity across modalities introduces distribution gaps and information redundancy, posing significant challenges for MMER. In this paper, we propose a novel fine-grained disentangled representation learning (FDRL) framework to address these challenges. Specifically, we design modality-shared and modality-private encoders to project each modality into modality-shared and modality-private subspaces, respectively. In the shared subspace, we introduce a fine-grained alignment component to learn modality-shared representations, thus capturing modal consistency. Subsequently, we tailor a fine-grained disparity component to constrain the private subspaces, thereby learning modality-private representations and enhancing their diversity. Lastly, we introduce a fine-grained predictor component to ensure that the labels of the output representations from the encoders remain unchanged. Experimental results on the IEMOCAP dataset show that FDRL outperforms the state-of-the-art methods, achieving 78.34% and 79.44% on WAR and UAR, respectively.
【14】 kNN-CTC: Enhancing ASR via Retrieval of CTC Pseudo Labels标题：KNN-CTC：通过检索CTC伪标签来增强ASR链接：https://arxiv.org/abs/2312.13560
作者：Jiaming Zhou,Shiwan Zhao,Yaqi Liu,Wenjia Zeng,Yong Chen,Yong Qin备注：Accepted by ICASSP 2024摘要：检索增强语言模型在各种自然语言处理（NLP）任务中的成功在自动语音识别（ASR）应用中受到限制，这是由于构建细粒度音频文本数据存储的挑战。本文提出了kNN-CTC，一种新的方法，克服了这些挑战，利用连接时间分类（CTC）伪标签建立帧级音频-文本键值对，规避了精确的地面实况对齐的需要。我们还引入了一个跳过空白的策略，战略性地忽略CTC空白帧，以减少存储器的大小。kNN-CTC将k-最近邻检索机制集成到预先训练的CTC ASR系统中，从而显著提高了性能。通过将k-最近邻检索机制整合到预先训练的CTC ASR系统中，并利用细粒度、修剪过的搜索引擎，kNN-CTC在各种实验设置下始终实现了性能的大幅提升。我们的代码可以在https://github.com/NKU-HLT/KNN-CTC上找到。摘要：The success of retrieval-augmented language models in various natural language processing (NLP) tasks has been constrained in automatic speech recognition (ASR) applications due to challenges in constructing fine-grained audio-text datastores. This paper presents kNN-CTC, a novel approach that overcomes these challenges by leveraging Connectionist Temporal Classification (CTC) pseudo labels to establish frame-level audio-text key-value pairs, circumventing the need for precise ground truth alignments. We further introduce a skip-blank strategy, which strategically ignores CTC blank frames, to reduce datastore size. kNN-CTC incorporates a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems, achieving significant improvements in performance. By incorporating a k-nearest neighbors retrieval mechanism into pre-trained CTC ASR systems and leveraging a fine-grained, pruned datastore, kNN-CTC consistently achieves substantial improvements in performance under various experimental settings. Our code is available at https://github.com/NKU-HLT/KNN-CTC.
【15】 Multi-Level Knowledge Distillation for Speech Emotion Recognition in  Noisy Conditions标题：噪声环境下语音情感识别的多层次知识提取链接：https://arxiv.org/abs/2312.13556
作者：Yang Liu,Haoqin Sun,Geng Chen,Qingyue Wang,Zhen Zhao,Xugang Lu,Longbiao Wang备注：Accepted by INTERSPEECH 2023摘要：语音情感识别（SER）的性能显着恶化，在噪声的存在下，使其具有挑战性，以实现有竞争力的性能在嘈杂的条件下。为此，我们提出了一种多级知识蒸馏（MLKD）方法，其目的是将知识从干净语音训练的教师模型转移到噪声语音训练的更简单的学生模型。具体来说，我们使用干净的语音特征提取的wav 2 vec-2.0作为学习目标和训练的wav 2 vec-2.0近似的原始wav 2 vec-2.0在噪声条件下的特征提取能力。此外，我们利用原始wav 2 vec-2.0的多级知识来监督wav 2 vec-2.0的单级输出。我们通过使用IEMOCAP数据集上的五种类型的噪声污染语音进行广泛的实验来评估我们所提出的方法的有效性，与最先进的模型相比，这些实验显示出有希望的结果。摘要：Speech emotion recognition (SER) performance deteriorates significantly in the presence of noise, making it challenging to achieve competitive performance in noisy conditions. To this end, we propose a multi-level knowledge distillation (MLKD) method, which aims to transfer the knowledge from a teacher model trained on clean speech to a simpler student model trained on noisy speech. Specifically, we use clean speech features extracted by the wav2vec-2.0 as the learning goal and train the distil wav2vec-2.0 to approximate the feature extraction ability of the original wav2vec-2.0 under noisy conditions. Furthermore, we leverage the multi-level knowledge of the original wav2vec-2.0 to supervise the single-level output of the distil wav2vec-2.0. We evaluate the effectiveness of our proposed method by conducting extensive experiments using five types of noise-contaminated speech on the IEMOCAP dataset, which show promising results compared to state-of-the-art models.
机器翻译由腾讯交互翻译提供，仅供参考