今日论文合集：cs.SD语音48篇，eess.AS音频处理51篇。【1】 A Study on Bias and Fairness In Deep Speaker Recognition标题：深度说话人识别中的偏差和公平性研究链接：http://arxiv.org/pdf/2303.08026v1
作者：Amirhossein Hajavi,Ali Etemad机构：Dept. ECE and Ingenuity Labs Research Institute, Queen’s University, Canada摘要：随着使用说话人识别（SR）系统作为个人认证和个性化服务的手段的智能设备的普遍存在，SR系统的公平性已经成为一个重要的焦点。在本文中，我们研究了公平的概念，在最近的SR系统的基础上，3个流行的和相关的定义，即统计平价，均衡赔率，和平等机会。我们检查了5种流行的神经结构和5种常用的损失函数在训练SR系统，同时评估他们的公平性对性别和民族群体。我们的详细实验阐明了这个概念，并证明了更复杂的编码器架构更好地符合公平性的定义。此外，我们发现损失函数的选择会显著影响随机共振模型的偏差。摘要：With the ubiquity of smart devices that use speaker recognition (SR) systems as a means of authenticating individuals and personalizing their services, fairness of SR systems has becomes an important point of focus. In this paper we study the notion of fairness in recent SR systems based on 3 popular and relevant definitions, namely Statistical Parity, Equalized Odds, and Equal Opportunity. We examine 5 popular neural architectures and 5 commonly used loss functions in training SR systems, while evaluating their fairness against gender and nationality groups. Our detailed experiments shed light on this concept and demonstrate that more sophisticated encoder architectures better align with the definitions of fairness. Additionally, we find that the choice of loss functions can significantly impact the bias of SR models.【2】 Improving Accented Speech Recognition with Multi-Domain Training标题：利用多域训练改进口音语音识别链接：http://arxiv.org/pdf/2303.07924v1
作者：Lucas Maison,Yannick Estève机构：Yannick Esteve⋆, ⋆ Laboratoire Informatique d’Avignon, Avignon, France, † Thales SIX, Multimedia Lab, Gennevilliers, France备注：5 pages, 2 figures. Accepted to ICASSP 2023摘要：由于自监督学习的兴起，自动语音识别（ASR）系统现在在各种各样的数据集上实现了接近人类的性能。然而，它们仍然缺乏泛化能力，并且对重音变化等领域变化不具有鲁棒性。在这项工作中，我们使用代表四种不同法语口音的语音音频来创建微调数据集，以提高预训练ASR模型的鲁棒性。通过在训练集中加入不同的重音，我们获得了域内和域外的改进。我们的数值实验表明，与单域训练相比，我们可以将非洲和比利时口音的错误率降低高达25%（相对），同时保持标准法语的良好性能。摘要：Thanks to the rise of self-supervised learning, automatic speech recognition (ASR) systems now achieve near-human performance on a wide variety of datasets. However, they still lack generalization capability and are not robust to domain shifts like accent variations. In this work, we use speech audio representing four different French accents to create fine-tuning datasets that improve the robustness of pre-trained ASR models. By incorporating various accents in the training set, we obtain both in-domain and out-of-domain improvements. Our numerical experiments show that we can reduce error rates by up to 25% (relative) on African and Belgian accents compared to single-domain training while keeping a good performance on standard French.【3】 BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data标题：BLAT：引导语言.基于AudioSet标签引导的合成数据的音频预训练链接：http://arxiv.org/pdf/2303.07902v1
作者：Xuenan Xu,Zhiling Zhang,Zelin Zhou,Pingyue Zhang,Zeyu Xie,Mengyue Wu,Kenny Q. Zhu机构：Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China摘要：与大量的视文本预训练研究相比，对听文本预训练的研究较少，这主要是由于缺乏足够的平行听文本数据。现有的方法大多以视觉通道为中心进行音频-文本预训练，不可避免地引入了数据噪声。在本文中，我们提出BLAT：Bootstrapping语言-基于标签引导合成数据的音频预训练。我们利用音频字幕直接从音频生成文本，而不需要视觉模态的帮助，从而消除了模态失配带来的潜在噪声。此外，提出了AudioSet标签指导下的字幕生成方法，使得字幕更加准确。通过以上两点改进，我们可以获得高质量、大规模的并行音频文本数据，并在此基础上进行音频文本预训练。对一系列下游任务的评估表明，BLAT在大多数数据集上达到了SOTA zero-shot分类性能，并且在下游任务上进行微调时性能有了显著提高，这表明了我们合成数据的有效性。摘要：Compared with ample visual-text pre-training research, few works explore audio-text pre-training, mostly due to the lack of sufficient parallel audio-text data. Most existing methods incorporate the visual modality as a pivot for audio-text pre-training, which inevitably induces data noise. In this paper, we propose BLAT: Bootstrapping Language-Audio pre-training based on Tag-guided synthetic data. We utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated. Furthermore, we propose caption generation under the guidance of AudioSet tags, leading to more accurate captions. With the above two improvements, we curate high-quality, large-scale parallel audio-text data, based on which we perform audio-text pre-training. Evaluation on a series of downstream tasks indicates that BLAT achieves SOTA zero-shot classification performance on most datasets and significant performance improvement when fine-tuned on downstream tasks, suggesting the effectiveness of our synthetic data.【4】 DiffuseRoll: Multi-track multi-category music generation based on diffusion model标题：DiffuseRoll：基于扩散模型的多轨多品类音乐生成链接：http://arxiv.org/pdf/2303.07794v1
作者：Hongfei Wang机构：Communication University of China, Beijing, China, nd Given Name Surname, dept. name of organization (of Aff.), City, Country, email address, rd Given Name Surname摘要：近年来，生成模型的研究在音乐生成领域取得了显著进展。然而，现有的方法大多集中于生成单声道或单声道音乐，而生成具有丰富属性的多声道和多音轨音乐仍然是一项具有挑战性的任务。在这篇论文中，我们提出了一个新的方法来产生多音轨，多属性的交响音乐使用扩散模型。具体来说，我们用扩散模型生成钢琴卷表示，并将它们映射为MIDI格式输出。为了获取丰富的属性信息，我们引入了颜色编码方案，将音符序列编码为代表音高、力度和乐器的颜色和位置信息。该方案实现了离散音乐序列和连续图像之间的无缝映射。我们还提出了一种后处理方法来优化生成的分数以获得更好的性能。实验结果表明，与图形方法相比，该方法在生成具有丰富属性信息的复调音乐方面优于现有方法。摘要：Recent advancements in generative models have shown remarkable progress in music generation. However, most existing methods focus on generating monophonic or homophonic music, while the generation of polyphonic and multi-track music with rich attributes is still a challenging task. In this paper, we propose a novel approach for multi-track, multi-attribute symphonic music generation using the diffusion model. Specifically, we generate piano-roll representations with a diffusion model and map them to MIDI format for output. To capture rich attribute information, we introduce a color coding scheme to encode note sequences into color and position information that represents pitch,velocity, and instrument. This scheme enables a seamless mapping between discrete music sequences and continuous images. We also propose a post-processing method to optimize the generated scores for better performance. Experimental results show that our method outperforms state-of-the-art methods in terms of polyphonic music generation with rich attribute information compared to the figure methods.【5】 Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis标题：语音合成中基于半监督风格提取和分层建模的交叉语体风格转换韵律改进链接：http://arxiv.org/pdf/2303.07711v1
作者：Chunyu Qiang,Peng Yang,Hao Che,Ying Zhang,Xiaorui Wang,Zhongyuan Wang机构：Kwai, Beijing, P.R. China备注：Accepted by ICASSP2023摘要：语音合成中的跨说话人风格迁移是指将源说话人的风格迁移到目标说话人的合成语音中。在大多数先前方法中，合成的细粒度韵律特征通常表示源说话人的平均风格，类似于一对多问题（即，多个韵律变化对应于同一文本）。针对这一问题，提出一种强度控制的半监督风格提取器，将风格与内容、音色分离，提高全局风格嵌入的表示性和可解释性，从而缓解韵律预测中的一对多映射和数据不均衡问题。提出了一种分层韵律预测器来改进韵律建模。我们发现，利用源语者易于预测的韵律特征可以实现更好的风格迁移。此外，还提出了一种说话人迁移方式的循环一致性损失，以帮助模型在训练阶段学习看不见的风格-音色组合。实验结果表明，该方法优于基线方法。我们提供了一个网站与音频样本。摘要：Cross-speaker style transfer in speech synthesis aims at transferring a style from source speaker to synthesized speech of a target speaker's timbre. In most previous methods, the synthesized fine-grained prosody features often represent the source speaker's average style, similar to the one-to-many problem(i.e., multiple prosody variations correspond to the same text). In response to this problem, a strength-controlled semi-supervised style extractor is proposed to disentangle the style from content and timbre, improving the representation and interpretability of the global style embedding, which can alleviate the one-to-many mapping and data imbalance problems in prosody prediction. A hierarchical prosody predictor is proposed to improve prosody modeling. We find that better style transfer can be achieved by using the source speaker's prosody features that are easily predicted. Additionally, a speaker-transfer-wise cycle consistency loss is proposed to assist the model in learning unseen style-timbre combinations during the training phase. Experimental results show that the method outperforms the baseline. We provide a website with audio samples.【6】 Dynamic Alignment Mask CTC: Improved Mask-CTC with Aligned Cross Entropy标题：动态对准掩模CTC：改进的对准交叉熵掩模CTC链接：http://arxiv.org/pdf/2303.07687v1
作者：Xulong Zhang,Haobin Tang,Jianzong Wang,Ning Cheng,Jian Luo,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., University of Science and Technology of China备注：Accepted by ICASSP 2023摘要：由于非自回归模型对所有目标标记进行并行预测，与传统的自回归模型相比，大大提高了语音识别的解码效率。本文提出了动态对准掩模CTC，介绍了两种方法：（1）对准交叉熵（AXE），通过动态编程找到使交叉熵损失最小化的单调对准，（2）动态校正，通过用模型预测的表征替换一些掩码来创建新的训练样本。AXE忽略了预测和地面事实句子之间的绝对位置对齐，而关注于相对顺序的标记匹配。动态校正方法使得模型能够模拟非掩码但可能错误的标记，即使它们具有高置信度。在WSJ数据集上的实验表明，AXE丢失和校正方法都能提高Mask CTC的WER性能。摘要：Because of predicting all the target tokens in parallel, the non-autoregressive models greatly improve the decoding efficiency of speech recognition compared with traditional autoregressive models. In this work, we present dynamic alignment Mask CTC, introducing two methods: (1) Aligned Cross Entropy (AXE), finding the monotonic alignment that minimizes the cross-entropy loss through dynamic programming, (2) Dynamic Rectification, creating new training samples by replacing some masks with model predicted tokens. The AXE ignores the absolute position alignment between prediction and ground truth sentence and focuses on tokens matching in relative order. The dynamic rectification method makes the model capable of simulating the non-mask but possible wrong tokens, even if they have high confidence. Our experiments on WSJ dataset demonstrated that not only AXE loss but also the rectification method could improve the WER performance of Mask CTC.【7】 QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis标题：QI-TTS：情感语音合成中的提问语调控制链接：http://arxiv.org/pdf/2303.07682v1
作者：Haobin Tang,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing XiaoWijayakulasooriya,Pantaleon Perera机构：Ping An Technology (Shenzhen) Co., Ltd., University of Science and Technology of China备注：Accepted by ICASSP 2023摘要：目前的表达性文语转换模型主要关注情感语音的合成，而忽略了语调等细粒度语音的合成。本文提出了QI-TTS，旨在更好地传递和控制语调，在传递参考语音情感的同时，进一步传递说话人的提问意图。我们提出一个多重样式撷取器，从两个不同层次撷取样式嵌入。句子级表示情感，最后的音节级表示语调。对于细粒度的语调控制，我们使用相对属性表示音节级的语调强度，实验验证了QI-TTS在情感语音合成中提高语调表达能力的有效性。摘要：Recent expressive text to speech (TTS) models focus on synthesizing emotional speech, but some fine-grained styles such as intonation are neglected. In this paper, we propose QI-TTS which aims to better transfer and control intonation to further deliver the speaker's questioning intention while transferring emotion from reference speech. We propose a multi-style extractor to extract style embedding from two different levels. While the sentence level represents emotion, the final syllable level represents intonation. For fine-grained intonation control, we use relative attributes to represent intonation intensity at the syllable level.Experiments have validated the effectiveness of QI-TTS for improving intonation expressiveness in emotional speech synthesis.【8】 Improving Music Genre Classification from multi-modal properties of music and genre correlations Perspective标题：从音乐的多模式属性和流派相关性角度改进音乐流派分类链接：http://arxiv.org/pdf/2303.07667v1
作者：Ganghui Ru,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., China备注：Accepted by ICASSP 2023摘要：音乐体裁分类由于其在音乐信息检索中的广泛应用，近年来得到了广泛的研究。以前的作品往往表现得不令人满意，因为那些方法只使用音频内容或低效地联合使用音频内容和歌词内容。此外，由于流派通常在音乐轨道中共同出现，因此期望捕获流派相关性并对其建模以改进多标签音乐流派分类的性能。针对这些问题，提出了一种新的多模态方法，利用音频-歌词对比度损失和两个对称的跨模态注意力，对音频和歌词的特征进行对齐和融合。此外，根据多标签分类的特点，提出了一个体裁相关性提取模块，用于捕获和建模潜在的体裁相关性。实验结果表明，该方法在Music 4All数据集上取得了较好的分类效果，显著优于其他多标签音乐流派分类方法.摘要：Music genre classification has been widely studied in past few years for its various applications in music information retrieval. Previous works tend to perform unsatisfactorily, since those methods only use audio content or jointly use audio content and lyrics content inefficiently. In addition, as genres normally co-occur in a music track, it is desirable to capture and model the genre correlations to improve the performance of multi-label music genre classification. To solve these issues, we present a novel multi-modal method leveraging audio-lyrics contrastive loss and two symmetric cross-modal attention, to align and fuse features from audio and lyrics. Furthermore, based on the nature of the multi-label classification, a genre correlations extraction module is presented to capture and model potential genre correlations. Extensive experiments demonstrate that our proposed method significantly surpasses other multi-label music genre classification methods and achieves state-of-the-art result on Music4All dataset.【9】 Cross-lingual Alzheimer's Disease detection based on paralinguistic and pre-trained features标题：基于副语言和预训练特征的跨语言阿尔茨海默病检测链接：http://arxiv.org/pdf/2303.07650v1
作者：Xuchu Chen,Yu Pu,Jinpeng Li,Wei-Qiang Zhang机构：Department of Electronic Engineering, Tsinghua University, Beijing, China备注：accepted by ICASSP 2023摘要：我们向ICASSP-SPGC-2023 ADReSS-M挑战任务提交了我们的意见书，该任务旨在研究哪些声学特征可以在阿尔茨海默病（AD）预测的语言之间进行概括和转移。这项挑战包括两项任务：一种是对AD患者和健康个体的语音进行分类，另一种是仅基于语音推断简易精神状态检查（MMSE）分数。困难主要体现在数据集的不匹配上，训练集是英文的，而测试集是希腊文的。利用openSmile工具包提取副语言特征，利用XLSR-53提取声学特征。此外，我们在将语音转录成文本之后提取语言特征。在我们的方法中，这些特征被用作AD检测的指标。我们的方法在分类任务上达到了69.6%的准确率，在回归任务上达到了4.788的均方根误差（RMSE）。实验结果表明，该方法有望实现基于自发语音的多语种阿尔茨海默病自动检测。摘要：We present our submission to the ICASSP-SPGC-2023 ADReSS-M Challenge Task, which aims to investigate which acoustic features can be generalized and transferred across languages for Alzheimer's Disease (AD) prediction. The challenge consists of two tasks: one is to classify the speech of AD patients and healthy individuals, and the other is to infer Mini Mental State Examination (MMSE) score based on speech only. The difficulty is mainly embodied in the mismatch of the dataset, in which the training set is in English while the test set is in Greek. We extract paralinguistic features using openSmile toolkit and acoustic features using XLSR-53. In addition, we extract linguistic features after transcribing the speech into text. These features are used as indicators for AD detection in our method. Our method achieves an accuracy of 69.6% on the classification task and a root mean squared error (RMSE) of 4.788 on the regression task. The results show that our proposed method is expected to achieve automatic multilingual Alzheimer's Disease detection through spontaneous speech.【10】 Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classification标题：面向一般声音分类的面向无数据知识提取的富特征音频模型反演链接：http://arxiv.org/pdf/2303.07643v1
作者：Zuheng Kang,Yayun He,Jianzong Wang,Junqing Peng,Xiaoyang Qu,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by ICASSP 2023. International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023)摘要：无数据知识提取（DFKD）近年来引起了学术界的广泛关注，尤其是计算机视觉领域的重大突破。尽管结果很有希望，但该技术还没有很好地应用于音频和信号处理。由于音频信号的持续时间是可变的，因此它有自己独特的建模方式。在这项工作中，我们提出了富特征音频模型反演（FRAMI），一个用于一般声音分类任务的无数据知识提取框架。该算法首先通过特征不变的对比度损失生成高质量、特征丰富的Mel谱图。然后，在对这些特征丰富的样本进行知识提取时，重用统计池层之前和之后的隐藏状态。在Urbansound 8 k、ESC-50和audioMNIST数据集上的实验结果表明，FRAMI能够生成特征丰富的样本.同时，通过重用隐藏状态进一步提高了学生模型的准确性，显著优于基线方法。摘要：Data-Free Knowledge Distillation (DFKD) has recently attracted growing attention in the academic community, especially with major breakthroughs in computer vision. Despite promising results, the technique has not been well applied to audio and signal processing. Due to the variable duration of audio signals, it has its own unique way of modeling. In this work, we propose feature-rich audio model inversion (FRAMI), a data-free knowledge distillation framework for general sound classification tasks. It first generates high-quality and feature-rich Mel-spectrograms through a feature-invariant contrastive loss. Then, the hidden states before and after the statistics pooling layer are reused when knowledge distillation is performed on these feature-rich samples. Experimental results on the Urbansound8k, ESC-50, and audioMNIST datasets demonstrate that FRAMI can generate feature-rich samples. Meanwhile, the accuracy of the student model is further improved by reusing the hidden state and significantly outperforms the baseline method.【11】CAT: Causal Audio Transformer for Audio Classification标题：CAT：用于音频分类的因果音频转换器链接：http://arxiv.org/pdf/2303.07626v1
作者：Xiaoyu Liu,Hanlin Lu,Jianbo Yuan,Xinyu Li机构：University of Maryland, College Park,ByteDance备注：Accepted to ICASSP 2023摘要：基于注意力的Transformers由于其全局感受域和处理长时相关性的能力而越来越多地应用于音频分类。然而，现有的框架主要是从Vision Transformers扩展而来的，并不完全兼容音频信号。本文介绍了一种因果音频Transformer（CAT），它由多分辨率多特征（MRMF）特征提取和声学注意块组成，用于更优化的音频建模。此外，我们提出了一个因果模型，以减轻过度拟合，帮助知识转移，并提高可解释性。CAT在ESC50、AudioSet和UrbanSound8K数据集上获得了更高或相当的最新分类性能，并且可以轻松推广到其他基于Transformer的模型。摘要：The attention-based Transformers have been increasingly applied to audio classification because of their global receptive field and ability to handle long-term dependency. However, the existing frameworks which are mainly extended from the Vision Transformers are not perfectly compatible with audio signals. In this paper, we introduce a Causal Audio Transformer (CAT) consisting of a Multi-Resolution Multi-Feature (MRMF) feature extraction with an acoustic attention block for more optimized audio modeling. In addition, we propose a causal module that alleviates over-fitting, helps with knowledge transfer, and improves interpretability. CAT obtains higher or comparable state-of-the-art classification performance on ESC50, AudioSet and UrbanSound8K datasets, and can be easily generalized to other Transformer-based models.【12】I3D: Transformer architectures with input-dependent dynamic depth for speech recognition标题：I3D：具有输入相关动态深度的语音识别转换器体系结构
链接：http://arxiv.org/pdf/2303.07624v1
作者：Yifan Peng,Jaesong Lee,Shinji Watanabe机构：Carnegie Mellon University, NAVER Corporation备注：Accepted at ICASSP 2023摘要：基于变换器的端到端语音识别已经取得了很大的成功。然而，大量的内存占用和计算开销使得这些模型很难在一些实际应用中部署。模型压缩技术可以减小模型大小和加快推理速度，但压缩后的模型具有固定的结构，这可能是次优的。提出了一种新的基于输入相关动态深度（I3 D）的Transformer编码器，以实现性能-效率的折衷。在推理时具有相似数量的层的情况下，基于I3 D的模型优于vanilla Transformer和通过迭代层修剪的静态修剪模型。我们还对门概率和输入依赖性进行了有趣的分析，这有助于我们更好地理解深度编码器。摘要：Transformer-based end-to-end speech recognition has achieved great success. However, the large footprint and computational overhead make it difficult to deploy these models in some real-world applications. Model compression techniques can reduce the model size and speed up inference, but the compressed model has a fixed architecture which might be suboptimal. We propose a novel Transformer encoder with Input-Dependent Dynamic Depth (I3D) to achieve strong performance-efficiency trade-offs. With a similar number of layers at inference time, I3D-based models outperform the vanilla Transformer and the static pruned model via iterative layer pruning. We also present interesting analysis on the gate probabilities and the input-dependency, which helps us better understand deep encoders.【13】 VANI: Very-lightweight Accent-controllable TTS for Native and Non-native speakers with Identity Preservation标题：Vani：非常轻量级、口音可控的TTS，适用于母语和非母语人士，具有身份保留功能链接：http://arxiv.org/pdf/2303.07578v1
作者：Rohan Badlani,Akshit Arora,Subhankar Ghosh,Rafael Valle,Kevin J. Shih,João Felipe Santos,Boris Ginsburg,Bryan Catanzaro机构：NVIDIA备注：Presentation accepted at ICASSP 2023摘要：本文介绍了一个轻量级的多语言重音可控语音合成系统VANI。该模型基于RADMMM提出的去纠缠策略，支持重音、语言、说话人以及细粒度的F0和能量特征的显式控制。我们利用印度语言数据集，作为ICASSP信号处理大挑战赛的一部分，为LIMMITS 2023发布，合成3种不同语言的语音。我们的模型支持转换说话者的语言，同时保留他们的声音和目标语言的本地口音。我们对赛道$1$采用大参数RADMMM模型，对赛道$2$和$3$采用轻量级VANI模型。摘要：We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system. Our model builds upon disentanglement strategies proposed in RADMMM and supports explicit control of accent, language, speaker and fine-grained $F_0$ and energy features for speech synthesis. We utilize the Indic languages dataset, released for LIMMITS 2023 as part of ICASSP Signal Processing Grand Challenge, to synthesize speech in 3 different languages. Our model supports transferring the language of a speaker while retaining their voice and the native accent of the target language. We utilize the large-parameter RADMMM model for Track $1$ and lightweight VANI model for Track $2$ and $3$ of the competition.【14】 HiSSNet: Sound Event Detection and Speaker Identification via Hierarchical Prototypical Networks for Low-Resource Headphones标题：HiSSNet：基于分层原型网络的低资源耳机声音事件检测与说话人识别链接：http://arxiv.org/pdf/2303.07538v1
作者：N Shashaank,Berker Banar,Mohammad Rasool Izadi,Jeremy Kemmerer,Shuo Zhang,Chuan-Che,Huang机构：Bose Corporation, MA, USA, Department of Computer Science, Columbia University, NY, USA, School of Electronic Engineering and Computer Science, Queen Mary University of London, UK摘要：现代的降噪耳机通过去除不必要的背景噪音，显著改善了用户的听觉体验，但它们也可以屏蔽掉对用户重要的声音。用于声音事件检测（SED）和说话人识别（SID）的机器学习（ML）模型可以使耳机选择性地通过重要声音;然而，为以用户为中心的体验实现这些模型提出了几个独特的挑战。首先，大多数人花有限的时间定制他们的耳机，所以声音检测应该工作得相当好的开箱即用。其次，模型应该能够随着时间的推移，根据用户的隐式和显式交互，学习对用户重要的特定声音。最后，这类型号应该有一个小的内存占用，以运行在低功耗耳机与有限的片上内存。本文提出使用HiSSNet（Hierarchical SED and SID Network）来解决这些问题。HiSSNet是一种SEID（SED和SID）模型，它使用分层原型网络来检测感兴趣的一般声音和特定声音，并表征警报类声音和语音声音。我们证明HiSSNet比使用非层次原型网络训练的SEID模型性能好6.9 - 8.6%。与专门针对SED或SID单独训练的最先进（SOTA）模型相比，HiSSNet可实现类似或更好的性能，同时减少支持设备上多种功能所需的内存占用。摘要：Modern noise-cancelling headphones have significantly improved users' auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to selectively pass through important sounds; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that are important to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet (Hierarchical SED and SID Network). HiSSNet is an SEID (SED and SID) model that uses a hierarchical prototypical network to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms an SEID model trained using non-hierarchical prototypical networks by 6.9 - 8.6 percent. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.【15】 Localizing Spatial Information in Neural Spatiospectral Filters标题：神经空间谱滤波中的空间信息定位链接：http://arxiv.org/pdf/2303.08052v1
作者：Annika Briegleb,Thomas Haubner,Vasileios Belagiannis,Walter Kellermann机构：Multimedia Communications and Signal Processing, Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Erlangen, Germany备注：Submitted to the 31st European Signal Processing Conference (EUSIPCO 2023), Helsinki, Finland. 5 pages, 3 figures摘要：用于多通道语音增强的波束形成依赖于对声学场景的空间特性的估计。在最简单的形式中，延迟求和波束形成器（DSB）向所有通道引入时间延迟，以对齐所需信号分量进行相长叠加。最近对神经空间谱滤波的研究表明，这些滤波器可以由类似于传统波束形成器的波束图来表征，这表明人工神经网络可以学习并显式地表示空间结构。使用复值空间自动编码器（COSPA）作为多通道语音增强的示例性神经空间频谱滤波器，我们研究了这种网络在哪里以及如何表示空间信息。我们通过聚类表明，对于COSPA，空间信息由同时访问所有信道的门控递归单元（GRU）层生成的特征表示，并且这些特征不是源，而是仅依赖于到达方向。摘要：Beamforming for multichannel speech enhancement relies on the estimation of spatial characteristics of the acoustic scene. In its simplest form, the delay-and-sum beamformer (DSB) introduces a time delay to all channels to align the desired signal components for constructive superposition. Recent investigations of neural spatiospectral filtering revealed that these filters can be characterized by a beampattern similar to one of traditional beamformers, which shows that artificial neural networks can learn and explicitly represent spatial structure. Using the Complex-valued Spatial Autoencoder (COSPA) as an exemplary neural spatiospectral filter for multichannel speech enhancement, we investigate where and how such networks represent spatial information. We show via clustering that for COSPA the spatial information is represented by the features generated by a gated recurrent unit (GRU) layer that has access to all channels simultaneously and that these features are not source -- but only direction of arrival-dependent.
【16】 A Hierarchical Regression Chain Framework for Affective Vocal Burst Recognition标题：一种用于情感突发语音识别的层次回归链框架链接：http://arxiv.org/pdf/2303.08027v1
作者：Jinchao Li,Xixin Wu,Kaitao Song,Dongsheng Li,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Microsoft Research Asia, Shanghai, China备注：5 pages, 3 figures, 5 tables摘要：爆发式发声作为一种常见的非语言发声表达情感的方式，在日常社会交往中发挥着重要作用。理解和建模人类的发声是发展鲁棒的和通用的人工智能不可或缺的。探索语音爆发的计算方法正引起越来越多的研究关注。在这项工作中，我们提出了一个层次框架，基于链回归模型，用于从VB中进行情感识别，它明确地考虑了多种关系：（i）情感状态和不同文化之间的差异;（ii）在低维（唤醒&效价）和高维（10个情感类别）情感空间之间;以及（iii）在高维空间内的各种情感类别之间。为了应对数据稀疏性的挑战，我们还使用了自监督学习（SSL）表示法，以及逐层和时态聚合模块。申报系统参加了2022年ACII情感爆发式发声（A-VB）挑战赛，并在“Two '”和“Culture'”任务中排名第一。基于ACII Challenge 2022数据集的实验结果验证了所提系统的优越性能和使用层次回归链模型考虑多重关系的有效性。摘要：As a common way of emotion signaling via non-linguistic vocalizations, vocal burst (VB) plays an important role in daily social interaction. Understanding and modeling human vocal bursts are indispensable for developing robust and general artificial intelligence. Exploring computational approaches for understanding vocal bursts is attracting increasing research attention. In this work, we propose a hierarchical framework, based on chain regression models, for affective recognition from VBs, that explicitly considers multiple relationships: (i) between emotional states and diverse cultures; (ii) between low-dimensional (arousal & valence) and high-dimensional (10 emotion classes) emotion spaces; and (iii) between various emotion classes within the high-dimensional space. To address the challenge of data sparsity, we also use self-supervised learning (SSL) representations with layer-wise and temporal aggregation modules. The proposed systems participated in the ACII Affective Vocal Burst (A-VB) Challenge 2022 and ranked first in the "TWO'' and "CULTURE'' tasks. Experimental results based on the ACII Challenge 2022 dataset demonstrate the superior performance of the proposed system and the effectiveness of considering multiple relationships using hierarchical regression chain models.
【17】 Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection标题：利用具有任务相关关键字的预先训练的表示法检测阿尔茨海默病链接：http://arxiv.org/pdf/2303.08019v1
作者：Jinchao Li,Kaitao Song,Junan Li,Bo Zheng,Dongsheng Li,Xixin Wu,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Microsoft Research Asia, Shanghai, China备注：5 pages, 3 figures, 3 tables摘要：随着全球人口迅速老龄化，阿尔茨海默病（AD）在老年人中尤为突出，其具有隐性发病并导致认知领域（记忆、交流等）的逐渐、不可逆恶化。基于言语的AD检测为广泛筛查和及时的疾病干预提供了可能性。预训练模型的最新进展促使AD检测建模从低级特征转移到高级表示。本文提出了几种有效的方法来从高级声学和语言特征中提取更好的广告相关线索。基于这些特点，本文还通过对参与者描述与认知任务之间关系的建模，提出了一种新的任务导向方法。实验在ADReSS数据集上进行，在二进制分类设置中，模型在看不见的测试集上进行评估。实验结果和近期文献的比较表明了所提出的声学、语言学和任务导向方法的有效性和优越性。研究结果还显示了语义和句法信息的重要性，以及使用有前途的纯音频和面向任务的方法进行广告检测任务的自动化和泛化的可行性。摘要：With the global population aging rapidly, Alzheimer's disease (AD) is particularly prominent in older adults, which has an insidious onset and leads to a gradual, irreversible deterioration in cognitive domains (memory, communication, etc.). Speech-based AD detection opens up the possibility of widespread screening and timely disease intervention. Recent advances in pre-trained models motivate AD detection modeling to shift from low-level features to high-level representations. This paper presents several efficient methods to extract better AD-related cues from high-level acoustic and linguistic features. Based on these features, the paper also proposes a novel task-oriented approach by modeling the relationship between the participants' description and the cognitive task. Experiments are carried out on the ADReSS dataset in a binary classification setup, and models are evaluated on the unseen test set. Results and comparison with recent literature demonstrate the efficiency and superior performance of proposed acoustic, linguistic and task-oriented methods. The findings also show the importance of semantic and syntactic information, and feasibility of automation and generalization with the promising audio-only and task-oriented methods for the AD detection task.
【18】 Native Multi-Band Audio Coding within Hyper-Autoencoded Reconstruction Propagation Networks标题：超自动编码重建传播网络中的原生多频带音频编码链接：http://arxiv.org/pdf/2303.08005v1
作者：Darius Petermann,Inseon Jang,Minje Kim机构：Indiana University, Department of Intelligent Systems Engineering, Bloomington, IN , USA, Electronics and Telecommunications Research Institute, Daejeon , South Korea备注：Accepted to ICASSP 2023. For resources and examples, see this https URL摘要：频谱子带不描绘相同的感知相关性。因此，在音频编码中，期望对每个组成频带具有独立的控制，使得可以有效地实现比特率分配和信号重构。在这项工作中，我们提出了一个新的神经音频编码网络，自然支持多频带编码范式。我们的模型扩展了基于U-Net的编解码器中压缩跳跃连接的思想，允许对核心和高频带特定的重建和比特分配进行独立控制。我们的系统主要从压缩的核心频带码重构全频带信号，从而最大限度地开发和展示其带宽扩展能力。同时，低比特率的高带编码有助于高带重构，类似于MPEG音频编解码器的频谱带宽复制。MUSHRA测试表明，该模型不仅通过显式分配更多的比特来提高核心频带的质量，而且在高频段也保持了良好的质量。摘要：Spectral sub-bands do not portray the same perceptual relevance. In audio coding, it is therefore desirable to have independent control over each of the constituent bands so that bitrate assignment and signal reconstruction can be achieved efficiently. In this work, we present a novel neural audio coding network that natively supports a multi-band coding paradigm. Our model extends the idea of compressed skip connections in the U-Net-based codec, allowing for independent control over both core and high band-specific reconstructions and bit allocation. Our system reconstructs the full-band signal mainly from the condensed core-band code, therefore exploiting and showcasing its bandwidth extension capabilities to its fullest. Meanwhile, the low-bitrate high-band code helps the high-band reconstruction similarly to MPEG audio codecs' spectral bandwidth replication. MUSHRA tests show that the proposed model not only improves the quality of the core band by explicitly assigning more bits to it but retains a good quality in the high-band as well.
【19】 Multi-Channel Masking with Learnable Filterbank for Sound Source Separation标题：用于声源分离的带学习滤波器组的多通道掩蔽链接：http://arxiv.org/pdf/2303.07816v1
作者：Wang Dai,Archontis Politis,Tuomas Virtanen机构：Audio Research Group, Tampere University, Tampere, Finland摘要：本文提出了一种基于多通道掩蔽框架的可学习滤波器组，用于多通道信号分离。可学习滤波器组是1D Conv层，它将原始波形转换为2D表示。与传统的单通道掩蔽方法相比，我们为每个单独的麦克风通道估计一个掩蔽。然后，像在传统的滤波和求和波束成形操作中一样，将估计的掩码应用于变换的波形表示。具体而言，每个掩码用于乘以对应通道的2D表示，然后对所有通道的掩码输出求和。最后，使用1D转置Conv层将求和的掩蔽信号转换到波形域。实验结果表明，在STGCSEN模型中，该方法优于单通道可学习滤波器组掩蔽，若将可学习滤波器组变换为更高的特征维数，则优于多通道STFT复谱掩蔽.空间响应分析也验证了可学习滤波器组域中的多通道掩蔽具有空间选择性。摘要：This work proposes a learnable filterbank based on a multi-channel masking framework for multi-channel source separation. The learnable filterbank is a 1D Conv layer, which transforms the raw waveform into a 2D representation. In contrast to the conventional single-channel masking method, we estimate a mask for each individual microphone channel. The estimated masks are then applied to the transformed waveform representation like in the traditional filter-and-sum beamforming operation. Specifically, each mask is used to multiply the corresponding channel's 2D representation, and the masked output of all channels are then summed. At last, a 1D transposed Conv layer is used to convert the summed masked signal into the waveform domain. The experimental results show our method outperforms single-channel masking with a learnable filterbank and can outperform multi-channel complex masking with STFT complex spectrum in the STGCSEN model if a learnable filterbank is transformed to a higher feature dimension. The spatial response analysis also verifies that multi-channel masking in the learnable filterbank domain has spatial selectivity.
【20】 Detecting post-stroke aphasia using EEG-based neural envelope tracking of natural speech标题：基于脑电的自然语音神经包络跟踪检测中风后失语链接：http://arxiv.org/pdf/2303.07739v1
作者：Pieter De Clercq,Jill Kries,Ramtin Mehraram,Jonas Vanthornhout,Tom Francart,Maaike Vandermosten机构：Experimental Oto-Rhino-Laryngology, Department of Neurosciences, Leuven Brain, Institute, KU Leuven, Belgium, Version:, Corresponding author:, arXiv:,.,v, [eess.SP] , Mar摘要：[目的]。中风后，三分之一的患者患有失语症，这是一种损害沟通能力的语言障碍。用于诊断失语症的标准行为测试耗时且生态效度低。语音包络的神经跟踪是研究大脑对自然语音反应的一种有前途的工具。言语包络是言语理解的关键，它包含了处理语言单位的线索。在本研究中，我们旨在测试神经包络追踪技术在失语症患者（IWA）中检测语言障碍的潜力。【办法】。我们记录了27名中风后慢性期的IWA和22名听故事的对照组的脑电图。我们使用互信息分析量化了宽带频率范围以及delta、theta、alpha、beta和gamma频带中的神经包络跟踪。除了神经跟踪测量的组差异外，我们还使用支持向量机（SVM）分类器测试了其检测失语症的适用性。我们进一步研究了SVM检测失语症和获得可靠结果所需的记录长度。[结果]。与对照组相比，IWA在宽、δ、θ和γ波段显示神经包络跟踪减少。这些频带中的神经跟踪在个体水平上有效地捕获失语症（SVM准确率84%，AUC 88%）。记录时间为5-7分钟，检测结果准确可靠。【意义】。我们的研究表明，语言的神经追踪是失语症的有效生物标志物。我们证明了其作为一种诊断工具的潜力，具有高可靠性、个体水平的失语症检测和时间效率评估。这项工作代表了一个重要的一步，更自动，客观，生态有效的评估语言障碍失语症。摘要：[Objective]. After a stroke, one-third of patients suffer from aphasia, a language disorder that impairs communication ability. The standard behavioral tests used to diagnose aphasia are time-consuming and have low ecological validity. Neural tracking of the speech envelope is a promising tool for investigating brain responses to natural speech. The speech envelope is crucial for speech understanding, encompassing cues for processing linguistic units. In this study, we aimed to test the potential of the neural envelope tracking technique for detecting language impairments in individuals with aphasia (IWA). [Approach]. We recorded EEG from 27 IWA in the chronic phase after stroke and 22 controls while they listened to a story. We quantified neural envelope tracking in a broadband frequency range as well as in the delta, theta, alpha, beta, and gamma frequency bands using mutual information analysis. Besides group differences in neural tracking measures, we also tested its suitability for detecting aphasia using a Support Vector Machine (SVM) classifier. We further investigated the required recording length for the SVM to detect aphasia and to obtain reliable outcomes. [Results]. IWA displayed decreased neural envelope tracking compared to controls in the broad, delta, theta, and gamma band. Neural tracking in these frequency bands effectively captured aphasia at the individual level (SVM accuracy 84%, AUC 88%). High-accuracy and reliable detection could be obtained with 5-7 minutes of recording time. [Significance]. Our study shows that neural tracking of speech is an effective biomarker for aphasia. We demonstrated its potential as a diagnostic tool with high reliability, individual-level detection of aphasia, and time-efficient assessment. This work represents a significant step towards more automatic, objective, and ecologically valid assessments of language impairments in aphasia.
【21】 TEA-PSE 3.0: Tencent-Ethereal-Audio-Lab Personalized Speech Enhancement System For ICASSP 2023 DNS Challenge标题：TEA-PSE 3.0：面向ICASSP 2023域名挑战赛的腾讯-Etheral-Audio-Lab个性化语音增强系统链接：http://arxiv.org/pdf/2303.07704v1
作者：Yukai Ju,Jun Chen,Shimin Zhang,Shulin He,Wei Rao,Weixin Zhu,Yannan Wang,Tao Yu,Shidong Shang机构：Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China, Northwestern Polytechnical University, Xi’an, China备注：Accepted by ICASSP 2023摘要：介绍了Unbeatable团队向ICASSP 2023深度噪声抑制（DNS）挑战赛提交的作品。我们将以前的工作TEA-PSE扩展到它的升级版本-- TEA-PSE 3.0。具体而言，TEA-PSE 3.0在压缩时间卷积网络（S-TCN）之后结合了剩余LSTM，以增强层序建模能力。此外，引入局部-全局表示（LGR）结构以提高说话人信息提取效率，并利用多STFT分辨率损失有效捕捉语音信号的时频特征。此外，在冻结训练策略的基础上采用再训练方法对系统进行微调。根据官方公布的结果，TEA-PSE 3.0在ICASSP 2023 DNS挑战赛第一和第二赛道均排名第一。摘要：This paper introduces the Unbeatable Team's submission to the ICASSP 2023 Deep Noise Suppression (DNS) Challenge. We expand our previous work, TEA-PSE, to its upgraded version -- TEA-PSE 3.0. Specifically, TEA-PSE 3.0 incorporates a residual LSTM after squeezed temporal convolution network (S-TCN) to enhance sequence modeling capabilities. Additionally, the local-global representation (LGR) structure is introduced to boost speaker information extraction, and multi-STFT resolution loss is used to effectively capture the time-frequency characteristics of the speech signals. Moreover, retraining methods are employed based on the freeze training strategy to fine-tune the system. According to the official results, TEA-PSE 3.0 ranks 1st in both ICASSP 2023 DNS-Challenge track 1 and track 2.
【22】 Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge标题：用于ICASSP 2023语音信号改善挑战的两级神经网络链接：http://arxiv.org/pdf/2303.07621v1
作者：Mingshuai Liu,Shubo Lv,Zihan Zhang,Runduo Han,Xiang Hao,Xianjun Xia,Li Chen,Yijian Xiao,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, ByteDance, China备注：Accepted by ICASSP 2023摘要：在ICASSP 2023语音信号改善挑战赛中，我们开发了一个双阶段神经模型，该模型以阶段分治的方式改善了由不同失真引起的语音信号质量。具体地，在第一阶段，语音改善网络集中于恢复频谱的缺失分量，而在第二阶段，我们的模型旨在进一步抑制由第一阶段模型引入的噪声、混响和伪像。最终得分为0.446，P.835得分为0.517，在非实时领域排名第四。摘要：In ICASSP 2023 speech signal improvement challenge, we developed a dual-stage neural model which improves speech signal quality induced by different distortions in a stage-wise divide-and-conquer fashion. Specifically, in the first stage, the speech improvement network focuses on recovering the missing components of the spectrum, while in the second stage, our model aims to further suppress noise, reverberation, and artifacts introduced by the first-stage model. Achieving 0.446 in the final score and 0.517 in the P.835 score, our system ranks 4th in the non-real-time track.
【23】 Lightweight feature encoder for wake-up word detection based on self-supervised speech representation标题：基于自监督语音表示的轻量级唤醒词检测特征编码器链接：http://arxiv.org/pdf/2303.07592v1
作者：Hyungjun Lim,Younggwan Kim,Kiho Yeom,Eunjoo Seo,Hoodong Lee,Stanley Jungkyu Choi,Honglak Lee机构：LG AI Research, Seoul, Republic of Korea备注：Accepted by ICASSP 2023摘要：提供广义语音表示的自监督学习方法近年来受到越来越多的关注。Wav2vec 2.0是最著名的例子，在众多下游语音处理任务中表现出了卓越的性能。尽管它取得了成功，但由于其昂贵的计算成本，直接将其用于移动设备上的唤醒词检测是具有挑战性的。在本文中，我们提出了LiteFEW，一个轻量级的特征编码器，用于唤醒字检测，它以最小的规模保留了wav2vec 2.0的固有能力。该方法通过引入基于自动编码器的降维技术，对预先训练好的wav2vec2. 0进行知识压缩，并提取到LiteFEW中.在开源“Hey Snips”数据集上的实验结果表明，所提方法应用于不同模型结构时，仅使用64k个参数，性能相对提高了20%以上。摘要：Self-supervised learning method that provides generalized speech representations has recently received increasing attention. Wav2vec 2.0 is the most famous example, showing remarkable performance in numerous downstream speech processing tasks. Despite its success, it is challenging to use it directly for wake-up word detection on mobile devices due to its expensive computational cost. In this work, we propose LiteFEW, a lightweight feature encoder for wake-up word detection that preserves the inherent ability of wav2vec 2.0 with a minimum scale. In the method, the knowledge of the pre-trained wav2vec 2.0 is compressed by introducing an auto-encoder-based dimensionality reduction technique and distilled to LiteFEW. Experimental results on the open-source "Hey Snips" dataset show that the proposed method applied to various model structures significantly improves the performance, achieving over 20% of relative improvements with only 64k parameters.
【24】 Towards Real-Time Single-Channel Speech Separation in Noisy and Reverberant Environments标题：噪声和混响环境下的实时单通道语音分离链接：http://arxiv.org/pdf/2303.07569v1
作者：Julian Neri,Sebastian Braun机构：McGill University, CIRMMT, Montreal, CA, Microsoft Research, Redmond, USA备注：to appear in ICASSP 2023摘要：实时单通道语音分离旨在将从单个麦克风捕获的包含多个人同时讲话、环境噪声和混响的音频流分解成多个去混响且无噪声的语音轨道，每个轨道仅包含一个讲话者。虽然大型的最先进的DNN可以从语音的无回声混合中实现出色的分离，但主要的挑战是创建可以在推断时间分离混响混合的紧凑和因果模型。在这篇论文中，我们探索低复杂度、资源有效、因果关系的DNN架构，以即时分离两个或多个同时说话人。三个神经网络模块的级联被训练以顺序地执行噪声抑制、分离和去混响。为了比较，训练较大的端到端模型以直接从有噪声的混响语音混合物输出两个无回声语音信号。我们提出一个有效的单解码器架构与减法分离实时递归语音分离两个或多个发言人。根据SI-SDR等语音分离度量、DNS-MOS等感知度量和一种新提出的通道分离度量，对语音混合的真实单声道记录进行评估，结果表明这些紧凑的因果模型能够以低延迟分离语音混合，并且与SepFormer等大型离线最先进模型的性能相当。摘要：Real-time single-channel speech separation aims to unmix an audio stream captured from a single microphone that contains multiple people talking at once, environmental noise, and reverberation into multiple de-reverberated and noise-free speech tracks, each track containing only one talker. While large state-of-the-art DNNs can achieve excellent separation from anechoic mixtures of speech, the main challenge is to create compact and causal models that can separate reverberant mixtures at inference time. In this paper, we explore low-complexity, resource-efficient, causal DNN architectures for real-time separation of two or more simultaneous speakers. A cascade of three neural network modules are trained to sequentially perform noise-suppression, separation, and de-reverberation. For comparison, a larger end-to-end model is trained to output two anechoic speech signals directly from noisy reverberant speech mixtures. We propose an efficient single-decoder architecture with subtractive separation for real-time recursive speech separation for two or more speakers. Evaluation on real monophonic recordings of speech mixtures, according to speech separation measures like SI-SDR, perceptual measures like DNS-MOS, and a novel proposed channel separation metric, show that these compact causal models can separate speech mixtures with low latency, and perform on par with large offline state-of-the-art models like SepFormer.
【25】 Speech Intelligibility Classifiers from 550k Disordered Speech Samples标题：55万无序语音样本中的语音清晰度分类器链接：http://arxiv.org/pdf/2303.07533v1
作者：Subhashini Venugopalan,Jimmy Tobin,Samuel J. Yang,Katie Seaver,Richard J. N. Cave,Pan-Pan Jiang,Neil Zeghidour,Rus Heywood,Jordan Green,Michael P. Brenner机构：Google Research, USA; ,Google Research, France;, MGH Institute of Health Professions, USA; ,Harvard University, USA备注：ICASSP 2023 camera-ready摘要：我们开发了构音障碍语音可懂度分类器，用于551,176个障碍语音样本，这些样本由468名不同的说话者提供，他们有一系列自我报告的说话障碍，并在5分制量表上对其整体可懂度进行评级。我们采用不同的深度学习方法训练了三个模型，并在100位发言者的约94 K次发言中对它们进行了评估。我们进一步发现模型在TORGO数据库（100%准确度）、UASpeech（0.93相关性）、ALS-TDI PMP（0.81 AUC）数据集以及我们收集的真实自发语音数据集（106例构音障碍和76例对照说话者，约2300个样本）上具有良好的泛化性（无需进一步训练）。为了推进这一领域的研究，我们在www.example.com上分享了我们的一个模型https: tfhub.dev google euphonia_spice classification 1。摘要：We developed dysarthric speech intelligibility classifiers on 551,176 disordered speech samples contributed by a diverse set of 468 speakers, with a range of self-reported speaking disorders and rated for their overall intelligibility on a five-point scale. We trained three models following different deep learning approaches and evaluated them on ~94K utterances from 100 speakers. We further found the models to generalize well (without further training) on the TORGO database (100% accuracy), UASpeech (0.93 correlation), ALS-TDI PMP (0.81 AUC) datasets as well as on a dataset of realistic unprompted speech we gathered (106 dysarthric and 76 control speakers,~2300 samples). To advance research in this domain, we share one of our models at https: tfhub.dev google euphonia_spice classification 1.
【26】 Guided Speech Enhancement Network标题：引导式语音增强网络链接：http://arxiv.org/pdf/2303.07486v1
作者：Yang Yang,Shao-Fu Shih,Hakan Erdogan,Jamie Menjay Lin,Chehung Lee,Yunpeng Li,George Sung,Matthias Grundmann机构：Google LLC, U.S.A.备注：Accepted to ICASSP 2023摘要：由于语音通信和人机接口的原因，高质量语音捕获已经被广泛研究。为了提高捕获性能，我们经常发现多麦克风语音增强技术被部署在各种设备上。多麦克风语音增强问题通常被分解为两个解耦的步骤：提供空间滤波的波束形成器和清除波束形成器输出的单通道语音增强模型。在这项工作中，我们提出了一个语音增强解决方案，采取原始麦克风和波束形成器输出作为输入的ML模型。我们设计了一种简单而有效的训练方案，允许模型通过对比两个输入来学习波束形成器的线索，并大大提高其空间抑制能力，同时执行去噪和去混响的一般任务。所提出的解决方案利用了经典的空间滤波算法，而不是与它们竞争。通过设计，波束形成器模块于是可以被单独地选择，并且不需要针对给定的形状因子优化大量数据，并且网络模型可以被认为是独立模块，其可以独立于麦克风阵列而被高度地转移。我们将解决方案中的ML模块命名为GSENet，即Guided Speech Enhancement Network的缩写。我们证明了它在多麦克风设备上收集的真实世界数据中抑制噪声和干扰语音的有效性。摘要：High quality speech capture has been widely studied for both voice communication and human computer interface reasons. To improve the capture performance, we can often find multi-microphone speech enhancement techniques deployed on various devices. Multi-microphone speech enhancement problem is often decomposed into two decoupled steps: a beamformer that provides spatial filtering and a single-channel speech enhancement model that cleans up the beamformer output. In this work, we propose a speech enhancement solution that takes both the raw microphone and beamformer outputs as the input for an ML model. We devise a simple yet effective training scheme that allows the model to learn from the cues of the beamformer by contrasting the two inputs and greatly boost its capability in spatial rejection, while conducting the general tasks of denoising and dereverberation. The proposed solution takes advantage of classical spatial filtering algorithms instead of competing with them. By design, the beamformer module then could be selected separately and does not require a large amount of data to be optimized for a given form factor, and the network model can be considered as a standalone module which is highly transferable independently from the microphone array. We name the ML module in our solution as GSENet, short for Guided Speech Enhancement Network. We demonstrate its effectiveness on real world data collected on multi-microphone devices in terms of the suppression of noise and interfering speech.
【27】Online Binaural Speech Separation of Moving Speakers With a Wavesplit Network标题：基于小波分解网络的运动说话人在线双耳语音分离链接：http://arxiv.org/pdf/2303.07458v1
作者：Cong Han,Nima Mesgarani机构：Department of Electrical Engineering, Columbia University, New York, NY备注：To appear in ICASSP 2023摘要：在真实世界场景中的双耳语音分离通常涉及移动的说话者。目前的语音分离方法大多采用话语级置换不变训练（u-PIT）进行训练。然而，在推理时间中，输出的顺序可能随时间不一致，特别是在长格式语音分离中。当扬声器在空间中不断移动时，被称为扬声器交换问题的这种情况甚至更成问题，并且因此对扬声器在输出声道中的一致放置提出了挑战。本文提出了一种基于小波分解网络的实时双耳语音分离模型，以解决说话人交换问题。该模型根据混合音频计算每个说话人在每个时间帧的说话人嵌入，使用在线聚类聚合嵌入，并使用聚类质心作为说话人简档来跟踪每个说话人的长持续时间。混响、长格式运动多说话人语音分离实验结果表明，该方法不易发生说话人交换，在分离精度和耳间线索保持方面与基于u-PIT模型的地面真值跟踪方法具有可比性。摘要：Binaural speech separation in real-world scenarios often involves moving speakers. Most current speech separation methods use utterance-level permutation invariant training (u-PIT) for training. In inference time, however, the order of outputs can be inconsistent over time particularly in long-form speech separation. This situation which is referred to as the speaker swap problem is even more problematic when speakers constantly move in space and therefore poses a challenge for consistent placement of speakers in output channels. Here, we describe a real-time binaural speech separation model based on a Wavesplit network to mitigate the speaker swap problem for moving speaker separation. Our model computes a speaker embedding for each speaker at each time frame from the mixed audio, aggregates embeddings using online clustering, and uses cluster centroids as speaker profiles to track each speaker throughout the long duration. Experimental results on reverberant, long-form moving multitalker speech separation show that the proposed method is less prone to speaker swap and achieves comparable performance with u-PIT based models with ground truth tracking in both separation accuracy and preserving the interaural cues.
【28】 Blind Acoustic Room Parameter Estimation Using Phase Features标题：基于相位特征的声室参数盲估计算法
链接：http://arxiv.org/pdf/2303.07449v1
作者：Christopher Ick,Adib Mehrabi,Wenyu Jin机构：New York University, New York, NY , United States, Sonos Experience Limited, United Kingdom , Sonos, Inc., Boston, MA , United States备注：4 pages + 1 page bibliography, 3 figures, to be published in proceedings of ICASSP 2023摘要：在现场环境中对房间声学进行建模涉及从噪声和混响音频中进行某种程度的盲参数估计。现代方法利用卷积神经网络（CNN）与时频表示相结合。使用短时傅立叶变换来开发这些类似频谱图的特征已经显示出有希望的结果，但是该方法隐含地丢弃了相位域中的大量音频信息。受语音增强研究的启发，本文提出利用新的相位相关特征来扩展现有的方法，盲估计混响指纹参数，即音量和RT60。这些特征的加入被证明优于现有的方法，现有的方法仅仅依赖于在广泛的声学空间范围内的基于震级的频谱特征。我们使用由公开可用的房间脉冲响应（RIR）、合成RIR和真实声学空间的内部测量组成的新型数据集，评估了这些新型特征在单参数和多参数估计策略中部署的有效性。摘要：Modeling room acoustics in a field setting involves some degree of blind parameter estimation from noisy and reverberant audio. Modern approaches leverage convolutional neural networks (CNNs) in tandem with time-frequency representation. Using short-time Fourier transforms to develop these spectrogram-like features has shown promising results, but this method implicitly discards a significant amount of audio information in the phase domain. Inspired by recent works in speech enhancement, we propose utilizing novel phase-related features to extend recent approaches to blindly estimate the so-called "reverberation fingerprint" parameters, namely, volume and RT60. The addition of these features is shown to outperform existing methods that rely solely on magnitude-based spectral features across a wide range of acoustics spaces. We evaluate the effectiveness of the deployment of these novel features in both single-parameter and multi-parameter estimation strategies, using a novel dataset that consists of publicly available room impulse responses (RIRs), synthesized RIRs, and in-house measurements of real acoustic spaces.
【29】 A processing framework to access large quantities of whispered speech found in ASMR标题：一种访问ASMR中发现的大量耳语的处理框架链接：http://arxiv.org/pdf/2303.07442v1
作者：Pablo Perez Zarazaga,Gustav Eje Henter,Zofia Malisz机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：Accepted at ICASSP 2023, 5 pages, 2 figures, 2 tables摘要：耳语是人类日常使用的一种无处不在的交流模式。尽管如此，由于资源和处理方法的短缺，现有语音技术很难提供耳语音。为了解决这一问题，本文提供了一个处理框架，使得能够访问大量和独特的高质量耳语音数据。作为ASMR媒体文化现象的一部分，我们从提交到在线平台的录音中获得数据。我们描述了我们的处理流水线和一种改进的耳语活动检测（WAD）在ASMR数据的方法。为了有效地获得标记的、干净的耳语音，我们通过使用Edyson（一种具有人在环的批量音频注释工具）来补充自动WAD。我们还解决了ASMR特有的一个问题：将耳语与该类型中存在的其它声学触发分离。我们表明，提出的WAD和有效的标记允许建立广泛的扩增数据和训练一个分类器，从ASMR音频提取干净的耳语片段。我们庞大且不断增长的数据集支持具备耳语能力的数据驱动语音技术和语言分析。它还打开了机会，例如，人机交互作为一种资源，可以引起听众的情绪，心理和神经生理反应。摘要：Whispering is a ubiquitous mode of communication that humans use daily. Despite this, whispered speech has been poorly served by existing speech technology due to a shortage of resources and processing methodology. To remedy this, this paper provides a processing framework that enables access to large and unique data of high-quality whispered speech. We obtain the data from recordings submitted to online platforms as part of the ASMR media-cultural phenomenon. We describe our processing pipeline and a method for improved whispered activity detection (WAD) in the ASMR data. To efficiently obtain labelled, clean whispered speech, we complement the automatic WAD by using Edyson, a bulk audio-annotation tool with human-in-the-loop. We also tackle a problem particular to ASMR: separation of whisper from other acoustic triggers present in the genre. We show that the proposed WAD and the efficient labelling allows to build extensively augmented data and train a classifier that extracts clean whisper segments from ASMR audio. Our large and growing dataset enables whisper-capable, data-driven speech technology and linguistic analysis. It also opens opportunities in e.g. HCI as a resource that may elicit emotional, psychological and neuro-physiological responses in the listener.
【30】 Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video标题：深度视觉强制对齐：学习将转录与人脸对话视频对齐链接：http://arxiv.org/pdf/2303.08670v1
作者：Minsu Kim,Chae Won Kim,Yong Man Ro机构：Image and Video Systems Lab, KAIST, South Korea备注：Accepted in AAAI2023摘要：强制对齐指的是将给定的转录与对应的语音时间对齐的技术。然而，由于已经使用语音音频开发了强制对准技术，所以当输入语音音频被噪声破坏或不可访问时，强制对准技术可能对准失败。我们关注于存在可以从其推断语音的另一个组件，语音视频（即，正在说话的面部视频）。针对基于音频的强制对准方法在音频信号条件较差时可以利用视频信息进行补偿的不足，提出了一种基于视频的强制对准方法。然而，与音频强制对准不同，开发可靠的视觉强制对准技术具有挑战性，原因如下两个：1）视觉语音识别（VSR）与基于音频的自动语音识别（ASR）相比具有低得多的性能，以及2）从文本到视频的转换不可靠，因此通常用于建立音频强制对准的方法不能用于开发视觉强制对准。为了解决这些问题，本文提出了一种适合于视觉强制对准的新方法，即深度视觉强制对准（DVFA）。所提出的DVFA可以比对输入转录（即，句子）与正在说话的面部视频进行比较，而不访问语音音频。此外，通过用异常情况检测来扩充对准任务，DVFA可以在执行对准的同时检测输入转录和输入视频之间的失配。因此，即使文本中存在错误单词，我们也可以鲁棒地将文本与说话的面部视频对齐。通过大量的实验，我们证明了所提出的DVFA算法不仅在对齐任务中有效，而且在解释VSR模型的输出方面也是有效的。摘要：Forced alignment refers to a technology that time-aligns a given transcription with a corresponding speech. However, as the forced alignment technologies have developed using speech audio, they might fail in alignment when the input speech audio is noise-corrupted or is not accessible. We focus on that there is another component that the speech can be inferred from, the speech video (i.e., talking face video). Since the drawbacks of audio-based forced alignment can be complemented using the visual information when the audio signal is under poor condition, we try to develop a novel video-based forced alignment method. However, different from audio forced alignment, it is challenging to develop a reliable visual forced alignment technology for the following two reasons: 1) Visual Speech Recognition (VSR) has a much lower performance compared to audio-based Automatic Speech Recognition (ASR), and 2) the translation from text to video is not reliable, so the method typically used for building audio forced alignment cannot be utilized in developing visual forced alignment. In order to alleviate these challenges, in this paper, we propose a new method that is appropriate for visual forced alignment, namely Deep Visual Forced Alignment (DVFA). The proposed DVFA can align the input transcription (i.e., sentence) with the talking face video without accessing the speech audio. Moreover, by augmenting the alignment task with anomaly case detection, DVFA can detect mismatches between the input transcription and the input video while performing the alignment. Therefore, we can robustly align the text with the talking face video even if there exist error words in the text. Through extensive experiments, we show the effectiveness of the proposed DVFA not only in the alignment task but also in interpreting the outputs of VSR models.
【31】 Blind Estimation of Audio Processing Graph标题：音频处理图的盲估计算法链接：http://arxiv.org/pdf/2303.08610v1
作者：Sungho Lee,Jaehyun Park,Seungryeol Paik,Kyogu Lee机构：Department of Intelligence and Information, Seoul National University, Interdisciplinary Program in Artificial Intelligence, Seoul National University, Artificial Intelligence Institute, Seoul National University备注：Accepted to ICASSP 2023摘要：音乐家和音频工程师通过连接多个处理器来塑造和转换声音，形成一个音频处理图。然而，大多数深度学习方法忽略了这种现实世界的实践，并假设固定的图形设置。为了弥补这一差距，我们开发了一个系统，重建整个图形从一个给定的参考音频。我们首先产生一个真实的图参考对数据集，并训练一个简单的盲估计系统，该系统由卷积参考编码器和基于变换的图解码器组成。我们将我们的模型应用于歌唱声音效果和鼓混合估计任务。仿真结果表明，该方法能够重构复杂的信号路由，包括多频带处理和边链处理。摘要：Musicians and audio engineers sculpt and transform their sounds by connecting multiple processors, forming an audio processing graph. However, most deep-learning methods overlook this real-world practice and assume fixed graph settings. To bridge this gap, we develop a system that reconstructs the entire graph from a given reference audio. We first generate a realistic graph-reference pair dataset and train a simple blind estimation system composed of a convolutional reference encoder and a transformer-based graph decoder. We apply our model to singing voice effects and drum mixing estimation tasks. Evaluation results show that our method can reconstruct complex signal routings, including multi-band processing and sidechaining.
【32】 PHONEix: Acoustic Feature Processing Strategy for Enhanced Singing Pronunciation with Phoneme Distribution Predictor标题：Phoneix：音素分布预测器增强歌唱发音的声学特征处理策略链接：http://arxiv.org/pdf/2303.08607v1
作者：Yuning Wu,Jiatong Shi,Tao Qian,Dongji Gao,Qin Jin机构：School of Information, Renmin University of China, Language Technologies Institute, Carnegie Mellon University, Center for Language and Speech Processing, Johns Hopkins University备注：Accepted by ICASSP 2023摘要：歌唱声合成作为一项从乐谱中生成歌唱声的具体任务，近年来引起了广泛的关注。SVS面临着在同一乐谱条件下，歌唱发音灵活性变化的挑战。SVS之前的大部分作品都不能很好地处理乐谱和实际演唱之间的错位。本文提出了一种带有音素分布预测器的声学特征处理策略PHONEix，以减小乐谱和人声之间的差距，该策略可以方便地应用于不同的SVS系统。在各种环境下进行的大量实验证明了PHONEix在客观和主观评价方面的有效性。摘要：Singing voice synthesis (SVS), as a specific task for generating the vocal singing voice from a music score, has drawn much attention in recent years. SVS faces the challenge that the singing has various pronunciation flexibility conditioned on the same music score. Most of the previous works of SVS can not well handle the misalignment between the music score and actual singing. In this paper, we propose an acoustic feature processing strategy, named PHONEix, with a phoneme distribution predictor, to alleviate the gap between the music score and the singing voice, which can be easily adopted in different SVS systems. Extensive experiments in various settings demonstrate the effectiveness of our PHONEix in both objective and subjective evaluations.
【33】 Enhancing Unsupervised Audio Representation Learning via Adversarial Sample Generation标题：通过对抗性样本生成增强无监督音频表示学习链接：http://arxiv.org/pdf/2303.08561v1
作者：Yulin Pan,Xiangteng He,Biao Gong,Yuxin Peng,Yiliang Lv机构：Alibaba Group, Hangzhou, China, Wangxuan Institute of Computer Technology, Peking University, Beijing, China备注：8 pages, 4 figures摘要：现有的音频分析方法一般先将音频流转换为声谱图，然后送入CNN进行进一步分析。标准的CNN在特征地图上识别特定的视觉模式，然后进行高级表示，该高级表示忽略了识别模式的位置信息。然而，与自然图像不同，音频谱图的语义对位置变化敏感，因为其垂直轴和水平轴指示音频的频率和时间信息，而不是简单的直角坐标。因此，CNN对位置变化的不敏感性对音频谱图编码起到了负面作用。为了解决这个问题，本文提出了一种新的自监督学习机制，其通过首先生成对抗样本（ textit{即}，负样本），然后驱动CNN来区分负对在潜在空间中的嵌入。大量实验表明，与已有方法相比，该方法在9个下游数据集上取得了最优或有竞争力的结果，验证了其在音频表示学习中的有效性。摘要：Existing audio analysis methods generally first transform the audio stream to spectrogram, and then feed it into CNN for further analysis. A standard CNN recognizes specific visual patterns over feature map, then pools for high-level representation, which overlooks the positional information of recognized patterns. However, unlike natural image, the semantic of an audio spectrogram is sensitive to positional change, as its vertical and horizontal axes indicate the frequency and temporal information of the audio, instead of naive rectangular coordinates. Thus, the insensitivity of CNN to positional change plays a negative role on audio spectrogram encoding. To address this issue, this paper proposes a new self-supervised learning mechanism, which enhances the audio representation by first generating adversarial samples ( textit{i.e.}, negative samples), then driving CNN to distinguish the embeddings of negative pairs in the latent space. Extensive experiments show that the proposed approach achieves best or competitive results on 9 downstream datasets compared with previous methods, which verifies its effectiveness on audio representation learning.【34】 Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring标题：观看或收听：具有视觉腐败建模和可靠性评分的健壮视听语音识别链接：http://arxiv.org/pdf/2303.08536v1
作者：Joanna Hong,Minsu Kim,Jeongsoo Choi,Yong Man Ro机构：Image and Video Systems Lab, KAIST备注：Accepted at CVPR 2023. Implementation available: this https URL摘要：本文研究了多模态输入错误情况下的音视频语音识别（AVSR）问题，即音频和视频输入同时受到错误影响的情况，这一问题在以往的研究方向中没有得到很好的解决。先前的研究集中在如何在干净的视觉输入可用的假设下用干净的视觉输入来补充损坏的音频输入。然而，在现实生活中，干净的视觉输入并不总是可获得的，并且甚至可能被遮挡的嘴唇区域或噪声破坏。因此，我们首先分析了与单模态模型相比，以前的AVSR模型对多模态输入流（音频和视频输入）的破坏并不真正具有鲁棒性。然后，我们设计了多模态输入损坏模型来发展强健的AVSR模型。最后，我们提出了一个新的AVSR框架，即音视频可靠性评分模块（AV-RelScore），它对多模态输入具有鲁棒性。AV-RelScore可以确定哪个输入模态流对于预测是可靠的或不可靠的，并且还可以在预测中利用更可靠的流。在LRS 2和LRS 3两个主流数据库上的综合实验验证了该方法的有效性。结果表明，AV-RelScore得到的可靠性分数能很好地反映多模态表示的破坏程度，使模型更关注可靠的多模态表示.摘要：This paper deals with Audio-Visual Speech Recognition (AVSR) under multimodal input corruption situations where audio inputs and visual inputs are both corrupted, which is not well addressed in previous research directions. Previous studies have focused on how to complement the corrupted audio inputs with the clean visual inputs with the assumption of the availability of clean visual inputs. However, in real life, clean visual inputs are not always accessible and can even be corrupted by occluded lip regions or noises. Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models. Then, we design multimodal input corruption modeling to develop robust AVSR models. Lastly, we propose a novel AVSR framework, namely Audio-Visual Reliability Scoring module (AV-RelScore), that is robust to the corrupted multimodal inputs. The AV-RelScore can determine which input modal stream is reliable or not for the prediction and also can exploit the more reliable streams in prediction. The effectiveness of the proposed method is evaluated with comprehensive experiments on popular benchmark databases, LRS2 and LRS3. We also show that the reliability scores obtained by AV-RelScore well reflect the degree of corruption and make the proposed model focus on the reliable multimodal representations.
【35】 Generating symbolic music using diffusion models标题：使用扩散模型生成符号音乐链接：http://arxiv.org/pdf/2303.08385v1
作者：Lilac Atassi机构：University of California, San Diego摘要：概率去噪扩散模型已经成为简单但非常强大的生成模型。扩散模型不像其他生成模型那样遭受模式崩溃，也不需要鉴别器来生成高质量样本。在这篇文章中，我们提出了一个扩散模型，使用二项式先验分布来产生钢琴卷。文中还提出了一种有效的模型训练和样本生成方法。所生成的音乐在上至训练钢琴卷片段的长度的时间尺度上具有连贯性。我们展示了这样一个模型是如何受输入条件的影响，并可以用来协调一个给定的旋律，完成一个不完整的钢琴卷或产生一个给定的作品的变化。代码是公开共享的，以鼓励社区使用和开发该方法。摘要：Probabilistic Denoising Diffusion models have emerged as simple yet very powerful generative models. Diffusion models unlike other generative models do not suffer from mode collapse nor require a discriminator to generate high quality samples. In this paper, we propose a diffusion model that uses a binomial prior distribution to generate piano-rolls. The paper also proposes an efficient method to train the model and generate samples. The generated music has coherence at time scales up to the length of the training piano-roll segments. We show how such a model is conditioned on the input and can be used to harmonize a given melody, complete an incomplete piano-roll or generate a variation of a given piece. The code is shared publicly to encourage the use and development of the method by the community.
【36】 Transfer Learning Based Diagnosis and Analysis of Lung Sound Aberrations标题：基于迁移学习的肺音异常诊断与分析链接：http://arxiv.org/pdf/2303.08362v1
作者：Hafsa Gulzar,Jiyun Li,Arslan Manzoor,Sadaf Rehmat,Usman Amjad,Hadiqa Jalil Khan机构：Computer Science and Technology, Donghua University, Shanghai, China, Mathematics and Computer Science, University of Catania, Catania, Italy, Computer Science Department, PIEAS, Islamabad, Pakistan备注：12 pages, 9 figures摘要：随着能够收集和分析大量数据的计算机系统的发展，医学界正在建立几种非侵入性工具。这项工作试图开发一种非侵入性技术，用于通过机器学习技术识别由听诊器和语音记录软件采集的呼吸音。本研究提出了一种经过训练和验证的基于神经网络的呼吸音分类方法。构造每个音频样本的视觉表示，从而允许使用类似于用于有效描述视觉的那些方法的方法来进行分类的资源标识。我们使用了一种叫做梅尔倒谱系数（MFCC）的技术。这里，通过VGG16（迁移学习）检索和分类特征，并使用5重交叉验证完成预测。呼吸音数据库采用多种数据分割技术，取得了最先进的结果，准确率为95%，精确度为88%，召回率为86%，F1评分为81%。ICBHI数据集用于训练和测试模型。摘要：With the development of computer -systems that can collect and analyze enormous volumes of data, the medical profession is establishing several non-invasive tools. This work attempts to develop a non-invasive technique for identifying respiratory sounds acquired by a stethoscope and voice recording software via machine learning techniques. This study suggests a trained and proven CNN-based approach for categorizing respiratory sounds. A visual representation of each audio sample is constructed, allowing resource identification for classification using methods like those used to effectively describe visuals. We used a technique called Mel Frequency Cepstral Coefficients (MFCCs). Here, features are retrieved and categorized via VGG16 (transfer learning) and prediction is accomplished using 5-fold cross-validation. Employing various data splitting techniques, Respiratory Sound Database obtained cutting-edge results, including accuracy of 95%, precision of 88%, recall score of 86%, and F1 score of 81%. The ICBHI dataset is used to train and test the model.
【37】 Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs标题：基于视觉输入和参与者链接输入的多模式融合的自主声景增强链接：http://arxiv.org/pdf/2303.08342v1
作者：Kenneth Ooi,Karn N. Watcharasupat,Bhan Lam,Zhen-Ting Ong,Woon-Seng Gan机构：School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Center for Music Technology, Georgia Institute of Technology, Atlanta, GA, USA备注：5 pages, 2 figures. Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：自主声景增强系统通常使用经训练的模型来挑选最佳掩蔽器以实现期望的感知改变。虽然声学信息对于这样的系统是至关重要的，但是包括参与者人口统计和视觉环境的上下文信息也影响声学感知。因此，我们建议对现有的基于注意力的深度神经网络进行模块化修改，以允许参与者关联、视觉和声学特征的早期、中期和晚期特征融合。使用ARAUS数据集对模块配置和相应融合方法进行的消融研究表明，上下文特征在统计学上显著改善了标准化ISO愉悦度的模型性能，性能最佳的全模态模型的均方误差为$0.1194 pm0.0012$，而纯音频模型的均方误差为$0.1217 pm0.0009$。声景增强系统由此可以利用多模态输入来改进性能。我们还调查了个别参与者相关因素的影响，使用训练模型来说明模型解释能力的提高。摘要：Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of $0.1194 pm0.0012$ for the best-performing all-modality model, against $0.1217 pm0.0009$ for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability.
【38】 Cross-speaker Emotion Transfer by Manipulating Speech Style Latents标题：操纵语体潜伏期的跨说话人情绪传递链接：http://arxiv.org/pdf/2303.08329v1
作者：Suhee Jo,Younggun Lee,Yookyung Shin,Yeongtae Hwang,Taesu Kim机构：Neosapience, Inc.备注：accepted to ICASSP 2023摘要：近年来，情感化文本语音转换取得了长足的发展。但是，它需要大量的标记数据，这些数据不容易访问。即使可以获取情感语音数据集，在控制情感强度方面仍然存在限制。在本论文中，我们提出了一种在潜在风格空间中使用向量运算的跨说话人情感传递和操纵的新方法。通过只利用少量的标记样本，我们在不丢失说话人身份的情况下从阅读风格的语音中生成情感语音。此外，情感强度可使用标量值容易地控制，从而为用户提供了操纵语音的直观方式。实验结果表明，该方法在表达性、自然性、可控性等方面具有较好的性能，并能保持说话人身份。摘要：In recent years, emotional text-to-speech has shown considerable progress. However, it requires a large amount of labeled data, which is not easily accessible. Even if it is possible to acquire an emotional speech dataset, there is still a limitation in controlling emotion intensity. In this work, we propose a novel method for cross-speaker emotion transfer and manipulation using vector arithmetic in latent style space. By leveraging only a few labeled samples, we generate emotional speech from reading-style speech without losing the speaker identity. Furthermore, emotion strength is readily controllable using a scalar value, providing an intuitive way for users to manipulate speech. Experimental results show the proposed method affords superior performance in terms of expressiveness, naturalness, and controllability, preserving speaker identity.
【39】 Chat with the Environment: Interactive Multimodal Perception using Large Language Models标题：与环境聊天：使用大型语言模型的交互式多通道感知链接：http://arxiv.org/pdf/2303.08268v1
作者：Xufeng Zhao,Mengdi Li,Cornelius Weber,Muhammad Burhan Hafez,Stefan Wermter机构：Universit¨at Hamburg备注：See website at this https URL摘要：在复杂的世界中，编程机器人行为面临着多个层面的挑战，从灵巧的低级技能到高级规划和推理。最近提出的预训练大语言模型（LLM）在zero-shot机器人规划中表现出了卓越的推理能力。然而，要使LLM具备多模式感觉输入和连续动作输出的能力，同时使机器人能够与环境交互，并随着其策略的展开获取新信息，仍然是一个挑战。我们开发了一个具有部分可观察状态的机器人交互场景，它需要机器人决定一系列认知行为，以便在能够正确执行任务之前从多个模态中采样感觉信息。因此，提出了一种以LLM为主干的交互式感知框架，利用LLM的能力指导认知行为，对产生的多模态感觉（视觉、声音、触觉、本体感觉）进行推理，并基于交互式获取的信息规划整个任务执行。我们的研究表明，LLM能够提供高水平的规划和推理技能，并在多模态环境中控制交互式机器人行为，而具有环境状态上下文的多模态模块有助于LLM扎根并扩展其处理能力。摘要：Programming robot behaviour in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in zero-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. An interactive perception framework is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behaviour in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability.
【40】Facilitating deep acoustic phenotyping: A basic coding scheme of infant vocalisations preluding computational analysis, machine learning and clinical reasoning标题：促进深层声学表型：包括计算分析、机器学习和临床推理的婴幼儿发声的基本编码方案链接：http://arxiv.org/pdf/2303.08239v1
作者：Tomas Kulvicius,Sigrun Lang,Claudius AA Widmann,Nina Hansmann,Daniel Holzinger,Luise Poustka,Dajie Zhang,Peter B Marschik机构：Klinik für Kinder- und Jugendpsychiatrie und Psychotherapie, Universitätsmedizin Göttingen, Abteilung für Computational Neuroscience, Drittes Physikalisches Institut, Forschungsinstitut für Entwicklungsmedizin, Johannes Kepler Universität, Linz, Österreich备注：This paper is in German摘要：理论背景：早期的语言发展还没有被完全理解，特别是在它的形成阶段。研究问题：能否定义一种可靠、易于使用的编码方案，用于早期婴儿发声的分类，并可用作进一步分析语言发展的基础？方法：在一项对45名神经典型婴儿的纵向研究中，我们分析了出生后前4个月的发声情况。音频片段被分配到5个类别：（1）浊音化和（2）清音化;（3）定义信号;（4）非目标性;（5）不可转让。结果：两名经验不同的女性编码员在未经强化培训的情况下获得了较高的一致性。讨论和结论：该可靠的方案可用于研究和临床设置中，用于有效地编码婴儿发声，作为详细的人工和机器分析的基础。摘要：Theoretical background: early verbal development is not yet fully understood, especially in its formative phase. Research question: can a reliable, easy-to-use coding scheme for the classification of early infant vocalizations be defined that is applicable as a basis for further analysis of language development? Methods: in a longitudinal study of 45 neurotypical infants, we analyzed vocalizations of the first 4 months of life. Audio segments were assigned to 5 classes: (1) Voiced and (2) Voiceless vocalizations; (3) Defined signal; (4) Non-target; (5) Nonassignable. Results: Two female coders with different experience achieved high agreement without intensive training. Discussion and Conclusion: The reliable scheme can be used in research and clinical settings for efficient coding of infant vocalizations, as a basis for detailed manual and machine analyses.
【41】 Beamformer-Guided Target Speaker Extraction标题：波束形成器引导的目标说话人提取
链接：http://arxiv.org/pdf/2303.08702v1
作者：Mohamed Elminshawi,Srikanth Raj Chetupalli,Emanuël A. P. Habets机构：International Audio Laboratories Erlangen†, Am Wolfsmantel , Erlangen, Germany备注：Submitted to the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：提出了一种基于波束形成器引导的目标说话人提取（BG-TSE）方法，用于从多通道录音中提取目标说话人的语音，该方法可以根据目标说话人的到达方向来确定目标说话人。所提出的方法采用朝向目标说话人操纵的前端波束形成器来向单通道TSE系统提供辅助信号。通过允许在单通道TSE块中进行时变嵌入，所提出的方法充分利用了前端波束形成器输出与麦克风信号中的目标语音之间的对应性。在消声和混响条件下对模拟的多通道2-说话人混合声进行了实验评估，结果表明该方法相对于现有的单通道和多通道基线方法具有优势。摘要：We propose a Beamformer-guided Target Speaker Extraction (BG-TSE) method to extract a target speaker's voice from a multi-channel recording informed by the direction of arrival of the target. The proposed method employs a front-end beamformer steered towards the target speaker to provide an auxiliary signal to a single-channel TSE system. By allowing for time-varying embeddings in the single-channel TSE block, the proposed method fully exploits the correspondence between the front-end beamformer output and the target speech in the microphone signal. Experimental evaluation on simulated multi-channel 2-speaker mixtures, in both anechoic and reverberant conditions, demonstrates the advantage of the proposed method compared to recent single-channel and multi-channel baselines.
【42】 Speech Signal Improvement Using Causal Generative Diffusion Models标题：基于因果生成扩散模型的语音信号改善链接：http://arxiv.org/pdf/2303.08674v1
作者：Julius Richter,Simon Welker,Jean-Marie Lemercier,Bunlong Lay,Tal Peer,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany备注：Accepted by ICASSP 2023摘要：在本文中，我们提出了一个因果语音信号改善系统，旨在处理不同类型的失真。该方法基于生成扩散模型，该模型已被证明在具有缺失数据和非线性损坏的场景中工作良好。为了保证因果处理，我们修改了我们先前工作的网络结构，用因果自适应增益控制代替全局归一化。我们生成包含广泛失真的各种训练数据。这项工作是在“ICASSP信号处理大挑战赛”的背景下进行的，并提交到“语音信号改进挑战赛2023”的非实时赛道，在那里排名第五。摘要：In this paper, we present a causal speech signal improvement system that is designed to handle different types of distortions. The method is based on a generative diffusion model which has been shown to work well in scenarios with missing data and non-linear corruptions. To guarantee causal processing, we modify the network architecture of our previous work and replace global normalization with causal adaptive gain control. We generate diverse training data containing a broad range of distortions. This work was performed in the context of an "ICASSP Signal Processing Grand Challenge" and submitted to the non-real-time track of the "Speech Signal Improvement Challenge 2023", where it was ranked fifth.
【43】 HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism标题：HYBRIDFORMER：用混合注意和NSR机制改进SqueezeFormer链接：http://arxiv.org/pdf/2303.08636v1
作者：Yuguang Yang,Yu Pan,Jingjing Yin,Jiangyu Han,Lei Ma,Heng Lu机构：Ximalaya Inc., ShangHai, China, University of Alberta, Edmonton, Canada备注：Accepted by ICASSP2023摘要：SqueezeFormer最近在自动语音识别（ASR）方面表现出了令人印象深刻的性能。然而，其推理速度受到softmax-attention（SA）算法二次复杂度的影响。此外，受卷积核尺寸较大的限制，SqueezeFormer的局部建模能力不足。本文提出了一种新的HybridFormer方法来快速有效地改进SqueezeFormer。具体来说，我们首先引入线性注意（LA），并提出一种混合LASA范式来提高模型的推理速度。其次，提出了一种混合神经结构搜索（NAS）引导的结构重参数化（SRep）机制，称为NSR，以增强模型提取局部交互作用的能力.在LibriSpeech数据集上进行的大量实验表明，与SqueezeFormer相比，我们提出的HybridFormer在test-other数据集上的相对单词错误率（WER）降低了9.1%。此外，当输入语音为30 s时，HybridFormer可将模型的推理速度提高18%。我们的源代码可以在线获得。摘要：SqueezeFormer has recently shown impressive performance in automatic speech recognition (ASR). However, its inference speed suffers from the quadratic complexity of softmax-attention (SA). In addition, limited by the large convolution kernel size, the local modeling ability of SqueezeFormer is insufficient. In this paper, we propose a novel method HybridFormer to improve SqueezeFormer in a fast and efficient way. Specifically, we first incorporate linear attention (LA) and propose a hybrid LASA paradigm to increase the model's inference speed. Second, a hybrid neural architecture search (NAS) guided structural re-parameterization (SRep) mechanism, termed NSR, is proposed to enhance the ability of the model to extract local interactions. Extensive experiments conducted on the LibriSpeech dataset demonstrate that our proposed HybridFormer can achieve a 9.1% relative word error rate (WER) reduction over SqueezeFormer on the test-other dataset. Furthermore, when input speech is 30s, the HybridFormer can improve the model's inference speed up to 18%. Our source code is available online.
【44】 Acoustic source localization in the spherical harmonics domain exploiting low-rank approximations标题：基于低阶近似的球谐波域声源定位链接：http://arxiv.org/pdf/2303.08480v1
作者：Maximo Cobos,Mirco Pezzoli,Fabio Antonacci,Augusto Sarti机构：Departament d’Inform´atica, Universitat de Valencia, Valencia, Spain, †Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, ItalyJournal-ref：2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：球谐函数域（spherical harmonics domain，SHD）中的声学信号处理是利用高阶传声器阵列采集的信号的一个活跃的研究领域。一个非常重要的任务是关于有源声源的定位。本文提出了一种简单而有效的方法来定位不利声学场景中的显著声源。通过对估计的球谐系数进行适当的归一化和排列，我们利用低秩近似来估计主源在每个时间帧的远场模态方向图。实验证明了该方法的有效性，与其他基于SHD的方法相比具有更好的性能。摘要：Acoustic signal processing in the spherical harmonics domain (SHD) is an active research area that exploits the signals acquired by higher order microphone arrays. A very important task is that concerning the localization of active sound sources. In this paper, we propose a simple yet effective method to localize prominent acoustic sources in adverse acoustic scenarios. By using a proper normalization and arrangement of the estimated spherical harmonic coefficients, we exploit low-rank approximations to estimate the far field modal directional pattern of the dominant source at each time-frame. The experiments confirm the validity of the proposed approach, with superior performance compared to other recent SHD-based approaches.
【45】 Implementing Continuous HRTF Measurement in Near-Field标题：实现近场HRTF连续测量链接：http://arxiv.org/pdf/2303.08379v1
作者：Ee-Leng Tan,Santi Peksi,Woon-Seng Gan机构：School of Electrical Engineering, Nanyang Technological University, Singapore备注：5 pages, 9 figures, Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：头部相关传递函数（HRTF）是通过耳机为虚拟现实（VR）和增强现实（AR）应用创造沉浸式听觉体验的重要组成部分。Metaverse将VR和AR相结合，打造沉浸式数字体验，用户极有可能在近场（NF）与虚拟物体进行交互。这种物体的HRTF是高度个性化的，并且依赖于方向和距离。因此，需要在NF中的不同距离处进行大量的HRTF测量。使用传统的静态停-走HRTF测量方法来获取这些测量结果对于人类收听者来说将是耗时和乏味的。在本文中，我们提出了一个针对NF的连续测量系统，并在45秒内有效地捕获水平面上的HRTF。在头和躯干相似体（HATS）和人类听者上进行了对比实验，以评估系统的一致性和鲁棒性。摘要：Head-related transfer function (HRTF) is an essential component to create an immersive listening experience over headphones for virtual reality (VR) and augmented reality (AR) applications. Metaverse combines VR and AR to create immersive digital experiences, and users are very likely to interact with virtual objects in the near-field (NF). The HRTFs of such objects are highly individualized and dependent on directions and distances. Hence, a significant number of HRTF measurements at different distances in the NF would be needed. Using conventional static stop-and-go HRTF measurement methods to acquire these measurements would be time-consuming and tedious for human listeners. In this paper, we propose a continuous measurement system targeted for the NF, and efficiently capturing HRTFs in the horizontal plane within 45 secs. Comparative experiments are performed on head and torso similar (HATS) and human listeners to evaluate system consistency and robustness.
【46】 Target Sound Extraction with Variable Cross-modality Clues标题：基于可变交叉通道线索的目标声提取链接：http://arxiv.org/pdf/2303.08372v1
作者：Chenda Li,Yao Qian,Zhuo Chen,Dongmei Wang,Takuya Yoshioka,Shujie Liu,Yanmin Qian,Michael Zeng机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Microsoft, Redmond, WA, USA备注：Accepted by ICASSP 2023摘要：自动目标声提取（TSE）是一种机器学习方法，用于模拟人类从混合源中注意感兴趣的声源的听觉感知能力。它经常使用以固定形式的目标声音线索为条件的模型，例如声音类标签，这限制了用户可以与模型交互以指定目标声音的方式。为了充分利用在推理阶段可获得的不同数量的跨模态线索，包括视频、声音事件类和文本字幕，我们提出了一个统一的基于变换器的TSE模型架构，其中多线索注意模块集成了跨模态的所有线索。由于没有现成的基准来评估我们提出的方法，我们建立了一个基于公共语料库，Audioset和AudioCaps的数据集。在可见和不可见目标声音评价集上的实验结果表明，本文提出的TSE模型能够有效地处理变化数量的线索，提高了TSE的性能和对部分线索受损的鲁棒性。摘要：Automatic target sound extraction (TSE) is a machine learning approach to mimic the human auditory perception capability of attending to a sound source of interest from a mixture of sources. It often uses a model conditioned on a fixed form of target sound clues, such as a sound class label, which limits the ways in which users can interact with the model to specify the target sounds. To leverage variable number of clues cross modalities available in the inference phase, including a video, a sound event class, and a text caption, we propose a unified transformer-based TSE model architecture, where a multi-clue attention module integrates all the clues across the modalities. Since there is no off-the-shelf benchmark to evaluate our proposed approach, we build a dataset based on public corpora, Audioset and AudioCaps. Experimental results for seen and unseen target-sound evaluation sets show that our proposed TSE model can effectively deal with a varying number of clues which improves the TSE performance and robustness against partially compromised clues.
【47】 Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models标题：微小常通环境语音识别模型中共享低阶同名权的研究链接：http://arxiv.org/pdf/2303.08343v1
作者：Steven M. Hernandez,Ding Zhao,Shaojin Ding,Antoine Bruguier,Rohit Prabhavalkar,Tara N. Sainath,Yanzhang He,Ian McGraw机构：Virginia Commonwealth University,Google备注：Accepted to IEEE ICASSP 2023摘要：机器学习技术的不断改进通过使用更大的模型和更大的训练数据集提供了令人兴奋的新机会。然而，在低功耗设备（如智能手机、可穿戴设备和其他只有低内存可用的嵌入式环境）上提供这些新功能的需求日益增长。为此，我们考虑用于将基于一致器的语音识别模型的模型大小减小到仅$5$M参数同时最小化对模型质量的影响的方法，所述基于一致器的语音识别模型通常需要具有大于1亿个参数的模型。这种模型允许我们在具有低存储器神经处理器的边缘设备上实现始终在线的环境语音识别。我们建议在模型架构内的不同级别重用模型权重：（i）重复全构象异构体块层，（ii）跨层共享特定构象异构体模块，（iii）共享每个构象异构体模块的子组分，和（iv）共享低秩分解后的分解子组分权重。通过在模型的不同级别共享权重，我们可以在内存中保留完整的模型，同时增加应用于输入的虚拟转换的数量。通过一系列的消融研究和评估，我们发现通过权重共享和低秩架构，我们可以在$5$M参数模型下分别实现Librispeech dev-clean和test-clean的WER为2.84和2.94。摘要：Continued improvements in machine learning techniques offer exciting new opportunities through the use of larger models and larger training datasets. However, there is a growing need to offer these new capabilities on-board low-powered devices such as smartphones, wearables and other embedded environments where only low memory is available. Towards this, we consider methods to reduce the model size of Conformer-based speech recognition models which typically require models with greater than 100M parameters down to just $5$M parameters while minimizing impact on model quality. Such a model allows us to achieve always-on ambient speech recognition on edge devices with low-memory neural processors. We propose model weight reuse at different levels within our model architecture: (i) repeating full conformer block layers, (ii) sharing specific conformer modules across layers, (iii) sharing sub-components per conformer module, and (iv) sharing decomposed sub-component weights after low-rank decomposition. By sharing weights at different levels of our model, we can retain the full model in-memory while increasing the number of virtual transformations applied to the input. Through a series of ablation studies and evaluations, we find that with weight sharing and a low-rank architecture, we can achieve a WER of 2.84 and 2.94 for Librispeech dev-clean and test-clean respectively with a $5$M parameter model.
【48】 A large-scale multimodal dataset of human speech recognition标题：一种大规模的多模式人类语音识别数据集链接：http://arxiv.org/pdf/2303.08295v1
作者：Yao Ge,Chong Tang,Haobo Li,Zikang Zhang,Wenda Li,Kevin Chetty,Daniele Faccio,Qammer H. Abbasi,Muhammad Imran机构：James Watt School of Engineering, University of Glasgow, Glasgow, G,QQ,UK, School of Physics & Astronomy, University of Glasgow, Glasgow, G,QQ,UK, Department of Security and Crime Science, University College London, London, WC,E ,BT, UK摘要：目前，非隐私小尺度运动检测在遥感语音识别中的研究越来越多。这些新模态被用于增强和恢复来自说话者的多种类型数据的语音信息。本文提出了一个包含超宽带（UWB）雷达的7.5 GHz信道脉冲响应（CIR）数据、毫米波（mmWave）雷达的77 GHz调频连续波（FMCW）数据和激光数据的数据集。同时，采用深度摄像机记录被试嘴唇和声音的特征点。提供了大约400分钟的注释语音简档，其是从20个参与者讲5个元音、15个单词和16个句子收集的。该数据集已经过验证，具有唇读和多模态语音识别研究的潜力。摘要：Nowadays, non-privacy small-scale motion detection has attracted an increasing amount of research in remote sensing in speech recognition. These new modalities are employed to enhance and restore speech information from speakers of multiple types of data. In this paper, we propose a dataset contains 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77-GHz frequency modulated continuous wave (FMCW) data from millimetre wave (mmWave) radar, and laser data. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words and 16 sentences. The dataset has been validated and has potential for the research of lip reading and multimodal speech recognition.【1】 Localizing Spatial Information in Neural Spatiospectral Filters标题：神经空间谱滤波中的空间信息定位链接：http://arxiv.org/pdf/2303.08052v1
作者：Annika Briegleb,Thomas Haubner,Vasileios Belagiannis,Walter Kellermann机构：Multimedia Communications and Signal Processing, Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Erlangen, Germany备注：Submitted to the 31st European Signal Processing Conference (EUSIPCO 2023), Helsinki, Finland. 5 pages, 3 figures摘要：用于多通道语音增强的波束形成依赖于对声学场景的空间特性的估计。在最简单的形式中，延迟求和波束形成器（DSB）向所有通道引入时间延迟，以对齐所需信号分量进行相长叠加。最近对神经空间谱滤波的研究表明，这些滤波器可以由类似于传统波束形成器的波束图来表征，这表明人工神经网络可以学习并显式地表示空间结构。使用复值空间自动编码器（COSPA）作为多通道语音增强的示例性神经空间频谱滤波器，我们研究了这种网络在哪里以及如何表示空间信息。我们通过聚类表明，对于COSPA，空间信息由同时访问所有信道的门控递归单元（GRU）层生成的特征表示，并且这些特征不是源，而是仅依赖于到达方向。摘要：Beamforming for multichannel speech enhancement relies on the estimation of spatial characteristics of the acoustic scene. In its simplest form, the delay-and-sum beamformer (DSB) introduces a time delay to all channels to align the desired signal components for constructive superposition. Recent investigations of neural spatiospectral filtering revealed that these filters can be characterized by a beampattern similar to one of traditional beamformers, which shows that artificial neural networks can learn and explicitly represent spatial structure. Using the Complex-valued Spatial Autoencoder (COSPA) as an exemplary neural spatiospectral filter for multichannel speech enhancement, we investigate where and how such networks represent spatial information. We show via clustering that for COSPA the spatial information is represented by the features generated by a gated recurrent unit (GRU) layer that has access to all channels simultaneously and that these features are not source -- but only direction of arrival-dependent.
【2】 A Hierarchical Regression Chain Framework for Affective Vocal Burst Recognition标题：一种用于情感突发语音识别的层次回归链框架链接：http://arxiv.org/pdf/2303.08027v1
作者：Jinchao Li,Xixin Wu,Kaitao Song,Dongsheng Li,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Microsoft Research Asia, Shanghai, China备注：5 pages, 3 figures, 5 tables摘要：爆发式发声作为一种常见的非语言发声表达情感的方式，在日常社会交往中发挥着重要作用。理解和建模人类的发声是发展鲁棒的和通用的人工智能不可或缺的。探索语音爆发的计算方法正引起越来越多的研究关注。在这项工作中，我们提出了一个层次框架，基于链回归模型，用于从VB中进行情感识别，它明确地考虑了多种关系：（i）情感状态和不同文化之间的差异;（ii）在低维（唤醒&效价）和高维（10个情感类别）情感空间之间;以及（iii）在高维空间内的各种情感类别之间。为了应对数据稀疏性的挑战，我们还使用了自监督学习（SSL）表示法，以及逐层和时态聚合模块。申报系统参加了2022年ACII情感爆发式发声（A-VB）挑战赛，并在“Two '”和“Culture'”任务中排名第一。基于ACII Challenge 2022数据集的实验结果验证了所提系统的优越性能和使用层次回归链模型考虑多重关系的有效性。摘要：As a common way of emotion signaling via non-linguistic vocalizations, vocal burst (VB) plays an important role in daily social interaction. Understanding and modeling human vocal bursts are indispensable for developing robust and general artificial intelligence. Exploring computational approaches for understanding vocal bursts is attracting increasing research attention. In this work, we propose a hierarchical framework, based on chain regression models, for affective recognition from VBs, that explicitly considers multiple relationships: (i) between emotional states and diverse cultures; (ii) between low-dimensional (arousal & valence) and high-dimensional (10 emotion classes) emotion spaces; and (iii) between various emotion classes within the high-dimensional space. To address the challenge of data sparsity, we also use self-supervised learning (SSL) representations with layer-wise and temporal aggregation modules. The proposed systems participated in the ACII Affective Vocal Burst (A-VB) Challenge 2022 and ranked first in the "TWO'' and "CULTURE'' tasks. Experimental results based on the ACII Challenge 2022 dataset demonstrate the superior performance of the proposed system and the effectiveness of considering multiple relationships using hierarchical regression chain models.
【3】 Leveraging Pretrained Representations with Task-related Keywords for Alzheimer's Disease Detection标题：利用具有任务相关关键字的预先训练的表示法检测阿尔茨海默病链接：http://arxiv.org/pdf/2303.08019v1
作者：Jinchao Li,Kaitao Song,Junan Li,Bo Zheng,Dongsheng Li,Xixin Wu,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Microsoft Research Asia, Shanghai, China备注：5 pages, 3 figures, 3 tables摘要：随着全球人口迅速老龄化，阿尔茨海默病（AD）在老年人中尤为突出，其具有隐性发病并导致认知领域（记忆、交流等）的逐渐、不可逆恶化。基于言语的AD检测为广泛筛查和及时的疾病干预提供了可能性。预训练模型的最新进展促使AD检测建模从低级特征转移到高级表示。本文提出了几种有效的方法来从高级声学和语言特征中提取更好的广告相关线索。基于这些特点，本文还通过对参与者描述与认知任务之间关系的建模，提出了一种新的任务导向方法。实验在ADReSS数据集上进行，在二进制分类设置中，模型在看不见的测试集上进行评估。实验结果和近期文献的比较表明了所提出的声学、语言学和任务导向方法的有效性和优越性。研究结果还显示了语义和句法信息的重要性，以及使用有前途的纯音频和面向任务的方法进行广告检测任务的自动化和泛化的可行性。摘要：With the global population aging rapidly, Alzheimer's disease (AD) is particularly prominent in older adults, which has an insidious onset and leads to a gradual, irreversible deterioration in cognitive domains (memory, communication, etc.). Speech-based AD detection opens up the possibility of widespread screening and timely disease intervention. Recent advances in pre-trained models motivate AD detection modeling to shift from low-level features to high-level representations. This paper presents several efficient methods to extract better AD-related cues from high-level acoustic and linguistic features. Based on these features, the paper also proposes a novel task-oriented approach by modeling the relationship between the participants' description and the cognitive task. Experiments are carried out on the ADReSS dataset in a binary classification setup, and models are evaluated on the unseen test set. Results and comparison with recent literature demonstrate the efficiency and superior performance of proposed acoustic, linguistic and task-oriented methods. The findings also show the importance of semantic and syntactic information, and feasibility of automation and generalization with the promising audio-only and task-oriented methods for the AD detection task.
【4】 Native Multi-Band Audio Coding within Hyper-Autoencoded Reconstruction Propagation Networks标题：超自动编码重建传播网络中的原生多频带音频编码链接：http://arxiv.org/pdf/2303.08005v1
作者：Darius Petermann,Inseon Jang,Minje Kim机构：Indiana University, Department of Intelligent Systems Engineering, Bloomington, IN , USA, Electronics and Telecommunications Research Institute, Daejeon , South Korea备注：Accepted to ICASSP 2023. For resources and examples, see this https URL摘要：频谱子带不描绘相同的感知相关性。因此，在音频编码中，期望对每个组成频带具有独立的控制，使得可以有效地实现比特率分配和信号重构。在这项工作中，我们提出了一个新的神经音频编码网络，自然支持多频带编码范式。我们的模型扩展了基于U-Net的编解码器中压缩跳跃连接的思想，允许对核心和高频带特定的重建和比特分配进行独立控制。我们的系统主要从压缩的核心频带码重构全频带信号，从而最大限度地开发和展示其带宽扩展能力。同时，低比特率的高带编码有助于高带重构，类似于MPEG音频编解码器的频谱带宽复制。MUSHRA测试表明，该模型不仅通过显式分配更多的比特来提高核心频带的质量，而且在高频段也保持了良好的质量。摘要：Spectral sub-bands do not portray the same perceptual relevance. In audio coding, it is therefore desirable to have independent control over each of the constituent bands so that bitrate assignment and signal reconstruction can be achieved efficiently. In this work, we present a novel neural audio coding network that natively supports a multi-band coding paradigm. Our model extends the idea of compressed skip connections in the U-Net-based codec, allowing for independent control over both core and high band-specific reconstructions and bit allocation. Our system reconstructs the full-band signal mainly from the condensed core-band code, therefore exploiting and showcasing its bandwidth extension capabilities to its fullest. Meanwhile, the low-bitrate high-band code helps the high-band reconstruction similarly to MPEG audio codecs' spectral bandwidth replication. MUSHRA tests show that the proposed model not only improves the quality of the core band by explicitly assigning more bits to it but retains a good quality in the high-band as well.
【5】 Multi-Channel Masking with Learnable Filterbank for Sound Source Separation标题：用于声源分离的带学习滤波器组的多通道掩蔽链接：http://arxiv.org/pdf/2303.07816v1
作者：Wang Dai,Archontis Politis,Tuomas Virtanen机构：Audio Research Group, Tampere University, Tampere, Finland摘要：本文提出了一种基于多通道掩蔽框架的可学习滤波器组，用于多通道信号分离。可学习滤波器组是1D Conv层，它将原始波形转换为2D表示。与传统的单通道掩蔽方法相比，我们为每个单独的麦克风通道估计一个掩蔽。然后，像在传统的滤波和求和波束成形操作中一样，将估计的掩码应用于变换的波形表示。具体而言，每个掩码用于乘以对应通道的2D表示，然后对所有通道的掩码输出求和。最后，使用1D转置Conv层将求和的掩蔽信号转换到波形域。实验结果表明，在STGCSEN模型中，该方法优于单通道可学习滤波器组掩蔽，若将可学习滤波器组变换为更高的特征维数，则优于多通道STFT复谱掩蔽.空间响应分析也验证了可学习滤波器组域中的多通道掩蔽具有空间选择性。摘要：This work proposes a learnable filterbank based on a multi-channel masking framework for multi-channel source separation. The learnable filterbank is a 1D Conv layer, which transforms the raw waveform into a 2D representation. In contrast to the conventional single-channel masking method, we estimate a mask for each individual microphone channel. The estimated masks are then applied to the transformed waveform representation like in the traditional filter-and-sum beamforming operation. Specifically, each mask is used to multiply the corresponding channel's 2D representation, and the masked output of all channels are then summed. At last, a 1D transposed Conv layer is used to convert the summed masked signal into the waveform domain. The experimental results show our method outperforms single-channel masking with a learnable filterbank and can outperform multi-channel complex masking with STFT complex spectrum in the STGCSEN model if a learnable filterbank is transformed to a higher feature dimension. The spatial response analysis also verifies that multi-channel masking in the learnable filterbank domain has spatial selectivity.
【6】 Detecting post-stroke aphasia using EEG-based neural envelope tracking of natural speech标题：基于脑电的自然语音神经包络跟踪检测中风后失语链接：http://arxiv.org/pdf/2303.07739v1
作者：Pieter De Clercq,Jill Kries,Ramtin Mehraram,Jonas Vanthornhout,Tom Francart,Maaike Vandermosten机构：Experimental Oto-Rhino-Laryngology, Department of Neurosciences, Leuven Brain, Institute, KU Leuven, Belgium, Version:, Corresponding author:, arXiv:,.,v, [eess.SP] , Mar摘要：[目的]。中风后，三分之一的患者患有失语症，这是一种损害沟通能力的语言障碍。用于诊断失语症的标准行为测试耗时且生态效度低。语音包络的神经跟踪是研究大脑对自然语音反应的一种有前途的工具。言语包络是言语理解的关键，它包含了处理语言单位的线索。在本研究中，我们旨在测试神经包络追踪技术在失语症患者（IWA）中检测语言障碍的潜力。【办法】。我们记录了27名中风后慢性期的IWA和22名听故事的对照组的脑电图。我们使用互信息分析量化了宽带频率范围以及delta、theta、alpha、beta和gamma频带中的神经包络跟踪。除了神经跟踪测量的组差异外，我们还使用支持向量机（SVM）分类器测试了其检测失语症的适用性。我们进一步研究了SVM检测失语症和获得可靠结果所需的记录长度。[结果]。与对照组相比，IWA在宽、δ、θ和γ波段显示神经包络跟踪减少。这些频带中的神经跟踪在个体水平上有效地捕获失语症（SVM准确率84%，AUC 88%）。记录时间为5-7分钟，检测结果准确可靠。【意义】。我们的研究表明，语言的神经追踪是失语症的有效生物标志物。我们证明了其作为一种诊断工具的潜力，具有高可靠性、个体水平的失语症检测和时间效率评估。这项工作代表了一个重要的一步，更自动，客观，生态有效的评估语言障碍失语症。摘要：[Objective]. After a stroke, one-third of patients suffer from aphasia, a language disorder that impairs communication ability. The standard behavioral tests used to diagnose aphasia are time-consuming and have low ecological validity. Neural tracking of the speech envelope is a promising tool for investigating brain responses to natural speech. The speech envelope is crucial for speech understanding, encompassing cues for processing linguistic units. In this study, we aimed to test the potential of the neural envelope tracking technique for detecting language impairments in individuals with aphasia (IWA). [Approach]. We recorded EEG from 27 IWA in the chronic phase after stroke and 22 controls while they listened to a story. We quantified neural envelope tracking in a broadband frequency range as well as in the delta, theta, alpha, beta, and gamma frequency bands using mutual information analysis. Besides group differences in neural tracking measures, we also tested its suitability for detecting aphasia using a Support Vector Machine (SVM) classifier. We further investigated the required recording length for the SVM to detect aphasia and to obtain reliable outcomes. [Results]. IWA displayed decreased neural envelope tracking compared to controls in the broad, delta, theta, and gamma band. Neural tracking in these frequency bands effectively captured aphasia at the individual level (SVM accuracy 84%, AUC 88%). High-accuracy and reliable detection could be obtained with 5-7 minutes of recording time. [Significance]. Our study shows that neural tracking of speech is an effective biomarker for aphasia. We demonstrated its potential as a diagnostic tool with high reliability, individual-level detection of aphasia, and time-efficient assessment. This work represents a significant step towards more automatic, objective, and ecologically valid assessments of language impairments in aphasia.
【7】 TEA-PSE 3.0: Tencent-Ethereal-Audio-Lab Personalized Speech Enhancement System For ICASSP 2023 DNS Challenge标题：TEA-PSE 3.0：面向ICASSP 2023域名挑战赛的腾讯-Etheral-Audio-Lab个性化语音增强系统链接：http://arxiv.org/pdf/2303.07704v1
作者：Yukai Ju,Jun Chen,Shimin Zhang,Shulin He,Wei Rao,Weixin Zhu,Yannan Wang,Tao Yu,Shidong Shang机构：Tencent Ethereal Audio Lab, Tencent Corporation, Shenzhen, China, Northwestern Polytechnical University, Xi’an, China备注：Accepted by ICASSP 2023摘要：介绍了Unbeatable团队向ICASSP 2023深度噪声抑制（DNS）挑战赛提交的作品。我们将以前的工作TEA-PSE扩展到它的升级版本-- TEA-PSE 3.0。具体而言，TEA-PSE 3.0在压缩时间卷积网络（S-TCN）之后结合了剩余LSTM，以增强层序建模能力。此外，引入局部-全局表示（LGR）结构以提高说话人信息提取效率，并利用多STFT分辨率损失有效捕捉语音信号的时频特征。此外，在冻结训练策略的基础上采用再训练方法对系统进行微调。根据官方公布的结果，TEA-PSE 3.0在ICASSP 2023 DNS挑战赛第一和第二赛道均排名第一。摘要：This paper introduces the Unbeatable Team's submission to the ICASSP 2023 Deep Noise Suppression (DNS) Challenge. We expand our previous work, TEA-PSE, to its upgraded version -- TEA-PSE 3.0. Specifically, TEA-PSE 3.0 incorporates a residual LSTM after squeezed temporal convolution network (S-TCN) to enhance sequence modeling capabilities. Additionally, the local-global representation (LGR) structure is introduced to boost speaker information extraction, and multi-STFT resolution loss is used to effectively capture the time-frequency characteristics of the speech signals. Moreover, retraining methods are employed based on the freeze training strategy to fine-tune the system. According to the official results, TEA-PSE 3.0 ranks 1st in both ICASSP 2023 DNS-Challenge track 1 and track 2.
【8】 Two-stage Neural Network for ICASSP 2023 Speech Signal Improvement Challenge标题：用于ICASSP 2023语音信号改善挑战的两级神经网络链接：http://arxiv.org/pdf/2303.07621v1
作者：Mingshuai Liu,Shubo Lv,Zihan Zhang,Runduo Han,Xiang Hao,Xianjun Xia,Li Chen,Yijian Xiao,Lei Xie机构：Northwestern Polytechnical University, Xi’an, China, ByteDance, China备注：Accepted by ICASSP 2023摘要：在ICASSP 2023语音信号改善挑战赛中，我们开发了一个双阶段神经模型，该模型以阶段分治的方式改善了由不同失真引起的语音信号质量。具体地，在第一阶段，语音改善网络集中于恢复频谱的缺失分量，而在第二阶段，我们的模型旨在进一步抑制由第一阶段模型引入的噪声、混响和伪像。最终得分为0.446，P.835得分为0.517，在非实时领域排名第四。摘要：In ICASSP 2023 speech signal improvement challenge, we developed a dual-stage neural model which improves speech signal quality induced by different distortions in a stage-wise divide-and-conquer fashion. Specifically, in the first stage, the speech improvement network focuses on recovering the missing components of the spectrum, while in the second stage, our model aims to further suppress noise, reverberation, and artifacts introduced by the first-stage model. Achieving 0.446 in the final score and 0.517 in the P.835 score, our system ranks 4th in the non-real-time track.
【9】 Lightweight feature encoder for wake-up word detection based on self-supervised speech representation标题：基于自监督语音表示的轻量级唤醒词检测特征编码器链接：http://arxiv.org/pdf/2303.07592v1
作者：Hyungjun Lim,Younggwan Kim,Kiho Yeom,Eunjoo Seo,Hoodong Lee,Stanley Jungkyu Choi,Honglak Lee机构：LG AI Research, Seoul, Republic of Korea备注：Accepted by ICASSP 2023摘要：提供广义语音表示的自监督学习方法近年来受到越来越多的关注。Wav2vec 2.0是最著名的例子，在众多下游语音处理任务中表现出了卓越的性能。尽管它取得了成功，但由于其昂贵的计算成本，直接将其用于移动设备上的唤醒词检测是具有挑战性的。在本文中，我们提出了LiteFEW，一个轻量级的特征编码器，用于唤醒字检测，它以最小的规模保留了wav2vec 2.0的固有能力。该方法通过引入基于自动编码器的降维技术，对预先训练好的wav2vec2. 0进行知识压缩，并提取到LiteFEW中.在开源“Hey Snips”数据集上的实验结果表明，所提方法应用于不同模型结构时，仅使用64k个参数，性能相对提高了20%以上。摘要：Self-supervised learning method that provides generalized speech representations has recently received increasing attention. Wav2vec 2.0 is the most famous example, showing remarkable performance in numerous downstream speech processing tasks. Despite its success, it is challenging to use it directly for wake-up word detection on mobile devices due to its expensive computational cost. In this work, we propose LiteFEW, a lightweight feature encoder for wake-up word detection that preserves the inherent ability of wav2vec 2.0 with a minimum scale. In the method, the knowledge of the pre-trained wav2vec 2.0 is compressed by introducing an auto-encoder-based dimensionality reduction technique and distilled to LiteFEW. Experimental results on the open-source "Hey Snips" dataset show that the proposed method applied to various model structures significantly improves the performance, achieving over 20% of relative improvements with only 64k parameters.
【10】 Towards Real-Time Single-Channel Speech Separation in Noisy and Reverberant Environments标题：噪声和混响环境下的实时单通道语音分离链接：http://arxiv.org/pdf/2303.07569v1
作者：Julian Neri,Sebastian Braun机构：McGill University, CIRMMT, Montreal, CA, Microsoft Research, Redmond, USA备注：to appear in ICASSP 2023摘要：实时单通道语音分离旨在将从单个麦克风捕获的包含多个人同时讲话、环境噪声和混响的音频流分解成多个去混响且无噪声的语音轨道，每个轨道仅包含一个讲话者。虽然大型的最先进的DNN可以从语音的无回声混合中实现出色的分离，但主要的挑战是创建可以在推断时间分离混响混合的紧凑和因果模型。在这篇论文中，我们探索低复杂度、资源有效、因果关系的DNN架构，以即时分离两个或多个同时说话人。三个神经网络模块的级联被训练以顺序地执行噪声抑制、分离和去混响。为了比较，训练较大的端到端模型以直接从有噪声的混响语音混合物输出两个无回声语音信号。我们提出一个有效的单解码器架构与减法分离实时递归语音分离两个或多个发言人。根据SI-SDR等语音分离度量、DNS-MOS等感知度量和一种新提出的通道分离度量，对语音混合的真实单声道记录进行评估，结果表明这些紧凑的因果模型能够以低延迟分离语音混合，并且与SepFormer等大型离线最先进模型的性能相当。摘要：Real-time single-channel speech separation aims to unmix an audio stream captured from a single microphone that contains multiple people talking at once, environmental noise, and reverberation into multiple de-reverberated and noise-free speech tracks, each track containing only one talker. While large state-of-the-art DNNs can achieve excellent separation from anechoic mixtures of speech, the main challenge is to create compact and causal models that can separate reverberant mixtures at inference time. In this paper, we explore low-complexity, resource-efficient, causal DNN architectures for real-time separation of two or more simultaneous speakers. A cascade of three neural network modules are trained to sequentially perform noise-suppression, separation, and de-reverberation. For comparison, a larger end-to-end model is trained to output two anechoic speech signals directly from noisy reverberant speech mixtures. We propose an efficient single-decoder architecture with subtractive separation for real-time recursive speech separation for two or more speakers. Evaluation on real monophonic recordings of speech mixtures, according to speech separation measures like SI-SDR, perceptual measures like DNS-MOS, and a novel proposed channel separation metric, show that these compact causal models can separate speech mixtures with low latency, and perform on par with large offline state-of-the-art models like SepFormer.
【11】 Speech Intelligibility Classifiers from 550k Disordered Speech Samples标题：55万无序语音样本中的语音清晰度分类器链接：http://arxiv.org/pdf/2303.07533v1
作者：Subhashini Venugopalan,Jimmy Tobin,Samuel J. Yang,Katie Seaver,Richard J. N. Cave,Pan-Pan Jiang,Neil Zeghidour,Rus Heywood,Jordan Green,Michael P. Brenner机构：Google Research, USA; ,Google Research, France;, MGH Institute of Health Professions, USA; ,Harvard University, USA备注：ICASSP 2023 camera-ready摘要：我们开发了构音障碍语音可懂度分类器，用于551,176个障碍语音样本，这些样本由468名不同的说话者提供，他们有一系列自我报告的说话障碍，并在5分制量表上对其整体可懂度进行评级。我们采用不同的深度学习方法训练了三个模型，并在100位发言者的约94 K次发言中对它们进行了评估。我们进一步发现模型在TORGO数据库（100%准确度）、UASpeech（0.93相关性）、ALS-TDI PMP（0.81 AUC）数据集以及我们收集的真实自发语音数据集（106例构音障碍和76例对照说话者，约2300个样本）上具有良好的泛化性（无需进一步训练）。为了推进这一领域的研究，我们在www.example.com上分享了我们的一个模型https: tfhub.dev google euphonia_spice classification 1。摘要：We developed dysarthric speech intelligibility classifiers on 551,176 disordered speech samples contributed by a diverse set of 468 speakers, with a range of self-reported speaking disorders and rated for their overall intelligibility on a five-point scale. We trained three models following different deep learning approaches and evaluated them on ~94K utterances from 100 speakers. We further found the models to generalize well (without further training) on the TORGO database (100% accuracy), UASpeech (0.93 correlation), ALS-TDI PMP (0.81 AUC) datasets as well as on a dataset of realistic unprompted speech we gathered (106 dysarthric and 76 control speakers,~2300 samples). To advance research in this domain, we share one of our models at https: tfhub.dev google euphonia_spice classification 1.
【12】 Guided Speech Enhancement Network标题：引导式语音增强网络链接：http://arxiv.org/pdf/2303.07486v1
作者：Yang Yang,Shao-Fu Shih,Hakan Erdogan,Jamie Menjay Lin,Chehung Lee,Yunpeng Li,George Sung,Matthias Grundmann机构：Google LLC, U.S.A.备注：Accepted to ICASSP 2023摘要：由于语音通信和人机接口的原因，高质量语音捕获已经被广泛研究。为了提高捕获性能，我们经常发现多麦克风语音增强技术被部署在各种设备上。多麦克风语音增强问题通常被分解为两个解耦的步骤：提供空间滤波的波束形成器和清除波束形成器输出的单通道语音增强模型。在这项工作中，我们提出了一个语音增强解决方案，采取原始麦克风和波束形成器输出作为输入的ML模型。我们设计了一种简单而有效的训练方案，允许模型通过对比两个输入来学习波束形成器的线索，并大大提高其空间抑制能力，同时执行去噪和去混响的一般任务。所提出的解决方案利用了经典的空间滤波算法，而不是与它们竞争。通过设计，波束形成器模块于是可以被单独地选择，并且不需要针对给定的形状因子优化大量数据，并且网络模型可以被认为是独立模块，其可以独立于麦克风阵列而被高度地转移。我们将解决方案中的ML模块命名为GSENet，即Guided Speech Enhancement Network的缩写。我们证明了它在多麦克风设备上收集的真实世界数据中抑制噪声和干扰语音的有效性。摘要：High quality speech capture has been widely studied for both voice communication and human computer interface reasons. To improve the capture performance, we can often find multi-microphone speech enhancement techniques deployed on various devices. Multi-microphone speech enhancement problem is often decomposed into two decoupled steps: a beamformer that provides spatial filtering and a single-channel speech enhancement model that cleans up the beamformer output. In this work, we propose a speech enhancement solution that takes both the raw microphone and beamformer outputs as the input for an ML model. We devise a simple yet effective training scheme that allows the model to learn from the cues of the beamformer by contrasting the two inputs and greatly boost its capability in spatial rejection, while conducting the general tasks of denoising and dereverberation. The proposed solution takes advantage of classical spatial filtering algorithms instead of competing with them. By design, the beamformer module then could be selected separately and does not require a large amount of data to be optimized for a given form factor, and the network model can be considered as a standalone module which is highly transferable independently from the microphone array. We name the ML module in our solution as GSENet, short for Guided Speech Enhancement Network. We demonstrate its effectiveness on real world data collected on multi-microphone devices in terms of the suppression of noise and interfering speech.
【13】Online Binaural Speech Separation of Moving Speakers With a Wavesplit Network标题：基于小波分解网络的运动说话人在线双耳语音分离链接：http://arxiv.org/pdf/2303.07458v1
作者：Cong Han,Nima Mesgarani机构：Department of Electrical Engineering, Columbia University, New York, NY备注：To appear in ICASSP 2023摘要：在真实世界场景中的双耳语音分离通常涉及移动的说话者。目前的语音分离方法大多采用话语级置换不变训练（u-PIT）进行训练。然而，在推理时间中，输出的顺序可能随时间不一致，特别是在长格式语音分离中。当扬声器在空间中不断移动时，被称为扬声器交换问题的这种情况甚至更成问题，并且因此对扬声器在输出声道中的一致放置提出了挑战。本文提出了一种基于小波分解网络的实时双耳语音分离模型，以解决说话人交换问题。该模型根据混合音频计算每个说话人在每个时间帧的说话人嵌入，使用在线聚类聚合嵌入，并使用聚类质心作为说话人简档来跟踪每个说话人的长持续时间。混响、长格式运动多说话人语音分离实验结果表明，该方法不易发生说话人交换，在分离精度和耳间线索保持方面与基于u-PIT模型的地面真值跟踪方法具有可比性。摘要：Binaural speech separation in real-world scenarios often involves moving speakers. Most current speech separation methods use utterance-level permutation invariant training (u-PIT) for training. In inference time, however, the order of outputs can be inconsistent over time particularly in long-form speech separation. This situation which is referred to as the speaker swap problem is even more problematic when speakers constantly move in space and therefore poses a challenge for consistent placement of speakers in output channels. Here, we describe a real-time binaural speech separation model based on a Wavesplit network to mitigate the speaker swap problem for moving speaker separation. Our model computes a speaker embedding for each speaker at each time frame from the mixed audio, aggregates embeddings using online clustering, and uses cluster centroids as speaker profiles to track each speaker throughout the long duration. Experimental results on reverberant, long-form moving multitalker speech separation show that the proposed method is less prone to speaker swap and achieves comparable performance with u-PIT based models with ground truth tracking in both separation accuracy and preserving the interaural cues.
【14】 Blind Acoustic Room Parameter Estimation Using Phase Features标题：基于相位特征的声室参数盲估计算法
链接：http://arxiv.org/pdf/2303.07449v1
作者：Christopher Ick,Adib Mehrabi,Wenyu Jin机构：New York University, New York, NY , United States, Sonos Experience Limited, United Kingdom , Sonos, Inc., Boston, MA , United States备注：4 pages + 1 page bibliography, 3 figures, to be published in proceedings of ICASSP 2023摘要：在现场环境中对房间声学进行建模涉及从噪声和混响音频中进行某种程度的盲参数估计。现代方法利用卷积神经网络（CNN）与时频表示相结合。使用短时傅立叶变换来开发这些类似频谱图的特征已经显示出有希望的结果，但是该方法隐含地丢弃了相位域中的大量音频信息。受语音增强研究的启发，本文提出利用新的相位相关特征来扩展现有的方法，盲估计混响指纹参数，即音量和RT60。这些特征的加入被证明优于现有的方法，现有的方法仅仅依赖于在广泛的声学空间范围内的基于震级的频谱特征。我们使用由公开可用的房间脉冲响应（RIR）、合成RIR和真实声学空间的内部测量组成的新型数据集，评估了这些新型特征在单参数和多参数估计策略中部署的有效性。摘要：Modeling room acoustics in a field setting involves some degree of blind parameter estimation from noisy and reverberant audio. Modern approaches leverage convolutional neural networks (CNNs) in tandem with time-frequency representation. Using short-time Fourier transforms to develop these spectrogram-like features has shown promising results, but this method implicitly discards a significant amount of audio information in the phase domain. Inspired by recent works in speech enhancement, we propose utilizing novel phase-related features to extend recent approaches to blindly estimate the so-called "reverberation fingerprint" parameters, namely, volume and RT60. The addition of these features is shown to outperform existing methods that rely solely on magnitude-based spectral features across a wide range of acoustics spaces. We evaluate the effectiveness of the deployment of these novel features in both single-parameter and multi-parameter estimation strategies, using a novel dataset that consists of publicly available room impulse responses (RIRs), synthesized RIRs, and in-house measurements of real acoustic spaces.
【15】 A processing framework to access large quantities of whispered speech found in ASMR标题：一种访问ASMR中发现的大量耳语的处理框架链接：http://arxiv.org/pdf/2303.07442v1
作者：Pablo Perez Zarazaga,Gustav Eje Henter,Zofia Malisz机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：Accepted at ICASSP 2023, 5 pages, 2 figures, 2 tables摘要：耳语是人类日常使用的一种无处不在的交流模式。尽管如此，由于资源和处理方法的短缺，现有语音技术很难提供耳语音。为了解决这一问题，本文提供了一个处理框架，使得能够访问大量和独特的高质量耳语音数据。作为ASMR媒体文化现象的一部分，我们从提交到在线平台的录音中获得数据。我们描述了我们的处理流水线和一种改进的耳语活动检测（WAD）在ASMR数据的方法。为了有效地获得标记的、干净的耳语音，我们通过使用Edyson（一种具有人在环的批量音频注释工具）来补充自动WAD。我们还解决了ASMR特有的一个问题：将耳语与该类型中存在的其它声学触发分离。我们表明，提出的WAD和有效的标记允许建立广泛的扩增数据和训练一个分类器，从ASMR音频提取干净的耳语片段。我们庞大且不断增长的数据集支持具备耳语能力的数据驱动语音技术和语言分析。它还打开了机会，例如，人机交互作为一种资源，可以引起听众的情绪，心理和神经生理反应。摘要：Whispering is a ubiquitous mode of communication that humans use daily. Despite this, whispered speech has been poorly served by existing speech technology due to a shortage of resources and processing methodology. To remedy this, this paper provides a processing framework that enables access to large and unique data of high-quality whispered speech. We obtain the data from recordings submitted to online platforms as part of the ASMR media-cultural phenomenon. We describe our processing pipeline and a method for improved whispered activity detection (WAD) in the ASMR data. To efficiently obtain labelled, clean whispered speech, we complement the automatic WAD by using Edyson, a bulk audio-annotation tool with human-in-the-loop. We also tackle a problem particular to ASMR: separation of whisper from other acoustic triggers present in the genre. We show that the proposed WAD and the efficient labelling allows to build extensively augmented data and train a classifier that extracts clean whisper segments from ASMR audio. Our large and growing dataset enables whisper-capable, data-driven speech technology and linguistic analysis. It also opens opportunities in e.g. HCI as a resource that may elicit emotional, psychological and neuro-physiological responses in the listener.
【16】 A Study on Bias and Fairness In Deep Speaker Recognition标题：深度说话人识别中的偏差和公平性研究链接：http://arxiv.org/pdf/2303.08026v1
作者：Amirhossein Hajavi,Ali Etemad机构：Dept. ECE and Ingenuity Labs Research Institute, Queen’s University, Canada摘要：随着使用说话人识别（SR）系统作为个人认证和个性化服务的手段的智能设备的普遍存在，SR系统的公平性已经成为一个重要的焦点。在本文中，我们研究了公平的概念，在最近的SR系统的基础上，3个流行的和相关的定义，即统计平价，均衡赔率，和平等机会。我们检查了5种流行的神经结构和5种常用的损失函数在训练SR系统，同时评估他们的公平性对性别和民族群体。我们的详细实验阐明了这个概念，并证明了更复杂的编码器架构更好地符合公平性的定义。此外，我们发现损失函数的选择会显著影响随机共振模型的偏差。摘要：With the ubiquity of smart devices that use speaker recognition (SR) systems as a means of authenticating individuals and personalizing their services, fairness of SR systems has becomes an important point of focus. In this paper we study the notion of fairness in recent SR systems based on 3 popular and relevant definitions, namely Statistical Parity, Equalized Odds, and Equal Opportunity. We examine 5 popular neural architectures and 5 commonly used loss functions in training SR systems, while evaluating their fairness against gender and nationality groups. Our detailed experiments shed light on this concept and demonstrate that more sophisticated encoder architectures better align with the definitions of fairness. Additionally, we find that the choice of loss functions can significantly impact the bias of SR models.
【17】 Improving Accented Speech Recognition with Multi-Domain Training标题：利用多域训练改进口音语音识别链接：http://arxiv.org/pdf/2303.07924v1
作者：Lucas Maison,Yannick Estève机构：Yannick Esteve⋆, ⋆ Laboratoire Informatique d’Avignon, Avignon, France, † Thales SIX, Multimedia Lab, Gennevilliers, France备注：5 pages, 2 figures. Accepted to ICASSP 2023摘要：由于自监督学习的兴起，自动语音识别（ASR）系统现在在各种各样的数据集上实现了接近人类的性能。然而，它们仍然缺乏泛化能力，并且对重音变化等领域变化不具有鲁棒性。在这项工作中，我们使用代表四种不同法语口音的语音音频来创建微调数据集，以提高预训练ASR模型的鲁棒性。通过在训练集中加入不同的重音，我们获得了域内和域外的改进。我们的数值实验表明，与单域训练相比，我们可以将非洲和比利时口音的错误率降低高达25%（相对），同时保持标准法语的良好性能。摘要：Thanks to the rise of self-supervised learning, automatic speech recognition (ASR) systems now achieve near-human performance on a wide variety of datasets. However, they still lack generalization capability and are not robust to domain shifts like accent variations. In this work, we use speech audio representing four different French accents to create fine-tuning datasets that improve the robustness of pre-trained ASR models. By incorporating various accents in the training set, we obtain both in-domain and out-of-domain improvements. Our numerical experiments show that we can reduce error rates by up to 25% (relative) on African and Belgian accents compared to single-domain training while keeping a good performance on standard French.
【18】 BLAT: Bootstrapping Language-Audio Pre-training based on AudioSet Tag-guided Synthetic Data标题：BLAT：引导语言.基于AudioSet标签引导的合成数据的音频预训练链接：http://arxiv.org/pdf/2303.07902v1
作者：Xuenan Xu,Zhiling Zhang,Zelin Zhou,Pingyue Zhang,Zeyu Xie,Mengyue Wu,Kenny Q. Zhu机构：Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China摘要：与大量的视文本预训练研究相比，对听文本预训练的研究较少，这主要是由于缺乏足够的平行听文本数据。现有的方法大多以视觉通道为中心进行音频-文本预训练，不可避免地引入了数据噪声。在本文中，我们提出BLAT：Bootstrapping语言-基于标签引导合成数据的音频预训练。我们利用音频字幕直接从音频生成文本，而不需要视觉模态的帮助，从而消除了模态失配带来的潜在噪声。此外，提出了AudioSet标签指导下的字幕生成方法，使得字幕更加准确。通过以上两点改进，我们可以获得高质量、大规模的并行音频文本数据，并在此基础上进行音频文本预训练。对一系列下游任务的评估表明，BLAT在大多数数据集上达到了SOTA zero-shot分类性能，并且在下游任务上进行微调时性能有了显著提高，这表明了我们合成数据的有效性。摘要：Compared with ample visual-text pre-training research, few works explore audio-text pre-training, mostly due to the lack of sufficient parallel audio-text data. Most existing methods incorporate the visual modality as a pivot for audio-text pre-training, which inevitably induces data noise. In this paper, we propose BLAT: Bootstrapping Language-Audio pre-training based on Tag-guided synthetic data. We utilize audio captioning to generate text directly from audio, without the aid of the visual modality so that potential noise from modality mismatch is eliminated. Furthermore, we propose caption generation under the guidance of AudioSet tags, leading to more accurate captions. With the above two improvements, we curate high-quality, large-scale parallel audio-text data, based on which we perform audio-text pre-training. Evaluation on a series of downstream tasks indicates that BLAT achieves SOTA zero-shot classification performance on most datasets and significant performance improvement when fine-tuned on downstream tasks, suggesting the effectiveness of our synthetic data.
【19】 DiffuseRoll: Multi-track multi-category music generation based on diffusion model标题：DiffuseRoll：基于扩散模型的多轨多品类音乐生成链接：http://arxiv.org/pdf/2303.07794v1
作者：Hongfei Wang机构：Communication University of China, Beijing, China, nd Given Name Surname, dept. name of organization (of Aff.), City, Country, email address, rd Given Name Surname摘要：近年来，生成模型的研究在音乐生成领域取得了显著进展。然而，现有的方法大多集中于生成单声道或单声道音乐，而生成具有丰富属性的多声道和多音轨音乐仍然是一项具有挑战性的任务。在这篇论文中，我们提出了一个新的方法来产生多音轨，多属性的交响音乐使用扩散模型。具体来说，我们用扩散模型生成钢琴卷表示，并将它们映射为MIDI格式输出。为了获取丰富的属性信息，我们引入了颜色编码方案，将音符序列编码为代表音高、力度和乐器的颜色和位置信息。该方案实现了离散音乐序列和连续图像之间的无缝映射。我们还提出了一种后处理方法来优化生成的分数以获得更好的性能。实验结果表明，与图形方法相比，该方法在生成具有丰富属性信息的复调音乐方面优于现有方法。摘要：Recent advancements in generative models have shown remarkable progress in music generation. However, most existing methods focus on generating monophonic or homophonic music, while the generation of polyphonic and multi-track music with rich attributes is still a challenging task. In this paper, we propose a novel approach for multi-track, multi-attribute symphonic music generation using the diffusion model. Specifically, we generate piano-roll representations with a diffusion model and map them to MIDI format for output. To capture rich attribute information, we introduce a color coding scheme to encode note sequences into color and position information that represents pitch,velocity, and instrument. This scheme enables a seamless mapping between discrete music sequences and continuous images. We also propose a post-processing method to optimize the generated scores for better performance. Experimental results show that our method outperforms state-of-the-art methods in terms of polyphonic music generation with rich attribute information compared to the figure methods.
【20】 Good Neighbors Are All You Need for Chinese Grapheme-to-Phoneme Conversion标题：好邻居是中文字音转换所需要的全部链接：http://arxiv.org/pdf/2303.07726v1
作者：Jungjun Kim,Changjin Han,Gyuhyeon Nam,Gyeongsu Chae机构：DeepBrain AI Inc., Seoul, Korea备注：Accepted to ICASSP 2023摘要：大多数中文字形到音素（G2P）系统采用三阶段框架，首先将输入序列转换为字符嵌入，使用语言模型获得语言信息，然后基于整个输入序列的全局上下文预测音素。然而，仅有语言知识往往是不够的。语言模型经常对句子的过于一般的结构进行编码，并且不能覆盖使用语音知识所需的特定情况。此外，还需要一个手工制作的后处理系统来解决与字符音调有关的问题。然而，该系统在词边界的分割中表现出不一致性，这因此降低了G2P系统的性能。为了解决这些问题，我们提出了增强器，它通过强调相邻字符之间的语音信息来帮助消除发音歧义，从而为语言模型提供强大的归纳偏差。实验结果表明，Reinforcer大大提升了前沿架构的性能。我们还将Reinforcer与一个大规模的预训练模型相结合，并证明了在知识转移场景中使用相邻上下文的有效性。摘要：Most Chinese Grapheme-to-Phoneme (G2P) systems employ a three-stage framework that first transforms input sequences into character embeddings, obtains linguistic information using language models, and then predicts the phonemes based on global context about the entire input sequence. However, linguistic knowledge alone is often inadequate. Language models frequently encode overly general structures of a sentence and fail to cover specific cases needed to use phonetic knowledge. Also, a handcrafted post-processing system is needed to address the problems relevant to the tone of the characters. However, the system exhibits inconsistency in the segmentation of word boundaries which consequently degrades the performance of the G2P system. To address these issues, we propose the Reinforcer that provides strong inductive bias for language models by emphasizing the phonological information between neighboring characters to help disambiguate pronunciations. Experimental results show that the Reinforcer boosts the cutting-edge architectures by a large margin. We also combine the Reinforcer with a large-scale pre-trained model and demonstrate the validity of using neighboring context in knowledge transfer scenarios.
【21】 Improving Prosody for Cross-Speaker Style Transfer by Semi-Supervised Style Extractor and Hierarchical Modeling in Speech Synthesis标题：语音合成中基于半监督风格提取和分层建模的交叉语体风格转换韵律改进链接：http://arxiv.org/pdf/2303.07711v1
作者：Chunyu Qiang,Peng Yang,Hao Che,Ying Zhang,Xiaorui Wang,Zhongyuan Wang机构：Kwai, Beijing, P.R. China备注：Accepted by ICASSP2023摘要：语音合成中的跨说话人风格迁移是指将源说话人的风格迁移到目标说话人的合成语音中。在大多数先前方法中，合成的细粒度韵律特征通常表示源说话人的平均风格，类似于一对多问题（即，多个韵律变化对应于同一文本）。针对这一问题，提出一种强度控制的半监督风格提取器，将风格与内容、音色分离，提高全局风格嵌入的表示性和可解释性，从而缓解韵律预测中的一对多映射和数据不均衡问题。提出了一种分层韵律预测器来改进韵律建模。我们发现，利用源语者易于预测的韵律特征可以实现更好的风格迁移。此外，还提出了一种说话人迁移方式的循环一致性损失，以帮助模型在训练阶段学习看不见的风格-音色组合。实验结果表明，该方法优于基线方法。我们提供了一个网站与音频样本。摘要：Cross-speaker style transfer in speech synthesis aims at transferring a style from source speaker to synthesized speech of a target speaker's timbre. In most previous methods, the synthesized fine-grained prosody features often represent the source speaker's average style, similar to the one-to-many problem(i.e., multiple prosody variations correspond to the same text). In response to this problem, a strength-controlled semi-supervised style extractor is proposed to disentangle the style from content and timbre, improving the representation and interpretability of the global style embedding, which can alleviate the one-to-many mapping and data imbalance problems in prosody prediction. A hierarchical prosody predictor is proposed to improve prosody modeling. We find that better style transfer can be achieved by using the source speaker's prosody features that are easily predicted. Additionally, a speaker-transfer-wise cycle consistency loss is proposed to assist the model in learning unseen style-timbre combinations during the training phase. Experimental results show that the method outperforms the baseline. We provide a website with audio samples.
【22】 Dynamic Alignment Mask CTC: Improved Mask-CTC with Aligned Cross Entropy标题：动态对准掩模CTC：改进的对准交叉熵掩模CTC链接：http://arxiv.org/pdf/2303.07687v1
作者：Xulong Zhang,Haobin Tang,Jianzong Wang,Ning Cheng,Jian Luo,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., University of Science and Technology of China备注：Accepted by ICASSP 2023摘要：由于非自回归模型对所有目标标记进行并行预测，与传统的自回归模型相比，大大提高了语音识别的解码效率。本文提出了动态对准掩模CTC，介绍了两种方法：（1）对准交叉熵（AXE），通过动态编程找到使交叉熵损失最小化的单调对准，（2）动态校正，通过用模型预测的表征替换一些掩码来创建新的训练样本。AXE忽略了预测和地面事实句子之间的绝对位置对齐，而关注于相对顺序的标记匹配。动态校正方法使得模型能够模拟非掩码但可能错误的标记，即使它们具有高置信度。在WSJ数据集上的实验表明，AXE丢失和校正方法都能提高Mask CTC的WER性能。摘要：Because of predicting all the target tokens in parallel, the non-autoregressive models greatly improve the decoding efficiency of speech recognition compared with traditional autoregressive models. In this work, we present dynamic alignment Mask CTC, introducing two methods: (1) Aligned Cross Entropy (AXE), finding the monotonic alignment that minimizes the cross-entropy loss through dynamic programming, (2) Dynamic Rectification, creating new training samples by replacing some masks with model predicted tokens. The AXE ignores the absolute position alignment between prediction and ground truth sentence and focuses on tokens matching in relative order. The dynamic rectification method makes the model capable of simulating the non-mask but possible wrong tokens, even if they have high confidence. Our experiments on WSJ dataset demonstrated that not only AXE loss but also the rectification method could improve the WER performance of Mask CTC.
【23】 QI-TTS: Questioning Intonation Control for Emotional Speech Synthesis标题：QI-TTS：情感语音合成中的提问语调控制链接：http://arxiv.org/pdf/2303.07682v1
作者：Haobin Tang,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing XiaoWijayakulasooriya,Pantaleon Perera机构：Ping An Technology (Shenzhen) Co., Ltd., University of Science and Technology of China备注：Accepted by ICASSP 2023摘要：目前的表达性文语转换模型主要关注情感语音的合成，而忽略了语调等细粒度语音的合成。本文提出了QI-TTS，旨在更好地传递和控制语调，在传递参考语音情感的同时，进一步传递说话人的提问意图。我们提出一个多重样式撷取器，从两个不同层次撷取样式嵌入。句子级表示情感，最后的音节级表示语调。对于细粒度的语调控制，我们使用相对属性表示音节级的语调强度，实验验证了QI-TTS在情感语音合成中提高语调表达能力的有效性。摘要：Recent expressive text to speech (TTS) models focus on synthesizing emotional speech, but some fine-grained styles such as intonation are neglected. In this paper, we propose QI-TTS which aims to better transfer and control intonation to further deliver the speaker's questioning intention while transferring emotion from reference speech. We propose a multi-style extractor to extract style embedding from two different levels. While the sentence level represents emotion, the final syllable level represents intonation. For fine-grained intonation control, we use relative attributes to represent intonation intensity at the syllable level.Experiments have validated the effectiveness of QI-TTS for improving intonation expressiveness in emotional speech synthesis.
【24】 Cross-lingual Alzheimer's Disease detection based on paralinguistic and pre-trained features标题：基于副语言和预训练特征的跨语言阿尔茨海默病检测链接：http://arxiv.org/pdf/2303.07650v1
作者：Xuchu Chen,Yu Pu,Jinpeng Li,Wei-Qiang Zhang机构：Department of Electronic Engineering, Tsinghua University, Beijing, China备注：accepted by ICASSP 2023摘要：我们向ICASSP-SPGC-2023 ADReSS-M挑战任务提交了我们的意见书，该任务旨在研究哪些声学特征可以在阿尔茨海默病（AD）预测的语言之间进行概括和转移。这项挑战包括两项任务：一种是对AD患者和健康个体的语音进行分类，另一种是仅基于语音推断简易精神状态检查（MMSE）分数。困难主要体现在数据集的不匹配上，训练集是英文的，而测试集是希腊文的。利用openSmile工具包提取副语言特征，利用XLSR-53提取声学特征。此外，我们在将语音转录成文本之后提取语言特征。在我们的方法中，这些特征被用作AD检测的指标。我们的方法在分类任务上达到了69.6%的准确率，在回归任务上达到了4.788的均方根误差（RMSE）。实验结果表明，该方法有望实现基于自发语音的多语种阿尔茨海默病自动检测。摘要：We present our submission to the ICASSP-SPGC-2023 ADReSS-M Challenge Task, which aims to investigate which acoustic features can be generalized and transferred across languages for Alzheimer's Disease (AD) prediction. The challenge consists of two tasks: one is to classify the speech of AD patients and healthy individuals, and the other is to infer Mini Mental State Examination (MMSE) score based on speech only. The difficulty is mainly embodied in the mismatch of the dataset, in which the training set is in English while the test set is in Greek. We extract paralinguistic features using openSmile toolkit and acoustic features using XLSR-53. In addition, we extract linguistic features after transcribing the speech into text. These features are used as indicators for AD detection in our method. Our method achieves an accuracy of 69.6% on the classification task and a root mean squared error (RMSE) of 4.788 on the regression task. The results show that our proposed method is expected to achieve automatic multilingual Alzheimer's Disease detection through spontaneous speech.
【25】 Feature-Rich Audio Model Inversion for Data-Free Knowledge Distillation Towards General Sound Classification标题：面向一般声音分类的面向无数据知识提取的富特征音频模型反演链接：http://arxiv.org/pdf/2303.07643v1
作者：Zuheng Kang,Yayun He,Jianzong Wang,Junqing Peng,Xiaoyang Qu,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by ICASSP 2023. International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023)摘要：无数据知识提取（DFKD）近年来引起了学术界的广泛关注，尤其是计算机视觉领域的重大突破。尽管结果很有希望，但该技术还没有很好地应用于音频和信号处理。由于音频信号的持续时间是可变的，因此它有自己独特的建模方式。在这项工作中，我们提出了富特征音频模型反演（FRAMI），一个用于一般声音分类任务的无数据知识提取框架。该算法首先通过特征不变的对比度损失生成高质量、特征丰富的Mel谱图。然后，在对这些特征丰富的样本进行知识提取时，重用统计池层之前和之后的隐藏状态。在Urbansound 8 k、ESC-50和audioMNIST数据集上的实验结果表明，FRAMI能够生成特征丰富的样本.同时，通过重用隐藏状态进一步提高了学生模型的准确性，显著优于基线方法。摘要：Data-Free Knowledge Distillation (DFKD) has recently attracted growing attention in the academic community, especially with major breakthroughs in computer vision. Despite promising results, the technique has not been well applied to audio and signal processing. Due to the variable duration of audio signals, it has its own unique way of modeling. In this work, we propose feature-rich audio model inversion (FRAMI), a data-free knowledge distillation framework for general sound classification tasks. It first generates high-quality and feature-rich Mel-spectrograms through a feature-invariant contrastive loss. Then, the hidden states before and after the statistics pooling layer are reused when knowledge distillation is performed on these feature-rich samples. Experimental results on the Urbansound8k, ESC-50, and audioMNIST datasets demonstrate that FRAMI can generate feature-rich samples. Meanwhile, the accuracy of the student model is further improved by reusing the hidden state and significantly outperforms the baseline method.
【26】CAT: Causal Audio Transformer for Audio Classification标题：CAT：用于音频分类的因果音频转换器链接：http://arxiv.org/pdf/2303.07626v1
作者：Xiaoyu Liu,Hanlin Lu,Jianbo Yuan,Xinyu Li机构：University of Maryland, College Park,ByteDance备注：Accepted to ICASSP 2023摘要：基于注意力的Transformers由于其全局感受域和处理长时相关性的能力而越来越多地应用于音频分类。然而，现有的框架主要是从Vision Transformers扩展而来的，并不完全兼容音频信号。本文介绍了一种因果音频Transformer（CAT），它由多分辨率多特征（MRMF）特征提取和声学注意块组成，用于更优化的音频建模。此外，我们提出了一个因果模型，以减轻过度拟合，帮助知识转移，并提高可解释性。CAT在ESC50、AudioSet和UrbanSound8K数据集上获得了更高或相当的最新分类性能，并且可以轻松推广到其他基于Transformer的模型。摘要：The attention-based Transformers have been increasingly applied to audio classification because of their global receptive field and ability to handle long-term dependency. However, the existing frameworks which are mainly extended from the Vision Transformers are not perfectly compatible with audio signals. In this paper, we introduce a Causal Audio Transformer (CAT) consisting of a Multi-Resolution Multi-Feature (MRMF) feature extraction with an acoustic attention block for more optimized audio modeling. In addition, we propose a causal module that alleviates over-fitting, helps with knowledge transfer, and improves interpretability. CAT obtains higher or comparable state-of-the-art classification performance on ESC50, AudioSet and UrbanSound8K datasets, and can be easily generalized to other Transformer-based models.
【27】 I3D: Transformer architectures with input-dependent dynamic depth for speech recognition标题：I3D：具有输入相关动态深度的语音识别转换器体系结构
链接：http://arxiv.org/pdf/2303.07624v1
作者：Yifan Peng,Jaesong Lee,Shinji Watanabe机构：Carnegie Mellon University, NAVER Corporation备注：Accepted at ICASSP 2023摘要：基于变换器的端到端语音识别已经取得了很大的成功。然而，大量的内存占用和计算开销使得这些模型很难在一些实际应用中部署。模型压缩技术可以减小模型大小和加快推理速度，但压缩后的模型具有固定的结构，这可能是次优的。提出了一种新的基于输入相关动态深度（I3 D）的Transformer编码器，以实现性能-效率的折衷。在推理时具有相似数量的层的情况下，基于I3 D的模型优于vanilla Transformer和通过迭代层修剪的静态修剪模型。我们还对门概率和输入依赖性进行了有趣的分析，这有助于我们更好地理解深度编码器。摘要：Transformer-based end-to-end speech recognition has achieved great success. However, the large footprint and computational overhead make it difficult to deploy these models in some real-world applications. Model compression techniques can reduce the model size and speed up inference, but the compressed model has a fixed architecture which might be suboptimal. We propose a novel Transformer encoder with Input-Dependent Dynamic Depth (I3D) to achieve strong performance-efficiency trade-offs. With a similar number of layers at inference time, I3D-based models outperform the vanilla Transformer and the static pruned model via iterative layer pruning. We also present interesting analysis on the gate probabilities and the input-dependency, which helps us better understand deep encoders.
【28】 VANI: Very-lightweight Accent-controllable TTS for Native and Non-native speakers with Identity Preservation标题：Vani：非常轻量级、口音可控的TTS，适用于母语和非母语人士，具有身份保留功能链接：http://arxiv.org/pdf/2303.07578v1
作者：Rohan Badlani,Akshit Arora,Subhankar Ghosh,Rafael Valle,Kevin J. Shih,João Felipe Santos,Boris Ginsburg,Bryan Catanzaro机构：NVIDIA备注：Presentation accepted at ICASSP 2023摘要：本文介绍了一个轻量级的多语言重音可控语音合成系统VANI。该模型基于RADMMM提出的去纠缠策略，支持重音、语言、说话人以及细粒度的F0和能量特征的显式控制。我们利用印度语言数据集，作为ICASSP信号处理大挑战赛的一部分，为LIMMITS 2023发布，合成3种不同语言的语音。我们的模型支持转换说话者的语言，同时保留他们的声音和目标语言的本地口音。我们对赛道$1$采用大参数RADMMM模型，对赛道$2$和$3$采用轻量级VANI模型。摘要：We introduce VANI, a very lightweight multi-lingual accent controllable speech synthesis system. Our model builds upon disentanglement strategies proposed in RADMMM and supports explicit control of accent, language, speaker and fine-grained $F_0$ and energy features for speech synthesis. We utilize the Indic languages dataset, released for LIMMITS 2023 as part of ICASSP Signal Processing Grand Challenge, to synthesize speech in 3 different languages. Our model supports transferring the language of a speaker while retaining their voice and the native accent of the target language. We utilize the large-parameter RADMMM model for Track $1$ and lightweight VANI model for Track $2$ and $3$ of the competition.
【29】 HiSSNet: Sound Event Detection and Speaker Identification via Hierarchical Prototypical Networks for Low-Resource Headphones标题：HiSSNet：基于分层原型网络的低资源耳机声音事件检测与说话人识别链接：http://arxiv.org/pdf/2303.07538v1
作者：N Shashaank,Berker Banar,Mohammad Rasool Izadi,Jeremy Kemmerer,Shuo Zhang,Chuan-Che,Huang机构：Bose Corporation, MA, USA, Department of Computer Science, Columbia University, NY, USA, School of Electronic Engineering and Computer Science, Queen Mary University of London, UK摘要：现代的降噪耳机通过去除不必要的背景噪音，显著改善了用户的听觉体验，但它们也可以屏蔽掉对用户重要的声音。用于声音事件检测（SED）和说话人识别（SID）的机器学习（ML）模型可以使耳机选择性地通过重要声音;然而，为以用户为中心的体验实现这些模型提出了几个独特的挑战。首先，大多数人花有限的时间定制他们的耳机，所以声音检测应该工作得相当好的开箱即用。其次，模型应该能够随着时间的推移，根据用户的隐式和显式交互，学习对用户重要的特定声音。最后，这类型号应该有一个小的内存占用，以运行在低功耗耳机与有限的片上内存。本文提出使用HiSSNet（Hierarchical SED and SID Network）来解决这些问题。HiSSNet是一种SEID（SED和SID）模型，它使用分层原型网络来检测感兴趣的一般声音和特定声音，并表征警报类声音和语音声音。我们证明HiSSNet比使用非层次原型网络训练的SEID模型性能好6.9 - 8.6%。与专门针对SED或SID单独训练的最先进（SOTA）模型相比，HiSSNet可实现类似或更好的性能，同时减少支持设备上多种功能所需的内存占用。摘要：Modern noise-cancelling headphones have significantly improved users' auditory experiences by removing unwanted background noise, but they can also block out sounds that matter to users. Machine learning (ML) models for sound event detection (SED) and speaker identification (SID) can enable headphones to selectively pass through important sounds; however, implementing these models for a user-centric experience presents several unique challenges. First, most people spend limited time customizing their headphones, so the sound detection should work reasonably well out of the box. Second, the models should be able to learn over time the specific sounds that are important to users based on their implicit and explicit interactions. Finally, such models should have a small memory footprint to run on low-power headphones with limited on-chip memory. In this paper, we propose addressing these challenges using HiSSNet (Hierarchical SED and SID Network). HiSSNet is an SEID (SED and SID) model that uses a hierarchical prototypical network to detect both general and specific sounds of interest and characterize both alarm-like and speech sounds. We show that HiSSNet outperforms an SEID model trained using non-hierarchical prototypical networks by 6.9 - 8.6 percent. When compared to state-of-the-art (SOTA) models trained specifically for SED or SID alone, HiSSNet achieves similar or better performance while reducing the memory footprint required to support multiple capabilities on-device.
【30】 Beamformer-Guided Target Speaker Extraction标题：波束形成器引导的目标说话人提取
链接：http://arxiv.org/pdf/2303.08702v1
作者：Mohamed Elminshawi,Srikanth Raj Chetupalli,Emanuël A. P. Habets机构：International Audio Laboratories Erlangen†, Am Wolfsmantel , Erlangen, Germany备注：Submitted to the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)摘要：提出了一种基于波束形成器引导的目标说话人提取（BG-TSE）方法，用于从多通道录音中提取目标说话人的语音，该方法可以根据目标说话人的到达方向来确定目标说话人。所提出的方法采用朝向目标说话人操纵的前端波束形成器来向单通道TSE系统提供辅助信号。通过允许在单通道TSE块中进行时变嵌入，所提出的方法充分利用了前端波束形成器输出与麦克风信号中的目标语音之间的对应性。在消声和混响条件下对模拟的多通道2-说话人混合声进行了实验评估，结果表明该方法相对于现有的单通道和多通道基线方法具有优势。摘要：We propose a Beamformer-guided Target Speaker Extraction (BG-TSE) method to extract a target speaker's voice from a multi-channel recording informed by the direction of arrival of the target. The proposed method employs a front-end beamformer steered towards the target speaker to provide an auxiliary signal to a single-channel TSE system. By allowing for time-varying embeddings in the single-channel TSE block, the proposed method fully exploits the correspondence between the front-end beamformer output and the target speech in the microphone signal. Experimental evaluation on simulated multi-channel 2-speaker mixtures, in both anechoic and reverberant conditions, demonstrates the advantage of the proposed method compared to recent single-channel and multi-channel baselines.
【31】 Speech Signal Improvement Using Causal Generative Diffusion Models标题：基于因果生成扩散模型的语音信号改善链接：http://arxiv.org/pdf/2303.08674v1
作者：Julius Richter,Simon Welker,Jean-Marie Lemercier,Bunlong Lay,Tal Peer,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany备注：Accepted by ICASSP 2023摘要：在本文中，我们提出了一个因果语音信号改善系统，旨在处理不同类型的失真。该方法基于生成扩散模型，该模型已被证明在具有缺失数据和非线性损坏的场景中工作良好。为了保证因果处理，我们修改了我们先前工作的网络结构，用因果自适应增益控制代替全局归一化。我们生成包含广泛失真的各种训练数据。这项工作是在“ICASSP信号处理大挑战赛”的背景下进行的，并提交到“语音信号改进挑战赛2023”的非实时赛道，在那里排名第五。摘要：In this paper, we present a causal speech signal improvement system that is designed to handle different types of distortions. The method is based on a generative diffusion model which has been shown to work well in scenarios with missing data and non-linear corruptions. To guarantee causal processing, we modify the network architecture of our previous work and replace global normalization with causal adaptive gain control. We generate diverse training data containing a broad range of distortions. This work was performed in the context of an "ICASSP Signal Processing Grand Challenge" and submitted to the non-real-time track of the "Speech Signal Improvement Challenge 2023", where it was ranked fifth.
【32】 HYBRIDFORMER: improving SqueezeFormer with hybrid attention and NSR mechanism标题：HYBRIDFORMER：用混合注意和NSR机制改进SqueezeFormer链接：http://arxiv.org/pdf/2303.08636v1
作者：Yuguang Yang,Yu Pan,Jingjing Yin,Jiangyu Han,Lei Ma,Heng Lu机构：Ximalaya Inc., ShangHai, China, University of Alberta, Edmonton, Canada备注：Accepted by ICASSP2023摘要：SqueezeFormer最近在自动语音识别（ASR）方面表现出了令人印象深刻的性能。然而，其推理速度受到softmax-attention（SA）算法二次复杂度的影响。此外，受卷积核尺寸较大的限制，SqueezeFormer的局部建模能力不足。本文提出了一种新的HybridFormer方法来快速有效地改进SqueezeFormer。具体来说，我们首先引入线性注意（LA），并提出一种混合LASA范式来提高模型的推理速度。其次，提出了一种混合神经结构搜索（NAS）引导的结构重参数化（SRep）机制，称为NSR，以增强模型提取局部交互作用的能力.在LibriSpeech数据集上进行的大量实验表明，与SqueezeFormer相比，我们提出的HybridFormer在test-other数据集上的相对单词错误率（WER）降低了9.1%。此外，当输入语音为30 s时，HybridFormer可将模型的推理速度提高18%。我们的源代码可以在线获得。摘要：SqueezeFormer has recently shown impressive performance in automatic speech recognition (ASR). However, its inference speed suffers from the quadratic complexity of softmax-attention (SA). In addition, limited by the large convolution kernel size, the local modeling ability of SqueezeFormer is insufficient. In this paper, we propose a novel method HybridFormer to improve SqueezeFormer in a fast and efficient way. Specifically, we first incorporate linear attention (LA) and propose a hybrid LASA paradigm to increase the model's inference speed. Second, a hybrid neural architecture search (NAS) guided structural re-parameterization (SRep) mechanism, termed NSR, is proposed to enhance the ability of the model to extract local interactions. Extensive experiments conducted on the LibriSpeech dataset demonstrate that our proposed HybridFormer can achieve a 9.1% relative word error rate (WER) reduction over SqueezeFormer on the test-other dataset. Furthermore, when input speech is 30s, the HybridFormer can improve the model's inference speed up to 18%. Our source code is available online.
【33】 Acoustic source localization in the spherical harmonics domain exploiting low-rank approximations标题：基于低阶近似的球谐波域声源定位链接：http://arxiv.org/pdf/2303.08480v1
作者：Maximo Cobos,Mirco Pezzoli,Fabio Antonacci,Augusto Sarti机构：Departament d’Inform´atica, Universitat de Valencia, Valencia, Spain, †Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, ItalyJournal-ref：2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：球谐函数域（spherical harmonics domain，SHD）中的声学信号处理是利用高阶传声器阵列采集的信号的一个活跃的研究领域。一个非常重要的任务是关于有源声源的定位。本文提出了一种简单而有效的方法来定位不利声学场景中的显著声源。通过对估计的球谐系数进行适当的归一化和排列，我们利用低秩近似来估计主源在每个时间帧的远场模态方向图。实验证明了该方法的有效性，与其他基于SHD的方法相比具有更好的性能。摘要：Acoustic signal processing in the spherical harmonics domain (SHD) is an active research area that exploits the signals acquired by higher order microphone arrays. A very important task is that concerning the localization of active sound sources. In this paper, we propose a simple yet effective method to localize prominent acoustic sources in adverse acoustic scenarios. By using a proper normalization and arrangement of the estimated spherical harmonic coefficients, we exploit low-rank approximations to estimate the far field modal directional pattern of the dominant source at each time-frame. The experiments confirm the validity of the proposed approach, with superior performance compared to other recent SHD-based approaches.
【34】 A practical distributed active noise control algorithm overcoming communication restrictions标题：一种克服通信限制的实用分布式有源噪声控制算法链接：http://arxiv.org/pdf/2303.08411v1
作者：Junwei Ji,Dongyuan Shi,Zhengding Luo,Xiaoyi Shen,Woon-Seng Gan机构：School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore.摘要：分布式多通道有源噪声控制（DMCANC）技术将传统多通道有源噪声控制（MCANC）系统中的大量计算任务分配给多个分布式控制节点，成为一种有效的全局降噪解决方案。然而，现有的DMCANC算法只是简单地通过结合邻居节点的信息来完成传统集中式算法的分散，而很少考虑分布式单元由于通信延迟和中断而导致的控制性能和系统稳定性下降。因此，本文提出了一种新的DMCANC算法，该算法利用补偿滤波器和邻居节点信息来抵消信道间的串扰效应，同时保持独立的权值更新。由于所需的邻居信息几乎不影响每个节点的局部控制滤波器更新，因此该方法可以在一定程度上容忍通信延迟和中断。仿真结果表明，该算法具有较好的降噪性能和较强的鲁棒性。摘要：By assigning the massive computing tasks of the traditional multichannel active noise control (MCANC) system to several distributed control nodes, distributed multichannel active noise control (DMCANC) techniques have become effective global noise reduction solutions with low computational costs. However, existing DMCANC algorithms simply complete the distribution of traditional centralized algorithms by combining neighbour nodes' information but rarely consider the degraded control performance and system stability of distributed units caused by delays and interruptions in communication. Hence, this paper develops a novel DMCANC algorithm that utilizes the compensation filters and neighbour nodes' information to counterbalance the cross-talk effect between channels while maintaining independent weight updating. Since the neighbours' information required barely affects the local control filter updating in each node, this approach can tolerate communication delay and interruption to some extent. Numerical simulations demonstrate that the proposed algorithm can achieve satisfactory noise reduction performance and high robustness to real-world communication challenges.
【35】A Momentum Two-gradient Direction Algorithm with Variable Step Size Applied to Solve Practical Output Constraint Issue for Active Noise Control标题：变步长动量双梯度方向算法在有源噪声控制实际输出约束问题中的应用链接：http://arxiv.org/pdf/2303.08397v1
作者：Xiaoyi Shen,Dongyuan Shi,Zhengding Luo,Junwei Ji,Woon-Seng Gan机构：School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore.备注：Paper is submitted and accepted by ICASSP2023摘要：有源噪声控制（ANC）已被广泛用于降低不期望的环境噪声。ANC的主要目的是使用次级源产生与初级噪声幅度相同但相位相反的抗噪声。然而，ANC应用的有效性受到扬声器输出饱和度的影响。本文提出了一种带动量因子的双梯度方向ANC算法，以更快的收敛速度解决饱和问题。为了使算法能够实时实现，采用了一种计算有效的变步长方法，进一步减小了梯度方向变化带来的稳态误差。分析了动量双梯度方向算法的时间常数和步长界。仿真结果表明，该算法在时变和非时变环境下均具有良好的性能。摘要：Active noise control (ANC) has been widely utilized to reduce unwanted environmental noise. The primary objective of ANC is to generate an anti-noise with the same amplitude but the opposite phase of the primary noise using the secondary source. However, the effectiveness of the ANC application is impacted by the speaker's output saturation. This paper proposes a two-gradient direction ANC algorithm with a momentum factor to solve the saturation with faster convergence. In order to make it implemented in real-time, a computation-effective variable step size approach is applied to further reduce the steady-state error brought on by the changing gradient directions. The time constant and step size bound for the momentum two-gradient direction algorithm is analyzed. Simulation results show that the proposed algorithm performs effectively in the time-unvaried and time-varied environment.
【36】 Implementing Continuous HRTF Measurement in Near-Field标题：实现近场HRTF连续测量链接：http://arxiv.org/pdf/2303.08379v1
作者：Ee-Leng Tan,Santi Peksi,Woon-Seng Gan机构：School of Electrical Engineering, Nanyang Technological University, Singapore备注：5 pages, 9 figures, Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：头部相关传递函数（HRTF）是通过耳机为虚拟现实（VR）和增强现实（AR）应用创造沉浸式听觉体验的重要组成部分。Metaverse将VR和AR相结合，打造沉浸式数字体验，用户极有可能在近场（NF）与虚拟物体进行交互。这种物体的HRTF是高度个性化的，并且依赖于方向和距离。因此，需要在NF中的不同距离处进行大量的HRTF测量。使用传统的静态停-走HRTF测量方法来获取这些测量结果对于人类收听者来说将是耗时和乏味的。在本文中，我们提出了一个针对NF的连续测量系统，并在45秒内有效地捕获水平面上的HRTF。在头和躯干相似体（HATS）和人类听者上进行了对比实验，以评估系统的一致性和鲁棒性。摘要：Head-related transfer function (HRTF) is an essential component to create an immersive listening experience over headphones for virtual reality (VR) and augmented reality (AR) applications. Metaverse combines VR and AR to create immersive digital experiences, and users are very likely to interact with virtual objects in the near-field (NF). The HRTFs of such objects are highly individualized and dependent on directions and distances. Hence, a significant number of HRTF measurements at different distances in the NF would be needed. Using conventional static stop-and-go HRTF measurement methods to acquire these measurements would be time-consuming and tedious for human listeners. In this paper, we propose a continuous measurement system targeted for the NF, and efficiently capturing HRTFs in the horizontal plane within 45 secs. Comparative experiments are performed on head and torso similar (HATS) and human listeners to evaluate system consistency and robustness.
【37】 Target Sound Extraction with Variable Cross-modality Clues标题：基于可变交叉通道线索的目标声提取链接：http://arxiv.org/pdf/2303.08372v1
作者：Chenda Li,Yao Qian,Zhuo Chen,Dongmei Wang,Takuya Yoshioka,Shujie Liu,Yanmin Qian,Michael Zeng机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Microsoft, Redmond, WA, USA备注：Accepted by ICASSP 2023摘要：自动目标声提取（TSE）是一种机器学习方法，用于模拟人类从混合源中注意感兴趣的声源的听觉感知能力。它经常使用以固定形式的目标声音线索为条件的模型，例如声音类标签，这限制了用户可以与模型交互以指定目标声音的方式。为了充分利用在推理阶段可获得的不同数量的跨模态线索，包括视频、声音事件类和文本字幕，我们提出了一个统一的基于变换器的TSE模型架构，其中多线索注意模块集成了跨模态的所有线索。由于没有现成的基准来评估我们提出的方法，我们建立了一个基于公共语料库，Audioset和AudioCaps的数据集。在可见和不可见目标声音评价集上的实验结果表明，本文提出的TSE模型能够有效地处理变化数量的线索，提高了TSE的性能和对部分线索受损的鲁棒性。摘要：Automatic target sound extraction (TSE) is a machine learning approach to mimic the human auditory perception capability of attending to a sound source of interest from a mixture of sources. It often uses a model conditioned on a fixed form of target sound clues, such as a sound class label, which limits the ways in which users can interact with the model to specify the target sounds. To leverage variable number of clues cross modalities available in the inference phase, including a video, a sound event class, and a text caption, we propose a unified transformer-based TSE model architecture, where a multi-clue attention module integrates all the clues across the modalities. Since there is no off-the-shelf benchmark to evaluate our proposed approach, we build a dataset based on public corpora, Audioset and AudioCaps. Experimental results for seen and unseen target-sound evaluation sets show that our proposed TSE model can effectively deal with a varying number of clues which improves the TSE performance and robustness against partially compromised clues.
【38】 Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models标题：微小常通环境语音识别模型中共享低阶同名权的研究链接：http://arxiv.org/pdf/2303.08343v1
作者：Steven M. Hernandez,Ding Zhao,Shaojin Ding,Antoine Bruguier,Rohit Prabhavalkar,Tara N. Sainath,Yanzhang He,Ian McGraw机构：Virginia Commonwealth University,Google备注：Accepted to IEEE ICASSP 2023摘要：机器学习技术的不断改进通过使用更大的模型和更大的训练数据集提供了令人兴奋的新机会。然而，在低功耗设备（如智能手机、可穿戴设备和其他只有低内存可用的嵌入式环境）上提供这些新功能的需求日益增长。为此，我们考虑用于将基于一致器的语音识别模型的模型大小减小到仅$5$M参数同时最小化对模型质量的影响的方法，所述基于一致器的语音识别模型通常需要具有大于1亿个参数的模型。这种模型允许我们在具有低存储器神经处理器的边缘设备上实现始终在线的环境语音识别。我们建议在模型架构内的不同级别重用模型权重：（i）重复全构象异构体块层，（ii）跨层共享特定构象异构体模块，（iii）共享每个构象异构体模块的子组分，和（iv）共享低秩分解后的分解子组分权重。通过在模型的不同级别共享权重，我们可以在内存中保留完整的模型，同时增加应用于输入的虚拟转换的数量。通过一系列的消融研究和评估，我们发现通过权重共享和低秩架构，我们可以在$5$M参数模型下分别实现Librispeech dev-clean和test-clean的WER为2.84和2.94。摘要：Continued improvements in machine learning techniques offer exciting new opportunities through the use of larger models and larger training datasets. However, there is a growing need to offer these new capabilities on-board low-powered devices such as smartphones, wearables and other embedded environments where only low memory is available. Towards this, we consider methods to reduce the model size of Conformer-based speech recognition models which typically require models with greater than 100M parameters down to just $5$M parameters while minimizing impact on model quality. Such a model allows us to achieve always-on ambient speech recognition on edge devices with low-memory neural processors. We propose model weight reuse at different levels within our model architecture: (i) repeating full conformer block layers, (ii) sharing specific conformer modules across layers, (iii) sharing sub-components per conformer module, and (iv) sharing decomposed sub-component weights after low-rank decomposition. By sharing weights at different levels of our model, we can retain the full model in-memory while increasing the number of virtual transformations applied to the input. Through a series of ablation studies and evaluations, we find that with weight sharing and a low-rank architecture, we can achieve a WER of 2.84 and 2.94 for Librispeech dev-clean and test-clean respectively with a $5$M parameter model.
【39】 A large-scale multimodal dataset of human speech recognition标题：一种大规模的多模式人类语音识别数据集链接：http://arxiv.org/pdf/2303.08295v1
作者：Yao Ge,Chong Tang,Haobo Li,Zikang Zhang,Wenda Li,Kevin Chetty,Daniele Faccio,Qammer H. Abbasi,Muhammad Imran机构：James Watt School of Engineering, University of Glasgow, Glasgow, G,QQ,UK, School of Physics & Astronomy, University of Glasgow, Glasgow, G,QQ,UK, Department of Security and Crime Science, University College London, London, WC,E ,BT, UK摘要：目前，非隐私小尺度运动检测在遥感语音识别中的研究越来越多。这些新模态被用于增强和恢复来自说话者的多种类型数据的语音信息。本文提出了一个包含超宽带（UWB）雷达的7.5 GHz信道脉冲响应（CIR）数据、毫米波（mmWave）雷达的77 GHz调频连续波（FMCW）数据和激光数据的数据集。同时，采用深度摄像机记录被试嘴唇和声音的特征点。提供了大约400分钟的注释语音简档，其是从20个参与者讲5个元音、15个单词和16个句子收集的。该数据集已经过验证，具有唇读和多模态语音识别研究的潜力。摘要：Nowadays, non-privacy small-scale motion detection has attracted an increasing amount of research in remote sensing in speech recognition. These new modalities are employed to enhance and restore speech information from speakers of multiple types of data. In this paper, we propose a dataset contains 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77-GHz frequency modulated continuous wave (FMCW) data from millimetre wave (mmWave) radar, and laser data. Meanwhile, a depth camera is adopted to record the landmarks of the subject's lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words and 16 sentences. The dataset has been validated and has potential for the research of lip reading and multimodal speech recognition.
【40】 Cascading and Direct Approaches to Unsupervised Constituency Parsing on Spoken Sentences标题：级联和直接无监督的口语句法分析方法链接：http://arxiv.org/pdf/2303.08809v1
作者：Yuan Tseng,Cheng-I Lai,Hung-yi Lee机构：Cheng-I Jeff Lai, National Taiwan University, MIT CSAIL备注：Accepted to ICASSP 2023摘要：过去关于无监督句法分析的工作局限于书面形式。在这篇论文中，我们提出了第一个在无标记口语句子和无配对文本数据下的无监督口语成分分析的研究。目标是以构成成分分析树的形式确定口语句子的分层句法结构，使得每个节点是对应于构成成分的音频跨度。我们比较两种方法：（1）级联无监督自动语音识别（ASR）模型和无监督语法分析器以获得关于ASR抄本的语法分析树，以及（2）在连续词级语音表示上直接训练无监督语法分析器。这是通过首先将话语分割成单词级片段序列，并且聚集片段内的自监督语音表示以获得片段嵌入来完成的。我们发现，单独地在不成对文本上训练解析器并直接将其应用于ASR转录本进行推理产生了更好的无监督解析结果。此外，我们的结果表明，准确的分割本身可能足以准确地分析口语句子。最后，我们证明了直接方法可以正确地学习中心语首和中心语尾的中心语方向性，而不存在任何明显的归纳偏见。摘要：Past work on unsupervised parsing is constrained to written form. In this paper, we present the first study on unsupervised spoken constituency parsing given unlabeled spoken sentences and unpaired textual data. The goal is to determine the spoken sentences' hierarchical syntactic structure in the form of constituency parse trees, such that each node is a span of audio that corresponds to a constituent. We compare two approaches: (1) cascading an unsupervised automatic speech recognition (ASR) model and an unsupervised parser to obtain parse trees on ASR transcripts, and (2) direct training an unsupervised parser on continuous word-level speech representations. This is done by first splitting utterances into sequences of word-level segments, and aggregating self-supervised speech representations within segments to obtain segment embeddings. We find that separately training a parser on the unpaired text and directly applying it on ASR transcripts for inference produces better results for unsupervised parsing. Additionally, our results suggest that accurate segmentation alone may be sufficient to parse spoken sentences accurately. Finally, we show the direct approach may learn head-directionality correctly for both head-initial and head-final languages without any explicit inductive bias.
【41】 Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video标题：深度视觉强制对齐：学习将转录与人脸对话视频对齐链接：http://arxiv.org/pdf/2303.08670v1
作者：Minsu Kim,Chae Won Kim,Yong Man Ro机构：Image and Video Systems Lab, KAIST, South Korea备注：Accepted in AAAI2023摘要：强制对齐指的是将给定的转录与对应的语音时间对齐的技术。然而，由于已经使用语音音频开发了强制对准技术，所以当输入语音音频被噪声破坏或不可访问时，强制对准技术可能对准失败。我们关注于存在可以从其推断语音的另一个组件，语音视频（即，正在说话的面部视频）。针对基于音频的强制对准方法在音频信号条件较差时可以利用视频信息进行补偿的不足，提出了一种基于视频的强制对准方法。然而，与音频强制对准不同，开发可靠的视觉强制对准技术具有挑战性，原因如下两个：1）视觉语音识别（VSR）与基于音频的自动语音识别（ASR）相比具有低得多的性能，以及2）从文本到视频的转换不可靠，因此通常用于建立音频强制对准的方法不能用于开发视觉强制对准。为了解决这些问题，本文提出了一种适合于视觉强制对准的新方法，即深度视觉强制对准（DVFA）。所提出的DVFA可以比对输入转录（即，句子）与正在说话的面部视频进行比较，而不访问语音音频。此外，通过用异常情况检测来扩充对准任务，DVFA可以在执行对准的同时检测输入转录和输入视频之间的失配。因此，即使文本中存在错误单词，我们也可以鲁棒地将文本与说话的面部视频对齐。通过大量的实验，我们证明了所提出的DVFA算法不仅在对齐任务中有效，而且在解释VSR模型的输出方面也是有效的。摘要：Forced alignment refers to a technology that time-aligns a given transcription with a corresponding speech. However, as the forced alignment technologies have developed using speech audio, they might fail in alignment when the input speech audio is noise-corrupted or is not accessible. We focus on that there is another component that the speech can be inferred from, the speech video (i.e., talking face video). Since the drawbacks of audio-based forced alignment can be complemented using the visual information when the audio signal is under poor condition, we try to develop a novel video-based forced alignment method. However, different from audio forced alignment, it is challenging to develop a reliable visual forced alignment technology for the following two reasons: 1) Visual Speech Recognition (VSR) has a much lower performance compared to audio-based Automatic Speech Recognition (ASR), and 2) the translation from text to video is not reliable, so the method typically used for building audio forced alignment cannot be utilized in developing visual forced alignment. In order to alleviate these challenges, in this paper, we propose a new method that is appropriate for visual forced alignment, namely Deep Visual Forced Alignment (DVFA). The proposed DVFA can align the input transcription (i.e., sentence) with the talking face video without accessing the speech audio. Moreover, by augmenting the alignment task with anomaly case detection, DVFA can detect mismatches between the input transcription and the input video while performing the alignment. Therefore, we can robustly align the text with the talking face video even if there exist error words in the text. Through extensive experiments, we show the effectiveness of the proposed DVFA not only in the alignment task but also in interpreting the outputs of VSR models.
【42】 Blind Estimation of Audio Processing Graph标题：音频处理图的盲估计算法链接：http://arxiv.org/pdf/2303.08610v1
作者：Sungho Lee,Jaehyun Park,Seungryeol Paik,Kyogu Lee机构：Department of Intelligence and Information, Seoul National University, Interdisciplinary Program in Artificial Intelligence, Seoul National University, Artificial Intelligence Institute, Seoul National University备注：Accepted to ICASSP 2023摘要：音乐家和音频工程师通过连接多个处理器来塑造和转换声音，形成一个音频处理图。然而，大多数深度学习方法忽略了这种现实世界的实践，并假设固定的图形设置。为了弥补这一差距，我们开发了一个系统，重建整个图形从一个给定的参考音频。我们首先产生一个真实的图参考对数据集，并训练一个简单的盲估计系统，该系统由卷积参考编码器和基于变换的图解码器组成。我们将我们的模型应用于歌唱声音效果和鼓混合估计任务。仿真结果表明，该方法能够重构复杂的信号路由，包括多频带处理和边链处理。摘要：Musicians and audio engineers sculpt and transform their sounds by connecting multiple processors, forming an audio processing graph. However, most deep-learning methods overlook this real-world practice and assume fixed graph settings. To bridge this gap, we develop a system that reconstructs the entire graph from a given reference audio. We first generate a realistic graph-reference pair dataset and train a simple blind estimation system composed of a convolutional reference encoder and a transformer-based graph decoder. We apply our model to singing voice effects and drum mixing estimation tasks. Evaluation results show that our method can reconstruct complex signal routings, including multi-band processing and sidechaining.
【43】 PHONEix: Acoustic Feature Processing Strategy for Enhanced Singing Pronunciation with Phoneme Distribution Predictor标题：Phoneix：音素分布预测器增强歌唱发音的声学特征处理策略链接：http://arxiv.org/pdf/2303.08607v1
作者：Yuning Wu,Jiatong Shi,Tao Qian,Dongji Gao,Qin Jin机构：School of Information, Renmin University of China, Language Technologies Institute, Carnegie Mellon University, Center for Language and Speech Processing, Johns Hopkins University备注：Accepted by ICASSP 2023摘要：歌唱声合成作为一项从乐谱中生成歌唱声的具体任务，近年来引起了广泛的关注。SVS面临着在同一乐谱条件下，歌唱发音灵活性变化的挑战。SVS之前的大部分作品都不能很好地处理乐谱和实际演唱之间的错位。本文提出了一种带有音素分布预测器的声学特征处理策略PHONEix，以减小乐谱和人声之间的差距，该策略可以方便地应用于不同的SVS系统。在各种环境下进行的大量实验证明了PHONEix在客观和主观评价方面的有效性。摘要：Singing voice synthesis (SVS), as a specific task for generating the vocal singing voice from a music score, has drawn much attention in recent years. SVS faces the challenge that the singing has various pronunciation flexibility conditioned on the same music score. Most of the previous works of SVS can not well handle the misalignment between the music score and actual singing. In this paper, we propose an acoustic feature processing strategy, named PHONEix, with a phoneme distribution predictor, to alleviate the gap between the music score and the singing voice, which can be easily adopted in different SVS systems. Extensive experiments in various settings demonstrate the effectiveness of our PHONEix in both objective and subjective evaluations.
【44】 Enhancing Unsupervised Audio Representation Learning via Adversarial Sample Generation标题：通过对抗性样本生成增强无监督音频表示学习链接：http://arxiv.org/pdf/2303.08561v1
作者：Yulin Pan,Xiangteng He,Biao Gong,Yuxin Peng,Yiliang Lv机构：Alibaba Group, Hangzhou, China, Wangxuan Institute of Computer Technology, Peking University, Beijing, China备注：8 pages, 4 figures摘要：现有的音频分析方法一般先将音频流转换为声谱图，然后送入CNN进行进一步分析。标准的CNN在特征地图上识别特定的视觉模式，然后进行高级表示，该高级表示忽略了识别模式的位置信息。然而，与自然图像不同，音频谱图的语义对位置变化敏感，因为其垂直轴和水平轴指示音频的频率和时间信息，而不是简单的直角坐标。因此，CNN对位置变化的不敏感性对音频谱图编码起到了负面作用。为了解决这个问题，本文提出了一种新的自监督学习机制，其通过首先生成对抗样本（ textit{即}，负样本），然后驱动CNN来区分负对在潜在空间中的嵌入。大量实验表明，与已有方法相比，该方法在9个下游数据集上取得了最优或有竞争力的结果，验证了其在音频表示学习中的有效性。摘要：Existing audio analysis methods generally first transform the audio stream to spectrogram, and then feed it into CNN for further analysis. A standard CNN recognizes specific visual patterns over feature map, then pools for high-level representation, which overlooks the positional information of recognized patterns. However, unlike natural image, the semantic of an audio spectrogram is sensitive to positional change, as its vertical and horizontal axes indicate the frequency and temporal information of the audio, instead of naive rectangular coordinates. Thus, the insensitivity of CNN to positional change plays a negative role on audio spectrogram encoding. To address this issue, this paper proposes a new self-supervised learning mechanism, which enhances the audio representation by first generating adversarial samples ( textit{i.e.}, negative samples), then driving CNN to distinguish the embeddings of negative pairs in the latent space. Extensive experiments show that the proposed approach achieves best or competitive results on 9 downstream datasets compared with previous methods, which verifies its effectiveness on audio representation learning.【45】 Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring标题：观看或收听：具有视觉腐败建模和可靠性评分的健壮视听语音识别链接：http://arxiv.org/pdf/2303.08536v1
作者：Joanna Hong,Minsu Kim,Jeongsoo Choi,Yong Man Ro机构：Image and Video Systems Lab, KAIST备注：Accepted at CVPR 2023. Implementation available: this https URL摘要：本文研究了多模态输入错误情况下的音视频语音识别（AVSR）问题，即音频和视频输入同时受到错误影响的情况，这一问题在以往的研究方向中没有得到很好的解决。先前的研究集中在如何在干净的视觉输入可用的假设下用干净的视觉输入来补充损坏的音频输入。然而，在现实生活中，干净的视觉输入并不总是可获得的，并且甚至可能被遮挡的嘴唇区域或噪声破坏。因此，我们首先分析了与单模态模型相比，以前的AVSR模型对多模态输入流（音频和视频输入）的破坏并不真正具有鲁棒性。然后，我们设计了多模态输入损坏模型来发展强健的AVSR模型。最后，我们提出了一个新的AVSR框架，即音视频可靠性评分模块（AV-RelScore），它对多模态输入具有鲁棒性。AV-RelScore可以确定哪个输入模态流对于预测是可靠的或不可靠的，并且还可以在预测中利用更可靠的流。在LRS 2和LRS 3两个主流数据库上的综合实验验证了该方法的有效性。结果表明，AV-RelScore得到的可靠性分数能很好地反映多模态表示的破坏程度，使模型更关注可靠的多模态表示.摘要：This paper deals with Audio-Visual Speech Recognition (AVSR) under multimodal input corruption situations where audio inputs and visual inputs are both corrupted, which is not well addressed in previous research directions. Previous studies have focused on how to complement the corrupted audio inputs with the clean visual inputs with the assumption of the availability of clean visual inputs. However, in real life, clean visual inputs are not always accessible and can even be corrupted by occluded lip regions or noises. Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models. Then, we design multimodal input corruption modeling to develop robust AVSR models. Lastly, we propose a novel AVSR framework, namely Audio-Visual Reliability Scoring module (AV-RelScore), that is robust to the corrupted multimodal inputs. The AV-RelScore can determine which input modal stream is reliable or not for the prediction and also can exploit the more reliable streams in prediction. The effectiveness of the proposed method is evaluated with comprehensive experiments on popular benchmark databases, LRS2 and LRS3. We also show that the reliability scores obtained by AV-RelScore well reflect the degree of corruption and make the proposed model focus on the reliable multimodal representations.
【46】 Generating symbolic music using diffusion models标题：使用扩散模型生成符号音乐链接：http://arxiv.org/pdf/2303.08385v1
作者：Lilac Atassi机构：University of California, San Diego摘要：概率去噪扩散模型已经成为简单但非常强大的生成模型。扩散模型不像其他生成模型那样遭受模式崩溃，也不需要鉴别器来生成高质量样本。在这篇文章中，我们提出了一个扩散模型，使用二项式先验分布来产生钢琴卷。文中还提出了一种有效的模型训练和样本生成方法。所生成的音乐在上至训练钢琴卷片段的长度的时间尺度上具有连贯性。我们展示了这样一个模型是如何受输入条件的影响，并可以用来协调一个给定的旋律，完成一个不完整的钢琴卷或产生一个给定的作品的变化。代码是公开共享的，以鼓励社区使用和开发该方法。摘要：Probabilistic Denoising Diffusion models have emerged as simple yet very powerful generative models. Diffusion models unlike other generative models do not suffer from mode collapse nor require a discriminator to generate high quality samples. In this paper, we propose a diffusion model that uses a binomial prior distribution to generate piano-rolls. The paper also proposes an efficient method to train the model and generate samples. The generated music has coherence at time scales up to the length of the training piano-roll segments. We show how such a model is conditioned on the input and can be used to harmonize a given melody, complete an incomplete piano-roll or generate a variation of a given piece. The code is shared publicly to encourage the use and development of the method by the community.
【47】 Transfer Learning Based Diagnosis and Analysis of Lung Sound Aberrations标题：基于迁移学习的肺音异常诊断与分析链接：http://arxiv.org/pdf/2303.08362v1
作者：Hafsa Gulzar,Jiyun Li,Arslan Manzoor,Sadaf Rehmat,Usman Amjad,Hadiqa Jalil Khan机构：Computer Science and Technology, Donghua University, Shanghai, China, Mathematics and Computer Science, University of Catania, Catania, Italy, Computer Science Department, PIEAS, Islamabad, Pakistan备注：12 pages, 9 figures摘要：随着能够收集和分析大量数据的计算机系统的发展，医学界正在建立几种非侵入性工具。这项工作试图开发一种非侵入性技术，用于通过机器学习技术识别由听诊器和语音记录软件采集的呼吸音。本研究提出了一种经过训练和验证的基于神经网络的呼吸音分类方法。构造每个音频样本的视觉表示，从而允许使用类似于用于有效描述视觉的那些方法的方法来进行分类的资源标识。我们使用了一种叫做梅尔倒谱系数（MFCC）的技术。这里，通过VGG16（迁移学习）检索和分类特征，并使用5重交叉验证完成预测。呼吸音数据库采用多种数据分割技术，取得了最先进的结果，准确率为95%，精确度为88%，召回率为86%，F1评分为81%。ICBHI数据集用于训练和测试模型。摘要：With the development of computer -systems that can collect and analyze enormous volumes of data, the medical profession is establishing several non-invasive tools. This work attempts to develop a non-invasive technique for identifying respiratory sounds acquired by a stethoscope and voice recording software via machine learning techniques. This study suggests a trained and proven CNN-based approach for categorizing respiratory sounds. A visual representation of each audio sample is constructed, allowing resource identification for classification using methods like those used to effectively describe visuals. We used a technique called Mel Frequency Cepstral Coefficients (MFCCs). Here, features are retrieved and categorized via VGG16 (transfer learning) and prediction is accomplished using 5-fold cross-validation. Employing various data splitting techniques, Respiratory Sound Database obtained cutting-edge results, including accuracy of 95%, precision of 88%, recall score of 86%, and F1 score of 81%. The ICBHI dataset is used to train and test the model.
【48】 Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs标题：基于视觉输入和参与者链接输入的多模式融合的自主声景增强链接：http://arxiv.org/pdf/2303.08342v1
作者：Kenneth Ooi,Karn N. Watcharasupat,Bhan Lam,Zhen-Ting Ong,Woon-Seng Gan机构：School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, Center for Music Technology, Georgia Institute of Technology, Atlanta, GA, USA备注：5 pages, 2 figures. Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：自主声景增强系统通常使用经训练的模型来挑选最佳掩蔽器以实现期望的感知改变。虽然声学信息对于这样的系统是至关重要的，但是包括参与者人口统计和视觉环境的上下文信息也影响声学感知。因此，我们建议对现有的基于注意力的深度神经网络进行模块化修改，以允许参与者关联、视觉和声学特征的早期、中期和晚期特征融合。使用ARAUS数据集对模块配置和相应融合方法进行的消融研究表明，上下文特征在统计学上显著改善了标准化ISO愉悦度的模型性能，性能最佳的全模态模型的均方误差为$0.1194 pm0.0012$，而纯音频模型的均方误差为$0.1217 pm0.0009$。声景增强系统由此可以利用多模态输入来改进性能。我们还调查了个别参与者相关因素的影响，使用训练模型来说明模型解释能力的提高。摘要：Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of $0.1194 pm0.0012$ for the best-performing all-modality model, against $0.1217 pm0.0009$ for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability.
【49】 Cross-speaker Emotion Transfer by Manipulating Speech Style Latents标题：操纵语体潜伏期的跨说话人情绪传递链接：http://arxiv.org/pdf/2303.08329v1
作者：Suhee Jo,Younggun Lee,Yookyung Shin,Yeongtae Hwang,Taesu Kim机构：Neosapience, Inc.备注：accepted to ICASSP 2023摘要：近年来，情感化文本语音转换取得了长足的发展。但是，它需要大量的标记数据，这些数据不容易访问。即使可以获取情感语音数据集，在控制情感强度方面仍然存在限制。在本论文中，我们提出了一种在潜在风格空间中使用向量运算的跨说话人情感传递和操纵的新方法。通过只利用少量的标记样本，我们在不丢失说话人身份的情况下从阅读风格的语音中生成情感语音。此外，情感强度可使用标量值容易地控制，从而为用户提供了操纵语音的直观方式。实验结果表明，该方法在表达性、自然性、可控性等方面具有较好的性能，并能保持说话人身份。摘要：In recent years, emotional text-to-speech has shown considerable progress. However, it requires a large amount of labeled data, which is not easily accessible. Even if it is possible to acquire an emotional speech dataset, there is still a limitation in controlling emotion intensity. In this work, we propose a novel method for cross-speaker emotion transfer and manipulation using vector arithmetic in latent style space. By leveraging only a few labeled samples, we generate emotional speech from reading-style speech without losing the speaker identity. Furthermore, emotion strength is readily controllable using a scalar value, providing an intuitive way for users to manipulate speech. Experimental results show the proposed method affords superior performance in terms of expressiveness, naturalness, and controllability, preserving speaker identity.
【50】 Chat with the Environment: Interactive Multimodal Perception using Large Language Models标题：与环境聊天：使用大型语言模型的交互式多通道感知链接：http://arxiv.org/pdf/2303.08268v1
作者：Xufeng Zhao,Mengdi Li,Cornelius Weber,Muhammad Burhan Hafez,Stefan Wermter机构：Universit¨at Hamburg备注：See website at this https URL摘要：在复杂的世界中，编程机器人行为面临着多个层面的挑战，从灵巧的低级技能到高级规划和推理。最近提出的预训练大语言模型（LLM）在zero-shot机器人规划中表现出了卓越的推理能力。然而，要使LLM具备多模式感觉输入和连续动作输出的能力，同时使机器人能够与环境交互，并随着其策略的展开获取新信息，仍然是一个挑战。我们开发了一个具有部分可观察状态的机器人交互场景，它需要机器人决定一系列认知行为，以便在能够正确执行任务之前从多个模态中采样感觉信息。因此，提出了一种以LLM为主干的交互式感知框架，利用LLM的能力指导认知行为，对产生的多模态感觉（视觉、声音、触觉、本体感觉）进行推理，并基于交互式获取的信息规划整个任务执行。我们的研究表明，LLM能够提供高水平的规划和推理技能，并在多模态环境中控制交互式机器人行为，而具有环境状态上下文的多模态模块有助于LLM扎根并扩展其处理能力。摘要：Programming robot behaviour in a complex world faces challenges on multiple levels, from dextrous low-level skills to high-level planning and reasoning. Recent pre-trained Large Language Models (LLMs) have shown remarkable reasoning ability in zero-shot robotic planning. However, it remains challenging to ground LLMs in multimodal sensory input and continuous action output, while enabling a robot to interact with its environment and acquire novel information as its policies unfold. We develop a robot interaction scenario with a partially observable state, which necessitates a robot to decide on a range of epistemic actions in order to sample sensory information among multiple modalities, before being able to execute the task correctly. An interactive perception framework is therefore proposed with an LLM as its backbone, whose ability is exploited to instruct epistemic actions and to reason over the resulting multimodal sensations (vision, sound, haptics, proprioception), as well as to plan an entire task execution based on the interactively acquired information. Our study demonstrates that LLMs can provide high-level planning and reasoning skills and control interactive robot behaviour in a multimodal environment, while multimodal modules with the context of the environmental state help ground the LLMs and extend their processing ability.
【51】Facilitating deep acoustic phenotyping: A basic coding scheme of infant vocalisations preluding computational analysis, machine learning and clinical reasoning标题：促进深层声学表型：包括计算分析、机器学习和临床推理的婴幼儿发声的基本编码方案链接：http://arxiv.org/pdf/2303.08239v1
作者：Tomas Kulvicius,Sigrun Lang,Claudius AA Widmann,Nina Hansmann,Daniel Holzinger,Luise Poustka,Dajie Zhang,Peter B Marschik机构：Klinik für Kinder- und Jugendpsychiatrie und Psychotherapie, Universitätsmedizin Göttingen, Abteilung für Computational Neuroscience, Drittes Physikalisches Institut, Forschungsinstitut für Entwicklungsmedizin, Johannes Kepler Universität, Linz, Österreich备注：This paper is in German摘要：理论背景：早期的语言发展还没有被完全理解，特别是在它的形成阶段。研究问题：能否定义一种可靠、易于使用的编码方案，用于早期婴儿发声的分类，并可用作进一步分析语言发展的基础？方法：在一项对45名神经典型婴儿的纵向研究中，我们分析了出生后前4个月的发声情况。音频片段被分配到5个类别：（1）浊音化和（2）清音化;（3）定义信号;（4）非目标性;（5）不可转让。结果：两名经验不同的女性编码员在未经强化培训的情况下获得了较高的一致性。讨论和结论：该可靠的方案可用于研究和临床设置中，用于有效地编码婴儿发声，作为详细的人工和机器分析的基础。摘要：Theoretical background: early verbal development is not yet fully understood, especially in its formative phase. Research question: can a reliable, easy-to-use coding scheme for the classification of early infant vocalizations be defined that is applicable as a basis for further analysis of language development? Methods: in a longitudinal study of 45 neurotypical infants, we analyzed vocalizations of the first 4 months of life. Audio segments were assigned to 5 classes: (1) Voiced and (2) Voiceless vocalizations; (3) Defined signal; (4) Non-target; (5) Nonassignable. Results: Two female coders with different experience achieved high agreement without intensive training. Discussion and Conclusion: The reliable scheme can be used in research and clinical settings for efficient coding of infant vocalizations, as a basis for detailed manual and machine analyses.
机器翻译，仅供参考
每日语音速递