语音/音频处理学术速递[9.15]

今日论文合集：cs.SD语音39篇，eess.AS音频处理38篇。

cs.SD语音

【1】 CiwaGAN: Articulatory information exchange标题：CiwaGan：发音信息交流链接：https://arxiv.org/abs/2309.07861

作者：Gašper Beguš,Thomas Lu,Alan Zhou,Peter Wu,Gopala K. Anumanchipalli机构：University of California, Berkeley,Johns Hopkins University摘要：人类通过控制发音器官将信息编码成声音，并使用听觉器官从声音中解码信息。本文介绍了CiwaGAN，一个模型的人类口语习得相结合的无监督发音建模与无监督模型的信息交换，通过听觉模态。虽然以前的研究包括无监督发音建模和信息交换分别，我们的模型是第一个结合这两个组件。本文还提出了一种改进的发音模型，具有更多的可解释的内部表示。提出的CiwaGAN模型是使用深度学习的人类口语习得的最现实的近似。因此，它对于人类言语行为的认知上合理的模拟是有用的。摘要：Humans encode information into sounds by controlling articulators and decode information from sounds using the auditory apparatus. This paper introduces CiwaGAN, a model of human spoken language acquisition that combines unsupervised articulatory modeling with an unsupervised model of information exchange through the auditory modality. While prior research includes unsupervised articulatory modeling and information exchange separately, our model is the first to combine the two components. The paper also proposes an improved articulatory model with more interpretable internal representations. The proposed CiwaGAN model is the most realistic approximation of human spoken language acquisition using deep learning. As such, it is useful for cognitively plausible simulations of the human speech act.

【2】 Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Tasks标题：Echotune：在ASR任务中利用语音的可变长度特性的模块化抽取器链接：https://arxiv.org/abs/2309.07765

作者：Sizhou Chen,Songyang Gao,Sen Fang机构：Chengdu University of Information Technology, China, Fudan University, China, Victoria University, Australia摘要：Transformer架构已被证明对自动语音识别（ASR）任务非常有效，成为该领域大量研究的基础组件。从历史上看，许多方法都依赖于固定长度的注意力窗口，这对于持续时间和复杂度不同的语音样本来说是有问题的，导致数据过度平滑和忽视必要的长期连接。针对这一限制，我们介绍回声MSA，一个灵活的模块，配备了一个可变长度的注意力机制，可容纳一系列的语音样本的复杂性和持续时间。该模块提供了跨各种粒度提取语音特征的灵活性，从帧和音素到单词和话语。所提出的设计捕获的语音的可变长度的功能，并解决了固定长度的注意力的局限性。我们的评估利用了一个并行的注意力架构，辅以动态门控机制，融合了传统的关注与回声-MSA模块输出。我们研究的经验证据表明，将Echo-MSA集成到主模型的训练机制中显着提高了单词错误率（WER）的性能，同时保持了原始模型的内在稳定性。摘要：The Transformer architecture has proven to be highly effective for Automatic Speech Recognition (ASR) tasks, becoming a foundational component for a plethora of research in the domain. Historically, many approaches have leaned on fixed-length attention windows, which becomes problematic for varied speech samples in duration and complexity, leading to data over-smoothing and neglect of essential long-term connectivity. Addressing this limitation, we introduce Echo-MSA, a nimble module equipped with a variable-length attention mechanism that accommodates a range of speech sample complexities and durations. This module offers the flexibility to extract speech features across various granularities, spanning from frames and phonemes to words and discourse. The proposed design captures the variable length feature of speech and addresses the limitations of fixed-length attention. Our evaluation leverages a parallel attention architecture complemented by a dynamic gating mechanism that amalgamates traditional attention with the Echo-MSA module output. Empirical evidence from our study reveals that integrating Echo-MSA into the primary model's training regime significantly enhances the word error rate (WER) performance, all while preserving the intrinsic stability of the original model.

【3】 The complementary roles of non-verbal cues for Robust Pronunciation Assessment标题：非语言线索在强健发音评估中的互补作用链接：https://arxiv.org/abs/2309.07739

作者：Yassine El Kheir,Shammur Absar Chowdhury,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar备注：5 pages, submitted to ICASSP 2024摘要：语音评估系统的研究侧重于利用非母语（L2）语音的语音和语音方面，往往忽略了隐藏在非语言线索中的丰富信息层。在这项研究中，我们提出了一个新的语音评估框架，IntraVerbalPA。%的框架创新地结合了细粒度的框架和抽象的话语水平的非语言线索，除了传统的语音和音素表示。此外，我们引入了“良好的音素持续时间”度量有效地模型持续时间分布的框架内。我们的研究结果验证了建议的IntraVerbalPA框架及其单独的组件的有效性，产生的性能，要么匹配或优于现有的研究工作。摘要：Research on pronunciation assessment systems focuses on utilizing phonetic and phonological aspects of non-native (L2) speech, often neglecting the rich layer of information hidden within the non-verbal cues. In this study, we proposed a novel pronunciation assessment framework, IntraVerbalPA. % The framework innovatively incorporates both fine-grained frame- and abstract utterance-level non-verbal cues, alongside the conventional speech and phoneme representations. Additionally, we introduce ''Goodness of phonemic-duration'' metric to effectively model duration distribution within the framework. Our results validate the effectiveness of the proposed IntraVerbalPA framework and its individual components, yielding performance that either matches or outperforms existing research works.

【4】 Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features标题：基于词级音段和副语言特征的语音分类模型解释链接：https://arxiv.org/abs/2309.07733

作者：Eliana Pastor,Alkis Koudounas,Giuseppe Attanasio,Dirk Hovy,Elena Baralis机构：♣ Politecnico di Torino, Turin, Italy, ♡ Bocconi University, Milan, Italy备注：8 pages摘要：可解释人工智能（XAI）的最新进展为视觉、语言和表格数据模型的运行方式提供了新的见解。然而，存在很少的方法来理解语音模型。现有的工作集中在一些口语理解（SLU）的任务，解释是很难解释的大多数用户。我们介绍了一种新的方法来解释语音分类模型。我们通过两个信息水平上的输入扰动生成易于解释的解释。1)单词级解释揭示了每个单词相关的音频片段如何影响结果。2)副语言特征（例如，韵律和背景噪音）回答反事实：``如果我们以这种方式编辑音频信号，模型预测会是什么？我们通过在英语和意大利语的两个语音分类任务上解释两个最先进的SLU模型来验证我们的方法。我们的研究结果表明，解释是忠实的模型的内部工作原理和合理的人类。我们的方法和研究结果铺平了道路，为今后的研究口译语音模型。摘要：Recent advances in eXplainable AI (XAI) have provided new insights into how models for vision, language, and tabular data operate. However, few approaches exist for understanding speech models. Existing work focuses on a few spoken language understanding (SLU) tasks, and explanations are difficult to interpret for most users. We introduce a new approach to explain speech classification models. We generate easy-to-interpret explanations via input perturbation on two information levels. 1) Word-level explanations reveal how each word-related audio segment impacts the outcome. 2) Paralinguistic features (e.g., prosody and background noise) answer the counterfactual: ``What would the model prediction be if we edited the audio signal in this way?'' We validate our approach by explaining two state-of-the-art SLU models on two speech classification tasks in English and Italian. Our findings demonstrate that the explanations are faithful to the model's inner workings and plausible to humans. Our method and findings pave the way for future research on interpreting speech models.

【5】 L1-aware Multilingual Mispronunciation Detection Framework标题：一种L1感知的多语种发音错误检测框架链接：https://arxiv.org/abs/2309.07719

作者：Yassine El Kheir,Shammur Absar Chwodhury,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar备注：5 papers, submitted to ICASSP 2024摘要：母语和非母语之间的语音差异是造成发音错误的主要因素。本文介绍了一种新的多语言MDD架构，L1-MultiMDD，丰富了L1感知语音表示。在输入信号及其对应的参考音素序列上训练端到端语音编码器。首先，部署注意机制以将输入音频与参考音素序列对准。然后，从辅助模型中提取L1-L2语音嵌入，在识别L1和L2语言的多任务设置中进行预训练，并注入主网络。最后，L1-MultiMDD然后针对统一的多语言音素识别任务使用目标语言的连接主义时间分类（CTC）损失进行优化：英语、阿拉伯语和普通话。我们的实验证明了所提出的L1-MultiMDD框架的有效性，在两个见- L2-ARTIC，LATIC，和AraVoiceL 2 v2;和看不见的数据集-- EpaDB和Speechocean 762。PER和错误拒绝率（FRR）在所有目标语言中的一致增益证实了我们的方法的鲁棒性、有效性和可推广性。摘要：The phonological discrepancies between a speaker's native (L1) and the non-native language (L2) serves as a major factor for mispronunciation. This paper introduces a novel multilingual MDD architecture, L1-MultiMDD, enriched with L1-aware speech representation. An end-to-end speech encoder is trained on the input signal and its corresponding reference phoneme sequence. First, an attention mechanism is deployed to align the input audio with the reference phoneme sequence. Afterwards, the L1-L2-speech embedding are extracted from an auxiliary model, pretrained in a multi-task setup identifying L1 and L2 language, and are infused with the primary network. Finally, the L1-MultiMDD is then optimized for a unified multilingual phoneme recognition task using connectionist temporal classification (CTC) loss for the target languages: English, Arabic, and Mandarin. Our experiments demonstrate the effectiveness of the proposed L1-MultiMDD framework on both seen -- L2-ARTIC, LATIC, and AraVoiceL2v2; and unseen -- EpaDB and Speechocean762 datasets. The consistent gains in PER, and false rejection rate (FRR) across all target languages confirm our approach's robustness, efficacy, and generalizability.

【6】 CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders标题：CoLLD：用于压缩多语种预训练语音编码器的对比性分层蒸馏链接：https://arxiv.org/abs/2309.07707

作者：Heng-Jui Chang,Ning Dong,Ruslan Mavlyutov,Sravya Popuri,Yu-An Chung机构：MIT CSAIL ,Meta AI备注：Submitted to ICASSP 2024摘要：大规模自监督预训练语音编码器在语音识别和翻译任务中优于传统方法。由于开发这些大型模型的成本较高，为新任务构建新的编码器并将其部署到设备上应用是不可行的。先前的研究提出了模型压缩方法来解决这个问题，但这些工作集中在较小的模型和不太现实的任务。因此，我们提出了对比层到层蒸馏（CoLLD），一种新的知识蒸馏方法压缩预训练的语音编码器利用掩蔽预测和对比学习训练学生模型复制一个大的教师模型的行为。CoLLD优于现有方法，并缩小了小型和大型模型之间的差距，多语言语音到文本翻译和识别基准。摘要：Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new tasks and deploying them to on-device applications are infeasible. Prior studies propose model compression methods to address this issue, but those works focus on smaller models and less realistic tasks. Thus, we propose Contrastive Layer-to-layer Distillation (CoLLD), a novel knowledge distillation method to compress pre-trained speech encoders by leveraging masked prediction and contrastive learning to train student models to copy the behavior of a large teacher model. CoLLD outperforms prior methods and closes the gap between small and large models on multilingual speech-to-text translation and recognition benchmarks.

【7】 DDSP-based Neural Waveform Synthesis of Polyphonic Guitar Performance from String-wise MIDI Input标题：基于DDSP的弦式MIDI输入复调吉他演奏的神经波形合成链接：https://arxiv.org/abs/2309.07658

作者：Nicolas Jonason,Xin Wang,Erica Cooper,Lauri Juvela,Bob L. T. Sturm,Junichi Yamagishi机构：KTH Royal Institute of Technology, Sweden ,National Institute of Informatics, Japan, Aalto University, Finland摘要：我们探讨了使用神经合成的原声吉他弦明智的MIDI输入。我们提出了四种不同的系统，并将它们与自然音频和基于样本的基线的客观指标和主观评价进行比较。我们迭代开发这四个系统的架构和中间任务，如预测音高和响度控制功能的各种考虑。我们发现，制定控制功能预测任务作为一个分类任务，而不是回归任务产生更好的结果。此外，我们发现，我们最简单的建议系统，直接预测合成参数的MIDI输入执行最好的四个建议系统。音频示例可在https://erl-j.github.io/neural-guitar-web-supplement上获得。摘要：We explore the use of neural synthesis for acoustic guitar from string-wise MIDI input. We propose four different systems and compare them with both objective metrics and subjective evaluation against natural audio and a sample-based baseline. We iteratively develop these four systems by making various considerations on the architecture and intermediate tasks, such as predicting pitch and loudness control features. We find that formulating the control feature prediction task as a classification task rather than a regression task yields better results. Furthermore, we find that our simplest proposed system, which directly predicts synthesis parameters from MIDI input performs the best out of the four proposed systems. Audio examples are available at https://erl-j.github.io/neural-guitar-web-supplement.

【8】 Multilingual Audio Captioning using machine translated data标题：使用机器翻译数据的多语言音频字幕链接：https://arxiv.org/abs/2309.07615

作者：Matéo Cousin,Étienne Labbé,Thomas Pellegrini机构：IRIT, Universit´e Paul Sabatier, CNRS, Toulouse, France摘要：自动音频字幕（AAC）系统试图生成自然语言句子，即字幕，其根据声音事件描述音频记录的内容。现有的数据集提供音频-字幕对，字幕仅用英文书写。在这项工作中，我们探索多语言AAC，使用机器翻译的字幕。我们自动翻译了两个著名的AAC数据集，AudioCaps和Clotho，从英语到法语，德语和西班牙语。我们在AudioCaps和Clotho上用四种语言训练和评估了单语系统。在所有情况下，模型都实现了类似的性能，AudioCaps上的CIDEr约为75%，Clotho上为43%。在法语中，我们获得了AudioCaps eval子集的手动字幕。在AudioCaps的机器翻译版本上训练的法语系统在手动评估子集上取得了显着更好的结果，与我们自动将输出翻译为法语的英语系统相比。这主张支持以目标语言构建系统，而不是简单地将英语系统中的英语字幕翻译成目标语言。最后，我们建立了一个多语言模型，它在每种语言中取得的结果与每个单语系统相当，同时使用的参数比使用单语系统的集合少得多。摘要：Automated Audio Captioning (AAC) systems attempt to generate a natural language sentence, a caption, that describes the content of an audio recording, in terms of sound events. Existing datasets provide audio-caption pairs, with captions written in English only. In this work, we explore multilingual AAC, using machine translated captions. We translated automatically two prominent AAC datasets, AudioCaps and Clotho, from English to French, German and Spanish. We trained and evaluated monolingual systems in the four languages, on AudioCaps and Clotho. In all cases, the models achieved similar performance, about 75% CIDEr on AudioCaps and 43% on Clotho. In French, we acquired manual captions of the AudioCaps eval subset. The French system, trained on the machine translated version of AudioCaps, achieved significantly better results on the manual eval subset, compared to the English system for which we automatically translated the outputs to French. This advocates in favor of building systems in a target language instead of simply translating to a target language the English captions from the English system. Finally, we built a multilingual model, which achieved results in each language comparable to each monolingual system, while using much less parameters than using a collection of monolingual systems.

【9】 AAS-VC: On the Generalization Ability of Automatic Alignment Search based Non-autoregressive Sequence-to-sequence Voice Conversion标题：AAS-VC：基于自动比对搜索的非自回归序列到序列语音转换的泛化能力链接：https://arxiv.org/abs/2309.07598

作者：Wen-Chin Huang,Kazuhiro Kobayashi,Tomoki Toda机构：Nagoya University, Japan, TARVO, Inc., Japan备注：Submitted to ICASSP 2024. Demo: this https URL Code: this https URL摘要：用于语音转换（VC）的非自回归（非AR）序列到序列（seq 2seq）模型在其有效地对时间结构进行建模的能力方面是有吸引力的，同时由于非AR建模而享受增强的可理解性和快速推理。然而，当前非AR seq 2seq VC模型对从外部AR模型提取的地面实况持续时间的依赖性极大地限制了其对较小训练数据集的泛化能力。在本文中，我们首先通过改变训练数据的大小来证明上述问题。然后，我们提出了AAS-VC，一个基于自动对齐搜索（AAS）的非AR seq 2seq VC模型，它消除了对外部持续时间的依赖，并作为一个适当的归纳偏差，为小数据集提供所需的泛化能力。实验结果表明，AAS-VC可以更好地推广到只有5分钟的训练数据集。我们还进行了消融研究，以证明几种模型设计的选择。音频示例和实现可在线获得。摘要：Non-autoregressive (non-AR) sequence-to-seqeunce (seq2seq) models for voice conversion (VC) is attractive in its ability to effectively model the temporal structure while enjoying boosted intelligibility and fast inference thanks to non-AR modeling. However, the dependency of current non-AR seq2seq VC models on ground truth durations extracted from an external AR model greatly limits its generalization ability to smaller training datasets. In this paper, we first demonstrate the above-mentioned problem by varying the training data size. Then, we present AAS-VC, a non-AR seq2seq VC model based on automatic alignment search (AAS), which removes the dependency on external durations and serves as a proper inductive bias to provide the required generalization ability for small datasets. Experimental results show that AAS-VC can generalize better to a training dataset of only 5 minutes. We also conducted ablation studies to justify several model design choices. The audio samples and implementation are available online.

【10】 Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer标题：基于离散单元风格转换的语音到语音翻译链接：https://arxiv.org/abs/2309.07566

作者：Yongqi Wang,Jionghao Bai,Rongjie Huang,Ruiqi Li,Zhiqing Hong,Zhou Zhao机构：Zhejiang University, China备注：5 pages, 1 figure. submitted to ICASSP 2024摘要：具有离散自监督表示的直接语音到语音翻译（S2ST）已经实现了显著的准确性，但是在翻译期间无法保留源语音的扬声器音色。同时，高质量的说话人平行数据的缺乏对源语音和目标语音之间的学习风格转移提出了挑战。我们提出了一个S2ST框架的声学语言模型的基础上从一个自监督模型和神经编解码器的风格转移的离散单元。声学语言模型利用自监督的上下文学习，在不依赖于任何说话者并行数据的情况下获得风格转移的能力，从而克服了数据稀缺的问题。通过使用大量的训练数据，我们的模型在以前看不见的源语言上实现了zero-shot跨语言风格迁移。实验表明，我们的模型生成具有高保真度和风格相似度的翻译语音。音频样本可在http://stylelm.github.io/获得。摘要：Direct speech-to-speech translation (S2ST) with discrete self-supervised representations has achieved remarkable accuracy, but is unable to preserve the speaker timbre of the source speech during translation. Meanwhile, the scarcity of high-quality speaker-parallel data poses a challenge for learning style transfer between source and target speech. We propose an S2ST framework with an acoustic language model based on discrete units from a self-supervised model and a neural codec for style transfer. The acoustic language model leverages self-supervised in-context learning, acquiring the ability for style transfer without relying on any speaker-parallel data, thereby overcoming the issue of data scarcity. By using extensive training data, our model achieves zero-shot cross-lingual style transfer on previously unseen source languages. Experiments show that our model generates translated speeches with high fidelity and style similarity. Audio samples are available at http://stylelm.github.io/ .

【11】 SingFake: Singing Voice Deepfake Detection标题：SingFake：歌唱声音的深伪检测链接：https://arxiv.org/abs/2309.07525

作者：Yongyi Zang,You Zhang,Mojtaba Heydari,Zhiyao Duan机构：Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA备注：Submitted to ICASSP 2024摘要：歌唱声音合成的兴起给艺术家和行业利益相关者带来了未经授权的声音使用的严峻挑战。与合成语音不同，合成歌声通常在包含强烈背景音乐的歌曲中释放，这可能隐藏合成伪像。此外，歌唱声音呈现出不同于言语话语的声学和语言特征。这些独特的特性使得歌唱语音deepfake检测成为与合成语音检测相关但显著不同的问题。在这项工作中，我们提出了唱歌声音deepfake检测任务。我们首先介绍SingFake，这是第一个精心策划的野外数据集，包括来自40位歌手的28.93小时的bonafide和29.40小时的deepfake歌曲剪辑。我们提供了训练/val/test拆分，其中测试集包括各种场景。然后，我们使用SingFake来评估四个国家的最先进的语音对抗系统的语音话语训练。我们发现，这些系统落后于他们的语音测试数据的性能显着。当在SingFake上进行训练时，无论是使用单独的音轨还是歌曲混合，这些系统都显示出了实质性的改进。然而，我们的评估也确定了与看不见的歌手，通信编解码器，语言和音乐背景相关的挑战，呼吁专门研究唱歌声音deepfake检测。SingFake数据集和相关资源可在线获取。摘要：The rise of singing voice synthesis presents critical challenges to artists and industry stakeholders over unauthorized voice usage. Unlike synthesized speech, synthesized singing voices are typically released in songs containing strong background music that may hide synthesis artifacts. Additionally, singing voices present different acoustic and linguistic characteristics from speech utterances. These unique properties make singing voice deepfake detection a relevant but significantly different problem from synthetic speech detection. In this work, we propose the singing voice deepfake detection task. We first present SingFake, the first curated in-the-wild dataset consisting of 28.93 hours of bonafide and 29.40 hours of deepfake song clips in five languages from 40 singers. We provide a train/val/test split where the test sets include various scenarios. We then use SingFake to evaluate four state-of-the-art speech countermeasure systems trained on speech utterances. We find these systems lag significantly behind their performance on speech test data. When trained on SingFake, either using separated vocal tracks or song mixtures, these systems show substantial improvement. However, our evaluations also identify challenges associated with unseen singers, communication codecs, languages, and musical contexts, calling for dedicated research into singing voice deepfake detection. The SingFake dataset and related resources are available online.

【12】 Outlier-aware Inlier Modeling and Multi-scale Scoring for Anomalous Sound Detection via Multitask Learning标题：基于多任务学习的异常点感知内点建模和多尺度评分链接：https://arxiv.org/abs/2309.07500

作者：Yucong Zhang,Hongbin Suo,Yulong Wan,Ming Li机构：Data Science Research Center, Duke Kunshan University, Kunshan, China, Data & AI Engineering System, OPPO, Beijing, China备注：accepted at INTERSPEECH 2023摘要：本文提出了一种异常声音检测的方法，将离群点暴露和内点建模在一个统一的框架内的多任务学习。虽然基于离群点曝光的方法可以有效地提取特征，但它不是鲁棒的。内点建模在生成鲁棒特征方面很好，但这些特征不是很有效。最近，提出了一系列的方法来结合这两种方法，但它仍然需要一个单独的训练步骤正常的数据建模。为了克服这些局限性，我们使用多任务学习训练一个基于一致性的编码器的异常感知内点建模。此外，我们的方法提供了多尺度的分数检测异常。MIMII和DCASE 2020任务2数据集上的实验结果表明，我们的方法优于最先进的单模型系统，并取得了与顶级多系统集成相当的结果。摘要：This paper proposes an approach for anomalous sound detection that incorporates outlier exposure and inlier modeling within a unified framework by multitask learning. While outlier exposure-based methods can extract features efficiently, it is not robust. Inlier modeling is good at generating robust features, but the features are not very effective. Recently, serial approaches are proposed to combine these two methods, but it still requires a separate training step for normal data modeling. To overcome these limitations, we use multitask learning to train a conformer-based encoder for outlier-aware inlier modeling. Moreover, our approach provides multi-scale scores for detecting anomalies. Experimental results on the MIMII and DCASE 2020 task 2 datasets show that our approach outperforms state-of-the-art single-model systems and achieves comparable results with top-ranked multi-system ensembles.

【13】 Direct Text to Speech Translation System using Acoustic Units标题：使用声学单元的直接文本到语音翻译系统链接：https://arxiv.org/abs/2309.07478

作者：Victoria Mingote,Pablo Gimeno,Luis Vicente,Sameer Khurana,Antoine Laurent,Jarod Duret机构：Duret is with LIA - Avignon University备注：5 pages, 4 figures摘要：本文提出了一个直接的文本到语音翻译系统使用离散声学单位。该框架采用不同源语言的文本作为输入来生成目标语言的语音，而不需要该语言的文本转录。在以前的作品中的直接语音到语音翻译系统的声学单元的成功的动机，我们使用相同的管道提取的声学单元，使用语音编码器结合聚类算法。一旦获得单元，就训练编码器-解码器架构来预测它们。然后，声码器从单元生成语音。我们的直接文本到语音翻译的方法进行了测试，在新的CVSS语料库与两个不同的文本mBART模型作为初始化。所提出的系统报告的竞争性能的大多数语言对评估。此外，结果显示，当使用更多语言预训练的模型初始化我们提出的架构时，有了显着的改善。摘要：This paper proposes a direct text to speech translation system using discrete acoustic units. This framework employs text in different source languages as input to generate speech in the target language without the need for text transcriptions in this language. Motivated by the success of acoustic units in previous works for direct speech to speech translation systems, we use the same pipeline to extract the acoustic units using a speech encoder combined with a clustering algorithm. Once units are obtained, an encoder-decoder architecture is trained to predict them. Then a vocoder generates speech from units. Our approach for direct text to speech translation was tested on the new CVSS corpus with two different text mBART models employed as initialisation. The systems presented report competitive performance for most of the language pairs evaluated. Besides, results show a remarkable improvement when initialising our proposed architecture with a model pre-trained with more languages.

【14】 Analysis of Speech Separation Performance Degradation on Emotional Speech Mixtures标题：情感语音混合的语音分离性能劣化分析链接：https://arxiv.org/abs/2309.07458

作者：Jia Qi Yip,Dianwen Ng,Bin Ma,Chng Eng Siong机构：∗ Speech Lab of DAMO Academy, Alibaba Group, † Nanyang Technological University, Singapore备注：Accepted by APSIPA ASC 2023摘要：尽管最近在语音分离方面取得了长足的进步，但大多数模型都是在具有中性情感的数据集上训练的。情感语音已经被称为降低模型的性能在各种语音任务，这降低了这些模型的有效性时，部署在现实世界的场景。在本文中，我们进行分析，以区分性能下降所产生的情绪在讲话中的影响域外的推理。这是使用精心设计的测试数据集Emo 2 Mix测量的，该数据集由所有情绪组合的平衡数据组成。我们表明，即使是具有强大的域外性能，如Sepformer模型仍然可以遭受显着的退化高达5.1 dB SI-SDRi的混合物与强烈的情绪。这证明了在现实世界的语音分离应用中考虑情绪的重要性。摘要：Despite recent strides made in Speech Separation, most models are trained on datasets with neutral emotions. Emotional speech has been known to degrade performance of models in a variety of speech tasks, which reduces the effectiveness of these models when deployed in real-world scenarios. In this paper we perform analysis to differentiate the performance degradation arising from the emotions in speech from the impact of out-of-domain inference. This is measured using a carefully designed test dataset, Emo2Mix, consisting of balanced data across all emotional combinations. We show that even models with strong out-of-domain performance such as Sepformer can still suffer significant degradation of up to 5.1 dB SI-SDRi on mixtures with strong emotions. This demonstrates the importance of accounting for emotions in real-world speech separation applications.

【15】 SpatialCodec: Neural Spatial Speech Coding标题：SpatialCodec：神经空间语音编码链接：https://arxiv.org/abs/2309.07432

作者：Zhongweiyang Xu,Yong Xu,Vinay Kothapally,Heming Wang,Muqiao Yang,Dong Yu机构：†Tencent AI Lab, ⋆ University of Illinois at Urbana-Champaign, ♯ The Ohio State University, ‡ Carnegie Mellon University备注：Paper in Submission摘要：在这项工作中，我们解决了使用深度学习技术对麦克风阵列捕获的语音进行编码的挑战，目的是保留和准确重建嵌入在多通道录音中的关键空间线索。我们提出了一个神经空间音频编码框架，实现了高压缩比，利用单通道神经子带编解码器和SpatialCodec。我们的方法包括两个阶段：（i）神经子带编解码器被设计为以低比特率对参考信道进行编码，以及（ii）SpatialCodec捕获相对空间信息以用于在解码器端处的准确多信道重构。此外，我们还提出了新的评估指标来评估空间线索保存：（i）空间相似性，其计算空间直观波束空间上的余弦相似性，以及（ii）波束成形音频质量。与高比特率基线和黑盒神经架构相比，我们的系统显示出优越的空间性能。演示可在https://xzwy.github.io/SpatialCodecDemo上查阅。代码和型号可在https://github.com/XZWY/SpatialCodec上获得。摘要：In this work, we address the challenge of encoding speech captured by a microphone array using deep learning techniques with the aim of preserving and accurately reconstructing crucial spatial cues embedded in multi-channel recordings. We propose a neural spatial audio coding framework that achieves a high compression ratio, leveraging single-channel neural sub-band codec and SpatialCodec. Our approach encompasses two phases: (i) a neural sub-band codec is designed to encode the reference channel with low bit rates, and (ii), a SpatialCodec captures relative spatial information for accurate multi-channel reconstruction at the decoder end. In addition, we also propose novel evaluation metrics to assess the spatial cue preservation: (i) spatial similarity, which calculates cosine similarity on a spatially intuitive beamspace, and (ii), beamformed audio quality. Our system shows superior spatial performance compared with high bitrate baselines and black-box neural architecture. Demos are available at https://xzwy.github.io/SpatialCodecDemo. Codes and models are available at https://github.com/XZWY/SpatialCodec.

【16】 Mandarin Lombard Flavor Classification标题：柑橘伦巴第香精分类链接：https://arxiv.org/abs/2309.07419

作者：Qingmu Liu,Yuhong Yang,Baifeng Li,Hongyang Chen,Weiping Tu,Song Lin机构： National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China., Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan, China.摘要：隆巴德效应是指个体对环境噪声水平变化的无意识调节，旨在提高语音清晰度。不同分贝水平和类型的背景噪声对隆巴德效应的影响仍不清楚。隆巴德语音的特点，个人调整他们的语音，以提高可懂度动态的基础上的自我反馈的语音，我们提出了一种隆巴德效应的风味分类方法。首先采集不同噪声条件下的普通话伦巴底语语音，然后模拟自反馈语音，最后对正确率进行统计检验。我们发现，SSN和串音噪声类型导致四个不同类别的普通话伦巴德语音在30至80分贝的范围内，具有不同的过渡点。摘要：The Lombard effect refers to individuals' unconscious modulation of vocal effort in response to variations in the ambient noise levels, intending to enhance speech intelligibility. The impact of different decibel levels and types of background noise on Lombard effects remains unclear. Building upon the characteristic of Lombard speech that individuals adjust their speech to improve intelligibility dynamically based on the self-feedback speech, we propose a flavor classification approach for the Lombard effect. We first collected Mandarin Lombard speech under different noise conditions, then simulated self-feedback speech, and ultimately conducted the statistical test on the word correct rate. We found that both SSN and babble noise types result in four distinct categories of Mandarin Lombard speech in the range of 30 to 80 dBA with different transition points.

【17】 M3-AUDIODEC: Multi-channel multi-speaker multi-spatial audio codec标题：M3-AUDIODEC：多声道多扬声器多空间音频编解码器链接：https://arxiv.org/abs/2309.07416

作者：Anton Ratnarajah,Shi-Xiong Zhang,Yi Luo,Dong Yu机构： University of Maryland, College Park, MD, USA , Tencent AI Lab, Bellevue, WA, USA备注：More results and source code are available at this https URL摘要：我们介绍M3-AUDIODEC，一种创新的神经空间音频编解码器，设计用于在单个和多个扬声器场景中有效压缩多通道（双耳）语音，同时保留每个扬声器的空间位置信息。该模型具有多功能性，允许根据预定的一组多声道、多扬声器和多空间重叠语音条件进行配置和训练。主要贡献如下：1）以前的神经编解码器从单声道音频扩展到多声道音频。2)我们提出的模型的能力，压缩和解码重叠的语音。3)一种突破性的架构，可分别压缩语音内容和空间线索，确保解码后保留每个说话者的空间上下文。4)M3-AUDIODEC在压缩双声道语音的带宽方面的熟练程度与单个双耳声道压缩相比降低了48%。令人印象深刻的是，在12.6 kbps的操作下，它的性能分别比Opus（24 kbps）和AUDIODEC（24 kbps）分别高出37%和52%。在我们的评估中，我们采用语音增强和室内声学指标来确定来自M3-AUDIODEC的干净语音和空间线索估计的准确性。音频演示和源代码可在线获得。摘要：We introduce M3-AUDIODEC, an innovative neural spatial audio codec designed for efficient compression of multi-channel (binaural) speech in both single and multi-speaker scenarios, while retaining the spatial location information of each speaker. This model boasts versatility, allowing configuration and training tailored to a predetermined set of multi-channel, multi-speaker, and multi-spatial overlapping speech conditions. Key contributions are as follows: 1) Previous neural codecs are extended from single to multi-channel audios. 2) The ability of our proposed model to compress and decode for overlapping speech. 3) A groundbreaking architecture that compresses speech content and spatial cues separately, ensuring the preservation of each speaker's spatial context after decoding. 4) M3-AUDIODEC's proficiency in reducing the bandwidth for compressing two-channel speech by 48% when compared to individual binaural channel compression. Impressively, at a 12.6 kbps operation, it outperforms Opus at 24 kbps and AUDIODEC at 24 kbps by 37% and 52%, respectively. In our assessment, we employed speech enhancement and room acoustic metrics to ascertain the accuracy of clean speech and spatial cue estimates from M3-AUDIODEC. Audio demonstrations and source code are available online.

【18】 CPPF: A contextual and post-processing-free model for automatic speech recognition标题：CPPF：一种无需上下文和后处理的自动语音识别模型链接：https://arxiv.org/abs/2309.07413

作者：Lei Zhang,Zhengkun Tian,Xiang Chen,Jiaming Sun,Hongyu Xiang,Ke Ding,Guanglu Wan机构：School of Computer Science and Technology, Soochow University, China, Meituan备注：Submitted to ICASSP2024摘要：近年来，ASR系统已经变得越来越普遍。然而，它们的文本输出通常需要后处理任务，然后才能实际使用。为了解决这个问题，我们从LLM和Whisper的多方面功能中汲取灵感，并专注于将与语音识别相关的多个ASR文本处理任务集成到ASR模型中。这种集成不仅缩短了多阶段流水线，而且还防止了级联错误的传播，从而直接生成后处理文本。在本研究中，我们专注于与ASR相关的加工任务，包括上下文ASR和多个ASR后处理任务。为了实现这一目标，我们引入了CPPF模型，它提供了一个通用的和高效的替代ASR处理。CPPF无缝地集成了这些任务，而没有任何显着的识别性能损失。摘要：ASR systems have become increasingly widespread in recent years. However, their textual outputs often require post-processing tasks before they can be practically utilized. To address this issue, we draw inspiration from the multifaceted capabilities of LLMs and Whisper, and focus on integrating multiple ASR text processing tasks related to speech recognition into the ASR model. This integration not only shortens the multi-stage pipeline, but also prevents the propagation of cascading errors, resulting in direct generation of post-processed text. In this study, we focus on ASR-related processing tasks, including Contextual ASR and multiple ASR post processing tasks. To achieve this objective, we introduce the CPPF model, which offers a versatile and highly effective alternative to ASR processing. CPPF seamlessly integrates these tasks without any significant loss in recognition performance.

【19】 FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec标题：FunCodec：一个基础的、可重现的、可集成的开源神经语音编解码工具包链接：https://arxiv.org/abs/2309.07405

作者：Zhihao Du,Shiliang Zhang,Kai Hu,Siqi Zheng机构：Speech Lab of DAMO Academy, Alibaba Group, China备注：5 pages, 3 figures, submitted to ICASSP 2024摘要：本文介绍了FunCodec，一个基本的神经语音编解码器工具包，这是一个开源的语音处理工具包FunASR的扩展。FunCodec为最新的神经语音编解码器模型（如SoundStream和Encodec）提供可重复的训练配方和推理脚本。得益于FunASR的统一设计，FunCodec可以轻松集成到下游任务中，例如语音识别。与FunCodec一起，还提供了预训练模型，可用于学术或一般目的。基于该工具包，我们进一步提出了频域编解码器模型，FreqCodec，它可以实现相当的语音质量与低得多的计算和参数复杂度。实验结果表明，在相同的压缩比下，FunCodec的重建质量优于其他工具包和已发布的模型。我们还证明了预训练的模型适用于下游任务，包括自动语音识别和个性化的文本到语音合成。该工具包可在https://github.com/alibaba-damo-academy/FunCodec上公开获取。摘要：This paper presents FunCodec, a fundamental neural speech codec toolkit, which is an extension of the open-source speech processing toolkit FunASR. FunCodec provides reproducible training recipes and inference scripts for the latest neural speech codec models, such as SoundStream and Encodec. Thanks to the unified design with FunASR, FunCodec can be easily integrated into downstream tasks, such as speech recognition. Along with FunCodec, pre-trained models are also provided, which can be used for academic or generalized purposes. Based on the toolkit, we further propose the frequency-domain codec models, FreqCodec, which can achieve comparable speech quality with much lower computation and parameter complexity. Experimental results show that, under the same compression ratio, FunCodec can achieve better reconstruction quality compared with other toolkits and released models. We also demonstrate that the pre-trained models are suitable for downstream tasks, including automatic speech recognition and personalized text-to-speech synthesis. This toolkit is publicly available at https://github.com/alibaba-damo-academy/FunCodec.

【20】 EnCodecMAE: Leveraging neural codecs for universal audio representation learning标题：EnCodecMAE：利用神经编解码器进行通用音频表示学习链接：https://arxiv.org/abs/2309.07391

作者：Leonardo Pepino,Pablo Riera,Luciana Ferrer机构：⋆Instituto de Investigaci´on en Ciencias de la Computaci´on (ICC), CONICET-UBA, Argentina, †Departamento de Computaci´on, FCEyN, Universidad de Buenos Aires (UBA), Argentina备注：Submitted to ICASSP 2024摘要：通用音频表示学习的目标是获得可用于涉及语音、音乐或环境声音的各种下游任务的基础模型。为了解决这个问题，受NLP的自监督模型启发的方法，如BERT，经常被用于音频。这些模型依赖于文本的离散性质，因此采用这种类型的方法进行音频处理需要改变学习目标或将音频信号映射到一组离散类。在这项工作中，我们探讨了使用EnCodec，神经音频编解码器，以生成离散目标，用于学习基于掩蔽自动编码器（MAE）的通用音频模型。我们评估这种方法，我们称之为EncodecMAE，在广泛的音频任务跨越语音，音乐和环境声音，实现性能相当或优于领先的音频表示模型。摘要：The goal of universal audio representation learning is to obtain foundational models that can be used for a variety of downstream tasks involving speech, music or environmental sounds. To approach this problem, methods inspired by self-supervised models from NLP, like BERT, are often used and adapted to audio. These models rely on the discrete nature of text, hence adopting this type of approach for audio processing requires either a change in the learning objective or mapping the audio signal to a set of discrete classes. In this work, we explore the use of EnCodec, a neural audio codec, to generate discrete targets for learning an universal audio model based on a masked autoencoder (MAE). We evaluate this approach, which we call EncodecMAE, on a wide range of audio tasks spanning speech, music and environmental sounds, achieving performances comparable or better than leading audio representation models.

【21】 AudioSR: Versatile Audio Super-resolution at Scale标题：AudioSR：多功能音频超分辨率链接：https://arxiv.org/abs/2309.07314

作者：Haohe Liu,Ke Chen,Qiao Tian,Wenwu Wang,Mark D. Plumbley机构：Centre for Vision Speech and Signal Processing, University of Surrey, University of California San Diego, Speech, Audio & Music Intelligence (SAMI), ByteDance备注：Under review. Demo and code: this https URL摘要：音频超分辨率是预测低分辨率音频的高频分量、提高数字应用中的音频质量的基本任务。先前的方法具有限制，例如音频类型的有限范围（例如，音乐、语音）和它们可以处理的特定带宽设置（例如，4kHz至8kHz）。在本文中，我们介绍了一个基于扩散的生成模型，AudioSR，能够执行强大的音频超分辨率的通用音频类型，包括声音效果，音乐和语音。具体而言，AudioSR可以将2kHz至16kHz带宽范围内的任何输入音频信号上采样为24kHz带宽的高分辨率音频信号，采样率为48kHz。对各种音频超分辨率基准的广泛客观评估表明，所提出的模型实现了强大的结果。此外，我们的主观评估表明，AudioSR可以作为一个即插即用模块，以提高广泛的音频生成模型的生成质量，包括AudioLDM，Fastspeech2和MusicGen。我们的代码和演示可在https://audioldm.github.io/audiosr上获得。摘要：Audio super-resolution is a fundamental task that predicts high-frequency components for low-resolution audio, enhancing audio quality in digital applications. Previous methods have limitations such as the limited scope of audio types (e.g., music, speech) and specific bandwidth settings they can handle (e.g., 4kHz to 8kHz). In this paper, we introduce a diffusion-based generative model, AudioSR, that is capable of performing robust audio super-resolution on versatile audio types, including sound effects, music, and speech. Specifically, AudioSR can upsample any input audio signal within the bandwidth range of 2kHz to 16kHz to a high-resolution audio signal at 24kHz bandwidth with a sampling rate of 48kHz. Extensive objective evaluation on various audio super-resolution benchmarks demonstrates the strong result achieved by the proposed model. In addition, our subjective evaluation shows that AudioSR can acts as a plug-and-play module to enhance the generation quality of a wide range of audio generative models, including AudioLDM, Fastspeech2, and MusicGen. Our code and demo are available at https://audioldm.github.io/audiosr.

【22】 Diffusion models for audio semantic communication标题：音频语义传播的扩散模型链接：https://arxiv.org/abs/2309.07195

作者：Eleonora Grassucci,Christian Marinoni,Andrea Rodriguez,Danilo Comminiello机构：Dept. of Information Eng., Electronics and Telecom., Sapienza University of Rome, Italy备注：Submitted to IEEE ICASSP 2024摘要：跨越噪声信道将音频信号从发射器直接发送到接收器可吸收一致带宽且在尝试恢复所发射位时易于出错。相反，最近的语义通信方法提出发送语义，然后在接收器处重新生成语义一致的内容，而不需要精确地恢复比特流。在本文中，我们提出了一个生成的音频语义通信框架，面临的通信问题作为一个逆问题，因此是强大的不同的腐败。我们的方法将音频信号和相关联的语义的低维表示发送到接收器，接收器生成特别关注其含义的对应信号（即，语义），这要归功于其核心的条件扩散模型。在生成过程期间，扩散模型同时从多个劣化中恢复接收到的信息，所述多个劣化包括由噪声信道上的传输引起的损坏噪声和丢失部分。我们表明，我们的框架优于竞争对手在现实世界中的情况下，不同的信道条件。访问项目页面以收听示例并访问代码：https://ispamm.github.io/diffusion-audio-semantic-communication/。摘要：Directly sending audio signals from a transmitter to a receiver across a noisy channel may absorb consistent bandwidth and be prone to errors when trying to recover the transmitted bits. On the contrary, the recent semantic communication approach proposes to send the semantics and then regenerate semantically consistent content at the receiver without exactly recovering the bitstream. In this paper, we propose a generative audio semantic communication framework that faces the communication problem as an inverse problem, therefore being robust to different corruptions. Our method transmits lower-dimensional representations of the audio signal and of the associated semantics to the receiver, which generates the corresponding signal with a particular focus on its meaning (i.e., the semantics) thanks to the conditional diffusion model at its core. During the generation process, the diffusion model restores the received information from multiple degradations at the same time including corruption noise and missing parts caused by the transmission over the noisy channel. We show that our framework outperforms competitors in a real-world scenario and with different channel conditions. Visit the project page to listen to samples and access the code: https://ispamm.github.io/diffusion-audio-semantic-communication/.

【23】 EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data标题：EMOCONV-DIFF：基于扩散的非平行野外语音情感转换链接：https://arxiv.org/abs/2309.07828

作者：Navin Raj Prabhu,Bunlong Lay,Simon Welker,Nale Lehmann-Willenbrock,Timo Gerkmann机构：⋆Signal Processing, Universität Hamburg, Germany, †Industrial and Organizational Psychology, Universität Hamburg, Germany备注：Submitted to ICASSP 2024摘要：言语情感转换是指在保持言语的词汇内容和说话人身份的前提下，将言语表达的情感转换为目标情感的过程。虽然大多数现有的语音情感转换工作依赖于行动出来的数据集和并行数据样本，在这项工作中，我们特别关注更具挑战性的野外场景，不依赖于并行数据。为此，我们提出了一个基于扩散的语音情感转换生成模型，EmoConv-Diff，这是训练重建输入话语，同时也调节其情绪。随后，在推理时，采用目标情感嵌入来将输入话语的情感转换为给定的目标情感。与在分类表征上执行情感转换相反，我们使用连续唤醒维度来表示情感，同时还实现强度控制。我们验证了所提出的方法在一个大的野外数据集，MSP-Podcast v1.10。我们的研究结果表明，所提出的扩散模型确实能够合成具有可控目标情绪的语音。至关重要的是，所提出的方法显示出改善的性能，沿唤醒的极端值，从而解决了一个共同的挑战，在语音情感转换文学。摘要：Speech emotion conversion is the task of converting the expressed emotion of a spoken utterance to a target emotion while preserving the lexical content and speaker identity. While most existing works in speech emotion conversion rely on acted-out datasets and parallel data samples, in this work we specifically focus on more challenging in-the-wild scenarios and do not rely on parallel data. To this end, we propose a diffusion-based generative model for speech emotion conversion, the EmoConv-Diff, that is trained to reconstruct an input utterance while also conditioning on its emotion. Subsequently, at inference, a target emotion embedding is employed to convert the emotion of the input utterance to the given target emotion. As opposed to performing emotion conversion on categorical representations, we use a continuous arousal dimension to represent emotions while also achieving intensity control. We validate the proposed methodology on a large in-the-wild dataset, the MSP-Podcast v1.10. Our results show that the proposed diffusion model is indeed capable of synthesizing speech with a controllable target emotion. Crucially, the proposed approach shows improved performance along the extreme values of arousal and thereby addresses a common challenge in the speech emotion conversion literature.

【24】 SnakeGAN: A Universal Vocoder Leveraging DDSP Prior Knowledge and Periodic Inductive Bias标题：SnakeGan：一种利用DDSP先验知识和周期性感应偏差的通用声码器链接：https://arxiv.org/abs/2309.07803

作者：Sipan Li,Songxiang Liu,Luwen Zhang,Xiang Li,Yanyao Bian,Chao Weng,Zhiyong Wu,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, AI Lab, Tencent Inc, Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by ICME 2023摘要：基于生成对抗网络（GAN）的神经声码器由于其高生成质量、高效推理和小计算占用空间而被广泛用于音频合成任务。然而，训练一个通用的声码器仍然具有挑战性，它可以很好地推广到域外（OOD）的情况下，如看不见的说话风格，非语音发声，唱歌，音乐作品。在这项工作中，我们提出了SnakeGAN，一个基于GAN的通用声码器，它可以在各种OOD场景中合成高保真音频。SnakeGAN以可微分数字信号处理（DDSP）模型产生的粗粒度信号为先验知识，旨在从Mel谱图中恢复高保真波形。我们引入周期性的非线性通过蛇激活函数和抗锯齿表示到生成器中，这进一步带来了所需的电感偏置音频合成，并显着提高了外推能力，在看不见的情况下，通用声码。为了验证我们所提出的方法的有效性，我们只使用语音数据训练SnakeGAN，并使用主观和客观指标评估其在各种OOD分布中的性能。实验结果表明，SnakeGAN显着优于比较的方法，可以生成高保真的音频样本，包括看不见的扬声器与看不见的风格，歌声，器乐作品，和非言语发声。摘要：Generative adversarial network (GAN)-based neural vocoders have been widely used in audio synthesis tasks due to their high generation quality, efficient inference, and small computation footprint. However, it is still challenging to train a universal vocoder which can generalize well to out-of-domain (OOD) scenarios, such as unseen speaking styles, non-speech vocalization, singing, and musical pieces. In this work, we propose SnakeGAN, a GAN-based universal vocoder, which can synthesize high-fidelity audio in various OOD scenarios. SnakeGAN takes a coarse-grained signal generated by a differentiable digital signal processing (DDSP) model as prior knowledge, aiming at recovering high-fidelity waveform from a Mel-spectrogram. We introduce periodic nonlinearities through the Snake activation function and anti-aliased representation into the generator, which further brings the desired inductive bias for audio synthesis and significantly improves the extrapolation capacity for universal vocoding in unseen scenarios. To validate the effectiveness of our proposed method, we train SnakeGAN with only speech data and evaluate its performance for various OOD distributions with both subjective and objective metrics. Experimental results show that SnakeGAN significantly outperforms the compared approaches and can generate high-fidelity audio samples including unseen speakers with unseen styles, singing voices, instrumental pieces, and nonverbal vocalization.

【25】 Complexity Scaling for Speech Denoising标题：一种语音去噪的复杂度衡量方法链接：https://arxiv.org/abs/2309.07757

作者：Hangting Chen,Jianwei Yu,Chao Weng机构：Tencent AI Lab, Audio and Speech Signal Processing Oteam备注：Submitted to ICASSP2024摘要：在为设备上的应用部署基于深度学习的语音去噪模型时，计算复杂性至关重要。大多数先前的研究集中在优化模型架构以满足特定的计算成本约束，通常针对不同的复杂性限制创建不同的神经网络架构。该研究对语音去噪任务进行复杂度缩放，旨在将具有各种复杂度的模型合并到一个统一的架构中。我们提出了一个基于多路径变换（MPT）的架构来处理低和高复杂度的情况。一系列的MPT网络呈现出高性能，覆盖了DNS挑战数据集上的广泛的计算复杂性。此外，受自然语言处理中的缩放实验的启发，我们探索了去噪任务中模型性能和计算成本之间的经验关系。当MPT网络上的乘法累加运算（MAC）的复杂度从50 M/s缩放到15 G/s时，我们观察到PESQ-WB和SI-SNR的值线性增加，与MAC的对数成比例，这可能有助于理解和应用语音去噪任务中的复杂度缩放。摘要：Computational complexity is critical when deploying deep learning-based speech denoising models for on-device applications. Most prior research focused on optimizing model architectures to meet specific computational cost constraints, often creating distinct neural network architectures for different complexity limitations. This study conducts complexity scaling for speech denoising tasks, aiming to consolidate models with various complexities into a unified architecture. We present a Multi-Path Transform-based (MPT) architecture to handle both low- and high-complexity scenarios. A series of MPT networks present high performance covering a wide range of computational complexities on the DNS challenge dataset. Moreover, inspired by the scaling experiments in natural language processing, we explore the empirical relationship between model performance and computational cost on the denoising task. As the complexity number of multiply-accumulate operations (MACs) is scaled from 50M/s to 15G/s on MPT networks, we observe a linear increase in the values of PESQ-WB and SI-SNR, proportional to the logarithm of MACs, which might contribute to the understanding and application of complexity scaling in speech denoising tasks.

【26】 Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer标题：基于类的语言模型在分解神经传感器中的命名实体识别链接：https://arxiv.org/abs/2309.07648

作者：Peng Wang,Yifan Yang,Zheng Liang,Tian Tan,Shiliang Zhang,Xie Chen机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Key Lab of Speech Acoustics and Content Understanding, Institute of Acoustics, CAS, China摘要：尽管近年来端到端（E2 E）模型在语音识别中取得了长足的进步，但命名实体识别仍然是具有挑战性的，但对于语义理解至关重要。为了提高E2 E模型中命名实体的识别能力，以往的研究主要集中在各种基于规则或基于注意力的上下文偏置算法。然而，它们的性能可能对偏置权重敏感，或者由于对命名实体列表的过度关注而降级，以及错误触发的风险。基于类的语言模型（LM）在传统的混合系统中命名实体识别的成功和有效的解耦的声学和语言信息的因式神经换能器（FNT）的启发，我们提出了一种新的E2 E模型，将基于类的LM到FNT，这是被称为C-FNT。在C-FNT中，命名实体的语言模型分数可以与名称类而不是其表面形式相关联。实验结果表明，我们提出的C-FNT提出了显着的错误减少命名实体，而不损害一般的文字识别性能。摘要：In spite of the excellent strides made by end-to-end (E2E) models in speech recognition in recent years, named entity recognition is still challenging but critical for semantic understanding. In order to enhance the ability to recognize named entities in E2E models, previous studies mainly focus on various rule-based or attention-based contextual biasing algorithms. However, their performance might be sensitive to the biasing weight or degraded by excessive attention to the named entity list, along with a risk of false triggering. Inspired by the success of the class-based language model (LM) in named entity recognition in conventional hybrid systems and the effective decoupling of acoustic and linguistic information in the factorized neural Transducer (FNT), we propose a novel E2E model to incorporate class-based LMs into FNT, which is referred as C-FNT. In C-FNT, the language model score of named entities can be associated with the name class instead of its surface form. The experimental results show that our proposed C-FNT presents significant error reduction in named entities without hurting performance in general word recognition.

【27】 StarGAN-VC++: Towards Emotion Preserving Voice Conversion Using Deep Embeddings标题：StarGan-VC++：基于深度嵌入的情感保真语音转换链接：https://arxiv.org/abs/2309.07592

作者：Arnab Das,Suhita Ghosh,Tim Polzehl,Sebastian Stober机构：Artificial Intelligence Lab (AILab), Otto-von-Guericke-University, Magdeburg, Germany, Mobile Dialog Systems, Otto-von-Guericke-University, Magdeburg, Germany, Speech and Language Technology, German Research Center for Artificial Intelligence (DFKI)备注：Accepted in 12th Speech Synthesis Workshop (SSW), Satellite event in Interspeech 2023摘要：语音转换（VC）是指在不改变语言内容的情况下，将说话人的声音转换为另一个人的声音。最近提出的一种基于生成对抗网络的VC方法StarGANv 2-VC在生成自然的转换方面非常成功。然而，该方法未能在转换后的样本中保留源说话人的情感。情感的保存是自然的人机交互所必需的。在本文中，我们证明了StarGANv 2-VC未能解开扬声器和情感表示，相关的保留情感。具体地，在训练时，存在来自用于捕获说话者嵌入的参考音频的情感泄漏。为了解决这个问题，我们提出了新的情感感知损失和无监督的方法，利用情感监督通过潜在的情感表示。客观和主观评价证明了所提出的策略在不同数据集、情感、性别等上的有效性。摘要：Voice conversion (VC) transforms an utterance to sound like another person without changing the linguistic content. A recently proposed generative adversarial network-based VC method, StarGANv2-VC is very successful in generating natural-sounding conversions. However, the method fails to preserve the emotion of the source speaker in the converted samples. Emotion preservation is necessary for natural human-computer interaction. In this paper, we show that StarGANv2-VC fails to disentangle the speaker and emotion representations, pertinent to preserve emotion. Specifically, there is an emotion leakage from the reference audio used to capture the speaker embeddings while training. To counter the problem, we propose novel emotion-aware losses and an unsupervised method which exploits emotion supervision through latent emotion representations. The objective and subjective evaluations prove the efficacy of the proposed strategy over diverse datasets, emotions, gender, etc.

【28】 Emo-StarGAN: A Semi-Supervised Any-to-Many Non-Parallel Emotion-Preserving Voice Conversion标题：EMO-StarGan：一种半监督的任意对多非并行情感保持语音转换链接：https://arxiv.org/abs/2309.07586

作者：Suhita Ghosh,Arnab Das,Yamini Sinha,Ingo Siegert,Tim Polzehl,Sebastian Stober机构：Artificial Intelligence Lab (AILab), Otto-von-Guericke-University, Magdeburg, Germany, Mobile Dialog Systems, Otto-von-Guericke-University, Magdeburg, Germany, Speech and Language Technology, German Research Center for Artificial Intelligence (DFKI)备注：Accepted in Interspeech 2023摘要：语音匿名化通过移除任何个人标识符同时至少保留语言内容来防止滥用口头数据。然而，情感保存对于自然的人机交互至关重要。众所周知的语音转换技术StarGANv 2-VC实现了匿名化，但未能保留情感。这项工作提出了一个任意对多的半监督StarGANv 2-VC变体，在部分情绪标记的非并行数据上进行训练。我们提出了情感感知损失计算的情感嵌入和声学特征相关的情感。此外，我们使用一个情感分类器，以提供直接的情感监督。客观和主观评价表明，该方法显着提高了香草StarGANv 2-VC的情感保存。这种相当大的改进可以在不同的数据集、情感、目标说话者和组间转换上看到，而不会影响可理解性和匿名化。摘要：Speech anonymisation prevents misuse of spoken data by removing any personal identifier while preserving at least linguistic content. However, emotion preservation is crucial for natural human-computer interaction. The well-known voice conversion technique StarGANv2-VC achieves anonymisation but fails to preserve emotion. This work presents an any-to-many semi-supervised StarGANv2-VC variant trained on partially emotion-labelled non-parallel data. We propose emotion-aware losses computed on the emotion embeddings and acoustic features correlated to emotion. Additionally, we use an emotion classifier to provide direct emotion supervision. Objective and subjective evaluations show that the proposed approach significantly improves emotion preservation over the vanilla StarGANv2-VC. This considerable improvement is seen over diverse datasets, emotions, target speakers, and inter-group conversions without compromising intelligibility and anonymisation.

【29】 Hierarchical Metadata Information Constrained Self-Supervised Learning for Anomalous Sound Detection Under Domain Shift标题：层次元数据信息约束自监督学习在域移异声检测中的应用链接：https://arxiv.org/abs/2309.07498

作者：Haiyan Lan,Qiaoxi Zhu,Jian Guan,Yuming Wei,Wenwu Wang机构：Group of Intelligent Signal Processing, College of Computer Science and Technology, Harbin Engineering University, China, Centre for Audio, Acoustics and Vibration, University of Technology Sydney, Australia摘要：自监督学习方法已经实现了在域移位下的异常声音检测（ASD）的有前途的性能，其中域移位的类型被考虑在特征学习中通过结合部分ID。然而，每个部分下的音频文件所附带的属性，如机器操作条件和噪声类型，尚未被考虑，尽管它们对于表征域偏移也是至关重要的。在本文中，我们提出了一种分层的元数据信息约束自监督（HMIC）ASD方法，其中的部分ID和属性之间的层次关系的构建，并作为约束，以获得更好的特征表示。此外，我们提出了一个属性组中心（AGC）为基础的方法，计算异常的分数下的域转移的条件。在DCASE 2022挑战任务2中进行实验以证明其在最先进的自监督方法上的改进性能。摘要：Self-supervised learning methods have achieved promising performance for anomalous sound detection (ASD) under domain shift, where the type of domain shift is considered in feature learning by incorporating section IDs. However, the attributes accompanying audio files under each section, such as machine operating conditions and noise types, have not been considered, although they are also crucial for characterizing domain shifts. In this paper, we present a hierarchical metadata information constrained self-supervised (HMIC) ASD method, where the hierarchical relation between section IDs and attributes is constructed, and used as constraints to obtain finer feature representation. In addition, we propose an attribute-group-center (AGC)-based method for calculating the anomaly score under the domain shift condition. Experiments are performed to demonstrate its improved performance over the state-of-the-art self-supervised methods in DCASE 2022 challenge Task 2.

【30】 Codec Data Augmentation for Time-domain Heart Sound Classification标题：用于心音分类的编解码器数据增强链接：https://arxiv.org/abs/2309.07466

作者：Ansh Mishra,Jia Qi Yip,Eng Siong Chng机构：Department of Physics, IIT Delhi, New Delhi, India, Interdisciplinary Graduate Programme, Nanyang Technological University (NTU), Singapore, School of Computer Science and Engineering备注：Accepted by ICAICTA 2023摘要：心脏听诊是早期发现心脏瓣膜病的一种低成本、有效的方法，可以挽救生命。然而，由于听诊的有效性取决于医生的技能，因此难以扩展这种筛查方法。因此，使用深度学习算法对心音进行自动分类的研究兴趣越来越大。然而，由于可用于训练的数据有限，目前难以开发出良好的心音分类模型。在这项工作中，我们提出了一个简单的时域方法，心音分类问题的基础分类错误率为0.8，并表明，通过编解码器模拟的数据增强可以提高分类错误率到0.2。通过数据增强，我们的方法优于现有的时域CNN-BiLSTM基线模型。关键的是，我们的实验表明，编解码器数据增强是有效的，绕过数据限制。摘要：Heart auscultations are a low-cost and effective way of detecting valvular heart diseases early, which can save lives. Nevertheless, it has been difficult to scale this screening method since the effectiveness of auscultations is dependent on the skill of doctors. As such, there has been increasing research interest in the automatic classification of heart sounds using deep learning algorithms. However, it is currently difficult to develop good heart sound classification models due to the limited data available for training. In this work, we propose a simple time domain approach, to the heart sound classification problem with a base classification error rate of 0.8 and show that augmentation of the data through codec simulation can improve the classification error rate to 0.2. With data augmentation, our approach outperforms the existing time-domain CNN-BiLSTM baseline model. Critically, our experiments show that codec data augmentation is effective in getting around the data limitation.

【31】 PromptASR for contextualized ASR with controllable style标题：风格可控的语境化ASR链接：https://arxiv.org/abs/2309.07414

作者：Xiaoyu Yang,Wei Kang,Zengwei Yao,Yifan Yang,Liyong Guo,Fangjun Kuang,Long Lin,Daniel Povey机构：Xiaomi Corp. Beijing, China备注：Submitted to ICASSP2024摘要：提示对于大型语言模型至关重要，因为它们提供了上下文信息，例如主题或逻辑关系。受此启发，我们提出了PromptASR，一个框架，集成提示端到端的自动语音识别（E2E ASR）系统，以实现上下文的ASR与可控风格的转录。具体地，专用文本编码器对文本提示进行编码，并且通过交叉关注来自两种模态的特征来将编码注入语音编码器中。当使用地面实况文本从前面的话语作为内容提示，所提出的系统实现了21.9%和6.8%的相对字错误率减少的书籍阅读数据集和内部数据集相比，基线ASR系统。该系统还可以采用词级偏置列表作为提示，以提高对生僻词的识别准确率。可以向文本编码器给出附加的样式提示，并引导ASR系统输出不同样式的转录。代码可在icefall获得。摘要：Prompts are crucial to large language models as they provide context information such as topic or logical relationships. Inspired by this, we propose PromptASR, a framework that integrates prompts in end-to-end automatic speech recognition (E2E ASR) systems to achieve contextualized ASR with controllable style of transcriptions. Specifically, a dedicated text encoder encodes the text prompts and the encodings are injected into the speech encoder by cross-attending the features from two modalities. When using the ground truth text from preceding utterances as content prompt, the proposed system achieves 21.9% and 6.8% relative word error rate reductions on a book reading dataset and an in-house dataset compared to a baseline ASR system. The system can also take word-level biasing lists as prompt to improve recognition accuracy on rare words. An additional style prompt can be given to the text encoder and guide the ASR system to output different styles of transcriptions. The code is available at icefall.

【32】 Multi-dimensional Speech Quality Assessment in Crowdsourcing标题：众包环境下的多维语音质量评估链接：https://arxiv.org/abs/2309.07385

作者：Babak Naderi,Ross Cutler,Nicolae-Catalin Ristea机构：Microsoft Corporation, Redmond, USA摘要：主观语音质量评价是评价语音增强处理和电信系统的金标准。通常使用的标准ITU-T Rec. P.800定义了如何在实验室环境中测量语音质量，ITU-T Rec. P.808扩展到众包。ITU-T Rec.P.835扩展P.800以测量存在噪声时的语音质量。ITU-T Rec.P.804以会话测试为目标，并引入了在会话的收听阶段测量的感知语音质量维度。感知维度是噪声、着色、不连续性和响度。我们创建了一个群众包实现的多维主观测试以下的尺度从P.804和扩展它包括混响，语音信号，和整体质量。我们表明该工具是准确的和可重复的。该工具已用于ICASSP 2023语音信号改进挑战，我们展示了这些语音质量维度在这一挑战中的效用。该工具将在https://github.com/microsoft/P.808上以开源形式公开提供。摘要：Subjective speech quality assessment is the gold standard for evaluating speech enhancement processing and telecommunication systems. The commonly used standard ITU-T Rec. P.800 defines how to measure speech quality in lab environments, and ITU-T Rec.~P.808 extended it for crowdsourcing. ITU-T Rec. P.835 extends P.800 to measure the quality of speech in the presence of noise. ITU-T Rec. P.804 targets the conversation test and introduces perceptual speech quality dimensions which are measured during the listening phase of the conversation. The perceptual dimensions are noisiness, coloration, discontinuity, and loudness. We create a crowdsourcing implementation of a multi-dimensional subjective test following the scales from P.804 and extend it to include reverberation, the speech signal, and overall quality. We show the tool is both accurate and reproducible. The tool has been used in the ICASSP 2023 Speech Signal Improvement challenge and we show the utility of these speech quality dimensions in this challenge. The tool will be publicly available as open-source at https://github.com/microsoft/P.808.

【33】 Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS标题：走向通用语音离散标记--以ASR和TTS为例链接：https://arxiv.org/abs/2309.07377

作者：Yifan Yang,Feiyu Shen,Chenpeng Du,Ziyang Ma,Kai Yu,Daniel Povey,Xie Chen机构： MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Xiaomi Corporation, Beijing, China摘要：语音相关任务中的自我监督学习（SSL）熟练程度已经推动了利用离散令牌进行语音任务（如识别和翻译）的研究，这些任务提供了较低的存储要求和采用自然语言处理技术的巨大潜力。然而，这些研究，主要是单任务的重点，面临的挑战，如过拟合和性能下降的语音识别任务，往往在牺牲多任务的情况下的性能为代价。本研究提出了一个全面的比较和优化的离散令牌所产生的各种领先的SSL模型在语音识别和合成任务。我们的目标是探索在多个语音任务的语音离散令牌的普遍性。实验结果表明，离散令牌实现可比的结果对系统训练的FBank功能的语音识别任务，并优于梅尔频谱图功能的语音合成中的主观和客观的指标。这些研究结果表明，通用离散令牌在各种语音相关的任务有巨大的潜力。我们的工作是开源和公开的，以促进这一方向的研究。摘要：Self-supervised learning (SSL) proficiency in speech-related tasks has driven research into utilizing discrete tokens for speech tasks like recognition and translation, which offer lower storage requirements and great potential to employ natural language processing techniques. However, these studies, mainly single-task focused, faced challenges like overfitting and performance degradation in speech recognition tasks, often at the cost of sacrificing performance in multi-task scenarios. This study presents a comprehensive comparison and optimization of discrete tokens generated by various leading SSL models in speech recognition and synthesis tasks. We aim to explore the universality of speech discrete tokens across multiple speech tasks. Experimental results demonstrate that discrete tokens achieve comparable results against systems trained on FBank features in speech recognition tasks and outperform mel-spectrogram features in speech synthesis in subjective and objective metrics. These findings suggest that universal discrete tokens have enormous potential in various speech-related tasks. Our work is open-source and publicly available to facilitate research in this direction.

【34】 Training Audio Captioning Models without Audio标题：无音频的训练音频字幕模型链接：https://arxiv.org/abs/2309.07372

作者：Soham Deshmukh,Benjamin Elizalde,Dimitra Emmanouilidou,Bhiksha Raj,Rita Singh,Huaming Wang机构：Microsoft,Carnegie Mellon University摘要：自动音频字幕（AAC）是在给定音频流的情况下生成自然语言描述的任务。典型的AAC系统需要手动策划的音频片段的训练数据和对应的文本字幕注释。创建这些音频-字幕对的成本很高，导致任务的一般数据稀缺。在这项工作中，我们解决了这个主要的限制，并提出了一种方法来训练AAC系统只使用文本。我们的方法利用了对比训练的音频文本模型（如CLAP）的多模态空间。在训练期间，解码器生成以预训练的CLAP文本编码器为条件的字幕。在推断期间，文本编码器被替换为预训练的CLAP音频编码器。为了弥合文本和音频嵌入之间的模态差距，我们建议在训练过程中使用噪声注入或可学习的适配器。我们发现，所提出的纯文本框架与用配对音频训练的最先进的模型具有竞争力，表明有效的文本到音频传输是可能的。最后，我们展示了程式化的音频字幕和字幕丰富，同时在没有音频或人类创建的文本字幕的情况下进行训练。摘要：Automated Audio Captioning (AAC) is the task of generating natural language descriptions given an audio stream. A typical AAC system requires manually curated training data of audio segments and corresponding text caption annotations. The creation of these audio-caption pairs is costly, resulting in general data scarcity for the task. In this work, we address this major limitation and propose an approach to train AAC systems using only text. Our approach leverages the multimodal space of contrastively trained audio-text models, such as CLAP. During training, a decoder generates captions conditioned on the pretrained CLAP text encoder. During inference, the text encoder is replaced with the pretrained CLAP audio encoder. To bridge the modality gap between text and audio embeddings, we propose the use of noise injection or a learnable adapter, during training. We find that the proposed text-only framework performs competitively with state-of-the-art models trained with paired audio, showing that efficient text-to-audio transfer is possible. Finally, we showcase both stylized audio captioning and caption enrichment while training without audio or human-created text captions.

【35】 Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation标题：高效语言模型自适应的基于注意力的混合编解码器模型链接：https://arxiv.org/abs/2309.07369

作者：Shaoshi Ling,Guoli Ye,Rui Zhao,Yifan Gong机构：Microsoft Could and AI摘要：基于注意力的编码器-解码器（AED）语音识别模型近年来获得了广泛的成功。然而，声学模型和语言模型端到端的联合优化给文本自适应带来了挑战。特别地，有效地、快速地和廉价地适配文本已经成为在工业中部署AED系统的主要关注点。为了解决这个问题，我们提出了一种新的模型，混合注意力为基础的编码器-解码器（HAED）的语音识别模型，保留了传统的混合自动语音识别系统的模块化。我们的HAED模型分离的声学和语言模型，允许使用传统的基于文本的语言模型适应技术。我们表明，建议的HAED模型产生21\%的字错误率（WER）的改善相对时，域外文本数据用于语言模型的适应，只有一个轻微的退化WER在一般的测试集相比，传统的AED模型。摘要：Attention-based encoder-decoder (AED) speech recognition model has been widely successful in recent years. However, the joint optimization of acoustic model and language model in end-to-end manner has created challenges for text adaptation. In particular, effectively, quickly and inexpensively adapting text has become a primary concern for deploying AED systems in industry. To address this issue, we propose a novel model, the hybrid attention-based encoder-decoder (HAED) speech recognition model that preserves the modularity of conventional hybrid automatic speech recognition systems. Our HAED model separates the acoustic and language models, allowing for the use of conventional text-based language model adaptation techniques. We demonstrate that the proposed HAED model yields 21\% Word Error Rate (WER) improvements in relative when out-of-domain text data is used for language model adaptation, and with only a minor degradation in WER on a general test set compared with conventional AED model.

【36】 Enhancing Child Vocalization Classification in Multi-Channel Child-Adult Conversations Through Wav2vec2 Children ASR Features标题：利用Wav2ve2儿童ASR特征增强多通道儿童-成人对话中的儿童发声分类链接：https://arxiv.org/abs/2309.07287

作者：Jialu Li,Mark Hasegawa-Johnson,Karrie Karahalios机构：Department of Electrical and Computer Engineering, University of Illinois, Beckman Institute for Advanced Science and Technology, University of Illinois, Department of Computer Science, University of Illinois备注：Submitted to ICASSP 2024摘要：自闭症谱系障碍（ASD）是一种神经发育障碍，经常出现在儿童早期。ASD评估通常涉及观察协议，包括由训练有素的临床医生进行的儿童社会行为的记录和评级。能够标记成人和儿童音频的健壮机器学习（ML）模型具有在手动编码儿童行为方面节省大量时间和劳动的潜力。这可以帮助临床医生捕获感兴趣的事件，更好地与父母沟通事件，并教育新的临床医生。在这项研究中，我们利用自我监督学习模型，Wav 2 Vec 2.0（W2 V2），预先训练了4300 h的家庭录音5岁以下的儿童，建立一个统一的系统，执行说话人diarization（SD）和发声分类（VC）的任务。我们将此系统应用于两通道的音频记录简短的3-5分钟的临床医生与儿童的互动使用Rapid-ABC语料库。我们提出了一种新的技术，通过引入辅助特征提取的W2 V2为基础的自动语音识别（ASR）系统4岁以下的儿童，以提高儿童的VC任务。我们测试我们提出的方法，提高儿童的VC任务两个语料库（Rapid-ABC和BabbleCor），并观察到一致的改进。此外，我们达到，或者说超越，最先进的BabbleCor性能。摘要：Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that often emerges in early childhood. ASD assessment typically involves an observation protocol including note-taking and ratings of child's social behavior conducted by a trained clinician. A robust machine learning (ML) model that is capable of labeling adult and child audio has the potential to save significant time and labor in manual coding children's behaviors. This may assist clinicians capture events of interest, better communicate events with parents, and educate new clinicians. In this study, we leverage the self-supervised learning model, Wav2Vec 2.0 (W2V2), pretrained on 4300h of home recordings of children under 5 years old, to build a unified system that performs both speaker diarization (SD) and vocalization classification (VC) tasks. We apply this system to two-channel audio recordings of brief 3-5 minute clinician-child interactions using the Rapid-ABC corpus. We propose a novel technique by introducing auxiliary features extracted from W2V2-based automatic speech recognition (ASR) system for children under 4 years old to improve children's VC task. We test our proposed method of improving children's VC task on two corpora (Rapid-ABC and BabbleCor) and observe consistent improvements. Furthermore, we reach, or perhaps outperform, the state-of-the-art performance of BabbleCor.

【37】 Audio-Based Classification of Respiratory Diseases using Advanced Signal Processing and Machine Learning for Assistive Diagnosis Support标题：采用先进信号处理和机器学习的呼吸系统疾病音频分类辅助诊断链接：https://arxiv.org/abs/2309.07183

作者：Constantino Álvarez Casado,Manuel Lage Cañellas,Matteo Pedone,Xiaoting Wu,Miguel Bordallo López机构：Miguel Bordallo L´opez, Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland备注：5 pages, 2 figures, 3 tables, Conference paper摘要：在全球医疗保健中，呼吸系统疾病是导致死亡的主要原因，这突出了对快速准确诊断的需求。为了通过听诊推进快速筛查技术，我们的研究重点是采用最大的公开可用的呼吸音医学数据库之一来训练能够对不同健康状况进行分类的多个机器学习模型。我们的方法结合了经验模式分解（EMD）和频谱分析，从声学数据中提取生理相关的生物信号，与心血管和呼吸模式密切相关，使我们的方法不同于传统的音频特征提取实践。我们使用功率谱密度分析和滤波技术来选择与潜在的生理现象强烈相关的固有模式函数（IMF）。这些生物信号经历用于预测建模的综合特征提取过程。最初，我们部署了一个二元分类模型，该模型在区分健康和患病个体方面表现出87%的平衡准确度。随后，我们采用了六类分类模型，在诊断肺炎和慢性阻塞性肺疾病（COPD）等特定呼吸疾病时，该模型的平衡准确率为72%。我们还首次引入了仅基于声学数据估计年龄和体重指数（BMI）的回归模型，以及性别分类模型。我们的研究结果强调了这种方法显着提高辅助和远程诊断能力的潜力。摘要：In global healthcare, respiratory diseases are a leading cause of mortality, underscoring the need for rapid and accurate diagnostics. To advance rapid screening techniques via auscultation, our research focuses on employing one of the largest publicly available medical database of respiratory sounds to train multiple machine learning models able to classify different health conditions. Our method combines Empirical Mode Decomposition (EMD) and spectral analysis to extract physiologically relevant biosignals from acoustic data, closely tied to cardiovascular and respiratory patterns, making our approach apart in its departure from conventional audio feature extraction practices. We use Power Spectral Density analysis and filtering techniques to select Intrinsic Mode Functions (IMFs) strongly correlated with underlying physiological phenomena. These biosignals undergo a comprehensive feature extraction process for predictive modeling. Initially, we deploy a binary classification model that demonstrates a balanced accuracy of 87% in distinguishing between healthy and diseased individuals. Subsequently, we employ a six-class classification model that achieves a balanced accuracy of 72% in diagnosing specific respiratory conditions like pneumonia and chronic obstructive pulmonary disease (COPD). For the first time, we also introduce regression models that estimate age and body mass index (BMI) based solely on acoustic data, as well as a model for gender classification. Our findings underscore the potential of this approach to significantly enhance assistive and remote diagnostic capabilities.

【38】 Hybrid ASR for Resource-Constrained Robots: HMM - Deep Learning Fusion标题：资源受限机器人的混合ASR：HMM-深度学习融合链接：https://arxiv.org/abs/2309.07164

作者：Anshul Ranjan,Kaushik Jegadeesan机构：Computer Science and Engineering, PES University, Bangalore, India备注：To be published in IEEE Access, 9 pages, 14 figures, Received valuable support from CCBD PESU, for associated code, see this https URL摘要：本文提出了一种新的混合自动语音识别（ASR）系统，专为资源受限的机器人。所提出的方法将隐马尔可夫模型（HMM）与深度学习模型相结合，并利用套接字编程来有效地分配处理任务。在这种架构中，基于HMM的处理发生在机器人内部，而单独的PC处理深度学习模型。HMM和深度学习之间的这种协同作用显著提高了语音识别的准确性。我们在各种机器人平台上进行了实验，展示了实时和精确的语音识别能力。值得注意的是，该系统表现出对不断变化的声学条件的适应性以及与低功耗硬件的兼容性，使其在计算资源有限的环境中非常有效。这种混合ASR范例为无缝人机交互开辟了有希望的可能性。总之，我们的研究为机器人定制的ASR技术引入了一个开创性的维度。通过采用套接字编程在不同的设备上分配处理任务，并将HMM与深度学习模型战略性地结合起来，我们的混合ASR系统展示了其潜力，使机器人能够熟练地理解和响应口语，即使在计算资源有限的环境中也是如此。这种模式为在广泛的现实世界场景中增强人机交互设置了一个创新的过程。摘要：This paper presents a novel hybrid Automatic Speech Recognition (ASR) system designed specifically for resource-constrained robots. The proposed approach combines Hidden Markov Models (HMMs) with deep learning models and leverages socket programming to distribute processing tasks effectively. In this architecture, the HMM-based processing takes place within the robot, while a separate PC handles the deep learning model. This synergy between HMMs and deep learning enhances speech recognition accuracy significantly. We conducted experiments across various robotic platforms, demonstrating real-time and precise speech recognition capabilities. Notably, the system exhibits adaptability to changing acoustic conditions and compatibility with low-power hardware, making it highly effective in environments with limited computational resources. This hybrid ASR paradigm opens up promising possibilities for seamless human-robot interaction. In conclusion, our research introduces a pioneering dimension to ASR techniques tailored for robotics. By employing socket programming to distribute processing tasks across distinct devices and strategically combining HMMs with deep learning models, our hybrid ASR system showcases its potential to enable robots to comprehend and respond to spoken language adeptly, even in environments with restricted computational resources. This paradigm sets a innovative course for enhancing human-robot interaction across a wide range of real-world scenarios.

【39】 DGSD: Dynamical Graph Self-Distillation for EEG-Based Auditory Spatial Attention Detection标题：DGSD：基于脑电的动态图形自提取听觉空间注意检测链接：https://arxiv.org/abs/2309.07147

作者：Cunhang Fan,Hongyu Zhang,Wei Huang,Jun Xue,Jianhua Tao,Jiangyan Yi,Zhao Lv,Xiaopei Wu摘要：听觉注意检测（AAD）的目的是在多说话人环境中从大脑信号中检测出目标说话人。尽管近年来基于EEG的AAD方法已经显示出有希望的结果，但目前的方法主要依赖于传统的卷积神经网络，该卷积神经网络被设计用于处理像图像这样的欧几里得数据。这使得处理具有非欧几里得特征的EEG信号具有挑战性。为了解决这个问题，本文提出了一种动态图自蒸馏（DGSD）的方法，AAD，它不需要语音刺激作为输入。具体而言，为了有效地表示EEG信号的非欧性质，动态图卷积网络被应用于表示EEG信号的图结构，其还可以提取EEG信号中与听觉空间注意相关的关键特征。此外，为了进一步提高AAD检测性能，自蒸馏，在每一层的特征蒸馏和分层蒸馏策略组成的，是集成的。这些策略利用来自最深网络层的特征和分类结果来指导浅层的学习。我们的实验进行了两个公开的数据集，KUL和DTU。在1秒的时间窗口下，我们在KUL和DTU上分别获得了90.0%和79.6%的准确度。我们比较我们的DGSD方法与竞争基线，实验结果表明，我们提出的DGSD方法的检测性能不仅优于最佳可重复的基线，但也显着减少了约100倍的可训练参数的数量。摘要：Auditory Attention Detection (AAD) aims to detect target speaker from brain signals in a multi-speaker environment. Although EEG-based AAD methods have shown promising results in recent years, current approaches primarily rely on traditional convolutional neural network designed for processing Euclidean data like images. This makes it challenging to handle EEG signals, which possess non-Euclidean characteristics. In order to address this problem, this paper proposes a dynamical graph self-distillation (DGSD) approach for AAD, which does not require speech stimuli as input. Specifically, to effectively represent the non-Euclidean properties of EEG signals, dynamical graph convolutional networks are applied to represent the graph structure of EEG signals, which can also extract crucial features related to auditory spatial attention in EEG signals. In addition, to further improve AAD detection performance, self-distillation, consisting of feature distillation and hierarchical distillation strategies at each layer, is integrated. These strategies leverage features and classification results from the deepest network layers to guide the learning of shallow layers. Our experiments are conducted on two publicly available datasets, KUL and DTU. Under a 1-second time window, we achieve results of 90.0\% and 79.6\% accuracy on KUL and DTU, respectively. We compare our DGSD method with competitive baselines, and the experimental results indicate that the detection performance of our proposed DGSD method is not only superior to the best reproducible baseline but also significantly reduces the number of trainable parameters by approximately 100 times.

eess.AS音频处理

【1】 EMOCONV-DIFF: Diffusion-based Speech Emotion Conversion for Non-parallel and In-the-wild Data标题：EMOCONV-DIFF：基于扩散的非平行野外语音情感转换链接：https://arxiv.org/abs/2309.07828

【2】 SnakeGAN: A Universal Vocoder Leveraging DDSP Prior Knowledge and Periodic Inductive Bias标题：SnakeGan：一种利用DDSP先验知识和周期性感应偏差的通用声码器链接：https://arxiv.org/abs/2309.07803

【3】 Complexity Scaling for Speech Denoising标题：一种语音去噪的复杂度衡量方法链接：https://arxiv.org/abs/2309.07757

【4】 Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer标题：基于类的语言模型在分解神经传感器中的命名实体识别链接：https://arxiv.org/abs/2309.07648

【5】 StarGAN-VC++: Towards Emotion Preserving Voice Conversion Using Deep Embeddings标题：StarGan-VC++：基于深度嵌入的情感保真语音转换链接：https://arxiv.org/abs/2309.07592

【6】 Emo-StarGAN: A Semi-Supervised Any-to-Many Non-Parallel Emotion-Preserving Voice Conversion标题：EMO-StarGan：一种半监督的任意对多非并行情感保持语音转换链接：https://arxiv.org/abs/2309.07586

【7】 Hierarchical Metadata Information Constrained Self-Supervised Learning for Anomalous Sound Detection Under Domain Shift标题：层次元数据信息约束自监督学习在域移异声检测中的应用链接：https://arxiv.org/abs/2309.07498

【8】 Codec Data Augmentation for Time-domain Heart Sound Classification标题：用于心音分类的编解码器数据增强链接：https://arxiv.org/abs/2309.07466

【9】 PromptASR for contextualized ASR with controllable style标题：风格可控的语境化ASR链接：https://arxiv.org/abs/2309.07414

【10】 Multi-dimensional Speech Quality Assessment in Crowdsourcing标题：众包环境下的多维语音质量评估链接：https://arxiv.org/abs/2309.07385

【11】 Towards Universal Speech Discrete Tokens: A Case Study for ASR and TTS标题：走向通用语音离散标记--以ASR和TTS为例链接：https://arxiv.org/abs/2309.07377

【12】 Training Audio Captioning Models without Audio标题：无音频的训练音频字幕模型链接：https://arxiv.org/abs/2309.07372

【13】 Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptation标题：高效语言模型自适应的基于注意力的混合编解码器模型链接：https://arxiv.org/abs/2309.07369

【14】 Enhancing Child Vocalization Classification in Multi-Channel Child-Adult Conversations Through Wav2vec2 Children ASR Features标题：利用Wav2ve2儿童ASR特征增强多通道儿童-成人对话中的儿童发声分类链接：https://arxiv.org/abs/2309.07287

【15】 Hybrid ASR for Resource-Constrained Robots: HMM - Deep Learning Fusion标题：资源受限机器人的混合ASR：HMM-深度学习融合链接：https://arxiv.org/abs/2309.07164

【16】 DGSD: Dynamical Graph Self-Distillation for EEG-Based Auditory Spatial Attention Detection标题：DGSD：基于脑电的动态图形自提取听觉空间注意检测链接：https://arxiv.org/abs/2309.07147

【17】 CiwaGAN: Articulatory information exchange标题：CiwaGan：发音信息交流链接：https://arxiv.org/abs/2309.07861

【18】 Echotune: A Modular Extractor Leveraging the Variable-Length Nature of Speech in ASR Tasks标题：Echotune：在ASR任务中利用语音的可变长度特性的模块化抽取器链接：https://arxiv.org/abs/2309.07765

【19】 The complementary roles of non-verbal cues for Robust Pronunciation Assessment标题：非语言线索在强健发音评估中的互补作用链接：https://arxiv.org/abs/2309.07739

【20】 Explaining Speech Classification Models via Word-Level Audio Segments and Paralinguistic Features标题：基于词级音段和副语言特征的语音分类模型解释链接：https://arxiv.org/abs/2309.07733

【21】 L1-aware Multilingual Mispronunciation Detection Framework标题：一种L1感知的多语种发音错误检测框架链接：https://arxiv.org/abs/2309.07719

【22】 CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders标题：CoLLD：用于压缩多语种预训练语音编码器的对比性分层蒸馏链接：https://arxiv.org/abs/2309.07707

【23】 DDSP-based Neural Waveform Synthesis of Polyphonic Guitar Performance from String-wise MIDI Input标题：基于DDSP的弦式MIDI输入复调吉他演奏的神经波形合成链接：https://arxiv.org/abs/2309.07658

【24】 Multilingual Audio Captioning using machine translated data标题：使用机器翻译数据的多语言音频字幕链接：https://arxiv.org/abs/2309.07615

【25】 AAS-VC: On the Generalization Ability of Automatic Alignment Search based Non-autoregressive Sequence-to-sequence Voice Conversion标题：AAS-VC：基于自动比对搜索的非自回归序列到序列语音转换的泛化能力链接：https://arxiv.org/abs/2309.07598

【26】 Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer标题：基于离散单元风格转换的语音到语音翻译链接：https://arxiv.org/abs/2309.07566

【27】 SingFake: Singing Voice Deepfake Detection标题：SingFake：歌唱声音的深伪检测链接：https://arxiv.org/abs/2309.07525

【28】 Outlier-aware Inlier Modeling and Multi-scale Scoring for Anomalous Sound Detection via Multitask Learning标题：基于多任务学习的异常点感知内点建模和多尺度评分链接：https://arxiv.org/abs/2309.07500

【29】 Direct Text to Speech Translation System using Acoustic Units标题：使用声学单元的直接文本到语音翻译系统链接：https://arxiv.org/abs/2309.07478

【30】 Analysis of Speech Separation Performance Degradation on Emotional Speech Mixtures标题：情感语音混合的语音分离性能劣化分析链接：https://arxiv.org/abs/2309.07458

【31】 SpatialCodec: Neural Spatial Speech Coding标题：SpatialCodec：神经空间语音编码链接：https://arxiv.org/abs/2309.07432

【32】 Mandarin Lombard Flavor Classification标题：柑橘伦巴第香精分类链接：https://arxiv.org/abs/2309.07419

【33】 M3-AUDIODEC: Multi-channel multi-speaker multi-spatial audio codec标题：M3-AUDIODEC：多声道多扬声器多空间音频编解码器链接：https://arxiv.org/abs/2309.07416

【34】 CPPF: A contextual and post-processing-free model for automatic speech recognition标题：CPPF：一种无需上下文和后处理的自动语音识别模型链接：https://arxiv.org/abs/2309.07413

【35】 FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec标题：FunCodec：一个基础的、可重现的、可集成的开源神经语音编解码工具包链接：https://arxiv.org/abs/2309.07405

【36】 EnCodecMAE: Leveraging neural codecs for universal audio representation learning标题：EnCodecMAE：利用神经编解码器进行通用音频表示学习链接：https://arxiv.org/abs/2309.07391

【37】 AudioSR: Versatile Audio Super-resolution at Scale标题：AudioSR：多功能音频超分辨率链接：https://arxiv.org/abs/2309.07314

【38】 Diffusion models for audio semantic communication标题：音频语义传播的扩散模型链接：https://arxiv.org/abs/2309.07195

机器翻译，仅供参考

语音/音频处理学术速递[9.15]

评论 0