语音/音频处理学术速递[5.24]

今日论文合集：cs.SD语音25篇，eess.AS音频处理35篇。

cs.SD语音

【1】 QFA2SR: Query-Free Adversarial Transfer Attacks to Speaker Recognition Systems标题：QFA2SR：对说话人识别系统的无查询对抗性转移攻击链接：https://arxiv.org/abs/2305.14097

作者：Guangke Chen,Yedi Zhang,Zhe Zhao,Fu Song机构： ShanghaiTech University, Automotive Software Innovation Center, Institute of Software, Chinese Academy of Sciences & University of Chinese Academy of Sciences备注：Accepted by the 32nd USENIX Security Symposium (2023 USENIX Security); Full Version摘要：目前针对说话人识别系统（SRS）的对抗性攻击需要对目标SRS进行白盒访问或大量黑盒查询，因此仍然落后于针对专有商业API和语音控制设备的实际攻击。为了填补这一空白，我们提出了QFA 2SR，这是一种有效且不可感知的无查询黑盒攻击，它利用了对抗性声音的可转移性。为了提高可转移性，我们提出了三种新的方法，定制的损失函数，SRS合奏，和时间-频率腐蚀。第一个是针对不同的攻击场景定制损失函数。后两个以两种不同的方式增强代理SRS。SRS集成结合了不同的替代SRS与新的策略，服从SRS的独特的评分特性。时间-频率腐蚀通过结合精心设计的时域/频域修改函数来增强代理SRS，所述修改函数模拟和近似在空中攻击期间引入的目标SRS和失真的决策边界。QFA 2SR在四个流行的商业API（Microsoft Azure，iFlytek，Jingdong和TalentedSoft）上将目标可转移性提高了20.9%-70.7%，在无查询设置下显著优于现有攻击，对不可感知性的影响可以忽略不计。QFA 2SR在针对三种广泛使用的语音助手（Google Assistant，Apple Siri和TMall Genie）进行无线发射时也非常有效，分别具有60%，46%和70%的目标可转移性。摘要：Current adversarial attacks against speaker recognition systems (SRSs) require either white-box access or heavy black-box queries to the target SRS, thus still falling behind practical attacks against proprietary commercial APIs and voice-controlled devices. To fill this gap, we propose QFA2SR, an effective and imperceptible query-free black-box attack, by leveraging the transferability of adversarial voices. To improve transferability, we present three novel methods, tailored loss functions, SRS ensemble, and time-freq corrosion. The first one tailors loss functions to different attack scenarios. The latter two augment surrogate SRSs in two different ways. SRS ensemble combines diverse surrogate SRSs with new strategies, amenable to the unique scoring characteristics of SRSs. Time-freq corrosion augments surrogate SRSs by incorporating well-designed time-/frequency-domain modification functions, which simulate and approximate the decision boundary of the target SRS and distortions introduced during over-the-air attacks. QFA2SR boosts the targeted transferability by 20.9%-70.7% on four popular commercial APIs (Microsoft Azure, iFlytek, Jingdong, and TalentedSoft), significantly outperforming existing attacks in query-free setting, with negligible effect on the imperceptibility. QFA2SR is also highly effective when launched over the air against three wide-spread voice assistants (Google Assistant, Apple Siri, and TMall Genie) with 60%, 46%, and 70% targeted transferability, respectively.

【2】 Disentangled Variational Autoencoder for Emotion Recognition in Conversations标题：一种用于会话中情感识别的解缠变分自动编码器链接：https://arxiv.org/abs/2305.14071

作者：Kailai Yang,Tianlin Zhang,Sophia Ananiadou机构： and Sophia Ananiadou are with NaCTeMand the Department of Computer Science, The University of Manchester备注：Accepted by IEEE Transactions on Affective Computing摘要：在会话情感识别中，目标话语的情感与其语境密切相关。因此，现有的作品训练模型来生成目标话语的响应，其目的是利用上下文信息来识别情绪。然而，相邻响应生成忽略了长程依赖性，并在许多情况下提供有限的情感信息。此外，大多数ERC模型为每个话语学习统一的分布式表示，这缺乏可解释性和鲁棒性。为了解决这些问题，我们提出了一个VAD解开变分自动编码器（VAD-VAE），它首先引入了一个目标话语重建任务的基础上变分自动编码器，然后解开三个影响表示价唤醒优势（VAD）从潜在空间。我们还通过引入来自情感词典的VAD监督信号和最小化VAD分布之间的互信息来增强解缠表示。实验表明，VAD-VAE优于国家的最先进的模型在两个数据集上。进一步的分析证明了每个建议的模块的有效性和解纠缠VAD表示的质量。该代码可在www.example.com上获得https://github.com/SteveKGYang/VAD-VAE。摘要：In Emotion Recognition in Conversations (ERC), the emotions of target utterances are closely dependent on their context. Therefore, existing works train the model to generate the response of the target utterance, which aims to recognise emotions leveraging contextual information. However, adjacent response generation ignores long-range dependencies and provides limited affective information in many cases. In addition, most ERC models learn a unified distributed representation for each utterance, which lacks interpretability and robustness. To address these issues, we propose a VAD-disentangled Variational AutoEncoder (VAD-VAE), which first introduces a target utterance reconstruction task based on Variational Autoencoder, then disentangles three affect representations Valence-Arousal-Dominance (VAD) from the latent space. We also enhance the disentangled representations by introducing VAD supervision signals from a sentiment lexicon and minimising the mutual information between VAD distributions. Experiments show that VAD-VAE outperforms the state-of-the-art model on two datasets. Further analysis proves the effectiveness of each proposed module and the quality of disentangled VAD representations. The code is available at https://github.com/SteveKGYang/VAD-VAE.

【3】 Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decoding标题：从声学和语义协同解码的多模式视角反思语音识别链接：https://arxiv.org/abs/2305.14049

作者：Tian-Hao Zhang,Hai-Bo Qin,Zhi-Hao Lai,Song-Lu Chen,Qi Liu,Feng Chen,Xinyuan Qian,Xu-Cheng Yin机构：School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing , China, USTB-EEasyTech Joint Lab of Artificial Intelligence, University of Science and Technology Beijing, Beijing , China备注：Accepted by Interspeech 2023摘要：基于注意力的编码器-解码器（AED）模型在ASR中表现出令人印象深刻的性能。然而，大多数现有的AED方法忽略了同时利用解码器中的声学和语义特征，这对于生成更准确和信息量更大的语义状态至关重要。在本文中，我们提出了一个声学和语义合作解码器（ASCD）的ASR。特别是，与在两个单独的阶段处理声学和语义特征的vanilla解码器不同，ASCD将它们协同集成。为了防止训练过程中的信息泄漏，我们设计了一个因果多模态掩码。此外，一个变种半ASCD提出了平衡精度和计算成本。我们的建议是在公开可用的AISHELL-1和aidatatang_200zh数据集上进行评估，分别使用Transformer，Conformer和Branchformer作为编码器。实验结果表明，ASCD显着提高了性能，同时利用声学和语义信息的合作。摘要：Attention-based encoder-decoder (AED) models have shown impressive performance in ASR. However, most existing AED methods neglect to simultaneously leverage both acoustic and semantic features in decoder, which is crucial for generating more accurate and informative semantic states. In this paper, we propose an Acoustic and Semantic Cooperative Decoder (ASCD) for ASR. In particular, unlike vanilla decoders that process acoustic and semantic features in two separate stages, ASCD integrates them cooperatively. To prevent information leakage during training, we design a Causal Multimodal Mask. Moreover, a variant Semi-ASCD is proposed to balance accuracy and computational cost. Our proposal is evaluated on the publicly available AISHELL-1 and aidatatang_200zh datasets using Transformer, Conformer, and Branchformer as encoders, respectively. The experimental results show that ASCD significantly improves the performance by leveraging both the acoustic and semantic information cooperatively.

【4】 Improving speech translation by fusing speech and text标题：通过语音和文本的融合提高语音翻译水平链接：https://arxiv.org/abs/2305.14042

作者：Wenbiao Yin,Zhicheng Liu,Chengqi Zhao,Tao Wang,Jian Tong,Rong Ye机构：Nanjing University ,ByteDance摘要：在语音翻译中，利用多模态数据来提高模型性能并解决单个模态的限制已经显示出显着的效果。在本文中，我们利用语音和文本，这是不同的模态的互补优势。我们观察到三个层次的模态之间的差距，表示模态输入表示，模态语义，模态隐藏状态。为了解决这些差距，我们提出了\textbf{F}use-\textbf{S}peech-\textbf{T}ext（\textbf{FST}），这是一个跨模态模型，支持三种不同的翻译输入模态：语音、文本和融合语音-文本。我们利用多种技术进行跨模态对齐，并进行全面分析，以评估其对语音翻译、机器翻译和融合语音文本翻译的影响。我们在MuST-C、GigaST和newstest基准测试上评估FST。实验结果表明，所提出的FST在MuST-C En$\rightarrow$De/Es/Fr（vs SOTA +1.1 BLEU）上实现了平均34.0 BLEU。进一步的实验表明，FST不会降低MT任务，在以前的作品中观察到的。相反，它比预训练的MT模型平均提高了3.2 BLEU。摘要：In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text, which are disparate modalities. We observe three levels of modality gap between them, denoted by Modal input representation, Modal semantic, and Modal hidden states. To tackle these gaps, we propose \textbf{F}use-\textbf{S}peech-\textbf{T}ext (\textbf{FST}), a cross-modal model which supports three distinct input modalities for translation: speech, text, and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation, and fused speech-text translation. We evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that the proposed FST achieves an average 34.0 BLEU on MuST-C En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FST does not degrade on MT task, as observed in prior works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model.

【5】 Can Self-Supervised Neural Networks Pre-Trained on Human Speech distinguish Animal Callers?标题：对人类语音进行预训练的自我监督神经网络能区分动物呼叫者吗？链接：https://arxiv.org/abs/2305.14035

作者：Eklavya Sarkar,Mathew Magimai. -Doss机构：Idiap Research Institute, Martigny, Switzerland, Ecole polytechnique f´ed´erale de Lausanne, Switzerland备注：Accepted at Interspeech 2023摘要：自监督学习（SSL）模型仅使用给定信号的固有结构，独立于其声学域，从输入到嵌入空间中提取基本信息。这意味着这样的表示的效用不限于单独建模人类语音。基于这种理解，本文探讨了从人类语音学习到的SSL神经表示的交叉转移性来分析生物声学信号。我们进行了呼叫者歧视分析和呼叫者检测研究狨猴发声使用11个SSL模型预先训练与各种借口的任务。结果表明，嵌入空间携带有意义的呼叫者信息，可以成功地区分狨猴呼叫者的个人身份，而无需微调。这表明，预训练的人类语音表示可以有效地应用于生物声学领域，为未来在该领域的调查提供有价值的见解。摘要：Self-supervised learning (SSL) models use only the intrinsic structure of a given signal, independent of its acoustic domain, to extract essential information from the input to an embedding space. This implies that the utility of such representations is not limited to modeling human speech alone. Building on this understanding, this paper explores the cross-transferability of SSL neural representations learned from human speech to analyze bio-acoustic signals. We conduct a caller discrimination analysis and a caller detection study on Marmoset vocalizations using eleven SSL models pre-trained with various pretext tasks. The results show that the embedding spaces carry meaningful caller information and can successfully distinguish the individual identities of Marmoset callers without fine-tuning. This demonstrates that representations pre-trained on human speech can be effectively applied to the bio-acoustics domain, providing valuable insights for future investigations in this field.

【6】 Happy or Evil Laughter? Analysing a Database of Natural Audio Samples标题：快乐还是邪恶的笑声？一个自然音频样本库的分析链接：https://arxiv.org/abs/2305.14023

作者：Aljoscha Düsterhöft,Felix Burkhardt,Björn W. Schuller机构：Technical University of Berlin, Germany,audEERING GmbH, Germany, Chair EIHW, University of Augsburg, Germany,GLAM, Imperial College London, UK摘要：我们在Google AudioSet数据库的基础上进行了一次数据收集，选择了一个带有\textit{laugh}注释的样本子集。选择的标准是要呈现一个交际行为，并有明确的内涵，要么是积极的（笑）或消极的（被嘲笑）。基于该注释数据，我们进行了两个实验：一方面，我们手动提取和分析语音特征。另一方面，我们通过系统地将几个自动提取的声学特征集与机器学习算法相结合来进行几个机器学习实验。这表明，表现最好的模型可以实现0.7的未加权平均召回率。摘要：We conducted a data collection on the basis of the Google AudioSet database by selecting a subset of the samples annotated with \textit{laughter}. The selection criterion was to be present a communicative act with clear connotation of being either positive (laughing with) or negative (being laughed at). On the basis of this annotated data, we performed two experiments: on the one hand, we manually extract and analyze phonetic features. On the other hand, we conduct several machine learning experiments by systematically combining several automatically extracted acoustic feature sets with machine learning algorithms. This shows that the best performing models can achieve and unweighted average recall of .7.

【7】 ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models标题：ZET-Speech：基于扩散和风格模型的Zero-Shot自适应情感可控文语合成链接：https://arxiv.org/abs/2305.13831

作者：Minki Kang,Wooseok Han,Sung Ju Hwang,Eunho Yang机构：AITRICS, KAIST备注：Accepted by INTERSPEECH 2023摘要：情感文本到语音（TTS）是系统开发中的重要任务（例如，类人对话代理），其需要自然和情感语音。然而，现有的方法，只针对在训练过程中看到的扬声器产生情感的TTS，而不考虑的泛化到看不见的扬声器。在本文中，我们提出了ZET-Speech，一个zero-shot自适应情绪可控的TTS模型，允许用户合成任何扬声器的情感语音只使用一个简短的，中性的语音段和目标情感标签。具体来说，为了使zero-shot自适应TTS模型合成情感语音，我们提出了域对抗学习和指导方法的扩散模型。实验结果表明，ZET-Speech成功地合成自然和情感的语音与所需的情感，为可见和不可见的扬声器。样品在www.example.com上https://ZET-Speech.github.io/ZET-Speech-Demo/。摘要：Emotional Text-To-Speech (TTS) is an important task in the development of systems (e.g., human-like dialogue agents) that require natural and emotional speech. Existing approaches, however, only aim to produce emotional TTS for seen speakers during training, without consideration of the generalization to unseen speakers. In this paper, we propose ZET-Speech, a zero-shot adaptive emotion-controllable TTS model that allows users to synthesize any speaker's emotional speech using only a short, neutral speech segment and the target emotion label. Specifically, to enable a zero-shot adaptive TTS model to synthesize emotional speech, we propose domain adversarial learning and guidance methods on the diffusion model. Experimental results demonstrate that ZET-Speech successfully synthesizes natural and emotional speech with the desired emotion for both seen and unseen speakers. Samples are at https://ZET-Speech.github.io/ZET-Speech-Demo/.

【8】 SE-Bridge: Speech Enhancement with Consistent Brownian Bridge标题：SE-Bridge：基于一致性布朗桥的语音增强链接：https://arxiv.org/abs/2305.13796

作者：Zhibin Qiu,Mengfan Fu,Fuchun Sun,Gulila Altenbek,Hao Huang机构： Fellow, IEEE, Member, IEEE摘要：我们提出了SE-Bridge，一种新的语音增强（SE）的方法。近年来，将扩散模型应用于语音增强，可以通过求解随机微分方程来实现语音增强。每个随机微分方程对应于一个概率流常微分方程（PF-ODE），PF-ODE解的轨迹由不同时刻的语音状态组成。我们的方法是基于一致性模型，确保任何语音状态相同的PF-ODE轨迹，对应于相同的初始状态。通过整合布朗桥过程，该模型能够生成高清晰度的语音样本，而无需对抗训练。这是将一致性模型应用于SE任务的第一次尝试，与基于扩散的基线相比，在几个指标中实现了最先进的结果，同时节省了15倍的采样所需时间。我们在多个数据集上的实验证明了SE-Bridge在SE中的有效性。此外，我们通过对下游任务的广泛实验，包括自动语音识别（ASR）和说话人验证（SV），SE-Bridge可以有效地支持多个下游任务。摘要：We propose SE-Bridge, a novel method for speech enhancement (SE). After recently applying the diffusion models to speech enhancement, we can achieve speech enhancement by solving a stochastic differential equation (SDE). Each SDE corresponds to a probabilistic flow ordinary differential equation (PF-ODE), and the trajectory of the PF-ODE solution consists of the speech states at different moments. Our approach is based on consistency model that ensure any speech states on the same PF-ODE trajectory, correspond to the same initial state. By integrating the Brownian Bridge process, the model is able to generate high-intelligibility speech samples without adversarial training. This is the first attempt that applies the consistency models to SE task, achieving state-of-the-art results in several metrics while saving 15 x the time required for sampling compared to the diffusion-based baseline. Our experiments on multiple datasets demonstrate the effectiveness of SE-Bridge in SE. Furthermore, we show through extensive experiments on downstream tasks, including Automatic Speech Recognition (ASR) and Speaker Verification (SV), that SE-Bridge can effectively support multiple downstream tasks.

【9】 ADD 2023: the Second Audio Deepfake Detection Challenge标题：ADD 2023：第二届音频深伪检测挑战赛链接：https://arxiv.org/abs/2305.13774

作者：Jiangyan Yi,Jianhua Tao,Ruibo Fu,Xinrui Yan,Chenglong Wang,Tao Wang,Chu Yuan Zhang,Xiaohui Zhang,Yan Zhao,Yong Ren,Le Xu,Junzuo Zhou,Hao Gu,Zhengqi Wen,Shan Liang,Zheng Lian,Shuai Nie,Haizhou Li机构：State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China, Department of Automation, Tsinghua University, Beijing, China摘要：音频deepfake检测是人工智能社区中的一个新兴话题。第二届音频Deepfake检测挑战赛（ADD 2023）旨在激励世界各地的研究人员开发新的创新技术，以进一步加速和促进检测和分析deepfake语音的研究。与之前的挑战不同（例如：ADD 2023专注于超越二进制真/假分类的约束，并且实际上定位部分假语音中的操纵间隔以及精确定位负责生成任何假音频的源。此外，ADD 2023包括针对伪音频游戏子挑战的更多轮评估。ADD 2023挑战包括三个子挑战：音频伪造游戏（FG）、操纵区域定位（RL）和深度伪造算法识别（AR）。本文介绍了数据集，评估指标和协议。在音频deepfake检测任务中也报告了一些发现。摘要：Audio deepfake detection is an emerging topic in the artificial intelligence community. The second Audio Deepfake Detection Challenge (ADD 2023) aims to spur researchers around the world to build new innovative technologies that can further accelerate and foster research on detecting and analyzing deepfake speech utterances. Different from previous challenges (e.g. ADD 2022), ADD 2023 focuses on surpassing the constraints of binary real/fake classification, and actually localizing the manipulated intervals in a partially fake speech as well as pinpointing the source responsible for generating any fake audio. Furthermore, ADD 2023 includes more rounds of evaluation for the fake audio game sub-challenge. The ADD 2023 challenge includes three subchallenges: audio fake game (FG), manipulation region location (RL) and deepfake algorithm recognition (AR). This paper describes the datasets, evaluation metrics, and protocols. Some findings are also reported in audio deepfake detection tasks.

【10】 A study of audio mixing methods for piano transcription in violin-piano ensembles标题：小提琴-钢琴重奏中钢琴改编的混音方法研究链接：https://arxiv.org/abs/2305.13758

作者：Hyemi Kim,Jiyun Park,Taegyun Kwon,Dasaem Jeong,Juhan Nam机构：♭Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea, ♮Electronics and Telecommunications Research Institute, Daejeon, South Korea, ♯Graduate School of Culture Technology, KAIST, Daejeon, South Korea, Sogang University, Seoul, South Korea备注：To Appear IEEE ICASSP 2023摘要：虽然钢琴音乐转录模型已经示出了独奏钢琴录音的高性能，但是当应用于合奏录音时，它们的性能降低。本研究旨在分析不同的数据增强方法对钢琴转录性能的影响，特别是集中于应用于小提琴-钢琴合奏的混音技术。我们应用混合方法，同时考虑谐波和时间特性的音频。为了创建本研究的数据集，我们生成了PFVN-synth数据集，该数据集通过渲染MIDI文件和相应的标签包含7小时的小提琴-钢琴合奏音频，并且还收集了无伴奏小提琴录音并将其与MAESTRO数据集混合。我们评估了合成和真实音频记录数据集的转录结果。摘要：While piano music transcription models have shown high performance for solo piano recordings, their performance degrades when applied to ensemble recordings. This study aims to analyze the impact of different data augmentation methods on piano transcription performance, specifically focusing on mixing techniques applied to violin-piano ensembles. We apply mixing methods that consider both harmonic and temporal characteristics of the audio. To create datasets for this study, we generated the PFVN-synth dataset, which contains 7 hours of violin-piano ensemble audio by rendering MIDI files and corresponding labels, and also collected unaccompanied violin recordings and mixed them with the MAESTRO dataset. We evaluated the transcription results on both synthesized and real audio recordings datasets.

【11】 ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddings标题：ChatGPT-EDSS：基于ChatGPT的上下文词嵌入训练的移情对话语音合成链接：https://arxiv.org/abs/2305.13724

作者：Yuki Saito,Shinnosuke Takamichi,Eiji Iimori,Kentaro Tachibana,Hiroshi Saruwatari机构：The University of Tokyo, Japan,LINE Corp., Japan.备注：5 pages, accepted for INTERSPEECH 2023摘要：我们提出ChatGPT-EDSS，一个移情对话语音合成（EDSS）方法，使用ChatGPT提取对话上下文。ChatGPT是一个聊天机器人，可以深入理解输入提示的内容和目的，并适当地响应用户的请求。我们专注于ChatGPT的阅读理解，并将其引入EDSS，这是一项合成语音的任务，可以与对话者的情感产生共鸣。我们的方法首先将聊天历史记录提供给ChatGPT，并要求它为聊天中的每一行生成三个代表意图，情感和说话风格的单词。然后，它使用ChatGPT派生的上下文词的嵌入作为条件特征来训练EDSS模型。实验结果表明，我们的方法执行相比，使用情感标签或神经网络衍生的上下文嵌入学习的聊天历史。所收集的ChatGPT衍生的上下文信息可在www.example.com获得https://sarulab-speech.github.io/demo_ChatGPT_EDSS/。摘要：We propose ChatGPT-EDSS, an empathetic dialogue speech synthesis (EDSS) method using ChatGPT for extracting dialogue context. ChatGPT is a chatbot that can deeply understand the content and purpose of an input prompt and appropriately respond to the user's request. We focus on ChatGPT's reading comprehension and introduce it to EDSS, a task of synthesizing speech that can empathize with the interlocutor's emotion. Our method first gives chat history to ChatGPT and asks it to generate three words representing the intention, emotion, and speaking style for each line in the chat. Then, it trains an EDSS model using the embeddings of ChatGPT-derived context words as the conditioning features. The experimental results demonstrate that our method performs comparably to ones using emotion labels or neural network-derived context embeddings learned from chat histories. The collected ChatGPT-derived context information is available at https://sarulab-speech.github.io/demo_ChatGPT_EDSS/.

【12】 BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR标题：BA-SOT：多人ASR的边界感知串行化输出训练链接：https://arxiv.org/abs/2305.13716

作者：Yuhao Liang,Fan Yu,Yangze Li,Pengcheng Guo,Shiliang Zhang,Qian Chen,Lei Xie机构：Northwestern Polytechnical University, China, Speech Lab of DAMO Academy, Alibaba Group, China备注：Accepted by INTERSPEECH 2023摘要：最近提出的序列化输出训练（SOT）简化了多说话人自动语音识别（ASR），通过生成由一个特殊的令牌分开的说话人转录。然而，频繁的说话者改变可能使得说话者改变预测困难。为了解决这个问题，我们提出了边界感知的串行输出训练（BA-SOT），它明确地将边界知识通过扬声器变化检测任务和边界约束损失的解码器。我们还介绍了一个两阶段的连接主义时间分类（CTC）的战略，采用令牌级SOT CTC恢复时间上下文信息。除了典型的字符错误率（CER），我们引入话语相关的字符错误率（UD-CER），以进一步衡量说话人变化预测的精度。与原始SOT相比，BA-SOT将CER/UD-CER降低了5.1%/14.0%，并且利用预训练的ASR模型进行BA-SOT模型初始化进一步将CER/UD-CER降低了8.4%/19.9%。摘要：The recently proposed serialized output training (SOT) simplifies multi-talker automatic speech recognition (ASR) by generating speaker transcriptions separated by a special token. However, frequent speaker changes can make speaker change prediction difficult. To address this, we propose boundary-aware serialized output training (BA-SOT), which explicitly incorporates boundary knowledge into the decoder via a speaker change detection task and boundary constraint loss. We also introduce a two-stage connectionist temporal classification (CTC) strategy that incorporates token-level SOT CTC to restore temporal context information. Besides typical character error rate (CER), we introduce utterance-dependent character error rate (UD-CER) to further measure the precision of speaker change prediction. Compared to original SOT, BA-SOT reduces CER/UD-CER by 5.1%/14.0%, and leveraging a pre-trained ASR model for BA-SOT model initialization further reduces CER/UD-CER by 8.4%/19.9%.

【13】 CALLS: Japanese Empathetic Dialogue Speech Corpus of Complaint Handling and Attentive Listening in Customer Center标题：来电：日语移情对话语料库投诉处理和客户中心专心倾听链接：https://arxiv.org/abs/2305.13713

作者：Yuki Saito,Eiji Iimori,Shinnosuke Takamichi,Kentaro Tachibana,Hiroshi Saruwatari机构：The University of Tokyo, Japan,LINE Corp., Japan.备注：5 pages, accepted for INTERSPEECH2023摘要：我们提出了呼叫，日本语音语料库，认为在客户中心的电话作为一个新的领域的移情口语对话。现有的研究语料库只涵盖了学校教师和学生之间的移情对话。为了扩展移情对话语音合成（EDSS）的应用范围，我们设计了我们的语料库，包括相同的女性扬声器的研究教师，作为一个操作员在模拟电话。我们描述了语料库的构建方法和分析记录的语音。我们还进行EDSS实验使用的呼叫和研究语料库，调查域差异的影响。结果表明，在训练过程中混合的两个语料库的合成语音的质量，由于不同程度的表现力的偏差的改善。我们的语料库项目页面是http://sython.org/Corpus/STUDIES-2。摘要：We present CALLS, a Japanese speech corpus that considers phone calls in a customer center as a new domain of empathetic spoken dialogue. The existing STUDIES corpus covers only empathetic dialogue between a teacher and student in a school. To extend the application range of empathetic dialogue speech synthesis (EDSS), we designed our corpus to include the same female speaker as the STUDIES teacher, acting as an operator in simulated phone calls. We describe a corpus construction methodology and analyze the recorded speech. We also conduct EDSS experiments using the CALLS and STUDIES corpora to investigate the effect of domain differences. The results show that mixing the two corpora during training causes biased improvements in the quality of synthetic speech due to the different degrees of expressiveness. Our project page of the corpus is http://sython.org/Corpus/STUDIES-2.

【14】 TO-Rawnet: Improving RawNet with TCN and Orthogonal Regularization for Fake Audio Detection标题：TO-Rawnet：利用TCN和正交正则化改进Rawnet进行伪音频检测链接：https://arxiv.org/abs/2305.13701

作者：Chenglong Wang,Jiangyan Yi,Jianhua Tao,Chuyuan Zhang,Shuai Zhang,Ruibo Fu,Xun Chen机构：University of Science and Technology of China, Hefei, China, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijng, China备注：Interspeech2023摘要：当前的虚假音频检测依赖于手工制作的特征，这些特征在提取期间丢失信息。为了克服这一点，最近的研究使用从原始音频信号的直接特征提取。例如，RawNet是端到端虚假音频检测的代表作品之一。然而，RawNet上的现有工作在训练期间没有优化Sinc-conv的参数，这限制了其性能。在本文中，我们提出将正交卷积纳入RawNet，这在优化Sinc-conv的参数时降低了滤波器之间的相关性，从而提高了可分辨性。此外，我们引入时间卷积网络（TCN）来捕获语音信号中的长期依赖关系。在ASVspoof 2019上的实验表明，与RawNet相比，我们的TO-RawNet系统在逻辑访问场景下可以相对降低EER 66.09\%，证明了其在检测虚假音频攻击方面的有效性。摘要：Current fake audio detection relies on hand-crafted features, which lose information during extraction. To overcome this, recent studies use direct feature extraction from raw audio signals. For example, RawNet is one of the representative works in end-to-end fake audio detection. However, existing work on RawNet does not optimize the parameters of the Sinc-conv during training, which limited its performance. In this paper, we propose to incorporate orthogonal convolution into RawNet, which reduces the correlation between filters when optimizing the parameters of Sinc-conv, thus improving discriminability. Additionally, we introduce temporal convolutional networks (TCN) to capture long-term dependencies in speech signals. Experiments on the ASVspoof 2019 show that the Our TO-RawNet system can relatively reduce EER by 66.09\% on logical access scenario compared with the RawNet, demonstrating its effectiveness in detecting fake audio attacks.

【15】 Detection of Cross-Dataset Fake Audio Based on Prosodic and Pronunciation Features标题：基于韵律和语音特征的跨数据集伪音频检测链接：https://arxiv.org/abs/2305.13700

作者：Chenglong Wang,Jiangyan Yi,Jianhua Tao,Chuyuan Zhang,Shuai Zhang,Xun Chen机构：University of Science and Technology of China, Hefei, China, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijng, China备注：Interspeech2023摘要：现有的虚假音频检测系统在域内测试中表现良好，但在域外测试中仍然面临许多挑战。这是由于训练和测试数据之间的不匹配，以及从有限的视图中提取的特征的泛化能力差。为了解决这个问题，我们提出了多视图功能的假音频检测，其目的是从韵律，发音和wav2vec维度捕获更广义的功能。具体地，基于大量语音数据从预先训练的模型中提取音素持续时间特征。对于发音特征，首先训练基于Conformer的音素识别模型，保持声学编码器部分作为深度嵌入的特征提取器。此外，韵律和发音特征与wav2vec特征融合的基础上的注意力机制，以提高假音频检测模型的泛化。结果表明，该方法实现了显着的性能增益在几个跨数据集的实验。摘要：Existing fake audio detection systems perform well in in-domain testing, but still face many challenges in out-of-domain testing. This is due to the mismatch between the training and test data, as well as the poor generalizability of features extracted from limited views. To address this, we propose multi-view features for fake audio detection, which aim to capture more generalized features from prosodic, pronunciation, and wav2vec dimensions. Specifically, the phoneme duration features are extracted from a pre-trained model based on a large amount of speech data. For the pronunciation features, a Conformer-based phoneme recognition model is first trained, keeping the acoustic encoder part as a deeply embedded feature extractor. Furthermore, the prosodic and pronunciation features are fused with wav2vec features based on an attention mechanism to improve the generalization of fake audio detection models. Results show that the proposed approach achieves significant performance gains in several cross-dataset experiments.

【16】 FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models标题：FluentSpeech：基于上下文感知扩散模型的卡顿语音自动编辑链接：https://arxiv.org/abs/2305.13612

作者：Ziyue Jiang,Qian Yang,Jialong Zuo,Zhenhui Ye,Rongjie Huang,Yi Ren,Zhou Zhao机构：Zhejiang University, Bytedance AI Lab备注：Accepted by ACL 2023 (Findings)摘要：口吃消除是语音编辑领域中的一个基本场景。然而，当语音记录包含口吃时，现有的基于文本的语音编辑方法仍然遭受：1）编辑语音中的过平滑问题; 2）由于断续引入的噪声而缺乏鲁棒性; 3）为了去除断续，需要用户手动确定编辑区域。为了解决口吃消除的挑战，我们提出了FluentSpeech，一个面向口吃的自动语音编辑模型。具体而言，1）我们提出了一个上下文感知的扩散模型，在上下文特征的指导下迭代地细化修改后的Mel谱图; 2）引入了口吃预测器模块，将口吃信息注入到隐藏序列中; 3）我们还提出了一个面向口吃的自动语音编辑（SASE）数据集，该数据集包含具有时间对齐的口吃标签的自发语音记录，以训练自动口吃定位模型。在VCTK和LibriTTS数据集上的实验结果表明，我们的模型在语音编辑方面达到了最先进的性能。在SASE数据集上的进一步实验表明，FluentSpeech在客观和主观指标上都能有效地提高口吃语音的流畅性。代码和音频示例可以在www.example.com上找到https://github.com/Zain-Jiang/Speech-Editing-Toolkit。摘要：Stutter removal is an essential scenario in the field of speech editing. However, when the speech recording contains stutters, the existing text-based speech editing approaches still suffer from: 1) the over-smoothing problem in the edited speech; 2) lack of robustness due to the noise introduced by stutter; 3) to remove the stutters, users are required to determine the edited region manually. To tackle the challenges in stutter removal, we propose FluentSpeech, a stutter-oriented automatic speech editing model. Specifically, 1) we propose a context-aware diffusion model that iteratively refines the modified mel-spectrogram with the guidance of context features; 2) we introduce a stutter predictor module to inject the stutter information into the hidden sequence; 3) we also propose a stutter-oriented automatic speech editing (SASE) dataset that contains spontaneous speech recordings with time-aligned stutter labels to train the automatic stutter localization model. Experimental results on VCTK and LibriTTS datasets demonstrate that our model achieves state-of-the-art performance on speech editing. Further experiments on our SASE dataset show that FluentSpeech can effectively improve the fluency of stuttering speech in terms of objective and subjective metrics. Code and audio samples can be found at https://github.com/Zain-Jiang/Speech-Editing-Toolkit.

【17】 Scaling Speech Technology to 1,000+ Languages标题：将语音技术扩展到1,000多种语言链接：https://arxiv.org/abs/2305.13516

作者：Vineel Pratap,Andros Tjandra,Bowen Shi,Paden Tomasello,Arun Babu,Sayani Kundu,Ali Elkahky,Zhaoheng Ni,Apoorv Vyas,Maryam Fazel-Zarandi,Alexei Baevski,Yossi Adi,Xiaohui Zhang,Wei-Ning Hsu,Alexis Conneau,Michael Auli机构：Meta AI, ♢ core team摘要：扩大语音技术的语言覆盖面有可能改善更多人获得信息的机会。然而，目前的语音技术仅限于大约100种语言，这是全世界使用的7,000多种语言中的一小部分。大规模多语言语音（MMS）项目将支持的语言数量增加了10- 40倍，具体取决于任务。主要成分是一个新的数据集，该数据集基于对公开宗教文本的阅读，并有效地利用了自我监督学习。我们构建了覆盖1，406种语言的预训练wav 2 vec 2.0模型，针对1，107种语言的单一多语言自动语音识别模型，针对相同数量语言的语音合成模型，以及针对4，017种语言的语言识别模型。实验表明，我们的多语种语音识别模型在FLEURS基准测试的54种语言上，在一小部分标记数据上进行训练时，Whisper的单词错误率减少了一半以上。摘要：Expanding the language coverage of speech technology has the potential to improve access to information for many more people. However, current speech technology is restricted to about one hundred languages which is a small fraction of the over 7,000 languages spoken around the world. The Massively Multilingual Speech (MMS) project increases the number of supported languages by 10-40x, depending on the task. The main ingredients are a new dataset based on readings of publicly available religious texts and effectively leveraging self-supervised learning. We built pre-trained wav2vec 2.0 models covering 1,406 languages, a single multilingual automatic speech recognition model for 1,107 languages, speech synthesis models for the same number of languages, as well as a language identification model for 4,017 languages. Experiments show that our multilingual speech recognition model more than halves the word error rate of Whisper on 54 languages of the FLEURS benchmark while being trained on a small fraction of the labeled data.

【18】 Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding标题：ChatGPT能检测到意图吗？评估用于口语理解的大型语言模型链接：https://arxiv.org/abs/2305.13512

作者：Mutian He,Philip N. Garner机构： Idiap Research Institute, Martigny, Switzerland, Ecole Polytechnique F´ed´erale de Lausanne, Switzerland备注：6 pages, 2 figures; Accepted by Interspeech 2023摘要：最近，大型预训练语言模型已经展示了强大的语言理解能力。这特别体现在他们通过提示对下游任务的zero-shot和上下文学习能力上。为了评估它们对口语理解（SLU）的影响，我们在多个基准上评估了几个这样的模型，如ChatGPT和不同大小的OPT。我们验证了最大模型所特有的紧急能力，因为它们可以达到接近监督模型的意图分类准确性，在各种语言上具有零或几个镜头，给出Oracle成绩单。相比之下，适合单个GPU的较小模型的结果远远落后。我们注意到，错误情况往往是由数据集的注释方案引起的; ChatGPT的回应仍然合理。然而，我们发现，该模型是在槽填充差，其性能是敏感的ASR错误，这些文本模型的应用SLU提出了严峻的挑战。摘要：Recently, large pretrained language models have demonstrated strong language understanding capabilities. This is particularly reflected in their zero-shot and in-context learning abilities on downstream tasks through prompting. To assess their impact on spoken language understanding (SLU), we evaluate several such models like ChatGPT and OPT of different sizes on multiple benchmarks. We verify the emergent ability unique to the largest models as they can reach intent classification accuracy close to that of supervised models with zero or few shots on various languages given oracle transcripts. By contrast, the results for smaller models fitting a single GPU fall far behind. We note that the error cases often arise from the annotation scheme of the dataset; responses from ChatGPT are still reasonable. We show, however, that the model is worse at slot filling, and its performance is sensitive to ASR errors, suggesting serious challenges for the application of those textual models on SLU.

【19】 Masked Modeling Duo for Speech: Specializing General-Purpose Audio Representation to Speech using Denoising Distillation标题：语音掩蔽建模二人组：使用去噪蒸馏将通用音频表示专业化到语音链接：https://arxiv.org/abs/2305.14079

作者：Daisuke Niizumi,Daiki Takeuchi,Yasunori Ohishi,Noboru Harada,Kunio Kashino机构：NTT Corporation, Japan备注：Interspeech 2023; 5 pages, 2 figures, 6 tables, Code: this https URL摘要：自监督学习通用音频表示在各种任务中表现出高性能。虽然它们可以通过微调针对应用程序进行优化，但是如果它们可以专门针对应用程序进行预训练，则可以预期甚至更高的性能。本文探讨的挑战和解决方案，在专门的通用音频表示为一个特定的应用程序，使用语音，一个要求很高的领域，作为一个例子。我们增强了Masked Modeling Duo（M2 D），一种通用模型，以缩小与最先进（SOTA）语音模型的性能差距。为此，我们提出了一个新的任务，去噪蒸馏，从细粒度聚类特征中学习，以及M2 D for Speech（M2 D-S），它联合学习去噪蒸馏任务和M2 D掩蔽预测任务。实验结果表明，M2 D-S在SUPERB基准测试中的表现与SOTA语音模型相当或优于SOTA语音模型，表明M2 D可以专注于要求苛刻的领域。我们的代码可在：https://github.com/nttcslab/m2d/tree/master/speech摘要：Self-supervised learning general-purpose audio representations have demonstrated high performance in a variety of tasks. Although they can be optimized for application by fine-tuning, even higher performance can be expected if they can be specialized to pre-train for an application. This paper explores the challenges and solutions in specializing general-purpose audio representations for a specific application using speech, a highly demanding field, as an example. We enhance Masked Modeling Duo (M2D), a general-purpose model, to close the performance gap with state-of-the-art (SOTA) speech models. To do so, we propose a new task, denoising distillation, to learn from fine-grained clustered features, and M2D for Speech (M2D-S), which jointly learns the denoising distillation task and M2D masked prediction task. Experimental results show that M2D-S performs comparably to or outperforms SOTA speech models on the SUPERB benchmark, demonstrating that M2D can specialize in a demanding field. Our code is available at: https://github.com/nttcslab/m2d/tree/master/speech

【20】 Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification标题：呼吸音分类的音频频谱变换Patch-Mix对比学习链接：https://arxiv.org/abs/2305.14032

作者：Sangmin Bae,June-Woo Kim,Won-Yang Cho,Hyerim Baek,Soyoun Son,Byungjo Lee,Changwan Ha,Kyongpil Tae,Sungnyun Kim,Se-Young Yun机构：KAIST AI, Department of AI, Kyungpook National University, SmartSound, Dongguk University, MODULABS备注：INTERSPEECH 2023, Code URL: this https URL摘要：呼吸音包含致命肺部疾病的早期诊断的关键信息。自2019冠状病毒病大流行以来，人们对基于电子听诊器的非接触式医疗护理越来越感兴趣。为此，开发了尖端的深度学习模型来诊断肺部疾病;然而，由于缺乏医疗数据，这项工作仍然具有挑战性。在这项研究中，我们证明了大规模视觉和音频数据集上的预训练模型可以推广到呼吸声分类任务。此外，我们引入了一个简单的补丁混合增强，它随机混合补丁之间的不同样本，与音频频谱图Transformer（AST）。我们进一步提出了一种新的和有效的补丁混合对比学习来区分潜在空间中的混合表示。我们的方法在ICBHI数据集上实现了最先进的性能，比之前的领先分数提高了4.08%。摘要：Respiratory sound contains crucial information for the early diagnosis of fatal lung diseases. Since the COVID-19 pandemic, there has been a growing interest in contact-free medical care based on electronic stethoscopes. To this end, cutting-edge deep learning models have been developed to diagnose lung diseases; however, it is still challenging due to the scarcity of medical data. In this study, we demonstrate that the pretrained model on large-scale visual and audio datasets can be generalized to the respiratory sound classification task. In addition, we introduce a straightforward Patch-Mix augmentation, which randomly mixes patches between different samples, with Audio Spectrogram Transformer (AST). We further propose a novel and effective Patch-Mix Contrastive Learning to distinguish the mixed representations in the latent space. Our method achieves state-of-the-art performance on the ICBHI dataset, outperforming the prior leading score by an improvement of 4.08%.

【21】 EfficientSpeech: An On-Device Text to Speech Model标题：EfficientSpeech：一种设备上的文语转换模型链接：https://arxiv.org/abs/2305.13905

作者：Rowel Atienza机构：Electrical and Electronics Engineering Institute and AI Graduate Program, University of the Philippines备注：To be presented at ICASSP 2023摘要：最先进的（SOTA）神经文本到语音（TTS）模型可以生成听起来自然的合成语音。这些模型的特点是大的内存占用和大量的操作，由于长期关注的语音质量与云推理。神经TTS模型通常不被设计为在资源受限且无互联网接入边缘设备上执行独立语音合成。在这项工作中，提出了一种高效的神经TTS称为EfficientSpeech，在ARM CPU上实时合成语音。EfficientSpeech使用一个浅的非自回归金字塔结构Transformer器，形成一个U网络。EfficientSpeech具有266 k个参数，仅消耗90 MFLOPS，或约为现代紧凑型模型（如Mixer-TTS）中计算大小和量的1%。EfficientSpeech在RPi 4上实现了104.3的平均mel生成实时因子。人工评估显示，与FastSpeech 2相比，音频质量仅略有下降。摘要：State of the art (SOTA) neural text to speech (TTS) models can generate natural-sounding synthetic voices. These models are characterized by large memory footprints and substantial number of operations due to the long-standing focus on speech quality with cloud inference in mind. Neural TTS models are generally not designed to perform standalone speech syntheses on resource-constrained and no Internet access edge devices. In this work, an efficient neural TTS called EfficientSpeech that synthesizes speech on an ARM CPU in real-time is proposed. EfficientSpeech uses a shallow non-autoregressive pyramid-structure transformer forming a U-Network. EfficientSpeech has 266k parameters and consumes 90 MFLOPS only or about 1% of the size and amount of computation in modern compact models such as Mixer-TTS. EfficientSpeech achieves an average mel generation real-time factor of 104.3 on an RPi4. Human evaluation shows only a slight degradation in audio quality as compared to FastSpeech2.

【22】 Multi-Stream Extension of Variational Bayesian HMM Clustering (MS-VBx) for Combined End-to-End and Vector Clustering-based Diarization标题：基于变分贝叶斯HMM聚类的多流扩展(MS-VBx)端到端和基于向量聚类的二值化链接：https://arxiv.org/abs/2305.13580

作者：Marc Delcroix,Naohiro Tawara,Mireia Diez,Federico Landini,Anna Silnova,Atsunori Ogawa,Tomohiro Nakatani,Lukas Burget,Shoko Araki机构： 2Brno University of Technology备注：Accepted at Interspeech 2023摘要：将端到端神经说话人日志化（EEND）与向量聚类（VC）（称为EEND-VC）相结合，已经获得了利用这两种方法的优势的兴趣。EEND-VC估计音频块内所有说话者的活动和说话者嵌入，并使用VC将这些活动与不同块的说话者身份相关联。EEND-VC因此生成多个嵌入流，一个用于块中的每个说话者。我们可以使用约束凝聚层次聚类（cAHC）对这些嵌入进行聚类，确保来自同一块的嵌入属于不同的聚类。本文介绍了一种替代的聚类方法，多流扩展的成功贝叶斯HMM聚类的x-向量（VBx），称为MS-VBx。在三个数据集上的实验表明，MS-VBx在日记化和说话人计数性能上优于cAHC。摘要：Combining end-to-end neural speaker diarization (EEND) with vector clustering (VC), known as EEND-VC, has gained interest for leveraging the strengths of both methods. EEND-VC estimates activities and speaker embeddings for all speakers within an audio chunk and uses VC to associate these activities with speaker identities across different chunks. EEND-VC generates thus multiple streams of embeddings, one for each speaker in a chunk. We can cluster these embeddings using constrained agglomerative hierarchical clustering (cAHC), ensuring embeddings from the same chunk belong to different clusters. This paper introduces an alternative clustering approach, a multi-stream extension of the successful Bayesian HMM clustering of x-vectors (VBx), called MS-VBx. Experiments on three datasets demonstrate that MS-VBx outperforms cAHC in diarization and speaker counting performance.

【23】 Modular Domain Adaptation for Conformer-Based Streaming ASR标题：基于整形器的流媒体ASR的模块化域自适应链接：https://arxiv.org/abs/2305.13408

作者：Qiujia Li,Bo Li,Dongseong Hwang,Tara N. Sainath,Pedro M. Mengibar机构：Google LLC备注：Accepted to Interspeech 2023摘要：来自不同领域的语音数据具有不同的声学和语言学特征。通常在来自所有域的数据的混合物上训练单个多域模型，例如用于语音识别的Conformer换能器。然而，更改一个域中的数据或添加新域将需要重新训练多域模型。为此，我们提出了一个称为模块化域自适应（MDA）的框架，该框架使单个模型能够处理多域数据，同时保持所有参数特定于域，即，每个参数仅由来自一个域的数据训练。实验结果表明，在一个流的构象传感器只训练视频字幕数据，基于MDA的模型可以达到类似的性能，如语音搜索和听写的其他领域的多域模型，通过添加每域适配器和每域前馈网络的构象编码器。摘要：Speech data from different domains has distinct acoustic and linguistic characteristics. It is common to train a single multidomain model such as a Conformer transducer for speech recognition on a mixture of data from all domains. However, changing data in one domain or adding a new domain would require the multidomain model to be retrained. To this end, we propose a framework called modular domain adaptation (MDA) that enables a single model to process multidomain data while keeping all parameters domain-specific, i.e., each parameter is only trained by data from one domain. On a streaming Conformer transducer trained only on video caption data, experimental results show that an MDA-based model can reach similar performance as the multidomain model on other domains such as voice search and dictation by adding per-domain adapters and per-domain feed-forward networks in the Conformer encoder.

【24】 Conditional Online Learning for Keyword Spotting标题：关键词识别的条件性在线学习链接：https://arxiv.org/abs/2305.13332

作者：Michel Meneses,Bruno Iwami机构：SiDi, Brazil摘要：关键词识别的现代方法依赖于在具有i.i.d.的大型静态数据集上训练深度神经网络。分布然而，由此产生的模型往往表现不佳时，在现实生活中的应用程序中不断变化的数据制度。这项工作研究了一种简单但有效的在线持续学习方法，该方法在新数据可用时通过SGD更新设备上的关键字识别器。与以前的研究相反，这项工作的重点是学习相同的KWS任务，它涵盖了大多数商业应用程序。在不同场景下的动态音频流实验中，该方法将预训练的小足迹模型的性能提高了34%。此外，实验表明，相比一个天真的在线学习实现，条件模型更新的基础上，其性能在一个小的坚持从训练分布集减轻灾难性的遗忘。摘要：Modern approaches for keyword spotting rely on training deep neural networks on large static datasets with i.i.d. distributions. However, the resulting models tend to underperform when presented with changing data regimes in real-life applications. This work investigates a simple but effective online continual learning method that updates a keyword spotter on-device via SGD as new data becomes available. Contrary to previous research, this work focuses on learning the same KWS task, which covers most commercial applications. During experiments with dynamic audio streams in different scenarios, that method improves the performance of a pre-trained small-footprint model by 34%. Moreover, experiments demonstrate that, compared to a naive online learning implementation, conditional model updates based on its performance in a small hold-out set drawn from the training distribution mitigate catastrophic forgetting.

【25】 Unsupervised ASR via Cross-Lingual Pseudo-Labeling标题：基于跨语言伪标记法的无监督ASR链接：https://arxiv.org/abs/2305.13330

作者：Tatiana Likhomanenko,Loren Lugosch,Ronan Collobert机构：Apple备注：under review摘要：最近的工作表明，可以只使用未配对的音频和文本来训练自动语音识别（ASR）系统。现有的无监督ASR方法假设没有标记数据可以用于训练。我们认为，即使一个人没有任何标记的音频为一种给定的语言，有$\textit{always}$标记的数据可用于其他语言。我们表明，它是可以使用字符级的声学模型（AM）从其他语言引导一个$\textit{无监督}$ AM在一个新的语言。这里，“无监督”意味着没有标记的音频可用于$\textit{target}$语言。我们的方法基于两个关键要素：（i）使用一些$\textit{other}$语言AM生成$\textit{target}$语言的伪标签（PL），以及（ii）使用$\textit{target language model}$约束这些PL。我们的方法在共同的声音上是有效的：例如，将英语AM转换为斯瓦希里语可实现18%的WER。它在LJSpeech上的绝对WER也比基于字符的wav 2 vec-U 2.0高出15%，使用800 h的标记德语数据，而不是60 k小时的未标记英语数据。摘要：Recent work has shown that it is possible to train an $\textit{unsupervised}$ automatic speech recognition (ASR) system using only unpaired audio and text. Existing unsupervised ASR methods assume that no labeled data can be used for training. We argue that even if one does not have any labeled audio for a given language, there is $\textit{always}$ labeled data available for other languages. We show that it is possible to use character-level acoustic models (AMs) from other languages to bootstrap an $\textit{unsupervised}$ AM in a new language. Here, "unsupervised" means no labeled audio is available for the $\textit{target}$ language. Our approach is based on two key ingredients: (i) generating pseudo-labels (PLs) of the $\textit{target}$ language using some $\textit{other}$ language AM and (ii) constraining these PLs with a $\textit{target language model}$. Our approach is effective on Common Voice: e.g. transfer of English AM to Swahili achieves 18% WER. It also outperforms character-based wav2vec-U 2.0 by 15% absolute WER on LJSpeech with 800h of labeled German data instead of 60k hours of unlabeled English data.

eess.AS音频处理

【1】 Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning标题：基于度量学习改善视觉语音识别中正常语音与无声语音的差距链接：https://arxiv.org/abs/2305.14203

作者：Sara Kashiwagi,Keitaro Tanaka,Qi Feng,Shigeo Morishima机构：Waseda University, Japan, Waseda Research Institute for Science and Engineering, Japan备注：Accepted by INTERSPEECH 2023摘要：本文提出了一种新的度量学习方法，以解决视觉语音识别（VSR）的正常和无声的语音之间的性能差距。两者之间的嘴唇运动的差异对现有的VSR模型提出了挑战，当应用于无声语音时，其表现出降低的准确性。为了解决这个问题，并解决沉默语音训练数据的稀缺性，我们建议利用正常和沉默语音之间的共享文字内容，并提出了一种基于视位的度量学习方法。具体来说，我们的目标是映射输入的两个语音类型彼此接近的潜在空间中，如果他们有类似的视位表示。通过最小化Kullback-Leibler散度的预测视位概率分布之间和两种语音类型内，我们的模型有效地学习和预测视位身份。我们的评估表明，我们的方法提高了无声VSR的准确性，即使在有限的训练数据。摘要：This paper presents a novel metric learning approach to address the performance gap between normal and silent speech in visual speech recognition (VSR). The difference in lip movements between the two poses a challenge for existing VSR models, which exhibit degraded accuracy when applied to silent speech. To solve this issue and tackle the scarcity of training data for silent speech, we propose to leverage the shared literal content between normal and silent speech and present a metric learning approach based on visemes. Specifically, we aim to map the input of two speech types close to each other in a latent space if they have similar viseme representations. By minimizing the Kullback-Leibler divergence of the predicted viseme probability distributions between and within the two speech types, our model effectively learns and predicts viseme identities. Our evaluation demonstrates that our method improves the accuracy of silent VSR, even when limited training data is available.

【2】 Understanding Spoken Language Development of Children with ASD Using Pre-trained Speech Embeddings标题：利用预先训练的语音嵌入了解自闭症儿童的口语发展链接：https://arxiv.org/abs/2305.14117

作者：Anfeng Xu,Rajat Hebbar,Rimita Lahiri,Tiantian Feng,Lindsay Butler,Lue Shen,Helen Tager-Flusberg,Shrikanth Narayanan机构：University of Southern California, Los Angeles, CA, USA, Boston University, Boston, MA, USA备注：Accepted to Interspeech 2023, 5 pages摘要：语音处理技术可用于分析自闭症谱系障碍（ASD）儿童的语音和语言发展，这些儿童在获得这些技能时往往会出现变化和延迟。早期识别和干预是至关重要的，但传统的评估方法，如护理报告是不够的必要的行为表型。自然语言样本（NLS）分析作为一种有前途的补充而受到关注。研究人员已经开发了ASD儿童口语能力的基准，可通过NLS分析获得。本文提出了语音处理技术的应用程序，支持自动评估儿童的口语发展之间的儿童和成人的语音和非言语发声NLS之间的分类，分别为82.6%和67.8%的F1宏分数，强调了潜在的准确和可扩展的工具，ASD的研究和临床使用。摘要：Speech processing techniques are useful for analyzing speech and language development in children with Autism Spectrum Disorder (ASD), who are often varied and delayed in acquiring these skills. Early identification and intervention are crucial, but traditional assessment methodologies such as caregiver reports are not adequate for the requisite behavioral phenotyping. Natural Language Sample (NLS) analysis has gained attention as a promising complement. Researchers have developed benchmarks for spoken language capabilities in children with ASD, obtainable through the analysis of NLS. This paper proposes applications of speech processing technologies in support of automated assessment of children's spoken language development by classification between child and adult speech and between speech and nonverbal vocalization in NLS, with respective F1 macro scores of 82.6% and 67.8%, underscoring the potential for accurate and scalable tools for ASD research and clinical use.

【3】 Masked Modeling Duo for Speech: Specializing General-Purpose Audio Representation to Speech using Denoising Distillation标题：语音掩蔽建模二人组：使用去噪蒸馏将通用音频表示专业化到语音链接：https://arxiv.org/abs/2305.14079

【4】 Patch-Mix Contrastive Learning with Audio Spectrogram Transformer on Respiratory Sound Classification标题：呼吸音分类的音频频谱变换Patch-Mix对比学习链接：https://arxiv.org/abs/2305.14032

【5】 Eeg2vec: Self-Supervised Electroencephalographic Representation Learning标题：Eeg2vec：自监督脑电表征学习链接：https://arxiv.org/abs/2305.13957

作者：Qiushi Zhu,Xiaoying Zhao,Jie Zhang,Yu Gu,Chao Weng,Yuchen Hu备注：5 pages摘要：最近，已经做出了许多努力来探索大脑如何使用脑电图（EEG）信号处理语音，其中基于深度学习的方法被证明适用于该领域。为了从EEG信号解码语音信号，通常以监督的方式使用线性网络、卷积神经网络（CNN）和长短期记忆网络。记录EEG语音标记数据是相当耗时和费力的，而未标记的EEG数据是丰富的。自监督方法是否有助于学习EEG表征以提高EEG听觉相关任务的性能尚未得到很好的探索。在这项工作中，我们首先提出了一个基于对比损失和重建损失的自监督模型来学习EEG表示，然后使用获得的预训练模型作为下游任务的特征提取器。其次，对于两个考虑的下游任务，我们使用CNN和Transformer网络分别学习局部特征和全局特征。最后，将来自其他通道的EEG数据混合到所选择的EEG数据中以用于增强。在ICASSP 2023听觉脑电挑战赛的脑电匹配-失配和脑电回归任务上验证了该方法的有效性。摘要：Recently, many efforts have been made to explore how the brain processes speech using electroencephalographic (EEG) signals, where deep learning-based approaches were shown to be applicable in this field. In order to decode speech signals from EEG signals, linear networks, convolutional neural networks (CNN) and long short-term memory networks are often used in a supervised manner. Recording EEG-speech labeled data is rather time-consuming and laborious, while unlabeled EEG data is abundantly available. Whether self-supervised methods are helpful to learn EEG representation to boost the performance of EEG auditory-related tasks has not been well explored. In this work, we first propose a self-supervised model based on contrastive loss and reconstruction loss to learn EEG representations, and then use the obtained pre-trained model as a feature extractor for downstream tasks. Second, for two considered downstream tasks, we use CNNs and Transformer networks to learn local features and global features, respectively. Finally, the EEG data from other channels are mixed into the chosen EEG data for augmentation. The effectiveness of our method is verified on the EEG match-mismatch and EEG regression tasks of the ICASSP2023 Auditory EEG Challenge.

【6】 EfficientSpeech: An On-Device Text to Speech Model标题：EfficientSpeech：一种设备上的文语转换模型链接：https://arxiv.org/abs/2305.13905

【7】 Sequence-Level Knowledge Distillation for Class-Incremental End-to-End Spoken Language Understanding标题：面向类增量式端到端口语理解的序列级知识提取链接：https://arxiv.org/abs/2305.13899

作者：Umberto Cappellazzo,Muqiao Yang,Daniele Falavigna,Alessio Brutti机构：University of Trento, Trento, Italy, Carnegie Mellon University, Pittsburgh, PA, USA, Fondazione Bruno Kessler, Trento, Italy备注：Accepted at INTERSPEECH 2023. Code available at this https URL摘要：连续学习新概念的能力是现代神经网络的一个主要弱点，这阻碍了它们在非平稳环境中的使用。他们倾向于适应当前的数据分布，以损害过去获得的知识，导致灾难性的遗忘问题。在这项工作中，我们解决的问题，口语理解适用于一个持续的学习环境。我们首先为SLURP数据集定义一个类增量场景。然后，我们提出了三种知识蒸馏（KD）方法来减轻序列到序列Transformer模型的遗忘：第一种KD方法应用于编码器输出（音频KD），另外两种方法作用于解码器输出，或者直接作用于令牌级（令牌KD），或者作用于序列级（序列KD）分布。我们表明，seq-KD大大提高了所有的性能指标，其与音频KD的组合进一步降低了平均WER，并提高了实体预测指标。摘要：The ability to learn new concepts sequentially is a major weakness for modern neural networks, which hinders their use in non-stationary environments. Their propensity to fit the current data distribution to the detriment of the past acquired knowledge leads to the catastrophic forgetting issue. In this work we tackle the problem of Spoken Language Understanding applied to a continual learning setting. We first define a class-incremental scenario for the SLURP dataset. Then, we propose three knowledge distillation (KD) approaches to mitigate forgetting for a sequence-to-sequence transformer model: the first KD method is applied to the encoder output (audio-KD), and the other two work on the decoder output, either directly on the token-level (tok-KD) or on the sequence-level (seq-KD) distributions. We show that the seq-KD substantially improves all the performance metrics, and its combination with the audio-KD further decreases the average WER and enhances the entity prediction metric.

【8】 MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra标题：MP-SENET：一种幅相谱并行去噪的语音增强模型链接：https://arxiv.org/abs/2305.13686

作者：Ye-Xin Lu,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, P. R. China备注：Accepted by Interspeech 2023摘要：本文提出了MP-SENet，一种新的语音增强网络，直接去噪幅度和相位谱并行。建议MP-SENet采用编解码器架构，其中编码器和解码器由卷积增强Transformers桥接。编码器的目的是从输入的噪声幅度和相位谱的时间-频率表示进行编码。该解码器由并行幅度掩模解码器和相位解码器组成，分别采用可学习的sigmoid激活和并行相位估计结构，直接恢复干净的幅度谱和干净的相位谱。定义在幅度谱、相位谱、短时复谱和时域波形上的多级损失被用于联合训练MP-SENet模型。实验结果表明，我们提出的MP-SENet实现了PESQ为3.50的公共VoiceBank+DEMAND数据集，并优于现有的先进的语音增强方法。摘要：This paper proposes MP-SENet, a novel Speech Enhancement Network which directly denoises Magnitude and Phase spectra in parallel. The proposed MP-SENet adopts a codec architecture in which the encoder and decoder are bridged by convolution-augmented transformers. The encoder aims to encode time-frequency representations from the input noisy magnitude and phase spectra. The decoder is composed of parallel magnitude mask decoder and phase decoder, directly recovering clean magnitude spectra and clean-wrapped phase spectra by incorporating learnable sigmoid activation and parallel phase estimation architecture, respectively. Multi-level losses defined on magnitude spectra, phase spectra, short-time complex spectra, and time-domain waveforms are used to train the MP-SENet model jointly. Experimental results show that our proposed MP-SENet achieves a PESQ of 3.50 on the public VoiceBank+DEMAND dataset and outperforms existing advanced speech enhancement methods.

【9】 TranUSR: Phoneme-to-word Transcoder Based Unified Speech Representation Learning for Cross-lingual Speech Recognition标题：TransUSR：基于音词转码的跨语言语音识别统一语音表示学习链接：https://arxiv.org/abs/2305.13629

作者：Hongfei Xue,Qijie Shao,Peikun Chen,Pengcheng Guo,Lei Xie,Jie Liu机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Huawei Cloud备注：5 pages, 3 figures. Accepted by INTERSPEECH 2023摘要：UniSpeech通过使用多任务自监督学习将潜在表示与音素单元显式对齐，在跨语言自动语音识别（ASR）方面取得了卓越的性能。虽然学习的表示从高资源到低资源语言转移良好，但在下游ASR中直接从这些语音表示预测单词具有挑战性。在本文中，我们提出了TranUSR，一个两阶段模型，包括一个预先训练的UniData2vec和一个音素到单词的转码器。与UniSpeech不同，UniData2vec将量化的离散表示替换为来自教师模型的连续和上下文表示，用于语音感知预训练。然后，转码器学习在额外文本的帮助下将音素翻译成单词，从而实现直接单词生成。在Common Voice上的实验表明，UniData2vec与UniSpeech相比，PER降低了5.3%，而Transcoder与字素微调相比，WER降低了14.4%。摘要：UniSpeech has achieved superior performance in cross-lingual automatic speech recognition (ASR) by explicitly aligning latent representations to phoneme units using multi-task self-supervised learning. While the learned representations transfer well from high-resource to low-resource languages, predicting words directly from these phonetic representations in downstream ASR is challenging. In this paper, we propose TranUSR, a two-stage model comprising a pre-trained UniData2vec and a phoneme-to-word Transcoder. Different from UniSpeech, UniData2vec replaces the quantized discrete representations with continuous and contextual representations from a teacher model for phonetically-aware pre-training. Then, Transcoder learns to translate phonemes to words with the aid of extra texts, enabling direct word generation. Experiments on Common Voice show that UniData2vec reduces PER by 5.3\% compared to UniSpeech, while Transcoder yields a 14.4\% WER reduction compared to grapheme fine-tuning.

【10】 Multimodal sensor fusion for real-time location-dependent defect detection in laser-directed energy deposition标题：激光定向能沉积中位置相关缺陷实时检测的多模式传感器融合链接：https://arxiv.org/abs/2305.13596

作者：Lequn Chen,Xiling Yao,Wenhe Feng,Youxiang Chew,Seung Ki Moon机构：Advanced Remanufacturing and, Technology Centre (ARTC), Nanyang Technological University, Singapore Institute of, Manufacturing Technology, (SIMTech), Singapore备注：8 pages, 10 figures. This paper has been accepted to be published in the proceedings of IDETC-CIE 2023摘要：实时缺陷检测在激光定向能量沉积（L-DED）增材制造（AM）中至关重要。传统的原位监测方法利用单个传感器（即，声学、视觉或热传感器）来捕获复杂的过程动态行为，这对于具有高精度和鲁棒性的缺陷检测是不够的。本文提出了一种新的多模态传感器融合方法的实时位置相关的缺陷检测在机器人L-DED过程。多模态融合源包括捕获激光-材料相互作用声音的麦克风传感器和捕获同轴熔池图像的可见光谱CCD相机。提出了一种混合卷积神经网络（CNN）融合声音和视觉数据。本研究的关键新颖性在于不再需要传统的手动特征提取程序，并且通过混合CNN模型直接融合原始熔池图像和声学信号，该模型在没有热传感模态的情况下实现了最高的缺陷预测精度（98.5%）。此外，与以前的基于区域的质量预测不同，所提出的混合CNN可以检测缺陷发生的开始。缺陷预测结果与原位获取的机器人工具中心点（TCP）数据同步并注册，这使得能够进行局部缺陷识别。所提出的多模态传感器融合方法提供了一个强大的解决方案，在原位缺陷检测。摘要：Real-time defect detection is crucial in laser-directed energy deposition (L-DED) additive manufacturing (AM). Traditional in-situ monitoring approach utilizes a single sensor (i.e., acoustic, visual, or thermal sensor) to capture the complex process dynamic behaviors, which is insufficient for defect detection with high accuracy and robustness. This paper proposes a novel multimodal sensor fusion method for real-time location-dependent defect detection in the robotic L-DED process. The multimodal fusion sources include a microphone sensor capturing the laser-material interaction sound and a visible spectrum CCD camera capturing the coaxial melt pool images. A hybrid convolutional neural network (CNN) is proposed to fuse acoustic and visual data. The key novelty in this study is that the traditional manual feature extraction procedures are no longer required, and the raw melt pool images and acoustic signals are fused directly by the hybrid CNN model, which achieved the highest defect prediction accuracy (98.5 %) without the thermal sensing modality. Moreover, unlike previous region-based quality prediction, the proposed hybrid CNN can detect the onset of defect occurrences. The defect prediction outcomes are synchronized and registered with in-situ acquired robot tool-center-point (TCP) data, which enables localized defect identification. The proposed multimodal sensor fusion method offers a robust solution for in-situ defect detection.

【11】 Multi-Stream Extension of Variational Bayesian HMM Clustering (MS-VBx) for Combined End-to-End and Vector Clustering-based Diarization标题：基于变分贝叶斯HMM聚类的多流扩展(MS-VBx)端到端和基于向量聚类的二值化链接：https://arxiv.org/abs/2305.13580

【12】 Modular Domain Adaptation for Conformer-Based Streaming ASR标题：基于整形器的流媒体ASR的模块化域自适应链接：https://arxiv.org/abs/2305.13408

【13】 Conditional Online Learning for Keyword Spotting标题：关键词识别的条件性在线学习链接：https://arxiv.org/abs/2305.13332

【14】 A New Benchmark of Aphasia Speech Recognition and Detection Based on E-Branchformer and Multi-task Learning标题：基于E-Branchformer和多任务学习的失语症语音识别与检测新基准链接：https://arxiv.org/abs/2305.13331

作者：Jiyang Tang,William Chen,Xuankai Chang,Shinji Watanabe,Brian MacWhinney机构：Language Technologies Institute, Carnegie Mellon University, USA, Department of Psychology, Carnegie Mellon University, USA备注：Accepted at INTERSPEECH 2023. Code: this https URL摘要：失语症是一种影响数百万患者说话能力的语言障碍。本文提出了一种新的失语症语音识别和检测任务的基准，使用最先进的语音识别技术与AphsiaBank数据集。具体来说，我们介绍了两个多任务学习方法的基础上的CTC/注意力架构，同时执行这两项任务。我们的系统实现了最先进的扬声器水平的检测精度（97.3%），和相对WER减少11%的中度失语症患者。此外，我们证明了我们的方法的普遍性，通过将其应用到另一个无序的语音数据库，DementiaBank皮特语料库。我们将公开我们的所有功能于一身的食谱和预训练模型，以促进可重复性。我们的标准化数据预处理管道和开源配方使研究人员能够直接比较结果，促进无序语音处理的进展。摘要：Aphasia is a language disorder that affects the speaking ability of millions of patients. This paper presents a new benchmark for Aphasia speech recognition and detection tasks using state-of-the-art speech recognition techniques with the AphsiaBank dataset. Specifically, we introduce two multi-task learning methods based on the CTC/Attention architecture to perform both tasks simultaneously. Our system achieves state-of-the-art speaker-level detection accuracy (97.3%), and a relative WER reduction of 11% for moderate Aphasia patients. In addition, we demonstrate the generalizability of our approach by applying it to another disordered speech database, the DementiaBank Pitt corpus. We will make our all-in-one recipes and pre-trained model publicly available to facilitate reproducibility. Our standardized data preprocessing pipeline and open-source recipes enable researchers to compare results directly, promoting progress in disordered speech processing.

【15】 Unsupervised ASR via Cross-Lingual Pseudo-Labeling标题：基于跨语言伪标记法的无监督ASR链接：https://arxiv.org/abs/2305.13330

【16】 QFA2SR: Query-Free Adversarial Transfer Attacks to Speaker Recognition Systems标题：QFA2SR：对说话人识别系统的无查询对抗性转移攻击链接：https://arxiv.org/abs/2305.14097

【17】 Disentangled Variational Autoencoder for Emotion Recognition in Conversations标题：一种用于会话中情感识别的解缠变分自动编码器链接：https://arxiv.org/abs/2305.14071

【18】 Rethinking Speech Recognition with A Multimodal Perspective via Acoustic and Semantic Cooperative Decoding标题：从声学和语义协同解码的多模式视角反思语音识别链接：https://arxiv.org/abs/2305.14049

【19】 Improving speech translation by fusing speech and text标题：通过语音和文本的融合提高语音翻译水平链接：https://arxiv.org/abs/2305.14042

【20】 Can Self-Supervised Neural Networks Pre-Trained on Human Speech distinguish Animal Callers?标题：对人类语音进行预训练的自我监督神经网络能区分动物呼叫者吗？链接：https://arxiv.org/abs/2305.14035

【21】 Happy or Evil Laughter? Analysing a Database of Natural Audio Samples标题：快乐还是邪恶的笑声？一个自然音频样本库的分析链接：https://arxiv.org/abs/2305.14023

【22】 ZET-Speech: Zero-shot adaptive Emotion-controllable Text-to-Speech Synthesis with Diffusion and Style-based Models标题：ZET-Speech：基于扩散和风格模型的Zero-Shot自适应情感可控文语合成链接：https://arxiv.org/abs/2305.13831

【23】 Personalized Predictive ASR for Latency Reduction in Voice Assistants标题：用于降低语音助理延迟的个性化预测ASR链接：https://arxiv.org/abs/2305.13794

作者：Andreas Schwarz,Di He,Maarten Van Segbroeck,Mohammed Hethnawi,Ariya Rastrow机构：Amazon Alexa, Germany, Amazon Alexa, USA备注：Accepted for Interspeech 2023摘要：语音助理中的流式自动语音识别（ASR）可以利用预取来部分隐藏响应生成的延迟。预取涉及将初步ASR假设传递到下游系统以便预取和高速缓存响应。如果端点检测之后的最终ASR假设与初步ASR假设匹配，则可以将缓存的响应递送给用户，从而节省延迟。在本文中，我们扩展了这个想法，通过引入预测自动语音识别，我们预测的完整的话语从部分观察到的话语，并预取的基础上预测的话语的响应。我们介绍了两种个性化的方法，并调查潜在的延迟收益从成功的预测和失败的预测成本增加之间的权衡。我们在内部语音助手数据集和公共SLURP数据集上评估了我们的方法。摘要：Streaming Automatic Speech Recognition (ASR) in voice assistants can utilize prefetching to partially hide the latency of response generation. Prefetching involves passing a preliminary ASR hypothesis to downstream systems in order to prefetch and cache a response. If the final ASR hypothesis after endpoint detection matches the preliminary one, the cached response can be delivered to the user, thus saving latency. In this paper, we extend this idea by introducing predictive automatic speech recognition, where we predict the full utterance from a partially observed utterance, and prefetch the response based on the predicted utterance. We introduce two personalization approaches and investigate the tradeoff between potential latency gains from successful predictions and the cost increase from failed predictions. We evaluate our methods on an internal voice assistant dataset as well as the public SLURP dataset.

【24】 ADD 2023: the Second Audio Deepfake Detection Challenge标题：ADD 2023：第二届音频深伪检测挑战赛链接：https://arxiv.org/abs/2305.13774

【25】 A study of audio mixing methods for piano transcription in violin-piano ensembles标题：小提琴-钢琴重奏中钢琴改编的混音方法研究链接：https://arxiv.org/abs/2305.13758

【26】 ChatGPT-EDSS: Empathetic Dialogue Speech Synthesis Trained from ChatGPT-derived Context Word Embeddings标题：ChatGPT-EDSS：基于ChatGPT的上下文词嵌入训练的移情对话语音合成链接：https://arxiv.org/abs/2305.13724

【27】 BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR标题：BA-SOT：多人ASR的边界感知串行化输出训练链接：https://arxiv.org/abs/2305.13716

【28】 CALLS: Japanese Empathetic Dialogue Speech Corpus of Complaint Handling and Attentive Listening in Customer Center标题：来电：日语移情对话语料库投诉处理和客户中心专心倾听链接：https://arxiv.org/abs/2305.13713

【29】 TO-Rawnet: Improving RawNet with TCN and Orthogonal Regularization for Fake Audio Detection标题：TO-Rawnet：利用TCN和正交正则化改进Rawnet进行伪音频检测链接：https://arxiv.org/abs/2305.13701

【30】 Detection of Cross-Dataset Fake Audio Based on Prosodic and Pronunciation Features标题：基于韵律和语音特征的跨数据集伪音频检测链接：https://arxiv.org/abs/2305.13700

【31】 Cross-lingual Knowledge Transfer and Iterative Pseudo-labeling for Low-Resource Speech Recognition with Transducers标题：基于换能器的低资源语音识别中的跨语言知识传递和迭代伪标注链接：https://arxiv.org/abs/2305.13652

作者：Jan Silovsky,Liuhui Deng,Arturo Argueta,Tresi Arvizo,Roger Hsiao,Sasha Kuznietsov,Yiu-Chang Lin,Xiaoqiang Xiao,Yuanyuan Zhang机构：Apple摘要：语音技术最近变得无处不在。然而，不同语言的准确性和体验差异很大，这使得该技术的包容性不一样。不同语言的数据可用性是影响准确性的关键因素之一，特别是在全神经端到端自动语音识别系统的训练中。跨语言知识转移和迭代伪标记是两种已被证明成功提高ASR系统准确性的技术，特别是对于低资源语言，如乌克兰语。我们的目标是训练一个基于全神经传感器的ASR系统，以取代没有手动注释训练数据的DNN-HMM混合系统。我们表明，使用混合系统产生的成绩单训练的换能器系统在字错误率方面实现了18%的减少。然而，使用相关语言的跨语言知识转移和迭代伪标记的组合，我们能够实现35%的错误率降低。摘要：Voice technology has become ubiquitous recently. However, the accuracy, and hence experience, in different languages varies significantly, which makes the technology not equally inclusive. The availability of data for different languages is one of the key factors affecting accuracy, especially in training of all-neural end-to-end automatic speech recognition systems. Cross-lingual knowledge transfer and iterative pseudo-labeling are two techniques that have been shown to be successful for improving the accuracy of ASR systems, in particular for low-resource languages, like Ukrainian. Our goal is to train an all-neural Transducer-based ASR system to replace a DNN-HMM hybrid system with no manually annotated training data. We show that the Transducer system trained using transcripts produced by the hybrid system achieves 18% reduction in terms of word error rate. However, using a combination of cross-lingual knowledge transfer from related languages and iterative pseudo-labeling, we are able to achieve 35% reduction of the error rate.

【32】 FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models标题：FluentSpeech：基于上下文感知扩散模型的卡顿语音自动编辑链接：https://arxiv.org/abs/2305.13612

【33】 Cross-Attention is Not Enough: Incongruity-Aware Multimodal Sentiment Analysis and Emotion Recognition标题：仅有交叉注意是不够的：不一致感知的多通道情绪分析和情绪识别链接：https://arxiv.org/abs/2305.13583

作者：Yaoting Wang,Yuanchao Li,Peter Bell,Catherine Lai机构：Institute for Language, Cognition and Computation, School of Informatics, University of Edinburgh备注：*Equal contribution摘要：融合多种形式的情感计算任务已被证明是有效的性能提高。然而，多模态融合的工作原理还没有得到很好的理解，它在现实世界中的使用通常会导致大的模型尺寸。在这项工作中，在情感和情绪分析，我们首先分析了如何显着的情感信息在一个通道可以影响其他跨通道的注意。我们发现，跨模态注意的存在，在潜在的层面上，跨模态注意。基于这一发现，我们提出了一个轻量级的模型，通过分层跨模态Transformer与模态门控（HCT-MG），它确定了一个主要的模态，根据其对目标任务的贡献，然后分层纳入辅助模态，以减轻模态间的不协调，减少信息冗余。在三个基准数据集上的实验评估：CMU-MOSI、CMU-MOSEI和IEMOCAP验证了我们的方法的有效性，表明它：1）通过取得有竞争力的结果而超越主要先前工作，并能够成功识别硬样本; 2）在潜在层面上减轻了情态间情感倾向不匹配时的情态间不协调; 3）将模型尺寸减小到小于1 M个参数，同时优于类似尺寸的现有模型。摘要：Fusing multiple modalities for affective computing tasks has proven effective for performance improvement. However, how multimodal fusion works is not well understood, and its use in the real world usually results in large model sizes. In this work, on sentiment and emotion analysis, we first analyze how the salient affective information in one modality can be affected by the other in crossmodal attention. We find that inter-modal incongruity exists at the latent level due to crossmodal attention. Based on this finding, we propose a lightweight model via Hierarchical Crossmodal Transformer with Modality Gating (HCT-MG), which determines a primary modality according to its contribution to the target task and then hierarchically incorporates auxiliary modalities to alleviate inter-modal incongruity and reduce information redundancy. The experimental evaluation on three benchmark datasets: CMU-MOSI, CMU-MOSEI, and IEMOCAP verifies the efficacy of our approach, showing that it: 1) outperforms major prior work by achieving competitive results and can successfully recognize hard samples; 2) mitigates the inter-modal incongruity at the latent level when modalities have mismatched affective tendencies; 3) reduces model size to less than 1M parameters while outperforming existing models of similar sizes.

【34】 Scaling Speech Technology to 1,000+ Languages标题：将语音技术扩展到1,000多种语言链接：https://arxiv.org/abs/2305.13516

【35】 Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding标题：ChatGPT能检测到意图吗？评估用于口语理解的大型语言模型链接：https://arxiv.org/abs/2305.13512

语音/音频处理学术速递[5.24]

评论 0