今日论文合集：cs.SD语音8篇，eess.AS音频处理12篇。【1】 Introducing STRAUSS: A flexible sonification Python package标题：介绍STRAUSS：一个灵活的发音Python包链接：https://arxiv.org/abs/2311.16847作者：James W. Trayford,Chris M. Harrison备注：6 pages, 4 figures, 28th International Conference on Auditory Display, see here for linked resources: this https URL摘要：我们介绍STRAUSS（使用声音合成进行分析的发音工具和资源），一个模块化的，自包含的和灵活的Python发音包，以自由和开源（FOSS）的能力运行。STRAUSS旨在成为一种灵活的工具，既适合科学数据探索和分析，也适合制作适合公共宣传和艺术背景的声音。我们解释了Strauss背后的动机，以及这些动机如何导致我们的设计选择。我们还描述了基本的代码结构和概念。然后，我们给出输出声化示例，具体地：（1）单变量数据的多个表示（即，单个数据系列）进行数据探索;（2）多变量数据如何映射到声音上，以帮助解释这些数据变量是如何相关的;（3）沉浸式虚拟现实的完整空间音频示例。我们总结，暗示一些未来的功能，因为施特劳斯的发展加速。摘要：We introduce STRAUSS (Sonification Tools and Resources for Analysis Using Sound Synthesis) a modular, self-contained and flexible Python sonification package, operating in a free and open source (FOSS) capacity. STRAUSS is intended to be a flexible tool suitable for both scientific data exploration and analysis as well as for producing sonifications that are suitable for public outreach and artistic contexts. We explain the motivations behind STRAUSS, and how these lead to our design choices. We also describe the basic code structure and concepts. We then present output sonification examples, specifically: (1) multiple representations of univariate data (i.e., single data series) for data exploration; (2) how multi-variate data can be mapped onto sound to help interpret how those data variables are related and; (3) a full spatial audio example for immersive Virtual Reality. We summarise, alluding to some of the future functionality as STRAUSS development accelerates.【2】 D4AM: A General Denoising Framework for Downstream Acoustic Models标题：D4AM：一种适用于下行声学模型的通用去噪框架链接：https://arxiv.org/abs/2311.16595作者：Chi-Chang Lee,Yu Tsao,Hsin-Min Wang,Chu-Song Chen摘要：在噪声环境中，声学模型的性能会显著下降。语音增强（SE）可以用作辅助自动语音识别（ASR）系统的前端策略。然而，SE方法的现有训练目标在整合语音-文本和噪声-干净配对数据以训练看不见的ASR系统方面并不完全有效。在这项研究中，我们提出了一个通用的去噪框架，D4 AM，各种下游声学模型。我们的框架根据特定的声学模型和相应的分类目标，用向后梯度微调SE模型。此外，我们的方法旨在考虑回归目标作为辅助损失，使SE模型推广到其他看不见的声学模型。为了联合训练具有回归和分类目标的SE单元，D4 AM使用调整方案来直接估计合适的加权系数，而不是经历具有额外训练成本的网格搜索过程。平差方案由梯度校正和回归目标加权两部分组成。实验结果表明，D4 AM可以一致和有效地提供各种看不见的声学模型的改进，并优于其他组合设置。具体来说，当在Google ASR API上使用SE训练期间完全看不到的真实噪声数据进行评估时，与直接输入噪声输入相比，D4 AM实现了24.65%的相对WER降低。据我们所知，这是第一个部署回归（去噪）和分类（ASR）目标的有效组合方案的工作，以获得适用于各种看不见的ASR系统的通用预处理器。我们的代码可在https://github.com/ChangLee0903/D4AM上获得。摘要：The performance of acoustic models degrades notably in noisy environments. Speech enhancement (SE) can be used as a front-end strategy to aid automatic speech recognition (ASR) systems. However, existing training objectives of SE methods are not fully effective at integrating speech-text and noisy-clean paired data for training toward unseen ASR systems. In this study, we propose a general denoising framework, D4AM, for various downstream acoustic models. Our framework fine-tunes the SE model with the backward gradient according to a specific acoustic model and the corresponding classification objective. In addition, our method aims to consider the regression objective as an auxiliary loss to make the SE model generalize to other unseen acoustic models. To jointly train an SE unit with regression and classification objectives, D4AM uses an adjustment scheme to directly estimate suitable weighting coefficients rather than undergoing a grid search process with additional training costs. The adjustment scheme consists of two parts: gradient calibration and regression objective weighting. The experimental results show that D4AM can consistently and effectively provide improvements to various unseen acoustic models and outperforms other combination setups. Specifically, when evaluated on the Google ASR API with real noisy data completely unseen during SE training, D4AM achieves a relative WER reduction of 24.65% compared with the direct feeding of noisy input. To our knowledge, this is the first work that deploys an effective combination scheme of regression (denoising) and classification (ASR) objectives to derive a general pre-processor applicable to various unseen ASR systems. Our code is available at https://github.com/ChangLee0903/D4AM.【3】 DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser标题：DiffusionTalker：语音驱动的3D人脸扩散器的个性化和加速链接：https://arxiv.org/abs/2311.16565作者：Peng Chen,Xiaobao Wei,Ming Lu,Yitong Zhu,Naiming Yao,Xingyu Xiao,Hui Chen摘要：语音驱动的三维人脸动画一直是学术界和工业界的热门课题。传统的方法主要集中在学习从语音到动画的确定性映射。最近的方法开始考虑语音驱动的3D人脸动画的非确定性事实，并采用扩散模型的任务。然而，个性化的面部动画和加速动画生成仍然是现有的基于扩散的方法的两个主要限制。为了解决上述限制，我们提出了DiffusionTalker，一种基于扩散的方法，利用对比学习来个性化3D面部动画和知识蒸馏来加速3D动画生成。具体来说，为了实现个性化，我们引入了一个可学习的谈话身份来聚合音频序列中的知识。所提出的身份嵌入提取定制的面部线索在不同的人在对比学习的方式。在推理过程中，用户可以根据输入音频获得个性化的面部动画，反映特定的谈话风格。使用具有数百步的训练扩散模型，我们将其提取为具有8步加速的轻量级模型。大量的实验表明，我们的方法优于国家的最先进的方法。代码将被释放。摘要：Speech-driven 3D facial animation has been an attractive task in both academia and industry. Traditional methods mostly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the non-deterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. However, personalizing facial animation and accelerating animation generation are still two major limitations of existing diffusion-based methods. To address the above limitations, we propose DiffusionTalker, a diffusion-based method that utilizes contrastive learning to personalize 3D facial animation and knowledge distillation to accelerate 3D animation generation. Specifically, to enable personalization, we introduce a learnable talking identity to aggregate knowledge in audio sequences. The proposed identity embeddings extract customized facial cues across different people in a contrastive learning manner. During inference, users can obtain personalized facial animation based on input audio, reflecting a specific talking style. With a trained diffusion model with hundreds of steps, we distill it into a lightweight model with 8 steps for acceleration. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released.【4】 StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models标题：StyleCap：基于语音和语言自我监督学习模型的语音自动字幕链接：https://arxiv.org/abs/2311.16509作者：Kazuki Yamauchi,Yusuke Ijima,Yuki Saito备注：Submitted to ICASSP 2024摘要：我们提出了StyleCap，一种方法来生成自然语言描述的讲话风格出现在讲话中。尽管大多数用于准/非语言信息识别的常规技术关注于类别分类或预定义标签的强度估计，但是它们不能以可解释的方式提供识别结果的推理。作为朝向用于从语音生成说话风格提示的端到端方法的第一步，即，在自动演讲式字幕方面，StyleCap使用语音和自然语言描述的配对数据来训练神经网络，该神经网络从语音表示向量预测输入到基于大语言模型（LLM）的文本解码器的前缀向量。我们探索一个合适的文本解码器和语音特征表示适合这个新的任务。实验结果表明，我们的StyleCap利用更丰富的LLM的文本解码器，语音自监督学习（SSL）功能和句子改写增强提高了准确性和多样性生成的说话风格的字幕。由我们的StyleCap生成的演讲式字幕的样本是公开的。摘要：We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of conventional techniques for para-/non-linguistic information recognition focus on the category classification or the intensity estimation of pre-defined labels, they cannot provide the reasoning of the recognition result in an interpretable manner. As a first step towards an end-to-end method for generating speaking-style prompts from speech, i.e., automatic speaking-style captioning, StyleCap uses paired data of speech and natural language descriptions to train neural networks that predict prefix vectors fed into a large language model (LLM)-based text decoder from a speech representation vector. We explore an appropriate text decoder and speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions. Samples of speaking-style captions generated by our StyleCap are publicly available.【5】 iMagLS: Interaural Level Difference with Magnitude Least-Squares Loss for Optimized First-Order Head-Related Transfer Function标题：IMAGLS：优化一阶头相关传递函数的幅度最小二乘损失耳间电平差链接：https://arxiv.org/abs/2311.16702作者：Or Berebi,Zamir Ben-Hur,David Lou Alon,Boaz Rafaely备注：3 pages, 2 figures, Forum Acusticum 2023摘要：用于基于耳机的收听的双耳再现是一个活跃的研究领域，这是由于其在诸如增强现实和虚拟现实（AR和VR）等不断发展的技术中的广泛使用。一方面，这些应用需要高质量的空间音频感知来保持沉浸感。另一方面，记录设备可能只有几个麦克风，导致低阶表示，例如一阶高保真度立体声（FOA）。然而，一阶高保真度立体声复制导致有限的外部化和空间分辨率。本文提出了一种新的头部相关传递函数（HRTF）预处理优化损失，并使用非线性规划最小化。新的方法，表示为iMagLS，涉及到一个耳间电平差（ILD）的误差项，现在广泛使用的MagLS优化损失的横向平面角度的介绍。结果表明，ILD误差可以大大减少，而HRTF的幅度误差保持类似的MagLS获得。这些结果可以证明有益于一阶高保真度立体声响复制的整体空间质量，而其他再现方法也可以从考虑这种修改的损失中受益。摘要：Binaural reproduction for headphone-based listening is an active research area due to its widespread use in evolving technologies such as augmented and virtual reality (AR and VR). On the one hand, these applications demand high quality spatial audio perception to preserve the sense of immersion. On the other hand, recording devices may only have a few microphones, leading to low-order representations such as first-order Ambisonics (FOA). However, first-order Ambisonics leads to limited externalization and spatial resolution. In this paper, a novel head-related transfer function (HRTF) preprocessing optimization loss is proposed, and is minimized using nonlinear programming. The new method, denoted iMagLS, involves the introduction of an interaural level difference (ILD) error term to the now widely used MagLS optimization loss for the lateral plane angles. Results indicate that the ILD error could be substantially reduced, while the HRTF magnitude error remains similar to that obtained with MagLS. These results could prove beneficial to the overall spatial quality of first-order Ambisonics, while other reproduction methods could also benefit from considering this modified loss.【6】 FAT-HuBERT: Front-end Adaptive Training of Hidden-unit BERT for Distortion-Invariant Robust Speech Recognition标题：Fat-Hubert：用于失真不变鲁棒语音识别的隐藏单元BERT的前端自适应训练链接：https://arxiv.org/abs/2311.17790
作者：Dongning Yang,Wei Wang,Yanmin Qian摘要：单声道语音增强（SE）技术的进步极大地改善了语音的感知质量。然而，将这些技术集成到自动语音识别（ASR）系统中并没有产生预期的性能增益，主要是由于在SE过程中引入了失真。在本文中，我们提出了一种新的方法称为FAT-HuBERT，它利用失真不变的自监督学习（SSL），以提高ASR的鲁棒性。为了解决由SE前端引入的失真，我们引入了分层融合模块，该模块结合了从观测到的噪声信号和增强信号中提取的特征。在训练过程中，SE前端是从模型池中随机选择的。我们评估了FAT-HuBERT在LibriSpeech生成的模拟噪声语音以及CHiME-4 1通道数据集的真实噪声语音上的性能。实验结果表明，显着的相对减少字错误率（WER）。摘要：Advancements in monaural speech enhancement (SE) techniques have greatly improved the perceptual quality of speech. However, integrating these techniques into automatic speech recognition (ASR) systems has not yielded the expected performance gains, primarily due to the introduction of distortions during the SE process. In this paper, we propose a novel approach called FAT-HuBERT, which leverages distortion-invariant self-supervised learning (SSL) to enhance the robustness of ASR. To address the distortions introduced by the SE frontends, we introduce layer-wise fusion modules that incorporate features extracted from both observed noisy signals and enhanced signals. During training, the SE frontend is randomly selected from a pool of models. We evaluate the performance of FAT-HuBERT on simulated noisy speech generated from LibriSpeech as well as real-world noisy speech from the CHiME-4 1-channel dataset. The experimental results demonstrate a significant relative reduction in word error rate (WER).【7】 End-to-end Joint Rich and Normalized ASR with a limited amount of rich training data标题：端到端联合丰富和标准化ASR，具有有限数量的丰富训练数据链接：https://arxiv.org/abs/2311.17741
作者：Can Cui,Imran Ahamad Sheikh,Mostafa Sadeghi,Emmanuel Vincent备注：Submitted to ICASSP 2024摘要：联合丰富和规范化的自动语音识别（ASR），产生transmittance与标点符号和大写，仍然是一个挑战。端到端（E2E）ASR模型提供了执行这种语音的联合转录的便利性和能力。训练这样的模型需要成对的语音和富文本数据，这并不广泛可用。在本文中，我们比较了两种不同的方法来训练一个无状态的基于传感器的E2E联合丰富和规范化的ASR系统，为流应用程序做好准备，丰富的标记数据量有限。第一种方法使用语言模型来生成规范化训练数据的伪富translation。第二种方法使用单个解码器，取决于输出的类型。第一种方法导致E2E丰富的ASR，其在域外数据上表现更好，错误相对减少高达9%。第二种方法证明了E2E联合丰富和归一化ASR系统的可行性，该系统使用低至5%的丰富训练数据，误差适度增加（绝对值为2.42%）。摘要：Joint rich and normalized automatic speech recognition (ASR), that produces transcriptions both with and without punctuation and capitalization, remains a challenge. End-to-end (E2E) ASR models offer both convenience and the ability to perform such joint transcription of speech. Training such models requires paired speech and rich text data, which is not widely available. In this paper, we compare two different approaches to train a stateless Transducer-based E2E joint rich and normalized ASR system, ready for streaming applications, with a limited amount of rich labeled data. The first approach uses a language model to generate pseudo-rich transcriptions of normalized training data. The second approach uses a single decoder conditioned on the type of the output. The first approach leads to E2E rich ASR which perform better on out-of-domain data, with up to 9% relative reduction in errors. The second approach demonstrates the feasibility of an E2E joint rich and normalized ASR system using as low as 5% rich training data with moderate (2.42% absolute) increase in errors.【8】 Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes标题：自动身份识别对视听Deepfakes的脆弱性链接：https://arxiv.org/abs/2311.17655
作者：Pavel Korshunov,Haolin Chen,Philip N. Garner,Sebastien Marcel备注：10 pages, 3 figures, 3 tables摘要：深度伪造检测的任务远未被语音或视觉研究人员解决。建立了几个公开可用的假合成视频和语音数据库，以帮助开发检测方法。然而，现有的数据库通常专注于视觉或语音模式，并没有提供证据证明他们的deepfake实际上可以冒充任何真实的人。在本文中，我们提出了第一个真实的深度伪造SWAN-DF视听数据库，其中嘴唇和语音同步良好，视频具有很高的视觉和音频质量。我们使用公开的SWAN数据集，使用DeepFaceLab的几种模型和混合技术来创建具有不同身份的真实视频的视听deepfake，用于面部交换和HiFiVC，DiffVC，YourTTS和FreeVC模型进行语音转换。从公开可用的语音数据集LibriTTS中，我们还使用几种最新的文本到语音方法创建了一个单独的音频deepfakes LibriTTS-DF数据库：YourTTS，Adaspeech和TorToiSe。我们展示了最先进的说话人识别系统，如ECAPA-TDNN为基础的模型SpeechBrain的脆弱性，合成的声音。同样，我们测试了基于MobileFaceNet架构的人脸识别系统，以测试我们的视觉deepfake的几个变体。漏洞评估表明，通过将现有的预训练的deepfake模型调整到特定的身份，人们可以在90%以上的时间内成功欺骗面部和说话人识别系统，并获得一个非常逼真的给定人的虚假视频。摘要：The task of deepfakes detection is far from being solved by speech or vision researchers. Several publicly available databases of fake synthetic video and speech were built to aid the development of detection methods. However, existing databases typically focus on visual or voice modalities and provide no proof that their deepfakes can in fact impersonate any real person. In this paper, we present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized and video have high visual and audio qualities. We took the publicly available SWAN dataset of real videos with different identities to create audio-visual deepfakes using several models from DeepFaceLab and blending techniques for face swapping and HiFiVC, DiffVC, YourTTS, and FreeVC models for voice conversion. From the publicly available speech dataset LibriTTS, we also created a separate database of only audio deepfakes LibriTTS-DF using several latest text to speech methods: YourTTS, Adaspeech, and TorToiSe. We demonstrate the vulnerability of a state of the art speaker recognition system, such as ECAPA-TDNN-based model from SpeechBrain, to the synthetic voices. Similarly, we tested face recognition system based on the MobileFaceNet architecture to several variants of our visual deepfakes. The vulnerability assessment show that by tuning the existing pretrained deepfake models to specific identities, one can successfully spoof the face and speaker recognition systems in more than 90% of the time and achieve a very realistic looking and sounding fake video of a given person.【1】 Study of speaker localization under dynamic and reverberant environments标题：动态混响环境下的说话人定位研究链接：https://arxiv.org/abs/2311.16927作者：Daniel A. Mitchell,Boaz Rafaely备注：None摘要：混响环境中说话人定位是音频信号处理中的一个基本问题。已经开发了许多解决方案来解决这个问题。然而，先前的算法通常假设麦克风阵列和声源都不移动的静止环境。随着可穿戴麦克风阵列的出现，声学场景随着移动源和阵列而变得动态。这就需要在动态环境中表现良好的算法。在这篇文章中，我们研究了在这样的环境中的说话人定位算法的性能。该研究基于最近发布的EasyCom语音数据集，该数据集使用眼镜上的可穿戴阵列在混响和嘈杂的环境中记录。虽然定位算法在静态环境中表现良好，但在EasyCom数据集上使用时，其性能大幅下降。本文提出了性能分析，并提出了改进的方法。摘要：Speaker localization in a reverberant environment is a fundamental problem in audio signal processing. Many solutions have been developed to tackle this problem. However, previous algorithms typically assume a stationary environment in which both the microphone array and the sound sources are not moving. With the emergence of wearable microphone arrays, acoustic scenes have become dynamic with moving sources and arrays. This calls for algorithms that perform well in dynamic environments. In this article, we study the performance of a speaker localization algorithm in such an environment. The study is based on the recently published EasyCom speech dataset recorded in reverberant and noisy environments using a wearable array on glasses. Although the localization algorithm performs well in static environments, its performance degraded substantially when used on the EasyCom dataset. The paper presents performance analysis and proposes methods for improvement.【2】 iMagLS: Interaural Level Difference with Magnitude Least-Squares Loss for Optimized First-Order Head-Related Transfer Function标题：IMAGLS：优化一阶头相关传递函数的幅度最小二乘损失耳间电平差链接：https://arxiv.org/abs/2311.16702作者：Or Berebi,Zamir Ben-Hur,David Lou Alon,Boaz Rafaely备注：3 pages, 2 figures, Forum Acusticum 2023摘要：用于基于耳机的收听的双耳再现是一个活跃的研究领域，这是由于其在诸如增强现实和虚拟现实（AR和VR）等不断发展的技术中的广泛使用。一方面，这些应用需要高质量的空间音频感知来保持沉浸感。另一方面，记录设备可能只有几个麦克风，导致低阶表示，例如一阶高保真度立体声（FOA）。然而，一阶高保真度立体声复制导致有限的外部化和空间分辨率。本文提出了一种新的头部相关传递函数（HRTF）预处理优化损失，并使用非线性规划最小化。新的方法，表示为iMagLS，涉及到一个耳间电平差（ILD）的误差项，现在广泛使用的MagLS优化损失的横向平面角度的介绍。结果表明，ILD误差可以大大减少，而HRTF的幅度误差保持类似的MagLS获得。这些结果可以证明有益于一阶高保真度立体声响复制的整体空间质量，而其他再现方法也可以从考虑这种修改的损失中受益。摘要：Binaural reproduction for headphone-based listening is an active research area due to its widespread use in evolving technologies such as augmented and virtual reality (AR and VR). On the one hand, these applications demand high quality spatial audio perception to preserve the sense of immersion. On the other hand, recording devices may only have a few microphones, leading to low-order representations such as first-order Ambisonics (FOA). However, first-order Ambisonics leads to limited externalization and spatial resolution. In this paper, a novel head-related transfer function (HRTF) preprocessing optimization loss is proposed, and is minimized using nonlinear programming. The new method, denoted iMagLS, involves the introduction of an interaural level difference (ILD) error term to the now widely used MagLS optimization loss for the lateral plane angles. Results indicate that the ILD error could be substantially reduced, while the HRTF magnitude error remains similar to that obtained with MagLS. These results could prove beneficial to the overall spatial quality of first-order Ambisonics, while other reproduction methods could also benefit from considering this modified loss.【3】 LC4SV: A Denoising Framework Learning to Compensate for Unseen Speaker Verification Models标题：LC4SV：一个用于补偿不可见说话人确认模型的去噪框架链接：https://arxiv.org/abs/2311.16604作者：Chi-Chang Lee,Hong-Wei Chen,Chu-Song Chen,Hsin-Min Wang,Tsung-Te Liu,Yu Tsao摘要：说话人确认（SV）模型在噪声环境中的性能会急剧下降。语音增强（SE）模块可以用作前端策略。然而，由于SE模型的预测信号中的伪影，现有的SE方法可能无法为下游SV系统带来性能改进。为了补偿伪影，我们提出了一个名为LC4SV的通用去噪框架，它可以作为各种未知下游SV模型的预处理器。在LC4SV中，我们采用基于学习的插值代理来自动生成增强信号与其噪声输入之间的适当系数，以提高SV在噪声环境中的性能。我们的实验结果表明，LC4SV一贯提高各种看不见的SV系统的性能。据我们所知，这项工作是第一次尝试开发一个基于学习的插值方案，旨在提高SV性能在嘈杂的环境中。摘要：The performance of speaker verification (SV) models may drop dramatically in noisy environments. A speech enhancement (SE) module can be used as a front-end strategy. However, existing SE methods may fail to bring performance improvements to downstream SV systems due to artifacts in the predicted signals of SE models. To compensate for artifacts, we propose a generic denoising framework named LC4SV, which can serve as a pre-processor for various unknown downstream SV models. In LC4SV, we employ a learning-based interpolation agent to automatically generate the appropriate coefficients between the enhanced signal and its noisy input to improve SV performance in noisy environments. Our experimental results demonstrate that LC4SV consistently improves the performance of various unseen SV systems. To the best of our knowledge, this work is the first attempt to develop a learning-based interpolation scheme aiming at improving SV performance in noisy environments.【4】 Introducing STRAUSS: A flexible sonification Python package标题：施特劳斯简介：一个灵活的发音Python包链接：https://arxiv.org/abs/2311.16847作者：James W. Trayford,Chris M. Harrison备注：6 pages, 4 figures, 28th International Conference on Auditory Display, see here for linked resources: this https URL摘要：我们介绍STRAUSS（使用声音合成进行分析的发音工具和资源），一个模块化的，自包含的和灵活的Python发音包，以自由和开源（FOSS）的能力运行。STRAUSS旨在成为一种灵活的工具，既适合科学数据探索和分析，也适合制作适合公共宣传和艺术背景的声音。我们解释了Strauss背后的动机，以及这些动机如何导致我们的设计选择。我们还描述了基本的代码结构和概念。然后，我们给出输出声化示例，具体地：（1）单变量数据的多个表示（即，单个数据系列）进行数据探索;（2）多变量数据如何映射到声音上，以帮助解释这些数据变量是如何相关的;（3）沉浸式虚拟现实的完整空间音频示例。我们总结，暗示一些未来的功能，因为施特劳斯的发展加速。摘要：We introduce STRAUSS (Sonification Tools and Resources for Analysis Using Sound Synthesis) a modular, self-contained and flexible Python sonification package, operating in a free and open source (FOSS) capacity. STRAUSS is intended to be a flexible tool suitable for both scientific data exploration and analysis as well as for producing sonifications that are suitable for public outreach and artistic contexts. We explain the motivations behind STRAUSS, and how these lead to our design choices. We also describe the basic code structure and concepts. We then present output sonification examples, specifically: (1) multiple representations of univariate data (i.e., single data series) for data exploration; (2) how multi-variate data can be mapped onto sound to help interpret how those data variables are related and; (3) a full spatial audio example for immersive Virtual Reality. We summarise, alluding to some of the future functionality as STRAUSS development accelerates.【5】 D4AM: A General Denoising Framework for Downstream Acoustic Models标题：D4AM：一种适用于下行声学模型的通用去噪框架链接：https://arxiv.org/abs/2311.16595作者：Chi-Chang Lee,Yu Tsao,Hsin-Min Wang,Chu-Song Chen摘要：在噪声环境中，声学模型的性能会显著下降。语音增强（SE）可以用作辅助自动语音识别（ASR）系统的前端策略。然而，SE方法的现有训练目标在整合语音-文本和噪声-干净配对数据以训练看不见的ASR系统方面并不完全有效。在这项研究中，我们提出了一个通用的去噪框架，D4 AM，各种下游声学模型。我们的框架根据特定的声学模型和相应的分类目标，用向后梯度微调SE模型。此外，我们的方法旨在考虑回归目标作为辅助损失，使SE模型推广到其他看不见的声学模型。为了联合训练具有回归和分类目标的SE单元，D4 AM使用调整方案来直接估计合适的加权系数，而不是经历具有额外训练成本的网格搜索过程。平差方案由梯度校正和回归目标加权两部分组成。实验结果表明，D4 AM可以一致和有效地提供各种看不见的声学模型的改进，并优于其他组合设置。具体来说，当在Google ASR API上使用SE训练期间完全看不到的真实噪声数据进行评估时，与直接输入噪声输入相比，D4 AM实现了24.65%的相对WER降低。据我们所知，这是第一个部署回归（去噪）和分类（ASR）目标的有效组合方案的工作，以获得适用于各种看不见的ASR系统的通用预处理器。我们的代码可在https://github.com/ChangLee0903/D4AM上获得。摘要：The performance of acoustic models degrades notably in noisy environments. Speech enhancement (SE) can be used as a front-end strategy to aid automatic speech recognition (ASR) systems. However, existing training objectives of SE methods are not fully effective at integrating speech-text and noisy-clean paired data for training toward unseen ASR systems. In this study, we propose a general denoising framework, D4AM, for various downstream acoustic models. Our framework fine-tunes the SE model with the backward gradient according to a specific acoustic model and the corresponding classification objective. In addition, our method aims to consider the regression objective as an auxiliary loss to make the SE model generalize to other unseen acoustic models. To jointly train an SE unit with regression and classification objectives, D4AM uses an adjustment scheme to directly estimate suitable weighting coefficients rather than undergoing a grid search process with additional training costs. The adjustment scheme consists of two parts: gradient calibration and regression objective weighting. The experimental results show that D4AM can consistently and effectively provide improvements to various unseen acoustic models and outperforms other combination setups. Specifically, when evaluated on the Google ASR API with real noisy data completely unseen during SE training, D4AM achieves a relative WER reduction of 24.65% compared with the direct feeding of noisy input. To our knowledge, this is the first work that deploys an effective combination scheme of regression (denoising) and classification (ASR) objectives to derive a general pre-processor applicable to various unseen ASR systems. Our code is available at https://github.com/ChangLee0903/D4AM.【6】 DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser标题：DiffusionTalker：语音驱动的3D人脸扩散器的个性化和加速链接：https://arxiv.org/abs/2311.16565作者：Peng Chen,Xiaobao Wei,Ming Lu,Yitong Zhu,Naiming Yao,Xingyu Xiao,Hui Chen摘要：语音驱动的三维人脸动画一直是学术界和工业界的热门课题。传统的方法主要集中在学习从语音到动画的确定性映射。最近的方法开始考虑语音驱动的3D人脸动画的非确定性事实，并采用扩散模型的任务。然而，个性化的面部动画和加速动画生成仍然是现有的基于扩散的方法的两个主要限制。为了解决上述限制，我们提出了DiffusionTalker，一种基于扩散的方法，利用对比学习来个性化3D面部动画和知识蒸馏来加速3D动画生成。具体来说，为了实现个性化，我们引入了一个可学习的谈话身份来聚合音频序列中的知识。所提出的身份嵌入提取定制的面部线索在不同的人在对比学习的方式。在推理过程中，用户可以根据输入音频获得个性化的面部动画，反映特定的谈话风格。使用具有数百步的训练扩散模型，我们将其提取为具有8步加速的轻量级模型。大量的实验表明，我们的方法优于国家的最先进的方法。代码将被释放。摘要：Speech-driven 3D facial animation has been an attractive task in both academia and industry. Traditional methods mostly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the non-deterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. However, personalizing facial animation and accelerating animation generation are still two major limitations of existing diffusion-based methods. To address the above limitations, we propose DiffusionTalker, a diffusion-based method that utilizes contrastive learning to personalize 3D facial animation and knowledge distillation to accelerate 3D animation generation. Specifically, to enable personalization, we introduce a learnable talking identity to aggregate knowledge in audio sequences. The proposed identity embeddings extract customized facial cues across different people in a contrastive learning manner. During inference, users can obtain personalized facial animation based on input audio, reflecting a specific talking style. With a trained diffusion model with hundreds of steps, we distill it into a lightweight model with 8 steps for acceleration. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released.【7】 StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Models标题：StyleCap：基于语音和语言自我监督学习模型的语音自动字幕链接：https://arxiv.org/abs/2311.16509作者：Kazuki Yamauchi,Yusuke Ijima,Yuki Saito备注：Submitted to ICASSP 2024摘要：我们提出了StyleCap，一种方法来生成自然语言描述的讲话风格出现在讲话中。尽管大多数用于准/非语言信息识别的常规技术关注于类别分类或预定义标签的强度估计，但是它们不能以可解释的方式提供识别结果的推理。作为朝向用于从语音生成说话风格提示的端到端方法的第一步，即，在自动演讲式字幕方面，StyleCap使用语音和自然语言描述的配对数据来训练神经网络，该神经网络从语音表示向量预测输入到基于大语言模型（LLM）的文本解码器的前缀向量。我们探索一个合适的文本解码器和语音特征表示适合这个新的任务。实验结果表明，我们的StyleCap利用更丰富的LLM的文本解码器，语音自监督学习（SSL）功能和句子改写增强提高了准确性和多样性生成的说话风格的字幕。由我们的StyleCap生成的演讲式字幕的样本是公开的。摘要：We propose StyleCap, a method to generate natural language descriptions of speaking styles appearing in speech. Although most of conventional techniques for para-/non-linguistic information recognition focus on the category classification or the intensity estimation of pre-defined labels, they cannot provide the reasoning of the recognition result in an interpretable manner. As a first step towards an end-to-end method for generating speaking-style prompts from speech, i.e., automatic speaking-style captioning, StyleCap uses paired data of speech and natural language descriptions to train neural networks that predict prefix vectors fed into a large language model (LLM)-based text decoder from a speech representation vector. We explore an appropriate text decoder and speech feature representation suitable for this new task. The experimental results demonstrate that our StyleCap leveraging richer LLMs for the text decoder, speech self-supervised learning (SSL) features, and sentence rephrasing augmentation improves the accuracy and diversity of generated speaking-style captions. Samples of speaking-style captions generated by our StyleCap are publicly available.【8】 Adapting OpenAI's Whisper for Speech Recognition on Code-Switch Mandarin-English SEAME and ASRU2019 Datasets标题：在语码转换汉英SEAM和ASRU2019数据集上采用OpenAI的Whisper进行语音识别链接：https://arxiv.org/abs/2311.17382
作者：Yuhang Yang,Yizhou Peng,Xionghu Zhong,Hao Huang,Eng Siong Chng备注：6 pages, 3 figures, 4 tables摘要：本文详细介绍了在SEAME和ASRU 2019语料库上采用OpenAI的Whisper模型进行码切换普通话-英语语音识别（ASR）的实验结果。我们做了两个实验：a）使用从1到100/200小时的适应数据来证明适应的有效性，b）检查Whisper提示上的不同语言ID设置。“混合错误率”结果显示，适应数据量可能低至$1\sim10$小时，以实现性能增益饱和（SEAME），而ASRU任务继续显示具有更多适应数据的性能（$>$100小时）。对于语言提示，结果表明，虽然不同的提示策略最初产生不同的结果，适应语码转换数据一致提高其性能的耳语模型。这些结果也可能是相关的社会时，应用耳语适应新的目标领域的相关任务。摘要：This paper details the experimental results of adapting the OpenAI's Whisper model for Code-Switch Mandarin-English Speech Recognition (ASR) on the SEAME and ASRU2019 corpora. We conducted 2 experiments: a) using adaptation data from 1 to 100/200 hours to demonstrate effectiveness of adaptation, b) examining different language ID setup on Whisper prompt. The Mixed Error Rate results show that the amount of adaptation data may be as low as $1\sim10$ hours to achieve saturation in performance gain (SEAME) while the ASRU task continued to show performance with more adaptation data ($>$100 hours). For the language prompt, the results show that although various prompting strategies initially produce different outcomes, adapting the Whisper model with code-switch data uniformly improves its performance. These results may be relevant also to the community when applying Whisper for related tasks of adapting to new target domains.【9】 Efficient Deep Speech Understanding at the Edge标题：在边缘有效的深度语音理解链接：https://arxiv.org/abs/2311.17065
作者：Rongxiang Wang,Felix Lin摘要：当代语音理解（SU）涉及一个复杂的管道：捕获实时语音输入，管道包含一个深度神经网络，其编码器-解码器架构通过波束搜索增强。该网络在其自回归输出中定期评估注意力和联结主义时间分类（CTC）分数。本文的目的是提高SU的边缘设备的性能与有限的资源。它追求两个相互交织的目标：加速设备上的执行，并有效地处理超过设备上模型容量的输入。虽然这些目标是既定的，我们推出创新的解决方案，专门解决SU的独特挑战：1。后期上下文化：在输入摄取期间启用模型的关注编码器的并行执行。2.导频解码：减轻时间负载不平衡。3.自回归输出斜坡：促进基于部分输出序列的卸载决策。我们的技术与现有的SU模型，管道和框架无缝集成，允许独立或组合应用。它们共同构成了边缘SU的混合解决方案，例如我们的原型XYZ。在配备6-8个Arm内核的平台上进行评估，我们的系统实现了最先进的（SOTA）精度，将端到端延迟降低了2倍，并将卸载需求减半。摘要：Contemporary Speech Understanding (SU) involves a sophisticated pipeline: capturing real-time voice input, the pipeline encompasses a deep neural network with an encoder-decoder architecture enhanced by beam search. This network periodically assesses attention and Connectionist Temporal Classification (CTC) scores in its autoregressive output. This paper aims to enhance SU performance on edge devices with limited resources. It pursues two intertwined goals: accelerating on-device execution and efficiently handling inputs that surpass the on-device model's capacity. While these objectives are well-established, we introduce innovative solutions that specifically address SU's distinctive challenges: 1. Late contextualization: Enables the parallel execution of a model's attentive encoder during input ingestion. 2. Pilot decoding: Alleviates temporal load imbalances. 3. Autoregression offramps: Facilitate offloading decisions based on partial output sequences. Our techniques seamlessly integrate with existing SU models, pipelines, and frameworks, allowing for independent or combined application. Together, they constitute a hybrid solution for edge SU, exemplified by our prototype, XYZ. Evaluated on platforms equipped with 6-8 Arm cores, our system achieves State-of-the-Art (SOTA) accuracy, reducing end-to-end latency by 2x and halving offloading requirements.【10】 FAT-HuBERT: Front-end Adaptive Training of Hidden-unit BERT for Distortion-Invariant Robust Speech Recognition标题：Fat-Hubert：用于失真不变鲁棒语音识别的隐藏单元BERT的前端自适应训练链接：https://arxiv.org/abs/2311.17790
作者：Dongning Yang,Wei Wang,Yanmin Qian摘要：单声道语音增强（SE）技术的进步极大地改善了语音的感知质量。然而，将这些技术集成到自动语音识别（ASR）系统中并没有产生预期的性能增益，主要是由于在SE过程中引入了失真。在本文中，我们提出了一种新的方法称为FAT-HuBERT，它利用失真不变的自监督学习（SSL），以提高ASR的鲁棒性。为了解决由SE前端引入的失真，我们引入了分层融合模块，该模块结合了从观测到的噪声信号和增强信号中提取的特征。在训练过程中，SE前端是从模型池中随机选择的。我们评估了FAT-HuBERT在LibriSpeech生成的模拟噪声语音以及CHiME-4 1通道数据集的真实噪声语音上的性能。实验结果表明，显着的相对减少字错误率（WER）。摘要：Advancements in monaural speech enhancement (SE) techniques have greatly improved the perceptual quality of speech. However, integrating these techniques into automatic speech recognition (ASR) systems has not yielded the expected performance gains, primarily due to the introduction of distortions during the SE process. In this paper, we propose a novel approach called FAT-HuBERT, which leverages distortion-invariant self-supervised learning (SSL) to enhance the robustness of ASR. To address the distortions introduced by the SE frontends, we introduce layer-wise fusion modules that incorporate features extracted from both observed noisy signals and enhanced signals. During training, the SE frontend is randomly selected from a pool of models. We evaluate the performance of FAT-HuBERT on simulated noisy speech generated from LibriSpeech as well as real-world noisy speech from the CHiME-4 1-channel dataset. The experimental results demonstrate a significant relative reduction in word error rate (WER).【11】 End-to-end Joint Rich and Normalized ASR with a limited amount of rich training data标题：端到端联合丰富和规范化ASR，具有有限的丰富训练数据链接：https://arxiv.org/abs/2311.17741
作者：Can Cui,Imran Ahamad Sheikh,Mostafa Sadeghi,Emmanuel Vincent备注：Submitted to ICASSP 2024摘要：联合丰富和规范化的自动语音识别（ASR），产生transmittance与标点符号和大写，仍然是一个挑战。端到端（E2E）ASR模型提供了执行这种语音的联合转录的便利性和能力。训练这样的模型需要成对的语音和富文本数据，这并不广泛可用。在本文中，我们比较了两种不同的方法来训练一个无状态的基于传感器的E2E联合丰富和规范化的ASR系统，为流应用程序做好准备，丰富的标记数据量有限。第一种方法使用语言模型来生成规范化训练数据的伪富translation。第二种方法使用单个解码器，取决于输出的类型。第一种方法导致E2E丰富的ASR，其在域外数据上表现更好，错误相对减少高达9%。第二种方法证明了E2E联合丰富和归一化ASR系统的可行性，该系统使用低至5%的丰富训练数据，误差适度增加（绝对值为2.42%）。摘要：Joint rich and normalized automatic speech recognition (ASR), that produces transcriptions both with and without punctuation and capitalization, remains a challenge. End-to-end (E2E) ASR models offer both convenience and the ability to perform such joint transcription of speech. Training such models requires paired speech and rich text data, which is not widely available. In this paper, we compare two different approaches to train a stateless Transducer-based E2E joint rich and normalized ASR system, ready for streaming applications, with a limited amount of rich labeled data. The first approach uses a language model to generate pseudo-rich transcriptions of normalized training data. The second approach uses a single decoder conditioned on the type of the output. The first approach leads to E2E rich ASR which perform better on out-of-domain data, with up to 9% relative reduction in errors. The second approach demonstrates the feasibility of an E2E joint rich and normalized ASR system using as low as 5% rich training data with moderate (2.42% absolute) increase in errors.【12】 Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes标题：自动身份识别对视听真伪的脆弱性链接：https://arxiv.org/abs/2311.17655
作者：Pavel Korshunov,Haolin Chen,Philip N. Garner,Sebastien Marcel备注：10 pages, 3 figures, 3 tables摘要：深度伪造检测的任务远未被语音或视觉研究人员解决。建立了几个公开可用的假合成视频和语音数据库，以帮助开发检测方法。然而，现有的数据库通常专注于视觉或语音模式，并没有提供证据证明他们的deepfake实际上可以冒充任何真实的人。在本文中，我们提出了第一个真实的深度伪造SWAN-DF视听数据库，其中嘴唇和语音同步良好，视频具有很高的视觉和音频质量。我们使用公开的SWAN数据集，使用DeepFaceLab的几种模型和混合技术来创建具有不同身份的真实视频的视听deepfake，用于面部交换和HiFiVC，DiffVC，YourTTS和FreeVC模型进行语音转换。从公开可用的语音数据集LibriTTS中，我们还使用几种最新的文本到语音方法创建了一个单独的音频deepfakes LibriTTS-DF数据库：YourTTS，Adaspeech和TorToiSe。我们展示了最先进的说话人识别系统，如ECAPA-TDNN为基础的模型SpeechBrain的脆弱性，合成的声音。同样，我们测试了基于MobileFaceNet架构的人脸识别系统，以测试我们的视觉deepfake的几个变体。漏洞评估表明，通过将现有的预训练的deepfake模型调整到特定的身份，人们可以在90%以上的时间内成功欺骗面部和说话人识别系统，并获得一个非常逼真的给定人的虚假视频。摘要：The task of deepfakes detection is far from being solved by speech or vision researchers. Several publicly available databases of fake synthetic video and speech were built to aid the development of detection methods. However, existing databases typically focus on visual or voice modalities and provide no proof that their deepfakes can in fact impersonate any real person. In this paper, we present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized and video have high visual and audio qualities. We took the publicly available SWAN dataset of real videos with different identities to create audio-visual deepfakes using several models from DeepFaceLab and blending techniques for face swapping and HiFiVC, DiffVC, YourTTS, and FreeVC models for voice conversion. From the publicly available speech dataset LibriTTS, we also created a separate database of only audio deepfakes LibriTTS-DF using several latest text to speech methods: YourTTS, Adaspeech, and TorToiSe. We demonstrate the vulnerability of a state of the art speaker recognition system, such as ECAPA-TDNN-based model from SpeechBrain, to the synthetic voices. Similarly, we tested face recognition system based on the MobileFaceNet architecture to several variants of our visual deepfakes. The vulnerability assessment show that by tuning the existing pretrained deepfake models to specific identities, one can successfully spoof the face and speaker recognition systems in more than 90% of the time and achieve a very realistic looking and sounding fake video of a given person. 
每日语音速递