今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。
cs.SD语音

【1】 QuickVC: Many-to-any Voice Conversion Using Inverse Short-time Fourier  Transform for Faster Conversion

标题：QuickVC：使用短时傅里叶逆变换实现多对任意语音转换，以实现更快的转换

链接：https://arxiv.org/abs/2302.08296

作者：Houjian Guo,Chaoran Liu,Carlos Toshinori Ishi,Hiroshi Ishiguro
机构：Interactive Robot Research Team, Guardian Robot Project, RIKEN, Japan, Graduate School of Engineering Science, Osaka University, Japan, Advanced Telecommunications Research Institute International, Japan
摘要：随着自动语音识别（ASR）和文本语音转换（TTS）技术的发展，通过提取源内容信息和目标说话人信息重构波形，可以实现高质量的语音转换（VC）。然而，当前方法在推理速度方面仍需要改进。在本研究中，我们提出了一个轻量级的基于VITS的VC模型，它使用HuBERT-Soft模型来提取内容信息特征，而不需要说话人信息。通过对合成语音的主观和客观实验，该模型在自然度和相似度方面表现出了较好的效果。重要的是，与原始的VTS模型不同，我们使用逆短时傅里叶变换（iSTFT）来替换计算量最大的部分。实验结果表明，该模型在3090 GPU和i9- 10900 K CPU上分别可以生成5000 kHz和250 kHz以上的样本，在相同的硬件配置下具有竞争力的速度。
摘要：With the development of automatic speech recognition (ASR) and text-to-speech (TTS) technology, high-quality voice conversion (VC) can be achieved by extracting source content information and target speaker information to reconstruct waveforms. However, current methods still require improvement in terms of inference speed. In this study, we propose a lightweight VITS-based VC model that uses the HuBERT-Soft model to extract content information features without speaker information. Through subjective and objective experiments on synthesized speech, the proposed model demonstrates competitive results in terms of naturalness and similarity. Importantly, unlike the original VITS model, we use the inverse short-time Fourier transform (iSTFT) to replace the most computationally expensive part. Experimental results show that our model can generate samples at over 5000 kHz on the 3090 GPU and over 250 kHz on the i9-10900K CPU, achieving competitive speed for the same hardware configuration.

【2】 ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly  Disentangled Self-supervised Speech Representations

标题：ACE-VC：使用显式解缠的自监督语音表示进行自适应和可控的语音转换

链接：https://arxiv.org/abs/2302.08137

作者：Shehzeen Hussain,Paarth Neekhara,Jocelyn Huang,Jason Li,Boris Ginsburg
机构：University of California, San Diego, CA, USA, NVIDIA Corporation, Santa Clara, CA, USA
备注：Published as a conference paper at ICASSP 2023
摘要：在这项工作中，我们提出了一种zero-shot语音转换方法，使用自监督学习训练的语音表示。首先，我们发展了一个多任务模型，将一个语音话语分解为语言内容、说话者特征和说话风格等特征。为了理清内容和说话人的表征，我们提出了一种基于暹罗网络的训练策略，该策略鼓励原始音频和音调偏移音频的内容表征之间的相似性。接下来，我们发展了一个具有音调和持续时间预测器的合成模型，它可以有效地从语音信号的分解表示中重构语音信号。我们的框架允许可控的和说话者自适应的合成来执行zero-shot任意到任意的语音转换，在评估说话者相似性、可理解性和自然度的指标上实现最先进的结果。我们的框架仅使用目标说话人的10秒数据就可以执行语音交换，并实现了对可见说话人和不可见说话人的说话人验证EER分别为5.5%和8.4%。
摘要：In this work, we propose a zero-shot voice conversion method using speech representations trained with self-supervised learning. First, we develop a multi-task model to decompose a speech utterance into features such as linguistic content, speaker characteristics, and speaking style. To disentangle content and speaker representations, we propose a training strategy based on Siamese networks that encourages similarity between the content representations of the original and pitch-shifted audio. Next, we develop a synthesis model with pitch and duration predictors that can effectively reconstruct the speech signal from its decomposed representation. Our framework allows controllable and speaker-adaptive synthesis to perform zero-shot any-to-any voice conversion achieving state-of-the-art results on metrics evaluating speaker similarity, intelligibility, and naturalness. Using just 10 seconds of data for a target speaker, our framework can perform voice swapping and achieves a speaker verification EER of 5.5% for seen speakers and 8.4% for unseen speakers.

【3】 An Attention-based Approach to Hierarchical Multi-label Music Instrument  Classification

标题：一种基于注意力的层次化多标签乐器分类方法

链接：https://arxiv.org/abs/2302.08136

作者：Zhi Zhong,Masato Hirano,Kazuki Shimada,Kazuya Tateishi,Shusuke Takahashi,Yuki Mitsufuji
机构：Sony Group Corporation, Tokyo, Japan
备注：To appear at ICASSP 2023
摘要：虽然音乐通常是多标签的，但是许多工作已经研究了具有简化设置（例如单标签数据）的分层音乐标签。此外，缺乏一个框架来描述多标签环境下的各种联合训练方法。为了讨论以上问题，本文引入了层次化多标签乐器分类任务。该任务提供其中假设多乐器真实音乐数据的现实设置。在深度学习和传统技术融合的背景下，总结和探索了联合训练DNN的各种分层方法。为了在多标签环境下有效地进行联合训练，提出了两种方法来建模细、粗标签之间的连接，一种方法使用基于规则的分组最大池，另一种方法使用数据驱动方式获得的注意机制。实验结果表明，该方法优于未进行联合训练的方法。此外，所提出的方法中的决策过程可以通过可视化注意图或参考固定规则来解释。
摘要：Although music is typically multi-label, many works have studied hierarchical music tagging with simplified settings such as single-label data. Moreover, there lacks a framework to describe various joint training methods under the multi-label setting. In order to discuss the above topics, we introduce hierarchical multi-label music instrument classification task. The task provides a realistic setting where multi-instrument real music data is assumed. Various hierarchical methods that jointly train a DNN are summarized and explored in the context of the fusion of deep learning and conventional techniques. For the effective joint training in the multi-label setting, we propose two methods to model the connection between fine- and coarse-level tags, where one uses rule-based grouped max-pooling, the other one uses the attention mechanism obtained in a data-driven manner. Our evaluation reveals that the proposed methods have advantages over the method without joint training. In addition, the decision procedure within the proposed methods can be interpreted by visualizing attention maps or referring to fixed rules.

【4】 Personalized Audio Quality Preference Prediction

标题：个性化音频质量偏好预测

链接：https://arxiv.org/abs/2302.08130

作者：Chung-Che Wang,Yu-Chun Lin,Yu-Teng Hsu,Jyh-Shing Roger Jang
机构：Dept. of Computer Science and Information Engineering, National Taiwan Univ., Taiwan
摘要：本文提出利用音频输入和主题信息来预测两个内容相同但质量不同的音频片段的个性化偏好。一个连体网络被用来比较输入和预测偏好。研究了几种不同的结构，其中以PANNs的CNN6作为编码器，多层感知器模块作为解码器的LDNet比仅使用音频输入的基线模型性能最好，其整体准确率从77.56%提高到78.04%。实验结果还表明，使用所有的主题信息，包括年龄、性别、头戴式耳机或耳机的规格，比只使用其中的一部分更有效。
摘要：This paper proposes to use both audio input and subject information to predict the personalized preference of two audio segments with the same content in different qualities. A siamese network is used to compare the inputs and predict the preference. Several different structures for each side of the siamese network are investigated, and an LDNet with PANNs' CNN6 as the encoder and a multi-layer perceptron block as the decoder outperforms a baseline model using only audio input the most, where the overall accuracy grows from 77.56% to 78.04%. Experimental results also show that using all the subject information, including age, gender, and the specifications of headphones or earphones, is more effective than using only a part of them.

【5】 Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech  Recognition

标题：说话人自适应视觉语音识别中深度神经网络的快速调整

链接：https://arxiv.org/abs/2302.08102

作者：Minsu Kim,Hyung-Il Kim,Yong Man Ro
摘要：视觉语音识别（VSR）的目标是仅根据嘴唇的运动将语音推断为文本。由于VSR模型关注视觉信息来对语音进行建模，因此其性能固有地对个人嘴唇外观和运动敏感，这使得VSR模型在应用于看不见的说话者时表现出降级的性能。本文针对VSR模型在未知说话人情况下性能下降的问题，提出了一种用于说话人自适应VSR的深度神经网络（DNNs）快速调整方法。特别地，受自然语言处理（NLP）的最新进展的推动，我们在目标说话人的适应数据上微调提示，而不是修改预先训练的模型参数。不同于以往主要局限于Transformer变体架构的提示调优方法，本文探索了不同类型的提示，如添加、填充和拼接形式的提示，这些提示可以应用于由CNN和Transformer组成的VSR模型。利用所提出的即时调谐，我们示出了通过使用少量的自适应数据（例如，少于5分钟），即使预先训练的模型已经开发为具有大的说话者变化。此外，通过分析不同类型提示的性能和参数，我们研究了提示调整何时优于微调方法。在LRW-ID和GRID两个词级和句子级的VSR数据库上对所提方法的有效性进行了评估。
摘要：Visual Speech Recognition (VSR) aims to infer speech into text depending on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers. In this paper, to remedy the performance degradation of the VSR model on unseen speakers, we propose prompt tuning methods of Deep Neural Networks (DNNs) for speaker-adaptive VSR. Specifically, motivated by recent advances in Natural Language Processing (NLP), we finetune prompts on adaptation data of target speakers instead of modifying the pre-trained model parameters. Different from the previous prompt tuning methods mainly limited to Transformer variant architecture, we explore different types of prompts, the addition, the padding, and the concatenation form prompts that can be applied to the VSR model which is composed of CNN and Transformer in general. With the proposed prompt tuning, we show that the performance of the pre-trained VSR model on unseen speakers can be largely improved by using a small amount of adaptation data (e.g., less than 5 minutes), even if the pre-trained model is already developed with large speaker variations. Moreover, by analyzing the performance and parameters of different types of prompts, we investigate when the prompt tuning is preferred over the finetuning methods. The effectiveness of the proposed method is evaluated on both word- and sentence-level VSR databases, LRW-ID and GRID.

【6】 PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech  Enhancement

标题：PAAPLoss：一种语音对齐的语音增强声学参数损失

链接：https://arxiv.org/abs/2302.08095

作者：Muqiao Yang,Joseph Konan,David Bick,Yunyang Zeng,Shuo Han,Anurag Kumar,Shinji Watanabe,Bhiksha Raj
机构：Carnegie Mellon University,Meta Reality Labs Research
备注：Accepted at ICASSP 2023
摘要：尽管近年来发展迅速，但是当前的语音增强模型经常产生在感知质量上与真实干净语音不同的语音。我们提出了一个学习目标，通过使用声学语音学领域知识，形式化感知质量的差异。我们识别出时间声学参数--如频谱倾斜、频谱通量、闪烁等--这些参数是不可微的，我们开发了一个神经网络估计器，可以准确地预测它们在整个话语中的时间序列值。我们还对每个特征的音素特定权重进行建模，因为已知声学参数在不同音素中表现出不同的行为。我们可以将该标准作为辅助损失添加到产生语音的任何模型，以优化语音输出，从而匹配这些特征中的干净语音的值。实验表明，它改善了语音增强工作流在时域和时频域，作为衡量标准的评估指标。我们还提供了对声学参数的音素依赖性改进的分析，证明了我们的方法提供的额外的可解释性。这种分析可以指出哪些特性是当前改进的瓶颈。
摘要：Despite rapid advancement in recent years, current speech enhancement models often produce speech that differs in perceptual quality from real clean speech. We propose a learning objective that formalizes differences in perceptual quality, by using domain knowledge of acoustic-phonetics. We identify temporal acoustic parameters -- such as spectral tilt, spectral flux, shimmer, etc. -- that are non-differentiable, and we develop a neural network estimator that can accurately predict their time-series values across an utterance. We also model phoneme-specific weights for each feature, as the acoustic parameters are known to show different behavior in different phonemes. We can add this criterion as an auxiliary loss to any model that produces speech, to optimize speech outputs to match the values of clean speech in these features. Experimentally we show that it improves speech enhancement workflows in both time-domain and time-frequency domain, as measured by standard evaluation metrics. We also provide an analysis of phoneme-dependent improvement on acoustic parameters, demonstrating the additional interpretability that our method provides. This analysis can suggest which features are currently the bottleneck for improvement.

【7】 TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement

标题：TAPLoss：一种用于语音增强的时间声学参数损失

链接：https://arxiv.org/abs/2302.08088

作者：Yunyang Zeng,Joseph Konan,Shuo Han,David Bick,Muqiao Yang,Anurag Kumar,Shinji Watanabe,Bhiksha Raj
机构：Carnegie Mellon University,Meta Reality Labs Research
备注：Accepted at ICASSP 2023
摘要：语音增强模型近年来有了很大的发展，但其语音输出的感知质量仍然有限。我们提出了一个基于时间声学参数的感知质量目标。这些基本语音特征在说话人识别和副语言分析等各种应用中起着至关重要的作用。我们提供了用于四类低级声学描述符的可微分估计器，包括：频率相关参数、能量或振幅相关参数、频谱平衡参数和时间特征。与关注聚集声学参数或少数类别的声学参数的先前工作不同，我们的时间声学参数（TAP）损失使得能够辅助优化和改进增强工作流程中的许多细粒度语音特征。我们表明，在语音增强中添加TAPLoss作为辅助目标产生具有改善的感知质量和可懂度的语音。我们使用深度噪声抑制2020挑战赛的数据来证明时域模型和时频域模型都可以从我们的方法中受益。
摘要：Speech enhancement models have greatly progressed in recent years, but still show limits in perceptual quality of their speech outputs. We propose an objective for perceptual quality based on temporal acoustic parameters. These are fundamental speech features that play an essential role in various applications, including speaker recognition and paralinguistic analysis. We provide a differentiable estimator for four categories of low-level acoustic descriptors involving: frequency-related parameters, energy or amplitude-related parameters, spectral balance parameters, and temporal features. Unlike prior work that looks at aggregated acoustic parameters or a few categories of acoustic parameters, our temporal acoustic parameter (TAP) loss enables auxiliary optimization and improvement of many fine-grain speech characteristics in enhancement workflows. We show that adding TAPLoss as an auxiliary objective in speech enhancement produces speech with improved perceptual quality and intelligibility. We use data from the Deep Noise Suppression 2020 Challenge to demonstrate that both time-domain models and time-frequency domain models can benefit from our method.

【8】 Speech Enhancement with Multi-granularity Vector Quantization

标题：基于多粒度矢量量化的语音增强

链接：https://arxiv.org/abs/2302.08342

作者：Xiao-Ying Zhao,Qiu-Shi Zhu,Jie Zhang
机构：NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China
摘要：随着深度学习技术的发展，基于神经网络的语音增强（SE）技术在近十年得到了迅速发展。同时，自监督预训练模型和矢量量化（VQ）在许多与语音相关的任务中取得了很好的性能，而在SE方面的研究较少。正如我们先前的工作所示，利用矢量量化模块离散化含噪语音表示有利于语音去噪，因此在本工作中，我们研究了在不同层使用不同数量码本的矢量量化的影响。不同的矢量量化模块确实能够提取多粒度语音特征。在注意机制之后，由预训练模型提取的上下文特征与由编码器提取的局部特征融合，使得全局和局部信息都被保留以重构增强的语音。在Valentini数据集上的实验结果表明，所提模型能有效提高SE性能，同时也揭示了预训练模型的选择对SE性能的影响。
摘要：With advances in deep learning, neural network based speech enhancement (SE) has developed rapidly in the last decade. Meanwhile, the self-supervised pre-trained model and vector quantization (VQ) have achieved excellent performance on many speech-related tasks, while they are less explored on SE. As it was shown in our previous work that utilizing a VQ module to discretize noisy speech representations is beneficial for speech denoising, in this work we therefore study the impact of using VQ at different layers with different number of codebooks. Different VQ modules indeed enable to extract multiple-granularity speech features. Following an attention mechanism, the contextual features extracted by a pre-trained model are fused with the local features extracted by the encoder, such that both global and local information are preserved to reconstruct the enhanced speech. Experimental results on the Valentini dataset show that the proposed model can improve the SE performance, where the impact of choosing pre-trained models is also revealed.

【9】 DeepSpace: Dynamic Spatial and Source Cue Based Source Separation for  Dialog Enhancement

标题：深度空间：基于动态空间和源线索的对话增强源分离

链接：https://arxiv.org/abs/2302.08202

作者：Aaron Master,Lie Lu,Jonas Samuelsson,Heidi-Maria Lehtonen,Scott Norcross,Nathan Swedlow,Audrey Howard
机构：Dolby Laboratories, Inc, Market St, San Francisco, CA , United States, Dolby Sweden AB, Gävlegatan ,A, Stockholm, Sweden
备注：5 pages, 4 figures. Accepted for publication in ICASSP 2023
摘要：对话增强（DE）是允许用户相对于非对话声音增加TV或电影内容中的对话的级别的特征。当只有原始混合信号可用时，差分方程是“非制导的”，需要源分离。本文描述了深空系统，该系统利用动态空间线索和源线索进行源分离，以支持非制导差分成像。它的技术包括空间级过滤（SLF）和基于深度学习的对话分类和去噪。使用主观听力测试，我们表明，与可供测试的最先进系统相比，深空展示了显著改善的整体性能。我们探索了使用现有自动化指标评估非制导DE系统的可行性。
摘要：Dialog Enhancement (DE) is a feature which allows a user to increase the level of dialog in TV or movie content relative to non-dialog sounds. When only the original mix is available, DE is "unguided," and requires source separation. In this paper, we describe the DeepSpace system, which performs source separation using both dynamic spatial cues and source cues to support unguided DE. Its technologies include spatio-level filtering (SLF) and deep-learning based dialog classification and denoising. Using subjective listening tests, we show that DeepSpace demonstrates significantly improved overall performance relative to state-of-the-art systems available for testing. We explore the feasibility of using existing automated metrics to evaluate unguided DE systems.

【10】 Multi-Channel Target Speaker Extraction with Refinement: The WavLab  Submission to the Second Clarity Enhancement Challenge

标题：改进的多通道目标说话人提取：WavLab提交给第二届清晰度增强挑战赛

链接：https://arxiv.org/abs/2302.07928

作者：Samuele Cornell,Zhong-Qiu Wang,Yoshiki Masuyama,Shinji Watanabe,Manuel Pariente,Nobutaka Ono
机构：Italy, Carnegie Mellon University, USA, Tokyo Metropolitan University, Japan, Pulse Audition, France
摘要：本文介绍了我们提交的第二届清晰度增强挑战赛（CEC 2），该挑战赛包括助听器（HA）设备在噪声混响环境中的目标语音增强，该环境存在多种干扰，如音乐和竞争扬声器。  我们的方法建立在我们最近工作中引入的强大的迭代神经/波束形成增强（iNeuBe）框架的基础上，本文将其扩展到目标说话人提取。因此，我们将拟定方法命名为iNeuBe-X，其中X代表浸提。为了解决在CEC 2环境中遇到的挑战，我们引入了四个主要的新功能：  (1)我们扩展了最先进的TF-GridNet模型，该模型最初设计用于单声道说话人分离，用于多通道、因果语音增强，并且通过用这种新的架构替换iNeuBe中使用的TCNDenseNet观察到了很大的改进;   (2)我们利用最近的具有未来帧预测的双窗口大小方法来确保iNueBe-X满足CEC 2所要求的算法延迟的5 ms约束;   (3)引入一个新的说话人条件分支实现TF-GridNet的目标说话人提取;   (4)我们提出了一个微调步骤，在该步骤中，我们计算相对于用听者听力图补偿的目标说话者信号的附加损耗。  在不使用外部数据的情况下，在官方开发集上，我们的最佳模型达到了0.942的助听器语音感知指数（HASPI）得分和18.8 dB的标度不变信号失真比改善（SI-SDRi）。鉴于CEC 2数据极具挑战性（例如：在显影装置上，混合SI-SDR为-12.3dB）。我们提交的系统的演示可在WAVLab CEC 2演示中获得。
摘要：This paper describes our submission to the Second Clarity Enhancement Challenge (CEC2), which consists of target speech enhancement for hearing-aid (HA) devices in noisy-reverberant environments with multiple interferers such as music and competing speakers.  Our approach builds upon the powerful iterative neural/beamforming enhancement (iNeuBe) framework introduced in our recent work, and this paper extends it for target speaker extraction. We therefore name the proposed approach as iNeuBe-X, where the X stands for extraction. To address the challenges encountered in the CEC2 setting, we introduce four major novelties:  (1) we extend the state-of-the-art TF-GridNet model, originally designed for monaural speaker separation, for multi-channel, causal speech enhancement, and large improvements are observed by replacing the TCNDenseNet used in iNeuBe with this new architecture;  (2) we leverage a recent dual window size approach with future-frame prediction to ensure that iNueBe-X satisfies the 5 ms constraint on algorithmic latency required by CEC2;  (3) we introduce a novel speaker-conditioning branch for TF-GridNet to achieve target speaker extraction;  (4) we propose a fine-tuning step, where we compute an additional loss with respect to the target speaker signal compensated with the listener audiogram.  Without using external data, on the official development set our best model reaches a hearing-aid speech perception index (HASPI) score of 0.942 and a scale-invariant signal-to-distortion ratio improvement (SI-SDRi) of 18.8 dB. These results are promising given the fact that the CEC2 data is extremely challenging (e.g., on the development set the mixture SI-SDR is -12.3 dB). A demo of our submitted system is available at WAVLab CEC2 demo.
eess.AS音频处理

【1】 Speech Enhancement with Multi-granularity Vector Quantization

标题：基于多粒度矢量量化的语音增强

链接：https://arxiv.org/abs/2302.08342

*与cs.SD语音【8】为同一篇

作者：Xiao-Ying Zhao,Qiu-Shi Zhu,Jie Zhang
机构：NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China
摘要：随着深度学习技术的发展，基于神经网络的语音增强（SE）技术在近十年得到了迅速发展。同时，自监督预训练模型和矢量量化（VQ）在许多与语音相关的任务中取得了很好的性能，而在SE方面的研究较少。正如我们先前的工作所示，利用矢量量化模块离散化含噪语音表示有利于语音去噪，因此在本工作中，我们研究了在不同层使用不同数量码本的矢量量化的影响。不同的矢量量化模块确实能够提取多粒度语音特征。在注意机制之后，由预训练模型提取的上下文特征与由编码器提取的局部特征融合，使得全局和局部信息都被保留以重构增强的语音。在Valentini数据集上的实验结果表明，所提模型能有效提高SE性能，同时也揭示了预训练模型的选择对SE性能的影响。
摘要：With advances in deep learning, neural network based speech enhancement (SE) has developed rapidly in the last decade. Meanwhile, the self-supervised pre-trained model and vector quantization (VQ) have achieved excellent performance on many speech-related tasks, while they are less explored on SE. As it was shown in our previous work that utilizing a VQ module to discretize noisy speech representations is beneficial for speech denoising, in this work we therefore study the impact of using VQ at different layers with different number of codebooks. Different VQ modules indeed enable to extract multiple-granularity speech features. Following an attention mechanism, the contextual features extracted by a pre-trained model are fused with the local features extracted by the encoder, such that both global and local information are preserved to reconstruct the enhanced speech. Experimental results on the Valentini dataset show that the proposed model can improve the SE performance, where the impact of choosing pre-trained models is also revealed.

【2】 DeepSpace: Dynamic Spatial and Source Cue Based Source Separation for  Dialog Enhancement

标题：深度空间：基于动态空间和源线索的对话增强源分离

链接：https://arxiv.org/abs/2302.08202

*与cs.SD语音【9】为同一篇

作者：Aaron Master,Lie Lu,Jonas Samuelsson,Heidi-Maria Lehtonen,Scott Norcross,Nathan Swedlow,Audrey Howard
机构：Dolby Laboratories, Inc, Market St, San Francisco, CA , United States, Dolby Sweden AB, Gävlegatan ,A, Stockholm, Sweden
备注：5 pages, 4 figures. Accepted for publication in ICASSP 2023
摘要：对话增强（DE）是允许用户相对于非对话声音增加TV或电影内容中的对话的级别的特征。当只有原始混合信号可用时，差分方程是“非制导的”，需要源分离。本文描述了深空系统，该系统利用动态空间线索和源线索进行源分离，以支持非制导差分成像。它的技术包括空间级过滤（SLF）和基于深度学习的对话分类和去噪。使用主观听力测试，我们表明，与可供测试的最先进系统相比，深空展示了显著改善的整体性能。我们探索了使用现有自动化指标评估非制导DE系统的可行性。
摘要：Dialog Enhancement (DE) is a feature which allows a user to increase the level of dialog in TV or movie content relative to non-dialog sounds. When only the original mix is available, DE is "unguided," and requires source separation. In this paper, we describe the DeepSpace system, which performs source separation using both dynamic spatial cues and source cues to support unguided DE. Its technologies include spatio-level filtering (SLF) and deep-learning based dialog classification and denoising. Using subjective listening tests, we show that DeepSpace demonstrates significantly improved overall performance relative to state-of-the-art systems available for testing. We explore the feasibility of using existing automated metrics to evaluate unguided DE systems.

【3】 Multi-Channel Target Speaker Extraction with Refinement: The WavLab  Submission to the Second Clarity Enhancement Challenge

标题：改进的多通道目标说话人提取：WavLab提交给第二届清晰度增强挑战赛

链接：https://arxiv.org/abs/2302.07928

*与cs.SD语音【10】为同一篇

作者：Samuele Cornell,Zhong-Qiu Wang,Yoshiki Masuyama,Shinji Watanabe,Manuel Pariente,Nobutaka Ono
机构：Italy, Carnegie Mellon University, USA, Tokyo Metropolitan University, Japan, Pulse Audition, France
摘要：本文介绍了我们提交的第二届清晰度增强挑战赛（CEC 2），该挑战赛包括助听器（HA）设备在噪声混响环境中的目标语音增强，该环境存在多种干扰，如音乐和竞争扬声器。  我们的方法建立在我们最近工作中引入的强大的迭代神经/波束形成增强（iNeuBe）框架的基础上，本文将其扩展到目标说话人提取。因此，我们将拟定方法命名为iNeuBe-X，其中X代表浸提。为了解决在CEC 2环境中遇到的挑战，我们引入了四个主要的新功能：  (1)我们扩展了最先进的TF-GridNet模型，该模型最初设计用于单声道说话人分离，用于多通道、因果语音增强，并且通过用这种新的架构替换iNeuBe中使用的TCNDenseNet观察到了很大的改进;   (2)我们利用最近的具有未来帧预测的双窗口大小方法来确保iNueBe-X满足CEC 2所要求的算法延迟的5 ms约束;   (3)引入一个新的说话人条件分支实现TF-GridNet的目标说话人提取;   (4)我们提出了一个微调步骤，在该步骤中，我们计算相对于用听者听力图补偿的目标说话者信号的附加损耗。  在不使用外部数据的情况下，在官方开发集上，我们的最佳模型达到了0.942的助听器语音感知指数（HASPI）得分和18.8 dB的标度不变信号失真比改善（SI-SDRi）。鉴于CEC 2数据极具挑战性（例如：在显影装置上，混合SI-SDR为-12.3dB）。我们提交的系统的演示可在WAVLab CEC 2演示中获得。
摘要：This paper describes our submission to the Second Clarity Enhancement Challenge (CEC2), which consists of target speech enhancement for hearing-aid (HA) devices in noisy-reverberant environments with multiple interferers such as music and competing speakers.  Our approach builds upon the powerful iterative neural/beamforming enhancement (iNeuBe) framework introduced in our recent work, and this paper extends it for target speaker extraction. We therefore name the proposed approach as iNeuBe-X, where the X stands for extraction. To address the challenges encountered in the CEC2 setting, we introduce four major novelties:  (1) we extend the state-of-the-art TF-GridNet model, originally designed for monaural speaker separation, for multi-channel, causal speech enhancement, and large improvements are observed by replacing the TCNDenseNet used in iNeuBe with this new architecture;  (2) we leverage a recent dual window size approach with future-frame prediction to ensure that iNueBe-X satisfies the 5 ms constraint on algorithmic latency required by CEC2;  (3) we introduce a novel speaker-conditioning branch for TF-GridNet to achieve target speaker extraction;  (4) we propose a fine-tuning step, where we compute an additional loss with respect to the target speaker signal compensated with the listener audiogram.  Without using external data, on the official development set our best model reaches a hearing-aid speech perception index (HASPI) score of 0.942 and a scale-invariant signal-to-distortion ratio improvement (SI-SDRi) of 18.8 dB. These results are promising given the fact that the CEC2 data is extremely challenging (e.g., on the development set the mixture SI-SDR is -12.3 dB). A demo of our submitted system is available at WAVLab CEC2 demo.

【4】 QuickVC: Many-to-any Voice Conversion Using Inverse Short-time Fourier  Transform for Faster Conversion

标题：QuickVC：使用短时傅里叶逆变换实现多对任意语音转换，以实现更快的转换

链接：https://arxiv.org/abs/2302.08296

*与cs.SD语音【1】为同一篇

作者：Houjian Guo,Chaoran Liu,Carlos Toshinori Ishi,Hiroshi Ishiguro
机构：Interactive Robot Research Team, Guardian Robot Project, RIKEN, Japan, Graduate School of Engineering Science, Osaka University, Japan, Advanced Telecommunications Research Institute International, Japan
摘要：随着自动语音识别（ASR）和文本语音转换（TTS）技术的发展，通过提取源内容信息和目标说话人信息重构波形，可以实现高质量的语音转换（VC）。然而，当前方法在推理速度方面仍需要改进。在本研究中，我们提出了一个轻量级的基于VITS的VC模型，它使用HuBERT-Soft模型来提取内容信息特征，而不需要说话人信息。通过对合成语音的主观和客观实验，该模型在自然度和相似度方面表现出了较好的效果。重要的是，与原始的VTS模型不同，我们使用逆短时傅里叶变换（iSTFT）来替换计算量最大的部分。实验结果表明，该模型在3090 GPU和i9- 10900 K CPU上分别可以生成5000 kHz和250 kHz以上的样本，在相同的硬件配置下具有竞争力的速度。
摘要：With the development of automatic speech recognition (ASR) and text-to-speech (TTS) technology, high-quality voice conversion (VC) can be achieved by extracting source content information and target speaker information to reconstruct waveforms. However, current methods still require improvement in terms of inference speed. In this study, we propose a lightweight VITS-based VC model that uses the HuBERT-Soft model to extract content information features without speaker information. Through subjective and objective experiments on synthesized speech, the proposed model demonstrates competitive results in terms of naturalness and similarity. Importantly, unlike the original VITS model, we use the inverse short-time Fourier transform (iSTFT) to replace the most computationally expensive part. Experimental results show that our model can generate samples at over 5000 kHz on the 3090 GPU and over 250 kHz on the i9-10900K CPU, achieving competitive speed for the same hardware configuration.

【5】 ACE-VC: Adaptive and Controllable Voice Conversion using Explicitly  Disentangled Self-supervised Speech Representations

标题：ACE-VC：使用显式解缠的自监督语音表示进行自适应和可控的语音转换

链接：https://arxiv.org/abs/2302.08137

*与cs.SD语音【2】为同一篇

作者：Shehzeen Hussain,Paarth Neekhara,Jocelyn Huang,Jason Li,Boris Ginsburg
机构：University of California, San Diego, CA, USA, NVIDIA Corporation, Santa Clara, CA, USA
备注：Published as a conference paper at ICASSP 2023
摘要：在这项工作中，我们提出了一种zero-shot语音转换方法，使用自监督学习训练的语音表示。首先，我们发展了一个多任务模型，将一个语音话语分解为语言内容、说话者特征和说话风格等特征。为了理清内容和说话人的表征，我们提出了一种基于暹罗网络的训练策略，该策略鼓励原始音频和音调偏移音频的内容表征之间的相似性。接下来，我们发展了一个具有音调和持续时间预测器的合成模型，它可以有效地从语音信号的分解表示中重构语音信号。我们的框架允许可控的和说话者自适应的合成来执行zero-shot任意到任意的语音转换，在评估说话者相似性、可理解性和自然度的指标上实现最先进的结果。我们的框架仅使用目标说话人的10秒数据就可以执行语音交换，并实现了对可见说话人和不可见说话人的说话人验证EER分别为5.5%和8.4%。
摘要：In this work, we propose a zero-shot voice conversion method using speech representations trained with self-supervised learning. First, we develop a multi-task model to decompose a speech utterance into features such as linguistic content, speaker characteristics, and speaking style. To disentangle content and speaker representations, we propose a training strategy based on Siamese networks that encourages similarity between the content representations of the original and pitch-shifted audio. Next, we develop a synthesis model with pitch and duration predictors that can effectively reconstruct the speech signal from its decomposed representation. Our framework allows controllable and speaker-adaptive synthesis to perform zero-shot any-to-any voice conversion achieving state-of-the-art results on metrics evaluating speaker similarity, intelligibility, and naturalness. Using just 10 seconds of data for a target speaker, our framework can perform voice swapping and achieves a speaker verification EER of 5.5% for seen speakers and 8.4% for unseen speakers.

【6】 An Attention-based Approach to Hierarchical Multi-label Music Instrument  Classification

标题：一种基于注意力的层次化多标签乐器分类方法

链接：https://arxiv.org/abs/2302.08136

*与cs.SD语音【3】为同一篇

作者：Zhi Zhong,Masato Hirano,Kazuki Shimada,Kazuya Tateishi,Shusuke Takahashi,Yuki Mitsufuji
机构：Sony Group Corporation, Tokyo, Japan
备注：To appear at ICASSP 2023
摘要：None
摘要：Although music is typically multi-label, many works have studied hierarchical music tagging with simplified settings such as single-label data. Moreover, there lacks a framework to describe various joint training methods under the multi-label setting. In order to discuss the above topics, we introduce hierarchical multi-label music instrument classification task. The task provides a realistic setting where multi-instrument real music data is assumed. Various hierarchical methods that jointly train a DNN are summarized and explored in the context of the fusion of deep learning and conventional techniques. For the effective joint training in the multi-label setting, we propose two methods to model the connection between fine- and coarse-level tags, where one uses rule-based grouped max-pooling, the other one uses the attention mechanism obtained in a data-driven manner. Our evaluation reveals that the proposed methods have advantages over the method without joint training. In addition, the decision procedure within the proposed methods can be interpreted by visualizing attention maps or referring to fixed rules.

【7】 Personalized Audio Quality Preference Prediction

标题：个性化音频质量偏好预测

链接：https://arxiv.org/abs/2302.08130

*与cs.SD语音【4】为同一篇

作者：Chung-Che Wang,Yu-Chun Lin,Yu-Teng Hsu,Jyh-Shing Roger Jang
机构：Dept. of Computer Science and Information Engineering, National Taiwan Univ., Taiwan
摘要：本文提出利用音频输入和主题信息来预测两个内容相同但质量不同的音频片段的个性化偏好。一个连体网络被用来比较输入和预测偏好。研究了几种不同的结构，其中以PANNs的CNN6作为编码器，多层感知器模块作为解码器的LDNet比仅使用音频输入的基线模型性能最好，其整体准确率从77.56%提高到78.04%。实验结果还表明，使用所有的主题信息，包括年龄、性别、头戴式耳机或耳机的规格，比只使用其中的一部分更有效。
摘要：This paper proposes to use both audio input and subject information to predict the personalized preference of two audio segments with the same content in different qualities. A siamese network is used to compare the inputs and predict the preference. Several different structures for each side of the siamese network are investigated, and an LDNet with PANNs' CNN6 as the encoder and a multi-layer perceptron block as the decoder outperforms a baseline model using only audio input the most, where the overall accuracy grows from 77.56% to 78.04%. Experimental results also show that using all the subject information, including age, gender, and the specifications of headphones or earphones, is more effective than using only a part of them.

【8】 Prompt Tuning of Deep Neural Networks for Speaker-adaptive Visual Speech  Recognition

标题：说话人自适应视觉语音识别中深度神经网络的快速调整

链接：https://arxiv.org/abs/2302.08102

*与cs.SD语音【5】为同一篇

作者：Minsu Kim,Hyung-Il Kim,Yong Man Ro
摘要：视觉语音识别（VSR）的目标是仅根据嘴唇的运动将语音推断为文本。由于VSR模型关注视觉信息来对语音进行建模，因此其性能固有地对个人嘴唇外观和运动敏感，这使得VSR模型在应用于看不见的说话者时表现出降级的性能。本文针对VSR模型在未知说话人情况下性能下降的问题，提出了一种用于说话人自适应VSR的深度神经网络（DNNs）快速调整方法。特别地，受自然语言处理（NLP）的最新进展的推动，我们在目标说话人的适应数据上微调提示，而不是修改预先训练的模型参数。不同于以往主要局限于Transformer变体架构的提示调优方法，本文探索了不同类型的提示，如添加、填充和拼接形式的提示，这些提示可以应用于由CNN和Transformer组成的VSR模型。利用所提出的即时调谐，我们示出了通过使用少量的自适应数据（例如，少于5分钟），即使预先训练的模型已经开发为具有大的说话者变化。此外，通过分析不同类型提示的性能和参数，我们研究了提示调整何时优于微调方法。在LRW-ID和GRID两个词级和句子级的VSR数据库上对所提方法的有效性进行了评估。
摘要：Visual Speech Recognition (VSR) aims to infer speech into text depending on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements, and this makes the VSR models show degraded performance when they are applied to unseen speakers. In this paper, to remedy the performance degradation of the VSR model on unseen speakers, we propose prompt tuning methods of Deep Neural Networks (DNNs) for speaker-adaptive VSR. Specifically, motivated by recent advances in Natural Language Processing (NLP), we finetune prompts on adaptation data of target speakers instead of modifying the pre-trained model parameters. Different from the previous prompt tuning methods mainly limited to Transformer variant architecture, we explore different types of prompts, the addition, the padding, and the concatenation form prompts that can be applied to the VSR model which is composed of CNN and Transformer in general. With the proposed prompt tuning, we show that the performance of the pre-trained VSR model on unseen speakers can be largely improved by using a small amount of adaptation data (e.g., less than 5 minutes), even if the pre-trained model is already developed with large speaker variations. Moreover, by analyzing the performance and parameters of different types of prompts, we investigate when the prompt tuning is preferred over the finetuning methods. The effectiveness of the proposed method is evaluated on both word- and sentence-level VSR databases, LRW-ID and GRID.

【9】 PAAPLoss: A Phonetic-Aligned Acoustic Parameter Loss for Speech  Enhancement

标题：PAAPLoss：一种语音对齐的语音增强声学参数损失

链接：https://arxiv.org/abs/2302.08095

*与cs.SD语音【6】为同一篇

作者：Muqiao Yang,Joseph Konan,David Bick,Yunyang Zeng,Shuo Han,Anurag Kumar,Shinji Watanabe,Bhiksha Raj
机构：Carnegie Mellon University,Meta Reality Labs Research
备注：Accepted at ICASSP 2023
摘要：尽管近年来发展迅速，但是当前的语音增强模型经常产生在感知质量上与真实干净语音不同的语音。我们提出了一个学习目标，通过使用声学语音学领域知识，形式化感知质量的差异。我们识别出时间声学参数--如频谱倾斜、频谱通量、闪烁等--这些参数是不可微的，我们开发了一个神经网络估计器，可以准确地预测它们在整个话语中的时间序列值。我们还对每个特征的音素特定权重进行建模，因为已知声学参数在不同音素中表现出不同的行为。我们可以将该标准作为辅助损失添加到产生语音的任何模型，以优化语音输出，从而匹配这些特征中的干净语音的值。实验表明，它改善了语音增强工作流在时域和时频域，作为衡量标准的评估指标。我们还提供了对声学参数的音素依赖性改进的分析，证明了我们的方法提供的额外的可解释性。这种分析可以指出哪些特性是当前改进的瓶颈。
摘要：Despite rapid advancement in recent years, current speech enhancement models often produce speech that differs in perceptual quality from real clean speech. We propose a learning objective that formalizes differences in perceptual quality, by using domain knowledge of acoustic-phonetics. We identify temporal acoustic parameters -- such as spectral tilt, spectral flux, shimmer, etc. -- that are non-differentiable, and we develop a neural network estimator that can accurately predict their time-series values across an utterance. We also model phoneme-specific weights for each feature, as the acoustic parameters are known to show different behavior in different phonemes. We can add this criterion as an auxiliary loss to any model that produces speech, to optimize speech outputs to match the values of clean speech in these features. Experimentally we show that it improves speech enhancement workflows in both time-domain and time-frequency domain, as measured by standard evaluation metrics. We also provide an analysis of phoneme-dependent improvement on acoustic parameters, demonstrating the additional interpretability that our method provides. This analysis can suggest which features are currently the bottleneck for improvement.

【10】 TAPLoss: A Temporal Acoustic Parameter Loss for Speech Enhancement

标题：TAPLoss：一种用于语音增强的时间声学参数损失

链接：https://arxiv.org/abs/2302.08088

*与cs.SD语音【7】为同一篇

作者：Yunyang Zeng,Joseph Konan,Shuo Han,David Bick,Muqiao Yang,Anurag Kumar,Shinji Watanabe,Bhiksha Raj
机构：Carnegie Mellon University,Meta Reality Labs Research
备注：Accepted at ICASSP 2023
摘要：None
摘要：Speech enhancement models have greatly progressed in recent years, but still show limits in perceptual quality of their speech outputs. We propose an objective for perceptual quality based on temporal acoustic parameters. These are fundamental speech features that play an essential role in various applications, including speaker recognition and paralinguistic analysis. We provide a differentiable estimator for four categories of low-level acoustic descriptors involving: frequency-related parameters, energy or amplitude-related parameters, spectral balance parameters, and temporal features. Unlike prior work that looks at aggregated acoustic parameters or a few categories of acoustic parameters, our temporal acoustic parameter (TAP) loss enables auxiliary optimization and improvement of many fine-grain speech characteristics in enhancement workflows. We show that adding TAPLoss as an auxiliary objective in speech enhancement produces speech with improved perceptual quality and intelligibility. We use data from the Deep Noise Suppression 2020 Challenge to demonstrate that both time-domain models and time-frequency domain models can benefit from our method.

机器翻译，仅供参考