【1】 Egocentric Audio-Visual Noise Suppression
标题：以自我为中心的视听噪声抑制
链接：https://arxiv.org/abs/2211.03643
作者：Roshan Sharma,Weipeng He,Ju Lin,Egor Lakomkin,Yang Liu,Kaustubh Kalgaonkar机构：Carnegie Mellon University, Pittsburgh, USA, Meta, Seattle, USA备注：Under Review at ICASSP 2023摘要：本文研究了以自我为中心的视频--说话人不在视频中--的音视频抑制问题。相反，潜在的噪声源在屏幕上是可见的，摄像机模拟屏幕外说话者对外部世界的看法。该设置不同于依赖于嘴唇和面部视觉的视听语音增强中的现有工作。本文首先证明了自我中心视觉信息有助于噪声抑制。我们比较了基于对象识别和动作分类的视觉特征提取器，并研究了对齐音频和视觉表示的方法。然后，我们检查对准特征的不同融合策略，以及噪声抑制模型内的位置，以合并视觉信息。实验表明，视觉特征在用于生成加性校正遮罩时是最有帮助的。最后，为了确保视觉特征相对于不同噪声类型是有区别的，我们引入了联合优化视听噪声抑制和基于视频的声学事件检测的多任务学习框架。此提议的多任务框架在所有指标上均优于仅音频基线，包括0.16 PESQ改善。广泛消融显示，在所有噪声类型和不同SNR下，使用多个主动干扰器的申报模型的性能有所改善。摘要：This paper studies audio-visual suppression for egocentric videos -- where the speaker is not captured in the video. Instead, potential noise sources are visible on screen with the camera emulating the off-screen speaker's view of the outside world. This setting is different from prior work in audio-visual speech enhancement that relies on lip and facial visuals. In this paper, we first demonstrate that egocentric visual information is helpful for noise suppression. We compare object recognition and action classification based visual feature extractors, and investigate methods to align audio and visual representations. Then, we examine different fusion strategies for the aligned features, and locations within the noise suppression model to incorporate visual information. Experiments demonstrate that visual features are most helpful when used to generate additive correction masks. Finally, in order to ensure that the visual features are discriminative with respect to different noise types, we introduce a multi-task learning framework that jointly optimizes audio-visual noise suppression and video based acoustic event detection. This proposed multi-task framework outperforms the audio only baseline on all metrics, including a 0.16 PESQ improvement. Extensive ablations reveal the improved performance of the proposed model with multiple active distractors, over all noise types and across different SNRs.
【2】 Hi,KIA: A Speech Emotion Recognition Dataset for Wake-Up Words
标题：Hi，KIA：一个用于唤醒词的语音情感识别数据集
链接：https://arxiv.org/abs/2211.03371
作者：Taesu Kim,SeungHeon Doh,Gyunpyo Lee,Hyungseok Jeon,Juhan Nam,Hyeon-Jeong Suk机构：∗ Department of Industrial Design, KAIST, Daejeon, South Korea, † Graduate School of Culture Technology, KAIST, Daejeon, South Korea, ‡ KIA Design Studio, Hyundai Motor Company, Hwaseong, South Korea, § Equally contributing authors备注：Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2022摘要：唤醒词（WUW）是用于激活语音识别系统以接收用户的语音输入的短句。WUW话语不仅包括用于唤醒系统的词汇信息，还包括诸如说话者身份或情感等非词汇信息。特别地，识别用户的情绪状态可以精心设计语音通信。然而，很少有数据集，其中的情绪状态的WUW话语标记。本文介绍了一个新的WUW数据集Hi，KIA，该数据集包含来自四位男性和四位女性说话人的488个韩国口音情感话语，每个话语都被标记为四种情感状态，包括愤怒、高兴、悲伤和中性。我们介绍了构建数据集的逐步过程，包括场景选择、后处理和标签一致性的人工验证。同时，我们还利用该数据集，提出了两种用于WUW语音情感识别的分类模型。一种是基于传统手工特征的方法，另一种是使用预先训练的神经网络的迁移学习方法。这些分类模型可以作为进一步研究的基准。摘要：Wake-up words (WUW) is a short sentence used to activate a speech recognition system to receive the user's speech input. WUW utterances include not only the lexical information for waking up the system but also non-lexical information such as speaker identity or emotion. In particular, recognizing the user's emotional state may elaborate the voice communication. However, there is few dataset where the emotional state of the WUW utterances is labeled. In this paper, we introduce Hi, KIA, a new WUW dataset which consists of 488 Korean accent emotional utterances collected from four male and four female speakers and each of utterances is labeled with four emotional states including anger, happy, sad, or neutral. We present the step-by-step procedure to build the dataset, covering scenario selection, post-processing, and human validation for label agreement. Also, we provide two classification models for WUW speech emotion recognition using the dataset. One is based on traditional hand-craft features and the other is a transfer-learning approach using a pre-trained neural network. These classification models could be used as benchmarks in further research.
【3】 "Seeing Sound": Audio Classification with the Wigner-Wille Distribution and Convolutional Neural Networks
链接：https://arxiv.org/abs/2211.03202
作者：Antonios Marios Christonasis,Stef van Eijndhoven,Peter Duin机构： Engineering Doctorate Data Science, Technical University of Eindhoven (TUe), Eindhoven, The, Dutch National Police, Eindhoven, The Netherlands摘要：随着大数据日益可用、物联网硬件被广泛采用以及AI功能日益强大，企业正在不断投资于传感。目前，来自传感器网络的数据与传感器融合和AI算法相结合，推动自动驾驶汽车等领域的创新。来自这些传感器的数据可用于许多使用案例，包括城市环境的安全系统中的警报，用于诸如枪击和爆炸之类的事件。此外，可以在低光照条件下或在相机不可用的位置处利用各种类型的传感器，诸如声音传感器。本文研究了声音传感器数据在城市环境中的应用潜力。在技术上，我们提出了一种新的方法来分类声音数据使用Wigner-Ville分布和卷积神经网络。在本文中，我们报告了该方法在开源数据集上的性能。本文所提出的概念和工作基于我的博士论文，该论文是作为埃因霍温大学数据科学工程博士学位项目的一部分，与荷兰国家警察合作完成的。在论文期间，对真实世界数据集进行了额外的工作，由于机密性，在此不作介绍。摘要：With big data becoming increasingly available, IoT hardware becoming widely adopted, and AI capabilities becoming more powerful, organizations are continuously investing in sensing. Data coming from sensor networks are currently combined with sensor fusion and AI algorithms to drive innovation in fields such as self-driving cars. Data from these sensors can be utilized in numerous use cases, including alerts in safety systems of urban settings, for events such as gun shots and explosions. Moreover, diverse types of sensors, such as sound sensors, can be utilized in low-light conditions or at locations where a camera is not available. This paper investigates the potential of the utilization of sound-sensor data in an urban context. Technically, we propose a novel approach of classifying sound data using the Wigner-Ville distribution and Convolutional Neural Networks. In this paper, we report on the performance of the approach on open-source datasets. The concept and work presented is based on my doctoral thesis, which was performed as part of the Engineering Doctorate program in Data Science at the University of Eindhoven, in collaboration with the Dutch National Police. Additional work on real-world datasets was performed during the thesis, which are not presented here due to confidentiality.
【4】 Going In Style: Audio Backdoors Through Stylistic Transformations
标题：走进风格：通过风格转换的音频后门
链接：https://arxiv.org/abs/2211.03117
作者：Stefanos Koffas,Luca Pajola,Stjepan Picek,Mauro Conti机构：Cybersecurity Group, Delft University of Technology, The Netherlands, Security and Privacy Research Group, University of Padua, Italy, Digital Security Group, Radboud University, The Netherlands, ∗Equal contribution摘要：后门攻击会在受害者的深度学习模型中放置触发器，以便在测试时实现有针对性的错误分类。通常，触发器是附加到样本的固定工件，使后门攻击易于发现。就在最近，人们提出了更难检测的新一代触发器：将风格变换应用于输入样本的风格触发器（例如，特定的书写风格）。 目前，文体学的后门文学缺乏一个恰当的形式化的攻击，本文建立了。而且，大多数关于文体触发的研究都集中在文本和图像上，而对它们是否能在声音中起作用却没有任何了解。这项工作填补了这一空白。我们提出了JingleBack，这是第一个基于音频变换（如合唱和增益）的风格后门攻击。在语音分类任务中使用444个模型，我们证实了音频中风格触发的可行性，实现了96%的攻击成功率。摘要：A backdoor attack places triggers in victims' deep learning models to enable a targeted misclassification at testing time. In general, triggers are fixed artifacts attached to samples, making backdoor attacks easy to spot. Only recently, a new trigger generation harder to detect has been proposed: the stylistic triggers that apply stylistic transformations to the input samples (e.g., a specific writing style). Currently, stylistic backdoor literature lacks a proper formalization of the attack, which is established in this paper. Moreover, most studies of stylistic triggers focus on text and images, while there is no understanding of whether they can work in sound. This work fills this gap. We propose JingleBack, the first stylistic backdoor attack based on audio transformations such as chorus and gain. Using 444 models in a speech classification task, we confirm the feasibility of stylistic triggers in audio, achieving 96% attack success.
【5】 I Hear Your True Colors: Image Guided Audio Generation
标题：我听到你的本色：图像引导的音频生成
链接：https://arxiv.org/abs/2211.03089
机构：School of Computer Science and Engineering, The Hebrew University of Jerusalem, Israel摘要：本文提出了一种图像引导的开放域音频生成系统Im 2 Wav。给定输入图像或图像序列，Im 2 Wav生成语义相关的声音。Im 2 Wav基于两个Transformer语言模型，其在从基于VQ-VAE的模型获得的分级离散音频表示上操作。我们首先使用语言模型产生低级音频表示。然后，我们使用附加的语言模型对音频令牌进行上采样，以生成高保真音频样本。我们使用预先训练的CLIP嵌入的丰富语义作为可视表示来调节语言模型。此外，为了将生成过程导向条件图像，我们应用了无分类器指导方法。结果表明，Im 2 Wav在保真度和相关性评价指标上均显著优于基线评价。此外，我们还提供了消融研究，以更好地评估每个方法组件对整体性能的影响。最后，为了更好地评估图像到音频模型，我们提出了一个域外图像数据集，表示为ImageHear。ImageHear可用作评估未来图像到音频模型的基准。样本和代码可以在手稿中找到。摘要：We propose Im2Wav, an image guided open-domain audio generation system. Given an input image or a sequence of images, Im2Wav generates a semantically relevant sound. Im2Wav is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-VAE based model. We first produce a low-level audio representation using a language model. Then, we upsample the audio tokens using an additional language model to generate a high-fidelity audio sample. We use the rich semantics of a pre-trained CLIP embedding as a visual representation to condition the language model. In addition, to steer the generation process towards the conditioning image, we apply the classifier-free guidance method. Results suggest that Im2Wav significantly outperforms the evaluated baselines in both fidelity and relevance evaluation metrics. Additionally, we provide an ablation study to better assess the impact of each of the method components on overall performance. Lastly, to better evaluate image-to-audio models, we propose an out-of-domain image dataset, denoted as ImageHear. ImageHear can be used as a benchmark for evaluating future image-to-audio models. Samples and code can be found inside the manuscript.
【6】 Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
标题：无监督ASR在语音和文本预训练模型之间的桥接
链接：https://arxiv.org/abs/2211.03025
作者：Jiatong Shi,Chan-Jan Hsu,Holam Chung,Dongji Gao,Paola Garcia,Shinji Watanabe,Ann Lee,Hung-yi Lee机构： Carnegie Mellon University, National Taiwan University, Johns Hopkins University, Meta AI摘要：口语理解是一项旨在从口语话语中提取高级语义的任务。先前的工作已经研究了语音自监督模型和文本预训练模型的使用，它们已经显示出对各种SLU任务的合理改进。然而，由于语音信号和文本标记之间的模态不匹配，以往的方法通常需要复杂的框架设计。本文提出了一种简单而有效的无监督范式，将语音和文本预训练模型连接起来，得到一个用于SLU中各种任务的无监督语音-语义预训练模型。具体地说，我们提出使用无监督自动语音识别（ASR）作为连接器，桥接语音和文本预训练模型中使用的不同模态。实验结果表明，无监督ASR本身可以改善语音自监督模型的表示。更重要的是，它被证明是语音和文本预训练模型之间的有效连接器，提高了五种不同的SLU任务的性能。值得注意的是，在口头问答方面，我们达到了最先进的结果，超过了具有挑战性的NMSQA基准。摘要：Spoken language understanding (SLU) is a task aiming to extract high-level semantics from spoken utterances. Previous works have investigated the use of speech self-supervised models and textual pre-trained models, which have shown reasonable improvements to various SLU tasks. However, because of the mismatched modalities between speech signals and text tokens, previous methods usually need complex designs of the frameworks. This work proposes a simple yet efficient unsupervised paradigm that connects speech and textual pre-trained models, resulting in an unsupervised speech-to-semantic pre-trained model for various tasks in SLU. To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models. Our experiments show that unsupervised ASR itself can improve the representations from speech self-supervised models. More importantly, it is shown as an efficient connector between speech and textual pre-trained models, improving the performances of five different SLU tasks. Notably, on spoken question answering, we reach the state-of-the-art result over the challenging NMSQA benchmark.
【7】 Effective Audio Classification Network Based on Paired Inverse Pyramid Structure and Dense MLP Block
标题：基于成对倒金字塔结构和密集MLP块的有效音频分类网络
链接：https://arxiv.org/abs/2211.02940
作者：Yunhao Chen,Yunjie Zhu,Zihui Yan,Lifang Chen机构：Jiangnan University, School of Artificial Intelligence and Computer Science, Wuxi, China, University of Leeds, School of Computing, Leeds, LS,JT, United Kingdom摘要：近年来，基于卷积神经网络（CNN）和自注意机制的大规模体系结构已经成为音频分类的必要条件。虽然这些技术是最先进的，但这些工作的有效性只能通过巨大的计算成本和参数、大量的数据扩增、从大数据集转移和一些其他技巧来保证。利用音频的轻量级特性，提出了一种高效的网络结构--成对逆金字塔结构（Paired Inverse Pyramid Structure，PIP）和成对逆金字塔结构MLP网络（Paired Inverse Pyramid Structure，MLP Network，PIPMN）。在UrbanSound 8 K数据集上，PIPMN的环境声音分类（ESC）准确率达到96%;在GTAZN数据集上，PIPMN的音乐类型分类（MGC）准确率达到93.2%，参数仅为100万.这两个结果都是在没有数据扩充或模型转移的情况下获得的。公共代码可从以下网址获得：https://github.com/JNAIC/PIPMN摘要：Recently, massive architectures based on Convolutional Neural Network (CNN) and self-attention mechanisms have become necessary for audio classification. While these techniques are state-of-the-art, these works' effectiveness can only be guaranteed with huge computational costs and parameters, large amounts of data augmentation, transfer from large datasets and some other tricks. By utilizing the lightweight nature of audio, we propose an efficient network structure called Paired Inverse Pyramid Structure (PIP) and a network called Paired Inverse Pyramid Structure MLP Network (PIPMN). The PIPMN reaches 96\% of Environmental Sound Classification (ESC) accuracy on the UrbanSound8K dataset and 93.2\% of Music Genre Classification (MGC) on the GTAZN dataset, with only 1 million parameters. Both of the results are achieved without data augmentation or model transfer. Public code is available at: https://github.com/JNAIC/PIPMN
【8】 VISinger 2: High-Fidelity End-to-End Singing Voice Synthesis Enhanced by Digital Signal Processing Synthesizer
标题：VISinger 2：由数字信号处理合成器增强的高保真端到端歌声合成
链接：https://arxiv.org/abs/2211.02903
作者：Yongmao Zhang,Heyang Xue,Hanzhao Li,Lei Xie,Tingwei Guo,Ruixiong Zhang,Caixia Gong机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, DiDi Chuxing, Beijing, China备注：Submitted to ICASSP 2023摘要：端到端的演唱语音合成（SVS）模型VISinger能够以较少的参数获得比典型的两阶段模型更好的性能。但是，VISinger有几个问题：文本到短语问题，端到端模型学习文本到短语的无意义映射;假信号问题，对应于有声段的周期性信号的谐波分量发生突然变化，具有可听伪像;采样率低，24 KHz的采样率不能满足全频段速率（44.1KHz或更高）高保真生成的应用需求。本文将数字信号处理（DSP）方法与VISinger相结合，提出了VISinger 2来解决这些问题。具体地说，受微分数字信号处理（DDSP）的最新进展的启发，我们将DSP合成器结合到解码器中以解决上述问题。DSP合成器由谐波合成器和噪声合成器组成，以分别从VISinger中的潜在表示z生成周期性和非周期性信号。它监督后级编码器提取没有相位信息的潜在表示，避免了前级编码器建模文本到相位的映射。为了避免假信号，HiFi-GAN被修改为接受由DSP合成器生成的波形作为产生歌声的条件。此外，通过改进的波形解码器，VISinger 2成功地生成了44.1kHz的歌唱音频，表现力更丰富，质量更好。在OpenCpop语料库上的实验表明，VISinger 2在主观和客观指标上都优于VISinger、CpopSing和RefineSinger。摘要：End-to-end singing voice synthesis (SVS) model VISinger can achieve better performance than the typical two-stage model with fewer parameters. However, VISinger has several problems: text-to-phase problem, the end-to-end model learns the meaningless mapping of text-to-phase; glitches problem, the harmonic components corresponding to the periodic signal of the voiced segment occurs a sudden change with audible artefacts; low sampling rate, the sampling rate of 24KHz does not meet the application needs of high-fidelity generation with the full-band rate (44.1KHz or higher). In this paper, we propose VISinger 2 to address these issues by integrating the digital signal processing (DSP) methods with VISinger. Specifically, inspired by recent advances in differentiable digital signal processing (DDSP), we incorporate a DSP synthesizer into the decoder to solve the above issues. The DSP synthesizer consists of a harmonic synthesizer and a noise synthesizer to generate periodic and aperiodic signals, respectively, from the latent representation z in VISinger. It supervises the posterior encoder to extract the latent representation without phase information and avoid the prior encoder modelling text-to-phase mapping. To avoid glitch artefacts, the HiFi-GAN is modified to accept the waveforms generated by the DSP synthesizer as a condition to produce the singing voice. Moreover, with the improved waveform decoder, VISinger 2 manages to generate 44.1kHz singing audio with richer expression and better quality. Experiments on OpenCpop corpus show that VISinger 2 outperforms VISinger, CpopSing and RefineSinger in both subjective and objective metrics.
【9】 LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers
标题：Lamassu：使用神经传感器的与流语言无关的多语言语音识别和翻译
链接：https://arxiv.org/abs/2211.02809
作者：Peidong Wang,Eric Sun,Jian Xue,Yu Wu,Long Zhou,Yashesh Gaur,Shujie Liu,Jinyu Li备注：Submitted to ICASSP 2023摘要：自动语音识别（ASR）和语音翻译（ST）的端到端公式化使得使用单个模型来同时处理多语言ASR和多对多ST变得容易。为了在LAMASSU中实现多语言文本生成，我们对指定和统一预测以及联合网络进行了系统比较。我们利用了一个与语言无关的多语言编码器，它的性能大大优于共享编码器。为了增强LAMASSU，我们建议将目标LID馈送到编码器。我们还将连接主义时间分类正则化应用于换能器训练。实验结果表明，LAMASSU不仅大大降低了模型的规模，而且性能优于单语ASR和双语ST模型。摘要：End-to-end formulation of automatic speech recognition (ASR) and speech translation (ST) makes it easy to use a single model for both multilingual ASR and many-to-many ST. In this paper, we propose streaming language-agnostic multilingual speech recognition and translation using neural transducers (LAMASSU). To enable multilingual text generation in LAMASSU, we conduct a systematic comparison between specified and unified prediction and joint networks. We leverage a language-agnostic multilingual encoder that substantially outperforms shared encoders. To enhance LAMASSU, we propose to feed target LID to encoders. We also apply connectionist temporal classification regularization to transducer training. Experimental results show that LAMASSU not only drastically reduces the model size but also outperforms monolingual ASR and bilingual ST models.
【10】 Fast and efficient speech enhancement with variational autoencoders
标题：利用可变自动编码器实现快速高效的语音增强
链接：https://arxiv.org/abs/2211.02728
作者：Mostafa Sadeghi,Romain Serizel机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：与常用的有监督语音增强方法相比，基于变分自动编码器的无监督语音增强方法具有较好的性能。该方法使用预先训练好的深度语音先验知识和参数噪声模型，其中噪声参数是通过基于期望最大化（EM）的方法从含噪语音信号中学习得到的。E-步骤涉及难以处理的潜在后验分布。现有的解决这一问题的算法要么是基于计算量大的蒙特卡罗马尔可夫链抽样方法和变分推理，要么是基于低效的优化方法。本文提出了一种基于Langevin动力学的新方法，该方法生成多个样本序列，并采用基于全变分的正则化方法来考虑潜在向量的时间相关性。实验结果表明，该算法在计算效率和增强质量之间取得了较好的折衷，优于现有算法.摘要：Unsupervised speech enhancement based on variational autoencoders has shown promising performance compared with the commonly used supervised methods. This approach involves the use of a pre-trained deep speech prior along with a parametric noise model, where the noise parameters are learned from the noisy speech signal with an expectationmaximization (EM)-based method. The E-step involves an intractable latent posterior distribution. Existing algorithms to solve this step are either based on computationally heavy Monte Carlo Markov Chain sampling methods and variational inference, or inefficient optimization-based methods. In this paper, we propose a new approach based on Langevin dynamics that generates multiple sequences of samples and comes with a total variation-based regularization to incorporate temporal correlations of latent vectors. Our experiments demonstrate that the developed framework makes an effective compromise between computational efficiency and enhancement quality, and outperforms existing methods.
【11】 Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion
标题：基于分层特征融合的语音基础模型资源高效转移学习
链接：https://arxiv.org/abs/2211.02712
作者：Zhouyuan Huo,Khe Chai Sim,Bo Li,Dongseong Hwang,Tara N. Sainath,Trevor Strohman摘要：语音基础模型的自监督预训练，随后的监督微调，已经在自动语音识别（ASR）任务上显示出令人印象深刻的质量改进。由于基础模型通常非常大，因此为许多下游任务微调单独的基础模型是昂贵的。参数高效的微调方法（例如适配器、稀疏更新方法）提供了一种替代范例，其中更新一小组参数以使基础模型适应新任务。然而，这些方法仍然遭受高的计算存储器成本和慢的训练速度，因为它们在每一步需要通过整个神经网络的反向传播。本文分析了基础模型不同层次特征在语音识别任务中的表现，提出了一种新的分层特征融合方法，用于语音基础模型的资源有效迁移学习。实验结果表明，与现有算法相比，该方法能够以更少的可训练参数、更少的计算内存开销和更快的训练速度获得更好的语音识别性能。在各层结合Adapters后，该方法可达到与微调整个模型相同的性能，但可训练的编码器参数减少了97%，训练速度提高了53%.摘要：Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with $97\%$ fewer trainable encoder parameters and $53\%$ faster training speed.
【12】 ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech
标题：Ernie-SAT：跨语言多说话人文语转换的语音和文本联合预训练
链接：https://arxiv.org/abs/2211.03545
作者：Xiaoran Fan,Chao Pang,Tian Yuan,He Bai,Renjie Zheng,Pengfei Zhu,Shuohuan Wang,Junkun Chen,Zeyu Chen,Liang Huang,Yu Sun,Hua Wu机构：⋆Baidu Inc., Beijing, China, University of Chinese Academy of Sciences, Beijing, China, University of Waterloo, Waterloo, ON, Canada, Oregon State University, Corvallis, OR, USA摘要：语音表示学习提高了单语言的语音理解和语音合成任务。然而，它在跨语言场景中的能力还没有被探索。本文将预训练方法扩展到跨语言多说话人语音合成任务，包括跨语言多说话人语音克隆和跨语言多说话人语音编辑。我们提出一个语音-文本联合预训练框架，其中我们随机屏蔽声谱图和音素给定的语音例子和它的转录。通过学习以不同语言重构输入的掩蔽部分，我们的模型显示了比基于说话人嵌入的多说话人TTS方法的巨大改进。此外，我们的框架是端到端的训练和推理，而不需要任何微调工作。在跨语言多说话人语音克隆和跨语言多说话人语音编辑任务中，我们的实验表明，我们的模型优于基于说话人嵌入的多说话人TTS方法。代码和模型可在PaddleSpeech上公开获得。摘要：Speech representation learning has improved both speech understanding and speech synthesis tasks for single language. However, its ability in cross-lingual scenarios has not been explored. In this paper, we extend the pretraining method for cross-lingual multi-speaker speech synthesis tasks, including cross-lingual multi-speaker voice cloning and cross-lingual multi-speaker speech editing. We propose a speech-text joint pretraining framework, where we randomly mask the spectrogram and the phonemes given a speech example and its transcription. By learning to reconstruct the masked parts of the input in different languages, our model shows great improvements over speaker-embedding-based multi-speaker TTS methods. Moreover, our framework is end-to-end for both the training and the inference without any finetuning effort. In cross-lingual multi-speaker voice cloning and cross-lingual multi-speaker speech editing tasks, our experiments show that our model outperforms speaker-embedding-based multi-speaker TTS methods. The code and model are publicly available at PaddleSpeech.
【13】 Multi-blank Transducers for Speech Recognition
标题：用于语音识别的多空白换能器
链接：https://arxiv.org/abs/2211.03541
作者：Hainan Xu,Fei Jia,Somshubra Majumdar,Shinji Watanabe,Boris Ginsburg机构：NVIDIA, USA, Carnegie Mellon University, PA, USA备注：Submitted to ICASSP 2023摘要：本文提出了一种用于自动语音识别的RNN-T模型的改进算法。在标准RNN-T中，空白符号的发射正好消耗一个输入帧;在我们提出的方法中，我们引入了附加的消隐符号，其在发射时消耗两个或更多个输入帧。我们将添加的符号称为大空白，并且将方法称为多空白RNN-T。针对多空白RNN-Ts的训练问题，提出了一种新的logit欠归一化方法，用于区分大空白的发射优先级.在多种语言和数据集上的实验结果表明，多空白RNN-T方法在英语Librispeech和德语Multilingual Librispeech数据集上的模型推理相对加速比分别达到了+90%/+139%。多空白RNN-T方法还可持续提高ASR准确度。我们将在NeMo（\url{https：//github.com/NVIDIA/NeMo}）工具包中发布该方法的实现。摘要：This paper proposes a modification to RNN-Transducer (RNN-T) models for automatic speech recognition (ASR). In standard RNN-T, the emission of a blank symbol consumes exactly one input frame; in our proposed method, we introduce additional blank symbols, which consume two or more input frames when emitted. We refer to the added symbols as big blanks, and the method multi-blank RNN-T. For training multi-blank RNN-Ts, we propose a novel logit under-normalization method in order to prioritize emissions of big blanks. With experiments on multiple languages and datasets, we show that multi-blank RNN-T methods could bring relative speedups of over +90%/+139% to model inference for English Librispeech and German Multilingual Librispeech datasets, respectively. The multi-blank RNN-T method also improves ASR accuracy consistently. We will release our implementation of the method in the NeMo (\url{https://github.com/NVIDIA/NeMo}) toolkit.
【14】 Accented Text-to-Speech Synthesis with a Conditional Variational Autoencoder
标题：基于条件变分自动编码器的重音文语合成
链接：https://arxiv.org/abs/2211.03316
作者：Jan Melechovsky,Ambuj Mehrish,Berrak Sisman,Dorien Herremans机构：Singapore University of Technology and Design, Singapore, Speech & Machine Learning Lab, The University of Texas at Dallas, Texas, USA备注：preprint submitted to a conference, under review摘要：口音在言语交际中起着重要的作用，影响着理解能力，也传达着一个人的身份。本文介绍了一种基于条件变分自动编码器的重音合成框架。它具有合成被转换成任何期望的目标口音的所选说话者的语音的能力。通过实验验证了该框架的有效性。实验结果表明，该算法在处理合成语音中重音的能力方面也表现出了显著的性能，为今后重音TTS的研究提供了一条有前景的途径。摘要：Accent plays a significant role in speech communication, influencing understanding capabilities and also conveying a person's identity. This paper introduces a novel and efficient framework for accented Text-to-Speech (TTS) synthesis based on a Conditional Variational Autoencoder. It has the ability to synthesize a selected speaker's speech that is converted to any desired target accent. Our thorough experiments validate the effectiveness of our proposed framework using both objective and subjective evaluations. The results also show remarkable performance in terms of the ability to manipulate accents in the synthesized speech and provide a promising avenue for future accented TTS research.
【15】 Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization
标题：峰值优先CTC：应用峰值优先正则化降低CTC模型的峰值潜伏期
链接：https://arxiv.org/abs/2211.03284
作者：Zhengkun Tian,Hongyu Xiang,Min Li,Feifei Lin,Ke Ding,Guanglu Wan机构：Meituan, Beijing, China备注：Submitted to ICASSP 2023(5 pages, 2 figures)摘要：CTC模型以其结构简单、性能优良、推理速度快等优点，被广泛应用于许多应用场景。在CTC模型预测的概率分布中存在许多峰值，每个峰值代表一个非空白标记。通过鼓励CTC模型更早地预测峰值，可以减少CTC模型的识别延迟。现有的减少延迟的方法需要修改前向-后向算法中令牌之间的转换关系以及梯度计算。这些方法中的一些甚至依赖于由其它预训练模型提供的强制对准结果。上述方法实现起来很复杂。为了减少峰值延迟，提出了一种简单新颖的峰值优先正则化方法，该方法利用逐帧的知识提取函数，迫使CTC模型的概率分布沿时间轴左移，而不是直接修改CTC丢失和梯度的计算过程.所有的实验都是在中文数据集AISHELL-1上进行的。我们分别在流式和非流式CTC模型上验证了所提出的正则化方法的有效性。实验结果表明，该方法可以在不降低识别率的情况下，将平均峰值延迟降低约100 ~ 200 ms。摘要：The CTC model has been widely applied to many application scenarios because of its simple structure, excellent performance, and fast inference speed. There are many peaks in the probability distribution predicted by the CTC models, and each peak represents a non-blank token. The recognition latency of CTC models can be reduced by encouraging the model to predict peaks earlier. Existing methods to reduce latency require modifying the transition relationship between tokens in the forward-backward algorithm, and the gradient calculation. Some of these methods even depend on the forced alignment results provided by other pretrained models. The above methods are complex to implement. To reduce the peak latency, we propose a simple and novel method named peak-first regularization, which utilizes a frame-wise knowledge distillation function to force the probability distribution of the CTC model to shift left along the time axis instead of directly modifying the calculation process of CTC loss and gradients. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. We have verified the effectiveness of the proposed regularization on both streaming and non-streaming CTC models respectively. The results show that the proposed method can reduce the average peak latency by about 100 to 200 milliseconds with almost no degradation of recognition accuracy.
【16】 Robust Total Least Mean M-Estimate normalized subband filter Adaptive Algorithm for impulse noises and noisy inputs
标题：用于脉冲噪声和噪声输入的稳健总体最小均值M估计归一化子带滤波自适应算法
链接：https://arxiv.org/abs/2211.03283
作者：Haiquan Zhao,Zian Cao,Yida Chen机构：Magnetic Suspension Technology and Maglev Vehicle, Ministry of Education, and the School of Electrical Engineering, Southwest Jiaotong University, Chengdu, China., (e-mail:, interference, the convergence performance of the traditional摘要：当输入信号为相关输入信号，且输入输出信号均受高斯噪声污染时，总体最小二乘归一化子带自适应滤波（TLS-NSAF）算法表现出良好的性能。然而，当受到脉冲噪声干扰时，TLS-NSAF算法的收敛性能急剧恶化。针对这一问题，提出了一种鲁棒的总体最小均值M估计归一化子带滤波（TLMM-NSAF）算法。此外，本文还对TLMM-NSAF算法进行了详细的理论性能分析，得到了该算法的稳定步长范围和理论稳态均方差（MSD）。为了进一步提高算法的性能，本文还提出了一种新的变步长算法。最后，通过不同噪声模型下的系统辨识和回波抵消的计算机仿真，验证了算法的鲁棒性和理论值与仿真值的一致性。摘要：When the input signal is correlated input signals, and the input and output signal is contaminated by Gaussian noise, the total least squares normalized subband adaptive filter (TLS-NSAF) algorithm shows good performance. However, when it is disturbed by impulse noise, the TLS-NSAF algorithm shows the rapidly deteriorating convergence performance. To solve this problem, this paper proposed the robust total minimum mean M-estimator normalized subband filter (TLMM-NSAF) algorithm. In addition, this paper also conducts a detailed theoretical performance analysis of the TLMM-NSAF algorithm and obtains the stable step size range and theoretical steady-state mean squared deviation (MSD) of the algorithm. To further improve the performance of the algorithm, we also propose a new variable step size (VSS) method of the algorithm. Finally, the robustness of our proposed algorithm and the consistency of theoretical and simulated values are verified by computer simulations of system identification and echo cancellation under different noise models.
【17】 A Context-Aware Computational Approach for Measuring Vocal Entrainment in Dyadic Conversations
标题：一种测量二元对话中发声训练的上下文感知计算方法
链接：https://arxiv.org/abs/2211.03279
作者：Rimita Lahiri,Md Nasir,Catherine Lord,So Hyun Kim,Shrikanth Narayanan机构：Signal Analysis and Interpretation Laboratory, University of Southern California, Los Angeles, USA, Microsoft AI for Good Research Lab, Redmond, Washington, USA, Semel Institute of Neuroscience and Human Behavior, University of California, Los Angeles, USA摘要：声音诱导是人类交往中的一种社会适应机制，对它的认识可以为个体的认知行为特征提供有益的启示。我们提出一种上下文感知的方法来测量二人会话中的语音夹带。我们使用构象器（卷积网络和转换器的组合）来捕获短期和长期会话上下文，以模拟跨不同域的交互中的夹带模式。具体来说，我们使用跨主题注意层来学习来自二元对话的个人内部和个人之间的信号。我们首先基于分类实验来验证所提出的方法，以区分真实（一致）和虚假（不一致/混洗）会话。关于自闭症谱系障碍患者相互作用的实验结果也表明，引入的诱导测量和与症状相关的临床评分之间存在统计学显著相关性，包括跨性别和年龄组。摘要：Vocal entrainment is a social adaptation mechanism in human interaction, knowledge of which can offer useful insights to an individual's cognitive-behavioral characteristics. We propose a context-aware approach for measuring vocal entrainment in dyadic conversations. We use conformers(a combination of convolutional network and transformer) for capturing both short-term and long-term conversational context to model entrainment patterns in interactions across different domains. Specifically we use cross-subject attention layers to learn intra- as well as inter-personal signals from dyadic conversations. We first validate the proposed method based on classification experiments to distinguish between real(consistent) and fake(inconsistent/shuffled) conversations. Experimental results on interactions involving individuals with Autism Spectrum Disorder also show evidence of a statistically-significant association between the introduced entrainment measure and clinical scores relevant to symptoms, including across gender and age groups.
【18】 An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Space
标题：基于元音空间的跨语言文语转换系统二语重音的实证研究
链接：https://arxiv.org/abs/2211.03078
作者：Jihwan Lee,Jae-Sung Bae,Seongkyu Mun,Heejin Choi,Joun Yeop Lee,Hoon-Young Cho,Chanwoo Kim机构：Samsung Research, Seoul, South Korea备注：Submitted to ICASSP 2023摘要：随着跨语言文语转换（TTS）系统的发展，第二语言（外语）重音问题日益突出。此外，对这种跨语言TTS系统进行主观评估是麻烦的。元音空间分析是一个很好的分析工具，它经常被用来研究语言的各个方面，包括L2重音。本研究运用元音空间分析方法，探讨跨语言TTS系统中的二语重音。通过元音空间分析，我们观察到以下三点：a）并行体系结构（Glow-TTS）比自回归体系结构（Tacotron）更少L2重音;（2）L2重音在语言对中非共有元音中占优势;跨语言TTS系统的L2重音与人类L2学习者的L2重音有一些共同的现象。我们的研究结果表明，TTS系统有必要根据语言对的语言特征（如非共享元音）对它们进行不同的处理。这也暗示我们可以进一步结合语言学知识来开发跨语言的TTS系统。摘要：With the recent developments in cross-lingual Text-to-Speech (TTS) systems, L2 (second-language, or foreign) accent problems arise. Moreover, running a subjective evaluation for such cross-lingual TTS systems is troublesome. The vowel space analysis, which is often utilized to explore various aspects of language including L2 accents, is a great alternative analysis tool. In this study, we apply the vowel space analysis method to explore L2 accents of cross-lingual TTS systems. Through the vowel space analysis, we observe the three followings: a) a parallel architecture (Glow-TTS) is less L2-accented than an auto-regressive one (Tacotron); b) L2 accents are more dominant in non-shared vowels in a language pair; and c) L2 accents of cross-lingual TTS systems share some phenomena with those of human L2 learners. Our findings imply that it is necessary for TTS systems to handle each language pair differently, depending on their linguistic characteristics such as non-shared vowels. They also hint that we can further incorporate linguistics knowledge in developing cross-lingual TTS systems.
【19】 Distinguishable Speaker Anonymization based on Formant and Fundamental Frequency Scaling
标题：基于共振峰和基频变换的可区分说话人匿名
链接：https://arxiv.org/abs/2211.03038
作者：Jixun Yao,Qing Wang,Yi Lei,Pengcheng Guo,Lei Xie,Namin Wang,Jie Liu机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Huawei Cloud备注：Submitted to ICASSP 2023摘要：由于社交媒体的出现，互联网上的语音数据呈指数级增长，共享此类个人数据引发了明显的安全和隐私问题。减轻这些问题的一种解决方案涉及在共享语音数据之前隐藏说话者身份，也称为说话者匿名化。在我们以前的工作中，我们已经开发了一个自动说话人确认（ASV）-无模型匿名化框架，以保护说话人隐私，同时保持语音可懂度。虽然该框架在VoicePrivacy 2022挑战赛中排名第一，但由于匿名语音的说话人可区分性下降，使得匿名化不完善。为了解决这一问题，本文直接对共振峰分布和基频（F0）进行建模来表示说话人身份，并通过均匀缩放共振峰和F0来实现源语音的匿名化。通过直接缩放共振峰和F0，防止了由于引入其他说话人而导致的匿名语音的说话人可分辨性下降。实验结果表明，本文提出的框架能够提高说话人区分度，在语音区分度方面明显优于已有的框架。此外，该方法还可以通过使用不同的缩放因子来实现隐私与效用的平衡.摘要：Speech data on the Internet are proliferating exponentially because of the emergence of social media, and the sharing of such personal data raises obvious security and privacy concerns. One solution to mitigate these concerns involves concealing speaker identities before sharing speech data, also referred to as speaker anonymization. In our previous work, we have developed an automatic speaker verification (ASV)-model-free anonymization framework to protect speaker privacy while preserving speech intelligibility. Although the framework ranked first place in VoicePrivacy 2022 challenge, the anonymization was imperfect, since the speaker distinguishability of the anonymized speech was deteriorated. To address this issue, in this paper, we directly model the formant distribution and fundamental frequency (F0) to represent speaker identity and anonymize the source speech by the uniformly scaling formant and F0. By directly scaling the formant and F0, the speaker distinguishability degradation of the anonymized speech caused by the introduction of other speakers is prevented. The experimental results demonstrate that our proposed framework can improve the speaker distinguishability and significantly outperforms our previous framework in voice distinctiveness. Furthermore, our proposed method also can trade off the privacy-utility by using different scaling factors.
【20】 Preserving background sound in noise-robust voice conversion via multi-task learning
标题：在噪声中保留背景声音-通过多任务学习实现稳健的语音转换
链接：https://arxiv.org/abs/2211.03036
作者：Jixun Yao,Yi Lei,Qing Wang,Pengcheng Guo,Ziqian Ning,Lei Xie,Hai Li,Junhui Liu,Danming Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, QIYI Inc, China备注：Submitted to ICASSP 2023摘要：背景声音是一种信息性的艺术形式，有助于在真实应用程序语音转换（VC）场景中提供更身临其境的体验。然而，以往对VC的研究主要集中在纯净语音上，对带背景音的VC研究较少。VC中保留背景音的关键问题是神经元分离模型不可避免的语音失真以及源分离模型与VC模型之间的级联失配。本文提出了一种基于多任务学习的端到端框架，该框架由源分离模块、瓶颈特征提取模块和VC模块依次级联而成。具体而言，源分离任务明确地考虑了临界相位信息，并限制了由不完美分离过程引起的失真。通过联合训练约束，源分离任务、典型VC任务和统一任务共享统一的重构损失，以减少SS和VC模块之间的失配。实验结果表明，本文提出的框架显著优于基线系统，同时获得了与使用干净数据训练的VC模型相当的质量和说话人相似度。摘要：Background sound is an informative form of art that is helpful in providing a more immersive experience in real-application voice conversion (VC) scenarios. However, prior research about VC, mainly focusing on clean voices, pay rare attention to VC with background sound. The critical problem for preserving background sound in VC is inevitable speech distortion by the neural separation model and the cascade mismatch between the source separation model and the VC model. In this paper, we propose an end-to-end framework via multi-task learning which sequentially cascades a source separation (SS) module, a bottleneck feature extraction module and a VC module. Specifically, the source separation task explicitly considers critical phase information and confines the distortion caused by the imperfect separation process. The source separation task, the typical VC task and the unified task shares a uniform reconstruction loss constrained by joint training to reduce the mismatch between the SS and VC modules. Experimental results demonstrate that our proposed framework significantly outperforms the baseline systems while achieving comparable quality and speaker similarity to the VC models trained with clean data.
【21】 Breaking the trade-off in personalized speech enhancement with cross-task knowledge distillation
标题：用跨任务知识提炼打破个性化语音增强中的权衡
链接：https://arxiv.org/abs/2211.02944
作者：Hassan Taherian,Sefik Emre Eskimez,Takuya Yoshioka机构：Microsoft, Redmond, WA, USA, The Ohio State University, Columbus, OH, USA备注：Submitted to ICASSP 2023摘要：与无条件语音增强模型相比，个性化语音增强模型由于能够去除背景噪声和干扰语音而取得了较好的效果。与无条件语音增强不同，因果PSE模型可能偶尔错误地去除目标语音。当目标说话人在延长的时段内沉默时，PSE模型也倾向于泄漏干扰语音。我们表明，现有的PSE方法在语音过抑制和干扰泄漏之间进行了权衡，解决了一个问题，而牺牲了另一个问题。我们提出一个新的PSE模型训练框架，使用跨任务的知识提取来缓解这种折衷。具体地，我们在训练期间利用个性化语音活动检测器（pVAD）来排除被错误地识别为包含具有硬分类或软分类的目标说话者的非目标语音帧。这防止了PSE模型过于激进，同时仍然允许模型学习以在输入语音可能由干扰说话者说出时抑制输入语音。给出了综合评估结果，涵盖了各种PSE使用场景。摘要：Personalized speech enhancement (PSE) models achieve promising results compared with unconditional speech enhancement models due to their ability to remove interfering speech in addition to background noise. Unlike unconditional speech enhancement, causal PSE models may occasionally remove the target speech by mistake. The PSE models also tend to leak interfering speech when the target speaker is silent for an extended period. We show that existing PSE methods suffer from a trade-off between speech over-suppression and interference leakage by addressing one problem at the expense of the other. We propose a new PSE model training framework using cross-task knowledge distillation to mitigate this trade-off. Specifically, we utilize a personalized voice activity detector (pVAD) during training to exclude the non-target speech frames that are wrongly identified as containing the target speaker with hard or soft classification. This prevents the PSE model from being too aggressive while still allowing the model to learn to suppress the input speech when it is likely to be spoken by interfering speakers. Comprehensive evaluation results are presented, covering various PSE usage scenarios.
【22】 Real-Time Joint Personalized Speech Enhancement and Acoustic Echo Cancellation with E3Net
标题：基于E3Net的实时联合个性化语音增强和声回波消除
链接：https://arxiv.org/abs/2211.02773
作者：Sefik Emre Eskimez,Takuya Yoshioka,Alex Ju,Min Tang,Tanel Parnamaa,Huaming Wang机构：Microsoft, Redmond, WA, USA备注：Submitted to ICASSP 23摘要：个性化语音增强（Personalized Speech Enhancement，PSE）是一种利用目标说话人的说话人嵌入向量实时估计干净的目标语音信号的方法，近年来随着全球在线会议的兴起而引起了研究界的广泛关注。对于实际的全双工通信，PSE模型需要声学回声消除（AEC）能力。在本文中，我们采用了最近提出的因果端到端增强网络（E3 Net），并对其进行了改进，得到了一个联合的PSE-AEC模型.我们将早期层专用于AEC任务，同时通过添加从早期层到掩码预测层的旁路连接来鼓励后期层进行个性化。这使我们能够采用多任务学习框架进行PSE和AEC联合培训。我们提供了大量的评估测试场景，包括模拟和真实世界的记录。实验结果表明，对于每个任务，我们的联合模型都接近专家模型，并且对于组合的PSE-AEC场景，我们的联合模型的性能明显更好。摘要：Personalized speech enhancement (PSE), a process of estimating a clean target speech signal in real time by leveraging a speaker embedding vector of the target talker, has garnered much attention from the research community due to the recent surge of online meetings across the globe. For practical full duplex communication, PSE models require an acoustic echo cancellation (AEC) capability. In this work, we employ a recently proposed causal end-to-end enhancement network (E3Net) and modify it to obtain a joint PSE-AEC model. We dedicate the early layers to the AEC task while encouraging later layers for personalization by adding a bypass connection from the early layers to the mask prediction layer. This allows us to employ a multi-task learning framework for joint PSE and AEC training. We provide extensive evaluation test scenarios with both simulated and real-world recordings. The results show that our joint model comes close to the expert models for each task and performs significantly better for the combined PSE-AEC scenario.
【23】 SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofing
标题：SAMO：用于语音反欺骗的扬声器吸引子多中心单类学习
链接：https://arxiv.org/abs/2211.02718
作者：Siwen Ding,You Zhang,Zhiyao Duan机构：Columbia University, New York, NY, USA, University of Rochester, Rochester, NY, USA摘要：语音反欺骗系统是自动说话人确认（ASV）系统的重要辅助设备。一个主要的挑战是由先进的语音合成技术支持的不可见的攻击造成的。我们以前的单类学习研究通过压缩真实语音在嵌入空间中的位置，提高了对不可见攻击的泛化能力。然而，这种紧凑性缺乏对说话者多样性的考虑。本文提出一种说话人吸引子多中心单类学习算法（SAMO），该算法将真实语音聚集在多个说话人吸引子周围，并在高维嵌入空间中从所有吸引子中排除欺骗攻击.在训练方面，我们提出了一个语音聚类和语音分类的联合优化算法。对于推理，我们提出了一些策略来使没有注册的说话者能够反欺骗。在ASVspoof 2019 LA评估集上，我们提出的系统在等错误率（EER）方面相对改进了38%，优于现有的最先进的单系统。摘要：Voice anti-spoofing systems are crucial auxiliaries for automatic speaker verification (ASV) systems. A major challenge is caused by unseen attacks empowered by advanced speech synthesis technologies. Our previous research on one-class learning has improved the generalization ability to unseen attacks by compacting the bona fide speech in the embedding space. However, such compactness lacks consideration of the diversity of speakers. In this work, we propose speaker attractor multi-center one-class learning (SAMO), which clusters bona fide speech around a number of speaker attractors and pushes away spoofing attacks from all the attractors in a high-dimensional embedding space. For training, we propose an algorithm for the co-optimization of bona fide speech clustering and bona fide/spoof classification. For inference, we propose strategies to enable anti-spoofing for speakers without enrollment. Our proposed system outperforms existing state-of-the-art single systems with a relative improvement of 38% on equal error rate (EER) on the ASVspoof2019 LA evaluation set.
【24】 Speech enhancement using ego-noise references with a microphone array embedded in an unmanned aerial vehicle
标题：利用嵌入无人机中的麦克风阵列的自噪声参考的语音增强
链接：https://arxiv.org/abs/2211.02690
作者：Elisa Tengan,Thomas Dietzen,Santiago Ruiz,Mansour Alkmim,João Cardenuto,Toon van Waterschoot摘要：提出了一种基于麦克风阵列的自噪声语音增强方法。利用位于无人机螺旋桨附近的麦克风捕获自我噪声参考信号，并将其用于先验知识多通道维纳滤波器（PK-MWF）中，以获得语音相关矩阵估计。可以估计语音存在概率（SPP），以用于检测来自语音源附近的外部麦克风的语音活动，从而提供性能基准，或者用于检测来自嵌入式麦克风之一的语音活动，从而假设更现实的场景。实验测量是在半消声室中进行的，无人机安装在支架上，扬声器播放语音信号，同时设置三个不同的和固定的螺旋桨转速，导致三个不同的信噪比（SNR）。使用在线获得的记录，将所提出的方法与使用标准多通道维纳滤波器（MWF）进行比较，在公式中使用和不使用螺旋桨麦克风进行估计。结果表明，与那些相比，使用PK-MWF在通过STOI和PESQ测量的语音可懂度和质量方面实现了更高水平的改善，而SNR改善相似。摘要：A method is proposed for performing speech enhancement using ego-noise references with a microphone array embedded in an unmanned aerial vehicle (UAV). The ego-noise reference signals are captured with microphones located near the UAV's propellers and used in the prior knowledge multichannel Wiener filter (PK-MWF) to obtain the speech correlation matrix estimate. Speech presence probability (SPP) can be estimated for detecting speech activity from an external microphone near the speech source, providing a performance benchmark, or from one of the embedded microphones, assuming a more realistic scenario. Experimental measurements are performed in a semi-anechoic chamber, with a UAV mounted on a stand and a loudspeaker playing a speech signal, while setting three distinct and fixed propeller rotation speeds, resulting in three different signal-to-noise ratios (SNRs). The recordings obtained and made available online are used to compare the proposed method to the use of the standard multichannel Wiener filter (MWF) estimated with and without the propellers' microphones being used in its formulation. Results show that compared to those, the use of PK-MWF achieves higher levels of improvement in speech intelligibility and quality, measured by STOI and PESQ, while the SNR improvement is similar.【1】 ERNIE-SAT: Speech and Text Joint Pretraining for Cross-Lingual Multi-Speaker Text-to-Speech标题：Ernie-SAT：跨语言多说话人文语转换的语音和文本联合预训练
链接：https://arxiv.org/abs/2211.03545
* 与cs.SD语音【12】为同一篇
作者：Xiaoran Fan,Chao Pang,Tian Yuan,He Bai,Renjie Zheng,Pengfei Zhu,Shuohuan Wang,Junkun Chen,Zeyu Chen,Liang Huang,Yu Sun,Hua Wu机构：⋆Baidu Inc., Beijing, China, University of Chinese Academy of Sciences, Beijing, China, University of Waterloo, Waterloo, ON, Canada, Oregon State University, Corvallis, OR, USA摘要：语音表示学习提高了单语言的语音理解和语音合成任务。然而，它在跨语言场景中的能力还没有被探索。本文将预训练方法扩展到跨语言多说话人语音合成任务，包括跨语言多说话人语音克隆和跨语言多说话人语音编辑。我们提出一个语音-文本联合预训练框架，其中我们随机屏蔽声谱图和音素给定的语音例子和它的转录。通过学习以不同语言重构输入的掩蔽部分，我们的模型显示了比基于说话人嵌入的多说话人TTS方法的巨大改进。此外，我们的框架是端到端的训练和推理，而不需要任何微调工作。在跨语言多说话人语音克隆和跨语言多说话人语音编辑任务中，我们的实验表明，我们的模型优于基于说话人嵌入的多说话人TTS方法。代码和模型可在PaddleSpeech上公开获得。摘要：Speech representation learning has improved both speech understanding and speech synthesis tasks for single language. However, its ability in cross-lingual scenarios has not been explored. In this paper, we extend the pretraining method for cross-lingual multi-speaker speech synthesis tasks, including cross-lingual multi-speaker voice cloning and cross-lingual multi-speaker speech editing. We propose a speech-text joint pretraining framework, where we randomly mask the spectrogram and the phonemes given a speech example and its transcription. By learning to reconstruct the masked parts of the input in different languages, our model shows great improvements over speaker-embedding-based multi-speaker TTS methods. Moreover, our framework is end-to-end for both the training and the inference without any finetuning effort. In cross-lingual multi-speaker voice cloning and cross-lingual multi-speaker speech editing tasks, our experiments show that our model outperforms speaker-embedding-based multi-speaker TTS methods. The code and model are publicly available at PaddleSpeech.
【2】 Multi-blank Transducers for Speech Recognition
标题：用于语音识别的多空白换能器
链接：https://arxiv.org/abs/2211.03541
* 与cs.SD语音【13】为同一篇
作者：Hainan Xu,Fei Jia,Somshubra Majumdar,Shinji Watanabe,Boris Ginsburg机构：NVIDIA, USA, Carnegie Mellon University, PA, USA备注：Submitted to ICASSP 2023摘要：本文提出了一种用于自动语音识别的RNN-T模型的改进算法。在标准RNN-T中，空白符号的发射正好消耗一个输入帧;在我们提出的方法中，我们引入了附加的消隐符号，其在发射时消耗两个或更多个输入帧。我们将添加的符号称为大空白，并且将方法称为多空白RNN-T。针对多空白RNN-Ts的训练问题，提出了一种新的logit欠归一化方法，用于区分大空白的发射优先级.在多种语言和数据集上的实验结果表明，多空白RNN-T方法在英语Librispeech和德语Multilingual Librispeech数据集上的模型推理相对加速比分别达到了+90%/+139%。多空白RNN-T方法还可持续提高ASR准确度。我们将在NeMo（\url{https：//github.com/NVIDIA/NeMo}）工具包中发布该方法的实现。摘要：This paper proposes a modification to RNN-Transducer (RNN-T) models for automatic speech recognition (ASR). In standard RNN-T, the emission of a blank symbol consumes exactly one input frame; in our proposed method, we introduce additional blank symbols, which consume two or more input frames when emitted. We refer to the added symbols as big blanks, and the method multi-blank RNN-T. For training multi-blank RNN-Ts, we propose a novel logit under-normalization method in order to prioritize emissions of big blanks. With experiments on multiple languages and datasets, we show that multi-blank RNN-T methods could bring relative speedups of over +90%/+139% to model inference for English Librispeech and German Multilingual Librispeech datasets, respectively. The multi-blank RNN-T method also improves ASR accuracy consistently. We will release our implementation of the method in the NeMo (\url{https://github.com/NVIDIA/NeMo}) toolkit.
【3】 Accented Text-to-Speech Synthesis with a Conditional Variational Autoencoder
标题：基于条件变分自动编码器的重音文语合成
链接：https://arxiv.org/abs/2211.03316
* 与cs.SD语音【14】为同一篇
作者：Jan Melechovsky,Ambuj Mehrish,Berrak Sisman,Dorien Herremans机构：Singapore University of Technology and Design, Singapore, Speech & Machine Learning Lab, The University of Texas at Dallas, Texas, USA备注：preprint submitted to a conference, under review摘要：口音在言语交际中起着重要的作用，影响着理解能力，也传达着一个人的身份。本文介绍了一种基于条件变分自动编码器的重音合成框架。它具有合成被转换成任何期望的目标口音的所选说话者的语音的能力。通过实验验证了该框架的有效性。实验结果表明，该算法在处理合成语音中重音的能力方面也表现出了显著的性能，为今后重音TTS的研究提供了一条有前景的途径。摘要：Accent plays a significant role in speech communication, influencing understanding capabilities and also conveying a person's identity. This paper introduces a novel and efficient framework for accented Text-to-Speech (TTS) synthesis based on a Conditional Variational Autoencoder. It has the ability to synthesize a selected speaker's speech that is converted to any desired target accent. Our thorough experiments validate the effectiveness of our proposed framework using both objective and subjective evaluations. The results also show remarkable performance in terms of the ability to manipulate accents in the synthesized speech and provide a promising avenue for future accented TTS research.
【4】 Peak-First CTC: Reducing the Peak Latency of CTC Models by Applying Peak-First Regularization
标题：峰值优先CTC：应用峰值优先正则化降低CTC模型的峰值潜伏期
链接：https://arxiv.org/abs/2211.03284
* 与cs.SD语音【15】为同一篇
作者：Zhengkun Tian,Hongyu Xiang,Min Li,Feifei Lin,Ke Ding,Guanglu Wan机构：Meituan, Beijing, China备注：Submitted to ICASSP 2023(5 pages, 2 figures)摘要：CTC模型以其结构简单、性能优良、推理速度快等优点，被广泛应用于许多应用场景。在CTC模型预测的概率分布中存在许多峰值，每个峰值代表一个非空白标记。通过鼓励CTC模型更早地预测峰值，可以减少CTC模型的识别延迟。现有的减少延迟的方法需要修改前向-后向算法中令牌之间的转换关系以及梯度计算。这些方法中的一些甚至依赖于由其它预训练模型提供的强制对准结果。上述方法实现起来很复杂。为了减少峰值延迟，提出了一种简单新颖的峰值优先正则化方法，该方法利用逐帧的知识提取函数，迫使CTC模型的概率分布沿时间轴左移，而不是直接修改CTC丢失和梯度的计算过程.所有的实验都是在中文数据集AISHELL-1上进行的。我们分别在流式和非流式CTC模型上验证了所提出的正则化方法的有效性。实验结果表明，该方法可以在不降低识别率的情况下，将平均峰值延迟降低约100 ~ 200 ms。摘要：The CTC model has been widely applied to many application scenarios because of its simple structure, excellent performance, and fast inference speed. There are many peaks in the probability distribution predicted by the CTC models, and each peak represents a non-blank token. The recognition latency of CTC models can be reduced by encouraging the model to predict peaks earlier. Existing methods to reduce latency require modifying the transition relationship between tokens in the forward-backward algorithm, and the gradient calculation. Some of these methods even depend on the forced alignment results provided by other pretrained models. The above methods are complex to implement. To reduce the peak latency, we propose a simple and novel method named peak-first regularization, which utilizes a frame-wise knowledge distillation function to force the probability distribution of the CTC model to shift left along the time axis instead of directly modifying the calculation process of CTC loss and gradients. All the experiments are conducted on a Chinese Mandarin dataset AISHELL-1. We have verified the effectiveness of the proposed regularization on both streaming and non-streaming CTC models respectively. The results show that the proposed method can reduce the average peak latency by about 100 to 200 milliseconds with almost no degradation of recognition accuracy.
【5】 Robust Total Least Mean M-Estimate normalized subband filter Adaptive Algorithm for impulse noises and noisy inputs
标题：用于脉冲噪声和噪声输入的稳健总体最小均值M估计归一化子带滤波自适应算法
链接：https://arxiv.org/abs/2211.03283
* 与cs.SD语音【16】为同一篇
作者：Haiquan Zhao,Zian Cao,Yida Chen机构：Magnetic Suspension Technology and Maglev Vehicle, Ministry of Education, and the School of Electrical Engineering, Southwest Jiaotong University, Chengdu, China., (e-mail:, interference, the convergence performance of the traditional摘要：当输入信号为相关输入信号，且输入输出信号均受高斯噪声污染时，总体最小二乘归一化子带自适应滤波（TLS-NSAF）算法表现出良好的性能。然而，当受到脉冲噪声干扰时，TLS-NSAF算法的收敛性能急剧恶化。针对这一问题，提出了一种鲁棒的总体最小均值M估计归一化子带滤波（TLMM-NSAF）算法。此外，本文还对TLMM-NSAF算法进行了详细的理论性能分析，得到了该算法的稳定步长范围和理论稳态均方差（MSD）。为了进一步提高算法的性能，本文还提出了一种新的变步长算法。最后，通过不同噪声模型下的系统辨识和回波抵消的计算机仿真，验证了算法的鲁棒性和理论值与仿真值的一致性。摘要：When the input signal is correlated input signals, and the input and output signal is contaminated by Gaussian noise, the total least squares normalized subband adaptive filter (TLS-NSAF) algorithm shows good performance. However, when it is disturbed by impulse noise, the TLS-NSAF algorithm shows the rapidly deteriorating convergence performance. To solve this problem, this paper proposed the robust total minimum mean M-estimator normalized subband filter (TLMM-NSAF) algorithm. In addition, this paper also conducts a detailed theoretical performance analysis of the TLMM-NSAF algorithm and obtains the stable step size range and theoretical steady-state mean squared deviation (MSD) of the algorithm. To further improve the performance of the algorithm, we also propose a new variable step size (VSS) method of the algorithm. Finally, the robustness of our proposed algorithm and the consistency of theoretical and simulated values are verified by computer simulations of system identification and echo cancellation under different noise models.
【6】 A Context-Aware Computational Approach for Measuring Vocal Entrainment in Dyadic Conversations
标题：一种测量二元对话中发声训练的上下文感知计算方法
链接：https://arxiv.org/abs/2211.03279
* 与cs.SD语音【17】为同一篇
作者：Rimita Lahiri,Md Nasir,Catherine Lord,So Hyun Kim,Shrikanth Narayanan机构：Signal Analysis and Interpretation Laboratory, University of Southern California, Los Angeles, USA, Microsoft AI for Good Research Lab, Redmond, Washington, USA, Semel Institute of Neuroscience and Human Behavior, University of California, Los Angeles, USA摘要：声音诱导是人类交往中的一种社会适应机制，对它的认识可以为个体的认知行为特征提供有益的启示。我们提出一种上下文感知的方法来测量二人会话中的语音夹带。我们使用构象器（卷积网络和转换器的组合）来捕获短期和长期会话上下文，以模拟跨不同域的交互中的夹带模式。具体来说，我们使用跨主题注意层来学习来自二元对话的个人内部和个人之间的信号。我们首先基于分类实验来验证所提出的方法，以区分真实（一致）和虚假（不一致/混洗）会话。关于自闭症谱系障碍患者相互作用的实验结果也表明，引入的诱导测量和与症状相关的临床评分之间存在统计学显著相关性，包括跨性别和年龄组。摘要：Vocal entrainment is a social adaptation mechanism in human interaction, knowledge of which can offer useful insights to an individual's cognitive-behavioral characteristics. We propose a context-aware approach for measuring vocal entrainment in dyadic conversations. We use conformers(a combination of convolutional network and transformer) for capturing both short-term and long-term conversational context to model entrainment patterns in interactions across different domains. Specifically we use cross-subject attention layers to learn intra- as well as inter-personal signals from dyadic conversations. We first validate the proposed method based on classification experiments to distinguish between real(consistent) and fake(inconsistent/shuffled) conversations. Experimental results on interactions involving individuals with Autism Spectrum Disorder also show evidence of a statistically-significant association between the introduced entrainment measure and clinical scores relevant to symptoms, including across gender and age groups.
【7】 An Empirical Study on L2 Accents of Cross-lingual Text-to-Speech Systems via Vowel Space
标题：基于元音空间的跨语言文语转换系统二语重音的实证研究
链接：https://arxiv.org/abs/2211.03078
* 与cs.SD语音【18】为同一篇
作者：Jihwan Lee,Jae-Sung Bae,Seongkyu Mun,Heejin Choi,Joun Yeop Lee,Hoon-Young Cho,Chanwoo Kim机构：Samsung Research, Seoul, South Korea备注：Submitted to ICASSP 2023摘要：随着跨语言文语转换（TTS）系统的发展，第二语言（外语）重音问题日益突出。此外，对这种跨语言TTS系统进行主观评估是麻烦的。元音空间分析是一个很好的分析工具，它经常被用来研究语言的各个方面，包括L2重音。本研究运用元音空间分析方法，探讨跨语言TTS系统中的二语重音。通过元音空间分析，我们观察到以下三点：a）并行体系结构（Glow-TTS）比自回归体系结构（Tacotron）更少L2重音;（2）L2重音在语言对中非共有元音中占优势;跨语言TTS系统的L2重音与人类L2学习者的L2重音有一些共同的现象。我们的研究结果表明，TTS系统有必要根据语言对的语言特征（如非共享元音）对它们进行不同的处理。这也暗示我们可以进一步结合语言学知识来开发跨语言的TTS系统。摘要：With the recent developments in cross-lingual Text-to-Speech (TTS) systems, L2 (second-language, or foreign) accent problems arise. Moreover, running a subjective evaluation for such cross-lingual TTS systems is troublesome. The vowel space analysis, which is often utilized to explore various aspects of language including L2 accents, is a great alternative analysis tool. In this study, we apply the vowel space analysis method to explore L2 accents of cross-lingual TTS systems. Through the vowel space analysis, we observe the three followings: a) a parallel architecture (Glow-TTS) is less L2-accented than an auto-regressive one (Tacotron); b) L2 accents are more dominant in non-shared vowels in a language pair; and c) L2 accents of cross-lingual TTS systems share some phenomena with those of human L2 learners. Our findings imply that it is necessary for TTS systems to handle each language pair differently, depending on their linguistic characteristics such as non-shared vowels. They also hint that we can further incorporate linguistics knowledge in developing cross-lingual TTS systems.
【8】 Distinguishable Speaker Anonymization based on Formant and Fundamental Frequency Scaling
标题：基于共振峰和基频变换的可区分说话人匿名
链接：https://arxiv.org/abs/2211.03038
* 与cs.SD语音【19】为同一篇
作者：Jixun Yao,Qing Wang,Yi Lei,Pengcheng Guo,Lei Xie,Namin Wang,Jie Liu机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Huawei Cloud备注：Submitted to ICASSP 2023摘要：由于社交媒体的出现，互联网上的语音数据呈指数级增长，共享此类个人数据引发了明显的安全和隐私问题。减轻这些问题的一种解决方案涉及在共享语音数据之前隐藏说话者身份，也称为说话者匿名化。在我们以前的工作中，我们已经开发了一个自动说话人确认（ASV）-无模型匿名化框架，以保护说话人隐私，同时保持语音可懂度。虽然该框架在VoicePrivacy 2022挑战赛中排名第一，但由于匿名语音的说话人可区分性下降，使得匿名化不完善。为了解决这一问题，本文直接对共振峰分布和基频（F0）进行建模来表示说话人身份，并通过均匀缩放共振峰和F0来实现源语音的匿名化。通过直接缩放共振峰和F0，防止了由于引入其他说话人而导致的匿名语音的说话人可分辨性下降。实验结果表明，本文提出的框架能够提高说话人区分度，在语音区分度方面明显优于已有的框架。此外，该方法还可以通过使用不同的缩放因子来实现隐私与效用的平衡.摘要：Speech data on the Internet are proliferating exponentially because of the emergence of social media, and the sharing of such personal data raises obvious security and privacy concerns. One solution to mitigate these concerns involves concealing speaker identities before sharing speech data, also referred to as speaker anonymization. In our previous work, we have developed an automatic speaker verification (ASV)-model-free anonymization framework to protect speaker privacy while preserving speech intelligibility. Although the framework ranked first place in VoicePrivacy 2022 challenge, the anonymization was imperfect, since the speaker distinguishability of the anonymized speech was deteriorated. To address this issue, in this paper, we directly model the formant distribution and fundamental frequency (F0) to represent speaker identity and anonymize the source speech by the uniformly scaling formant and F0. By directly scaling the formant and F0, the speaker distinguishability degradation of the anonymized speech caused by the introduction of other speakers is prevented. The experimental results demonstrate that our proposed framework can improve the speaker distinguishability and significantly outperforms our previous framework in voice distinctiveness. Furthermore, our proposed method also can trade off the privacy-utility by using different scaling factors.
【9】 Preserving background sound in noise-robust voice conversion via multi-task learning
标题：在噪声中保留背景声音-通过多任务学习实现稳健的语音转换
链接：https://arxiv.org/abs/2211.03036
* 与cs.SD语音【20】为同一篇
作者：Jixun Yao,Yi Lei,Qing Wang,Pengcheng Guo,Ziqian Ning,Lei Xie,Hai Li,Junhui Liu,Danming Xie机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, QIYI Inc, China备注：Submitted to ICASSP 2023摘要：背景声音是一种信息性的艺术形式，有助于在真实应用程序语音转换（VC）场景中提供更身临其境的体验。然而，以往对VC的研究主要集中在纯净语音上，对带背景音的VC研究较少。VC中保留背景音的关键问题是神经元分离模型不可避免的语音失真以及源分离模型与VC模型之间的级联失配。本文提出了一种基于多任务学习的端到端框架，该框架由源分离模块、瓶颈特征提取模块和VC模块依次级联而成。具体而言，源分离任务明确地考虑了临界相位信息，并限制了由不完美分离过程引起的失真。通过联合训练约束，源分离任务、典型VC任务和统一任务共享统一的重构损失，以减少SS和VC模块之间的失配。实验结果表明，本文提出的框架显著优于基线系统，同时获得了与使用干净数据训练的VC模型相当的质量和说话人相似度。摘要：Background sound is an informative form of art that is helpful in providing a more immersive experience in real-application voice conversion (VC) scenarios. However, prior research about VC, mainly focusing on clean voices, pay rare attention to VC with background sound. The critical problem for preserving background sound in VC is inevitable speech distortion by the neural separation model and the cascade mismatch between the source separation model and the VC model. In this paper, we propose an end-to-end framework via multi-task learning which sequentially cascades a source separation (SS) module, a bottleneck feature extraction module and a VC module. Specifically, the source separation task explicitly considers critical phase information and confines the distortion caused by the imperfect separation process. The source separation task, the typical VC task and the unified task shares a uniform reconstruction loss constrained by joint training to reduce the mismatch between the SS and VC modules. Experimental results demonstrate that our proposed framework significantly outperforms the baseline systems while achieving comparable quality and speaker similarity to the VC models trained with clean data.
【10】 Breaking the trade-off in personalized speech enhancement with cross-task knowledge distillation
标题：用跨任务知识提炼打破个性化语音增强中的权衡
链接：https://arxiv.org/abs/2211.02944
* 与cs.SD语音【21】为同一篇
作者：Hassan Taherian,Sefik Emre Eskimez,Takuya Yoshioka机构：Microsoft, Redmond, WA, USA, The Ohio State University, Columbus, OH, USA备注：Submitted to ICASSP 2023摘要：与无条件语音增强模型相比，个性化语音增强模型由于能够去除背景噪声和干扰语音而取得了较好的效果。与无条件语音增强不同，因果PSE模型可能偶尔错误地去除目标语音。当目标说话人在延长的时段内沉默时，PSE模型也倾向于泄漏干扰语音。我们表明，现有的PSE方法在语音过抑制和干扰泄漏之间进行了权衡，解决了一个问题，而牺牲了另一个问题。我们提出一个新的PSE模型训练框架，使用跨任务的知识提取来缓解这种折衷。具体地，我们在训练期间利用个性化语音活动检测器（pVAD）来排除被错误地识别为包含具有硬分类或软分类的目标说话者的非目标语音帧。这防止了PSE模型过于激进，同时仍然允许模型学习以在输入语音可能由干扰说话者说出时抑制输入语音。给出了综合评估结果，涵盖了各种PSE使用场景。摘要：Personalized speech enhancement (PSE) models achieve promising results compared with unconditional speech enhancement models due to their ability to remove interfering speech in addition to background noise. Unlike unconditional speech enhancement, causal PSE models may occasionally remove the target speech by mistake. The PSE models also tend to leak interfering speech when the target speaker is silent for an extended period. We show that existing PSE methods suffer from a trade-off between speech over-suppression and interference leakage by addressing one problem at the expense of the other. We propose a new PSE model training framework using cross-task knowledge distillation to mitigate this trade-off. Specifically, we utilize a personalized voice activity detector (pVAD) during training to exclude the non-target speech frames that are wrongly identified as containing the target speaker with hard or soft classification. This prevents the PSE model from being too aggressive while still allowing the model to learn to suppress the input speech when it is likely to be spoken by interfering speakers. Comprehensive evaluation results are presented, covering various PSE usage scenarios.
【11】 Real-Time Joint Personalized Speech Enhancement and Acoustic Echo Cancellation with E3Net
标题：基于E3Net的实时联合个性化语音增强和声回波消除
链接：https://arxiv.org/abs/2211.02773
* 与cs.SD语音【22】为同一篇
作者：Sefik Emre Eskimez,Takuya Yoshioka,Alex Ju,Min Tang,Tanel Parnamaa,Huaming Wang机构：Microsoft, Redmond, WA, USA备注：Submitted to ICASSP 23摘要：个性化语音增强（Personalized Speech Enhancement，PSE）是一种利用目标说话人的说话人嵌入向量实时估计干净的目标语音信号的方法，近年来随着全球在线会议的兴起而引起了研究界的广泛关注。对于实际的全双工通信，PSE模型需要声学回声消除（AEC）能力。在本文中，我们采用了最近提出的因果端到端增强网络（E3 Net），并对其进行了改进，得到了一个联合的PSE-AEC模型.我们将早期层专用于AEC任务，同时通过添加从早期层到掩码预测层的旁路连接来鼓励后期层进行个性化。这使我们能够采用多任务学习框架进行PSE和AEC联合培训。我们提供了大量的评估测试场景，包括模拟和真实世界的记录。实验结果表明，对于每个任务，我们的联合模型都接近专家模型，并且对于组合的PSE-AEC场景，我们的联合模型的性能明显更好。摘要：Personalized speech enhancement (PSE), a process of estimating a clean target speech signal in real time by leveraging a speaker embedding vector of the target talker, has garnered much attention from the research community due to the recent surge of online meetings across the globe. For practical full duplex communication, PSE models require an acoustic echo cancellation (AEC) capability. In this work, we employ a recently proposed causal end-to-end enhancement network (E3Net) and modify it to obtain a joint PSE-AEC model. We dedicate the early layers to the AEC task while encouraging later layers for personalization by adding a bypass connection from the early layers to the mask prediction layer. This allows us to employ a multi-task learning framework for joint PSE and AEC training. We provide extensive evaluation test scenarios with both simulated and real-world recordings. The results show that our joint model comes close to the expert models for each task and performs significantly better for the combined PSE-AEC scenario.
【12】 SAMO: Speaker Attractor Multi-Center One-Class Learning for Voice Anti-Spoofing
标题：SAMO：用于语音反欺骗的扬声器吸引子多中心单类学习
链接：https://arxiv.org/abs/2211.02718
* 与cs.SD语音【23】为同一篇
作者：Siwen Ding,You Zhang,Zhiyao Duan机构：Columbia University, New York, NY, USA, University of Rochester, Rochester, NY, USA摘要：语音反欺骗系统是自动说话人确认（ASV）系统的重要辅助设备。一个主要的挑战是由先进的语音合成技术支持的不可见的攻击造成的。我们以前的单类学习研究通过压缩真实语音在嵌入空间中的位置，提高了对不可见攻击的泛化能力。然而，这种紧凑性缺乏对说话者多样性的考虑。本文提出一种说话人吸引子多中心单类学习算法（SAMO），该算法将真实语音聚集在多个说话人吸引子周围，并在高维嵌入空间中从所有吸引子中排除欺骗攻击.在训练方面，我们提出了一个语音聚类和语音分类的联合优化算法。对于推理，我们提出了一些策略来使没有注册的说话者能够反欺骗。在ASVspoof 2019 LA评估集上，我们提出的系统在等错误率（EER）方面相对改进了38%，优于现有的最先进的单系统。摘要：Voice anti-spoofing systems are crucial auxiliaries for automatic speaker verification (ASV) systems. A major challenge is caused by unseen attacks empowered by advanced speech synthesis technologies. Our previous research on one-class learning has improved the generalization ability to unseen attacks by compacting the bona fide speech in the embedding space. However, such compactness lacks consideration of the diversity of speakers. In this work, we propose speaker attractor multi-center one-class learning (SAMO), which clusters bona fide speech around a number of speaker attractors and pushes away spoofing attacks from all the attractors in a high-dimensional embedding space. For training, we propose an algorithm for the co-optimization of bona fide speech clustering and bona fide/spoof classification. For inference, we propose strategies to enable anti-spoofing for speakers without enrollment. Our proposed system outperforms existing state-of-the-art single systems with a relative improvement of 38% on equal error rate (EER) on the ASVspoof2019 LA evaluation set.
【13】 Speech enhancement using ego-noise references with a microphone array embedded in an unmanned aerial vehicle
标题：利用嵌入无人机中的麦克风阵列的自噪声参考的语音增强
链接：https://arxiv.org/abs/2211.02690
* 与cs.SD语音【24】为同一篇
作者：Elisa Tengan,Thomas Dietzen,Santiago Ruiz,Mansour Alkmim,João Cardenuto,Toon van Waterschoot摘要：提出了一种基于麦克风阵列的自噪声语音增强方法。利用位于无人机螺旋桨附近的麦克风捕获自我噪声参考信号，并将其用于先验知识多通道维纳滤波器（PK-MWF）中，以获得语音相关矩阵估计。可以估计语音存在概率（SPP），以用于检测来自语音源附近的外部麦克风的语音活动，从而提供性能基准，或者用于检测来自嵌入式麦克风之一的语音活动，从而假设更现实的场景。实验测量是在半消声室中进行的，无人机安装在支架上，扬声器播放语音信号，同时设置三个不同的和固定的螺旋桨转速，导致三个不同的信噪比（SNR）。使用在线获得的记录，将所提出的方法与使用标准多通道维纳滤波器（MWF）进行比较，在公式中使用和不使用螺旋桨麦克风进行估计。结果表明，与那些相比，使用PK-MWF在通过STOI和PESQ测量的语音可懂度和质量方面实现了更高水平的改善，而SNR改善相似。摘要：A method is proposed for performing speech enhancement using ego-noise references with a microphone array embedded in an unmanned aerial vehicle (UAV). The ego-noise reference signals are captured with microphones located near the UAV's propellers and used in the prior knowledge multichannel Wiener filter (PK-MWF) to obtain the speech correlation matrix estimate. Speech presence probability (SPP) can be estimated for detecting speech activity from an external microphone near the speech source, providing a performance benchmark, or from one of the embedded microphones, assuming a more realistic scenario. Experimental measurements are performed in a semi-anechoic chamber, with a UAV mounted on a stand and a loudspeaker playing a speech signal, while setting three distinct and fixed propeller rotation speeds, resulting in three different signal-to-noise ratios (SNRs). The recordings obtained and made available online are used to compare the proposed method to the use of the standard multichannel Wiener filter (MWF) estimated with and without the propellers' microphones being used in its formulation. Results show that compared to those, the use of PK-MWF achieves higher levels of improvement in speech intelligibility and quality, measured by STOI and PESQ, while the SNR improvement is similar.
【14】 Egocentric Audio-Visual Noise Suppression
标题：以自我为中心的视听噪声抑制
链接：https://arxiv.org/abs/2211.03643
* 与cs.SD语音【1】为同一篇
作者：Roshan Sharma,Weipeng He,Ju Lin,Egor Lakomkin,Yang Liu,Kaustubh Kalgaonkar机构：Carnegie Mellon University, Pittsburgh, USA, Meta, Seattle, USA备注：Under Review at ICASSP 2023摘要：本文研究了以自我为中心的视频--说话人不在视频中--的音视频抑制问题。相反，潜在的噪声源在屏幕上是可见的，摄像机模拟屏幕外说话者对外部世界的看法。该设置不同于依赖于嘴唇和面部视觉的视听语音增强中的现有工作。本文首先证明了自我中心视觉信息有助于噪声抑制。我们比较了基于对象识别和动作分类的视觉特征提取器，并研究了对齐音频和视觉表示的方法。然后，我们检查对准特征的不同融合策略，以及噪声抑制模型内的位置，以合并视觉信息。实验表明，视觉特征在用于生成加性校正遮罩时是最有帮助的。最后，为了确保视觉特征相对于不同噪声类型是有区别的，我们引入了联合优化视听噪声抑制和基于视频的声学事件检测的多任务学习框架。此提议的多任务框架在所有指标上均优于仅音频基线，包括0.16 PESQ改善。广泛消融显示，在所有噪声类型和不同SNR下，使用多个主动干扰器的申报模型的性能有所改善。摘要：This paper studies audio-visual suppression for egocentric videos -- where the speaker is not captured in the video. Instead, potential noise sources are visible on screen with the camera emulating the off-screen speaker's view of the outside world. This setting is different from prior work in audio-visual speech enhancement that relies on lip and facial visuals. In this paper, we first demonstrate that egocentric visual information is helpful for noise suppression. We compare object recognition and action classification based visual feature extractors, and investigate methods to align audio and visual representations. Then, we examine different fusion strategies for the aligned features, and locations within the noise suppression model to incorporate visual information. Experiments demonstrate that visual features are most helpful when used to generate additive correction masks. Finally, in order to ensure that the visual features are discriminative with respect to different noise types, we introduce a multi-task learning framework that jointly optimizes audio-visual noise suppression and video based acoustic event detection. This proposed multi-task framework outperforms the audio only baseline on all metrics, including a 0.16 PESQ improvement. Extensive ablations reveal the improved performance of the proposed model with multiple active distractors, over all noise types and across different SNRs.
【15】 Hi,KIA: A Speech Emotion Recognition Dataset for Wake-Up Words
标题：Hi，KIA：一个用于唤醒词的语音情感识别数据集
链接：https://arxiv.org/abs/2211.03371
* 与cs.SD语音【2】为同一篇
作者：Taesu Kim,SeungHeon Doh,Gyunpyo Lee,Hyungseok Jeon,Juhan Nam,Hyeon-Jeong Suk机构：∗ Department of Industrial Design, KAIST, Daejeon, South Korea, † Graduate School of Culture Technology, KAIST, Daejeon, South Korea, ‡ KIA Design Studio, Hyundai Motor Company, Hwaseong, South Korea, § Equally contributing authors备注：Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2022摘要：唤醒词（WUW）是用于激活语音识别系统以接收用户的语音输入的短句。WUW话语不仅包括用于唤醒系统的词汇信息，还包括诸如说话者身份或情感等非词汇信息。特别地，识别用户的情绪状态可以精心设计语音通信。然而，很少有数据集，其中的情绪状态的WUW话语标记。本文介绍了一个新的WUW数据集Hi，KIA，该数据集包含来自四位男性和四位女性说话人的488个韩国口音情感话语，每个话语都被标记为四种情感状态，包括愤怒、高兴、悲伤和中性。我们介绍了构建数据集的逐步过程，包括场景选择、后处理和标签一致性的人工验证。同时，我们还利用该数据集，提出了两种用于WUW语音情感识别的分类模型。一种是基于传统手工特征的方法，另一种是使用预先训练的神经网络的迁移学习方法。这些分类模型可以作为进一步研究的基准。摘要：Wake-up words (WUW) is a short sentence used to activate a speech recognition system to receive the user's speech input. WUW utterances include not only the lexical information for waking up the system but also non-lexical information such as speaker identity or emotion. In particular, recognizing the user's emotional state may elaborate the voice communication. However, there is few dataset where the emotional state of the WUW utterances is labeled. In this paper, we introduce Hi, KIA, a new WUW dataset which consists of 488 Korean accent emotional utterances collected from four male and four female speakers and each of utterances is labeled with four emotional states including anger, happy, sad, or neutral. We present the step-by-step procedure to build the dataset, covering scenario selection, post-processing, and human validation for label agreement. Also, we provide two classification models for WUW speech emotion recognition using the dataset. One is based on traditional hand-craft features and the other is a transfer-learning approach using a pre-trained neural network. These classification models could be used as benchmarks in further research.
【16】 "Seeing Sound": Audio Classification with the Wigner-Wille Distribution and Convolutional Neural Networks
链接：https://arxiv.org/abs/2211.03202
* 与cs.SD语音【3】为同一篇
作者：Antonios Marios Christonasis,Stef van Eijndhoven,Peter Duin机构： Engineering Doctorate Data Science, Technical University of Eindhoven (TUe), Eindhoven, The, Dutch National Police, Eindhoven, The Netherlands摘要：随着大数据日益可用、物联网硬件被广泛采用以及AI功能日益强大，企业正在不断投资于传感。目前，来自传感器网络的数据与传感器融合和AI算法相结合，推动自动驾驶汽车等领域的创新。来自这些传感器的数据可用于许多使用案例，包括城市环境的安全系统中的警报，用于诸如枪击和爆炸之类的事件。此外，可以在低光照条件下或在相机不可用的位置处利用各种类型的传感器，诸如声音传感器。本文研究了声音传感器数据在城市环境中的应用潜力。在技术上，我们提出了一种新的方法来分类声音数据使用Wigner-Ville分布和卷积神经网络。在本文中，我们报告了该方法在开源数据集上的性能。本文所提出的概念和工作基于我的博士论文，该论文是作为埃因霍温大学数据科学工程博士学位项目的一部分，与荷兰国家警察合作完成的。在论文期间，对真实世界数据集进行了额外的工作，由于机密性，在此不作介绍。摘要：With big data becoming increasingly available, IoT hardware becoming widely adopted, and AI capabilities becoming more powerful, organizations are continuously investing in sensing. Data coming from sensor networks are currently combined with sensor fusion and AI algorithms to drive innovation in fields such as self-driving cars. Data from these sensors can be utilized in numerous use cases, including alerts in safety systems of urban settings, for events such as gun shots and explosions. Moreover, diverse types of sensors, such as sound sensors, can be utilized in low-light conditions or at locations where a camera is not available. This paper investigates the potential of the utilization of sound-sensor data in an urban context. Technically, we propose a novel approach of classifying sound data using the Wigner-Ville distribution and Convolutional Neural Networks. In this paper, we report on the performance of the approach on open-source datasets. The concept and work presented is based on my doctoral thesis, which was performed as part of the Engineering Doctorate program in Data Science at the University of Eindhoven, in collaboration with the Dutch National Police. Additional work on real-world datasets was performed during the thesis, which are not presented here due to confidentiality.
【17】 Going In Style: Audio Backdoors Through Stylistic Transformations
标题：走进风格：通过风格转换的音频后门
链接：https://arxiv.org/abs/2211.03117
* 与cs.SD语音【4】为同一篇
作者：Stefanos Koffas,Luca Pajola,Stjepan Picek,Mauro Conti机构：Cybersecurity Group, Delft University of Technology, The Netherlands, Security and Privacy Research Group, University of Padua, Italy, Digital Security Group, Radboud University, The Netherlands, ∗Equal contribution摘要：后门攻击会在受害者的深度学习模型中放置触发器，以便在测试时实现有针对性的错误分类。通常，触发器是附加到样本的固定工件，使后门攻击易于发现。就在最近，人们提出了更难检测的新一代触发器：将风格变换应用于输入样本的风格触发器（例如，特定的书写风格）。 目前，文体学的后门文学缺乏一个恰当的形式化的攻击，本文建立了。而且，大多数关于文体触发的研究都集中在文本和图像上，而对它们是否能在声音中起作用却没有任何了解。这项工作填补了这一空白。我们提出了JingleBack，这是第一个基于音频变换（如合唱和增益）的风格后门攻击。在语音分类任务中使用444个模型，我们证实了音频中风格触发的可行性，实现了96%的攻击成功率。摘要：A backdoor attack places triggers in victims' deep learning models to enable a targeted misclassification at testing time. In general, triggers are fixed artifacts attached to samples, making backdoor attacks easy to spot. Only recently, a new trigger generation harder to detect has been proposed: the stylistic triggers that apply stylistic transformations to the input samples (e.g., a specific writing style). Currently, stylistic backdoor literature lacks a proper formalization of the attack, which is established in this paper. Moreover, most studies of stylistic triggers focus on text and images, while there is no understanding of whether they can work in sound. This work fills this gap. We propose JingleBack, the first stylistic backdoor attack based on audio transformations such as chorus and gain. Using 444 models in a speech classification task, we confirm the feasibility of stylistic triggers in audio, achieving 96% attack success.
【18】 I Hear Your True Colors: Image Guided Audio Generation
标题：我听到你的本色：图像引导的音频生成
链接：https://arxiv.org/abs/2211.03089
* 与cs.SD语音【5】为同一篇
机构：School of Computer Science and Engineering, The Hebrew University of Jerusalem, Israel摘要：本文提出了一种图像引导的开放域音频生成系统Im 2 Wav。给定输入图像或图像序列，Im 2 Wav生成语义相关的声音。Im 2 Wav基于两个Transformer语言模型，其在从基于VQ-VAE的模型获得的分级离散音频表示上操作。我们首先使用语言模型产生低级音频表示。然后，我们使用附加的语言模型对音频令牌进行上采样，以生成高保真音频样本。我们使用预先训练的CLIP嵌入的丰富语义作为可视表示来调节语言模型。此外，为了将生成过程导向条件图像，我们应用了无分类器指导方法。结果表明，Im 2 Wav在保真度和相关性评价指标上均显著优于基线评价。此外，我们还提供了消融研究，以更好地评估每个方法组件对整体性能的影响。最后，为了更好地评估图像到音频模型，我们提出了一个域外图像数据集，表示为ImageHear。ImageHear可用作评估未来图像到音频模型的基准。样本和代码可以在手稿中找到。摘要：We propose Im2Wav, an image guided open-domain audio generation system. Given an input image or a sequence of images, Im2Wav generates a semantically relevant sound. Im2Wav is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-VAE based model. We first produce a low-level audio representation using a language model. Then, we upsample the audio tokens using an additional language model to generate a high-fidelity audio sample. We use the rich semantics of a pre-trained CLIP embedding as a visual representation to condition the language model. In addition, to steer the generation process towards the conditioning image, we apply the classifier-free guidance method. Results suggest that Im2Wav significantly outperforms the evaluated baselines in both fidelity and relevance evaluation metrics. Additionally, we provide an ablation study to better assess the impact of each of the method components on overall performance. Lastly, to better evaluate image-to-audio models, we propose an out-of-domain image dataset, denoted as ImageHear. ImageHear can be used as a benchmark for evaluating future image-to-audio models. Samples and code can be found inside the manuscript.
【19】 Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
标题：无监督ASR在语音和文本预训练模型之间的桥接
链接：https://arxiv.org/abs/2211.03025
* 与cs.SD语音【6】为同一篇
作者：Jiatong Shi,Chan-Jan Hsu,Holam Chung,Dongji Gao,Paola Garcia,Shinji Watanabe,Ann Lee,Hung-yi Lee机构： Carnegie Mellon University, National Taiwan University, Johns Hopkins University, Meta AI摘要：口语理解是一项旨在从口语话语中提取高级语义的任务。先前的工作已经研究了语音自监督模型和文本预训练模型的使用，它们已经显示出对各种SLU任务的合理改进。然而，由于语音信号和文本标记之间的模态不匹配，以往的方法通常需要复杂的框架设计。本文提出了一种简单而有效的无监督范式，将语音和文本预训练模型连接起来，得到一个用于SLU中各种任务的无监督语音-语义预训练模型。具体地说，我们提出使用无监督自动语音识别（ASR）作为连接器，桥接语音和文本预训练模型中使用的不同模态。实验结果表明，无监督ASR本身可以改善语音自监督模型的表示。更重要的是，它被证明是语音和文本预训练模型之间的有效连接器，提高了五种不同的SLU任务的性能。值得注意的是，在口头问答方面，我们达到了最先进的结果，超过了具有挑战性的NMSQA基准。摘要：Spoken language understanding (SLU) is a task aiming to extract high-level semantics from spoken utterances. Previous works have investigated the use of speech self-supervised models and textual pre-trained models, which have shown reasonable improvements to various SLU tasks. However, because of the mismatched modalities between speech signals and text tokens, previous methods usually need complex designs of the frameworks. This work proposes a simple yet efficient unsupervised paradigm that connects speech and textual pre-trained models, resulting in an unsupervised speech-to-semantic pre-trained model for various tasks in SLU. To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models. Our experiments show that unsupervised ASR itself can improve the representations from speech self-supervised models. More importantly, it is shown as an efficient connector between speech and textual pre-trained models, improving the performances of five different SLU tasks. Notably, on spoken question answering, we reach the state-of-the-art result over the challenging NMSQA benchmark.
【20】 Effective Audio Classification Network Based on Paired Inverse Pyramid Structure and Dense MLP Block
标题：基于成对倒金字塔结构和密集MLP块的有效音频分类网络
链接：https://arxiv.org/abs/2211.02940
* 与cs.SD语音【7】为同一篇
作者：Yunhao Chen,Yunjie Zhu,Zihui Yan,Lifang Chen机构：Jiangnan University, School of Artificial Intelligence and Computer Science, Wuxi, China, University of Leeds, School of Computing, Leeds, LS,JT, United Kingdom摘要：近年来，基于卷积神经网络（CNN）和自注意机制的大规模体系结构已经成为音频分类的必要条件。虽然这些技术是最先进的，但这些工作的有效性只能通过巨大的计算成本和参数、大量的数据扩增、从大数据集转移和一些其他技巧来保证。利用音频的轻量级特性，提出了一种高效的网络结构--成对逆金字塔结构（Paired Inverse Pyramid Structure，PIP）和成对逆金字塔结构MLP网络（Paired Inverse Pyramid Structure，MLP Network，PIPMN）。在UrbanSound 8 K数据集上，PIPMN的环境声音分类（ESC）准确率达到96%;在GTAZN数据集上，PIPMN的音乐类型分类（MGC）准确率达到93.2%，参数仅为100万.这两个结果都是在没有数据扩充或模型转移的情况下获得的。公共代码可从以下网址获得：https://github.com/JNAIC/PIPMN摘要：Recently, massive architectures based on Convolutional Neural Network (CNN) and self-attention mechanisms have become necessary for audio classification. While these techniques are state-of-the-art, these works' effectiveness can only be guaranteed with huge computational costs and parameters, large amounts of data augmentation, transfer from large datasets and some other tricks. By utilizing the lightweight nature of audio, we propose an efficient network structure called Paired Inverse Pyramid Structure (PIP) and a network called Paired Inverse Pyramid Structure MLP Network (PIPMN). The PIPMN reaches 96\% of Environmental Sound Classification (ESC) accuracy on the UrbanSound8K dataset and 93.2\% of Music Genre Classification (MGC) on the GTAZN dataset, with only 1 million parameters. Both of the results are achieved without data augmentation or model transfer. Public code is available at: https://github.com/JNAIC/PIPMN
【21】 VISinger 2: High-Fidelity End-to-End Singing Voice Synthesis Enhanced by Digital Signal Processing Synthesizer
标题：VISinger 2：由数字信号处理合成器增强的高保真端到端歌声合成
链接：https://arxiv.org/abs/2211.02903
* 与cs.SD语音【8】为同一篇
作者：Yongmao Zhang,Heyang Xue,Hanzhao Li,Lei Xie,Tingwei Guo,Ruixiong Zhang,Caixia Gong机构：School of Computer Science, Northwestern Polytechnical University, Xi’an, China, DiDi Chuxing, Beijing, China备注：Submitted to ICASSP 2023摘要：端到端的演唱语音合成（SVS）模型VISinger能够以较少的参数获得比典型的两阶段模型更好的性能。但是，VISinger有几个问题：文本到短语问题，端到端模型学习文本到短语的无意义映射;假信号问题，对应于有声段的周期性信号的谐波分量发生突然变化，具有可听伪像;采样率低，24 KHz的采样率不能满足全频段速率（44.1KHz或更高）高保真生成的应用需求。本文将数字信号处理（DSP）方法与VISinger相结合，提出了VISinger 2来解决这些问题。具体地说，受微分数字信号处理（DDSP）的最新进展的启发，我们将DSP合成器结合到解码器中以解决上述问题。DSP合成器由谐波合成器和噪声合成器组成，以分别从VISinger中的潜在表示z生成周期性和非周期性信号。它监督后级编码器提取没有相位信息的潜在表示，避免了前级编码器建模文本到相位的映射。为了避免假信号，HiFi-GAN被修改为接受由DSP合成器生成的波形作为产生歌声的条件。此外，通过改进的波形解码器，VISinger 2成功地生成了44.1kHz的歌唱音频，表现力更丰富，质量更好。在OpenCpop语料库上的实验表明，VISinger 2在主观和客观指标上都优于VISinger、CpopSing和RefineSinger。摘要：End-to-end singing voice synthesis (SVS) model VISinger can achieve better performance than the typical two-stage model with fewer parameters. However, VISinger has several problems: text-to-phase problem, the end-to-end model learns the meaningless mapping of text-to-phase; glitches problem, the harmonic components corresponding to the periodic signal of the voiced segment occurs a sudden change with audible artefacts; low sampling rate, the sampling rate of 24KHz does not meet the application needs of high-fidelity generation with the full-band rate (44.1KHz or higher). In this paper, we propose VISinger 2 to address these issues by integrating the digital signal processing (DSP) methods with VISinger. Specifically, inspired by recent advances in differentiable digital signal processing (DDSP), we incorporate a DSP synthesizer into the decoder to solve the above issues. The DSP synthesizer consists of a harmonic synthesizer and a noise synthesizer to generate periodic and aperiodic signals, respectively, from the latent representation z in VISinger. It supervises the posterior encoder to extract the latent representation without phase information and avoid the prior encoder modelling text-to-phase mapping. To avoid glitch artefacts, the HiFi-GAN is modified to accept the waveforms generated by the DSP synthesizer as a condition to produce the singing voice. Moreover, with the improved waveform decoder, VISinger 2 manages to generate 44.1kHz singing audio with richer expression and better quality. Experiments on OpenCpop corpus show that VISinger 2 outperforms VISinger, CpopSing and RefineSinger in both subjective and objective metrics.
【22】 Fast and efficient speech enhancement with variational autoencoders
标题：利用可变自动编码器实现快速高效的语音增强
链接：https://arxiv.org/abs/2211.02728
* 与cs.SD语音【10】为同一篇
作者：Mostafa Sadeghi,Romain Serizel机构：Universit´e de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：与常用的有监督语音增强方法相比，基于变分自动编码器的无监督语音增强方法具有较好的性能。该方法使用预先训练好的深度语音先验知识和参数噪声模型，其中噪声参数是通过基于期望最大化（EM）的方法从含噪语音信号中学习得到的。E-步骤涉及难以处理的潜在后验分布。现有的解决这一问题的算法要么是基于计算量大的蒙特卡罗马尔可夫链抽样方法和变分推理，要么是基于低效的优化方法。本文提出了一种基于Langevin动力学的新方法，该方法生成多个样本序列，并采用基于全变分的正则化方法来考虑潜在向量的时间相关性。实验结果表明，该算法在计算效率和增强质量之间取得了较好的折衷，优于现有算法.摘要：Unsupervised speech enhancement based on variational autoencoders has shown promising performance compared with the commonly used supervised methods. This approach involves the use of a pre-trained deep speech prior along with a parametric noise model, where the noise parameters are learned from the noisy speech signal with an expectationmaximization (EM)-based method. The E-step involves an intractable latent posterior distribution. Existing algorithms to solve this step are either based on computationally heavy Monte Carlo Markov Chain sampling methods and variational inference, or inefficient optimization-based methods. In this paper, we propose a new approach based on Langevin dynamics that generates multiple sequences of samples and comes with a total variation-based regularization to incorporate temporal correlations of latent vectors. Our experiments demonstrate that the developed framework makes an effective compromise between computational efficiency and enhancement quality, and outperforms existing methods.
【23】 Resource-Efficient Transfer Learning From Speech Foundation Model Using Hierarchical Feature Fusion
标题：基于分层特征融合的语音基础模型资源高效转移学习
链接：https://arxiv.org/abs/2211.02712
* 与cs.SD语音【11】为同一篇
作者：Zhouyuan Huo,Khe Chai Sim,Bo Li,Dongseong Hwang,Tara N. Sainath,Trevor Strohman摘要：语音基础模型的自监督预训练，随后的监督微调，已经在自动语音识别（ASR）任务上显示出令人印象深刻的质量改进。由于基础模型通常非常大，因此为许多下游任务微调单独的基础模型是昂贵的。参数高效的微调方法（例如适配器、稀疏更新方法）提供了一种替代范例，其中更新一小组参数以使基础模型适应新任务。然而，这些方法仍然遭受高的计算存储器成本和慢的训练速度，因为它们在每一步需要通过整个神经网络的反向传播。本文分析了基础模型不同层次特征在语音识别任务中的表现，提出了一种新的分层特征融合方法，用于语音基础模型的资源有效迁移学习。实验结果表明，与现有算法相比，该方法能够以更少的可训练参数、更少的计算内存开销和更快的训练速度获得更好的语音识别性能。在各层结合Adapters后，该方法可达到与微调整个模型相同的性能，但可训练的编码器参数减少了97%，训练速度提高了53%.摘要：Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with $97\%$ fewer trainable encoder parameters and $53\%$ faster training speed.
机器翻译，仅供参考
每日语音速递