今日论文合集：cs.SD语音6篇，eess.AS音频处理10篇。
【1】 Hopf Physical Reservoir Computer for Reconfigurable Sound Recognition
标题：用于可重构声音识别的Hopf物性储集层计算机
链接：https://arxiv.org/abs/2212.10370
作者：Md Raf E Ul Shougat,XiaoFu Li,Siyao Shao,Kathleen Walden McGarvey,Edmon Perkins机构：Perkins,†, Mechanical & Aerospace Engineering Department, North, Carolina State University, Entrepreneur Drive, Raleigh, NC, USA., LAB, Atwood, OK, USA., TandemLaunch, Av. Brewster, Montreal, H,C,K, Canada., echosonic, Av. Brewster, Montreal, H,C,K, Canada.摘要：霍普夫振子是一种非线性振子，表现出极限环运动。这种水库计算机利用振荡器的振动性质，这使它成为可重新配置的声音识别任务的理想候选者。本文系统地论证了Hopf水库计算机进行声音识别的能力。这项工作表明，与传统方法相比，Hopf水库计算机可以提供优越的声音识别精度（例如，梅尔谱+机器学习方法）。更重要的是，作为声音识别系统操作的Hopf水库计算机不需要音频预处理，并且具有非常简单的设置，同时仍然提供高度的可重构性。这些特性为在低功耗边缘设备中应用物理库计算进行声音识别铺平了道路。摘要：The Hopf oscillator is a nonlinear oscillator that exhibits limit cycle motion. This reservoir computer utilizes the vibratory nature of the oscillator, which makes it an ideal candidate for reconfigurable sound recognition tasks. In this paper, the capabilities of the Hopf reservoir computer performing sound recognition are systematically demonstrated. This work shows that the Hopf reservoir computer can offer superior sound recognition accuracy compared to legacy approaches (e.g., a Mel spectrum + machine learning approach). More importantly, the Hopf reservoir computer operating as a sound recognition system does not require audio preprocessing and has a very simple setup while still offering a high degree of reconfigurability. These features pave the way of applying physical reservoir computing for sound recognition in low power edge devices.
【2】 Emotion Selectable End-to-End Text-based Speech Editing
标题：基于情感选择的端到端文本语音编辑
链接：https://arxiv.org/abs/2212.10191
作者：Tao Wang,Jiangyan Yi,Ruibo Fu,Jianhua Tao,Zhengqi Wen,Chu Yuan Zhang机构：and also with the School of Artificial Intelligence, University of ChineseAcademy of Sciences备注：Under review, 12 pages, 11 figures, demo page is available at this https URL摘要：基于文本的语音编辑允许用户通过直观地剪切、复制和粘贴文本来编辑语音，以加快编辑语音的过程。在前期工作中，提出了CampNet（context-aware mask prediction network）来实现基于文本的语音编辑，显著提高了编辑后语音的质量。本文针对一个新课题：在基于文本的语音编辑过程中，向编辑语音添加情感效果，以使生成的语音更具表现力。为了实现这一目标，提出了Emo-CampNet（emotion CampNet），它可以为基于文本的语音编辑中生成的语音提供情感属性选项，并具有一次性编辑未见过说话人语音的能力。首先，提出了一种端到端的基于文本的情感选择语音编辑模型。该模型的核心思想是在上下文感知掩码预测网络的基础上，通过引入附加情感属性来控制生成语音的情感。其次，为了避免生成语音的情感受到原始语音中情感成分的干扰，提出了一种中性内容生成器，用于去除原始语音中的情感，并通过生成式对抗框架进行了优化。第三，提出了两种数据增强方法，丰富了训练集的情感和发音信息，使模型能够编辑未见过说话人的语音。实验结果表明：1）在基于文本的语音编辑过程中，Emo-CampNet能够有效地控制生成语音的情感;还能编辑看不见的说话人的讲话。2)详细的烧蚀实验进一步证明了情感选择和数据扩充方法的有效性。演示页可从www.example.com获得https://hairuo55.github.io/Emo-CampNet/摘要：Text-based speech editing allows users to edit speech by intuitively cutting, copying, and pasting text to speed up the process of editing speech. In the previous work, CampNet (context-aware mask prediction network) is proposed to realize text-based speech editing, significantly improving the quality of edited speech. This paper aims at a new task: adding emotional effect to the editing speech during the text-based speech editing to make the generated speech more expressive. To achieve this task, we propose Emo-CampNet (emotion CampNet), which can provide the option of emotional attributes for the generated speech in text-based speech editing and has the one-shot ability to edit unseen speakers' speech. Firstly, we propose an end-to-end emotion-selectable text-based speech editing model. The key idea of the model is to control the emotion of generated speech by introducing additional emotion attributes based on the context-aware mask prediction network. Secondly, to prevent the emotion of the generated speech from being interfered by the emotional components in the original speech, a neutral content generator is proposed to remove the emotion from the original speech, which is optimized by the generative adversarial framework. Thirdly, two data augmentation methods are proposed to enrich the emotional and pronunciation information in the training set, which can enable the model to edit the unseen speaker's speech. The experimental results that 1) Emo-CampNet can effectively control the emotion of the generated speech in the process of text-based speech editing; And can edit unseen speakers' speech. 2) Detailed ablation experiments further prove the effectiveness of emotional selectivity and data augmentation methods. The demo page is available at https://hairuo55.github.io/Emo-CampNet/
【3】 VSVC: Backdoor attack against Keyword Spotting based on Voiceprint Selection and Voice Conversion
标题：VSVC：基于声纹选择和语音转换的关键字检测后门攻击
链接：https://arxiv.org/abs/2212.10103
作者：Hanbo Cai,Pengcheng Zhang,Hai Dong,Yan Xiao,Shunhui Ji机构：College of Computer and Information, Hohai University, Nanjing, China, School of Computing Technologies, RMIT University, Melbourne, Australia, School of Computing, National University of Singapore, Singapore摘要：基于深度神经网络（DNNs）的关键词识别（KWS）在语音控制领域取得了巨大成功。然而，这种基于DNN的KWS系统的训练通常需要大量的数据和硬件资源。制造商通常将此过程委托给第三方平台。这使得训练过程无法控制，攻击者可以通过操纵第三方训练数据在模型中植入后门。有效的后门攻击可以迫使模型在某些条件下做出指定的判断，即，触发器。本文设计了一种基于声纹选择和语音转换的后门攻击方案，简称VSVC。实验结果表明，在4种受害者模型中，当中毒数据小于1%时，VSVC攻击成功率平均接近97%.摘要：Keyword spotting (KWS) based on deep neural networks (DNNs) has achieved massive success in voice control scenarios. However, training of such DNN-based KWS systems often requires significant data and hardware resources. Manufacturers often entrust this process to a third-party platform. This makes the training process uncontrollable, where attackers can implant backdoors in the model by manipulating third-party training data. An effective backdoor attack can force the model to make specified judgments under certain conditions, i.e., triggers. In this paper, we design a backdoor attack scheme based on Voiceprint Selection and Voice Conversion, abbreviated as VSVC. Experimental results demonstrated that VSVC is feasible to achieve an average attack success rate close to 97% in four victim models when poisoning less than 1% of the training data.
【4】 Visual Transformers for Primates Classification and Covid Detection
标题：视觉转换器在灵长类分类和衣物检测中的应用
链接：https://arxiv.org/abs/2212.10093
作者：Steffen Illium,Robert Müller,Andreas Sedlmeier,Claudia-Linnhoff Popien机构：Mobile and Distributed Systems Group, LMU Munich摘要：我们将视觉转换器（一种围绕注意力机制构建的深度机器学习模型）应用于原始音频记录的梅尔谱图表示。当添加基于mel的数据扩增技术和样本加权时，我们在CompParE21的两个（PRS和CCS挑战）任务上实现了相当的性能，优于大多数单一模型基线。我们进一步介绍了重叠垂直修补，并评估了参数配置的影响。索引术语：音频分类、注意力、梅尔谱图、不平衡数据集、计算副语言学摘要：We apply the vision transformer, a deep machine learning model build around the attention mechanism, on mel-spectrogram representations of raw audio recordings. When adding mel-based data augmentation techniques and sample-weighting, we achieve comparable performance on both (PRS and CCS challenge) tasks of ComParE21, outperforming most single model baselines. We further introduce overlapping vertical patching and evaluate the influence of parameter configurations. Index Terms: audio classification, attention, mel-spectrogram, unbalanced data-sets, computational paralinguistics
【5】 Exploring Effective Fusion Algorithms for Speech Based Self-Supervised Learning Models
标题：探索基于语音的自监督学习模型的有效融合算法
链接：https://arxiv.org/abs/2212.10092
作者：Changli Tang,Yujin Wang,Xie Chen,Wei-Qiang Zhang机构：Department of Electronic Engineering, Tsinghua University, Beijing, China, MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by NCMMSC2022摘要：自监督学习在语音处理等领域取得了巨大的成功。最近，在SUPERB基准测试中，与传统的手工特征（例如，FBank、MFCC）相比，基于语音的SSL模型能够在一定范围的下游任务上提取出优越的通用表示。但是，不同类型的SSL模型在不同的下游任务上可能表现出不同的优势。为了更好地利用SSL模型的潜在能力，本文探讨了多个SSL模型的有效融合。结合Hubert和Data 2 vec两种SSL模型，针对SUPERB基准测试中的两个典型任务--说话人识别（SID）和自动语音识别（ASR），研究并比较了一系列模型融合算法。实验结果表明，本文提出的融合算法能进一步显著增强个体模型的性能。摘要：Self-supervised learning (SSL) has achieved great success in various areas including speech processing. Recently, it is proven that speech based SSL models are able to extract superior universal representations on a range of downstream tasks compared to traditional hand-craft feature (e.g. FBank, MFCC) in the SUPERB benchmark. However, different types of SSL models might exhibit distinct strengths on different downstream tasks. In order to better utilize the potential power of SSL models, in this work, we explore the effective fusion on multiple SSL models. A series of model fusion algorithms are investigated and compared by combining two types of SSL models, Hubert and Data2vec, on two representative tasks from SUPERB benchmark, which are speaker identification (SID) and automatic speech recognition (ASR) tasks. The experimental results demonstrate that our proposed fusion algorithms can further boost the individual model significantly.
【6】 Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data
标题：联合语音转录与翻译：非分布数据的伪标注
链接：https://arxiv.org/abs/2212.09982
作者：Mozhdeh Gheini,Tatiana Likhomanenko,Matthias Sperber,Hendra Setiawan机构：⋄Information Sciences Institute, University of Southern California, †Apple摘要：自我训练已被证明有助于解决许多领域的数据稀缺问题，包括视觉、言语和语言。具体来说，自训练或伪标记会标记非监督数据并将其添加到训练池中。在这项工作中，我们研究和使用伪标记最近提出的新设置：语音的联合转录和翻译，其遭受缺乏足够的数据资源。在这种数据缺失的情况下，未标记数据可能与监督数据在域上发生显著变化，从而导致伪标记质量下降。我们研究了两类补救措施，它们不需要额外的监督，并且针对域不匹配：伪标签过滤和数据扩充。我们表明，伪标签分析和处理本身会在普通伪标签设置的基础上产生额外的增益，从而导致高达0.6%绝对WER和2.2 BLEU点的总改善。摘要：Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points.
【1】 TTS-Guided Training for Accent Conversion Without Parallel Data
标题：无平行数据的TTS指导的重音转换训练
链接：https://arxiv.org/abs/2212.10204
作者：Yi Zhou,Zhizheng Wu,Mingyang Zhang,Xiaohai Tian,Haizhou Li备注：5 pages, 4 figures, submitted to signal processing letter摘要：重音转换（AC）旨在将语音从一个（源）改变为另一个（目标），同时保持语音内容和说话者身份。然而，许多AC方法依赖于源-目标并行语音数据。提出了一种无需并行数据的重音转换框架。具体地，首先用目标重音语音数据预训练文本到语音（TTS）系统。此TTS模型及其隐藏表示预期仅与目标重音相关联。然后，训练语音编码器以在预先训练的TTS模型的监督下转换语音的重音。在这样做时，源重音语音及其对应的转录被分别转发到语音编码器和预训练的TTS。语音编码器的输出被优化为与TTS系统中嵌入的文本相同。在运行时，语音编码器与预先训练的TTS解码器相结合，以将源重音语音向目标转换。在实验中，我们将英语中的两种源口音（汉语和印度语）转换为目标口音（美国口音/英国口音/加拿大口音）。客观测试和主观听力测试都成功地验证了该方法在不需要任何并行数据的情况下，生成了接近目标口音的语音样本，具有较高的语音质量。摘要：Accent Conversion (AC) seeks to change the accent of speech from one (source) to another (target) while preserving the speech content and speaker identity. However, many AC approaches rely on source-target parallel speech data. We propose a novel accent conversion framework without the need of parallel data. Specifically, a text-to-speech (TTS) system is first pretrained with target-accented speech data. This TTS model and its hidden representations are expected to be associated only with the target accent. Then, a speech encoder is trained to convert the accent of the speech under the supervision of the pretrained TTS model. In doing so, the source-accented speech and its corresponding transcription are forwarded to the speech encoder and the pretrained TTS, respectively. The output of the speech encoder is optimized to be the same as the text embedding in the TTS system. At run-time, the speech encoder is combined with the pretrained TTS decoder to convert the source-accented speech toward the target. In the experiments, we converted English with two source accents (Chinese and Indian) to the target accent (American/British/Canadian). Both objective metrics and subjective listening tests successfully validate that, without any parallel data, the proposed approach generates speech samples that are close to the target accent with high speech quality.
【2】 A Simple Feature Method for Prosody Rhythm Comparison
标题：韵律节奏比较的一种简单特征方法
链接：https://arxiv.org/abs/2212.10201
作者：Mariana Julião,Alberto Abad,Helena Moniz机构：INESC-ID, Lisbon, Portugal, Instituto Superior T´ecnico, University of Lisbon, Portugal, FLUL - School of Arts and Humanities, University of Lisbon, Portugal摘要：Of all components of Prosody, Rhythm has been regarded as the hardest to address, as it is utterly linked to Pitch and Intensity. Nevertheless, Rhythm is a very good indicator of a speaker's fluency in a foreign language or even of some diseases. Canonical ways to measure Rhythm, such as $\Delta C$ or $\%V$, involve a cumbersome process of segment alignment, often leading to modest and questionable results. Perceptively, however, rhythm does not sound as difficult, as humans can grasp it even when the text is not fully intelligible. In this work, we develop an empirical and unsupervised method of rhythm assessment, which does not rely on the content. We have created a fixed-length representation of each utterance, Peak Embedding (PE), which codifies the proportional distance between peaks of the chosen Low-Level Descriptors. Clustering pairs of small sentence-like units, we have attained averages of 0.444 for Silhouette Coefficient using PE with Loudness, and 0.979 for Global Separability Index with a combination of PE with Pitch and Loudness. Clustering same-structure words, we have attained averages of 0.196 for Silhouette Coefficient and 0.864 for Global Separability Index for PE with Loudness.
【3】 Improving the quality of neural TTS using long-form content and multi-speaker multi-style modeling
标题：使用长格式内容和多说话人多风格建模提高神经TTS的质量
链接：https://arxiv.org/abs/2212.10075
作者：Tuomo Raitio,Javier Latorre,Andrea Davis,Ladan Golipour备注：Submitted to ICASSP 2023摘要：如果有足够数量的高质量语音材料可用于训练，则神经文本到语音（TTS）可以提供接近自然语音的质量。然而，获取用于TTS训练的语音数据是昂贵且耗时的，特别是如果目标是生成不同的说话风格。在这项工作中，我们表明，我们可以转移说话人之间的说话风格，并提高合成语音的质量，通过训练一个多说话人多风格（MSMS）模型与长格式录音，除了定期TTS录音。特别地，我们表明1）多说话人建模提高了整体TTS质量，2）当利用额外的多说话人数据时，所提出的MSMS方法优于预训练和微调方法，以及3）无论目标文本域如何，长格式说话风格都得到了很高的评价。摘要：Neural text-to-speech (TTS) can provide quality close to natural speech if an adequate amount of high-quality speech material is available for training. However, acquiring speech data for TTS training is costly and time-consuming, especially if the goal is to generate different speaking styles. In this work, we show that we can transfer speaking style across speakers and improve the quality of synthetic speech by training a multi-speaker multi-style (MSMS) model with long-form recordings, in addition to regular TTS recordings. In particular, we show that 1) multi-speaker modeling improves the overall TTS quality, 2) the proposed MSMS approach outperforms pre-training and fine-tuning approach when utilizing additional multi-speaker data, and 3) long-form speaking style is highly rated regardless of the target text domain.
【4】 SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks
标题：SLUE阶段-2：不同口语理解任务的基准套件
链接：https://arxiv.org/abs/2212.10525
作者：Suwon Shon,Siddhant Arora,Chyi-Jiunn Lin,Ankita Pasad,Felix Wu,Roshan Sharma,Wei-Lun Wu,Hung-Yi Lee,Karen Livescu,Shinji Watanabe机构：ASAPP, Carnegie Mellon University, National Taiwan University, Toyota Technological Institute at Chicago摘要：口语理解（SLU）任务在语音研究界已经研究了几十年，但是还没有像语音和说话人识别这样的低水平任务那样得到那么多的关注。特别是，SLU任务基准并没有那么多，而且许多现有的基准使用的数据并不是所有研究人员都能免费获得的。最近的工作已经开始为一些任务引入这样的基准数据集。在这项工作中，我们介绍了几个新的注释SLU基准测试任务，基于免费可用的语音数据，补充现有的基准测试，并解决SLU评估景观的差距。我们致力于四项任务：问题回答和概括涉及对较长语音序列的推断;命名实体定位处理定位信号中的目标内容的语音特定任务;对话行为分类识别给定语音的功能。我们遵循口语理解评估（SLUE）基准测试套件的蓝图。为了促进SLU模型的开发，利用预先训练的语音表示的成功，我们将为每个任务发布（i）相对较小的微调集的注释，（ii）注释的开发和测试集，以及（iii）基线模型，以便于再现和比较。在这项工作中，我们介绍了数据收集和注释的细节以及基线模型的性能。我们还使用了20多个最先进的语音识别模型，进行了管道模型（语音识别器+文本模型）性能对语音识别准确率的敏感性分析。摘要：Spoken language understanding (SLU) tasks have been studied for many decades in the speech research community, but have not received as much attention as lower-level tasks like speech and speaker recognition. In particular, there are not nearly as many SLU task benchmarks, and many of the existing ones use data that is not freely available to all researchers. Recent work has begun to introduce such benchmark datasets for several tasks. In this work, we introduce several new annotated SLU benchmark tasks based on freely available speech data, which complement existing benchmarks and address gaps in the SLU evaluation landscape. We contribute four tasks: question answering and summarization involve inference over longer speech sequences; named entity localization addresses the speech-specific task of locating the targeted content in the signal; dialog act classification identifies the function of a given speech utterance. We follow the blueprint of the Spoken Language Understanding Evaluation (SLUE) benchmark suite. In order to facilitate the development of SLU models that leverage the success of pre-trained speech representations, we will be publishing for each task (i) annotations for a relatively small fine-tuning set, (ii) annotated development and test sets, and (iii) baseline models for easy reproducibility and comparisons. In this work, we present the details of data collection and annotation and the performance of the baseline models. We also perform sensitivity analysis of pipeline models' performance (speech recognizer + text model) to the speech recognition accuracy, using more than 20 state-of-the-art speech recognition models.
【5】 Hopf Physical Reservoir Computer for Reconfigurable Sound Recognition
标题：用于可重构声音识别的Hopf物性储集层计算机
链接：https://arxiv.org/abs/2212.10370
*与cs.SD语音【1】为同一篇
作者：Md Raf E Ul Shougat,XiaoFu Li,Siyao Shao,Kathleen Walden McGarvey,Edmon Perkins机构：Perkins,†, Mechanical & Aerospace Engineering Department, North, Carolina State University, Entrepreneur Drive, Raleigh, NC, USA., LAB, Atwood, OK, USA., TandemLaunch, Av. Brewster, Montreal, H,C,K, Canada., echosonic, Av. Brewster, Montreal, H,C,K, Canada.摘要：霍普夫振子是一种非线性振子，表现出极限环运动。这种水库计算机利用振荡器的振动性质，这使它成为可重新配置的声音识别任务的理想候选者。本文系统地论证了Hopf水库计算机进行声音识别的能力。这项工作表明，与传统方法相比，Hopf水库计算机可以提供优越的声音识别精度（例如，梅尔谱+机器学习方法）。更重要的是，作为声音识别系统操作的Hopf水库计算机不需要音频预处理，并且具有非常简单的设置，同时仍然提供高度的可重构性。这些特性为在低功耗边缘设备中应用物理库计算进行声音识别铺平了道路。摘要：The Hopf oscillator is a nonlinear oscillator that exhibits limit cycle motion. This reservoir computer utilizes the vibratory nature of the oscillator, which makes it an ideal candidate for reconfigurable sound recognition tasks. In this paper, the capabilities of the Hopf reservoir computer performing sound recognition are systematically demonstrated. This work shows that the Hopf reservoir computer can offer superior sound recognition accuracy compared to legacy approaches (e.g., a Mel spectrum + machine learning approach). More importantly, the Hopf reservoir computer operating as a sound recognition system does not require audio preprocessing and has a very simple setup while still offering a high degree of reconfigurability. These features pave the way of applying physical reservoir computing for sound recognition in low power edge devices.
【6】 Emotion Selectable End-to-End Text-based Speech Editing
标题：基于情感选择的端到端文本语音编辑
链接：https://arxiv.org/abs/2212.10191
*与cs.SD语音【2】为同一篇
作者：Tao Wang,Jiangyan Yi,Ruibo Fu,Jianhua Tao,Zhengqi Wen,Chu Yuan Zhang机构：and also with the School of Artificial Intelligence, University of ChineseAcademy of Sciences备注：Under review, 12 pages, 11 figures, demo page is available at this https URL摘要：基于文本的语音编辑允许用户通过直观地剪切、复制和粘贴文本来编辑语音，以加快编辑语音的过程。在前期工作中，提出了CampNet（context-aware mask prediction network）来实现基于文本的语音编辑，显著提高了编辑后语音的质量。本文针对一个新课题：在基于文本的语音编辑过程中，向编辑语音添加情感效果，以使生成的语音更具表现力。为了实现这一目标，提出了Emo-CampNet（emotion CampNet），它可以为基于文本的语音编辑中生成的语音提供情感属性选项，并具有一次性编辑未见过说话人语音的能力。首先，提出了一种端到端的基于文本的情感选择语音编辑模型。该模型的核心思想是在上下文感知掩码预测网络的基础上，通过引入附加情感属性来控制生成语音的情感。其次，为了避免生成语音的情感受到原始语音中情感成分的干扰，提出了一种中性内容生成器，用于去除原始语音中的情感，并通过生成式对抗框架进行了优化。第三，提出了两种数据增强方法，丰富了训练集的情感和发音信息，使模型能够编辑未见过说话人的语音。实验结果表明：1）在基于文本的语音编辑过程中，Emo-CampNet能够有效地控制生成语音的情感;还能编辑看不见的说话人的讲话。2)详细的烧蚀实验进一步证明了情感选择和数据扩充方法的有效性。演示页可从www.example.com获得https://hairuo55.github.io/Emo-CampNet/摘要：Text-based speech editing allows users to edit speech by intuitively cutting, copying, and pasting text to speed up the process of editing speech. In the previous work, CampNet (context-aware mask prediction network) is proposed to realize text-based speech editing, significantly improving the quality of edited speech. This paper aims at a new task: adding emotional effect to the editing speech during the text-based speech editing to make the generated speech more expressive. To achieve this task, we propose Emo-CampNet (emotion CampNet), which can provide the option of emotional attributes for the generated speech in text-based speech editing and has the one-shot ability to edit unseen speakers' speech. Firstly, we propose an end-to-end emotion-selectable text-based speech editing model. The key idea of the model is to control the emotion of generated speech by introducing additional emotion attributes based on the context-aware mask prediction network. Secondly, to prevent the emotion of the generated speech from being interfered by the emotional components in the original speech, a neutral content generator is proposed to remove the emotion from the original speech, which is optimized by the generative adversarial framework. Thirdly, two data augmentation methods are proposed to enrich the emotional and pronunciation information in the training set, which can enable the model to edit the unseen speaker's speech. The experimental results that 1) Emo-CampNet can effectively control the emotion of the generated speech in the process of text-based speech editing; And can edit unseen speakers' speech. 2) Detailed ablation experiments further prove the effectiveness of emotional selectivity and data augmentation methods. The demo page is available at https://hairuo55.github.io/Emo-CampNet/
【7】 VSVC: Backdoor attack against Keyword Spotting based on Voiceprint Selection and Voice Conversion
标题：VSVC：基于声纹选择和语音转换的关键字检测后门攻击
链接：https://arxiv.org/abs/2212.10103
*与cs.SD语音【3】为同一篇
作者：Hanbo Cai,Pengcheng Zhang,Hai Dong,Yan Xiao,Shunhui Ji机构：College of Computer and Information, Hohai University, Nanjing, China, School of Computing Technologies, RMIT University, Melbourne, Australia, School of Computing, National University of Singapore, Singapore摘要：基于深度神经网络（DNNs）的关键词识别（KWS）在语音控制领域取得了巨大成功。然而，这种基于DNN的KWS系统的训练通常需要大量的数据和硬件资源。制造商通常将此过程委托给第三方平台。这使得训练过程无法控制，攻击者可以通过操纵第三方训练数据在模型中植入后门。有效的后门攻击可以迫使模型在某些条件下做出指定的判断，即，触发器。本文设计了一种基于声纹选择和语音转换的后门攻击方案，简称VSVC。实验结果表明，在4种受害者模型中，当中毒数据小于1%时，VSVC攻击成功率平均接近97%.摘要：Keyword spotting (KWS) based on deep neural networks (DNNs) has achieved massive success in voice control scenarios. However, training of such DNN-based KWS systems often requires significant data and hardware resources. Manufacturers often entrust this process to a third-party platform. This makes the training process uncontrollable, where attackers can implant backdoors in the model by manipulating third-party training data. An effective backdoor attack can force the model to make specified judgments under certain conditions, i.e., triggers. In this paper, we design a backdoor attack scheme based on Voiceprint Selection and Voice Conversion, abbreviated as VSVC. Experimental results demonstrated that VSVC is feasible to achieve an average attack success rate close to 97% in four victim models when poisoning less than 1% of the training data.
【8】 Visual Transformers for Primates Classification and Covid Detection
标题：视觉转换器在灵长类分类和衣物检测中的应用
链接：https://arxiv.org/abs/2212.10093
*与cs.SD语音【4】为同一篇
作者：Steffen Illium,Robert Müller,Andreas Sedlmeier,Claudia-Linnhoff Popien机构：Mobile and Distributed Systems Group, LMU Munich摘要：我们将视觉转换器（一种围绕注意力机制构建的深度机器学习模型）应用于原始音频记录的梅尔谱图表示。当添加基于mel的数据扩增技术和样本加权时，我们在CompParE21的两个（PRS和CCS挑战）任务上实现了相当的性能，优于大多数单一模型基线。我们进一步介绍了重叠垂直修补，并评估了参数配置的影响。索引术语：音频分类、注意力、梅尔谱图、不平衡数据集、计算副语言学摘要：We apply the vision transformer, a deep machine learning model build around the attention mechanism, on mel-spectrogram representations of raw audio recordings. When adding mel-based data augmentation techniques and sample-weighting, we achieve comparable performance on both (PRS and CCS challenge) tasks of ComParE21, outperforming most single model baselines. We further introduce overlapping vertical patching and evaluate the influence of parameter configurations. Index Terms: audio classification, attention, mel-spectrogram, unbalanced data-sets, computational paralinguistics
【9】 Exploring Effective Fusion Algorithms for Speech Based Self-Supervised Learning Models
标题：探索基于语音的自监督学习模型的有效融合算法
链接：https://arxiv.org/abs/2212.10092
*与cs.SD语音【5】为同一篇
作者：Changli Tang,Yujin Wang,Xie Chen,Wei-Qiang Zhang机构：Department of Electronic Engineering, Tsinghua University, Beijing, China, MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by NCMMSC2022摘要：自监督学习在语音处理等领域取得了巨大的成功。最近，在SUPERB基准测试中，与传统的手工特征（例如，FBank、MFCC）相比，基于语音的SSL模型能够在一定范围的下游任务上提取出优越的通用表示。但是，不同类型的SSL模型在不同的下游任务上可能表现出不同的优势。为了更好地利用SSL模型的潜在能力，本文探讨了多个SSL模型的有效融合。结合Hubert和Data 2 vec两种SSL模型，针对SUPERB基准测试中的两个典型任务--说话人识别（SID）和自动语音识别（ASR），研究并比较了一系列模型融合算法。实验结果表明，本文提出的融合算法能进一步显著增强个体模型的性能。摘要：Self-supervised learning (SSL) has achieved great success in various areas including speech processing. Recently, it is proven that speech based SSL models are able to extract superior universal representations on a range of downstream tasks compared to traditional hand-craft feature (e.g. FBank, MFCC) in the SUPERB benchmark. However, different types of SSL models might exhibit distinct strengths on different downstream tasks. In order to better utilize the potential power of SSL models, in this work, we explore the effective fusion on multiple SSL models. A series of model fusion algorithms are investigated and compared by combining two types of SSL models, Hubert and Data2vec, on two representative tasks from SUPERB benchmark, which are speaker identification (SID) and automatic speech recognition (ASR) tasks. The experimental results demonstrate that our proposed fusion algorithms can further boost the individual model significantly.
【10】 Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data
标题：联合语音转录与翻译：非分布数据的伪标注
链接：https://arxiv.org/abs/2212.09982
*与cs.SD语音【6】为同一篇
作者：Mozhdeh Gheini,Tatiana Likhomanenko,Matthias Sperber,Hendra Setiawan机构：⋄Information Sciences Institute, University of Southern California, †Apple摘要：自我训练已被证明有助于解决许多领域的数据稀缺问题，包括视觉、言语和语言。具体来说，自训练或伪标记会标记非监督数据并将其添加到训练池中。在这项工作中，我们研究和使用伪标记最近提出的新设置：语音的联合转录和翻译，其遭受缺乏足够的数据资源。在这种数据缺失的情况下，未标记数据可能与监督数据在域上发生显著变化，从而导致伪标记质量下降。我们研究了两类补救措施，它们不需要额外的监督，并且针对域不匹配：伪标签过滤和数据扩充。我们表明，伪标签分析和处理本身会在普通伪标签设置的基础上产生额外的增益，从而导致高达0.6%绝对WER和2.2 BLEU点的总改善。摘要：Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to the training pool. In this work, we investigate and use pseudo-labeling for a recently proposed novel setup: joint transcription and translation of speech, which suffers from an absence of sufficient data resources. We show that under such data-deficient circumstances, the unlabeled data can significantly vary in domain from the supervised data, which results in pseudo-label quality degradation. We investigate two categories of remedies that require no additional supervision and target the domain mismatch: pseudo-label filtering and data augmentation. We show that pseudo-label analysis and processing as such results in additional gains on top of the vanilla pseudo-labeling setup resulting in total improvements of up to 0.6% absolute WER and 2.2 BLEU points.
机器翻译，仅供参考
每日语音速递