今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。
cs.SD语音
【1】 Can one hear the position of nodes?
标题：人们能听到节点的位置吗？
链接：https://arxiv.org/abs/2211.06325
机构：Department of Software and Information Systems Engineering, Ben-Gurion University of the Negev, Beer-Sheva, Israel备注：Presented at Complex Networks 2022, Palermo, Italy摘要：波通过网络的节点和链路的传播形成了谱图理论的基础。然而，由网络形成的谐振腔中的节点发出的声音还没有得到很好的研究。由单个节点的振动发出的声音不仅反映了整个网络拓扑的结构，而且还反映了节点在网络中的位置。在这篇文章中，一个声音识别神经网络被训练来从节点的波形推断中心性度量。除了推进网络表征学习之外，节点发出的声音在大多数情况下是可信的。网络拓扑的可听化可能会开辟艺术的新方向，与网络可视化竞争。摘要：Wave propagation through nodes and links of a network forms the basis of spectral graph theory. Nevertheless, the sound emitted by nodes within the resonating chamber formed by a network are not well studied. The sound emitted by vibrations of individual nodes reflects the structure of the overall network topology but also the location of the node within the network. In this article, a sound recognition neural network is trained to infer centrality measures from the nodes' wave-forms. In addition to advancing network representation learning, sounds emitted by nodes are plausible in most cases. Auralization of the network topology may open new directions in arts, competing with network visualization.
【2】 MaskedSpeech: Context-aware Speech Synthesis with Masking Strategy
标题：MaskedSpeech：基于屏蔽策略的上下文感知语音合成
链接：https://arxiv.org/abs/2211.06170
作者：Ya-Jie Zhang,Wei Song,Yanghao Yue,Zhengchen Zhang,Youzheng Wu,Xiaodong He摘要：人类经常以连续的方式说话，这导致跨相邻话语的连贯和一致的韵律属性。然而，现有的语音合成系统大多只考虑句子中的信息，而忽略了上下文语义和声学特征。这使得不足以生成要求高表现力和自然度的高质量段落级别语音。为了合成自然、有表现力的段落语音，提出了一个上下文感知的语音合成系统MaskedSpeech，该系统综合考虑了上下文语义和声学特征。受语音编辑研究中掩蔽策略的启发，将当前句子的声学特征掩蔽出来，并与上下文语音的声学特征连接起来，进一步作为附加的模型输入。音素编码器将相邻句子的连接音素序列作为输入，并从上下文文本中学习细粒度的语义信息。此外，采用跨话语的粗粒度语义特征来提高韵律的生成效率。该模型被训练来重建掩蔽的声学特征，同时增强上下文语义和声学特征。实验结果表明，MaskedSpeech在自然度和表现力方面明显优于基准系统。摘要：Humans often speak in a continuous manner which leads to coherent and consistent prosody properties across neighboring utterances. However, most state-of-the-art speech synthesis systems only consider the information within each sentence and ignore the contextual semantic and acoustic features. This makes it inadequate to generate high-quality paragraph-level speech which requires high expressiveness and naturalness. To synthesize natural and expressive speech for a paragraph, a context-aware speech synthesis system named MaskedSpeech is proposed in this paper, which considers both contextual semantic and acoustic features. Inspired by the masking strategy in the speech editing research, the acoustic features of the current sentence are masked out and concatenated with those of contextual speech, and further used as additional model input. The phoneme encoder takes the concatenated phoneme sequence from neighboring sentences as input and learns fine-grained semantic information from contextual text. Furthermore, cross-utterance coarse-grained semantic features are employed to improve the prosody generation. The model is trained to reconstruct the masked acoustic features with the augmentation of both the contextual semantic and acoustic features. Experimental results demonstrate that the proposed MaskedSpeech outperformed the baseline system significantly in terms of naturalness and expressiveness.
【3】 SceneFake: An Initial Dataset and Benchmarks for Scene Fake Audio Detection
标题：SceneFake：一种用于场景伪音频检测的初始数据集和基准
链接：https://arxiv.org/abs/2211.06073
作者：Jiangyan Yi,Chenglong Wang,Jianhua Tao,Zhengkun Tian,Cunhang Fan,Haoxin Ma,Ruibo Fu机构：Institute of Automation, Chinese Academy of Sciences, University of Science and Technology of China, Anhui University摘要：先前的数据库已经被设计成进一步开发伪造音频检测。然而，虚假话语大多是通过改变原始音频的音色、韵律、语言内容或信道噪声而产生的。他们忽略了一个虚假的情况，其中攻击者操纵一个声学场景的原始音频与另一个伪造的。如果某些人出于恶意目的滥用被操纵的音频，将对我们的社会构成重大威胁。因此，这促使我们填补空白。本文设计了一个用于场景假音频检测（SceneFake）的数据集。SceneFake数据集中的处理音频仅涉及通过使用语音增强技术来篡改发声的声学场景。我们不仅可以检测可见测试集上的虚假话语，还可以评估虚假检测模型对不可见操纵攻击的泛化能力。SceneFake数据集中描述了一些基准测试结果。此外，还对不同语音增强技术和信噪比下的伪语音攻击进行了分析。实验结果表明，现有的ASVspoof 2019基线模型不能可靠地检测出场景操纵话语。此外，对未看见的场景操纵音频的检测仍然是具有挑战性的。摘要：Previous databases have been designed to further the development of fake audio detection. However, fake utterances are mostly generated by altering timbre, prosody, linguistic content or channel noise of original audios. They ignore a fake situation, in which the attacker manipulates an acoustic scene of the original audio with another forgery one. It will pose a major threat to our society if some people misuse the manipulated audio with malicious purpose. Therefore, this motivates us to fill in the gap. This paper designs such a dataset for scene fake audio detection (SceneFake). A manipulated audio in the SceneFake dataset involves only tampering the acoustic scene of an utterance by using speech enhancement technologies. We can not only detect fake utterances on a seen test set but also evaluate the generalization of fake detection models to unseen manipulation attacks. Some benchmark results are described on the SceneFake dataset. Besides, an analysis of fake attacks with different speech enhancement technologies and signal-to-noise ratios are presented on the dataset. The results show that scene manipulated utterances can not be detected reliably by the existing baseline models of ASVspoof 2019. Furthermore, the detection of unseen scene manipulation audio is still challenging.
【4】 Continuous Soft Pseudo-Labeling in ASR
标题：ASR中的连续软伪标记
链接：https://arxiv.org/abs/2211.06007
作者：Tatiana Likhomanenko,Ronan Collobert,Navdeep Jaitly,Samy Bengio摘要：连续伪标记（PL）算法，如slimIPL，最近已经成为语音识别中半监督学习的一种强有力的策略。与在训练模型和用其生成伪标签（PL）之间交替的早期策略相比，这里随着训练的进行以端到端的方式生成PL，从而提高了训练速度和最终模型的准确性。PL与教师-学生模型（如蒸馏）共享一个共同的主题，因为教师模型生成需要由正在训练的学生模型模仿的目标。然而，有趣的是，PL策略通常使用硬标签，而蒸馏使用标签上的分布作为模拟的目标。受蒸馏的启发，我们期望指定序列上的整体分布（又称软标签）作为未标记数据的目标，而不是单一的最佳通过伪标记转录物（硬标签），这应该会提高PL性能和收敛性。令人惊讶和出乎意料的是，我们发现软标签目标可以导致训练发散，其中模型崩溃为每帧的退化令牌分布。我们假设，硬标签不会发生这种情况的原因是，硬标签上的训练丢失强加了序列级一致性，这使得模型不会崩溃为退化解。在本文中，我们展示了几个支持这一假设的实验，并实验了几种正则化方法，这些方法可以改善使用软标签时的退化崩溃。这些方法可以使软标签的准确性更接近硬标签的准确性，虽然它们还不能胜过硬标签，但它们作为进一步改进的有用框架。摘要：Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently emerged as a powerful strategy for semi-supervised learning in speech recognition. In contrast with earlier strategies that alternated between training a model and generating pseudo-labels (PLs) with it, here PLs are generated in end-to-end manner as training proceeds, improving training speed and the accuracy of the final model. PL shares a common theme with teacher-student models such as distillation in that a teacher model generates targets that need to be mimicked by the student model being trained. However, interestingly, PL strategies in general use hard-labels, whereas distillation uses the distribution over labels as the target to mimic. Inspired by distillation we expect that specifying the whole distribution (aka soft-labels) over sequences as the target for unlabeled data, instead of a single best pass pseudo-labeled transcript (hard-labels) should improve PL performance and convergence. Surprisingly and unexpectedly, we find that soft-labels targets can lead to training divergence, with the model collapsing to a degenerate token distribution per frame. We hypothesize that the reason this does not happen with hard-labels is that training loss on hard-labels imposes sequence-level consistency that keeps the model from collapsing to the degenerate solution. In this paper, we show several experiments that support this hypothesis, and experiment with several regularization approaches that can ameliorate the degenerate collapse when using soft-labels. These approaches can bring the accuracy of soft-labels closer to that of hard-labels, and while they are unable to outperform them yet, they serve as a useful framework for further improvements.
【5】 Acoustic Pornography Recognition Using Convolutional Neural Networks and Bag of Refinements
标题：基于卷积神经网络和精化包的声学色情识别
链接：https://arxiv.org/abs/2211.05983
作者：Lifeng Zhou,Kaifeng Wei,Yuke Li,Yiya Hao,Weiqiang Yang,Haoqi Zhu机构：∗ Netease YiDun AI Lab, Hangzhou, China, † Netease CommsEase AudioLab, Hangzhou, China摘要：互联网上公开的大量色情音频严重威胁着儿童的身心健康，但这些音频却很少被检测和过滤。本文首先提出了一种基于卷积神经网络（CNN）的声学色情识别模型。然后，我们研究了一系列改进，并通过消融研究验证了其有效性。最后，我们将所有的细化叠加在一起，以验证它们是否能进一步提高模型的精度。在224127个色情音频和274206个正常音频样本的大数据集上的实验结果表明了所提模型和这些改进的有效性.具体而言，所提出的模型实现了92.46%的准确度，并且当所有细化组合时，准确度进一步提高到97.19%。摘要：A large number of pornographic audios publicly available on the Internet seriously threaten the mental and physical health of children, but these audios are rarely detected and filtered. In this paper, we firstly propose a convolutional neural networks (CNN) based model for acoustic pornography recognition. Then, we research a collection of refinements and verify their effectiveness through ablation studies. Finally, we stack all refinements together to verify whether they can further improve the accuracy of the model. Experimental results on our newly-collected large dataset consisting of 224127 pornographic audios and 274206 normal samples demonstrate the effectiveness of our proposed model and these refinements. Specifically, the proposed model achieves an accuracy of 92.46% and the accuracy is further improved to 97.19% when all refinements are combined.
【6】 A Gait Triaging Toolkit for Overlapping Acoustic Events in Indoor Environments
标题：室内环境中交叠声事件步态检测工具包
链接：https://arxiv.org/abs/2211.05944
作者：Kelvin Summoogum,Debayan Das,Parvati Jayakumar摘要：步态已被用于临床和医疗保健应用，以评估老年人的身体和认知健康。基于声学的步态检测是一种被动、非侵入式采集老年人步态数据的有效方法。然而，在开发能够在家庭和护理院的嘈杂的多音声学场景中操作的基于声学的步态检测器方面的工作有限。本文提出了一种基于机器学习的滤波器，它可以对步态音频样本进行分类，从而训练出适合于步态检测的机器学习模型。该滤波器通过消除f（1）分数为0.85的噪声样本并优先考虑具有不同频谱特征和最小噪声的步态样本来实现这一点。为了证明该滤波器的有效性，我们在应用和不应用该滤波器的老年人步态数据集上训练和评估深度学习模型。当使用过滤的步态样本进行训练时，该模型在其f（1）分数中对看不见的真实世界步态数据记录了25个点的增加。所提出的滤波器将有助于自动化手动注释步态样本的任务，以训练室内环境中老年人的基于声学的步态检测模型。摘要：Gait has been used in clinical and healthcare applications to assess the physical and cognitive health of older adults. Acoustic based gait detection is a promising approach to collect gait data of older adults passively and non-intrusively. However, there has been limited work in developing acoustic based gait detectors that can operate in noisy polyphonic acoustic scenes of homes and care homes. We attribute this to the lack of good quality gait datasets from the real-world to train a gait detector on. In this paper, we put forward a novel machine learning based filter which can triage gait audio samples suitable for training machine learning models for gait detection. The filter achieves this by eliminating noisy samples at an f(1) score of 0.85 and prioritising gait samples with distinct spectral features and minimal noise. To demonstrate the effectiveness of the filter, we train and evaluate a deep learning model on gait datasets collected from older adults with and without applying the filter. The model registers an increase of 25 points in its f(1) score on unseen real-word gait data when trained with the filtered gait samples. The proposed filter will help automate the task of manual annotation of gait samples for training acoustic based gait detection models for older adults in indoor environments.
【7】 Optimal Condition Training for Target Source Separation
标题：目标源分离的最优条件训练
链接：https://arxiv.org/abs/2211.05927
作者：Efthymios Tzinis,Gordon Wichern,Paris Smaragdis,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, University of Illinois at Urbana-Champaign, Urbana, IL, USA备注：Submitted to ICASSP 2023摘要：最近的研究已经显示了在利用多个无关的条件和非互斥的语义概念用于声源分离方面的显著性能，允许基于多个不同的查询来提取给定目标源的灵活性。本文提出了一种新的单通道目标源分离的最优条件训练（OCT）方法，该方法基于贪婪参数更新，使用与给定目标源相关的等效条件中的最高性能条件。实验表明，与单一条件模型相比，不同语义概念所携带的互补信息显著地有助于更有效地理清和隔离感兴趣的来源。此外，我们提出了一种条件细化的OCT变体，其中初始条件向量适应于给定的混合物，并被转换为更适合的表示，用于目标源提取。我们展示了OCT在不同源分离实验中的有效性，其中它改进了带有神谕分配的排列不变模型，并在更具挑战性的基于文本的源分离任务中获得了最先进的性能，甚至优于专用的纯文本条件模型。摘要：Recent research has shown remarkable performance in leveraging multiple extraneous conditional and non-mutually exclusive semantic concepts for sound source separation, allowing the flexibility to extract a given target source based on multiple different queries. In this work, we propose a new optimal condition training (OCT) method for single-channel target source separation, based on greedy parameter updates using the highest performing condition among equivalent conditions associated with a given target source. Our experiments show that the complementary information carried by the diverse semantic concepts significantly helps to disentangle and isolate sources of interest much more efficiently compared to single-conditioned models. Moreover, we propose a variation of OCT with condition refinement, in which an initial conditional vector is adapted to the given mixture and transformed to a more amenable representation for target source extraction. We showcase the effectiveness of OCT on diverse source separation experiments where it improves upon permutation invariant models with oracle assignment and obtains state-of-the-art performance in the more challenging task of text-based source separation, outperforming even dedicated text-only conditioned models.
【8】 A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding
标题：用于口语理解的预训练SSL、ASR、LM和SLU模型的整合研究
链接：https://arxiv.org/abs/2211.05869
作者：Yifan Peng,Siddhant Arora,Yosuke Higuchi,Yushi Ueda,Sujay Kumar,Karthik Ganesan,Siddharth Dalmia,Xuankai Chang,Shinji Watanabe机构：Carnegie Mellon University, Pittsburgh, PA, USA摘要：为口语理解（SLU）收集足够的标记数据是昂贵且耗时的。最近的研究通过在低资源场景下使用预训练模型取得了令人鼓舞的结果。受此启发，我们的目标是问：哪些（如果有）预培训策略可以提高SLU基准测试的性能？为了回答这个问题，我们采用了四种类型的预训练模型及其组合来进行SLU。我们利用在大量非成对数据上预先训练的自监督语音和语言模型（LM）来提取强语音和文本表示。我们还探索了使用在更大的外部自动语音识别（ASR）或SLU语料库上预先训练的监督模型。我们对SLU评估（SLUE）基准进行了广泛的实验，并观察到自监督预训练模型更强大，其中预训练LM和语音模型分别最有利于情感分析和命名实体识别任务。摘要：Collecting sufficient labeled data for spoken language understanding (SLU) is expensive and time-consuming. Recent studies achieved promising results by using pre-trained models in low-resource scenarios. Inspired by this, we aim to ask: which (if any) pre-training strategies can improve performance across SLU benchmarks? To answer this question, we employ four types of pre-trained models and their combinations for SLU. We leverage self-supervised speech and language models (LM) pre-trained on large quantities of unpaired data to extract strong speech and text representations. We also explore using supervised models pre-trained on larger external automatic speech recognition (ASR) or SLU corpora. We conduct extensive experiments on the SLU Evaluation (SLUE) benchmark and observe self-supervised pre-trained models to be more powerful, with pre-trained LM and speech models being most beneficial for the Sentiment Analysis and Named Entity Recognition task, respectively.
【9】 Continuous Emotional Intensity Controllable Speech Synthesis using Semi-supervised Learning
标题：基于半监督学习的连续情感强度可控语音合成
链接：https://arxiv.org/abs/2211.06160
作者：Yoori Oh,Juheon Lee,Yoseob Han,Kyogu Lee机构：Department of Intelligence and Information,Institute of New Media and Communications, Interdisciplinary Program in Artificial Intelligence,Artificial Intelligence Institute, Seoul National University, Member, IEEE备注：Submitted to ICASSP 2023摘要：随着语音合成系统的快速发展，最近的文本到语音模型已经达到了生成类似于人类所说的自然语音的水平。但在表现力方面仍有局限性。特别地，现有的情感语音合成模型已经显示出在情感潜在空间中使用具有缩放参数的内插特征的可控性。然而，由于情感、说话人等特征的纠缠，现有模型生成的情感潜空间难以控制连续的情感强度.本文提出一种基于半监督学习的情感强度控制方法.该模型使用从语音信息的音素级序列生成的伪标签来学习中等强度的情绪。从所提出的模型构建的嵌入空间满足具有情感基础的均匀网格几何。此外，为了提高中间情感语音的自然度，将鉴别器应用于低级别元素如持续时间、音调和能量的生成。实验结果表明，该方法具有较好的可控性和自然性。合成的语音样本可在www.example.com上获得https://tinyurl.com/34zaehh2摘要：With the rapid development of the speech synthesis system, recent text-to-speech models have reached the level of generating natural speech similar to what humans say. But there still have limitations in terms of expressiveness. In particular, the existing emotional speech synthesis models have shown controllability using interpolated features with scaling parameters in emotional latent space. However, the emotional latent space generated from the existing models is difficult to control the continuous emotional intensity because of the entanglement of features like emotions, speakers, etc. In this paper, we propose a novel method to control the continuous intensity of emotions using semi-supervised learning. The model learns emotions of intermediate intensity using pseudo-labels generated from phoneme-level sequences of speech information. An embedding space built from the proposed model satisfies the uniform grid geometry with an emotional basis. In addition, to improve the naturalness of intermediate emotional speech, a discriminator is applied to the generation of low-level elements like duration, pitch and energy. The experimental results showed that the proposed method was superior in controllability and naturalness. The synthesized speech samples are available at https://tinyurl.com/34zaehh2
【10】 Topology in Sound Synthesis and Digital Signal Processing -- DAFx2022 Lecture Notes
标题：声音合成与数字信号处理中的拓扑学--DAFx2022讲座笔记
链接：https://arxiv.org/abs/2211.05821
摘要：在奥地利维也纳举行的数字音频效果国际会议（DAFx-22）上举办的声音合成和数字信号处理中的拓扑学教程的演讲笔记。摘要：Lecture notes of a tutorial on topology in sound synthesis and digital signal processing held at international conference for digital audio effects (DAFx-22) in Vienna, Austria.
【1】 Vocal Breath Sound Based Gender Classification
标题：基于声音呼吸音的性别分类
链接：https://arxiv.org/abs/2211.06371
作者：Mohammad Shaique Solanki,Ashutosh M Bharadwaj,Jeevan K,Prasanta Kumar Ghosh机构：SPIRE Laboratory, Indian Institute of Science, Bengaluru, India备注：Submitted in ICASSP 2023摘要：已知诸如连续语音之类的浊音语音信号具有诸如音调（F0）和共振峰频率（F1、F2、F3）之类的声学特征，这些声学特征可以用于性别分类。然而，使用非语音信号（如有声呼吸音）的性别分类研究尚未被探索，因为它们缺乏典型的性别特异性声学特征。在这项工作中，我们探讨了是否声乐呼吸声编码性别信息，如果是这样，在多大程度上可以用于自动性别分类。在本研究中，我们探讨了使用数据驱动和基于知识的特征，从声乐呼吸声以及分类器的复杂性性别分类。我们还探讨了用于自动分类的呼吸信号段的位置和持续时间的重要性。用54.23分钟的男性和51.83分钟的女性呼吸音进行的实验表明，具有低复杂度分类器的基于知识的特征，即MFCC统计，与具有较高复杂度分类器的数据驱动特征相比表现得相当。无论位置如何，平均持续时间为3秒的呼吸段都是最佳选择，这避免了对呼吸循环边界注释的需要。摘要：Voiced speech signals such as continuous speech are known to have acoustic features such as pitch(F0), and formant frequencies(F1, F2, F3) which can be used for gender classification. However, gender classification studies using non-speech signals such as vocal breath sounds have not been explored as they lack typical gender-specific acoustic features. In this work, we explore whether vocal breath sounds encode gender information and if so, to what extent it can be used for automatic gender classification. In this study, we explore the use of data-driven and knowledge-based features from vocal breath sounds as well as the classifier complexity for gender classification. We also explore the importance of the location and duration of breath signal segments to be used for automatic classification. Experiments with 54.23 minutes of male and 51.83 minutes of female breath sounds reveal that knowledge-based features, namely MFCC statistics, with low-complexity classifier perform comparably to the data-driven features with classifiers of higher complexity. Breath segments with an average duration of 3 seconds are found to be the best choice irrespective of the location which avoids the need for breath cycle boundary annotation.
【2】 Enhancing and Adversarial: Improve ASR with Speaker Labels
标题：增强和对抗：使用扬声器标签提高ASR
链接：https://arxiv.org/abs/2211.06369
作者：Wei Zhou,Haotian Wu,Jingjing Xu,Mohammad Zeineldeen,Christoph Lüscher,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：submitted to ICASSP 2023摘要：ASR可以通过具有域增强或域对抗训练的多任务学习（MTL）来改进，这是两个相反的目标，其目的分别是增加/减少域向域感知/不可知ASR的域方差。在本研究中，我们研究如何最好地应用这两个相反的目标与说话人标签，以改善基于符合者的ASR。我们还提出了一个新的自适应梯度反转层，用于稳定和有效的对抗训练，而无需调整工作。通过详细的分析和实验验证，给出了ASR神经网络中应用说话人增强和对抗性训练的最佳位置。我们还探索了它们的组合以进一步改进，实现与i向量加对抗训练相同的性能。我们最好的基于扬声器的MTL在Switchboard Hub 5 '00集上实现了7%的相对改进。我们还研究了这种基于说话者的MTL对r.t.较干净的数据集和较弱的ASR NN。摘要：ASR can be improved by multi-task learning (MTL) with domain enhancing or domain adversarial training, which are two opposite objectives with the aim to increase/decrease domain variance towards domain-aware/agnostic ASR, respectively. In this work, we study how to best apply these two opposite objectives with speaker labels to improve conformer-based ASR. We also propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. We also explore their combination for further improvement, achieving the same performance as i-vectors plus adversarial training. Our best speaker-based MTL achieves 7\% relative improvement on the Switchboard Hub5'00 set. We also investigate the effect of such speaker-based MTL w.r.t. cleaner dataset and weaker ASR NN.
【3】 Continuous Emotional Intensity Controllable Speech Synthesis using Semi-supervised Learning
标题：基于半监督学习的连续情感强度可控语音合成
链接：https://arxiv.org/abs/2211.06160
* 与cs.SD语音【9】为同一篇
作者：Yoori Oh,Juheon Lee,Yoseob Han,Kyogu Lee机构：Department of Intelligence and Information,Institute of New Media and Communications, Interdisciplinary Program in Artificial Intelligence,Artificial Intelligence Institute, Seoul National University, Member, IEEE备注：Submitted to ICASSP 2023摘要：随着语音合成系统的快速发展，最近的文本到语音模型已经达到了生成类似于人类所说的自然语音的水平。但在表现力方面仍有局限性。特别地，现有的情感语音合成模型已经显示出在情感潜在空间中使用具有缩放参数的内插特征的可控性。然而，由于情感、说话人等特征的纠缠，现有模型生成的情感潜空间难以控制连续的情感强度.本文提出一种基于半监督学习的情感强度控制方法.该模型使用从语音信息的音素级序列生成的伪标签来学习中等强度的情绪。从所提出的模型构建的嵌入空间满足具有情感基础的均匀网格几何。此外，为了提高中间情感语音的自然度，将鉴别器应用于低级别元素如持续时间、音调和能量的生成。实验结果表明，该方法具有较好的可控性和自然性。合成的语音样本可在www.example.com上获得https://tinyurl.com/34zaehh2摘要：With the rapid development of the speech synthesis system, recent text-to-speech models have reached the level of generating natural speech similar to what humans say. But there still have limitations in terms of expressiveness. In particular, the existing emotional speech synthesis models have shown controllability using interpolated features with scaling parameters in emotional latent space. However, the emotional latent space generated from the existing models is difficult to control the continuous emotional intensity because of the entanglement of features like emotions, speakers, etc. In this paper, we propose a novel method to control the continuous intensity of emotions using semi-supervised learning. The model learns emotions of intermediate intensity using pseudo-labels generated from phoneme-level sequences of speech information. An embedding space built from the proposed model satisfies the uniform grid geometry with an emotional basis. In addition, to improve the naturalness of intermediate emotional speech, a discriminator is applied to the generation of low-level elements like duration, pitch and energy. The experimental results showed that the proposed method was superior in controllability and naturalness. The synthesized speech samples are available at https://tinyurl.com/34zaehh2
【4】 An Adapter based Multi-label Pre-training for Speech Separation and Enhancement
标题：一种基于适配器的多标签预训练语音分离与增强方法
链接：https://arxiv.org/abs/2211.06041
作者：Tianrui Wang,Xie Chen,Zhuo Chen,Shu Yu,Weibin Zhu机构：Institute of Information Science, Beijing Jiaotong University, Beijing, China, MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China, Peng Cheng Laboratory, Shenzhen, China摘要：近年来，自监督学习（SSL）由于其能够从大量无标记数据中提取表示而在各种语音任务中取得了巨大的成功。然而，与诸如语音识别（ASR）之类的任务相比，SSL表示在语音分离（SS）和增强（SE）方面的改进相当小。基于HuBERT，本文研究了针对SS和SE的SSL模型的改进。首先，通过综合分离和去噪项更新HuBERT的掩蔽语音预测（MSP）目标，得到多伪标签预训练方案，该方案显著改善了HuBERT的掩蔽语音和SE性能，但降低了ASR性能。为了保持其在ASR上的性能增益，我们进一步为HuBERT的Transformer编码器提出了一种基于适配器的架构，其中仅将每层的少数参数调整为多伪标签MSP，而其他参数保持冻结为默认HuBERT。实验结果表明，本文提出的基于自适应的多伪标签HuBERT算法在SE、SS和ASR任务上取得了一致和显著的性能提升，并且在仅增加边缘参数的情况下，预训练速度更快.摘要：In recent years, self-supervised learning (SSL) has achieved tremendous success in various speech tasks due to its power to extract representations from massive unlabeled data. However, compared with tasks such as speech recognition (ASR), the improvements from SSL representation in speech separation (SS) and enhancement (SE) are considerably smaller. Based on HuBERT, this work investigates improving the SSL model for SS and SE. We first update HuBERT's masked speech prediction (MSP) objective by integrating the separation and denoising terms, resulting in a multiple pseudo label pre-training scheme, which significantly improves HuBERT's performance on SS and SE but degrades the performance on ASR. To maintain its performance gain on ASR, we further propose an adapter-based architecture for HuBERT's Transformer encoder, where only a few parameters of each layer are adjusted to the multiple pseudo label MSP while other parameters remain frozen as default HuBERT. Experimental results show that our proposed adapter-based multiple pseudo label HuBERT yield consistent and significant performance improvements on SE, SS, and ASR tasks, with a faster pre-training speed, at only marginal parameters increase.
【5】 Remap, warp and attend: Non-parallel many-to-many accent conversion with Normalizing Flows
标题：重新映射、扭曲和加入：使用规格化流程进行非并行多对多重音转换
链接：https://arxiv.org/abs/2211.05850
作者：Abdelhamid Ezzerg,Thomas Merritt,Kayoko Yanagisawa,Piotr Bilinski,Magdalena Proszewska,Kamil Pokora,Renard Korzeniowski,Roberto Barra-Chicote,Daniel Korzekwa机构：Amazon Alexa , Jagiellonian University, Poland备注：IEEE Spoken Language Technology Workshop 2022摘要：同一语言的地区口音不仅影响单词的发音方式（即，语音内容），而且还影响语音的韵律方面，例如语速和语调。本文研究了一种基于流的重音转换方法。拟议的方法围绕三个步骤：重新映射语音条件以更好地匹配目标重音，扭曲转换后的语音的持续时间以更好地适合目标音素，以及隐式地对准源和目标语音序列的注意机制。所提出的重映射-扭曲-注意系统使得能够适应语音的语音和韵律方面，同时允许源语音信号和转换后的语音信号具有不同的长度。客观和主观评价表明，该方法在与目标重音的相似度、自然度和可懂度方面明显优于同类CopyCat基准模型。摘要：Regional accents of the same language affect not only how words are pronounced (i.e., phonetic content), but also impact prosodic aspects of speech such as speaking rate and intonation. This paper investigates a novel flow-based approach to accent conversion using normalizing flows. The proposed approach revolves around three steps: remapping the phonetic conditioning, to better match the target accent, warping the duration of the converted speech, to better suit the target phonemes, and an attention mechanism that implicitly aligns source and target speech sequences. The proposed remap-warp-attend system enables adaptation of both phonetic and prosodic aspects of speech while allowing for source and converted speech signals to be of different lengths. Objective and subjective evaluations show that the proposed approach significantly outperforms a competitive CopyCat baseline model in terms of similarity to the target accent, naturalness and intelligibility.
【6】 MaskedSpeech: Context-aware Speech Synthesis with Masking Strategy
标题：MaskedSpeech：基于屏蔽策略的上下文感知语音合成
链接：https://arxiv.org/abs/2211.06170
* 与cs.SD语音【2】为同一篇
作者：Ya-Jie Zhang,Wei Song,Yanghao Yue,Zhengchen Zhang,Youzheng Wu,Xiaodong He摘要：人类经常以连续的方式说话，这导致跨相邻话语的连贯和一致的韵律属性。然而，现有的语音合成系统大多只考虑句子中的信息，而忽略了上下文语义和声学特征。这使得不足以生成要求高表现力和自然度的高质量段落级别语音。为了合成自然、有表现力的段落语音，提出了一个上下文感知的语音合成系统MaskedSpeech，该系统综合考虑了上下文语义和声学特征。受语音编辑研究中掩蔽策略的启发，将当前句子的声学特征掩蔽出来，并与上下文语音的声学特征连接起来，进一步作为附加的模型输入。音素编码器将相邻句子的连接音素序列作为输入，并从上下文文本中学习细粒度的语义信息。此外，采用跨话语的粗粒度语义特征来提高韵律的生成效率。该模型被训练来重建掩蔽的声学特征，同时增强上下文语义和声学特征。实验结果表明，MaskedSpeech在自然度和表现力方面明显优于基准系统。摘要：Humans often speak in a continuous manner which leads to coherent and consistent prosody properties across neighboring utterances. However, most state-of-the-art speech synthesis systems only consider the information within each sentence and ignore the contextual semantic and acoustic features. This makes it inadequate to generate high-quality paragraph-level speech which requires high expressiveness and naturalness. To synthesize natural and expressive speech for a paragraph, a context-aware speech synthesis system named MaskedSpeech is proposed in this paper, which considers both contextual semantic and acoustic features. Inspired by the masking strategy in the speech editing research, the acoustic features of the current sentence are masked out and concatenated with those of contextual speech, and further used as additional model input. The phoneme encoder takes the concatenated phoneme sequence from neighboring sentences as input and learns fine-grained semantic information from contextual text. Furthermore, cross-utterance coarse-grained semantic features are employed to improve the prosody generation. The model is trained to reconstruct the masked acoustic features with the augmentation of both the contextual semantic and acoustic features. Experimental results demonstrate that the proposed MaskedSpeech outperformed the baseline system significantly in terms of naturalness and expressiveness.
【7】 SceneFake: An Initial Dataset and Benchmarks for Scene Fake Audio Detection
标题：SceneFake：一种用于场景伪音频检测的初始数据集和基准
链接：https://arxiv.org/abs/2211.06073
* 与cs.SD语音【3】为同一篇
作者：Jiangyan Yi,Chenglong Wang,Jianhua Tao,Zhengkun Tian,Cunhang Fan,Haoxin Ma,Ruibo Fu机构：Institute of Automation, Chinese Academy of Sciences, University of Science and Technology of China, Anhui University摘要：Previous databases have been designed to further the development of fake audio detection. However, fake utterances are mostly generated by altering timbre, prosody, linguistic content or channel noise of original audios. They ignore a fake situation, in which the attacker manipulates an acoustic scene of the original audio with another forgery one. It will pose a major threat to our society if some people misuse the manipulated audio with malicious purpose. Therefore, this motivates us to fill in the gap. This paper designs such a dataset for scene fake audio detection (SceneFake). A manipulated audio in the SceneFake dataset involves only tampering the acoustic scene of an utterance by using speech enhancement technologies. We can not only detect fake utterances on a seen test set but also evaluate the generalization of fake detection models to unseen manipulation attacks. Some benchmark results are described on the SceneFake dataset. Besides, an analysis of fake attacks with different speech enhancement technologies and signal-to-noise ratios are presented on the dataset. The results show that scene manipulated utterances can not be detected reliably by the existing baseline models of ASVspoof 2019. Furthermore, the detection of unseen scene manipulation audio is still challenging.
【8】 Continuous Soft Pseudo-Labeling in ASR
标题：ASR中的连续软伪标记
链接：https://arxiv.org/abs/2211.06007
* 与cs.SD语音【4】为同一篇
作者：Tatiana Likhomanenko,Ronan Collobert,Navdeep Jaitly,Samy Bengio摘要：连续伪标记（PL）算法，如slimIPL，最近已经成为语音识别中半监督学习的一种强有力的策略。与在训练模型和用其生成伪标签（PL）之间交替的早期策略相比，这里随着训练的进行以端到端的方式生成PL，从而提高了训练速度和最终模型的准确性。PL与教师-学生模型（如蒸馏）共享一个共同的主题，因为教师模型生成需要由正在训练的学生模型模仿的目标。然而，有趣的是，PL策略通常使用硬标签，而蒸馏使用标签上的分布作为模拟的目标。受蒸馏的启发，我们期望指定序列上的整体分布（又称软标签）作为未标记数据的目标，而不是单一的最佳通过伪标记转录物（硬标签），这应该会提高PL性能和收敛性。令人惊讶和出乎意料的是，我们发现软标签目标可以导致训练发散，其中模型崩溃为每帧的退化令牌分布。我们假设，硬标签不会发生这种情况的原因是，硬标签上的训练丢失强加了序列级一致性，这使得模型不会崩溃为退化解。在本文中，我们展示了几个支持这一假设的实验，并实验了几种正则化方法，这些方法可以改善使用软标签时的退化崩溃。这些方法可以使软标签的准确性更接近硬标签的准确性，虽然它们还不能胜过硬标签，但它们作为进一步改进的有用框架。摘要：Continuous pseudo-labeling (PL) algorithms such as slimIPL have recently emerged as a powerful strategy for semi-supervised learning in speech recognition. In contrast with earlier strategies that alternated between training a model and generating pseudo-labels (PLs) with it, here PLs are generated in end-to-end manner as training proceeds, improving training speed and the accuracy of the final model. PL shares a common theme with teacher-student models such as distillation in that a teacher model generates targets that need to be mimicked by the student model being trained. However, interestingly, PL strategies in general use hard-labels, whereas distillation uses the distribution over labels as the target to mimic. Inspired by distillation we expect that specifying the whole distribution (aka soft-labels) over sequences as the target for unlabeled data, instead of a single best pass pseudo-labeled transcript (hard-labels) should improve PL performance and convergence. Surprisingly and unexpectedly, we find that soft-labels targets can lead to training divergence, with the model collapsing to a degenerate token distribution per frame. We hypothesize that the reason this does not happen with hard-labels is that training loss on hard-labels imposes sequence-level consistency that keeps the model from collapsing to the degenerate solution. In this paper, we show several experiments that support this hypothesis, and experiment with several regularization approaches that can ameliorate the degenerate collapse when using soft-labels. These approaches can bring the accuracy of soft-labels closer to that of hard-labels, and while they are unable to outperform them yet, they serve as a useful framework for further improvements.
【9】 Align, Write, Re-order: Explainable End-to-End Speech Translation via Operation Sequence Generation
标题：对齐、写入、重新排序：通过操作序列生成实现可解释的端到端语音翻译
链接：https://arxiv.org/abs/2211.05967
作者：Motoi Omachi,Brian Yan,Siddharth Dalmia,Yuya Fujita,Shinji Watanabe机构：Yahoo Japan Corporation, Tokyo, JAPAN; ,Carnegie Mellon University, PA, USA摘要：The black-box nature of end-to-end speech translation (E2E ST) systems makes it difficult to understand how source language inputs are being mapped to the target language. To solve this problem, we would like to simultaneously generate automatic speech recognition (ASR) and ST predictions such that each source language word is explicitly mapped to a target language word. A major challenge arises from the fact that translation is a non-monotonic sequence transduction task due to word ordering differences between languages -- this clashes with the monotonic nature of ASR. Therefore, we propose to generate ST tokens out-of-order while remembering how to re-order them later. We achieve this by predicting a sequence of tuples consisting of a source word, the corresponding target words, and post-editing operations dictating the correct insertion points for the target word. We examine two variants of such operation sequences which enable generation of monotonic transcriptions and non-monotonic translations from the same speech input simultaneously. We apply our approach to offline and real-time streaming models, demonstrating that we can provide explainable translations without sacrificing quality or latency. In fact, the delayed re-ordering ability of our approach improves performance during streaming. As an added benefit, our method performs ASR and ST simultaneously, making it faster than using two separate systems to perform these tasks.
【10】 Optimal Condition Training for Target Source Separation
标题：目标源分离的最优条件训练
链接：https://arxiv.org/abs/2211.05927
* 与cs.SD语音【7】为同一篇
作者：Efthymios Tzinis,Gordon Wichern,Paris Smaragdis,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, University of Illinois at Urbana-Champaign, Urbana, IL, USA备注：Submitted to ICASSP 2023摘要：最近的研究已经显示了在利用多个无关的条件和非互斥的语义概念用于声源分离方面的显著性能，允许基于多个不同的查询来提取给定目标源的灵活性。本文提出了一种新的单通道目标源分离的最优条件训练（OCT）方法，该方法基于贪婪参数更新，使用与给定目标源相关的等效条件中的最高性能条件。实验表明，与单一条件模型相比，不同语义概念所携带的互补信息显著地有助于更有效地理清和隔离感兴趣的来源。此外，我们提出了一种条件细化的OCT变体，其中初始条件向量适应于给定的混合物，并被转换为更适合的表示，用于目标源提取。我们展示了OCT在不同源分离实验中的有效性，其中它改进了带有神谕分配的排列不变模型，并在更具挑战性的基于文本的源分离任务中获得了最先进的性能，甚至优于专用的纯文本条件模型。摘要：Recent research has shown remarkable performance in leveraging multiple extraneous conditional and non-mutually exclusive semantic concepts for sound source separation, allowing the flexibility to extract a given target source based on multiple different queries. In this work, we propose a new optimal condition training (OCT) method for single-channel target source separation, based on greedy parameter updates using the highest performing condition among equivalent conditions associated with a given target source. Our experiments show that the complementary information carried by the diverse semantic concepts significantly helps to disentangle and isolate sources of interest much more efficiently compared to single-conditioned models. Moreover, we propose a variation of OCT with condition refinement, in which an initial conditional vector is adapted to the given mixture and transformed to a more amenable representation for target source extraction. We showcase the effectiveness of OCT on diverse source separation experiments where it improves upon permutation invariant models with oracle assignment and obtains state-of-the-art performance in the more challenging task of text-based source separation, outperforming even dedicated text-only conditioned models.
机器翻译，仅供参考
每日语音速递