今天跟大家分享一篇语音相关的论文合集：cs.SD语音28篇，eess.AS音频处理38篇。
cs.SD语音
【1】 HSD: A hierarchical singing annotation dataset
标题：HSD：一个层次化的歌唱标注数据集
链接：https://arxiv.org/abs/2209.15640
作者：Xiao Fu,Xin Yuan,Jinglu Hu机构：Graduate School of Information Production and Systems, Waseda University, Kitakyushu, Fukuoka, Japan摘要：一般音乐都有明显的层次结构，尤其是流行歌曲中作为主旋律的演唱部分。然而，目前的演唱标注数据集大多只记录了音乐音符的符号信息，而忽略了音乐的结构。本文提出了一个包含68首Youtube流行歌曲的分层演唱标注数据集。该数据集以增强的LyRiCs格式记录每个音符的开始/偏移时间、音高、持续时间和歌词，以呈现音乐的分层结构。我们分两个阶段为每首歌曲添加注释：首先，建立对应乐谱及歌词档案的初始标签;第二步，参照原始音频手动校准这些标签。我们主要通过与一个自动歌唱转录（AST）数据集的比较来验证该数据集的标注准确性。实验结果表明，该数据集达到了AST数据集的标注精度。摘要：Commonly music has an obvious hierarchical structure, especially for the singing parts which usually act as the main melody in pop songs. However, most of the current singing annotation datasets only record symbolic information of music notes, ignoring the structure of music. In this paper, we propose a hierarchical singing annotation dataset that consists of 68 pop songs from Youtube. This dataset records the onset/offset time, pitch, duration, and lyric of each musical note in an enhanced LyRiCs format to present the hierarchical structure of music. We annotate each song in a two-stage process: first, create initial labels with the corresponding musical notation and lyrics file; second, manually calibrate these labels referring to the raw audio. We mainly validate the labeling accuracy of the proposed dataset by comparing it with an automatic singing transcription (AST) dataset. The result indicates that the proposed dataset reaches the labeling accuracy of AST datasets.
【2】 Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio
标题：匹配取胜：分析语音和音频中有效自我监督学习的序列长度
链接：https://arxiv.org/abs/2209.15575
作者：Yan Gao,Javier Fernandez-Marques,Titouan Parcollet,Pedro P. B. de Gusmao,Nicholas D. Lane机构：University of Cambridge,Samsung AI,Avignon University摘要：自监督学习（SSL）已被证明在语音和音频相关应用中至关重要。该范例在未标记的数据上训练通用模型，这些数据稍后可用于解决特定的下游任务。这种类型的模型训练起来成本很高，因为它需要操纵只能由强大的集中式服务器处理的长输入序列。令人惊讶的是，尽管有许多尝试通过模型压缩来提高训练效率，但是还没有研究截断输入序列长度以减少计算的效果。在本文中，我们提供了SSL预训练的第一个经验研究，用于不同的指定序列长度，并将其与各种下游任务联系起来。我们发现，短序列训练可以显著降低资源成本，同时保持所有任务的满意性能。这一简单的一行更改将促进SSL培训从数据中心向用户端边缘设备的迁移，以实现更真实和个性化的应用。摘要：Self-supervised learning (SSL) has proven vital in speech and audio-related applications. The paradigm trains a general model on unlabeled data that can later be used to solve specific downstream tasks. This type of model is costly to train as it requires manipulating long input sequences that can only be handled by powerful centralised servers. Surprisingly, despite many attempts to increase training efficiency through model compression, the effects of truncating input sequence lengths to reduce computation have not been studied. In this paper, we provide the first empirical study of SSL pre-training for different specified sequence lengths and link this to various downstream tasks. We find that training on short sequences can dramatically reduce resource costs while retaining a satisfactory performance for all tasks. This simple one-line change would promote the migration of SSL training from data centres to user-end edge devices for more realistic and personalised applications.
【3】 AudioGen: Textually Guided Audio Generation
标题：AudioGen：文本引导的音频生成
链接：https://arxiv.org/abs/2209.15352
作者：Felix Kreuk,Gabriel Synnaeve,Adam Polyak,Uriel Singer,Alexandre Défossez,Jade Copet,Devi Parikh,Yaniv Taigman,Yossi Adi机构：FAIR Team, Meta AI, The Hebrew University of Jerusalem摘要：我们解决了生成以描述性文本字幕为条件的音频样本的问题。在本研究中，我们提出了一个自动回归的产生式模型AaudioGen，它可以产生以文字输入为条件的声音样本。AudioGen在已学习的离散音频表示上运行。文本到音频生成的任务提出了多个挑战。由于音频通过介质的传播方式，区分"对象“可能是一项困难的任务（例如，分开多个人同时说话）。这由于真实世界的记录条件（例如，背景噪声、混响等）。缺少文字注释会造成另一个限制，限制缩放模型的能力。最后，对高保真音频建模需要以高采样率对音频进行编码，从而导致极长的序列。为了缓解上述挑战，我们提出了一种增强技术，该技术混合不同的音频样本，驱动模型内部学习以分离多个源。我们整理了10个包含不同类型音频和文本注释的数据集，以处理文本-音频数据点的稀缺性。为了加快推理速度，我们探索了多流建模的使用，允许使用较短的序列，同时保持相似的比特率和感知质量。我们应用无分类器指导来提高对文本的遵守。与评价的基线相比，AudioGen在客观和主观指标方面均优于基线。最后，我们探讨了所提出的方法有条件地和无条件地生成音频连续的能力。示例：https://tinyurl.com/audiogen-text2audio摘要：We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive generative model that generates audio samples conditioned on text inputs. AudioGen operates on a learnt discrete audio representation. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. We apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally. Samples: https://tinyurl.com/audiogen-text2audio
【4】 ChordMics: Acoustic Signal Purification with Distributed Microphones
标题：ChordMics：分布式麦克风的声学信号净化
链接：https://arxiv.org/abs/2209.15334
作者：Weiguo Wang,Jinming Li,Meng Jin,Yuan He机构：School of Software and BNRist, Tsinghua University摘要：声信号作为许多系统的基本输入。然而，纯声学信号很难提取，尤其是在噪声环境中。现有的波束形成系统能够提取从某些方向发射的信号。然而，由于麦克风是集中部署的，因此这些系统具有有限的覆盖范围和低空间分辨率。我们克服了上述局限性，提出了一种分布式波束形成系统ChordMics。通过利用分布式麦克风的空间多样性，ChordMics能够从任意点提取声学信号。为了实现这样的系统，我们进一步解决分布式波束成形中的基本挑战：对准由分布式和非同步的麦克风捕获的信号。我们实现了ChordMics，并在LOS和NLOS两种场景下评估了其性能。仿真结果表明，ChordMics比集中式麦克风阵列具有更高的信干噪比。平均性能增益高达15dB。摘要：Acoustic signal acts as an essential input to many systems. However, the pure acoustic signal is very difficult to extract, especially in noisy environments. Existing beamforming systems are able to extract the signal transmitted from certain directions. However, since microphones are centrally deployed, these systems have limited coverage and low spatial resolution. We overcome the above limitations and present ChordMics, a distributed beamforming system. By leveraging the spatial diversity of the distributed microphones, ChordMics is able to extract the acoustic signal from arbitrary points. To realize such a system, we further address the fundamental challenge in distributed beamforming: aligning the signals captured by distributed and unsynchronized microphones. We implement ChordMics and evaluate its performance under both LOS and NLOS scenarios. The evaluation results tell that ChordMics can deliver higher SINR than the centralized microphone array. The average performance gain is up to 15dB.
【5】 Symphony: Localizing Multiple Acoustic Sources with a Single Microphone Array
标题：交响乐：使用单个麦克风阵列定位多个声源
链接：https://arxiv.org/abs/2209.15325
作者：Weiguo Wang,Jinming Li,Yuan He,Yunhao Liu机构：School of Software and BNRist, Tsinghua University, Department of Computer Science and Engineering, Michigan State University摘要：声音识别是智能设备的一项重要且受欢迎的功能。声音的位置是与声源相关联的基本信息。除了声音识别之外，声源是否能够被定位在很大程度上影响着智能设备交互功能的能力和质量。在这项工作中，我们研究了使用智能设备（例如，亚马逊Alexa等智能扬声器）。现有的方法或者只能定位单个源，或者需要部署麦克风阵列的分布式网络来起作用。我们提出的Symphony方案是第一个使用单个麦克风阵列解决上述问题的方法。Symphony的设计理念是，阵列上麦克风的几何布局决定了来自同一信号源、沿同一到达路径的信号之间的独特关系，而信号源的位置决定了信号沿不同到达路径的DoA（到达方向）。Symphony因此包括用于区分来自沿不同路径的不同源的信号的基于几何的滤波模块和用于识别来自相同源的信号的基于相干性的模块。我们使用不同类型的商用现成麦克风阵列来实现Symphony，并评估其在不同设置下的性能。结果表明，Symphony的中值定位误差为0.694m，比最先进的方法的中值定位误差小68%。摘要：Sound recognition is an important and popular function of smart devices. The location of sound is basic information associated with the acoustic source. Apart from sound recognition, whether the acoustic sources can be localized largely affects the capability and quality of the smart device's interactive functions. In this work, we study the problem of concurrently localizing multiple acoustic sources with a smart device (e.g., a smart speaker like Amazon Alexa). The existing approaches either can only localize a single source, or require deploying a distributed network of microphone arrays to function. Our proposal called Symphony is the first approach to tackle the above problem with a single microphone array. The insight behind Symphony is that the geometric layout of microphones on the array determines the unique relationship among signals from the same source along the same arriving path, while the source's location determines the DoAs (direction-of-arrival) of signals along different arriving paths. Symphony therefore includes a geometry-based filtering module to distinguish signals from different sources along different paths and a coherence-based module to identify signals from the same source. We implement Symphony with different types of commercial off-the-shelf microphone arrays and evaluate its performance under different settings. The results show that Symphony has a median localization error of 0.694m, which is 68% less than that of the state-of-the-art approach.
【6】 Wake Word Detection Based on Res2Net
标题：基于Res2Net的唤醒词检测
链接：https://arxiv.org/abs/2209.15296
作者：Qiuchen Yu,Ruohua Zhou摘要：本文提出了一种基于Res2Net的唤醒词检测系统。Res2Net是ResNet的一个变种，它首先被应用于目标检测。Res2Net通过增加可能的感受野来实现多个特征尺度。这种多尺度机制显著提高了对不同时长唤醒词的检测能力。与基于ResNet的模型相比，Res2Net模型显著减小了模型规模，更适合于唤醒词的检测。所提出的系统可以在没有任何额外辅助的情况下从音频流中确定唤醒词的位置。在包含两个唤醒词的Mobvoi数据集上验证了该方法的有效性.在每小时0.5的误报警率下，该系统将两个唤醒字的误拒绝比现有工作减少了12%以上。摘要：This letter proposes a new wake word detection system based on Res2Net. As a variant of ResNet, Res2Net was first applied to objection detection. Res2Net realizes multiple feature scales by increasing possible receptive fields. This multiple scaling mechanism significantly improves the detection ability of wake words with different durations. Compared with the ResNet-based model, Res2Net also significantly reduces the model size and is more suitable for detecting wake words. The proposed system can determine the positions of wake words from the audio stream without any additional assistance. The proposed method is verified on the Mobvoi dataset containing two wake words. At a false alarm rate of 0.5 per hour, the system reduced the false rejection of the two wake words by more than 12% over prior works.
【7】 An efficient encoder-decoder architecture with top-down attention for speech separation
标题：一种自顶向下关注语音分离的高效编解码器结构
链接：https://arxiv.org/abs/2209.15200
作者：Kai Li,Runxuan Yang,Xiaolin Hu机构：Department of Computer Science and Technology, Tsinghua Laboratory of Brain and Intelligence (THBI), IDGMcGovern Institute of Brain Research Tsinghua University, Beijing, China摘要：深度神经网络在语音分离任务中显示出良好的前景。然而，在保持低模型复杂度的同时获得好的结果在现实世界的应用中仍然具有挑战性。本文提出了一种模拟大脑自上而下注意力的生物启发式高效编解码器架构TDANet，在不牺牲性能的前提下降低了模型复杂度。TDANet中的自顶向下注意力由全局注意力（GA）模块和级联的局部注意力（LA）层提取。遗传算法模块以多尺度声学特征作为输入，提取全局注意力信号，然后通过直接自顶向下的连接方式对不同尺度的特征进行调制。LA层使用相邻层的特征作为输入来提取局部注意信号，该局部注意信号用于以自顶向下的方式调制横向输入。在三个基准数据集上，TDANet始终实现了与先前最先进的（SOTA）方法具有竞争力的分离性能和更高的效率。具体地说，TDANet的乘累加运算（MACs）仅为Sepformer的5%，CPU推理时间仅为Sepformer的10%.此外，在3个数据集上，TDANet的大规模版本获得了SOTA结果，其MAC仍仅为Sepformer的10%，CPU推理时间仅为Sepformer的24%.我们的研究表明，自上而下的注意可能是一种更有效的语音分离策略。摘要：Deep neural networks have shown excellent prospects in speech separation tasks. However, obtaining good results while keeping a low model complexity remains challenging in real-world applications. In this paper, we provide a bio-inspired efficient encoder-decoder architecture by mimicking the brain's top-down attention, called TDANet, with decreased model complexity without sacrificing performance. The top-down attention in TDANet is extracted by the global attention (GA) module and the cascaded local attention (LA) layers. The GA module takes multi-scale acoustic features as input to extract global attention signal, which then modulates features of different scales by direct top-down connections. The LA layers use features of adjacent layers as input to extract the local attention signal, which is used to modulate the lateral input in a top-down manner. On three benchmark datasets, TDANet consistently achieved competitive separation performance to previous state-of-the-art (SOTA) methods with higher efficiency. Specifically, TDANet's multiply-accumulate operations (MACs) are only 5\% of Sepformer, one of the previous SOTA models, and CPU inference time is only 10\% of Sepformer. In addition, a large-size version of TDANet obtained SOTA results on three datasets, with MACs still only 10\% of Sepformer and the CPU inference time only 24\% of Sepformer. Our study suggests that top-down attention can be a more efficient strategy for speech separation.
【8】 An empirical study of weakly supervised audio tagging embeddings for general audio representations
标题：用于一般音频表示的弱监督音频标记嵌入的实验研究
链接：https://arxiv.org/abs/2209.15167
作者：Heinrich Dinkel,Zhiyong Yan,Yongqing Wang,Junbo Zhang,Yujun Wang机构：Xiaomi Corporation, Beijing, China摘要：我们研究了预训练的弱监督音频标记（AT）模型作为一般音频表示的特征提取器的可用性。我们主要分析了将这些嵌入转移到语音和声音领域的其他任务中的可行性。具体而言，我们将弱监督预训练模型（MobileNetV 2和EfficientNet-B 0）与现代自监督学习方法（BYOL-A）作为特征提取器进行基准测试。14个下游任务用于从乐器分类到语言分类的评估。我们的研究结果表明，AT预训练模型是音乐、事件和情绪识别任务的一个很好的迁移学习选择。此外，微调AT模型还可以有益于语音相关任务，例如关键字定位和意图分类。摘要：We study the usability of pre-trained weakly supervised audio tagging (AT) models as feature extractors for general audio representations. We mainly analyze the feasibility of transferring those embeddings to other tasks within the speech and sound domains. Specifically, we benchmark weakly supervised pre-trained models (MobileNetV2 and EfficientNet-B0) against modern self-supervised learning methods (BYOL-A) as feature extractors. Fourteen downstream tasks are used for evaluation ranging from music instrument classification to language classification. Our results indicate that AT pre-trained models are an excellent transfer learning choice for music, event, and emotion recognition tasks. Further, finetuning AT models can also benefit speech-related tasks such as keyword spotting and intent classification.
【9】 Music Source Separation with Band-split RNN
标题：基于频带分割RNN的音乐源分离
链接：https://arxiv.org/abs/2209.15174
摘要：近年来，由于新型神经网络结构和训练流水线的发展，音乐源分离（MSS）模型的性能得到了很大的提高。然而，目前的MSS模型设计主要是基于其他音频处理任务或其他研究领域，而音乐信号的内在特征和模式并没有被完全发现。本文提出了一种频域模型--频带分割RNN（BSRNN），它将混合信号的频谱图明确地分割成子带，并进行交织的频带级和序列级建模。子带的带宽的选择可以通过关于目标源的特性的先验知识或专家知识来确定，以便优化特定类型的目标乐器上的性能。为了更好地利用未标记的数据，我们还描述了一种半监督模型微调流水线，其可以进一步提高模型的性能。实验结果表明，仅在MUSDB 18-HQ数据集上训练的BSRNN显著优于2021年MDX挑战赛中排名靠前的几个模型，并且半监督微调阶段进一步提高了在所有四个乐器音轨上的性能。摘要：The performance of music source separation (MSS) models has been greatly improved in recent years thanks to the development of novel neural network architectures and training pipelines. However, recent model designs for MSS were mainly motivated by other audio processing tasks or other research fields, while the intrinsic characteristics and patterns of the music signals were not fully discovered. In this paper, we propose band-split RNN (BSRNN), a frequency-domain model that explictly splits the spectrogram of the mixture into subbands and perform interleaved band-level and sequence-level modeling. The choices of the bandwidths of the subbands can be determined by a priori knowledge or expert knowledge on the characteristics of the target source in order to optimize the performance on a certain type of target musical instrument. To better make use of unlabeled data, we also describe a semi-supervised model finetuning pipeline that can further improve the performance of the model. Experiment results show that BSRNN trained only on MUSDB18-HQ dataset significantly outperforms several top-ranking models in Music Demixing (MDX) Challenge 2021, and the semi-supervised finetuning stage further improves the performance on all four instrument tracks.
【10】 That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation标题：听起来很对：动态机器人操作的听觉自我监督链接：https://arxiv.org/abs/2210.01116
作者：Abitha Thankaraj,Lerrel Pinto机构：New York University备注：Videos and audio data are best seen on our project website: audio-robot-learning.github.io摘要：从原始的感觉数据中学习产生接触丰富的动态行为一直是机器人领域的一个长期挑战。突出的方法主要集中在使用视觉或触觉传感，其中不幸的是，一个不能捕捉高频交互，而另一个可能太微妙，无法进行大规模数据收集。在本文中，我们提出了一种以数据为中心的动态操作方法，该方法使用了一个经常被忽略的信息源：声音。我们首先使用商用接触式麦克风收集了五个动态任务中25k个交互声音对的数据集。然后，根据这些数据，我们利用自监督学习来加速根据声音进行的行为预测。我们的实验表明，这种自我监督的“预训练”对于实现高性能是至关重要的，与普通监督学习相比，MSE降低了34.5%，与视觉训练相比，MSE降低了54.3%。重要的是，我们发现，当要求生成所需的声音配置文件时，在UR10机器人上在线推出我们的模型可以产生动态行为，在音频相似性度量方面比监督学习平均提高11.5%。摘要：Learning to produce contact-rich, dynamic behaviors from raw sensory data has been a longstanding challenge in robotics. Prominent approaches primarily focus on using visual or tactile sensing, where unfortunately one fails to capture high-frequency interaction, while the other can be too delicate for large-scale data collection. In this work, we propose a data-centric approach to dynamic manipulation that uses an often ignored source of information: sound. We first collect a dataset of 25k interaction-sound pairs across five dynamic tasks using commodity contact microphones. Then, given this data, we leverage self-supervised learning to accelerate behavior prediction from sound. Our experiments indicate that this self-supervised 'pretraining' is crucial to achieving high performance, with a 34.5% lower MSE than plain supervised learning and a 54.3% lower MSE over visual training. Importantly, we find that when asked to generate desired sound profiles, online rollouts of our models on a UR10 robot can produce dynamic behavior that achieves an average of 11.5% improvement over supervised learning on audio similarity metrics.
【11】 Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection标题：Push-Pull：表征视听主动说话人检测的对抗稳健性链接：https://arxiv.org/abs/2210.00753
作者：Xuanjun Chen,Haibin Wu,Helen Meng,Hung-yi Lee,Jyh-Shing Roger Jang机构：Department. of Computer Science Information Engineering, National Taiwan University, Graduate Institute of Communication Engineering, National Taiwan University, Centre for Perceptual and Interactive Intelligence, The Chinese University of Hong Kong备注：Accepted by SLT 2022摘要：音视频主动说话人检测（AVASD）是一种成熟的检测技术，目前已成为多模态应用中不可缺少的前端技术。然而，据我们所知，AVASD模型的对抗鲁棒性还没有得到研究，更不用说对此类攻击的有效防御了。本文首先通过大量的实验揭示了AVASD模型在纯音频攻击、纯视频攻击和音频视频对抗攻击下的脆弱性。此外，我们还提出了一个新的视听交互损失（AVIL），使攻击者难以找到可行的敌对例子在分配的攻击预算。这种损失的目的是使类间嵌入（即非语音簇和语音簇）充分地分散，并使类内嵌入尽可能地靠近以保持它们的紧凑。实验结果表明，AVIL在多模态攻击下的性能比对抗性训练提高了33.14mAP（%）。摘要：Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.
【12】 Efficient acoustic feature transformation in mismatched environments using a Guided-GAN标题：利用导向型GaN实现失配环境下的高效声特征变换链接：https://arxiv.org/abs/2210.00721
作者：Walter Heymans,Marelie H. Davel,Charl van Heerden机构：North-West University, South Africa备注：None摘要：本文提出了一种新的框架，以改善在资源稀缺的环境中的自动语音识别（ASR）系统使用一个生成式对抗网络（GAN）的声学输入特征操作。GAN用于在解码之前增强不匹配数据的特征，或者可以可选地用于微调声学模型。我们实现了与多风格训练（MTR）相当的改进，但计算成本更低。利用少于一个小时的数据，在高质量数据上训练并在不匹配的音频上评估的ASR系统的相对字错误率（WER）提高了11.5%到19.7%。实验表明，该框架在训练数据和计算资源有限的资源不足的环境中非常有用。GAN不需要并行训练数据，因为它利用基线声学模型来提供附加损耗项，该附加损耗项引导发生器创建通过基线更好地分类的声学特征。摘要：We propose a new framework to improve automatic speech recognition (ASR) systems in resource-scarce environments using a generative adversarial network (GAN) operating on acoustic input features. The GAN is used to enhance the features of mismatched data prior to decoding, or can optionally be used to fine-tune the acoustic model. We achieve improvements that are comparable to multi-style training (MTR), but at a lower computational cost. With less than one hour of data, an ASR system trained on good quality data, and evaluated on mismatched audio is improved by between 11.5% and 19.7% relative word error rate (WER). Experiments demonstrate that the framework can be very useful in under-resourced environments where training data and computational resources are limited. The GAN does not require parallel training data, because it utilises a baseline acoustic model to provide an additional loss term that guides the generator to create acoustic features that are better classified by the baseline.
【13】 SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model标题：SpeechCLIP：将语音与预先训练的视觉和语言模型相结合链接：https://arxiv.org/abs/2210.00705
作者：Yi-Jen Shih,Hsuan-Fu Wang,Heng-Jui Chang,Layne Berry,Hung-yi Lee,David Harwath机构：National Taiwan University, MIT CSAIL, The University of Texas at Austin备注：Accepted to IEEE SLT 2022摘要：数据驱动的语音处理模型通常在大量的文本监督下表现良好，但是收集转录的语音数据是昂贵的。因此，我们提出了一个新的框架SpeechCLIP，它通过图像将语音和文本连接起来，以增强语音模型而无需转录。我们利用最先进的预先训练的HuBERT和CLIP，通过成对的图像和语音字幕将它们对齐，只需最少的微调。SpeechCLIP在图像-语音检索方面优于现有技术，并且在没有来自转录的直接监督的情况下执行zero-shot语音-文本检索。此外，SpeechCLIP可以直接从语音中检索语义相关的关键词。摘要：Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.
【14】 Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition标题：用于设备语音识别的换能器多级累进压缩链接：https://arxiv.org/abs/2210.00169
作者：Jash Rathod,Nauman Dawalatabad,Shatrughan Singh,Dhananjaya Gowda机构：Samsung Research, Massachusetts Institute of Technology备注：None摘要：智能设备中较小的存储器带宽促使开发较小的自动语音识别（ASR）模型。为了获得较小的模型，可以采用模型压缩技术。知识蒸馏（KD）是一种流行的模型压缩方法，它可以实现更小的模型大小，同时模型性能的下降相对较小。在该方法中，知识从经训练的大尺寸教师模型提取到较小尺寸的学生模型。此外，基于传感器的模型最近已经显示出对于设备上流式ASR任务执行良好，而构象器模型在处理长期依赖性方面是有效的。因此，在这项工作中，我们采用了一个流式传感器架构与构象作为编码器。我们提出了一个多阶段渐进式的方法来压缩构象传感器模型使用KD。在多阶段设置中，我们使用提取的学生模型逐步更新我们的教师模型。在标准的LibriSpeech数据集上，我们的实验结果成功地实现了60%以上的压缩率，与更大的教师模型相比，性能没有明显下降。摘要：The smaller memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. To obtain a smaller model, one can employ the model compression techniques. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size with relatively lesser degradation in the model performance. In this approach, knowledge is distilled from a trained large size teacher model to a smaller size student model. Also, the transducer based models have recently shown to perform well for on-device streaming ASR task, while the conformer models are efficient in handling long term dependencies. Hence in this work we employ a streaming transducer architecture with conformer as the encoder. We propose a multi-stage progressive approach to compress the conformer transducer model using KD. We progressively update our teacher model with the distilled student model in a multi-stage setup. On standard LibriSpeech dataset, our experimental results have successfully achieved compression rates greater than 60% without significant degradation in the performance compared to the larger teacher model.
【15】 WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration标题：WaveFit：一种基于定点迭代的迭代非自回归神经声码器链接：https://arxiv.org/abs/2210.01029
作者：Yuma Koizumi,Kohei Yatabe,Heiga Zen,Michiel Bacchiani机构：Google Research, Japan, Tokyo University of Agriculture and Technology, Japan备注：Accepted to IEEE SLT 2022摘要：去噪扩散概率模型（DDPMs）和生成式对抗网络（GANs）是神经声码器常用的生成式模型。DDPM和GANs可以分别由迭代去噪框架和对抗训练来表征。本文提出了一种快速、高质量的神经声码器\textit{WaveFit}，它将GANs的本质融入到一个基于定点迭代的类DDPM迭代框架中。WaveFit迭代地对输入信号去噪，并且训练深度神经网络（DNN），用于最小化在所有迭代处从中间输出计算的对抗损失。主观（并排）听音测试显示，人类自然语音和通过WaveFit五次迭代合成的语音之间在自然度方面没有统计学显著差异。此外，WaveFit的推理速度比WaveRNN快240倍。音频演示可从以下网址获得：摘要：Denoising diffusion probabilistic models (DDPMs) and generative adversarial networks (GANs) are popular generative models for neural vocoders. The DDPMs and GANs can be characterized by the iterative denoising framework and adversarial training, respectively. This study proposes a fast and high-quality neural vocoder called \textit{WaveFit}, which integrates the essence of GANs into a DDPM-like iterative framework based on fixed-point iteration. WaveFit iteratively denoises an input signal, and trains a deep neural network (DNN) for minimizing an adversarial loss calculated from intermediate outputs at all iterations. Subjective (side-by-side) listening tests showed no statistically significant differences in naturalness between human natural speech and those synthesized by WaveFit with five iterations. Furthermore, the inference speed of WaveFit was more than 240 times faster than WaveRNN. Audio demos are available at \url{google.github.io/df-conformer/wavefit/}.
【16】 Simple Pooling Front-ends For Efficient Audio Classification标题：简单的池化前端，实现高效的音频分类链接：https://arxiv.org/abs/2210.00943
作者：Xubo Liu,Haohe Liu,Qiuqiang Kong,Xinhao Mei,Mark D. Plumbley,Wenwu Wang机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK, Speech, Audio, and Music Intelligence (SAMI) Group, ByteDance, China备注：Submitted to ICASSP 2023摘要：近来，对于构建用于设备上场景的高效音频神经网络的兴趣日益增加。而大多数现有的方法被设计成使用诸如模型修剪的方法来减小音频神经网络的大小。在这项工作中，我们表明，不是使用复杂的方法来减少模型大小，而是消除输入音频特征中的时间冗余（例如，Mel-spectrogram）是一种有效的音频分类方法。为此，我们提出了一系列简单池前端（SimPF），它们使用简单的非参数池操作来减少Mel谱图中的冗余信息。我们在四个音频分类任务上进行了广泛的实验来评估SimPF的性能。实验结果表明，SimPF算法可以将现有的音频神经网络的FLOP减少一半以上，而音频分类性能的下降可以忽略不计，甚至有相当大的提高。摘要：Recently, there has been increasing interest in building efficient audio neural networks for on-device scenarios. While most existing approaches are designed to reduce the size of audio neural networks using methods such as model pruning. In this work, we show that instead of reducing model size using complex methods, eliminating the temporal redundancy in the input audio features (e.g., Mel-spectrogram) could be an effective approach for efficient audio classification. To do so, we proposed a family of simple pooling front-ends (SimPFs) which use simple non-parametric pooling operations to reduce the redundant information within the Mel-spectrogram. We perform extensive experiments on four audio classification tasks to evaluate the performance of SimPFs. Experimental results show that SimPFs can achieve a reduction in more than half of the FLOPs for off-the-shelf audio neural networks, with negligible degradation or even decent improvement in audio classification performance.
【17】 Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings标题：音乐到文本通感：从音乐录音生成描述性文本链接：https://arxiv.org/abs/2210.00434
作者：Zhihuan Kuang,Shi Zong,Jianbing Zhang,Jiajun Chen,Hongfu Liu机构：Nanjing University, Brandeis University摘要：本文提出了一个新的研究课题--音乐与文本通感。与传统的音乐标注问题不同，音乐-文本通感问题的目标是从音乐记录中生成描述性文本，以供进一步理解。虽然这是机器学习领域一个新的有趣的应用，但据我们所知，现有的音乐相关数据集不包含音乐录音的语义描述，无法完成音乐到文本的通感任务。有鉴于此，我们收集了一个新的数据集，其中包含1,955对对齐的古典音乐录音和文本描述。在此基础上，我们建立了一个计算模型，生成能够描述音乐录音内容的句子。针对古典音乐的高度非区分性，在计算模型中设计了一种群拓扑保持损失，将更多的样本作为群参考，保持了不同样本之间的相对拓扑关系.实验结果定性和定量地证明了该模型在5种启发式或预先训练的竞争方法及其变体上的有效性.摘要：In this paper, we consider a novel research problem, music-to-text synaesthesia. Different from the classical music tagging problem that classifies a music recording into pre-defined categories, the music-to-text synaesthesia aims to generate descriptive texts from music recordings for further understanding. Although this is a new and interesting application to the machine learning community, to our best knowledge, the existing music-related datasets do not contain the semantic descriptions on music recordings and cannot serve the music-to-text synaesthesia task. In light of this, we collect a new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions. Based on this, we build a computational model to generate sentences that can describe the content of the music recording. To tackle the highly non-discriminative classical music, we design a group topology-preservation loss in our computational model, which considers more samples as a group reference and preserves the relative topology among different samples. Extensive experimental results qualitatively and quantitatively demonstrate the effectiveness of our proposed model over five heuristics or pre-trained competitive methods and their variants on our collected dataset.
【18】 Voice Spoofing Countermeasures: Taxonomy, State-of-the-art, experimental analysis of generalizability, open challenges, and the way forward标题：语音欺骗对策：分类、最新技术、可概括性实验分析、开放挑战和前进方向链接：https://arxiv.org/abs/2210.00417
作者：Awais Khan,Khalid Mahmood Malik,James Ryan,Mikul Saravanan机构：Department of Computer Science and Engineering, Oakland University, Rochester, MI , USA摘要：恶意行为者可能试图使用不同的语音欺骗攻击来欺骗ASV系统，甚至使用它们来传播错误信息。已经提出了各种对策来检测这些欺骗攻击。由于在过去的6-7年中在自动说话人确认（ASV）系统中的欺骗检测方面所做的大量工作，因此有必要对研究进行分类，并对最新的对策进行定性和定量比较。此外，没有现有的调查报告回顾了语音欺骗评估和说话人验证、欺骗对策的对抗性/反取证攻击和ASV本身的集成解决方案，或使用单个模型检测多个攻击的统一解决方案。此外，还没有对已发表的对策进行对等比较，以便通过跨语料库评估它们来评估它们的普遍性。在本文中，我们对使用手工特征、深度学习、端到端和通用欺骗对策解决方案检测语音合成（SS）、语音转换（VC）和重放攻击的欺骗检测文献进行了回顾。此外，我们还回顾了语音欺骗评估和说话人验证、对抗性和反取证攻击的语音对抗和ASV的集成解决方案。最后，分析了现有电子欺骗对抗措施的局限性和面临的挑战。我们报告了这些对策在几个数据集上的表现，并在语料库中进行了评估。对于实验，我们使用ASVspoof 2019和VSDC数据集以及GMM、SVM、CNN和CNN-GRU分类器。(For测试平台的代码可以在我们的GitHub Repository中找到。摘要：Malicious actors may seek to use different voice-spoofing attacks to fool ASV systems and even use them for spreading misinformation. Various countermeasures have been proposed to detect these spoofing attacks. Due to the extensive work done on spoofing detection in automated speaker verification (ASV) systems in the last 6-7 years, there is a need to classify the research and perform qualitative and quantitative comparisons on state-of-the-art countermeasures. Additionally, no existing survey paper has reviewed integrated solutions to voice spoofing evaluation and speaker verification, adversarial/antiforensics attacks on spoofing countermeasures, and ASV itself, or unified solutions to detect multiple attacks using a single model. Further, no work has been done to provide an apples-to-apples comparison of published countermeasures in order to assess their generalizability by evaluating them across corpora. In this work, we conduct a review of the literature on spoofing detection using hand-crafted features, deep learning, end-to-end, and universal spoofing countermeasure solutions to detect speech synthesis (SS), voice conversion (VC), and replay attacks. Additionally, we also review integrated solutions to voice spoofing evaluation and speaker verification, adversarial and anti-forensics attacks on voice countermeasures, and ASV. The limitations and challenges of the existing spoofing countermeasures are also presented. We report the performance of these countermeasures on several datasets and evaluate them across corpora. For the experiments, we employ the ASVspoof2019 and VSDC datasets along with GMM, SVM, CNN, and CNN-GRU classifiers. (For reproduceability of the results, the code of the test bed can be found in our GitHub Repository.
【19】 Optimized Decoders for Mixed-Order Ambisonics标题：一种适用于混合阶数的优化解码器链接：https://arxiv.org/abs/2210.00378
作者：Aaron Heller,Eric Benjamin,Fernando Lopez-Lezcano机构：Artificial Intelligence Center, SRI International, Menlo Park, CA, Surround Research, Pacifica, CA, Center for Computer Research in Music and Acoustics (CCRMA), Stanford University, Stanford, CA备注：None摘要：在本文中，我们讨论的动机，设计，和分析的立体声解码器的系统，其中垂直阶比水平阶少，称为混合阶立体声系统。这可能是由于使用了强调水平空间分辨率的麦克风阵列或提供垂直稀疏覆盖的扬声器阵列。首先，我们回顾了Gerzon定义的双声道再现标准，并总结了最近关于各种标准的相对感知重要性的结果。然后，我们表明，使用全阶解码器与混合顺序节目材料导致比使用适当设计的混合顺序解码器更差的性能。然后，我们介绍了解码器优化器的一种新实现，该优化器利用机器学习技术实现快速和稳健的收敛，讨论了目标函数的构造，并将其应用于设计混合阶信号集和非均匀扬声器布局的双频段解码器的问题。本文总结了非正式听力测试的结果，并讨论了未来的发展方向。摘要：In this paper we discuss the motivation, design, and analysis of ambisonic decoders for systems where the vertical order is less than the horizontal order, known as mixed-order Ambisonic systems. This can be due to the use of microphone arrays that emphasize horizontal spatial resolution or speaker arrays that provide sparser coverage vertically. First, we review Ambisonic reproduction criteria, as defined by Gerzon, and summarize recent results on the relative perceptual importance of the various criteria. Then we show that using full-order decoders with mixed-order program material results in poorer performance than with a properly designed mixed-order decoder. We then introduce a new implementation of a decoder optimizer that draws upon techniques from machine learning for quick and robust convergence, discuss the construction of the objective function, and apply it to the problem of designing two-band decoders for mixed-order signal sets and non-uniform loudspeaker layouts. Results of informal listening tests are summarized and future directions discussed.
【20】 Fine-tuning Wav2vec for Vocal-burst Emotion Recognition标题：微调Wav2vec用于发声情感识别链接：https://arxiv.org/abs/2210.00263
作者：Dang-Khanh Nguyen,Sudarshan Pant,Ngoc-Huynh Ho,Guee-Sang Lee,Soo-Huyng Kim,Hyung-Jeong Yang机构：Department of AI Convergence, Chonnam National University, Gwangju, South Korea摘要：ACII情感爆发式发声（A-VB）竞赛引入了情感计算的一个新主题，即使用人类的非语言声音来理解情感表达。我们熟悉通过言语、声音或面部表情进行情绪识别。然而，诸如笑、哭和手势之类的声音突发没有被利用，即使它们对于行为分析是非常有用的。A-VB竞赛包括四项任务，探索不同空间中的非语言信息。本技术报告描述了SclabCNU团队完成挑战任务的方法和结果。与组织者提供的基线模型相比，我们取得了令人鼓舞的结果。摘要：The ACII Affective Vocal Bursts (A-VB) competition introduces a new topic in affective computing, which is understanding emotional expression using the non-verbal sound of humans. We are familiar with emotion recognition via verbal vocal or facial expression. However, the vocal bursts such as laughs, cries, and signs, are not exploited even though they are very informative for behavior analysis. The A-VB competition comprises four tasks that explore non-verbal information in different spaces. This technical report describes the method and the result of SclabCNU Team for the tasks of the challenge. We achieved promising results compared to the baseline model provided by the organizers.
【21】 Pre-trained Speech Representations as Feature Extractors for Speech Quality Assessment in Online Conferencing Applications标题：用于在线会议语音质量评估的预训练语音表示作为特征提取链接：https://arxiv.org/abs/2210.00259
作者：Bastiaan Tamm,Helena Balabin,Rik Vandenberghe,Hugo Van hamme机构：Laboratory for Cognitive Neurology, KU Leuven, Belgium, Processing Speech and Images, KU Leuven, Belgium备注：None摘要：在线会议应用中的语音质量通常通过平均意见得分（MOS）度量形式的人类判断来评估。由于这种劳动密集型方法在大多数设置中对于大规模语音质量评估是不可行的，所以焦点已经转移到通过深度神经网络（DNN）的端到端训练的自动MOS预测。我们建议利用来自预训练的基于wav 2 vec的XLS-R模型的语音表示，而不是从头开始训练网络。然而，这种模型的参数数量超过任务特定的DNN几个数量级，这对在较小数据集上产生的微调过程提出了挑战。因此，我们选择在特征提取中使用来自XLS-R的预先训练的语音表示，而不是微调设置，从而显著减少可训练模型参数的数量。我们将我们提出的基于XLS-R的特征提取器与基于Mel频率倒谱系数（MFCC）的特征提取器进行了比较，并对双向长短期记忆（Bi-LSTM）和注意力池前馈（AttPoolFF）网络的各种组合进行了实验，这些组合是在特征提取器的输出上训练的。我们在ConferencingSpeech 2022 MOS预测任务中证明了预训练XLS-R嵌入在降低均方根误差（RMSE）方面的性能提高。摘要：Speech quality in online conferencing applications is typically assessed through human judgements in the form of the mean opinion score (MOS) metric. Since such a labor-intensive approach is not feasible for large-scale speech quality assessments in most settings, the focus has shifted towards automated MOS prediction through end-to-end training of deep neural networks (DNN). Instead of training a network from scratch, we propose to leverage the speech representations from the pre-trained wav2vec-based XLS-R model. However, the number of parameters of such a model exceeds task-specific DNNs by several orders of magnitude, which poses a challenge for resulting fine-tuning procedures on smaller datasets. Therefore, we opt to use pre-trained speech representations from XLS-R in a feature extraction rather than a fine-tuning setting, thereby significantly reducing the number of trainable model parameters. We compare our proposed XLS-R-based feature extractor to a Mel-frequency cepstral coefficient (MFCC)-based one, and experiment with various combinations of bidirectional long short term memory (Bi-LSTM) and attention pooling feedforward (AttPoolFF) networks trained on the output of the feature extractors. We demonstrate the increased performance of pre-trained XLS-R embeddings in terms a reduced root mean squared error (RMSE) on the ConferencingSpeech 2022 MOS prediction task.
【22】 Blind Signal Dereverberation for Machine Speech Recognition标题：机器语音识别中的盲信号去混响链接：https://arxiv.org/abs/2210.00117
作者：Samik Sadhu,Hynek Hermansky机构：Center for Language and Speech Processing, Johns Hopkins University, USA, Human Language Technology Center of Excellence, Johns Hopkins University, USA摘要：我们提出了一种利用来自混响环境的一定量的训练语音数据和任何可用的非混响语音数据来去除由记录环境的混响引入到语音中的未知卷积噪声的方法。使用在长时间窗上计算的傅里叶变换，其理想地覆盖整个房间脉冲响应，我们将房间引起的卷积转换为对数谱域中的加法。接下来，我们根据在对数频谱域中的混响语音以及干净语音上收集的统计来计算频谱归一化向量。在操作期间，该归一化矢量被用于减轻来自在相同混响条件下记录的复杂语音频谱的混响。这种去混响的复语音频谱被用来计算复FDLP频谱图，以用于自动语音识别。摘要：We present a method to remove unknown convolutive noise introduced to speech by reverberations of recording environments, utilizing some amount of training speech data from the reverberant environment, and any available non-reverberant speech data. Using Fourier transform computed over long temporal windows, which ideally cover the entire room impulse response, we convert room induced convolution to additions in the log spectral domain. Next, we compute a spectral normalization vector from statistics gathered over reverberated as well as over clean speech in the log spectral domain. During operation, this normalization vectors are used to alleviate reverberations from complex speech spectra recorded under the same reverberant conditions . Such dereverberated complex speech spectra are used to compute complex FDLP-spectrograms for use in automatic speech recognition.
【23】 Learning the Spectrogram Temporal Resolution for Audio Classification标题：用于音频分类的语谱图时间分辨率学习链接：https://arxiv.org/abs/2210.01719
作者：Haohe Liu,Xubo Liu,Qiuqiang Kong,Wenwu Wang,Mark D. Plumbley备注：Under review. Code open-sourced at this https URL摘要：音频谱图是一种时频表示，已被广泛用于音频分类。频谱图的时间分辨率取决于跳数大小。以前的工作通常假设跳大小应该是一个恒定值，例如10毫秒。然而，固定的跳跃大小或分辨率对于不同类型的声音并不总是最佳的。提出了一种新的基于差分时间分辨率学习的音频分类方法DiffRes，该方法能够提高音频分类模型的性能。给定一个用固定跳数计算的频谱图，DiffRes合并非必要的时间帧，同时保留重要的帧。DiffRes充当音频频谱图和分类器之间的“插入”模块，并且可以进行端到端优化。我们在mel谱图上评估DiffRes，然后使用最先进的分类器骨干，并将其应用于五个不同的子任务。与固定分辨率的Mel谱图相比，基于DiffRes的方法在特征层面上至少减少了25%的时间维度，同时降低了计算成本，达到了相同甚至更好的分类精度。从一个高时间分辨率的频谱图（如1毫秒的跳数）开始，我们证明了DiffRes可以在相同的计算复杂度下提高分类精度。摘要：The audio spectrogram is a time-frequency representation that has been widely used for audio classification. The temporal resolution of a spectrogram depends on hop size. Previous works generally assume the hop size should be a constant value such as ten milliseconds. However, a fixed hop size or resolution is not always optimal for different types of sound. This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution learning to improve the performance of audio classification models. Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames. DiffRes acts as a "drop-in" module between an audio spectrogram and a classifier, and can be end-to-end optimized. We evaluate DiffRes on the mel-spectrogram, followed by state-of-the-art classifier backbones, and apply it to five different subtasks. Compared with using the fixed-resolution mel-spectrogram, the DiffRes-based method can achieve the same or better classification accuracy with at least 25% fewer temporal dimensions on the feature level, which alleviates the computational cost at the same time. Starting from a high-temporal-resolution spectrogram such as one-millisecond hop size, we show that DiffRes can improve classification accuracy with the same computational complexity.
【24】 Improving Label-Deficient Keyword Spotting Using Self-Supervised Pretraining标题：基于自监督预训练的标签缺陷关键词检测方法链接：https://arxiv.org/abs/2210.01703
作者：Holger Severin Bovbjerg,Zheng-Hua Tan备注：8 pages, 3 figures, 4 tables, Submitted to Northern Lights Deep Learning Conference 2023摘要：近年来，准确的深度关键词识别（KWS）模型的发展已经导致KWS技术被嵌入到诸如语音助手的许多技术中。其中许多模型都依赖于大量的标记数据来实现良好的性能。结果，它们的使用被限制在可以获得大的标记语音数据集的应用中。自监督学习试图通过利用更容易大量获得的未标记数据来减少对大的标记数据集的需要。然而，大多数自监督方法仅针对非常大的模型进行了研究，而KWS模型期望是小的。在这篇文章中，我们研究了在标签缺失的情况下使用自监督预训练来训练较小的KWS模型。我们使用自监督框架Data2Vec对关键词Transformer模型进行了预训练，并在Google Speech Commands数据集的标签缺失设置上进行了实验。实验结果表明，经过预训练的模型性能明显优于未经过预训练的模型，表明Data2Vec预训练可以提高KWS模型在标签缺失场景下的性能。源代码是公开的。摘要：In recent years, the development of accurate deep keyword spotting (KWS) models has resulted in KWS technology being embedded in a number of technologies such as voice assistants. Many of these models rely on large amounts of labelled data to achieve good performance. As a result, their use is restricted to applications for which a large labelled speech data set can be obtained. Self-supervised learning seeks to mitigate the need for large labelled data sets by leveraging unlabelled data, which is easier to obtain in large amounts. However, most self-supervised methods have only been investigated for very large models, whereas KWS models are desired to be small. In this paper, we investigate the use of self-supervised pretraining for the smaller KWS models in a label-deficient scenario. We pretrain the Keyword Transformer model using the self-supervised framework Data2Vec and carry out experiments on a label-deficient setup of the Google Speech Commands data set. It is found that the pretrained models greatly outperform the models without pretraining, showing that Data2Vec pretraining can increase the performance of KWS models in label-deficient scenarios. The source code is made publicly available.
【25】 Code-Switching without Switching: Language Agnostic End-to-End Speech Translation标题：语码转换不转换：语言不可知的端到端语音翻译链接：https://arxiv.org/abs/2210.01512
作者：Christian Huber,Enes Yavuz Ugan,Alexander Waibel摘要：我们提出了一个语言不可知的端到端语音翻译模型（LAST），以及一个数据扩充策略来提高代码转换（CS）的性能。随着全球化的发展，在流利的讲话中，多种语言越来越多地可互换地使用。这样的CS使传统的语音识别和翻译复杂化，因为我们必须首先识别说的是哪种语言，然后应用语言相关的识别器和随后的翻译组件来生成期望的目标语言输出。这样的流水线引入了等待时间和错误。在本文中，我们通过将语音识别和翻译视为一个统一的端到端语音翻译问题来消除对这一点的需要。通过用两种输入语言训练LAST，我们将语音解码成一种目标语言，而不管输入语言。LAST在单语使用中提供了相当的识别和语音翻译准确性，同时在观察到CS时大大减少了延迟和错误率。摘要：We propose a) a Language Agnostic end-to-end Speech Translation model (LAST), and b) a data augmentation strategy to increase code-switching (CS) performance. With increasing globalization, multiple languages are increasingly used interchangeably during fluent speech. Such CS complicates traditional speech recognition and translation, as we must recognize which language was spoken first and then apply a language-dependent recognizer and subsequent translation component to generate the desired target language output. Such a pipeline introduces latency and errors. In this paper, we eliminate the need for that, by treating speech recognition and translation as one unified end-to-end speech translation problem. By training LAST with both input languages, we decode speech into one target language, regardless of the input language. LAST delivers comparable recognition and speech translation accuracy in monolingual usage, while reducing latency and error rate considerably when CS is observed.
【26】 Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings标题：韵律手势：基于分层神经嵌入的韵律感知协同语音手势合成链接：https://arxiv.org/abs/2210.01448
作者：Tenglong Ao,Qingzhe Gao,Yuke Lou,Baoquan Chen,Libin Liu备注：SIGGRAPH Asia 2022 (Journal Track); Project Page: this https URL摘要：真实感共语手势的自动合成是人工具身智能体创建中一个日益重要但又具有挑战性的任务。现有的手势生成系统主要关注于端到端的手势生成，由于语音和手势之间复杂而微妙的和谐关系，导致难以挖掘出清晰的韵律和语义。提出了一种新颖的共语手势合成方法，该方法在韵律和语义上都取得了令人信服的结果。对于节奏，我们的系统包含了一个健壮的基于节奏的分割流水线，以明确地确保发声和手势之间的时间连贯性。对于手势语义，我们基于语言学理论设计了一种机制来有效地解开语音和运动的低层和高层神经嵌入。高级嵌入对应于语义，而低级嵌入涉及细微的变化。最后，我们建立了语音和动作的层次嵌入之间的对应关系，从而实现了节奏和语义感知的手势合成。与现有的客观指标、新提出的节奏指标和人工反馈的评估表明，我们的方法明显优于最先进的系统。摘要：Automatic synthesis of realistic co-speech gestures is an increasingly important yet challenging task in artificial embodied agent creation. Previous systems mainly focus on generating gestures in an end-to-end manner, which leads to difficulties in mining the clear rhythm and semantics due to the complex yet subtle harmony between speech and gestures. We present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory. The high-level embedding corresponds to semantics, while the low-level embedding relates to subtle variations. Lastly, we build correspondence between the hierarchical embeddings of the speech and the motion, resulting in rhythm- and semantics-aware gesture synthesis. Evaluations with existing objective metrics, a newly proposed rhythmic metric, and human feedback show that our method outperforms state-of-the-art systems by a clear margin.
【27】 Pay Self-Attention to Audio-Visual Navigation标题：关注视听导航链接：https://arxiv.org/abs/2210.01353
作者：Yinfeng Yu,Lele Cao,Fuchun Sun,Xiaohong Liu,Liejun Wang摘要：视听具身导航是一个研究热点，其目的是利用以自我为中心的视觉（来自安装在机器人上的传感器）和音频（从目标发出）输入训练机器人到达音频目标。视听信息融合策略对于导航性能自然是重要的，但是现有的方法仍然简单地连接视觉和音频特征，潜在地忽略了上下文的直接影响。此外，现有的方法需要逐段训练或额外的辅助（例如拓扑图和声音语义）。到目前为止，处理具有移动目标的更具挑战性的设置的工作仍然很少。因此，我们提出了一个端到端的框架FSAAVN（feature self-attention audio-visual navigation）来学习如何使用上下文感知的视听融合策略来追踪运动的音频目标，并将其实现为自注意模块。实验结果验证了FSAAVN算法在定性和定量两个方面的优越性，并对视觉模态、视/音频编码器骨干和融合模式的选择提供了独特的见解。摘要：Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.
【28】 And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?标题：如果两个音乐版本不共享旋律、和声、节奏或歌词怎么办？链接：https://arxiv.org/abs/2210.01256
作者：Mathilde Abrassart,Guillaume Doras摘要：在过去的几年里，版本识别（VI）已经取得了实质性的进展。一方面，度量学习范式的引入有利于可扩展但精确的VI系统的出现。另一方面，使用专注于音乐作品的特定方面的特征，如旋律、和声或歌词，产生了可解释的和有希望的表现。在这项工作中，我们建立在这些最新进展的基础上，并提出了一个基于度量学习的系统，系统地利用四个维度来传达版本之间的音乐相似性：旋律线、和声结构、节奏模式和歌词。我们描述了我们刻意简化的模型结构，并且我们特别地示出了歌词的近似表示是区分版本和非版本的有效代理。然后，我们描述了这些特征如何相互补充，并在两个公开可用的数据集上产生新的最先进的性能。我们最后建议，使用旋律、和声、节奏和歌词特征的组合的VI系统理论上可以达到在这些数据集上可获得的最佳性能。摘要：Version identification (VI) has seen substantial progress over the past few years. On the one hand, the introduction of the metric learning paradigm has favored the emergence of scalable yet accurate VI systems. On the other hand, using features focusing on specific aspects of musical pieces, such as melody, harmony, or lyrics, yielded interpretable and promising performances. In this work, we build upon these recent advances and propose a metric learning-based system systematically leveraging four dimensions commonly admitted to convey musical similarity between versions: melodic line, harmonic structure, rhythmic patterns, and lyrics. We describe our deliberately simple model architecture, and we show in particular that an approximated representation of the lyrics is an efficient proxy to discriminate between versions and non-versions. We then describe how these features complement each other and yield new state-of-the-art performances on two publicly available datasets. We finally suggest that a VI system using a combination of melodic, harmonic, rhythmic and lyrics features could theoretically reach the optimal performances obtainable on these datasets.
【1】 Binaural Speech Enhancement Using STOI-Optimal Masks标题：基于STOI最优掩码的双耳语音增强链接：https://arxiv.org/abs/2209.15472
作者：Vikas Tokala,Mike Brookes,Patrick A. Naylor机构：Department of Electrical and Electronic Engineering, Imperial College London备注：Accepted at IWAENC 2022摘要：STOI最优掩蔽先前已经被提出和开发用于单通道语音增强。在这篇论文中，我们考虑双耳语音增强任务的扩展，其中已知空间信息对语音理解是重要的，因此应该通过增强处理来保留。分别为每个双耳通道估计掩蔽，并通过选择两个掩蔽中的最大值来计算"更好耳朵收听“掩蔽。所估计的掩码用于向最佳修改对数谱幅度（OM-LSA）增强器提供关于每个时间-频率仓中的语音存在的概率信息。实验结果表明，该方法不仅可以提高双耳信号的信噪比，而且可以保持双耳信号的清晰度和线索。摘要：STOI-optimal masking has been previously proposed and developed for single-channel speech enhancement. In this paper, we consider the extension to the task of binaural speech enhancement in which spatial information is known to be important to speech understanding and therefore should be preserved by the enhancement processing. Masks are estimated for each of the binaural channels individually and a `better-ear listening' mask is computed by choosing the maximum of the two masks. The estimated mask is used to supply probability information about the speech presence in each time-frequency bin to an Optimally-modified Log Spectral Amplitude (OM-LSA) enhancer. We show that using the proposed method for binaural signals with a directional noise not only improves the SNR of the noisy signal but also preserves the binaural cues and intelligibility.
【2】 End-to-End Label Uncertainty Modeling in Speech Emotion Recognition using Bayesian Neural Networks and Label Distribution Learning标题：基于贝叶斯神经网络和标签分布学习的语音情感识别端到端标签不确定性建模链接：https://arxiv.org/abs/2209.15449
作者：Navin Raj Prabhu,Nale Lehmann-Willenbrock,Timo Gerkman机构：Universit¨at Hamburg备注：arXiv admin note: text overlap with arXiv:2207.12135摘要：为了训练机器学习算法来根据唤醒和效价预测情绪表达，需要注释数据集。然而，由于不同的人对他人情绪表达的感知不同，他们的注释本身就具有主观性。为此，通常从多个注释器收集注释，并求平均值以获得地面实况标签。然而，当专门在这个平均的地面实况上训练时，训练的网络对于情感表达中的固有主观性是不可知的。因此，在这项工作中，我们提出了一个端到端的贝叶斯神经网络，能够在标签分布上进行训练，以捕获基于主观性的标签不确定性。我们使用Student 's t分布（也考虑了注释的数量）对标签分布建模，而不是高斯分布。我们推导出相应的Kullback-Leibler散度损失，并使用它来训练标签分布的估计量，从中可以推断出平均值和不确定性。我们使用两个野外数据集验证了所提出的方法。实验结果表明，该方法在语音情感识别中取得了良好的不确定性建模效果，并且在跨语料库评价中取得了一致的结果.此外，分析显示，t分布相对于高斯分布的优势随着注释器间相关性的增加和注释器数量的减少而增长。摘要：To train machine learning algorithms to predict emotional expressions in terms of arousal and valence, annotated datasets are needed. However, as different people perceive others' emotional expressions differently, their annotations are per se subjective. For this, annotations are typically collected from multiple annotators and averaged to obtain ground-truth labels. However, when exclusively trained on this averaged ground-truth, the trained network is agnostic to the inherent subjectivity in emotional expressions. In this work, we therefore propose an end-to-end Bayesian neural network capable of being trained on a distribution of labels to also capture the subjectivity-based label uncertainty. Instead of a Gaussian, we model the label distribution using Student's t-distribution, which also accounts for the number of annotations. We derive the corresponding Kullback-Leibler divergence loss and use it to train an estimator for the distribution of labels, from which the mean and uncertainty can be inferred. We validate the proposed method using two in-the-wild datasets. We show that the proposed t-distribution based approach achieves state-of-the-art uncertainty modeling results in speech emotion recognition, and also consistent results in cross-corpora evaluations. Furthermore, analyses reveal that the advantage of a t-distribution over a Gaussian grows with increasing inter-annotator correlation and a decreasing number of annotators.
【3】 Music Source Separation with Band-split RNN标题：基于频带分割RNN的音乐源分离链接：https://arxiv.org/abs/2209.15174
* 与cs.SD语音【9】为同一篇
作者：Yi Luo,Jianwei Yu摘要：近年来，由于新型神经网络结构和训练流水线的发展，音乐源分离（MSS）模型的性能得到了很大的提高。然而，目前的MSS模型设计主要是基于其他音频处理任务或其他研究领域，而音乐信号的内在特征和模式并没有被完全发现。本文提出了一种频域模型--频带分割RNN（BSRNN），它将混合信号的频谱图明确地分割成子带，并进行交织的频带级和序列级建模。子带的带宽的选择可以通过关于目标源的特性的先验知识或专家知识来确定，以便优化特定类型的目标乐器上的性能。为了更好地利用未标记的数据，我们还描述了一种半监督模型微调流水线，其可以进一步提高模型的性能。实验结果表明，仅在MUSDB 18-HQ数据集上训练的BSRNN显著优于2021年MDX挑战赛中排名靠前的几个模型，并且半监督微调阶段进一步提高了在所有四个乐器音轨上的性能。摘要：The performance of music source separation (MSS) models has been greatly improved in recent years thanks to the development of novel neural network architectures and training pipelines. However, recent model designs for MSS were mainly motivated by other audio processing tasks or other research fields, while the intrinsic characteristics and patterns of the music signals were not fully discovered. In this paper, we propose band-split RNN (BSRNN), a frequency-domain model that explictly splits the spectrogram of the mixture into subbands and perform interleaved band-level and sequence-level modeling. The choices of the bandwidths of the subbands can be determined by a priori knowledge or expert knowledge on the characteristics of the target source in order to optimize the performance on a certain type of target musical instrument. To better make use of unlabeled data, we also describe a semi-supervised model finetuning pipeline that can further improve the performance of the model. Experiment results show that BSRNN trained only on MUSDB18-HQ dataset significantly outperforms several top-ranking models in Music Demixing (MDX) Challenge 2021, and the semi-supervised finetuning stage further improves the performance on all four instrument tracks.
【4】 Detection of Prosodic Boundaries in Speech Using Wav2Vec 2.0标题：基于Wav2Vec2.0的语音韵律边界检测链接：https://arxiv.org/abs/2209.15032
作者：Marie Kunešová,Markéta Řezáčková机构：Řezáčková[,−,−,−,], New Technologies for the Information Society and Department of Cybernetics, University of West Bohemia, Pilsen, Czech Republic备注：None摘要：语音中的韵律边界与语音合成和音频标注都有很大的关系。本文将wav2vec 2.0框架应用于语音信号中这些边界的检测任务，仅使用声学信息。我们在一组由语音专家标注的捷克广播新闻录音上测试了该方法，并将其与现有的基于文本的预测器进行了比较，后者使用了相同数据的转录本。尽管使用了相对少量的标记数据，但wav2vec2模型在句内韵律边界上实现了94%的准确率和83%的F1测量（或在所有韵律边界上实现了95%和89%），优于基于文本的方法。然而，通过组合两个不同模型的输出，我们可以进一步改进结果。摘要：Prosodic boundaries in speech are of great relevance to both speech synthesis and audio annotation. In this paper, we apply the wav2vec 2.0 framework to the task of detecting these boundaries in speech signal, using only acoustic information. We test the approach on a set of recordings of Czech broadcast news, labeled by phonetic experts, and compare it to an existing text-based predictor, which uses the transcripts of the same data. Despite using a relatively small amount of labeled data, the wav2vec2 model achieves an accuracy of 94% and F1 measure of 83% on within-sentence prosodic boundaries (or 95% and 89% on all prosodic boundaries), outperforming the text-based approach. However, by combining the outputs of the two different models we can improve the results even further.
【5】 Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio标题：匹配取胜：分析语音和音频中有效自我监督学习的序列长度链接：https://arxiv.org/abs/2209.15575
* 与cs.SD语音【2】为同一篇
作者：Yan Gao,Javier Fernandez-Marques,Titouan Parcollet,Pedro P. B. de Gusmao,Nicholas D. Lane机构：University of Cambridge,Samsung AI,Avignon University摘要：自监督学习（SSL）已被证明在语音和音频相关应用中至关重要。该范例在未标记的数据上训练通用模型，这些数据稍后可用于解决特定的下游任务。这种类型的模型训练起来成本很高，因为它需要操纵只能由强大的集中式服务器处理的长输入序列。令人惊讶的是，尽管有许多尝试通过模型压缩来提高训练效率，但是还没有研究截断输入序列长度以减少计算的效果。在本文中，我们提供了SSL预训练的第一个经验研究，用于不同的指定序列长度，并将其与各种下游任务联系起来。我们发现，短序列训练可以显著降低资源成本，同时保持所有任务的满意性能。这一简单的一行更改将促进SSL培训从数据中心向用户端边缘设备的迁移，以实现更真实和个性化的应用。摘要：Self-supervised learning (SSL) has proven vital in speech and audio-related applications. The paradigm trains a general model on unlabeled data that can later be used to solve specific downstream tasks. This type of model is costly to train as it requires manipulating long input sequences that can only be handled by powerful centralised servers. Surprisingly, despite many attempts to increase training efficiency through model compression, the effects of truncating input sequence lengths to reduce computation have not been studied. In this paper, we provide the first empirical study of SSL pre-training for different specified sequence lengths and link this to various downstream tasks. We find that training on short sequences can dramatically reduce resource costs while retaining a satisfactory performance for all tasks. This simple one-line change would promote the migration of SSL training from data centres to user-end edge devices for more realistic and personalised applications.
【6】 On The Robustness of Self-Supervised Representations for Spoken Language Modeling标题：口语建模中自监督表征的稳健性研究链接：https://arxiv.org/abs/2209.15483
作者：Itai Gat,Felix Kreuk,Ann Lee,Jade Copet,Gabriel Synnaeve,Emmanuel Dupoux,Yossi Adi机构：FAIR Team, Meta AI Research, ENS, INRIA, INSERM, UPEC, PSL Research University, The Hebrew University of Jerusalem摘要：自监督表征已经被广泛地研究用于区分和生成任务。然而，尚未对它们的耐用性进行广泛研究。本文主要研究口语生成语言模型的自监督表示。首先，我们经验性地证明了当前现有技术的语音表示模型如何缺乏对不改变口语信息的基本信号变化的鲁棒性。为了克服这一问题，我们提出了一种用于生成式口语建模的鲁棒自监督语音表示学习方法。提出的方法是基于对语音信号应用一组信号变换，并使用迭代伪标记方案优化模型。当考虑编码度量时，我们的方法相对于评估的基线有显著改进。我们还在语音到语音的翻译任务上评估了我们的方法。我们考虑了西班牙语-英语和法语-英语转换，并实证证明了遵循所建议的方法的好处。摘要：Self-supervised representations have been extensively studied for discriminative and generative tasks. However, their robustness capabilities have not been extensively investigated. This work focuses on self-supervised representations for spoken generative language models. First, we empirically demonstrate how current state-of-the-art speech representation models lack robustness to basic signal variations that do not alter the spoken information. To overcome this, we propose an effective and efficient method to learn robust self-supervised speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. Our method significantly improves over the evaluated baselines when considering encoding metrics. We additionally evaluate our method on the speech-to-speech translation task. We consider Spanish-English and French-English conversions and empirically demonstrate the benefits of following the proposed approach.
【7】 AudioGen: Textually Guided Audio Generation标题：AudioGen：文本引导的音频生成链接：https://arxiv.org/abs/2209.15352
* 与cs.SD语音【3】为同一篇
作者：Felix Kreuk,Gabriel Synnaeve,Adam Polyak,Uriel Singer,Alexandre Défossez,Jade Copet,Devi Parikh,Yaniv Taigman,Yossi Adi机构：FAIR Team, Meta AI, The Hebrew University of Jerusalem摘要：我们解决了生成以描述性文本字幕为条件的音频样本的问题。在本研究中，我们提出了一个自动回归的产生式模型AaudioGen，它可以产生以文字输入为条件的声音样本。AudioGen在已学习的离散音频表示上运行。文本到音频生成的任务提出了多个挑战。由于音频通过介质的传播方式，区分"对象“可能是一项困难的任务（例如，分开多个人同时说话）。这由于真实世界的记录条件（例如，背景噪声、混响等）。缺少文字注释会造成另一个限制，限制缩放模型的能力。最后，对高保真音频建模需要以高采样率对音频进行编码，从而导致极长的序列。为了缓解上述挑战，我们提出了一种增强技术，该技术混合不同的音频样本，驱动模型内部学习以分离多个源。我们整理了10个包含不同类型音频和文本注释的数据集，以处理文本-音频数据点的稀缺性。为了加快推理速度，我们探索了多流建模的使用，允许使用较短的序列，同时保持相似的比特率和感知质量。我们应用无分类器指导来提高对文本的遵守。与评价的基线相比，AudioGen在客观和主观指标方面均优于基线。最后，我们探讨了所提出的方法有条件地和无条件地生成音频连续的能力。示例：https://tinyurl.com/audiogen-text2audio摘要：We tackle the problem of generating audio samples conditioned on descriptive text captions. In this work, we propose AaudioGen, an auto-regressive generative model that generates audio samples conditioned on text inputs. AudioGen operates on a learnt discrete audio representation. The task of text-to-audio generation poses multiple challenges. Due to the way audio travels through a medium, differentiating ``objects'' can be a difficult task (e.g., separating multiple people simultaneously speaking). This is further complicated by real-world recording conditions (e.g., background noise, reverberation, etc.). Scarce text annotations impose another constraint, limiting the ability to scale models. Finally, modeling high-fidelity audio requires encoding audio at high sampling rate, leading to extremely long sequences. To alleviate the aforementioned challenges we propose an augmentation technique that mixes different audio samples, driving the model to internally learn to separate multiple sources. We curated 10 datasets containing different types of audio and text annotations to handle the scarcity of text-audio data points. For faster inference, we explore the use of multi-stream modeling, allowing the use of shorter sequences while maintaining a similar bitrate and perceptual quality. We apply classifier-free guidance to improve adherence to text. Comparing to the evaluated baselines, AudioGen outperforms over both objective and subjective metrics. Finally, we explore the ability of the proposed method to generate audio continuation conditionally and unconditionally. Samples: https://tinyurl.com/audiogen-text2audio
【8】 ChordMics: Acoustic Signal Purification with Distributed Microphones标题：ChordMics：分布式麦克风的声学信号净化链接：https://arxiv.org/abs/2209.15334
* 与cs.SD语音【4】为同一篇
作者：Weiguo Wang,Jinming Li,Meng Jin,Yuan He机构：School of Software and BNRist, Tsinghua University摘要：声信号作为许多系统的基本输入。然而，纯声学信号很难提取，尤其是在噪声环境中。现有的波束形成系统能够提取从某些方向发射的信号。然而，由于麦克风是集中部署的，因此这些系统具有有限的覆盖范围和低空间分辨率。我们克服了上述局限性，提出了一种分布式波束形成系统ChordMics。通过利用分布式麦克风的空间多样性，ChordMics能够从任意点提取声学信号。为了实现这样的系统，我们进一步解决分布式波束成形中的基本挑战：对准由分布式和非同步的麦克风捕获的信号。我们实现了ChordMics，并在LOS和NLOS两种场景下评估了其性能。仿真结果表明，ChordMics比集中式麦克风阵列具有更高的信干噪比。平均性能增益高达15dB。摘要：Acoustic signal acts as an essential input to many systems. However, the pure acoustic signal is very difficult to extract, especially in noisy environments. Existing beamforming systems are able to extract the signal transmitted from certain directions. However, since microphones are centrally deployed, these systems have limited coverage and low spatial resolution. We overcome the above limitations and present ChordMics, a distributed beamforming system. By leveraging the spatial diversity of the distributed microphones, ChordMics is able to extract the acoustic signal from arbitrary points. To realize such a system, we further address the fundamental challenge in distributed beamforming: aligning the signals captured by distributed and unsynchronized microphones. We implement ChordMics and evaluate its performance under both LOS and NLOS scenarios. The evaluation results tell that ChordMics can deliver higher SINR than the centralized microphone array. The average performance gain is up to 15dB.
【9】 SpeechLM: Enhanced Speech Pre-Training with Unpaired Textual Data标题：SpeechLM：使用不成对的文本数据进行增强的语音预训练链接：https://arxiv.org/abs/2209.15329
作者：Ziqiang Zhang,Sanyuan Chen,Long Zhou,Yu Wu,Shuo Ren,Shujie Liu,Zhuoyuan Yao,Xun Gong,Lirong Dai,Jinyu Li,Furu Wei机构：University of Science and Technology of China, Harbin Institute of Technology, Microsoft备注：14 pages摘要：由于语音和文本是具有不同特征的不同模态，如何利用文本数据来增强语音预训练一直是一个未解决的问题。本文提出了一种跨模态的语音和语言模型（SpeechLM），通过预先定义的统一离散表示来显式地对齐语音和文本预训练。具体地说，我们引入了两种可选的离散标记器来桥接语音和文本模态，包括音素单元标记器和隐藏单元标记器，它们可以使用少量成对的语音-文本数据来训练。基于训练好的标记器，我们将未标记的语音和文本数据转换为音素单元或隐藏单元的标记。预训练的目标是用统一的Transformer网络将语音和文本统一到同一个离散语义空间中。在LibriSpeech ASR公共基准测试中，我们的SpeechLM仅利用10 K文本句子，与最佳基础模型性能（从6.8到5.7）相比，相对WER降低了16\%。此外，在CoVoST-2语音翻译任务中，参数较少的SpeechLM模型甚至优于以前的SOTA模型。我们还在通用表征评估框架SUPERB下评估了SpeechLM在各种口语处理任务上的表现，证明了其在内容相关任务上的显著改进。我们的代码和模型可在www.example.com上获得https://aka.ms/SpeechLM。摘要：How to boost speech pre-training with textual data is an unsolved problem due to the fact that speech and text are very different modalities with distinct characteristics. In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities, including phoneme-unit and hidden-unit tokenizers, which can be trained using a small amount of paired speech-text data. Based on the trained tokenizers, we convert the unlabeled speech and text data into tokens of phoneme units or hidden units. The pre-training objective is designed to unify the speech and the text into the same discrete semantic space with a unified Transformer network. Leveraging only 10K text sentences, our SpeechLM gets a 16\% relative WER reduction over the best base model performance (from 6.8 to 5.7) on the public LibriSpeech ASR benchmark. Moreover, SpeechLM with fewer parameters even outperforms previous SOTA models on CoVoST-2 speech translation tasks. We also evaluate our SpeechLM on various spoken language processing tasks under the universal representation evaluation framework SUPERB, demonstrating significant improvements on content-related tasks. Our code and models are available at https://aka.ms/SpeechLM.
【10】 Symphony: Localizing Multiple Acoustic Sources with a Single Microphone Array标题：交响乐：使用单个麦克风阵列定位多个声源链接：https://arxiv.org/abs/2209.15325
* 与cs.SD语音【5】为同一篇
作者：Weiguo Wang,Jinming Li,Yuan He,Yunhao Liu机构：School of Software and BNRist, Tsinghua University, Department of Computer Science and Engineering, Michigan State University摘要：声音识别是智能设备的一项重要且受欢迎的功能。声音的位置是与声源相关联的基本信息。除了声音识别之外，声源是否能够被定位在很大程度上影响着智能设备交互功能的能力和质量。在这项工作中，我们研究了使用智能设备（例如，亚马逊Alexa等智能扬声器）。现有的方法或者只能定位单个源，或者需要部署麦克风阵列的分布式网络来起作用。我们提出的Symphony方案是第一个使用单个麦克风阵列解决上述问题的方法。Symphony的设计理念是，阵列上麦克风的几何布局决定了来自同一信号源、沿同一到达路径的信号之间的独特关系，而信号源的位置决定了信号沿不同到达路径的DoA（到达方向）。Symphony因此包括用于区分来自沿不同路径的不同源的信号的基于几何的滤波模块和用于识别来自相同源的信号的基于相干性的模块。我们使用不同类型的商用现成麦克风阵列来实现Symphony，并评估其在不同设置下的性能。结果表明，Symphony的中值定位误差为0.694m，比最先进的方法的中值定位误差小68%。摘要：Sound recognition is an important and popular function of smart devices. The location of sound is basic information associated with the acoustic source. Apart from sound recognition, whether the acoustic sources can be localized largely affects the capability and quality of the smart device's interactive functions. In this work, we study the problem of concurrently localizing multiple acoustic sources with a smart device (e.g., a smart speaker like Amazon Alexa). The existing approaches either can only localize a single source, or require deploying a distributed network of microphone arrays to function. Our proposal called Symphony is the first approach to tackle the above problem with a single microphone array. The insight behind Symphony is that the geometric layout of microphones on the array determines the unique relationship among signals from the same source along the same arriving path, while the source's location determines the DoAs (direction-of-arrival) of signals along different arriving paths. Symphony therefore includes a geometry-based filtering module to distinguish signals from different sources along different paths and a coherence-based module to identify signals from the same source. We implement Symphony with different types of commercial off-the-shelf microphone arrays and evaluate its performance under different settings. The results show that Symphony has a median localization error of 0.694m, which is 68% less than that of the state-of-the-art approach.
【11】 Wake Word Detection Based on Res2Net标题：基于Res2Net的唤醒词检测链接：https://arxiv.org/abs/2209.15296
* 与cs.SD语音【6】为同一篇
作者：Qiuchen Yu,Ruohua Zhou摘要：本文提出了一种基于Res2Net的唤醒词检测系统。Res2Net是ResNet的一个变种，它首先被应用于目标检测。Res2Net通过增加可能的感受野来实现多个特征尺度。这种多尺度机制显著提高了对不同时长唤醒词的检测能力。与基于ResNet的模型相比，Res2Net模型显著减小了模型规模，更适合于唤醒词的检测。所提出的系统可以在没有任何额外辅助的情况下从音频流中确定唤醒词的位置。在包含两个唤醒词的Mobvoi数据集上验证了该方法的有效性.在每小时0.5的误报警率下，该系统将两个唤醒字的误拒绝比现有工作减少了12%以上。摘要：This letter proposes a new wake word detection system based on Res2Net. As a variant of ResNet, Res2Net was first applied to objection detection. Res2Net realizes multiple feature scales by increasing possible receptive fields. This multiple scaling mechanism significantly improves the detection ability of wake words with different durations. Compared with the ResNet-based model, Res2Net also significantly reduces the model size and is more suitable for detecting wake words. The proposed system can determine the positions of wake words from the audio stream without any additional assistance. The proposed method is verified on the Mobvoi dataset containing two wake words. At a false alarm rate of 0.5 per hour, the system reduced the false rejection of the two wake words by more than 12% over prior works.
【12】 An efficient encoder-decoder architecture with top-down attention for speech separation标题：一种自顶向下关注语音分离的高效编解码器结构链接：https://arxiv.org/abs/2209.15200
* 与cs.SD语音【7】为同一篇
作者：Kai Li,Runxuan Yang,Xiaolin Hu机构：Department of Computer Science and Technology, Tsinghua Laboratory of Brain and Intelligence (THBI), IDGMcGovern Institute of Brain Research Tsinghua University, Beijing, China备注：13 pages, 4 figures摘要：深度神经网络在语音分离任务中显示出良好的前景。然而，在保持低模型复杂度的同时获得好的结果在现实世界的应用中仍然具有挑战性。本文提出了一种模拟大脑自上而下注意力的生物启发式高效编解码器架构TDANet，在不牺牲性能的前提下降低了模型复杂度。TDANet中的自顶向下注意力由全局注意力（GA）模块和级联的局部注意力（LA）层提取。遗传算法模块以多尺度声学特征作为输入，提取全局注意力信号，然后通过直接自顶向下的连接方式对不同尺度的特征进行调制。LA层使用相邻层的特征作为输入来提取局部注意信号，该局部注意信号用于以自顶向下的方式调制横向输入。在三个基准数据集上，TDANet始终实现了与先前最先进的（SOTA）方法具有竞争力的分离性能和更高的效率。具体地说，TDANet的乘累加运算（MACs）仅为Sepformer的5%，CPU推理时间仅为Sepformer的10%.此外，在3个数据集上，TDANet的大规模版本获得了SOTA结果，其MAC仍仅为Sepformer的10%，CPU推理时间仅为Sepformer的24%.我们的研究表明，自上而下的注意可能是一种更有效的语音分离策略。摘要：Deep neural networks have shown excellent prospects in speech separation tasks. However, obtaining good results while keeping a low model complexity remains challenging in real-world applications. In this paper, we provide a bio-inspired efficient encoder-decoder architecture by mimicking the brain's top-down attention, called TDANet, with decreased model complexity without sacrificing performance. The top-down attention in TDANet is extracted by the global attention (GA) module and the cascaded local attention (LA) layers. The GA module takes multi-scale acoustic features as input to extract global attention signal, which then modulates features of different scales by direct top-down connections. The LA layers use features of adjacent layers as input to extract the local attention signal, which is used to modulate the lateral input in a top-down manner. On three benchmark datasets, TDANet consistently achieved competitive separation performance to previous state-of-the-art (SOTA) methods with higher efficiency. Specifically, TDANet's multiply-accumulate operations (MACs) are only 5\% of Sepformer, one of the previous SOTA models, and CPU inference time is only 10\% of Sepformer. In addition, a large-size version of TDANet obtained SOTA results on three datasets, with MACs still only 10\% of Sepformer and the CPU inference time only 24\% of Sepformer. Our study suggests that top-down attention can be a more efficient strategy for speech separation.
【13】 An empirical study of weakly supervised audio tagging embeddings for general audio representations标题：用于一般音频表示的弱监督音频标记嵌入的实验研究链接：https://arxiv.org/abs/2209.15167
* 与cs.SD语音【8】为同一篇
作者：Heinrich Dinkel,Zhiyong Yan,Yongqing Wang,Junbo Zhang,Yujun Wang机构：Xiaomi Corporation, Beijing, China备注：Odyssey 2022摘要：我们研究了预训练的弱监督音频标记（AT）模型作为一般音频表示的特征提取器的可用性。我们主要分析了将这些嵌入转移到语音和声音领域的其他任务中的可行性。具体而言，我们将弱监督预训练模型（MobileNetV 2和EfficientNet-B 0）与现代自监督学习方法（BYOL-A）作为特征提取器进行基准测试。14个下游任务用于从乐器分类到语言分类的评估。我们的研究结果表明，AT预训练模型是音乐、事件和情绪识别任务的一个很好的迁移学习选择。此外，微调AT模型还可以有益于语音相关任务，例如关键字定位和意图分类。摘要：We study the usability of pre-trained weakly supervised audio tagging (AT) models as feature extractors for general audio representations. We mainly analyze the feasibility of transferring those embeddings to other tasks within the speech and sound domains. Specifically, we benchmark weakly supervised pre-trained models (MobileNetV2 and EfficientNet-B0) against modern self-supervised learning methods (BYOL-A) as feature extractors. Fourteen downstream tasks are used for evaluation ranging from music instrument classification to language classification. Our results indicate that AT pre-trained models are an excellent transfer learning choice for music, event, and emotion recognition tasks. Further, finetuning AT models can also benefit speech-related tasks such as keyword spotting and intent classification.
【14】 WaveFit: An Iterative and Non-autoregressive Neural Vocoder based on Fixed-Point Iteration标题：WaveFit：一种基于定点迭代的迭代非自回归神经声码器链接：https://arxiv.org/abs/2210.01029
* 与cs.SD语音【15】为同一篇
作者：Yuma Koizumi,Kohei Yatabe,Heiga Zen,Michiel Bacchiani机构：Google Research, Japan, Tokyo University of Agriculture and Technology, Japan备注：Accepted to IEEE SLT 2022摘要：去噪扩散概率模型（DDPMs）和生成式对抗网络（GANs）是神经声码器常用的生成式模型。DDPM和GANs可以分别由迭代去噪框架和对抗训练来表征。本文提出了一种快速、高质量的神经声码器\textit{WaveFit}，它将GANs的本质融入到一个基于定点迭代的类DDPM迭代框架中。WaveFit迭代地对输入信号去噪，并且训练深度神经网络（DNN），用于最小化在所有迭代处从中间输出计算的对抗损失。主观（并排）听音测试显示，人类自然语音和通过WaveFit五次迭代合成的语音之间在自然度方面没有统计学显著差异。此外，WaveFit的推理速度比WaveRNN快240倍。音频演示可从以下网址获得：摘要：Denoising diffusion probabilistic models (DDPMs) and generative adversarial networks (GANs) are popular generative models for neural vocoders. The DDPMs and GANs can be characterized by the iterative denoising framework and adversarial training, respectively. This study proposes a fast and high-quality neural vocoder called \textit{WaveFit}, which integrates the essence of GANs into a DDPM-like iterative framework based on fixed-point iteration. WaveFit iteratively denoises an input signal, and trains a deep neural network (DNN) for minimizing an adversarial loss calculated from intermediate outputs at all iterations. Subjective (side-by-side) listening tests showed no statistically significant differences in naturalness between human natural speech and those synthesized by WaveFit with five iterations. Furthermore, the inference speed of WaveFit was more than 240 times faster than WaveRNN. Audio demos are available at \url{google.github.io/df-conformer/wavefit/}.
【15】 Simple Pooling Front-ends For Efficient Audio Classification标题：简单的池化前端，实现高效的音频分类链接：https://arxiv.org/abs/2210.00943
* 与cs.SD语音【16】为同一篇
作者：Xubo Liu,Haohe Liu,Qiuqiang Kong,Xinhao Mei,Mark D. Plumbley,Wenwu Wang机构：Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK, Speech, Audio, and Music Intelligence (SAMI) Group, ByteDance, China备注：Submitted to ICASSP 2023摘要：近来，对于构建用于设备上场景的高效音频神经网络的兴趣日益增加。而大多数现有的方法被设计成使用诸如模型修剪的方法来减小音频神经网络的大小。在这项工作中，我们表明，不是使用复杂的方法来减少模型大小，而是消除输入音频特征中的时间冗余（例如，Mel-spectrogram）是一种有效的音频分类方法。为此，我们提出了一系列简单池前端（SimPF），它们使用简单的非参数池操作来减少Mel谱图中的冗余信息。我们在四个音频分类任务上进行了广泛的实验来评估SimPF的性能。实验结果表明，SimPF算法可以将现有的音频神经网络的FLOP减少一半以上，而音频分类性能的下降可以忽略不计，甚至有相当大的提高。摘要：Recently, there has been increasing interest in building efficient audio neural networks for on-device scenarios. While most existing approaches are designed to reduce the size of audio neural networks using methods such as model pruning. In this work, we show that instead of reducing model size using complex methods, eliminating the temporal redundancy in the input audio features (e.g., Mel-spectrogram) could be an effective approach for efficient audio classification. To do so, we proposed a family of simple pooling front-ends (SimPFs) which use simple non-parametric pooling operations to reduce the redundant information within the Mel-spectrogram. We perform extensive experiments on four audio classification tasks to evaluate the performance of SimPFs. Experimental results show that SimPFs can achieve a reduction in more than half of the FLOPs for off-the-shelf audio neural networks, with negligible degradation or even decent improvement in audio classification performance.
【16】 Learnable Acoustic Frontends in Bird Activity Detection标题：可学习声学前沿在鸟类活动检测中的应用链接：https://arxiv.org/abs/2210.00889
作者：Mark Anderson,Naomi Harte机构：SIGMEDIA Lab, School of Engineering, Trinity College Dublin, Ireland, ©, IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including备注：Submitted and presented at IWAENC, September 2022, 5 Pages, 1 Figure, 3 Tables摘要：自主记录单元和被动声学监测提供了收集生物声学数据的最小侵入方法。将该数据与物种不可知的鸟类活动检测系统相结合，使得能够监测鸟类种群的活动水平。不幸的是，环境噪声水平和对象距离的可变性导致难以在记录中准确地检测鸟类活动。声学前端的选择直接影响这些问题对系统性能的影响。在本文中，我们使用来自DCASE2018 BAD挑战赛的数据，对传统固定参数声学前端与新一代可学习前端在广泛的鸟类音频检测任务中进行了基准测试。我们观察到，每通道能量归一化是最好的整体表现，实现了89.9%的准确性，并且在一般情况下，可学习的前端显著优于传统方法。我们还确定了学习鸟类音频滤波器组的挑战。摘要：Autonomous recording units and passive acoustic monitoring present minimally intrusive methods of collecting bioacoustics data. Combining this data with species agnostic bird activity detection systems enables the monitoring of activity levels of bird populations. Unfortunately, variability in ambient noise levels and subject distance contribute to difficulties in accurately detecting bird activity in recordings. The choice of acoustic frontend directly affects the impact these issues have on system performance. In this paper, we benchmark traditional fixed-parameter acoustic frontends against the new generation of learnable frontends on a wide-ranging bird audio detection task using data from the DCASE2018 BAD Challenge. We observe that Per-Channel Energy Normalization is the best overall performer, achieving an accuracy of 89.9%, and that in general learnable frontends significantly outperform traditional methods. We also identify challenges in learning filterbanks for bird audio.
【17】 Music-to-Text Synaesthesia: Generating Descriptive Text from Music Recordings标题：音乐到文本通感：从音乐录音生成描述性文本链接：https://arxiv.org/abs/2210.00434
* 与cs.SD语音【17】为同一篇
作者：Zhihuan Kuang,Shi Zong,Jianbing Zhang,Jiajun Chen,Hongfu Liu机构：Nanjing University, Brandeis University摘要：本文提出了一个新的研究课题--音乐与文本通感。与传统的音乐标注问题不同，音乐-文本通感问题的目标是从音乐记录中生成描述性文本，以供进一步理解。虽然这是机器学习领域一个新的有趣的应用，但据我们所知，现有的音乐相关数据集不包含音乐录音的语义描述，无法完成音乐到文本的通感任务。有鉴于此，我们收集了一个新的数据集，其中包含1,955对对齐的古典音乐录音和文本描述。在此基础上，我们建立了一个计算模型，生成能够描述音乐录音内容的句子。针对古典音乐的高度非区分性，在计算模型中设计了一种群拓扑保持损失，将更多的样本作为群参考，保持了不同样本之间的相对拓扑关系.实验结果定性和定量地证明了该模型在5种启发式或预先训练的竞争方法及其变体上的有效性.摘要：In this paper, we consider a novel research problem, music-to-text synaesthesia. Different from the classical music tagging problem that classifies a music recording into pre-defined categories, the music-to-text synaesthesia aims to generate descriptive texts from music recordings for further understanding. Although this is a new and interesting application to the machine learning community, to our best knowledge, the existing music-related datasets do not contain the semantic descriptions on music recordings and cannot serve the music-to-text synaesthesia task. In light of this, we collect a new dataset that contains 1,955 aligned pairs of classical music recordings and text descriptions. Based on this, we build a computational model to generate sentences that can describe the content of the music recording. To tackle the highly non-discriminative classical music, we design a group topology-preservation loss in our computational model, which considers more samples as a group reference and preserves the relative topology among different samples. Extensive experimental results qualitatively and quantitatively demonstrate the effectiveness of our proposed model over five heuristics or pre-trained competitive methods and their variants on our collected dataset.
【18】 Voice Spoofing Countermeasures: Taxonomy, State-of-the-art, experimental analysis of generalizability, open challenges, and the way forward标题：语音欺骗对策：分类、最新技术、可概括性实验分析、开放挑战和前进方向链接：https://arxiv.org/abs/2210.00417
* 与cs.SD语音【18】为同一篇
作者：Awais Khan,Khalid Mahmood Malik,James Ryan,Mikul Saravanan机构：Department of Computer Science and Engineering, Oakland University, Rochester, MI , USA摘要：恶意行为者可能试图使用不同的语音欺骗攻击来欺骗ASV系统，甚至使用它们来传播错误信息。已经提出了各种对策来检测这些欺骗攻击。由于在过去的6-7年中在自动说话人确认（ASV）系统中的欺骗检测方面所做的大量工作，因此有必要对研究进行分类，并对最新的对策进行定性和定量比较。此外，没有现有的调查报告回顾了语音欺骗评估和说话人验证、欺骗对策的对抗性/反取证攻击和ASV本身的集成解决方案，或使用单个模型检测多个攻击的统一解决方案。此外，还没有对已发表的对策进行对等比较，以便通过跨语料库评估它们来评估它们的普遍性。在本文中，我们对使用手工特征、深度学习、端到端和通用欺骗对策解决方案检测语音合成（SS）、语音转换（VC）和重放攻击的欺骗检测文献进行了回顾。此外，我们还回顾了语音欺骗评估和说话人验证、对抗性和反取证攻击的语音对抗和ASV的集成解决方案。最后，分析了现有电子欺骗对抗措施的局限性和面临的挑战。我们报告了这些对策在几个数据集上的表现，并在语料库中进行了评估。对于实验，我们使用ASVspoof 2019和VSDC数据集以及GMM、SVM、CNN和CNN-GRU分类器。(For测试平台的代码可以在我们的GitHub Repository中找到。摘要：Malicious actors may seek to use different voice-spoofing attacks to fool ASV systems and even use them for spreading misinformation. Various countermeasures have been proposed to detect these spoofing attacks. Due to the extensive work done on spoofing detection in automated speaker verification (ASV) systems in the last 6-7 years, there is a need to classify the research and perform qualitative and quantitative comparisons on state-of-the-art countermeasures. Additionally, no existing survey paper has reviewed integrated solutions to voice spoofing evaluation and speaker verification, adversarial/antiforensics attacks on spoofing countermeasures, and ASV itself, or unified solutions to detect multiple attacks using a single model. Further, no work has been done to provide an apples-to-apples comparison of published countermeasures in order to assess their generalizability by evaluating them across corpora. In this work, we conduct a review of the literature on spoofing detection using hand-crafted features, deep learning, end-to-end, and universal spoofing countermeasure solutions to detect speech synthesis (SS), voice conversion (VC), and replay attacks. Additionally, we also review integrated solutions to voice spoofing evaluation and speaker verification, adversarial and anti-forensics attacks on voice countermeasures, and ASV. The limitations and challenges of the existing spoofing countermeasures are also presented. We report the performance of these countermeasures on several datasets and evaluate them across corpora. For the experiments, we employ the ASVspoof2019 and VSDC datasets along with GMM, SVM, CNN, and CNN-GRU classifiers. (For reproduceability of the results, the code of the test bed can be found in our GitHub Repository.
【19】 Optimized Decoders for Mixed-Order Ambisonics标题：一种适用于混合阶数的优化解码器链接：https://arxiv.org/abs/2210.00378
* 与cs.SD语音【19】为同一篇
作者：Aaron Heller,Eric Benjamin,Fernando Lopez-Lezcano机构：Artificial Intelligence Center, SRI International, Menlo Park, CA, Surround Research, Pacifica, CA, Center for Computer Research in Music and Acoustics (CCRMA), Stanford University, Stanford, CA备注：None摘要：在本文中，我们讨论的动机，设计，和分析的立体声解码器的系统，其中垂直阶比水平阶少，称为混合阶立体声系统。这可能是由于使用了强调水平空间分辨率的麦克风阵列或提供垂直稀疏覆盖的扬声器阵列。首先，我们回顾了Gerzon定义的双声道再现标准，并总结了最近关于各种标准的相对感知重要性的结果。然后，我们表明，使用全阶解码器与混合顺序节目材料导致比使用适当设计的混合顺序解码器更差的性能。然后，我们介绍了解码器优化器的一种新实现，该优化器利用机器学习技术实现快速和稳健的收敛，讨论了目标函数的构造，并将其应用于设计混合阶信号集和非均匀扬声器布局的双频段解码器的问题。本文总结了非正式听力测试的结果，并讨论了未来的发展方向。摘要：In this paper we discuss the motivation, design, and analysis of ambisonic decoders for systems where the vertical order is less than the horizontal order, known as mixed-order Ambisonic systems. This can be due to the use of microphone arrays that emphasize horizontal spatial resolution or speaker arrays that provide sparser coverage vertically. First, we review Ambisonic reproduction criteria, as defined by Gerzon, and summarize recent results on the relative perceptual importance of the various criteria. Then we show that using full-order decoders with mixed-order program material results in poorer performance than with a properly designed mixed-order decoder. We then introduce a new implementation of a decoder optimizer that draws upon techniques from machine learning for quick and robust convergence, discuss the construction of the objective function, and apply it to the problem of designing two-band decoders for mixed-order signal sets and non-uniform loudspeaker layouts. Results of informal listening tests are summarized and future directions discussed.
【20】 A Comparison of Transformer, Convolutional, and Recurrent Neural Networks on Phoneme Recognition标题：变换神经网络、卷积神经网络和递归神经网络在音素识别中的比较链接：https://arxiv.org/abs/2210.00367
作者：Kyuhong Shim,Wonyong Sung机构：Department of Electrical and Computer Engineering, Seoul National University, Korea摘要：音素识别是语音识别中非常重要的一部分，它要求能够从多个帧中提取语音特征。本文对CNN、RNN、Transformer和Conformer四种模型在音素识别中的应用进行了比较和分析。对于CNN，使用ContextNet模型进行实验。首先，我们比较了不同结构在不同约束条件下的精度，如感受野长度、参数大小和层深度。其次，我们解释了这些模型的性能差异，特别是当可观察的序列长度变化时。我们的分析表明，Transformer和顺应者模型受益于通过输入框架的自我注意的远程可及性。摘要：Phoneme recognition is a very important part of speech recognition that requires the ability to extract phonetic features from multiple frames. In this paper, we compare and analyze CNN, RNN, Transformer, and Conformer models using phoneme recognition. For CNN, the ContextNet model is used for the experiments. First, we compare the accuracy of various architectures under different constraints, such as the receptive field length, parameter size, and layer depth. Second, we interpret the performance difference of these models, especially when the observable sequence length varies. Our analyses show that Transformer and Conformer models benefit from the long-range accessibility of self-attention through input frames.
【21】 Fine-tuning Wav2vec for Vocal-burst Emotion Recognition标题：微调Wav2vec用于发声情感识别链接：https://arxiv.org/abs/2210.00263
* 与cs.SD语音【20】为同一篇
作者：Dang-Khanh Nguyen,Sudarshan Pant,Ngoc-Huynh Ho,Guee-Sang Lee,Soo-Huyng Kim,Hyung-Jeong Yang机构：Department of AI Convergence, Chonnam National University, Gwangju, South Korea摘要：ACII情感爆发式发声（A-VB）竞赛引入了情感计算的一个新主题，即使用人类的非语言声音来理解情感表达。我们熟悉通过言语、声音或面部表情进行情绪识别。然而，诸如笑、哭和手势之类的声音突发没有被利用，即使它们对于行为分析是非常有用的。A-VB竞赛包括四项任务，探索不同空间中的非语言信息。本技术报告描述了SclabCNU团队完成挑战任务的方法和结果。与组织者提供的基线模型相比，我们取得了令人鼓舞的结果。摘要：The ACII Affective Vocal Bursts (A-VB) competition introduces a new topic in affective computing, which is understanding emotional expression using the non-verbal sound of humans. We are familiar with emotion recognition via verbal vocal or facial expression. However, the vocal bursts such as laughs, cries, and signs, are not exploited even though they are very informative for behavior analysis. The A-VB competition comprises four tasks that explore non-verbal information in different spaces. This technical report describes the method and the result of SclabCNU Team for the tasks of the challenge. We achieved promising results compared to the baseline model provided by the organizers.
【22】 Pre-trained Speech Representations as Feature Extractors for Speech Quality Assessment in Online Conferencing Applications标题：用于在线会议语音质量评估的预训练语音表示作为特征提取链接：https://arxiv.org/abs/2210.00259
* 与cs.SD语音【21】为同一篇
作者：Bastiaan Tamm,Helena Balabin,Rik Vandenberghe,Hugo Van hamme机构：Laboratory for Cognitive Neurology, KU Leuven, Belgium, Processing Speech and Images, KU Leuven, Belgium备注：None摘要：在线会议应用中的语音质量通常通过平均意见得分（MOS）度量形式的人类判断来评估。由于这种劳动密集型方法在大多数设置中对于大规模语音质量评估是不可行的，所以焦点已经转移到通过深度神经网络（DNN）的端到端训练的自动MOS预测。我们建议利用来自预训练的基于wav 2 vec的XLS-R模型的语音表示，而不是从头开始训练网络。然而，这种模型的参数数量超过任务特定的DNN几个数量级，这对在较小数据集上产生的微调过程提出了挑战。因此，我们选择在特征提取中使用来自XLS-R的预先训练的语音表示，而不是微调设置，从而显著减少可训练模型参数的数量。我们将我们提出的基于XLS-R的特征提取器与基于Mel频率倒谱系数（MFCC）的特征提取器进行了比较，并对双向长短期记忆（Bi-LSTM）和注意力池前馈（AttPoolFF）网络的各种组合进行了实验，这些组合是在特征提取器的输出上训练的。我们在ConferencingSpeech 2022 MOS预测任务中证明了预训练XLS-R嵌入在降低均方根误差（RMSE）方面的性能提高。摘要：Speech quality in online conferencing applications is typically assessed through human judgements in the form of the mean opinion score (MOS) metric. Since such a labor-intensive approach is not feasible for large-scale speech quality assessments in most settings, the focus has shifted towards automated MOS prediction through end-to-end training of deep neural networks (DNN). Instead of training a network from scratch, we propose to leverage the speech representations from the pre-trained wav2vec-based XLS-R model. However, the number of parameters of such a model exceeds task-specific DNNs by several orders of magnitude, which poses a challenge for resulting fine-tuning procedures on smaller datasets. Therefore, we opt to use pre-trained speech representations from XLS-R in a feature extraction rather than a fine-tuning setting, thereby significantly reducing the number of trainable model parameters. We compare our proposed XLS-R-based feature extractor to a Mel-frequency cepstral coefficient (MFCC)-based one, and experiment with various combinations of bidirectional long short term memory (Bi-LSTM) and attention pooling feedforward (AttPoolFF) networks trained on the output of the feature extractors. We demonstrate the increased performance of pre-trained XLS-R embeddings in terms a reduced root mean squared error (RMSE) on the ConferencingSpeech 2022 MOS prediction task.
【23】 Blind Signal Dereverberation for Machine Speech Recognition标题：机器语音识别中的盲信号去混响链接：https://arxiv.org/abs/2210.00117
* 与cs.SD语音【22】为同一篇
作者：Samik Sadhu,Hynek Hermansky机构：Center for Language and Speech Processing, Johns Hopkins University, USA, Human Language Technology Center of Excellence, Johns Hopkins University, USA摘要：我们提出了一种利用来自混响环境的一定量的训练语音数据和任何可用的非混响语音数据来去除由记录环境的混响引入到语音中的未知卷积噪声的方法。使用在长时间窗上计算的傅里叶变换，其理想地覆盖整个房间脉冲响应，我们将房间引起的卷积转换为对数谱域中的加法。接下来，我们根据在对数频谱域中的混响语音以及干净语音上收集的统计来计算频谱归一化向量。在操作期间，该归一化矢量被用于减轻来自在相同混响条件下记录的复杂语音频谱的混响。这种去混响的复语音频谱被用来计算复FDLP频谱图，以用于自动语音识别。摘要：We present a method to remove unknown convolutive noise introduced to speech by reverberations of recording environments, utilizing some amount of training speech data from the reverberant environment, and any available non-reverberant speech data. Using Fourier transform computed over long temporal windows, which ideally cover the entire room impulse response, we convert room induced convolution to additions in the log spectral domain. Next, we compute a spectral normalization vector from statistics gathered over reverberated as well as over clean speech in the log spectral domain. During operation, this normalization vectors are used to alleviate reverberations from complex speech spectra recorded under the same reverberant conditions . Such dereverberated complex speech spectra are used to compute complex FDLP-spectrograms for use in automatic speech recognition.
【24】 E-Branchformer: Branchformer with Enhanced merging for speech recognition标题：E-Branchformer：用于语音识别的增强合并的Branchformer链接：https://arxiv.org/abs/2210.00077
作者：Kwangyoun Kim,Felix Wu,Yifan Peng,Jing Pan,Prashant Sridhar,Kyu J. Han,Shinji Watanabe机构：ASAPP Inc., Mountain View, CA, USA, Carnegie Mellon University, Pittsburgh, PA, USA备注：Accepted to SLT 2022摘要：Conformer算法将卷积和自注意相结合，能够同时捕获语音的局部和全局信息，具有很好的识别性能，是目前自动语音识别（ASR）领域的研究热点。其他几项研究也探索了卷积和自我注意力的整合，但它们都没有达到顺从者的表现。最近引入的Branchformer通过使用卷积和自关注的专用分支以及合并来自每个分支的局部和全局上下文，实现了与Conformer相当的性能。本文提出了E-Branchformer算法，该算法通过采用有效的合并方法和堆叠额外的逐点模块，对Branchformer算法进行了改进。E-Branchformer在不使用任何外部训练数据的情况下，在LibriSpeech测试-清洁和测试-其他集上设置了新的最先进的单词错误率（WER）1.81%和3.65%。摘要：Conformer, combining convolution and self-attention sequentially to capture both local and global information, has shown remarkable performance and is currently regarded as the state-of-the-art for automatic speech recognition (ASR). Several other studies have explored integrating convolution and self-attention but they have not managed to match Conformer's performance. The recently introduced Branchformer achieves comparable performance to Conformer by using dedicated branches of convolution and self-attention and merging local and global context from each branch. In this paper, we propose E-Branchformer, which enhances Branchformer by applying an effective merging method and stacking additional point-wise modules. E-Branchformer sets new state-of-the-art word error rates (WERs) 1.81% and 3.65% on LibriSpeech test-clean and test-other sets without using any external training data.
【25】 That Sounds Right: Auditory Self-Supervision for Dynamic Robot Manipulation标题：听起来很对：动态机器人操作的听觉自我监督链接：https://arxiv.org/abs/2210.01116
* 与cs.SD语音【10】为同一篇
作者：Abitha Thankaraj,Lerrel Pinto机构：New York University备注：Videos and audio data are best seen on our project website: audio-robot-learning.github.io摘要：从原始的感觉数据中学习产生接触丰富的动态行为一直是机器人领域的一个长期挑战。突出的方法主要集中在使用视觉或触觉传感，其中不幸的是，一个不能捕捉高频交互，而另一个可能太微妙，无法进行大规模数据收集。在本文中，我们提出了一种以数据为中心的动态操作方法，该方法使用了一个经常被忽略的信息源：声音。我们首先使用商用接触式麦克风收集了五个动态任务中25k个交互声音对的数据集。然后，根据这些数据，我们利用自监督学习来加速根据声音进行的行为预测。我们的实验表明，这种自我监督的“预训练”对于实现高性能是至关重要的，与普通监督学习相比，MSE降低了34.5%，与视觉训练相比，MSE降低了54.3%。重要的是，我们发现，当要求生成所需的声音配置文件时，在UR10机器人上在线推出我们的模型可以产生动态行为，在音频相似性度量方面比监督学习平均提高11.5%。摘要：Learning to produce contact-rich, dynamic behaviors from raw sensory data has been a longstanding challenge in robotics. Prominent approaches primarily focus on using visual or tactile sensing, where unfortunately one fails to capture high-frequency interaction, while the other can be too delicate for large-scale data collection. In this work, we propose a data-centric approach to dynamic manipulation that uses an often ignored source of information: sound. We first collect a dataset of 25k interaction-sound pairs across five dynamic tasks using commodity contact microphones. Then, given this data, we leverage self-supervised learning to accelerate behavior prediction from sound. Our experiments indicate that this self-supervised 'pretraining' is crucial to achieving high performance, with a 34.5% lower MSE than plain supervised learning and a 54.3% lower MSE over visual training. Importantly, we find that when asked to generate desired sound profiles, online rollouts of our models on a UR10 robot can produce dynamic behavior that achieves an average of 11.5% improvement over supervised learning on audio similarity metrics.
【26】 Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection标题：Push-Pull：表征视听主动说话人检测的对抗稳健性链接：https://arxiv.org/abs/2210.00753
* 与cs.SD语音【11】为同一篇
作者：Xuanjun Chen,Haibin Wu,Helen Meng,Hung-yi Lee,Jyh-Shing Roger Jang机构：Department. of Computer Science Information Engineering, National Taiwan University, Graduate Institute of Communication Engineering, National Taiwan University, Centre for Perceptual and Interactive Intelligence, The Chinese University of Hong Kong备注：Accepted by SLT 2022摘要：音视频主动说话人检测（AVASD）是一种成熟的检测技术，目前已成为多模态应用中不可缺少的前端技术。然而，据我们所知，AVASD模型的对抗鲁棒性还没有得到研究，更不用说对此类攻击的有效防御了。本文首先通过大量的实验揭示了AVASD模型在纯音频攻击、纯视频攻击和音频视频对抗攻击下的脆弱性。此外，我们还提出了一个新的视听交互损失（AVIL），使攻击者难以找到可行的敌对例子在分配的攻击预算。这种损失的目的是使类间嵌入（即非语音簇和语音簇）充分地分散，并使类内嵌入尽可能地靠近以保持它们的紧凑。实验结果表明，AVIL在多模态攻击下的性能比对抗性训练提高了33.14mAP（%）。摘要：Audio-visual active speaker detection (AVASD) is well-developed, and now is an indispensable front-end for several multi-modal applications. However, to the best of our knowledge, the adversarial robustness of AVASD models hasn't been investigated, not to mention the effective defense against such attacks. In this paper, we are the first to reveal the vulnerability of AVASD models under audio-only, visual-only, and audio-visual adversarial attacks through extensive experiments. What's more, we also propose a novel audio-visual interaction loss (AVIL) for making attackers difficult to find feasible adversarial examples under an allocated attack budget. The loss aims at pushing the inter-class embeddings to be dispersed, namely non-speech and speech clusters, sufficiently disentangled, and pulling the intra-class embeddings as close as possible to keep them compact. Experimental results show the AVIL outperforms the adversarial training by 33.14 mAP (%) under multi-modal attacks.
【27】 Efficient acoustic feature transformation in mismatched environments using a Guided-GAN标题：利用导向型GaN实现失配环境下的高效声特征变换链接：https://arxiv.org/abs/2210.00721
* 与cs.SD语音【12】为同一篇
作者：Walter Heymans,Marelie H. Davel,Charl van Heerden机构：North-West University, South Africa备注：None摘要：本文提出了一种新的框架，以改善在资源稀缺的环境中的自动语音识别（ASR）系统使用一个生成式对抗网络（GAN）的声学输入特征操作。GAN用于在解码之前增强不匹配数据的特征，或者可以可选地用于微调声学模型。我们实现了与多风格训练（MTR）相当的改进，但计算成本更低。利用少于一个小时的数据，在高质量数据上训练并在不匹配的音频上评估的ASR系统的相对字错误率（WER）提高了11.5%到19.7%。实验表明，该框架在训练数据和计算资源有限的资源不足的环境中非常有用。GAN不需要并行训练数据，因为它利用基线声学模型来提供附加损耗项，该附加损耗项引导发生器创建通过基线更好地分类的声学特征。摘要：We propose a new framework to improve automatic speech recognition (ASR) systems in resource-scarce environments using a generative adversarial network (GAN) operating on acoustic input features. The GAN is used to enhance the features of mismatched data prior to decoding, or can optionally be used to fine-tune the acoustic model. We achieve improvements that are comparable to multi-style training (MTR), but at a lower computational cost. With less than one hour of data, an ASR system trained on good quality data, and evaluated on mismatched audio is improved by between 11.5% and 19.7% relative word error rate (WER). Experiments demonstrate that the framework can be very useful in under-resourced environments where training data and computational resources are limited. The GAN does not require parallel training data, because it utilises a baseline acoustic model to provide an additional loss term that guides the generator to create acoustic features that are better classified by the baseline.
【28】 SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model标题：SpeechCLIP：将语音与预先训练的视觉和语言模型相结合链接：https://arxiv.org/abs/2210.00705
* 与cs.SD语音【13】为同一篇
作者：Yi-Jen Shih,Hsuan-Fu Wang,Heng-Jui Chang,Layne Berry,Hung-yi Lee,David Harwath机构：National Taiwan University, MIT CSAIL, The University of Texas at Austin备注：Accepted to IEEE SLT 2022摘要：数据驱动的语音处理模型通常在大量的文本监督下表现良好，但是收集转录的语音数据是昂贵的。因此，我们提出了一个新的框架SpeechCLIP，它通过图像将语音和文本连接起来，以增强语音模型而无需转录。我们利用最先进的预先训练的HuBERT和CLIP，通过成对的图像和语音字幕将它们对齐，只需最少的微调。SpeechCLIP在图像-语音检索方面优于现有技术，并且在没有来自转录的直接监督的情况下执行zero-shot语音-文本检索。此外，SpeechCLIP可以直接从语音中检索语义相关的关键词。摘要：Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.
【29】 Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition标题：用于设备语音识别的换能器多级累进压缩链接：https://arxiv.org/abs/2210.00169
* 与cs.SD语音【14】为同一篇
作者：Jash Rathod,Nauman Dawalatabad,Shatrughan Singh,Dhananjaya Gowda机构：Samsung Research, Massachusetts Institute of Technology备注：None摘要：智能设备中较小的存储器带宽促使开发较小的自动语音识别（ASR）模型。为了获得较小的模型，可以采用模型压缩技术。知识蒸馏（KD）是一种流行的模型压缩方法，它可以实现更小的模型大小，同时模型性能的下降相对较小。在该方法中，知识从经训练的大尺寸教师模型提取到较小尺寸的学生模型。此外，基于传感器的模型最近已经显示出对于设备上流式ASR任务执行良好，而构象器模型在处理长期依赖性方面是有效的。因此，在这项工作中，我们采用了一个流式传感器架构与构象作为编码器。我们提出了一个多阶段渐进式的方法来压缩构象传感器模型使用KD。在多阶段设置中，我们使用提取的学生模型逐步更新我们的教师模型。在标准的LibriSpeech数据集上，我们的实验结果成功地实现了60%以上的压缩率，与更大的教师模型相比，性能没有明显下降。摘要：The smaller memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. To obtain a smaller model, one can employ the model compression techniques. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size with relatively lesser degradation in the model performance. In this approach, knowledge is distilled from a trained large size teacher model to a smaller size student model. Also, the transducer based models have recently shown to perform well for on-device streaming ASR task, while the conformer models are efficient in handling long term dependencies. Hence in this work we employ a streaming transducer architecture with conformer as the encoder. We propose a multi-stage progressive approach to compress the conformer transducer model using KD. We progressively update our teacher model with the distilled student model in a multi-stage setup. On standard LibriSpeech dataset, our experimental results have successfully achieved compression rates greater than 60% without significant degradation in the performance compared to the larger teacher model.
【30】 The DKU-DukeECE Diarization System for the VoxCeleb Speaker Recognition Challenge 2022标题：用于2022年VoxCeleb说话人识别挑战赛的DKU-DukeECA二元化系统链接：https://arxiv.org/abs/2210.01677
作者：Weiqing Wang,Xiaoyi Qin,Ming Cheng,Yucong Zhang,Kangyue Wang,Ming Li备注：arXiv admin note: substantial text overlap with arXiv:2109.02002摘要：本文介绍了DKU-DukeECE提交的2022年VoxCeleb说话人识别挑战赛（VoxSRC-22）第四轮比赛的参赛作品。我们的系统包含了一个融合的语音活动检测模型，一个基于聚类的二分法模型，和一个基于目标说话人语音活动检测的重叠检测模型。总体而言，提交的系统与我们去年在VoxSRC-21中的系统相似。不同的是，我们使用了更好的说话人嵌入和融合的语音活动检测，这大大提高了性能。最后，我们利用DOVER-lap融合了4个不同的系统，并取得了4.75的二值化错误率，在第4轨道中排名第一。摘要：This paper discribes the DKU-DukeECE submission to the 4th track of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our system contains a fused voice activity detection model, a clustering-based diarization model, and a target-speaker voice activity detection-based overlap detection model. Overall, the submitted system is similar to our previous year's system in VoxSRC-21. The difference is that we use a much better speaker embedding and a fused voice activity detection, which significantly improves the performance. Finally, we fuse 4 different systems using DOVER-lap and achieve 4.75 of the diarization error rate, which ranks the 1st place in track 4.
【31】 An attention-based backend allowing efficient fine-tuning of transformer models for speaker verification标题：基于注意力的后端，允许高效微调Transformer模型以进行说话人验证链接：https://arxiv.org/abs/2210.01273
作者：Junyi Peng,Oldrich Plchot,Themos Stafylakis,Ladislav Mosner,Lukas Burget,Jan Cernocky备注：Accepted by SLT2022摘要：近年来，自监督学习范式因其在各种下游任务中取得的巨大成功而受到广泛关注。然而，如何将这些预训练好的模型应用于说话人确认任务的微调策略还有待于充分探索。在本文中，我们分析了几种基于预训练模型的特征提取方法，以及正则化和学习速率计划，以稳定微调过程并进一步提高性能：提出了多头分解注意力池以将说话者表示的比较分解成多个语音聚类。我们向着预训练模型的参数进行正则化，并且我们在微调期间为预训练模型的每一层设置不同的学习速率。实验结果表明，该方法可以将训练时间显著缩短到4小时，并达到SOTA性能：在Vox 1-O、Vox 1-E和Vox 1-H上的EER分别为0.59%、0.79%和1.77%。摘要：In recent years, self-supervised learning paradigm has received extensive attention due to its great success in various down-stream tasks. However, the fine-tuning strategies for adapting those pre-trained models to speaker verification task have yet to be fully explored. In this paper, we analyze several feature extraction approaches built on top of a pre-trained model, as well as regularization and learning rate schedule to stabilize the fine-tuning process and further boost performance: multi-head factorized attentive pooling is proposed to factorize the comparison of speaker representations into multiple phonetic clusters. We regularize towards the parameters of the pre-trained model and we set different learning rates for each layer of the pre-trained model during fine-tuning. The experimental results show our method can significantly shorten the training time to 4 hours and achieve SOTA performance: 0.59%, 0.79% and 1.77% EER on Vox1-O, Vox1-E and Vox1-H, respectively.
【32】 Learning the Spectrogram Temporal Resolution for Audio Classification标题：用于音频分类的语谱图时间分辨率学习链接：https://arxiv.org/abs/2210.01719
* 与cs.SD语音【23】为同一篇
作者：Haohe Liu,Xubo Liu,Qiuqiang Kong,Wenwu Wang,Mark D. Plumbley备注：Under review. Code open-sourced at this https URL摘要：音频谱图是一种时频表示，已被广泛用于音频分类。频谱图的时间分辨率取决于跳数大小。以前的工作通常假设跳大小应该是一个恒定值，例如10毫秒。然而，固定的跳跃大小或分辨率对于不同类型的声音并不总是最佳的。提出了一种新的基于差分时间分辨率学习的音频分类方法DiffRes，该方法能够提高音频分类模型的性能。给定一个用固定跳数计算的频谱图，DiffRes合并非必要的时间帧，同时保留重要的帧。DiffRes充当音频频谱图和分类器之间的“插入”模块，并且可以进行端到端优化。我们在mel谱图上评估DiffRes，然后使用最先进的分类器骨干，并将其应用于五个不同的子任务。与固定分辨率的Mel谱图相比，基于DiffRes的方法在特征层面上至少减少了25%的时间维度，同时降低了计算成本，达到了相同甚至更好的分类精度。从一个高时间分辨率的频谱图（如1毫秒的跳数）开始，我们证明了DiffRes可以在相同的计算复杂度下提高分类精度。摘要：The audio spectrogram is a time-frequency representation that has been widely used for audio classification. The temporal resolution of a spectrogram depends on hop size. Previous works generally assume the hop size should be a constant value such as ten milliseconds. However, a fixed hop size or resolution is not always optimal for different types of sound. This paper proposes a novel method, DiffRes, that enables differentiable temporal resolution learning to improve the performance of audio classification models. Given a spectrogram calculated with a fixed hop size, DiffRes merges non-essential time frames while preserving important frames. DiffRes acts as a "drop-in" module between an audio spectrogram and a classifier, and can be end-to-end optimized. We evaluate DiffRes on the mel-spectrogram, followed by state-of-the-art classifier backbones, and apply it to five different subtasks. Compared with using the fixed-resolution mel-spectrogram, the DiffRes-based method can achieve the same or better classification accuracy with at least 25% fewer temporal dimensions on the feature level, which alleviates the computational cost at the same time. Starting from a high-temporal-resolution spectrogram such as one-millisecond hop size, we show that DiffRes can improve classification accuracy with the same computational complexity.
【33】 Improving Label-Deficient Keyword Spotting Using Self-Supervised Pretraining标题：基于自监督预训练的标签缺陷关键词检测方法链接：https://arxiv.org/abs/2210.01703
* 与cs.SD语音【24】为同一篇
作者：Holger Severin Bovbjerg,Zheng-Hua Tan备注：8 pages, 3 figures, 4 tables, Submitted to Northern Lights Deep Learning Conference 2023摘要：近年来，准确的深度关键词识别（KWS）模型的发展已经导致KWS技术被嵌入到诸如语音助手的许多技术中。其中许多模型都依赖于大量的标记数据来实现良好的性能。结果，它们的使用被限制在可以获得大的标记语音数据集的应用中。自监督学习试图通过利用更容易大量获得的未标记数据来减少对大的标记数据集的需要。然而，大多数自监督方法仅针对非常大的模型进行了研究，而KWS模型期望是小的。在这篇文章中，我们研究了在标签缺失的情况下使用自监督预训练来训练较小的KWS模型。我们使用自监督框架Data2Vec对关键词Transformer模型进行了预训练，并在Google Speech Commands数据集的标签缺失设置上进行了实验。实验结果表明，经过预训练的模型性能明显优于未经过预训练的模型，表明Data2Vec预训练可以提高KWS模型在标签缺失场景下的性能。源代码是公开的。摘要：In recent years, the development of accurate deep keyword spotting (KWS) models has resulted in KWS technology being embedded in a number of technologies such as voice assistants. Many of these models rely on large amounts of labelled data to achieve good performance. As a result, their use is restricted to applications for which a large labelled speech data set can be obtained. Self-supervised learning seeks to mitigate the need for large labelled data sets by leveraging unlabelled data, which is easier to obtain in large amounts. However, most self-supervised methods have only been investigated for very large models, whereas KWS models are desired to be small. In this paper, we investigate the use of self-supervised pretraining for the smaller KWS models in a label-deficient scenario. We pretrain the Keyword Transformer model using the self-supervised framework Data2Vec and carry out experiments on a label-deficient setup of the Google Speech Commands data set. It is found that the pretrained models greatly outperform the models without pretraining, showing that Data2Vec pretraining can increase the performance of KWS models in label-deficient scenarios. The source code is made publicly available.
【34】 Code-Switching without Switching: Language Agnostic End-to-End Speech Translation标题：语码转换不转换：语言不可知的端到端语音翻译链接：https://arxiv.org/abs/2210.01512
* 与cs.SD语音【25】为同一篇
作者：Christian Huber,Enes Yavuz Ugan,Alexander Waibel摘要：我们提出了一个语言不可知的端到端语音翻译模型（LAST），以及一个数据扩充策略来提高代码转换（CS）的性能。随着全球化的发展，在流利的讲话中，多种语言越来越多地可互换地使用。这样的CS使传统的语音识别和翻译复杂化，因为我们必须首先识别说的是哪种语言，然后应用语言相关的识别器和随后的翻译组件来生成期望的目标语言输出。这样的流水线引入了等待时间和错误。在本文中，我们通过将语音识别和翻译视为一个统一的端到端语音翻译问题来消除对这一点的需要。通过用两种输入语言训练LAST，我们将语音解码成一种目标语言，而不管输入语言。LAST在单语使用中提供了相当的识别和语音翻译准确性，同时在观察到CS时大大减少了延迟和错误率。摘要：We propose a) a Language Agnostic end-to-end Speech Translation model (LAST), and b) a data augmentation strategy to increase code-switching (CS) performance. With increasing globalization, multiple languages are increasingly used interchangeably during fluent speech. Such CS complicates traditional speech recognition and translation, as we must recognize which language was spoken first and then apply a language-dependent recognizer and subsequent translation component to generate the desired target language output. Such a pipeline introduces latency and errors. In this paper, we eliminate the need for that, by treating speech recognition and translation as one unified end-to-end speech translation problem. By training LAST with both input languages, we decode speech into one target language, regardless of the input language. LAST delivers comparable recognition and speech translation accuracy in monolingual usage, while reducing latency and error rate considerably when CS is observed.
【35】 Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddings标题：韵律手势：基于分层神经嵌入的韵律感知协同语音手势合成链接：https://arxiv.org/abs/2210.01448
* 与cs.SD语音【26】为同一篇
作者：Tenglong Ao,Qingzhe Gao,Yuke Lou,Baoquan Chen,Libin Liu备注：SIGGRAPH Asia 2022 (Journal Track); Project Page: this https URL摘要：真实感共语手势的自动合成是人工具身智能体创建中一个日益重要但又具有挑战性的任务。现有的手势生成系统主要关注于端到端的手势生成，由于语音和手势之间复杂而微妙的和谐关系，导致难以挖掘出清晰的韵律和语义。提出了一种新颖的共语手势合成方法，该方法在韵律和语义上都取得了令人信服的结果。对于节奏，我们的系统包含了一个健壮的基于节奏的分割流水线，以明确地确保发声和手势之间的时间连贯性。对于手势语义，我们基于语言学理论设计了一种机制来有效地解开语音和运动的低层和高层神经嵌入。高级嵌入对应于语义，而低级嵌入涉及细微的变化。最后，我们建立了语音和动作的层次嵌入之间的对应关系，从而实现了节奏和语义感知的手势合成。与现有的客观指标、新提出的节奏指标和人工反馈的评估表明，我们的方法明显优于最先进的系统。摘要：Automatic synthesis of realistic co-speech gestures is an increasingly important yet challenging task in artificial embodied agent creation. Previous systems mainly focus on generating gestures in an end-to-end manner, which leads to difficulties in mining the clear rhythm and semantics due to the complex yet subtle harmony between speech and gestures. We present a novel co-speech gesture synthesis method that achieves convincing results both on the rhythm and semantics. For the rhythm, our system contains a robust rhythm-based segmentation pipeline to ensure the temporal coherence between the vocalization and gestures explicitly. For the gesture semantics, we devise a mechanism to effectively disentangle both low- and high-level neural embeddings of speech and motion based on linguistic theory. The high-level embedding corresponds to semantics, while the low-level embedding relates to subtle variations. Lastly, we build correspondence between the hierarchical embeddings of the speech and the motion, resulting in rhythm- and semantics-aware gesture synthesis. Evaluations with existing objective metrics, a newly proposed rhythmic metric, and human feedback show that our method outperforms state-of-the-art systems by a clear margin.
【36】 Pay Self-Attention to Audio-Visual Navigation标题：关注视听导航链接：https://arxiv.org/abs/2210.01353
* 与cs.SD语音【27】为同一篇
作者：Yinfeng Yu,Lele Cao,Fuchun Sun,Xiaohong Liu,Liejun Wang摘要：视听具身导航是一个研究热点，其目的是利用以自我为中心的视觉（来自安装在机器人上的传感器）和音频（从目标发出）输入训练机器人到达音频目标。视听信息融合策略对于导航性能自然是重要的，但是现有的方法仍然简单地连接视觉和音频特征，潜在地忽略了上下文的直接影响。此外，现有的方法需要逐段训练或额外的辅助（例如拓扑图和声音语义）。到目前为止，处理具有移动目标的更具挑战性的设置的工作仍然很少。因此，我们提出了一个端到端的框架FSAAVN（feature self-attention audio-visual navigation）来学习如何使用上下文感知的视听融合策略来追踪运动的音频目标，并将其实现为自注意模块。实验结果验证了FSAAVN算法在定性和定量两个方面的优越性，并对视觉模态、视/音频编码器骨干和融合模式的选择提供了独特的见解。摘要：Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.
【37】 And what if two musical versions don't share melody, harmony, rhythm, or lyrics ?标题：如果两个音乐版本不共享旋律、和声、节奏或歌词怎么办？链接：https://arxiv.org/abs/2210.01256
* 与cs.SD语音【28】为同一篇
作者：Mathilde Abrassart,Guillaume Doras摘要：在过去的几年里，版本识别（VI）已经取得了实质性的进展。一方面，度量学习范式的引入有利于可扩展但精确的VI系统的出现。另一方面，使用专注于音乐作品的特定方面的特征，如旋律、和声或歌词，产生了可解释的和有希望的表现。在这项工作中，我们建立在这些最新进展的基础上，并提出了一个基于度量学习的系统，系统地利用四个维度来传达版本之间的音乐相似性：旋律线、和声结构、节奏模式和歌词。我们描述了我们刻意简化的模型结构，并且我们特别地示出了歌词的近似表示是区分版本和非版本的有效代理。然后，我们描述了这些特征如何相互补充，并在两个公开可用的数据集上产生新的最先进的性能。我们最后建议，使用旋律、和声、节奏和歌词特征的组合的VI系统理论上可以达到在这些数据集上可获得的最佳性能。摘要：Version identification (VI) has seen substantial progress over the past few years. On the one hand, the introduction of the metric learning paradigm has favored the emergence of scalable yet accurate VI systems. On the other hand, using features focusing on specific aspects of musical pieces, such as melody, harmony, or lyrics, yielded interpretable and promising performances. In this work, we build upon these recent advances and propose a metric learning-based system systematically leveraging four dimensions commonly admitted to convey musical similarity between versions: melodic line, harmonic structure, rhythmic patterns, and lyrics. We describe our deliberately simple model architecture, and we show in particular that an approximated representation of the lyrics is an efficient proxy to discriminate between versions and non-versions. We then describe how these features complement each other and yield new state-of-the-art performances on two publicly available datasets. We finally suggest that a VI system using a combination of melodic, harmonic, rhythmic and lyrics features could theoretically reach the optimal performances obtainable on these datasets.
【38】 Diagnosis of Parkinson's Disease Based on Voice Signals Using SHAP and Hard Voting Ensemble Method标题：基于Shap和硬投票集成方法的语音信号帕金森病诊断链接：https://arxiv.org/abs/2210.01205
作者：Paria Ghaheri,Hamid Nasiri,Ahmadreza Shateri,Arman Homafar摘要：背景和目的：帕金森病（Parkinson 's disease，PD）是继阿尔茨海默病之后第二常见的进行性神经疾病，以运动和非运动症状为特征。开发一种方法来诊断条件在其开始阶段是必不可少的，因为有相当数量的个人遭受这种疾病。PD通常使用运动症状或其他神经成像技术（如DATSCAN和SPECT）进行识别。这些方法是昂贵的、耗时的，并且对于一般公众是不可用的;而且，它们不是很精确。这些限制促使我们开发了一种新的基于语音信号的SHAP和硬投票集成方法。方法：利用Pearson相关系数分析输入特征与输出特征之间的关系，最终选择相关性高的输入特征。通过极端梯度提升（XGBoost）、轻度梯度提升机（LightGBM）、梯度提升和Bagging对这些选定的特征进行分类。在此基础上，根据四种分类器的性能，确定了硬投票集成方法。在最后阶段，我们提出了Shapley可加性解释（SHAP），根据其在诊断帕金森病中的重要性对特征进行排序。结果和结论：该方法的准确度为85.42%，F1评分为84.94%，精密度为86.77%，特异性为87.62%，灵敏度为83.20%。该研究的结果表明，所提出的方法优于最先进的方法，可以帮助医生诊断帕金森病病例。摘要：Background and Objective: Parkinson's disease (PD) is the second most common progressive neurological condition after Alzheimer's, characterized by motor and non-motor symptoms. Developing a method to diagnose the condition in its beginning phases is essential because of the significant number of individuals afflicting with this illness. PD is typically identified using motor symptoms or other Neuroimaging techniques, such as DATSCAN and SPECT. These methods are expensive, time-consuming, and unavailable to the general public; furthermore, they are not very accurate. These constraints encouraged us to develop a novel technique using SHAP and Hard Voting Ensemble Method based on voice signals. Methods: In this article, we used Pearson Correlation Coefficients to understand the relationship between input features and the output, and finally, input features with high correlation were selected. These selected features were classified by the Extreme Gradient Boosting (XGBoost), Light Gradient Boosting Machine (LightGBM), Gradient Boosting, and Bagging. Moreover, the Hard Voting Ensemble Method was determined based on the performance of the four classifiers. At the final stage, we proposed Shapley Additive exPlanations (SHAP) to rank the features according to their significance in diagnosing Parkinson's disease. Results and Conclusion: The proposed method achieved 85.42% accuracy, 84.94% F1-score, 86.77% precision, 87.62% specificity, and 83.20% sensitivity. The study's findings demonstrated that the proposed method outperformed state-of-the-art approaches and can assist physicians in diagnosing Parkinson's cases.
机器翻译，仅供参考
每日语音速递