今日论文合集：cs.SD语音6篇，eess.AS音频处理8篇。
cs.SD语音
【1】 MAViL: Masked Audio-Video Learners
标题：MAViL：蒙面视听学习者
链接：https://arxiv.org/abs/2212.08071
作者：Po-Yao Huang,Vasu Sharma,Hu Xu,Chaitanya Ryali,Haoqi Fan,Yanghao Li,Shang-Wen Li,Gargi Ghosh,Jitendra Malik,Christoph Feichtenhofer摘要：We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks. Code will be available soon.
【2】 UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
标题：Unity：离散单元的两遍直接语音到语音翻译
链接：https://arxiv.org/abs/2212.08055
作者：Hirofumi Inaguma,Sravya Popuri,Ilia Kulikov,Peng-Jen Chen,Changhan Wang,Yu-An Chung,Yun Tang,Ann Lee,Shinji Watanabe,Juan Pino机构：FAIR, Meta AI♥, Carnegie Mellon University♣备注：Early draft. Work in progress摘要：直接语音到语音翻译（S2 ST），其中所有组件可以被联合优化，比级联方法更有利于用简化的流水线实现快速推理。我们提出了一种新颖的两遍直接S2 ST架构{\textitUnitY}，其首先生成文本表示，随后预测离散声学单元。我们通过在第一遍解码器中的子字预测、改进的两遍解码器结构设计和搜索策略以及更好的训练正则化来增强模型性能。为了充分利用大量未标记文本数据，我们基于自监督去噪自动编码任务对第一遍文本解码器进行预训练。在各种数据规模的基准数据集上的实验评估表明，UnitY比单遍语音到单元转换模型性能高2.5-4.2 ASR-BLEU，解码速度提高2.83倍。我们的结果表明，即使在第二次预测谱图时，所提出的方法也能提高性能。然而，与该情况相比，预测离散单元实现了2.51倍的解码加速。摘要：Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, {\textit UnitY}, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.
【3】 Vision Transformers are Parameter-Efficient Audio-Visual Learners
标题：视觉Transformer是参数高效的视听学习者
链接：https://arxiv.org/abs/2212.07983
作者：Yan-Bo Lin,Yi-Lin Sung,Jie Lei,Mohit Bansal,Gedas Bertasius机构：Department of Computer Science, UNC Chapel Hill备注：project page: this https URL摘要：Vision Transformers（ViTs）在过去的几年中在各种计算机视觉任务中取得了令人印象深刻的成果。在这项工作中，我们研究了冻结的ViT的能力，只在视觉数据上进行预训练，以推广到视听数据，而无需微调其原始参数。为此，我们提出了一种潜在视听混合（LAVISH）适配器，通过向冻结ViT的每一层注入少量可训练参数，使预先训练的ViT适应视听任务。为了有效地融合视觉和听觉线索，我们的LAVISH适配器使用了一小组潜在标记，这些标记形成了注意力瓶颈，从而消除了标准交叉注意的二次成本。与现有的特定于模态的视听方法相比，我们的方法在各种视听任务上实现了有竞争力的甚至更好的性能，同时使用更少的可调参数，并且不依赖于昂贵的音频预训练或外部音频编码器。我们的代码可从www.example.com获得https://genjib.github.io/project_page/LAVISH/摘要：Vision transformers (ViTs) have achieved impressive results on various computer vision tasks in the last several years. In this work, we study the capability of frozen ViTs, pretrained only on visual data, to generalize to audio-visual data without finetuning any of its original parameters. To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT. To efficiently fuse visual and audio cues, our LAVISH adapter uses a small set of latent tokens, which form an attention bottleneck, thus, eliminating the quadratic cost of standard cross-attention. Compared to the existing modality-specific audio-visual methods, our approach achieves competitive or even better performance on various audio-visual tasks while using fewer tunable parameters and without relying on costly audio pretraining or external audio encoders. Our code is available at https://genjib.github.io/project_page/LAVISH/
【4】 RWEN-TTS: Relation-aware Word Encoding Network for Natural Text-to-Speech Synthesis
标题：RWEN-TTS：面向自然文语合成的关系感知单词编码网络
链接：https://arxiv.org/abs/2212.07939
作者：Shinhyeok Oh,HyeongRae Noh,Yoonseok Hong,Insoo Oh摘要：随着深度学习技术的出现，大量能够产生类人语音的文本语音转换（TTS）模型应运而生。近年来，通过引入输入文本的句法和语义信息，人们提出了各种方法来丰富TTS模型的自然性和表达性。尽管这些策略取得了令人瞩目的效果，但在利用语言信息方面仍存在一定的局限性。首先，大多数方法只使用图网络来利用句法和语义信息，而没有考虑语言特征。第二，大多数以前的工作在对句法和语义信息进行编码时并没有明确地考虑相邻词，尽管很明显，相邻词在对当前词进行编码时通常是有意义的。为了解决这些问题，我们提出了关系感知的词编码网络（Relation-aware Word Encoding Network，RWEN），它有效地允许基于两个模块的句法和语义信息（即，语义层次关系编码和邻词关系编码）。实验结果表明，与以往的工作相比有了很大的改进。摘要：With the advent of deep learning, a huge number of text-to-speech (TTS) models which produce human-like speech have emerged. Recently, by introducing syntactic and semantic information w.r.t the input text, various approaches have been proposed to enrich the naturalness and expressiveness of TTS models. Although these strategies showed impressive results, they still have some limitations in utilizing language information. First, most approaches only use graph networks to utilize syntactic and semantic information without considering linguistic features. Second, most previous works do not explicitly consider adjacent words when encoding syntactic and semantic information, even though it is obvious that adjacent words are usually meaningful when encoding the current word. To address these issues, we propose Relation-aware Word Encoding Network (RWEN), which effectively allows syntactic and semantic information based on two modules (i.e., Semantic-level Relation Encoding and Adjacent Word Relation Encoding). Experimental results show substantial improvements compared to previous works.
【5】 A large-scale and PCR-referenced vocal audio dataset for COVID-19
标题：基于聚合酶链式反应的新冠肺炎大规模语音数据集
链接：https://arxiv.org/abs/2212.07738
作者：Jobie Budd,Kieran Baker,Emma Karoune,Harry Coppock,Selina Patel,Ana Tendero Cañadas,Alexander Titcomb,Richard Payne,David Hurley,Sabrina Egglestone,Lorraine Butler,Jonathon Mellor,George Nicholson,Ivan Kiskin,Vasiliki Koutra,Radka Jersakova,Rachel A. McKendry,Peter Diggle,Sylvia Richardson,Björn W. Schuller,Steven Gilmour,Davide Pigoli,Stephen Roberts,Josef Packham,Tracey Thornley,Chris Holmes机构：London Centre for Nanotechnology, University College London, UK, Division of Medicine, University College London, UK, King's College London, UK, The Alan Turing Institute, UK, Imperial College London, UK, UK Health Security Agency, UK摘要：The UK COVID-19 Vocal Audio Dataset is designed for the training and evaluation of machine learning models that classify SARS-CoV-2 infection status or associated respiratory symptoms using vocal audio. The UK Health Security Agency recruited voluntary participants through the national Test and Trace programme and the REACT-1 survey in England from March 2021 to March 2022, during dominant transmission of the Alpha and Delta SARS-CoV-2 variants and some Omicron variant sublineages. Audio recordings of volitional coughs, exhalations, and speech were collected in the 'Speak up to help beat coronavirus' digital survey alongside demographic, self-reported symptom and respiratory condition data, and linked to SARS-CoV-2 test results. The UK COVID-19 Vocal Audio Dataset represents the largest collection of SARS-CoV-2 PCR-referenced audio recordings to date. PCR results were linked to 70,794 of 72,999 participants and 24,155 of 25,776 positive cases. Respiratory symptoms were reported by 45.62% of participants. This dataset has additional potential uses for bioacoustics research, with 11.30% participants reporting asthma, and 27.20% with linked influenza PCR test results.
【6】 Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
标题：视觉、语音和语言的语境化目标表征的高效自监督学习
链接：https://arxiv.org/abs/2212.07525
作者：Alexei Baevski,Arun Babu,Wei-Ning Hsu,Michael Auli摘要：当前的自监督学习算法通常是模态特定的，并且需要大量的计算资源。为了解决这些问题，我们提高了data 2 vec的训练效率，data 2 vec是一个跨几种模态的学习目标。我们不对掩码令牌进行编码，使用快速卷积解码器，并分摊构建教师表示的工作。data2vec2.0得益于data 2 vec中引入的丰富的上下文化目标表示，其允许快速的自我监督学习器。ImageNet-1 K图像分类的实验表明，data 2 vec 2.0在预训练时间减少16.4倍的情况下达到了掩蔽自动编码器的准确性，在Librispeech语音识别方面，它在时间减少10.6倍的情况下与wav 2 vec 2.0一样好，在GLUE自然语言理解方面，它在一半的时间内达到了重新训练的RoBERTa模型。用一些速度换取准确度，结果ImageNet-1 K top-1准确度为86.8%，ViT-L模型经过150个时期的训练。摘要：Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8\% with a ViT-L model trained for 150 epochs.
【1】 Improving Fast-slow Encoder based Transducer with Streaming Deliberation
标题：基于流处理的快慢编码变送器的改进
链接：https://arxiv.org/abs/2212.07650
作者：Ke Li,Jay Mahadeokar,Jinxi Guo,Yangyang Shi,Gil Keren,Ozlem Kalinli,Michael L. Seltzer,Duc Le机构：Meta AI, USA备注：Submitted to ICASSP 2023摘要：本文介绍了一种基于快慢编码器的流式审议转换器，用于端到端的自动语音识别。我们的目标是提高基于快慢编码器的传感器的识别精度，同时通过集成流式审议模型保持其低延迟。具体地，审议模型利用来自流式快速编码器的部分假设，并且隐式地学习以校正识别错误。我们改进了基于快-慢编码器的换能器的平行波束搜索算法，使其更有效并与审议模型兼容。此外，设计了审议模型来处理流数据。为了进一步提高审议性能，提出了一种简单的文本增强方法。我们还比较了LSTM和Conformer模型对部分假设的编码。在Librispeech和内部数据上的实验表明，与基于快慢编码器的换能器相比，相对WER减少（WERR）从3%到5%，模型大小略有增加，额外的令牌发射延迟可忽略不计。与普通神经传感器相比，所提出的审议模型与基于快慢编码器的传感器一起在Librispeech上获得相对10-11%的WERR，在内部数据上获得大约6%的WERR，具有较小的发射延迟。摘要：This paper introduces a fast-slow encoder based transducer with streaming deliberation for end-to-end automatic speech recognition. We aim to improve the recognition accuracy of the fast-slow encoder based transducer while keeping its latency low by integrating a streaming deliberation model. Specifically, the deliberation model leverages partial hypotheses from the streaming fast encoder and implicitly learns to correct recognition errors. We modify the parallel beam search algorithm for fast-slow encoder based transducer to be efficient and compatible with the deliberation model. In addition, the deliberation model is designed to process streaming data. To further improve the deliberation performance, a simple text augmentation approach is explored. We also compare LSTM and Conformer models for encoding partial hypotheses. Experiments on Librispeech and in-house data show relative WER reductions (WERRs) from 3% to 5% with a slight increase in model size and negligible extra token emission latency compared with fast-slow encoder based transducer. Compared with vanilla neural transducers, the proposed deliberation model together with fast-slow encoder based transducer obtains relative 10-11% WERRs on Librispeech and around relative 6% WERR on in-house data with smaller emission delays.
【2】 DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement
标题：DIFT-AN：用于多通道语音增强的密集频率-时间注意网络
链接：https://arxiv.org/abs/2212.07570
作者：Dongheon Lee,Jung-Woo Choi机构：of Science and Technology (KAIST), Daejeon , South Korea (e-mail:, Fourier transform (STFT) data [,], [,]. However, end-to-end, methods using time-domain waveforms [,]–[,] have been de-, veloped to overcome the shortcomings of STFT approaches备注：5 pages, 5 figures, 3 tables摘要：在本研究中，我们提出一个密集频率-时间注意网路（DeFT-AN）来进行多通道语音增强。DeFT-AN是预测用于抑制嵌入在输入信号的短时傅立叶变换（STFT）中的噪声和混响的复频谱掩蔽模式的掩蔽估计网络。所提出的掩蔽估计网络包含三种不同类型的块，用于聚集空间、光谱和时间维度的信息。它利用了一个改进的前馈网络的频谱Transformer和一个连续扩张卷积的时间构象器。专用于音频信号的三种不同特性的密集块和Transformers的使用实现了更全面的去噪和去混响。DeFT-AN相对于最先进的多信道模型的显著性能基于两个流行的噪声和混响数据集在语音质量和可懂度的各种度量方面得到了证明。摘要：In this study, we propose a dense frequency-time attentive network (DeFT-AN) for multichannel speech enhancement. DeFT-AN is a mask estimation network that predicts a complex spectral masking pattern for suppressing the noise and reverberation embedded in the short-time Fourier transform (STFT) of an input signal. The proposed mask estimation network incorporates three different types of blocks for aggregating information in the spatial, spectral, and temporal dimensions. It utilizes a spectral transformer with a modified feed-forward network and a temporal conformer with sequential dilated convolutions. The use of dense blocks and transformers dedicated to the three different characteristics of audio signals enables more comprehensive denoising and dereverberation. The remarkable performance of DeFT-AN over state-of-the-art multichannel models is demonstrated based on two popular noisy and reverberant datasets in terms of various metrics for speech quality and intelligibility.
【3】 MAViL: Masked Audio-Video Learners
标题：MAViL：蒙面视听学习者
链接：https://arxiv.org/abs/2212.08071
*与cs.SD语音【1】为同一篇
作者：Po-Yao Huang,Vasu Sharma,Hu Xu,Chaitanya Ryali,Haoqi Fan,Yanghao Li,Shang-Wen Li,Gargi Ghosh,Jitendra Malik,Christoph Feichtenhofer摘要：我们提出蒙面视听学习者（MAViL）来训练视听表征。我们的方法通过三种互补形式的自我监督来学习：（1）重构掩蔽的音频和视频输入数据，（2）利用掩蔽的模态内和模态间对比学习，以及（3）通过重构从前两个目标学习的联合音频-视频上下文化特征来进行自训练。利用MAViL进行预训练不仅能够使模型在视听分类和检索任务中表现良好，而且还能够改善每种模态的孤立表示，而无需使用来自其他模态的信息进行微调或推理。根据经验，MAViL在AudioSet（53.1 mAP）和VGGSound（67.1%准确度）上设置了新的最先进技术。在这些基准测试中，自我监督的视听模型首次超过了使用外部监督的模型。代码将很快提供。摘要：We present Masked Audio-Video Learners (MAViL) to train audio-visual representations. Our approach learns with three complementary forms of self-supervision: (1) reconstruction of masked audio and video input data, (2) intra- and inter-modal contrastive learning with masking, and (3) self-training by reconstructing joint audio-video contextualized features learned from the first two objectives. Pre-training with MAViL not only enables the model to perform well in audio-visual classification and retrieval tasks but also improves representations of each modality in isolation, without using information from the other modality for fine-tuning or inference. Empirically, MAViL sets a new state-of-the-art on AudioSet (53.1 mAP) and VGGSound (67.1% accuracy). For the first time, a self-supervised audio-visual model outperforms ones that use external supervision on these benchmarks. Code will be available soon.
【4】 UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units
标题：Unity：离散单元的两遍直接语音到语音翻译
链接：https://arxiv.org/abs/2212.08055
*与cs.SD语音【2】为同一篇
作者：Hirofumi Inaguma,Sravya Popuri,Ilia Kulikov,Peng-Jen Chen,Changhan Wang,Yu-An Chung,Yun Tang,Ann Lee,Shinji Watanabe,Juan Pino机构：FAIR, Meta AI♥, Carnegie Mellon University♣备注：Early draft. Work in progress摘要：直接语音到语音翻译（S2 ST），其中所有组件可以被联合优化，比级联方法更有利于用简化的流水线实现快速推理。我们提出了一种新颖的两遍直接S2 ST架构{\textitUnitY}，其首先生成文本表示，随后预测离散声学单元。我们通过在第一遍解码器中的子字预测、改进的两遍解码器结构设计和搜索策略以及更好的训练正则化来增强模型性能。为了充分利用大量未标记文本数据，我们基于自监督去噪自动编码任务对第一遍文本解码器进行预训练。在各种数据规模的基准数据集上的实验评估表明，UnitY比单遍语音到单元转换模型性能高2.5-4.2 ASR-BLEU，解码速度提高2.83倍。我们的结果表明，即使在第二次预测谱图时，所提出的方法也能提高性能。然而，与该情况相比，预测离散单元实现了2.51倍的解码加速。摘要：Direct speech-to-speech translation (S2ST), in which all components can be optimized jointly, is advantageous over cascaded approaches to achieve fast inference with a simplified pipeline. We present a novel two-pass direct S2ST architecture, {\textit UnitY}, which first generates textual representations and predicts discrete acoustic units subsequently. We enhance the model performance by subword prediction in the first-pass decoder, advanced two-pass decoder architecture design and search strategy, and better training regularization. To leverage large amounts of unlabeled text data, we pre-train the first-pass text decoder based on the self-supervised denoising auto-encoding task. Experimental evaluations on benchmark datasets at various data scales demonstrate that UnitY outperforms a single-pass speech-to-unit translation model by 2.5-4.2 ASR-BLEU with 2.83x decoding speed-up. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass. However, predicting discrete units achieves 2.51x decoding speed-up compared to that case.
【5】 Vision Transformers are Parameter-Efficient Audio-Visual Learners
标题：视觉Transformer是参数高效的视听学习者
链接：https://arxiv.org/abs/2212.07983
*与cs.SD语音【3】为同一篇
作者：Yan-Bo Lin,Yi-Lin Sung,Jie Lei,Mohit Bansal,Gedas Bertasius机构：Department of Computer Science, UNC Chapel Hill备注：project page: this https URL摘要：Vision Transformers（ViTs）在过去的几年中在各种计算机视觉任务中取得了令人印象深刻的成果。在这项工作中，我们研究了冻结的ViT的能力，只在视觉数据上进行预训练，以推广到视听数据，而无需微调其原始参数。为此，我们提出了一种潜在视听混合（LAVISH）适配器，通过向冻结ViT的每一层注入少量可训练参数，使预先训练的ViT适应视听任务。为了有效地融合视觉和听觉线索，我们的LAVISH适配器使用了一小组潜在标记，这些标记形成了注意力瓶颈，从而消除了标准交叉注意的二次成本。与现有的特定于模态的视听方法相比，我们的方法在各种视听任务上实现了有竞争力的甚至更好的性能，同时使用更少的可调参数，并且不依赖于昂贵的音频预训练或外部音频编码器。我们的代码可从www.example.com获得https://genjib.github.io/project_page/LAVISH/摘要：Vision transformers (ViTs) have achieved impressive results on various computer vision tasks in the last several years. In this work, we study the capability of frozen ViTs, pretrained only on visual data, to generalize to audio-visual data without finetuning any of its original parameters. To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT. To efficiently fuse visual and audio cues, our LAVISH adapter uses a small set of latent tokens, which form an attention bottleneck, thus, eliminating the quadratic cost of standard cross-attention. Compared to the existing modality-specific audio-visual methods, our approach achieves competitive or even better performance on various audio-visual tasks while using fewer tunable parameters and without relying on costly audio pretraining or external audio encoders. Our code is available at https://genjib.github.io/project_page/LAVISH/
【6】 RWEN-TTS: Relation-aware Word Encoding Network for Natural Text-to-Speech Synthesis
标题：RWEN-TTS：面向自然文语合成的关系感知单词编码网络
链接：https://arxiv.org/abs/2212.07939
*与cs.SD语音【4】为同一篇
作者：Shinhyeok Oh,HyeongRae Noh,Yoonseok Hong,Insoo Oh摘要：随着深度学习技术的出现，大量能够产生类人语音的文本语音转换（TTS）模型应运而生。近年来，通过引入输入文本的句法和语义信息，人们提出了各种方法来丰富TTS模型的自然性和表达性。尽管这些策略取得了令人瞩目的效果，但在利用语言信息方面仍存在一定的局限性。首先，大多数方法只使用图网络来利用句法和语义信息，而没有考虑语言特征。第二，大多数以前的工作在对句法和语义信息进行编码时并没有明确地考虑相邻词，尽管很明显，相邻词在对当前词进行编码时通常是有意义的。为了解决这些问题，我们提出了关系感知的词编码网络（Relation-aware Word Encoding Network，RWEN），它有效地允许基于两个模块的句法和语义信息（即，语义层次关系编码和邻词关系编码）。实验结果表明，与以往的工作相比有了很大的改进。摘要：With the advent of deep learning, a huge number of text-to-speech (TTS) models which produce human-like speech have emerged. Recently, by introducing syntactic and semantic information w.r.t the input text, various approaches have been proposed to enrich the naturalness and expressiveness of TTS models. Although these strategies showed impressive results, they still have some limitations in utilizing language information. First, most approaches only use graph networks to utilize syntactic and semantic information without considering linguistic features. Second, most previous works do not explicitly consider adjacent words when encoding syntactic and semantic information, even though it is obvious that adjacent words are usually meaningful when encoding the current word. To address these issues, we propose Relation-aware Word Encoding Network (RWEN), which effectively allows syntactic and semantic information based on two modules (i.e., Semantic-level Relation Encoding and Adjacent Word Relation Encoding). Experimental results show substantial improvements compared to previous works.
【7】 A large-scale and PCR-referenced vocal audio dataset for COVID-19
标题：基于聚合酶链式反应的新冠肺炎大规模语音数据集
链接：https://arxiv.org/abs/2212.07738
*与cs.SD语音【5】为同一篇
作者：Jobie Budd,Kieran Baker,Emma Karoune,Harry Coppock,Selina Patel,Ana Tendero Cañadas,Alexander Titcomb,Richard Payne,David Hurley,Sabrina Egglestone,Lorraine Butler,Jonathon Mellor,George Nicholson,Ivan Kiskin,Vasiliki Koutra,Radka Jersakova,Rachel A. McKendry,Peter Diggle,Sylvia Richardson,Björn W. Schuller,Steven Gilmour,Davide Pigoli,Stephen Roberts,Josef Packham,Tracey Thornley,Chris Holmes机构：London Centre for Nanotechnology, University College London, UK, Division of Medicine, University College London, UK, King's College London, UK, The Alan Turing Institute, UK, Imperial College London, UK, UK Health Security Agency, UK摘要：英国COVID-19语音音频数据集设计用于训练和评估机器学习模型，该模型使用语音音频对SARS-CoV-2感染状态或相关呼吸道症状进行分类。2021年3月至2022年3月，在SARS-CoV-2 α和δ变异株以及部分Omicron变异株亚系的优势传播期间，英国卫生安全局通过国家检测和溯源计划以及REACT-1调查在英格兰招募志愿参与者。在“大声说话，帮助战胜冠状病毒”数字调查中，收集了自愿咳嗽、呼气和讲话的录音，以及人口统计学、自我报告的症状和呼吸状况数据，并与SARS-CoV-2检测结果相关联。英国COVID-19声乐音频数据集代表了迄今为止最大的SARS-CoV-2 PCR参考音频记录集合。PCR结果与72,999例参与者中的70,794例和25,776例阳性病例中的24,155例相关。45.62%的受试者报告了呼吸道症状。该数据集对生物声学研究有额外的潜在用途，11.30%的参与者报告哮喘，27.20%的参与者报告流感PCR检测结果。摘要：The UK COVID-19 Vocal Audio Dataset is designed for the training and evaluation of machine learning models that classify SARS-CoV-2 infection status or associated respiratory symptoms using vocal audio. The UK Health Security Agency recruited voluntary participants through the national Test and Trace programme and the REACT-1 survey in England from March 2021 to March 2022, during dominant transmission of the Alpha and Delta SARS-CoV-2 variants and some Omicron variant sublineages. Audio recordings of volitional coughs, exhalations, and speech were collected in the 'Speak up to help beat coronavirus' digital survey alongside demographic, self-reported symptom and respiratory condition data, and linked to SARS-CoV-2 test results. The UK COVID-19 Vocal Audio Dataset represents the largest collection of SARS-CoV-2 PCR-referenced audio recordings to date. PCR results were linked to 70,794 of 72,999 participants and 24,155 of 25,776 positive cases. Respiratory symptoms were reported by 45.62% of participants. This dataset has additional potential uses for bioacoustics research, with 11.30% participants reporting asthma, and 27.20% with linked influenza PCR test results.
【8】 Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Language
标题：视觉、语音和语言的语境化目标表征的高效自监督学习
链接：https://arxiv.org/abs/2212.07525
*与cs.SD语音【6】为同一篇
作者：Alexei Baevski,Arun Babu,Wei-Ning Hsu,Michael Auli摘要：当前的自监督学习算法通常是模态特定的，并且需要大量的计算资源。为了解决这些问题，我们提高了data 2 vec的训练效率，data 2 vec是一个跨几种模态的学习目标。我们不对掩码令牌进行编码，使用快速卷积解码器，并分摊构建教师表示的工作。data2vec2.0得益于data 2 vec中引入的丰富的上下文化目标表示，其允许快速的自我监督学习器。ImageNet-1 K图像分类的实验表明，data 2 vec 2.0在预训练时间减少16.4倍的情况下达到了掩蔽自动编码器的准确性，在Librispeech语音识别方面，它在时间减少10.6倍的情况下与wav 2 vec 2.0一样好，在GLUE自然语言理解方面，它在一半的时间内达到了重新训练的RoBERTa模型。用一些速度换取准确度，结果ImageNet-1 K top-1准确度为86.8%，ViT-L模型经过150个时期的训练。摘要：Current self-supervised learning algorithms are often modality-specific and require large amounts of computational resources. To address these issues, we increase the training efficiency of data2vec, a learning objective that generalizes across several modalities. We do not encode masked tokens, use a fast convolutional decoder and amortize the effort to build teacher representations. data2vec 2.0 benefits from the rich contextualized target representations introduced in data2vec which enable a fast self-supervised learner. Experiments on ImageNet-1K image classification show that data2vec 2.0 matches the accuracy of Masked Autoencoders in 16.4x lower pre-training time, on Librispeech speech recognition it performs as well as wav2vec 2.0 in 10.6x less time, and on GLUE natural language understanding it matches a retrained RoBERTa model in half the time. Trading some speed for accuracy results in ImageNet-1K top-1 accuracy of 86.8\% with a ViT-L model trained for 150 epochs.
机器翻译，仅供参考
每日语音速递