今日论文合集：cs.SD语音11篇，eess.AS音频处理16篇。
cs.SD语音

【1】 LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading标题：LIPSFUS：用于唇读视听感觉融合的神经形态数据集链接：http://arxiv.org/pdf/2304.01080v1

作者：Antonio Rios-Navarro,Enrique Piñero-Fuentes,Salvador Canas-Moreno,Aqib Javed,Jin Harkin,Alejandro Linares-Barranco机构：University of Seville备注：Submitted to ISCAS2023, 4 pages, plus references, github link provided摘要：本文提出了一种感觉融合神经形态数据集收集精确的时间同步使用一组地址事件表示传感器和工具。目标应用程序是不同机器学习应用程序的几个关键字的唇读，例如数字，机器人命令和辅助丰富的语音短单词。该数据集被放大的尖峰版本的视听唇读数据集收集基于帧的相机。LIPSFUS是公开可用的，并且已经通过用于音频和视觉分类的深度学习架构进行了验证。它旨在用于基于人工和尖峰神经网络算法的感觉融合架构。摘要：This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.

【2】 The Music Annotation Pattern标题：音乐的注解模式链接：http://arxiv.org/pdf/2304.00988v1

作者：Jacopo de Berardinis,Albert Meroño-Peñuela,Andrea Poltronieri,Valentina Presutti机构：King’s College London, University of Bologna备注：12 pages, 3 figures. Proceedings of the 13th Workshop on Ontology Design and Patterns, edited by V. Sv'atek et al., WOP, 2022摘要：音乐内容的诠释是一个复杂的表现过程，这是由于音乐内容本身所固有的多面性、主观性和跨学科性。在过去的几十年中，已经开发了许多用于注释音乐的系统和约定作为独立的标准。很少有人做，使他们的互操作性，这危及跨语料库的研究，因为它需要用户熟悉大量的公约。这些系统中的大多数缺乏表示音乐语言的复杂性所需的语义表达能力，并且不能对源自音频和符号源的多模态注释进行建模。在这篇文章中，我们介绍了音乐注释模式，本体设计模式（ODP）同质化不同的注释系统，并表示几种类型的音乐对象（如和弦，模式，结构）。该ODP在不同级别和时间粒度上保留对象内容的语义。此外，我们的ODP占多模态前期，描述来自不同来源的注释，它是第一个实现大规模集成音乐数据集。摘要：The annotation of music content is a complex process to represent due to its inherent multifaceted, subjectivity, and interdisciplinary nature. Numerous systems and conventions for annotating music have been developed as independent standards over the past decades. Little has been done to make them interoperable, which jeopardises cross-corpora studies as it requires users to familiarise with a multitude of conventions. Most of these systems lack the semantic expressiveness needed to represent the complexity of the musical language and cannot model multi-modal annotations originating from audio and symbolic sources. In this article, we introduce the Music Annotation Pattern, an Ontology Design Pattern (ODP) to homogenise different annotation systems and to represent several types of musical objects (e.g. chords, patterns, structures). This ODP preserves the semantics of the object's content at different levels and temporal granularity. Moreover, our ODP accounts for multi-modality upfront, to describe annotations derived from different sources, and it is the first to enable the integration of music datasets at a large scale.

【3】 The Music Note Ontology标题：音符本体论链接：http://arxiv.org/pdf/2304.00986v1

作者：Andrea Poltronieri,Aldo Gangemi机构：Gangemi,[,−,−,−,], LILEC, University of Bologna, Bologna, Italy, FICLIT, University of Bologna, Bologna, Italy备注：12 pages, 1 figure, 1 table. Proceedings of the 12th Workshop on Ontology Design and Patterns (WOP 2021), Online, edited by K. Hammar et al., 2021摘要：在本文中，我们提出了音符本体论，一个本体论建模音符及其实现。本体论解决了符号表示系统中表示的音符与其实现（即音乐表演）之间的关系。因此，这项工作的目的是解决建模和表示问题时出现的分析抽象的符号特征和相应的物理特征的音频信号之间的关系。本体是由三个不同的本体设计模式（ODP），模型的结构的分数（分数部分模式），注意的符号符号表示法（音符模式）和其实现（音乐对象模式）。摘要：In this paper we propose the Music Note Ontology, an ontology for modelling music notes and their realisation. The ontology addresses the relation between a note represented in a symbolic representation system, and its realisation, i.e. a musical performance. This work therefore aims to solve the modelling and representation issues that arise when analysing the relationships between abstract symbolic features and the corresponding physical features of an audio signal. The ontology is composed of three different Ontology Design Patterns (ODP), which model the structure of the score (Score Part Pattern), the note in the symbolic notation (Music Note Pattern) and its realisation (Musical Object Pattern).

【4】 Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP标题：语音情感识别系统的设计与评估：IEMOCAP的现实检验案例研究链接：http://arxiv.org/pdf/2304.00860v1

作者：Nikolaos Antoniou,Athanasios Katsamanis,Theodoros Giannakopoulos,Shrikanth Narayanan机构：† Behavioral Signal Technologies, Los Angeles, CA, USA, ∗ Institute for Language and Speech Processing, Athena Research Center, Athens, Greece, + SAIL-University of Southern California, Los Angeles, CA, USA备注：Accepted at ICASSP 2023摘要：有一个迫切需要的指导方针和标准的测试集，以允许直接和公平的比较语音情感识别（SER）。虽然资源，如交互式情感二元运动捕捉（IEMOCAP）数据库，已成为广泛采用的参考语料库的研究人员开发和测试模型SER，发表的工作揭示了广泛的假设和多样性，在其使用的挑战再现性和泛化。基于对使用IEMOCAP作为用例的SER的最新进展的批判性审查，我们的工作旨在做出两个贡献：首先，使用最近的文献分析，包括假设和其中使用的指标，我们提供了一套SER评估准则。其次，使用最近的出版物与开源的实现，我们专注于在SER的再现性评估。摘要：There is an imminent need for guidelines and standard test sets to allow direct and fair comparisons of speech emotion recognition (SER). While resources, such as the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database, have emerged as widely-adopted reference corpora for researchers to develop and test models for SER, published work reveals a wide range of assumptions and variety in its use that challenge reproducibility and generalization. Based on a critical review of the latest advances in SER using IEMOCAP as the use case, our work aims at two contributions: First, using an analysis of the recent literature, including assumptions made and metrics used therein, we provide a set of SER evaluation guidelines. Second, using recent publications with open-sourced implementations, we focus on reproducibility assessment in SER.

【5】 AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models标题：审计：遵循潜在扩散模型的说明进行音频编辑链接：http://arxiv.org/pdf/2304.00830v1

作者：Yuancheng Wang,Zeqian Ju,Xu Tan,Lei He,Zhizheng Wu,Jiang Bian,Sheng Zhao机构：Microsoft,The Chinese University of Hong Kong, Shenzhen摘要：音频编辑适用于各种目的，例如添加背景声音效果、替换乐器以及修复损坏的音频。最近，一些基于扩散的方法通过使用以输出音频的文本描述为条件的扩散和去噪过程来实现zero-shot音频编辑。然而，这些方法仍然存在一些问题：1）没有经过编辑工作的培训，不能保证良好的编辑效果;2）它们可能错误地修改不需要编辑的音频段;3）它们需要输出音频的完整描述，这在实际场景中并不总是可用或必要的。在这项工作中，我们提出了AUDIT，一个指令指导的音频编辑模型的基础上潜在的扩散模型。具体来说，AUDIT有三个主要设计特点：1）我们为不同的音频编辑任务构建三元组训练数据（指令、输入音频、输出音频），并使用指令和输入（待编辑）音频作为条件来训练扩散模型，并生成输出（编辑）音频;2）它可以通过比较输入和输出音频之间的差异来自动学习仅修改需要编辑的片段;3）它只需要编辑指令而不是完整的目标音频描述作为文本输入。AUDIT在针对若干音频编辑任务的客观和主观度量两者中实现了最先进的结果（例如，添加、删除、替换、修补、超分辨率）。演示示例可在www.example.com上获得https: audit-demo.github.io 。摘要：Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https: audit-demo.github.io .

【6】 Musical creativity enabled by nonlinear oscillations of a bubble in water标题：水中气泡的非线性振荡带来的音乐创造力链接：http://arxiv.org/pdf/2304.00822v1

作者：Ivan S. Maksymov机构：Artificial Intelligence and Cyber Futures Institute, Charles Sturt University, Bathurst, NSW , Australia摘要：制作原创和安排现有的音乐成果是一门需要多年学习和实践才能掌握的艺术。然而，尽管人工智能驱动的音乐创造力领域不断取得进展，但制作高质量的音乐作品仍然是人类的特权。在这里，我们证明了一个单一的气泡在水中可以用来产生创造性的音乐成果，当它的非线性振荡下的声压信号，编码一段古典音乐。气泡的响应的音频信号类似于原始作曲的电吉他版本。我们建议，并提供合理的理论支持论据，泡沫的这种特性可以用来创建物理启发的人工智能系统，能够模拟人类在音乐编排和作曲方面的创造力。摘要：Producing original and arranging existing musical outcomes is an art that takes years of learning and practice to master. Yet, despite the constant advances in the field of AI-powered musical creativity, production of quality musical outcomes remains a prerogative of the humans. Here we demonstrate that a single bubble in water can be used to produce creative musical outcomes, when it nonlinearly oscillates under an acoustic pressure signal that encodes a piece of classical music. The audio signal of the response of the bubble resembles an electric guitar version of the original composition. We suggest, and provide plausible theoretical supporting arguments, that this property of the bubble can be used to create physics-inspired AI systems capable of simulating human creativity in arrangement and composition of music.

【7】 Multilingual Word Error Rate Estimation: e-WER3标题：多语种错误率估计：E-WER3链接：http://arxiv.org/pdf/2304.00649v1

作者：Shammur Absar Chowdhury,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar备注：Accepted in ICASSP, Multilingual WER estimation, End-to-End systems, multilingual model, automatic word error rate estimation摘要：多语种自动语音识别系统的成功为许多语音驱动的应用程序提供了支持。然而，测量这样的系统的性能仍然是一个重大的挑战，由于其依赖于手动转录的语音数据在单语言和多语言的情况。在本文中，我们提出了一种新的多语言框架-eWER 3-联合训练的声学和词汇表示，以估计单词错误率。我们证明了eWER 3的有效性（i）预测WER，而不使用任何内部状态的ASR和（ii）使用多语言共享的潜在空间，以推动密切相关的语言的性能。我们表明，我们提出的多语言模型优于以前的单语单词错误率估计方法（eWER 2）的绝对9 %的皮尔逊相关系数（PCC）的增加，更好的整体估计之间的预测和参考WER。摘要：The success of the multilingual automatic speech recognition systems empowered many voice-driven applications. However, measuring the performance of such systems remains a major challenge, due to its dependency on manually transcribed speech data in both mono- and multilingual scenarios. In this paper, we propose a novel multilingual framework -- eWER3 -- jointly trained on acoustic and lexical representation to estimate word error rate. We demonstrate the effectiveness of eWER3 to (i) predict WER without using any internal states from the ASR and (ii) use the multilingual shared latent space to push the performance of the close-related languages. We show our proposed multilingual model outperforms the previous monolingual word error rate estimation method (eWER2) by an absolute 9 % increase in Pearson correlation coefficient (PCC), with better overall estimation between the predicted and reference WER.

【8】 A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation标题：一种用于高效语音驱动的语音驱动的统一压缩框架链接：http://arxiv.org/pdf/2304.00471v1

作者：Bo-Kyeong Kim,Jaemin Kang,Daeun Seo,Hancheol Park,Shinkook Choi,Hyungshin Kim,Sungsu Lim机构：2Chungnam National University备注：MLSys Workshop on On-Device Intelligence, 2023摘要：虚拟人已经在许多行业中获得了相当大的关注，娱乐和电子商务。作为一项核心技术，从目标语音和面部身份合成逼真的人脸帧已经被生成对抗网络积极研究。尽管现代说话脸生成模型的显着成果，他们往往需要高的计算负担，这限制了他们的有效部署。本研究的目的是开发一个轻量级的语音驱动的说话人脸合成模型。我们建立了一个紧凑的发电机，通过去除残留的块，减少通道宽度从Wav 2Lip，一个流行的说话的脸发生器。我们还提出了一个知识蒸馏方案，以稳定而有效地训练小容量发电机，而无需对抗性学习。我们将参数和MAC的数量减少了28$ times$，同时保留了原始模型的性能。此外，为了减轻将整个生成器转换为INT 8精度时的严重性能下降，我们采用了选择性量化方法，该方法使用FP 16用于量化敏感层，并使用INT 8用于其他层。使用这种混合精度，我们在边缘GPU上实现了高达19$ times$的加速，而不会明显影响生成质量。摘要：Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28$ times$ while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19$ times$ speedup on edge GPUs without noticeably compromising the generation quality.

【9】 Lego-Features: Exporting modular encoder features for streaming and deliberation ASR标题：乐高-功能：导出模块化编码器功能以用于流媒体和审议ASR链接：http://arxiv.org/pdf/2304.00173v1

作者：Rami Botros,Rohit Prabhavalkar,Johan Schalkwyk,Ciprian Chelba,Tara N. Sainath,Françoise Beaufays机构：Google LLC, USA摘要：在端到端（E2 E）语音识别模型中，编码器和解码器之间不可避免地出现代表性的紧耦合。我们建立在最近的工作，已经开始探索建设编码器与模块化编码表示，这样的编码器和解码器，从不同的模型可以拼接在一起，在一个zero-shot的方式，而无需进一步微调。虽然以前的研究只涉及全上下文语音模型，我们也在流媒体环境中探索这个问题。我们的框架建立在现有编码表示的基础上，将它们转换为模块化特征，称为Lego-Features，而无需修改预先训练的模型。当模型用不同的初始化重新训练时，特征保持可互换。虽然稀疏，但我们表明，当使用RNN-T或LAS解码器进行测试时，Lego-Features是强大的，保持了高质量的下游性能。它们也足够丰富，可以代表两遍审议期间的第一遍预测。在这种情况下，它们优于N个最佳假设，因为它们不需要补充声学特征来提供最佳结果。此外，生成乐高特征不需要波束搜索或自回归计算。总的来说，他们提出了一个模块化的，强大的和廉价的替代标准的编码器输出，以及N-最好的假设。摘要：In end-to-end (E2E) speech recognition models, a representational tight-coupling inevitably emerges between the encoder and the decoder. We build upon recent work that has begun to explore building encoders with modular encoded representations, such that encoders and decoders from different models can be stitched together in a zero-shot manner without further fine-tuning. While previous research only addresses full-context speech models, we explore the problem in a streaming setting as well. Our framework builds on top of existing encoded representations, converting them to modular features, dubbed as Lego-Features, without modifying the pre-trained model. The features remain interchangeable when the model is retrained with distinct initializations. Though sparse, we show that the Lego-Features are powerful when tested with RNN-T or LAS decoders, maintaining high-quality downstream performance. They are also rich enough to represent the first-pass prediction during two-pass deliberation. In this scenario, they outperform the N-best hypotheses, since they do not need to be supplemented with acoustic features to deliver the best results. Moreover, generating the Lego-Features does not require beam search or auto-regressive computation. Overall, they present a modular, powerful and cheap alternative to the standard encoder output, as well as the N-best hypotheses.

【10】 Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR标题：实用整形器：优化设备上和云ASR的整形器的大小、速度和故障链接：http://arxiv.org/pdf/2304.00171v1

作者：Rami Botros,Anmol Gulati,Tara N. Sainath,Krzysztof Choromanski,Ruoming Pang,Trevor Strohman,Weiran Wang,Jiahui Yu机构：Google LLC, USA摘要：构象模型保持了大量的内部状态，其中绝大多数与自我注意层相关。在有限的存储器带宽的情况下，在每个推理步骤从存储器读取这些可能会减慢推理。在本文中，我们设计了一个优化的构象，是小到足以满足设备上的限制，并具有快速推理TPU。我们探索了各种想法来提高执行速度，包括用仅卷积块替换较低的构象块，战略性地缩小架构，以及利用RNNAttention Performer。我们的优化构象可以很容易地纳入级联编码器设置，允许第二遍解码器操作其输出，并提高精度，每当更多的资源可用。总而言之，我们发现这些优化可以将延迟降低6.8倍，并在质量上进行合理的权衡。与级联的第二通，我们表明，识别精度是完全恢复。因此，我们提出的编码器可以在设备上作为强大的独立编码器，并作为高性能ASR流水线的第一部分。摘要：Conformer models maintain a large number of internal states, the vast majority of which are associated with self-attention layers. With limited memory bandwidth, reading these from memory at each inference step can slow down inference. In this paper, we design an optimized conformer that is small enough to meet on-device restrictions and has fast inference on TPUs. We explore various ideas to improve the execution speed, including replacing lower conformer blocks with convolution-only blocks, strategically downsizing the architecture, and utilizing an RNNAttention-Performer. Our optimized conformer can be readily incorporated into a cascaded-encoder setting, allowing a second-pass decoder to operate on its output and improve the accuracy whenever more resources are available. Altogether, we find that these optimizations can reduce latency by a factor of 6.8x, and come at a reasonable trade-off in quality. With the cascaded second-pass, we show that the recognition accuracy is completely recoverable. Thus, our proposed encoder can double as a strong standalone encoder in on device, and as the first part of a high-performance ASR pipeline.

【11】Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labels标题：基于时间梯度伪标的无监督分词链接：http://arxiv.org/pdf/2304.00993v1

作者：Tzeviya Sylvia Fuchs,Yedid Hoshen机构：Bar-Ilan University, Ramat-Gan, Israel, The Hebrew University of Jerusalem备注：ICASSP 2023摘要：音频话语中的无监督分词是具有挑战性的，因为在语音中，单词之间通常没有间隙。在一个初步的实验中，我们表明，最近的深度自监督功能是非常有效的分词，但需要监督训练分类头。为了将其有效性扩展到无监督分词，我们提出了一种伪标签策略。我们的方法依赖于这样的观察，即嵌入的时间梯度幅度（即，后续帧的嵌入之间的距离）通常在远离边界处最小并且在靠近边界处更高。我们使用时间梯度幅度上的阈值函数来定义用于措辞的伪标签。我们训练一个线性分类器，将单个帧的嵌入映射到伪标签。最后，我们使用分类器得分来预测帧是单词还是边界。在实证调查中，我们的方法，尽管它的简单性和快速的运行时间，显着优于所有以前的方法在两个数据集。摘要：Unsupervised word segmentation in audio utterances is challenging as, in speech, there is typically no gap between words. In a preliminary experiment, we show that recent deep self-supervised features are very effective for word segmentation but require supervision for training the classification head. To extend their effectiveness to unsupervised word segmentation, we propose a pseudo-labeling strategy. Our approach relies on the observation that the temporal gradient magnitude of the embeddings (i.e. the distance between the embeddings of subsequent frames) is typically minimal far from the boundaries and higher nearer the boundaries. We use a thresholding function on the temporal gradient magnitude to define a psuedo-label for wordness. We train a linear classifier, mapping the embedding of a single frame to the pseudo-label. Finally, we use the classifier score to predict whether a frame is a word or a boundary. In an empirical investigation, our method, despite its simplicity and fast run time, is shown to significantly outperform all previous methods on two datasets.
eess.AS音频处理

【1】Unsupervised Word Segmentation Using Temporal Gradient Pseudo-Labels标题：基于时间梯度伪标的无监督分词链接：http://arxiv.org/pdf/2304.00993v1

*与cs.SD语音【11】为同一篇

作者：Tzeviya Sylvia Fuchs,Yedid Hoshen机构：Bar-Ilan University, Ramat-Gan, Israel, The Hebrew University of Jerusalem备注：ICASSP 2023摘要：音频话语中的无监督分词是具有挑战性的，因为在语音中，单词之间通常没有间隙。在一个初步的实验中，我们表明，最近的深度自监督功能是非常有效的分词，但需要监督训练分类头。为了将其有效性扩展到无监督分词，我们提出了一种伪标签策略。我们的方法依赖于这样的观察，即嵌入的时间梯度幅度（即，后续帧的嵌入之间的距离）通常在远离边界处最小并且在靠近边界处更高。我们使用时间梯度幅度上的阈值函数来定义用于措辞的伪标签。我们训练一个线性分类器，将单个帧的嵌入映射到伪标签。最后，我们使用分类器得分来预测帧是单词还是边界。在实证调查中，我们的方法，尽管它的简单性和快速的运行时间，显着优于所有以前的方法在两个数据集。摘要：Unsupervised word segmentation in audio utterances is challenging as, in speech, there is typically no gap between words. In a preliminary experiment, we show that recent deep self-supervised features are very effective for word segmentation but require supervision for training the classification head. To extend their effectiveness to unsupervised word segmentation, we propose a pseudo-labeling strategy. Our approach relies on the observation that the temporal gradient magnitude of the embeddings (i.e. the distance between the embeddings of subsequent frames) is typically minimal far from the boundaries and higher nearer the boundaries. We use a thresholding function on the temporal gradient magnitude to define a psuedo-label for wordness. We train a linear classifier, mapping the embedding of a single frame to the pseudo-label. Finally, we use the classifier score to predict whether a frame is a word or a boundary. In an empirical investigation, our method, despite its simplicity and fast run time, is shown to significantly outperform all previous methods on two datasets.

【2】 Self-Supervised Learning-Based Source Separation for Meeting Data标题：基于自监督学习的会议数据源分离链接：http://arxiv.org/pdf/2304.00871v1

作者：Yuang Li,Xianrui Zheng,Philip C. Woodland机构：Cambridge University Engineering Dept., Trumpington St., Cambridge, CB,PZ U.K.备注：To appear in Proc. ICASSP2023摘要：None摘要：Source separation can improve automatic speech recognition (ASR) under multi-party meeting scenarios by extracting single-speaker signals from overlapped speech. Despite the success of self-supervised learning models in single-channel source separation, most studies have focused on simulated setups. In this paper, seven SSL models were compared on both simulated and real-world corpora. Then, we propose to integrate the best-performing model WavLM into an automatic transcription system through a novel iterative source selection method. To improve real-world performance, time-domain unsupervised mixture invariant training was adapted to the time-frequency domain. Experiments showed that in the transcription system when source separation was inserted before an ASR model fine-tuned on separated speech, absolute reductions of 1.9% and 1.5% in concatenated minimum-permutation word error rate for an unknown number of speakers (cpWER-us) were observed on the AMI dev and test sets.

【3】Ensemble prosody prediction for expressive speech synthesis标题：表现性语音合成中的集成韵律预测链接：http://arxiv.org/pdf/2304.00714v1

作者：Tian Huey Teh,Vivian Hu,Devang S Ram Mohan,Zack Hodari,Christopher G. R. Wallis,Tomás Gomez Ibarrondo,Alexandra Torresquintero,James Leoni,Mark Gales,Simon King机构：Tom´as G´omez Ibarrondo†, † Papercup Technologies Ltd., United Kingdom, ‡ University of Cambridge, United Kingdom, § University of Edinburgh, United Kingdom备注：ICASSP 2023摘要：生成具有丰富多样韵律的表达性语音仍然是文本到语音的挑战。大多数努力都集中在旨在更好地建模数据分布的复杂神经架构上。然而，在评估中，通常发现，没有一个单一的模型是首选的所有输入文本。这表明了一种以前很少用于文本到语音的方法：模特儿的集合我们将集成学习应用于韵律预测。我们构造简单的合奏韵律预测通过改变模型结构或模型参数值。为了在执行文本到语音转换时自动选择合奏中的模型，我们提出了一种新的，计算上微不足道的，基于方差的标准。我们证明，即使是一个小的合奏韵律预测产生有用的多样性，其中，结合建议的选择标准，优于任何个人的模型从合奏。摘要：Generating expressive speech with rich and varied prosody continues to be a challenge for Text-to-Speech. Most efforts have focused on sophisticated neural architectures intended to better model the data distribution. Yet, in evaluations it is generally found that no single model is preferred for all input texts. This suggests an approach that has rarely been used before for Text-to-Speech: an ensemble of models. We apply ensemble learning to prosody prediction. We construct simple ensembles of prosody predictors by varying either model architecture or model parameter values. To automatically select amongst the models in the ensemble when performing Text-to-Speech, we propose a novel, and computationally trivial, variance-based criterion. We demonstrate that even a small ensemble of prosody predictors yields useful diversity, which, combined with the proposed selection criterion, outperforms any individual model from the ensemble.

【4】 Improving Meeting Inclusiveness using Speech Interruption Analysis标题：利用语音中断分析提高会议包容性

链接：http://arxiv.org/pdf/2304.00658v1

作者：Szu-Wei Fu,Yaran Fan,Yasaman Hosseinkashi,Jayant Gupchup,Ross Cutler机构：Microsoft, Redmond, WA, USA摘要：会议是所有类型公司和组织中普遍存在的沟通方式，自COVID-19大流行以来，使用远程协作系统进行会议的情况急剧增加。然而，并非所有会议都是包容性的，特别是在与会者的参与率方面。在微软最近进行的一项大规模调查中，会议参与者提出的提高包容性的最佳建议是提高远程参与者在会议期间中断和获得发言权的能力。我们表明，使用虚拟举手（VRH）功能，可以导致在微软的预测会议包容性的增加。一个挑战是VRH在所有会议中的使用率不到1%。为了推动采用其使用，以提高包容性（和参与），我们提出了一个基于机器学习的系统，预测当会议参与者试图获得发言权，但未能中断（称为“失败的中断”）。该预测可以用于轻推用户在会议内举起他们的虚拟手。我们相信这是第一个失败的语音中断检测器，并且在现实测试集上的性能具有0.95的曲线下面积（AUC），其中真阳性率（TPR）为50%，假阳性率（FPR）〈1%。据我们所知，这也是远程会议的中断类别（包括失败的中断类别）的第一个数据集。最后，我们相信这是第一个这样的系统，旨在提高会议的包容性，通过语音中断分析和积极干预。摘要：Meetings are a pervasive method of communication within all types of companies and organizations, and using remote collaboration systems to conduct meetings has increased dramatically since the COVID-19 pandemic. However, not all meetings are inclusive, especially in terms of the participation rates among attendees. In a recent large-scale survey conducted at Microsoft, the top suggestion given by meeting participants for improving inclusiveness is to improve the ability of remote participants to interrupt and acquire the floor during meetings. We show that the use of the virtual raise hand (VRH) feature can lead to an increase in predicted meeting inclusiveness at Microsoft. One challenge is that VRH is used in less than 1% of all meetings. In order to drive adoption of its usage to improve inclusiveness (and participation), we present a machine learning-based system that predicts when a meeting participant attempts to obtain the floor, but fails to interrupt (termed a failed interruption'). This prediction can be used to nudge the user to raise their virtual hand within the meeting. We believe this is the first failed speech interruption detector, and the performance on a realistic test set has an area under curve (AUC) of 0.95 with a true positive rate (TPR) of 50% at a false positive rate (FPR) of <1%. To our knowledge, this is also the first dataset of interruption categories (including the failed interruption category) for remote meetings. Finally, we believe this is the first such system designed to improve meeting inclusiveness through speech interruption analysis and active intervention.

【5】 From Smart to Intelligent Utility Meters in Natural Gas Distribution Networks标题：天然气管网从智能电表到智能电表链接：http://arxiv.org/pdf/2304.00251v1

作者：Ghaith Matalkah,Edward J. Coyle机构：School of Electrical and, Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA摘要：我们提出了一种新的方法，用于监测气体分配网络（GDN），使用智能传感器节点，可以与现有的智能燃气表（智能仪表）。该方法旨在实时检测和定位GDN中的气体泄漏。智能仪表利用现有智能仪表中的无线连接来协作实现该方法，该方法包括主动声学探测阶段和被动线性成像阶段。在主动声学阶段中，智能仪表使用新颖的声学脉冲反射计技术协作地发现被监测的管道网络的拓扑结构。在无源线性成像阶段，智能仪表使用它们对管道网络拓扑结构的了解，使用时间曝光声学（TEA）算法来创建管道的线性图像。所得到的图像揭示了网络中活性气体泄漏的存在和位置。我们提出了该方法的理论基础，并显示在实验室收集的实验数据实施它的结果。摘要：We propose a novel method for monitoring gas distribution networks (GDNs) using intelligent sensor nodes that can be integrated with existing smart gas meters (intelligent meters). The method aims at detecting and locating gas leaks in GDNs in real time. The intelligent meters leverage wireless connectivity in existing smart meters to collaborate in implementing this method, which comprises an active acoustic probing phase and a passive linear imaging phase. In the active acoustic phase, the intelligent meters collaboratively discover the topology of the monitored pipeline network using a novel acoustic pulse reflectometry technique. In the passive linear imaging phase, the intelligent meters use their knowledge of the pipeline network topology to create a linear image of the pipeline using the Time-Exposure Acoustic (TEA) algorithm. The resulting image reveals the presence and locations of active gas leaks in the network. We present the theoretical basis of the method and show results of implementing it on experimental data collected in the lab.

【6】 LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading标题：LIPSFUS：用于唇读视听感觉融合的神经形态数据集链接：http://arxiv.org/pdf/2304.01080v1

*与cs.SD语音【1】为同一篇

作者：Antonio Rios-Navarro,Enrique Piñero-Fuentes,Salvador Canas-Moreno,Aqib Javed,Jin Harkin,Alejandro Linares-Barranco机构：University of Seville备注：Submitted to ISCAS2023, 4 pages, plus references, github link provided摘要：本文提出了一种感觉融合神经形态数据集收集精确的时间同步使用一组地址事件表示传感器和工具。目标应用程序是不同机器学习应用程序的几个关键字的唇读，例如数字，机器人命令和辅助丰富的语音短单词。该数据集被放大的尖峰版本的视听唇读数据集收集基于帧的相机。LIPSFUS是公开可用的，并且已经通过用于音频和视觉分类的深度学习架构进行了验证。它旨在用于基于人工和尖峰神经网络算法的感觉融合架构。摘要：This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.

【7】 The Music Annotation Pattern标题：音乐的注解模式链接：http://arxiv.org/pdf/2304.00988v1

*与cs.SD语音【2】为同一篇

作者：Jacopo de Berardinis,Albert Meroño-Peñuela,Andrea Poltronieri,Valentina Presutti机构：King’s College London, University of Bologna备注：12 pages, 3 figures. Proceedings of the 13th Workshop on Ontology Design and Patterns, edited by V. Sv'atek et al., WOP, 2022摘要：音乐内容的诠释是一个复杂的表现过程，这是由于音乐内容本身所固有的多面性、主观性和跨学科性。在过去的几十年中，已经开发了许多用于注释音乐的系统和约定作为独立的标准。很少有人做，使他们的互操作性，这危及跨语料库的研究，因为它需要用户熟悉大量的公约。这些系统中的大多数缺乏表示音乐语言的复杂性所需的语义表达能力，并且不能对源自音频和符号源的多模态注释进行建模。在这篇文章中，我们介绍了音乐注释模式，本体设计模式（ODP）同质化不同的注释系统，并表示几种类型的音乐对象（如和弦，模式，结构）。该ODP在不同级别和时间粒度上保留对象内容的语义。此外，我们的ODP占多模态前期，描述来自不同来源的注释，它是第一个实现大规模集成音乐数据集。摘要：The annotation of music content is a complex process to represent due to its inherent multifaceted, subjectivity, and interdisciplinary nature. Numerous systems and conventions for annotating music have been developed as independent standards over the past decades. Little has been done to make them interoperable, which jeopardises cross-corpora studies as it requires users to familiarise with a multitude of conventions. Most of these systems lack the semantic expressiveness needed to represent the complexity of the musical language and cannot model multi-modal annotations originating from audio and symbolic sources. In this article, we introduce the Music Annotation Pattern, an Ontology Design Pattern (ODP) to homogenise different annotation systems and to represent several types of musical objects (e.g. chords, patterns, structures). This ODP preserves the semantics of the object's content at different levels and temporal granularity. Moreover, our ODP accounts for multi-modality upfront, to describe annotations derived from different sources, and it is the first to enable the integration of music datasets at a large scale.

【8】 The Music Note Ontology标题：音符本体论链接：http://arxiv.org/pdf/2304.00986v1

*与cs.SD语音【3】为同一篇

作者：Andrea Poltronieri,Aldo Gangemi机构：Gangemi,[,−,−,−,], LILEC, University of Bologna, Bologna, Italy, FICLIT, University of Bologna, Bologna, Italy备注：12 pages, 1 figure, 1 table. Proceedings of the 12th Workshop on Ontology Design and Patterns (WOP 2021), Online, edited by K. Hammar et al., 2021摘要：在本文中，我们提出了音符本体论，一个本体论建模音符及其实现。本体论解决了符号表示系统中表示的音符与其实现（即音乐表演）之间的关系。因此，这项工作的目的是解决建模和表示问题时出现的分析抽象的符号特征和相应的物理特征的音频信号之间的关系。本体是由三个不同的本体设计模式（ODP），模型的结构的分数（分数部分模式），注意的符号符号表示法（音符模式）和其实现（音乐对象模式）。摘要：In this paper we propose the Music Note Ontology, an ontology for modelling music notes and their realisation. The ontology addresses the relation between a note represented in a symbolic representation system, and its realisation, i.e. a musical performance. This work therefore aims to solve the modelling and representation issues that arise when analysing the relationships between abstract symbolic features and the corresponding physical features of an audio signal. The ontology is composed of three different Ontology Design Patterns (ODP), which model the structure of the score (Score Part Pattern), the note in the symbolic notation (Music Note Pattern) and its realisation (Musical Object Pattern).

【9】 Artificial Dendritic Computation: The case for dendrites in neuromorphic circuits标题：人工树突计算：神经形态回路中的树突链接：http://arxiv.org/pdf/2304.00951v1

作者：Daniel John Mannion,Anthony John Kenyon机构：Kenyon 1 1Department of Electronic & Electrical Engineering, University College London (UCL)备注：11 pages, 3 figures摘要：生物启发计算专注于神经元和突触，并取得了巨大成功。然而，树突之间的连接也起着重要的作用。在本文中，我们调查的动机复制树突状计算，并提出了一个框架，以指导未来的尝试，在他们的建设。该框架确定的树突的关键属性，并提出了一个例子的树突计算的任务中的声音定位。我们评估了树突对BiLSTM神经网络性能的影响，发现树突预处理降低了阈值性能所需的网络大小。摘要：Bio-inspired computing has focused on neuron and synapses with great success. However, the connections between these, the dendrites, also play an important role. In this paper, we investigate the motivation for replicating dendritic computation and present a framework to guide future attempts in their construction. The framework identifies key properties of the dendrites and presents and example of dendritic computation in the task of sound localisation. We evaluate the impact of dendrites on an BiLSTM neural network's performance, finding that dendrite pre-processing reduce the size of network required for a threshold performance.

【10】 Designing and Evaluating Speech Emotion Recognition Systems: A reality check case study with IEMOCAP标题：语音情感识别系统的设计与评估：IEMOCAP的现实检验案例研究链接：http://arxiv.org/pdf/2304.00860v1

*与cs.SD语音【4】为同一篇

作者：Nikolaos Antoniou,Athanasios Katsamanis,Theodoros Giannakopoulos,Shrikanth Narayanan机构：† Behavioral Signal Technologies, Los Angeles, CA, USA, ∗ Institute for Language and Speech Processing, Athena Research Center, Athens, Greece, + SAIL-University of Southern California, Los Angeles, CA, USA备注：Accepted at ICASSP 2023摘要：有一个迫切需要的指导方针和标准的测试集，以允许直接和公平的比较语音情感识别（SER）。虽然资源，如交互式情感二元运动捕捉（IEMOCAP）数据库，已成为广泛采用的参考语料库的研究人员开发和测试模型SER，发表的工作揭示了广泛的假设和多样性，在其使用的挑战再现性和泛化。基于对使用IEMOCAP作为用例的SER的最新进展的批判性审查，我们的工作旨在做出两个贡献：首先，使用最近的文献分析，包括假设和其中使用的指标，我们提供了一套SER评估准则。其次，使用最近的出版物与开源的实现，我们专注于在SER的再现性评估。摘要：There is an imminent need for guidelines and standard test sets to allow direct and fair comparisons of speech emotion recognition (SER). While resources, such as the Interactive Emotional Dyadic Motion Capture (IEMOCAP) database, have emerged as widely-adopted reference corpora for researchers to develop and test models for SER, published work reveals a wide range of assumptions and variety in its use that challenge reproducibility and generalization. Based on a critical review of the latest advances in SER using IEMOCAP as the use case, our work aims at two contributions: First, using an analysis of the recent literature, including assumptions made and metrics used therein, we provide a set of SER evaluation guidelines. Second, using recent publications with open-sourced implementations, we focus on reproducibility assessment in SER.

【11】 AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models标题：审计：遵循潜在扩散模型的说明进行音频编辑链接：http://arxiv.org/pdf/2304.00830v1

*与cs.SD语音【5】为同一篇

作者：Yuancheng Wang,Zeqian Ju,Xu Tan,Lei He,Zhizheng Wu,Jiang Bian,Sheng Zhao机构：Microsoft,The Chinese University of Hong Kong, Shenzhen摘要：音频编辑适用于各种目的，例如添加背景声音效果、替换乐器以及修复损坏的音频。最近，一些基于扩散的方法通过使用以输出音频的文本描述为条件的扩散和去噪过程来实现zero-shot音频编辑。然而，这些方法仍然存在一些问题：1）没有经过编辑工作的培训，不能保证良好的编辑效果;2）它们可能错误地修改不需要编辑的音频段;3）它们需要输出音频的完整描述，这在实际场景中并不总是可用或必要的。在这项工作中，我们提出了AUDIT，一个指令指导的音频编辑模型的基础上潜在的扩散模型。具体来说，AUDIT有三个主要设计特点：1）我们为不同的音频编辑任务构建三元组训练数据（指令、输入音频、输出音频），并使用指令和输入（待编辑）音频作为条件来训练扩散模型，并生成输出（编辑）音频;2）它可以通过比较输入和输出音频之间的差异来自动学习仅修改需要编辑的片段;3）它只需要编辑指令而不是完整的目标音频描述作为文本输入。AUDIT在针对若干音频编辑任务的客观和主观度量两者中实现了最先进的结果（例如，添加、删除、替换、修补、超分辨率）。演示示例可在www.example.com上获得https: audit-demo.github.io 。摘要：Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https: audit-demo.github.io .

【12】 Musical creativity enabled by nonlinear oscillations of a bubble in water标题：水中气泡的非线性振荡带来的音乐创造力链接：http://arxiv.org/pdf/2304.00822v1

*与cs.SD语音【6】为同一篇

作者：Ivan S. Maksymov机构：Artificial Intelligence and Cyber Futures Institute, Charles Sturt University, Bathurst, NSW , Australia摘要：制作原创和安排现有的音乐成果是一门需要多年学习和实践才能掌握的艺术。然而，尽管人工智能驱动的音乐创造力领域不断取得进展，但制作高质量的音乐作品仍然是人类的特权。在这里，我们证明了一个单一的气泡在水中可以用来产生创造性的音乐成果，当它的非线性振荡下的声压信号，编码一段古典音乐。气泡的响应的音频信号类似于原始作曲的电吉他版本。我们建议，并提供合理的理论支持论据，泡沫的这种特性可以用来创建物理启发的人工智能系统，能够模拟人类在音乐编排和作曲方面的创造力。摘要：Producing original and arranging existing musical outcomes is an art that takes years of learning and practice to master. Yet, despite the constant advances in the field of AI-powered musical creativity, production of quality musical outcomes remains a prerogative of the humans. Here we demonstrate that a single bubble in water can be used to produce creative musical outcomes, when it nonlinearly oscillates under an acoustic pressure signal that encodes a piece of classical music. The audio signal of the response of the bubble resembles an electric guitar version of the original composition. We suggest, and provide plausible theoretical supporting arguments, that this property of the bubble can be used to create physics-inspired AI systems capable of simulating human creativity in arrangement and composition of music.

【13】 Multilingual Word Error Rate Estimation: e-WER3标题：多语种错误率估计：E-WER3链接：http://arxiv.org/pdf/2304.00649v1

*与cs.SD语音【7】为同一篇

作者：Shammur Absar Chowdhury,Ahmed Ali机构：Qatar Computing Research Institute, HBKU, Doha, Qatar备注：Accepted in ICASSP, Multilingual WER estimation, End-to-End systems, multilingual model, automatic word error rate estimation摘要：多语种自动语音识别系统的成功为许多语音驱动的应用程序提供了支持。然而，测量这样的系统的性能仍然是一个重大的挑战，由于其依赖于手动转录的语音数据在单语言和多语言的情况。在本文中，我们提出了一种新的多语言框架-eWER 3-联合训练的声学和词汇表示，以估计单词错误率。我们证明了eWER 3的有效性（i）预测WER，而不使用任何内部状态的ASR和（ii）使用多语言共享的潜在空间，以推动密切相关的语言的性能。我们表明，我们提出的多语言模型优于以前的单语单词错误率估计方法（eWER 2）的绝对9 %的皮尔逊相关系数（PCC）的增加，更好的整体估计之间的预测和参考WER。摘要：The success of the multilingual automatic speech recognition systems empowered many voice-driven applications. However, measuring the performance of such systems remains a major challenge, due to its dependency on manually transcribed speech data in both mono- and multilingual scenarios. In this paper, we propose a novel multilingual framework -- eWER3 -- jointly trained on acoustic and lexical representation to estimate word error rate. We demonstrate the effectiveness of eWER3 to (i) predict WER without using any internal states from the ASR and (ii) use the multilingual shared latent space to push the performance of the close-related languages. We show our proposed multilingual model outperforms the previous monolingual word error rate estimation method (eWER2) by an absolute 9 % increase in Pearson correlation coefficient (PCC), with better overall estimation between the predicted and reference WER.

【14】 A Unified Compression Framework for Efficient Speech-Driven Talking-Face Generation标题：一种用于高效语音驱动的语音驱动的统一压缩框架链接：http://arxiv.org/pdf/2304.00471v1

*与cs.SD语音【8】为同一篇

作者：Bo-Kyeong Kim,Jaemin Kang,Daeun Seo,Hancheol Park,Shinkook Choi,Hyungshin Kim,Sungsu Lim机构：2Chungnam National University备注：MLSys Workshop on On-Device Intelligence, 2023摘要：虚拟人已经在许多行业中获得了相当大的关注，娱乐和电子商务。作为一项核心技术，从目标语音和面部身份合成逼真的人脸帧已经被生成对抗网络积极研究。尽管现代说话脸生成模型的显着成果，他们往往需要高的计算负担，这限制了他们的有效部署。本研究的目的是开发一个轻量级的语音驱动的说话人脸合成模型。我们建立了一个紧凑的发电机，通过去除残留的块，减少通道宽度从Wav 2Lip，一个流行的说话的脸发生器。我们还提出了一个知识蒸馏方案，以稳定而有效地训练小容量发电机，而无需对抗性学习。我们将参数和MAC的数量减少了28$ times$，同时保留了原始模型的性能。此外，为了减轻将整个生成器转换为INT 8精度时的严重性能下降，我们采用了选择性量化方法，该方法使用FP 16用于量化敏感层，并使用INT 8用于其他层。使用这种混合精度，我们在边缘GPU上实现了高达19$ times$的加速，而不会明显影响生成质量。摘要：Virtual humans have gained considerable attention in numerous industries, e.g., entertainment and e-commerce. As a core technology, synthesizing photorealistic face frames from target speech and facial identity has been actively studied with generative adversarial networks. Despite remarkable results of modern talking-face generation models, they often entail high computational burdens, which limit their efficient deployment. This study aims to develop a lightweight model for speech-driven talking-face synthesis. We build a compact generator by removing the residual blocks and reducing the channel width from Wav2Lip, a popular talking-face generator. We also present a knowledge distillation scheme to stably yet effectively train the small-capacity generator without adversarial learning. We reduce the number of parameters and MACs by 28$ times$ while retaining the performance of the original model. Moreover, to alleviate a severe performance drop when converting the whole generator to INT8 precision, we adopt a selective quantization method that uses FP16 for the quantization-sensitive layers and INT8 for the other layers. Using this mixed precision, we achieve up to a 19$ times$ speedup on edge GPUs without noticeably compromising the generation quality.

【15】 Lego-Features: Exporting modular encoder features for streaming and deliberation ASR标题：乐高-功能：导出模块化编码器功能以用于流媒体和审议ASR链接：http://arxiv.org/pdf/2304.00173v1

*与cs.SD语音【9】为同一篇

作者：Rami Botros,Rohit Prabhavalkar,Johan Schalkwyk,Ciprian Chelba,Tara N. Sainath,Françoise Beaufays机构：Google LLC, USA摘要：在端到端（E2 E）语音识别模型中，编码器和解码器之间不可避免地出现代表性的紧耦合。我们建立在最近的工作，已经开始探索建设编码器与模块化编码表示，这样的编码器和解码器，从不同的模型可以拼接在一起，在一个zero-shot的方式，而无需进一步微调。虽然以前的研究只涉及全上下文语音模型，我们也在流媒体环境中探索这个问题。我们的框架建立在现有编码表示的基础上，将它们转换为模块化特征，称为Lego-Features，而无需修改预先训练的模型。当模型用不同的初始化重新训练时，特征保持可互换。虽然稀疏，但我们表明，当使用RNN-T或LAS解码器进行测试时，Lego-Features是强大的，保持了高质量的下游性能。它们也足够丰富，可以代表两遍审议期间的第一遍预测。在这种情况下，它们优于N个最佳假设，因为它们不需要补充声学特征来提供最佳结果。此外，生成乐高特征不需要波束搜索或自回归计算。总的来说，他们提出了一个模块化的，强大的和廉价的替代标准的编码器输出，以及N-最好的假设。摘要：In end-to-end (E2E) speech recognition models, a representational tight-coupling inevitably emerges between the encoder and the decoder. We build upon recent work that has begun to explore building encoders with modular encoded representations, such that encoders and decoders from different models can be stitched together in a zero-shot manner without further fine-tuning. While previous research only addresses full-context speech models, we explore the problem in a streaming setting as well. Our framework builds on top of existing encoded representations, converting them to modular features, dubbed as Lego-Features, without modifying the pre-trained model. The features remain interchangeable when the model is retrained with distinct initializations. Though sparse, we show that the Lego-Features are powerful when tested with RNN-T or LAS decoders, maintaining high-quality downstream performance. They are also rich enough to represent the first-pass prediction during two-pass deliberation. In this scenario, they outperform the N-best hypotheses, since they do not need to be supplemented with acoustic features to deliver the best results. Moreover, generating the Lego-Features does not require beam search or auto-regressive computation. Overall, they present a modular, powerful and cheap alternative to the standard encoder output, as well as the N-best hypotheses.

【16】 Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR标题：实用整形器：优化设备上和云ASR的整形器的大小、速度和故障链接：http://arxiv.org/pdf/2304.00171v1

*与cs.SD语音【10】为同一篇

作者：Rami Botros,Anmol Gulati,Tara N. Sainath,Krzysztof Choromanski,Ruoming Pang,Trevor Strohman,Weiran Wang,Jiahui Yu机构：Google LLC, USA摘要：构象模型保持了大量的内部状态，其中绝大多数与自我注意层相关。在有限的存储器带宽的情况下，在每个推理步骤从存储器读取这些可能会减慢推理。在本文中，我们设计了一个优化的构象，是小到足以满足设备上的限制，并具有快速推理TPU。我们探索了各种想法来提高执行速度，包括用仅卷积块替换较低的构象块，战略性地缩小架构，以及利用RNNAttention Performer。我们的优化构象可以很容易地纳入级联编码器设置，允许第二遍解码器操作其输出，并提高精度，每当更多的资源可用。总而言之，我们发现这些优化可以将延迟降低6.8倍，并在质量上进行合理的权衡。与级联的第二通，我们表明，识别精度是完全恢复。因此，我们提出的编码器可以在设备上作为强大的独立编码器，并作为高性能ASR流水线的第一部分。摘要：Conformer models maintain a large number of internal states, the vast majority of which are associated with self-attention layers. With limited memory bandwidth, reading these from memory at each inference step can slow down inference. In this paper, we design an optimized conformer that is small enough to meet on-device restrictions and has fast inference on TPUs. We explore various ideas to improve the execution speed, including replacing lower conformer blocks with convolution-only blocks, strategically downsizing the architecture, and utilizing an RNNAttention-Performer. Our optimized conformer can be readily incorporated into a cascaded-encoder setting, allowing a second-pass decoder to operate on its output and improve the accuracy whenever more resources are available. Altogether, we find that these optimizations can reduce latency by a factor of 6.8x, and come at a reasonable trade-off in quality. With the cascaded second-pass, we show that the recognition accuracy is completely recoverable. Thus, our proposed encoder can double as a strong standalone encoder in on device, and as the first part of a high-performance ASR pipeline.

机器翻译，仅供参考