【1】 Development of Hybrid ASR Systems for Low Resource Medical Domain Conversational Telephone Speech
标题：低资源医疗领域通话语音混合ASR系统的研制
链接：https://arxiv.org/abs/2210.13397
作者：Christoph Lüscher,Mohammad Zeineldeen,Zijian Yang,Peter Vieting,Khai Le-Duc,Weiyue Wang,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition Group, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：ASR System Paper for HYKIST project摘要：近年来，ASR系统在有足够数量的训练数据可用的特定任务（例如LibriSpeech）上已经达到了显著的性能。然而，变化的声学和记录条件和说话风格以及缺乏足够的域内训练数据仍然对准确模型的开发提出了挑战。在这项工作中，我们介绍了我们为医疗领域中三种语言（阿拉伯语、德语、越南语）的会话式电话语音翻译任务开发ASR系统的努力，以支持医生和患者之间跨越语言障碍的急诊室交互。我们研究了不同的训练计划和数据组合方法，以提高系统的性能，并分析在哪里最有效地使用有限的可用数据。 摘要：In recent years, ASR systems have reached remarkable performance on specific tasks for which sufficient amounts of training data are available, like e.g. LibriSpeech. However, varying acoustic and recording conditions and speaking styles and a lack of sufficient in-domain training data still pose challenges to the development of accurate models. In this work, we present our efforts for the development of ASR systems for a conversational telephone speech translation task in the medical domain for three languages (Arabic, German, Vietnamese) to support emergency room interaction between physician and patient across language barriers. We study different training schedules and data combination approaches in order to improve the system's performance, as well as analyze where limited available data is used most efficiently.
【2】 ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition
标题：ESB：一种多域端到端语音识别基准
链接：https://arxiv.org/abs/2210.13352
作者：Sanchit Gandhi,Patrick von Platen,Alexander M. Rush备注：25 pages, 1 figure, submitted to ICLR 2023摘要：语音识别应用覆盖了一系列不同的音频和文本分布，具有不同的说话风格、背景噪声、转录标点符号和字符大小写。然而，许多语音识别系统需要数据集特定的调整（音频过滤、标点符号去除和大小写的归一化），因此假定音频和文本分布的先验知识。这种调优要求可能导致系统无法推广到其他数据集和域。为了促进多领域语音识别系统的发展，引入了端到端语音基准测试（ESB），用于评估单个自动语音识别（ASR）系统在大量语音数据集上的性能。基准测试系统必须在数据集之间使用相同的数据预处理和后处理算法-假设音频和文本数据分布是先验未知的。我们在此基准测试上比较了一系列最先进的（SoTA）端到端（E2 E）系统，展示了如何在广泛的数据分布上应用和评估单个语音系统。我们发现E2 E系统在数据集上都是有效的：在一个公平的比较中，E2 E系统实现了在SoTA系统的2.6%之内调谐到一个特定的数据集。我们的分析表明，转录伪像，如标点符号和大小写，造成困难的ASR系统，并应纳入评估。我们相信在一系列数据集上的E2 E基准测试促进了多领域语音识别系统的研究。ESB可从www.example.com获得https://huggingface.co/esb。摘要：Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalisation of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalise to other datasets and domains. To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data pre- and post-processing algorithm across datasets - assuming the audio and text data distributions are a-priori unknown. We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speech system can be applied and evaluated on a wide range of data distributions. We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation. We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems. ESB is available at https://huggingface.co/esb.
【3】 Proficiency assessment of L2 spoken English using wav2vec 2.0
标题：用Wave2vec 2.0评估二语口语水平
链接：https://arxiv.org/abs/2210.13168
作者：Stefano Bannò,Marco Matassoni机构：Fondazione Bruno Kessler, Trento, Italy, University of Trento, Trento, Italy摘要：作为第二语言的英语学习需求的增长导致了对自动评估口语能力的方法的兴趣的增长。大多数方法使用手工制作的特征，但它们的有效性依赖于它们特定的潜在假设，并且它们有可能丢弃关于熟练程度的潜在显著信息。其它方法依赖于ASR系统产生的转录，其可能不提供学习者话语在特定场景中的忠实再现（例如，非母语儿童的自发言语）。此外，转录不产生任何关于相关方面的信息，例如语调、节奏或韵律。在本文中，我们研究了使用wav2vec 2.0评估两个小数据集（其中一个是公开的）的总体和个人方面的熟练程度。我们发现，这种方法明显优于基于BERT的基线系统，该基线系统在ASR和用于比较的手动转录上训练。摘要：The increasing demand for learning English as a second language has led to a growing interest in methods for automatically assessing spoken language proficiency. Most approaches use hand-crafted features, but their efficacy relies on their particular underlying assumptions and they risk discarding potentially salient information about proficiency. Other approaches rely on transcriptions produced by ASR systems which may not provide a faithful rendition of a learner's utterance in specific scenarios (e.g., non-native children's spontaneous speech). Furthermore, transcriptions do not yield any information about relevant aspects such as intonation, rhythm or prosody. In this paper, we investigate the use of wav2vec 2.0 for assessing overall and individual aspects of proficiency on two small datasets, one of which is publicly available. We find that this approach significantly outperforms the BERT-based baseline system trained on ASR and manual transcriptions used for comparison.
【4】 10 hours data is all you need
标题：10小时的数据是您所需要的全部
链接：https://arxiv.org/abs/2210.13067
作者：Zeping Min,Qian Ge,Zhong Li机构：⋆ Peking University, † Microsoft Research Asia摘要：本文提出了一种新的伪普通话语音数据的生成方法CAMP（character audio mixed up），其目的是从字符尺度生成音频。本文还提出了一种建立适应CAMP的普通话文字尺度音频数据库的方法--META-AUDIO，该方法充分利用了音频数据，极大地提高了数据库的多样性。实验结果表明，该方法简单有效。例如，我们使用AISHELL-1中的10小时音频数据和CAMP生成的伪音频数据训练模型，并实现了具有竞争力的11.07个字符的错误率（CER）。此外，我们还使用AIDATATANG数据集中仅10小时的音频数据和CAMP生成的伪音频数据进行了训练，再次获得了具有竞争力的8.26 CER。摘要：We propose a novel procedure to generate pseudo mandarin speech data named as CAMP (character audio mix up), which aims at generating audio from a character scale. We also raise a method for building a mandarin character scale audio database adaptive to CAMP named as META-AUDIO, which makes full use of audio data and can greatly increase the data diversity of the database. Experiments show that our CAMP method is simple and quite effective. For example, we train models with 10 hours of audio data in AISHELL-1 and pseudo audio data generated by CAMP, and achieve a competitive 11.07 character error rate (CER). Besides, we also perform training with only 10 hours of audio data in AIDATATANG dataset and pseudo audio data generated by CAMP, which again achieves a competitive 8.26 CER.
【5】 Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing
标题：预先训练的语音编码器的自我监督重新布线：在语音处理中以更少的标签实现更快的微调
链接：https://arxiv.org/abs/2210.13030
作者：Hao Yang,Jinming Zhao,Gholamreza Haffari,Ehsan Shareghi机构：Department of Data Science & AI, Monash University摘要：预先训练的语音变换器已经促进了各种语音处理任务的巨大成功。然而，为下游任务微调这些编码器需要足够大的训练数据来收敛或实现现有技术。在文本域中，这部分归因于预训练的变换器中的表示空间的次优性。在这项工作中，我们冷静地看待预先训练的语音编码器，并重新布线他们的表示空间，而不需要任何任务特定的标签。我们的方法利用音频输入的中性合成版本以及帧掩蔽来构造用于对比自监督学习的正对。当用于扩充wav2vec 2编码器时，我们观察到表示空间中的各向同性的一致改进。我们在6个语音处理任务上的实验表明，在任务微调过程中，收敛速度显著加快，并且任务性能持续改善，特别是在低资源环境下。摘要：Pre-trained speech Transformers have facilitated great success across various speech processing tasks. However, fine-tuning these encoders for downstream tasks require sufficiently large training data to converge or to achieve state-of-the-art. In text domain this has been partly attributed to sub-optimality of the representation space in pre-trained Transformers. In this work, we take a sober look into pre-trained speech encoders and rewire their representation space without requiring any task-specific labels. Our method utilises neutrally synthesised version of audio inputs along with frame masking to construct positive pairs for contrastive self-supervised learning. When used for augmenting the wav2vec 2 encoder, we observe consistent improvement of isotropy in the representation space. Our experiments on 6 speech processing tasks, exhibit a significant convergence speedup during task fine-tuning as well as consistent task improvement, specially in low-resource settings.
【6】 Investigating the effect of domain selection on automatic speech recognition performance: a case study on Bangladeshi Bangla标题：领域选择对自动语音识别性能影响的研究--以孟加拉国孟加拉为例链接：https://arxiv.org/abs/2210.12921
作者：Ahnaf Mozib Samin,M. Humayan Kobir,Md. Mushtaq Shahriyar Rafee,M. Firoz Ahmed,Shafkat Kibria,M. Shahidur Rahman机构： Department of Computer Science and Engineering, Shahjalal University of Science and TechnologyThe performance of data-driven natural language processing systems is contingent upon the quality of corpora备注：To be submitted摘要：数据驱动的自然语言处理系统的性能取决于语料库的质量。然而，主要的语料库设计标准通常没有被充分地确定和检查，特别是在语音处理学科。语音语料库的开发需要额外关注清晰/嘈杂、朗读/自发、多说话者语音、口音/方言等。域选择也是语音语料库开发中的一个关键决策点。在本研究中，我们通过评估一个最先进的孟加拉语自动语音识别（ASR）模型，在一个新的多领域孟加拉语ASR评估基准- BanSpeech上，证明了领域选择的重要性，该基准包含7.2小时的语音和来自19个不同领域的9802句话。采用深度卷积神经网络（CNN）、层归一化技术和连接主义时态分类（CTC）损失准则，在低资源、形态丰富的孟加拉语SUBAK. KO语料库上训练ASR模型。实验结果表明，SUBAK.KO上的ASR模型在识别以自发语音为主的语音域时存在困难，并且存在大量的词汇表外（OOV）单词。另一方面，相同的ASR模型在读取语音域中执行得更好，并且包含更少的OOV单词。此外，我们还报告了我们对层归一化、输入特征提取、卷积层数等的实验结果，并在SUBAK.KO上设置基线。BanSpeech将公开提供，以满足对孟加拉语ASR具有挑战性的评估基准的需求。摘要：The performance of data-driven natural language processing systems is contingent upon the quality of corpora. However, principal corpus design criteria are often not identified and examined adequately, particularly in the speech processing discipline. Speech corpora development requires additional attention with regard to clean/noisy, read/spontaneous, multi-talker speech, accents/dialects, etc. Domain selection is also a crucial decision point in speech corpus development. In this study, we demonstrate the significance of domain selection by assessing a state-of-the-art Bangla automatic speech recognition (ASR) model on a novel multi-domain Bangladeshi Bangla ASR evaluation benchmark - BanSpeech, which contains 7.2 hours of speech and 9802 utterances from 19 distinct domains. The ASR model has been trained with deep convolutional neural network (CNN), layer normalization technique, and Connectionist Temporal Classification (CTC) loss criterion on SUBAK.KO, a mostly read speech corpus for the low-resource and morphologically rich language Bangla. Experimental evaluation reveals the ASR model on SUBAK.KO faces difficulty recognizing speech from domains with mostly spontaneous speech and has a high number of out-of-vocabulary (OOV) words. The same ASR model, on the other hand, performs better in read speech domains and contains fewer OOV words. In addition, we report the outcomes of our experiments with layer normalization, input feature extraction, number of convolutional layers, etc., and set a baseline on SUBAK.KO. The BanSpeech will be publicly available to meet the need for a challenging evaluation benchmark for Bangla ASR.
【7】 Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings标题：通过听力引导意义：口语句子嵌入的无监督学习链接：https://arxiv.org/abs/2210.12857
作者：Jian Zhu,Zuoyu Tian,Yadong Liu,Cong Zhang,Chia-wen Lo机构：BUniversity of Michigan, Ann Arbor, University of British Columbia, XIndiana University Bloomington, QNewcastle University, MMax Planck Institute for Human Cognitive and Brain Sciences备注：Findings of EMNLP 2022摘要：直接从语音信号中推导出语义表示是一项极具挑战性的任务，但在语音挖掘和口语理解中有许多有用的应用。本研究探讨口语语意表征的无监督学习。通过将语音信号转换为由声学单元发现产生的隐藏单元，提出了WavEmbed，它是一种多模态序列自动编码器，可以从语音的密集表示中预测隐藏单元。其次，本文提出了S-HuBERT算法，该算法首先在隐单元上训练句子嵌入模型，然后通过对比学习将其知识传递给语音编码器，从而通过知识提取来提取语义。表现最好的模型与人类判断达到了中等相关性（0.5~0.6），而不依赖于任何标签或转录。此外，这些模型还可以容易地扩展以利用语音的文本转录来学习与人类注释强相关的好得多的语音嵌入。我们提出的方法适用于开发纯数据驱动的语音挖掘、索引和搜索系统。摘要：Inducing semantic representations directly from speech signals is a highly challenging task but has many useful applications in speech mining and spoken language understanding. This study tackles the unsupervised learning of semantic representations for spoken utterances. Through converting speech signals into hidden units generated from acoustic unit discovery, we propose WavEmbed, a multimodal sequential autoencoder that predicts hidden units from a dense representation of speech. Secondly, we also propose S-HuBERT to induce meaning through knowledge distillation, in which a sentence embedding model is first trained on hidden units and passes its knowledge to a speech encoder through contrastive learning. The best performing model achieves a moderate correlation (0.5~0.6) with human judgments, without relying on any labels or transcriptions. Furthermore, these models can also be easily extended to leverage textual transcriptions of speech to learn much better speech embeddings that are strongly correlated with human annotations. Our proposed methods are applicable to the development of purely data-driven systems for speech mining, indexing and search.
【8】 Clarinet: A Music Retrieval System标题：单簧管：一个音乐检索系统链接：https://arxiv.org/abs/2210.12648
作者：Kshitij Alwadhi,Rohan Sharma,Siddhant Sharma机构：Dept. of Electrical Engineering, Indian Institute of Technology Delhi摘要：本文提出并实现了一种基于MIDI的音乐识别方法。我们的单簧管音乐检索系统是专为搜索钢琴MIDI文件与高召回率和速度。我们设计了一种新颖的旋律提取算法，使召回率提高了10%以上。我们还实现了三个检索算法--两个自行设计的算法（RSA Note和RSA Time），以及一个改进的Mongeau Sankoff算法。本文还讨论了实现速度和尺度不变性的算法。本白皮书还包含使用四种不同指标进行的详细实验和基准测试。单簧管的回忆率超过94%。摘要：A MIDI based approach for music recognition is proposed and implemented in this paper. Our Clarinet music retrieval system is designed to search piano MIDI files with high recall and speed. We design a novel melody extraction algorithm that improves recall results by more than 10%. We also implement 3 algorithms for retrieval-two self designed (RSA Note and RSA Time), and a modified version of the Mongeau Sankoff Algorithm. Algorithms to achieve tempo and scale invariance are also discussed in this paper. The paper also contains detailed experimentation and benchmarks with four different metrics. Clarinet achieves recall scores of more than 94%.
【9】 Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation标题：用于目标说话人分离的登记说话人嵌入被忽略方面的定量证据链接：https://arxiv.org/abs/2210.12635
作者：Xiaoyu Liu,Xu Li,Joan Serrà机构：Joan Serra, Dolby Laboratories摘要：单通道目标说话人分离（TSS）的目的是从多个说话人的混合语音中提取出说话人的语音。一个典型的深度学习TSS框架由一个获得注册说话人嵌入的上游模型和一个根据嵌入执行分离的下游模型组成。本文对语音识别嵌入技术中几个重要但被忽略的方面进行了研究，包括广泛使用的说话人识别嵌入技术的适用性，log-mel滤波器组和自监督嵌入技术的引入，以及嵌入技术的跨数据集泛化能力。实验结果表明，说话人识别嵌入可能会由于次优度量、训练目标或常规预处理而丢失相关信息。相比之下，滤波器组和自监督嵌入都保持了说话人信息的完整性，但在跨数据集评估中，前者始终优于后者。之前被忽略的滤波器组嵌入的竞争性分离和泛化性能在我们的研究中是一致的，这需要进一步研究更好的上游特征。摘要：Single channel target speaker separation (TSS) aims at extracting a speaker's voice from a mixture of multiple talkers given an enrollment utterance of that speaker. A typical deep learning TSS framework consists of an upstream model that obtains enrollment speaker embeddings and a downstream model that performs the separation conditioned on the embeddings. In this paper, we look into several important but overlooked aspects of the enrollment embeddings, including the suitability of the widely used speaker identification embeddings, the introduction of the log-mel filterbank and self-supervised embeddings, and the embeddings' cross-dataset generalization capability. Our results show that the speaker identification embeddings could lose relevant information due to a sub-optimal metric, training objective, or common pre-processing. In contrast, both the filterbank and the self-supervised embeddings preserve the integrity of the speaker information, but the former consistently outperforms the latter in a cross-dataset evaluation. The competitive separation and generalization performance of the previously overlooked filterbank embedding is consistent across our study, which calls for future research on better upstream features.
【10】 GCT: Gated Contextual Transformer for Sequential Audio Tagging标题：GCT：用于顺序音频标注的门控上下文转换器链接：https://arxiv.org/abs/2210.12541
作者：Yuanbo Hou,Yun Wang,Wenwu Wang,Dick Botteldooren机构：WAVES, Ghent University, Belgium, Meta AI, USA, CVSSP, University of Surrey, UK摘要：音频标注是指为音频片段分配预定义的标签，以表示音频事件的类别信息。顺序音频标记（SAT）是指检测音频事件的类别信息以及它们在音频片段中出现的顺序。现有的SAT方法大多基于连接主义时态分类（CTC）。然而，由于不同时刻输出之间的条件独立性假设，CTC无法有效捕获事件之间的关联。上下文转换器（cTransformer）通过利用SAT中的上下文信息来解决这个问题。然而，cTransformer在利用上下文信息方面也受到限制，因为它在推理中仅使用前向信息。本文提出了一种基于前向-后向推理的门控上下文转换器。此外，在GCT中提出了一个门控上下文多层感知器（GCMLP）模块，从结构上改善了cTransformer的性能。在两个真实音频数据集上的实验结果表明，GCMLP与FBI相结合的GCT算法的性能优于基于CTC的算法和cTransformer.为了促进SAT的研究，发布了两个数据集的手动注释顺序标签。摘要：Audio tagging aims to assign predefined tags to audio clips to indicate the class information of audio events. Sequential audio tagging (SAT) means detecting both the class information of audio events, and the order in which they occur within the audio clip. Most existing methods for SAT are based on connectionist temporal classification (CTC). However, CTC cannot effectively capture connections between events due to the conditional independence assumption between outputs at different times. The contextual Transformer (cTransformer) addresses this issue by exploiting contextual information in SAT. Nevertheless, cTransformer is also limited in exploiting contextual information as it only uses forward information in inference. This paper proposes a gated contextual Transformer (GCT) with forward-backward inference (FBI). In addition, a gated contextual multi-layer perceptron (GCMLP) block is proposed in GCT to improve the performance of cTransformer structurally. Experiments on two real-life audio datasets show that the proposed GCT with GCMLP and FBI performs better than the CTC-based methods and cTransformer. To promote research on SAT, the manually annotated sequential labels for the two datasets are released.
【11】 Speech Emotion Recognition via an Attentive Time-Frequency Neural Network标题：基于时频神经网络的语音情感识别链接：https://arxiv.org/abs/2210.12430
作者：Cheng Lu,Wenming Zheng,Hailun Lian,Yuan Zong,Chuangao Tang,Sunan Li,Yan Zhao备注：This paper has been accepted as a regular paper on IEEE Transactions on Computational Social Systems摘要：在语音情感识别中，语音谱图通常被用作深度神经网络的输入特征，用于学习语音信号的高层时频模式。\textcolor{black}{通常，不同的情感在谱图上对应于频带和时间帧内的特定能量激活，这表明对于SER，频域和时域都是表示情感的必要条件。然而，最近基于谱图的工作主要集中在时域中建模长期依赖性，导致这些方法遇到以下两个问题：（1）在时频联合学习过程中，忽略了对情感相关性在频域内的建模;（2）忽略捕捉与情绪相关的特定频带。}针对这一问题，提出了一种基于注意的时频神经网络（ATFNN），包括时频神经网络（TFNN）和时频注意。具体来说，针对第一个问题，我们设计了一个TFNN，其中频域编码器（F-Encoder）基于Transformer编码器，时域编码器（T-Encoder）基于双向长短时记忆（Bi-LSTM）。F-Encoder和T-Encoder分别对频带内和时间帧内的相关性进行建模，并将其嵌入时频联合学习策略中，以获得语音情感的时频模式。此外，为了解决第二个问题，我们还采用了时频注意，即频率-注意网络（F-Attention）和时间-注意网络（T-Attention）来关注与情感相关的频带范围和时间帧范围，从而提高了语音情感特征的区分度。摘要：Spectrogram is commonly used as the input feature of deep neural networks to learn the high(er)-level time-frequency pattern of speech signal for speech emotion recognition (SER). \textcolor{black}{Generally, different emotions correspond to specific energy activations both within frequency bands and time frames on spectrogram, which indicates the frequency and time domains are both essential to represent the emotion for SER. However, recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain, leading to these methods encountering the following two issues: (1) neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning; (2) ignoring to capture the specific frequency bands associated with emotions.} To cope with the issues, we propose an attentive time-frequency neural network (ATFNN) for SER, including a time-frequency neural network (TFNN) and time-frequency attention. Specifically, aiming at the first issue, we design a TFNN with a frequency-domain encoder (F-Encoder) based on the Transformer encoder and a time-domain encoder (T-Encoder) based on the Bidirectional Long Short-Term Memory (Bi-LSTM). The F-Encoder and T-Encoder model the correlations within frequency bands and time frames, respectively, and they are embedded into a time-frequency joint learning strategy to obtain the time-frequency patterns for speech emotions. Moreover, to handle the second issue, we also adopt time-frequency attention with a frequency-attention network (F-Attention) and a time-attention network (T-Attention) to focus on the emotion-related frequency band ranges and time frame ranges, which can enhance the discriminability of speech emotion features.
【12】 Information-Transport-based Policy for Simultaneous Translation标题：基于信息传输的同声传译策略链接：https://arxiv.org/abs/2210.12357
作者：Shaolei Zhang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：Accept to EMNLP 2022 main conference. 22 pages, 16 figures, 8 tables摘要：同步翻译（ST）在接收源输入的同时输出翻译，因此需要一种策略来确定是翻译目标令牌还是等待下一个源令牌。ST的主要挑战是每个目标标记只能基于当前接收到的源标记进行翻译，其中接收到的源信息将直接影响翻译质量。因此，对于当前目标令牌的翻译，接收到多少源信息被认为是ST策略在翻译和等待之间做出决定的关键证据。本文将翻译看作是信息从源语言到目标语言的传递，并据此提出了基于信息传递的同声翻译。ITST量化从每个源令牌到当前目标令牌的传输信息权重，然后根据目标令牌累积的接收信息来决定是否翻译目标令牌。对文本到文本ST和语音到文本ST（也称为，流式语音翻译）任务的测试结果表明，ITST的性能优于强基线，并达到了最新的性能。摘要：Simultaneous translation (ST) outputs translation while receiving the source inputs, and hence requires a policy to determine whether to translate a target token or wait for the next source token. The major challenge of ST is that each target token can only be translated based on the current received source tokens, where the received source information will directly affect the translation quality. So naturally, how much source information is received for the translation of the current target token is supposed to be the pivotal evidence for the ST policy to decide between translating and waiting. In this paper, we treat the translation as information transport from source to target and accordingly propose an Information-Transport-based Simultaneous Translation (ITST). ITST quantifies the transported information weight from each source token to the current target token, and then decides whether to translate the target token according to its accumulated received information. Experiments on both text-to-text ST and speech-to-text ST (a.k.a., streaming speech translation) tasks show that ITST outperforms strong baselines and achieves state-of-the-art performance.
【13】 Neural Sound Field Decomposition with Super-resolution of Sound Direction标题：声向超分辨的神经声场分解链接：https://arxiv.org/abs/2210.12345
作者：Qiuqiang Kong,Shilei Liu,Junjie Shi,Xuzhou Ye,Yin Cao,Qiaoxi Zhu,Yong Xu,Yuxuan Wang机构： ByteDance, Shanghai, China, University of Surrey, Guildford, UK, University of Technology Sydney, Sydney, Australia, Tencent AI Lab, Bellevue, USA备注：12 pages摘要：声场分解使用来自有限数量的麦克风的信号作为输入来预测任意方向上的波形。声场分解是下游任务的基础，包括源定位、源分离和空间音频再现。传统的声场分解方法，例如立体声音响，具有有限的空间分解分辨率。提出了一种基于学习的神经声场分解（NeSD）框架，利用几个麦克风在任意位置的拾音盒录音，实现了具有良好空间方向分辨率的声场分解。NeSD系统的输入包括麦克风信号、麦克风位置和所查询的方向。NeSD的输出包括查询位置的波形和存在概率。分别用全连接、时滞和递归神经网络对NeSD系统进行建模。我们证明了NeSD系统在语音、音乐和声音事件数据集上的声场分解和源定位方面优于传统的Ambisonics和DOANet方法。演示可在www.example.com上获得https://www.youtube.com/watch? v=0GIr6doj3BQ.摘要：Sound field decomposition predicts waveforms in arbitrary directions using signals from a limited number of microphones as inputs. Sound field decomposition is fundamental to downstream tasks, including source localization, source separation, and spatial audio reproduction. Conventional sound field decomposition methods such as Ambisonics have limited spatial decomposition resolution. This paper proposes a learning-based Neural Sound field Decomposition (NeSD) framework to allow sound field decomposition with fine spatial direction resolution, using recordings from microphone capsules of a few microphones at arbitrary positions. The inputs of a NeSD system include microphone signals, microphone positions, and queried directions. The outputs of a NeSD include the waveform and the presence probability of a queried position. We model the NeSD systems respectively with different neural networks, including fully connected, time delay, and recurrent neural networks. We show that the NeSD systems outperform conventional Ambisonics and DOANet methods in sound field decomposition and source localization on speech, music, and sound events datasets. Demos are available at https://www.youtube.com/watch?v=0GIr6doj3BQ.
【14】 Guided contrastive self-supervised pre-training for automatic speech recognition标题：用于自动语音识别的引导式对比自监督预训练链接：https://arxiv.org/abs/2210.12335
作者：Aparna Khare,Minhua Wu,Saurabhchand Bhati,Jasha Droppo,Roland Maas机构：Amazon Alexa, USA, Johns Hopkins University, USA备注：To appear in SLT 2022摘要：对比预测编码（CPC）是一种表示学习方法，它最大化中间潜在表示和给定模型的输出之间的互信息。它可用于有效地初始化自动语音识别（ASR）模型的编码器。本文提出了一种新的CPC改进方案--引导对比预测编码（Guided Contrasive Predictive Coding，GCPC）。我们提出的方法最大化了来自先验知识模型的表示和被预训练的模型的输出之间的互信息，允许在预训练期间的先验知识注入。我们在3个ASR任务上验证了我们的方法：德语、法语和英语。在德语、法语和英语（Librispeech）三个语料库上，与从头开始训练相比，我们的方法在所有三个数据集上的表现都优于CPC预训练，相对于从头开始训练，单词错误率（WER）分别降低了4.44%、6.55%和15.43%，而CPC预训练仅分别带来了2.96%、1.01%和14.39%的相对WER降低。摘要：Contrastive Predictive Coding (CPC) is a representation learning method that maximizes the mutual information between intermediate latent representations and the output of a given model. It can be used to effectively initialize the encoder of an Automatic Speech Recognition (ASR) model. We present a novel modification of CPC called Guided Contrastive Predictive Coding (GCPC). Our proposed method maximizes the mutual information between representations from a prior-knowledge model and the output of the model being pre-trained, allowing prior knowledge injection during pre-training. We validate our method on 3 ASR tasks: German, French and English. Our method outperforms CPC pre-training on all three datasets, reducing the Word Error Rate (WER) by 4.44%, 6.55% and 15.43% relative on the German, French and English (Librispeech) tasks respectively, compared to training from scratch, while CPC pre-training only brings 2.96%, 1.01% and 14.39% relative WER reduction respectively.
【15】 BEANS: The Benchmark of Animal Sounds标题：豆类：动物声音的标杆链接：https://arxiv.org/abs/2210.12300
作者：Masato Hagiwara,Benjamin Hoffman,Jen-Yu Liu,Maddie Cusimano,Felix Effenberger,Katie Zacarian机构：Earth Species Project摘要：近年来，基于机器学习的技术在生物声学领域中的应用越来越普遍。基于ML的技术的成功应用的基本要求是策划、商定、高质量的数据集和在给定数据集上学习的基准任务。然而，迄今为止，生物声学领域缺乏这样的公共基准，其覆盖多个任务和物种，以受控和标准化的方式测量ML技术的性能，并允许将新提出的技术与现有技术进行基准比较。在这里，我们提出BEANS（动物声音的BEnchmark），一个生物声学任务和公共数据集的集合，专门设计用于测量机器学习算法在生物声学领域的性能。这里提出的基准测试由生物声学中的两个常见任务组成：分类和检测。它包括12个数据集，涵盖各种物种，包括鸟类、陆地和海洋哺乳动物、无尾目动物和昆虫。除了数据集，我们还提出了一套标准的ML方法的性能作为任务性能的基线。基准测试和基线代码可在\url{https：//github.com/earthspecies/beans}上公开获取，希望为基于ML的生物声学研究建立一个新的标准数据集。摘要：The use of machine learning (ML) based techniques has become increasingly popular in the field of bioacoustics over the last years. Fundamental requirements for the successful application of ML based techniques are curated, agreed upon, high-quality datasets and benchmark tasks to be learned on a given dataset. However, the field of bioacoustics so far lacks such public benchmarks which cover multiple tasks and species to measure the performance of ML techniques in a controlled and standardized way and that allows for benchmarking newly proposed techniques to existing ones. Here, we propose BEANS (the BEnchmark of ANimal Sounds), a collection of bioacoustics tasks and public datasets, specifically designed to measure the performance of machine learning algorithms in the field of bioacoustics. The benchmark proposed here consists of two common tasks in bioacoustics: classification and detection. It includes 12 datasets covering various species, including birds, land and marine mammals, anurans, and insects. In addition to the datasets, we also present the performance of a set of standard ML methods as the baseline for task performance. The benchmark and baseline code is made publicly available at \url{https://github.com/earthspecies/beans} in the hope of establishing a new standard dataset for ML-based bioacoustic research.
【16】 Low-Resource Multilingual and Zero-Shot Multispeaker TTS标题：低资源多语言零激发多人话音合成技术链接：https://arxiv.org/abs/2210.12223
作者：Florian Lux,Julia Koch,Ngoc Thang Vu机构：University of Stuttgart备注：Accepted to AACL 2022摘要：虽然用于文本到语音（TTS）的神经方法在建模多个说话人方面显示出巨大的进步，即使在零触发设置中，这些方法所需的数据量通常对于世界上6,000多种口语中的绝大多数来说是不可行的。在这项工作中，我们将零拍语音克隆和多语言低资源TTS的任务结合在一起。使用语言不可知元学习（LAML）过程和对TTS编码器的修改，我们表明系统可以仅使用5分钟的训练数据来学习说一种新语言，同时保持推断新学习语言中甚至看不见的说话者的声音的能力。我们使用客观指标以及人类研究，在可懂度、自然度和与目标说话人的相似性方面展示了我们提出的方法的成功，并提供我们的代码和开源训练模型。摘要：While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world's over 6,000 spoken languages. In this work, we bring together the tasks of zero-shot voice cloning and multilingual low-resource TTS. Using the language agnostic meta learning (LAML) procedure and modifications to a TTS encoder, we show that it is possible for a system to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language. We show the success of our proposed approach in terms of intelligibility, naturalness and similarity to target speaker using objective metrics as well as human studies and provide our code and trained models open source.
【17】 Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation标题：用合成语码转换文本生成优化双语神经传感器链接：https://arxiv.org/abs/2210.12214
作者：Thien Nguyen,Nathalie Tran,Liuhui Deng,Thiago Fraga da Silva,Matthew Radzihovsky,Roger Hsiao,Henry Mason,Stefan Braun,Erik McDermott,Dogan Can,Pawel Swietojanski,Lyan Verwimp,Sibel Oyman,Tresi Arvizo,Honza Silovsky,Arnab Ghoshal,Mathieu Martel,Bharat Ram Ambati,Mohamed Ali机构：Apple备注：5 pages, 1 figure, submitted to ICASSP 2023, *: equal contributions摘要：语码转换是指在同一句话中使用多种语言的行为。在本研究中，我们研究如何优化一个基于神经传感器的双语自动语音识别（ASR）模型，用于语码转换语音。针对无监督语码转换语音训练ASR模型的场景，我们发现半监督训练和合成语码转换语音可以改善双语ASR系统对语码转换语音的处理。我们通过测量编码器特定的召回值来分析神经传感器的每个编码器如何对代码转换性能做出贡献，并在ASCEND数据集上评估我们的英语/汉语系统。我们的最终系统在ASCEND英语/普通话代码转换测试集上实现了25%的混合错误率（MER）--与以前的文献相比，MER绝对值降低了2.1%--同时在单语测试集上保持了良好的准确率。摘要：Code-switching describes the practice of using more than one language in the same sentence. In this study, we investigate how to optimize a neural transducer based bilingual automatic speech recognition (ASR) model for code-switching speech. Focusing on the scenario where the ASR model is trained without supervised code-switching data, we found that semi-supervised training and synthetic code-switched data can improve the bilingual ASR system on code-switching speech. We analyze how each of the neural transducer's encoders contributes towards code-switching performance by measuring encoder-specific recall values, and evaluate our English/Mandarin system on the ASCEND data set. Our final system achieves 25% mixed error rate (MER) on the ASCEND English/Mandarin code-switching test set -- reducing the MER by 2.1% absolute compared to the previous literature -- while maintaining good accuracy on the monolingual test sets.
【18】 High Fidelity Neural Audio Compression标题：高保真神经音频压缩链接：https://arxiv.org/abs/2210.13438
作者：Alexandre Défossez,Jade Copet,Gabriel Synnaeve,Yossi Adi机构：Meta AI, FAIR Team, Paris, France, Meta AI, FAIR Team, Tel-Aviv, Israel备注：Preprint摘要：我们介绍了一种利用神经网络的实时、高保真音频编解码器。它存在于具有以端到端方式训练的量化潜在空间的流编码器-解码器架构中。我们通过使用单个多尺度谱图对手来简化和加速训练，该对手有效地减少伪影并产生高质量的样本。我们引入了一种新颖的损耗均衡器机制来稳定训练：现在，损失的权重定义了它应该表示的总梯度的分数，从而将该超参数的选择与损失的典型尺度分离。最后，我们研究了如何使用轻量级Transformer模型将获得的表示进一步压缩高达40%，同时保持比实时更快的速度。我们提供了所提议模型的关键设计选择的详细描述，包括：训练目标、结构变化和对各种知觉丧失功能的研究。我们提出了一个广泛的主观评价（MUSHRA测试）与消融研究的带宽和音频领域的范围，包括语音，噪声混响语音和音乐。考虑到24 kHz单声道和48 kHz立体声音频，我们的方法在所有评估设置中均优于基线方法。有关代码和型号的信息，请访问github.com/facebookresearch/encodec。摘要：We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.
【19】 Time-Domain Speech Enhancement for Robust Automatic Speech Recognition标题：用于鲁棒自动语音识别的时域语音增强链接：https://arxiv.org/abs/2210.13318
作者：Yufeng Yang,Ashutosh Pandey,DeLiang Wang机构：Department of Computer Science and Engineering, The Ohio State University, USA, Center for Cognitive and Brain Sciences, The Ohio State University, USA备注：5 pages, 2 figures, submitted to ICASSP 2023摘要：研究表明，语音增强算法可以提高含噪语音的可懂度。然而，与直接在含噪语音上训练的ASR模型相比，语音增强还没有被确立为鲁棒自动语音识别（ASR）的有效前端。语音增强和ASR之间的鸿沟阻碍了鲁棒ASR系统的发展，特别是近年来语音增强取得了很大的进展。在本研究中，我们将重点放在消除这种分裂，并使用一种基于ARN（注意力递归网络）的时域增强模型。所提出的系统完全解耦语音增强和仅在干净语音上训练的声学模型。在CHiME-2语料库上的结果显示ARN增强的语音翻译成改进的ASR结果。该系统的平均误字率为6.28%，比以前的最佳系统高出19.3%.摘要：It has been shown that the intelligibility of noisy speech can be improved by speech enhancement algorithms. However, speech enhancement has not been established as an effective front-end for robust automatic speech recognition (ASR) in comparison with an ASR model trained on noisy speech directly. The divide between speech enhancement and ASR impedes the progress of robust ASR systems especially as speech enhancement has made big strides in recent years. In this work, we focus on eliminating such divide with an ARN (attentive recurrent network) based time-domain enhancement model. The proposed system fully decouples speech enhancement and an acoustic model trained only on clean speech. Results on the CHiME-2 corpus show that ARN enhanced speech translates to improved ASR results. The proposed system achieves $6.28\%$ average word error rate, outperforming the previous best by $19.3\%$.
【20】 An Acoustical Machine Learning Approach to Determine Abrasive Belt Wear of Wide Belt Sanders标题：确定宽带砂带磨损量的声学机器学习方法链接：https://arxiv.org/abs/2210.13273
作者：Maximilian Bundscherer,Thomas H. Schmitt,Sebastian Bayerl,Thomas Auerbach,Tobias Bocklet机构：†Department of Computer Science - Technische Hochschule N¨urnberg Georg Simon Ohm N¨urnberg, Germany, ‡Production Technology - Hans Weber Maschinenfabrik GmbH Kronach, Germany备注：Conference IEEE SENSORS 2022摘要：本文描述了一种机器学习方法，用于根据声学数据确定工业过程中使用的宽带砂光机的砂带磨损，而不考虑砂光工艺相关参数、进给速度、磨料粒度和材料类型。我们的方法利用决策树、随机森林、k-最近邻和神经网络分类器从谱图、Mel谱图、MFCC、IMFCC和LFCC检测皮带磨损，在五个皮带磨损级别上产生高达86.1%的准确度。使用专门用于不同撒砂参数配置的不同决策树分类器可实现96%的准确度。分类器还可以以97%的准确度确定机器当前是在砂磨还是空闲，并且以98.4%和98.8%的准确度检测砂磨参数进给速度和砂砾尺寸。我们可以表明，高维特征的低维映射可以用于有意义地可视化皮带磨损和打磨参数。摘要：This paper describes a machine learning approach to determine the abrasive belt wear of wide belt sanders used in industrial processes based on acoustic data, regardless of the sanding process-related parameters, Feed speed, Grit Size, and Type of material. Our approach utilizes Decision Tree, Random Forest, k-nearest Neighbors, and Neural network Classifiers to detect the belt wear from Spectrograms, Mel Spectrograms, MFCC, IMFCC, and LFCC, yielding an accuracy of up to 86.1% on five levels of belt wear. A 96% accuracy could be achieved with different Decision Tree Classifiers specialized in different sanding parameter configurations. The classifiers could also determine with an accuracy of 97% if the machine is currently sanding or is idle and with an accuracy of 98.4% and 98.8% detect the sanding parameters Feed speed and Grit Size. We can show that low-dimensional mappings of high-dimensional features can be used to visualize belt wear and sanding parameters meaningfully.
【21】 Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimation标题：Brouhaha：语音活动检测、语音噪声比和C50房间声学估计的多任务训练链接：https://arxiv.org/abs/2210.13248
作者：Marvin Lavechin,Marianne Métais,Hadrien Titeux,Alodie Boissonnet,Jade Copet,Morgane Rivière,Elika Bergelson,Alejandrina Cristia,Emmanuel Dupoux,Hervé Bredin机构：Marianne M´etais⋆, Morgane Riviere, Herv´e Bredin, LSCP, DEC, ENS, EHESS, CNRS, PSL University, Paris, France, INRIA, Paris, France, Meta AI Research, Paris, France, Deepmind, Paris, France, Duke University, North Carolina, USA摘要：大多数自动语音处理系统对声学环境敏感，当应用于有噪声或混响的语音时性能降低。但是，如何分辨语音是嘈杂的还是回响的呢？我们提出了一个模拟在噪声和混响条件下录制的音频片段的流水线--Brouhaha。然后，我们使用模拟音频来联合训练Brouhaha模型，以进行语音活动检测、信噪比估计和C50房间声学预测。我们展示了如何使用预测的SNR和C50值来调查和帮助诊断自动语音处理工具（例如pyannote.audio用于扬声器diarization，或OpenAI的Whisper用于自动语音识别）所造成的错误。我们的管道和预先训练的模型都是开源的，并与语音社区共享。摘要：Most automatic speech processing systems are sensitive to the acoustic environment, with degraded performance when applied to noisy or reverberant speech. But how can one tell whether speech is noisy or reverberant? We propose Brouhaha, a pipeline to simulate audio segments recorded in noisy and reverberant conditions. We then use the simulated audio to jointly train the Brouhaha model for voice activity detection, signal-to-noise ratio estimation, and C50 room acoustics prediction. We show how the predicted SNR and C50 values can be used to investigate and help diagnose errors made by automatic speech processing tools (such as pyannote.audio for speaker diarization or OpenAI's Whisper for automatic speech recognition). Both our pipeline and a pretrained model are open source and shared with the speech community.
【22】 Weak-Supervised Dysarthria-invariant Features for Spoken Language Understanding using an FHVAE and Adversarial Training标题：使用FHVAE和对抗性训练的弱监督构音障碍不变特征用于口语理解链接：https://arxiv.org/abs/2210.13144
作者：Jinzi Qi,Hugo Van hamme机构：KULeuven, Department Electrical Engineering-ESAT-PSI, Kasteelpark Arenberg , Leuven, Belgium摘要：构音障碍语音中训练数据的缺乏和说话人的大变化导致构音障碍语音的口语理解系统的准确性和说话人泛化能力较差。通过对语音特征的研究，我们致力于在有限的构音障碍数据下提高模型的泛化能力。无监督训练的分解层次变分自动编码器（FHVAE）在分离内容和说话人表征方面显示出了其优势。早期的工作表明构音障碍表现在两个特征向量中。在这里，我们增加对抗性训练，以弥合控制和构音障碍语音数据域之间的差距。我们使用弱监督提取构音障碍和说话人不变特征。所提取的特征在口语理解任务上被评估，并且与来自基本FHVAE模型或普通滤波器组的特征相比，在具有更严重构音障碍的看不见的说话者上产生更高的准确度。摘要：The scarcity of training data and the large speaker variation in dysarthric speech lead to poor accuracy and poor speaker generalization of spoken language understanding systems for dysarthric speech. Through work on the speech features, we focus on improving the model generalization ability with limited dysarthric data. Factorized Hierarchical Variational Auto-Encoders (FHVAE) trained unsupervisedly have shown their advantage in disentangling content and speaker representations. Earlier work showed that the dysarthria shows in both feature vectors. Here, we add adversarial training to bridge the gap between the control and dysarthric speech data domains. We extract dysarthric and speaker invariant features using weak supervision. The extracted features are evaluated on a Spoken Language Understanding task and yield a higher accuracy on unseen speakers with more severe dysarthria compared to features from the basic FHVAE model or plain filterbanks.
【23】 A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids标题：一种基于云的多模式助听器视听语音增强框架结构链接：https://arxiv.org/abs/2210.13127
作者：Abhijeet Bishnu,Ankit Gupta,Mandar Gogate,Kia Dashtipour,Ahsan Adeel,Amir Hussain,Mathini Sellathurai,Tharmalingam Ratnarajah机构：∗ School of Engineering, University of Edinburgh, Edinburgh, United Kingdom, † School of Engineering & Physical Sciences, Heriot-Watt Watt University, Edinburgh, United Kingdom, ‡ School of Computing, Edinburgh Napier University, Edinburgh, United Kingdom摘要：本文设计了一种基于云的音视频语音增强（SE）收发器（PHY层）原型，以满足未来多通道听力辅助技术对高数据速率和低延迟的要求。该创新设计需要满足多个挑战性约束，包括上行/下行链路通信、传输和信号处理延迟以及实时AV SE模型处理。该收发器包括设备检测、帧检测、频率偏移估计和信道估计能力。我们根据数据速率和延迟要求开发了上行链路（助听器到云）和下行链路（云到助听器）帧结构。由于上行链路信息（音频和唇读）的变化性质，上行链路信道支持多数据速率帧结构，而下行链路信道具有固定数据速率帧结构。此外，我们还使用LabVIEW NXG评估了所开发的帧结构的收发器的不同PHY层模块的延迟。这可以与软件定义无线电（例如通用软件无线电外设）一起用于实时演示场景。摘要：In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.
【24】 TridentSE: Guiding Speech Enhancement with 32 Global Tokens标题：TridentSE：使用32个全局令牌引导语音增强链接：https://arxiv.org/abs/2210.12995
作者：Dacheng Yin,Zhiyuan Zhao,Chuanxin Tang,Zhiwei Xiong,Chong Luo机构：University of Science and Technology of China, Hefei, China, Microsoft Research Asia, Beijing, China备注：5 pages, 2 figures, 3 tables摘要：本文提出了一种新的语音增强结构TridentSE，它能够有效地捕获全局信息和局部细节信息。TridentSE保持T-F bin级表示以捕获细节，并使用少量全局令牌来处理全局信息。信息通过交叉注意模块在局部和全局表示之间传播。为了捕获帧间和帧内信息，将全局表征分为两组，分别沿时间轴和频率轴进行处理。进一步使用度量鉴别器来指导我们的模型获得更高的感知质量。TridentSE算法在VoiceBank+DEMAND数据集上的PESQ为3.47，在DNS无混响测试集上的PESQ为3.44，比之前的各种语音增强算法的性能都要好。可视化显示，全局令牌学习多样的和可解释的全局模式。摘要：In this paper, we present TridentSE, a novel architecture for speech enhancement, which is capable of efficiently capturing both global information and local details. TridentSE maintains T-F bin level representation to capture details, and uses a small number of global tokens to process the global information. Information is propagated between the local and the global representations through cross attention modules. To capture both inter- and intra-frame information, the global tokens are divided into two groups to process along the time and the frequency axis respectively. A metric discriminator is further employed to guide our model to achieve higher perceptual quality. Even with significantly lower computational cost, TridentSE outperforms a variety of previous speech enhancement methods, achieving a PESQ of 3.47 on VoiceBank+DEMAND dataset and a PESQ of 3.44 on DNS no-reverb test set. Visualization shows that the global tokens learn diverse and interpretable global patterns.
【25】 HiFi-WaveGAN: Generative Adversarial Network with Auxiliary Spectrogram-Phase Loss for High-Fidelity Singing Voice Generation标题：HIFI-WaveGAN：高保真歌声生成的带辅助谱图的生成对抗性网络--相位损失链接：https://arxiv.org/abs/2210.12740
作者：Chunhui Wang,Chang Zeng,Xing He机构：Xiaobing.ai, China ,National Institute of Informatics, Japan ,SOKENDAI, Japan备注：submitted to icassp2023摘要：面向娱乐的歌唱声音合成（SVS）需要声码器来生成高保真度（例如48 kHz）音频。然而，即使用于文本到语音（TTS）的神经声码器已经取得了显著的进展，大多数TTS声码器在这种情况下也不能很好地工作。本文提出了一种用于实时合成48 kHz高质量歌唱声的HiFi-WaveGAN算法。具体来说，它由改进的WaveNet发生器、与HiFiGAN相同的多周期鉴别器和借鉴UnivNet的多分辨率谱图鉴别器组成。为了更好地从全频带Mel谱图中重构高频部分，设计了一种新的辅助谱图-相位损失来训练神经网络，并加快了训练过程。实验结果表明，在48 kHz的SVS任务中，HiFi-WaveGAN的平均意见得分（MOS）指标明显优于并行WaveGAN（PWG）和HiFiGAN等神经声码器。通过对有无相位损失项的HiFi-WaveGAN的对比研究，证明了相位损失项确实提高了训练速度。此外，我们还将HiFi-WaveGAN和PWG生成的频谱图进行了比较，结果表明，HiFi-WaveGAN对高频部分的建模能力更强。摘要：Entertainment-oriented singing voice synthesis (SVS) requires a vocoder to generate high-fidelity (e.g. 48kHz) audio. However, most text-to-speech (TTS) vocoders cannot work well in this scenario even if the neural vocoder for TTS has achieved significant progress. In this paper, we propose HiFi-WaveGAN which is designed for synthesizing the 48kHz high-quality singing voices from the full-band mel-spectrogram in real-time. Specifically, it consists of a generator improved from WaveNet, a multi-period discriminator same to HiFiGAN, and a multi-resolution spectrogram discriminator borrowed from UnivNet. To better reconstruct the high-frequency part from the full-band mel-spectrogram, we design a novel auxiliary spectrogram-phase loss to train the neural network, which can also accelerate the training process. The experimental result shows that our proposed HiFi-WaveGAN significantly outperforms other neural vocoders such as Parallel WaveGAN (PWG) and HiFiGAN in the mean opinion score (MOS) metric for the 48kHz SVS task. And a comparative study of HiFi-WaveGAN with/without phase loss term proves that phase loss indeed improves the training speed. Besides, we also compare the spectrogram generated by our HiFi-WaveGAN and PWG, which shows our HiFi-WaveGAN has a more powerful ability to model the high-frequency parts.
【26】 Speaker Identification from emotional and noisy speech data using learned voice segregation and Speech VGG标题：基于学习语音分离和语音VGG的情感和噪声语音数据说话人识别链接：https://arxiv.org/abs/2210.12701
作者：Shibani Hamsa,Ismail Shahin,Youssef Iraqi,Ernesto Damiani,Naoufel Werghi机构：Center for Cyber-Physical Systems (C,PS), Dept. of ECE, Khalifa University, Abu Dhabi, UAE., Dept. of Electrical Engineering, University of Sharjah, Sharjah, UAE., School of Computer Science, Mohammed VI Polytechnic University, Morocco备注：Journal摘要：语音信号比其他信号更容易受到声学干扰和情感因素的影响。噪声和情感交织的语音数据对实时语音处理应用是一个挑战。找到一种有效的方法将主导信号与其他外部影响分离是至关重要的。理想的系统应该具有从不利情况下拍摄的复杂场景中准确识别所需听觉事件的能力。本文提出了一种基于深度神经网络模板和语音VGG的说话人识别方法，该方法能够在情感和干扰等不利条件下识别说话人。该模型在英语和阿拉伯语情感语音数据集上的识别率分别为85.2%、87.0%和86.6%，优于现有的文献.摘要：Speech signals are subjected to more acoustic interference and emotional factors than other signals. Noisy emotion-riddled speech data is a challenge for real-time speech processing applications. It is essential to find an effective way to segregate the dominant signal from other external influences. An ideal system should have the capacity to accurately recognize required auditory events from a complex scene taken in an unfavorable situation. This paper proposes a novel approach to speaker identification in unfavorable conditions such as emotion and interference using a pre-trained Deep Neural Network mask and speech VGG. The proposed model obtained superior performance over the recent literature in English and Arabic emotional speech data and reported an average speaker identification rate of 85.2\%, 87.0\%, and 86.6\% using the Ryerson audio-visual dataset (RAVDESS), speech under simulated and actual stress (SUSAS) dataset and Emirati-accented Speech dataset (ESD) respectively.
【27】 Deep domain adaptation for polyphonic melody extraction标题：用于复调旋律提取的深域自适应链接：https://arxiv.org/abs/2210.12532
作者：Kavya Ranjan Saxena,Vipul Arora机构：Department of Electrical Engineering, Indian Institute of Technology Kanpur, India备注：Submitted to ICASSP 2023摘要：从复调音频中提取主音高是音乐信息检索和计算音乐学领域的基本任务之一。为了使用机器学习来完成该任务，需要大量标记的音频数据来训练预测音高轮廓的模型。但是，在来自一个域（源）的数据（例如，特定歌手或流派的歌曲）上预先训练的经典模型在从其他域（目标）提取旋律时可能执行得不相对好。通过使用目标域中的一些注释数据来调整模型，可以提高这样的模型的性能。在这项工作中，我们研究了各种适应技术应用于机器学习模型的复调旋律提取。实验结果表明，基于元学习的自适应方法比简单的微调方法具有更好的性能。此外，我们发现该方法优于现有的最先进的非自适应复调旋律提取算法。摘要：Extraction of the predominant pitch from polyphonic audio is one of the fundamental tasks in the field of music information retrieval and computational musicology. To accomplish this task using machine learning, a large amount of labeled audio data is required to train the model that predicts the pitch contour. But a classical model pre-trained on data from one domain (source), e.g, songs of a particular singer or genre, may not perform comparatively well in extracting melody from other domains (target). The performance of such models can be boosted by adapting the model using some annotated data in the target domain. In this work, we study various adaptation techniques applied to machine learning models for polyphonic melody extraction. Experimental results show that meta-learning-based adaptation performs better than simple fine-tuning. In addition to this, we find that this method outperforms the existing state-of-the-art non-adaptive polyphonic melody extraction algorithms.
【1】 High Fidelity Neural Audio Compression标题：高保真神经音频压缩链接：https://arxiv.org/abs/2210.13438
* 与cs.SD语音【18】为同一篇
作者：Alexandre Défossez,Jade Copet,Gabriel Synnaeve,Yossi Adi机构：Meta AI, FAIR Team, Paris, France, Meta AI, FAIR Team, Tel-Aviv, Israel备注：Preprint摘要：我们介绍了一种利用神经网络的实时、高保真音频编解码器。它存在于具有以端到端方式训练的量化潜在空间的流编码器-解码器架构中。我们通过使用单个多尺度谱图对手来简化和加速训练，该对手有效地减少伪影并产生高质量的样本。我们引入了一种新颖的损耗均衡器机制来稳定训练：现在，损失的权重定义了它应该表示的总梯度的分数，从而将该超参数的选择与损失的典型尺度分离。最后，我们研究了如何使用轻量级Transformer模型将获得的表示进一步压缩高达40%，同时保持比实时更快的速度。我们提供了所提议模型的关键设计选择的详细描述，包括：训练目标、结构变化和对各种知觉丧失功能的研究。我们提出了一个广泛的主观评价（MUSHRA测试）与消融研究的带宽和音频领域的范围，包括语音，噪声混响语音和音乐。考虑到24 kHz单声道和48 kHz立体声音频，我们的方法在所有评估设置中均优于基线方法。有关代码和型号的信息，请访问github.com/facebookresearch/encodec。摘要：We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.
【2】 Time-Domain Speech Enhancement for Robust Automatic Speech Recognition标题：用于鲁棒自动语音识别的时域语音增强链接：https://arxiv.org/abs/2210.13318
* 与cs.SD语音【19】为同一篇
作者：Yufeng Yang,Ashutosh Pandey,DeLiang Wang机构：Department of Computer Science and Engineering, The Ohio State University, USA, Center for Cognitive and Brain Sciences, The Ohio State University, USA备注：5 pages, 2 figures, submitted to ICASSP 2023摘要：研究表明，语音增强算法可以提高含噪语音的可懂度。然而，与直接在含噪语音上训练的ASR模型相比，语音增强还没有被确立为鲁棒自动语音识别（ASR）的有效前端。语音增强和ASR之间的鸿沟阻碍了鲁棒ASR系统的发展，特别是近年来语音增强取得了很大的进展。在本研究中，我们将重点放在消除这种分裂，并使用一种基于ARN（注意力递归网络）的时域增强模型。所提出的系统完全解耦语音增强和仅在干净语音上训练的声学模型。在CHiME-2语料库上的结果显示ARN增强的语音翻译成改进的ASR结果。该系统的平均误字率为6.28%，比以前的最佳系统高出19.3%.摘要：It has been shown that the intelligibility of noisy speech can be improved by speech enhancement algorithms. However, speech enhancement has not been established as an effective front-end for robust automatic speech recognition (ASR) in comparison with an ASR model trained on noisy speech directly. The divide between speech enhancement and ASR impedes the progress of robust ASR systems especially as speech enhancement has made big strides in recent years. In this work, we focus on eliminating such divide with an ARN (attentive recurrent network) based time-domain enhancement model. The proposed system fully decouples speech enhancement and an acoustic model trained only on clean speech. Results on the CHiME-2 corpus show that ARN enhanced speech translates to improved ASR results. The proposed system achieves $6.28\%$ average word error rate, outperforming the previous best by $19.3\%$.
【3】 An Acoustical Machine Learning Approach to Determine Abrasive Belt Wear of Wide Belt Sanders标题：确定宽带砂带磨损量的声学机器学习方法链接：https://arxiv.org/abs/2210.13273
* 与cs.SD语音【20】为同一篇
作者：Maximilian Bundscherer,Thomas H. Schmitt,Sebastian Bayerl,Thomas Auerbach,Tobias Bocklet机构：†Department of Computer Science - Technische Hochschule N¨urnberg Georg Simon Ohm N¨urnberg, Germany, ‡Production Technology - Hans Weber Maschinenfabrik GmbH Kronach, Germany备注：Conference IEEE SENSORS 2022摘要：本文描述了一种机器学习方法，用于根据声学数据确定工业过程中使用的宽带砂光机的砂带磨损，而不考虑砂光工艺相关参数、进给速度、磨料粒度和材料类型。我们的方法利用决策树、随机森林、k-最近邻和神经网络分类器从谱图、Mel谱图、MFCC、IMFCC和LFCC检测皮带磨损，在五个皮带磨损级别上产生高达86.1%的准确度。使用专门用于不同撒砂参数配置的不同决策树分类器可实现96%的准确度。分类器还可以以97%的准确度确定机器当前是在砂磨还是空闲，并且以98.4%和98.8%的准确度检测砂磨参数进给速度和砂砾尺寸。我们可以表明，高维特征的低维映射可以用于有意义地可视化皮带磨损和打磨参数。摘要：This paper describes a machine learning approach to determine the abrasive belt wear of wide belt sanders used in industrial processes based on acoustic data, regardless of the sanding process-related parameters, Feed speed, Grit Size, and Type of material. Our approach utilizes Decision Tree, Random Forest, k-nearest Neighbors, and Neural network Classifiers to detect the belt wear from Spectrograms, Mel Spectrograms, MFCC, IMFCC, and LFCC, yielding an accuracy of up to 86.1% on five levels of belt wear. A 96% accuracy could be achieved with different Decision Tree Classifiers specialized in different sanding parameter configurations. The classifiers could also determine with an accuracy of 97% if the machine is currently sanding or is idle and with an accuracy of 98.4% and 98.8% detect the sanding parameters Feed speed and Grit Size. We can show that low-dimensional mappings of high-dimensional features can be used to visualize belt wear and sanding parameters meaningfully.
【4】 Brouhaha: multi-task training for voice activity detection, speech-to-noise ratio, and C50 room acoustics estimation标题：Brouhaha：语音活动检测、语音噪声比和C50房间声学估计的多任务训练链接：https://arxiv.org/abs/2210.13248
* 与cs.SD语音【21】为同一篇
作者：Marvin Lavechin,Marianne Métais,Hadrien Titeux,Alodie Boissonnet,Jade Copet,Morgane Rivière,Elika Bergelson,Alejandrina Cristia,Emmanuel Dupoux,Hervé Bredin机构：Marianne M´etais⋆, Morgane Riviere, Herv´e Bredin, LSCP, DEC, ENS, EHESS, CNRS, PSL University, Paris, France, INRIA, Paris, France, Meta AI Research, Paris, France, Deepmind, Paris, France, Duke University, North Carolina, USA摘要：大多数自动语音处理系统对声学环境敏感，当应用于有噪声或混响的语音时性能降低。但是，如何分辨语音是嘈杂的还是回响的呢？我们提出了一个模拟在噪声和混响条件下录制的音频片段的流水线--Brouhaha。然后，我们使用模拟音频来联合训练Brouhaha模型，以进行语音活动检测、信噪比估计和C50房间声学预测。我们展示了如何使用预测的SNR和C50值来调查和帮助诊断自动语音处理工具（例如pyannote.audio用于扬声器diarization，或OpenAI的Whisper用于自动语音识别）所造成的错误。我们的管道和预先训练的模型都是开源的，并与语音社区共享。摘要：Most automatic speech processing systems are sensitive to the acoustic environment, with degraded performance when applied to noisy or reverberant speech. But how can one tell whether speech is noisy or reverberant? We propose Brouhaha, a pipeline to simulate audio segments recorded in noisy and reverberant conditions. We then use the simulated audio to jointly train the Brouhaha model for voice activity detection, signal-to-noise ratio estimation, and C50 room acoustics prediction. We show how the predicted SNR and C50 values can be used to investigate and help diagnose errors made by automatic speech processing tools (such as pyannote.audio for speaker diarization or OpenAI's Whisper for automatic speech recognition). Both our pipeline and a pretrained model are open source and shared with the speech community.
【5】 Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?标题：视觉情境能改善具体化代理的自动语音识别吗？链接：https://arxiv.org/abs/2210.13189
作者：Pradip Pramanick,Chayan Sarkar机构：Robotics & Autonomous Systems, TCS Research, India备注：Accepted in EMNLP '22摘要：从个人助理到聊天机器人、家庭和工业自动化系统等，自动语音识别（ASR）系统的使用正变得无处不在。现代机器人还配备有用于与人类交互的ASR能力，因为语音是最自然的交互模态。然而，与个人助理相比，机器人中的ASR面临着额外的挑战。作为一个具体化的代理，机器人必须识别它周围的物理实体，从而可靠地识别包含这些实体描述的语音。然而，由于ASR训练中的限制，例如通用数据集和开放词汇表建模，当前的ASR系统通常不能这样做。此外，推断期间的不利条件，例如噪声、重音和远场语音，使得转录不准确。在这项工作中，我们提出了一种方法，将机器人的视觉信息融入到ASR系统中，并提高了对包含可视实体的口语话语的识别。具体而言，我们提出了一种新的解码器偏置技术，以结合视觉上下文，同时确保ASR输出不会因不正确的上下文而降级。与未修改的ASR系统相比，我们实现了59%的WER相对降低。摘要：The usage of automatic speech recognition (ASR) systems are becoming omnipresent ranging from personal assistant to chatbots, home, and industrial automation systems, etc. Modern robots are also equipped with ASR capabilities for interacting with humans as speech is the most natural interaction modality. However, ASR in robots faces additional challenges as compared to a personal assistant. Being an embodied agent, a robot must recognize the physical entities around it and therefore reliably recognize the speech containing the description of such entities. However, current ASR systems are often unable to do so due to limitations in ASR training, such as generic datasets and open-vocabulary modeling. Also, adverse conditions during inference, such as noise, accented, and far-field speech makes the transcription inaccurate. In this work, we present a method to incorporate a robot's visual information into an ASR system and improve the recognition of a spoken utterance containing a visible entity. Specifically, we propose a new decoder biasing technique to incorporate the visual context while ensuring the ASR output does not degrade for incorrect context. We achieve a 59% relative reduction in WER from an unmodified ASR system.
【6】 Weak-Supervised Dysarthria-invariant Features for Spoken Language Understanding using an FHVAE and Adversarial Training标题：使用FHVAE和对抗性训练的弱监督构音障碍不变特征用于口语理解链接：https://arxiv.org/abs/2210.13144
* 与cs.SD语音【22】为同一篇
作者：Jinzi Qi,Hugo Van hamme机构：KULeuven, Department Electrical Engineering-ESAT-PSI, Kasteelpark Arenberg , Leuven, Belgium摘要：构音障碍语音中训练数据的缺乏和说话人的大变化导致构音障碍语音的口语理解系统的准确性和说话人泛化能力较差。通过对语音特征的研究，我们致力于在有限的构音障碍数据下提高模型的泛化能力。无监督训练的分解层次变分自动编码器（FHVAE）在分离内容和说话人表征方面显示出了其优势。早期的工作表明构音障碍表现在两个特征向量中。在这里，我们增加对抗性训练，以弥合控制和构音障碍语音数据域之间的差距。我们使用弱监督提取构音障碍和说话人不变特征。所提取的特征在口语理解任务上被评估，并且与来自基本FHVAE模型或普通滤波器组的特征相比，在具有更严重构音障碍的看不见的说话者上产生更高的准确度。摘要：The scarcity of training data and the large speaker variation in dysarthric speech lead to poor accuracy and poor speaker generalization of spoken language understanding systems for dysarthric speech. Through work on the speech features, we focus on improving the model generalization ability with limited dysarthric data. Factorized Hierarchical Variational Auto-Encoders (FHVAE) trained unsupervisedly have shown their advantage in disentangling content and speaker representations. Earlier work showed that the dysarthria shows in both feature vectors. Here, we add adversarial training to bridge the gap between the control and dysarthric speech data domains. We extract dysarthric and speaker invariant features using weak supervision. The extracted features are evaluated on a Spoken Language Understanding task and yield a higher accuracy on unseen speakers with more severe dysarthria compared to features from the basic FHVAE model or plain filterbanks.
【7】 A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids标题：一种基于云的多模式助听器视听语音增强框架结构链接：https://arxiv.org/abs/2210.13127
* 与cs.SD语音【23】为同一篇
作者：Abhijeet Bishnu,Ankit Gupta,Mandar Gogate,Kia Dashtipour,Ahsan Adeel,Amir Hussain,Mathini Sellathurai,Tharmalingam Ratnarajah机构：∗ School of Engineering, University of Edinburgh, Edinburgh, United Kingdom, † School of Engineering & Physical Sciences, Heriot-Watt Watt University, Edinburgh, United Kingdom, ‡ School of Computing, Edinburgh Napier University, Edinburgh, United Kingdom摘要：本文设计了一种基于云的音视频语音增强（SE）收发器（PHY层）原型，以满足未来多通道听力辅助技术对高数据速率和低延迟的要求。该创新设计需要满足多个挑战性约束，包括上行/下行链路通信、传输和信号处理延迟以及实时AV SE模型处理。该收发器包括设备检测、帧检测、频率偏移估计和信道估计能力。我们根据数据速率和延迟要求开发了上行链路（助听器到云）和下行链路（云到助听器）帧结构。由于上行链路信息（音频和唇读）的变化性质，上行链路信道支持多数据速率帧结构，而下行链路信道具有固定数据速率帧结构。此外，我们还使用LabVIEW NXG评估了所开发的帧结构的收发器的不同PHY层模块的延迟。这可以与软件定义无线电（例如通用软件无线电外设）一起用于实时演示场景。摘要：In this paper, we design a first of its kind transceiver (PHY layer) prototype for cloud-based audio-visual (AV) speech enhancement (SE) complying with high data rate and low latency requirements of future multimodal hearing assistive technology. The innovative design needs to meet multiple challenging constraints including up/down link communications, delay of transmission and signal processing, and real-time AV SE models processing. The transceiver includes device detection, frame detection, frequency offset estimation, and channel estimation capabilities. We develop both uplink (hearing aid to the cloud) and downlink (cloud to hearing aid) frame structures based on the data rate and latency requirements. Due to the varying nature of uplink information (audio and lip-reading), the uplink channel supports multiple data rate frame structure, while the downlink channel has a fixed data rate frame structure. In addition, we evaluate the latency of different PHY layer blocks of the transceiver for developed frame structures using LabVIEW NXG. This can be used with software defined radio (such as Universal Software Radio Peripheral) for real-time demonstration scenarios.
【8】 TridentSE: Guiding Speech Enhancement with 32 Global Tokens标题：TridentSE：使用32个全局令牌引导语音增强链接：https://arxiv.org/abs/2210.12995
* 与cs.SD语音【24】为同一篇
作者：Dacheng Yin,Zhiyuan Zhao,Chuanxin Tang,Zhiwei Xiong,Chong Luo机构：University of Science and Technology of China, Hefei, China, Microsoft Research Asia, Beijing, China备注：5 pages, 2 figures, 3 tables摘要：本文提出了一种新的语音增强结构TridentSE，它能够有效地捕获全局信息和局部细节信息。TridentSE保持T-F bin级表示以捕获细节，并使用少量全局令牌来处理全局信息。信息通过交叉注意模块在局部和全局表示之间传播。为了捕获帧间和帧内信息，将全局表征分为两组，分别沿时间轴和频率轴进行处理。进一步使用度量鉴别器来指导我们的模型获得更高的感知质量。TridentSE算法在VoiceBank+DEMAND数据集上的PESQ为3.47，在DNS无混响测试集上的PESQ为3.44，比之前的各种语音增强算法的性能都要好。可视化显示，全局令牌学习多样的和可解释的全局模式。摘要：In this paper, we present TridentSE, a novel architecture for speech enhancement, which is capable of efficiently capturing both global information and local details. TridentSE maintains T-F bin level representation to capture details, and uses a small number of global tokens to process the global information. Information is propagated between the local and the global representations through cross attention modules. To capture both inter- and intra-frame information, the global tokens are divided into two groups to process along the time and the frequency axis respectively. A metric discriminator is further employed to guide our model to achieve higher perceptual quality. Even with significantly lower computational cost, TridentSE outperforms a variety of previous speech enhancement methods, achieving a PESQ of 3.47 on VoiceBank+DEMAND dataset and a PESQ of 3.44 on DNS no-reverb test set. Visualization shows that the global tokens learn diverse and interpretable global patterns.
【9】 HiFi-WaveGAN: Generative Adversarial Network with Auxiliary Spectrogram-Phase Loss for High-Fidelity Singing Voice Generation标题：HIFI-WaveGAN：高保真歌声生成的带辅助谱图的生成对抗性网络--相位损失链接：https://arxiv.org/abs/2210.12740
* 与cs.SD语音【25】为同一篇
作者：Chunhui Wang,Chang Zeng,Xing He机构：Xiaobing.ai, China ,National Institute of Informatics, Japan ,SOKENDAI, Japan备注：submitted to icassp2023摘要：面向娱乐的歌唱声音合成（SVS）需要声码器来生成高保真度（例如48 kHz）音频。然而，即使用于文本到语音（TTS）的神经声码器已经取得了显著的进展，大多数TTS声码器在这种情况下也不能很好地工作。本文提出了一种用于实时合成48 kHz高质量歌唱声的HiFi-WaveGAN算法。具体来说，它由改进的WaveNet发生器、与HiFiGAN相同的多周期鉴别器和借鉴UnivNet的多分辨率谱图鉴别器组成。为了更好地从全频带Mel谱图中重构高频部分，设计了一种新的辅助谱图-相位损失来训练神经网络，并加快了训练过程。实验结果表明，在48 kHz的SVS任务中，HiFi-WaveGAN的平均意见得分（MOS）指标明显优于并行WaveGAN（PWG）和HiFiGAN等神经声码器。通过对有无相位损失项的HiFi-WaveGAN的对比研究，证明了相位损失项确实提高了训练速度。此外，我们还将HiFi-WaveGAN和PWG生成的频谱图进行了比较，结果表明，HiFi-WaveGAN对高频部分的建模能力更强。摘要：Entertainment-oriented singing voice synthesis (SVS) requires a vocoder to generate high-fidelity (e.g. 48kHz) audio. However, most text-to-speech (TTS) vocoders cannot work well in this scenario even if the neural vocoder for TTS has achieved significant progress. In this paper, we propose HiFi-WaveGAN which is designed for synthesizing the 48kHz high-quality singing voices from the full-band mel-spectrogram in real-time. Specifically, it consists of a generator improved from WaveNet, a multi-period discriminator same to HiFiGAN, and a multi-resolution spectrogram discriminator borrowed from UnivNet. To better reconstruct the high-frequency part from the full-band mel-spectrogram, we design a novel auxiliary spectrogram-phase loss to train the neural network, which can also accelerate the training process. The experimental result shows that our proposed HiFi-WaveGAN significantly outperforms other neural vocoders such as Parallel WaveGAN (PWG) and HiFiGAN in the mean opinion score (MOS) metric for the 48kHz SVS task. And a comparative study of HiFi-WaveGAN with/without phase loss term proves that phase loss indeed improves the training speed. Besides, we also compare the spectrogram generated by our HiFi-WaveGAN and PWG, which shows our HiFi-WaveGAN has a more powerful ability to model the high-frequency parts.
【10】 Speaker Identification from emotional and noisy speech data using learned voice segregation and Speech VGG标题：基于学习语音分离和语音VGG的情感和噪声语音数据说话人识别链接：https://arxiv.org/abs/2210.12701
* 与cs.SD语音【26】为同一篇
作者：Shibani Hamsa,Ismail Shahin,Youssef Iraqi,Ernesto Damiani,Naoufel Werghi机构：Center for Cyber-Physical Systems (C,PS), Dept. of ECE, Khalifa University, Abu Dhabi, UAE., Dept. of Electrical Engineering, University of Sharjah, Sharjah, UAE., School of Computer Science, Mohammed VI Polytechnic University, Morocco备注：Journal摘要：语音信号比其他信号更容易受到声学干扰和情感因素的影响。噪声和情感交织的语音数据对实时语音处理应用是一个挑战。找到一种有效的方法将主导信号与其他外部影响分离是至关重要的。理想的系统应该具有从不利情况下拍摄的复杂场景中准确识别所需听觉事件的能力。本文提出了一种基于深度神经网络模板和语音VGG的说话人识别方法，该方法能够在情感和干扰等不利条件下识别说话人。该模型在英语和阿拉伯语情感语音数据集上的识别率分别为85.2%、87.0%和86.6%，优于现有的文献.摘要：Speech signals are subjected to more acoustic interference and emotional factors than other signals. Noisy emotion-riddled speech data is a challenge for real-time speech processing applications. It is essential to find an effective way to segregate the dominant signal from other external influences. An ideal system should have the capacity to accurately recognize required auditory events from a complex scene taken in an unfavorable situation. This paper proposes a novel approach to speaker identification in unfavorable conditions such as emotion and interference using a pre-trained Deep Neural Network mask and speech VGG. The proposed model obtained superior performance over the recent literature in English and Arabic emotional speech data and reported an average speaker identification rate of 85.2\%, 87.0\%, and 86.6\% using the Ryerson audio-visual dataset (RAVDESS), speech under simulated and actual stress (SUSAS) dataset and Emirati-accented Speech dataset (ESD) respectively.
【11】 chowdsp_wdf: An Advanced C++ Library for Wave Digital Circuit Modelling标题：Chowdsp_wdf：一个用于波形数字电路建模的高级C++库链接：https://arxiv.org/abs/2210.12554
作者：Jatin Chowdhury机构：Chowdhury DSP, Denver, CO摘要：chowdsp_wdf是一个C++库，用于实现模拟电路的实时波形数字模型。chowdsp_wdf与现有的wave数字建模库的不同之处在于，它提供了用于对具有固定拓扑的电路进行建模的模板元编程接口，并提供了对显式SIMD加速的支持。本文描述了该库的动机和设计，以及现实世界的用例，并与其他波浪数字建模库进行了性能比较。摘要：chowdsp_wdf is a C++ library for implementing real-time wave digital models of analog circuits. chowdsp_wdf differs from existing wave digital modelling libraries by providing a template meta-programming interface for modelling circuits with a fixed topology, and providing support for explicit SIMD acceleration. The motivation and design of the library are described, as well as real-world use-cases, and performance comparisons with other wave digital modelling libraries.
【12】 Deep domain adaptation for polyphonic melody extraction标题：用于复调旋律提取的深域自适应链接：https://arxiv.org/abs/2210.12532
* 与cs.SD语音【27】为同一篇
作者：Kavya Ranjan Saxena,Vipul Arora机构：Department of Electrical Engineering, Indian Institute of Technology Kanpur, India备注：Submitted to ICASSP 2023摘要：从复调音频中提取主音高是音乐信息检索和计算音乐学领域的基本任务之一。为了使用机器学习来完成该任务，需要大量标记的音频数据来训练预测音高轮廓的模型。但是，在来自一个域（源）的数据（例如，特定歌手或流派的歌曲）上预先训练的经典模型在从其他域（目标）提取旋律时可能执行得不相对好。通过使用目标域中的一些注释数据来调整模型，可以提高这样的模型的性能。在这项工作中，我们研究了各种适应技术应用于机器学习模型的复调旋律提取。实验结果表明，基于元学习的自适应方法比简单的微调方法具有更好的性能。此外，我们发现该方法优于现有的最先进的非自适应复调旋律提取算法。摘要：Extraction of the predominant pitch from polyphonic audio is one of the fundamental tasks in the field of music information retrieval and computational musicology. To accomplish this task using machine learning, a large amount of labeled audio data is required to train the model that predicts the pitch contour. But a classical model pre-trained on data from one domain (source), e.g, songs of a particular singer or genre, may not perform comparatively well in extracting melody from other domains (target). The performance of such models can be boosted by adapting the model using some annotated data in the target domain. In this work, we study various adaptation techniques applied to machine learning models for polyphonic melody extraction. Experimental results show that meta-learning-based adaptation performs better than simple fine-tuning. In addition to this, we find that this method outperforms the existing state-of-the-art non-adaptive polyphonic melody extraction algorithms.
【13】 Development of Hybrid ASR Systems for Low Resource Medical Domain Conversational Telephone Speech标题：低资源医疗领域通话语音混合ASR系统的研制链接：https://arxiv.org/abs/2210.13397
* 与cs.SD语音【1】为同一篇
作者：Christoph Lüscher,Mohammad Zeineldeen,Zijian Yang,Peter Vieting,Khai Le-Duc,Weiyue Wang,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition Group, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：ASR System Paper for HYKIST project摘要：近年来，ASR系统在有足够数量的训练数据可用的特定任务（例如LibriSpeech）上已经达到了显著的性能。然而，变化的声学和记录条件和说话风格以及缺乏足够的域内训练数据仍然对准确模型的开发提出了挑战。在这项工作中，我们介绍了我们为医疗领域中三种语言（阿拉伯语、德语、越南语）的会话式电话语音翻译任务开发ASR系统的努力，以支持医生和患者之间跨越语言障碍的急诊室交互。我们研究了不同的训练计划和数据组合方法，以提高系统的性能，并分析在哪里最有效地使用有限的可用数据。摘要：In recent years, ASR systems have reached remarkable performance on specific tasks for which sufficient amounts of training data are available, like e.g. LibriSpeech. However, varying acoustic and recording conditions and speaking styles and a lack of sufficient in-domain training data still pose challenges to the development of accurate models. In this work, we present our efforts for the development of ASR systems for a conversational telephone speech translation task in the medical domain for three languages (Arabic, German, Vietnamese) to support emergency room interaction between physician and patient across language barriers. We study different training schedules and data combination approaches in order to improve the system's performance, as well as analyze where limited available data is used most efficiently.
【14】 ESB: A Benchmark For Multi-Domain End-to-End Speech Recognition标题：ESB：一种多域端到端语音识别基准链接：https://arxiv.org/abs/2210.13352
* 与cs.SD语音【2】为同一篇
作者：Sanchit Gandhi,Patrick von Platen,Alexander M. Rush机构：Hugging Face备注：25 pages, 1 figure, submitted to ICLR 2023摘要：语音识别应用覆盖了一系列不同的音频和文本分布，具有不同的说话风格、背景噪声、转录标点符号和字符大小写。然而，许多语音识别系统需要数据集特定的调整（音频过滤、标点符号去除和大小写的归一化），因此假定音频和文本分布的先验知识。这种调优要求可能导致系统无法推广到其他数据集和域。为了促进多领域语音识别系统的发展，引入了端到端语音基准测试（ESB），用于评估单个自动语音识别（ASR）系统在大量语音数据集上的性能。基准测试系统必须在数据集之间使用相同的数据预处理和后处理算法-假设音频和文本数据分布是先验未知的。我们在此基准测试上比较了一系列最先进的（SoTA）端到端（E2 E）系统，展示了如何在广泛的数据分布上应用和评估单个语音系统。我们发现E2 E系统在数据集上都是有效的：在一个公平的比较中，E2 E系统实现了在SoTA系统的2.6%之内调谐到一个特定的数据集。我们的分析表明，转录伪像，如标点符号和大小写，造成困难的ASR系统，并应纳入评估。我们相信在一系列数据集上的E2 E基准测试促进了多领域语音识别系统的研究。ESB可从www.example.com获得https://huggingface.co/esb。摘要：Speech recognition applications cover a range of different audio and text distributions, with different speaking styles, background noise, transcription punctuation and character casing. However, many speech recognition systems require dataset-specific tuning (audio filtering, punctuation removal and normalisation of casing), therefore assuming a-priori knowledge of both the audio and text distributions. This tuning requirement can lead to systems failing to generalise to other datasets and domains. To promote the development of multi-domain speech systems, we introduce the End-to-end Speech Benchmark (ESB) for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. Benchmarked systems must use the same data pre- and post-processing algorithm across datasets - assuming the audio and text data distributions are a-priori unknown. We compare a series of state-of-the-art (SoTA) end-to-end (E2E) systems on this benchmark, demonstrating how a single speech system can be applied and evaluated on a wide range of data distributions. We find E2E systems to be effective across datasets: in a fair comparison, E2E systems achieve within 2.6% of SoTA systems tuned to a specific dataset. Our analysis reveals that transcription artefacts, such as punctuation and casing, pose difficulties for ASR systems and should be included in evaluation. We believe E2E benchmarking over a range of datasets promotes the research of multi-domain speech recognition systems. ESB is available at https://huggingface.co/esb.
【15】 Proficiency assessment of L2 spoken English using wav2vec 2.0标题：用Wave2vec 2.0评估二语口语水平链接：https://arxiv.org/abs/2210.13168
* 与cs.SD语音【3】为同一篇
作者：Stefano Bannò,Marco Matassoni机构：Fondazione Bruno Kessler, Trento, Italy, University of Trento, Trento, Italy备注：Accepted at SLT 2022摘要：作为第二语言的英语学习需求的增长导致了对自动评估口语能力的方法的兴趣的增长。大多数方法使用手工制作的特征，但它们的有效性依赖于它们特定的潜在假设，并且它们有可能丢弃关于熟练程度的潜在显著信息。其它方法依赖于ASR系统产生的转录，其可能不提供学习者话语在特定场景中的忠实再现（例如，非母语儿童的自发言语）。此外，转录不产生任何关于相关方面的信息，例如语调、节奏或韵律。在本文中，我们研究了使用wav2vec 2.0评估两个小数据集（其中一个是公开的）的总体和个人方面的熟练程度。我们发现，这种方法明显优于基于BERT的基线系统，该基线系统在ASR和用于比较的手动转录上训练。摘要：The increasing demand for learning English as a second language has led to a growing interest in methods for automatically assessing spoken language proficiency. Most approaches use hand-crafted features, but their efficacy relies on their particular underlying assumptions and they risk discarding potentially salient information about proficiency. Other approaches rely on transcriptions produced by ASR systems which may not provide a faithful rendition of a learner's utterance in specific scenarios (e.g., non-native children's spontaneous speech). Furthermore, transcriptions do not yield any information about relevant aspects such as intonation, rhythm or prosody. In this paper, we investigate the use of wav2vec 2.0 for assessing overall and individual aspects of proficiency on two small datasets, one of which is publicly available. We find that this approach significantly outperforms the BERT-based baseline system trained on ASR and manual transcriptions used for comparison.
【16】 10 hours data is all you need标题：10小时的数据是您所需要的全部链接：https://arxiv.org/abs/2210.13067
* 与cs.SD语音【4】为同一篇
作者：Zeping Min,Qian Ge,Zhong Li机构：⋆ Peking University, † Microsoft Research Asia摘要：本文提出了一种新的伪普通话语音数据的生成方法CAMP（character audio mixed up），其目的是从字符尺度生成音频。本文还提出了一种建立适应CAMP的普通话文字尺度音频数据库的方法--META-AUDIO，该方法充分利用了音频数据，极大地提高了数据库的多样性。实验结果表明，该方法简单有效。例如，我们使用AISHELL-1中的10小时音频数据和CAMP生成的伪音频数据训练模型，并实现了具有竞争力的11.07个字符的错误率（CER）。此外，我们还使用AIDATATANG数据集中仅10小时的音频数据和CAMP生成的伪音频数据进行了训练，再次获得了具有竞争力的8.26 CER。摘要：We propose a novel procedure to generate pseudo mandarin speech data named as CAMP (character audio mix up), which aims at generating audio from a character scale. We also raise a method for building a mandarin character scale audio database adaptive to CAMP named as META-AUDIO, which makes full use of audio data and can greatly increase the data diversity of the database. Experiments show that our CAMP method is simple and quite effective. For example, we train models with 10 hours of audio data in AISHELL-1 and pseudo audio data generated by CAMP, and achieve a competitive 11.07 character error rate (CER). Besides, we also perform training with only 10 hours of audio data in AIDATATANG dataset and pseudo audio data generated by CAMP, which again achieves a competitive 8.26 CER.
【17】 Self-supervised Rewiring of Pre-trained Speech Encoders: Towards Faster Fine-tuning with Less Labels in Speech Processing标题：预先训练的语音编码器的自我监督重新布线：在语音处理中以更少的标签实现更快的微调链接：https://arxiv.org/abs/2210.13030
* 与cs.SD语音【5】为同一篇
作者：Hao Yang,Jinming Zhao,Gholamreza Haffari,Ehsan Shareghi机构：Department of Data Science & AI, Monash University备注：8 pages, 3 figures摘要：预先训练的语音变换器已经促进了各种语音处理任务的巨大成功。然而，为下游任务微调这些编码器需要足够大的训练数据来收敛或实现现有技术。在文本域中，这部分归因于预训练的变换器中的表示空间的次优性。在这项工作中，我们冷静地看待预先训练的语音编码器，并重新布线他们的表示空间，而不需要任何任务特定的标签。我们的方法利用音频输入的中性合成版本以及帧掩蔽来构造用于对比自监督学习的正对。当用于扩充wav2vec 2编码器时，我们观察到表示空间中的各向同性的一致改进。我们在6个语音处理任务上的实验表明，在任务微调过程中，收敛速度显著加快，并且任务性能持续改善，特别是在低资源环境下。摘要：Pre-trained speech Transformers have facilitated great success across various speech processing tasks. However, fine-tuning these encoders for downstream tasks require sufficiently large training data to converge or to achieve state-of-the-art. In text domain this has been partly attributed to sub-optimality of the representation space in pre-trained Transformers. In this work, we take a sober look into pre-trained speech encoders and rewire their representation space without requiring any task-specific labels. Our method utilises neutrally synthesised version of audio inputs along with frame masking to construct positive pairs for contrastive self-supervised learning. When used for augmenting the wav2vec 2 encoder, we observe consistent improvement of isotropy in the representation space. Our experiments on 6 speech processing tasks, exhibit a significant convergence speedup during task fine-tuning as well as consistent task improvement, specially in low-resource settings.
【18】 Investigating the effect of domain selection on automatic speech recognition performance: a case study on Bangladeshi Bangla标题：领域选择对自动语音识别性能影响的研究--以孟加拉国孟加拉为例链接：https://arxiv.org/abs/2210.12921
* 与cs.SD语音【6】为同一篇
作者：Ahnaf Mozib Samin,M. Humayan Kobir,Md. Mushtaq Shahriyar Rafee,M. Firoz Ahmed,Shafkat Kibria,M. Shahidur Rahman机构： Department of Computer Science and Engineering, Shahjalal University of Science and TechnologyThe performance of data-driven natural language processing systems is contingent upon the quality of corpora备注：To be submitted摘要：数据驱动的自然语言处理系统的性能取决于语料库的质量。然而，主要的语料库设计标准通常没有被充分地确定和检查，特别是在语音处理学科。语音语料库的开发需要额外关注清晰/嘈杂、朗读/自发、多说话者语音、口音/方言等。域选择也是语音语料库开发中的一个关键决策点。在本研究中，我们通过评估一个最先进的孟加拉语自动语音识别（ASR）模型，在一个新的多领域孟加拉语ASR评估基准- BanSpeech上，证明了领域选择的重要性，该基准包含7.2小时的语音和来自19个不同领域的9802句话。采用深度卷积神经网络（CNN）、层归一化技术和连接主义时态分类（CTC）损失准则，在低资源、形态丰富的孟加拉语SUBAK. KO语料库上训练ASR模型。实验结果表明，SUBAK.KO上的ASR模型在识别以自发语音为主的语音域时存在困难，并且存在大量的词汇表外（OOV）单词。另一方面，相同的ASR模型在读取语音域中执行得更好，并且包含更少的OOV单词。此外，我们还报告了我们对层归一化、输入特征提取、卷积层数等的实验结果，并在SUBAK.KO上设置基线。BanSpeech将公开提供，以满足对孟加拉语ASR具有挑战性的评估基准的需求。摘要：The performance of data-driven natural language processing systems is contingent upon the quality of corpora. However, principal corpus design criteria are often not identified and examined adequately, particularly in the speech processing discipline. Speech corpora development requires additional attention with regard to clean/noisy, read/spontaneous, multi-talker speech, accents/dialects, etc. Domain selection is also a crucial decision point in speech corpus development. In this study, we demonstrate the significance of domain selection by assessing a state-of-the-art Bangla automatic speech recognition (ASR) model on a novel multi-domain Bangladeshi Bangla ASR evaluation benchmark - BanSpeech, which contains 7.2 hours of speech and 9802 utterances from 19 distinct domains. The ASR model has been trained with deep convolutional neural network (CNN), layer normalization technique, and Connectionist Temporal Classification (CTC) loss criterion on SUBAK.KO, a mostly read speech corpus for the low-resource and morphologically rich language Bangla. Experimental evaluation reveals the ASR model on SUBAK.KO faces difficulty recognizing speech from domains with mostly spontaneous speech and has a high number of out-of-vocabulary (OOV) words. The same ASR model, on the other hand, performs better in read speech domains and contains fewer OOV words. In addition, we report the outcomes of our experiments with layer normalization, input feature extraction, number of convolutional layers, etc., and set a baseline on SUBAK.KO. The BanSpeech will be publicly available to meet the need for a challenging evaluation benchmark for Bangla ASR.
【19】 Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings标题：通过听力引导意义：口语句子嵌入的无监督学习链接：https://arxiv.org/abs/2210.12857
* 与cs.SD语音【7】为同一篇
作者：Jian Zhu,Zuoyu Tian,Yadong Liu,Cong Zhang,Chia-wen Lo机构：BUniversity of Michigan, Ann Arbor, University of British Columbia, XIndiana University Bloomington, QNewcastle University, MMax Planck Institute for Human Cognitive and Brain Sciences备注：Findings of EMNLP 2022摘要：直接从语音信号中推导出语义表示是一项极具挑战性的任务，但在语音挖掘和口语理解中有许多有用的应用。本研究探讨口语语意表征的无监督学习。通过将语音信号转换为由声学单元发现产生的隐藏单元，提出了WavEmbed，它是一种多模态序列自动编码器，可以从语音的密集表示中预测隐藏单元。其次，本文提出了S-HuBERT算法，该算法首先在隐单元上训练句子嵌入模型，然后通过对比学习将其知识传递给语音编码器，从而通过知识提取来提取语义。表现最好的模型与人类判断达到了中等相关性（0.5~0.6），而不依赖于任何标签或转录。此外，这些模型还可以容易地扩展以利用语音的文本转录来学习与人类注释强相关的好得多的语音嵌入。我们提出的方法适用于开发纯数据驱动的语音挖掘、索引和搜索系统。摘要：Inducing semantic representations directly from speech signals is a highly challenging task but has many useful applications in speech mining and spoken language understanding. This study tackles the unsupervised learning of semantic representations for spoken utterances. Through converting speech signals into hidden units generated from acoustic unit discovery, we propose WavEmbed, a multimodal sequential autoencoder that predicts hidden units from a dense representation of speech. Secondly, we also propose S-HuBERT to induce meaning through knowledge distillation, in which a sentence embedding model is first trained on hidden units and passes its knowledge to a speech encoder through contrastive learning. The best performing model achieves a moderate correlation (0.5~0.6) with human judgments, without relying on any labels or transcriptions. Furthermore, these models can also be easily extended to leverage textual transcriptions of speech to learn much better speech embeddings that are strongly correlated with human annotations. Our proposed methods are applicable to the development of purely data-driven systems for speech mining, indexing and search.
【20】 Clarinet: A Music Retrieval System标题：单簧管：一个音乐检索系统链接：https://arxiv.org/abs/2210.12648
* 与cs.SD语音【8】为同一篇
作者：Kshitij Alwadhi,Rohan Sharma,Siddhant Sharma机构：Dept. of Electrical Engineering, Indian Institute of Technology Delhi摘要：本文提出并实现了一种基于MIDI的音乐识别方法。我们的单簧管音乐检索系统是专为搜索钢琴MIDI文件与高召回率和速度。我们设计了一种新颖的旋律提取算法，使召回率提高了10%以上。我们还实现了三个检索算法--两个自行设计的算法（RSA Note和RSA Time），以及一个改进的Mongeau Sankoff算法。本文还讨论了实现速度和尺度不变性的算法。本白皮书还包含使用四种不同指标进行的详细实验和基准测试。单簧管的回忆率超过94%。摘要：A MIDI based approach for music recognition is proposed and implemented in this paper. Our Clarinet music retrieval system is designed to search piano MIDI files with high recall and speed. We design a novel melody extraction algorithm that improves recall results by more than 10%. We also implement 3 algorithms for retrieval-two self designed (RSA Note and RSA Time), and a modified version of the Mongeau Sankoff Algorithm. Algorithms to achieve tempo and scale invariance are also discussed in this paper. The paper also contains detailed experimentation and benchmarks with four different metrics. Clarinet achieves recall scores of more than 94%.
【21】 Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separation标题：用于目标说话人分离的登记说话人嵌入被忽略方面的定量证据链接：https://arxiv.org/abs/2210.12635
* 与cs.SD语音【9】为同一篇
作者：Xiaoyu Liu,Xu Li,Joan Serrà机构：Joan Serra, Dolby Laboratories摘要：单通道目标说话人分离（TSS）的目的是从多个说话人的混合语音中提取出说话人的语音。一个典型的深度学习TSS框架由一个获得注册说话人嵌入的上游模型和一个根据嵌入执行分离的下游模型组成。本文对语音识别嵌入技术中几个重要但被忽略的方面进行了研究，包括广泛使用的说话人识别嵌入技术的适用性，log-mel滤波器组和自监督嵌入技术的引入，以及嵌入技术的跨数据集泛化能力。实验结果表明，说话人识别嵌入可能会由于次优度量、训练目标或常规预处理而丢失相关信息。相比之下，滤波器组和自监督嵌入都保持了说话人信息的完整性，但在跨数据集评估中，前者始终优于后者。之前被忽略的滤波器组嵌入的竞争性分离和泛化性能在我们的研究中是一致的，这需要进一步研究更好的上游特征。摘要：Single channel target speaker separation (TSS) aims at extracting a speaker's voice from a mixture of multiple talkers given an enrollment utterance of that speaker. A typical deep learning TSS framework consists of an upstream model that obtains enrollment speaker embeddings and a downstream model that performs the separation conditioned on the embeddings. In this paper, we look into several important but overlooked aspects of the enrollment embeddings, including the suitability of the widely used speaker identification embeddings, the introduction of the log-mel filterbank and self-supervised embeddings, and the embeddings' cross-dataset generalization capability. Our results show that the speaker identification embeddings could lose relevant information due to a sub-optimal metric, training objective, or common pre-processing. In contrast, both the filterbank and the self-supervised embeddings preserve the integrity of the speaker information, but the former consistently outperforms the latter in a cross-dataset evaluation. The competitive separation and generalization performance of the previously overlooked filterbank embedding is consistent across our study, which calls for future research on better upstream features.
【22】 GCT: Gated Contextual Transformer for Sequential Audio Tagging标题：GCT：用于顺序音频标注的门控上下文转换器链接：https://arxiv.org/abs/2210.12541
* 与cs.SD语音【10】为同一篇
作者：Yuanbo Hou,Yun Wang,Wenwu Wang,Dick Botteldooren机构：WAVES, Ghent University, Belgium, Meta AI, USA, CVSSP, University of Surrey, UK摘要：音频标注是指为音频片段分配预定义的标签，以表示音频事件的类别信息。顺序音频标记（SAT）是指检测音频事件的类别信息以及它们在音频片段中出现的顺序。现有的SAT方法大多基于连接主义时态分类（CTC）。然而，由于不同时刻输出之间的条件独立性假设，CTC无法有效捕获事件之间的关联。上下文转换器（cTransformer）通过利用SAT中的上下文信息来解决这个问题。然而，cTransformer在利用上下文信息方面也受到限制，因为它在推理中仅使用前向信息。本文提出了一种基于前向-后向推理的门控上下文转换器。此外，在GCT中提出了一个门控上下文多层感知器（GCMLP）模块，从结构上改善了cTransformer的性能。在两个真实音频数据集上的实验结果表明，GCMLP与FBI相结合的GCT算法的性能优于基于CTC的算法和cTransformer.为了促进SAT的研究，发布了两个数据集的手动注释顺序标签。摘要：Audio tagging aims to assign predefined tags to audio clips to indicate the class information of audio events. Sequential audio tagging (SAT) means detecting both the class information of audio events, and the order in which they occur within the audio clip. Most existing methods for SAT are based on connectionist temporal classification (CTC). However, CTC cannot effectively capture connections between events due to the conditional independence assumption between outputs at different times. The contextual Transformer (cTransformer) addresses this issue by exploiting contextual information in SAT. Nevertheless, cTransformer is also limited in exploiting contextual information as it only uses forward information in inference. This paper proposes a gated contextual Transformer (GCT) with forward-backward inference (FBI). In addition, a gated contextual multi-layer perceptron (GCMLP) block is proposed in GCT to improve the performance of cTransformer structurally. Experiments on two real-life audio datasets show that the proposed GCT with GCMLP and FBI performs better than the CTC-based methods and cTransformer. To promote research on SAT, the manually annotated sequential labels for the two datasets are released.
【23】 Speech Emotion Recognition via an Attentive Time-Frequency Neural Network标题：基于时频神经网络的语音情感识别链接：https://arxiv.org/abs/2210.12430
* 与cs.SD语音【11】为同一篇
作者：Cheng Lu,Wenming Zheng,Hailun Lian,Yuan Zong,Chuangao Tang,Sunan Li,Yan Zhao备注：This paper has been accepted as a regular paper on IEEE Transactions on Computational Social Systems摘要：在语音情感识别中，语音谱图通常被用作深度神经网络的输入特征，用于学习语音信号的高层时频模式。\textcolor{black}{通常，不同的情感在谱图上对应于频带和时间帧内的特定能量激活，这表明对于SER，频域和时域都是表示情感的必要条件。然而，最近基于谱图的工作主要集中在时域中建模长期依赖性，导致这些方法遇到以下两个问题：（1）在时频联合学习过程中，忽略了对情感相关性在频域内的建模;（2）忽略捕捉与情绪相关的特定频带。}针对这一问题，提出了一种基于注意的时频神经网络（ATFNN），包括时频神经网络（TFNN）和时频注意。具体来说，针对第一个问题，我们设计了一个TFNN，其中频域编码器（F-Encoder）基于Transformer编码器，时域编码器（T-Encoder）基于双向长短时记忆（Bi-LSTM）。F-Encoder和T-Encoder分别对频带内和时间帧内的相关性进行建模，并将其嵌入时频联合学习策略中，以获得语音情感的时频模式。此外，为了解决第二个问题，我们还采用了时频注意，即频率-注意网络（F-Attention）和时间-注意网络（T-Attention）来关注与情感相关的频带范围和时间帧范围，从而提高了语音情感特征的区分度。摘要：Spectrogram is commonly used as the input feature of deep neural networks to learn the high(er)-level time-frequency pattern of speech signal for speech emotion recognition (SER). \textcolor{black}{Generally, different emotions correspond to specific energy activations both within frequency bands and time frames on spectrogram, which indicates the frequency and time domains are both essential to represent the emotion for SER. However, recent spectrogram-based works mainly focus on modeling the long-term dependency in time domain, leading to these methods encountering the following two issues: (1) neglecting to model the emotion-related correlations within frequency domain during the time-frequency joint learning; (2) ignoring to capture the specific frequency bands associated with emotions.} To cope with the issues, we propose an attentive time-frequency neural network (ATFNN) for SER, including a time-frequency neural network (TFNN) and time-frequency attention. Specifically, aiming at the first issue, we design a TFNN with a frequency-domain encoder (F-Encoder) based on the Transformer encoder and a time-domain encoder (T-Encoder) based on the Bidirectional Long Short-Term Memory (Bi-LSTM). The F-Encoder and T-Encoder model the correlations within frequency bands and time frames, respectively, and they are embedded into a time-frequency joint learning strategy to obtain the time-frequency patterns for speech emotions. Moreover, to handle the second issue, we also adopt time-frequency attention with a frequency-attention network (F-Attention) and a time-attention network (T-Attention) to focus on the emotion-related frequency band ranges and time frame ranges, which can enhance the discriminability of speech emotion features.
【24】 Information-Transport-based Policy for Simultaneous Translation标题：基于信息传输的同声传译策略链接：https://arxiv.org/abs/2210.12357
* 与cs.SD语音【12】为同一篇
作者：Shaolei Zhang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：Accept to EMNLP 2022 main conference. 22 pages, 16 figures, 8 tables摘要：同步翻译（ST）在接收源输入的同时输出翻译，因此需要一种策略来确定是翻译目标令牌还是等待下一个源令牌。ST的主要挑战是每个目标标记只能基于当前接收到的源标记进行翻译，其中接收到的源信息将直接影响翻译质量。因此，对于当前目标令牌的翻译，接收到多少源信息被认为是ST策略在翻译和等待之间做出决定的关键证据。本文将翻译看作是信息从源语言到目标语言的传递，并据此提出了基于信息传递的同声翻译。ITST量化从每个源令牌到当前目标令牌的传输信息权重，然后根据目标令牌累积的接收信息来决定是否翻译目标令牌。对文本到文本ST和语音到文本ST（也称为，流式语音翻译）任务的测试结果表明，ITST的性能优于强基线，并达到了最新的性能。摘要：Simultaneous translation (ST) outputs translation while receiving the source inputs, and hence requires a policy to determine whether to translate a target token or wait for the next source token. The major challenge of ST is that each target token can only be translated based on the current received source tokens, where the received source information will directly affect the translation quality. So naturally, how much source information is received for the translation of the current target token is supposed to be the pivotal evidence for the ST policy to decide between translating and waiting. In this paper, we treat the translation as information transport from source to target and accordingly propose an Information-Transport-based Simultaneous Translation (ITST). ITST quantifies the transported information weight from each source token to the current target token, and then decides whether to translate the target token according to its accumulated received information. Experiments on both text-to-text ST and speech-to-text ST (a.k.a., streaming speech translation) tasks show that ITST outperforms strong baselines and achieves state-of-the-art performance.
【25】 Neural Sound Field Decomposition with Super-resolution of Sound Direction标题：声向超分辨的神经声场分解链接：https://arxiv.org/abs/2210.12345
* 与cs.SD语音【13】为同一篇
作者：Qiuqiang Kong,Shilei Liu,Junjie Shi,Xuzhou Ye,Yin Cao,Qiaoxi Zhu,Yong Xu,Yuxuan Wang机构： ByteDance, Shanghai, China, University of Surrey, Guildford, UK, University of Technology Sydney, Sydney, Australia, Tencent AI Lab, Bellevue, USA备注：12 pages摘要：声场分解使用来自有限数量的麦克风的信号作为输入来预测任意方向上的波形。声场分解是下游任务的基础，包括源定位、源分离和空间音频再现。传统的声场分解方法，例如立体声音响，具有有限的空间分解分辨率。提出了一种基于学习的神经声场分解（NeSD）框架，利用几个麦克风在任意位置的拾音盒录音，实现了具有良好空间方向分辨率的声场分解。NeSD系统的输入包括麦克风信号、麦克风位置和所查询的方向。NeSD的输出包括查询位置的波形和存在概率。分别用全连接、时滞和递归神经网络对NeSD系统进行建模。我们证明了NeSD系统在语音、音乐和声音事件数据集上的声场分解和源定位方面优于传统的Ambisonics和DOANet方法。演示可在www.example.com上获得https://www.youtube.com/watch? v=0GIr6doj3BQ.摘要：Sound field decomposition predicts waveforms in arbitrary directions using signals from a limited number of microphones as inputs. Sound field decomposition is fundamental to downstream tasks, including source localization, source separation, and spatial audio reproduction. Conventional sound field decomposition methods such as Ambisonics have limited spatial decomposition resolution. This paper proposes a learning-based Neural Sound field Decomposition (NeSD) framework to allow sound field decomposition with fine spatial direction resolution, using recordings from microphone capsules of a few microphones at arbitrary positions. The inputs of a NeSD system include microphone signals, microphone positions, and queried directions. The outputs of a NeSD include the waveform and the presence probability of a queried position. We model the NeSD systems respectively with different neural networks, including fully connected, time delay, and recurrent neural networks. We show that the NeSD systems outperform conventional Ambisonics and DOANet methods in sound field decomposition and source localization on speech, music, and sound events datasets. Demos are available at https://www.youtube.com/watch?v=0GIr6doj3BQ.
【26】 Guided contrastive self-supervised pre-training for automatic speech recognition标题：用于自动语音识别的引导式对比自监督预训练链接：https://arxiv.org/abs/2210.12335
* 与cs.SD语音【14】为同一篇
作者：Aparna Khare,Minhua Wu,Saurabhchand Bhati,Jasha Droppo,Roland Maas机构：Amazon Alexa, USA, Johns Hopkins University, USA备注：To appear in SLT 2022摘要：对比预测编码（CPC）是一种表示学习方法，它最大化中间潜在表示和给定模型的输出之间的互信息。它可用于有效地初始化自动语音识别（ASR）模型的编码器。本文提出了一种新的CPC改进方案--引导对比预测编码（Guided Contrasive Predictive Coding，GCPC）。我们提出的方法最大化了来自先验知识模型的表示和被预训练的模型的输出之间的互信息，允许在预训练期间的先验知识注入。我们在3个ASR任务上验证了我们的方法：德语、法语和英语。在德语、法语和英语（Librispeech）三个语料库上，与从头开始训练相比，我们的方法在所有三个数据集上的表现都优于CPC预训练，相对于从头开始训练，单词错误率（WER）分别降低了4.44%、6.55%和15.43%，而CPC预训练仅分别带来了2.96%、1.01%和14.39%的相对WER降低。摘要：Contrastive Predictive Coding (CPC) is a representation learning method that maximizes the mutual information between intermediate latent representations and the output of a given model. It can be used to effectively initialize the encoder of an Automatic Speech Recognition (ASR) model. We present a novel modification of CPC called Guided Contrastive Predictive Coding (GCPC). Our proposed method maximizes the mutual information between representations from a prior-knowledge model and the output of the model being pre-trained, allowing prior knowledge injection during pre-training. We validate our method on 3 ASR tasks: German, French and English. Our method outperforms CPC pre-training on all three datasets, reducing the Word Error Rate (WER) by 4.44%, 6.55% and 15.43% relative on the German, French and English (Librispeech) tasks respectively, compared to training from scratch, while CPC pre-training only brings 2.96%, 1.01% and 14.39% relative WER reduction respectively.
【27】 BEANS: The Benchmark of Animal Sounds标题：豆类：动物声音的标杆链接：https://arxiv.org/abs/2210.12300
* 与cs.SD语音【15】为同一篇
作者：Masato Hagiwara,Benjamin Hoffman,Jen-Yu Liu,Maddie Cusimano,Felix Effenberger,Katie Zacarian机构：Earth Species Project摘要：近年来，基于机器学习的技术在生物声学领域中的应用越来越普遍。基于ML的技术的成功应用的基本要求是策划、商定、高质量的数据集和在给定数据集上学习的基准任务。然而，迄今为止，生物声学领域缺乏这样的公共基准，其覆盖多个任务和物种，以受控和标准化的方式测量ML技术的性能，并允许将新提出的技术与现有技术进行基准比较。在这里，我们提出BEANS（动物声音的BEnchmark），一个生物声学任务和公共数据集的集合，专门设计用于测量机器学习算法在生物声学领域的性能。这里提出的基准测试由生物声学中的两个常见任务组成：分类和检测。它包括12个数据集，涵盖各种物种，包括鸟类、陆地和海洋哺乳动物、无尾目动物和昆虫。除了数据集，我们还提出了一套标准的ML方法的性能作为任务性能的基线。基准测试和基线代码可在\url{https：//github.com/earthspecies/beans}上公开获取，希望为基于ML的生物声学研究建立一个新的标准数据集。摘要：The use of machine learning (ML) based techniques has become increasingly popular in the field of bioacoustics over the last years. Fundamental requirements for the successful application of ML based techniques are curated, agreed upon, high-quality datasets and benchmark tasks to be learned on a given dataset. However, the field of bioacoustics so far lacks such public benchmarks which cover multiple tasks and species to measure the performance of ML techniques in a controlled and standardized way and that allows for benchmarking newly proposed techniques to existing ones. Here, we propose BEANS (the BEnchmark of ANimal Sounds), a collection of bioacoustics tasks and public datasets, specifically designed to measure the performance of machine learning algorithms in the field of bioacoustics. The benchmark proposed here consists of two common tasks in bioacoustics: classification and detection. It includes 12 datasets covering various species, including birds, land and marine mammals, anurans, and insects. In addition to the datasets, we also present the performance of a set of standard ML methods as the baseline for task performance. The benchmark and baseline code is made publicly available at \url{https://github.com/earthspecies/beans} in the hope of establishing a new standard dataset for ML-based bioacoustic research.
【28】 Low-Resource Multilingual and Zero-Shot Multispeaker TTS标题：低资源多语言零激发多人话音合成技术链接：https://arxiv.org/abs/2210.12223
* 与cs.SD语音【16】为同一篇
作者：Florian Lux,Julia Koch,Ngoc Thang Vu机构：University of Stuttgart备注：Accepted to AACL 2022摘要：虽然用于文本到语音（TTS）的神经方法在建模多个说话人方面显示出巨大的进步，即使在零触发设置中，这些方法所需的数据量通常对于世界上6,000多种口语中的绝大多数来说是不可行的。在这项工作中，我们将零拍语音克隆和多语言低资源TTS的任务结合在一起。使用语言不可知元学习（LAML）过程和对TTS编码器的修改，我们表明系统可以仅使用5分钟的训练数据来学习说一种新语言，同时保持推断新学习语言中甚至看不见的说话者的声音的能力。我们使用客观指标以及人类研究，在可懂度、自然度和与目标说话人的相似性方面展示了我们提出的方法的成功，并提供我们的代码和开源训练模型。摘要：While neural methods for text-to-speech (TTS) have shown great advances in modeling multiple speakers, even in zero-shot settings, the amount of data needed for those approaches is generally not feasible for the vast majority of the world's over 6,000 spoken languages. In this work, we bring together the tasks of zero-shot voice cloning and multilingual low-resource TTS. Using the language agnostic meta learning (LAML) procedure and modifications to a TTS encoder, we show that it is possible for a system to learn speaking a new language using just 5 minutes of training data while retaining the ability to infer the voice of even unseen speakers in the newly learned language. We show the success of our proposed approach in terms of intelligibility, naturalness and similarity to target speaker using objective metrics as well as human studies and provide our code and trained models open source.
【29】 Optimizing Bilingual Neural Transducer with Synthetic Code-switching Text Generation标题：用合成语码转换文本生成优化双语神经传感器链接：https://arxiv.org/abs/2210.12214
* 与cs.SD语音【17】为同一篇
作者：Thien Nguyen,Nathalie Tran,Liuhui Deng,Thiago Fraga da Silva,Matthew Radzihovsky,Roger Hsiao,Henry Mason,Stefan Braun,Erik McDermott,Dogan Can,Pawel Swietojanski,Lyan Verwimp,Sibel Oyman,Tresi Arvizo,Honza Silovsky,Arnab Ghoshal,Mathieu Martel,Bharat Ram Ambati,Mohamed Ali机构：Apple备注：5 pages, 1 figure, submitted to ICASSP 2023, *: equal contributions摘要：语码转换是指在同一句话中使用多种语言的行为。在本研究中，我们研究如何优化一个基于神经传感器的双语自动语音识别（ASR）模型，用于语码转换语音。针对无监督语码转换语音训练ASR模型的场景，我们发现半监督训练和合成语码转换语音可以改善双语ASR系统对语码转换语音的处理。我们通过测量编码器特定的召回值来分析神经传感器的每个编码器如何对代码转换性能做出贡献，并在ASCEND数据集上评估我们的英语/汉语系统。我们的最终系统在ASCEND英语/普通话代码转换测试集上实现了25%的混合错误率（MER）--与以前的文献相比，MER绝对值降低了2.1%--同时在单语测试集上保持了良好的准确率。摘要：Code-switching describes the practice of using more than one language in the same sentence. In this study, we investigate how to optimize a neural transducer based bilingual automatic speech recognition (ASR) model for code-switching speech. Focusing on the scenario where the ASR model is trained without supervised code-switching data, we found that semi-supervised training and synthetic code-switched data can improve the bilingual ASR system on code-switching speech. We analyze how each of the neural transducer's encoders contributes towards code-switching performance by measuring encoder-specific recall values, and evaluate our English/Mandarin system on the ASCEND data set. Our final system achieves 25% mixed error rate (MER) on the ASCEND English/Mandarin code-switching test set -- reducing the MER by 2.1% absolute compared to the previous literature -- while maintaining good accuracy on the monolingual test sets.
机器翻译，仅供参考
每日语音速递