【1】 Effects of language mismatch in automatic forensic voice comparison using deep learning embeddings
标题：深度学习嵌入在法医语音自动比对中语言失配的影响
链接：https://arxiv.org/abs/2209.12602
作者：Dávid Sztahó,Attila Fejes机构：Budapest University of Technology and Economics, Budapest, Magyar tudósok körútja ,., Hungary, University of Public Service Doctoral School of Law Enforcement, Ludovika tér ,., Budapest, Hungary摘要：近10年来，说话人嵌入技术在法庭语音比对中得到了广泛的应用。大多数预训练的说话人嵌入都是在英语语料库上训练的，因为英语语料库容易获取。因此，语言依赖性可以是自动法庭语音比较中的重要因素，尤其是当目标语言在语言上非常不同时。有许多可用的商业系统，但它们的模型主要是在与目标语言不同的语言（主要是英语）上训练的。对于资源不足的语言，开发一个用于取证目的的语料库，其中包含足够多的说话者来训练深度学习模型，成本非常高。本研究的目的是调查一个在英语语料库上预先训练的模型是否可以用于一个目标低资源语言（这里是匈牙利语），不同于模型训练的目标语言。因此，在有和没有可疑（已知）说话者的说话者登记的情况下成对比较样本。两个语料库是专门为取证目的而开发的，第三个语料库是为传统的说话人确认而开发的。使用两种基于深度学习的说话人嵌入向量提取方法：x矢量和ECAPA-TDNN。在似然比框架下对说话人确认进行评估。对语言组合（建模、LR校准、评估）进行比较。通过minClrr和EER指标对结果进行评价。实验结果表明，该模型在不同语言的语料库中训练后，在语言不匹配的语料库中表现良好。此外，还考察了样本时长和说话风格对学习者口语表达的影响。发现所讨论的样本的持续时间越长，性能越好。此外，如果应用不同的说话风格，也没有真正的区别。摘要：In forensic voice comparison the speaker embedding has become widely popular in the last 10 years. Most of the pretrained speaker embeddings are trained on English corpora, because it is easily accessible. Thus, language dependency can be an important factor in automatic forensic voice comparison, especially when the target language is linguistically very different. There are numerous commercial systems available, but their models are mainly trained on a different language (mostly English) than the target language. In the case of a low-resource language, developing a corpus for forensic purposes containing enough speakers to train deep learning models is costly. This study aims to investigate whether a model pre-trained on English corpus can be used on a target low-resource language (here, Hungarian), different from the model is trained on. Also, often multiple samples are not available from the offender (unknown speaker). Therefore, samples are compared pairwise with and without speaker enrollment for suspect (known) speakers. Two corpora are applied that were developed especially for forensic purposes, and a third that is meant for traditional speaker verification. Two deep learning based speaker embedding vector extraction methods are used: the x-vector and ECAPA-TDNN. Speaker verification was evaluated in the likelihood-ratio framework. A comparison is made between the language combinations (modeling, LR calibration, evaluation). The results were evaluated by minCllr and EER metrics. It was found that the model pre-trained on a different language but on a corpus with a huge amount of speakers performs well on samples with language mismatch. The effect of sample durations and speaking styles were also examined. It was found that the longer the duration of the sample in question the better the performance is. Also, there is no real difference if various speaking styles are applied.
【2】 Speech Forensics: Blind Voice Mimicry Detection
标题：语音取证：盲音模仿检测
链接：https://arxiv.org/abs/2209.12573
作者：Sahar Al Ajmi,Khizar Hayat,Alaa M. Al Obaidi,Naresh Kumar,Munaf Najmuldeen,Baptiste Magnier机构：University of Nizwa, Sultanate of Oman, Euromov Digital Health in Motion, Univ Montpellier, IMT Mines Ales, Ales, France备注：11 pages, 4 figures (6 if you count subfigures), 2 tables摘要：音频是人类最常用的交流方式之一，但同时它也很容易被滥用来欺骗人们。随着人工智能的革命，相关技术现在几乎每个人都可以访问，从而使犯罪分子很容易实施犯罪和伪造。在这项工作中，我们引入了一种深度学习方法来开发一个分类器，该分类器将盲地将输入音频分类为真实音频或模仿音频。该模型在从大量音频数据集中提取的一组重要特征上进行训练，得到一个分类器，并在来自不同音频的同一组特征上进行测试。为这项工作创建了两个数据集;一个全英文数据集和一个混合数据集（阿拉伯文和英文）。这些数据集已经通过GitHub提供给www.example.com上的研究社区使用https://github.com/SaSs7/Dataset。为了进行比较，还通过人的观察对音频进行了分类，受试者是母语使用者。随后的结果是有趣的，并表现出可怕的准确性。摘要：Audio is one of the most used way of human communication, but at the same time it can be easily misused by to trick people. With the revolution of AI, the related technologies are now accessible to almost everyone thus making it simple for the criminals to commit crimes and forgeries. In this work, we introduce a deep learning method to develop a classifier that will blindly classify an input audio as real or mimicked. The proposed model was trained on a set of important features extracted from a large dataset of audios to get a classifier that was tested on the same set of features from different audios. Two datasets were created for this work; an all English data set and a mixed data set (Arabic and English). These datasets have been made available through GitHub for the use of the research community at https://github.com/SaSs7/Dataset. For the purpose of comparison, the audios were also classified through human inspection with the subjects being the native speakers. The ensued results were interesting and exhibited formidable accuracy.
【3】 Multi-Task Adversarial Training Algorithm for Multi-Speaker Neural Text-to-Speech
标题：多说话人神经文语转换的多任务对抗性训练算法
链接：https://arxiv.org/abs/2209.12549
作者：Yusuke Nakai,Yuki Saito,Kenta Udagawa,Hiroshi Saruwatari机构：Graduate School of Information Science and Technology, The University of Tokyo, Japan.备注：6 pages, 1 figure, Accepted for APSIPA ASC 2022摘要：提出了一种基于多任务对抗训练的多说话人神经文语转换（TTS）模型的训练算法.传统的基于生成式对抗网络（GAN）的训练算法通过减少自然语音和合成语音之间的统计差异来显著提高合成语音的质量。然而，该算法不能保证训练的TTS模型在合成未被包括在训练数据中的未看见的说话者的语音时的泛化性能。我们的算法交替训练两个深度神经网络：多任务鉴别器和多说话人神经TTS模型（即，GANs生成器）。训练鉴别器不仅区分自然语音和合成语音，而且验证输入语音的说话者是存在的还是不存在的（即，通过内插所见说话人的嵌入向量而新生成的）。同时，训练生成器使语音重构损失和对抗损失的加权和最小，以欺骗鉴别器，即使目标说话人不可见，也能得到高质量的多说话人TTS。实验结果表明，与传统的GANSpeech算法相比，该算法提高了合成语音的质量。摘要：We propose a novel training algorithm for a multi-speaker neural text-to-speech (TTS) model based on multi-task adversarial training. A conventional generative adversarial network (GAN)-based training algorithm significantly improves the quality of synthetic speech by reducing the statistical difference between natural and synthetic speech. However, the algorithm does not guarantee the generalization performance of the trained TTS model in synthesizing voices of unseen speakers who are not included in the training data. Our algorithm alternatively trains two deep neural networks: multi-task discriminator and multi-speaker neural TTS model (i.e., generator of GANs). The discriminator is trained not only to distinguish between natural and synthetic speech but also to verify the speaker of input speech is existent or non-existent (i.e., newly generated by interpolating seen speakers' embedding vectors). Meanwhile, the generator is trained to minimize the weighted sum of the speech reconstruction loss and adversarial loss for fooling the discriminator, which achieves high-quality multi-speaker TTS even if the target speaker is unseen. Experimental evaluation shows that our algorithm improves the quality of synthetic speech better than a conventional GANSpeech algorithm.
【4】 Multimodal Exponentially Modified Gaussian Oscillators
标题：多峰指数修正高斯振荡器
链接：https://arxiv.org/abs/2209.12202
机构：Member, IEEE, University of Bern, Bern, Switzerland备注：IEEE International Ultrasonic Symposium 2022摘要：声学建模服务于音频处理任务中的去噪、数据重构、基于模型的测试和分类。以前的工作通过多个高斯分布或单个非对称高斯曲线来处理波包络的信号参数化，这两种方法都不能充分好地表示叠加的回波。本文提出了一个三阶段的多模态指数修正高斯（MEMG）模型，该模型带有一个可选的振荡项，将捕获的回波视为时域上单变量概率分布的叠加。这样，可以完全恢复遭受伪影的合成超声信号，这由定量评估支持。进行真实数据实验以证明所获取的特征的分类能力，其中在不同时间点检测到对象反射。该代码可在www.example.com上获得https://github.com/hahnec/multimodal_emg。摘要：Acoustic modeling serves de-noising, data reconstruction, model-based testing and classification in audio processing tasks. Previous work dealt with signal parameterization of wave envelopes either by multiple Gaussian distributions or a single asymmetric Gaussian curve, which both fall short in representing super-imposed echoes sufficiently well. This study presents a three-stage Multimodal Exponentially Modified Gaussian (MEMG) model with an optional oscillating term that regards captured echoes as a superposition of univariate probability distributions in the temporal domain. With this, synthetic ultrasound signals suffering from artifacts can be fully recovered, which is backed by quantitative assessment. Real data experimentation is carried out to demonstrate the classification capability of the acquired features with object reflections being detected at different points in time. The code is available at https://github.com/hahnec/multimodal_emg.
【5】 Song Emotion Recognition: a Performance Comparison Between Audio Features and Artificial Neural Networks
标题：歌曲情感识别：音频特征与人工神经网络的性能比较
链接：https://arxiv.org/abs/2209.12045
作者：Karen Rosero,Arthur Nicholas dos Santos,Pedro Benevenuto Valadares,Bruno Sanches Masiero机构：School of Electrical and Computer Engineering, University of Campinas, Brazil摘要：当创作或表演歌曲时，歌手/词曲作者常常有通过歌曲表达感情或情绪的意图。对于人类来说，将音乐作品或表演中的情感与观众的主观感知相匹配可能是相当具有挑战性的。幸运的是，解决这个问题的机器学习方法更简单。通常，它需要一个数据集，从该数据集提取音频特征以将该信息呈现给数据驱动模型，该数据驱动模型又将训练以预测给定歌曲与目标情感匹配的概率是多少。在本文中，我们研究了最近出版的用于解决这个问题的最常见的特征和模型，揭示了哪些特征和模型最适合识别无伴奏合唱歌曲中的情感。摘要：When songs are composed or performed, there is often an intent by the singer/songwriter of expressing feelings or emotions through it. For humans, matching the emotiveness in a musical composition or performance with the subjective perception of an audience can be quite challenging. Fortunately, the machine learning approach for this problem is simpler. Usually, it takes a data-set, from which audio features are extracted to present this information to a data-driven model, that will, in turn, train to predict what is the probability that a given song matches a target emotion. In this paper, we studied the most common features and models used in recent publications to tackle this problem, revealing which ones are best suited for recognizing emotion in a cappella songs.
【6】 Unsupervised domain adaptation for speech recognition with unsupervised error correction
标题：用于具有无监督纠错的语音识别的无监督域自适应
链接：https://arxiv.org/abs/2209.12043
作者：Long Mai,Julie Carson-Berndsen机构：ML-Labs, School of Computer Science, University College Dublin, Ireland摘要：当转录来自看不见的域的音频时，自动语音识别（ASR）系统的转录质量显著降低。提出了一种无监督ASR域自适应纠错方法，旨在恢复域错配引起的转录错误.与依赖于转录音频进行训练的现有校正方法不同，我们的方法仅需要目标域的未标记数据，其中应用伪标记技术来生成校正训练样本。为了减少对伪数据的过拟合，我们还提出了一种编码器-解码器校正模型，该模型可以考虑诸如对话上下文和声学特征等附加信息。实验结果表明，与未自适应的ASR系统相比，该方法显著降低了误字率（WER）.校正模型还可以应用于其他适应方法之上，以带来相对10%的额外改进。摘要：The transcription quality of automatic speech recognition (ASR) systems degrades significantly when transcribing audios coming from unseen domains. We propose an unsupervised error correction method for unsupervised ASR domain adaption, aiming to recover transcription errors caused by domain mismatch. Unlike existing correction methods that rely on transcribed audios for training, our approach requires only unlabeled data of the target domains in which a pseudo-labeling technique is applied to generate correction training samples. To reduce over-fitting to the pseudo data, we also propose an encoder-decoder correction model that can take into account additional information such as dialogue context and acoustic features. Experiment results show that our method obtains a significant word error rate (WER) reduction over non-adapted ASR systems. The correction model can also be applied on top of other adaptation approaches to bring an additional improvement of 10% relatively.
【7】 Joint Speech Activity and Overlap Detection with Multi-Exit Architecture
标题：多出口结构下的联合语音活动和重叠检测
链接：https://arxiv.org/abs/2209.11906
作者：Ziqing Du,Kai Liu,Xucheng Wan,Huan Zhou机构：Artificial Intelligence Application Research Center, Huawei Technologies, Shenzhen, PRC摘要：重叠语音检测（OSD）是多方通话场景下语音应用的关键。尽管OSD的研究取得了许多进展，但与语音激活检测（VAD）相比，OSD仍是一个开放的挑战，其整体性能还远远不能令人满意。大多数现有研究通常将OSD问题表述为标准分类问题，以在帧级识别具有二进制（OSD）或三类标签（联合VAD和OSD）的语音。与主流研究不同，本研究从一个新的角度对VAD和OSD联合任务进行了研究。特别地，我们提出用多出口结构扩展传统分类网络。这样的体系结构使我们的系统具有独特的能力来使用来自早期退出的低级特征或来自最后退出的高级特征来识别类。此外，采用了知识提取和密集连接两种训练方案，进一步提高了系统性能。在标准数据集AMI和DIHARD-III上的实验结果验证了该系统的有效性和通用性.我们的消融进一步揭示了所提出方案的互补贡献。在AMI和DIHARD-III上的$F_1$得分分别为0.792和0.625，我们提出的系统在这些数据集上的性能优于几个性能最好的模型，而且在两个数据集上也以很大的幅度超过了当前最先进的模型。除了性能优势外，我们提出的系统还提供了另一种具有吸引力的质量-复杂性权衡潜力，这对于高效的OSD部署是非常优选的。摘要：Overlapped speech detection (OSD) is critical for speech applications in scenario of multi-party conversion. Despite numerous research efforts and progresses, comparing with speech activity detection (VAD), OSD remains an open challenge and its overall performance is far from satisfactory. The majority of prior research typically formulates the OSD problem as a standard classification problem, to identify speech with binary (OSD) or three-class label (joint VAD and OSD) at frame level. In contrast to the mainstream, this study investigates the joint VAD and OSD task from a new perspective. In particular, we propose to extend traditional classification network with multi-exit architecture. Such an architecture empowers our system with unique capability to identify class using either low-level features from early exits or high-level features from last exit. In addition, two training schemes, knowledge distillation and dense connection, are adopted to further boost our system performance. Experimental results on benchmark datasets (AMI and DIHARD-III) validated the effectiveness and generality of our proposed system. Our ablations further reveal the complementary contribution of proposed schemes. With $F_1$ score of 0.792 on AMI and 0.625 on DIHARD-III, our proposed system outperforms several top performing models on these datasets, but also surpasses the current state-of-the-art by large margins across both datasets. Besides the performance benefit, our proposed system offers another appealing potential for quality-complexity trade-offs, which is highly preferred for efficient OSD deployment.
【8】 Speech Enhancement with Perceptually-motivated Optimization and Dual Transformations
标题：基于感知激励优化和对偶变换的语音增强
链接：https://arxiv.org/abs/2209.11905
作者：Xucheng Wan,Kai Liu,Ziqing Du,Huan Zhou机构：Artificial Intelligence Application Research Center, Huawei Technologies, Shenzhen, PRC摘要：为了解决单耳语音增强问题，已经进行了大量的研究，以通过在时域中对从语音混合学习的内部域进行操作或者在时频域中对固定的全频带短时傅里叶变换（STFT）谱图进行操作来增强语音。最近，已经提出了一些关于基于子带的语音增强的研究。通过对子带频谱图进行运算来增强语音，这些研究在DNS 2020的基准数据集上表现出了有竞争力的性能。尽管这一新的研究方向颇具吸引力，但尚未得到充分的探索，仍有改进的空间。因此，本研究针对目前语音增强的最新研究方向，提出了一种基于子带的感知优化和双变换的语音增强系统，称为PT-FSE。特别地，我们提出的PT-FSE模型通过三个方面的努力改进了其主干--全带和子带融合模型。首先，我们设计了一个频率变换模块，旨在增强全局频率相关性。然后引入时间变换来捕获长范围的时间上下文。最后，提出了一种新的基于人类听觉感知特性的损失模型，使该模型更侧重于低频增强。为了验证所提模型的有效性，在DNS 2020数据集上进行了大量实验.实验结果表明，PT-FSE系统在性能上比现有的SOTA系统有了很大的提高，同时比SOTA系统的体积小了27%.我们的系统在基准数据集上的平均NB-PESQ为3.57，提供了迄今为止报告的最好的语音增强结果。摘要：To address the monaural speech enhancement problem, numerous research studies have been conducted to enhance speech via operations either in time-domain on the inner-domain learned from the speech mixture or in time--frequency domain on the fixed full-band short time Fourier transform (STFT) spectrograms. Very recently, a few studies on sub-band based speech enhancement have been proposed. By enhancing speech via operations on sub-band spectrograms, those studies demonstrated competitive performances on the benchmark dataset of DNS2020. Despite attractive, this new research direction has not been fully explored and there is still room for improvement. As such, in this study, we delve into the latest research direction and propose a sub-band based speech enhancement system with perceptually-motivated optimization and dual transformations, called PT-FSE. Specially, our proposed PT-FSE model improves its backbone, a full-band and sub-band fusion model, by three efforts. First, we design a frequency transformation module that aims to strengthen the global frequency correlation. Then a temporal transformation is introduced to capture long range temporal contexts. Lastly, a novel loss, with leverage of properties of human auditory perception, is proposed to facilitate the model to focus on low frequency enhancement. To validate the effectiveness of our proposed model, extensive experiments are conducted on the DNS2020 dataset. Experimental results show that our PT-FSE system achieves substantial improvements over its backbone, but also outperforms the current state-of-the-art while being 27\% smaller than the SOTA. With average NB-PESQ of 3.57 on the benchmark dataset, our system offers the best speech enhancement results reported till date.
【9】 Impact of temporal resolution on convolutional recurrent networks\\ for audio tagging and sound event detection
标题：时间分辨率对用于音频标记和声音事件检测的卷积递归网络的影响
链接：https://arxiv.org/abs/2209.12843
作者：Wim Boes,Hugo Van hamme机构：ESAT, KU Leuven, Belgium备注：Submitted to DCASE 2022 Workshop摘要：用于音频标记和声音事件检测的许多现有技术系统采用卷积递归神经架构。典型地，他们在平均教师设置中被训练以处理可用数据的异类注释。 在这项工作中，我们对改变这些卷积递归神经网络的时间分辨率--这可以通过简单地调整它们的池化操作来完成--如何影响它们的性能进行了彻底的分析。通过使用各种评估指标，我们研究了在涉及时间定位方面的不同需求的几种声音识别场景下调整该设计参数的效果。摘要：Many state-of-the-art systems for audio tagging and sound event detection employ convolutional recurrent neural architectures. Typically, they are trained in a mean teacher setting to deal with the heterogeneous annotation of the available data. In this work, we present a thorough analysis of how changing the temporal resolution of these convolutional recurrent neural networks - which can be done by simply adapting their pooling operations - impacts their performance. By using a variety of evaluation metrics, we investigate the effects of adapting this design parameter under several sound recognition scenarios involving different needs in terms of temporal localization.
【10】 Multi-encoder attention-based architectures for sound recognition with partial visual assistance
标题：基于多编码者注意力的部分视觉辅助声音识别体系结构
链接：https://arxiv.org/abs/2209.12826
作者：Wim Boes,Hugo Van hamme备注：Submitted to EURASIP Journal on Audio, Speech, and Music Processing摘要：大规模声音识别数据集通常由从多媒体库获得的声学记录组成。因此，通常可以利用音频以外的模态来改进为相关任务设计的模型的输出。然而，通常并非所有内容都可用于这样的集合的所有样本：例如，原始素材可能在某个点已经从源平台移除，因此，不能再获取非听觉特征。 我们证明了多编码器框架可以通过将该方法应用于基于注意力的深度学习系统来处理这一问题，这是目前声音识别领域最先进的一部分。更具体地说，我们证明了所提出的模型扩展可以成功地用于将部分可用的视觉信息结合到这种网络的操作过程中，这种网络通常在训练和推理过程中只使用听觉特征。实验上，我们验证了所考虑的方法在与音频标记和声音事件检测有关的多个评估场景中导致改进的预测。此外，我们还详细研究了所提出的技术的一些特性和局限性。摘要：Large-scale sound recognition data sets typically consist of acoustic recordings obtained from multimedia libraries. As a consequence, modalities other than audio can often be exploited to improve the outputs of models designed for associated tasks. Frequently, however, not all contents are available for all samples of such a collection: For example, the original material may have been removed from the source platform at some point, and therefore, non-auditory features can no longer be acquired. We demonstrate that a multi-encoder framework can be employed to deal with this issue by applying this method to attention-based deep learning systems, which are currently part of the state of the art in the domain of sound recognition. More specifically, we show that the proposed model extension can successfully be utilized to incorporate partially available visual information into the operational procedures of such networks, which normally only use auditory features during training and inference. Experimentally, we verify that the considered approach leads to improved predictions in a number of evaluation scenarios pertaining to audio tagging and sound event detection. Additionally, we scrutinize some properties and limitations of the presented technique.
【11】 Investigating self-supervised learning for lyrics recognition
标题：歌词识别中的自监督学习研究
链接：https://arxiv.org/abs/2209.12702
作者：Xiangyu Zhang,Leibny Paola Garcia,Zhanhong He,Shuyue Stella Li,Roberto Togneri机构：Johns Hopkins University, University of Western Australia摘要：歌词识别是音乐处理中的一项重要任务。尽管传统的HMM-TDNN模型等算法取得了很好的性能，但应用端到端模型和自监督学习（SSL）的研究还很有限。本文首先建立了歌词识别的端到端基线，然后研究了SSL模型的性能。我们基于四种上游SSL模型的训练方法（掩蔽重建、掩蔽预测、自回归重建、对比模型）对它们进行了评估。在应用SSL模型后，即使没有使用大型语料库训练的语言模型，与以前的最先进的基线系统相比，开发集的最佳性能提高了5.23%，测试集的最佳性能提高了2.4%。此外，考虑到这些模型没有在音乐数据集上训练，我们研究了SSL特征的泛化能力。摘要：Lyrics recognition is an important task in music processing. Despite the great number of traditional algorithms such as the hybrid HMM-TDNN model achieving good performance, studies on applying end-to-end models and self-supervised learning (SSL) are limited. In this paper, we first establish an end-to-end baseline for lyrics recognition and then explore the performance of SSL models. We evaluate four upstream SSL models based on their training method (masked reconstruction, masked prediction, autoregressive reconstruction, contrastive model). After applying the SSL model, the best performance improved by 5.23% for the dev set and 2.4% for the test set compared with the previous state-of-art baseline system even without a language model trained by a large corpus. Moreover, we study the generalization ability of the SSL features considering that those models were not trained on music datasets.
【12】 Spatial-aware Speaker Diarization for Multi-channel Multi-party Meeting
标题：多通道多方会议中的空间感知发言人二值化
链接：https://arxiv.org/abs/2209.12002
作者：Jie Wang,Yuji Liu,Binling Wang,Yiming Zhi,Song Li,Shipeng Xia,Jiayang Zhang,Feng Tong,Lin Li,Qingyang Hong机构：School of Electronic Science and Engineering, Xiamen University, China, School of Informatics, Xiamen University, China, College of Ocean and Earth Sciences, Xiamen University, Key Laboratory of Underwater Acoustic Communication and备注：Accepted by Interspeech 2022. arXiv admin note: text overlap with arXiv:2202.05744摘要：针对多通道多方会议，提出了一种基于空间感知的发言人识别系统。语音识别系统通过麦克风阵列获取说话人的方向信息。说话人空间嵌入由超定向波束形成（SDB）得到的x向量和s向量生成，使得嵌入更加鲁棒。提出了一种新的多通道序列到序列神经网络结构--区分多流神经网络（DMSNet），该结构由注意超定向波束形成（ASDB）模块和Conformer编码器组成。所提出的ASDB是一种自适应通道式模块，其通过对通道之间的相互依赖性进行建模来提取阵列音频的潜在空间特征。我们探索DMSNet来解决多声道音频上的重叠语音问题，在评估集上取得了93.53%的正确率。通过实现基于DMSNet的重叠语音检测（OSD）模块，使基于聚类的二值化系统的二值化错误率（DER）从13.45%显著降低到7.64%。摘要：This paper describes a spatial-aware speaker diarization system for the multi-channel multi-party meeting. The diarization system obtains direction information of speaker by microphone array. Speaker spatial embedding is generated by xvector and s-vector derived from superdirective beamforming (SDB) which makes the embedding more robust. Specifically, we propose a novel multi-channel sequence-to-sequence neural network architecture named discriminative multi-stream neural network (DMSNet) which consists of attention superdirective beamforming (ASDB) block and Conformer encoder. The proposed ASDB is a self-adapted channel-wise block that extracts the latent spatial features of array audios by modeling interdependencies between channels. We explore DMSNet to address overlapped speech problem on multi-channel audio and achieve 93.53% accuracy on evaluation set. By performing DMSNet based overlapped speech detection (OSD) module, the diarization error rate (DER) of cluster-based diarization system decrease significantly from 13.45% to 7.64%.
【13】 NWPU-ASLP System for the VoicePrivacy 2022 Challenge
标题：NWPU-针对语音隐私2022挑战的ASLP系统
链接：https://arxiv.org/abs/2209.11969
作者：Jixun Yao,Qing Wang,Li Zhang,Pengcheng Guo,Yuhao Liang,Lei Xie机构：Northwestern Polytechnical University, China备注：VoicePrivacy 2022 Challenge摘要：本文介绍了NWPU-ASLP的语音保密挑战赛发言人匿名系统。我们的提交不涉及额外的自动说话人验证（ASV）模型或x向量池。该系统由特征提取模块、声学模型模块、匿名化模块和神经声码器模块组成。首先，特征提取器从输入语音信号中提取语音后验图（PPG）和音调。然后，我们从说话人查找表（LUT）中保留一个伪说话人ID，该ID随后被馈送到说话人编码器中以生成不对应于任何真实说话人的伪说话人嵌入。为了保证伪说话人的可识别性，我们进一步对随机选择的说话人嵌入进行平均，并将其与伪说话人嵌入加权连接，生成匿名说话人嵌入。最后，声学模型输出来自匿名说话人嵌入的匿名mel谱图，并且HifiGAN的修改版本将mel谱图转换为匿名语音波形。实验结果验证了该匿名系统的有效性。摘要：This paper presents the NWPU-ASLP speaker anonymization system for VoicePrivacy 2022 Challenge. Our submission does not involve additional Automatic Speaker Verification (ASV) model or x-vector pool. Our system consists of four modules, including feature extractor, acoustic model, anonymization module, and neural vocoder. First, the feature extractor extracts the Phonetic Posteriorgram (PPG) and pitch from the input speech signal. Then, we reserve a pseudo speaker ID from a speaker look-up table (LUT), which is subsequently fed into a speaker encoder to generate the pseudo speaker embedding that is not corresponding to any real speaker. To ensure the pseudo speaker is distinguishable, we further average the randomly selected speaker embedding and weighted concatenate it with the pseudo speaker embedding to generate the anonymized speaker embedding. Finally, the acoustic model outputs the anonymized mel-spectrogram from the anonymized speaker embedding and a modified version of HifiGAN transforms the mel-spectrogram into the anonymized speech waveform. Experimental results demonstrate the effectiveness of our proposed anonymization system.
【14】 ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Rhythm
标题：ControlVC：音高和节奏随时间变化控制的零发声语音转换
链接：https://arxiv.org/abs/2209.11866
作者：Meiying Chen,Zhiyao Duan机构：Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA备注：Demo page with audio samples: this https URL摘要：近年来，神经语音合成和声码技术的发展重新激发了人们对语音转换的兴趣。除了音色传递之外，在许多应用场景中部署VC系统时，实现对诸如音高和节奏等副语言参数的可控性是至关重要的。然而，现有的研究要么只提供了话语层面的全局控制，要么缺乏对控制的可解释性。本文提出了ControlVC，这是第一个实现了对音高和节奏进行时变控制的神经语音转换系统。ControlVC使用预先训练的编码器来计算来自源话语的音高嵌入和语言嵌入以及来自目标话语的说话者嵌入。这些嵌入然后被连接并使用声码器转换成语音。该算法通过TD-PSOLA对源语音进行预处理，实现节奏控制;通过对基音轮廓进行处理，实现基音控制。通过系统的主观和客观评价来评估语音质量和可控性。实验结果表明，在非并行和zero-shot转换任务中，ControlVC的语音质量明显优于其他两种自构建的基线，并且能够成功实现时变基音控制。摘要：Recent developments in neural speech synthesis and vocoding have sparked a renewed interest in voice conversion (VC). Beyond timbre transfer, achieving controllability on para-linguistic parameters such as pitch and rhythm is critical in deploying VC systems in many application scenarios. Existing studies, however, either only provide utterance-level global control or lack interpretability on the controls. In this paper, we propose ControlVC, the first neural voice conversion system that achieves time-varying controls on pitch and rhythm. ControlVC uses pre-trained encoders to compute pitch embeddings and linguistic embeddings from the source utterance and speaker embeddings from the target utterance. These embeddings are then concatenated and converted to speech using a vocoder. It achieves rhythm control through TD-PSOLA pre-processing on the source utterance, and achieves pitch control by manipulating the pitch contour before feeding it to the pitch encoder. Systematic subjective and objective evaluations are conducted to assess the speech quality and controllability. Results show that, on non-parallel and zero-shot conversion tasks, ControlVC significantly outperforms two other self-constructed baselines on speech quality, and it can successfully achieve time-varying pitch control.
【1】 Impact of temporal resolution on convolutional recurrent networks\\ for audio tagging and sound event detection
标题：时间分辨率对用于音频标记和声音事件检测的卷积递归网络的影响
链接：https://arxiv.org/abs/2209.12843
* 与cs.SD语音【9】为同一篇
作者：Wim Boes,Hugo Van hamme机构：ESAT, KU Leuven, Belgium备注：Submitted to DCASE 2022 Workshop摘要：用于音频标记和声音事件检测的许多现有技术系统采用卷积递归神经架构。典型地，他们在平均教师设置中被训练以处理可用数据的异类注释。 在这项工作中，我们对改变这些卷积递归神经网络的时间分辨率--这可以通过简单地调整它们的池化操作来完成--如何影响它们的性能进行了彻底的分析。通过使用各种评估指标，我们研究了在涉及时间定位方面的不同需求的几种声音识别场景下调整该设计参数的效果。摘要：Many state-of-the-art systems for audio tagging and sound event detection employ convolutional recurrent neural architectures. Typically, they are trained in a mean teacher setting to deal with the heterogeneous annotation of the available data. In this work, we present a thorough analysis of how changing the temporal resolution of these convolutional recurrent neural networks - which can be done by simply adapting their pooling operations - impacts their performance. By using a variety of evaluation metrics, we investigate the effects of adapting this design parameter under several sound recognition scenarios involving different needs in terms of temporal localization.
【2】 Multi-encoder attention-based architectures for sound recognition with partial visual assistance
标题：基于多编码者注意力的部分视觉辅助声音识别体系结构
链接：https://arxiv.org/abs/2209.12826
* 与cs.SD语音【10】为同一篇
作者：Wim Boes,Hugo Van hamme备注：Submitted to EURASIP Journal on Audio, Speech, and Music Processing摘要：大规模声音识别数据集通常由从多媒体库获得的声学记录组成。因此，通常可以利用音频以外的模态来改进为相关任务设计的模型的输出。然而，通常并非所有内容都可用于这样的集合的所有样本：例如，原始素材可能在某个点已经从源平台移除，因此，不能再获取非听觉特征。 我们证明了多编码器框架可以通过将该方法应用于基于注意力的深度学习系统来处理这一问题，这是目前声音识别领域最先进的一部分。更具体地说，我们证明了所提出的模型扩展可以成功地用于将部分可用的视觉信息结合到这种网络的操作过程中，这种网络通常在训练和推理过程中只使用听觉特征。实验上，我们验证了所考虑的方法在与音频标记和声音事件检测有关的多个评估场景中导致改进的预测。此外，我们还详细研究了所提出的技术的一些特性和局限性。摘要：Large-scale sound recognition data sets typically consist of acoustic recordings obtained from multimedia libraries. As a consequence, modalities other than audio can often be exploited to improve the outputs of models designed for associated tasks. Frequently, however, not all contents are available for all samples of such a collection: For example, the original material may have been removed from the source platform at some point, and therefore, non-auditory features can no longer be acquired. We demonstrate that a multi-encoder framework can be employed to deal with this issue by applying this method to attention-based deep learning systems, which are currently part of the state of the art in the domain of sound recognition. More specifically, we show that the proposed model extension can successfully be utilized to incorporate partially available visual information into the operational procedures of such networks, which normally only use auditory features during training and inference. Experimentally, we verify that the considered approach leads to improved predictions in a number of evaluation scenarios pertaining to audio tagging and sound event detection. Additionally, we scrutinize some properties and limitations of the presented technique.
【3】 Investigating self-supervised learning for lyrics recognition
标题：歌词识别中的自监督学习研究
链接：https://arxiv.org/abs/2209.12702
* 与cs.SD语音【11】为同一篇
作者：Xiangyu Zhang,Leibny Paola Garcia,Zhanhong He,Shuyue Stella Li,Roberto Togneri机构：Johns Hopkins University, University of Western Australia摘要：歌词识别是音乐处理中的一项重要任务。尽管传统的HMM-TDNN模型等算法取得了很好的性能，但应用端到端模型和自监督学习（SSL）的研究还很有限。本文首先建立了歌词识别的端到端基线，然后研究了SSL模型的性能。我们基于四种上游SSL模型的训练方法（掩蔽重建、掩蔽预测、自回归重建、对比模型）对它们进行了评估。在应用SSL模型后，即使没有使用大型语料库训练的语言模型，与以前的最先进的基线系统相比，开发集的最佳性能提高了5.23%，测试集的最佳性能提高了2.4%。此外，考虑到这些模型没有在音乐数据集上训练，我们研究了SSL特征的泛化能力。摘要：Lyrics recognition is an important task in music processing. Despite the great number of traditional algorithms such as the hybrid HMM-TDNN model achieving good performance, studies on applying end-to-end models and self-supervised learning (SSL) are limited. In this paper, we first establish an end-to-end baseline for lyrics recognition and then explore the performance of SSL models. We evaluate four upstream SSL models based on their training method (masked reconstruction, masked prediction, autoregressive reconstruction, contrastive model). After applying the SSL model, the best performance improved by 5.23% for the dev set and 2.4% for the test set compared with the previous state-of-art baseline system even without a language model trained by a large corpus. Moreover, we study the generalization ability of the SSL features considering that those models were not trained on music datasets.
【4】 Spatial-aware Speaker Diarization for Multi-channel Multi-party Meeting
标题：多通道多方会议中的空间感知发言人二值化
链接：https://arxiv.org/abs/2209.12002
* 与cs.SD语音【12】为同一篇
作者：Jie Wang,Yuji Liu,Binling Wang,Yiming Zhi,Song Li,Shipeng Xia,Jiayang Zhang,Feng Tong,Lin Li,Qingyang Hong机构：School of Electronic Science and Engineering, Xiamen University, China, School of Informatics, Xiamen University, China, College of Ocean and Earth Sciences, Xiamen University, Key Laboratory of Underwater Acoustic Communication and备注：Accepted by Interspeech 2022. arXiv admin note: text overlap with arXiv:2202.05744摘要：针对多通道多方会议，提出了一种基于空间感知的发言人识别系统。语音识别系统通过麦克风阵列获取说话人的方向信息。说话人空间嵌入由超定向波束形成（SDB）得到的x向量和s向量生成，使得嵌入更加鲁棒。提出了一种新的多通道序列到序列神经网络结构--区分多流神经网络（DMSNet），该结构由注意超定向波束形成（ASDB）模块和Conformer编码器组成。所提出的ASDB是一种自适应通道式模块，其通过对通道之间的相互依赖性进行建模来提取阵列音频的潜在空间特征。我们探索DMSNet来解决多声道音频上的重叠语音问题，在评估集上取得了93.53%的正确率。通过实现基于DMSNet的重叠语音检测（OSD）模块，使基于聚类的二值化系统的二值化错误率（DER）从13.45%显著降低到7.64%。摘要：This paper describes a spatial-aware speaker diarization system for the multi-channel multi-party meeting. The diarization system obtains direction information of speaker by microphone array. Speaker spatial embedding is generated by xvector and s-vector derived from superdirective beamforming (SDB) which makes the embedding more robust. Specifically, we propose a novel multi-channel sequence-to-sequence neural network architecture named discriminative multi-stream neural network (DMSNet) which consists of attention superdirective beamforming (ASDB) block and Conformer encoder. The proposed ASDB is a self-adapted channel-wise block that extracts the latent spatial features of array audios by modeling interdependencies between channels. We explore DMSNet to address overlapped speech problem on multi-channel audio and achieve 93.53% accuracy on evaluation set. By performing DMSNet based overlapped speech detection (OSD) module, the diarization error rate (DER) of cluster-based diarization system decrease significantly from 13.45% to 7.64%.
【5】 NWPU-ASLP System for the VoicePrivacy 2022 Challenge
标题：NWPU-针对语音隐私2022挑战的ASLP系统
链接：https://arxiv.org/abs/2209.11969
* 与cs.SD语音【13】为同一篇
作者：Jixun Yao,Qing Wang,Li Zhang,Pengcheng Guo,Yuhao Liang,Lei Xie机构：Northwestern Polytechnical University, China备注：VoicePrivacy 2022 Challenge摘要：本文介绍了NWPU-ASLP的语音保密挑战赛发言人匿名系统。我们的提交不涉及额外的自动说话人验证（ASV）模型或x向量池。该系统由特征提取模块、声学模型模块、匿名化模块和神经声码器模块组成。首先，特征提取器从输入语音信号中提取语音后验图（PPG）和音调。然后，我们从说话人查找表（LUT）中保留一个伪说话人ID，该ID随后被馈送到说话人编码器中以生成不对应于任何真实说话人的伪说话人嵌入。为了保证伪说话人的可识别性，我们进一步对随机选择的说话人嵌入进行平均，并将其与伪说话人嵌入加权连接，生成匿名说话人嵌入。最后，声学模型输出来自匿名说话人嵌入的匿名mel谱图，并且HifiGAN的修改版本将mel谱图转换为匿名语音波形。实验结果验证了该匿名系统的有效性。摘要：This paper presents the NWPU-ASLP speaker anonymization system for VoicePrivacy 2022 Challenge. Our submission does not involve additional Automatic Speaker Verification (ASV) model or x-vector pool. Our system consists of four modules, including feature extractor, acoustic model, anonymization module, and neural vocoder. First, the feature extractor extracts the Phonetic Posteriorgram (PPG) and pitch from the input speech signal. Then, we reserve a pseudo speaker ID from a speaker look-up table (LUT), which is subsequently fed into a speaker encoder to generate the pseudo speaker embedding that is not corresponding to any real speaker. To ensure the pseudo speaker is distinguishable, we further average the randomly selected speaker embedding and weighted concatenate it with the pseudo speaker embedding to generate the anonymized speaker embedding. Finally, the acoustic model outputs the anonymized mel-spectrogram from the anonymized speaker embedding and a modified version of HifiGAN transforms the mel-spectrogram into the anonymized speech waveform. Experimental results demonstrate the effectiveness of our proposed anonymization system.
【6】 Unsupervised active speaker detection in media content using cross-modal information
标题：使用跨模式信息的媒体内容中的无监督主动说话人检测
链接：https://arxiv.org/abs/2209.11896
作者：Rahul Sharma,Shrikanth Narayanan机构：Universityof Southern California备注：Under review at IEEE Transactions on Image Processing摘要：提出了一种跨模态无监督的主动说话人检测框架，用于检测电视节目和电影等媒体内容中的主动说话人。机器学习的进步使得从语音和面部图像中识别个人的性能令人印象深刻。我们利用语音和人脸的说话人身份信息，将主动说话人检测公式化为一个语音-人脸分配任务，使得主动说话人的人脸和潜在的语音识别同一个人（角色）。我们根据语音段与所有其它语音段的相关说话者身份距离来表示语音段，以捕获视频的相对身份结构。然后，我们从同时出现的面部中为每个语音片段分配活动说话人的面部，使得所获得的活动说话人面部集合显示相似的相对身份结构。此外，我们提出了一种简单有效的方法来解决说话人出现在屏幕外的语音片段。我们在三个基准数据集-- Visual Person Clustering数据集、AVA活动说话人数据集和Columbia数据集--上对所提出的系统进行了评估，结果表明，该系统的性能与现有的全监督方法相当.摘要：We present a cross-modal unsupervised framework for active speaker detection in media content such as TV shows and movies. Machine learning advances have enabled impressive performance in identifying individuals from speech and facial images. We leverage speaker identity information from speech and faces, and formulate active speaker detection as a speech-face assignment task such that the active speaker's face and the underlying speech identify the same person (character). We express the speech segments in terms of their associated speaker identity distances, from all other speech segments, to capture a relative identity structure for the video. Then we assign an active speaker's face to each speech segment from the concurrently appearing faces such that the obtained set of active speaker faces displays a similar relative identity structure. Furthermore, we propose a simple and effective approach to address speech segments where speakers are present off-screen. We evaluate the proposed system on three benchmark datasets -- Visual Person Clustering dataset, AVA-active speaker dataset, and Columbia dataset -- consisting of videos from entertainment and broadcast media, and show competitive performance to state-of-the-art fully supervised methods.
【7】 ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Rhythm
标题：ControlVC：音高和节奏随时间变化控制的零发声语音转换
链接：https://arxiv.org/abs/2209.11866
* 与cs.SD语音【14】为同一篇
作者：Meiying Chen,Zhiyao Duan机构：Department of Electrical and Computer Engineering, University of Rochester, Rochester, NY, USA备注：Demo page with audio samples: this https URL摘要：近年来，神经语音合成和声码技术的发展重新激发了人们对语音转换的兴趣。除了音色传递之外，在许多应用场景中部署VC系统时，实现对诸如音高和节奏等副语言参数的可控性是至关重要的。然而，现有的研究要么只提供了话语层面的全局控制，要么缺乏对控制的可解释性。本文提出了ControlVC，这是第一个实现了对音高和节奏进行时变控制的神经语音转换系统。ControlVC使用预先训练的编码器来计算来自源话语的音高嵌入和语言嵌入以及来自目标话语的说话者嵌入。这些嵌入然后被连接并使用声码器转换成语音。该算法通过TD-PSOLA对源语音进行预处理，实现节奏控制;通过对基音轮廓进行处理，实现基音控制。通过系统的主观和客观评价来评估语音质量和可控性。实验结果表明，在非并行和zero-shot转换任务中，ControlVC的语音质量明显优于其他两种自构建的基线，并且能够成功实现时变基音控制。摘要：Recent developments in neural speech synthesis and vocoding have sparked a renewed interest in voice conversion (VC). Beyond timbre transfer, achieving controllability on para-linguistic parameters such as pitch and rhythm is critical in deploying VC systems in many application scenarios. Existing studies, however, either only provide utterance-level global control or lack interpretability on the controls. In this paper, we propose ControlVC, the first neural voice conversion system that achieves time-varying controls on pitch and rhythm. ControlVC uses pre-trained encoders to compute pitch embeddings and linguistic embeddings from the source utterance and speaker embeddings from the target utterance. These embeddings are then concatenated and converted to speech using a vocoder. It achieves rhythm control through TD-PSOLA pre-processing on the source utterance, and achieves pitch control by manipulating the pitch contour before feeding it to the pitch encoder. Systematic subjective and objective evaluations are conducted to assess the speech quality and controllability. Results show that, on non-parallel and zero-shot conversion tasks, ControlVC significantly outperforms two other self-constructed baselines on speech quality, and it can successfully achieve time-varying pitch control.
【8】 Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers
标题：FAST-FNET：通过高效傅立叶层加速Transformer编码器模型
链接：https://arxiv.org/abs/2209.12816
作者：Nurullah Sevim,Ege Ozan Özyedek,Furkan Şahinuç,Aykut Koç机构：Koc¸ are with the Department of Electrical and ElectronicsEngineering, Bilkent University备注：Submitted to IEEE Transactions on Audio, Speech and Language Processing. 12 pages摘要：基于转换器的语言模型在几乎所有的自然语言处理（NLP）任务中都利用了注意机制来显著提高性能。类似的注意力结构也在其他几个领域得到了广泛的研究。虽然注意机制显著提高了模型的性能，但其二次复杂度阻碍了对长序列的有效处理。最近的工作集中在消除计算效率低的缺点，并表明基于变压器的模型仍然可以达到竞争的结果没有注意层。一项开创性的研究提出了FNet，它用Transformer编码器架构中的傅立叶变换（FT）来代替注意层。FNet在消除注意机制的计算负担的同时，实现了与原始Transformer编码器模型的竞争性能，同时加快了训练过程。然而，FNet模型忽略了来自经典信号处理的FT的基本属性，其可以被利用来进一步提高模型效率。我们提出了不同的方法来在Transformer编码器模型中有效地部署FT。我们提出的架构具有更少的模型参数、更短的训练时间、更少的内存使用，以及一些额外的性能改进。我们通过在常见基准测试上进行大量实验来演示这些改进。摘要：Transformer-based language models utilize the attention mechanism for substantial performance improvements in almost all natural language processing (NLP) tasks. Similar attention structures are also extensively studied in several other areas. Although the attention mechanism enhances the model performances significantly, its quadratic complexity prevents efficient processing of long sequences. Recent works focused on eliminating the disadvantages of computational inefficiency and showed that transformer-based models can still reach competitive results without the attention layer. A pioneering study proposed the FNet, which replaces the attention layer with the Fourier Transform (FT) in the transformer encoder architecture. FNet achieves competitive performances concerning the original transformer encoder model while accelerating training process by removing the computational burden of the attention mechanism. However, the FNet model ignores essential properties of the FT from the classical signal processing that can be leveraged to increase model efficiency further. We propose different methods to deploy FT efficiently in transformer encoder models. Our proposed architectures have smaller number of model parameters, shorter training times, less memory usage, and some additional performance improvements. We demonstrate these improvements through extensive experiments on common benchmarks.
【9】 AI-powered Language Assessment Tools for Dementia
标题：人工智能支持的痴呆症语言评估工具
链接：https://arxiv.org/abs/2209.12652
作者：Mahboobeh Parsapoor,Muhammad Raisul Alam,Alex Mihailidis机构：CRIM, Montreal, Department Occupational Science and Occupational Therapy, University of Toronto备注：27 Pages, 11 Tables, 16 Figure摘要：本文的主要目标是提出一种开发人工智能语言评估工具的方法。这些工具可用于评估老年人痴呆症相关的语言障碍。机器学习（ML）分类器是我们提出的方法的主要部分，因此为了开发具有高灵敏度和特异性的准确工具，我们考虑了不同的二元分类器并评估它们的性能。我们还通过比较不同类型的语言任务、特征和记录介质对ML分类器性能的影响来评估我们方法的信度和效度。摘要：The main objective of this paper is to propose an approach for developing an Artificial Intelligence (AI)-powered Language Assessment (LA) tool. Such tools can be used to assess language impairments associated with dementia in older adults. The Machine Learning (ML) classifiers are the main parts of our proposed approach, therefore to develop an accurate tool with high sensitivity and specificity, we consider different binary classifiers and evaluate their performances. We also assess the reliability and validity of our approach by comparing the impact of different types of language tasks, features, and recording media on the performance of ML classifiers.
【10】 Bangla-Wave: Improving Bangla Automatic Speech Recognition Utilizing N-gram Language Models
标题：孟加拉浪潮：利用N元语法语言模型改进孟加拉自动语音识别
链接：https://arxiv.org/abs/2209.12650
作者：Mohammed Rakib,Md. Ismail Hossain,Nabeel Mohammed,Fuad Rahman机构：∗Apurba-NSU R&D Lab, Department of Electrical and Computer Engineering, North South University, Dhaka, Bangladesh, †Apurba Technologies, N. Wolfe Rd., Sunnyvale, CA , USA摘要：虽然全世界有超过3亿人说孟加拉语，但由于孟加拉语是一种资源匮乏的语言，在提高孟加拉语语音到文本的转录方面做得很少。然而，随着孟加拉语Common Voice 9.0语音数据集的引入，自动语音识别（ASR）模型现在可以得到显著改进。拥有399小时的语音录音，孟加拉语共同之声是世界上最大和最多样化的开源孟加拉语语音语料库。在本文中，我们通过在普通语音数据集上微调预训练的wav2vec2模型，从而优于SOTA预训练的孟加拉语ASR模型。我们还演示了如何通过添加n元语法语言模型作为后处理器来显著提高ASR模型的性能。最后，通过实验和超参数调整，得到了一个鲁棒的Bangla ASR模型，该模型优于现有的ASR模型。摘要：Although over 300M around the world speak Bangla, scant work has been done in improving Bangla voice-to-text transcription due to Bangla being a low-resource language. However, with the introduction of the Bengali Common Voice 9.0 speech dataset, Automatic Speech Recognition (ASR) models can now be significantly improved. With 399hrs of speech recordings, Bengali Common Voice is the largest and most diversified open-source Bengali speech corpus in the world. In this paper, we outperform the SOTA pretrained Bengali ASR models by finetuning a pretrained wav2vec2 model on the common voice dataset. We also demonstrate how to significantly improve the performance of an ASR model by adding an n-gram language model as a post-processor. Finally, we do some experiments and hyperparameter tuning to generate a robust Bangla ASR model that is better than the existing ASR models.
【11】 Effects of language mismatch in automatic forensic voice comparison using deep learning embeddings
标题：深度学习嵌入在法医语音自动比对中语言失配的影响
链接：https://arxiv.org/abs/2209.12602
* 与cs.SD语音【1】为同一篇
作者：Dávid Sztahó,Attila Fejes机构：Budapest University of Technology and Economics, Budapest, Magyar tudósok körútja ,., Hungary, University of Public Service Doctoral School of Law Enforcement, Ludovika tér ,., Budapest, Hungary摘要：近10年来，说话人嵌入技术在法庭语音比对中得到了广泛的应用。大多数预训练的说话人嵌入都是在英语语料库上训练的，因为英语语料库容易获取。因此，语言依赖性可以是自动法庭语音比较中的重要因素，尤其是当目标语言在语言上非常不同时。有许多可用的商业系统，但它们的模型主要是在与目标语言不同的语言（主要是英语）上训练的。对于资源不足的语言，开发一个用于取证目的的语料库，其中包含足够多的说话者来训练深度学习模型，成本非常高。本研究的目的是调查一个在英语语料库上预先训练的模型是否可以用于一个目标低资源语言（这里是匈牙利语），不同于模型训练的目标语言。因此，在有和没有可疑（已知）说话者的说话者登记的情况下成对比较样本。两个语料库是专门为取证目的而开发的，第三个语料库是为传统的说话人确认而开发的。使用两种基于深度学习的说话人嵌入向量提取方法：x矢量和ECAPA-TDNN。在似然比框架下对说话人确认进行评估。对语言组合（建模、LR校准、评估）进行比较。通过minClrr和EER指标对结果进行评价。实验结果表明，该模型在不同语言的语料库中训练后，在语言不匹配的语料库中表现良好。此外，还考察了样本时长和说话风格对学习者口语表达的影响。发现所讨论的样本的持续时间越长，性能越好。此外，如果应用不同的说话风格，也没有真正的区别。摘要：In forensic voice comparison the speaker embedding has become widely popular in the last 10 years. Most of the pretrained speaker embeddings are trained on English corpora, because it is easily accessible. Thus, language dependency can be an important factor in automatic forensic voice comparison, especially when the target language is linguistically very different. There are numerous commercial systems available, but their models are mainly trained on a different language (mostly English) than the target language. In the case of a low-resource language, developing a corpus for forensic purposes containing enough speakers to train deep learning models is costly. This study aims to investigate whether a model pre-trained on English corpus can be used on a target low-resource language (here, Hungarian), different from the model is trained on. Also, often multiple samples are not available from the offender (unknown speaker). Therefore, samples are compared pairwise with and without speaker enrollment for suspect (known) speakers. Two corpora are applied that were developed especially for forensic purposes, and a third that is meant for traditional speaker verification. Two deep learning based speaker embedding vector extraction methods are used: the x-vector and ECAPA-TDNN. Speaker verification was evaluated in the likelihood-ratio framework. A comparison is made between the language combinations (modeling, LR calibration, evaluation). The results were evaluated by minCllr and EER metrics. It was found that the model pre-trained on a different language but on a corpus with a huge amount of speakers performs well on samples with language mismatch. The effect of sample durations and speaking styles were also examined. It was found that the longer the duration of the sample in question the better the performance is. Also, there is no real difference if various speaking styles are applied.
【12】 Speech Forensics: Blind Voice Mimicry Detection
标题：语音取证：盲音模仿检测
链接：https://arxiv.org/abs/2209.12573
* 与cs.SD语音【2】为同一篇
作者：Sahar Al Ajmi,Khizar Hayat,Alaa M. Al Obaidi,Naresh Kumar,Munaf Najmuldeen,Baptiste Magnier机构：University of Nizwa, Sultanate of Oman, Euromov Digital Health in Motion, Univ Montpellier, IMT Mines Ales, Ales, France备注：11 pages, 4 figures (6 if you count subfigures), 2 tables摘要：音频是人类最常用的交流方式之一，但同时它也很容易被滥用来欺骗人们。随着人工智能的革命，相关技术现在几乎每个人都可以访问，从而使犯罪分子很容易实施犯罪和伪造。在这项工作中，我们引入了一种深度学习方法来开发一个分类器，该分类器将盲地将输入音频分类为真实音频或模仿音频。该模型在从大量音频数据集中提取的一组重要特征上进行训练，得到一个分类器，并在来自不同音频的同一组特征上进行测试。为这项工作创建了两个数据集;一个全英文数据集和一个混合数据集（阿拉伯文和英文）。这些数据集已经通过GitHub提供给www.example.com上的研究社区使用https://github.com/SaSs7/Dataset。为了进行比较，还通过人的观察对音频进行了分类，受试者是母语使用者。随后的结果是有趣的，并表现出可怕的准确性。摘要：Audio is one of the most used way of human communication, but at the same time it can be easily misused by to trick people. With the revolution of AI, the related technologies are now accessible to almost everyone thus making it simple for the criminals to commit crimes and forgeries. In this work, we introduce a deep learning method to develop a classifier that will blindly classify an input audio as real or mimicked. The proposed model was trained on a set of important features extracted from a large dataset of audios to get a classifier that was tested on the same set of features from different audios. Two datasets were created for this work; an all English data set and a mixed data set (Arabic and English). These datasets have been made available through GitHub for the use of the research community at https://github.com/SaSs7/Dataset. For the purpose of comparison, the audios were also classified through human inspection with the subjects being the native speakers. The ensued results were interesting and exhibited formidable accuracy.
【13】 Multi-Task Adversarial Training Algorithm for Multi-Speaker Neural Text-to-Speech
标题：多说话人神经文语转换的多任务对抗性训练算法
链接：https://arxiv.org/abs/2209.12549
* 与cs.SD语音【3】为同一篇
作者：Yusuke Nakai,Yuki Saito,Kenta Udagawa,Hiroshi Saruwatari机构：Graduate School of Information Science and Technology, The University of Tokyo, Japan.备注：6 pages, 1 figure, Accepted for APSIPA ASC 2022摘要：提出了一种基于多任务对抗训练的多说话人神经文语转换（TTS）模型的训练算法.传统的基于生成式对抗网络（GAN）的训练算法通过减少自然语音和合成语音之间的统计差异来显著提高合成语音的质量。然而，该算法不能保证训练的TTS模型在合成未被包括在训练数据中的未看见的说话者的语音时的泛化性能。我们的算法交替训练两个深度神经网络：多任务鉴别器和多说话人神经TTS模型（即，GANs生成器）。训练鉴别器不仅区分自然语音和合成语音，而且验证输入语音的说话者是存在的还是不存在的（即，通过内插所见说话人的嵌入向量而新生成的）。同时，训练生成器使语音重构损失和对抗损失的加权和最小，以欺骗鉴别器，即使目标说话人不可见，也能得到高质量的多说话人TTS。实验结果表明，与传统的GANSpeech算法相比，该算法提高了合成语音的质量。摘要：We propose a novel training algorithm for a multi-speaker neural text-to-speech (TTS) model based on multi-task adversarial training. A conventional generative adversarial network (GAN)-based training algorithm significantly improves the quality of synthetic speech by reducing the statistical difference between natural and synthetic speech. However, the algorithm does not guarantee the generalization performance of the trained TTS model in synthesizing voices of unseen speakers who are not included in the training data. Our algorithm alternatively trains two deep neural networks: multi-task discriminator and multi-speaker neural TTS model (i.e., generator of GANs). The discriminator is trained not only to distinguish between natural and synthetic speech but also to verify the speaker of input speech is existent or non-existent (i.e., newly generated by interpolating seen speakers' embedding vectors). Meanwhile, the generator is trained to minimize the weighted sum of the speech reconstruction loss and adversarial loss for fooling the discriminator, which achieves high-quality multi-speaker TTS even if the target speaker is unseen. Experimental evaluation shows that our algorithm improves the quality of synthetic speech better than a conventional GANSpeech algorithm.
【14】 Multimodal Exponentially Modified Gaussian Oscillators
标题：多峰指数修正高斯振荡器
链接：https://arxiv.org/abs/2209.12202
* 与cs.SD语音【4】为同一篇
机构：Member, IEEE, University of Bern, Bern, Switzerland备注：IEEE International Ultrasonic Symposium 2022摘要：声学建模服务于音频处理任务中的去噪、数据重构、基于模型的测试和分类。以前的工作通过多个高斯分布或单个非对称高斯曲线来处理波包络的信号参数化，这两种方法都不能充分好地表示叠加的回波。本文提出了一个三阶段的多模态指数修正高斯（MEMG）模型，该模型带有一个可选的振荡项，将捕获的回波视为时域上单变量概率分布的叠加。这样，可以完全恢复遭受伪影的合成超声信号，这由定量评估支持。进行真实数据实验以证明所获取的特征的分类能力，其中在不同时间点检测到对象反射。该代码可在www.example.com上获得https://github.com/hahnec/multimodal_emg。摘要：Acoustic modeling serves de-noising, data reconstruction, model-based testing and classification in audio processing tasks. Previous work dealt with signal parameterization of wave envelopes either by multiple Gaussian distributions or a single asymmetric Gaussian curve, which both fall short in representing super-imposed echoes sufficiently well. This study presents a three-stage Multimodal Exponentially Modified Gaussian (MEMG) model with an optional oscillating term that regards captured echoes as a superposition of univariate probability distributions in the temporal domain. With this, synthetic ultrasound signals suffering from artifacts can be fully recovered, which is backed by quantitative assessment. Real data experimentation is carried out to demonstrate the classification capability of the acquired features with object reflections being detected at different points in time. The code is available at https://github.com/hahnec/multimodal_emg.
【15】 Song Emotion Recognition: a Performance Comparison Between Audio Features and Artificial Neural Networks
标题：歌曲情感识别：音频特征与人工神经网络的性能比较
链接：https://arxiv.org/abs/2209.12045
* 与cs.SD语音【5】为同一篇
作者：Karen Rosero,Arthur Nicholas dos Santos,Pedro Benevenuto Valadares,Bruno Sanches Masiero机构：School of Electrical and Computer Engineering, University of Campinas, Brazil摘要：当创作或表演歌曲时，歌手/词曲作者常常有通过歌曲表达感情或情绪的意图。对于人类来说，将音乐作品或表演中的情感与观众的主观感知相匹配可能是相当具有挑战性的。幸运的是，解决这个问题的机器学习方法更简单。通常，它需要一个数据集，从该数据集提取音频特征以将该信息呈现给数据驱动模型，该数据驱动模型又将训练以预测给定歌曲与目标情感匹配的概率是多少。在本文中，我们研究了最近出版的用于解决这个问题的最常见的特征和模型，揭示了哪些特征和模型最适合识别无伴奏合唱歌曲中的情感。摘要：When songs are composed or performed, there is often an intent by the singer/songwriter of expressing feelings or emotions through it. For humans, matching the emotiveness in a musical composition or performance with the subjective perception of an audience can be quite challenging. Fortunately, the machine learning approach for this problem is simpler. Usually, it takes a data-set, from which audio features are extracted to present this information to a data-driven model, that will, in turn, train to predict what is the probability that a given song matches a target emotion. In this paper, we studied the most common features and models used in recent publications to tackle this problem, revealing which ones are best suited for recognizing emotion in a cappella songs.
【16】 Unsupervised domain adaptation for speech recognition with unsupervised error correction
标题：用于具有无监督纠错的语音识别的无监督域自适应
链接：https://arxiv.org/abs/2209.12043
* 与cs.SD语音【6】为同一篇
作者：Long Mai,Julie Carson-Berndsen机构：ML-Labs, School of Computer Science, University College Dublin, Ireland摘要：当转录来自看不见的域的音频时，自动语音识别（ASR）系统的转录质量显著降低。提出了一种无监督ASR域自适应纠错方法，旨在恢复域错配引起的转录错误.与依赖于转录音频进行训练的现有校正方法不同，我们的方法仅需要目标域的未标记数据，其中应用伪标记技术来生成校正训练样本。为了减少对伪数据的过拟合，我们还提出了一种编码器-解码器校正模型，该模型可以考虑诸如对话上下文和声学特征等附加信息。实验结果表明，与未自适应的ASR系统相比，该方法显著降低了误字率（WER）.校正模型还可以应用于其他适应方法之上，以带来相对10%的额外改进。摘要：The transcription quality of automatic speech recognition (ASR) systems degrades significantly when transcribing audios coming from unseen domains. We propose an unsupervised error correction method for unsupervised ASR domain adaption, aiming to recover transcription errors caused by domain mismatch. Unlike existing correction methods that rely on transcribed audios for training, our approach requires only unlabeled data of the target domains in which a pseudo-labeling technique is applied to generate correction training samples. To reduce over-fitting to the pseudo data, we also propose an encoder-decoder correction model that can take into account additional information such as dialogue context and acoustic features. Experiment results show that our method obtains a significant word error rate (WER) reduction over non-adapted ASR systems. The correction model can also be applied on top of other adaptation approaches to bring an additional improvement of 10% relatively.
【17】 Joint Speech Activity and Overlap Detection with Multi-Exit Architecture
标题：多出口结构下的联合语音活动和重叠检测
链接：https://arxiv.org/abs/2209.11906
* 与cs.SD语音【7】为同一篇
作者：Ziqing Du,Kai Liu,Xucheng Wan,Huan Zhou机构：Artificial Intelligence Application Research Center, Huawei Technologies, Shenzhen, PRC摘要：重叠语音检测（OSD）是多方通话场景下语音应用的关键。尽管OSD的研究取得了许多进展，但与语音激活检测（VAD）相比，OSD仍是一个开放的挑战，其整体性能还远远不能令人满意。大多数现有研究通常将OSD问题表述为标准分类问题，以在帧级识别具有二进制（OSD）或三类标签（联合VAD和OSD）的语音。与主流研究不同，本研究从一个新的角度对VAD和OSD联合任务进行了研究。特别地，我们提出用多出口结构扩展传统分类网络。这样的体系结构使我们的系统具有独特的能力来使用来自早期退出的低级特征或来自最后退出的高级特征来识别类。此外，采用了知识提取和密集连接两种训练方案，进一步提高了系统性能。在标准数据集AMI和DIHARD-III上的实验结果验证了该系统的有效性和通用性.我们的消融进一步揭示了所提出方案的互补贡献。在AMI和DIHARD-III上的$F_1$得分分别为0.792和0.625，我们提出的系统在这些数据集上的性能优于几个性能最好的模型，而且在两个数据集上也以很大的幅度超过了当前最先进的模型。除了性能优势外，我们提出的系统还提供了另一种具有吸引力的质量-复杂性权衡潜力，这对于高效的OSD部署是非常优选的。摘要：Overlapped speech detection (OSD) is critical for speech applications in scenario of multi-party conversion. Despite numerous research efforts and progresses, comparing with speech activity detection (VAD), OSD remains an open challenge and its overall performance is far from satisfactory. The majority of prior research typically formulates the OSD problem as a standard classification problem, to identify speech with binary (OSD) or three-class label (joint VAD and OSD) at frame level. In contrast to the mainstream, this study investigates the joint VAD and OSD task from a new perspective. In particular, we propose to extend traditional classification network with multi-exit architecture. Such an architecture empowers our system with unique capability to identify class using either low-level features from early exits or high-level features from last exit. In addition, two training schemes, knowledge distillation and dense connection, are adopted to further boost our system performance. Experimental results on benchmark datasets (AMI and DIHARD-III) validated the effectiveness and generality of our proposed system. Our ablations further reveal the complementary contribution of proposed schemes. With $F_1$ score of 0.792 on AMI and 0.625 on DIHARD-III, our proposed system outperforms several top performing models on these datasets, but also surpasses the current state-of-the-art by large margins across both datasets. Besides the performance benefit, our proposed system offers another appealing potential for quality-complexity trade-offs, which is highly preferred for efficient OSD deployment.
【18】 Speech Enhancement with Perceptually-motivated Optimization and Dual Transformations
标题：基于感知激励优化和对偶变换的语音增强
链接：https://arxiv.org/abs/2209.11905
* 与cs.SD语音【8】为同一篇
作者：Xucheng Wan,Kai Liu,Ziqing Du,Huan Zhou机构：Artificial Intelligence Application Research Center, Huawei Technologies, Shenzhen, PRC摘要：为了解决单耳语音增强问题，已经进行了大量的研究，以通过在时域中对从语音混合学习的内部域进行操作或者在时频域中对固定的全频带短时傅里叶变换（STFT）谱图进行操作来增强语音。最近，已经提出了一些关于基于子带的语音增强的研究。通过对子带频谱图进行运算来增强语音，这些研究在DNS 2020的基准数据集上表现出了有竞争力的性能。尽管这一新的研究方向颇具吸引力，但尚未得到充分的探索，仍有改进的空间。因此，本研究针对目前语音增强的最新研究方向，提出了一种基于子带的感知优化和双变换的语音增强系统，称为PT-FSE。特别地，我们提出的PT-FSE模型通过三个方面的努力改进了其主干--全带和子带融合模型。首先，我们设计了一个频率变换模块，旨在增强全局频率相关性。然后引入时间变换来捕获长范围的时间上下文。最后，提出了一种新的基于人类听觉感知特性的损失模型，使该模型更侧重于低频增强。为了验证所提模型的有效性，在DNS 2020数据集上进行了大量实验.实验结果表明，PT-FSE系统在性能上比现有的SOTA系统有了很大的提高，同时比SOTA系统的体积小了27%.我们的系统在基准数据集上的平均NB-PESQ为3.57，提供了迄今为止报告的最好的语音增强结果。摘要：To address the monaural speech enhancement problem, numerous research studies have been conducted to enhance speech via operations either in time-domain on the inner-domain learned from the speech mixture or in time--frequency domain on the fixed full-band short time Fourier transform (STFT) spectrograms. Very recently, a few studies on sub-band based speech enhancement have been proposed. By enhancing speech via operations on sub-band spectrograms, those studies demonstrated competitive performances on the benchmark dataset of DNS2020. Despite attractive, this new research direction has not been fully explored and there is still room for improvement. As such, in this study, we delve into the latest research direction and propose a sub-band based speech enhancement system with perceptually-motivated optimization and dual transformations, called PT-FSE. Specially, our proposed PT-FSE model improves its backbone, a full-band and sub-band fusion model, by three efforts. First, we design a frequency transformation module that aims to strengthen the global frequency correlation. Then a temporal transformation is introduced to capture long range temporal contexts. Lastly, a novel loss, with leverage of properties of human auditory perception, is proposed to facilitate the model to focus on low frequency enhancement. To validate the effectiveness of our proposed model, extensive experiments are conducted on the DNS2020 dataset. Experimental results show that our PT-FSE system achieves substantial improvements over its backbone, but also outperforms the current state-of-the-art while being 27\% smaller than the SOTA. With average NB-PESQ of 3.57 on the benchmark dataset, our system offers the best speech enhancement results reported till date.
机器翻译，仅供参考
每日语音速递