今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。
【1】 Topological Data Analysis for Speech Processing
标题：语音处理中的拓扑数据分析
链接：https://arxiv.org/abs/2211.17223
作者：Eduard Tulchinskii,Kristian Kuznetsov,Laida Kushnareva,Daniil Cherniavskii,Serguei Barannikov,Irina Piontkovskaya,Sergey Nikolenko,Evgeny Burnaev机构：Noah’s Ark Lab, Huawei备注：Submitted to ICASSP 2023 conference, awaiting review摘要：我们将拓扑数据分析（TDA）应用于语音分类问题和预训练语音模型HuBERT的内省。为此，我们引入了一些从Transformer注意图和嵌入导出的拓扑和代数特征。我们证明了一个简单的线性分类器建立在这样的特征之上优于一个微调的分类头。特别地，在四个常见的数据集上，我们取得了约9%的准确率和5%的错误率的改进;在CREMA-D上，所提出的特征集达到了新的技术性能状态，准确度为$80.155$。我们还表明，拓扑特征能够揭示语音转换器头部的功能角色;例如，可以是，我们发现头部能够区分成对的样本源（自然/合成）或语音，而无需任何下游微调。实验结果表明，TDA是一种很有前途的语音分析新方法，尤其适用于需要结构预测的任务。摘要：We apply topological data analysis (TDA) to speech classification problems and to the introspection of a pretrained speech model, HuBERT. To this end, we introduce a number of topological and algebraic features derived from Transformer attention maps and embeddings. We show that a simple linear classifier built on top of such features outperforms a fine-tuned classification head. In particular, we achieve an improvement of about $9\%$ accuracy and $5\%$ ERR on four common datasets; on CREMA-D, the proposed feature set reaches a new state of the art performance with accuracy $80.155$. We also show that topological features are able to reveal functional roles of speech Transformer heads; e.g., we find the heads capable to distinguish between pairs of sample sources (natural/synthetic) or voices without any downstream fine-tuning. Our results demonstrate that TDA is a promising new approach for speech analysis, especially for tasks that require structural prediction.
【2】 EURO: ESPnet Unsupervised ASR Open-source Toolkit
标题：欧元：ESPnet非监督ASR开源工具包
链接：https://arxiv.org/abs/2211.17196
作者：Dongji Gao,Jiatong Shi,Shun-Po Chuang,Leibny Paola Garcia,Hung-yi Lee,Shinji Watanabe,Sanjeev Khudanpur机构：Johns Hopkins University,Carnegie Mellon University, National Taiwan University摘要：介绍了ESPnet无监督自动语音识别开源工具包（EURO），它是一个用于无监督自动语音识别（UASR）的端到端开源工具包。EURO采用了Wav 2 vec-U引入的最先进的UASR学习方法，该方法最初在FAIRSEQ实施，它利用了自监督语音表示和对抗训练。除了wav 2 vec 2，EURO通过集成S3 PRL和k2扩展了UASR任务的功能并提高了可再现性，从而从27个自监督模型和各种基于图的解码策略中产生了灵活的前端。EURO在ESPnet中实现，并遵循其统一的管道，以提供具有完整设置的UASR配方。这提高了管道的效率，并允许EURO容易地应用于ESPnet中的现有数据集。在TIMIT和LibriSpeech数据集上对三种主流的自监督模型进行了大量实验，实验结果表明了该工具包的有效性，并取得了最佳的UASR性能. EURO将在https：//github.com/espnet/espnet上公开，旨在通过开源活动促进这一基于UASR的令人兴奋的新兴研究领域。摘要：This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR). EURO adopts the state-of-the-art UASR learning method introduced by the Wav2vec-U, originally implemented at FAIRSEQ, which leverages self-supervised speech representations and adversarial training. In addition to wav2vec2, EURO extends the functionality and promotes reproducibility for UASR tasks by integrating S3PRL and k2, resulting in flexible frontends from 27 self-supervised models and various graph-based decoding strategies. EURO is implemented in ESPnet and follows its unified pipeline to provide UASR recipes with a complete setup. This improves the pipeline's efficiency and allows EURO to be easily applied to existing datasets in ESPnet. Extensive experiments on three mainstream self-supervised models demonstrate the toolkit's effectiveness and achieve state-of-the-art UASR performance on TIMIT and LibriSpeech datasets. EURO will be publicly available at https://github.com/espnet/espnet, aiming to promote this exciting and emerging research area based on UASR through open-source activity.
【3】 How to (virtually) train your sound source localizer
标题：如何(虚拟地)训练您的声源定位器
链接：https://arxiv.org/abs/2211.16958
作者：Prerak Srivastava,Antoine Deleforge,Archontis Politis,Emmanuel Vincent机构：Universit´e de Lorraine, CNRS, Inria, Loria, F-, Nancy, France, Audio and Speech Processing Research Group, Tampere University, Finland摘要：基于学习的方法在声源定位（SSL）中已经变得普遍。现有的系统依赖于模拟训练集，因为缺乏足够大的、多样的和注释的真实数据集。大多数用于此目的的室内声学模拟器依赖于镜像源方法（ISM），因为其计算效率高。本文认为，仔细扩展ISM，将更真实的表面、源和麦克风响应纳入训练集，可以显著提高SSL系统的真实性能。结果表明，提高现有技术的波达方向估计器的训练集真实性，可以在三个不同的真实测试集上产生一致的改进，这三个测试集以不同房间和不同麦克风阵列中的人类说话者为特征。消融研究进一步显示，每增加一层真实感都会对这些改善产生积极影响。摘要：Learning-based methods have become ubiquitous in sound source localization (SSL). Existing systems rely on simulated training sets for the lack of sufficiently large, diverse and annotated real datasets. Most room acoustic simulators used for this purpose rely on the image source method (ISM) because of its computational efficiency. This paper argues that carefully extending the ISM to incorporate more realistic surface, source and microphone responses into training sets can significantly boost the real-world performance of SSL systems. It is shown that increasing the training-set realism of a state-of-the-art direction-of-arrival estimator yields consistent improvements across three different real test sets featuring human speakers in a variety of rooms and various microphone arrays. An ablation study further reveals that every added layer of realism contributes positively to these improvements.
【4】 A General Deep Learning Speech Enhancement Framework Motivated by Taylor's Theorem
题：一种基于泰勒定理的通用深度学习语音增强框架
链接：https://arxiv.org/abs/2211.16764
作者：Andong Li,Guochen Yu,Chengshi Zheng,Wenzhe Liu,Xiaodong Li机构： Communication University of China备注：Submitted to TASLP, 13 pages摘要：虽然深度神经网络极大地促进了语音增强领域的发展，但大多数现有方法是根据启发式或盲优化标准开发的，这严重妨碍了可解释性和透明性。受泰勒定理的启发，我们提出了一个通用的展开框架，用于单通道和多通道语音增强任务。具体地，将复谱恢复转化为含噪混合信号邻域空间的谱幅值映射，并引入稀疏先验进行相位修正。在此基础上，将映射函数分解为泰勒级数的0阶多项式和高阶多项式的叠加，其中0阶多项式在幅度域粗略去除干扰，高阶多项式在复谱域逐步补充剩余的谱细节。此外，我们还研究了相邻阶项之间的关系，揭示了每个高阶项可以用其低阶项递归估计，并提出了用一个带有可训练权值的代理函数对每个高阶项进行估计，从而使整个系统能够以端到端的方式进行训练.在WSJ 0-SI 84、DNS-Challenge、Voicebank+Demand和空间化Librispeech数据集上进行了广泛的实验。定量结果表明，该方法不仅具有较好的性能，而且具有良好的内部透明性和灵活性.摘要：While deep neural networks greatly facilitate the proliferation of the speech enhancement field, most of the existing methods are developed following either heuristic or blind optimization criteria, which severely hampers interpretability and transparency. Inspired by Taylor's theorem, we propose a general unfolding framework for both single- and multi-channel speech enhancement tasks. Concretely, we formulate the complex spectrum recovery into the spectral magnitude mapping in the neighboring space of the noisy mixture, in which the sparse prior is introduced for phase modification in advance. Based on that, the mapping function is decomposed into the superimposition of the 0th-order and high-order polynomials in Taylor's series, where the former coarsely removes the interference in the magnitude domain and the latter progressively complements the remaining spectral detail in the complex spectrum domain. In addition, we study the relation between adjacent order term and reveal that each high-order term can be recursively estimated with its lower-order term, and each high-order term is then proposed to evaluate using a surrogate function with trainable weights, so that the whole system can be trained in an end-to-end manner. Extensive experiments are conducted on WSJ0-SI84, DNS-Challenge, Voicebank+Demand, and spatialized Librispeech datasets. Quantitative results show that the proposed approach not only yields competitive performance over existing top-performed approaches, but also enjoys decent internal transparency and flexibility.
【5】 Assisted RTF-Vector-Based Binaural Direction of Arrival Estimation Exploiting a Calibrated External Microphone Array
标题：基于校准外麦克风阵列的RTF矢量辅助双耳波达方向估计
链接：https://arxiv.org/abs/2211.17202
作者：Daniel Fejgin,Simon Doclo机构：University of Oldenburg, Department of Medical Physics and Acoustics, and Cluster of Excellence Hearing,all, Oldenburg, Germany备注：Submitted to ICASSP 2023摘要：最近，已经提出了一种基于相对传递函数（RTF）矢量的方法来估计双耳助听器设置的目标说话者的到达方向（DOA），假设外部麦克风可用。该方法利用外部麦克风来估计对应于双耳助听器的RTF向量，并且通过将估计的RTF向量与若干方向的无回声原型RTF向量的数据库进行比较来构建一维空间频谱。在本文中，我们假设校准的外部麦克风阵列可用，其特征在于消声原型RTF矢量的第二数据库。我们提出了一种方法，其中外部麦克风不仅用于估计对应于双耳助听器的RTF矢量，而且还辅助估计目标说话人的DOA。基于所有麦克风和两个原型数据库的估计RTF矢量，构造二维空间谱，从该二维空间谱估计DOA。在混响环境下的实验结果表明，辅助DOA估计优于不使用表征外部麦克风阵列的原型数据库的DOA估计。摘要：Recently, a relative transfer function (RTF)-vector-based method has been proposed to estimate the direction of arrival (DOA) of a target speaker for a binaural hearing aid setup, assuming the availability of external microphones. This method exploits the external microphones to estimate the RTF vector corresponding to the binaural hearing aid and constructs a one-dimensional spatial spectrum by comparing the estimated RTF vector against a database of anechoic prototype RTF vectors for several directions. In this paper we assume the availability of a calibrated array of external microphones, which is characterized by a second database of anechoic prototype RTF vectors. We propose a method, where the external microphones are not only exploited to estimate the RTF vector corresponding to the binaural hearing aid but also assist in estimating the DOA of the target speaker. Based on the estimated RTF vector for all microphones and both prototype databases, a two-dimensional spatial spectrum is constructed from which the DOA is estimated. Experimental results for a reverberant environment with diffuse-like noise show that assisted DOA estimation outperforms DOA estimation where the prototype database characterizing the array of external microphones is not used.
【6】 Better Transcription of UK Supreme Court Hearings
标题：更好地抄写英国最高法院听证会
链接：https://arxiv.org/abs/2211.17094
作者：Hadeel Saadany,Constantin Orăsan,Catherine Breslin机构：Centre for Translation Studies, University of Surrey, United Kingdom, Constantin Or˘asan, Kingfisher Labs Ltd摘要：法律诉讼的笔录对于实现司法救助非常重要。然而，语音转录是昂贵且缓慢的过程。在本文中，我们描述了一个联合研究和工业项目的一部分，建立一个自动转录工具，专门为司法部门在英国。我们解释了在法庭听证会上转录所涉及的挑战，以及我们用来解决这些挑战的自然语言处理（NLP）技术。我们将证明，使用领域内语言模型微调通用现成的预训练自动语音识别（ASR）系统，以及注入用搭配检测模型提取的常见短语，不仅可以改善转录听证会的单词错误率（WER），还可以避免英国法院常用的法律行话和术语特有的关键错误。摘要：Transcription of legal proceedings is very important to enable access to justice. However, speech transcription is an expensive and slow process. In this paper we describe part of a combined research and industrial project for building an automated transcription tool designed specifically for the Justice sector in the UK. We explain the challenges involved in transcribing court room hearings and the Natural Language Processing (NLP) techniques we employ to tackle these challenges. We will show that fine-tuning a generic off-the-shelf pre-trained Automatic Speech Recognition (ASR) system with an in-domain language model as well as infusing common phrases extracted with a collocation detection model can improve not only the Word Error Rate (WER) of the transcribed hearings but avoid critical errors that are specific of the legal jargon and terminology commonly used in British courts.
【7】 Extreme Audio Time Stretching Using Neural Synthesis
标题：基于神经合成的极端音频时间伸展
链接：https://arxiv.org/abs/2211.16992
作者：Leonardo Fierro,Alec Wright,Vesa Välimäki,Matti Hämäläinen机构：Acoustics Lab, Dept. of Signal Processing and Acoustics, Aalto University, Espoo, Finland, Matti H¨am¨al¨ainen, Nokia Technologies, Tampere, Finland备注：Submitted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023 on Oct 27, 2022摘要：针对环境声音，提出了一种基于深度神经网络的时间尺度修正方法（TSM）。当TSM因子为4或更大时，传统TSM伪像（如瞬态拖尾、不在场和相位）会严重加重，并导致音频质量较差。通常基于相位声码器结构的已建立TSM方法的弱点在于声音的瞬变和噪声分量或细微差别的不良描述和缩放。我们的新颖解决方案将正弦-瞬态-噪声分解与独立的WaveNet合成器相结合，以提供对噪声分量的更好描述，并改善大拉伸因子的声音质量。通过与其他四种TSM算法的主观听觉测试，结果表明该方法具有更好的性能。所提出的方法是立体声兼容的，并且具有与媒体内容的慢动作相关的广泛应用。摘要：A deep neural network solution for time-scale modification (TSM) focused on large stretching factors is proposed, targeting environmental sounds. Traditional TSM artifacts such as transient smearing, loss of presence, and phasiness are heavily accentuated and cause poor audio quality when the TSM factor is four or larger. The weakness of established TSM methods, often based on a phase vocoder structure, lies in the poor description and scaling of the transient and noise components, or nuances, of a sound. Our novel solution combines a sines-transients-noise decomposition with an independent WaveNet synthesizer to provide a better description of the noise component and an improve sound quality for large stretching factors. Results of a subjective listening test against four other TSM algorithms are reported, showing the proposed method to be often superior. The proposed method is stereo compatible and has a wide range of applications related to the slow motion of media content.
【8】 SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
标题：SNAC：基于流的Zero-Shot多说话人文语转换中说话人归一化仿射耦合层
链接：https://arxiv.org/abs/2211.16866
作者：Byoung Jin Choi,Myeonghun Jeong,Joun Yeop Lee,Nam Soo Kim机构： and Nam Soo Kim are with the De-partment of Electrical and Computer Engineering and with the Institute of NewMedia and Communications, Seoul National University备注：Accepted to IEEE Signal Processing Letters摘要：零拍多说话人文本到语音（ZSM-TTS）模型的目标是生成具有不可见说话人声音特征的语音样本。ZSM-TTS的主要挑战是增加未看过的说话人的总体说话人相似度。在基于流的多说话人文本-语音（TTS）模型中，最成功的说话人条件化方法之一是根据给定的说话人嵌入向量，利用预测仿射耦合层尺度和偏置参数的函数。在这封信中，我们通过引入一个说话人归一化仿射耦合（SNAC）层来改进先前的说话人条件化方法，该层允许以零拍方式利用基于归一化的条件化技术来合成看不见的说话人语音。新设计的耦合层通过训练时说话人嵌入向量预测的参数显式归一化输入，使得在推理时新说话人嵌入的反归一化过程成为可能。在ZSM-TTS环境下，所提出的条件化方案在语音质量和说话人相似度方面产生了最先进的性能。摘要：Zero-shot multi-speaker text-to-speech (ZSM-TTS) models aim to generate a speech sample with the voice characteristic of an unseen speaker. The main challenge of ZSM-TTS is to increase the overall speaker similarity for unseen speakers. One of the most successful speaker conditioning methods for flow-based multi-speaker text-to-speech (TTS) models is to utilize the functions which predict the scale and bias parameters of the affine coupling layers according to the given speaker embedding vector. In this letter, we improve on the previous speaker conditioning method by introducing a speaker-normalized affine coupling (SNAC) layer which allows for unseen speaker speech synthesis in a zero-shot manner leveraging a normalization-based conditioning technique. The newly designed coupling layer explicitly normalizes the input by the parameters predicted from a speaker embedding vector while training, enabling an inverse process of denormalizing for a new speaker embedding at inference. The proposed conditioning scheme yields the state-of-the-art performance in terms of the speech quality and speaker similarity in a ZSM-TTS setting.
【9】 MSV Challenge 2022: NPU-HC Speaker Verification System for Low-resource Indian Languages
标题：MSV挑战2022：针对低资源印度语言的NPU-HC说话人验证系统
链接：https://arxiv.org/abs/2211.16694
作者：Yue Li,Li Zhang,Namin Wang,Jie Liu,Lei Xie机构：Northwestern Polytechnical University (NPU), China, Huawei Cloud备注：6pages, submitted to the 9th International Workshop on Vietnamese Language and Speech Processing摘要：本报告描述了提交给O-COCOSDA多语言说话人确认（MSV）挑战赛2022的NPU-HC说话人确认系统，该挑战赛专注于开发资源较少的亚洲语言的说话人确认系统。我们参与了I-MSV项目，该项目旨在为各种印度语言开发说话人验证系统。在这个挑战中，我们首先探索不同的神经网络框架用于低资源说话人确认。然后，我们利用香草微调和权重转移微调将域外预训练模型转移到域内印度数据集。具体地说，权值转移微调旨在约束预训练模型和微调模型之间的权值距离，从而充分利用了大规模域外数据集预先获得的区分能力，同时避免了灾难性遗忘和过拟合.最后，采用分数融合进一步提高性能。加上上述贡献，我们在公共评估集中获得了0.223%的EER，在排行榜上排名第二。在私有评估集上，我们提交的系统对于I-MSV轨道的约束和无约束子任务的EER分别为2.123%和0.630%，导致排名均为第二位。摘要：This report describes the NPU-HC speaker verification system submitted to the O-COCOSDA Multi-lingual Speaker Verification (MSV) Challenge 2022, which focuses on developing speaker verification systems for low-resource Asian languages. We participate in the I-MSV track, which aims to develop speaker verification systems for various Indian languages. In this challenge, we first explore different neural network frameworks for low-resource speaker verification. Then we leverage vanilla fine-tuning and weight transfer fine-tuning to transfer the out-domain pre-trained models to the in-domain Indian dataset. Specifically, the weight transfer fine-tuning aims to constrain the distance of the weights between the pre-trained model and the fine-tuned model, which takes advantage of the previously acquired discriminative ability from the large-scale out-domain datasets and avoids catastrophic forgetting and overfitting at the same time. Finally, score fusion is adopted to further improve performance. Together with the above contributions, we obtain 0.223% EER on the public evaluation set, ranking 2nd place on the leaderboard. On the private evaluation set, the EER of our submitted system is 2.123% and 0.630% for the constrained and unconstrained sub-tasks of the I-MSV track respectively, leading to both the 2nd place in the ranking.
【10】 Enhancing and Adversarial: Improve ASR with Speaker Labels
标题：增强和对抗：使用扬声器标签提高ASR
链接：https://arxiv.org/abs/2211.06369
作者：Wei Zhou,Haotian Wu,Jingjing Xu,Mohammad Zeineldeen,Christoph Lüscher,Ralf Schlüter,Hermann Ney机构：Human Language Technology and Pattern Recognition, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：submitted to ICASSP 2023摘要：ASR可以通过具有领域增强或领域对抗训练的多任务学习（MTL）来改进，这是两个相反的目标，其目的分别是增加/减少朝向领域感知/不可知ASR的领域方差。在本研究中，我们研究如何最好地应用这两个相反的目标与说话人标签，以改善基于符合者的ASR。我们还提出了一个新的自适应梯度反转层，用于稳定和有效的对抗训练，而无需调整工作。通过详细的分析和实验验证，给出了ASR神经网络中应用说话人增强和对抗性训练的最佳位置。我们还探索了它们的组合以进一步改进，实现与i向量加对抗训练相同的性能。我们最好的基于扬声器的MTL在Switchboard Hub 5 '00集上实现了7%的相对改进。我们还研究了这种基于说话者的MTL对r.t.较干净的数据集和较弱的ASR NN。摘要：ASR can be improved by multi-task learning (MTL) with domain enhancing or domain adversarial training, which are two opposite objectives with the aim to increase/decrease domain variance towards domain-aware/agnostic ASR, respectively. In this work, we study how to best apply these two opposite objectives with speaker labels to improve conformer-based ASR. We also propose a novel adaptive gradient reversal layer for stable and effective adversarial training without tuning effort. Detailed analysis and experimental verification are conducted to show the optimal positions in the ASR neural network (NN) to apply speaker enhancing and adversarial training. We also explore their combination for further improvement, achieving the same performance as i-vectors plus adversarial training. Our best speaker-based MTL achieves 7\% relative improvement on the Switchboard Hub5'00 set. We also investigate the effect of such speaker-based MTL w.r.t. cleaner dataset and weaker ASR NN.
【1】 Assisted RTF-Vector-Based Binaural Direction of Arrival Estimation Exploiting a Calibrated External Microphone Array
标题：基于校准外麦克风阵列的RTF矢量辅助双耳波达方向估计
链接：https://arxiv.org/abs/2211.17202
* 与cs.SD语音【5】为同一篇
作者：Daniel Fejgin,Simon Doclo机构：University of Oldenburg, Department of Medical Physics and Acoustics, and Cluster of Excellence Hearing,all, Oldenburg, Germany备注：Submitted to ICASSP 2023摘要：最近，已经提出了一种基于相对传递函数（RTF）矢量的方法来估计双耳助听器设置的目标说话者的到达方向（DOA），假设外部麦克风可用。该方法利用外部麦克风来估计对应于双耳助听器的RTF向量，并且通过将估计的RTF向量与若干方向的无回声原型RTF向量的数据库进行比较来构建一维空间频谱。在本文中，我们假设校准的外部麦克风阵列可用，其特征在于消声原型RTF矢量的第二数据库。我们提出了一种方法，其中外部麦克风不仅用于估计对应于双耳助听器的RTF矢量，而且还辅助估计目标说话人的DOA。基于所有麦克风和两个原型数据库的估计RTF矢量，构造二维空间谱，从该二维空间谱估计DOA。在混响环境下的实验结果表明，辅助DOA估计优于不使用表征外部麦克风阵列的原型数据库的DOA估计。摘要：Recently, a relative transfer function (RTF)-vector-based method has been proposed to estimate the direction of arrival (DOA) of a target speaker for a binaural hearing aid setup, assuming the availability of external microphones. This method exploits the external microphones to estimate the RTF vector corresponding to the binaural hearing aid and constructs a one-dimensional spatial spectrum by comparing the estimated RTF vector against a database of anechoic prototype RTF vectors for several directions. In this paper we assume the availability of a calibrated array of external microphones, which is characterized by a second database of anechoic prototype RTF vectors. We propose a method, where the external microphones are not only exploited to estimate the RTF vector corresponding to the binaural hearing aid but also assist in estimating the DOA of the target speaker. Based on the estimated RTF vector for all microphones and both prototype databases, a two-dimensional spatial spectrum is constructed from which the DOA is estimated. Experimental results for a reverberant environment with diffuse-like noise show that assisted DOA estimation outperforms DOA estimation where the prototype database characterizing the array of external microphones is not used.
【2】 Better Transcription of UK Supreme Court Hearings
标题：更好地抄写英国最高法院听证会
链接：https://arxiv.org/abs/2211.17094
* 与cs.SD语音【6】为同一篇
作者：Hadeel Saadany,Constantin Orăsan,Catherine Breslin机构：Centre for Translation Studies, University of Surrey, United Kingdom, Constantin Or˘asan, Kingfisher Labs Ltd摘要：法律诉讼的笔录对于实现司法救助非常重要。然而，语音转录是昂贵且缓慢的过程。在本文中，我们描述了一个联合研究和工业项目的一部分，建立一个自动转录工具，专门为司法部门在英国。我们解释了在法庭听证会上转录所涉及的挑战，以及我们用来解决这些挑战的自然语言处理（NLP）技术。我们将证明，使用领域内语言模型微调通用现成的预训练自动语音识别（ASR）系统，以及注入用搭配检测模型提取的常见短语，不仅可以改善转录听证会的单词错误率（WER），还可以避免英国法院常用的法律行话和术语特有的关键错误。摘要：Transcription of legal proceedings is very important to enable access to justice. However, speech transcription is an expensive and slow process. In this paper we describe part of a combined research and industrial project for building an automated transcription tool designed specifically for the Justice sector in the UK. We explain the challenges involved in transcribing court room hearings and the Natural Language Processing (NLP) techniques we employ to tackle these challenges. We will show that fine-tuning a generic off-the-shelf pre-trained Automatic Speech Recognition (ASR) system with an in-domain language model as well as infusing common phrases extracted with a collocation detection model can improve not only the Word Error Rate (WER) of the transcribed hearings but avoid critical errors that are specific of the legal jargon and terminology commonly used in British courts.
【3】 Extreme Audio Time Stretching Using Neural Synthesis
标题：基于神经合成的极端音频时间伸展
链接：https://arxiv.org/abs/2211.16992
* 与cs.SD语音【7】为同一篇
作者：Leonardo Fierro,Alec Wright,Vesa Välimäki,Matti Hämäläinen机构：Acoustics Lab, Dept. of Signal Processing and Acoustics, Aalto University, Espoo, Finland, Matti H¨am¨al¨ainen, Nokia Technologies, Tampere, Finland备注：Submitted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2023 on Oct 27, 2022摘要：A deep neural network solution for time-scale modification (TSM) focused on large stretching factors is proposed, targeting environmental sounds. Traditional TSM artifacts such as transient smearing, loss of presence, and phasiness are heavily accentuated and cause poor audio quality when the TSM factor is four or larger. The weakness of established TSM methods, often based on a phase vocoder structure, lies in the poor description and scaling of the transient and noise components, or nuances, of a sound. Our novel solution combines a sines-transients-noise decomposition with an independent WaveNet synthesizer to provide a better description of the noise component and an improve sound quality for large stretching factors. Results of a subjective listening test against four other TSM algorithms are reported, showing the proposed method to be often superior. The proposed method is stereo compatible and has a wide range of applications related to the slow motion of media content.
【4】 SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker text-to-speech
标题：SNAC：基于流的Zero-Shot多说话人文语转换中说话人归一化仿射耦合层
链接：https://arxiv.org/abs/2211.16866
* 与cs.SD语音【8】为同一篇
作者：Byoung Jin Choi,Myeonghun Jeong,Joun Yeop Lee,Nam Soo Kim机构： and Nam Soo Kim are with the De-partment of Electrical and Computer Engineering and with the Institute of NewMedia and Communications, Seoul National University备注：Accepted to IEEE Signal Processing Letters摘要：零拍多说话人文本到语音（ZSM-TTS）模型的目标是生成具有不可见说话人声音特征的语音样本。ZSM-TTS的主要挑战是增加未看过的说话人的总体说话人相似度。在基于流的多说话人文本语音（TTS）模型中，最成功的说话人条件化方法之一是根据给定的说话人嵌入向量，利用预测仿射耦合层尺度和偏置参数的函数。在这封信中，我们通过引入一个说话人归一化仿射耦合（SNAC）层来改进先前的说话人条件化方法，该层允许以零拍方式利用基于归一化的条件化技术来合成看不见的说话人语音。新设计的耦合层通过训练时说话人嵌入向量预测的参数显式归一化输入，使得在推理时新说话人嵌入的反归一化过程成为可能。在ZSM-TTS环境下，所提出的条件化方案在语音质量和说话人相似度方面产生了最先进的性能。摘要：Zero-shot multi-speaker text-to-speech (ZSM-TTS) models aim to generate a speech sample with the voice characteristic of an unseen speaker. The main challenge of ZSM-TTS is to increase the overall speaker similarity for unseen speakers. One of the most successful speaker conditioning methods for flow-based multi-speaker text-to-speech (TTS) models is to utilize the functions which predict the scale and bias parameters of the affine coupling layers according to the given speaker embedding vector. In this letter, we improve on the previous speaker conditioning method by introducing a speaker-normalized affine coupling (SNAC) layer which allows for unseen speaker speech synthesis in a zero-shot manner leveraging a normalization-based conditioning technique. The newly designed coupling layer explicitly normalizes the input by the parameters predicted from a speaker embedding vector while training, enabling an inverse process of denormalizing for a new speaker embedding at inference. The proposed conditioning scheme yields the state-of-the-art performance in terms of the speech quality and speaker similarity in a ZSM-TTS setting.
【5】 MSV Challenge 2022: NPU-HC Speaker Verification System for Low-resource Indian Languages
标题：MSV挑战2022：针对低资源印度语言的NPU-HC说话人验证系统
链接：https://arxiv.org/abs/2211.16694
* 与cs.SD语音【9】为同一篇
作者：Yue Li,Li Zhang,Namin Wang,Jie Liu,Lei Xie机构：Northwestern Polytechnical University (NPU), China, Huawei Cloud备注：6pages, submitted to the 9th International Workshop on Vietnamese Language and Speech Processing摘要：本报告描述了提交给O-COCOSDA多语言说话人验证（MSV）挑战赛2022的NPU-HC说话人验证系统，该挑战赛专注于开发资源较少的亚洲语言的说话人验证系统。我们参与了I-MSV项目，该项目旨在为各种印度语言开发说话人验证系统。在这个挑战中，我们首先探索不同的神经网络框架用于低资源说话人确认。然后我们利用香草微调和权重转移微调将域外预训练模型转移到域内印度数据集。具体地说，权值转移微调旨在约束预训练模型和微调模型之间的权值距离，从而充分利用了大规模域外数据集预先获得的区分能力，同时避免了灾难性遗忘和过拟合.最后，采用分数融合进一步提高性能。加上上述贡献，我们在公共评估集中获得了0.223%的EER，在排行榜上排名第二。在私有评估集上，我们提交的系统对于I-MSV轨道的约束和无约束子任务的EER分别为2.123%和0.630%，导致两者在排名中均为第二位。摘要：This report describes the NPU-HC speaker verification system submitted to the O-COCOSDA Multi-lingual Speaker Verification (MSV) Challenge 2022, which focuses on developing speaker verification systems for low-resource Asian languages. We participate in the I-MSV track, which aims to develop speaker verification systems for various Indian languages. In this challenge, we first explore different neural network frameworks for low-resource speaker verification. Then we leverage vanilla fine-tuning and weight transfer fine-tuning to transfer the out-domain pre-trained models to the in-domain Indian dataset. Specifically, the weight transfer fine-tuning aims to constrain the distance of the weights between the pre-trained model and the fine-tuned model, which takes advantage of the previously acquired discriminative ability from the large-scale out-domain datasets and avoids catastrophic forgetting and overfitting at the same time. Finally, score fusion is adopted to further improve performance. Together with the above contributions, we obtain 0.223% EER on the public evaluation set, ranking 2nd place on the leaderboard. On the private evaluation set, the EER of our submitted system is 2.123% and 0.630% for the constrained and unconstrained sub-tasks of the I-MSV track respectively, leading to both the 2nd place in the ranking.
【6】 Topological Data Analysis for Speech Processing
标题：语音处理中的拓扑数据分析
链接：https://arxiv.org/abs/2211.17223
* 与cs.SD语音【1】为同一篇
作者：Eduard Tulchinskii,Kristian Kuznetsov,Laida Kushnareva,Daniil Cherniavskii,Serguei Barannikov,Irina Piontkovskaya,Sergey Nikolenko,Evgeny Burnaev机构：Noah’s Ark Lab, Huawei备注：Submitted to ICASSP 2023 conference, awaiting review摘要：我们将拓扑数据分析（TDA）应用于语音分类问题和预训练语音模型HuBERT的内省。为此，我们引入了一些从Transformer注意图和嵌入导出的拓扑和代数特征。我们证明了基于这些特征的简单线性分类器优于微调的分类头。特别地，在四个常见的数据集上，我们取得了约9%的准确率和5%的错误率的改进;在CREMA-D上，所提出的特征集达到了新的技术性能状态，准确度为$80.155$。我们还表明，拓扑特征能够揭示语音转换器头部的功能角色;例如，可以是，我们发现头部能够区分成对的样本源（自然/合成）或语音，而无需任何下游微调。实验结果表明，TDA是一种很有前途的语音分析新方法，尤其适用于需要结构预测的任务。摘要：We apply topological data analysis (TDA) to speech classification problems and to the introspection of a pretrained speech model, HuBERT. To this end, we introduce a number of topological and algebraic features derived from Transformer attention maps and embeddings. We show that a simple linear classifier built on top of such features outperforms a fine-tuned classification head. In particular, we achieve an improvement of about $9\%$ accuracy and $5\%$ ERR on four common datasets; on CREMA-D, the proposed feature set reaches a new state of the art performance with accuracy $80.155$. We also show that topological features are able to reveal functional roles of speech Transformer heads; e.g., we find the heads capable to distinguish between pairs of sample sources (natural/synthetic) or voices without any downstream fine-tuning. Our results demonstrate that TDA is a promising new approach for speech analysis, especially for tasks that require structural prediction.
【7】 EURO: ESPnet Unsupervised ASR Open-source Toolkit
标题：欧元：ESPnet非监督ASR开源工具包
链接：https://arxiv.org/abs/2211.17196
* 与cs.SD语音【2】为同一篇
作者：Dongji Gao,Jiatong Shi,Shun-Po Chuang,Leibny Paola Garcia,Hung-yi Lee,Shinji Watanabe,Sanjeev Khudanpur机构：Johns Hopkins University,Carnegie Mellon University, National Taiwan University摘要：介绍了ESPnet无监督自动语音识别开源工具包（EURO），它是一个用于无监督自动语音识别（UASR）的端到端开源工具包。EURO采用了Wav 2 vec-U引入的最先进的UASR学习方法，该方法最初在FAIRSEQ实施，它利用了自监督语音表示和对抗训练。除了wav 2 vec 2，EURO通过集成S3 PRL和k2扩展了UASR任务的功能并提高了可重复性，从而从27个自监督模型和各种基于图的解码策略中产生了灵活的前端。EURO在ESPnet中实现，并遵循其统一的管道，以提供具有完整设置的UASR配方。这提高了管道的效率，并允许EURO容易地应用于ESPnet中的现有数据集。在TIMIT和LibriSpeech数据集上对三种主流的自监督模型进行了大量实验，实验结果表明了该工具包的有效性，并取得了最佳的UASR性能. EURO将在https：//github.com/espnet/espnet上公开，旨在通过开源活动促进这一基于UASR的令人兴奋的新兴研究领域。摘要：This paper describes the ESPnet Unsupervised ASR Open-source Toolkit (EURO), an end-to-end open-source toolkit for unsupervised automatic speech recognition (UASR). EURO adopts the state-of-the-art UASR learning method introduced by the Wav2vec-U, originally implemented at FAIRSEQ, which leverages self-supervised speech representations and adversarial training. In addition to wav2vec2, EURO extends the functionality and promotes reproducibility for UASR tasks by integrating S3PRL and k2, resulting in flexible frontends from 27 self-supervised models and various graph-based decoding strategies. EURO is implemented in ESPnet and follows its unified pipeline to provide UASR recipes with a complete setup. This improves the pipeline's efficiency and allows EURO to be easily applied to existing datasets in ESPnet. Extensive experiments on three mainstream self-supervised models demonstrate the toolkit's effectiveness and achieve state-of-the-art UASR performance on TIMIT and LibriSpeech datasets. EURO will be publicly available at https://github.com/espnet/espnet, aiming to promote this exciting and emerging research area based on UASR through open-source activity.
【8】 How to (virtually) train your sound source localizer
标题：如何(虚拟地)训练您的声源定位器
链接：https://arxiv.org/abs/2211.16958
* 与cs.SD语音【3】为同一篇
作者：Prerak Srivastava,Antoine Deleforge,Archontis Politis,Emmanuel Vincent机构：Universit´e de Lorraine, CNRS, Inria, Loria, F-, Nancy, France, Audio and Speech Processing Research Group, Tampere University, Finland摘要：基于学习的方法在声源定位（SSL）中已经变得普遍。现有的系统依赖于模拟训练集，因为缺乏足够大的、多样的和注释的真实数据集。大多数用于此目的的室内声学模拟器依赖于镜像源方法（ISM），因为其计算效率高。本文认为，仔细扩展ISM，将更真实的表面、源和麦克风响应纳入训练集，可以显著提高SSL系统的真实性能。结果表明，提高现有技术的波达方向估计器的训练集真实性，可以在三个不同的真实测试集上产生一致的改进，这三个测试集以不同房间和不同麦克风阵列中的人类说话者为特征。消融研究进一步显示，每增加一层真实感都会对这些改善产生积极影响。摘要：Learning-based methods have become ubiquitous in sound source localization (SSL). Existing systems rely on simulated training sets for the lack of sufficiently large, diverse and annotated real datasets. Most room acoustic simulators used for this purpose rely on the image source method (ISM) because of its computational efficiency. This paper argues that carefully extending the ISM to incorporate more realistic surface, source and microphone responses into training sets can significantly boost the real-world performance of SSL systems. It is shown that increasing the training-set realism of a state-of-the-art direction-of-arrival estimator yields consistent improvements across three different real test sets featuring human speakers in a variety of rooms and various microphone arrays. An ablation study further reveals that every added layer of realism contributes positively to these improvements.
【9】 VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing
标题：Video Dubber：用于视频配音的语音感知长度控制机器翻译
链接：https://arxiv.org/abs/2211.16934
作者：Yihan Wu,Junliang Guo,Xu Tan,Chen Zhang,Bohan Li,Ruihua Song,Lei He,Sheng Zhao,Arul Menezes,Jiang Bian机构： Gaoling School of Artificial Intelligence, Renmin University of China, Microsoft Research Asia, Microsoft Azure Speech, Microsoft Azure Translation备注：AAAI 2023 camera version摘要：视频配音是将影视节目中的原始语音翻译成目标语言的语音，它可以通过语音识别、机器翻译和语音合成的级联系统来实现。为了确保翻译后的语音与对应的视频很好地对齐，翻译后的语音的长度/持续时间应尽可能接近原始语音的长度/持续时间，这需要严格的长度控制。以往的工作通常控制机器翻译模型生成的单词或字符的数量与源句子相似，而没有考虑不同语言中单词/字符的语音持续时间不同时语音的等时性。本文提出了一种适合视频配音任务的机器翻译系统，该系统在翻译过程中直接考虑每个单词的语音时长，以匹配源语音和目标语音的长度。具体地说，我们通过用持续时间信息指导每个词的预测来控制所生成句子的语音长度，所述持续时间信息包括自身的语音持续时间以及剩余词的持续时间。在德语-〉英语，西班牙语-〉英语，汉语-英语四个语言方向上进行实验<->，结果表明，与基准方法相比，该方法对生成的语音具有更好的长度控制能力。为了弥补真实数据集的不足，我们还构建了一个从电影中收集的真实测试集，以提供对视频配音任务的综合评价。摘要：Video dubbing aims to translate the original speech in a film or television program into the speech in a target language, which can be achieved with a cascaded system consisting of speech recognition, machine translation and speech synthesis. To ensure the translated speech to be well aligned with the corresponding video, the length/duration of the translated speech should be as close as possible to that of the original speech, which requires strict length control. Previous works usually control the number of words or characters generated by the machine translation model to be similar to the source sentence, without considering the isochronicity of speech as the speech duration of words/characters in different languages varies. In this paper, we propose a machine translation system tailored for the task of video dubbing, which directly considers the speech duration of each token in translation, to match the length of source and target speech. Specifically, we control the speech length of generated sentence by guiding the prediction of each word with the duration information, including the speech duration of itself as well as how much duration is left for the remaining words. We design experiments on four language directions (German -> English, Spanish -> English, Chinese <-> English), and the results show that the proposed method achieves better length control ability on the generated speech than baseline methods. To make up the lack of real-world datasets, we also construct a real-world test set collected from films to provide comprehensive evaluations on the video dubbing task.
【10】 A General Deep Learning Speech Enhancement Framework Motivated by Taylor's Theorem
标题：一种基于泰勒定理的通用深度学习语音增强框架
链接：https://arxiv.org/abs/2211.16764
* 与cs.SD语音【4】为同一篇
作者：Andong Li,Guochen Yu,Chengshi Zheng,Wenzhe Liu,Xiaodong Li机构： Communication University of China备注：Submitted to TASLP, 13 pages摘要：虽然深度神经网络极大地促进了语音增强领域的发展，但大多数现有方法是根据启发式或盲优化标准开发的，这严重妨碍了可解释性和透明性。受泰勒定理的启发，我们提出了一个通用的展开框架，用于单通道和多通道语音增强任务。具体地，将复谱恢复转化为含噪混合信号邻域空间的谱幅值映射，并引入稀疏先验进行相位修正。在此基础上，将映射函数分解为泰勒级数的0阶多项式和高阶多项式的叠加，其中0阶多项式在幅度域粗略去除干扰，高阶多项式在复谱域逐步补充剩余的谱细节。此外，我们还研究了相邻阶项之间的关系，揭示了每个高阶项可以用其低阶项递归估计，并提出了用一个带有可训练权值的代理函数对每个高阶项进行估计，从而使整个系统能够以端到端的方式进行训练.在WSJ 0-SI 84、DNS-Challenge、Voicebank+Demand和空间化Librispeech数据集上进行了广泛的实验。定量结果表明，该方法不仅具有较好的性能，而且具有良好的内部透明性和灵活性.摘要：While deep neural networks greatly facilitate the proliferation of the speech enhancement field, most of the existing methods are developed following either heuristic or blind optimization criteria, which severely hampers interpretability and transparency. Inspired by Taylor's theorem, we propose a general unfolding framework for both single- and multi-channel speech enhancement tasks. Concretely, we formulate the complex spectrum recovery into the spectral magnitude mapping in the neighboring space of the noisy mixture, in which the sparse prior is introduced for phase modification in advance. Based on that, the mapping function is decomposed into the superimposition of the 0th-order and high-order polynomials in Taylor's series, where the former coarsely removes the interference in the magnitude domain and the latter progressively complements the remaining spectral detail in the complex spectrum domain. In addition, we study the relation between adjacent order term and reveal that each high-order term can be recursively estimated with its lower-order term, and each high-order term is then proposed to evaluate using a surrogate function with trainable weights, so that the whole system can be trained in an end-to-end manner. Extensive experiments are conducted on WSJ0-SI84, DNS-Challenge, Voicebank+Demand, and spatialized Librispeech datasets. Quantitative results show that the proposed approach not only yields competitive performance over existing top-performed approaches, but also enjoys decent internal transparency and flexibility.
机器翻译，仅供参考
每日语音速递