今日论文合集：cs.SD语音16篇，eess.AS音频处理17篇。
cs.SD语音

【1】 Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding标题：SURE语音理解基准上参数高效迁移学习方法的评价链接：http://arxiv.org/pdf/2303.03267v1

作者：Yingting Li,Ambuj Mehrish,Shuai Zhao,Rishabh Bhardwaj,Amir Zadeh,Navonil Majumder,Rada Mihalcea,Soujanya Poria机构：Bo Cheng, Beijing University of Posts and Telecommunications, China, University of Michigan, USA, Singapore University of Technology and Design, Singapore, Amazon Science, USA备注：ICASSP 2023摘要：微调被广泛用作从预先训练的模型进行迁移学习的默认算法。然而，当在迁移学习期间需要针对各个下游任务更新大型预训练模型的所有参数时，可能出现参数低效。随着参数数量的增长，微调容易出现过拟合和灾难性遗忘。此外，当模型用于许多任务时，完全微调可能变得非常昂贵。为了缓解这个问题，已经提出了参数有效的迁移学习算法，例如适配器和前缀调整，作为引入一些可训练参数的方式，这些参数可以插入到大的预先训练的语言模型中，例如BERT和HuBERT。在这篇文章中，我们介绍了语音理解评估（SURE）基准，用于各种语音处理任务的参数有效学习。此外，我们还引入了一种基于1D卷积的新适配器ConvAdapter。我们证明ConvAdapter优于标准适配器，同时显示出与前缀调整和LoRA相当的性能，在SURE中的一些任务上仅使用0.94%的可训练参数。我们进一步探讨了参数有效迁移学习在语音合成任务如文语转换（TTS）中的有效性。摘要：Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and LoRA with only 0.94% of trainable parameters on some of the task in SURE. We further explore the effectiveness of parameter efficient transfer learning for speech synthesis task such as Text-to-Speech (TTS).

【2】 Revisiting Wright: Improving supervised classification of rat ultrasonic vocalisations using synthetic training data标题：重访赖特：使用合成训练数据改进大鼠超声发声的监督分类链接：http://arxiv.org/pdf/2303.03183v1

作者：K. Jack Scott,Lucinda J. Speers,David K. Bilkey机构：⁺ Department of Psychology, University of Otago, New Zealand., All correspondence to:, William James Building, Leith Walk, University of Otago, Dunedin , New Zealand., ORC ID:, Supported by funding from the University of Otago.备注：25 pages, 5 main figures, 4 supplementary figures, 2 tables摘要：啮齿动物通过超声波发声（USVs）进行交流。这些叫声之所以令人感兴趣，是因为它们提供了对声音交流的发展和功能的深入了解，并可能被证明是神经发育障碍模型中功能障碍的有用生物标志物。啮齿类USV可分为不同的部件，虽然人工分类耗时，但神经计算的进步允许快速准确地识别和分类。在这里，我们采用了一个卷积神经网络（CNN），VocalMat，创建用于分析小鼠USV，用于大鼠。我们编纂了一个修改的模式，改编自Wright等人（2010年）之前提出的模式，用于分类，并将我们改编的VocalMat与基准CNN DeepSqueak的性能进行了比较。此外，我们测试了将合成USV插入到分类网络的训练数据中的效果，以减少生成训练集所涉及的工作量。我们的结果表明，修改后的VocalMat在呼叫识别和分类方面都优于基准软件。此外，我们发现，当VocalMat随后用于分析新数据时，使用合成图像增强训练数据可显著提高VocalMat的准确性。修改后的Wright分类的准确度足够高，允许在实验室条件下将该软件应用于大鼠USV分类。我们的研究结果还表明，将合成USV呼叫插入到训练集中会导致准确性的提高，而几乎没有额外的时间成本。摘要：Rodents communicate through ultrasonic vocalizations (USVs). These calls are of interest because they provide insight into the development and function of vocal communication, and may prove to be useful as a biomarker for dysfunction in models of neurodevelopmental disorders. Rodent USVs can be categorised into different components and while manual classification is time consuming, advances in neural computing have allowed for fast and accurate identification and classification. Here, we adapt a convolutional neural network (CNN), VocalMat, created for analysing mice USVs, for use with rats. We codify a modified schema, adapted from that previously proposed by Wright et al. (2010), for classification, and compare the performance of our adaptation of VocalMat with a benchmark CNN, DeepSqueak. Additionally, we test the effect of inserting synthetic USVs into the training data of our classification network in order to reduce the workload involved in generating a training set. Our results show that the modified VocalMat outperformed the benchmark software on measures of both call identification, and classification. Additionally, we found that the augmentation of training data with synthetic images resulted in a marked improvement in the accuracy of VocalMat when it was subsequently used to analyse novel data. The resulting accuracy on the modified Wright categorizations was sufficiently high to allow for the application of this software in rat USV classification in laboratory conditions. Our findings also show that inserting synthetic USV calls into the training set leads to improvements in accuracy with little extra time-cost.

【3】 IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining标题：IPA-CLIP：将语音先验知识融入视觉和语言预训练链接：http://arxiv.org/pdf/2303.03144v1

作者：Chihaya Matsuhira,Marc A. Kastner,Takahiro Komamizu,Takatsugu Hirayama,Keisuke Doman,Yasutomo Kawanishi,Ichiro Ide机构：Nagoya University, Aichi, Japan, Kyoto University, Kyoto, Japan, University of Human Environments, Chukyo University, RIKEN备注：11 pages, 8 figures, 5 Tables摘要：近年来，大规模的视觉和语言（V &L）预训练已经成为许多多媒体系统的标准主干。虽然它甚至在看不见的情况下也表现出了非凡的性能，但它的表现方式往往与人类的直觉不符。特别地，它们通常不考虑输入的发音，而人类将利用该发音来理解语言，尤其是当涉及未知单词时。为此，本文在对比语言-图像预训练（CLIP）模型中加入语音先验知识，使其考虑语音输入之间的语音相似性。为了实现这一点，我们首先提出了一种音素嵌入方法，该方法利用国际音标（IPA）表提供的音素关系作为语音先验。接下来，通过提取冻结的CLIP文本编码器，我们训练基于IPA的嵌入训练语音编码器。IPA-CLIP模型由语音编码器和原始的CLIP编码器（图像和文本）组成。定量评价结果表明，使用所提出的音素嵌入方法，音素在嵌入空间上的分布能更准确地反映语音关系。此外，在一些多模态检索任务中，我们证实了所提出的发音编码器增强了文本编码器的性能，并且发音编码器以比文本编码器更语音化的方式处理无意义单词。最后，定性评价验证了语音编码器与人类感知语音相似度之间的相关性。摘要：Recently, large-scale Vision and Language (V &L) pretraining has become the standard backbone of many multimedia systems. While it has shown remarkable performance even in unseen situations, it often performs in ways not intuitive to humans. Particularly, they usually do not consider the pronunciation of the input, which humans would utilize to understand language, especially when it comes to unknown words. Thus, this paper inserts phonetic prior into Contrastive Language-Image Pretraining (CLIP), one of the V &L pretrained models, to make it consider the pronunciation similarity among its pronunciation inputs. To achieve this, we first propose a phoneme embedding that utilizes the phoneme relationships provided by the International Phonetic Alphabet (IPA) chart as a phonetic prior. Next, by distilling the frozen CLIP text encoder, we train a pronunciation encoder employing the IPA-based embedding. The proposed model named IPA-CLIP comprises this pronunciation encoder and the original CLIP encoders (image and text). Quantitative evaluation reveals that the phoneme distribution on the embedding space represents phonetic relationships more accurately when using the proposed phoneme embedding. Furthermore, in some multimodal retrieval tasks, we confirm that the proposed pronunciation encoder enhances the performance of the text encoder and that the pronunciation encoder handles nonsense words in a more phonetic manner than the text encoder. Finally, qualitative evaluation verifies the correlation between the pronunciation encoder and human perception regarding pronunc

【4】 Scaling strategies for on-device low-complexity source separation with Conv-Tasnet标题：基于Conv-Tasnet的设备上低复杂度信源分离扩展策略链接：http://arxiv.org/pdf/2303.03005v1

作者：Mohamed Nabih Ali,Francesco Paissan,Daniele Falavigna,Alessio Brutti机构：University of Trento, Trento, Italy, Fondazione Bruno Kessler, Trento, Italy摘要：最近，文献中提出了几种非常有效的单通道语音分离的神经方法。然而，由于这些模型的尺寸和复杂性，它们在低资源设备（例如助听器和耳机）上的使用仍然是一个挑战，并且还没有成熟的解决方案可用。尽管已经提出了基于修剪或压缩神经模型的方法，但是适合于特定应用领域的模型架构的设计通常需要启发式过程，而该启发式过程不容易移植到不同的低资源平台。给定公知的Conv-Tasnet语音分离架构的模块化性质，在本文中我们考虑直接控制模型的总体大小的三个参数，即：残差块的数目、分离块的重复数目和深度方向卷积中的通道数目，并且实验地评估它们如何影响语音分离性能。特别地，在Libri 2 Mix上执行的实验表明，膨胀的1D-Conv块的数量是最关键的参数，并且在残余块中使用额外膨胀允许减少性能下降。摘要：Recently, several very effective neural approaches for single-channel speech separation have been presented in the literature. However, due to the size and complexity of these models, their use on low-resource devices, e.g. for hearing aids, and earphones, is still a challenge and established solutions are not available yet. Although approaches based on either pruning or compressing neural models have been proposed, the design of a model architecture suitable for a certain application domain often requires heuristic procedures not easily portable to different low-resource platforms. Given the modular nature of the well-known Conv-Tasnet speech separation architecture, in this paper we consider three parameters that directly control the overall size of the model, namely: the number of residual blocks, the number of repetitions of the separation blocks and the number of channels in the depth-wise convolutions, and experimentally evaluate how they affect the speech separation performance. In particular, experiments carried out on the Libri2Mix show that the number of dilated 1D-Conv blocks is the most critical parameter and that the usage of extra-dilation in the residual blocks allows reducing the performance drop.

【5】 Time-frequency Network for Robust Speaker Recognition标题：用于稳健说话人识别的时频网络链接：http://arxiv.org/pdf/2303.02673v1

作者：Jiguo Li,Tianzi Zhang,Xiaobin Liu,Lirong Zheng机构：the School of information science and technology, Fudan University备注：5pages, 3 figures摘要：基于语音的生物特征识别系统的广泛应用通常需要高性能的说话人识别算法。然而，大多数用于说话人识别的现有工作或者在频域或者在时域中处理语音，这可能产生次优结果，因为时域和频域对于说话人识别都是重要的。本文试图从时域和频域两个方面对语音信号进行分析，通过提取和融合两个域的特征，提出了用于说话人识别的时频网络~（TFN）。基于深度神经网络的最新进展，提出了一种卷积神经网络，将原始语音波形和频谱编码为特定领域的特征，然后将这些特征融合并转换为分类特征空间，用于说话人识别。在TIMIT和LibriSpeech两个公开数据集上的实验结果表明，该框架能够有效地融合两个领域的信息，且识别效果优于现有的说话人识别方法.摘要：The wide deployment of speech-based biometric systems usually demands high-performance speaker recognition algorithms. However, most of the prior works for speaker recognition either process the speech in the frequency domain or time domain, which may produce suboptimal results because both time and frequency domains are important for speaker recognition. In this paper, we attempt to analyze the speech signal in both time and frequency domains and propose the time-frequency network~(TFN) for speaker recognition by extracting and fusing the features in the two domains. Based on the recent advance of deep neural networks, we propose a convolution neural network to encode the raw speech waveform and the frequency spectrum into domain-specific features, which are then fused and transformed into a classification feature space for speaker recognition. Experimental results on the publicly available datasets TIMIT and LibriSpeech show that our framework is effective to combine the information in the two domains and performs better than the state-of-the-art methods for speaker recognition.

【6】 Heterogeneous Graph Learning for Acoustic Event Classification标题：异构图学习用于声事件分类链接：http://arxiv.org/pdf/2303.02665v1

作者：Amir Shirian,Mona Ahmadian,Krishna Somandepalli,Tanaya Guha机构：University of Warwick, UK,University of Surrey, UK,Google Research, USA, University of Glasgow, UK备注：arXiv admin note: text overlap with arXiv:2207.07935摘要：异构图提供了一种紧凑、高效和可扩展的方式来对涉及多种不同模态的数据进行建模。这使得使用异构图来建模视听数据成为一种有吸引力的选择。然而，图结构在视听数据中并不自然地出现。用于视听数据的图是手动构造的，这既困难又次优。在这项工作中，我们通过（i）提出一种参数图构造策略来构造模态内边，以及（ii）学习跨模态边来解决这个问题。为此，我们发展了一个新的模型，异构图跨模态网络（HGCN），学习跨模态边。由于模型的参数化构造，该模型能够适应不同的时空尺度，而可学习的跨模态边有效地连接了跨模态的相关节点。在大型基准数据集（AudioSet）上的实验表明，该模型是最先进的（平均精度为0. 53），优于基于变换器的模型和其他基于图的模型。摘要：Heterogeneous graphs provide a compact, efficient, and scalable way to model data involving multiple disparate modalities. This makes modeling audiovisual data using heterogeneous graphs an attractive option. However, graph structure does not appear naturally in audiovisual data. Graphs for audiovisual data are constructed manually which is both difficult and sub-optimal. In this work, we address this problem by (i) proposing a parametric graph construction strategy for the intra-modal edges, and (ii) learning the crossmodal edges. To this end, we develop a new model, heterogeneous graph crossmodal network (HGCN) that learns the crossmodal edges. Our proposed model can adapt to various spatial and temporal scales owing to its parametric construction, while the learnable crossmodal edges effectively connect the relevant nodes across modalities. Experiments on a large benchmark dataset (AudioSet) show that our model is state-of-the-art (0.53 mean average precision), outperforming transformer-based models and other graph-based models.

【7】 Hybrid Y-Net Architecture for Singing Voice Separation标题：歌唱语音分离的混合Y-Net结构链接：http://arxiv.org/pdf/2303.02599v1

作者：Rashen Fernando,Pamudu Ranasinghe,Udula Ranasinghe,Janaka Wijayakulasooriya,Pantaleon Perera机构：dept. Electrical & Electronic Eng., University of Peradeniya, Peradeniya, Sri Lanka, dept. Electrical & Electronic Engineering, dept. Engineering Mathematics备注：Submitted for EUSIPCO23: 5 Pages, 7 figures摘要：提出了一种基于深度学习的神经网络结构, -Y网络，用于实现音乐源分离。该结构通过从频谱图和波形域提取特征来实现端到端的混合源分离。受U-Net架构的启发，Y-Net预测一个声谱图掩模来从混合信号中分离声源。实验结果证明了该结构在参数较少的情况下对音乐源分离的有效性。总之，我们的工作为提高音乐源分离的准确性和效率提供了一种有前景的方法。摘要：This research paper presents a novel deep learning-based neural network architecture, named Y-Net, for achieving music source separation. The proposed architecture performs end-to-end hybrid source separation by extracting features from both spectrogram and waveform domains. Inspired by the U-Net architecture, Y-Net predicts a spectrogram mask to separate vocal sources from a mixture signal. Our results demonstrate the effectiveness of the proposed architecture for music source separation with fewer parameters. Overall, our work presents a promising approach for improving the accuracy and efficiency of music source separation.

【8】 A General Framework for Learning Procedural Audio Models of Environmental Sounds标题：学习环境声音程序音频模型的通用框架链接：http://arxiv.org/pdf/2303.02396v1

作者：Danzel Serrano,Mark Cartwright机构：New Jersey Institute of Technology, Newark, NJ, USA备注：Accepted at ICASSP 2023摘要：本文介绍了程序（音频）变分自动编码器（ProVE）框架，它是学习环境声音的程序音频PA模型的通用方法，可提高合成的真实感，同时通过可调参数保持对生成声音的控制。该框架包括两个阶段：（i）音频类表示，其中通过训练音频自动编码器来定义潜在表示空间，以及（ii）控制映射，其中学习从音频导出的静态 时间控制变量和均匀噪声的随机样本的联合函数以代替音频编码器。我们通过各种表面上的脚步声效果示例来演示ProVE的使用。我们的结果表明，ProVE模型在声音保真度方面优于经典PA模型和基于逆向的方法，如通过Fr “echet音频距离（FAD）、最大平均差异（MMD）和主观评价所测量的，使其成为声音设计工作流的可行工具。摘要：This paper introduces the Procedural (audio) Variational autoEncoder (ProVE) framework as a general approach to learning Procedural Audio PA models of environmental sounds with an improvement to the realism of the synthesis while maintaining provision of control over the generated sound through adjustable parameters. The framework comprises two stages: (i) Audio Class Representation, in which a latent representation space is defined by training an audio autoencoder, and (ii) Control Mapping, in which a joint function of static temporal control variables derived from the audio and a random sample of uniform noise is learned to replace the audio encoder. We demonstrate the use of ProVE through the example of footstep sound effects on various surfaces. Our results show that ProVE models outperform both classical PA models and an adversarial-based approach in terms of sound fidelity, as measured by Fr 'echet Audio Distance (FAD), Maximum Mean Discrepancy (MMD), and subjective evaluations, making them feasible tools for sound design workflows.

【9】 The DKU Post-Challenge Audio-Visual Wake Word Spotting System for the 2021 MISP Challenge: Deep Analysis标题：2021年MISP挑战赛DKU赛后视听寻词系统：深度分析链接：http://arxiv.org/pdf/2303.02348v1

作者：Haoxu Wang,Ming Cheng,Qiang Fu,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China, Alibaba Group, China备注：Accepted by ICASSP 2023摘要：本文进一步探讨了我们之前在MISP挑战赛2021赛道1中排名第二的唤醒词识别系统。首先，我们研究了一种基于3D和2D卷积的鲁棒单峰方法，并采用简单注意模块（SimAM）来提高系统的性能。第二，我们探索了数据扩充方法的不同组合以获得更好的性能。最后，研究了分数级融合、级联融合和神经网络融合等融合策略。我们所提出的多模态系统利用多模态特征并使用互补的视觉信息来减轻在复杂声学场景中纯音频系统的性能下降。在竞争数据库的评估集中，我们的系统获得了2.15%的误拒率和3.44%的误报警率，与以前的系统相比，相对提高了21%，达到了新的最佳性能。摘要：This paper further explores our previous wake word spotting system ranked 2-nd in Track 1 of the MISP Challenge 2021. First, we investigate a robust unimodal approach based on 3D and 2D convolution and adopt the simple attention module (SimAM) for our system to improve performance. Second, we explore different combinations of data augmentation methods for better performance. Finally, we study the fusion strategies, including score-level, cascaded and neural fusion. Our proposed multimodal system leverages multimodal features and uses the complementary visual information to mitigate the performance degradation of audio-only systems in complex acoustic scenarios. Our system obtains a false reject rate of 2.15% and a false alarm rate of 3.44% in the evaluation set of the competition database, which achieves the new state-of-the-art performance by 21% relative improvement compared to previous systems.

【10】 End-to-End Speech Recognition: A Survey标题：端到端语音识别：综述链接：http://arxiv.org/pdf/2303.03329v1

作者：Rohit Prabhavalkar,Takaaki Hori,Tara N. Sainath,Ralf Schlüter,Shinji Watanabe备注：Submitted to IEEEACM Transactions on Audio, Speech, and Language Processing摘要：在过去十年的自动语音识别（ASR）研究中，与没有深度学习的建模相比，深度学习的引入大大降低了单词错误率，相对减少了50%以上。随着这一转变，许多全神经ASR架构被引入。这些所谓的端到端（E2E）模型提供了高度集成的、完全神经的ASR模型，这些模型强烈依赖于一般机器学习知识，从数据中学习的一致性更高，同时较少依赖于ASR领域特定的经验。深度学习的成功和热情采用伴随着更通用的模型架构，导致E2E模型现在成为突出的ASR方法。本文的目标是对E2E ASR模型进行分类和改进，讨论它们的特性以及与经典的基于隐马尔可夫模型（HMM）的ASR体系结构的关系。本工作涵盖了E2E ASR的所有相关方面：建模、训练、解码和外部语言模型集成，同时讨论性能和部署机会，以及对潜在未来开发的展望。摘要：In the last decade of automatic speech recognition (ASR) research, the introduction of deep learning brought considerable reductions in word error rate of more than 50% relative, compared to modeling without deep learning. In the wake of this transition, a number of all-neural ASR architectures were introduced. These so-called end-to-end (E2E) models provide highly integrated, completely neural ASR models, which rely strongly on general machine learning knowledge, learn more consistently from data, while depending less on ASR domain-specific experience. The success and enthusiastic adoption of deep learning accompanied by more generic model architectures lead to E2E models now becoming the prominent ASR approach. The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements, and to discuss their properties and their relation to the classical hidden Markov model (HMM) based ASR architecture. All relevant aspects of E2E ASR are covered in this work: modeling, training, decoding, and external language model integration, accompanied by discussions of performance and deployment opportunities, as well as an outlook into potential future developments.

【11】Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis标题：语音情感分析中的预训练模型表示及其抗噪能力链接：http://arxiv.org/pdf/2303.03177v1

作者：Vikramjit Mitra,Vasudha Kowtha,Hsiang-Yun Sherry Chien,Erdrin Azemi,Carlos Avendano机构：Apple备注：5 pages, conference摘要：预训练的模型表示已经在语音识别、自然语言处理和其他应用中展示了最先进的性能。语音模型，例如来自Transformers的双向编码器表示（BERT）和隐藏单元BERT（HuBERT），已经使得能够生成词汇和声学表示以有益于语音识别应用。我们调查了使用预先训练的模型表示来估计维度情绪，如激活，效价和优势，从语音。我们观察到，虽然配价可能严重依赖于词汇表征，但激活和优势主要依赖于声学信息。在这项工作中，我们使用来自预训练模型的多模态融合表示来生成最先进的语音情感估计，并且我们显示与标准声学和词汇基线相比，效价估计的一致性相关系数（CCC）相对提高了100%和30%。最后，我们研究了预训练模型表示对噪声和混响退化的鲁棒性，并注意到词汇和声学表示受到不同的影响。我们发现词汇表征比声学表征对失真更鲁棒，并证明了从多模态模型中提取知识有助于提高基于声学的模型的噪声鲁棒性。摘要：Pre-trained model representations have demonstrated state-of-the-art performance in speech recognition, natural language processing, and other applications. Speech models, such as Bidirectional Encoder Representations from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled generating lexical and acoustic representations to benefit speech recognition applications. We investigated the use of pre-trained model representations for estimating dimensional emotions, such as activation, valence, and dominance, from speech. We observed that while valence may rely heavily on lexical representations, activation and dominance rely mostly on acoustic information. In this work, we used multi-modal fusion representations from pre-trained models to generate state-of-the-art speech emotion estimation, and we showed a 100% and 30% relative improvement in concordance correlation coefficient (CCC) on valence estimation compared to standard acoustic and lexical baselines. Finally, we investigated the robustness of pre-trained model representations against noise and reverberation degradation and noticed that lexical and acoustic representations are impacted differently. We discovered that lexical representations are more robust to distortions compared to acoustic representations, and demonstrated that knowledge distillation from a multi-modal model helps to improve the noise-robustness of acoustic-based models.

【12】Cross-Lingual Transfer Learning for Alzheimer's Detection From Spontaneous Speech标题：基于跨语言迁移学习的自然语音阿尔茨海默病检测

链接：http://arxiv.org/pdf/2303.03049v1

作者：Bastiaan Tamm,Rik Vandenberghe,Hugo Van hamme机构：KU Leuven, Belgium备注：2 pages, submitted to ICASSP 2023摘要：阿尔茨海默病（AD）是一种进行性神经退行性疾病，最常与记忆缺陷和认知下降有关。随着人口老龄化，人们对自动化的认知障碍检测方法产生了浓厚的兴趣。近年来引起注意的一种方法是通过自发语音进行AD检测。虽然结果是有希望的，但不确定学习的语音特征是否可以跨语言推广。为了填补这一空白，组织了ADReSS-M挑战赛。本文介绍我们为ICASSP-2023信号处理大挑战赛（SPGC）提交的参赛作品。该模型在图片描述任务的228个英语样本上进行训练，并且仅使用8个样本就被转移到希腊语。我们获得了82.6%的AD检测准确率，4.345的认知评分预测均方根误差，在24名竞争者中排名第二。摘要：Alzheimer's disease (AD) is a progressive neurodegenerative disease most often associated with memory deficits and cognitive decline. With the aging population, there has been much interest in automated methods for cognitive impairment detection. One approach that has attracted attention in recent years is AD detection through spontaneous speech. While the results are promising, it is not certain whether the learned speech features can be generalized across languages. To fill this gap, the ADReSS-M challenge was organized. This paper presents our submission to this ICASSP-2023 Signal Processing Grand Challenge (SPGC). The model was trained on 228 English samples of a picture description task and was transferred to Greek using only 8 samples. We obtained an accuracy of 82.6% for AD detection, a root-mean-square error of 4.345 for cognitive score prediction, and ranked 2nd place in the competition out of 24 competitors.

【13】 FoundationTTS: Text-to-Speech for ASR Custmization with Generative Language Model标题：FoundationTTS：基于产生式语言模型的ASR定制文语转换链接：http://arxiv.org/pdf/2303.02939v1

作者：Ruiqing Xue,Yanqing Liu,Lei He,Xu Tan,Linquan Liu,Edward Lin,Sheng Zhao机构：Microsoft摘要：神经文本到语音（TTS）通常由具有单独优化的声学模型和声码器的级联结构或具有连续梅尔谱图或自提取语音帧作为中间表示以在联合训练中桥接声学模型和声码器的端到端结构组成，其遭受两个限制：1）连续的声学帧很难仅用音素来预测，还需要持续时间或音高等声学信息来解决一对多问题，这在大规模和噪声数据集上不容易扩展;2）多样性语音输出对于连续语音特征不是直接的，并且经常需要复杂的VAE或基于流的模型。本文提出了一种新的语音合成系统--FoundationTTS，该系统从神经音频编解码器中提取离散语音标记，并基于大语言建模的声学模型来同时优化语言标记和声学标记。具体来说，1）我们提出了一种基于对抗训练矢量量化自动编码器（VQ-GAN）的分层编解码网络，首先用细粒度编解码器提取连续帧级语音表示，粗粒度编解码器用较少的量化器重构连续语音帧;2）将语音标记、语言标记、说话人标记与一个大的语言模型联合优化，对离散的语音标记进行自回归预测。实验结果表明，与基线系统相比，FoundationTTS的MOS增益为+0.14;在ASR定制任务中，该方法在两个强定制ASR基线上的WERR分别为7.09%和10.35%.摘要：Neural text to speech (TTS) generally consists of cascaded architecture with separately optimized acoustic model and vocoder or end-to-end architecture with continuous mel-spectrograms or self-extracted speech frames as the intermediate representations to bridge acoustic model and vocoder in joint training, which suffers from two limitations: 1) the continuous acoustic frames are hard to predict with phoneme only, acoustic information are also needed like duration or pitch to solve the one-to-many problem, which is not easy to scale on large scale and noise dataset; 2) diverse speech output is not straightforward with continuous speech features and complex VAE or flow based models are often needed. In this paper, we propose FoundationTTS, a new speech synthesis system with discrete speech tokens extraction from neural audio codec and large language modelling based acoustic model for optimizing linguistic and acoustic tokens simultaneously. Specifically, 1) we propose a hierarchical codec network based on vector-quantized auto-encoders with adversarial training (VQ-GAN) to first extract continuous frame-level speech representations with fine-grained codec, and the coarse-grained codec reconstructs the continuous speech frame with less quantizers; 2) we jointly optimize speech token, linguistic tokens, speaker token together with a large language model and autoregressively predict the discrete speech tokens. Experiments show that FoundationTTS achieves a MOS gain +0.14 compared to baseline system, and in ASR customizati

【14】 A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS标题：阅读和自发性文语转换中自我监督言语表征的对比研究链接：http://arxiv.org/pdf/2303.02719v1

作者：Siyang Wang,Gustav Eje Henter,Joakim Gustafson,Éva Székely机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 2 figures. In submission摘要：最近的工作已经探索了使用诸如wav2vec2.0的自监督学习（SSL）语音表示作为标准两阶段TTS中的表示媒介，来代替传统使用的Mel谱图。然而，目前还不清楚哪种语音SSL更适合TTS，以及读TTS和自发TTS之间的性能是否不同，后者可能更具挑战性。本研究旨在通过在阅读语料库和自发语料库上使用两阶段TTS测试几种语音SSL（包括同一SSL的不同层）来解决这些问题，同时保持TTS模型架构和训练设置不变。听力测试结果表明，12层wav2vec2. 0（ASR finetuned）中的第9层在朗读和自发TTS中均优于其他测试的SSL和Mel谱图。我们的工作阐明了语音SSL如何能够容易地改进当前的TTS系统，以及SSL如何在具有挑战性的TTS生成任务中进行比较。音频示例可在www.example.com上找到https: www.speech.kth.se tts-demos ssr_tts摘要：Recent work has explored using self-supervised learning (SSL) speech representations such as wav2vec2.0 as the representation medium in standard two-stage TTS, in place of conventionally used mel-spectrograms. It is however unclear which speech SSL is the better fit for TTS, and whether or not the performance differs between read and spontaneous TTS, the later of which is arguably more challenging. This study aims at addressing these questions by testing several speech SSLs, including different layers of the same SSL, in two-stage TTS on both read and spontaneous corpora, while maintaining constant TTS model architecture and training settings. Results from listening tests show that the 9th layer of 12-layer wav2vec2.0 (ASR finetuned) outperforms other tested SSLs and mel-spectrogram, in both read and spontaneous TTS. Our work sheds light on both how speech SSL can readily improve current TTS systems, and how SSLs compare in the challenging generative task of TTS. Audio examples can be found at https: www.speech.kth.se tts-demos ssr_tts

【15】 Requirements for Mass Adoption of Assistive Listening Technology by the General Public标题：大众大众广泛采用辅助听力技术的要求链接：http://arxiv.org/pdf/2303.02523v1

作者：Thomas B. Kaufmann,Mehdi Foroogozar,Julie Liss,Visar Berisha机构：Arizona State University, Department of Speech and Hearing Science, Tempe, AZ, USA备注：Submitted to ICASSP 2023摘要：辅助听音系统（ALS）极大地提高了语音可懂度并减少了听音努力。这是非常有可能的，基本上每个人，不仅是听力损失的个人，将受益于增加信噪比的ALS提供几乎任何听力情况。然而，除了患有严重到极重度听力损失的人之外，很少有人使用ALS。迄今为止，尚未系统地调查这种不良采用的原因。作者假设辅助听力技术的不良采用的原因包括（1）不能使用个人拥有的接收设备，（2）缺乏高保真立体声，（3）接收设备不能提供无阻塞的听力体验，（4）来自对准延迟的失真，以及（5）缺乏到可用辅助听力音频信号的自动连接。我们针对这些问题提出解决方案，为辅助听力技术的广泛应用铺平道路。摘要：Assistive listening systems (ALSs) dramatically increase speech intelligibility and reduce listening effort. It is very likely that essentially everyone, not only individuals with hearing loss, would benefit from the increased signal-to-noise ratio an ALS provides in almost any listening scenario. However, ALSs are rarely used by anyone other than people with severe to profound hearing losses. To date, the reasons for this poor adoption have not been systematically investigated. The authors hypothesize that the reasons for poor adoption of assistive listening technology include (1) an inability to use personally owned receiving devices, (2) a lack of high-fidelity stereo sound, (3) receiving devices not providing an unoccluded listening experience, (4) distortion from alignment delay and (5) a lack of automatic connectivity to an available assistive listening audio signal. We propose solutions to each of these problems in an effort to pave the way for mass adoption of assistive listening technology.

【16】 DeFT-AN: Dense Frequency-Time Attentive Network for Multichannel Speech Enhancement标题：DIFT-AN：用于多通道语音增强的密集频率-时间注意网络链接：http://arxiv.org/pdf/2212.07570v5

作者：Dongheon Lee,Jung-Woo Choi机构：of Science and Technology (KAIST), Daejeon , South Korea (e-mail:, methods using time-domain waveforms [,] have been devel-, oped to overcome the shortcomings of STFT approaches such, as the phase reconstruction issue [,]. In particular, with the dis-Journal-ref：IEEE Signal Processing Letters, Vol. 30, pp. 155-159, 2023摘要：None摘要：In this study, we propose a dense frequency-time attentive network (DeFT-AN) for multichannel speech enhancement. DeFT-AN is a mask estimation network that predicts a complex spectral masking pattern for suppressing the noise and reverberation embedded in the short-time Fourier transform (STFT) of an input signal. The proposed mask estimation network incorporates three different types of blocks for aggregating information in the spatial, spectral, and temporal dimensions. It utilizes a spectral transformer with a modified feed-forward network and a temporal conformer with sequential dilated convolutions. The use of dense blocks and transformers dedicated to the three different characteristics of audio signals enables more comprehensive enhancement in noisy and reverberant environments. The remarkable performance of DeFT-AN over state-of-the-art multichannel models is demonstrated based on two popular noisy and reverberant datasets in terms of various metrics for speech quality and intelligibility.
eess.AS音频处理

【1】 End-to-End Speech Recognition: A Survey标题：端到端语音识别：综述链接：http://arxiv.org/pdf/2303.03329v1

*与cs.SD语音【10】为同一篇

作者：Rohit Prabhavalkar,Takaaki Hori,Tara N. Sainath,Ralf Schlüter,Shinji Watanabe备注：Submitted to IEEEACM Transactions on Audio, Speech, and Language Processing摘要：在过去十年的自动语音识别（ASR）研究中，与没有深度学习的建模相比，深度学习的引入大大降低了单词错误率，相对减少了50%以上。随着这一转变，许多全神经ASR架构被引入。这些所谓的端到端（E2E）模型提供了高度集成的、完全神经的ASR模型，这些模型强烈依赖于一般机器学习知识，从数据中学习的一致性更高，同时较少依赖于ASR领域特定的经验。深度学习的成功和热情采用伴随着更通用的模型架构，导致E2E模型现在成为突出的ASR方法。本文的目标是对E2E ASR模型进行分类和改进，讨论它们的特性以及与经典的基于隐马尔可夫模型（HMM）的ASR体系结构的关系。本工作涵盖了E2E ASR的所有相关方面：建模、训练、解码和外部语言模型集成，同时讨论性能和部署机会，以及对潜在未来开发的展望。摘要：In the last decade of automatic speech recognition (ASR) research, the introduction of deep learning brought considerable reductions in word error rate of more than 50% relative, compared to modeling without deep learning. In the wake of this transition, a number of all-neural ASR architectures were introduced. These so-called end-to-end (E2E) models provide highly integrated, completely neural ASR models, which rely strongly on general machine learning knowledge, learn more consistently from data, while depending less on ASR domain-specific experience. The success and enthusiastic adoption of deep learning accompanied by more generic model architectures lead to E2E models now becoming the prominent ASR approach. The goal of this survey is to provide a taxonomy of E2E ASR models and corresponding improvements, and to discuss their properties and their relation to the classical hidden Markov model (HMM) based ASR architecture. All relevant aspects of E2E ASR are covered in this work: modeling, training, decoding, and external language model integration, accompanied by discussions of performance and deployment opportunities, as well as an outlook into potential future developments.

【2】Pre-trained Model Representations and their Robustness against Noise for Speech Emotion Analysis标题：语音情感分析中的预训练模型表示及其抗噪能力链接：http://arxiv.org/pdf/2303.03177v1

*与cs.SD语音【11】为同一篇

作者：Vikramjit Mitra,Vasudha Kowtha,Hsiang-Yun Sherry Chien,Erdrin Azemi,Carlos Avendano机构：Apple备注：5 pages, conference摘要：预训练的模型表示已经在语音识别、自然语言处理和其他应用中展示了最先进的性能。语音模型，例如来自Transformers的双向编码器表示（BERT）和隐藏单元BERT（HuBERT），已经使得能够生成词汇和声学表示以有益于语音识别应用。我们调查了使用预先训练的模型表示来估计维度情绪，如激活，效价和优势，从语音。我们观察到，虽然配价可能严重依赖于词汇表征，但激活和优势主要依赖于声学信息。在这项工作中，我们使用来自预训练模型的多模态融合表示来生成最先进的语音情感估计，并且我们显示与标准声学和词汇基线相比，效价估计的一致性相关系数（CCC）相对提高了100%和30%。最后，我们研究了预训练模型表示对噪声和混响退化的鲁棒性，并注意到词汇和声学表示受到不同的影响。我们发现词汇表征比声学表征对失真更鲁棒，并证明了从多模态模型中提取知识有助于提高基于声学的模型的噪声鲁棒性。摘要：Pre-trained model representations have demonstrated state-of-the-art performance in speech recognition, natural language processing, and other applications. Speech models, such as Bidirectional Encoder Representations from Transformers (BERT) and Hidden units BERT (HuBERT), have enabled generating lexical and acoustic representations to benefit speech recognition applications. We investigated the use of pre-trained model representations for estimating dimensional emotions, such as activation, valence, and dominance, from speech. We observed that while valence may rely heavily on lexical representations, activation and dominance rely mostly on acoustic information. In this work, we used multi-modal fusion representations from pre-trained models to generate state-of-the-art speech emotion estimation, and we showed a 100% and 30% relative improvement in concordance correlation coefficient (CCC) on valence estimation compared to standard acoustic and lexical baselines. Finally, we investigated the robustness of pre-trained model representations against noise and reverberation degradation and noticed that lexical and acoustic representations are impacted differently. We discovered that lexical representations are more robust to distortions compared to acoustic representations, and demonstrated that knowledge distillation from a multi-modal model helps to improve the noise-robustness of acoustic-based models.

【3】 Cross-Lingual Transfer Learning for Alzheimer's Detection From Spontaneous Speech标题：基于跨语言迁移学习的自然语音阿尔茨海默病检测

链接：http://arxiv.org/pdf/2303.03049v1

*与cs.SD语音【12】为同一篇

作者：Bastiaan Tamm,Rik Vandenberghe,Hugo Van hamme机构：KU Leuven, Belgium备注：2 pages, submitted to ICASSP 2023摘要：阿尔茨海默病（AD）是一种进行性神经退行性疾病，最常与记忆缺陷和认知下降有关。随着人口老龄化，人们对自动化的认知障碍检测方法产生了浓厚的兴趣。近年来引起注意的一种方法是通过自发语音进行AD检测。虽然结果是有希望的，但不确定学习的语音特征是否可以跨语言推广。为了填补这一空白，组织了ADReSS-M挑战赛。本文介绍我们为ICASSP-2023信号处理大挑战赛（SPGC）提交的参赛作品。该模型在图片描述任务的228个英语样本上进行训练，并且仅使用8个样本就被转移到希腊语。我们获得了82.6%的AD检测准确率，4.345的认知评分预测均方根误差，在24名竞争者中排名第二。摘要：Alzheimer's disease (AD) is a progressive neurodegenerative disease most often associated with memory deficits and cognitive decline. With the aging population, there has been much interest in automated methods for cognitive impairment detection. One approach that has attracted attention in recent years is AD detection through spontaneous speech. While the results are promising, it is not certain whether the learned speech features can be generalized across languages. To fill this gap, the ADReSS-M challenge was organized. This paper presents our submission to this ICASSP-2023 Signal Processing Grand Challenge (SPGC). The model was trained on 228 English samples of a picture description task and was transferred to Greek using only 8 samples. We obtained an accuracy of 82.6% for AD detection, a root-mean-square error of 4.345 for cognitive score prediction, and ranked 2nd place in the competition out of 24 competitors.

【4】 Variational Inference of Structured Line Spectra Exploiting Group-Sparsity标题：利用群稀疏性的结构线谱的变分推断

链接：http://arxiv.org/pdf/2303.03017v1

作者：Jakob Möderl,Franz Pernkopf,Klaus Witrisal,Erik Leitinger机构：The authors are with the Signal Processing and Speech CommunicationsLaboratory at Graz University of Technology备注：13 Pages, 5 Figures. Submitted to IEEE Transactions on Signal Processing on 6th of March, 2023摘要：本文提出了一种变分推理算法，将信号分解为多组相关谱线。每个组中的谱线与该组内所有谱线共有的组参数相关联。该算法利用Bernoulli-Gamma-Gaussian分层先验模型联合估计组参数、组内谱线数和组数，提高了稀疏解。变分推理以最大化证据下界（ELBO）为目标，提供后验概率密度函数（PDF）的解析近似，并给出测量噪声方差等附加模型参数的估计。当组的激活变量和相关组参数（例如基频和相应的高阶谐波）被估计为点估计时，剩余参数例如谱线的复振幅及其精度参数被估计为近似后验PDF。我们在三个不同的推理问题上证明了所提算法的通用性和性能。特别地，所提出的算法被应用于多基音估计问题、基于雷达信号的扩展目标估计问题、使用合成测量的变分模式分解（VMD）以及使用Bach-10数据集的实数多基音估计问题。实验结果表明，该算法在所有三个推理问题上都优于现有的基于模型的算法和预训练算法。摘要：In this paper, we present a variational inference algorithm that decomposes a signal into multiple groups of related spectral lines. The spectral lines in each group are associated with a group parameter common to all spectral lines within the group. The proposed algorithm jointly estimates the group parameters, the number of spetral lines within a group, and the number of groups exploiting a Bernoulli-Gamma-Gaussian hierarchical prior model which promotes sparse solutions. Aiming to maximize the evidence lower bound (ELBO), variational inference provides analytic approximations of the posterior probability density functions (PDFs) and also gives estimates of the additional model parameters such as the measurement noise variance. While the activation variables of the groups and the associated group parameters (such as fundamental frequencies and the corresponding higher order harmonics) are estimated as point estimates, the remaining parameters such as the complex amplitudes of the spectral lines and their precision parameters are estimated as approximate posterior PDFs. We demonstrate the versatility and performance of the proposed algorithm on three different inference problems. In particular, the proposed algorithm is applied to the multi-pitch estimation problem, the radar signal-based extended object estimation problem, and variational mode decomposition (VMD) using synthetic measurements and to real multi-pitch estimation problem using the Bach-10 dataset. The results show that the proposed algorithm outperforms state-of-the-art model-based and pre-trained algorithms on all three inference problems.

【5】 FoundationTTS: Text-to-Speech for ASR Custmization with Generative Language Model标题：FoundationTTS：基于产生式语言模型的ASR定制文语转换链接：http://arxiv.org/pdf/2303.02939v1

*与cs.SD语音【13】为同一篇

作者：Ruiqing Xue,Yanqing Liu,Lei He,Xu Tan,Linquan Liu,Edward Lin,Sheng Zhao机构：Microsoft摘要：神经文本到语音（TTS）通常由具有单独优化的声学模型和声码器的级联结构或具有连续梅尔谱图或自提取语音帧作为中间表示以在联合训练中桥接声学模型和声码器的端到端结构组成，其遭受两个限制：1）连续的声学帧很难仅用音素来预测，还需要持续时间或音高等声学信息来解决一对多问题，这在大规模和噪声数据集上不容易扩展;2）多样性语音输出对于连续语音特征不是直接的，并且经常需要复杂的VAE或基于流的模型。本文提出了一种新的语音合成系统--FoundationTTS，该系统从神经音频编解码器中提取离散语音标记，并基于大语言建模的声学模型来同时优化语言标记和声学标记。具体来说，1）我们提出了一种基于对抗训练矢量量化自动编码器（VQ-GAN）的分层编解码网络，首先用细粒度编解码器提取连续帧级语音表示，粗粒度编解码器用较少的量化器重构连续语音帧;2）将语音标记、语言标记、说话人标记与一个大的语言模型联合优化，对离散的语音标记进行自回归预测。实验结果表明，与基线系统相比，FoundationTTS的MOS增益为+0.14;在ASR定制任务中，该方法在两个强定制ASR基线上的WERR分别为7.09%和10.35%.摘要：Neural text to speech (TTS) generally consists of cascaded architecture with separately optimized acoustic model and vocoder or end-to-end architecture with continuous mel-spectrograms or self-extracted speech frames as the intermediate representations to bridge acoustic model and vocoder in joint training, which suffers from two limitations: 1) the continuous acoustic frames are hard to predict with phoneme only, acoustic information are also needed like duration or pitch to solve the one-to-many problem, which is not easy to scale on large scale and noise dataset; 2) diverse speech output is not straightforward with continuous speech features and complex VAE or flow based models are often needed. In this paper, we propose FoundationTTS, a new speech synthesis system with discrete speech tokens extraction from neural audio codec and large language modelling based acoustic model for optimizing linguistic and acoustic tokens simultaneously. Specifically, 1) we propose a hierarchical codec network based on vector-quantized auto-encoders with adversarial training (VQ-GAN) to first extract continuous frame-level speech representations with fine-grained codec, and the coarse-grained codec reconstructs the continuous speech frame with less quantizers; 2) we jointly optimize speech token, linguistic tokens, speaker token together with a large language model and autoregressively predict the discrete speech tokens. Experiments show that FoundationTTS achieves a MOS gain +0.14 compared to baseline system, and in ASR customizati

【6】 A Comparative Study of Self-Supervised Speech Representations in Read and Spontaneous TTS标题：阅读和自发性文语转换中自我监督言语表征的对比研究链接：http://arxiv.org/pdf/2303.02719v1

*与cs.SD语音【14】为同一篇

作者：Siyang Wang,Gustav Eje Henter,Joakim Gustafson,Éva Székely机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Stockholm, Sweden备注：5 pages, 2 figures. In submission摘要：最近的工作已经探索了使用诸如wav2vec2.0的自监督学习（SSL）语音表示作为标准两阶段TTS中的表示媒介，来代替传统使用的Mel谱图。然而，目前还不清楚哪种语音SSL更适合TTS，以及读TTS和自发TTS之间的性能是否不同，后者可能更具挑战性。本研究旨在通过在阅读语料库和自发语料库上使用两阶段TTS测试几种语音SSL（包括同一SSL的不同层）来解决这些问题，同时保持TTS模型架构和训练设置不变。听力测试结果表明，12层wav2vec2. 0（ASR finetuned）中的第9层在朗读和自发TTS中均优于其他测试的SSL和Mel谱图。我们的工作阐明了语音SSL如何能够容易地改进当前的TTS系统，以及SSL如何在具有挑战性的TTS生成任务中进行比较。音频示例可在www.example.com上找到https: www.speech.kth.se tts-demos ssr_tts摘要：Recent work has explored using self-supervised learning (SSL) speech representations such as wav2vec2.0 as the representation medium in standard two-stage TTS, in place of conventionally used mel-spectrograms. It is however unclear which speech SSL is the better fit for TTS, and whether or not the performance differs between read and spontaneous TTS, the later of which is arguably more challenging. This study aims at addressing these questions by testing several speech SSLs, including different layers of the same SSL, in two-stage TTS on both read and spontaneous corpora, while maintaining constant TTS model architecture and training settings. Results from listening tests show that the 9th layer of 12-layer wav2vec2.0 (ASR finetuned) outperforms other tested SSLs and mel-spectrogram, in both read and spontaneous TTS. Our work sheds light on both how speech SSL can readily improve current TTS systems, and how SSLs compare in the challenging generative task of TTS. Audio examples can be found at https: www.speech.kth.se tts-demos ssr_tts

【7】 Requirements for Mass Adoption of Assistive Listening Technology by the General Public标题：公众广泛采用辅助听力技术的要求链接：http://arxiv.org/pdf/2303.02523v1

*与cs.SD语音【15】为同一篇

作者：Thomas B. Kaufmann,Mehdi Foroogozar,Julie Liss,Visar Berisha机构：Arizona State University, Department of Speech and Hearing Science, Tempe, AZ, USA备注：Submitted to ICASSP 2023摘要：辅助听音系统（ALS）极大地提高了语音可懂度并减少了听音努力。这是非常有可能的，基本上每个人，不仅是听力损失的个人，将受益于增加信噪比的ALS提供几乎任何听力情况。然而，除了患有严重到极重度听力损失的人之外，很少有人使用ALS。迄今为止，尚未系统地调查这种不良采用的原因。作者假设辅助听力技术的不良采用的原因包括（1）不能使用个人拥有的接收设备，（2）缺乏高保真立体声，（3）接收设备不能提供无阻塞的听力体验，（4）来自对准延迟的失真，以及（5）缺乏到可用辅助听力音频信号的自动连接。我们针对这些问题提出解决方案，为辅助听力技术的广泛应用铺平道路。摘要：Assistive listening systems (ALSs) dramatically increase speech intelligibility and reduce listening effort. It is very likely that essentially everyone, not only individuals with hearing loss, would benefit from the increased signal-to-noise ratio an ALS provides in almost any listening scenario. However, ALSs are rarely used by anyone other than people with severe to profound hearing losses. To date, the reasons for this poor adoption have not been systematically investigated. The authors hypothesize that the reasons for poor adoption of assistive listening technology include (1) an inability to use personally owned receiving devices, (2) a lack of high-fidelity stereo sound, (3) receiving devices not providing an unoccluded listening experience, (4) distortion from alignment delay and (5) a lack of automatic connectivity to an available assistive listening audio signal. We propose solutions to each of these problems in an effort to pave the way for mass adoption of assistive listening technology.

【8】 Fixed-point quantization aware training for on-device keyword-spotting标题：用于设备上关键词检测的定点量化感知训练链接：http://arxiv.org/pdf/2303.02523v1

作者：Sashank Macha,Om Oza,Alex Escott,Francesco Caliva,Robbie Armitano,Santosh Kumar Cheekatmalla,Sree Hari Krishnan Parthasarathi,Yuzong Liu机构：Francesco Caliv´a⋆, ⋆Alexa Perceptual Technologies, †Alexa AI, AmazonJournal-ref：International Conference on Acoustics, Speech, and Signal Processing, 2023摘要：定点（FXP）推理已被证明适用于计算资源有限的嵌入式设备，但模型训练仍在浮点（FLP）中持续执行。FXP训练还没有被充分探索，从FLP到FXP的非平凡转换呈现不可避免的性能下降。提出了一种训练和获取FXP卷积关键字识别模型的新方法。我们将我们的方法与两种量化感知训练（QAT）技术-压缩权重分布和模型参数的绝对余弦正则化相结合，并提出了将QAT扩展到瞬态变量的技术，否则会被先前的范式所忽略。在Google Speech Commands v2数据集上的实验结果表明，我们可以在不损失准确度的情况下将模型精度降低到4位。此外，在一个内部KWS数据集上，我们表明，与全精度FLP模型相比，我们的8位FXP-QAT模型在固定错误拒绝率下的相对错误发现率有4-6%的改善。在推理过程中，我们认为FXP-QAT消除了q格式归一化，并支持使用低位累加器，同时最大化SIMD吞吐量，以减少用户感知的延迟。我们证明，我们可以减少68%的执行时间，而不损害KWS模型的预测性能或要求模型架构的变化。我们的工作提供了新的发现，有助于该领域的未来研究，并实现准确和有效的模型。摘要：Fixed-point (FXP) inference has proven suitable for embedded devices with limited computational resources, and yet model training is continually performed in floating-point (FLP). FXP training has not been fully explored and the non-trivial conversion from FLP to FXP presents unavoidable performance drop. We propose a novel method to train and obtain FXP convolutional keyword-spotting (KWS) models. We combine our methodology with two quantization-aware-training (QAT) techniques - squashed weight distribution and absolute cosine regularization for model parameters, and propose techniques for extending QAT over transient variables, otherwise neglected by previous paradigms. Experimental results on the Google Speech Commands v2 dataset show that we can reduce model precision up to 4-bit with no loss in accuracy. Furthermore, on an in-house KWS dataset, we show that our 8-bit FXP-QAT models have a 4-6% improvement in relative false discovery rate at fixed false reject rate compared to full precision FLP models. During inference we argue that FXP-QAT eliminates q-format normalization and enables the use of low-bit accumulators while maximizing SIMD throughput to reduce user perceived latency. We demonstrate that we can reduce execution time by 68% without compromising KWS model's predictive performance or requiring model architectural changes. Our work provides novel findings that aid future research in this area and enable accurate and efficient models.

【9】 Evaluating Parameter-Efficient Transfer Learning Approaches on SURE Benchmark for Speech Understanding标题：SURE语音理解基准上参数高效迁移学习方法的评价链接：http://arxiv.org/pdf/2303.03267v1

*与cs.SD语音【1】为同一篇

作者：Yingting Li,Ambuj Mehrish,Shuai Zhao,Rishabh Bhardwaj,Amir Zadeh,Navonil Majumder,Rada Mihalcea,Soujanya Poria机构：Bo Cheng, Beijing University of Posts and Telecommunications, China, University of Michigan, USA, Singapore University of Technology and Design, Singapore, Amazon Science, USA备注：ICASSP 2023摘要：微调被广泛用作从预先训练的模型进行迁移学习的默认算法。然而，当在迁移学习期间需要针对各个下游任务更新大型预训练模型的所有参数时，可能出现参数低效。随着参数数量的增长，微调容易出现过拟合和灾难性遗忘。此外，当模型用于许多任务时，完全微调可能变得非常昂贵。为了缓解这个问题，已经提出了参数有效的迁移学习算法，例如适配器和前缀调整，作为引入一些可训练参数的方式，这些参数可以插入到大的预先训练的语言模型中，例如BERT和HuBERT。在这篇文章中，我们介绍了语音理解评估（SURE）基准，用于各种语音处理任务的参数有效学习。此外，我们还引入了一种基于1D卷积的新适配器ConvAdapter。我们证明ConvAdapter优于标准适配器，同时显示出与前缀调整和LoRA相当的性能，在SURE中的一些任务上仅使用0.94%的可训练参数。我们进一步探讨了参数有效迁移学习在语音合成任务如文语转换（TTS）中的有效性。摘要：Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models. Parameter inefficiency can however arise when, during transfer learning, all the parameters of a large pre-trained model need to be updated for individual downstream tasks. As the number of parameters grows, fine-tuning is prone to overfitting and catastrophic forgetting. In addition, full fine-tuning can become prohibitively expensive when the model is used for many tasks. To mitigate this issue, parameter-efficient transfer learning algorithms, such as adapters and prefix tuning, have been proposed as a way to introduce a few trainable parameters that can be plugged into large pre-trained language models such as BERT, and HuBERT. In this paper, we introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks. Additionally, we introduce a new adapter, ConvAdapter, based on 1D convolution. We show that ConvAdapter outperforms the standard adapters while showing comparable performance against prefix tuning and LoRA with only 0.94% of trainable parameters on some of the task in SURE. We further explore the effectiveness of parameter efficient transfer learning for speech synthesis task such as Text-to-Speech (TTS).

【10】 Revisiting Wright: Improving supervised classification of rat ultrasonic vocalisations using synthetic training data标题：重访赖特：使用合成训练数据改进大鼠超声发声的监督分类链接：http://arxiv.org/pdf/2303.03183v1

*与cs.SD语音【2】为同一篇

作者：K. Jack Scott,Lucinda J. Speers,David K. Bilkey机构：⁺ Department of Psychology, University of Otago, New Zealand., All correspondence to:, William James Building, Leith Walk, University of Otago, Dunedin , New Zealand., ORC ID:, Supported by funding from the University of Otago.备注：25 pages, 5 main figures, 4 supplementary figures, 2 tables摘要：啮齿动物通过超声波发声（USVs）进行交流。这些叫声之所以令人感兴趣，是因为它们提供了对声音交流的发展和功能的深入了解，并可能被证明是神经发育障碍模型中功能障碍的有用生物标志物。啮齿类USV可分为不同的部件，虽然人工分类耗时，但神经计算的进步允许快速准确地识别和分类。在这里，我们采用了一个卷积神经网络（CNN），VocalMat，创建用于分析小鼠USV，用于大鼠。我们编纂了一个修改的模式，改编自Wright等人（2010年）之前提出的模式，用于分类，并将我们改编的VocalMat与基准CNN DeepSqueak的性能进行了比较。此外，我们测试了将合成USV插入到分类网络的训练数据中的效果，以减少生成训练集所涉及的工作量。我们的结果表明，修改后的VocalMat在呼叫识别和分类方面都优于基准软件。此外，我们发现，当VocalMat随后用于分析新数据时，使用合成图像增强训练数据可显著提高VocalMat的准确性。修改后的Wright分类的准确度足够高，允许在实验室条件下将该软件应用于大鼠USV分类。我们的研究结果还表明，将合成USV呼叫插入到训练集中会导致准确性的提高，而几乎没有额外的时间成本。摘要：Rodents communicate through ultrasonic vocalizations (USVs). These calls are of interest because they provide insight into the development and function of vocal communication, and may prove to be useful as a biomarker for dysfunction in models of neurodevelopmental disorders. Rodent USVs can be categorised into different components and while manual classification is time consuming, advances in neural computing have allowed for fast and accurate identification and classification. Here, we adapt a convolutional neural network (CNN), VocalMat, created for analysing mice USVs, for use with rats. We codify a modified schema, adapted from that previously proposed by Wright et al. (2010), for classification, and compare the performance of our adaptation of VocalMat with a benchmark CNN, DeepSqueak. Additionally, we test the effect of inserting synthetic USVs into the training data of our classification network in order to reduce the workload involved in generating a training set. Our results show that the modified VocalMat outperformed the benchmark software on measures of both call identification, and classification. Additionally, we found that the augmentation of training data with synthetic images resulted in a marked improvement in the accuracy of VocalMat when it was subsequently used to analyse novel data. The resulting accuracy on the modified Wright categorizations was sufficiently high to allow for the application of this software in rat USV classification in laboratory conditions. Our findings also show that inserting synthetic USV calls into the training set leads to improvements in accuracy with little extra time-cost.

【11】 IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining标题：IPA-CLIP：将语音先验知识融入视觉和语言预训练链接：http://arxiv.org/pdf/2303.03144v1

*与cs.SD语音【3】为同一篇

作者：Chihaya Matsuhira,Marc A. Kastner,Takahiro Komamizu,Takatsugu Hirayama,Keisuke Doman,Yasutomo Kawanishi,Ichiro Ide机构：Nagoya University, Aichi, Japan, Kyoto University, Kyoto, Japan, University of Human Environments, Chukyo University, RIKEN备注：11 pages, 8 figures, 5 Tables摘要：近年来，大规模的视觉和语言（V &L）预训练已经成为许多多媒体系统的标准主干。虽然它甚至在看不见的情况下也表现出了非凡的性能，但它的表现方式往往与人类的直觉不符。特别地，它们通常不考虑输入的发音，而人类将利用该发音来理解语言，尤其是当涉及未知单词时。为此，本文在对比语言-图像预训练（CLIP）模型中加入语音先验知识，使其考虑语音输入之间的语音相似性。为了实现这一点，我们首先提出了一种音素嵌入方法，该方法利用国际音标（IPA）表提供的音素关系作为语音先验。接下来，通过提取冻结的CLIP文本编码器，我们训练基于IPA的嵌入训练语音编码器。IPA-CLIP模型由语音编码器和原始的CLIP编码器（图像和文本）组成。定量评价结果表明，使用所提出的音素嵌入方法，音素在嵌入空间上的分布能更准确地反映语音关系。此外，在一些多模态检索任务中，我们证实了所提出的发音编码器增强了文本编码器的性能，并且发音编码器以比文本编码器更语音化的方式处理无意义单词。最后，定性评价验证了语音编码器与人类感知语音相似度之间的相关性。摘要：Recently, large-scale Vision and Language (V &L) pretraining has become the standard backbone of many multimedia systems. While it has shown remarkable performance even in unseen situations, it often performs in ways not intuitive to humans. Particularly, they usually do not consider the pronunciation of the input, which humans would utilize to understand language, especially when it comes to unknown words. Thus, this paper inserts phonetic prior into Contrastive Language-Image Pretraining (CLIP), one of the V &L pretrained models, to make it consider the pronunciation similarity among its pronunciation inputs. To achieve this, we first propose a phoneme embedding that utilizes the phoneme relationships provided by the International Phonetic Alphabet (IPA) chart as a phonetic prior. Next, by distilling the frozen CLIP text encoder, we train a pronunciation encoder employing the IPA-based embedding. The proposed model named IPA-CLIP comprises this pronunciation encoder and the original CLIP encoders (image and text). Quantitative evaluation reveals that the phoneme distribution on the embedding space represents phonetic relationships more accurately when using the proposed phoneme embedding. Furthermore, in some multimodal retrieval tasks, we confirm that the proposed pronunciation encoder enhances the performance of the text encoder and that the pronunciation encoder handles nonsense words in a more phonetic manner than the text encoder. Finally, qualitative evaluation verifies the correlation between the pronunciation encoder and human perception regarding pronunc

【12】 Scaling strategies for on-device low-complexity source separation with Conv-Tasnet标题：基于Conv-Tasnet的设备上低复杂度信源分离扩展策略链接：http://arxiv.org/pdf/2303.03005v1

*与cs.SD语音【4】为同一篇

作者：Mohamed Nabih Ali,Francesco Paissan,Daniele Falavigna,Alessio Brutti机构：University of Trento, Trento, Italy, Fondazione Bruno Kessler, Trento, Italy摘要：最近，文献中提出了几种非常有效的单通道语音分离的神经方法。然而，由于这些模型的尺寸和复杂性，它们在低资源设备（例如助听器和耳机）上的使用仍然是一个挑战，并且还没有成熟的解决方案可用。尽管已经提出了基于修剪或压缩神经模型的方法，但是适合于特定应用领域的模型架构的设计通常需要启发式过程，而该启发式过程不容易移植到不同的低资源平台。给定公知的Conv-Tasnet语音分离架构的模块化性质，在本文中我们考虑直接控制模型的总体大小的三个参数，即：残差块的数目、分离块的重复数目和深度方向卷积中的通道数目，并且实验地评估它们如何影响语音分离性能。特别地，在Libri 2 Mix上执行的实验表明，膨胀的1D-Conv块的数量是最关键的参数，并且在残余块中使用额外膨胀允许减少性能下降。摘要：Recently, several very effective neural approaches for single-channel speech separation have been presented in the literature. However, due to the size and complexity of these models, their use on low-resource devices, e.g. for hearing aids, and earphones, is still a challenge and established solutions are not available yet. Although approaches based on either pruning or compressing neural models have been proposed, the design of a model architecture suitable for a certain application domain often requires heuristic procedures not easily portable to different low-resource platforms. Given the modular nature of the well-known Conv-Tasnet speech separation architecture, in this paper we consider three parameters that directly control the overall size of the model, namely: the number of residual blocks, the number of repetitions of the separation blocks and the number of channels in the depth-wise convolutions, and experimentally evaluate how they affect the speech separation performance. In particular, experiments carried out on the Libri2Mix show that the number of dilated 1D-Conv blocks is the most critical parameter and that the usage of extra-dilation in the residual blocks allows reducing the performance drop.

【13】 Time-frequency Network for Robust Speaker Recognition标题：用于稳健说话人识别的时频网络链接：http://arxiv.org/pdf/2303.02673v1

*与cs.SD语音【5】为同一篇

作者：Jiguo Li,Tianzi Zhang,Xiaobin Liu,Lirong Zheng机构：the School of information science and technology, Fudan University备注：5pages, 3 figures摘要：基于语音的生物特征识别系统的广泛应用通常需要高性能的说话人识别算法。然而，大多数用于说话人识别的现有工作或者在频域或者在时域中处理语音，这可能产生次优结果，因为时域和频域对于说话人识别都是重要的。本文试图从时域和频域两个方面对语音信号进行分析，通过提取和融合两个域的特征，提出了用于说话人识别的时频网络~（TFN）。基于深度神经网络的最新进展，提出了一种卷积神经网络，将原始语音波形和频谱编码为特定领域的特征，然后将这些特征融合并转换为分类特征空间，用于说话人识别。在TIMIT和LibriSpeech两个公开数据集上的实验结果表明，该框架能够有效地融合两个领域的信息，且识别效果优于现有的说话人识别方法.摘要：The wide deployment of speech-based biometric systems usually demands high-performance speaker recognition algorithms. However, most of the prior works for speaker recognition either process the speech in the frequency domain or time domain, which may produce suboptimal results because both time and frequency domains are important for speaker recognition. In this paper, we attempt to analyze the speech signal in both time and frequency domains and propose the time-frequency network~(TFN) for speaker recognition by extracting and fusing the features in the two domains. Based on the recent advance of deep neural networks, we propose a convolution neural network to encode the raw speech waveform and the frequency spectrum into domain-specific features, which are then fused and transformed into a classification feature space for speaker recognition. Experimental results on the publicly available datasets TIMIT and LibriSpeech show that our framework is effective to combine the information in the two domains and performs better than the state-of-the-art methods for speaker recognition.

【14】 Heterogeneous Graph Learning for Acoustic Event Classification标题：异构图学习用于声事件分类链接：http://arxiv.org/pdf/2303.02665v1

*与cs.SD语音【6】为同一篇

作者：Amir Shirian,Mona Ahmadian,Krishna Somandepalli,Tanaya Guha机构：University of Warwick, UK,University of Surrey, UK,Google Research, USA, University of Glasgow, UK备注：arXiv admin note: text overlap with arXiv:2207.07935摘要：异构图提供了一种紧凑、高效和可扩展的方式来对涉及多种不同模态的数据进行建模。这使得使用异构图来建模视听数据成为一种有吸引力的选择。然而，图结构在视听数据中并不自然地出现。用于视听数据的图是手动构造的，这既困难又次优。在这项工作中，我们通过（i）提出一种参数图构造策略来构造模态内边，以及（ii）学习跨模态边来解决这个问题。为此，我们发展了一个新的模型，异构图跨模态网络（HGCN），学习跨模态边。由于模型的参数化构造，该模型能够适应不同的时空尺度，而可学习的跨模态边有效地连接了跨模态的相关节点。在大型基准数据集（AudioSet）上的实验表明，该模型是最先进的（平均精度为0. 53），优于基于变换器的模型和其他基于图的模型。摘要：Heterogeneous graphs provide a compact, efficient, and scalable way to model data involving multiple disparate modalities. This makes modeling audiovisual data using heterogeneous graphs an attractive option. However, graph structure does not appear naturally in audiovisual data. Graphs for audiovisual data are constructed manually which is both difficult and sub-optimal. In this work, we address this problem by (i) proposing a parametric graph construction strategy for the intra-modal edges, and (ii) learning the crossmodal edges. To this end, we develop a new model, heterogeneous graph crossmodal network (HGCN) that learns the crossmodal edges. Our proposed model can adapt to various spatial and temporal scales owing to its parametric construction, while the learnable crossmodal edges effectively connect the relevant nodes across modalities. Experiments on a large benchmark dataset (AudioSet) show that our model is state-of-the-art (0.53 mean average precision), outperforming transformer-based models and other graph-based models.

【15】 Hybrid Y-Net Architecture for Singing Voice Separation标题：歌唱语音分离的混合Y-Net结构链接：http://arxiv.org/pdf/2303.02599v1

*与cs.SD语音【7】为同一篇

作者：Rashen Fernando,Pamudu Ranasinghe,Udula Ranasinghe,Janaka Wijayakulasooriya,Pantaleon Perera机构：dept. Electrical & Electronic Eng., University of Peradeniya, Peradeniya, Sri Lanka, dept. Electrical & Electronic Engineering, dept. Engineering Mathematics备注：Submitted for EUSIPCO23: 5 Pages, 7 figures摘要：提出了一种基于深度学习的神经网络结构, -Y网络，用于实现音乐源分离。该结构通过从频谱图和波形域提取特征来实现端到端的混合源分离。受U-Net架构的启发，Y-Net预测一个声谱图掩模来从混合信号中分离声源。实验结果证明了该结构在参数较少的情况下对音乐源分离的有效性。总之，我们的工作为提高音乐源分离的准确性和效率提供了一种有前景的方法。摘要：This research paper presents a novel deep learning-based neural network architecture, named Y-Net, for achieving music source separation. The proposed architecture performs end-to-end hybrid source separation by extracting features from both spectrogram and waveform domains. Inspired by the U-Net architecture, Y-Net predicts a spectrogram mask to separate vocal sources from a mixture signal. Our results demonstrate the effectiveness of the proposed architecture for music source separation with fewer parameters. Overall, our work presents a promising approach for improving the accuracy and efficiency of music source separation.

【16】 A General Framework for Learning Procedural Audio Models of Environmental Sounds标题：学习环境声音程序音频模型的通用框架链接：http://arxiv.org/pdf/2303.02396v1

*与cs.SD语音【8】为同一篇

作者：Danzel Serrano,Mark Cartwright机构：New Jersey Institute of Technology, Newark, NJ, USA备注：Accepted at ICASSP 2023摘要：本文介绍了程序（音频）变分自动编码器（ProVE）框架，它是学习环境声音的程序音频PA模型的通用方法，可提高合成的真实感，同时通过可调参数保持对生成声音的控制。该框架包括两个阶段：（i）音频类表示，其中通过训练音频自动编码器来定义潜在表示空间，以及（ii）控制映射，其中学习从音频导出的静态 时间控制变量和均匀噪声的随机样本的联合函数以代替音频编码器。我们通过各种表面上的脚步声效果示例来演示ProVE的使用。我们的结果表明，ProVE模型在声音保真度方面优于经典PA模型和基于逆向的方法，如通过Fr “echet音频距离（FAD）、最大平均差异（MMD）和主观评价所测量的，使其成为声音设计工作流的可行工具。摘要：This paper introduces the Procedural (audio) Variational autoEncoder (ProVE) framework as a general approach to learning Procedural Audio PA models of environmental sounds with an improvement to the realism of the synthesis while maintaining provision of control over the generated sound through adjustable parameters. The framework comprises two stages: (i) Audio Class Representation, in which a latent representation space is defined by training an audio autoencoder, and (ii) Control Mapping, in which a joint function of static temporal control variables derived from the audio and a random sample of uniform noise is learned to replace the audio encoder. We demonstrate the use of ProVE through the example of footstep sound effects on various surfaces. Our results show that ProVE models outperform both classical PA models and an adversarial-based approach in terms of sound fidelity, as measured by Fr 'echet Audio Distance (FAD), Maximum Mean Discrepancy (MMD), and subjective evaluations, making them feasible tools for sound design workflows.

【17】 The DKU Post-Challenge Audio-Visual Wake Word Spotting System for the 2021 MISP Challenge: Deep Analysis标题：2021年MISP挑战赛DKU赛后视听寻词系统：深度分析链接：http://arxiv.org/pdf/2303.02348v1

*与cs.SD语音【9】为同一篇

作者：Haoxu Wang,Ming Cheng,Qiang Fu,Ming Li机构：School of Computer Science, Wuhan University, Wuhan, China, Data Science Research Center, Duke Kunshan University, Kunshan, China, Alibaba Group, China备注：Accepted by ICASSP 2023摘要：本文进一步探讨了我们之前在MISP挑战赛2021赛道1中排名第二的唤醒词识别系统。首先，我们研究了一种基于3D和2D卷积的鲁棒单峰方法，并采用简单注意模块（SimAM）来提高系统的性能。第二，我们探索了数据扩充方法的不同组合以获得更好的性能。最后，研究了分数级融合、级联融合和神经网络融合等融合策略。我们所提出的多模态系统利用多模态特征并使用互补的视觉信息来减轻在复杂声学场景中纯音频系统的性能下降。在竞争数据库的评估集中，我们的系统获得了2.15%的误拒率和3.44%的误报警率，与以前的系统相比，相对提高了21%，达到了新的最佳性能。摘要：This paper further explores our previous wake word spotting system ranked 2-nd in Track 1 of the MISP Challenge 2021. First, we investigate a robust unimodal approach based on 3D and 2D convolution and adopt the simple attention module (SimAM) for our system to improve performance. Second, we explore different combinations of data augmentation methods for better performance. Finally, we study the fusion strategies, including score-level, cascaded and neural fusion. Our proposed multimodal system leverages multimodal features and uses the complementary visual information to mitigate the performance degradation of audio-only systems in complex acoustic scenarios. Our system obtains a false reject rate of 2.15% and a false alarm rate of 3.44% in the evaluation set of the competition database, which achieves the new state-of-the-art performance by 21% relative improvement compared to previous systems.

机器翻译，仅供参考