今天跟大家分享一篇语音相关的论文合集：cs.SD语音3篇，eess.AS音频处理5篇。
cs.SD语音
【1】 The Ability of Self-Supervised Speech Models for Audio Representations
标题：自监督语音模型的音频表示能力
链接：https://arxiv.org/abs/2209.12900
作者：Tung-Yu Wu,Chen-An Li,Tzu-Han Lin,Tsu-Yuan Hsu,Hung-Yi Lee机构：Department of Electrical Engineering, National Taiwan University, Department of Computer Science and Information Engineering, Editors: Joseph Turian, Bj¨orn W. Schuller, Dorien Herremans, Katrin Kirchhoff, Paola Garcia, Perera, Philippe Esling备注：18 pages, 2 figures, 6 tables. Submitted to Proceedings of Machine Learning Research (PMLR): NeurIPS 2021 Competition Track. Under review摘要：自监督学习（Self-supervised learning，SSL）语音模型在语音表征学习中取得了前所未有的成功，但其表征能力仍存在一些问题。本文将讨论其中的两个问题：（1）SSL语音模型可以处理非语音音频吗？(2)不同的SSL语音模型是否能够深入了解音频特性的各个方面？为了回答这两个问题，我们在大量的语音和非语音音频数据集上进行了大量的实验，以评估目前最先进的SSL语音模型（wav 2 vec 2. 0和HuBERT）的表示能力。这些实验是在NeurIPS 2021 HEAR挑战赛期间进行的，作为竞赛官员提供的标准评估管道。实验结果表明：（1）SSL语音模型能够提取出大量非语音音频的有意义特征，但在某些类型的数据集上可能会失败;（2）不同的SSL语音模型具有对音频特征的不同方面的洞察。这两个结论为表征模型的集成提供了基础。我们进一步提出了一个集成框架来融合语音表示模型的嵌入。我们的框架性能优于最先进的SSL语音/音频模型，并且与HEAR Challenge中的其他团队相比，在丰富的数据集上具有普遍优越的性能。您可以在www.example.com -NTU-GURA上取得我们的代码https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge。摘要：Self-supervised learning (SSL) speech models have achieved unprecedented success in speech representation learning, but some questions regarding their representation ability remain unanswered. This paper addresses two of them: (1) Can SSL speech models deal with non-speech audio?; (2) Would different SSL speech models have insights into diverse aspects of audio features? To answer the two questions, we conduct extensive experiments on abundant speech and non-speech audio datasets to evaluate the representation ability of currently state-of-the-art SSL speech models, which are wav2vec 2.0 and HuBERT in this paper. These experiments are carried out during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline provided by competition officials. Results show that (1) SSL speech models could extract meaningful features of a wide range of non-speech audio, while they may also fail on certain types of datasets; (2) different SSL speech models have insights into different aspects of audio features. The two conclusions provide a foundation for the ensemble of representation models. We further propose an ensemble framework to fuse speech representation models' embeddings. Our framework outperforms state-of-the-art SSL speech/audio models and has generally superior performance on abundant datasets compared with other teams in HEAR Challenge. Our code is available at https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge -- NTU-GURA.
【2】 Beyond Heart Murmur Detection: Automatic Murmur Grading from Phonocardiogram
标题：心外杂音检测：心音图杂音自动分级
链接：https://arxiv.org/abs/2209.13385
作者：Andoni Elola,Elisabete Aramendi,Jorge Oliveira,Francesco Renna,Miguel T. Coimbra,Matthew A. Reyna,Reza Sameni,Gari D. Clifford,Ali Bahrami Rad机构：Jorge Oliveira is with Universidade Portucalense Infante D摘要：目的：杂音是由专家通过心脏听诊所识别的异常心音。杂音等级是杂音强度的定量测量，与患者的临床状况密切相关。这项工作的目的是估计每个患者的杂音等级（即，缺少、柔和、响亮）的多个听诊位置心音图（PCG）。方法：将每个心音记录的Mel谱图表示提供给15个具有通道式注意机制的卷积残差神经网络集成，以分类每个心音记录。根据建议的决策规则并考虑可用记录的所有估计标签，得出每例患者的最终杂音等级。使用分层的10倍交叉验证，在由1007例患者的3456个PCG记录组成的数据集上对所提出的方法进行交叉验证。此外，在由442名患者的1538个PCG记录组成的隐藏测试集上测试了该方法。结果：根据敏感性和F1评分的未加权平均值，患者水平杂音分级的总体交叉验证性能分别为86.3%和81.6%。无杂音、柔和杂音和响亮杂音的敏感性（和F1评分）分别为90.7%（93.6%）、75.8%（66.8%）和92.3%（84.2%）。在测试集上，该算法实现了80.4%的灵敏度的未加权平均值和75.8%的F1得分。结论：本研究为低资源环境下专家筛选成本相对较高的算法预筛选提供了一种潜在的方法。意义：所提出的方法代表了超出杂音检测的重要一步，提供了强度的表征，其可以提供临床结局的增强分类。摘要：Objective: Murmurs are abnormal heart sounds, identified by experts through cardiac auscultation. The murmur grade, a quantitative measure of the murmur intensity, is strongly correlated with the patient's clinical condition. This work aims to estimate each patient's murmur grade (i.e., absent, soft, loud) from multiple auscultation location phonocardiograms (PCGs) of a large population of pediatric patients from a low-resource rural area. Methods: The Mel spectrogram representation of each PCG recording is given to an ensemble of 15 convolutional residual neural networks with channel-wise attention mechanisms to classify each PCG recording. The final murmur grade for each patient is derived based on the proposed decision rule and considering all estimated labels for available recordings. The proposed method is cross-validated on a dataset consisting of 3456 PCG recordings from 1007 patients using a stratified ten-fold cross-validation. Additionally, the method was tested on a hidden test set comprised of 1538 PCG recordings from 442 patients. Results: The overall cross-validation performances for patient-level murmur gradings are 86.3% and 81.6% in terms of the unweighted average of sensitivities and F1-scores, respectively. The sensitivities (and F1-scores) for absent, soft, and loud murmurs are 90.7% (93.6%), 75.8% (66.8%), and 92.3% (84.2%), respectively. On the test set, the algorithm achieves an unweighted average of sensitivities of 80.4% and an F1-score of 75.8%. Conclusions: This study provides a potential approach for algorithmic pre-screening in low-resource settings with relatively high expert screening costs. Significance: The proposed method represents a significant step beyond detection of murmurs, providing characterization of intensity which may provide a enhanced classification of clinical outcomes.
【3】 Automated Sex Classification of Children's Voices and Changes in Differentiating Factors with Age
标题：儿童嗓音的自动性别分类及区分因素随年龄的变化
链接：https://arxiv.org/abs/2209.13112
作者：Fuling Chen,Roberto Togneri,Murray Maybery,Diana Weiting Tan机构：)Dept. of Electrical, Electronic and Computer Engineering, University of Western Australia, Perth, Western Australia, Australia, )School of Psychological Science, University of Western Australia, Perth, Western Australia摘要：儿童声音的性别分类允许研究第二性征的发展，这一直是语音分析领域的一个关键兴趣。本研究调查了来自脚本和自发语音的广泛声学特征，并应用基于层次聚类的机器学习模型来区分5至15岁儿童的性别。我们提出了一个最佳特征集，我们的模型在所有年龄段实现了0.84的平均F1得分（精确度和召回率的调和平均值）。我们的结果表明，当为每个年龄组而不是为4岁年龄段的儿童开发一个模型时，性别分类通常更准确，年龄较大的年龄组的分类准确性更好。我们发现自发言语比照本宣科言语更能提供性别分类的线索，尤其是对于7岁以下的儿童。对于较年轻的年龄组，广泛的声学因素对性别分类的贡献是均匀的，而对于较年长的年龄组，发现与F0相关的声学因素通常是最关键的预测因素。对于较年长年龄组的其他重要声学因素包括声道长度估计器、谱通量、响度和清音特征。摘要：Sex classification of children's voices allows for an investigation of the development of secondary sex characteristics which has been a key interest in the field of speech analysis. This research investigated a broad range of acoustic features from scripted and spontaneous speech and applied a hierarchical clustering-based machine learning model to distinguish the sex of children aged between 5 and 15 years. We proposed an optimal feature set and our modelling achieved an average F1 score (the harmonic mean of the precision and recall) of 0.84 across all ages. Our results suggest that the sex classification is generally more accurate when a model is developed for each year group rather than for children in 4-year age bands, with classification accuracy being better for older age groups. We found that spontaneous speech could provide more helpful cues in sex classification than scripted speech, especially for children younger than 7 years. For younger age groups, a broad range of acoustic factors contributed evenly to sex classification, while for older age groups, F0-related acoustic factors were found to be the most critical predictors generally. Other important acoustic factors for older age groups include vocal tract length estimators, spectral flux, loudness and unvoiced features.
【1】 Hyperbolic Timbre Embedding for Musical Instrument Sound Synthesis Based on Variational Autoencoders
标题：基于变分自动编码器的乐器声音合成中的双曲音色嵌入
链接：https://arxiv.org/abs/2209.13211
作者：Futa Nakashima,Tomohiko Nakamura,Norihiro Takamune,Satoru Fukayama,Hiroshi Saruwatari机构：∗The University of Tokyo, Tokyo, Japan, †National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan备注：8 pages, 4 figures, to be published in Asia Pacific Signal and Information Processing Association Annual Summit and Conference 2022摘要：本文提出了一种基于变分自动编码器的乐器声音合成方法，该方法具有层次诱导的音色潜在空间。基于VAE的MISS方法将输入信号嵌入到低维潜在表示中，该低维潜在表示捕获输入的特征。适当地操纵这种表示会导致声音变形和音色替换。虽然大多数基于VAE的MISS方法寻求音高和音色的分离表示，但如何捕捉音色中的潜在结构仍然是一个悬而未决的问题。为了解决这一问题，我们关注的事实是，乐器可以在其物理机制的基础上进行分层分类。基于这种层次结构，我们通过引入双曲空间来描述音色，提出了一种基于VAE的MISS方法。由于双曲空间具有指数增长特性，因此它比欧氏空间更能有效地表示树状数据。实验结果表明，与基于欧氏空间的方法相比，该方法能够有效地表征音色的潜在特征。摘要：In this paper, we propose a musical instrument sound synthesis (MISS) method based on a variational autoencoder (VAE) that has a hierarchy-inducing latent space for timbre. VAE-based MISS methods embed an input signal into a low-dimensional latent representation that captures the characteristics of the input. Adequately manipulating this representation leads to sound morphing and timbre replacement. Although most VAE-based MISS methods seek a disentangled representation of pitch and timbre, how to capture an underlying structure in timbre remains an open problem. To address this problem, we focus on the fact that musical instruments can be hierarchically classified on the basis of their physical mechanisms. Motivated by this hierarchy, we propose a VAE-based MISS method by introducing a hyperbolic space for timbre. The hyperbolic space can represent treelike data more efficiently than the Euclidean space owing to its exponential growth property. Results of experiments show that the proposed method provides an efficient latent representation of timbre compared with the method using the Euclidean space.
【2】 Predicting Affective Vocal Bursts with Finetuned wav2vec 2.0
标题：用精细调整的Wave2vec 2.0预测情感发声
链接：https://arxiv.org/abs/2209.13146
作者：Bagus Tris Atmaja,Akira Sasou摘要：从人类声音预测情感状态的研究在很大程度上依赖于语音。事实上，这项研究探讨了人类情感状态的识别，从他们的声音爆发，一个简短的非言语发声。借鉴wav2vec 2.0最近成功的思想，我们评估了来自不同数据集的微调wav2vec 2.0模型，以从说话者的声音突发预测说话者的情感状态。然后，在语音突发数据上训练微调的wav2vec 2.0模型。实验结果表明，经过微调的wav2vec 2.0模型，特别是在情感语音数据集上，优于手工制作的声学特征的基线模型。然而，在非情感语音数据集和情感语音数据集上调整的模型没有太大的差距。摘要：The studies of predicting affective states from human voices have relied heavily on speech. This study, indeed, explores the recognition of humans' affective state from their vocal burst, a short non-verbal vocalization. Borrowing the idea from the recent success of wav2vec 2.0, we evaluated finetuned wav2vec 2.0 models from different datasets to predict the affective state of the speaker from their vocal burst. The finetuned wav2vec 2.0 models are then trained on the vocal burst data. The results show that the finetuned wav2vec 2.0 models, particularly on an affective speech dataset, outperform the baseline model, which is handcrafted acoustic features. However, there is no large gap between the model finetuned on non-affective speech dataset and affective speech dataset.
【3】 Automated Sex Classification of Children's Voices and Changes in Differentiating Factors with Age
标题：儿童嗓音的自动性别分类及区分因素随年龄的变化
链接：https://arxiv.org/abs/2209.13112
* 与cs.SD语音【3】为同一篇
作者：Fuling Chen,Roberto Togneri,Murray Maybery,Diana Weiting Tan机构：)Dept. of Electrical, Electronic and Computer Engineering, University of Western Australia, Perth, Western Australia, Australia, )School of Psychological Science, University of Western Australia, Perth, Western Australia摘要：儿童声音的性别分类允许研究第二性征的发展，这一直是语音分析领域的一个关键兴趣。本研究调查了来自脚本和自发语音的广泛声学特征，并应用基于层次聚类的机器学习模型来区分5至15岁儿童的性别。我们提出了一个最佳特征集，我们的模型在所有年龄段实现了0.84的平均F1得分（精确度和召回率的调和平均值）。我们的结果表明，当为每个年龄组而不是为4岁年龄段的儿童开发一个模型时，性别分类通常更准确，年龄较大的年龄组的分类准确性更好。我们发现自发言语比照本宣科言语更能提供性别分类的线索，尤其是对于7岁以下的儿童。对于较年轻的年龄组，广泛的声学因素对性别分类的贡献是均匀的，而对于较年长的年龄组，发现与F0相关的声学因素通常是最关键的预测因素。对于较年长年龄组的其他重要声学因素包括声道长度估计器、谱通量、响度和清音特征。摘要：Sex classification of children's voices allows for an investigation of the development of secondary sex characteristics which has been a key interest in the field of speech analysis. This research investigated a broad range of acoustic features from scripted and spontaneous speech and applied a hierarchical clustering-based machine learning model to distinguish the sex of children aged between 5 and 15 years. We proposed an optimal feature set and our modelling achieved an average F1 score (the harmonic mean of the precision and recall) of 0.84 across all ages. Our results suggest that the sex classification is generally more accurate when a model is developed for each year group rather than for children in 4-year age bands, with classification accuracy being better for older age groups. We found that spontaneous speech could provide more helpful cues in sex classification than scripted speech, especially for children younger than 7 years. For younger age groups, a broad range of acoustic factors contributed evenly to sex classification, while for older age groups, F0-related acoustic factors were found to be the most critical predictors generally. Other important acoustic factors for older age groups include vocal tract length estimators, spectral flux, loudness and unvoiced features.
【4】 Beyond Heart Murmur Detection: Automatic Murmur Grading from Phonocardiogram
标题：心外杂音检测：心音图杂音自动分级
链接：https://arxiv.org/abs/2209.13385
* 与cs.SD语音【2】为同一篇
作者：Andoni Elola,Elisabete Aramendi,Jorge Oliveira,Francesco Renna,Miguel T. Coimbra,Matthew A. Reyna,Reza Sameni,Gari D. Clifford,Ali Bahrami Rad机构：Jorge Oliveira is with Universidade Portucalense Infante D摘要：目的：杂音是由专家通过心脏听诊所识别的异常心音。杂音等级是杂音强度的定量测量，与患者的临床状况密切相关。这项工作的目的是估计每个患者的杂音等级（即，缺少、柔和、响亮）的多个听诊位置心音图（PCG）。方法：将每个心音记录的Mel谱图表示提供给15个具有通道式注意机制的卷积残差神经网络集成，以分类每个心音记录。根据建议的决策规则并考虑可用记录的所有估计标签，得出每例患者的最终杂音等级。使用分层的10倍交叉验证，在由1007例患者的3456个PCG记录组成的数据集上对所提出的方法进行交叉验证。此外，在由442名患者的1538个PCG记录组成的隐藏测试集上测试了该方法。结果：根据敏感性和F1评分的未加权平均值，患者水平杂音分级的总体交叉验证性能分别为86.3%和81.6%。无杂音、柔和杂音和响亮杂音的敏感性（和F1评分）分别为90.7%（93.6%）、75.8%（66.8%）和92.3%（84.2%）。在测试集上，该算法实现了80.4%的灵敏度的未加权平均值和75.8%的F1得分。结论：本研究为低资源环境下专家筛选成本相对较高的算法预筛选提供了一种潜在的方法。意义：所提出的方法代表了超出杂音检测的重要一步，提供了强度的表征，其可以提供临床结局的增强分类。摘要：Objective: Murmurs are abnormal heart sounds, identified by experts through cardiac auscultation. The murmur grade, a quantitative measure of the murmur intensity, is strongly correlated with the patient's clinical condition. This work aims to estimate each patient's murmur grade (i.e., absent, soft, loud) from multiple auscultation location phonocardiograms (PCGs) of a large population of pediatric patients from a low-resource rural area. Methods: The Mel spectrogram representation of each PCG recording is given to an ensemble of 15 convolutional residual neural networks with channel-wise attention mechanisms to classify each PCG recording. The final murmur grade for each patient is derived based on the proposed decision rule and considering all estimated labels for available recordings. The proposed method is cross-validated on a dataset consisting of 3456 PCG recordings from 1007 patients using a stratified ten-fold cross-validation. Additionally, the method was tested on a hidden test set comprised of 1538 PCG recordings from 442 patients. Results: The overall cross-validation performances for patient-level murmur gradings are 86.3% and 81.6% in terms of the unweighted average of sensitivities and F1-scores, respectively. The sensitivities (and F1-scores) for absent, soft, and loud murmurs are 90.7% (93.6%), 75.8% (66.8%), and 92.3% (84.2%), respectively. On the test set, the algorithm achieves an unweighted average of sensitivities of 80.4% and an F1-score of 75.8%. Conclusions: This study provides a potential approach for algorithmic pre-screening in low-resource settings with relatively high expert screening costs. Significance: The proposed method represents a significant step beyond detection of murmurs, providing characterization of intensity which may provide a enhanced classification of clinical outcomes.
【5】 The Ability of Self-Supervised Speech Models for Audio Representations
标题：自监督语音模型的音频表示能力
链接：https://arxiv.org/abs/2209.12900
* 与cs.SD语音【1】为同一篇
作者：Tung-Yu Wu,Chen-An Li,Tzu-Han Lin,Tsu-Yuan Hsu,Hung-Yi Lee机构：Department of Electrical Engineering, National Taiwan University, Department of Computer Science and Information Engineering, Editors: Joseph Turian, Bj¨orn W. Schuller, Dorien Herremans, Katrin Kirchhoff, Paola Garcia, Perera, Philippe Esling备注：18 pages, 2 figures, 6 tables. Submitted to Proceedings of Machine Learning Research (PMLR): NeurIPS 2021 Competition Track. Under review摘要：自监督学习（Self-supervised learning，SSL）语音模型在语音表征学习中取得了前所未有的成功，但其表征能力仍存在一些问题。本文将讨论其中的两个问题：（1）SSL语音模型可以处理非语音音频吗？(2)不同的SSL语音模型是否能够深入了解音频特性的各个方面？为了回答这两个问题，我们在大量的语音和非语音音频数据集上进行了大量的实验，以评估目前最先进的SSL语音模型（wav 2 vec 2. 0和HuBERT）的表示能力。这些实验是在NeurIPS 2021 HEAR挑战赛期间进行的，作为竞赛官员提供的标准评估管道。实验结果表明：（1）SSL语音模型能够提取出大量非语音音频的有意义特征，但在某些类型的数据集上可能会失败;（2）不同的SSL语音模型具有对音频特征的不同方面的洞察。这两个结论为表征模型的集成提供了基础。我们进一步提出了一个集成框架来融合语音表示模型的嵌入。我们的框架性能优于最先进的SSL语音/音频模型，并且与HEAR Challenge中的其他团队相比，在丰富的数据集上具有普遍优越的性能。您可以在www.example.com -NTU-GURA上取得我们的代码https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge。摘要：Self-supervised learning (SSL) speech models have achieved unprecedented success in speech representation learning, but some questions regarding their representation ability remain unanswered. This paper addresses two of them: (1) Can SSL speech models deal with non-speech audio?; (2) Would different SSL speech models have insights into diverse aspects of audio features? To answer the two questions, we conduct extensive experiments on abundant speech and non-speech audio datasets to evaluate the representation ability of currently state-of-the-art SSL speech models, which are wav2vec 2.0 and HuBERT in this paper. These experiments are carried out during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline provided by competition officials. Results show that (1) SSL speech models could extract meaningful features of a wide range of non-speech audio, while they may also fail on certain types of datasets; (2) different SSL speech models have insights into different aspects of audio features. The two conclusions provide a foundation for the ensemble of representation models. We further propose an ensemble framework to fuse speech representation models' embeddings. Our framework outperforms state-of-the-art SSL speech/audio models and has generally superior performance on abundant datasets compared with other teams in HEAR Challenge. Our code is available at https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge -- NTU-GURA.
机器翻译，仅供参考
每日语音速递