今日论文合集：cs.SD语音7篇，eess.AS音频处理7篇。
cs.SD语音

【1】Self-supervised Adaptive Pre-training of Multilingual Speech Models for  Language and Dialect Identification

标题：用于语言和方言识别的多语言语音模型的自监督自适应预训练链接：https://arxiv.org/abs/2312.07338

作者：Mohammed Maqsood Shaik,Dietrich Klakow,Badr M. Abdullah备注：Submitted to ICASSP 2024摘要：经过预训练的基于transformer的语音模型在对自动语音识别和口语识别（SLID）等各种下游任务进行微调时表现出惊人的性能。然而，域不匹配的问题仍然是该领域的一个挑战，其中预训练数据的域可能与用于微调的下游标记数据的域不同。在SLID等多语言任务中，预先训练的语音模型可能不支持下游任务中的所有语言。为了应对这一挑战，我们提出了自监督自适应预训练（SAPT），以使预训练模型适应下游任务的目标领域和语言。我们将SAPT应用于XLSR-128模型，并研究这种方法对SLID任务的有效性。首先，我们证明了SAPT提高了XLSR性能的FLEURS基准大幅收益高达40.1%的代表性不足的语言。其次，我们在Few-Shot学习环境中对四个不同的数据集应用SAPT，表明我们的方法在微调过程中提高了XLSR的样本效率。我们的实验提供了强有力的经验证据，通过自我监督的持续适应提高了多语言语音模型的下游性能。摘要：Pre-trained Transformer-based speech models have shown striking performance when fine-tuned on various downstream tasks such as automatic speech recognition and spoken language identification (SLID). However, the problem of domain mismatch remains a challenge in this area, where the domain of the pre-training data might differ from that of the downstream labeled data used for fine-tuning. In multilingual tasks such as SLID, the pre-trained speech model may not support all the languages in the downstream task. To address this challenge, we propose self-supervised adaptive pre-training (SAPT) to adapt the pre-trained model to the target domain and languages of the downstream task. We apply SAPT to the XLSR-128 model and investigate the effectiveness of this approach for the SLID task. First, we demonstrate that SAPT improves XLSR performance on the FLEURS benchmark with substantial gains up to 40.1% for under-represented languages. Second, we apply SAPT on four different datasets in a few-shot learning setting, showing that our approach improves the sample efficiency of XLSR during fine-tuning. Our experiments provide strong empirical evidence that continual adaptation via self-supervision improves downstream performance for multilingual speech models.

【2】 More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module  for Multimodal Fusion Based on Signal Theory标题：Beyond Vanilla Fusion：一种基于信号理论的简单、无解耦的多模式融合注意模块链接：https://arxiv.org/abs/2312.07212
作者：Peiwen Sun,Yifan Zhang,Zishan Liu,Donghao Chen,Honggang Zhang摘要：香草融合方法仍然占主流视听任务的很大比例。然而，从理论角度来看，香草融合的有效性仍然值得讨论。因此，本文从仿生学的角度重新考虑了多模态情况下融合的信号，并提出了一个简单的，即插即用，注意力模块香草融合的基础上，信号理论和不确定性理论。此外，以前的工作多模态动态梯度调制仍然依赖于解耦的方式。因此，一个无抖动的梯度调制方案已被设计与上述的注意力模块，它具有各种优势，比解耦的。实验结果表明，只需几行代码，就可以实现高达2.0%的性能提高几个多模态分类方法。最后，其他融合任务的定量评估揭示了额外的应用场景的潜力。摘要：The vanilla fusion methods still dominate a large percentage of mainstream audio-visual tasks. However, the effectiveness of vanilla fusion from a theoretical perspective is still worth discussing. Thus, this paper reconsiders the signal fused in the multimodal case from a bionics perspective and proposes a simple, plug-and-play, attention module for vanilla fusion based on fundamental signal theory and uncertainty theory. In addition, previous work on multimodal dynamic gradient modulation still relies on decoupling the modalities. So, a decoupling-free gradient modulation scheme has been designed in conjunction with the aforementioned attention module, which has various advantages over the decoupled one. Experiment results show that just a few lines of code can achieve up to 2.0% performance improvements to several multimodal classification methods. Finally, quantitative evaluation of other fusion tasks reveals the potential for additional application scenarios.
【3】 Robust End-to-End Diarization with Domain Adaptive Training and  Multi-Task Learning标题：基于领域自适应训练和多任务学习的端到端稳健二值化链接：https://arxiv.org/abs/2312.07136
作者：Ivan Fung,Lahiru Samarakoon,Samuel J. Broughton备注：7 pages, 2 figures, ASRU 2023摘要：由于公开可用的日志数据的稀缺性，可以通过使用来自不同领域的数据训练单个模型来提高模型性能。在这项工作中，我们建议将域信息训练一个单一的端到端的多个域的日记模型。首先，我们采用域自适应训练与参数有效的适配器在飞行模型重新配置。其次，我们引入了一个辅助的领域分类任务，使日志化模型更加领域感知。对于可见域，与基线相比，我们提出的方法的组合将绝对DER从17.66%降低到16.59%。在推理期间，来自地面实况域的适配器不可用于看不见的域。我们证明了我们的模型表现出更强的泛化能力，看不见的域时，适配器被删除。对于两个看不见的域，这将DER性能分别从基线的39.91%提高到23.09%和25.32%提高到18.76%。摘要：Due to the scarcity of publicly available diarization data, the model performance can be improved by training a single model with data from different domains. In this work, we propose to incorporate domain information to train a single end-to-end diarization model for multiple domains. First, we employ domain adaptive training with parameter-efficient adapters for on-the-fly model reconfiguration. Second, we introduce an auxiliary domain classification task to make the diarization model more domain-aware. For seen domains, the combination of our proposed methods reduces the absolute DER from 17.66% to 16.59% when compared with the baseline. During inference, adapters from ground-truth domains are not available for unseen domains. We demonstrate our model exhibits a stronger generalizability to unseen domains when adapters are removed. For two unseen domains, this improves the DER performance from 39.91% to 23.09% and 25.32% to 18.76% over the baseline, respectively.
【4】 LSTM-CNN Network for Audio Signature Analysis in Noisy Environments标题：LSTM-CNN网络用于噪声环境下的音频特征分析链接：https://arxiv.org/abs/2312.07059
作者：Praveen Damacharla,Hamid Rajabalipanah,Mohammad Hosein Fakheri备注：10th Annual Conf. on Computational Science & Computational Intelligence (CSCI'23)摘要：有多个应用程序可以在工作，展览，商场，销售和工业用途中自动计算人数并指定其性别。虽然目前的语音检测方法应该运行良好，在大多数情况下，除了性别，当前说话人的数量是未知的，分类方法是不合适的，由于许多可能的类。在这项研究中，我们专注于长短期记忆卷积神经网络（LSTM-CNN）来提取声音数据的时间和/或频率相关特征，以估计在嘈杂环境中每帧同时活动的说话者的数量/性别。考虑到扬声器的最大数量为10，我们已经利用了19000音频样本的男性，女性和背景噪声的不同组合，在公共城市，工业情况，商场，展览，工作场所和自然学习的目的。该概念证明显示出在检测计数和性别方面具有约0.019/0.017的训练/验证MSE值的有希望的性能。摘要：There are multiple applications to automatically count people and specify their gender at work, exhibitions, malls, sales, and industrial usage. Although current speech detection methods are supposed to operate well, in most situations, in addition to genders, the number of current speakers is unknown and the classification methods are not suitable due to many possible classes. In this study, we focus on a long-short-term memory convolutional neural network (LSTM-CNN) to extract time and / or frequency-dependent features of the sound data to estimate the number / gender of simultaneous active speakers at each frame in noisy environments. Considering the maximum number of speakers as 10, we have utilized 19000 audio samples with diverse combinations of males, females, and background noise in public cities, industrial situations, malls, exhibitions, workplaces, and nature for learning purposes. This proof of concept shows promising performance with training/validation MSE values of about 0.019/0.017 in detecting count and gender.
【5】 Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus标题：在闽南语语料库上评估自我监督语音模型链接：https://arxiv.org/abs/2312.06668
作者：Yi-Hui Chou,Kalvin Chang,Meng-Ju Wu,Winston Ou,Alice Wen-Hsin Bi,Carol Yang,Bryan Y. Chen,Rong-Wei Pai,Po-Yen Yeh,Jo-Peng Chiang,Iu-Tshian Phoann,Winnie Chang,Chenxuan Cui,Noel Chen,Jiatong Shi备注：Accepted to ASRU 2023摘要：台湾闽南语的使用和地位正在下降，因为台湾的语言转向普通话。这就是为什么它在今天的NLP和语音研究中是一种低资源语言的部分原因。为了确保语音处理的最新技术不会让台湾闽南语落后，我们将一个1.5小时的台湾闽南语数据集贡献给ML-SUPERB的隐藏集。在我们的数据集上评估ML-SUPERB的自监督学习（SSL）语音表示套件，我们发现模型大小并不总是决定性能。事实上，某些较小的模型优于较大的模型。此外，预训练数据和目标语言之间的语言对齐起着至关重要的作用。摘要：Taiwanese Hokkien is declining in use and status due to a language shift towards Mandarin in Taiwan. This is partly why it is a low resource language in NLP and speech research today. To ensure that the state of the art in speech processing does not leave Taiwanese Hokkien behind, we contribute a 1.5-hour dataset of Taiwanese Hokkien to ML-SUPERB's hidden set. Evaluating ML-SUPERB's suite of self-supervised learning (SSL) speech representations on our dataset, we find that model size does not consistently determine performance. In fact, certain smaller models outperform larger ones. Furthermore, linguistic alignment between pretraining data and the target language plays a crucial role.
【6】 NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint  Auditory Attention Detection标题：NeuroHeed+：利用联合听觉注意检测改进神经控制说话人提取链接：https://arxiv.org/abs/2312.07513
作者：Zexu Pan,Gordon Wichern,Francois G. Germain,Sameer Khurana,Jonathan Le Roux摘要：神经导向说话人提取的目的是从多说话人的语音信号中提取听者的大脑注意语音信号，其中注意力来自于大脑皮层的活动。这种活动通常使用脑电图（EEG）设备记录。虽然很有前途，但目前的方法通常具有较高的扬声器混淆错误，其中干扰扬声器被提取而不是被关注的扬声器，从而降低了收听体验。在这项工作中，我们的目标是通过一个共同微调的辅助听觉注意检测模型，以减少在神经操纵的说话人提取模型的说话人混淆错误。后者增强了所提取的目标语音信号与EEG表示之间的一致性，也改善了EEG表示。实验结果表明，所提出的网络显着优于基线的扬声器混淆和整体信号质量在两个说话人的情况下。摘要：Neuro-steered speaker extraction aims to extract the listener's brain-attended speech signal from a multi-talker speech signal, in which the attention is derived from the cortical activity. This activity is usually recorded using electroencephalography (EEG) devices. Though promising, current methods often have a high speaker confusion error, where the interfering speaker is extracted instead of the attended speaker, degrading the listening experience. In this work, we aim to reduce the speaker confusion error in the neuro-steered speaker extraction model through a jointly fine-tuned auxiliary auditory attention detection model. The latter reinforces the consistency between the extracted target speech signal and the EEG representation, and also improves the EEG representation. Experimental results show that the proposed network significantly outperforms the baseline in terms of speaker confusion and overall signal quality in two-talker scenarios.
【7】 w2v-SELD: A Sound Event Localization and Detection Framework for  Self-Supervised Spatial Audio Pre-Training标题：W2v-SELD：一种用于自监督空间音频预训练的声音事件定位和检测框架链接：https://arxiv.org/abs/2312.06907
作者：Orlem Lima dos Santos,Karen Rosero,Roberto de Alencar Lotufo备注：17 pages, 5 figures摘要：声音事件检测和定位（SELD）是一项复杂的任务，它依赖于大量的多声道音频记录，其中包含注释的声音事件及其各自的位置。在本文中，我们介绍了一种自监督的SELD方法，该方法改编自wav 2 vec 2.0的预训练方法，该方法直接从原始音频数据中学习表示，无需监督。通过将这种方法应用于SELD，我们可以利用大量未标记的3D音频数据来学习声音事件及其位置的鲁棒表示。我们的方法包括两个主要阶段：预训练和微调。在预训练阶段，利用未标记的3D音频数据集来训练我们的w2 v-SELD模型，捕获音频信号中固有的复杂高级特征和上下文信息。随后，在微调阶段，具有标记SELD数据的较小数据集对预训练模型进行微调。基准数据集上的实验结果表明，所提出的自监督方法的SELD的有效性。该模型超越了数据集提供的基线系统，并实现了与最先进的监督方法相当的竞争性能。我们的w2 v-SELD模型的代码和预训练参数在此存储库中可用。摘要：Sound Event Detection and Localization (SELD) constitutes a complex task that depends on extensive multichannel audio recordings with annotated sound events and their respective locations. In this paper, we introduce a self-supervised approach for SELD adapted from the pre-training methodology of wav2vec 2.0, which learns representations directly from raw audio data, eliminating the need for supervision. By applying this approach to SELD, we can leverage a substantial amount of unlabeled 3D audio data to learn robust representations of sound events and their locations. Our method comprises two primary stages: pre-training and fine-tuning. In the pre-training phase, unlabeled 3D audio datasets are utilized to train our w2v-SELD model, capturing intricate high-level features and contextual information inherent in audio signals. Subsequently, in the fine-tuning stage, a smaller dataset with labeled SELD data fine-tunes the pre-trained model. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed self-supervised approach for SELD. The model surpasses baseline systems provided with the datasets and achieves competitive performance comparable to state-of-the-art supervised methods. The code and pre-trained parameters of our w2v-SELD model are available in this repository.
eess.AS音频处理

【1】 NeuroHeed+: Improving Neuro-steered Speaker Extraction with Joint  Auditory Attention Detection标题：NeuroHeed+：利用联合听觉注意检测改进神经控制说话人提取链接：https://arxiv.org/abs/2312.07513

作者：Zexu Pan,Gordon Wichern,Francois G. Germain,Sameer Khurana,Jonathan Le Roux摘要：神经导向说话人提取的目的是从多说话人的语音信号中提取听者的大脑注意语音信号，其中注意力来自于大脑皮层的活动。这种活动通常使用脑电图（EEG）设备记录。虽然很有前途，但目前的方法通常具有较高的扬声器混淆错误，其中干扰扬声器被提取而不是被关注的扬声器，从而降低了收听体验。在这项工作中，我们的目标是通过一个共同微调的辅助听觉注意检测模型，以减少在神经操纵的说话人提取模型的说话人混淆错误。后者增强了所提取的目标语音信号与EEG表示之间的一致性，也改善了EEG表示。实验结果表明，所提出的网络显着优于基线的扬声器混淆和整体信号质量在两个说话人的情况下。摘要：Neuro-steered speaker extraction aims to extract the listener's brain-attended speech signal from a multi-talker speech signal, in which the attention is derived from the cortical activity. This activity is usually recorded using electroencephalography (EEG) devices. Though promising, current methods often have a high speaker confusion error, where the interfering speaker is extracted instead of the attended speaker, degrading the listening experience. In this work, we aim to reduce the speaker confusion error in the neuro-steered speaker extraction model through a jointly fine-tuned auxiliary auditory attention detection model. The latter reinforces the consistency between the extracted target speech signal and the EEG representation, and also improves the EEG representation. Experimental results show that the proposed network significantly outperforms the baseline in terms of speaker confusion and overall signal quality in two-talker scenarios.

【2】 w2v-SELD: A Sound Event Localization and Detection Framework for  Self-Supervised Spatial Audio Pre-Training标题：W2v-SELD：一种用于自监督空间音频预训练的声音事件定位和检测框架链接：https://arxiv.org/abs/2312.06907
作者：Orlem Lima dos Santos,Karen Rosero,Roberto de Alencar Lotufo备注：17 pages, 5 figures摘要：声音事件检测和定位（SELD）是一项复杂的任务，它依赖于大量的多声道音频记录，其中包含注释的声音事件及其各自的位置。在本文中，我们介绍了一种自监督的SELD方法，该方法改编自wav 2 vec 2.0的预训练方法，该方法直接从原始音频数据中学习表示，无需监督。通过将这种方法应用于SELD，我们可以利用大量未标记的3D音频数据来学习声音事件及其位置的鲁棒表示。我们的方法包括两个主要阶段：预训练和微调。在预训练阶段，利用未标记的3D音频数据集来训练我们的w2 v-SELD模型，捕获音频信号中固有的复杂高级特征和上下文信息。随后，在微调阶段，具有标记SELD数据的较小数据集对预训练模型进行微调。基准数据集上的实验结果表明，所提出的自监督方法的SELD的有效性。该模型超越了数据集提供的基线系统，并实现了与最先进的监督方法相当的竞争性能。我们的w2 v-SELD模型的代码和预训练参数在此存储库中可用。摘要：Sound Event Detection and Localization (SELD) constitutes a complex task that depends on extensive multichannel audio recordings with annotated sound events and their respective locations. In this paper, we introduce a self-supervised approach for SELD adapted from the pre-training methodology of wav2vec 2.0, which learns representations directly from raw audio data, eliminating the need for supervision. By applying this approach to SELD, we can leverage a substantial amount of unlabeled 3D audio data to learn robust representations of sound events and their locations. Our method comprises two primary stages: pre-training and fine-tuning. In the pre-training phase, unlabeled 3D audio datasets are utilized to train our w2v-SELD model, capturing intricate high-level features and contextual information inherent in audio signals. Subsequently, in the fine-tuning stage, a smaller dataset with labeled SELD data fine-tunes the pre-trained model. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed self-supervised approach for SELD. The model surpasses baseline systems provided with the datasets and achieves competitive performance comparable to state-of-the-art supervised methods. The code and pre-trained parameters of our w2v-SELD model are available in this repository.
【3】 Self-supervised Adaptive Pre-training of Multilingual Speech Models for  Language and Dialect Identification标题：用于语言和方言识别的多语言语音模型的自监督自适应预训练链接：https://arxiv.org/abs/2312.07338
作者：Mohammed Maqsood Shaik,Dietrich Klakow,Badr M. Abdullah备注：Submitted to ICASSP 2024摘要：经过预训练的基于transformer的语音模型在对自动语音识别和口语识别（SLID）等各种下游任务进行微调时表现出惊人的性能。然而，域不匹配的问题仍然是该领域的一个挑战，其中预训练数据的域可能与用于微调的下游标记数据的域不同。在SLID等多语言任务中，预先训练的语音模型可能不支持下游任务中的所有语言。为了应对这一挑战，我们提出了自监督自适应预训练（SAPT），以使预训练模型适应下游任务的目标领域和语言。我们将SAPT应用于XLSR-128模型，并研究这种方法对SLID任务的有效性。首先，我们证明了SAPT提高了XLSR性能的FLEURS基准大幅收益高达40.1%的代表性不足的语言。其次，我们在Few-Shot学习环境中对四个不同的数据集应用SAPT，表明我们的方法在微调过程中提高了XLSR的样本效率。我们的实验提供了强有力的经验证据，通过自我监督的持续适应提高了多语言语音模型的下游性能。摘要：Pre-trained Transformer-based speech models have shown striking performance when fine-tuned on various downstream tasks such as automatic speech recognition and spoken language identification (SLID). However, the problem of domain mismatch remains a challenge in this area, where the domain of the pre-training data might differ from that of the downstream labeled data used for fine-tuning. In multilingual tasks such as SLID, the pre-trained speech model may not support all the languages in the downstream task. To address this challenge, we propose self-supervised adaptive pre-training (SAPT) to adapt the pre-trained model to the target domain and languages of the downstream task. We apply SAPT to the XLSR-128 model and investigate the effectiveness of this approach for the SLID task. First, we demonstrate that SAPT improves XLSR performance on the FLEURS benchmark with substantial gains up to 40.1% for under-represented languages. Second, we apply SAPT on four different datasets in a few-shot learning setting, showing that our approach improves the sample efficiency of XLSR during fine-tuning. Our experiments provide strong empirical evidence that continual adaptation via self-supervision improves downstream performance for multilingual speech models.
【4】 More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module  for Multimodal Fusion Based on Signal Theory标题：Beyond Vanilla Fusion：一种基于信号理论的简单、无解耦的多模式融合注意模块链接：https://arxiv.org/abs/2312.07212
作者：Peiwen Sun,Yifan Zhang,Zishan Liu,Donghao Chen,Honggang Zhang摘要：香草融合方法仍然占主流视听任务的很大比例。然而，从理论角度来看，香草融合的有效性仍然值得讨论。因此，本文从仿生学的角度重新考虑了多模态情况下融合的信号，并提出了一个简单的，即插即用，注意力模块香草融合的基础上，信号理论和不确定性理论。此外，以前的工作多模态动态梯度调制仍然依赖于解耦的方式。因此，一个无抖动的梯度调制方案已被设计与上述的注意力模块，它具有各种优势，比解耦的。实验结果表明，只需几行代码，就可以实现高达2.0%的性能提高几个多模态分类方法。最后，其他融合任务的定量评估揭示了额外的应用场景的潜力。摘要：The vanilla fusion methods still dominate a large percentage of mainstream audio-visual tasks. However, the effectiveness of vanilla fusion from a theoretical perspective is still worth discussing. Thus, this paper reconsiders the signal fused in the multimodal case from a bionics perspective and proposes a simple, plug-and-play, attention module for vanilla fusion based on fundamental signal theory and uncertainty theory. In addition, previous work on multimodal dynamic gradient modulation still relies on decoupling the modalities. So, a decoupling-free gradient modulation scheme has been designed in conjunction with the aforementioned attention module, which has various advantages over the decoupled one. Experiment results show that just a few lines of code can achieve up to 2.0% performance improvements to several multimodal classification methods. Finally, quantitative evaluation of other fusion tasks reveals the potential for additional application scenarios.
【5】 Robust End-to-End Diarization with Domain Adaptive Training and  Multi-Task Learning标题：基于领域自适应训练和多任务学习的端到端稳健二值化链接：https://arxiv.org/abs/2312.07136
作者：Ivan Fung,Lahiru Samarakoon,Samuel J. Broughton备注：7 pages, 2 figures, ASRU 2023摘要：由于公开可用的日志数据的稀缺性，可以通过使用来自不同领域的数据训练单个模型来提高模型性能。在这项工作中，我们建议将域信息训练一个单一的端到端的多个域的日记模型。首先，我们采用域自适应训练与参数有效的适配器在飞行模型重新配置。其次，我们引入了一个辅助的领域分类任务，使日志化模型更加领域感知。对于可见域，与基线相比，我们提出的方法的组合将绝对DER从17.66%降低到16.59%。在推理期间，来自地面实况域的适配器不可用于看不见的域。我们证明了我们的模型表现出更强的泛化能力，看不见的域时，适配器被删除。对于两个看不见的域，这将DER性能分别从基线的39.91%提高到23.09%和25.32%提高到18.76%。摘要：Due to the scarcity of publicly available diarization data, the model performance can be improved by training a single model with data from different domains. In this work, we propose to incorporate domain information to train a single end-to-end diarization model for multiple domains. First, we employ domain adaptive training with parameter-efficient adapters for on-the-fly model reconfiguration. Second, we introduce an auxiliary domain classification task to make the diarization model more domain-aware. For seen domains, the combination of our proposed methods reduces the absolute DER from 17.66% to 16.59% when compared with the baseline. During inference, adapters from ground-truth domains are not available for unseen domains. We demonstrate our model exhibits a stronger generalizability to unseen domains when adapters are removed. For two unseen domains, this improves the DER performance from 39.91% to 23.09% and 25.32% to 18.76% over the baseline, respectively.
【6】 LSTM-CNN Network for Audio Signature Analysis in Noisy Environments标题：噪声环境下音频特征分析的LSTM-CNN网络链接：https://arxiv.org/abs/2312.07059
作者：Praveen Damacharla,Hamid Rajabalipanah,Mohammad Hosein Fakheri备注：10th Annual Conf. on Computational Science & Computational Intelligence (CSCI'23)摘要：有多个应用程序可以在工作，展览，商场，销售和工业用途中自动计算人数并指定其性别。虽然目前的语音检测方法应该运行良好，在大多数情况下，除了性别，当前说话人的数量是未知的，分类方法是不合适的，由于许多可能的类。在这项研究中，我们专注于长短期记忆卷积神经网络（LSTM-CNN）来提取声音数据的时间和/或频率相关特征，以估计在嘈杂环境中每帧同时活动的说话者的数量/性别。考虑到扬声器的最大数量为10，我们已经利用了19000音频样本的男性，女性和背景噪声的不同组合，在公共城市，工业情况，商场，展览，工作场所和自然学习的目的。该概念证明显示出在检测计数和性别方面具有约0.019/0.017的训练/验证MSE值的有希望的性能。摘要：There are multiple applications to automatically count people and specify their gender at work, exhibitions, malls, sales, and industrial usage. Although current speech detection methods are supposed to operate well, in most situations, in addition to genders, the number of current speakers is unknown and the classification methods are not suitable due to many possible classes. In this study, we focus on a long-short-term memory convolutional neural network (LSTM-CNN) to extract time and / or frequency-dependent features of the sound data to estimate the number / gender of simultaneous active speakers at each frame in noisy environments. Considering the maximum number of speakers as 10, we have utilized 19000 audio samples with diverse combinations of males, females, and background noise in public cities, industrial situations, malls, exhibitions, workplaces, and nature for learning purposes. This proof of concept shows promising performance with training/validation MSE values of about 0.019/0.017 in detecting count and gender.
【7】 Evaluating Self-supervised Speech Models on a Taiwanese Hokkien Corpus标题：自监督语音模型在台湾闽南语语料库上的评价链接：https://arxiv.org/abs/2312.06668
作者：Yi-Hui Chou,Kalvin Chang,Meng-Ju Wu,Winston Ou,Alice Wen-Hsin Bi,Carol Yang,Bryan Y. Chen,Rong-Wei Pai,Po-Yen Yeh,Jo-Peng Chiang,Iu-Tshian Phoann,Winnie Chang,Chenxuan Cui,Noel Chen,Jiatong Shi备注：Accepted to ASRU 2023摘要：台湾闽南语的使用和地位正在下降，因为台湾的语言转向普通话。这就是为什么它在今天的NLP和语音研究中是一种低资源语言的部分原因。为了确保语音处理的最新技术不会让台湾闽南语落后，我们将一个1.5小时的台湾闽南语数据集贡献给ML-SUPERB的隐藏集。在我们的数据集上评估ML-SUPERB的自监督学习（SSL）语音表示套件，我们发现模型大小并不总是决定性能。事实上，某些较小的模型优于较大的模型。此外，预训练数据和目标语言之间的语言对齐起着至关重要的作用。摘要：Taiwanese Hokkien is declining in use and status due to a language shift towards Mandarin in Taiwan. This is partly why it is a low resource language in NLP and speech research today. To ensure that the state of the art in speech processing does not leave Taiwanese Hokkien behind, we contribute a 1.5-hour dataset of Taiwanese Hokkien to ML-SUPERB's hidden set. Evaluating ML-SUPERB's suite of self-supervised learning (SSL) speech representations on our dataset, we find that model size does not consistently determine performance. In fact, certain smaller models outperform larger ones. Furthermore, linguistic alignment between pretraining data and the target language plays a crucial role.
机器翻译由腾讯交互翻译提供，仅供参