今日论文合集：cs.SD语音6篇，eess.AS音频处理6篇。
cs.SD语音

【1】Underwater Acoustic Signal Recognition Based on Salient Features

标题：基于显著特征的水声信号识别链接：https://arxiv.org/abs/2312.13143

作者：Minghao Chen摘要：随着科学技术的飞速进步，复杂环境下水声信号的识别变得越来越关键。目前，主流的水声信号识别方法主要依靠时频分析来提取信号的频谱特征，在该领域得到了广泛的应用。然而，现有的识别方法严重依赖于专家系统，面临着局限性，如有限的知识库和处理复杂关系的挑战。这些限制源于与规则或推理引擎相关的复杂性和维护困难。认识到深度学习在处理复杂关系方面的潜在优势，本文提出了一种利用神经网络进行水声信号识别的方法。所提出的方法涉及不断学习的特征提取的频谱分类的水声信号。深度学习模型可以自动从数据中学习抽象特征，并在训练过程中不断调整权重，以提高分类性能。摘要：With the rapid advancement of technology, the recognition of underwater acoustic signals in complex environments has become increasingly crucial. Currently, mainstream underwater acoustic signal recognition relies primarily on time-frequency analysis to extract spectral features, finding widespread applications in the field. However, existing recognition methods heavily depend on expert systems, facing limitations such as restricted knowledge bases and challenges in handling complex relationships. These limitations stem from the complexity and maintenance difficulties associated with rules or inference engines. Recognizing the potential advantages of deep learning in handling intricate relationships, this paper proposes a method utilizing neural networks for underwater acoustic signal recognition. The proposed approach involves continual learning of features extracted from spectra for the classification of underwater acoustic signals. Deep learning models can automatically learn abstract features from data and continually adjust weights during training to enhance classification performance.

【2】 FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous  Self-Supervised Learning标题：FusDOM：域内和域外知识相结合的连续自监督学习链接：https://arxiv.org/abs/2312.13026
作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha备注：Accepted at ICASSP 2024. Code: this https URL摘要：持续预训练（CP）提供了多种优势，例如目标域自适应和利用在线可用的未标记数据的连续流的潜力。然而，对域外分布的持续预训练通常会导致对先前获得的知识的灾难性遗忘，从而导致次优ASR性能。本文介绍了FusDom，一种简单而新颖的基于SSL的持续预训练方法。FusDom学习的语音表示是鲁棒和自适应的，但不会忘记过去看到的概念。FusDom不是在单个模型的输出表示上解决SSL预文本任务，而是利用两个相同的预训练SSL模型，一个教师和一个学生，用修改后的预训练头来解决CP SSL预文本任务。这个头在两个模型的表示之间采用交叉注意机制，而只有学生接收梯度更新，而教师不接收。最后，对学生进行ASR微调。在实践中，FusDom在不同设置下的表现明显优于我们的所有基线，目标域中的WER改进范围为0.2 WER - 7.3 WER，同时保留了早期域中的性能。摘要：Continued pre-training (CP) offers multiple advantages, like target domain adaptation and the potential to exploit the continuous stream of unlabeled data available online. However, continued pre-training on out-of-domain distributions often leads to catastrophic forgetting of previously acquired knowledge, leading to sub-optimal ASR performance. This paper presents FusDom, a simple and novel methodology for SSL-based continued pre-training. FusDom learns speech representations that are robust and adaptive yet not forgetful of concepts seen in the past. Instead of solving the SSL pre-text task on the output representations of a single model, FusDom leverages two identical pre-trained SSL models, a teacher and a student, with a modified pre-training head to solve the CP SSL pre-text task. This head employs a cross-attention mechanism between the representations of both models while only the student receives gradient updates and the teacher does not. Finally, the student is fine-tuned for ASR. In practice, FusDom outperforms all our baselines across settings significantly, with WER improvements in the range of 0.2 WER - 7.3 WER in the target domain while retaining the performance in the earlier domain.
【3】 CST-former: Transformer with Channel-Spectro-Temporal Attention for  Sound Event Localization and Detection标题：CST-FORFER：用于声事件定位和检测的通道-频谱-时间注意转换器链接：https://arxiv.org/abs/2312.12821
作者：Yusun Shul,Jung-Woo Choi备注：Accepted to ICASSP 2024摘要：声事件定位与检测（SELD）是利用多通道声信号对声事件进行分类和到达方向（DoA）定位的任务。先前的研究采用频谱和通道信息作为时间注意的嵌入。然而，这种用法限制了深度神经网络从光谱或空间域中提取有意义的特征。因此，我们在本文中的调查提出了一种新的框架，称为光谱-时间Transformer（CST前），支持SELD性能通过独立的应用程序的注意机制，以不同的领域。CST-前架构采用不同的注意力机制来独立地处理通道、光谱和时间信息。此外，我们提出了一个展开的本地嵌入（ULE）技术的通道注意力（CA）生成信息嵌入向量，包括本地频谱和时间信息。通过对2022年和2023年DCASE挑战task 3数据集的实验进行实证验证，证实了采用跨每个领域分离的注意力机制的有效性以及ULE在增强SELD性能方面的优势。摘要：Sound event localization and detection (SELD) is a task for the classification of sound events and the localization of direction of arrival (DoA) utilizing multichannel acoustic signals. Prior studies employ spectral and channel information as the embedding for temporal attention. However, this usage limits the deep neural network from extracting meaningful features from the spectral or spatial domains. Therefore, our investigation in this paper presents a novel framework termed the Channel-Spectro-Temporal Transformer (CST-former) that bolsters SELD performance through the independent application of attention mechanisms to distinct domains. The CST-former architecture employs distinct attention mechanisms to independently process channel, spectral, and temporal information. In addition, we propose an unfolded local embedding (ULE) technique for channel attention (CA) to generate informative embedding vectors including local spectral and temporal information. Empirical validation through experimentation on the 2022 and 2023 DCASE Challenge task3 datasets affirms the efficacy of employing attention mechanisms separated across each domain and the benefit of ULE, in enhancing SELD performance.
【4】 Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and  Detection标题：不流利语音转录和检测的无约束不流利建模链接：https://arxiv.org/abs/2312.12810
作者：Jiachen Lian,Carly Feng,Naasir Farooqi,Steve Li,Anshul Kashyap,Cheol Jun Cho,Peter Wu,Robbie Netzorg,Tingle Li,Gopala Krishna Anumanchipalli备注：2023 ASRU摘要：不流利语音建模需要在单词级和语音级都具有时间准确性和沉默意识的转录。然而，目前的研究主要集中在转录或检测的不流利建模，和每个方面的性能仍然有限。在这项工作中，我们提出了一个无约束的不流畅建模（UDM）的方法，解决了转录和检测的自动和分层的方式。UDM通过提供全面的解决方案消除了大量手动注释的需要。此外，我们还引入了一个名为VCTK++的模拟不流利数据集，以增强UDM在语音转录方面的能力。我们的实验结果表明，我们提出的方法在转录和检测任务的有效性和鲁棒性。摘要：Dysfluent speech modeling requires time-accurate and silence-aware transcription at both the word-level and phonetic-level. However, current research in dysfluency modeling primarily focuses on either transcription or detection, and the performance of each aspect remains limited. In this work, we present an unconstrained dysfluency modeling (UDM) approach that addresses both transcription and detection in an automatic and hierarchical manner. UDM eliminates the need for extensive manual annotation by providing a comprehensive solution. Furthermore, we introduce a simulated dysfluent dataset called VCTK++ to enhance the capabilities of UDM in phonetic transcription. Our experimental results demonstrate the effectiveness and robustness of our proposed methods in both transcription and detection tasks.
【5】 Stable Distillation: Regularizing Continued Pre-training for  Low-Resource Automatic Speech Recognition标题：稳定蒸馏：规范低资源自动语音识别的持续预训练链接：https://arxiv.org/abs/2312.12783
作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha备注：Accepted to ICASSP 2024. Code: this https URL摘要：持续自监督（SSL）预训练，以适应现有的SSL模型的目标领域已被证明是非常有效的低资源自动语音识别（ASR）。本文提出了稳定蒸馏，这是一种简单而新颖的基于SSL的持续预训练方法，可以提高目标域中的ASR性能，其中标记和未标记数据都是有限的。Stable Distillation采用自蒸馏作为持续预训练的正则化，缓解了过度拟合问题，这是源域和目标域不同时持续预训练面临的常见问题。具体来说，首先，我们对目标领域ASR数据集上的初始SSL预训练模型执行vanilla持续预训练，并将其称为教师。接下来，我们将相同的初始预训练模型作为学生来执行继续的预训练，同时强制其隐藏表示接近教师的隐藏表示（通过MSE损失）。然后，该学生用于对目标数据集进行下游ASR微调。在实践中，稳定蒸馏在各种实验设置中进行评估时，其性能优于我们所有的基线0.8 - 7 WER。摘要：Continued self-supervised (SSL) pre-training for adapting existing SSL models to the target domain has shown to be extremely effective for low-resource Automatic Speech Recognition (ASR). This paper proposes Stable Distillation, a simple and novel approach for SSL-based continued pre-training that boosts ASR performance in the target domain where both labeled and unlabeled data are limited. Stable Distillation employs self-distillation as regularization for continued pre-training, alleviating the over-fitting issue, a common problem continued pre-training faces when the source and target domains differ. Specifically, first, we perform vanilla continued pre-training on an initial SSL pre-trained model on the target domain ASR dataset and call it the teacher. Next, we take the same initial pre-trained model as a student to perform continued pre-training while enforcing its hidden representations to be close to that of the teacher (via MSE loss). This student is then used for downstream ASR fine-tuning on the target dataset. In practice, Stable Distillation outperforms all our baselines by 0.8 - 7 WER when evaluated in various experimental settings.
【6】 Lattice Rescoring Based on Large Ensemble of Complementary Neural  Language Models标题：基于互补神经语言模型大集合格的研究链接：https://arxiv.org/abs/2312.12764
作者：Atsunori Ogawa,Naohiro Tawara,Marc Delcroix,Shoko Araki备注：Accepted to ICASSP 2022摘要：我们研究了使用大型集成的高级神经语言模型（NLM）的自动语音识别（ASR）的假设格rescoring的有效性。以前的研究已经报道了结合少量NLM的有效性。相比之下，在这项研究中，我们结合了多达八个NLM，即，前向/后向长短期记忆/用两个不同的随机初始化种子训练的变换器LM。我们结合这些NLM通过迭代格生成。由于这些NLM相互补充，通过在每次重新评分迭代中逐个组合它们，可以逐渐细化附加到给定格弧的语言分数。因此，ASR假设的误差可以逐渐减小。我们还研究了携带上下文信息（以前的rescoring结果）跨越一个长的语音，如演讲演讲的格序列的有效性。在实验中，使用演讲语音语料库，通过结合八个NLM和使用上下文结转，我们得到了24.4%的相对字错误率降低从ASR 1-最佳基线。为了进一步比较，我们同时进行（即，非迭代）NLM组合和使用大NLM系综的100最佳重评分，证实了迭代NLM组合的格重评分的优势。摘要：We investigate the effectiveness of using a large ensemble of advanced neural language models (NLMs) for lattice rescoring on automatic speech recognition (ASR) hypotheses. Previous studies have reported the effectiveness of combining a small number of NLMs. In contrast, in this study, we combine up to eight NLMs, i.e., forward/backward long short-term memory/Transformer-LMs that are trained with two different random initialization seeds. We combine these NLMs through iterative lattice generation. Since these NLMs work complementarily with each other, by combining them one by one at each rescoring iteration, language scores attached to given lattice arcs can be gradually refined. Consequently, errors of the ASR hypotheses can be gradually reduced. We also investigate the effectiveness of carrying over contextual information (previous rescoring results) across a lattice sequence of a long speech such as a lecture speech. In experiments using a lecture speech corpus, by combining the eight NLMs and using context carry-over, we obtained a 24.4% relative word error rate reduction from the ASR 1-best baseline. For further comparison, we performed simultaneous (i.e., non-iterative) NLM combination and 100-best rescoring using the large ensemble of NLMs, which confirmed the advantage of lattice rescoring with iterative NLM combination.
eess.AS音频处理
【1】 FusDom: Combining In-Domain and Out-of-Domain Knowledge for Continuous  Self-Supervised Learning标题：FusDOM：域内和域外知识相结合的连续自监督学习链接：https://arxiv.org/abs/2312.13026
作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha备注：Accepted at ICASSP 2024. Code: this https URL摘要：持续预训练（CP）提供了多种优势，例如目标域自适应和利用在线可用的未标记数据的连续流的潜力。然而，对域外分布的持续预训练通常会导致对先前获得的知识的灾难性遗忘，从而导致次优ASR性能。本文介绍了FusDom，一种简单而新颖的基于SSL的持续预训练方法。FusDom学习的语音表示是鲁棒和自适应的，但不会忘记过去看到的概念。FusDom不是在单个模型的输出表示上解决SSL预文本任务，而是利用两个相同的预训练SSL模型，一个教师和一个学生，用修改后的预训练头来解决CP SSL预文本任务。这个头在两个模型的表示之间采用交叉注意机制，而只有学生接收梯度更新，而教师不接收。最后，对学生进行ASR微调。在实践中，FusDom在不同设置下的表现明显优于我们的所有基线，目标域中的WER改进范围为0.2 WER - 7.3 WER，同时保留了早期域中的性能。摘要：Continued pre-training (CP) offers multiple advantages, like target domain adaptation and the potential to exploit the continuous stream of unlabeled data available online. However, continued pre-training on out-of-domain distributions often leads to catastrophic forgetting of previously acquired knowledge, leading to sub-optimal ASR performance. This paper presents FusDom, a simple and novel methodology for SSL-based continued pre-training. FusDom learns speech representations that are robust and adaptive yet not forgetful of concepts seen in the past. Instead of solving the SSL pre-text task on the output representations of a single model, FusDom leverages two identical pre-trained SSL models, a teacher and a student, with a modified pre-training head to solve the CP SSL pre-text task. This head employs a cross-attention mechanism between the representations of both models while only the student receives gradient updates and the teacher does not. Finally, the student is fine-tuned for ASR. In practice, FusDom outperforms all our baselines across settings significantly, with WER improvements in the range of 0.2 WER - 7.3 WER in the target domain while retaining the performance in the earlier domain.
【2】 CST-former: Transformer with Channel-Spectro-Temporal Attention for  Sound Event Localization and Detection标题：CST-FORFER：用于声事件定位和检测的通道-频谱-时间注意转换器链接：https://arxiv.org/abs/2312.12821
作者：Yusun Shul,Jung-Woo Choi备注：Accepted to ICASSP 2024摘要：声事件定位与检测（SELD）是利用多通道声信号对声事件进行分类和到达方向（DoA）定位的任务。先前的研究采用频谱和通道信息作为时间注意的嵌入。然而，这种用法限制了深度神经网络从光谱或空间域中提取有意义的特征。因此，我们在本文中的调查提出了一种新的框架，称为光谱-时间Transformer（CST前），支持SELD性能通过独立的应用程序的注意机制，以不同的领域。CST-前架构采用不同的注意力机制来独立地处理通道、光谱和时间信息。此外，我们提出了一个展开的本地嵌入（ULE）技术的通道注意力（CA）生成信息嵌入向量，包括本地频谱和时间信息。通过对2022年和2023年DCASE挑战task 3数据集的实验进行实证验证，证实了采用跨每个领域分离的注意力机制的有效性以及ULE在增强SELD性能方面的优势。摘要：Sound event localization and detection (SELD) is a task for the classification of sound events and the localization of direction of arrival (DoA) utilizing multichannel acoustic signals. Prior studies employ spectral and channel information as the embedding for temporal attention. However, this usage limits the deep neural network from extracting meaningful features from the spectral or spatial domains. Therefore, our investigation in this paper presents a novel framework termed the Channel-Spectro-Temporal Transformer (CST-former) that bolsters SELD performance through the independent application of attention mechanisms to distinct domains. The CST-former architecture employs distinct attention mechanisms to independently process channel, spectral, and temporal information. In addition, we propose an unfolded local embedding (ULE) technique for channel attention (CA) to generate informative embedding vectors including local spectral and temporal information. Empirical validation through experimentation on the 2022 and 2023 DCASE Challenge task3 datasets affirms the efficacy of employing attention mechanisms separated across each domain and the benefit of ULE, in enhancing SELD performance.
【3】 Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and  Detection标题：不流利语音转录和检测的无约束不流利建模链接：https://arxiv.org/abs/2312.12810
作者：Jiachen Lian,Carly Feng,Naasir Farooqi,Steve Li,Anshul Kashyap,Cheol Jun Cho,Peter Wu,Robbie Netzorg,Tingle Li,Gopala Krishna Anumanchipalli备注：2023 ASRU摘要：不流利语音建模需要在单词级和语音级都具有时间准确性和沉默意识的转录。然而，目前的研究主要集中在转录或检测的不流利建模，和每个方面的性能仍然有限。在这项工作中，我们提出了一个无约束的不流畅建模（UDM）的方法，解决了转录和检测的自动和分层的方式。UDM通过提供全面的解决方案消除了大量手动注释的需要。此外，我们还引入了一个名为VCTK++的模拟不流利数据集，以增强UDM在语音转录方面的能力。我们的实验结果表明，我们提出的方法在转录和检测任务的有效性和鲁棒性。摘要：Dysfluent speech modeling requires time-accurate and silence-aware transcription at both the word-level and phonetic-level. However, current research in dysfluency modeling primarily focuses on either transcription or detection, and the performance of each aspect remains limited. In this work, we present an unconstrained dysfluency modeling (UDM) approach that addresses both transcription and detection in an automatic and hierarchical manner. UDM eliminates the need for extensive manual annotation by providing a comprehensive solution. Furthermore, we introduce a simulated dysfluent dataset called VCTK++ to enhance the capabilities of UDM in phonetic transcription. Our experimental results demonstrate the effectiveness and robustness of our proposed methods in both transcription and detection tasks.
【4】 Stable Distillation: Regularizing Continued Pre-training for  Low-Resource Automatic Speech Recognition标题：稳定蒸馏：规范低资源自动语音识别的持续预训练链接：https://arxiv.org/abs/2312.12783
作者：Ashish Seth,Sreyan Ghosh,S. Umesh,Dinesh Manocha备注：Accepted to ICASSP 2024. Code: this https URL摘要：持续自监督（SSL）预训练，以适应现有的SSL模型的目标领域已被证明是非常有效的低资源自动语音识别（ASR）。本文提出了稳定蒸馏，这是一种简单而新颖的基于SSL的持续预训练方法，可以提高目标域中的ASR性能，其中标记和未标记数据都是有限的。Stable Distillation采用自蒸馏作为持续预训练的正则化，缓解了过度拟合问题，这是源域和目标域不同时持续预训练面临的常见问题。具体来说，首先，我们对目标领域ASR数据集上的初始SSL预训练模型执行vanilla持续预训练，并将其称为教师。接下来，我们将相同的初始预训练模型作为学生来执行继续的预训练，同时强制其隐藏表示接近教师的隐藏表示（通过MSE损失）。然后，该学生用于对目标数据集进行下游ASR微调。在实践中，稳定蒸馏在各种实验设置中进行评估时，其性能优于我们所有的基线0.8 - 7 WER。摘要：Continued self-supervised (SSL) pre-training for adapting existing SSL models to the target domain has shown to be extremely effective for low-resource Automatic Speech Recognition (ASR). This paper proposes Stable Distillation, a simple and novel approach for SSL-based continued pre-training that boosts ASR performance in the target domain where both labeled and unlabeled data are limited. Stable Distillation employs self-distillation as regularization for continued pre-training, alleviating the over-fitting issue, a common problem continued pre-training faces when the source and target domains differ. Specifically, first, we perform vanilla continued pre-training on an initial SSL pre-trained model on the target domain ASR dataset and call it the teacher. Next, we take the same initial pre-trained model as a student to perform continued pre-training while enforcing its hidden representations to be close to that of the teacher (via MSE loss). This student is then used for downstream ASR fine-tuning on the target dataset. In practice, Stable Distillation outperforms all our baselines by 0.8 - 7 WER when evaluated in various experimental settings.
【5】 Lattice Rescoring Based on Large Ensemble of Complementary Neural  Language Models标题：基于互补神经语言模型大集合格的研究链接：https://arxiv.org/abs/2312.12764
作者：Atsunori Ogawa,Naohiro Tawara,Marc Delcroix,Shoko Araki备注：Accepted to ICASSP 2022摘要：我们研究了使用大型集成的高级神经语言模型（NLM）的自动语音识别（ASR）的假设格rescoring的有效性。以前的研究已经报道了结合少量NLM的有效性。相比之下，在这项研究中，我们结合了多达八个NLM，即，前向/后向长短期记忆/用两个不同的随机初始化种子训练的变换器LM。我们结合这些NLM通过迭代格生成。由于这些NLM相互补充，通过在每次重新评分迭代中逐个组合它们，可以逐渐细化附加到给定格弧的语言分数。因此，ASR假设的误差可以逐渐减小。我们还研究了携带上下文信息（以前的rescoring结果）跨越一个长的语音，如演讲演讲的格序列的有效性。在实验中，使用演讲语音语料库，通过结合八个NLM和使用上下文结转，我们得到了24.4%的相对字错误率降低从ASR 1-最佳基线。为了进一步比较，我们同时进行（即，非迭代）NLM组合和使用大NLM系综的100最佳重评分，证实了迭代NLM组合的格重评分的优势。摘要：We investigate the effectiveness of using a large ensemble of advanced neural language models (NLMs) for lattice rescoring on automatic speech recognition (ASR) hypotheses. Previous studies have reported the effectiveness of combining a small number of NLMs. In contrast, in this study, we combine up to eight NLMs, i.e., forward/backward long short-term memory/Transformer-LMs that are trained with two different random initialization seeds. We combine these NLMs through iterative lattice generation. Since these NLMs work complementarily with each other, by combining them one by one at each rescoring iteration, language scores attached to given lattice arcs can be gradually refined. Consequently, errors of the ASR hypotheses can be gradually reduced. We also investigate the effectiveness of carrying over contextual information (previous rescoring results) across a lattice sequence of a long speech such as a lecture speech. In experiments using a lecture speech corpus, by combining the eight NLMs and using context carry-over, we obtained a 24.4% relative word error rate reduction from the ASR 1-best baseline. For further comparison, we performed simultaneous (i.e., non-iterative) NLM combination and 100-best rescoring using the large ensemble of NLMs, which confirmed the advantage of lattice rescoring with iterative NLM combination.
【6】 Underwater Acoustic Signal Recognition Based on Salient Features标题：基于显著特征的水声信号识别链接：https://arxiv.org/abs/2312.13143
作者：Minghao Chen摘要：随着科学技术的飞速进步，复杂环境下水声信号的识别变得越来越关键。目前，主流的水声信号识别方法主要依靠时频分析来提取信号的频谱特征，在该领域得到了广泛的应用。然而，现有的识别方法严重依赖于专家系统，面临着局限性，如有限的知识库和处理复杂关系的挑战。这些限制源于与规则或推理引擎相关的复杂性和维护困难。认识到深度学习在处理复杂关系方面的潜在优势，本文提出了一种利用神经网络进行水声信号识别的方法。所提出的方法涉及不断学习的特征提取的频谱分类的水声信号。深度学习模型可以自动从数据中学习抽象特征，并在训练过程中不断调整权重，以提高分类性能。摘要：With the rapid advancement of technology, the recognition of underwater acoustic signals in complex environments has become increasingly crucial. Currently, mainstream underwater acoustic signal recognition relies primarily on time-frequency analysis to extract spectral features, finding widespread applications in the field. However, existing recognition methods heavily depend on expert systems, facing limitations such as restricted knowledge bases and challenges in handling complex relationships. These limitations stem from the complexity and maintenance difficulties associated with rules or inference engines. Recognizing the potential advantages of deep learning in handling intricate relationships, this paper proposes a method utilizing neural networks for underwater acoustic signal recognition. The proposed approach involves continual learning of features extracted from spectra for the classification of underwater acoustic signals. Deep learning models can automatically learn abstract features from data and continually adjust weights during training to enhance classification performance.
机器翻译由腾讯交互翻译提供，仅供参考