今日论文合集：cs.SD语音4篇，eess.AS音频处理4篇。
【1】 Efficient Domain Adaptation for Speech Foundation Models
标题：一种高效的语音基础模型域自适应算法
链接：https://arxiv.org/abs/2302.01496
作者：Bo Li,Dongseong Hwang,Zhouyuan Huo,Junwen Bai,Guru Prakash,Tara N. Sainath,Khe Chai Sim,Yu Zhang,Wei Han,Trevor Strohman,Francoise Beaufays摘要：基础模型（FM）是在大规模的广泛数据上训练的，并可适应广泛的下游任务，已引起研究界的极大兴趣。基础模型得益于不同的数据源，如不同的模态、语言和应用领域，表现出了强大的泛化和知识转移能力。在本文中，我们提出了一个开创性的研究，为建立一个有效的解决方案，调频为基础的语音识别系统。我们采用最新发展的自监督BEST-RQ进行预训练，并提出了基于JUST Hydra的源域和无监督目标域数据的联合微调。FM编码器适配器和解码器然后被微调到具有少量受监督的域内数据的目标域。在大规模的YouTube和语音搜索任务中，我们的方法被证明是数据和模型参数有效的。与731.1M模型在额外的300 M监督域内数据上从头训练相比，它仅用21.6M监督域内数据和130.8M微调参数就实现了相同的质量。摘要：Foundation models (FMs), that are trained on broad data at scale and are adaptable to a wide range of downstream tasks, have brought large interest in the research community. Benefiting from the diverse data sources such as different modalities, languages and application domains, foundation models have demonstrated strong generalization and knowledge transfer capabilities. In this paper, we present a pioneering study towards building an efficient solution for FM-based speech recognition systems. We adopt the recently developed self-supervised BEST-RQ for pretraining, and propose the joint finetuning with both source and unsupervised target domain data using JUST Hydra. The FM encoder adapter and decoder are then finetuned to the target domain with a small amount of supervised in-domain data. On a large-scale YouTube and Voice Search task, our method is shown to be both data and model parameter efficient. It achieves the same quality with only 21.6M supervised in-domain data and 130.8M finetuned parameters, compared to the 731.1M model trained from scratch on additional 300M supervised in-domain data.
【2】 SPADE: Self-supervised Pretraining for Acoustic DisEntanglement
标题：Spade：用于声学解缠的自监督预训练
链接：https://arxiv.org/abs/2302.01483
作者：John Harvill,Jarred Barber,Arun Nair,Ramin Pishehvar机构：University of Illinois Urbana-Champaign, USA, Amazon Alexa AI, USA摘要：自监督表征学习方法由于能够在大量未标记数据上训练模型而变得越来越流行，并且已经在诸如自然语言处理、计算机视觉和语音等不同领域中证明了成功。以往在语音领域的自我监督工作已经解开了语音的多个属性，如语言内容、说话人身份和节奏。在这项工作中，我们介绍了一种自监督的方法来从语音中分离房间声学，并将声学表示用于设备仲裁的下游任务。我们的结果表明，当标记的训练数据稀少时，我们提出的方法显著地改善了基线上的性能，这表明我们的预训练方案学习编码房间声学信息，同时保持对语音信号的其他属性不变。摘要：Self-supervised representation learning approaches have grown in popularity due to the ability to train models on large amounts of unlabeled data and have demonstrated success in diverse fields such as natural language processing, computer vision, and speech. Previous self-supervised work in the speech domain has disentangled multiple attributes of speech such as linguistic content, speaker identity, and rhythm. In this work, we introduce a self-supervised approach to disentangle room acoustics from speech and use the acoustic representation on the downstream task of device arbitration. Our results demonstrate that our proposed approach significantly improves performance over a baseline when labeled training data is scarce, indicating that our pretraining scheme learns to encode room acoustic information while remaining invariant to other attributes of the speech signal.
【3】 Machine Learning Extreme Acoustic Non-reciprocity in a Linear Waveguide with Multiple Nonlinear Asymmetric Gates
标题：具有多个非线性非对称门的线性波导中的机器学习极值声学非互易
链接：https://arxiv.org/abs/2302.01746
作者：Anargyros Michaloliakos,Chongan Wang,Alexander F. Vakakis机构：a Department of Mechanical Science and Engineering, University of Illinois, Urbana-Champaign, a Department of Mechanical and Aerospace Engineering, University of California, San Diego摘要：本文研究了一个含有两个局部强非线性非对称门的无源一维线性波导所表现出的声学非互易性。两个局部非线性门破坏了波导的对称性和线性，产生了强的全局非互易声学，其方式是根据谐波激励的应用侧而发生极其不同的声学响应。据作者所知，本发明的双门波导能够具有极高的声学非互易性，其水平远远高于当前文献中有源或无源器件所报道的水平;此外，在波传播的期望方向上，这种极端的性能与可接受水平的传输率相结合。机器学习被用于根据可传输性和非互易性的测量来预测设计该选通波导，目的是减少高维参数空间分析所需的计算时间。这项研究为这些介质的物理学提供了新的思路，并考虑了使用神经网络分析这类物理问题的优势和局限性。在预测的强非互易性理想参数空间中，最大传输率高达40%，并且从上游到下游的传输能量变化高达9个数量级，这取决于波传输的方向。机器学习工具以及本工作的数值方法可以为实际的非互易波导和声学超材料的预测设计提供信息，这些波导和声学超材料包含局部非线性门。本文表明，非线性门的组合可以导致极高的非互易性，同时保持所需的可传递性水平。摘要：This work is a study of acoustic non-reciprocity exhibited by a passive one-dimensional linear waveguide incorporating two local strongly nonlinear, asymmetric gates. Two local nonlinear gates break the symmetry and linearity of the waveguide, yielding strong global non-reciprocal acoustics, in the way that extremely different acoustical responses occur depending on the side of application of harmonic excitation. To the authors' best knowledge that the present two-gated waveguide is capable of extremely high acoustic non-reciprocity, at a much higher level to what is reported by active or passive devices in the current literature; moreover, this extreme performance combines with acceptable levels of transmissibility in the desired direction of wave propagation. Machine learning is utilized for predictive design of this gated waveguide in terms of the measures of transmissibility and non-reciprocity, with the aim of reducing the required computational time for high-dimensional parameter space analysis. The study sheds new light into the physics of these media and considers the advantages and limitations of using neural networks to analyze this type of physical problems. In the predicted desirable parameter space for intense non-reciprocity, the maximum transmissibility reaches as much as 40%, and the transmitted energy from upstream to downstream varies up to nine orders of magnitude, depending on the direction of wave transmission. The machine learning tools along with the numerical methods of this work can inform predictive designs of practical non-reciprocal waveguides and acoustic metamaterials that incorporate local nonlinear gates. The current paper shows that combinations of nonlinear gates can lead to extremely high non-reciprocity while maintaining desired levels of transmissibility.
【4】 Relating EEG to continuous speech using deep neural networks: a review
标题：利用深度神经网络将脑电与连续语音联系起来的研究进展
链接：https://arxiv.org/abs/2302.01736
作者：Corentin Puffay,Bernd Accou,Lies Bollens,Mohammad Jalilpour Monesi,Jonas Vanthornhout,Hugo Van hamme,Tom Francart机构：KU Leuven, Dept. Neurosciences, ExpORL, Leuven, Belgium, KU Leuven, Dept. of Electrical engineering (ESAT), PSI, Leuven, Belgium摘要：客观。当一个人听到连续的语音时，在大脑中会引起相应的反应，并且可以使用脑电图（EEG）来记录。线性模型目前用于将EEG记录与对应的语音信号相关联。线性模型找到这两个信号之间的映射的能力被用作语音的神经跟踪的度量。这种模型是有限的，因为它们假设EEG-语音关系是线性的，这忽略了大脑的非线性动态。作为替代方案，深度学习模型最近被用于将EEG与连续语音相关联，特别是在听觉注意力解码（AAD）和单语音源范例中。过来本文回顾和评论了基于深度学习的研究，这些研究将EEG与AAD和单语音源范式中的连续语音联系起来。我们指出了反复出现的方法学陷阱和模型分析标准基准的必要性。主要结果。我们收集了28项研究。我们发现的主要方法学问题是有偏倚的交叉验证、导致过拟合模型的数据泄漏或与模型复杂性相比不成比例的数据大小。此外，我们还解决了标准基准模型分析的需求，例如公共数据集、公共评估度量标准和匹配-不匹配任务的良好实践。意义。我们是第一个提交综述论文的人，总结了基于深度学习的主要研究，这些研究将EEG与语音联系起来，同时解决了这个新兴领域的方法论陷阱和重要考虑因素。鉴于深度学习在脑电语音解码中的应用越来越广泛，我们的研究尤其重要。摘要：Objective. When a person listens to continuous speech, a corresponding response is elicited in the brain and can be recorded using electroencephalography (EEG). Linear models are presently used to relate the EEG recording to the corresponding speech signal. The ability of linear models to find a mapping between these two signals is used as a measure of neural tracking of speech. Such models are limited as they assume linearity in the EEG-speech relationship, which omits the nonlinear dynamics of the brain. As an alternative, deep learning models have recently been used to relate EEG to continuous speech, especially in auditory attention decoding (AAD) and single-speech-source paradigms. Approach. This paper reviews and comments on deep-learning-based studies that relate EEG to continuous speech in AAD and single-speech-source paradigms. We point out recurrent methodological pitfalls and the need for a standard benchmark of model analysis. Main results. We gathered 28 studies. The main methodological issues we found are biased cross-validations, data leakage leading to over-fitted models, or disproportionate data size compared to the model's complexity. In addition, we address requirements for a standard benchmark model analysis, such as public datasets, common evaluation metrics, and good practices for the match-mismatch task. Significance. We are the first to present a review paper summarizing the main deep-learning-based studies that relate EEG to speech while addressing methodological pitfalls and important considerations for this newly expanding field. Our study is particularly relevant given the growing application of deep learning in EEG-speech decoding.【1】 Machine Learning Extreme Acoustic Non-reciprocity in a Linear Waveguide with Multiple Nonlinear Asymmetric Gates
标题：具有多个非线性非对称门的线性波导中的机器学习极值声学非互易
链接：https://arxiv.org/abs/2302.01746
*与cs.SD语音【3】为同一篇
作者：Anargyros Michaloliakos,Chongan Wang,Alexander F. Vakakis机构：a Department of Mechanical Science and Engineering, University of Illinois, Urbana-Champaign, a Department of Mechanical and Aerospace Engineering, University of California, San Diego摘要：本文研究了一个含有两个局部强非线性非对称门的无源一维线性波导所表现出的声学非互易性。两个局部非线性门破坏了波导的对称性和线性，产生了强的全局非互易声学，其方式是根据谐波激励的应用侧而发生极其不同的声学响应。据作者所知，本发明的双门波导能够具有极高的声学非互易性，其水平远远高于当前文献中有源或无源器件所报道的水平;此外，在波传播的期望方向上，这种极端的性能与可接受水平的传输率相结合。机器学习被用于根据可传输性和非互易性的测量来预测设计该选通波导，目的是减少高维参数空间分析所需的计算时间。这项研究为这些介质的物理学提供了新的思路，并考虑了使用神经网络分析这类物理问题的优势和局限性。在预测的强非互易性理想参数空间中，最大传输率高达40%，并且从上游到下游的传输能量变化高达9个数量级，这取决于波传输的方向。机器学习工具以及本工作的数值方法可以为实际的非互易波导和声学超材料的预测设计提供信息，这些波导和声学超材料包含局部非线性门。本文表明，非线性门的组合可以导致极高的非互易性，同时保持所需的可传递性水平。摘要：This work is a study of acoustic non-reciprocity exhibited by a passive one-dimensional linear waveguide incorporating two local strongly nonlinear, asymmetric gates. Two local nonlinear gates break the symmetry and linearity of the waveguide, yielding strong global non-reciprocal acoustics, in the way that extremely different acoustical responses occur depending on the side of application of harmonic excitation. To the authors' best knowledge that the present two-gated waveguide is capable of extremely high acoustic non-reciprocity, at a much higher level to what is reported by active or passive devices in the current literature; moreover, this extreme performance combines with acceptable levels of transmissibility in the desired direction of wave propagation. Machine learning is utilized for predictive design of this gated waveguide in terms of the measures of transmissibility and non-reciprocity, with the aim of reducing the required computational time for high-dimensional parameter space analysis. The study sheds new light into the physics of these media and considers the advantages and limitations of using neural networks to analyze this type of physical problems. In the predicted desirable parameter space for intense non-reciprocity, the maximum transmissibility reaches as much as 40%, and the transmitted energy from upstream to downstream varies up to nine orders of magnitude, depending on the direction of wave transmission. The machine learning tools along with the numerical methods of this work can inform predictive designs of practical non-reciprocal waveguides and acoustic metamaterials that incorporate local nonlinear gates. The current paper shows that combinations of nonlinear gates can lead to extremely high non-reciprocity while maintaining desired levels of transmissibility.
【2】 Relating EEG to continuous speech using deep neural networks: a review
标题：利用深度神经网络将脑电与连续语音联系起来的研究进展
链接：https://arxiv.org/abs/2302.01736
*与cs.SD语音【4】为同一篇
作者：Corentin Puffay,Bernd Accou,Lies Bollens,Mohammad Jalilpour Monesi,Jonas Vanthornhout,Hugo Van hamme,Tom Francart机构：KU Leuven, Dept. Neurosciences, ExpORL, Leuven, Belgium, KU Leuven, Dept. of Electrical engineering (ESAT), PSI, Leuven, Belgium摘要：Objective. When a person listens to continuous speech, a corresponding response is elicited in the brain and can be recorded using electroencephalography (EEG). Linear models are presently used to relate the EEG recording to the corresponding speech signal. The ability of linear models to find a mapping between these two signals is used as a measure of neural tracking of speech. Such models are limited as they assume linearity in the EEG-speech relationship, which omits the nonlinear dynamics of the brain. As an alternative, deep learning models have recently been used to relate EEG to continuous speech, especially in auditory attention decoding (AAD) and single-speech-source paradigms. Approach. This paper reviews and comments on deep-learning-based studies that relate EEG to continuous speech in AAD and single-speech-source paradigms. We point out recurrent methodological pitfalls and the need for a standard benchmark of model analysis. Main results. We gathered 28 studies. The main methodological issues we found are biased cross-validations, data leakage leading to over-fitted models, or disproportionate data size compared to the model's complexity. In addition, we address requirements for a standard benchmark model analysis, such as public datasets, common evaluation metrics, and good practices for the match-mismatch task. Significance. We are the first to present a review paper summarizing the main deep-learning-based studies that relate EEG to speech while addressing methodological pitfalls and important considerations for this newly expanding field. Our study is particularly relevant given the growing application of deep learning in EEG-speech decoding.
【3】 Efficient Domain Adaptation for Speech Foundation Models
标题：一种高效的语音基础模型域自适应算法
链接：https://arxiv.org/abs/2302.01496
*与cs.SD语音【1】为同一篇
作者：Bo Li,Dongseong Hwang,Zhouyuan Huo,Junwen Bai,Guru Prakash,Tara N. Sainath,Khe Chai Sim,Yu Zhang,Wei Han,Trevor Strohman,Francoise Beaufays摘要：基础模型（FM）是在大规模的广泛数据上训练的，并可适应广泛的下游任务，已引起研究界的极大兴趣。基础模型得益于不同的数据源，如不同的模态、语言和应用领域，表现出了强大的泛化和知识转移能力。在本文中，我们提出了一个开创性的研究，为建立一个有效的解决方案，调频为基础的语音识别系统。我们采用最新发展的自监督BEST-RQ进行预训练，并提出了基于JUST Hydra的源域和无监督目标域数据的联合微调。FM编码器适配器和解码器然后被微调到具有少量受监督的域内数据的目标域。在大规模的YouTube和语音搜索任务中，我们的方法被证明是数据和模型参数有效的。与731.1M模型在额外的300 M监督域内数据上从头训练相比，它仅用21.6M监督域内数据和130.8M微调参数就实现了相同的质量。摘要：Foundation models (FMs), that are trained on broad data at scale and are adaptable to a wide range of downstream tasks, have brought large interest in the research community. Benefiting from the diverse data sources such as different modalities, languages and application domains, foundation models have demonstrated strong generalization and knowledge transfer capabilities. In this paper, we present a pioneering study towards building an efficient solution for FM-based speech recognition systems. We adopt the recently developed self-supervised BEST-RQ for pretraining, and propose the joint finetuning with both source and unsupervised target domain data using JUST Hydra. The FM encoder adapter and decoder are then finetuned to the target domain with a small amount of supervised in-domain data. On a large-scale YouTube and Voice Search task, our method is shown to be both data and model parameter efficient. It achieves the same quality with only 21.6M supervised in-domain data and 130.8M finetuned parameters, compared to the 731.1M model trained from scratch on additional 300M supervised in-domain data.
【4】 SPADE: Self-supervised Pretraining for Acoustic DisEntanglement
标题：Spade：用于声学解缠的自监督预训练
链接：https://arxiv.org/abs/2302.01483
*与cs.SD语音【2】为同一篇
作者：John Harvill,Jarred Barber,Arun Nair,Ramin Pishehvar机构：University of Illinois Urbana-Champaign, USA, Amazon Alexa AI, USA摘要：自监督表征学习方法由于能够在大量未标记数据上训练模型而变得越来越流行，并且已经在诸如自然语言处理、计算机视觉和语音等不同领域中证明了成功。以往在语音领域的自我监督工作已经解开了语音的多个属性，如语言内容、说话人身份和节奏。在这项工作中，我们介绍了一种自监督的方法来从语音中分离房间声学，并将声学表示用于设备仲裁的下游任务。我们的结果表明，当标记的训练数据稀少时，我们提出的方法显著地改善了基线上的性能，这表明我们的预训练方案学习编码房间声学信息，同时保持对语音信号的其他属性不变。摘要：Self-supervised representation learning approaches have grown in popularity due to the ability to train models on large amounts of unlabeled data and have demonstrated success in diverse fields such as natural language processing, computer vision, and speech. Previous self-supervised work in the speech domain has disentangled multiple attributes of speech such as linguistic content, speaker identity, and rhythm. In this work, we introduce a self-supervised approach to disentangle room acoustics from speech and use the acoustic representation on the downstream task of device arbitration. Our results demonstrate that our proposed approach significantly improves performance over a baseline when labeled training data is scarce, indicating that our pretraining scheme learns to encode room acoustic information while remaining invariant to other attributes of the speech signal.机器翻译，仅供参考
每日语音速递