今日论文合集：cs.SD语音5篇，eess.AS音频处理7篇。
cs.SD语音

【1】 PD-ADSV: An Automated Diagnosing System Using Voice Signals and Hard Voting Ensemble Method for Parkinson's Disease标题：PD-ADSV：一种基于语音信号和硬投票集成方法的帕金森病自动诊断系统链接：http://arxiv.org/pdf/2304.06016v1

作者：Paria Ghaheri,Ahmadreza Shateri,Hamid Nasiri机构：Electrical and Computer Engineering Department, Semnan University, Semnan, Iran, Department of Computer Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran.备注：This work has been submitted to the Software Impacts journal (Elsevier) for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：帕金森氏病（PD）是最普遍的运动病症和继阿尔茨海默氏病之后的第二最常见的神经退行性疾病。运动症状和成像技术是诊断这种疾病最常用的方法。然而，它们并不准确和快速，可能只有少数人可以访问。本研究提供了一种自主系统，即，PD-ADSV，用于基于语音信号的PD诊断，它使用四个机器学习分类器和硬投票集成方法来达到最高的准确率。PD-ADSV使用Python和Gradio Web框架开发。摘要：Parkinson's disease (PD) is the most widespread movement condition and the second most common neurodegenerative disorder, following Alzheimer's. Movement symptoms and imaging techniques are the most popular ways to diagnose this disease. However, they are not accurate and fast and may only be accessible to a few people. This study provides an autonomous system, i.e., PD-ADSV, for diagnosing PD based on voice signals, which uses four machine learning classifiers and the hard voting ensemble method to achieve the highest accuracy. PD-ADSV is developed using Python and the Gradio web framework.

【2】 A Phoneme-Informed Neural Network Model for Note-Level Singing Transcription标题：音符水平歌唱转录的音素信息神经网络模型链接：http://arxiv.org/pdf/2304.05917v1

作者：Sangeon Yong,Li Su,Juhan Nam机构：Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea, Institute of Information Science, Academia Sinica, Taipei, Taiwan备注：Accepted at ICASSP 2023摘要：音符级的自动音乐转录是最具代表性的音乐信息检索任务之一，并且已经被研究用于各种乐器理解音乐。然而，由于缺乏高质量的标记数据，许多仪器的转录仍然是一项具有挑战性的任务。特别是在歌唱的情况下，由于其在音高、音色和力度上的表现力，很难找到准确的音符。在本文中，我们提出了一种方法，找到注意到的开端，更准确地利用歌唱的语言特点，这是没有看到在其他仪器。该模型使用梅尔缩放的声谱图和语音后验图（PPG），音素的帧似然性，作为发病检测网络的输入，而PPG是由预先训练的网络与歌唱和语音数据生成的。为了验证语言特征如何影响起始点检测，我们通过不同语言的数据集比较评估结果，并划分起始点类型进行详细分析。我们的方法大大提高了歌唱转录的性能，因此强调了语言特征在歌唱分析中的重要性。摘要：Note-level automatic music transcription is one of the most representative music information retrieval (MIR) tasks and has been studied for various instruments to understand music. However, due to the lack of high-quality labeled data, transcription of many instruments is still a challenging task. In particular, in the case of singing, it is difficult to find accurate notes due to its expressiveness in pitch, timbre, and dynamics. In this paper, we propose a method of finding note onsets of singing voice more accurately by leveraging the linguistic characteristics of singing, which are not seen in other instruments. The proposed model uses mel-scaled spectrogram and phonetic posteriorgram (PPG), a frame-wise likelihood of phoneme, as an input of the onset detection network while PPG is generated by the pre-trained network with singing and speech data. To verify how linguistic features affect onset detection, we compare the evaluation results through the dataset with different languages and divide onset types for detailed analysis. Our approach substantially improves the performance of singing transcription and therefore emphasizes the importance of linguistic features in singing analysis.

【3】 Self-Supervised Learning with Cluster-Aware-DINO for High-Performance Robust Speaker Verification标题：基于簇感知Dino的自监督学习用于高性能稳健说话人确认链接：http://arxiv.org/pdf/2304.05754v1

作者：Bing Han,Zhengyang Chen,Yanmin Qian机构：Department of ComputerScience and Engineering & MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University备注：Submitted to TASLP in July 19, 2022摘要：自动说话人确认任务已经取得了很大的成就，使用深度学习方法与大规模手动标注的数据集。然而，收集大量标记良好的数据用于系统构建非常困难和昂贵。在本文中，我们提出了一种新的和先进的自监督学习框架，可以构建一个高性能的说话人确认系统，而不使用任何标记的数据。为了避免假负对的影响，我们采用无标签自蒸馏（DINO）框架作为初始模型，可以在不利用负对的情况下进行训练。然后，我们引入了一种簇感知的DINO训练策略来提高数据的多样性。在迭代学习阶段，由于聚类产生的大量不可靠标签，伪标签的质量对系统训练至关重要。这促使我们提出动态损失门和标签校正（DLG-LC）的方法，以减轻不可靠的标签所造成的性能下降。更具体地说，我们模型的损失分布与GMM和损失门阈值动态获得区分可靠和不可靠的标签。此外，我们采用模型预测来纠正不可靠的标签，更好地利用不可靠的数据，而不是直接丢弃它们。此外，我们将DLG-LC扩展到多模态，以进一步提高性能。实验是在常用的Voxceleb数据集上进行的。与最著名的自监督说话人确认系统相比，我们提出的方法在Vox-O，Vox-E和Vox-H测试集上获得了22.17%，27.94%和25.56%的相对EER改进，即使迭代次数更少，模型更小，聚类方法更简单。更重要的是，新提出的系统甚至实现了与完全监督系统相当的结果，但不使用任何人类标记的数据。摘要：Automatic speaker verification task has made great achievements using deep learning approaches with the large-scale manually annotated dataset. However, it's very difficult and expensive to collect a large amount of well-labeled data for system building. In this paper, we propose a novel and advanced self-supervised learning framework which can construct a high performance speaker verification system without using any labeled data. To avoid the impact of false negative pairs, we adopt the self-distillation with no labels (DINO) framework as the initial model, which can be trained without exploiting negative pairs. Then, we introduce a cluster-aware training strategy for DINO to improve the diversity of data. In the iteration learning stage, due to a mass of unreliable labels from clustering, the quality of pseudo labels is important for the system training. This motivates us to propose dynamic loss-gate and label correction (DLG-LC) methods to alleviate the performance degradation caused by unreliable labels. More specifically, we model the loss distribution with GMM and obtain the loss-gate threshold dynamically to distinguish the reliable and unreliable labels. Besides, we adopt the model predictions to correct the unreliable label, for better utilizing the unreliable data rather than dropping them directly. Moreover, we extend the DLG-LC to multi-modality to further improve the performance. The experiments are performed on the commonly used Voxceleb dataset. Compared to the best-known self-supervised speaker verification system, our proposed method obtain 22.17%, 27.94% and 25.56% relative EER improvement on Vox-O, Vox-E and Vox-H test sets, even with fewer iterations, smaller models, and simpler clustering methods. More importantly, the newly proposed system even achieves comparable results with the fully supervised system, but without using any human labeled data.

【4】 Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning标题：看起来相似，听起来不同：利用反事实的跨通道对进行视听表征学习链接：http://arxiv.org/pdf/2304.05600v1

作者：Nikhil Singh,Chih-Wei Wu,Iroro Orife,Mahdi Kalayeh机构：Massachusetts Institute of Technology, Netflix备注：17 pages, 5 figures摘要：视听表征学习通常依赖于视觉和声音之间的对应关系。然而，通常存在可以与视觉场景相对应的多个音轨。例如，考虑在同一条拥挤的街道上的不同对话。这种反事实对视听表征学习的影响以前没有被探索过。为了研究这一点，我们使用配音版本的电影，以增强跨模态对比学习。我们的方法学习代表交替的音轨，不同的语音内容，类似于相同的视频。我们的研究结果表明，配音增强训练提高了一系列听觉和视听任务的性能，而不会显着影响语言任务的整体性能。我们还将这种方法与预训练前删除语音的强基线进行了比较，发现配音增强训练更有效，包括副语言和视听任务，其中语音删除导致性能下降。这些研究结果强调了在学习场景级视听对应时考虑语音变化的重要性，并表明配音音频可以是一种有用的增强技术，用于训练视听模型以获得更强大的性能。摘要：Audiovisual representation learning typically relies on the correspondence between sight and sound. However, there are often multiple audio tracks that can correspond with a visual scene. Consider, for example, different conversations on the same crowded street. The effect of such counterfactual pairs on audiovisual representation learning has not been previously explored. To investigate this, we use dubbed versions of movies to augment cross-modal contrastive learning. Our approach learns to represent alternate audio tracks, differing only in speech content, similarly to the same video. Our results show that dub-augmented training improves performance on a range of auditory and audiovisual tasks, without significantly affecting linguistic task performance overall. We additionally compare this approach to a strong baseline where we remove speech before pretraining, and find that dub-augmented training is more effective, including for paralinguistic and audiovisual tasks where speech removal leads to worse performance. These findings highlight the importance of considering speech variation when learning scene-level audiovisual correspondences and suggest that dubbed audio can be a useful augmentation technique for training audiovisual models toward more robust performance.

【5】 Filler Word Detection with Hard Category Mining and Inter-Category Focal Loss标题：基于硬类别挖掘和类别间焦点损失的填充词检测链接：http://arxiv.org/pdf/2304.05922v1

作者：Zhiyuan Zhao,Lijun Wu,Chuanxin Tang,Dacheng Yin,Yucheng Zhao,Chong Luo机构：Microsoft Research Asia, Beijing, China, University of Science and Technology of China, Hefei, China备注：accepted by ICASSP23摘要：像“um”或“uh”这样的填充词在自发言语中很常见。期望在记录中自动检测和移除它们，因为它们影响语音的流畅性、自信度和专业性。先前的研究和我们的初步实验表明，在填充词检测的最大挑战是，填充词可以很容易地与其他硬类别，如“a”或“I”混淆。在本文中，我们提出了一种新的填充词检测方法，有效地解决了这一挑战，通过动态添加辅助类别和应用额外的类别间焦点损失。辅助类别迫使模型通过挖掘硬类别来显式地对易混淆的词进行建模。此外，类别间焦点损失自适应地调整“填充”和"非填充”类别之间的惩罚权重，以处理"非填充”类别中的其他易混淆单词。在PodcastFillers数据集上，与其他方法相比，我们的系统取得了最好的结果，有了巨大的改进。摘要：Filler words like um" or uh" are common in spontaneous speech. It is desirable to automatically detect and remove them in recordings, as they affect the fluency, confidence, and professionalism of speech. Previous studies and our preliminary experiments reveal that the biggest challenge in filler word detection is that fillers can be easily confused with other hard categories like a" or I". In this paper, we propose a novel filler word detection method that effectively addresses this challenge by adding auxiliary categories dynamically and applying an additional inter-category focal loss. The auxiliary categories force the model to explicitly model the confusing words by mining hard categories. In addition, inter-category focal loss adaptively adjusts the penalty weight between filler" and non-filler" categories to deal with other confusing words left in the non-filler" category. Our system achieves the best results, with a huge improvement compared to other methods on the PodcastFillers dataset.
eess.AS音频处理

【1】 Regularizing Contrastive Predictive Coding for Speech Applications标题：用于语音应用的正则化对比预测编码链接：http://arxiv.org/pdf/2304.05974v1

作者：Saurabhchand Bhati,Jesús Villalba,Piotr Żelasko,Laureano Moro-Velazquez,Najim Dehak机构：†Center for Language and Speech Processing, Johns Hopkins University, USA, ‡Human Language Technology Center of Excellence, Johns Hopkins University, USA, ∗Meaning.Team Inc, USA摘要：自监督方法，如对比预测编码（CPC），大大提高了质量的无监督表示。这些表示显著减少了下游任务性能（例如自动语音识别）所需的标记数据的量。CPC通过学习在给定当前帧的情况下预测未来帧来学习表示。基于声学信息（例如，电话，变化慢于特征提取率在CPC，我们提出了正则化技术，施加慢约束的功能。在这里，我们提出了两种正则化技术：自表达约束和左或右正则化。我们评估所提出的模型ABX和线性电话分类任务，声学单元发现，和自动语音识别。在100小时的未标记数据上训练的正则化CPC与在360小时的未标记数据上训练的基线CPC的性能相匹配。我们还表明，我们的正则化技术是互补的数据增强，可以进一步提高系统的性能。在单语言、跨语言或多语言设置中，无论是否使用数据扩充，无论用于训练的数据量如何，我们的正则化模型都优于基线CPC模型。摘要：Self-supervised methods such as Contrastive predictive Coding (CPC) have greatly improved the quality of the unsupervised representations. These representations significantly reduce the amount of labeled data needed for downstream task performance, such as automatic speech recognition. CPC learns representations by learning to predict future frames given current frames. Based on the observation that the acoustic information, e.g., phones, changes slower than the feature extraction rate in CPC, we propose regularization techniques that impose slowness constraints on the features. Here we propose two regularization techniques: Self-expressing constraint and Left-or-right regularization. We evaluate the proposed model on ABX and linear phone classification tasks, acoustic unit discovery, and automatic speech recognition. The regularized CPC trained on 100 hours of unlabeled data matches the performance of the baseline CPC trained on 360 hours of unlabeled data. We also show that our regularization techniques are complementary to data augmentation and can further boost the system's performance. In monolingual, cross-lingual, or multilingual settings, with without data augmentation, regardless of the amount of data used for training, our regularized models outperformed the baseline CPC models.

【2】 Filler Word Detection with Hard Category Mining and Inter-Category Focal Loss标题：基于硬类别挖掘和类别间焦点损失的填充词检测链接：http://arxiv.org/pdf/2304.05922v1

*与cs.SD语音【5】为同一篇

作者：Zhiyuan Zhao,Lijun Wu,Chuanxin Tang,Dacheng Yin,Yucheng Zhao,Chong Luo机构：Microsoft Research Asia, Beijing, China, University of Science and Technology of China, Hefei, China备注：accepted by ICASSP23摘要：像“um”或“uh”这样的填充词在自发言语中很常见。期望在记录中自动检测和移除它们，因为它们影响语音的流畅性、自信度和专业性。先前的研究和我们的初步实验表明，在填充词检测的最大挑战是，填充词可以很容易地与其他硬类别，如“a”或“I”混淆。在本文中，我们提出了一种新的填充词检测方法，有效地解决了这一挑战，通过动态添加辅助类别和应用额外的类别间焦点损失。辅助类别迫使模型通过挖掘硬类别来显式地对易混淆的词进行建模。此外，类别间焦点损失自适应地调整“填充”和"非填充”类别之间的惩罚权重，以处理"非填充”类别中的其他易混淆单词。在PodcastFillers数据集上，与其他方法相比，我们的系统取得了最好的结果，有了巨大的改进。摘要：Filler words like um" or uh" are common in spontaneous speech. It is desirable to automatically detect and remove them in recordings, as they affect the fluency, confidence, and professionalism of speech. Previous studies and our preliminary experiments reveal that the biggest challenge in filler word detection is that fillers can be easily confused with other hard categories like a" or I". In this paper, we propose a novel filler word detection method that effectively addresses this challenge by adding auxiliary categories dynamically and applying an additional inter-category focal loss. The auxiliary categories force the model to explicitly model the confusing words by mining hard categories. In addition, inter-category focal loss adaptively adjusts the penalty weight between filler" and non-filler" categories to deal with other confusing words left in the non-filler" category. Our system achieves the best results, with a huge improvement compared to other methods on the PodcastFillers dataset.

【3】 Speech Reconstruction from Silent Tongue and Lip Articulation By Pseudo Target Generation and Domain Adversarial Training标题：基于伪靶生成和领域对抗性训练的无声和唇音语音重建链接：http://arxiv.org/pdf/2304.05574v1

作者：Rui-Chen Zheng,Yang Ai,Zhen-Hua Ling机构：National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, P. R. China备注：To be published in ICASSP2023摘要：None摘要：This paper studies the task of speech reconstruction from ultrasound tongue images and optical lip videos recorded in a silent speaking mode, where people only activate their intra-oral and extra-oral articulators without producing sound. This task falls under the umbrella of articulatory-to-acoustic conversion, and may also be refered to as a silent speech interface. We propose to employ a method built on pseudo target generation and domain adversarial training with an iterative training strategy to improve the intelligibility and naturalness of the speech recovered from silent tongue and lip articulation. Experiments show that our proposed method significantly improves the intelligibility and naturalness of the reconstructed speech in silent speaking mode compared to the baseline TaLNet model. When using an automatic speech recognition (ASR) model to measure intelligibility, the word error rate (WER) of our proposed method decreases by over 15% compared to the baseline. In addition, our proposed method also outperforms the baseline on the intelligibility of the speech reconstructed in vocalized articulating mode, reducing the WER by approximately 10%.

【4】 PD-ADSV: An Automated Diagnosing System Using Voice Signals and Hard Voting Ensemble Method for Parkinson's Disease标题：PD-ADSV：一种基于语音信号和硬投票集成方法的帕金森病自动诊断系统链接：http://arxiv.org/pdf/2304.06016v1

*与cs.SD语音【1】为同一篇

作者：Paria Ghaheri,Ahmadreza Shateri,Hamid Nasiri机构：Electrical and Computer Engineering Department, Semnan University, Semnan, Iran, Department of Computer Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran.备注：This work has been submitted to the Software Impacts journal (Elsevier) for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：帕金森氏病（PD）是最普遍的运动病症和继阿尔茨海默氏病之后的第二最常见的神经退行性疾病。运动症状和成像技术是诊断这种疾病最常用的方法。然而，它们并不准确和快速，可能只有少数人可以访问。本研究提供了一种自主系统，即，PD-ADSV，用于基于语音信号的PD诊断，它使用四个机器学习分类器和硬投票集成方法来达到最高的准确率。PD-ADSV使用Python和Gradio Web框架开发。摘要：Parkinson's disease (PD) is the most widespread movement condition and the second most common neurodegenerative disorder, following Alzheimer's. Movement symptoms and imaging techniques are the most popular ways to diagnose this disease. However, they are not accurate and fast and may only be accessible to a few people. This study provides an autonomous system, i.e., PD-ADSV, for diagnosing PD based on voice signals, which uses four machine learning classifiers and the hard voting ensemble method to achieve the highest accuracy. PD-ADSV is developed using Python and the Gradio web framework.

【5】 A Phoneme-Informed Neural Network Model for Note-Level Singing Transcription标题：音符水平歌唱转录的音素信息神经网络模型链接：http://arxiv.org/pdf/2304.05917v1

*与cs.SD语音【2】为同一篇

作者：Sangeon Yong,Li Su,Juhan Nam机构：Graduate School of Culture Technology, KAIST, Daejeon, Republic of Korea, Institute of Information Science, Academia Sinica, Taipei, Taiwan备注：Accepted at ICASSP 2023摘要：音符级的自动音乐转录是最具代表性的音乐信息检索任务之一，并且已经被研究用于各种乐器理解音乐。然而，由于缺乏高质量的标记数据，许多仪器的转录仍然是一项具有挑战性的任务。特别是在歌唱的情况下，由于其在音高、音色和力度上的表现力，很难找到准确的音符。在本文中，我们提出了一种方法，找到注意到的开端，更准确地利用歌唱的语言特点，这是没有看到在其他仪器。该模型使用梅尔缩放的声谱图和语音后验图（PPG），音素的帧似然性，作为发病检测网络的输入，而PPG是由预先训练的网络与歌唱和语音数据生成的。为了验证语言特征如何影响起始点检测，我们通过不同语言的数据集比较评估结果，并划分起始点类型进行详细分析。我们的方法大大提高了歌唱转录的性能，因此强调了语言特征在歌唱分析中的重要性。摘要：Note-level automatic music transcription is one of the most representative music information retrieval (MIR) tasks and has been studied for various instruments to understand music. However, due to the lack of high-quality labeled data, transcription of many instruments is still a challenging task. In particular, in the case of singing, it is difficult to find accurate notes due to its expressiveness in pitch, timbre, and dynamics. In this paper, we propose a method of finding note onsets of singing voice more accurately by leveraging the linguistic characteristics of singing, which are not seen in other instruments. The proposed model uses mel-scaled spectrogram and phonetic posteriorgram (PPG), a frame-wise likelihood of phoneme, as an input of the onset detection network while PPG is generated by the pre-trained network with singing and speech data. To verify how linguistic features affect onset detection, we compare the evaluation results through the dataset with different languages and divide onset types for detailed analysis. Our approach substantially improves the performance of singing transcription and therefore emphasizes the importance of linguistic features in singing analysis.

【6】 Self-Supervised Learning with Cluster-Aware-DINO for High-Performance Robust Speaker Verification标题：基于簇感知Dino的自监督学习用于高性能稳健说话人确认链接：http://arxiv.org/pdf/2304.05754v1

*与cs.SD语音【3】为同一篇

作者：Bing Han,Zhengyang Chen,Yanmin Qian机构：Department of ComputerScience and Engineering & MoE Key Laboratory of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University备注：Submitted to TASLP in July 19, 2022摘要：自动说话人确认任务已经取得了很大的成就，使用深度学习方法与大规模手动标注的数据集。然而，收集大量标记良好的数据用于系统构建非常困难和昂贵。在本文中，我们提出了一种新的和先进的自监督学习框架，可以构建一个高性能的说话人确认系统，而不使用任何标记的数据。为了避免假负对的影响，我们采用无标签自蒸馏（DINO）框架作为初始模型，可以在不利用负对的情况下进行训练。然后，我们引入了一种簇感知的DINO训练策略来提高数据的多样性。在迭代学习阶段，由于聚类产生的大量不可靠标签，伪标签的质量对系统训练至关重要。这促使我们提出动态损失门和标签校正（DLG-LC）的方法，以减轻不可靠的标签所造成的性能下降。更具体地说，我们模型的损失分布与GMM和损失门阈值动态获得区分可靠和不可靠的标签。此外，我们采用模型预测来纠正不可靠的标签，更好地利用不可靠的数据，而不是直接丢弃它们。此外，我们将DLG-LC扩展到多模态，以进一步提高性能。实验是在常用的Voxceleb数据集上进行的。与最著名的自监督说话人确认系统相比，我们提出的方法在Vox-O，Vox-E和Vox-H测试集上获得了22.17%，27.94%和25.56%的相对EER改进，即使迭代次数更少，模型更小，聚类方法更简单。更重要的是，新提出的系统甚至实现了与完全监督系统相当的结果，但不使用任何人类标记的数据。摘要：Automatic speaker verification task has made great achievements using deep learning approaches with the large-scale manually annotated dataset. However, it's very difficult and expensive to collect a large amount of well-labeled data for system building. In this paper, we propose a novel and advanced self-supervised learning framework which can construct a high performance speaker verification system without using any labeled data. To avoid the impact of false negative pairs, we adopt the self-distillation with no labels (DINO) framework as the initial model, which can be trained without exploiting negative pairs. Then, we introduce a cluster-aware training strategy for DINO to improve the diversity of data. In the iteration learning stage, due to a mass of unreliable labels from clustering, the quality of pseudo labels is important for the system training. This motivates us to propose dynamic loss-gate and label correction (DLG-LC) methods to alleviate the performance degradation caused by unreliable labels. More specifically, we model the loss distribution with GMM and obtain the loss-gate threshold dynamically to distinguish the reliable and unreliable labels. Besides, we adopt the model predictions to correct the unreliable label, for better utilizing the unreliable data rather than dropping them directly. Moreover, we extend the DLG-LC to multi-modality to further improve the performance. The experiments are performed on the commonly used Voxceleb dataset. Compared to the best-known self-supervised speaker verification system, our proposed method obtain 22.17%, 27.94% and 25.56% relative EER improvement on Vox-O, Vox-E and Vox-H test sets, even with fewer iterations, smaller models, and simpler clustering methods. More importantly, the newly proposed system even achieves comparable results with the fully supervised system, but without using any human labeled data.

【7】 Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning标题：看起来相似，听起来不同：利用反事实的跨通道对进行视听表征学习链接：http://arxiv.org/pdf/2304.05600v1

*与cs.SD语音【4】为同一篇

作者：Nikhil Singh,Chih-Wei Wu,Iroro Orife,Mahdi Kalayeh机构：Massachusetts Institute of Technology, Netflix备注：17 pages, 5 figures摘要：视听表征学习通常依赖于视觉和声音之间的对应关系。然而，通常存在可以与视觉场景相对应的多个音轨。例如，考虑在同一条拥挤的街道上的不同对话。这种反事实对视听表征学习的影响以前没有被探索过。为了研究这一点，我们使用配音版本的电影，以增强跨模态对比学习。我们的方法学习代表交替的音轨，不同的语音内容，类似于相同的视频。我们的研究结果表明，配音增强训练提高了一系列听觉和视听任务的性能，而不会显着影响语言任务的整体性能。我们还将这种方法与预训练前删除语音的强基线进行了比较，发现配音增强训练更有效，包括副语言和视听任务，其中语音删除导致性能下降。这些研究结果强调了在学习场景级视听对应时考虑语音变化的重要性，并表明配音音频可以是一种有用的增强技术，用于训练视听模型以获得更强大的性能。摘要：Audiovisual representation learning typically relies on the correspondence between sight and sound. However, there are often multiple audio tracks that can correspond with a visual scene. Consider, for example, different conversations on the same crowded street. The effect of such counterfactual pairs on audiovisual representation learning has not been previously explored. To investigate this, we use dubbed versions of movies to augment cross-modal contrastive learning. Our approach learns to represent alternate audio tracks, differing only in speech content, similarly to the same video. Our results show that dub-augmented training improves performance on a range of auditory and audiovisual tasks, without significantly affecting linguistic task performance overall. We additionally compare this approach to a strong baseline where we remove speech before pretraining, and find that dub-augmented training is more effective, including for paralinguistic and audiovisual tasks where speech removal leads to worse performance. These findings highlight the importance of considering speech variation when learning scene-level audiovisual correspondences and suggest that dubbed audio can be a useful augmentation technique for training audiovisual models toward more robust performance.

机器翻译，仅供参考