今日论文合集：cs.SD语音5篇，eess.AS音频处理6篇。
cs.SD语音

【1】 Dual-Quaternions: Theory and Applications in Sound标题：对偶四元数：理论及其在声音中的应用链接：http://arxiv.org/pdf/2303.12692v1

作者：Benjamin Kenwright摘要：声音是基本的、丰富的信息来源;在从人文和社会科学到工程和数学的许多领域发挥着关键作用。声音不仅仅是数据信号。它包含了身体、感官和情感，以及社会、文化和环境因素。声音有助于我们的经验、环境和信念的转变。声音就在我们身边，无处不在。因此，声音是一个复杂的多成分实体，具有各种各样的特性和应用，这一点也不奇怪。当然，一个重要的问题是，什么是数字化存储和表示声音以捕捉这些特征的最佳方法？哪种模型或方法最适合处理、提取和过滤声音？有大量的表示和模型，然而，一种尚未用于声音的方法是双四元数。虽然双四元数已经在许多科学和计算领域中确立了自身作为用于提供表示多分量数据的明确的、不麻烦的、计算上有效的手段的有效数学模型。声音是一个尚未探索和收获双四元数（使用声音和音频相关的双四元数模型）的好处的领域。本文旨在探索双四元数在与基于声音的模型（包括但不限于应用程序，工具，机器学习，统计和计算声音相关算法）相结合时所提供的令人兴奋的潜力和可能性。摘要：Sound is a fundamental and rich source of information; playing a key role in many areas from humanities and social sciences through to engineering and mathematics. Sound is more than just data 'signals'. It encapsulates physical, sensorial and emotional, as well as social, cultural and environmental factors. Sound contributes to the transformation of our experiences, environments and beliefs. Sound is all around us and everywhere. Hence, it should come as no surprise that sound is a complex multicomponent entity with a vast assortment of characteristics and applications. Of course, an important question is, what is the best way to store and represent sound digitally to capture these characteristics? What model or method is best for manipulating, extracting and filtering sounds? There are a large number of representations and models, however, one approach that has yet to be used with sound is dual-quaternions. While dual-quaternions have established themselves in many fields of science and computing as an efficient mathematical model for providing an unambiguous, un-cumbersome, computationally effective means of representing multi-component data. Sound is one area that has yet to explore and reap the benefits of dual-quaternions (using sound and audio-related dual-quaternion models). This article aims to explore the exciting potential and possibilities dual-quaternions offer when applied and combined with sound-based models (including but not limited to the applications, tools, machine-learning, statistical and computational sound-related algorithms).

【2】 Posthoc Interpretation via Quantization标题：量化后解释链接：http://arxiv.org/pdf/2303.12659v1

作者：Cem Subakan,Francesco Paissan,Mirco Ravanelli机构：Université Laval,Concordia University,Mila, Québec AI Institute,University of Trento备注：Equal contribution摘要：在本文中，我们介绍了一种新的方法，称为“事后解释通过量化（PIQ）”，用于解释由训练的分类器作出的决定。我们的方法利用矢量量化的分类器的表示转换成一个离散的，类特定的潜在空间。特定于类的码本充当瓶颈，其迫使解释器集中于分类器认为与做出预测相关的输入数据的部分。我们通过定量和定性研究评估了我们的方法，发现与文献中的其他几种解释方法相比，PIQ生成的解释更容易被参与者理解。摘要：In this paper, we introduce a new approach, called "Posthoc Interpretation via Quantization (PIQ)", for interpreting decisions made by trained classifiers. Our method utilizes vector quantization to transform the representations of a classifier into a discrete, class-specific latent space. The class-specific codebooks act as a bottleneck that forces the interpreter to focus on the parts of the input data deemed relevant by the classifier for making a prediction. We evaluated our method through quantitative and qualitative studies and found that PIQ generates interpretations that are more easily understood by participants to our user studies when compared to several other interpretation methods in the literature.

【3】 Music-Driven Group Choreography标题：音乐驱动的集体编舞链接：http://arxiv.org/pdf/2303.12337v1

作者：Nhat Le,Thang Pham,Tuong Do,Erman Tjiputra,Quang D. Tran,Anh Nguyen机构：AIOZ, Singapore ,University of Liverpool, UK备注：accepted in cvpr 2023摘要：音乐驱动的编排是一个具有挑战性的问题，具有广泛的工业应用。近来，已经提出了许多方法来从单个舞蹈者的音乐合成舞蹈动作。然而，为一组生成舞蹈动作仍然是一个悬而未决的问题。在本文中，我们提出了$ rm AIOZ-GDANCE$，一个新的大型数据集的音乐驱动的群舞生成。与现有的数据集，只支持单一的舞蹈，我们的新数据集包含集体舞蹈视频，从而支持集体舞蹈的研究。我们提出了一种半自主的标签方法，人类在循环中获得我们的数据集的3D地面真相。建议的数据集包括16.7 $小时的配对音乐和3D运动，从在野外的视频，涵盖$7$舞蹈风格和$16$音乐流派。我们发现单纯地应用单一舞蹈产生技术来产生群舞动作可能会导致不理想的结果，例如动作不一致和舞者之间的碰撞。基于我们的新数据集，我们提出了一种新的方法，需要一个输入的音乐序列和一组舞者的3D位置，以有效地产生多组连贯的编舞。我们提出了新的评估指标，测量集体舞的质量和密集的实验，以证明我们的方法的有效性。摘要：Music-driven choreography is a challenging problem with a wide variety of industrial applications. Recently, many methods have been proposed to synthesize dance motions from music for a single dancer. However, generating dance motion for a group remains an open problem. In this paper, we present $ rm AIOZ-GDANCE$, a new large-scale dataset for music-driven group dance generation. Unlike existing datasets that only support single dance, our new dataset contains group dance videos, hence supporting the study of group choreography. We propose a semi-autonomous labeling method with humans in the loop to obtain the 3D ground truth for our dataset. The proposed dataset consists of $16.7$ hours of paired music and 3D motion from in-the-wild videos, covering $7$ dance styles and $16$ music genres. We show that naively applying single dance generation technique to creating group dance motion may lead to unsatisfactory results, such as inconsistent movements and collisions between dancers. Based on our new dataset, we propose a new method that takes an input music sequence and a set of 3D positions of dancers to efficiently produce multiple group-coherent choreographies. We propose new evaluation metrics for measuring group dance quality and perform intensive experiments to demonstrate the effectiveness of our method.

【4】 Exploring Turkish Speech Recognition via Hybrid CTCAttention Architecture and Multi-feature Fusion Network标题：基于混合CTC注意结构和多特征融合网络的土耳其语语音识别研究链接：http://arxiv.org/pdf/2303.12300v1

作者：Zeyu Ren,Nurmement Yolwas,Huiru Wang,Wushour Slamu机构：†Xinjiang Multilingual Information Technology Laboratory, College of Information Science and Engineering, Xinjiang University, Urumqi, China, ‡Xinjiang Key Laboratory of Signal Detection, College of Information Science and Engineering摘要：近年来，基于深度学习的端到端语音识别技术发展迅速。由于缺乏土耳其语语音数据，土耳其语语音识别系统的性能较差。本文首先研究了一系列语音识别调优技术。结果表明，采用速度扰动与加噪相结合的数据增强技术，波束搜索宽度设为16。其次，为了最大限度地利用有效特征信息，提高特征提取的准确性，本文提出了一种新的特征提取器LSPC。LSPC和LiGRU网络相结合，形成共享编码器结构，实现模型压缩。实验结果表明，在仅使用Fbank特征时，LSPC的性能优于MSPC和VGGnet，WER分别提高了1.01%和2.53%。最后，基于以上两点，提出了一种新的多特征融合网络作为编码器的主要结构。实验结果表明，基于LSPC的特征融合网络的WER比基于LSPC的单一特征（Fbank特征和Spectrogram特征）提取分别提高了0.82%和1.94%。我们的模型实现的性能与先进的端到端模型。摘要：In recent years, End-to-End speech recognition technology based on deep learning has developed rapidly. Due to the lack of Turkish speech data, the performance of Turkish speech recognition system is poor. Firstly, this paper studies a series of speech recognition tuning technologies. The results show that the performance of the model is the best when the data enhancement technology combining speed perturbation with noise addition is adopted and the beam search width is set to 16. Secondly, to maximize the use of effective feature information and improve the accuracy of feature extraction, this paper proposes a new feature extractor LSPC. LSPC and LiGRU network are combined to form a shared encoder structure, and model compression is realized. The results show that the performance of LSPC is better than MSPC and VGGnet when only using Fbank features, and the WER is improved by 1.01% and 2.53% respectively. Finally, based on the above two points, a new multi-feature fusion network is proposed as the main structure of the encoder. The results show that the WER of the proposed feature fusion network based on LSPC is improved by 0.82% and 1.94% again compared with the single feature (Fbank feature and Spectrogram feature) extraction using LSPC. Our model achieves performance comparable to that of advanced End-to-End models.

【5】 Practice of the conformer enhanced AUDIO-VISUAL HUBERT on Mandarin and English标题：形象化增强视听休BERT在普通话和英语中的实践链接：http://arxiv.org/pdf/2303.12187v1

作者：Xiaoming Ren,Chao Li,Shenjian Wang,Biao Li机构：Beijing OPPO telecommunications corp., ltd., Beijing, China摘要：考虑到人类语音感知的双峰性质，嘴唇和牙齿运动在自动语音识别中具有关键作用。得益于相关的和噪声不变的视觉信息，视听识别系统增强了在多种场景下的鲁棒性。在以前的工作中，视听HuBERT似乎是最好的做法，结合模态知识。本文概述了一种混合的方法，命名为构象增强的AV-HuBERT，进一步提高了AV-HuBERT系统的性能。与基线AV-HuBERT相比，我们的方法在清洁和嘈杂的条件下的一个阶段的评估实现了7%和16%的相对WER减少英语AVSR基准数据集LRS 3。此外，我们建立了一个新的1000小时普通话AVSR数据集CSTS。在基线AV-HuBERT的基础上，通过使用该数据集进行预训练，我们在MISP和CMLR方面分别超过WeNet ASR系统14%和18%。与基准AV-HuBERT系统相比，我们提出的一致性增强的AV-HuBERT在MISP上带来7%的减少，在CMLR上带来6%的CER减少。摘要：Considering the bimodal nature of human speech perception, lips, and teeth movement has a pivotal role in automatic speech recognition. Benefiting from the correlated and noise-invariant visual information, audio-visual recognition systems enhance robustness in multiple scenarios. In previous work, audio-visual HuBERT appears to be the finest practice incorporating modality knowledge. This paper outlines a mixed methodology, named conformer enhanced AV-HuBERT, boosting the AV-HuBERT system's performance a step further. Compared with baseline AV-HuBERT, our method in the one-phase evaluation of clean and noisy conditions achieves 7% and 16% relative WER reduction on the English AVSR benchmark dataset LRS3. Furthermore, we establish a novel 1000h Mandarin AVSR dataset CSTS. On top of the baseline AV-HuBERT, we exceed the WeNet ASR system by 14% and 18% relatively on MISP and CMLR by pre-training with this dataset. The conformer-enhanced AV-HuBERT we proposed brings 7% on MISP and 6% CER reduction on CMLR, compared with the baseline AV-HuBERT system.
eess.AS音频处理

【1】 Self-Supervised Representations for Singing Voice Conversion标题：用于歌唱声音转换的自监督表示法链接：http://arxiv.org/pdf/2303.12197v1

作者：Tejas Jayashankar,Jilong Wu,Leda Sari,David Kant,Vimal Manohar,Qing He机构：Massachusetts Institute of Technology, Meta AI摘要：演唱声音转换模型将任意源演唱者的声音中的歌曲转换成目标演唱者的声音。最近，利用自监督音频表示的方法，如HuBERT和Wav 2 Vec 2.0，有助于进一步发展最先进的技术。虽然这些方法产生更自然和旋律的歌唱输出，但它们通常依赖于混淆和解缠损失来呈现自监督表示扬声器和音高不变。在本文中，我们绕过了解纠缠训练，并提出了一种新的模型，该模型利用ASR微调的自监督表示作为HiFi-GAN神经声码器的输入，用于歌唱语音转换。我们对不同的f0编码方案进行了实验，结果表明，使用并行转置卷积库（PBTC）和ASR微调Wav 2 Vec 2.0特性的f0谐波生成模块可实现最佳的歌声转换质量。此外，该模型能够使语音唱歌。我们还表明，一个简单的F0移位方案在推理过程中有助于保留歌手的身份，并支持我们的歌声转换模型的性能。我们的结果得到了广泛的MOS研究的支持，这些研究比较了不同的消融和基线。摘要：A singing voice conversion model converts a song in the voice of an arbitrary source singer to the voice of a target singer. Recently, methods that leverage self-supervised audio representations such as HuBERT and Wav2Vec 2.0 have helped further the state-of-the-art. Though these methods produce more natural and melodic singing outputs, they often rely on confusion and disentanglement losses to render the self-supervised representations speaker and pitch-invariant. In this paper, we circumvent disentanglement training and propose a new model that leverages ASR fine-tuned self-supervised representations as inputs to a HiFi-GAN neural vocoder for singing voice conversion. We experiment with different f0 encoding schemes and show that an f0 harmonic generation module that uses a parallel bank of transposed convolutions (PBTC) alongside ASR fine-tuned Wav2Vec 2.0 features results in the best singing voice conversion quality. Additionally, the model is capable of making a spoken voice sing. We also show that a simple f0 shifting scheme during inference helps retain singer identity and bolsters the performance of our singing voice conversion model. Our results are backed up by extensive MOS studies that compare different ablations and baselines.

【2】 Practice of the conformer enhanced AUDIO-VISUAL HUBERT on Mandarin and English标题：形象化增强视听休BERT在普通话和英语中的实践链接：http://arxiv.org/pdf/2303.12187v1

*与cs.SD语音【5】为同一篇

作者：Xiaoming Ren,Chao Li,Shenjian Wang,Biao Li机构：Beijing OPPO telecommunications corp., ltd., Beijing, China摘要：考虑到人类语音感知的双峰性质，嘴唇和牙齿运动在自动语音识别中具有关键作用。得益于相关的和噪声不变的视觉信息，视听识别系统增强了在多种场景下的鲁棒性。在以前的工作中，视听HuBERT似乎是最好的做法，结合模态知识。本文概述了一种混合的方法，命名为构象增强的AV-HuBERT，进一步提高了AV-HuBERT系统的性能。与基线AV-HuBERT相比，我们的方法在清洁和嘈杂的条件下的一个阶段的评估实现了7%和16%的相对WER减少英语AVSR基准数据集LRS 3。此外，我们建立了一个新的1000小时普通话AVSR数据集CSTS。在基线AV-HuBERT的基础上，通过使用该数据集进行预训练，我们在MISP和CMLR方面分别超过WeNet ASR系统14%和18%。与基准AV-HuBERT系统相比，我们提出的一致性增强的AV-HuBERT在MISP上带来7%的减少，在CMLR上带来6%的CER减少。摘要：Considering the bimodal nature of human speech perception, lips, and teeth movement has a pivotal role in automatic speech recognition. Benefiting from the correlated and noise-invariant visual information, audio-visual recognition systems enhance robustness in multiple scenarios. In previous work, audio-visual HuBERT appears to be the finest practice incorporating modality knowledge. This paper outlines a mixed methodology, named conformer enhanced AV-HuBERT, boosting the AV-HuBERT system's performance a step further. Compared with baseline AV-HuBERT, our method in the one-phase evaluation of clean and noisy conditions achieves 7% and 16% relative WER reduction on the English AVSR benchmark dataset LRS3. Furthermore, we establish a novel 1000h Mandarin AVSR dataset CSTS. On top of the baseline AV-HuBERT, we exceed the WeNet ASR system by 14% and 18% relatively on MISP and CMLR by pre-training with this dataset. The conformer-enhanced AV-HuBERT we proposed brings 7% on MISP and 6% CER reduction on CMLR, compared with the baseline AV-HuBERT system.

【3】 Dual-Quaternions: Theory and Applications in Sound标题：对偶四元数：理论及其在声音中的应用链接：http://arxiv.org/pdf/2303.12692v1

*与cs.SD语音【1】为同一篇

作者：Benjamin Kenwright摘要：声音是基本的、丰富的信息来源;在从人文和社会科学到工程和数学的许多领域发挥着关键作用。声音不仅仅是数据信号。它包含了身体、感官和情感，以及社会、文化和环境因素。声音有助于我们的经验、环境和信念的转变。声音就在我们身边，无处不在。因此，声音是一个复杂的多成分实体，具有各种各样的特性和应用，这一点也不奇怪。当然，一个重要的问题是，什么是数字化存储和表示声音以捕捉这些特征的最佳方法？哪种模型或方法最适合处理、提取和过滤声音？有大量的表示和模型，然而，一种尚未用于声音的方法是双四元数。虽然双四元数已经在许多科学和计算领域中确立了自身作为用于提供表示多分量数据的明确的、不麻烦的、计算上有效的手段的有效数学模型。声音是一个尚未探索和收获双四元数（使用声音和音频相关的双四元数模型）的好处的领域。本文旨在探索双四元数在与基于声音的模型（包括但不限于应用程序，工具，机器学习，统计和计算声音相关算法）相结合时所提供的令人兴奋的潜力和可能性。摘要：Sound is a fundamental and rich source of information; playing a key role in many areas from humanities and social sciences through to engineering and mathematics. Sound is more than just data 'signals'. It encapsulates physical, sensorial and emotional, as well as social, cultural and environmental factors. Sound contributes to the transformation of our experiences, environments and beliefs. Sound is all around us and everywhere. Hence, it should come as no surprise that sound is a complex multicomponent entity with a vast assortment of characteristics and applications. Of course, an important question is, what is the best way to store and represent sound digitally to capture these characteristics? What model or method is best for manipulating, extracting and filtering sounds? There are a large number of representations and models, however, one approach that has yet to be used with sound is dual-quaternions. While dual-quaternions have established themselves in many fields of science and computing as an efficient mathematical model for providing an unambiguous, un-cumbersome, computationally effective means of representing multi-component data. Sound is one area that has yet to explore and reap the benefits of dual-quaternions (using sound and audio-related dual-quaternion models). This article aims to explore the exciting potential and possibilities dual-quaternions offer when applied and combined with sound-based models (including but not limited to the applications, tools, machine-learning, statistical and computational sound-related algorithms).

【4】 Posthoc Interpretation via Quantization标题：量化后解释链接：http://arxiv.org/pdf/2303.12659v1

*与cs.SD语音【2】为同一篇

作者：Cem Subakan,Francesco Paissan,Mirco Ravanelli机构：Université Laval,Concordia University,Mila, Québec AI Institute,University of Trento备注：Equal contribution摘要：在本文中，我们介绍了一种新的方法，称为“事后解释通过量化（PIQ）”，用于解释由训练的分类器作出的决定。我们的方法利用矢量量化的分类器的表示转换成一个离散的，类特定的潜在空间。特定于类的码本充当瓶颈，其迫使解释器集中于分类器认为与做出预测相关的输入数据的部分。我们通过定量和定性研究评估了我们的方法，发现与文献中的其他几种解释方法相比，PIQ生成的解释更容易被参与者理解。摘要：In this paper, we introduce a new approach, called "Posthoc Interpretation via Quantization (PIQ)", for interpreting decisions made by trained classifiers. Our method utilizes vector quantization to transform the representations of a classifier into a discrete, class-specific latent space. The class-specific codebooks act as a bottleneck that forces the interpreter to focus on the parts of the input data deemed relevant by the classifier for making a prediction. We evaluated our method through quantitative and qualitative studies and found that PIQ generates interpretations that are more easily understood by participants to our user studies when compared to several other interpretation methods in the literature.

【5】 Music-Driven Group Choreography标题：音乐驱动的集体编舞链接：http://arxiv.org/pdf/2303.12337v1

*与cs.SD语音【3】为同一篇

作者：Nhat Le,Thang Pham,Tuong Do,Erman Tjiputra,Quang D. Tran,Anh Nguyen机构：AIOZ, Singapore ,University of Liverpool, UK备注：accepted in cvpr 2023摘要：音乐驱动的编排是一个具有挑战性的问题，具有广泛的工业应用。近来，已经提出了许多方法来从单个舞蹈者的音乐合成舞蹈动作。然而，为一组生成舞蹈动作仍然是一个悬而未决的问题。在本文中，我们提出了$ rm AIOZ-GDANCE$，一个新的大型数据集的音乐驱动的群舞生成。与现有的数据集，只支持单一的舞蹈，我们的新数据集包含集体舞蹈视频，从而支持集体舞蹈的研究。我们提出了一种半自主的标签方法，人类在循环中获得我们的数据集的3D地面真相。建议的数据集包括16.7 $小时的配对音乐和3D运动，从在野外的视频，涵盖$7$舞蹈风格和$16$音乐流派。我们发现单纯地应用单一舞蹈产生技术来产生群舞动作可能会导致不理想的结果，例如动作不一致和舞者之间的碰撞。基于我们的新数据集，我们提出了一种新的方法，需要一个输入的音乐序列和一组舞者的3D位置，以有效地产生多组连贯的编舞。我们提出了新的评估指标，测量集体舞的质量和密集的实验，以证明我们的方法的有效性。摘要：Music-driven choreography is a challenging problem with a wide variety of industrial applications. Recently, many methods have been proposed to synthesize dance motions from music for a single dancer. However, generating dance motion for a group remains an open problem. In this paper, we present $ rm AIOZ-GDANCE$, a new large-scale dataset for music-driven group dance generation. Unlike existing datasets that only support single dance, our new dataset contains group dance videos, hence supporting the study of group choreography. We propose a semi-autonomous labeling method with humans in the loop to obtain the 3D ground truth for our dataset. The proposed dataset consists of $16.7$ hours of paired music and 3D motion from in-the-wild videos, covering $7$ dance styles and $16$ music genres. We show that naively applying single dance generation technique to creating group dance motion may lead to unsatisfactory results, such as inconsistent movements and collisions between dancers. Based on our new dataset, we propose a new method that takes an input music sequence and a set of 3D positions of dancers to efficiently produce multiple group-coherent choreographies. We propose new evaluation metrics for measuring group dance quality and perform intensive experiments to demonstrate the effectiveness of our method.

【6】 Exploring Turkish Speech Recognition via Hybrid CTCAttention Architecture and Multi-feature Fusion Network标题：基于混合CTC注意结构和多特征融合网络的土耳其语语音识别研究链接：http://arxiv.org/pdf/2303.12300v1

*与cs.SD语音【4】为同一篇

作者：Zeyu Ren,Nurmement Yolwas,Huiru Wang,Wushour Slamu机构：†Xinjiang Multilingual Information Technology Laboratory, College of Information Science and Engineering, Xinjiang University, Urumqi, China, ‡Xinjiang Key Laboratory of Signal Detection, College of Information Science and Engineering摘要：近年来，基于深度学习的端到端语音识别技术发展迅速。由于缺乏土耳其语语音数据，土耳其语语音识别系统的性能较差。本文首先研究了一系列语音识别调优技术。结果表明，采用速度扰动与加噪相结合的数据增强技术，波束搜索宽度设为16。其次，为了最大限度地利用有效特征信息，提高特征提取的准确性，本文提出了一种新的特征提取器LSPC。LSPC和LiGRU网络相结合，形成共享编码器结构，实现模型压缩。实验结果表明，在仅使用Fbank特征时，LSPC的性能优于MSPC和VGGnet，WER分别提高了1.01%和2.53%。最后，基于以上两点，提出了一种新的多特征融合网络作为编码器的主要结构。实验结果表明，基于LSPC的特征融合网络的WER比基于LSPC的单一特征（Fbank特征和Spectrogram特征）提取分别提高了0.82%和1.94%。我们的模型实现的性能与先进的端到端模型。摘要：In recent years, End-to-End speech recognition technology based on deep learning has developed rapidly. Due to the lack of Turkish speech data, the performance of Turkish speech recognition system is poor. Firstly, this paper studies a series of speech recognition tuning technologies. The results show that the performance of the model is the best when the data enhancement technology combining speed perturbation with noise addition is adopted and the beam search width is set to 16. Secondly, to maximize the use of effective feature information and improve the accuracy of feature extraction, this paper proposes a new feature extractor LSPC. LSPC and LiGRU network are combined to form a shared encoder structure, and model compression is realized. The results show that the performance of LSPC is better than MSPC and VGGnet when only using Fbank features, and the WER is improved by 1.01% and 2.53% respectively. Finally, based on the above two points, a new multi-feature fusion network is proposed as the main structure of the encoder. The results show that the WER of the proposed feature fusion network based on LSPC is improved by 0.82% and 1.94% again compared with the single feature (Fbank feature and Spectrogram feature) extraction using LSPC. Our model achieves performance comparable to that of advanced End-to-End models.

机器翻译，仅供参考