今天跟大家分享一篇语音相关的论文合集：cs.SD语音7篇，eess.AS音频处理8篇。

cs.SD语音

【1】 Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First  Results

标题：自发幽默的多模式预测：一个新的数据集和初步结果

链接：https://arxiv.org/abs/2209.14272

作者：Lukas Christ,Shahin Amiriparian,Alexander Kathan,Niklas Müller,Andreas König,Björn W. Schuller
机构：and Bj¨orn W. Schuller, Fellow, IEEE
备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
摘要：幽默是人类情感和认知的重要组成部分。它的自动理解可以促进更自然的人机交互和人工智能的人性化。当前的幽默检测方法仅仅基于分阶段的数据，这使得它们不适合“真实世界”的应用。我们通过引入新的帕绍-自发足球教练幽默（帕绍-SFCH）数据集来解决这一缺陷，该数据集包括约11小时的记录。Passau-SFCH数据集注释了幽默的存在及其维度（情感和方向），如马丁的幽默风格问卷所提出的。我们进行了一系列的实验，使用了预先训练的Transformers、卷积神经网络和专家设计的特征。分析了各种模态（文本、音频、视频）对自发幽默识别的性能，并研究了它们的互补性。我们的研究结果表明，对于幽默及其情感的自动分析，面部表情是最有希望的，而幽默方向可以通过基于文本的特征进行最好的建模。结果显示，不同的被试在幽默的使用和风格上存在着很大的差异，突出了幽默使用和风格的个性。此外，我们观察到决策级融合产生最好的识别结果。最后，我们在www.example.com上公开了我们的代码https://www.github.com/EIHW/passau-sfch。Passau-SFCH数据集可应要求提供。
摘要：Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.


【2】 Deepfake audio detection by speaker verification

标题：基于说话人验证的深伪音频检测

链接：https://arxiv.org/abs/2209.14098

作者：Alessandro Pianese,Davide Cozzolino,Giovanni Poggi,Luisa Verdoliva
机构：University Federico II of Naples, Italy
摘要：得益于深度学习领域的最新进展，如今已经存在复杂的生成工具，可以生成极其逼真的合成语音。然而，恶意使用此类工具是可能的，对我们的社会构成严重威胁。因此，合成语音检测成为一个迫切的研究课题，并且近来已经提出了各种各样的检测方法。不幸的是，它们很难推广到由在训练阶段从未见过的工具生成的合成音频，这使得它们不适合面对现实世界的场景。在本研究中，我们提出了一种新的检测方法来克服这一问题，该方法仅利用说话人的生物特征，而不参考特定的操作。由于检测器仅在真实数据上训练，因此自动确保泛化。所提出的方法可以基于现成的说话人确认工具来实现。我们在三个流行的测试集上测试了几个这样的解决方案，获得了良好的性能、高泛化能力和对音频损伤的高鲁棒性。
摘要：Thanks to recent advances in deep learning, sophisticated generation tools exist, nowadays, that produce extremely realistic synthetic speech. However, malicious uses of such tools are possible and likely, posing a serious threat to our society. Hence, synthetic voice detection has become a pressing research topic, and a large variety of detection methods have been recently proposed. Unfortunately, they hardly generalize to synthetic audios generated by tools never seen in the training phase, which makes them unfit to face real-world scenarios. In this work, we aim at overcoming this issue by proposing a new detection approach that leverages only the biometric characteristics of the speaker, with no reference to specific manipulations. Since the detector is trained only on real data, generalization is automatically ensured. The proposed approach can be implemented based on off-the-shelf speaker verification tools. We test several such solutions on three popular test sets, obtaining good performance, high generalization ability, and high robustness to audio impairment.


【3】 MeWEHV: Mel and Wave Embeddings for Human Voice Tasks

标题：MeWEHV：用于人类语音任务的MEL和WAVE嵌入

链接：https://arxiv.org/abs/2209.14078

作者：Andrés Vasco-Carofilis,Laura Fernández-Robles,Enrique Alegre,Eduardo Fidalgo
机构：Dept. of Electrical, Systems and Automation Engineering. School of Industrial, Computer, and Aerospace Engineering, Universidad de Le´on, Campus Vegazana , Le´on, Spain, Researcher at INCIBE (Spanish National Cybersecurity Institute), Le´on, Spain
备注：Submitted to Expert Systems with Applications
摘要：语音处理的最新趋势是使用通过机器学习模型创建的嵌入，所述机器学习模型针对具有大数据集的特定任务进行训练。通过利用已经获得的知识，这些模型可以在可用数据量很小的新任务中重用。提出了一种新的语音嵌入模型MeWEHV（Mel and Wave Embeddings for Human Voice Tasks），该模型能够为语音处理生成鲁棒的嵌入。MeWEHV结合了由预先训练的原始音频波形编码器模型生成的嵌入，以及使用卷积神经网络（CNNs）从梅尔频率倒谱系数（MFCC）提取的深度特征。我们在三个任务上评估MeWEHV的性能：说话者、语言和口音识别。对于第一个问题，我们使用VoxCeleb1数据集，并提供YouSpeakers204，这是一个新的公开可用的英语说话人识别数据集，包含来自204个人以六种不同口音说话的19607个音频剪辑，允许其他研究人员使用非常平衡的数据集，并创建对多种口音鲁棒的新模型。为了评估语言识别任务，我们使用了VoxForge和Common Language数据集。最后，对于口音识别，我们使用拉丁美洲西班牙语语料库（LASC）和普通语音数据集。我们的方法允许在所有测试数据集上显著提高最先进的模型的性能，同时具有低的额外计算成本。
摘要：A recent trend in speech processing is the use of embeddings created through machine learning models trained on a specific task with large datasets. By leveraging the knowledge already acquired, these models can be reused in new tasks where the amount of available data is small. This paper proposes a pipeline to create a new model, called Mel and Wave Embeddings for Human Voice Tasks (MeWEHV), capable of generating robust embeddings for speech processing. MeWEHV combines the embeddings generated by a pre-trained raw audio waveform encoder model, and deep features extracted from Mel Frequency Cepstral Coefficients (MFCCs) using Convolutional Neural Networks (CNNs). We evaluate the performance of MeWEHV on three tasks: speaker, language, and accent identification. For the first one, we use the VoxCeleb1 dataset and present YouSpeakers204, a new and publicly available dataset for English speaker identification that contains 19607 audio clips from 204 persons speaking in six different accents, allowing other researchers to work with a very balanced dataset, and to create new models that are robust to multiple accents. For evaluating the language identification task, we use the VoxForge and Common Language datasets. Finally, for accent identification, we use the Latin American Spanish Corpora (LASC) and Common Voice datasets. Our approach allows a significant increase in the performance of state-of-the-art models on all the tested datasets, with a low additional computational cost.


【4】 Entangling Practice with Artistic and Educational Aims: Interviews on  Technology-based Movement Sound Interactions

标题：将实践与艺术和教育目标纠缠在一起：基于技术的运动声音互动访谈

链接：https://arxiv.org/abs/2209.13921

作者：Victor Paredes,Jules Françoise,Frédéric Bevilacqua
机构：License: Creative Commons Attribution ,., International License (CC-BY ,.,)
备注：New Interfaces for Musical Expression (NIME), Jun 2022, Auckland, New Zealand
摘要：运动-声音交互系统是不同艺术和教育实践的接口。在这种用途的多样性中，我们从学习、占用和与技术系统的关系方面考察了共同的特性。虽然这些主题已经在NIME报告过，但我们想调查来自不同角度的从业者如何与这些问题相关。我们采访了6位从事运动声音互动的艺术家：1名表演者、1名表演者/作曲家、1名作曲家、1名教师/作曲家、1名舞蹈家/教师、1名舞蹈家。通过对成绩单的主题分析，我们确定了三个主要主题：（1）技术工具的中介作用;（2）可用性和规范性;（3）学习和实践。这些结果为运动-声音交互系统的设计和研究提供了理论依据。
摘要：Movement-sound interactive systems are at the interface of different artistic and educational practices. Within this multiplicity of uses, we examine common denominators in terms of learning, appropriation and relationship to technological systems. While these topics have been previously reported at NIME, we wanted to investigate how practitioners, coming from different perspectives, relate to these questions. We conducted interviews with 6 artists who are engaged in movementsound interactions: 1 performer, 1 performer/composer, 1 composer, 1 teacher/composer, 1 dancer/teacher, 1 dancer. Through a thematic analysis of the transcripts we identified three main themes related to (1) the mediating role of technological tools (2) usability and normativity, and (3) learning and practice. These results provide ground for discussion about the design and study of movement-sound interactive systems.


【5】 An Efficient Multitask Learning Architecture for Affective Vocal Burst  Analysis

标题：一种用于情感发声分析的高效多任务学习结构

链接：https://arxiv.org/abs/2209.13914

作者：Tobias Hallmen,Silvan Mertes,Dominik Schiller,Elisabeth André
机构：Chair for Human-Centered Artificial Intelligence, Augsburg University, Germany, Elisabeth Andr´e摘要：情感语音分析是一个正在进行的研究课题。在这个领域中，一个相对较新的问题是分析突发声，这是非语言的发声，如笑或叹息。目前的情感突发语音分析方法大多基于wav2vec2或HuBERT特征。在本文中，我们研究了wav2vec后继data2vec与多任务学习流水线的结合使用，以同时处理不同的分析问题。为了评估我们高效的多任务学习架构的性能，我们参加了2022年ACII情感爆发式发声挑战赛，结果显示我们的方法在三个不同的子任务中大大优于那里建立的基线。
摘要：Affective speech analysis is an ongoing topic of research. A relatively new problem in this field is the analysis of vocal bursts, which are nonverbal vocalisations such as laughs or sighs. Current state-of-the-art approaches to address affective vocal burst analysis are mostly based on wav2vec2 or HuBERT features. In this paper, we investigate the use of the wav2vec successor data2vec in combination with a multitask learning pipeline to tackle different analysis problems at once. To assess the performance of our efficient multitask learning architecture, we participate in the 2022 ACII Affective Vocal Burst Challenge, showing that our approach substantially outperforms the baseline established there in three different subtasks.


【6】 Computing Melodic Templates in Oral Music Traditions

标题：计算口述音乐传统中的旋律模板

链接：https://arxiv.org/abs/2209.13598

作者：Sergey Bereg,José-Miguel Díaz-Báñez,Nadine Kroher,Inmaculada Ventura
机构：Jos´e-Miguel D´ıaz-B´a˜nez†
摘要：术语旋律模板或骨架指的是在音乐表演期间经受变化的基本旋律。在许多口头音乐传统中，这些模板被隐式地代代相传，而从来没有在乐谱中被正式化。本文引入一个新的几何优化问题--跨越管问题，来近似一组对应于特定风格的口头音乐传统的有标签的演奏转录的旋律模板。给定一组$n$个分段线性函数，我们解决了寻找连续函数$f^*$和最小值$\varepsilon^*$的问题，使得以$（x，f^*（x））$为中心的长度为$2\varepsilon^*$的垂直线段至少与$p$个函数（$p\leq n$）相交。本文所探讨的方法也提供了一种新的工具来定量评估跨表演发生的旋律变化量。
摘要：The term melodic template or skeleton refers to a basic melody which is subject to variation during a music performance. In many oral music tradition, these templates are implicitly passed throughout generations without ever being formalized in a score. In this work, we introduce a new geometric optimization problem, the spanning tube problem, to approximate a melodic template for a set of labeled performance transcriptions corresponding to an specific style in oral music traditions. Given a set of $n$ piecewise linear functions, we solve the problem of finding a continuous function, $f^*$, and a minimum value, $\varepsilon^*$, such that, the vertical segment of length $2\varepsilon^*$ centered at $(x,f^*(x))$ intersects at least $p$ functions ($p\leq n$). The method explored here also provide a novel tool for quantitatively assess the amount of melodic variation which occurs across performances.


【7】 Speech Enhancement Using Self-Supervised Pre-Trained Model and Vector  Quantization

标题：基于自监督预训练模型和矢量量化的语音增强

链接：https://arxiv.org/abs/2209.14150

作者：Xiao-Ying Zhao,Qiu-Shi Zhu,Jie Zhang
机构：∗University of Science and Technology of China (USTC), Hefei, Anhui, China, †National Engineering Research Center for Speech and Language Information Processing
备注：Accepted to APSIPA ASC 2022
摘要：随着深度学习技术的发展，基于神经网络的语音增强（SE）模型表现出了优异的性能。同时，研究表明，自监督预训练模型的发展可以应用于各种下游任务。在本文中，我们将考虑将预先训练好的模型应用于实时SE问题。具体而言，利用自监督预训练WavLM模型对DEMUCS模型的编码器和瓶颈层进行初始化，用因果卷积代替编码器中的卷积，瓶颈层的Transformer编码器基于因果注意掩码。此外，由于离散化含噪语音表示更有利于去噪，我们利用量化模块对瓶颈层输出的表示进行离散化，然后将离散化后的表示送入解码器重构出干净的语音波形。在Valentini数据集和一个内部数据集上的实验结果表明，基于预训练模型的初始化可以提高SE性能，离散化操作在一定程度上抑制了表示中的噪声成分，从而进一步提高了性能。
摘要：With the development of deep learning, neural network-based speech enhancement (SE) models have shown excellent performance. Meanwhile, it was shown that the development of self-supervised pre-trained models can be applied to various downstream tasks. In this paper, we will consider the application of the pre-trained model to the real-time SE problem. Specifically, the encoder and bottleneck layer of the DEMUCS model are initialized using the self-supervised pretrained WavLM model, the convolution in the encoder is replaced by causal convolution, and the transformer encoder in the bottleneck layer is based on causal attention mask. In addition, as discretizing the noisy speech representations is more beneficial for denoising, we utilize a quantization module to discretize the representation output from the bottleneck layer, which is then fed into the decoder to reconstruct the clean speech waveform. Experimental results on the Valentini dataset and an internal dataset show that the pre-trained model based initialization can improve the SE performance and the discretization operation suppresses the noise component in the representations to some extent, which can further improve the performance.


eess.AS音频处理

【1】 Audio Retrieval with WavText5K and CLAP Training

标题：用WavText5K和CLAP训练进行音频检索

链接：https://arxiv.org/abs/2209.14275

作者：Soham Deshmukh,Benjamin Elizalde,Huaming Wang
机构：Microsoft
摘要：音频-文本检索采用自然语言查询来检索数据库中的相关音频文件。而文本-音频检索则是将音频文件作为查询来检索相关的自然语言描述。大多数文献使用一个音频字幕数据集来训练检索系统，但是评估使用多个数据集训练的益处的探索不足。此外，检索系统必须学习描述从几秒到几分钟的可变长度的音频内容的详细句子之间的对齐。在这项工作中，我们提出了一个新的网络音频-文本对集合和一个新的检索框架。首先，我们提供了一个新的集合，其中包含大约5000个Web音频-文本对，我们称之为WavText 5 K。当用于训练我们的检索系统时，WavText 5 K比其他音频字幕数据集提高了性能。第二，我们的框架通过使用一个文本编码器、两个音频编码器和一个对比学习目标来学习连接语言和音频内容。组合两种音频编码器有助于处理可变长度的音频。这两个贡献在文本-音频检索方面分别比AudioCaps和Clotho的最新性能高出2%和16%，在音频-文本检索方面分别高出6%和23%。
摘要：Audio-Text retrieval takes a natural language query to retrieve relevant audio files in a database. Conversely, Text-Audio retrieval takes an audio file as a query to retrieve relevant natural language descriptions. Most of the literature train retrieval systems with one audio captioning dataset, but evaluating the benefit of training with multiple datasets is underexplored. Moreover, retrieval systems have to learn the alignment between elaborated sentences describing audio content of variable length ranging from a few seconds to several minutes. In this work, we propose a new collection of web audio-text pairs and a new framework for retrieval. First, we provide a new collection of about five thousand web audio-text pairs that we refer to as WavText5K. When used to train our retrieval system, WavText5K improved performance more than other audio captioning datasets. Second, our framework learns to connect language and audio content by using a text encoder, two audio encoders, and a contrastive learning objective. Combining both audio encoders helps to process variable length audio. The two contributions beat state of the art performance for AudioCaps and Clotho on Text-Audio retrieval by a relative 2% and 16%, and Audio-Text retrieval by 6% and 23%.


【2】 Speech Enhancement Using Self-Supervised Pre-Trained Model and Vector  Quantization

标题：基于自监督预训练模型和矢量量化的语音增强

链接：https://arxiv.org/abs/2209.14150

* 与cs.SD语音【7】为同一篇

作者：Xiao-Ying Zhao,Qiu-Shi Zhu,Jie Zhang
机构：∗University of Science and Technology of China (USTC), Hefei, Anhui, China, †National Engineering Research Center for Speech and Language Information Processing
备注：Accepted to APSIPA ASC 2022
摘要：随着深度学习技术的发展，基于神经网络的语音增强（SE）模型表现出了优异的性能。同时，研究表明，自监督预训练模型的发展可以应用于各种下游任务。在本文中，我们将考虑将预先训练好的模型应用于实时SE问题。具体而言，利用自监督预训练WavLM模型对DEMUCS模型的编码器和瓶颈层进行初始化，用因果卷积代替编码器中的卷积，瓶颈层的Transformer编码器基于因果注意掩码。此外，由于离散化含噪语音表示更有利于去噪，我们利用量化模块对瓶颈层输出的表示进行离散化，然后将离散化后的表示送入解码器重构出干净的语音波形。在Valentini数据集和一个内部数据集上的实验结果表明，基于预训练模型的初始化可以提高SE性能，离散化操作在一定程度上抑制了表示中的噪声成分，从而进一步提高了性能。
摘要：With the development of deep learning, neural network-based speech enhancement (SE) models have shown excellent performance. Meanwhile, it was shown that the development of self-supervised pre-trained models can be applied to various downstream tasks. In this paper, we will consider the application of the pre-trained model to the real-time SE problem. Specifically, the encoder and bottleneck layer of the DEMUCS model are initialized using the self-supervised pretrained WavLM model, the convolution in the encoder is replaced by causal convolution, and the transformer encoder in the bottleneck layer is based on causal attention mask. In addition, as discretizing the noisy speech representations is more beneficial for denoising, we utilize a quantization module to discretize the representation output from the bottleneck layer, which is then fed into the decoder to reconstruct the clean speech waveform. Experimental results on the Valentini dataset and an internal dataset show that the pre-trained model based initialization can improve the SE performance and the discretization operation suppresses the noise component in the representations to some extent, which can further improve the performance.


【3】 Multimodal Prediction of Spontaneous Humour: A Novel Dataset and First  Results

标题：自发幽默的多模式预测：一个新的数据集和初步结果

链接：https://arxiv.org/abs/2209.14272

* 与cs.SD语音【1】为同一篇

作者：Lukas Christ,Shahin Amiriparian,Alexander Kathan,Niklas Müller,Andreas König,Björn W. Schuller
机构：and Bj¨orn W. Schuller, Fellow, IEEE
备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible
摘要：幽默是人类情感和认知的重要组成部分。它的自动理解可以促进更自然的人机交互和人工智能的人性化。当前的幽默检测方法仅仅基于分阶段的数据，这使得它们不适合“真实世界”的应用。我们通过引入新的帕绍-自发足球教练幽默（帕绍-SFCH）数据集来解决这一缺陷，该数据集包括约11小时的记录。Passau-SFCH数据集注释了幽默的存在及其维度（情感和方向），如马丁的幽默风格问卷所提出的。我们进行了一系列的实验，使用了预先训练的Transformers、卷积神经网络和专家设计的特征。分析了各种模态（文本、音频、视频）对自发幽默识别的性能，并研究了它们的互补性。我们的研究结果表明，对于幽默及其情感的自动分析，面部表情是最有希望的，而幽默方向可以通过基于文本的特征进行最好的建模。结果显示，不同的被试在幽默的使用和风格上存在着很大的差异，突出了幽默使用和风格的个性。此外，我们观察到决策级融合产生最好的识别结果。最后，我们在www.example.com上公开了我们的代码https://www.github.com/EIHW/passau-sfch。Passau-SFCH数据集可应要求提供。
摘要：Humour is a substantial element of human affect and cognition. Its automatic understanding can facilitate a more naturalistic human-device interaction and the humanisation of artificial intelligence. Current methods of humour detection are solely based on staged data making them inadequate for 'real-world' applications. We address this deficiency by introducing the novel Passau-Spontaneous Football Coach Humour (Passau-SFCH) dataset, comprising of about 11 hours of recordings. The Passau-SFCH dataset is annotated for the presence of humour and its dimensions (sentiment and direction) as proposed in Martin's Humor Style Questionnaire. We conduct a series of experiments, employing pretrained Transformers, convolutional neural networks, and expert-designed features. The performance of each modality (text, audio, video) for spontaneous humour recognition is analysed and their complementarity is investigated. Our findings suggest that for the automatic analysis of humour and its sentiment, facial expressions are most promising, while humour direction can be best modelled via text-based features. The results reveal considerable differences among various subjects, highlighting the individuality of humour usage and style. Further, we observe that a decision-level fusion yields the best recognition result. Finally, we make our code publicly available at https://www.github.com/EIHW/passau-sfch. The Passau-SFCH dataset is available upon request.


【4】 Deepfake audio detection by speaker verification

标题：基于说话人验证的深伪音频检测

链接：https://arxiv.org/abs/2209.14098

* 与cs.SD语音【2】为同一篇

作者：Alessandro Pianese,Davide Cozzolino,Giovanni Poggi,Luisa Verdoliva
机构：University Federico II of Naples, Italy
摘要：得益于深度学习领域的最新进展，如今已经存在复杂的生成工具，可以生成极其逼真的合成语音。然而，恶意使用此类工具是可能的，对我们的社会构成严重威胁。因此，合成语音检测成为一个迫切的研究课题，并且近来已经提出了各种各样的检测方法。不幸的是，它们很难推广到由在训练阶段从未见过的工具生成的合成音频，这使得它们不适合面对现实世界的场景。在本研究中，我们提出了一种新的检测方法来克服这一问题，该方法仅利用说话人的生物特征，而不参考特定的操作。由于检测器仅在真实数据上训练，因此自动确保泛化。所提出的方法可以基于现成的说话人确认工具来实现。我们在三个流行的测试集上测试了几个这样的解决方案，获得了良好的性能、高泛化能力和对音频损伤的高鲁棒性。
摘要：Thanks to recent advances in deep learning, sophisticated generation tools exist, nowadays, that produce extremely realistic synthetic speech. However, malicious uses of such tools are possible and likely, posing a serious threat to our society. Hence, synthetic voice detection has become a pressing research topic, and a large variety of detection methods have been recently proposed. Unfortunately, they hardly generalize to synthetic audios generated by tools never seen in the training phase, which makes them unfit to face real-world scenarios. In this work, we aim at overcoming this issue by proposing a new detection approach that leverages only the biometric characteristics of the speaker, with no reference to specific manipulations. Since the detector is trained only on real data, generalization is automatically ensured. The proposed approach can be implemented based on off-the-shelf speaker verification tools. We test several such solutions on three popular test sets, obtaining good performance, high generalization ability, and high robustness to audio impairment.


【5】 MeWEHV: Mel and Wave Embeddings for Human Voice Tasks

标题：MeWEHV：用于人类语音任务的MEL和WAVE嵌入

链接：https://arxiv.org/abs/2209.14078

* 与cs.SD语音【3】为同一篇

作者：Andrés Vasco-Carofilis,Laura Fernández-Robles,Enrique Alegre,Eduardo Fidalgo
机构：Dept. of Electrical, Systems and Automation Engineering. School of Industrial, Computer, and Aerospace Engineering, Universidad de Le´on, Campus Vegazana , Le´on, Spain, Researcher at INCIBE (Spanish National Cybersecurity Institute), Le´on, Spain
备注：Submitted to Expert Systems with Applications
摘要：语音处理的最新趋势是使用通过机器学习模型创建的嵌入，所述机器学习模型针对具有大数据集的特定任务进行训练。通过利用已经获得的知识，这些模型可以在可用数据量很小的新任务中重用。提出了一种新的语音嵌入模型MeWEHV（Mel and Wave Embeddings for Human Voice Tasks），该模型能够为语音处理生成鲁棒的嵌入。MeWEHV结合了由预先训练的原始音频波形编码器模型生成的嵌入，以及使用卷积神经网络（CNNs）从梅尔频率倒谱系数（MFCC）提取的深度特征。我们在三个任务上评估MeWEHV的性能：说话者、语言和口音识别。对于第一个问题，我们使用VoxCeleb1数据集，并提供YouSpeakers204，这是一个新的公开可用的英语说话人识别数据集，包含来自204个人以六种不同口音说话的19607个音频剪辑，允许其他研究人员使用非常平衡的数据集，并创建对多种口音鲁棒的新模型。为了评估语言识别任务，我们使用了VoxForge和Common Language数据集。最后，对于口音识别，我们使用拉丁美洲西班牙语语料库（LASC）和普通语音数据集。我们的方法允许在所有测试数据集上显著提高最先进的模型的性能，同时具有低的额外计算成本。
摘要：A recent trend in speech processing is the use of embeddings created through machine learning models trained on a specific task with large datasets. By leveraging the knowledge already acquired, these models can be reused in new tasks where the amount of available data is small. This paper proposes a pipeline to create a new model, called Mel and Wave Embeddings for Human Voice Tasks (MeWEHV), capable of generating robust embeddings for speech processing. MeWEHV combines the embeddings generated by a pre-trained raw audio waveform encoder model, and deep features extracted from Mel Frequency Cepstral Coefficients (MFCCs) using Convolutional Neural Networks (CNNs). We evaluate the performance of MeWEHV on three tasks: speaker, language, and accent identification. For the first one, we use the VoxCeleb1 dataset and present YouSpeakers204, a new and publicly available dataset for English speaker identification that contains 19607 audio clips from 204 persons speaking in six different accents, allowing other researchers to work with a very balanced dataset, and to create new models that are robust to multiple accents. For evaluating the language identification task, we use the VoxForge and Common Language datasets. Finally, for accent identification, we use the Latin American Spanish Corpora (LASC) and Common Voice datasets. Our approach allows a significant increase in the performance of state-of-the-art models on all the tested datasets, with a low additional computational cost.


【6】 Entangling Practice with Artistic and Educational Aims: Interviews on  Technology-based Movement Sound Interactions

标题：将实践与艺术和教育目标纠缠在一起：基于技术的运动声音互动访谈

链接：https://arxiv.org/abs/2209.13921

* 与cs.SD语音【4】为同一篇

作者：Victor Paredes,Jules Françoise,Frédéric Bevilacqua
机构：License: Creative Commons Attribution ,., International License (CC-BY ,.,)
备注：New Interfaces for Musical Expression (NIME), Jun 2022, Auckland, New Zealand
摘要：运动-声音交互系统是不同艺术和教育实践的接口。在这种用途的多样性中，我们从学习、占用和与技术系统的关系方面考察了共同的特性。虽然这些主题已经在NIME报告过，但我们想调查来自不同角度的从业者如何与这些问题相关。我们采访了6位从事运动声音互动的艺术家：1名表演者、1名表演者/作曲家、1名作曲家、1名教师/作曲家、1名舞蹈家/教师、1名舞蹈家。通过对成绩单的主题分析，我们确定了三个主要主题：（1）技术工具的中介作用;（2）可用性和规范性;（3）学习和实践。这些结果为运动-声音交互系统的设计和研究提供了理论依据。
摘要：Movement-sound interactive systems are at the interface of different artistic and educational practices. Within this multiplicity of uses, we examine common denominators in terms of learning, appropriation and relationship to technological systems. While these topics have been previously reported at NIME, we wanted to investigate how practitioners, coming from different perspectives, relate to these questions. We conducted interviews with 6 artists who are engaged in movementsound interactions: 1 performer, 1 performer/composer, 1 composer, 1 teacher/composer, 1 dancer/teacher, 1 dancer. Through a thematic analysis of the transcripts we identified three main themes related to (1) the mediating role of technological tools (2) usability and normativity, and (3) learning and practice. These results provide ground for discussion about the design and study of movement-sound interactive systems.


【7】 An Efficient Multitask Learning Architecture for Affective Vocal Burst  Analysis

标题：一种用于情感发声分析的高效多任务学习结构

链接：https://arxiv.org/abs/2209.13914

* 与cs.SD语音【5】为同一篇

作者：Tobias Hallmen,Silvan Mertes,Dominik Schiller,Elisabeth André
机构：Chair for Human-Centered Artificial Intelligence, Augsburg University, Germany, Elisabeth Andr´e摘要：情感语音分析是一个正在进行的研究课题。在这个领域中，一个相对较新的问题是分析突发声，这是非语言的发声，如笑或叹息。目前的情感突发语音分析方法大多基于wav2vec2或HuBERT特征。在本文中，我们研究了wav2vec后继data2vec与多任务学习流水线的结合使用，以同时处理不同的分析问题。为了评估我们高效的多任务学习架构的性能，我们参加了2022年ACII情感爆发式发声挑战赛，结果显示我们的方法在三个不同的子任务中大大优于那里建立的基线。
摘要：Affective speech analysis is an ongoing topic of research. A relatively new problem in this field is the analysis of vocal bursts, which are nonverbal vocalisations such as laughs or sighs. Current state-of-the-art approaches to address affective vocal burst analysis are mostly based on wav2vec2 or HuBERT features. In this paper, we investigate the use of the wav2vec successor data2vec in combination with a multitask learning pipeline to tackle different analysis problems at once. To assess the performance of our efficient multitask learning architecture, we participate in the 2022 ACII Affective Vocal Burst Challenge, showing that our approach substantially outperforms the baseline established there in three different subtasks.


【8】 Computing Melodic Templates in Oral Music Traditions

标题：计算口述音乐传统中的旋律模板

链接：https://arxiv.org/abs/2209.13598

* 与cs.SD语音【6】为同一篇

作者：Sergey Bereg,José-Miguel Díaz-Báñez,Nadine Kroher,Inmaculada Ventura
机构：Jos´e-Miguel D´ıaz-B´a˜nez†
摘要：术语旋律模板或骨架指的是在音乐表演期间经受变化的基本旋律。在许多口头音乐传统中，这些模板被隐式地代代相传，而从来没有在乐谱中被正式化。本文引入一个新的几何优化问题--跨越管问题，来近似一组对应于特定风格的口头音乐传统的有标签的演奏转录的旋律模板。给定一组$n$个分段线性函数，我们解决了寻找连续函数$f^*$和最小值$\varepsilon^*$的问题，使得以$（x，f^*（x））$为中心的长度为$2\varepsilon^*$的垂直线段至少与$p$个函数（$p\leq n$）相交。本文所探讨的方法也提供了一种新的工具来定量评估跨表演发生的旋律变化量。
摘要：The term melodic template or skeleton refers to a basic melody which is subject to variation during a music performance. In many oral music tradition, these templates are implicitly passed throughout generations without ever being formalized in a score. In this work, we introduce a new geometric optimization problem, the spanning tube problem, to approximate a melodic template for a set of labeled performance transcriptions corresponding to an specific style in oral music traditions. Given a set of $n$ piecewise linear functions, we solve the problem of finding a continuous function, $f^*$, and a minimum value, $\varepsilon^*$, such that, the vertical segment of length $2\varepsilon^*$ centered at $(x,f^*(x))$ intersects at least $p$ functions ($p\leq n$). The method explored here also provide a novel tool for quantitatively assess the amount of melodic variation which occurs across performances.

机器翻译，仅供参考