【1】 Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters标题：基于带适配器的自监督语音表示模型的抗噪零发声文语合成链接：https://arxiv.org/abs/2401.05111作者：Kenichi Fujita,Hiroshi Sato,Takanori Ashihara,Hiroki Kanagawa,Marc Delcroix,Takafumi Moriya,Yusuke Ijima备注：5 pages,3 figures, Accepted to IEEE ICASSP 2024摘要：zero-shot文语转换（TTS）方法基于使用自监督学习（SSL）语音表示从参考语音提取的说话人嵌入，可以非常准确地再现说话人特征。然而，当参考语音包含噪声时，该方法遭受语音合成质量的劣化。在本文中，我们提出了一种噪声鲁棒的zero-shot TTS方法。我们将适配器合并到SSL模型中，并使用噪声参考语音与TTS模型进行微调。此外，为了进一步提高性能，我们采用了语音增强（SE）前端。通过这些改进，我们提出的基于SSL的zero-shot TTS实现了高质量的语音合成与噪声参考语音。通过客观和主观评价，我们证实了所提出的方法对参考语音中的噪声具有很强的鲁棒性，并且有效地与SE相结合。摘要：The zero-shot text-to-speech (TTS) method, based on speaker embeddings extracted from reference speech using self-supervised learning (SSL) speech representations, can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method. We incorporated adapters into the SSL model, which we fine-tuned with the TTS model using noisy reference speech. In addition, to further improve performance, we adopted a speech enhancement (SE) front-end. With these improvements, our proposed SSL-based zero-shot TTS achieved high-quality speech synthesis with noisy reference speech. Through the objective and subjective evaluations, we confirmed that the proposed method is highly robust to noise in reference speech, and effectively works in combination with SE.【2】 Singer Identity Representation Learning using Self-Supervised Techniques标题：使用自监督技术的歌手身份表征学习链接：https://arxiv.org/abs/2401.05064作者：Bernardo Torres,Stefan Lattner,Gaël Richard备注：None摘要：在使用语音数据创建语音身份表示方面已经取得了重大进展。然而，在歌唱方面没有取得同样的进展。为了弥合这一差距，我们提出了一个框架，用于训练歌手身份编码器，以提取适合于各种唱歌相关任务的表示，如唱歌的声音相似性和合成。我们在大量孤立的音轨上探索不同的自监督学习技术，并在训练过程中应用数据增强，以确保表示对音高和内容变化不变。我们评估了多个数据集上歌手相似性和识别任务的结果表示的质量，特别强调域外泛化。我们提出的框架产生了高质量的嵌入，在44.1 kHz下运行时，其性能优于说话人验证和wav2vec 2.0预先训练的歌唱声音基线。我们发布我们的代码和训练模型，以促进对歌唱声音和相关领域的进一步研究。摘要：Significant strides have been made in creating voice identity representations using speech data. However, the same level of progress has not been achieved for singing voices. To bridge this gap, we suggest a framework for training singer identity encoders to extract representations suitable for various singing-related tasks, such as singing voice similarity and synthesis. We explore different self-supervised learning techniques on a large collection of isolated vocal tracks and apply data augmentations during training to ensure that the representations are invariant to pitch and content variations. We evaluate the quality of the resulting representations on singer similarity and identification tasks across multiple datasets, with a particular emphasis on out-of-domain generalization. Our proposed framework produces high-quality embeddings that outperform both speaker verification and wav2vec 2.0 pre-trained baselines on singing voice while operating at 44.1 kHz. We release our code and trained models to facilitate further research on singing voice and related areas.【3】 MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector标题：MuTox：通用多语言音频毒性数据集和零触发检测器链接：https://arxiv.org/abs/2401.05060作者：Marta R. Costa-jussà,Mariano Coria Meglioli,Pierre Andrews,David Dale,Prangthip Hansanti,Elahe Kalbassi,Alex Mourachko,Christophe Ropers,Carleigh Wood摘要：语音模态（基于音频）的自然语言处理中的毒性检测研究非常有限，特别是对于英语以外的语言。为了解决这些限制并为真正的多语言音频毒性检测奠定基础，我们引入了MuTox，这是第一个带有毒性标签的高度多语言音频数据集。该数据集包括英语和西班牙语的20，000个音频话语，以及其他19种语言的4，000个。为了证明该数据集的质量，我们训练了基于MuTox音频的毒性分类器，该分类器能够在各种语言中进行zero-shot毒性检测。该分类器比现有的基于文本的可训练分类器的AUC高出1%以上，同时将语言覆盖范围扩大了十倍以上。与覆盖类似语言数量的基于单词列表的分类器相比，MuTox将精确度和召回率提高了约2.5倍。这一重大改进强调了MuTox在推进基于音频的毒性检测领域的潜力。摘要：Research in toxicity detection in natural language processing for the speech modality (audio-based) is quite limited, particularly for languages other than English. To address these limitations and lay the groundwork for truly multilingual audio-based toxicity detection, we introduce MuTox, the first highly multilingual audio-based dataset with toxicity labels. The dataset comprises 20,000 audio utterances for English and Spanish, and 4,000 for the other 19 languages. To demonstrate the quality of this dataset, we trained the MuTox audio-based toxicity classifier, which enables zero-shot toxicity detection across a wide range of languages. This classifier outperforms existing text-based trainable classifiers by more than 1% AUC, while expanding the language coverage more than tenfold. When compared to a wordlist-based classifier that covers a similar number of languages, MuTox improves precision and recall by approximately 2.5 times. This significant improvement underscores the potential of MuTox in advancing the field of audio-based toxicity detection.【4】 Learning Audio Concepts from Counterfactual Natural Language标题：从反事实自然语言中学习音频概念链接：https://arxiv.org/abs/2401.04935作者：Ali Vosoughi,Luca Bondi,Ho-Hsiang Wu,Chenliang Xu备注：Accepted at ICASSP 2024摘要：传统的音频分类依赖于预定义的类，缺乏从自由形式文本中学习的能力。最近的方法从描述自然语言中音频的原始音频文本对中解锁学习联合音频文本嵌入。尽管最近取得了进展，但很少探索系统的方法来训练模型，以识别替代场景中的声音事件和来源，例如在类似情况下区分户外活动中的烟花和枪声。本研究介绍了音频领域的因果推理和反事实分析。我们使用反事实实例，并将它们包含在我们的模型中的不同方面。我们的模型考虑了人类注释的参考文本的声学特性和声源信息。为了验证我们模型的有效性，我们利用多个音频字幕数据集进行了预训练。然后，我们用几个常见的下游任务进行评估，证明了所提出的方法作为利用音频域中的反事实信息的第一个作品之一的优点。具体来说，开放式语言音频检索任务的前1名准确率提高了43%以上。摘要：Conventional audio classification relied on predefined classes, lacking the ability to learn from free-form text. Recent methods unlock learning joint audio-text embeddings from raw audio-text pairs describing audio in natural language. Despite recent advancements, there is little exploration of systematic methods to train models for recognizing sound events and sources in alternative scenarios, such as distinguishing fireworks from gunshots at outdoor events in similar situations. This study introduces causal reasoning and counterfactual analysis in the audio domain. We use counterfactual instances and include them in our model across different aspects. Our model considers acoustic characteristics and sound source information from human-annotated reference texts. To validate the effectiveness of our model, we conducted pre-training utilizing multiple audio captioning datasets. We then evaluate with several common downstream tasks, demonstrating the merits of the proposed method as one of the first works leveraging counterfactual information in audio domain. Specifically, the top-1 accuracy in open-ended language-based audio retrieval task increased by more than 43%.【5】 Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection标题：基于语音活动投影的实时连续话轮切换预测链接：https://arxiv.org/abs/2401.04868作者：Koji Inoue,Bing'er Jiang,Erik Ekstedt,Tatsuya Kawahara,Gabriel Skantze备注：This paper has been accepted for presentation at International Workshop on Spoken Dialogue Systems Technology 2024 (IWSDS 2024) and represents the author's version of the work摘要：一个实时和连续的话轮转换预测系统的演示。该系统基于语音活动投影（VAP）模型，该模型直接将对话立体声音频映射到未来的语音活动。VAP模型包括对比预测编码（CPC）和自我注意Transformers，然后是交叉注意变换器。我们研究了输入上下文音频长度的影响，并证明了所提出的系统可以在CPU设置的实时操作，以最小的性能下降。摘要：A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.【6】 DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation标题：DiffSHEG：一种基于扩散的实时语音驱动整体3D表情和手势生成方法链接：https://arxiv.org/abs/2401.04747作者：Junming Chen,Yunfei Liu,Jianan Wang,Ailing Zeng,Yu Li,Qifeng Chen摘要：我们提出了DiffSHEG，一种基于扩散的方法，用于语音驱动的整体3D表达和任意长度的手势生成。虽然以前的作品集中在共同语音手势或表达生成单独的，同步的表情和手势的联合生成仍然很少探索。为了解决这个问题，我们的基于扩散的共同语音运动生成Transformer实现了从表情到手势的单向信息流，促进了联合表情-手势分布的改进匹配。此外，我们引入了一个基于outpainting的采样策略，用于扩散模型中的任意长序列生成，提供了灵活性和计算效率。我们的方法提供了一个实用的解决方案，产生高质量的同步表达和手势生成语音驱动。在两个公共数据集上进行评估，我们的方法在定量和定性方面都达到了最先进的性能。此外，一项用户研究证实了DiffSHEG优于以前的方法。通过实现表达和同步运动的实时生成，DiffSHEG展示了其在数字人类和具体代理开发中的各种应用的潜力。摘要：We propose DiffSHEG, a Diffusion-based approach for Speech-driven Holistic 3D Expression and Gesture generation with arbitrary length. While previous works focused on co-speech gesture or expression generation individually, the joint generation of synchronized expressions and gestures remains barely explored. To address this, our diffusion-based co-speech motion generation transformer enables uni-directional information flow from expression to gesture, facilitating improved matching of joint expression-gesture distributions. Furthermore, we introduce an outpainting-based sampling strategy for arbitrary long sequence generation in diffusion models, offering flexibility and computational efficiency. Our method provides a practical solution that produces high-quality synchronized expression and gesture generation driven by speech. Evaluated on two public datasets, our approach achieves state-of-the-art performance both quantitatively and qualitatively. Additionally, a user study confirms the superiority of DiffSHEG over prior approaches. By enabling the real-time generation of expressive and synchronized motions, DiffSHEG showcases its potential for various applications in the development of digital humans and embodied agents.【7】 Music Genre Classification: A Comparative Analysis of CNN and XGBoost Approaches with Mel-frequency cepstral coefficients and Mel Spectrograms标题：音乐类型分类：基于Mel倒谱系数和Mel谱图的CNN和XGBoost方法的比较分析链接：https://arxiv.org/abs/2401.04737作者：Yigang Meng摘要：近年来，各种精心设计的算法使音乐平台能够根据个人喜好提供内容。音乐类型是从各个方面来定义的，包括声学特征和文化因素。音乐流派分类与基于内容的过滤配合得很好，后者根据音乐相似性向用户推荐内容。给定一个相当大的数据集，一个前提是使用机器学习或深度学习方法进行自动注释，可以有效地对音频文件进行分类。系统的有效性在很大程度上取决于特征和模型的选择，因为不同的架构和特征可以相互促进并产生不同的结果。在这项研究中，我们进行了一项比较研究，调查三种模型的性能：一个建议的卷积神经网络（CNN），VGG16与全连接层（FC），和一个极端梯度提升（XGBoost）的方法对不同的功能：30秒梅尔频谱图和3秒梅尔频率倒谱系数（MFCC）。结果表明，MFCC XGBoost模型的性能优于其他模型。此外，在数据预处理阶段应用数据分割可以显着提高CNN的性能。摘要：In recent years, various well-designed algorithms have empowered music platforms to provide content based on one's preferences. Music genres are defined through various aspects, including acoustic features and cultural considerations. Music genre classification works well with content-based filtering, which recommends content based on music similarity to users. Given a considerable dataset, one premise is automatic annotation using machine learning or deep learning methods that can effectively classify audio files. The effectiveness of systems largely depends on feature and model selection, as different architectures and features can facilitate each other and yield different results. In this study, we conduct a comparative study investigating the performances of three models: a proposed convolutional neural network (CNN), the VGG16 with fully connected layers (FC), and an eXtreme Gradient Boosting (XGBoost) approach on different features: 30-second Mel spectrogram and 3-second Mel-frequency cepstral coefficients (MFCCs). The results show that the MFCC XGBoost model outperformed the others. Furthermore, applying data segmentation in the data preprocessing phase can significantly enhance the performance of the CNNs.【8】 ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video标题：Anim-400K：用于视频端到端自动配音的大规模数据集链接：https://arxiv.org/abs/2401.05314作者：Kevin Cai,Chonghua Liu,David M. Chan备注：To appear in ICASSP 2024摘要：互联网上丰富的内容，高达60%的内容是用英语发布的，这与全球人口形成鲜明对比，全球人口中只有18.8%的人说英语，只有5.1%的人认为这是他们的母语，导致在线信息获取的差异。不幸的是，用于视频配音的自动化过程-用翻译的替代方案替换视频的音轨-由于流水线而仍然是一项复杂且具有挑战性的任务，需要精确的定时、面部运动同步和韵律匹配。虽然端到端配音提供了一种解决方案，但数据稀缺仍然阻碍着端到端和基于管道的方法的进展。在这项工作中，我们介绍了Anim-400 K，这是一个综合数据集，包含超过425 K的日语和英语对齐动画视频片段，支持各种视频相关任务，包括自动配音，同声翻译，引导视频摘要和流派/主题/风格分类。我们的数据集在https://github.com/davidmchan/Anim400K上公开提供，用于研究目的。摘要：The Internet's wealth of content, with up to 60% published in English, starkly contrasts the global population, where only 18.8% are English speakers, and just 5.1% consider it their native language, leading to disparities in online information access. Unfortunately, automated processes for dubbing of video - replacing the audio track of a video with a translated alternative - remains a complex and challenging task due to pipelines, necessitating precise timing, facial movement synchronization, and prosody matching. While end-to-end dubbing offers a solution, data scarcity continues to impede the progress of both end-to-end and pipeline-based methods. In this work, we introduce Anim-400K, a comprehensive dataset of over 425K aligned animated video segments in Japanese and English supporting various video-related tasks, including automated dubbing, simultaneous translation, guided video summarization, and genre/theme/style classification. Our dataset is made publicly available for research purposes at https://github.com/davidmchan/Anim400K.【9】 Full-frequency dynamic convolution: a physical frequency-dependent convolution for sound event detection标题：全频动态卷积：一种用于声事件检测的物理频率相关卷积链接：https://arxiv.org/abs/2401.04976作者：Haobo Yue,Zhicheng Zhang,Da Mu,Yonghao Dang,Jianqin Yin,Jin Tang备注：6 pages, 4 figures, submitted to ICME2024摘要：最近，2D卷积已被发现不合格的声音事件检测（SED）。它使声音事件沿频率轴具有平移等变性，而频率轴不是平移不变维。为了解决这个问题，使用动态卷积来对声音事件的频率依赖性进行建模。在本文中，我们提出了第一个全动态的方法命名为全频动态卷积（FFDConv）。FFDConv为每个频带生成频率核，该频率核直接在频率相关建模的结构中设计。它在物理上提供了具有频率相关建模能力的2D卷积。FFDConv在DESED真实验证数据集上的PSDS 1测试结果比基线测试结果高出6.6%，且优于其他全动态方法。此外，通过可视化声音事件的特征，我们观察到FFDConv可以有效地提取特定频带中的相干特征，与声音事件的声音连续性一致。这证明了FFDConv具有很强的频率依赖感知能力。摘要：Recently, 2D convolution has been found unqualified in sound event detection (SED). It enforces translation equivariance on sound events along frequency axis, which is not a shift-invariant dimension. To address this issue, dynamic convolution is used to model the frequency dependency of sound events. In this paper, we proposed the first full-dynamic method named \emph{full-frequency dynamic convolution} (FFDConv). FFDConv generates frequency kernels for every frequency band, which is designed directly in the structure for frequency-dependent modeling. It physically furnished 2D convolution with the capability of frequency-dependent modeling. FFDConv outperforms not only the baseline by 6.6\% in DESED real validation dataset in terms of PSDS1, but outperforms the other full-dynamic methods. In addition, by visualizing features of sound events, we observed that FFDConv could effectively extract coherent features in specific frequency bands, consistent with the vocal continuity of sound events. This proves that FFDConv has great frequency-dependent perception ability.【10】 Self-supervised speech representation and contextual text embedding for match-mismatch classification with EEG recording标题：用于匹配-失配分类和脑电记录的自监督语音表示和上下文文本嵌入链接：https://arxiv.org/abs/2401.04964作者：Bo Wang,Xiran Xu,Zechen Zhang,Haolin Zhu,YuJie Yan,Xihong Wu,Jing Chen备注：2 pages, 2 figures, submitted to ICASSP 2024摘要：将语音与EEG相关联具有相当大的重要性，但具有挑战性。在这项研究中，深度卷积网络提取时空特征的EEG数据。自监督语音表示和上下文文本嵌入被用作语音特征。对比学习被用于将EEG特征与语音特征相关联。实验结果证明了使用自监督语音表示和上下文文本嵌入的好处。通过特征融合和模型集成，实现了60.29%的准确率，性能在听觉脑电挑战赛（ICASSP 2024）的任务1中排名第二。摘要：Relating speech to EEG holds considerable importance but challenging. In this study, deep convolutional network was employed to extract spatiotemporal features from EEG data. Self-supervised speech representation and contextual text embedding were used as speech features. Contrastive learning was used to related EEG features to speech features. The experimental results demonstrate the benefits of using self-supervised speech representation and contextual text embedding. Through feature fusion and model ensemble, an accuracy of 60.29% was achieved, and the performance was ranked as No.2 in Task1 of the Auditory EEG Challenge (ICASSP 2024).eess.AS音频处理【1】 ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video标题：Anim-400K：用于视频端到端自动配音的大规模数据集链接：https://arxiv.org/abs/2401.05314作者：Kevin Cai,Chonghua Liu,David M. Chan备注：To appear in ICASSP 2024摘要：互联网上丰富的内容，高达60%的内容是用英语发布的，这与全球人口形成鲜明对比，全球人口中只有18.8%的人说英语，只有5.1%的人认为这是他们的母语，导致在线信息获取的差异。不幸的是，用于视频配音的自动化过程-用翻译的替代方案替换视频的音轨-由于流水线而仍然是一项复杂且具有挑战性的任务，需要精确的定时、面部运动同步和韵律匹配。虽然端到端配音提供了一种解决方案，但数据稀缺仍然阻碍着端到端和基于管道的方法的进展。在这项工作中，我们介绍了Anim-400 K，这是一个综合数据集，包含超过425 K的日语和英语对齐动画视频片段，支持各种视频相关任务，包括自动配音，同声翻译，引导视频摘要和流派/主题/风格分类。我们的数据集在https://github.com/davidmchan/Anim400K上公开提供，用于研究目的。摘要：The Internet's wealth of content, with up to 60% published in English, starkly contrasts the global population, where only 18.8% are English speakers, and just 5.1% consider it their native language, leading to disparities in online information access. Unfortunately, automated processes for dubbing of video - replacing the audio track of a video with a translated alternative - remains a complex and challenging task due to pipelines, necessitating precise timing, facial movement synchronization, and prosody matching. While end-to-end dubbing offers a solution, data scarcity continues to impede the progress of both end-to-end and pipeline-based methods. In this work, we introduce Anim-400K, a comprehensive dataset of over 425K aligned animated video segments in Japanese and English supporting various video-related tasks, including automated dubbing, simultaneous translation, guided video summarization, and genre/theme/style classification. Our dataset is made publicly available for research purposes at https://github.com/davidmchan/Anim400K.【2】 Decoding of Selective Attention to Speech From Ear-EEG Recordings标题：从耳-脑电记录中解码对语音的选择性注意链接：https://arxiv.org/abs/2401.05187作者：Mike Thornton,Danilo Mandic,Tobias Reichenbach摘要：许多听力损失患者在拥挤的听觉场景中难以理解语音，即使他们使用助听器。未来的听力技术可以识别听者听觉注意力的焦点，并选择性地单独放大声音，可以改善这一患者群体使用助听器的体验。在这项工作中，我们提出了我们的实验结果与超可穿戴式耳内脑电图（EEG）监测设备。参与者在听觉注意力实验中听两个竞争的演讲者，同时记录他们的EEG。我们表明，典型的神经反应的语音信封，以及它的发作，可以从这样的设备中恢复，并且记录的反应的形态确实是由选择性注意语音调制。被关注和被忽略的语音流的特征也可以从EEG记录中重建，重建质量用作选择性听觉注意的标记。使用的刺激重建方法，我们表明，与此设备的听觉注意力可以解码的EEG记录，这是短短几秒钟的持续时间的短片段。研究结果进一步证明，耳脑电系统为可穿戴听觉监测以及未来的认知导向助听器提供了良好的前景。摘要：Many people with hearing loss struggle to comprehend speech in crowded auditory scenes, even when they are using hearing aids. Future hearing technologies which can identify the focus of a listener's auditory attention, and selectively amplify that sound alone, could improve the experience that this patient group has with their hearing aids. In this work, we present the results of our experiments with an ultra-wearable in-ear electroencephalography (EEG) monitoring device. Participants listened to two competing speakers in an auditory attention experiment whilst their EEG was recorded. We show that typical neural responses to the speech envelope, as well as its onsets, can be recovered from such a device, and that the morphology of the recorded responses is indeed modulated by selective attention to speech. Features of the attended and ignored speech stream can also be reconstructed from the EEG recordings, with the reconstruction quality serving as a marker of selective auditory attention. Using the stimulus-reconstruction method, we show that with this device auditory attention can be decoded from short segments of EEG recordings which are of just a few seconds in duration. The results provide further evidence that ear-EEG systems offer good prospects for wearable auditory monitoring as well as future cognitively-steered hearing aids.【3】 Full-frequency dynamic convolution: a physical frequency-dependent convolution for sound event detection标题：全频动态卷积：一种用于声事件检测的物理频率相关卷积链接：https://arxiv.org/abs/2401.04976作者：Haobo Yue,Zhicheng Zhang,Da Mu,Yonghao Dang,Jianqin Yin,Jin Tang备注：6 pages, 4 figures, submitted to ICME2024摘要：最近，2D卷积已被发现不合格的声音事件检测（SED）。它使声音事件沿频率轴具有平移等变性，而频率轴不是平移不变维。为了解决这个问题，使用动态卷积来对声音事件的频率依赖性进行建模。在本文中，我们提出了第一个全动态的方法命名为全频动态卷积（FFDConv）。FFDConv为每个频带生成频率核，该频率核直接在频率相关建模的结构中设计。它在物理上提供了具有频率相关建模能力的2D卷积。FFDConv在DESED真实验证数据集上的PSDS 1测试结果比基线测试结果高出6.6%，且优于其他全动态方法。此外，通过可视化声音事件的特征，我们观察到FFDConv可以有效地提取特定频带中的相干特征，与声音事件的声音连续性一致。这证明了FFDConv具有很强的频率依赖感知能力。摘要：Recently, 2D convolution has been found unqualified in sound event detection (SED). It enforces translation equivariance on sound events along frequency axis, which is not a shift-invariant dimension. To address this issue, dynamic convolution is used to model the frequency dependency of sound events. In this paper, we proposed the first full-dynamic method named \emph{full-frequency dynamic convolution} (FFDConv). FFDConv generates frequency kernels for every frequency band, which is designed directly in the structure for frequency-dependent modeling. It physically furnished 2D convolution with the capability of frequency-dependent modeling. FFDConv outperforms not only the baseline by 6.6\% in DESED real validation dataset in terms of PSDS1, but outperforms the other full-dynamic methods. In addition, by visualizing features of sound events, we observed that FFDConv could effectively extract coherent features in specific frequency bands, consistent with the vocal continuity of sound events. This proves that FFDConv has great frequency-dependent perception ability.【4】 Self-supervised speech representation and contextual text embedding for match-mismatch classification with EEG recording标题：用于匹配-失配分类和脑电记录的自监督语音表示和上下文文本嵌入链接：https://arxiv.org/abs/2401.04964作者：Bo Wang,Xiran Xu,Zechen Zhang,Haolin Zhu,YuJie Yan,Xihong Wu,Jing Chen备注：2 pages, 2 figures, submitted to ICASSP 2024摘要：将语音与EEG相关联具有相当大的重要性，但具有挑战性。在这项研究中，深度卷积网络提取时空特征的EEG数据。自监督语音表示和上下文文本嵌入被用作语音特征。对比学习被用于将EEG特征与语音特征相关联。实验结果证明了使用自监督语音表示和上下文文本嵌入的好处。通过特征融合和模型集成，实现了60.29%的准确率，性能在听觉脑电挑战赛（ICASSP 2024）的任务1中排名第二。摘要：Relating speech to EEG holds considerable importance but challenging. In this study, deep convolutional network was employed to extract spatiotemporal features from EEG data. Self-supervised speech representation and contextual text embedding were used as speech features. Contrastive learning was used to related EEG features to speech features. The experimental results demonstrate the benefits of using self-supervised speech representation and contextual text embedding. Through feature fusion and model ensemble, an accuracy of 60.29% was achieved, and the performance was ranked as No.2 in Task1 of the Auditory EEG Challenge (ICASSP 2024).【5】 Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters标题：基于带适配器的自监督语音表示模型的抗噪零发声文语合成链接：https://arxiv.org/abs/2401.05111作者：Kenichi Fujita,Hiroshi Sato,Takanori Ashihara,Hiroki Kanagawa,Marc Delcroix,Takafumi Moriya,Yusuke Ijima备注：5 pages,3 figures, Accepted to IEEE ICASSP 2024摘要：zero-shot文语转换（TTS）方法基于使用自监督学习（SSL）语音表示从参考语音提取的说话人嵌入，可以非常准确地再现说话人特征。然而，当参考语音包含噪声时，该方法遭受语音合成质量的劣化。在本文中，我们提出了一种噪声鲁棒的zero-shot TTS方法。我们将适配器合并到SSL模型中，并使用噪声参考语音与TTS模型进行微调。此外，为了进一步提高性能，我们采用了语音增强（SE）前端。通过这些改进，我们提出的基于SSL的zero-shot TTS实现了高质量的语音合成与噪声参考语音。通过客观和主观评价，我们证实了所提出的方法对参考语音中的噪声具有很强的鲁棒性，并且有效地与SE相结合。摘要：The zero-shot text-to-speech (TTS) method, based on speaker embeddings extracted from reference speech using self-supervised learning (SSL) speech representations, can reproduce speaker characteristics very accurately. However, this approach suffers from degradation in speech synthesis quality when the reference speech contains noise. In this paper, we propose a noise-robust zero-shot TTS method. We incorporated adapters into the SSL model, which we fine-tuned with the TTS model using noisy reference speech. In addition, to further improve performance, we adopted a speech enhancement (SE) front-end. With these improvements, our proposed SSL-based zero-shot TTS achieved high-quality speech synthesis with noisy reference speech. Through the objective and subjective evaluations, we confirmed that the proposed method is highly robust to noise in reference speech, and effectively works in combination with SE.【6】 Singer Identity Representation Learning using Self-Supervised Techniques标题：基于自监督技术的歌手身份表征学习链接：https://arxiv.org/abs/2401.05064作者：Bernardo Torres,Stefan Lattner,Gaël Richard备注：None摘要：在使用语音数据创建语音身份表示方面已经取得了重大进展。然而，在歌唱方面没有取得同样的进展。为了弥合这一差距，我们提出了一个框架，用于训练歌手身份编码器，以提取适合于各种唱歌相关任务的表示，如唱歌的声音相似性和合成。我们在大量孤立的音轨上探索不同的自监督学习技术，并在训练过程中应用数据增强，以确保表示对音高和内容变化不变。我们评估了多个数据集上歌手相似性和识别任务的结果表示的质量，特别强调域外泛化。我们提出的框架产生了高质量的嵌入，在44.1 kHz下运行时，其性能优于说话人验证和wav2vec 2.0预先训练的歌唱声音基线。我们发布我们的代码和训练模型，以促进对歌唱声音和相关领域的进一步研究。摘要：Significant strides have been made in creating voice identity representations using speech data. However, the same level of progress has not been achieved for singing voices. To bridge this gap, we suggest a framework for training singer identity encoders to extract representations suitable for various singing-related tasks, such as singing voice similarity and synthesis. We explore different self-supervised learning techniques on a large collection of isolated vocal tracks and apply data augmentations during training to ensure that the representations are invariant to pitch and content variations. We evaluate the quality of the resulting representations on singer similarity and identification tasks across multiple datasets, with a particular emphasis on out-of-domain generalization. Our proposed framework produces high-quality embeddings that outperform both speaker verification and wav2vec 2.0 pre-trained baselines on singing voice while operating at 44.1 kHz. We release our code and trained models to facilitate further research on singing voice and related areas.【7】 MuTox: Universal MUltilingual Audio-based TOXicity Dataset and Zero-shot Detector标题：MuTox：基于音频的通用多语言毒性数据集和零射探测器链接：https://arxiv.org/abs/2401.05060作者：Marta R. Costa-jussà,Mariano Coria Meglioli,Pierre Andrews,David Dale,Prangthip Hansanti,Elahe Kalbassi,Alex Mourachko,Christophe Ropers,Carleigh Wood摘要：语音模态（基于音频）的自然语言处理中的毒性检测研究非常有限，特别是对于英语以外的语言。为了解决这些限制并为真正的多语言音频毒性检测奠定基础，我们引入了MuTox，这是第一个带有毒性标签的高度多语言音频数据集。该数据集包括英语和西班牙语的20，000个音频话语，以及其他19种语言的4，000个。为了证明该数据集的质量，我们训练了基于MuTox音频的毒性分类器，该分类器能够在各种语言中进行zero-shot毒性检测。该分类器比现有的基于文本的可训练分类器的AUC高出1%以上，同时将语言覆盖范围扩大了十倍以上。与覆盖类似语言数量的基于单词列表的分类器相比，MuTox将精确度和召回率提高了约2.5倍。这一重大改进强调了MuTox在推进基于音频的毒性检测领域的潜力。摘要：Research in toxicity detection in natural language processing for the speech modality (audio-based) is quite limited, particularly for languages other than English. To address these limitations and lay the groundwork for truly multilingual audio-based toxicity detection, we introduce MuTox, the first highly multilingual audio-based dataset with toxicity labels. The dataset comprises 20,000 audio utterances for English and Spanish, and 4,000 for the other 19 languages. To demonstrate the quality of this dataset, we trained the MuTox audio-based toxicity classifier, which enables zero-shot toxicity detection across a wide range of languages. This classifier outperforms existing text-based trainable classifiers by more than 1% AUC, while expanding the language coverage more than tenfold. When compared to a wordlist-based classifier that covers a similar number of languages, MuTox improves precision and recall by approximately 2.5 times. This significant improvement underscores the potential of MuTox in advancing the field of audio-based toxicity detection.【8】 Learning Audio Concepts from Counterfactual Natural Language标题：从反事实自然语言中学习音频概念链接：https://arxiv.org/abs/2401.04935作者：Ali Vosoughi,Luca Bondi,Ho-Hsiang Wu,Chenliang Xu备注：Accepted at ICASSP 2024摘要：传统的音频分类依赖于预定义的类，缺乏从自由形式文本中学习的能力。最近的方法从描述自然语言中音频的原始音频文本对中解锁学习联合音频文本嵌入。尽管最近取得了进展，但很少探索系统的方法来训练模型，以识别替代场景中的声音事件和来源，例如在类似情况下区分户外活动中的烟花和枪声。本研究介绍了音频领域的因果推理和反事实分析。我们使用反事实实例，并将它们包含在我们的模型中的不同方面。我们的模型考虑了人类注释的参考文本的声学特性和声源信息。为了验证我们模型的有效性，我们利用多个音频字幕数据集进行了预训练。然后，我们用几个常见的下游任务进行评估，证明了所提出的方法作为利用音频域中的反事实信息的第一个作品之一的优点。具体来说，开放式语言音频检索任务的前1名准确率提高了43%以上。摘要：Conventional audio classification relied on predefined classes, lacking the ability to learn from free-form text. Recent methods unlock learning joint audio-text embeddings from raw audio-text pairs describing audio in natural language. Despite recent advancements, there is little exploration of systematic methods to train models for recognizing sound events and sources in alternative scenarios, such as distinguishing fireworks from gunshots at outdoor events in similar situations. This study introduces causal reasoning and counterfactual analysis in the audio domain. We use counterfactual instances and include them in our model across different aspects. Our model considers acoustic characteristics and sound source information from human-annotated reference texts. To validate the effectiveness of our model, we conducted pre-training utilizing multiple audio captioning datasets. We then evaluate with several common downstream tasks, demonstrating the merits of the proposed method as one of the first works leveraging counterfactual information in audio domain. Specifically, the top-1 accuracy in open-ended language-based audio retrieval task increased by more than 43%.【9】 Real-time and Continuous Turn-taking Prediction Using Voice Activity Projection标题：基于语音活动投影的实时连续话轮切换预测链接：https://arxiv.org/abs/2401.04868作者：Koji Inoue,Bing'er Jiang,Erik Ekstedt,Tatsuya Kawahara,Gabriel Skantze备注：This paper has been accepted for presentation at International Workshop on Spoken Dialogue Systems Technology 2024 (IWSDS 2024) and represents the author's version of the work摘要：一个实时和连续的话轮转换预测系统的演示。该系统基于语音活动投影（VAP）模型，该模型直接将对话立体声音频映射到未来的语音活动。VAP模型包括对比预测编码（CPC）和自我注意Transformers，然后是交叉注意变换器。我们研究了输入上下文音频长度的影响，并证明了所提出的系统可以在CPU设置的实时操作，以最小的性能下降。摘要：A demonstration of a real-time and continuous turn-taking prediction system is presented. The system is based on a voice activity projection (VAP) model, which directly maps dialogue stereo audio to future voice activities. The VAP model includes contrastive predictive coding (CPC) and self-attention transformers, followed by a cross-attention transformer. We examine the effect of the input context audio length and demonstrate that the proposed system can operate in real-time with CPU settings, with minimal performance degradation.【10】 DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation标题：DiffSHEG：一种基于扩散的实时语音驱动整体3D表情和手势生成方法链接：https://arxiv.org/abs/2401.04747作者：Junming Chen,Yunfei Liu,Jianan Wang,Ailing Zeng,Yu Li,Qifeng Chen摘要：我们提出了DiffSHEG，一种基于扩散的方法，用于语音驱动的整体3D表达和任意长度的手势生成。虽然以前的作品集中在共同语音手势或表达生成单独的，同步的表情和手势的联合生成仍然很少探索。为了解决这个问题，我们的基于扩散的共同语音运动生成Transformer实现了从表情到手势的单向信息流，促进了联合表情-手势分布的改进匹配。此外，我们引入了一个基于outpainting的采样策略，用于扩散模型中的任意长序列生成，提供了灵活性和计算效率。我们的方法提供了一个实用的解决方案，产生高质量的同步表达和手势生成语音驱动。在两个公共数据集上进行评估，我们的方法在定量和定性方面都达到了最先进的性能。此外，一项用户研究证实了DiffSHEG优于以前的方法。通过实现表达和同步运动的实时生成，DiffSHEG展示了其在数字人类和具体代理开发中的各种应用的潜力。摘要：We propose DiffSHEG, a Diffusion-based approach for Speech-driven Holistic 3D Expression and Gesture generation with arbitrary length. While previous works focused on co-speech gesture or expression generation individually, the joint generation of synchronized expressions and gestures remains barely explored. To address this, our diffusion-based co-speech motion generation transformer enables uni-directional information flow from expression to gesture, facilitating improved matching of joint expression-gesture distributions. Furthermore, we introduce an outpainting-based sampling strategy for arbitrary long sequence generation in diffusion models, offering flexibility and computational efficiency. Our method provides a practical solution that produces high-quality synchronized expression and gesture generation driven by speech. Evaluated on two public datasets, our approach achieves state-of-the-art performance both quantitatively and qualitatively. Additionally, a user study confirms the superiority of DiffSHEG over prior approaches. By enabling the real-time generation of expressive and synchronized motions, DiffSHEG showcases its potential for various applications in the development of digital humans and embodied agents.【11】 Music Genre Classification: A Comparative Analysis of CNN and XGBoost Approaches with Mel-frequency cepstral coefficients and Mel Spectrograms标题：音乐类型分类：基于Mel倒谱系数和Mel谱图的CNN和XGBoost方法的比较分析链接：https://arxiv.org/abs/2401.04737作者：Yigang Meng摘要：近年来，各种精心设计的算法使音乐平台能够根据个人喜好提供内容。音乐类型是从各个方面来定义的，包括声学特征和文化因素。音乐流派分类与基于内容的过滤配合得很好，后者根据音乐相似性向用户推荐内容。给定一个相当大的数据集，一个前提是使用机器学习或深度学习方法进行自动注释，可以有效地对音频文件进行分类。系统的有效性在很大程度上取决于特征和模型的选择，因为不同的架构和特征可以相互促进并产生不同的结果。在这项研究中，我们进行了一项比较研究，调查三种模型的性能：一个建议的卷积神经网络（CNN），VGG16与全连接层（FC），和一个极端梯度提升（XGBoost）的方法对不同的功能：30秒梅尔频谱图和3秒梅尔频率倒谱系数（MFCC）。结果表明，MFCC XGBoost模型的性能优于其他模型。此外，在数据预处理阶段应用数据分割可以显着提高CNN的性能。摘要：In recent years, various well-designed algorithms have empowered music platforms to provide content based on one's preferences. Music genres are defined through various aspects, including acoustic features and cultural considerations. Music genre classification works well with content-based filtering, which recommends content based on music similarity to users. Given a considerable dataset, one premise is automatic annotation using machine learning or deep learning methods that can effectively classify audio files. The effectiveness of systems largely depends on feature and model selection, as different architectures and features can facilitate each other and yield different results. In this study, we conduct a comparative study investigating the performances of three models: a proposed convolutional neural network (CNN), the VGG16 with fully connected layers (FC), and an eXtreme Gradient Boosting (XGBoost) approach on different features: 30-second Mel spectrogram and 3-second Mel-frequency cepstral coefficients (MFCCs). The results show that the MFCC XGBoost model outperformed the others. Furthermore, applying data segmentation in the data preprocessing phase can significantly enhance the performance of the CNNs.机器翻译由腾讯交互翻译提供，仅供参考
每日语音速递
