今日论文合集：cs.SD语音8篇，eess.AS音频处理10篇。
cs.SD语音
【1】 tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models标题：TinyCLAP：提取对比语言--音频预训练模型链接：https://arxiv.org/abs/2311.14517
作者：Francesco Paissan,Elisabetta Farella摘要：对比存储-音频预训练（CLAP）在音频和语音处理领域具有重要意义。它的应用范围从声音事件检测到文本到音频生成。然而，其中一个主要的限制是训练过程中需要大量的数据和推理过程中的整体计算复杂性。本文研究了如何降低对比性语言音频预训练模型的复杂性，从而产生一个高效的模型，我们称之为tinyCLAP。我们从第一原理推导出单峰蒸馏损失，并探讨如何通过修剪减少共享的多模态潜在空间的维数。TinyCLAP仅使用原始Microsoft CLAP参数的6%，在测试的三个声音事件检测数据集上，zero-shot分类性能降低最小（小于5%摘要：Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. TinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested
【2】 Overview Of The 2023 Icassp Sp Clarity Challenge: Speech Enhancement For  Hearing Aids标题：2023年Icassp Sp清晰度挑战综述：助听器语音增强链接：https://arxiv.org/abs/2311.14490
作者：Trevor J. Cox,Jon Barker,Will Bailey,Simone Graetzer,Michael A. Akeroyd,John F. Culling,Graham Naylor备注：ICASSP 2023摘要：本文报告了ICASSP SP清晰度挑战赛的设计和结果：助听器的语音增强。场景是一个听众在嘈杂的家庭环境中倾听目标说话者。有多个听众和头部旋转。该挑战通过固定助听器的放大阶段扩展了第二个清晰度增强挑战（CEC 2）;使用语音清晰度和质量的组合度量进行评估;并提供两个评估集，一个基于模拟，另一个基于真实房间测量。五个团队改进了模拟评估集的基线系统，但测量评估集的性能要差得多。调查正在进行中，以确定模拟和测量数据集之间不匹配的确切原因。测量中存在换能器噪声、低阶高保真度立体声响复制损害系统利用双耳线索的能力以及真实和模拟房间脉冲响应之间的差异是建议的原因摘要：This paper reports on the design and outcomes of the ICASSP SP Clarity Challenge: Speech Enhancement for Hearing Aids. The scenario was a listener attending to a target speaker in a noisy, domestic environment. There were multiple interferers and head rotation by the listener. The challenge extended the second Clarity Enhancement Challenge (CEC2) by fixing the amplification stage of the hearing aid; evaluating with a combined metric for speech intelligibility and quality; and providing two evaluation sets, one based on simulation and the other on real-room measurements. Five teams improved on the baseline system for the simulated evaluation set, but the performance on the measured evaluation set was much poorer. Investigations are on-going to determine the exact cause of the mismatch between the simulated and measured data sets. The presence of transducer noise in the measurements, lower order Ambisonics harming the ability for systems to exploit binaural cues and the differences between real and simulated room impulse responses are suggested causes
【3】 Cooperative Dual Attention for Audio-Visual Speech Enhancement with  Facial Cues标题：基于人脸线索的协同双注意视听语音增强链接：https://arxiv.org/abs/2311.14275
作者：Feixiang Wang,Shuang Yang,Shiguang Shan,Xilin Chen备注：Accepted to BMVC 2023 15 pages, 2 figures摘要：在这项工作中，我们专注于利用面部线索超出嘴唇区域强大的视听语音增强（AVSE）。包括嘴唇区域的面部区域反映了额外的语音相关属性，例如性别、肤色、国籍等，这有助于AVSE的有效性。然而，静态和动态的语音无关的属性也存在，导致外观的变化，在讲话。为了解决这些挑战，我们提出了一个双重注意力合作框架，DualAVSE，忽略语音无关的信息，捕捉语音相关的信息与面部线索，并动态地将其与音频信号AVSE。具体来说，我们引入了一个基于空间注意力的视觉编码器来捕获和增强嘴唇区域以外的视觉语音信息，结合全局面部上下文，并自动忽略语音无关的信息，以实现鲁棒的视觉特征提取。此外，一个动态的视觉特征融合策略，通过整合一个时间维的自我注意模块，使模型能够鲁棒地处理人脸变化。说话过程中的声学噪声是可变的，影响音频质量。因此，一个动态的融合策略，为音频和视觉功能的介绍，以解决这个问题。通过在视觉编码器和视听融合策略中集成合作双注意，我们的模型有效地从音频和视觉线索中提取有益的语音信息，用于AVSE。对不同数据集的彻底分析和比较，包括具有不可靠或缺乏视觉信息的正常和具有挑战性的情况，一致表明我们的模型在多个指标上优于现有方法。摘要：In this work, we focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE). The facial region, encompassing the lip region, reflects additional speech-related attributes such as gender, skin color, nationality, etc., which contribute to the effectiveness of AVSE. However, static and dynamic speech-unrelated attributes also exist, causing appearance changes during speech. To address these challenges, we propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE. Specifically, we introduce a spatial attention-based visual encoder to capture and enhance visual speech information beyond the lip region, incorporating global facial context and automatically ignoring speech-unrelated information for robust visual feature extraction. Additionally, a dynamic visual feature fusion strategy is introduced by integrating a temporal-dimensional self-attention module, enabling the model to robustly handle facial variations. The acoustic noise in the speaking process is variable, impacting audio quality. Therefore, a dynamic fusion strategy for both audio and visual features is introduced to address this issue. By integrating cooperative dual attention in the visual encoder and audio-visual fusion strategy, our model effectively extracts beneficial speech information from both audio and visual cues for AVSE. Thorough analysis and comparison on different datasets, including normal and challenging cases with unreliable or absent visual information, consistently show our model outperforming existing methods across multiple metrics.
【4】 Allpass impulse response modelling标题：全通脉冲响应建模链接：https://arxiv.org/abs/2311.14239
作者：Matt R. Flax摘要：本文档定义了一种FIR系统建模方法，该方法非常简单，因为它仅依赖于相位引入和移除（全通滤波器）。由于幅度不改变，处理在数值上是稳定的。它仅限于相位改变，其保持时域幅度以迫使系统处于其线性极限内。摘要：This document defines a method for FIR system modelling which is very trivial as it only depends on phase introduction and removal (allpass filters). As magnitude is not altered, the processing is numerically stable. It is limited to phase alteration which maintains the time domain magnitude to force a system within its linear limits.
【5】 Learning to Solve Inverse Problems for Perceptual Sound Matching标题：学习解决知觉声音匹配的逆问题链接：https://arxiv.org/abs/2311.14213
作者：Han Han,Vincent Lostanlen,Mathieu Lagrange摘要：感知声音匹配（PSM）的目的是找到合成器的输入参数，以便最好地模仿音频目标。PSM的深度学习优化了神经网络，以分析和重建预先记录的样本。在这种情况下，我们的文章解决的问题，设计一个合适的损失函数时，由一个可微合成器生成的训练集。我们的主要贡献是感知神经物理损失（PNP），其目的是解决感知相关性和计算效率之间的权衡。PNP背后的关键思想是线性化合成参数对每个训练样本附近的听觉特征的影响。线性化过程是大规模可线性化的，可以预先计算，并提供了100倍的速度比可微分数字信号处理（DDSP）在梯度下降。我们证明PNP的两个数据集的非平稳的声音：AM/FM琶音器和矩形膜的物理模型。我们表明，PNP能够加速DDSP与联合时频散射变换（JTFS）的听觉特征，同时保持其感知保真度。此外，我们还评估了PSM中其他设计选择的影响：参数重新调整、预训练、听觉表示和梯度裁剪。我们在两个数据集上报告了最先进的结果，发现PNP加速的JTFS对PSM性能的影响比任何其他设计选择都大。摘要：Perceptual sound matching (PSM) aims to find the input parameters to a synthesizer so as to best imitate an audio target. Deep learning for PSM optimizes a neural network to analyze and reconstruct prerecorded samples. In this context, our article addresses the problem of designing a suitable loss function when the training set is generated by a differentiable synthesizer. Our main contribution is perceptual-neural-physical loss (PNP), which aims at addressing a tradeoff between perceptual relevance and computational efficiency. The key idea behind PNP is to linearize the effect of synthesis parameters upon auditory features in the vicinity of each training sample. The linearization procedure is massively paralellizable, can be precomputed, and offers a 100-fold speedup during gradient descent compared to differentiable digital signal processing (DDSP). We demonstrate PNP on two datasets of nonstationary sounds: an AM/FM arpeggiator and a physical model of rectangular membranes. We show that PNP is able to accelerate DDSP with joint time-frequency scattering transform (JTFS) as auditory feature, while preserving its perceptual fidelity. Additionally, we evaluate the impact of other design choices in PSM: parameter rescaling, pretraining, auditory representation, and gradient clipping. We report state-of-the-art results on both datasets and find that PNP-accelerated JTFS has greater influence on PSM performance than any other design choice.
【6】 Beat-Aligned Spectrogram-to-Sequence Generation of Rhythm-Game Charts标题：节拍对齐的节律游戏曲线图的逐次生成链接：https://arxiv.org/abs/2311.13687
作者：Jayeon Yi,Sungho Lee,Kyogu Lee备注：ISMIR 2023 LBD. Demo videos and code at stet-stet.github.io/goct摘要：在“节奏游戏”的核心--玩家必须与一段音乐同步执行动作的游戏--是“图表”，即给玩家的指令。我们新制定的图表生成作为一个序列生成任务和训练一个Transformer使用一个大的数据集。我们还介绍了节奏知情的预处理和培训程序，其中一些建议是一个成功的培训不可或缺的。我们的模型在大型数据集上的表现优于基线，并且还受益于预训练和微调。摘要：In the heart of "rhythm games" - games where players must perform actions in sync with a piece of music - are "charts", the directives to be given to players. We newly formulate chart generation as a sequence generation task and train a Transformer using a large dataset. We also introduce tempo-informed preprocessing and training procedures, some of which are suggested to be integral for a successful training. Our model is found to outperform the baselines on a large dataset, and is also found to benefit from pretraining and finetuning.
【7】 SER_AMPEL: A multi-source dataset for SER of Italian older adults标题：Ser_Ampel：意大利老年人SER的多源数据集链接：https://arxiv.org/abs/2311.14483
作者：Alessandra Grossi,Francesca Gasparini备注：11 pages, 1 Figure, 7 Tables, submitted to ForItAAL 2023 (12{\deg} Forum Italiano Ambient Assisted Living)摘要：本文提出了一种用于语音情感识别的多源数据集SER_AMPEL。该数据集的特点是，它是收集的目的是提供一个参考的情况下，意大利老年人的语音情感识别。数据集是按照不同的协议收集的，特别是考虑到从电影和电视剧中提取的表演对话，以及记录通过适当的问题引发情感的自然对话。需要这样一个数据集的证据出现从分析的最先进的。初步考虑SER的关键问题的报告分析的分类结果的一个子集的建议的数据集。摘要：In this paper, SER_AMPEL, a multi-source dataset for speech emotion recognition (SER) is presented. The peculiarity of the dataset is that it is collected with the aim of providing a reference for speech emotion recognition in case of Italian older adults. The dataset is collected following different protocols, in particular considering acted conversations, extracted from movies and TV series, and recording natural conversations where the emotions are elicited by proper questions. The evidence of the need for such a dataset emerges from the analysis of the state of the art. Preliminary considerations on the critical issues of SER are reported analyzing the classification results on a subset of the proposed dataset.
【8】 Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark标题：Jam-alt：一种支持格式的歌词转录基准链接：https://arxiv.org/abs/2311.13987
作者：Ondřej Cífka,Constantinos Dimitriou,Cheng-i Wang,Hendrik Schreiber,Luke Miner,Fabian-Robert Stöter备注：6 pages (3 pages main content); website: this https URL; data: this https URL; code: this https URL摘要：目前的自动歌词转录（ALT）基准只关注单词内容，而忽略了书面歌词的细微差别，包括格式和标点符号，这导致了与音乐家和词曲作者的创意产品以及听众体验的潜在不一致。例如，断行在传达有关节奏、情感重点、韵律和高级结构的信息方面很重要。为了解决这个问题，我们介绍了Jam-ALT，一个新的歌词转录基准的基础上JamendoLyrics数据集。我们的贡献是双重的。首先，对成绩单进行全面修订，专门针对ALT评估，遵循新创建的注释指南，该指南统一了音乐行业的指导方针，涵盖了标点符号，换行符，拼写，背景人声和非单词声音等方面。其次，一套评估指标的设计，不同于传统的字错误率，捕捉这种现象。我们希望拟议的基准有助于ALT任务，使转录系统的评估更加精确和可靠，并增强歌词应用程序的用户体验，如实时字幕或卡拉OK的字幕渲染。摘要：Current automatic lyrics transcription (ALT) benchmarks focus exclusively on word content and ignore the finer nuances of written lyrics including formatting and punctuation, which leads to a potential misalignment with the creative products of musicians and songwriters as well as listeners' experiences. For example, line breaks are important in conveying information about rhythm, emotional emphasis, rhyme, and high-level structure. To address this issue, we introduce Jam-ALT, a new lyrics transcription benchmark based on the JamendoLyrics dataset. Our contribution is twofold. Firstly, a complete revision of the transcripts, geared specifically towards ALT evaluation by following a newly created annotation guide that unifies the music industry's guidelines, covering aspects such as punctuation, line breaks, spelling, background vocals, and non-word sounds. Secondly, a suite of evaluation metrics designed, unlike the traditional word error rate, to capture such phenomena. We hope that the proposed benchmark contributes to the ALT task, enabling more precise and reliable assessments of transcription systems and enhancing the user experience in lyrics applications such as subtitle renderings for live captioning or karaoke.
eess.AS音频处理
【1】 SER_AMPEL: A multi-source dataset for SER of Italian older adults标题：Ser_Ampel：意大利老年人SER的多源数据集链接：https://arxiv.org/abs/2311.14483
作者：Alessandra Grossi,Francesca Gasparini备注：11 pages, 1 Figure, 7 Tables, submitted to ForItAAL 2023 (12{\deg} Forum Italiano Ambient Assisted Living)摘要：本文提出了一种用于语音情感识别的多源数据集SER_AMPEL。该数据集的特点是，它是收集的目的是提供一个参考的情况下，意大利老年人的语音情感识别。数据集是按照不同的协议收集的，特别是考虑到从电影和电视剧中提取的表演对话，以及记录通过适当的问题引发情感的自然对话。需要这样一个数据集的证据出现从分析的最先进的。初步考虑SER的关键问题的报告分析的分类结果的一个子集的建议的数据集。摘要：In this paper, SER_AMPEL, a multi-source dataset for speech emotion recognition (SER) is presented. The peculiarity of the dataset is that it is collected with the aim of providing a reference for speech emotion recognition in case of Italian older adults. The dataset is collected following different protocols, in particular considering acted conversations, extracted from movies and TV series, and recording natural conversations where the emotions are elicited by proper questions. The evidence of the need for such a dataset emerges from the analysis of the state of the art. Preliminary considerations on the critical issues of SER are reported analyzing the classification results on a subset of the proposed dataset.
【2】 Interactive Dual Attention Conformer With Scene-Based Mask For Soft  Sound Event Detection标题：基于场景掩模的交互式双注意力一致性软音事件检测链接：https://arxiv.org/abs/2311.14068
作者：Han Yin,Jisheng Bai,Mou Wang,Dongyuan Shi,Woon-Seng Gan,Jianfeng Chen备注：Submitted to ICASSP 2024摘要：用于声音事件检测的软标记数据的出现有效地克服了传统强标记数据的不足。然而，基于这种软标签的当前SED系统的性能仍然不能令人满意。在这项工作中，我们引入了一个双分支SED模型，旨在利用软标签内的信息。提出了四种相互作用的卷积模块，以研究信息交互的有效机制。此外，我们将估计器生成的基于场景的掩码直接应用于SED模型的预测。实验结果表明，该模板估计器可以获得与手工设计的模板相当甚至更好的性能，并显着提高SED的性能。该方法在DCASE 2023 Task4B挑战赛中获得了最高排名。摘要：The emergence of soft-labeled data for sound event detection (SED) effectively overcomes the lack of traditional strong-labeled data. However, the performance of present SED systems based on such soft labels is still unsatisfactory. In this work, we introduce a dual-branch SED model designed to leverage the information within soft labels. Four variations of the interacted convolutional module are presented to investigate the effective mechanism for information interaction. Furthermore, we incorporate the scene-based mask generated by an estimator to directly apply to the prediction of SED models. Experimental results show that the mask estimator can achieve comparable or even better performance than the manually-designed mask and significantly improve the performance of SED. The proposed approach achieved the top ranking in the DCASE 2023 Task4B Challenge.
【3】 Jam-ALT: A Formatting-Aware Lyrics Transcription Benchmark标题：Jam-alt：一种支持格式的歌词转录基准链接：https://arxiv.org/abs/2311.13987
作者：Ondřej Cífka,Constantinos Dimitriou,Cheng-i Wang,Hendrik Schreiber,Luke Miner,Fabian-Robert Stöter备注：6 pages (3 pages main content); website: this https URL; data: this https URL; code: this https URL摘要：目前的自动歌词转录（ALT）基准只关注单词内容，而忽略了书面歌词的细微差别，包括格式和标点符号，这导致了与音乐家和词曲作者的创意产品以及听众体验的潜在不一致。例如，断行在传达有关节奏、情感重点、韵律和高级结构的信息方面很重要。为了解决这个问题，我们介绍了Jam-ALT，一个新的歌词转录基准的基础上JamendoLyrics数据集。我们的贡献是双重的。首先，对成绩单进行全面修订，专门针对ALT评估，遵循新创建的注释指南，该指南统一了音乐行业的指导方针，涵盖了标点符号，换行符，拼写，背景人声和非单词声音等方面。其次，一套评估指标的设计，不同于传统的字错误率，捕捉这种现象。我们希望拟议的基准有助于ALT任务，使转录系统的评估更加精确和可靠，并增强歌词应用程序的用户体验，如实时字幕或卡拉OK的字幕渲染。摘要：Current automatic lyrics transcription (ALT) benchmarks focus exclusively on word content and ignore the finer nuances of written lyrics including formatting and punctuation, which leads to a potential misalignment with the creative products of musicians and songwriters as well as listeners' experiences. For example, line breaks are important in conveying information about rhythm, emotional emphasis, rhyme, and high-level structure. To address this issue, we introduce Jam-ALT, a new lyrics transcription benchmark based on the JamendoLyrics dataset. Our contribution is twofold. Firstly, a complete revision of the transcripts, geared specifically towards ALT evaluation by following a newly created annotation guide that unifies the music industry's guidelines, covering aspects such as punctuation, line breaks, spelling, background vocals, and non-word sounds. Secondly, a suite of evaluation metrics designed, unlike the traditional word error rate, to capture such phenomena. We hope that the proposed benchmark contributes to the ALT task, enabling more precise and reliable assessments of transcription systems and enhancing the user experience in lyrics applications such as subtitle renderings for live captioning or karaoke.
【4】 End-to-end Transfer Learning for Speaker-independent Cross-language  Speech Emotion Recognition标题：端到端迁移学习用于非特定人跨语言语音情感识别链接：https://arxiv.org/abs/2311.13678
作者：Duowei Tang,Peter Kuppens,Luc Geurts,Toon van Waterschoot备注：15 pages, 6 figures, 4 tables摘要：数据驱动模型在语音情感识别（SER）中取得了成功的结果。然而，这些基于一般声学特征或端到端方法的模型在测试集具有不同语言（即，跨语言设置）而不是训练集或者当它们来自不同的数据集时（即，跨语料库设置）。针对这一问题，本文提出了一种基于迁移学习的端到端深度神经网络（DNN）模型，用于跨语言SER识别。我们使用wav 2 vec 2.0预训练模型将不同语言、不同说话人、不同录音条件下的音频时域波形转换到多语言共享的特征空间中，从而降低语音特征中的语言变异性。接下来，我们提出了一个新的Deep-Within-Class Co-variance Normalisation（Deep-WCCN）层，可以插入到DNN模型中，它旨在减少其他可变性，包括扬声器可变性，通道可变性等。整个模型以端到端的方式对组合损失进行微调，并在来自三种语言的数据集上进行验证（即，英文、德文、中文）。实验结果表明，我们提出的方法不仅优于基线模型的基础上，共同的声学特征集SER在语言内设置，但也显着优于基线模型的跨语言设置。此外，我们还通过实验验证了Deep-WCCN的有效性，可以进一步提高模型的性能。最后，通过与使用相同测试数据集的最新文献的结果比较，我们提出的模型在跨语言SER中表现出明显优于其他最先进的模型的性能。摘要：Data-driven models achieve successful results in Speech Emotion Recognition (SER). However, these models, which are based on general acoustic features or end-to-end approaches, show poor performance when the testing set has a different language (i.e., the cross-language setting) than the training set or when they come from a different dataset (i.e., the cross-corpus setting). To alleviate this problem, this paper presents an end-to-end Deep Neural Network (DNN) model based on transfer learning for cross-language SER. We use the wav2vec 2.0 pre-trained model to transform audio time-domain waveforms from different languages, different speakers and different recording conditions into a feature space shared by multiple languages, thereby it reduces the language variabilities in the speech features. Next, we propose a new Deep-Within-Class Co-variance Normalisation (Deep-WCCN) layer that can be inserted into the DNN model and it aims to reduce other variabilities including speaker variability, channel variability and so on. The whole model is fine-tuned in an end-to-end manner on a combined loss and is validated on datasets from three languages (i.e., English, German, Chinese). Experiment results show that our proposed method not only outperforms the baseline model that is based on common acoustic feature sets for SER in the within-language setting, but also significantly outperforms the baseline model for cross-language setting. In addition, we also experimentally validate the effectiveness of Deep-WCCN, which can further improve the model performance. Finally, to comparing the results in the recent literatures that use the same testing datasets, our proposed model shows significantly better performance than other state-of-the-art models in cross-language SER.
【5】 tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models标题：TinyCLAP：提取对比语言--音频预训练模型链接：https://arxiv.org/abs/2311.14517
作者：Francesco Paissan,Elisabetta Farella摘要：对比存储-音频预训练（CLAP）在音频和语音处理领域具有重要意义。它的应用范围从声音事件检测到文本到音频生成。然而，其中一个主要的限制是训练过程中需要大量的数据和推理过程中的整体计算复杂性。本文研究了如何降低对比性语言音频预训练模型的复杂性，从而产生一个高效的模型，我们称之为tinyCLAP。我们从第一原理推导出单峰蒸馏损失，并探讨如何通过修剪减少共享的多模态潜在空间的维数。TinyCLAP仅使用原始Microsoft CLAP参数的6%，在测试的三个声音事件检测数据集上，zero-shot分类性能降低最小（小于5%摘要：Contrastive Language-Audio Pretraining (CLAP) became of crucial importance in the field of audio and speech processing. Its employment ranges from sound event detection to text-to-audio generation. However, one of the main limitations is the considerable amount of data required in the training process and the overall computational complexity during inference. This paper investigates how we can reduce the complexity of contrastive language-audio pre-trained models, yielding an efficient model that we call tinyCLAP. We derive an unimodal distillation loss from first principles and explore how the dimensionality of the shared, multimodal latent space can be reduced via pruning. TinyCLAP uses only 6% of the original Microsoft CLAP parameters with a minimal reduction (less than 5%) in zero-shot classification performance across the three sound event detection datasets on which it was tested
【6】 Overview Of The 2023 Icassp Sp Clarity Challenge: Speech Enhancement For  Hearing Aids标题：2023年Icassp Sp清晰度挑战综述：助听器语音增强链接：https://arxiv.org/abs/2311.14490
作者：Trevor J. Cox,Jon Barker,Will Bailey,Simone Graetzer,Michael A. Akeroyd,John F. Culling,Graham Naylor备注：ICASSP 2023摘要：None摘要：This paper reports on the design and outcomes of the ICASSP SP Clarity Challenge: Speech Enhancement for Hearing Aids. The scenario was a listener attending to a target speaker in a noisy, domestic environment. There were multiple interferers and head rotation by the listener. The challenge extended the second Clarity Enhancement Challenge (CEC2) by fixing the amplification stage of the hearing aid; evaluating with a combined metric for speech intelligibility and quality; and providing two evaluation sets, one based on simulation and the other on real-room measurements. Five teams improved on the baseline system for the simulated evaluation set, but the performance on the measured evaluation set was much poorer. Investigations are on-going to determine the exact cause of the mismatch between the simulated and measured data sets. The presence of transducer noise in the measurements, lower order Ambisonics harming the ability for systems to exploit binaural cues and the differences between real and simulated room impulse responses are suggested causes
【7】 Cooperative Dual Attention for Audio-Visual Speech Enhancement with  Facial Cues标题：基于人脸线索的协同双注意视听语音增强链接：https://arxiv.org/abs/2311.14275
作者：Feixiang Wang,Shuang Yang,Shiguang Shan,Xilin Chen备注：Accepted to BMVC 2023 15 pages, 2 figures摘要：在这项工作中，我们专注于利用面部线索超出嘴唇区域强大的视听语音增强（AVSE）。包括嘴唇区域的面部区域反映了额外的语音相关属性，例如性别、肤色、国籍等，这有助于AVSE的有效性。然而，静态和动态的语音无关的属性也存在，导致外观的变化，在讲话。为了解决这些挑战，我们提出了一个双重注意力合作框架，DualAVSE，忽略语音无关的信息，捕捉语音相关的信息与面部线索，并动态地将其与音频信号AVSE。具体来说，我们引入了一个基于空间注意力的视觉编码器来捕获和增强嘴唇区域以外的视觉语音信息，结合全局面部上下文，并自动忽略语音无关的信息，以实现鲁棒的视觉特征提取。此外，一个动态的视觉特征融合策略，通过整合一个时间维的自我注意模块，使模型能够鲁棒地处理人脸变化。说话过程中的声学噪声是可变的，影响音频质量。因此，一个动态的融合策略，为音频和视觉功能的介绍，以解决这个问题。通过在视觉编码器和视听融合策略中集成合作双注意，我们的模型有效地从音频和视觉线索中提取有益的语音信息，用于AVSE。对不同数据集的彻底分析和比较，包括具有不可靠或缺乏视觉信息的正常和具有挑战性的情况，一致表明我们的模型在多个指标上优于现有方法。摘要：In this work, we focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE). The facial region, encompassing the lip region, reflects additional speech-related attributes such as gender, skin color, nationality, etc., which contribute to the effectiveness of AVSE. However, static and dynamic speech-unrelated attributes also exist, causing appearance changes during speech. To address these challenges, we propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE. Specifically, we introduce a spatial attention-based visual encoder to capture and enhance visual speech information beyond the lip region, incorporating global facial context and automatically ignoring speech-unrelated information for robust visual feature extraction. Additionally, a dynamic visual feature fusion strategy is introduced by integrating a temporal-dimensional self-attention module, enabling the model to robustly handle facial variations. The acoustic noise in the speaking process is variable, impacting audio quality. Therefore, a dynamic fusion strategy for both audio and visual features is introduced to address this issue. By integrating cooperative dual attention in the visual encoder and audio-visual fusion strategy, our model effectively extracts beneficial speech information from both audio and visual cues for AVSE. Thorough analysis and comparison on different datasets, including normal and challenging cases with unreliable or absent visual information, consistently show our model outperforming existing methods across multiple metrics.
【8】 Allpass impulse response modelling标题：全通脉冲响应建模链接：https://arxiv.org/abs/2311.14239
作者：Matt R. Flax摘要：本文档定义了一种FIR系统建模方法，该方法非常简单，因为它仅依赖于相位引入和移除（全通滤波器）。由于幅度不改变，处理在数值上是稳定的。它仅限于相位改变，其保持时域幅度以迫使系统处于其线性极限内。摘要：This document defines a method for FIR system modelling which is very trivial as it only depends on phase introduction and removal (allpass filters). As magnitude is not altered, the processing is numerically stable. It is limited to phase alteration which maintains the time domain magnitude to force a system within its linear limits.
【9】 Learning to Solve Inverse Problems for Perceptual Sound Matching标题：学习解决感知声音匹配的逆问题链接：https://arxiv.org/abs/2311.14213
作者：Han Han,Vincent Lostanlen,Mathieu Lagrange摘要：感知声音匹配（PSM）的目的是找到合成器的输入参数，以便最好地模仿音频目标。PSM的深度学习优化了神经网络，以分析和重建预先记录的样本。在这种情况下，我们的文章解决的问题，设计一个合适的损失函数时，由一个可微合成器生成的训练集。我们的主要贡献是感知神经物理损失（PNP），其目的是解决感知相关性和计算效率之间的权衡。PNP背后的关键思想是线性化合成参数对每个训练样本附近的听觉特征的影响。线性化过程是大规模可线性化的，可以预先计算，并提供了100倍的速度比可微分数字信号处理（DDSP）在梯度下降。我们证明PNP的两个数据集的非平稳的声音：AM/FM琶音器和矩形膜的物理模型。我们表明，PNP能够加速DDSP与联合时频散射变换（JTFS）的听觉特征，同时保持其感知保真度。此外，我们还评估了PSM中其他设计选择的影响：参数重新调整、预训练、听觉表示和梯度裁剪。我们在两个数据集上报告了最先进的结果，发现PNP加速的JTFS对PSM性能的影响比任何其他设计选择都大。摘要：Perceptual sound matching (PSM) aims to find the input parameters to a synthesizer so as to best imitate an audio target. Deep learning for PSM optimizes a neural network to analyze and reconstruct prerecorded samples. In this context, our article addresses the problem of designing a suitable loss function when the training set is generated by a differentiable synthesizer. Our main contribution is perceptual-neural-physical loss (PNP), which aims at addressing a tradeoff between perceptual relevance and computational efficiency. The key idea behind PNP is to linearize the effect of synthesis parameters upon auditory features in the vicinity of each training sample. The linearization procedure is massively paralellizable, can be precomputed, and offers a 100-fold speedup during gradient descent compared to differentiable digital signal processing (DDSP). We demonstrate PNP on two datasets of nonstationary sounds: an AM/FM arpeggiator and a physical model of rectangular membranes. We show that PNP is able to accelerate DDSP with joint time-frequency scattering transform (JTFS) as auditory feature, while preserving its perceptual fidelity. Additionally, we evaluate the impact of other design choices in PSM: parameter rescaling, pretraining, auditory representation, and gradient clipping. We report state-of-the-art results on both datasets and find that PNP-accelerated JTFS has greater influence on PSM performance than any other design choice.
【10】 Beat-Aligned Spectrogram-to-Sequence Generation of Rhythm-Game Charts标题：节拍对齐的节律游戏曲线图的逐次生成链接：https://arxiv.org/abs/2311.13687
作者：Jayeon Yi,Sungho Lee,Kyogu Lee备注：ISMIR 2023 LBD. Demo videos and code at stet-stet.github.io/goct摘要：在“节奏游戏”的核心--玩家必须与一段音乐同步执行动作的游戏--是“图表”，即给玩家的指令。我们新制定的图表生成作为一个序列生成任务和训练一个Transformer使用一个大的数据集。我们还介绍了节奏知情的预处理和培训程序，其中一些建议是一个成功的培训不可或缺的。我们的模型在大型数据集上的表现优于基线，并且还受益于预训练和微调。摘要：In the heart of "rhythm games" - games where players must perform actions in sync with a piece of music - are "charts", the directives to be given to players. We newly formulate chart generation as a sequence generation task and train a Transformer using a large dataset. We also introduce tempo-informed preprocessing and training procedures, some of which are suggested to be integral for a successful training. Our model is found to outperform the baselines on a large dataset, and is also found to benefit from pretraining and finetuning.
机器翻译，仅供参考