【1】Contrastive Loss Based Frame-wise Feature disentanglement for Polyphonic Sound Event Detection标题：基于对比损失的帧级特征解纠缠多音事件检测链接：https://arxiv.org/abs/2401.05850作者：Yadong Guan,Jiqing Han,Hongwei Song,Wenjie Song,Guibin Zheng,Tieran Zheng,Yongjun He备注：accepted by icassp2024摘要：重叠声音事件在现实环境中普遍存在，但现有的端到端声音事件检测（SED）方法仍然难以有效地检测它们。一个关键的原因是，这些方法表示重叠的事件使用共享和纠缠帧的功能，这降低了功能的歧视。为了解决这个问题，我们提出了一个分离的特征学习框架来学习特定于类别的表示。具体来说，我们使用不同的投影仪来学习每个类别的帧特征。为了确保这些功能不包含其他类别的信息，我们最大限度地提高同一类别内的帧之间的共同信息，并提出了一个帧的对比损失。此外，考虑到所提出的方法使用的标记数据是有限的，我们提出了一种半监督的逐帧对比损失，可以利用大量的未标记数据来实现特征解纠缠。实验结果证明了该方法的有效性。摘要：Overlapping sound events are ubiquitous in real-world environments, but existing end-to-end sound event detection (SED) methods still struggle to detect them effectively. A critical reason is that these methods represent overlapping events using shared and entangled frame-wise features, which degrades the feature discrimination. To solve the problem, we propose a disentangled feature learning framework to learn a category-specific representation. Specifically, we employ different projectors to learn the frame-wise features for each category. To ensure that these feature does not contain information of other categories, we maximize the common information between frame-wise features within the same category and propose a frame-wise contrastive loss. In addition, considering that the labeled data used by the proposed method is limited, we propose a semi-supervised frame-wise contrastive loss that can leverage large amounts of unlabeled data to achieve feature disentanglement. The experimental results demonstrate the effectiveness of our method.【2】 Intuitive Control of Scraping and Rubbing Through Audio-tactile Synthesis标题：通过音频-触觉合成直观地控制刮擦和摩擦链接：https://arxiv.org/abs/2401.05757作者：Mitsuko Aramaki,Corentin Bernard,Richard Kronland-Martinet,Samuel Poirot,Sølvi Ystad备注：None摘要：对合成过程的直观控制是听觉感知和认知领域内的一个持续挑战。之前的声音建模与心理物理测试相结合的工作使我们的团队能够开发出一种合成器，该合成器基于声源的语义描述提供对动作和对象的直观控制。在这个演示中，我们提出了一个增强版本的合成器，其中我们添加了触觉刺激，以增加与模拟对象的真实连续摩擦相互作用（摩擦和刮擦）的感觉。这是有意义的，原因有几个。首先，它能够评估我们的声音模型在其他模态刺激存在下的真实性。其次，它能够比较触觉和听觉信号结构链接到相同的唤起，第三，它提供了一个工具来研究多模态感知和如何刺激从不同的方式应该结合起来，以提供逼真的用户界面。摘要：Intuitive control of synthesis processes is an ongoing challenge within the domain of auditory perception and cognition. Previous works on sound modelling combined with psychophysical tests have enabled our team to develop a synthesizer that provides intuitive control of actions and objects based on semantic descriptions for sound sources. In this demo we present an augmented version of the synthesizer in which we added tactile stimulations to increase the sensation of true continuous friction interactions (rubbing and scratching) with the simulated objects. This is of interest for several reasons. Firstly, it enables to evaluate the realism of our sound model in presence of stimulations from other modalities. Secondly it enables to compare tactile and auditory signal structures linked to the same evocation, and thirdly it provides a tool to investigate multimodal perception and how stimulations from different modalities should be combined to provide realistic user interfaces.【3】 HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition标题：HiCMAE：用于自监督视听情感识别的分层对比掩模自动编码器链接：https://arxiv.org/abs/2401.05698作者：Licai Sun,Zheng Lian,Bin Liu,Jianhua Tao备注：19 pages, 8 figures, 15 tables. Full Abstract is shown in the pdf file. Codes and models will be publicly available at this https URL摘要：近年来，视听情感识别（AVER）因其在创造情感智能机器中的关键作用而受到越来越多的关注。以前在这方面的努力主要是监督学习范式。尽管取得了重大进展，但由于AVER中长期存在的数据稀缺问题，监督学习正在遇到瓶颈。受自监督学习最新进展的启发，我们提出了分层对比掩蔽自动编码器（HiCMAE），这是一种新型的自监督框架，它利用大量未标记的视听数据进行大规模自监督预训练，以促进AVER的进步。在自监督视听表示学习的现有技术之后，HiCMAE采用两种主要形式的自监督进行预训练，即掩蔽数据建模和对比学习。与只关注顶层表征而忽视中间层的明确指导不同，HiCMAE开发了一种三管齐下的策略来促进分层视听特征学习并提高学习表征的整体质量。为了验证HiCMAE的有效性，我们在9个数据集上进行了广泛的实验，涵盖了分类和维度AVER任务。实验结果表明，我们的方法显着优于国家的最先进的监督和自我监督的视听方法，这表明HiCMAE是一个强大的视听情感表征学习器。代码和模型将在https://github.com/sunlicai/HiCMAE上公开。摘要：Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models will be publicly available at https://github.com/sunlicai/HiCMAE.【4】 UCorrect: An Unsupervised Framework for Automatic Speech Recognition Error Correction标题：UCorrect：一个用于自动语音识别纠错的无监督框架链接：https://arxiv.org/abs/2401.05689作者：Jiaxin Guo,Minghan Wang,Xiaosong Qiao,Daimeng Wei,Hengchao Shang,Zongyao Li,Zhengzhe Yu,Yinglu Li,Chang Su,Min Zhang,Shimin Tao,Hao Yang备注：Accepted in ICASSP 2023摘要：纠错技术已经被用来改进自动语音识别（ASR）模型的输出句子，并实现较低的单词错误率（WER）。以往的研究多采用端到端模型，对伪配对数据和原始配对数据有很强的依赖性。但是，当仅对伪配对数据进行预训练时，以前的模型对校正有负面影响。在对原始配对数据进行微调时，源端数据必须由经过良好训练的ASR模型转录，这需要大量时间并且不通用。在本文中，我们提出了UCorrect，一个用于ASR纠错的无监督检测器生成器框架。UCorrect不依赖于前面提到的训练数据。整个过程首先检测字符是否错误，然后产生候选字符，最后选择最可信的字符替换错误字符。在公开的AISHELL-1数据集和WenetSpeech数据集上的实验结果表明，UCorrect对ASR纠错的有效性：1）UCorrect显著降低了WER，在不进行微调的情况下WER降低了6.83%，微调后WER降低了14.29%;以及3）它是一种通用方法，因为它减少了具有不同解码策略的ASR模型的所有WER，并且减少了在不同规模数据集上训练的ASR模型的所有WER。摘要：Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve a lower word error rate (WER). Previous works usually adopt end-to-end models and has strong dependency on Pseudo Paired Data and Original Paired Data. But when only pre-training on Pseudo Paired Data, previous models have negative effect on correction. While fine-tuning on Original Paired Data, the source side data must be transcribed by a well-trained ASR model, which takes a lot of time and not universal. In this paper, we propose UCorrect, an unsupervised Detector-Generator-Selector framework for ASR Error Correction. UCorrect has no dependency on the training data mentioned before. The whole procedure is first to detect whether the character is erroneous, then to generate some candidate characters and finally to select the most confident one to replace the error character. Experiments on the public AISHELL-1 dataset and WenetSpeech dataset show the effectiveness of UCorrect for ASR error correction: 1) it achieves significant WER reduction, achieves 6.83\% even without fine-tuning and 14.29\% after fine-tuning; 2) it outperforms the popular NAR correction models by a large margin with a competitive low latency; and 3) it is an universal method, as it reduces all WERs of the ASR model with different decoding strategies and reduces all WERs of ASR models trained on different scale datasets.【5】 Self-Attention and Hybrid Features for Replay and Deep-Fake Audio Detection标题：用于重放和深度伪音频检测的自我注意和混合特征链接：https://arxiv.org/abs/2401.05614作者：Lian Huang,Chi-Man Pun摘要：由于深度学习的成功应用，音频欺骗检测取得了重大进展。具有语音合成或语音转换的欺骗音频可以通过许多对策很好地检测。然而，自动说话人验证系统仍然容易受到欺骗攻击，例如重放或Deep-Fake音频。Deep-Fake音频是指使用文本到语音（TTS）和语音转换（VC）算法生成的欺骗性话语。在这里，我们提出了一个新的框架的基础上的混合功能与自我注意力机制。期望可以使用混合特征来获得更大的区分能力。首先，深度学习特征和Mel频谱图特征将通过两条并行路径提取，而不是只有一种传统特征：卷积神经网络和短时傅立叶变换（STFT），然后是Mel频率。其次，特征将通过最大池化层连接起来。第三，有一个自我注意机制，专注于基本要素。最后，建立ResNet和线性层来获得结果。实验结果表明，与传统特征相比，混合特征能够覆盖更多的话语细节。我们在ASVspoof 2021数据集上的物理访问（PA）场景中实现了9.67\%的最佳等错误率（EER），在深度伪造任务中实现了8.94\%。与最佳基线系统相比，该方法分别提高了74.60%和60.05%。摘要：Due to the successful application of deep learning, audio spoofing detection has made significant progress. Spoofed audio with speech synthesis or voice conversion can be well detected by many countermeasures. However, an automatic speaker verification system is still vulnerable to spoofing attacks such as replay or Deep-Fake audio. Deep-Fake audio means that the spoofed utterances are generated using text-to-speech (TTS) and voice conversion (VC) algorithms. Here, we propose a novel framework based on hybrid features with the self-attention mechanism. It is expected that hybrid features can be used to get more discrimination capacity. Firstly, instead of only one type of conventional feature, deep learning features and Mel-spectrogram features will be extracted by two parallel paths: convolution neural networks and a short-time Fourier transform (STFT) followed by Mel-frequency. Secondly, features will be concatenated by a max-pooling layer. Thirdly, there is a Self-attention mechanism for focusing on essential elements. Finally, ResNet and a linear layer are built to get the results. Experimental results reveal that the hybrid features, compared with conventional features, can cover more details of an utterance. We achieve the best Equal Error Rate (EER) of 9.67\% in the physical access (PA) scenario and 8.94\% in the Deep fake task on the ASVspoof 2021 dataset. Compared with the best baseline system, the proposed approach improves by 74.60\% and 60.05\%, respectively.【6】 Useful Blunders: Can Automated Speech Recognition Errors Improve Downstream Dementia Classification?标题：有用的错误：自动语音识别错误能否改善下游痴呆症分类？链接：https://arxiv.org/abs/2401.05551作者：Changye Li,Weizhe Xu,Trevor Cohen,Serguei Pakhomov备注：To appear on Journal of Biomedical Informatics摘要：\textbf{目标}：我们的目的是研究自动语音识别（ASR）系统的错误如何影响痴呆症分类的准确性，特别是在“Cookie盗窃”的图片描述任务。我们的目的是评估不完美的ASR生成的转录本是否可以提供有价值的信息，用于区分认知健康个体和阿尔茨海默病（AD）患者的语言样本。 \textbf{方法}：我们使用各种ASR模型进行实验，使用后编辑技术改进其转录本。这些不完美的ASR转录本和手动转录的转录本都被用作下游痴呆分类的输入。我们进行了全面的误差分析，以比较模型性能并评估ASR生成的转录本在痴呆症分类中的有效性。 \textbf{结果}：不完美的ASR生成的转录本令人惊讶地优于手动转录，用于区分AD患者和“Cookie盗窃”任务中没有的人。这些基于ASR的模型超越了以前最先进的方法，表明ASR错误可能包含与痴呆症相关的有价值的线索。ASR和分类模型之间的协同作用提高了痴呆症分类的整体准确性。 \textbf{结论}：不完美的ASR成绩单有效地捕捉与痴呆症相关的语言异常，提高分类任务的准确性。ASR和分类模型之间的这种协同作用强调了ASR作为评估认知障碍和相关临床应用的有价值工具的潜力。摘要：\textbf{Objectives}: We aimed to investigate how errors from automatic speech recognition (ASR) systems affect dementia classification accuracy, specifically in the ``Cookie Theft'' picture description task. We aimed to assess whether imperfect ASR-generated transcripts could provide valuable information for distinguishing between language samples from cognitively healthy individuals and those with Alzheimer's disease (AD). \textbf{Methods}: We conducted experiments using various ASR models, refining their transcripts with post-editing techniques. Both these imperfect ASR transcripts and manually transcribed ones were used as inputs for the downstream dementia classification. We conducted comprehensive error analysis to compare model performance and assess ASR-generated transcript effectiveness in dementia classification. \textbf{Results}: Imperfect ASR-generated transcripts surprisingly outperformed manual transcription for distinguishing between individuals with AD and those without in the ``Cookie Theft'' task. These ASR-based models surpassed the previous state-of-the-art approach, indicating that ASR errors may contain valuable cues related to dementia. The synergy between ASR and classification models improved overall accuracy in dementia classification. \textbf{Conclusion}: Imperfect ASR transcripts effectively capture linguistic anomalies linked to dementia, improving accuracy in classification tasks. This synergy between ASR and classification models underscores ASR's potential as a valuable tool in assessing cognitive impairment and related clinical applications.【7】 VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognition标题：VI-PAN：利用迁移学习和不确定性感知变分推理改进音频模式识别中的泛化链接：https://arxiv.org/abs/2401.05531作者：John Fischer,Marko Orescanin,Eric Eckstrand备注：Submitted to IEEE Access摘要：迁移学习（TL）是一种越来越受欢迎的训练深度学习（DL）模型的方法，它利用了通过在不同的大规模数据集上训练基础模型所获得的知识，用于下游任务，其中特定于领域或任务的数据较少。文献是丰富的TL技术和应用，但是，大部分的研究利用确定性DL模型，往往是未经校准，缺乏沟通的能力，在预测的认知（模型）的不确定性的措施。与确定性模型不同，贝叶斯DL（BDL）模型通常经过良好校准，可以获得预测的认知不确定性，并且能够实现有竞争力的预测性能。在这项研究中，我们提出了变分推理预训练的音频神经网络（VI-PANNs）。VI-PANN是流行的ResNet-54架构的变分推理变体，它在AudioSet（一个大规模音频事件检测数据集）上进行了预训练。我们使用ESC-50，UrbanSound 8 K和DCASE 2013数据集将知识从VI-PANN转移到其他下游声学分类任务时，评估了由此产生的不确定性的质量。我们证明，第一次，它是可能的，随着上游任务的知识转移校准的不确定性信息，以提高模型的能力，执行下游任务。摘要：Transfer learning (TL) is an increasingly popular approach to training deep learning (DL) models that leverages the knowledge gained by training a foundation model on diverse, large-scale datasets for use on downstream tasks where less domain- or task-specific data is available. The literature is rich with TL techniques and applications; however, the bulk of the research makes use of deterministic DL models which are often uncalibrated and lack the ability to communicate a measure of epistemic (model) uncertainty in prediction. Unlike their deterministic counterparts, Bayesian DL (BDL) models are often well-calibrated, provide access to epistemic uncertainty for a prediction, and are capable of achieving competitive predictive performance. In this study, we propose variational inference pre-trained audio neural networks (VI-PANNs). VI-PANNs are a variational inference variant of the popular ResNet-54 architecture which are pre-trained on AudioSet, a large-scale audio event detection dataset. We evaluate the quality of the resulting uncertainty when transferring knowledge from VI-PANNs to other downstream acoustic classification tasks using the ESC-50, UrbanSound8K, and DCASE2013 datasets. We demonstrate, for the first time, that it is possible to transfer calibrated uncertainty information along with knowledge from upstream tasks to enhance a model's capability to perform downstream tasks.【8】 Neural Ambisonics encoding for compact irregular microphone arrays标题：用于紧凑型不规则麦克风阵列的神经高保真度立体声编码链接：https://arxiv.org/abs/2401.05916作者：Mikko Heikkinen,Archontis Politis,Tuomas Virtanen备注：Accepted for publication in Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：麦克风阵列信号的高保真度立体声编码可以实现各种空间音频应用，例如虚拟现实或远程呈现，但它通常被设计用于均匀间隔的球形麦克风阵列。本文提出了一种用于高保真度立体声响复制编码的方法，该方法使用深度神经网络（DNN）来估计从麦克风输入到高保真度立体声响复制信号的信号变换。该方法使用由具有可学习预处理的U-Net结构组成的DNN，以及由平均误差，空间相关性和能量保存组件组成的损失函数。该方法是验证两个麦克风阵列与规则和不规则的形状有四个麦克风，在模拟混响场景与多个来源。验证结果表明，所提出的方法可以满足或超过传统的信号无关的高保真度立体声编码器的性能上的一些错误度量。摘要：Ambisonics encoding of microphone array signals can enable various spatial audio applications, such as virtual reality or telepresence, but it is typically designed for uniformly-spaced spherical microphone arrays. This paper proposes a method for Ambisonics encoding that uses a deep neural network (DNN) to estimate a signal transform from microphone inputs to Ambisonics signals. The approach uses a DNN consisting of a U-Net structure with a learnable preprocessing as well as a loss function consisting of mean average error, spatial correlation, and energy preservation components. The method is validated on two microphone arrays with regular and irregular shapes having four microphones, on simulated reverberant scenes with multiple sources. The results of the validation show that the proposed method can meet or exceed the performance of a conventional signal-independent Ambisonics encoder on a number of error metrics.【9】 Localizing Acoustic Energy in Sound Field Synthesis by Directionally Weighted Exterior Radiation Suppression标题：定向加权外辐射抑制声场合成中声能的局部化链接：https://arxiv.org/abs/2401.05809作者：Yoshihide Tomita,Shoichi Koyama,Hiroshi Saruwatari备注：Accepted to International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024摘要：提出了一种利用方向性加权抑制外部辐射功率的同时合成期望声场的方法。声场合成系统中扬声器的外部辐射在实际情况中可能是有问题的。尽管已经提出了几种抑制外部辐射的方法，但是在所有向外方向上的抑制通常是困难的，特别是当扬声器的数量不够大时。我们提出了方向加权的外部辐射表示优先考虑的抑制方向，将其纳入声场合成的优化问题。通过使用所提出的表示，在优先方向上的外部辐射可以显着减少，同时保持高的内部合成精度，由于放松了对外部辐射的约束。它的性能进行了评估与应用所提出的表示幅度匹配的数值实验。摘要：A method for synthesizing the desired sound field while suppressing the exterior radiation power with directional weighting is proposed. The exterior radiation from the loudspeakers in sound field synthesis systems can be problematic in practical situations. Although several methods to suppress the exterior radiation have been proposed, suppression in all outward directions is generally difficult, especially when the number of loudspeakers is not sufficiently large. We propose the directionally weighted exterior radiation representation to prioritize the suppression directions by incorporating it into the optimization problem of sound field synthesis. By using the proposed representation, the exterior radiation in the prioritized directions can be significantly reduced while maintaining high interior synthesis accuracy, owing to the relaxed constraint on the exterior radiation. Its performance is evaluated with the application of the proposed representation to amplitude matching in numerical experiments.【10】 Segment Boundary Detection via Class Entropy Measurements in Connectionist Phoneme Recognition标题：联结式音素识别中基于类熵测量的分段边界检测链接：https://arxiv.org/abs/2401.05717作者：Giampiero Salvi备注：None摘要：本文探讨了利用联结主义音素识别器输出的类熵来预测语音类之间的时间边界的可能性。基本原理是熵的值应该在识别网络很好地建模（已知）的两个段之间的过渡附近增加，因为它是不确定性的度量。这种措施的优点是它的简单性，因为每个类的后验概率可用于连接主义音素识别。熵和基于熵的微分的一些测量被单独地和组合地使用。用于预测边界的决策方法从简单的阈值到基于神经网络的过程。不同的方法进行比较，其精度，测量的参考10或20毫秒内的预测边界的数量C和预测边界的总数之间的比率，和召回，测量为C和参考边界的总数之间的比率。摘要：This article investigates the possibility to use the class entropy of the output of a connectionist phoneme recogniser to predict time boundaries between phonetic classes. The rationale is that the value of the entropy should increase in proximity of a transition between two segments that are well modelled (known) by the recognition network since it is a measure of uncertainty. The advantage of this measure is its simplicity as the posterior probabilities of each class are available in connectionist phoneme recognition. The entropy and a number of measures based on differentiation of the entropy are used in isolation and in combination. The decision methods for predicting the boundaries range from simple thresholds to neural network based procedure. The different methods are compared with respect to their precision, measured in terms of the ratio between the number C of predicted boundaries within 10 or 20 msec of the reference and the total number of predicted boundaries, and recall, measured as the ratio between C and the total number of reference boundaries.eess.AS音频处理【1】 Neural Ambisonics encoding for compact irregular microphone arrays标题：用于紧凑型非规则麦克风阵列的神经两用编码链接：https://arxiv.org/abs/2401.05916作者：Mikko Heikkinen,Archontis Politis,Tuomas Virtanen备注：Accepted for publication in Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing摘要：麦克风阵列信号的高保真度立体声编码可以实现各种空间音频应用，例如虚拟现实或远程呈现，但它通常被设计用于均匀间隔的球形麦克风阵列。本文提出了一种用于高保真度立体声响复制编码的方法，该方法使用深度神经网络（DNN）来估计从麦克风输入到高保真度立体声响复制信号的信号变换。该方法使用由具有可学习预处理的U-Net结构组成的DNN，以及由平均误差，空间相关性和能量保存组件组成的损失函数。该方法是验证两个麦克风阵列与规则和不规则的形状有四个麦克风，在模拟混响场景与多个来源。验证结果表明，所提出的方法可以满足或超过传统的信号无关的高保真度立体声编码器的性能上的一些错误度量。摘要：Ambisonics encoding of microphone array signals can enable various spatial audio applications, such as virtual reality or telepresence, but it is typically designed for uniformly-spaced spherical microphone arrays. This paper proposes a method for Ambisonics encoding that uses a deep neural network (DNN) to estimate a signal transform from microphone inputs to Ambisonics signals. The approach uses a DNN consisting of a U-Net structure with a learnable preprocessing as well as a loss function consisting of mean average error, spatial correlation, and energy preservation components. The method is validated on two microphone arrays with regular and irregular shapes having four microphones, on simulated reverberant scenes with multiple sources. The results of the validation show that the proposed method can meet or exceed the performance of a conventional signal-independent Ambisonics encoder on a number of error metrics.【2】 Localizing Acoustic Energy in Sound Field Synthesis by Directionally Weighted Exterior Radiation Suppression标题：方向加权外辐射抑制声场合成中的声能局部化链接：https://arxiv.org/abs/2401.05809作者：Yoshihide Tomita,Shoichi Koyama,Hiroshi Saruwatari备注：Accepted to International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2024摘要：提出了一种利用方向性加权抑制外部辐射功率的同时合成期望声场的方法。声场合成系统中扬声器的外部辐射在实际情况中可能是有问题的。尽管已经提出了几种抑制外部辐射的方法，但是在所有向外方向上的抑制通常是困难的，特别是当扬声器的数量不够大时。我们提出了方向加权的外部辐射表示优先考虑的抑制方向，将其纳入声场合成的优化问题。通过使用所提出的表示，在优先方向上的外部辐射可以显着减少，同时保持高的内部合成精度，由于放松了对外部辐射的约束。它的性能进行了评估与应用所提出的表示幅度匹配的数值实验。摘要：A method for synthesizing the desired sound field while suppressing the exterior radiation power with directional weighting is proposed. The exterior radiation from the loudspeakers in sound field synthesis systems can be problematic in practical situations. Although several methods to suppress the exterior radiation have been proposed, suppression in all outward directions is generally difficult, especially when the number of loudspeakers is not sufficiently large. We propose the directionally weighted exterior radiation representation to prioritize the suppression directions by incorporating it into the optimization problem of sound field synthesis. By using the proposed representation, the exterior radiation in the prioritized directions can be significantly reduced while maintaining high interior synthesis accuracy, owing to the relaxed constraint on the exterior radiation. Its performance is evaluated with the application of the proposed representation to amplitude matching in numerical experiments.【3】 Segment Boundary Detection via Class Entropy Measurements in Connectionist Phoneme Recognition标题：基于类熵测度的连接主义音素识别中的音段边界检测链接：https://arxiv.org/abs/2401.05717作者：Giampiero Salvi备注：None摘要：本文探讨了利用联结主义音素识别器输出的类熵来预测语音类之间的时间边界的可能性。基本原理是熵的值应该在识别网络很好地建模（已知）的两个段之间的过渡附近增加，因为它是不确定性的度量。这种措施的优点是它的简单性，因为每个类的后验概率可用于连接主义音素识别。熵和基于熵的微分的一些测量被单独地和组合地使用。用于预测边界的决策方法从简单的阈值到基于神经网络的过程。不同的方法进行比较，其精度，测量的参考10或20毫秒内的预测边界的数量C和预测边界的总数之间的比率，和召回，测量为C和参考边界的总数之间的比率。摘要：This article investigates the possibility to use the class entropy of the output of a connectionist phoneme recogniser to predict time boundaries between phonetic classes. The rationale is that the value of the entropy should increase in proximity of a transition between two segments that are well modelled (known) by the recognition network since it is a measure of uncertainty. The advantage of this measure is its simplicity as the posterior probabilities of each class are available in connectionist phoneme recognition. The entropy and a number of measures based on differentiation of the entropy are used in isolation and in combination. The decision methods for predicting the boundaries range from simple thresholds to neural network based procedure. The different methods are compared with respect to their precision, measured in terms of the ratio between the number C of predicted boundaries within 10 or 20 msec of the reference and the total number of predicted boundaries, and recall, measured as the ratio between C and the total number of reference boundaries.【4】 Contrastive Loss Based Frame-wise Feature disentanglement for Polyphonic Sound Event Detection标题：基于对比损失的帧级特征解纠缠多音事件检测链接：https://arxiv.org/abs/2401.05850作者：Yadong Guan,Jiqing Han,Hongwei Song,Wenjie Song,Guibin Zheng,Tieran Zheng,Yongjun He备注：accepted by icassp2024摘要：重叠声音事件在现实环境中普遍存在，但现有的端到端声音事件检测（SED）方法仍然难以有效地检测它们。一个关键的原因是，这些方法表示重叠的事件使用共享和纠缠帧的功能，这降低了功能的歧视。为了解决这个问题，我们提出了一个分离的特征学习框架来学习特定于类别的表示。具体来说，我们使用不同的投影仪来学习每个类别的帧特征。为了确保这些功能不包含其他类别的信息，我们最大限度地提高同一类别内的帧之间的共同信息，并提出了一个帧的对比损失。此外，考虑到所提出的方法使用的标记数据是有限的，我们提出了一种半监督的逐帧对比损失，可以利用大量的未标记数据来实现特征解纠缠。实验结果证明了该方法的有效性。摘要：Overlapping sound events are ubiquitous in real-world environments, but existing end-to-end sound event detection (SED) methods still struggle to detect them effectively. A critical reason is that these methods represent overlapping events using shared and entangled frame-wise features, which degrades the feature discrimination. To solve the problem, we propose a disentangled feature learning framework to learn a category-specific representation. Specifically, we employ different projectors to learn the frame-wise features for each category. To ensure that these feature does not contain information of other categories, we maximize the common information between frame-wise features within the same category and propose a frame-wise contrastive loss. In addition, considering that the labeled data used by the proposed method is limited, we propose a semi-supervised frame-wise contrastive loss that can leverage large amounts of unlabeled data to achieve feature disentanglement. The experimental results demonstrate the effectiveness of our method.【5】 Intuitive Control of Scraping and Rubbing Through Audio-tactile Synthesis标题：通过音频-触觉合成直观地控制刮擦和摩擦链接：https://arxiv.org/abs/2401.05757作者：Mitsuko Aramaki,Corentin Bernard,Richard Kronland-Martinet,Samuel Poirot,Sølvi Ystad备注：None摘要：对合成过程的直观控制是听觉感知和认知领域内的一个持续挑战。之前的声音建模与心理物理测试相结合的工作使我们的团队能够开发出一种合成器，该合成器基于声源的语义描述提供对动作和对象的直观控制。在这个演示中，我们提出了一个增强版本的合成器，其中我们添加了触觉刺激，以增加与模拟对象的真实连续摩擦相互作用（摩擦和刮擦）的感觉。这是有意义的，原因有几个。首先，它能够评估我们的声音模型在其他模态刺激存在下的真实性。其次，它能够比较触觉和听觉信号结构链接到相同的唤起，第三，它提供了一个工具来研究多模态感知和如何刺激从不同的方式应该结合起来，以提供逼真的用户界面。摘要：Intuitive control of synthesis processes is an ongoing challenge within the domain of auditory perception and cognition. Previous works on sound modelling combined with psychophysical tests have enabled our team to develop a synthesizer that provides intuitive control of actions and objects based on semantic descriptions for sound sources. In this demo we present an augmented version of the synthesizer in which we added tactile stimulations to increase the sensation of true continuous friction interactions (rubbing and scratching) with the simulated objects. This is of interest for several reasons. Firstly, it enables to evaluate the realism of our sound model in presence of stimulations from other modalities. Secondly it enables to compare tactile and auditory signal structures linked to the same evocation, and thirdly it provides a tool to investigate multimodal perception and how stimulations from different modalities should be combined to provide realistic user interfaces.【6】 HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition标题：HiCMAE：用于自监督视听情感识别的分层对比掩模自动编码器链接：https://arxiv.org/abs/2401.05698作者：Licai Sun,Zheng Lian,Bin Liu,Jianhua Tao备注：19 pages, 8 figures, 15 tables. Full Abstract is shown in the pdf file. Codes and models will be publicly available at this https URL摘要：近年来，视听情感识别（AVER）因其在创造情感智能机器中的关键作用而受到越来越多的关注。以前在这方面的努力主要是监督学习范式。尽管取得了重大进展，但由于AVER中长期存在的数据稀缺问题，监督学习正在遇到瓶颈。受自监督学习最新进展的启发，我们提出了分层对比掩蔽自动编码器（HiCMAE），这是一种新型的自监督框架，它利用大量未标记的视听数据进行大规模自监督预训练，以促进AVER的进步。在自监督视听表示学习的现有技术之后，HiCMAE采用两种主要形式的自监督进行预训练，即掩蔽数据建模和对比学习。与只关注顶层表征而忽视中间层的明确指导不同，HiCMAE开发了一种三管齐下的策略来促进分层视听特征学习并提高学习表征的整体质量。为了验证HiCMAE的有效性，我们在9个数据集上进行了广泛的实验，涵盖了分类和维度AVER任务。实验结果表明，我们的方法显着优于国家的最先进的监督和自我监督的视听方法，这表明HiCMAE是一个强大的视听情感表征学习器。代码和模型将在https://github.com/sunlicai/HiCMAE上公开。摘要：Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models will be publicly available at https://github.com/sunlicai/HiCMAE.【7】 UCorrect: An Unsupervised Framework for Automatic Speech Recognition Error Correction标题：一种无监督的语音识别自动纠错框架链接：https://arxiv.org/abs/2401.05689作者：Jiaxin Guo,Minghan Wang,Xiaosong Qiao,Daimeng Wei,Hengchao Shang,Zongyao Li,Zhengzhe Yu,Yinglu Li,Chang Su,Min Zhang,Shimin Tao,Hao Yang备注：Accepted in ICASSP 2023摘要：纠错技术已经被用来改进自动语音识别（ASR）模型的输出句子，并实现较低的单词错误率（WER）。以往的研究多采用端到端模型，对伪配对数据和原始配对数据有很强的依赖性。但是，当仅对伪配对数据进行预训练时，以前的模型对校正有负面影响。在对原始配对数据进行微调时，源端数据必须由经过良好训练的ASR模型转录，这需要大量时间并且不通用。在本文中，我们提出了UCorrect，一个用于ASR纠错的无监督检测器生成器框架。UCorrect不依赖于前面提到的训练数据。整个过程首先检测字符是否错误，然后产生候选字符，最后选择最可信的字符替换错误字符。在公开的AISHELL-1数据集和WenetSpeech数据集上的实验结果表明，UCorrect对ASR纠错的有效性：1）UCorrect显著降低了WER，在不进行微调的情况下WER降低了6.83%，微调后WER降低了14.29%;以及3）它是一种通用方法，因为它减少了具有不同解码策略的ASR模型的所有WER，并且减少了在不同规模数据集上训练的ASR模型的所有WER。摘要：Error correction techniques have been used to refine the output sentences from automatic speech recognition (ASR) models and achieve a lower word error rate (WER). Previous works usually adopt end-to-end models and has strong dependency on Pseudo Paired Data and Original Paired Data. But when only pre-training on Pseudo Paired Data, previous models have negative effect on correction. While fine-tuning on Original Paired Data, the source side data must be transcribed by a well-trained ASR model, which takes a lot of time and not universal. In this paper, we propose UCorrect, an unsupervised Detector-Generator-Selector framework for ASR Error Correction. UCorrect has no dependency on the training data mentioned before. The whole procedure is first to detect whether the character is erroneous, then to generate some candidate characters and finally to select the most confident one to replace the error character. Experiments on the public AISHELL-1 dataset and WenetSpeech dataset show the effectiveness of UCorrect for ASR error correction: 1) it achieves significant WER reduction, achieves 6.83\% even without fine-tuning and 14.29\% after fine-tuning; 2) it outperforms the popular NAR correction models by a large margin with a competitive low latency; and 3) it is an universal method, as it reduces all WERs of the ASR model with different decoding strategies and reduces all WERs of ASR models trained on different scale datasets.【8】 Self-Attention and Hybrid Features for Replay and Deep-Fake Audio Detection标题：用于重放和深度伪音频检测的自我注意和混合特征链接：https://arxiv.org/abs/2401.05614作者：Lian Huang,Chi-Man Pun摘要：由于深度学习的成功应用，音频欺骗检测取得了重大进展。具有语音合成或语音转换的欺骗音频可以通过许多对策很好地检测。然而，自动说话人验证系统仍然容易受到欺骗攻击，例如重放或Deep-Fake音频。Deep-Fake音频是指使用文本到语音（TTS）和语音转换（VC）算法生成的欺骗性话语。在这里，我们提出了一个新的框架的基础上的混合功能与自我注意力机制。期望可以使用混合特征来获得更大的区分能力。首先，深度学习特征和Mel频谱图特征将通过两条并行路径提取，而不是只有一种传统特征：卷积神经网络和短时傅立叶变换（STFT），然后是Mel频率。其次，特征将通过最大池化层连接起来。第三，有一个自我注意机制，专注于基本要素。最后，建立ResNet和线性层来获得结果。实验结果表明，与传统特征相比，混合特征能够覆盖更多的话语细节。我们在ASVspoof 2021数据集上的物理访问（PA）场景中实现了9.67\%的最佳等错误率（EER），在深度伪造任务中实现了8.94\%。与最佳基线系统相比，该方法分别提高了74.60%和60.05%。摘要：Due to the successful application of deep learning, audio spoofing detection has made significant progress. Spoofed audio with speech synthesis or voice conversion can be well detected by many countermeasures. However, an automatic speaker verification system is still vulnerable to spoofing attacks such as replay or Deep-Fake audio. Deep-Fake audio means that the spoofed utterances are generated using text-to-speech (TTS) and voice conversion (VC) algorithms. Here, we propose a novel framework based on hybrid features with the self-attention mechanism. It is expected that hybrid features can be used to get more discrimination capacity. Firstly, instead of only one type of conventional feature, deep learning features and Mel-spectrogram features will be extracted by two parallel paths: convolution neural networks and a short-time Fourier transform (STFT) followed by Mel-frequency. Secondly, features will be concatenated by a max-pooling layer. Thirdly, there is a Self-attention mechanism for focusing on essential elements. Finally, ResNet and a linear layer are built to get the results. Experimental results reveal that the hybrid features, compared with conventional features, can cover more details of an utterance. We achieve the best Equal Error Rate (EER) of 9.67\% in the physical access (PA) scenario and 8.94\% in the Deep fake task on the ASVspoof 2021 dataset. Compared with the best baseline system, the proposed approach improves by 74.60\% and 60.05\%, respectively.【9】 Useful Blunders: Can Automated Speech Recognition Errors Improve Downstream Dementia Classification?标题：有用的错误：自动语音识别错误能否改善下游痴呆症分类？链接：https://arxiv.org/abs/2401.05551作者：Changye Li,Weizhe Xu,Trevor Cohen,Serguei Pakhomov备注：To appear on Journal of Biomedical Informatics摘要：\textbf{目标}：我们的目的是研究自动语音识别（ASR）系统的错误如何影响痴呆症分类的准确性，特别是在“Cookie盗窃”的图片描述任务。我们的目的是评估不完美的ASR生成的转录本是否可以提供有价值的信息，用于区分认知健康个体和阿尔茨海默病（AD）患者的语言样本。 \textbf{方法}：我们使用各种ASR模型进行实验，使用后编辑技术改进其转录本。这些不完美的ASR转录本和手动转录的转录本都被用作下游痴呆分类的输入。我们进行了全面的误差分析，以比较模型性能并评估ASR生成的转录本在痴呆症分类中的有效性。 \textbf{结果}：不完美的ASR生成的转录本令人惊讶地优于手动转录，用于区分AD患者和“Cookie盗窃”任务中没有的人。这些基于ASR的模型超越了以前最先进的方法，表明ASR错误可能包含与痴呆症相关的有价值的线索。ASR和分类模型之间的协同作用提高了痴呆症分类的整体准确性。 \textbf{结论}：不完美的ASR成绩单有效地捕捉与痴呆症相关的语言异常，提高分类任务的准确性。ASR和分类模型之间的这种协同作用强调了ASR作为评估认知障碍和相关临床应用的有价值工具的潜力。摘要：\textbf{Objectives}: We aimed to investigate how errors from automatic speech recognition (ASR) systems affect dementia classification accuracy, specifically in the ``Cookie Theft'' picture description task. We aimed to assess whether imperfect ASR-generated transcripts could provide valuable information for distinguishing between language samples from cognitively healthy individuals and those with Alzheimer's disease (AD). \textbf{Methods}: We conducted experiments using various ASR models, refining their transcripts with post-editing techniques. Both these imperfect ASR transcripts and manually transcribed ones were used as inputs for the downstream dementia classification. We conducted comprehensive error analysis to compare model performance and assess ASR-generated transcript effectiveness in dementia classification. \textbf{Results}: Imperfect ASR-generated transcripts surprisingly outperformed manual transcription for distinguishing between individuals with AD and those without in the ``Cookie Theft'' task. These ASR-based models surpassed the previous state-of-the-art approach, indicating that ASR errors may contain valuable cues related to dementia. The synergy between ASR and classification models improved overall accuracy in dementia classification. \textbf{Conclusion}: Imperfect ASR transcripts effectively capture linguistic anomalies linked to dementia, improving accuracy in classification tasks. This synergy between ASR and classification models underscores ASR's potential as a valuable tool in assessing cognitive impairment and related clinical applications.【10】 VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognition标题：VI-PAN：利用迁移学习和不确定性感知变分推理改进音频模式识别中的泛化链接：https://arxiv.org/abs/2401.05531作者：John Fischer,Marko Orescanin,Eric Eckstrand备注：Submitted to IEEE Access摘要：迁移学习（TL）是一种越来越受欢迎的训练深度学习（DL）模型的方法，它利用了通过在不同的大规模数据集上训练基础模型所获得的知识，用于下游任务，其中特定于领域或任务的数据较少。文献是丰富的TL技术和应用，但是，大部分的研究利用确定性DL模型，往往是未经校准，缺乏沟通的能力，在预测的认知（模型）的不确定性的措施。与确定性模型不同，贝叶斯DL（BDL）模型通常经过良好校准，可以获得预测的认知不确定性，并且能够实现有竞争力的预测性能。在这项研究中，我们提出了变分推理预训练的音频神经网络（VI-PANNs）。VI-PANN是流行的ResNet-54架构的变分推理变体，它在AudioSet（一个大规模音频事件检测数据集）上进行了预训练。我们使用ESC-50，UrbanSound 8 K和DCASE 2013数据集将知识从VI-PANN转移到其他下游声学分类任务时，评估了由此产生的不确定性的质量。我们证明，第一次，它是可能的，随着上游任务的知识转移校准的不确定性信息，以提高模型的能力，执行下游任务。摘要：Transfer learning (TL) is an increasingly popular approach to training deep learning (DL) models that leverages the knowledge gained by training a foundation model on diverse, large-scale datasets for use on downstream tasks where less domain- or task-specific data is available. The literature is rich with TL techniques and applications; however, the bulk of the research makes use of deterministic DL models which are often uncalibrated and lack the ability to communicate a measure of epistemic (model) uncertainty in prediction. Unlike their deterministic counterparts, Bayesian DL (BDL) models are often well-calibrated, provide access to epistemic uncertainty for a prediction, and are capable of achieving competitive predictive performance. In this study, we propose variational inference pre-trained audio neural networks (VI-PANNs). VI-PANNs are a variational inference variant of the popular ResNet-54 architecture which are pre-trained on AudioSet, a large-scale audio event detection dataset. We evaluate the quality of the resulting uncertainty when transferring knowledge from VI-PANNs to other downstream acoustic classification tasks using the ESC-50, UrbanSound8K, and DCASE2013 datasets. We demonstrate, for the first time, that it is possible to transfer calibrated uncertainty information along with knowledge from upstream tasks to enhance a model's capability to perform downstream tasks.机器翻译由腾讯交互翻译提供，仅供参考
每日语音速递
