今日论文合集：cs.SD语音26篇，eess.AS音频处理27篇。【1】WikiMuTe: A web-sourced dataset of semantic descriptions for music audio
标题： WikiMuTe：一个基于Web的音乐音频语义描述数据集链接：https://arxiv.org/abs/2312.09207作者：Benno Weck,Holger Kirchhoff,Peter Grosche,Xavier Serra备注：Submitted to 30th International Conference on MultiMedia Modeling (MMM2024). This preprint has not undergone peer review or any post-submission improvements or corrections摘要：用于匹配自由文本与音乐的多模态深度学习技术在音乐信息检索（MIR）领域显示出了良好的效果。先前的工作通常基于大型专有数据，而公开的数据集很少且规模较小。在这项研究中，我们提出了WikiMuTe，一个新的和开放的数据集，包含丰富的音乐语义描述。这些数据来自维基百科丰富的音乐作品条目目录。使用专用的文本挖掘管道，我们提取了长形式和短形式的描述，涵盖了与音乐内容相关的广泛主题，如流派，风格，情绪，乐器和节奏。为了展示这些数据的使用，我们训练了一个模型，该模型联合学习文本和音频表示并执行跨模态检索。该模型进行评估的两个任务：基于标签的音乐检索和音乐自动标注。结果表明，虽然我们的方法在多个任务上具有最先进的性能，但仍然观察到性能差异取决于用于训练的数据。摘要：Multi-modal deep learning techniques for matching free-form text with music have shown promising results in the field of Music Information Retrieval (MIR). Prior work is often based on large proprietary data while publicly available datasets are few and small in size. In this study, we present WikiMuTe, a new and open dataset containing rich semantic descriptions of music. The data is sourced from Wikipedia's rich catalogue of articles covering musical works. Using a dedicated text-mining pipeline, we extract both long and short-form descriptions covering a wide range of topics related to music content such as genre, style, mood, instrumentation, and tempo. To show the use of this data, we train a model that jointly learns text and audio representations and performs cross-modal retrieval. The model is evaluated on two tasks: tag-based music retrieval and music auto-tagging. The results show that while our approach has state-of-the-art performance on multiple tasks, but still observe a difference in performance depending on the data used for training.【2】 F1-EV Score: Measuring the Likelihood of Estimating a Good Decision Threshold for Semi-Supervised Anomaly Detection标题：F1-EV评分：衡量估计半监督异常检测良好决策阈值的可能性链接：https://arxiv.org/abs/2312.09143作者：Kevin Wilkinghoff,Keisuke Imoto备注：Accepted for presentation at IEEE ICASSP 2024摘要：异常声音检测（ASD）系统通常使用阈值无关的性能指标，如AUC-ROC进行比较。然而，对于实际应用，需要一个判定阈值来判定给定的测试样本是正常还是异常。在只有正常训练样本可用的半监督设置中，估计这样的阈值是非常重要的。在这项工作中，F1-EV提出了一种新的阈值独立的ASD系统，也包括估计一个良好的决策阈值的可能性的性能指标，并激励使用特定的玩具的例子。在实验评估中，对提交给DCASE Challenge 2023 ASD任务的所有系统进行了多项性能指标评估。结果表明，F1-EV与AUC-ROC强相关，同时与使用估计和最佳决策阈值获得的F1评分的相关性显著强于AUC-ROC。摘要：Anomalous sound detection (ASD) systems are usually compared by using threshold-independent performance measures such as AUC-ROC. However, for practical applications a decision threshold is needed to decide whether a given test sample is normal or anomalous. Estimating such a threshold is highly non-trivial in a semi-supervised setting where only normal training samples are available. In this work, F1-EV a novel threshold-independent performance measure for ASD systems that also includes the likelihood of estimating a good decision threshold is proposed and motivated using specific toy examples. In experimental evaluations, multiple performance measures are evaluated for all systems submitted to the ASD task of the DCASE Challenge 2023. It is shown that F1-EV is strongly correlated with AUC-ROC while having a significantly stronger correlation with the F1-score obtained with estimated and optimal decision thresholds than AUC-ROC.【3】 STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models标题： STaR：轻量级语音自监督学习模型的语音时序关系提取链接：https://arxiv.org/abs/2312.09040作者：Kangwook Jang,Sungnyun Kim,Hoirin Kim备注：Accepted at ICASSP 2024摘要：尽管基于transformer的语音自监督学习（SSL）模型具有很好的性能，但其较大的参数大小和计算成本使其不利于使用。在这项研究中，我们提出了压缩语音SSL模型提取语音时间关系（STaR）。与以往直接匹配每个语音帧的表示不同，STaR蒸馏传输语音帧之间的时间关系，这更适合于能力有限的轻量级学生。我们探讨了三个STaR蒸馏目标，并选择最佳组合作为最终的STaR损失。我们从HuBERT BASE中提取的模型在SUPERB基准测试中获得了79.8的总得分，在多达2700万个参数的模型中表现最好。我们表明，我们的方法是适用于不同的语音SSL模型，并保持强大的性能与进一步减少参数。摘要：Albeit great performance of Transformer-based speech selfsupervised learning (SSL) models, their large parameter size and computational cost make them unfavorable to utilize. In this study, we propose to compress the speech SSL models by distilling speech temporal relation (STaR). Unlike previous works that directly match the representation for each speech frame, STaR distillation transfers temporal relation between speech frames, which is more suitable for lightweight student with limited capacity. We explore three STaR distillation objectives and select the best combination as the final STaR loss. Our model distilled from HuBERT BASE achieves an overall score of 79.8 on SUPERB benchmark, the best performance among models with up to 27 million parameters. We show that our method is applicable across different speech SSL models and maintains robust performance with further reduced parameters.【4】 Multi-CMGAN+/+: Leveraging Multi-Objective Speech Quality Metric Prediction for Speech Enhancement标题：多CMGAN+/+：利用多目标语音质量度量预测进行语音增强链接：https://arxiv.org/abs/2312.08979作者：George Close,William Ravenscroft,Thomas Hain,Stefan Goetze备注：Accepted @ ICASSP 2024摘要：基于神经网络的语音增强方法已被证明是特别强大的，能够利用数据驱动的方法，从而导致与其他方法相比显着的性能增益。这种方法依赖于人工创建的标记训练数据，使得可以使用将模型的输出与干净的参考语音进行比较的侵入性损失函数来训练神经模型。这种系统在增强真实世界音频时的性能通常相对于它们在模拟测试数据上的性能而受损。在这项工作中，引入了一种非侵入性的多度量预测方法，其中使用对抗训练的度量预测神经网络的推理在人工标记的数据上训练模型。所提出的方法显示了改进的性能与最先进的系统在最近的CHiME-7挑战\ac{UDASE}任务评估集。摘要：Neural network based approaches to speech enhancement have shown to be particularly powerful, being able to leverage a data-driven approach to result in a significant performance gain versus other approaches. Such approaches are reliant on artificially created labelled training data such that the neural model can be trained using intrusive loss functions which compare the output of the model with clean reference speech. Performance of such systems when enhancing real-world audio often suffers relative to their performance on simulated test data. In this work, a non-intrusive multi-metric prediction approach is introduced, wherein a model trained on artificial labelled data using inference of an adversarially trained metric prediction neural network. The proposed approach shows improved performance versus state-of-the-art systems on the recent CHiME-7 challenge \ac{UDASE} task evaluation sets.【5】 N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding标题：N-Gram无监督组合和特征注入更好地理解符号音乐链接：https://arxiv.org/abs/2312.08931作者：Jinhao Tian,Zuchao Li,Jiajia Li,Ping Wang备注：8 pages, 2 figures, aaai2024摘要：将深度学习技术应用于符号音乐理解的第一步是将音乐作品（主要是以MIDI格式）转换为预定义的标记序列，如音符音高，音符速度和和弦。随后，这些序列被送入神经序列模型以完成特定任务。音乐序列在相邻元素之间表现出很强的相关性，使其成为自然语言处理（NLP）中N-gram技术的主要候选者。以古典钢琴音乐为例：特定的旋律可能会在整首曲子中反复出现，每次都有微妙的变化。在本文中，我们提出了一种新的方法，NG-Midiformer，理解符号音乐序列，利用N-gram的方法。我们的方法涉及到第一次处理音乐作品成词样序列与我们提出的无监督的复合，其次是使用我们的N-gram Transformer编码器，它可以有效地将N-gram信息，以提高主要编码器的一部分，更好地理解音乐序列。在大规模音乐数据集上的预训练过程使模型能够彻底学习音乐序列中包含的N-gram信息，并随后在微调阶段应用这些信息进行推断。在不同数据集上的实验证明了该方法的有效性，并在一系列音乐理解下游任务上取得了最先进的性能。代码和模型重量将在https://github.com/WouuYoauin/NG-Midiformer上发布。摘要：The first step to apply deep learning techniques for symbolic music understanding is to transform musical pieces (mainly in MIDI format) into sequences of predefined tokens like note pitch, note velocity, and chords. Subsequently, the sequences are fed into a neural sequence model to accomplish specific tasks. Music sequences exhibit strong correlations between adjacent elements, making them prime candidates for N-gram techniques from Natural Language Processing (NLP). Consider classical piano music: specific melodies might recur throughout a piece, with subtle variations each time. In this paper, we propose a novel method, NG-Midiformer, for understanding symbolic music sequences that leverages the N-gram approach. Our method involves first processing music pieces into word-like sequences with our proposed unsupervised compoundation, followed by using our N-gram Transformer encoder, which can effectively incorporate N-gram information to enhance the primary encoder part for better understanding of music sequences. The pre-training process on large-scale music datasets enables the model to thoroughly learn the N-gram information contained within music sequences, and subsequently apply this information for making inferences during the fine-tuning stage. Experiment on various datasets demonstrate the effectiveness of our method and achieved state-of-the-art performance on a series of music understanding downstream tasks. The code and model weights will be released at https://github.com/WouuYoauin/NG-Midiformer.【6】 Hourglass-AVSR: Down-Up Sampling-based Computational Efficiency Model for Audio-Visual Speech Recognition标题：沙漏-AVSR：基于上下采样的视听语音识别计算效率模型链接：https://arxiv.org/abs/2312.08850作者：Fan Yu,Haoxu Wang,Ziyang Ma,Shiliang Zhang备注：Accepted by ICASSP 2024摘要：近年来，视听语音识别（AVSR），它更好地利用视频模态作为附加信息来扩展自动语音识别（ASR），在复杂的声学环境中显示出良好的效果。然而，由于视觉模块的复杂计算和视听模态的无效融合，仍有很大的改进空间。为了消除这些缺点，我们提出了一个向下向上采样的AVSR模型（沙漏AVSR）享受高效率和性能，其时间长度在中间处理过程中缩放，类似于沙漏。首先，我们提出了一种上下文和残差感知的视频上采样方法，以提高识别性能，它利用视觉表示的上下文信息，并捕获相邻视频帧之间的残差信息。其次，我们在上采样期间引入了一种视觉-音频对齐方法，该方法通过显式地引入边界约束损失。此外，我们提出了一个跨层的注意力融合，以捕捉每个视觉编码器层内的模态依赖。在MISP-AVSR数据集上进行的实验表明，我们提出的Hourglass-AVSR模型在远场和中场测试集上的相对级联最小排列字符错误率（cpCER）分别降低了12.9%和20.8%，优于ASR模型。此外，与其他最先进的AVSR模型相比，我们的模型在视觉模块的cpCER中表现出最高的改进。此外，在我们的向下向上采样方法的好处，沙漏AVSR模型减少了54.2%的整体计算成本与轻微的性能下降。摘要：Recently audio-visual speech recognition (AVSR), which better leverages video modality as additional information to extend automatic speech recognition (ASR), has shown promising results in complex acoustic environments. However, there is still substantial space to improve as complex computation of visual modules and ineffective fusion of audio-visual modalities. To eliminate these drawbacks, we propose a down-up sampling-based AVSR model (Hourglass-AVSR) to enjoy high efficiency and performance, whose time length is scaled during the intermediate processing, resembling an hourglass. Firstly, we propose a context and residual aware video upsampling approach to improve the recognition performance, which utilizes contextual information from visual representations and captures residual information between adjacent video frames. Secondly, we introduce a visual-audio alignment approach during the upsampling by explicitly incorporating boundary constraint loss. Besides, we propose a cross-layer attention fusion to capture the modality dependencies within each visual encoder layer. Experiments conducted on the MISP-AVSR dataset reveal that our proposed Hourglass-AVSR model outperforms ASR model by 12.9% and 20.8% relative concatenated minimum permutation character error rate (cpCER) reduction on far-field and middle-field test sets, respectively. Moreover, compared to other state-of-the-art AVSR models, our model exhibits the highest improvement in cpCER for the visual module. Furthermore, on the benefit of our down-up sampling approach, Hourglass-AVSR model reduces 54.2% overall computation costs with minor performance degradation.【7】 TIA: A Teaching Intonation Assessment Dataset in Real Teaching Situations标题：TIA：一个真实教学情境下的教学语调评价数据集链接：https://arxiv.org/abs/2312.08732作者：Shuhua Liu,Chunyu Zhang,Binshuai Li,Niantong Qin,Huanting Cheng,Huayu Zhang备注：4 pages, 3 figures, 4 tables, accepted by 2024 International Conference on Acoustics, Speech, and Signal Processing (ICASSP2024)摘要：语调是影响教学语言艺术的重要因素之一，利用人工智能技术对教师语调进行评价是一个亟待解决的问题。然而，语调评估数据集的缺乏阻碍了该领域的发展。为此，本文首次在真实教学情境下构建了教学语调评估数据集。该数据集涵盖9个学科，396名教师，共计11，444个话语样本，长度为15秒。为了检验数据集的有效性，本文提出了一个基于语音低层和深层特征的教学语调评估模型（TIAM）。实验结果表明，基于本文构建的数据集的TIAM评价结果与人工评价结果基本一致，且评价结果优于基线模型，证明了评价模型的有效性。摘要：Intonation is one of the important factors affecting the teaching language arts, so it is an urgent problem to be addressed by evaluating the teachers' intonation through artificial intelligence technology. However, the lack of an intonation assessment dataset has hindered the development of the field. To this end, this paper constructs a Teaching Intonation Assessment (TIA) dataset for the first time in real teaching situations. This dataset covers 9 disciplines, 396 teachers, total of 11,444 utterance samples with a length of 15 seconds. In order to test the validity of the dataset, this paper proposes a teaching intonation assessment model (TIAM) based on low-level and deep-level features of speech. The experimental results show that TIAM based on the dataset constructed in this paper is basically consistent with the results of manual evaluation, and the results are better than the baseline models, which proves the effectiveness of the evaluation model.【8】 StemGen: A music generation model that listens标题：StemGen：倾听的音乐生成模式链接：https://arxiv.org/abs/2312.08723作者：Julian D. Parker,Janne Spijkervet,Katerina Kosta,Furkan Yesiler,Boris Kuznetsov,Ju-Chiang Wang,Matt Avent,Jitong Chen,Duc Le备注：Accepted for publication at ICASSP 2024摘要：使用深度学习技术的端到端音乐音频生成最近出现了爆炸式增长。然而，大多数模型专注于生成完全混合的音乐，以响应抽象的条件信息。在这项工作中，我们提出了一个替代的范式，生产音乐生成模型，可以听和响应音乐的背景。我们描述了如何使用非自回归，基于变换器的模型架构构建这样的模型，并提出了一些新的架构和采样改进。我们在开源和专有数据集上训练所描述的架构。我们使用标准的质量指标和一种新的方法，基于音乐信息检索描述符的模型进行评估。由此产生的模型达到了最先进的文本条件模型的音频质量，以及表现出很强的音乐连贯性与其上下文。摘要：End-to-end generation of musical audio using deep learning techniques has seen an explosion of activity recently. However, most models concentrate on generating fully mixed music in response to abstract conditioning information. In this work, we present an alternative paradigm for producing music generation models that can listen and respond to musical context. We describe how such a model can be constructed using a non-autoregressive, transformer-based model architecture and present a number of novel architectural and sampling improvements. We train the described architecture on both an open-source and a proprietary dataset. We evaluate the produced models using standard quality metrics and a new approach based on music information retrieval descriptors. The resulting model reaches the audio quality of state-of-the-art text-conditioned models, as well as exhibiting strong musical coherence with its context.【9】 SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention标题：SEF-VC：嵌入交叉注意的自由零发声语音转换的扬声器链接：https://arxiv.org/abs/2312.08676作者：Junjie Li,Yiwei Guo,Xie Chen,Kai Yu备注：5 pages,2 figures, submitted to ICASSP 2024摘要：Zero-shot语音转换（VC）的目的是在保持语言内容不变的情况下，将源说话人的音色转换为目标说话人的音色。虽然可以通过提供目标说话人的说话人嵌入来控制所生成的语音的声音，但是说话人相似性仍然落后于地面实况记录。在本文中，我们提出了SEF-VC，一个说话人嵌入自由语音转换模型，它的目的是通过一个强大的位置不可知的交叉注意机制，学习和纳入说话人的音色从参考语音，然后重建波形从HuBERT语义令牌在非自回归的方式。SEF-VC的简洁设计提高了训练稳定性和语音转换性能。客观和主观评价表明，即使对于非常短的参考语音，SEF-VC也比强zero-shot VC基线生成与目标参考具有更好相似性的高质量语音。摘要：Zero-shot voice conversion (VC) aims to transfer the source speaker timbre to arbitrary unseen target speaker timbre, while keeping the linguistic content unchanged. Although the voice of generated speech can be controlled by providing the speaker embedding of the target speaker, the speaker similarity still lags behind the ground truth recordings. In this paper, we propose SEF-VC, a speaker embedding free voice conversion model, which is designed to learn and incorporate speaker timbre from reference speech via a powerful position-agnostic cross-attention mechanism, and then reconstruct waveform from HuBERT semantic tokens in a non-autoregressive manner. The concise design of SEF-VC enhances its training stability and voice conversion performance. Objective and subjective evaluations demonstrate the superiority of SEF-VC to generate high-quality speech with better similarity to target reference than strong zero-shot VC baselines, even for very short reference speeches.【10】 Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation标题：视听之外的片段：视听语义切分中部分缺失通道的处理链接：https://arxiv.org/abs/2312.08673作者：Renjie Wu,Hu Wang,Feras Dayoub,Hsiang-Ting Chen备注：Accepted by AAAI-24摘要：增强现实（AR）设备作为突出的移动交互平台出现，面临着用户安全方面的挑战，特别是关于迎面而来的车辆。虽然一些解决方案利用板载相机阵列，但这些相机通常具有有限的前视角或下视角的视场（FoV）。针对这一点，我们提出了一个新的视图外的语义分割任务和分段超越视图（SBV），一种新的视听语义分割方法。SBV补充视觉模态，错过了FoV以外的信息，与听觉信息使用师生蒸馏模型（Omni2Ego）。该模型包括一个视觉教师利用全景信息，一个听觉教师与8通道音频，和一个视听的学生，认为有限的FoV和双耳音频作为输入，并产生语义分割FoV以外的对象。SBV在比较评估中优于现有模型，并在不同的FoV范围和单声道音频设置中显示出一致的性能。摘要：Augmented Reality (AR) devices, emerging as prominent mobile interaction platforms, face challenges in user safety, particularly concerning oncoming vehicles. While some solutions leverage onboard camera arrays, these cameras often have limited field-of-view (FoV) with front or downward perspectives. Addressing this, we propose a new out-of-view semantic segmentation task and Segment Beyond View (SBV), a novel audio-visual semantic segmentation method. SBV supplements the visual modality, which miss the information beyond FoV, with the auditory information using a teacher-student distillation model (Omni2Ego). The model consists of a vision teacher utilising panoramic information, an auditory teacher with 8-channel audio, and an audio-visual student that takes views with limited FoV and binaural audio as input and produce semantic segmentation for objects outside FoV. SBV outperforms existing models in comparative evaluations and shows a consistent performance across varying FoV ranges and in monaural audio settings.【11】 Low-rank constrained multichannel signal denoising considering channel-dependent sensitivity inspired by self-supervised learning for optical fiber sensing标题：基于自监督学习的低阶约束多通道光纤传感信号去噪链接：https://arxiv.org/abs/2312.08660作者：Noriyuki Tonami,Wataru Kohno,Sakiko Mishima,Yumi Arai,Reishi Kondo,Tomoyuki Hino备注：Accepted for ICASSP2024摘要：光纤传感是使用光纤检测音频、振动和温度的技术;特别是音频/振动感知传感被称为分布式声学传感（DAS）。在DAS中，观测数据由多通道数据组成，由于光学噪声或安装方法的原因，观测数据遭受严重的噪声水平。在用于对DAS数据进行去噪的常规方法中，已经研究了基于信号处理或深度神经网络（DNN）的模型。基于信号处理的方法具有可解释性，即，非黑盒基于DNN的方法擅长灵活设计网络结构和目标函数，即先验。然而，在DAS研究中，先验的可解释性和灵活性之间没有平衡。基于DNN的方法通常也需要大量的训练数据。为了解决这些问题，本文提出了一种基于DNN结构的信号处理去噪方法。作为DAS的先验，我们采用空间知识;使用基于DNN的结构的低秩和通道依赖的灵敏度。光纤声传感实验结果表明，该方法优于传统方法，且对空间秩数具有较好的鲁棒性。此外，所提出的方法的优化参数表明与通道灵敏度的关系，可解释性。摘要：Optical fiber sensing is a technology wherein audio, vibrations, and temperature are detected using an optical fiber; especially the audio/vibrations-aware sensing is called distributed acoustic sensing (DAS). In DAS, observed data, which is comprised of multichannel data, has suffered from severe noise levels because of the optical noise or the installation methods. In conventional methods for denoising DAS data, signal-processing- or deep-neural-network (DNN)-based models have been studied. The signal-processing-based methods have the interpretability, i.e., non-black box. The DNN-based methods are good at flexibility designing network architectures and objective functions, that is, priors. However, there is no balance between the interpretability and the flexibility of priors in the DAS studies. The DNN-based methods also require a large amount of training data in general. To address the problems, we propose a DNN-structure signal-processing-based denoising method in this paper. As the priors of DAS, we employ spatial knowledge; low rank and channel-dependent sensitivity using the DNN-based structure. The result of fiber-acoustic sensing shows that the proposed method outperforms the conventional methods and the robustness to the number of the spatial ranks. Moreover, the optimized parameters of the proposed method indicate the relationship with the channel sensitivity; the interpretability.【12】 PhasePerturbation: Speech Data Augmentation via Phase Perturbation for Automatic Speech Recognition标题：相位扰动：用于自动语音识别的相位扰动语音数据增强链接：https://arxiv.org/abs/2312.08571作者：Chengxi Lei,Satwinder Singh,Feng Hou,Xiaoyun Jia,Ruili Wang摘要：目前的语音数据增强方法大多是对语音的原始波形或幅度谱进行增强。在本文中，我们提出了一种新的语音数据增强方法称为相位扰动，动态操作的语音相位谱。PhasePerturbation利用三个动态相位谱操作，即，随机化操作、频率掩蔽操作和时间掩蔽操作，以增强语音数据的多样性。我们对wav2vec2.0预训练的ASR模型进行实验，通过PhasePerturbation增强TIMIT语料库对其进行微调。实验结果表明，与基线模型相比，在没有任何增强操作微调的字错误率（WER）的相对减少10.9%。此外，所提出的方法实现了额外的改进（12.9%和15.9%），通过补充声道长度扰动（VTLP）和SpecAug，这两个幅度谱为基础的增强方法在WER。结果突出了相位扰动的能力，以改善目前的幅度谱为基础的增强方法。摘要：Most of the current speech data augmentation methods operate on either the raw waveform or the amplitude spectrum of speech. In this paper, we propose a novel speech data augmentation method called PhasePerturbation that operates dynamically on the phase spectrum of speech. Instead of statically rotating a phase by a constant degree, PhasePerturbation utilizes three dynamic phase spectrum operations, i.e., a randomization operation, a frequency masking operation, and a temporal masking operation, to enhance the diversity of speech data. We conduct experiments on wav2vec2.0 pre-trained ASR models by fine-tuning them with the PhasePerturbation augmented TIMIT corpus. The experimental results demonstrate 10.9\% relative reduction in the word error rate (WER) compared with the baseline model fine-tuned without any augmentation operation. Furthermore, the proposed method achieves additional improvements (12.9\% and 15.9\%) in WER by complementing the Vocal Tract Length Perturbation (VTLP) and the SpecAug, which are both amplitude spectrum-based augmentation methods. The results highlight the capability of PhasePerturbation to improve the current amplitude spectrum-based augmentation methods.【13】 PerMod: Perceptually Grounded Voice Modification with Latent Diffusion Models标题：PerMod：基于潜在扩散模型的感知基础语音修正链接：https://arxiv.org/abs/2312.08494作者：Robin Netzorg,Ajil Jalal,Luna McNulty,Gopala Krishna Anumanchipalli摘要：语音的感知修改是一个难以捉摸的目标。虽然非专家可以用可用的工具在感知上修改图像或句子，但不清楚如何沿着感知轴类似地修改语音。语音转换确实可以将一种语音转换为另一种语音，但这些修改是由黑盒模型处理的，并且修改哪些感知质量以及如何修改它们的细节尚不清楚。为了让更大的感知控制的声音，我们引入PerMod，一个有条件的潜在扩散模型，需要在一个输入的声音和感知质量向量，并产生一个声音与匹配的感知质量。与以前的工作不同，PerMod生成一个新的声音对应于特定的感知修改。从人类和预测标签的RMSE评估感知质量向量，我们证明了PerMod产生的声音与典型的声音所需的感知质量，但表现不佳的非典型的声音。摘要：Perceptual modification of voice is an elusive goal. While non-experts can modify an image or sentence perceptually with available tools, it is not clear how to similarly modify speech along perceptual axes. Voice conversion does make it possible to convert one voice to another, but these modifications are handled by black box models, and the specifics of what perceptual qualities to modify and how to modify them are unclear. Towards allowing greater perceptual control over voice, we introduce PerMod, a conditional latent diffusion model that takes in an input voice and a perceptual qualities vector, and produces a voice with the matching perceptual qualities. Unlike prior work, PerMod generates a new voice corresponding to specific perceptual modifications. Evaluating perceptual quality vectors with RMSE from both human and predicted labels, we demonstrate that PerMod produces voices with the desired perceptual qualities for typical voices, but performs poorly on atypical voices.【14】 FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models标题：FaceTalk：用于神经参数头部模型的音频驱动运动扩散链接：https://arxiv.org/abs/2312.08459作者：Shivangi Aneja,Justus Thies,Angela Dai,Matthias Nießner备注：Paper Video: this https URL Project Page: this https URL摘要：我们介绍FaceTalk，一种新的生成方法，设计用于从输入音频信号合成高保真的3D运动序列的说话人的头。为了捕捉人类头部的表现力，细节的性质，包括头发，耳朵和更精细的眼球运动，我们建议将语音信号与神经参数头部模型的潜在空间耦合，以创建高保真，时间相干的运动序列。为此，我们提出了一个新的潜在扩散模型，在神经参数头部模型的表达空间中操作，以合成音频驱动的逼真头部序列。在没有与音频对应的NPHM表达式的数据集的情况下，我们针对这些对应关系进行优化，以产生适合人们谈话的音频-视频记录的时间优化的NPHM表达式的数据集。据我们所知，这是第一个工作，提出了一个生成的方法，逼真和高质量的运动合成的体积人头，代表了音频驱动的3D动画领域的重大进展。值得注意的是，我们的方法脱颖而出的能力，以产生合理的运动序列，可以产生高保真度的头部动画加上NPHM形状空间。我们的实验结果证实了FaceTalk的有效性，始终实现卓越和视觉自然的运动，包括不同的面部表情和风格，在感知用户研究评估中超过现有方法75%。摘要：We introduce FaceTalk, a novel generative approach designed for synthesizing high-fidelity 3D motion sequences of talking human heads from input audio signal. To capture the expressive, detailed nature of human heads, including hair, ears, and finer-scale eye movements, we propose to couple speech signal with the latent space of neural parametric head models to create high-fidelity, temporally coherent motion sequences. We propose a new latent diffusion model for this task, operating in the expression space of neural parametric head models, to synthesize audio-driven realistic head sequences. In the absence of a dataset with corresponding NPHM expressions to audio, we optimize for these correspondences to produce a dataset of temporally-optimized NPHM expressions fit to audio-video recordings of people talking. To the best of our knowledge, this is the first work to propose a generative approach for realistic and high-quality motion synthesis of volumetric human heads, representing a significant advancement in the field of audio-driven 3D animation. Notably, our approach stands out in its ability to generate plausible motion sequences that can produce high-fidelity head animation coupled with the NPHM shape space. Our experimental results substantiate the effectiveness of FaceTalk, consistently achieving superior and visually natural motion, encompassing diverse facial expressions and styles, outperforming existing methods by 75% in perceptual user study evaluation.【15】 EmbAu: A Novel Technique to Embed Audio Data Using Shuffled Frog Leaping Algorithm标题： EmbAu：一种基于混合蛙跳算法的音频数据嵌入技术链接：https://arxiv.org/abs/2312.08417作者：Sahil Nokhwal,Saurabh Pahune,Ankit Chaudhary摘要：隐写算法的目的是确定宿主图像或封面图像中合适的像素位置，在这些位置可以隐藏敏感信息以进行数据加密。目前正在努力提高整合敏感信息的能力，并保持隐写图像的视觉外观。因此，隐写术是一个具有挑战性的研究领域。在我们目前提出的图像隐写技术中，我们使用了混洗蛙跳算法（SFLA）来确定敏感信息可以放置在封面图像中的像素顺序。为了实现更大的嵌入容量，仔细选择来自封面图像的空间域的像素并用于放置敏感数据。通过图像隐写术的支持，嵌入后的最终图像可以抵抗隐写分析攻击。SFLA算法用于任何彩色（RGB）覆盖图像的最佳像素选择，用于秘密比特嵌入。使用适应度函数，SFLA通过在可接受的时间量内达到最小成本值而受益。嵌入的像素是精心选择的，以最大限度地减少嵌入时宿主图像的失真。此外，已采取的努力，使隐藏图像中的嵌入数据的检测是一个艰巨的挑战。由于当今世界对音频数据加密的巨大需求，我们认为我们提出的方法在现实世界的应用中具有巨大的潜力。在本文中，我们提出并比较我们的策略，现有的隐写方法。摘要：The aim of steganographic algorithms is to identify the appropriate pixel positions in the host or cover image, where bits of sensitive information can be concealed for data encryption. Work is being done to improve the capacity to integrate sensitive information and to maintain the visual appearance of the steganographic image. Consequently, steganography is a challenging research area. In our currently proposed image steganographic technique, we used the Shuffled Frog Leaping Algorithm (SFLA) to determine the order of pixels by which sensitive information can be placed in the cover image. To achieve greater embedding capacity, pixels from the spatial domain of the cover image are carefully chosen and used for placing the sensitive data. Bolstered via image steganography, the final image after embedding is resistant to steganalytic attacks. The SFLA algorithm serves in the optimal pixels selection of any colored (RGB) cover image for secret bit embedding. Using the fitness function, the SFLA benefits by reaching a minimum cost value in an acceptable amount of time. The pixels for embedding are meticulously chosen to minimize the host image's distortion upon embedding. Moreover, an effort has been taken to make the detection of embedded data in the steganographic image a formidable challenge. Due to the enormous need for audio data encryption in the current world, we feel that our suggested method has significant potential in real-world applications. In this paper, we propose and compare our strategy to existing steganographic methods.【16】 FastInject: Injecting Unpaired Text Data into CTC-based ASR training标题： FastInject：将未配对文本数据注入基于CTC的ASR训练链接：https://arxiv.org/abs/2312.09100作者：Keqi Deng,Philip C. Woodland备注：Accepted by ICASSP2024摘要：最近，基于连接主义时态分类（CTC）的端到端（E2 E）自动语音识别（ASR）模型取得了令人印象深刻的结果，特别是随着自监督学习的发展。然而，在成对的语音-文本数据上训练的E2 E ASR模型经常遭受从训练到测试的域转移。为了缓解这个问题，本文提出了一种平坦开始的联合训练方法，命名为FastInject，该方法有效地将多域不成对文本数据注入到基于CTC的ASR训练中。为了保持训练效率，对文本单元进行预上采样，并将其表示与语音特征一起输入CTC模型。为了弥合语音和文本之间的模态鸿沟，提出了一种基于注意力的模态匹配机制（AM3），该机制保留了E2 E平坦开始训练。实验结果表明，该算法对Librispeech-100 h域内数据的相对WER降低率为22%，对域外数据的相对WER降低率为20%.摘要：Recently, connectionist temporal classification (CTC)-based end-to-end (E2E) automatic speech recognition (ASR) models have achieved impressive results, especially with the development of self-supervised learning. However, E2E ASR models trained on paired speech-text data often suffer from domain shifts from training to testing. To alleviate this issue, this paper proposes a flat-start joint training method, named FastInject, which efficiently injects multi-domain unpaired text data into CTC-based ASR training. To maintain training efficiency, text units are pre-upsampled, and their representations are fed into the CTC model along with speech features. To bridge the modality gap between speech and text, an attention-based modality matching mechanism (AM3) is proposed, which retains the E2E flat-start training. Experiments show that the proposed FastInject gave a 22\% relative WER reduction (WERR) for intra-domain Librispeech-100h data and 20\% relative WERR on out-of-domain test sets.【17】 Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection标题：用于声事件定位和检测的音视频融合嵌入链接：https://arxiv.org/abs/2312.09034作者：Davide Berghi,Peipei Wu,Jinzheng Zhao,Wenwu Wang,Philip J. B. Jackson备注：ICASSP 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：声事件定位与检测（SELD）包括两个子任务：声事件检测（SED）和到达方向（DOA）估计。SELD通常作为一个仅音频的问题来处理，但最近已经包括了视觉信息。很少有视听（AV）-SELD作品已经出版，大多数采用通过面部/对象边界框或人体姿势关键点的视觉。相比之下，我们探索了用预训练的深度网络提取的音频和视觉特征嵌入的集成。对于视觉模态，我们测试了ResNet 50和Inflated 3D ConvNet（I3 D）。我们的AV融合方法的比较，包括AV一致性和跨模态注意力融合（CMAF）模型。我们最好的模型在STARSS 23数据集的开发集上远远优于DCASE 2023 Task 3纯音频和AV基线，使其在AV挑战的最先进结果中具有竞争力，无需模型集成，大量数据增强或预测后处理。这些技术和进一步的预先培训可以作为提高业绩的下一步。摘要：Sound event localization and detection (SELD) combines two subtasks: sound event detection (SED) and direction of arrival (DOA) estimation. SELD is usually tackled as an audio-only problem, but visual information has been recently included. Few audio-visual (AV)-SELD works have been published and most employ vision via face/object bounding boxes, or human pose keypoints. In contrast, we explore the integration of audio and visual feature embeddings extracted with pre-trained deep networks. For the visual modality, we tested ResNet50 and Inflated 3D ConvNet (I3D). Our comparison of AV fusion methods includes the AV-Conformer and Cross-Modal Attentive Fusion (CMAF) model. Our best models outperform the DCASE 2023 Task3 audio-only and AV baselines by a wide margin on the development set of the STARSS23 dataset, making them competitive amongst state-of-the-art results of the AV challenge, without model ensembling, heavy data augmentation, or prediction post-processing. Such techniques and further pre-training could be applied as next steps to improve performance.【18】 Design, construction and evaluation of emotional multimodal pathological speech database标题：情感多通道病态语音库的设计、构建与评价链接：https://arxiv.org/abs/2312.08998作者：Ting Zhu,Shufei Duan,Huizhi Liang,Wei Zhang摘要：情感病理学数据库的缺乏是研究构音障碍患者情感表达状况的主要障碍之一。本文构建了第一个包含多视角信息的汉语多模态情感病理语音数据库。包括29名对照组和39名不同程度的运动性构音障碍患者，表达快乐、悲伤、愤怒和中性情绪。所有情感语音都通过开发的微信小程序标记为可理解度，类型和离散维度情感。主观分析从情感识别准确率、言语清晰度、效价-唤醒空间分布、SCL-90与疾病严重程度的相关性等方面进行验证。自动识别测试语音和声门数据，平均准确率为78%的控制和60%的患者在音频中，而51%的控制和38%的患者在声门数据，表明疾病的情绪表达的影响。摘要：The lack of an available emotion pathology database is one of the key obstacles in studying the emotion expression status of patients with dysarthria. The first Chinese multimodal emotional pathological speech database containing multi-perspective information is constructed in this paper. It includes 29 controls and 39 patients with different degrees of motor dysarthria, expressing happy, sad, angry and neutral emotions. All emotional speech was labeled for intelligibility, types and discrete dimensional emotions by developed WeChat mini-program. The subjective analysis justifies from emotion discrimination accuracy, speech intelligibility, valence-arousal spatial distribution, and correlation between SCL-90 and disease severity. The automatic recognition tested on speech and glottal data, with average accuracy of 78% for controls and 60% for patients in audio, while 51% for controls and 38% for patients in glottal data, indicating an influence of the disease on emotional expression.【19】 Attention-Guided Adaptation for Code-Switching Speech Recognition标题： 注意力引导的自适应码转换语音识别链接：https://arxiv.org/abs/2312.08856作者：Bobbi Aditya,Mahdin Rohmatillah,Liang-Hsuan Tai,Jen-Tzung Chien备注：Accepted to ICASSP 2024摘要：Whisper等多种语言模型的出现，极大地推动了语音识别的研究。然而，这些模型通常难以处理代码切换设置，这在多语言语音识别中至关重要。最近的研究试图通过分离不同语言的模块来解决这个问题，以确保语言的不同潜在表征。其他一些方法考虑了基于语言识别的切换机制。在这项研究中，提出了一种新的注意力引导的适应进行参数有效的学习双语ASR。该方法在模型中选取与语言身份密切相关的注意中心，并引导这些注意中心与其对应的语言进行正确的注意。在汉英语码转换语音语料库上的实验表明，该方法的混合错误率为14.2%，超过了现有的只训练了5.6%的Whisper附加参数的方法。摘要：The prevalence of the powerful multilingual models, such as Whisper, has significantly advanced the researches on speech recognition. However, these models often struggle with handling the code-switching setting, which is essential in multilingual speech recognition. Recent studies have attempted to address this setting by separating the modules for different languages to ensure distinct latent representations for languages. Some other methods considered the switching mechanism based on language identification. In this study, a new attention-guided adaptation is proposed to conduct parameter-efficient learning for bilingual ASR. This method selects those attention heads in a model which closely express language identities and then guided those heads to be correctly attended with their corresponding languages. The experiments on the Mandarin-English code-switching speech corpus show that the proposed approach achieves a 14.2% mixed error rate, surpassing state-of-the-art method, where only 5.6% additional parameters over Whisper are trained.【20】 Reconstruction of Sound Field through Diffusion Models标题：基于扩散模型的声场重建链接：https://arxiv.org/abs/2312.08821作者：Federico Miotello,Luca Comanducci,Mirco Pezzoli,Alberto Bernardini,Fabio Antonacci,Augusto Sarti备注：Accepted for publication at ICASSP 2024摘要：重建房间中的声场对于诸如声音控制和增强（AR）或虚拟现实（VR）之类的若干应用来说是一项重要任务。在本文中，我们提出了一个数据驱动的生成模型，用于重建房间中的声场幅度，重点是模态频率范围。我们介绍，第一次，使用一个条件去噪扩散概率模型（DDPM）训练，以重建声场（SF-Diff）在扩展域。该架构被设计为以不同频率的一组有限的可用测量为条件，并在目标未知位置生成声场。结果表明，SF-Diff能够提供准确的重建，优于基于核插值的最先进的基线。摘要：Reconstructing the sound field in a room is an important task for several applications, such as sound control and augmented (AR) or virtual reality (VR). In this paper, we propose a data-driven generative model for reconstructing the magnitude of acoustic fields in rooms with a focus on the modal frequency range. We introduce, for the first time, the use of a conditional Denoising Diffusion Probabilistic Model (DDPM) trained in order to reconstruct the sound field (SF-Diff) over an extended domain. The architecture is devised in order to be conditioned on a set of limited available measurements at different frequencies and generate the sound field in target, unknown, locations. The results show that SF-Diff is able to provide accurate reconstructions, outperforming a state-of-the-art baseline based on kernel interpolation.【21】 Towards Automatic Data Augmentation for Disordered Speech Recognition标题： 无序语音识别中的自动数据扩充链接：https://arxiv.org/abs/2312.08641作者：Zengrui Jin,Xurong Xie,Tianzi Wang,Mengzhe Geng,Jiajun Deng,Guinan Li,Shujie Hu,Xunying Liu备注：To appear at IEEE ICASSP 2024摘要：迄今为止，由于数据稀缺，无序语音的自动识别仍然是一项极具挑战性的任务。本文提出了一种基于强化学习（RL）的动态数据增强方法，用于在此类数据上训练最先进的PyChain TDNN和端到端Conformer ASR系统。标准SpecAugment方法中的手工时间和频谱掩码操作依赖于任务和系统，以及额外引入的这些时间频率掩码的最小和最大截止值，现在可以使用基于RNN的策略控制器自动学习，并与ASR系统训练紧密集成。UASpeech语料库上的实验表明，所提出的基于RL的数据增强方法始终产生优于或可比的性能，使用专家或手工制作的SpecAugment政策。我们的RL自动增强PyChain TDNN系统在16个构音障碍扬声器的UASpeech测试集上产生了28.79%的总体WER。摘要：Automatic recognition of disordered speech remains a highly challenging task to date due to data scarcity. This paper presents a reinforcement learning (RL) based on-the-fly data augmentation approach for training state-of-the-art PyChain TDNN and end-to-end Conformer ASR systems on such data. The handcrafted temporal and spectral mask operations in the standard SpecAugment method that are task and system dependent, together with additionally introduced minimum and maximum cut-offs of these time-frequency masks, are now automatically learned using an RNN-based policy controller and tightly integrated with ASR system training. Experiments on the UASpeech corpus suggest the proposed RL-based data augmentation approach consistently produced performance superior or comparable that obtained using expert or handcrafted SpecAugment policies. Our RL auto-augmented PyChain TDNN system produced an overall WER of 28.79% on the UASpeech test set of 16 dysarthric speakers.【22】 Scalable Ensemble-based Detection Method against Adversarial Attacks for speaker verification标题： 一种抗对抗性攻击的可扩展Ensemble说话人确认检测方法链接：https://arxiv.org/abs/2312.08622作者：Haibin Wu,Heng-Cheng Kuo,Yu Tsao,Hung-yi Lee备注：Submitted to 2024 ICASSP摘要：自动说话人确认（ASV）是一种非常容易受到对抗性攻击的方法。净化模块通常被用作预处理以减轻对抗性噪声。然而，它们通常在不同的实验环境中实施，使得直接比较具有挑战性。本文在一个统一的框架内对主流净化技术进行了综合比较。我们发现这些方法通常面临用户体验和安全性之间的权衡，因为它们很难同时保持真实的样本性能并减少对抗性干扰。为了应对这一挑战，一些努力已经扩展了净化模块以包括检测能力，旨在减轻权衡。然而，先进的净化模块总是会进入阶段，以超越以往的检测方法。因此，我们进一步提出了一种易于遵循的集成方法，该方法集成了先进的净化模块进行检测，在对抗对抗性噪声方面实现了最先进的（SOTA）性能。我们的合奏方法有很大的潜力，由于其与未来先进的净化技术的兼容性。摘要：Automatic speaker verification (ASV) is highly susceptible to adversarial attacks. Purification modules are usually adopted as a pre-processing to mitigate adversarial noise. However, they are commonly implemented across diverse experimental settings, rendering direct comparisons challenging. This paper comprehensively compares mainstream purification techniques in a unified framework. We find these methods often face a trade-off between user experience and security, as they struggle to simultaneously maintain genuine sample performance and reduce adversarial perturbations. To address this challenge, some efforts have extended purification modules to encompass detection capabilities, aiming to alleviate the trade-off. However, advanced purification modules will always come into the stage to surpass previous detection method. As a result, we further propose an easy-to-follow ensemble approach that integrates advanced purification modules for detection, achieving state-of-the-art (SOTA) performance in countering adversarial noise. Our ensemble method has great potential due to its compatibility with future advanced purification techniques.【23】 A computationally efficient semi-blind source separation based approach for nonlinear echo cancellation based on an element-wise iterative source steering标题：一种计算高效的基于半盲源分离的基于单元迭代信源操纵的非线性回波抵消方法链接：https://arxiv.org/abs/2312.08610作者：Kunxing Lu,Xianrui Wang,Tetsuya Ueda,Shoji Makino,Jingdong Chen摘要：与传统的自适应算法相比，基于半盲源分离的回声消除算法（SBSS-AEC）在双端通话时具有良好的性能，因此受到了广泛的关注，但它存在系统延迟和非线性失真的问题。为了克服这些缺点，近年来发展起来的卷积传递函数（CTF）逼近和非线性展开思想被用于基于迭代投影（IP）的半盲源分离（SBSS）算法中。然而，由于CTF近似和非线性扩展的引入，该算法变得非常昂贵的计算，这使得它很难在嵌入式系统中实现。因此，我们试图在本文中，以改善这种基于IP的算法，从而开发一个元素的迭代源转向（EISS）算法。与基于IP的SBSS算法相比，该算法的计算效率更高，特别是在非线性展开阶数较高和CTF滤波器长度较长的情况下。同时，其AEC性能与基于IP的SBSS相当。摘要：While the semi-blind source separation-based acoustic echo cancellation (SBSS-AEC) has received much research attention due to its promising performance during double-talk compared to the traditional adaptive algorithms, it suffers from system latency and nonlinear distortions. To circumvent these drawbacks, the recently developed ideas on convolutive transfer function (CTF) approximation and nonlinear expansion have been used in the iterative projection (IP)-based semi-blind source separation (SBSS) algorithm. However, because of the introduction of CTF approximation and nonlinear expansion, this algorithm becomes computationally very expensive, which makes it difficult to implement in embedded systems. Thus, we attempt in this paper to improve this IP-based algorithm, thereby developing an element-wise iterative source steering (EISS) algorithm. In comparison with the IP-based SBSS algorithm, the proposed algorithm is computationally much more efficient, especially when the nonlinear expansion order is high and the length of the CTF filter is long. Meanwhile, its AEC performance is as good as that of IP-based SBSS.【24】 NeXt-TDNN: Modernizing Multi-Scale Temporal Convolution Backbone for Speaker Verification标题：Next-TDNN：用于说话人确认的多尺度时间卷积主干的现代化链接：https://arxiv.org/abs/2312.08603作者：Hyun-Jun Heo,Ui-Hyeop Shin,Ran Lee,YoungJu Cheon,Hyung-Min Park备注：Accepted by ICASSP 2024摘要：在说话人确认中，ECAPA-TDNN通过使用一维（1D）Res 2Net块和挤压和激励（SE）模块以及多层特征聚合（MFA），表现出显着的改进。同时，在视觉任务中，ConvNet结构通过引用Transformer进行了现代化，从而提高了性能。在本文中，我们提出了一种改进的块设计TDNN在说话人确认。受最近ConvNet结构的启发，我们将ECAPA-TDNN中的SE-Res 2Net块替换为一个新的1D两步多尺度ConvNeXt块，我们称之为\textit{TS-ConvNeXt}。TS-ConvNeXt块使用两个独立的子模块构建：时间多尺度卷积（MSC）和逐帧前馈网络（FFN）。这种两步设计允许灵活地捕获帧间和帧内上下文。此外，我们为FFN模块引入了全局响应归一化（GRN），以实现更具选择性的特征传播，类似于ECAPA-TDNN中的SE模块。实验结果表明，NeXt-TDNN，与现代化的骨干块，显着提高了说话人验证任务的性能，同时减少参数大小和推理时间。我们已经发布了我们的代码，供未来研究使用。摘要：In speaker verification, ECAPA-TDNN has shown remarkable improvement by utilizing one-dimensional(1D) Res2Net block and squeeze-and-excitation(SE) module, along with multi-layer feature aggregation (MFA). Meanwhile, in vision tasks, ConvNet structures have been modernized by referring to Transformer, resulting in improved performance. In this paper, we present an improved block design for TDNN in speaker verification. Inspired by recent ConvNet structures, we replace the SE-Res2Net block in ECAPA-TDNN with a novel 1D two-step multi-scale ConvNeXt block, which we call \textit{TS-ConvNeXt}. The TS-ConvNeXt block is constructed using two separated sub-modules: a temporal multi-scale convolution (MSC) and a frame-wise feed-forward network (FFN). This two-step design allows for flexible capturing of inter-frame and intra-frame contexts. Additionally, we introduce global response normalization (GRN) for the FFN modules to enable more selective feature propagation, similar to the SE module in ECAPA-TDNN. Experimental results demonstrate that NeXt-TDNN, with a modernized backbone block, significantly improved performance in speaker verification tasks while reducing parameter size and inference time. We have released our code for future studies.【25】 USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models标题：USM-Lite：基于通用语音模型的量化和稀疏性精调链接：https://arxiv.org/abs/2312.08553作者：Shaojin Ding,Qiu David,David Rim,Yanzhang He,Oleg Rybakov,Bo Li,Rohit Prabhavalkar,Weiran Wang,Tara N. Sainath,Shivani Agrawal,Zhonglin Han,Jian Li,Amir Yazdanbakhsh备注：Accepted by ICASSP 2024. Preprint摘要：随着最近大规模通用语音模型（USM）的发展，端到端自动语音识别（ASR）模型已经看到了革命性的质量提升。然而，部署这些大规模的USM是非常昂贵的，由于巨大的内存使用和计算成本。因此，模型压缩是一个重要的研究课题，以适应基于USM的ASR在预算下，在现实世界的场景。在这项研究中，我们提出了一个USM微调方法的ASR，低比特量化和N：M结构稀疏意识的范式模型的权重，从参数精度和矩阵拓扑的角度降低模型的复杂性。我们在大规模语音搜索数据集上使用20亿参数USM进行了广泛的实验，以评估我们提出的方法。一系列消融研究验证了高达int4量化和2：4稀疏度的有效性。然而，在包括int2量化和1：4稀疏度的极端设置下，单一压缩技术无法很好地恢复性能。相比之下，我们提出的方法可以压缩模型的大小为9.4%，成本只有7.3%的相对字错误率（WER）回归。我们还对结果进行了深入分析，并就局限性和潜在解决方案进行了讨论，这对未来的研究很有价值。摘要：End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N:M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a large-scale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.【26】 Metrological support of acoustic measuring installations mid-frequency devices标题：声学测量装置中频装置的计量支持链接：https://arxiv.org/abs/2312.08496作者：A. N. Grekov,N. A. Grekov,E. N. Sychev备注：None摘要：本文讨论了测量液体中声速、声散射、声衰减和声吸收的方法，并在此基础上绘制了现代装置的结构图。在实际条件下，声学测量方案并不理想，并且根据具体结构可能给出不同的结果。本文介绍了现代声学装置测量通道的技术和工艺特点。文中对GOST在声学测量中的应用提出了建议，并指出新研制的测量仪器在现场条件下的声特性达到了国家一级标准。摘要：The article discusses methods for measuring the speed of sound, scattering, attenuation and absorption of sound in liquids, on the basis of which the structural diagrams of modern devices have been developed. In real conditions, acoustic measurement schemes are not ideal and depending on specific structure may give different results. The technical and metrological characteristics of the measuring channels of modern acoustic devices are presented. Recommendations are given for the use of GOST in acoustic measurements and it is stated that the metrological characteristics of newly created measuring instruments operating in in situ conditions are at the level of State primary standards.【1】 FastInject: Injecting Unpaired Text Data into CTC-based ASR training标题：FastInject：将未配对的文本数据注入基于CTC的ASR训练链接：https://arxiv.org/abs/2312.09100作者：Keqi Deng,Philip C. Woodland备注：Accepted by ICASSP2024摘要：最近，基于连接主义时态分类（CTC）的端到端（E2 E）自动语音识别（ASR）模型取得了令人印象深刻的结果，特别是随着自监督学习的发展。然而，在成对的语音-文本数据上训练的E2 E ASR模型经常遭受从训练到测试的域转移。为了缓解这个问题，本文提出了一种平坦开始的联合训练方法，命名为FastInject，该方法有效地将多域不成对文本数据注入到基于CTC的ASR训练中。为了保持训练效率，对文本单元进行预上采样，并将其表示与语音特征一起输入CTC模型。为了弥合语音和文本之间的模态鸿沟，提出了一种基于注意力的模态匹配机制（AM3），该机制保留了E2 E平坦开始训练。实验结果表明，该算法对Librispeech-100 h域内数据的相对WER降低率为22%，对域外数据的相对WER降低率为20%.摘要：Recently, connectionist temporal classification (CTC)-based end-to-end (E2E) automatic speech recognition (ASR) models have achieved impressive results, especially with the development of self-supervised learning. However, E2E ASR models trained on paired speech-text data often suffer from domain shifts from training to testing. To alleviate this issue, this paper proposes a flat-start joint training method, named FastInject, which efficiently injects multi-domain unpaired text data into CTC-based ASR training. To maintain training efficiency, text units are pre-upsampled, and their representations are fed into the CTC model along with speech features. To bridge the modality gap between speech and text, an attention-based modality matching mechanism (AM3) is proposed, which retains the E2E flat-start training. Experiments show that the proposed FastInject gave a 22\% relative WER reduction (WERR) for intra-domain Librispeech-100h data and 20\% relative WERR on out-of-domain test sets.【2】 Fusion of Audio and Visual Embeddings for Sound Event Localization and Detection标题：用于声事件定位和检测的音视频融合嵌入链接：https://arxiv.org/abs/2312.09034作者：Davide Berghi,Peipei Wu,Jinzheng Zhao,Wenwu Wang,Philip J. B. Jackson备注：ICASSP 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)摘要：声事件定位与检测（SELD）包括两个子任务：声事件检测（SED）和到达方向（DOA）估计。SELD通常作为一个仅音频的问题来处理，但最近已经包括了视觉信息。很少有视听（AV）-SELD作品已经出版，大多数采用通过面部/对象边界框或人体姿势关键点的视觉。相比之下，我们探索了用预训练的深度网络提取的音频和视觉特征嵌入的集成。对于视觉模态，我们测试了ResNet 50和Inflated 3D ConvNet（I3 D）。我们的AV融合方法的比较，包括AV一致性和跨模态注意力融合（CMAF）模型。我们最好的模型在STARSS 23数据集的开发集上远远优于DCASE 2023 Task 3纯音频和AV基线，使其在AV挑战的最先进结果中具有竞争力，无需模型集成，大量数据增强或预测后处理。这些技术和进一步的预先培训可以作为提高业绩的下一步。摘要：Sound event localization and detection (SELD) combines two subtasks: sound event detection (SED) and direction of arrival (DOA) estimation. SELD is usually tackled as an audio-only problem, but visual information has been recently included. Few audio-visual (AV)-SELD works have been published and most employ vision via face/object bounding boxes, or human pose keypoints. In contrast, we explore the integration of audio and visual feature embeddings extracted with pre-trained deep networks. For the visual modality, we tested ResNet50 and Inflated 3D ConvNet (I3D). Our comparison of AV fusion methods includes the AV-Conformer and Cross-Modal Attentive Fusion (CMAF) model. Our best models outperform the DCASE 2023 Task3 audio-only and AV baselines by a wide margin on the development set of the STARSS23 dataset, making them competitive amongst state-of-the-art results of the AV challenge, without model ensembling, heavy data augmentation, or prediction post-processing. Such techniques and further pre-training could be applied as next steps to improve performance.【3】 Design, construction and evaluation of emotional multimodal pathological speech database标题：情感多通道病态语音库的设计、构建与评价链接：https://arxiv.org/abs/2312.08998作者：Ting Zhu,Shufei Duan,Huizhi Liang,Wei Zhang摘要：情感病理学数据库的缺乏是研究构音障碍患者情感表达状况的主要障碍之一。本文构建了第一个包含多视角信息的汉语多模态情感病理语音数据库。包括29名对照组和39名不同程度的运动性构音障碍患者，表达快乐、悲伤、愤怒和中性情绪。所有情感语音都通过开发的微信小程序标记为可理解度，类型和离散维度情感。主观分析从情感识别准确率、言语清晰度、效价-唤醒空间分布、SCL-90与疾病严重程度的相关性等方面进行验证。自动识别测试语音和声门数据，平均准确率为78%的控制和60%的患者在音频中，而51%的控制和38%的患者在声门数据，表明疾病的情绪表达的影响。摘要：The lack of an available emotion pathology database is one of the key obstacles in studying the emotion expression status of patients with dysarthria. The first Chinese multimodal emotional pathological speech database containing multi-perspective information is constructed in this paper. It includes 29 controls and 39 patients with different degrees of motor dysarthria, expressing happy, sad, angry and neutral emotions. All emotional speech was labeled for intelligibility, types and discrete dimensional emotions by developed WeChat mini-program. The subjective analysis justifies from emotion discrimination accuracy, speech intelligibility, valence-arousal spatial distribution, and correlation between SCL-90 and disease severity. The automatic recognition tested on speech and glottal data, with average accuracy of 78% for controls and 60% for patients in audio, while 51% for controls and 38% for patients in glottal data, indicating an influence of the disease on emotional expression.【4】 Multi-Microphone Noise Data Augmentation for DNN-based Own Voice Reconstruction for Hearables in Noisy Environments标题：噪声环境下基于DNN的自语音重建中的多麦克风噪声数据增强链接：https://arxiv.org/abs/2312.08908作者：Mattes Ohlenbusch,Christian Rollwage,Simon Doclo备注：ICASSP 2024 (c) 2024 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works摘要：具有集成麦克风的可听设备可以在嘈杂的工作环境中提供通信益处，例如，通过传输用户自己录制的语音。旨在从嘈杂的麦克风录音中重建干净和全带宽自己的语音的系统通常基于监督学习。记录训练这种系统所需的足够量的噪声是昂贵的，因为外部麦克风和内部麦克风之间的噪声传输单独地变化。以前提出的方法要么不考虑噪声，只考虑外部麦克风处的噪声，要么假设内部和外部麦克风噪声在训练期间是独立的，并且还不清楚个性化噪声是否可以有益于语音重建系统的训练和自身。在本文中，我们研究了几个噪声数据增强技术的基础上测量的传递函数来模拟多麦克风噪声。利用增强噪声，我们训练了一个多通道自声重建系统。使用真实噪声进行实验，以调查的泛化能力。结果表明，将增强噪声产生很大的好处，特别是考虑到个性化的噪声增强导致更高的性能。摘要：Hearables with integrated microphones may offer communication benefits in noisy working environments, e.g. by transmitting the recorded own voice of the user. Systems aiming at reconstructing the clean and full-bandwidth own voice from noisy microphone recordings are often based on supervised learning. Recording a sufficient amount of noise required for training such a system is costly since noise transmission between outer and inner microphones varies individually. Previously proposed methods either do not consider noise, only consider noise at outer microphones or assume inner and outer microphone noise to be independent during training, and it is not yet clear whether individualized noise can benefit the training of and own voice reconstruction system. In this paper, we investigate several noise data augmentation techniques based on measured transfer functions to simulate multi-microphone noise. Using augmented noise, we train a multi-channel own voice reconstruction system. Experiments using real noise are carried out to investigate the generalization capability. Results show that incorporating augmented noise yields large benefits, in particular considering individualized noise augmentation leads to higher performance.【5】 Attention-Guided Adaptation for Code-Switching Speech Recognition标题：码型转换语音识别中的注意力引导自适应链接：https://arxiv.org/abs/2312.08856作者：Bobbi Aditya,Mahdin Rohmatillah,Liang-Hsuan Tai,Jen-Tzung Chien备注：Accepted to ICASSP 2024摘要：Whisper等多种语言模型的出现，极大地推动了语音识别的研究。然而，这些模型通常难以处理代码切换设置，这在多语言语音识别中至关重要。最近的研究试图通过分离不同语言的模块来解决这个问题，以确保语言的不同潜在表征。其他一些方法考虑了基于语言识别的切换机制。在这项研究中，提出了一种新的注意力引导的适应进行参数有效的学习双语ASR。该方法在模型中选取与语言身份密切相关的注意中心，并引导这些注意中心与其对应的语言进行正确的注意。在汉英语码转换语音语料库上的实验表明，该方法的混合错误率为14.2%，超过了现有的只训练了5.6%的Whisper附加参数的方法。摘要：The prevalence of the powerful multilingual models, such as Whisper, has significantly advanced the researches on speech recognition. However, these models often struggle with handling the code-switching setting, which is essential in multilingual speech recognition. Recent studies have attempted to address this setting by separating the modules for different languages to ensure distinct latent representations for languages. Some other methods considered the switching mechanism based on language identification. In this study, a new attention-guided adaptation is proposed to conduct parameter-efficient learning for bilingual ASR. This method selects those attention heads in a model which closely express language identities and then guided those heads to be correctly attended with their corresponding languages. The experiments on the Mandarin-English code-switching speech corpus show that the proposed approach achieves a 14.2% mixed error rate, surpassing state-of-the-art method, where only 5.6% additional parameters over Whisper are trained.【6】 Reconstruction of Sound Field through Diffusion Models标题：基于扩散模型的声场重建链接：https://arxiv.org/abs/2312.08821作者：Federico Miotello,Luca Comanducci,Mirco Pezzoli,Alberto Bernardini,Fabio Antonacci,Augusto Sarti备注：Accepted for publication at ICASSP 2024摘要：重建房间中的声场对于诸如声音控制和增强（AR）或虚拟现实（VR）之类的若干应用来说是一项重要任务。在本文中，我们提出了一个数据驱动的生成模型，用于重建房间中的声场幅度，重点是模态频率范围。我们介绍，第一次，使用一个条件去噪扩散概率模型（DDPM）训练，以重建声场（SF-Diff）在扩展域。该架构被设计为以不同频率的一组有限的可用测量为条件，并在目标未知位置生成声场。结果表明，SF-Diff能够提供准确的重建，优于基于核插值的最先进的基线。摘要：Reconstructing the sound field in a room is an important task for several applications, such as sound control and augmented (AR) or virtual reality (VR). In this paper, we propose a data-driven generative model for reconstructing the magnitude of acoustic fields in rooms with a focus on the modal frequency range. We introduce, for the first time, the use of a conditional Denoising Diffusion Probabilistic Model (DDPM) trained in order to reconstruct the sound field (SF-Diff) over an extended domain. The architecture is devised in order to be conditioned on a set of limited available measurements at different frequencies and generate the sound field in target, unknown, locations. The results show that SF-Diff is able to provide accurate reconstructions, outperforming a state-of-the-art baseline based on kernel interpolation.【7】 Towards Automatic Data Augmentation for Disordered Speech Recognition标题：面向无序语音识别的数据自动增强链接：https://arxiv.org/abs/2312.08641作者：Zengrui Jin,Xurong Xie,Tianzi Wang,Mengzhe Geng,Jiajun Deng,Guinan Li,Shujie Hu,Xunying Liu备注：To appear at IEEE ICASSP 2024摘要：迄今为止，由于数据稀缺，无序语音的自动识别仍然是一项极具挑战性的任务。本文提出了一种基于强化学习（RL）的动态数据增强方法，用于在此类数据上训练最先进的PyChain TDNN和端到端Conformer ASR系统。标准SpecAugment方法中的手工时间和频谱掩码操作依赖于任务和系统，以及额外引入的这些时间频率掩码的最小和最大截止值，现在可以使用基于RNN的策略控制器自动学习，并与ASR系统训练紧密集成。UASpeech语料库上的实验表明，所提出的基于RL的数据增强方法始终产生优于或可比的性能，使用专家或手工制作的SpecAugment政策。我们的RL自动增强PyChain TDNN系统在16个构音障碍扬声器的UASpeech测试集上产生了28.79%的总体WER。摘要：Automatic recognition of disordered speech remains a highly challenging task to date due to data scarcity. This paper presents a reinforcement learning (RL) based on-the-fly data augmentation approach for training state-of-the-art PyChain TDNN and end-to-end Conformer ASR systems on such data. The handcrafted temporal and spectral mask operations in the standard SpecAugment method that are task and system dependent, together with additionally introduced minimum and maximum cut-offs of these time-frequency masks, are now automatically learned using an RNN-based policy controller and tightly integrated with ASR system training. Experiments on the UASpeech corpus suggest the proposed RL-based data augmentation approach consistently produced performance superior or comparable that obtained using expert or handcrafted SpecAugment policies. Our RL auto-augmented PyChain TDNN system produced an overall WER of 28.79% on the UASpeech test set of 16 dysarthric speakers.【8】 Scalable Ensemble-based Detection Method against Adversarial Attacks for speaker verification标题： 一种抗对抗性攻击的可扩展Ensemble说话人确认检测方法链接：https://arxiv.org/abs/2312.08622作者：Haibin Wu,Heng-Cheng Kuo,Yu Tsao,Hung-yi Lee备注：Submitted to 2024 ICASSP摘要：自动说话人确认（ASV）是一种非常容易受到对抗性攻击的方法。净化模块通常被用作预处理以减轻对抗性噪声。然而，它们通常在不同的实验环境中实施，使得直接比较具有挑战性。本文在一个统一的框架内对主流净化技术进行了综合比较。我们发现这些方法通常面临用户体验和安全性之间的权衡，因为它们很难同时保持真实的样本性能并减少对抗性干扰。为了应对这一挑战，一些努力已经扩展了净化模块以包括检测能力，旨在减轻权衡。然而，先进的净化模块总是会进入阶段，以超越以往的检测方法。因此，我们进一步提出了一种易于遵循的集成方法，该方法集成了先进的净化模块进行检测，在对抗对抗性噪声方面实现了最先进的（SOTA）性能。我们的合奏方法有很大的潜力，由于其与未来先进的净化技术的兼容性。摘要：Automatic speaker verification (ASV) is highly susceptible to adversarial attacks. Purification modules are usually adopted as a pre-processing to mitigate adversarial noise. However, they are commonly implemented across diverse experimental settings, rendering direct comparisons challenging. This paper comprehensively compares mainstream purification techniques in a unified framework. We find these methods often face a trade-off between user experience and security, as they struggle to simultaneously maintain genuine sample performance and reduce adversarial perturbations. To address this challenge, some efforts have extended purification modules to encompass detection capabilities, aiming to alleviate the trade-off. However, advanced purification modules will always come into the stage to surpass previous detection method. As a result, we further propose an easy-to-follow ensemble approach that integrates advanced purification modules for detection, achieving state-of-the-art (SOTA) performance in countering adversarial noise. Our ensemble method has great potential due to its compatibility with future advanced purification techniques.【9】 A computationally efficient semi-blind source separation based approach for nonlinear echo cancellation based on an element-wise iterative source steering标题：一种计算高效的基于半盲源分离的基于单元迭代信源操纵的非线性回波抵消方法链接：https://arxiv.org/abs/2312.08610作者：Kunxing Lu,Xianrui Wang,Tetsuya Ueda,Shoji Makino,Jingdong Chen摘要：与传统的自适应算法相比，基于半盲源分离的回声消除算法（SBSS-AEC）在双端通话时具有良好的性能，因此受到了广泛的关注，但它存在系统延迟和非线性失真的问题。为了克服这些缺点，近年来发展起来的卷积传递函数（CTF）逼近和非线性展开思想被用于基于迭代投影（IP）的半盲源分离（SBSS）算法中。然而，由于CTF近似和非线性扩展的引入，该算法变得非常昂贵的计算，这使得它很难在嵌入式系统中实现。因此，我们试图在本文中，以改善这种基于IP的算法，从而开发一个元素的迭代源转向（EISS）算法。与基于IP的SBSS算法相比，该算法的计算效率更高，特别是在非线性展开阶数较高和CTF滤波器长度较长的情况下。同时，其AEC性能与基于IP的SBSS相当。摘要：While the semi-blind source separation-based acoustic echo cancellation (SBSS-AEC) has received much research attention due to its promising performance during double-talk compared to the traditional adaptive algorithms, it suffers from system latency and nonlinear distortions. To circumvent these drawbacks, the recently developed ideas on convolutive transfer function (CTF) approximation and nonlinear expansion have been used in the iterative projection (IP)-based semi-blind source separation (SBSS) algorithm. However, because of the introduction of CTF approximation and nonlinear expansion, this algorithm becomes computationally very expensive, which makes it difficult to implement in embedded systems. Thus, we attempt in this paper to improve this IP-based algorithm, thereby developing an element-wise iterative source steering (EISS) algorithm. In comparison with the IP-based SBSS algorithm, the proposed algorithm is computationally much more efficient, especially when the nonlinear expansion order is high and the length of the CTF filter is long. Meanwhile, its AEC performance is as good as that of IP-based SBSS.【10】 NeXt-TDNN: Modernizing Multi-Scale Temporal Convolution Backbone for Speaker Verification标题： NeXt-TDNN：用于说话人确认的多尺度时间卷积骨干的现代化链接：https://arxiv.org/abs/2312.08603作者：Hyun-Jun Heo,Ui-Hyeop Shin,Ran Lee,YoungJu Cheon,Hyung-Min Park备注：Accepted by ICASSP 2024摘要：在说话人确认中，ECAPA-TDNN通过使用一维（1D）Res 2Net块和挤压和激励（SE）模块以及多层特征聚合（MFA），表现出显着的改进。同时，在视觉任务中，ConvNet结构通过引用Transformer进行了现代化，从而提高了性能。在本文中，我们提出了一种改进的块设计TDNN在说话人确认。受最近ConvNet结构的启发，我们将ECAPA-TDNN中的SE-Res 2Net块替换为一个新的1D两步多尺度ConvNeXt块，我们称之为\textit{TS-ConvNeXt}。TS-ConvNeXt块使用两个独立的子模块构建：时间多尺度卷积（MSC）和逐帧前馈网络（FFN）。这种两步设计允许灵活地捕获帧间和帧内上下文。此外，我们为FFN模块引入了全局响应归一化（GRN），以实现更具选择性的特征传播，类似于ECAPA-TDNN中的SE模块。实验结果表明，NeXt-TDNN，与现代化的骨干块，显着提高了说话人验证任务的性能，同时减少参数大小和推理时间。我们已经发布了我们的代码，供未来研究使用。摘要：In speaker verification, ECAPA-TDNN has shown remarkable improvement by utilizing one-dimensional(1D) Res2Net block and squeeze-and-excitation(SE) module, along with multi-layer feature aggregation (MFA). Meanwhile, in vision tasks, ConvNet structures have been modernized by referring to Transformer, resulting in improved performance. In this paper, we present an improved block design for TDNN in speaker verification. Inspired by recent ConvNet structures, we replace the SE-Res2Net block in ECAPA-TDNN with a novel 1D two-step multi-scale ConvNeXt block, which we call \textit{TS-ConvNeXt}. The TS-ConvNeXt block is constructed using two separated sub-modules: a temporal multi-scale convolution (MSC) and a frame-wise feed-forward network (FFN). This two-step design allows for flexible capturing of inter-frame and intra-frame contexts. Additionally, we introduce global response normalization (GRN) for the FFN modules to enable more selective feature propagation, similar to the SE module in ECAPA-TDNN. Experimental results demonstrate that NeXt-TDNN, with a modernized backbone block, significantly improved performance in speaker verification tasks while reducing parameter size and inference time. We have released our code for future studies.【11】 USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models标题：USM-Lite：基于通用语音模型的量化和稀疏性精调链接：https://arxiv.org/abs/2312.08553作者：Shaojin Ding,Qiu David,David Rim,Yanzhang He,Oleg Rybakov,Bo Li,Rohit Prabhavalkar,Weiran Wang,Tara N. Sainath,Shivani Agrawal,Zhonglin Han,Jian Li,Amir Yazdanbakhsh备注：Accepted by ICASSP 2024. Preprint摘要：随着最近大规模通用语音模型（USM）的发展，端到端自动语音识别（ASR）模型已经看到了革命性的质量提升。然而，部署这些大规模的USM是非常昂贵的，由于巨大的内存使用和计算成本。因此，模型压缩是一个重要的研究课题，以适应基于USM的ASR在预算下，在现实世界的场景。在这项研究中，我们提出了一个USM微调方法的ASR，低比特量化和N：M结构稀疏意识的范式模型的权重，从参数精度和矩阵拓扑的角度降低模型的复杂性。我们在大规模语音搜索数据集上使用20亿参数USM进行了广泛的实验，以评估我们提出的方法。一系列消融研究验证了高达int4量化和2：4稀疏度的有效性。然而，在包括int2量化和1：4稀疏度的极端设置下，单一压缩技术无法很好地恢复性能。相比之下，我们提出的方法可以压缩模型的大小为9.4%，成本只有7.3%的相对字错误率（WER）回归。我们还对结果进行了深入分析，并就局限性和潜在解决方案进行了讨论，这对未来的研究很有价值。摘要：End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N:M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a large-scale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.【12】 Metrological support of acoustic measuring installations mid-frequency devices标题：声学测量装置中频装置的计量支持链接：https://arxiv.org/abs/2312.08496作者：A. N. Grekov,N. A. Grekov,E. N. Sychev备注：None摘要：本文讨论了测量液体中声速、声散射、声衰减和声吸收的方法，并在此基础上绘制了现代装置的结构图。在实际条件下，声学测量方案并不理想，并且根据具体结构可能给出不同的结果。本文介绍了现代声学装置测量通道的技术和工艺特点。文中对GOST在声学测量中的应用提出了建议，并指出新研制的测量仪器在现场条件下的声特性达到了国家一级标准。摘要：The article discusses methods for measuring the speed of sound, scattering, attenuation and absorption of sound in liquids, on the basis of which the structural diagrams of modern devices have been developed. In real conditions, acoustic measurement schemes are not ideal and depending on specific structure may give different results. The technical and metrological characteristics of the measuring channels of modern acoustic devices are presented. Recommendations are given for the use of GOST in acoustic measurements and it is stated that the metrological characteristics of newly created measuring instruments operating in in situ conditions are at the level of State primary standards.【13】 WikiMuTe: A web-sourced dataset of semantic descriptions for music audio标题：WikiMuTe：一个基于Web的音乐音频语义描述数据集链接：https://arxiv.org/abs/2312.09207作者：Benno Weck,Holger Kirchhoff,Peter Grosche,Xavier Serra备注：Submitted to 30th International Conference on MultiMedia Modeling (MMM2024). This preprint has not undergone peer review or any post-submission improvements or corrections摘要：用于匹配自由文本与音乐的多模态深度学习技术在音乐信息检索（MIR）领域显示出了良好的效果。先前的工作通常基于大型专有数据，而公开的数据集很少且规模较小。在这项研究中，我们提出了WikiMuTe，一个新的和开放的数据集，包含丰富的音乐语义描述。这些数据来自维基百科丰富的音乐作品条目目录。使用专用的文本挖掘管道，我们提取了长形式和短形式的描述，涵盖了与音乐内容相关的广泛主题，如流派，风格，情绪，乐器和节奏。为了展示这些数据的使用，我们训练了一个模型，该模型联合学习文本和音频表示并执行跨模态检索。该模型进行评估的两个任务：基于标签的音乐检索和音乐自动标注。结果表明，虽然我们的方法在多个任务上具有最先进的性能，但仍然观察到性能差异取决于用于训练的数据。摘要：Multi-modal deep learning techniques for matching free-form text with music have shown promising results in the field of Music Information Retrieval (MIR). Prior work is often based on large proprietary data while publicly available datasets are few and small in size. In this study, we present WikiMuTe, a new and open dataset containing rich semantic descriptions of music. The data is sourced from Wikipedia's rich catalogue of articles covering musical works. Using a dedicated text-mining pipeline, we extract both long and short-form descriptions covering a wide range of topics related to music content such as genre, style, mood, instrumentation, and tempo. To show the use of this data, we train a model that jointly learns text and audio representations and performs cross-modal retrieval. The model is evaluated on two tasks: tag-based music retrieval and music auto-tagging. The results show that while our approach has state-of-the-art performance on multiple tasks, but still observe a difference in performance depending on the data used for training.【14】 F1-EV Score: Measuring the Likelihood of Estimating a Good Decision Threshold for Semi-Supervised Anomaly Detection标题：F1-EV评分：衡量估计半监督异常检测良好决策阈值的可能性链接：https://arxiv.org/abs/2312.09143作者：Kevin Wilkinghoff,Keisuke Imoto备注：Accepted for presentation at IEEE ICASSP 2024摘要：异常声音检测（ASD）系统通常使用阈值无关的性能指标，如AUC-ROC进行比较。然而，对于实际应用，需要一个判定阈值来判定给定的测试样本是正常还是异常。在只有正常训练样本可用的半监督设置中，估计这样的阈值是非常重要的。在这项工作中，F1-EV提出了一种新的阈值独立的ASD系统，也包括估计一个良好的决策阈值的可能性的性能指标，并激励使用特定的玩具的例子。在实验评估中，对提交给DCASE Challenge 2023 ASD任务的所有系统进行了多项性能指标评估。结果表明，F1-EV与AUC-ROC强相关，同时与使用估计和最佳决策阈值获得的F1评分的相关性显著强于AUC-ROC。摘要：Anomalous sound detection (ASD) systems are usually compared by using threshold-independent performance measures such as AUC-ROC. However, for practical applications a decision threshold is needed to decide whether a given test sample is normal or anomalous. Estimating such a threshold is highly non-trivial in a semi-supervised setting where only normal training samples are available. In this work, F1-EV a novel threshold-independent performance measure for ASD systems that also includes the likelihood of estimating a good decision threshold is proposed and motivated using specific toy examples. In experimental evaluations, multiple performance measures are evaluated for all systems submitted to the ASD task of the DCASE Challenge 2023. It is shown that F1-EV is strongly correlated with AUC-ROC while having a significantly stronger correlation with the F1-score obtained with estimated and optimal decision thresholds than AUC-ROC.【15】 STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models标题：STAR：用于轻量级语音自监督学习模型的语音时间关系提取链接：https://arxiv.org/abs/2312.09040作者：Kangwook Jang,Sungnyun Kim,Hoirin Kim备注：Accepted at ICASSP 2024摘要：尽管基于transformer的语音自监督学习（SSL）模型具有很好的性能，但其较大的参数大小和计算成本使其不利于使用。在这项研究中，我们提出了压缩语音SSL模型提取语音时间关系（STaR）。与以往直接匹配每个语音帧的表示不同，STaR蒸馏传输语音帧之间的时间关系，这更适合于能力有限的轻量级学生。我们探讨了三个STaR蒸馏目标，并选择最佳组合作为最终的STaR损失。我们从HuBERT BASE中提取的模型在SUPERB基准测试中获得了79.8的总得分，在多达2700万个参数的模型中表现最好。我们表明，我们的方法是适用于不同的语音SSL模型，并保持强大的性能与进一步减少参数。摘要：Albeit great performance of Transformer-based speech selfsupervised learning (SSL) models, their large parameter size and computational cost make them unfavorable to utilize. In this study, we propose to compress the speech SSL models by distilling speech temporal relation (STaR). Unlike previous works that directly match the representation for each speech frame, STaR distillation transfers temporal relation between speech frames, which is more suitable for lightweight student with limited capacity. We explore three STaR distillation objectives and select the best combination as the final STaR loss. Our model distilled from HuBERT BASE achieves an overall score of 79.8 on SUPERB benchmark, the best performance among models with up to 27 million parameters. We show that our method is applicable across different speech SSL models and maintains robust performance with further reduced parameters.【16】 Multi-CMGAN+/+: Leveraging Multi-Objective Speech Quality Metric Prediction for Speech Enhancement标题：多CMGAN+/+：利用多目标语音质量度量预测进行语音增强链接：https://arxiv.org/abs/2312.08979作者：George Close,William Ravenscroft,Thomas Hain,Stefan Goetze备注：Accepted @ ICASSP 2024摘要：基于神经网络的语音增强方法已被证明是特别强大的，能够利用数据驱动的方法，从而导致与其他方法相比显着的性能增益。这种方法依赖于人工创建的标记训练数据，使得可以使用将模型的输出与干净的参考语音进行比较的侵入性损失函数来训练神经模型。这种系统在增强真实世界音频时的性能通常相对于它们在模拟测试数据上的性能而受损。在这项工作中，引入了一种非侵入性的多度量预测方法，其中使用对抗训练的度量预测神经网络的推理在人工标记的数据上训练模型。所提出的方法显示了改进的性能与最先进的系统在最近的CHiME-7挑战\ac{UDASE}任务评估集。摘要：Neural network based approaches to speech enhancement have shown to be particularly powerful, being able to leverage a data-driven approach to result in a significant performance gain versus other approaches. Such approaches are reliant on artificially created labelled training data such that the neural model can be trained using intrusive loss functions which compare the output of the model with clean reference speech. Performance of such systems when enhancing real-world audio often suffers relative to their performance on simulated test data. In this work, a non-intrusive multi-metric prediction approach is introduced, wherein a model trained on artificial labelled data using inference of an adversarially trained metric prediction neural network. The proposed approach shows improved performance versus state-of-the-art systems on the recent CHiME-7 challenge \ac{UDASE} task evaluation sets.【17】 N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding标题：N-Gram无监督组合和特征注入更好地理解符号音乐链接：https://arxiv.org/abs/2312.08931作者：Jinhao Tian,Zuchao Li,Jiajia Li,Ping Wang备注：8 pages, 2 figures, aaai2024摘要：将深度学习技术应用于符号音乐理解的第一步是将音乐作品（主要是以MIDI格式）转换为预定义的标记序列，如音符音高，音符速度和和弦。随后，这些序列被送入神经序列模型以完成特定任务。音乐序列在相邻元素之间表现出很强的相关性，使其成为自然语言处理（NLP）中N-gram技术的主要候选者。以古典钢琴音乐为例：特定的旋律可能会在整首曲子中反复出现，每次都有微妙的变化。在本文中，我们提出了一种新的方法，NG-Midiformer，理解符号音乐序列，利用N-gram的方法。我们的方法涉及到第一次处理音乐作品成词样序列与我们提出的无监督的复合，其次是使用我们的N-gram Transformer编码器，它可以有效地将N-gram信息，以提高主要编码器的一部分，更好地理解音乐序列。在大规模音乐数据集上的预训练过程使模型能够彻底学习音乐序列中包含的N-gram信息，并随后在微调阶段应用这些信息进行推断。在不同数据集上的实验证明了该方法的有效性，并在一系列音乐理解下游任务上取得了最先进的性能。代码和模型重量将在https://github.com/WouuYoauin/NG-Midiformer上发布。摘要：The first step to apply deep learning techniques for symbolic music understanding is to transform musical pieces (mainly in MIDI format) into sequences of predefined tokens like note pitch, note velocity, and chords. Subsequently, the sequences are fed into a neural sequence model to accomplish specific tasks. Music sequences exhibit strong correlations between adjacent elements, making them prime candidates for N-gram techniques from Natural Language Processing (NLP). Consider classical piano music: specific melodies might recur throughout a piece, with subtle variations each time. In this paper, we propose a novel method, NG-Midiformer, for understanding symbolic music sequences that leverages the N-gram approach. Our method involves first processing music pieces into word-like sequences with our proposed unsupervised compoundation, followed by using our N-gram Transformer encoder, which can effectively incorporate N-gram information to enhance the primary encoder part for better understanding of music sequences. The pre-training process on large-scale music datasets enables the model to thoroughly learn the N-gram information contained within music sequences, and subsequently apply this information for making inferences during the fine-tuning stage. Experiment on various datasets demonstrate the effectiveness of our method and achieved state-of-the-art performance on a series of music understanding downstream tasks. The code and model weights will be released at https://github.com/WouuYoauin/NG-Midiformer.【18】 Hourglass-AVSR: Down-Up Sampling-based Computational Efficiency Model for Audio-Visual Speech Recognition标题：沙漏-AVSR：基于上下采样的视听语音识别计算效率模型链接：https://arxiv.org/abs/2312.08850作者：Fan Yu,Haoxu Wang,Ziyang Ma,Shiliang Zhang备注：Accepted by ICASSP 2024摘要：近年来，视听语音识别（AVSR），它更好地利用视频模态作为附加信息来扩展自动语音识别（ASR），在复杂的声学环境中显示出良好的效果。然而，由于视觉模块的复杂计算和视听模态的无效融合，仍有很大的改进空间。为了消除这些缺点，我们提出了一个向下向上采样的AVSR模型（沙漏AVSR）享受高效率和性能，其时间长度在中间处理过程中缩放，类似于沙漏。首先，我们提出了一种上下文和残差感知的视频上采样方法，以提高识别性能，它利用视觉表示的上下文信息，并捕获相邻视频帧之间的残差信息。其次，我们在上采样期间引入了一种视觉-音频对齐方法，该方法通过显式地引入边界约束损失。此外，我们提出了一个跨层的注意力融合，以捕捉每个视觉编码器层内的模态依赖。在MISP-AVSR数据集上进行的实验表明，我们提出的Hourglass-AVSR模型在远场和中场测试集上的相对级联最小排列字符错误率（cpCER）分别降低了12.9%和20.8%，优于ASR模型。此外，与其他最先进的AVSR模型相比，我们的模型在视觉模块的cpCER中表现出最高的改进。此外，在我们的向下向上采样方法的好处，沙漏AVSR模型减少了54.2%的整体计算成本与轻微的性能下降。摘要：Recently audio-visual speech recognition (AVSR), which better leverages video modality as additional information to extend automatic speech recognition (ASR), has shown promising results in complex acoustic environments. However, there is still substantial space to improve as complex computation of visual modules and ineffective fusion of audio-visual modalities. To eliminate these drawbacks, we propose a down-up sampling-based AVSR model (Hourglass-AVSR) to enjoy high efficiency and performance, whose time length is scaled during the intermediate processing, resembling an hourglass. Firstly, we propose a context and residual aware video upsampling approach to improve the recognition performance, which utilizes contextual information from visual representations and captures residual information between adjacent video frames. Secondly, we introduce a visual-audio alignment approach during the upsampling by explicitly incorporating boundary constraint loss. Besides, we propose a cross-layer attention fusion to capture the modality dependencies within each visual encoder layer. Experiments conducted on the MISP-AVSR dataset reveal that our proposed Hourglass-AVSR model outperforms ASR model by 12.9% and 20.8% relative concatenated minimum permutation character error rate (cpCER) reduction on far-field and middle-field test sets, respectively. Moreover, compared to other state-of-the-art AVSR models, our model exhibits the highest improvement in cpCER for the visual module. Furthermore, on the benefit of our down-up sampling approach, Hourglass-AVSR model reduces 54.2% overall computation costs with minor performance degradation.【19】 TIA: A Teaching Intonation Assessment Dataset in Real Teaching Situations标题： TIA：一个真实教学情境下的教学语调评估数据集链接：https://arxiv.org/abs/2312.08732作者：Shuhua Liu,Chunyu Zhang,Binshuai Li,Niantong Qin,Huanting Cheng,Huayu Zhang备注：4 pages, 3 figures, 4 tables, accepted by 2024 International Conference on Acoustics, Speech, and Signal Processing (ICASSP2024)摘要：语调是影响教学语言艺术的重要因素之一，利用人工智能技术对教师语调进行评价是一个亟待解决的问题。然而，语调评估数据集的缺乏阻碍了该领域的发展。为此，本文首次在真实教学情境下构建了教学语调评估数据集。该数据集涵盖9个学科，396名教师，共计11，444个话语样本，长度为15秒。为了检验数据集的有效性，本文提出了一个基于语音低层和深层特征的教学语调评估模型（TIAM）。实验结果表明，基于本文构建的数据集的TIAM评价结果与人工评价结果基本一致，且评价结果优于基线模型，证明了评价模型的有效性。摘要：Intonation is one of the important factors affecting the teaching language arts, so it is an urgent problem to be addressed by evaluating the teachers' intonation through artificial intelligence technology. However, the lack of an intonation assessment dataset has hindered the development of the field. To this end, this paper constructs a Teaching Intonation Assessment (TIA) dataset for the first time in real teaching situations. This dataset covers 9 disciplines, 396 teachers, total of 11,444 utterance samples with a length of 15 seconds. In order to test the validity of the dataset, this paper proposes a teaching intonation assessment model (TIAM) based on low-level and deep-level features of speech. The experimental results show that TIAM based on the dataset constructed in this paper is basically consistent with the results of manual evaluation, and the results are better than the baseline models, which proves the effectiveness of the evaluation model.【20】 StemGen: A music generation model that listens标题：StemGen：倾听的音乐生成模式链接：https://arxiv.org/abs/2312.08723作者：Julian D. Parker,Janne Spijkervet,Katerina Kosta,Furkan Yesiler,Boris Kuznetsov,Ju-Chiang Wang,Matt Avent,Jitong Chen,Duc Le备注：Accepted for publication at ICASSP 2024摘要：使用深度学习技术的端到端音乐音频生成最近出现了爆炸式增长。然而，大多数模型专注于生成完全混合的音乐，以响应抽象的条件信息。在这项工作中，我们提出了一个替代的范式，生产音乐生成模型，可以听和响应音乐的背景。我们描述了如何使用非自回归，基于变换器的模型架构构建这样的模型，并提出了一些新的架构和采样改进。我们在开源和专有数据集上训练所描述的架构。我们使用标准的质量指标和一种新的方法，基于音乐信息检索描述符的模型进行评估。由此产生的模型达到了最先进的文本条件模型的音频质量，以及表现出很强的音乐连贯性与其上下文。摘要：End-to-end generation of musical audio using deep learning techniques has seen an explosion of activity recently. However, most models concentrate on generating fully mixed music in response to abstract conditioning information. In this work, we present an alternative paradigm for producing music generation models that can listen and respond to musical context. We describe how such a model can be constructed using a non-autoregressive, transformer-based model architecture and present a number of novel architectural and sampling improvements. We train the described architecture on both an open-source and a proprietary dataset. We evaluate the produced models using standard quality metrics and a new approach based on music information retrieval descriptors. The resulting model reaches the audio quality of state-of-the-art text-conditioned models, as well as exhibiting strong musical coherence with its context.【21】 SEF-VC: Speaker Embedding Free Zero-Shot Voice Conversion with Cross Attention标题：SEF-VC：嵌入交叉注意的自由零发声语音转换的扬声器链接：https://arxiv.org/abs/2312.08676作者：Junjie Li,Yiwei Guo,Xie Chen,Kai Yu备注：5 pages,2 figures, submitted to ICASSP 2024摘要：Zero-shot语音转换（VC）的目的是在保持语言内容不变的情况下，将源说话人的音色转换为目标说话人的音色。虽然可以通过提供目标说话人的说话人嵌入来控制所生成的语音的声音，但是说话人相似性仍然落后于地面实况记录。在本文中，我们提出了SEF-VC，一个说话人嵌入自由语音转换模型，它的目的是通过一个强大的位置不可知的交叉注意机制，学习和纳入说话人的音色从参考语音，然后重建波形从HuBERT语义令牌在非自回归的方式。SEF-VC的简洁设计提高了训练稳定性和语音转换性能。客观和主观评价表明，即使对于非常短的参考语音，SEF-VC也比强zero-shot VC基线生成与目标参考具有更好相似性的高质量语音。摘要：Zero-shot voice conversion (VC) aims to transfer the source speaker timbre to arbitrary unseen target speaker timbre, while keeping the linguistic content unchanged. Although the voice of generated speech can be controlled by providing the speaker embedding of the target speaker, the speaker similarity still lags behind the ground truth recordings. In this paper, we propose SEF-VC, a speaker embedding free voice conversion model, which is designed to learn and incorporate speaker timbre from reference speech via a powerful position-agnostic cross-attention mechanism, and then reconstruct waveform from HuBERT semantic tokens in a non-autoregressive manner. The concise design of SEF-VC enhances its training stability and voice conversion performance. Objective and subjective evaluations demonstrate the superiority of SEF-VC to generate high-quality speech with better similarity to target reference than strong zero-shot VC baselines, even for very short reference speeches.【22】 Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation标题：视听之外的片段：视听语义切分中部分缺失通道的处理链接：https://arxiv.org/abs/2312.08673作者：Renjie Wu,Hu Wang,Feras Dayoub,Hsiang-Ting Chen备注：Accepted by AAAI-24摘要：增强现实（AR）设备作为突出的移动交互平台出现，面临着用户安全方面的挑战，特别是关于迎面而来的车辆。虽然一些解决方案利用板载相机阵列，但这些相机通常具有有限的前视角或下视角的视场（FoV）。针对这一点，我们提出了一个新的视图外的语义分割任务和分段超越视图（SBV），一种新的视听语义分割方法。SBV补充视觉模态，错过了FoV以外的信息，与听觉信息使用师生蒸馏模型（Omni2Ego）。该模型包括一个视觉教师利用全景信息，一个听觉教师与8通道音频，和一个视听的学生，认为有限的FoV和双耳音频作为输入，并产生语义分割FoV以外的对象。SBV在比较评估中优于现有模型，并在不同的FoV范围和单声道音频设置中显示出一致的性能。摘要：Augmented Reality (AR) devices, emerging as prominent mobile interaction platforms, face challenges in user safety, particularly concerning oncoming vehicles. While some solutions leverage onboard camera arrays, these cameras often have limited field-of-view (FoV) with front or downward perspectives. Addressing this, we propose a new out-of-view semantic segmentation task and Segment Beyond View (SBV), a novel audio-visual semantic segmentation method. SBV supplements the visual modality, which miss the information beyond FoV, with the auditory information using a teacher-student distillation model (Omni2Ego). The model consists of a vision teacher utilising panoramic information, an auditory teacher with 8-channel audio, and an audio-visual student that takes views with limited FoV and binaural audio as input and produce semantic segmentation for objects outside FoV. SBV outperforms existing models in comparative evaluations and shows a consistent performance across varying FoV ranges and in monaural audio settings.【23】 Low-rank constrained multichannel signal denoising considering channel-dependent sensitivity inspired by self-supervised learning for optical fiber sensing标题：基于自监督学习的低阶约束多通道光纤传感信号去噪链接：https://arxiv.org/abs/2312.08660作者：Noriyuki Tonami,Wataru Kohno,Sakiko Mishima,Yumi Arai,Reishi Kondo,Tomoyuki Hino备注：Accepted for ICASSP2024摘要：光纤传感是使用光纤检测音频、振动和温度的技术;特别是音频/振动感知传感被称为分布式声学传感（DAS）。在DAS中，观测数据由多通道数据组成，由于光学噪声或安装方法的原因，观测数据遭受严重的噪声水平。在用于对DAS数据进行去噪的常规方法中，已经研究了基于信号处理或深度神经网络（DNN）的模型。基于信号处理的方法具有可解释性，即，非黑盒基于DNN的方法擅长灵活设计网络结构和目标函数，即先验。然而，在DAS研究中，先验的可解释性和灵活性之间没有平衡。基于DNN的方法通常也需要大量的训练数据。为了解决这些问题，本文提出了一种基于DNN结构的信号处理去噪方法。作为DAS的先验，我们采用空间知识;使用基于DNN的结构的低秩和通道依赖的灵敏度。光纤声传感实验结果表明，该方法优于传统方法，且对空间秩数具有较好的鲁棒性。此外，所提出的方法的优化参数表明与通道灵敏度的关系，可解释性。摘要：Optical fiber sensing is a technology wherein audio, vibrations, and temperature are detected using an optical fiber; especially the audio/vibrations-aware sensing is called distributed acoustic sensing (DAS). In DAS, observed data, which is comprised of multichannel data, has suffered from severe noise levels because of the optical noise or the installation methods. In conventional methods for denoising DAS data, signal-processing- or deep-neural-network (DNN)-based models have been studied. The signal-processing-based methods have the interpretability, i.e., non-black box. The DNN-based methods are good at flexibility designing network architectures and objective functions, that is, priors. However, there is no balance between the interpretability and the flexibility of priors in the DAS studies. The DNN-based methods also require a large amount of training data in general. To address the problems, we propose a DNN-structure signal-processing-based denoising method in this paper. As the priors of DAS, we employ spatial knowledge; low rank and channel-dependent sensitivity using the DNN-based structure. The result of fiber-acoustic sensing shows that the proposed method outperforms the conventional methods and the robustness to the number of the spatial ranks. Moreover, the optimized parameters of the proposed method indicate the relationship with the channel sensitivity; the interpretability.【24】 PhasePerturbation: Speech Data Augmentation via Phase Perturbation for Automatic Speech Recognition标题： PhasePerturbation：用于自动语音识别的通过相位扰动的语音数据增强链接：https://arxiv.org/abs/2312.08571作者：Chengxi Lei,Satwinder Singh,Feng Hou,Xiaoyun Jia,Ruili Wang摘要：目前的语音数据增强方法大多是对语音的原始波形或幅度谱进行增强。在本文中，我们提出了一种新的语音数据增强方法称为相位扰动，动态操作的语音相位谱。PhasePerturbation利用三个动态相位谱操作，即，随机化操作、频率掩蔽操作和时间掩蔽操作，以增强语音数据的多样性。我们对wav2vec2.0预训练的ASR模型进行实验，通过PhasePerturbation增强TIMIT语料库对其进行微调。实验结果表明，与基线模型相比，在没有任何增强操作微调的字错误率（WER）的相对减少10.9%。此外，所提出的方法实现了额外的改进（12.9%和15.9%），通过补充声道长度扰动（VTLP）和SpecAug，这两个幅度谱为基础的增强方法在WER。结果突出了相位扰动的能力，以改善目前的幅度谱为基础的增强方法。摘要：Most of the current speech data augmentation methods operate on either the raw waveform or the amplitude spectrum of speech. In this paper, we propose a novel speech data augmentation method called PhasePerturbation that operates dynamically on the phase spectrum of speech. Instead of statically rotating a phase by a constant degree, PhasePerturbation utilizes three dynamic phase spectrum operations, i.e., a randomization operation, a frequency masking operation, and a temporal masking operation, to enhance the diversity of speech data. We conduct experiments on wav2vec2.0 pre-trained ASR models by fine-tuning them with the PhasePerturbation augmented TIMIT corpus. The experimental results demonstrate 10.9\% relative reduction in the word error rate (WER) compared with the baseline model fine-tuned without any augmentation operation. Furthermore, the proposed method achieves additional improvements (12.9\% and 15.9\%) in WER by complementing the Vocal Tract Length Perturbation (VTLP) and the SpecAug, which are both amplitude spectrum-based augmentation methods. The results highlight the capability of PhasePerturbation to improve the current amplitude spectrum-based augmentation methods.【25】 PerMod: Perceptually Grounded Voice Modification with Latent Diffusion Models标题：PerMod：基于潜在扩散模型的感知基础语音修正链接：https://arxiv.org/abs/2312.08494作者：Robin Netzorg,Ajil Jalal,Luna McNulty,Gopala Krishna Anumanchipalli摘要：语音的感知修改是一个难以捉摸的目标。虽然非专家可以用可用的工具在感知上修改图像或句子，但不清楚如何沿着感知轴类似地修改语音。语音转换确实可以将一种语音转换为另一种语音，但这些修改是由黑盒模型处理的，并且修改哪些感知质量以及如何修改它们的细节尚不清楚。为了让更大的感知控制的声音，我们引入PerMod，一个有条件的潜在扩散模型，需要在一个输入的声音和感知质量向量，并产生一个声音与匹配的感知质量。与以前的工作不同，PerMod生成一个新的声音对应于特定的感知修改。从人类和预测标签的RMSE评估感知质量向量，我们证明了PerMod产生的声音与典型的声音所需的感知质量，但表现不佳的非典型的声音。摘要：Perceptual modification of voice is an elusive goal. While non-experts can modify an image or sentence perceptually with available tools, it is not clear how to similarly modify speech along perceptual axes. Voice conversion does make it possible to convert one voice to another, but these modifications are handled by black box models, and the specifics of what perceptual qualities to modify and how to modify them are unclear. Towards allowing greater perceptual control over voice, we introduce PerMod, a conditional latent diffusion model that takes in an input voice and a perceptual qualities vector, and produces a voice with the matching perceptual qualities. Unlike prior work, PerMod generates a new voice corresponding to specific perceptual modifications. Evaluating perceptual quality vectors with RMSE from both human and predicted labels, we demonstrate that PerMod produces voices with the desired perceptual qualities for typical voices, but performs poorly on atypical voices.【26】 FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models标题：FaceTalk：用于神经参数头部模型的音频驱动运动扩散链接：https://arxiv.org/abs/2312.08459作者：Shivangi Aneja,Justus Thies,Angela Dai,Matthias Nießner备注：Paper Video: this https URL Project Page: this https URL摘要：我们介绍FaceTalk，一种新的生成方法，设计用于从输入音频信号合成高保真的3D运动序列的说话人的头。为了捕捉人类头部的表现力，细节的性质，包括头发，耳朵和更精细的眼球运动，我们建议将语音信号与神经参数头部模型的潜在空间耦合，以创建高保真，时间相干的运动序列。为此，我们提出了一个新的潜在扩散模型，在神经参数头部模型的表达空间中操作，以合成音频驱动的逼真头部序列。在没有与音频对应的NPHM表达式的数据集的情况下，我们针对这些对应关系进行优化，以产生适合人们谈话的音频-视频记录的时间优化的NPHM表达式的数据集。据我们所知，这是第一个工作，提出了一个生成的方法，逼真和高质量的运动合成的体积人头，代表了音频驱动的3D动画领域的重大进展。值得注意的是，我们的方法脱颖而出的能力，以产生合理的运动序列，可以产生高保真度的头部动画加上NPHM形状空间。我们的实验结果证实了FaceTalk的有效性，始终实现卓越和视觉自然的运动，包括不同的面部表情和风格，在感知用户研究评估中超过现有方法75%。摘要：We introduce FaceTalk, a novel generative approach designed for synthesizing high-fidelity 3D motion sequences of talking human heads from input audio signal. To capture the expressive, detailed nature of human heads, including hair, ears, and finer-scale eye movements, we propose to couple speech signal with the latent space of neural parametric head models to create high-fidelity, temporally coherent motion sequences. We propose a new latent diffusion model for this task, operating in the expression space of neural parametric head models, to synthesize audio-driven realistic head sequences. In the absence of a dataset with corresponding NPHM expressions to audio, we optimize for these correspondences to produce a dataset of temporally-optimized NPHM expressions fit to audio-video recordings of people talking. To the best of our knowledge, this is the first work to propose a generative approach for realistic and high-quality motion synthesis of volumetric human heads, representing a significant advancement in the field of audio-driven 3D animation. Notably, our approach stands out in its ability to generate plausible motion sequences that can produce high-fidelity head animation coupled with the NPHM shape space. Our experimental results substantiate the effectiveness of FaceTalk, consistently achieving superior and visually natural motion, encompassing diverse facial expressions and styles, outperforming existing methods by 75% in perceptual user study evaluation.【27】 EmbAu: A Novel Technique to Embed Audio Data Using Shuffled Frog Leaping Algorithm标题：Embau：一种基于混洗蛙跳算法的音频数据嵌入新技术链接：https://arxiv.org/abs/2312.08417作者：Sahil Nokhwal,Saurabh Pahune,Ankit Chaudhary摘要：隐写算法的目的是确定宿主图像或封面图像中合适的像素位置，在这些位置可以隐藏敏感信息以进行数据加密。目前正在努力提高整合敏感信息的能力，并保持隐写图像的视觉外观。因此，隐写术是一个具有挑战性的研究领域。在我们目前提出的图像隐写技术中，我们使用了混洗蛙跳算法（SFLA）来确定敏感信息可以放置在封面图像中的像素顺序。为了实现更大的嵌入容量，仔细选择来自封面图像的空间域的像素并用于放置敏感数据。通过图像隐写术的支持，嵌入后的最终图像可以抵抗隐写分析攻击。SFLA算法用于任何彩色（RGB）覆盖图像的最佳像素选择，用于秘密比特嵌入。使用适应度函数，SFLA通过在可接受的时间量内达到最小成本值而受益。嵌入的像素是精心选择的，以最大限度地减少嵌入时宿主图像的失真。此外，已采取的努力，使隐藏图像中的嵌入数据的检测是一个艰巨的挑战。由于当今世界对音频数据加密的巨大需求，我们认为我们提出的方法在现实世界的应用中具有巨大的潜力。在本文中，我们提出并比较我们的策略，现有的隐写方法。摘要：The aim of steganographic algorithms is to identify the appropriate pixel positions in the host or cover image, where bits of sensitive information can be concealed for data encryption. Work is being done to improve the capacity to integrate sensitive information and to maintain the visual appearance of the steganographic image. Consequently, steganography is a challenging research area. In our currently proposed image steganographic technique, we used the Shuffled Frog Leaping Algorithm (SFLA) to determine the order of pixels by which sensitive information can be placed in the cover image. To achieve greater embedding capacity, pixels from the spatial domain of the cover image are carefully chosen and used for placing the sensitive data. Bolstered via image steganography, the final image after embedding is resistant to steganalytic attacks. The SFLA algorithm serves in the optimal pixels selection of any colored (RGB) cover image for secret bit embedding. Using the fitness function, the SFLA benefits by reaching a minimum cost value in an acceptable amount of time. The pixels for embedding are meticulously chosen to minimize the host image's distortion upon embedding. Moreover, an effort has been taken to make the detection of embedded data in the steganographic image a formidable challenge. Due to the enormous need for audio data encryption in the current world, we feel that our suggested method has significant potential in real-world applications. In this paper, we propose and compare our strategy to existing steganographic methods. 
每日语音速递
