今日论文合集：cs.SD语音16篇，eess.AS音频处理14篇。
cs.SD语音
【1】 Back Translation for Speech-to-text Translation Without Transcripts标题：无抄本的语音到文本翻译的回译链接：https://arxiv.org/abs/2305.08709
作者：Qingkai Fang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：ACL 2023 main conference摘要：端到端语音到文本翻译（ST）的成功通常通过利用源转录本来实现，例如，通过使用自动语音识别（ASR）和机器翻译（MT）任务进行预训练，或者通过引入额外的ASR和MT数据。遗憾的是，由于世界上存在着许多不成文的语言，因此有时只能获得抄本。在本文中，我们的目标是利用大量的目标侧单语数据，以提高ST没有成绩单。在MT的显着成功的回翻译的动机，我们开发了一个回翻译算法ST（BT4 ST）合成伪ST数据从单语目标数据。为了缓解短到长的一代和一对多的映射所带来的挑战，我们引入了自我监督的离散单元，并通过级联目标到单元模型和单元到语音模型来实现反向翻译。通过我们的合成ST数据，我们在MuST-C En-De，En-Fr和En-Es数据集上实现了2.3 BLEU的平均提升。更多的实验表明，我们的方法是特别有效的低资源的情况下。摘要：The success of end-to-end speech-to-text translation (ST) is often achieved by utilizing source transcripts, e.g., by pre-training with automatic speech recognition (ASR) and machine translation (MT) tasks, or by introducing additional ASR and MT data. Unfortunately, transcripts are only sometimes available since numerous unwritten languages exist worldwide. In this paper, we aim to utilize large amounts of target-side monolingual data to enhance ST without transcripts. Motivated by the remarkable success of back translation in MT, we develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolingual target data. To ease the challenges posed by short-to-long generation and one-to-many mapping, we introduce self-supervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model. With our synthetic ST data, we achieve an average boost of 2.3 BLEU on MuST-C En-De, En-Fr, and En-Es datasets. More experiments show that our method is especially effective in low-resource scenarios.
【2】 Understanding and Bridging the Modality Gap for Speech Translation标题：理解和弥合言语翻译中的情态鸿沟链接：https://arxiv.org/abs/2305.08706
作者：Qingkai Fang,Yang Feng机构：Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICTCAS), University of Chinese Academy of Sciences, Beijing, China备注：ACL 2023 main conference摘要：如何利用（文本）机器翻译（MT）数据实现更好的端到端语音翻译（ST）？在各种现有技术中，多任务学习是ST和MT之间共享知识的有效方法之一，其中附加的MT数据可以帮助学习源到目标映射。然而，由于语音和文本的差异，ST和MT之间始终存在着差距。在本文中，我们的目标首先是从目标端表示差异中理解这种模态差距，并将模态差距与神经机器翻译中的另一个众所周知的问题联系起来：暴露偏倚我们发现，除了一些困难的情况下，在训练过程中的模态差距是相对较小的，但不断增加的推理过程中，由于级联效应。为了解决这些问题，我们提出了跨模态正则化与预定采样（Cress）方法。具体来说，我们正则化ST和MT的输出预测，其目标侧上下文是通过在地面真值词和自生成词之间以不同的概率进行采样而得出的。此外，我们引入令牌级自适应训练，为目标令牌分配不同的训练权重，以处理具有大模态间隙的困难情况。实验和分析表明，我们的方法有效地弥合了模态差距，并取得了可喜的结果，在所有八个方向的MuST-C数据集。摘要：How to achieve better end-to-end speech translation (ST) by leveraging (text) machine translation (MT) data? Among various existing techniques, multi-task learning is one of the effective ways to share knowledge between ST and MT in which additional MT data can help to learn source-to-target mapping. However, due to the differences between speech and text, there is always a gap between ST and MT. In this paper, we first aim to understand this modality gap from the target-side representation differences, and link the modality gap to another well-known problem in neural machine translation: exposure bias. We find that the modality gap is relatively small during training except for some difficult cases, but keeps increasing during inference due to the cascading effect. To address these problems, we propose the Cross-modal Regularization with Scheduled Sampling (Cress) method. Specifically, we regularize the output predictions of ST and MT, whose target-side contexts are derived by sampling between ground truth words and self-generated words with a varying probability. Furthermore, we introduce token-level adaptive training which assigns different training weights to target tokens to handle difficult cases with large modality gaps. Experiments and analysis show that our approach effectively bridges the modality gap, and achieves promising results in all eight directions of the MuST-C dataset.
【3】 Ripple sparse self-attention for monaural speech enhancement标题：用于单声道语音增强的波纹稀疏自我注意链接：https://arxiv.org/abs/2305.08541
作者：Qiquan Zhang,Hongxu Zhu,Qi Song,Xinyuan Qian,Zhaoheng Ni,Haizhou Li机构：School of Electrical Engineering and Telecommunications, University of New South Wales, Australia, Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Alibaba Group, China, Meta, United States备注：5 pages, ICASSP 2023 published摘要：Transformer的使用代表了语音增强的最新成功。然而，作为其核心组成部分，自我注意遭受二次复杂性，这是计算上禁止长的语音记录。此外，它允许每个时间帧参加所有的时间帧，忽略了语音信号的强局部相关性。这项研究提出了一个简单而有效的稀疏的语音增强，称为纹波的注意力，同时进行细粒度和粗粒度建模的本地和全球的依赖关系，分别。具体而言，我们采用本地频带注意力，使每个帧参加其最近的邻居帧在一个窗口中的细粒度，而采用扩大注意力的窗口外的模型的全局依赖性在一个粗略的粒度。我们评估了我们的涟漪注意力的两个常用的训练目标的语音增强的功效。广泛的实验结果一致证实了卓越的性能的波纹注意设计超过标准的完全自我注意，块式注意，和双路径注意（Sep-Former）的语音质量和清晰度。摘要：The use of Transformer represents a recent success in speech enhancement. However, as its core component, self-attention suffers from quadratic complexity, which is computationally prohibited for long speech recordings. Moreover, it allows each time frame to attend to all time frames, neglecting the strong local correlations of speech signals. This study presents a simple yet effective sparse self-attention for speech enhancement, called ripple attention, which simultaneously performs fine- and coarse-grained modeling for local and global dependencies, respectively. Specifically, we employ local band attention to enable each frame to attend to its closest neighbor frames in a window at fine granularity, while employing dilated attention outside the window to model the global dependencies at a coarse granularity. We evaluate the efficacy of our ripple attention for speech enhancement on two commonly used training objectives. Extensive experimental results consistently confirm the superior performance of the ripple attention design over standard full self-attention, blockwise attention, and dual-path attention (Sep-Former) in terms of speech quality and intelligibility.
【4】 ForkNet: Simultaneous Time and Time-Frequency Domain Modeling for Speech  Enhancement标题：ForkNet：语音增强的同时时频域建模链接：https://arxiv.org/abs/2305.08292
作者：Feng Dang,Qi Hu,Pengyuan Zhang,Yonghong Yan机构：University of Chinese Academy of Sciences摘要：语音增强中的先前研究主要集中于单独建模时间或时间-频率域信息，很少考虑同时建模两个域的潜在益处。由于这些域包含互补信息，因此组合它们可以提高模型的性能。在这封信中，我们提出了一种新的方法，同时在一个单一的模型的时间和时间-频率域的信息模型。我们开始与DPT-FSNet（因果版本）模型作为基线和修改的编码器结构，取代原来的编码器与三个单独的编码器，每个专用于建模时域，实虚，和幅度信息，分别。此外，我们在双路径处理块之前和之后引入了特征融合模块，以更好地利用来自不同域的信息。我们的实验结果表明，所提出的方法相比，现有的国家的最先进的因果模型，同时保持一个相对紧凑的模型大小和较低的计算复杂度，实现了卓越的性能。摘要：Previous research in speech enhancement has mostly focused on modeling time or time-frequency domain information alone, with little consideration given to the potential benefits of simultaneously modeling both domains. Since these domains contain complementary information, combining them may improve the performance of the model. In this letter, we propose a new approach to simultaneously model time and time-frequency domain information in a single model. We begin with the DPT-FSNet (causal version) model as a baseline and modify the encoder structure by replacing the original encoder with three separate encoders, each dedicated to modeling time-domain, real-imaginary, and magnitude information, respectively. Additionally, we introduce a feature fusion module both before and after the dual-path processing blocks to better leverage information from the different domains. The outcomes of our experiments reveal that the proposed approach achieves superior performance compared to existing state-of-the-art causal models, while preserving a relatively compact model size and low computational complexity.
【5】 Self-supervised Neural Factor Analysis for Disentangling Utterance-level  Speech Representations标题：解纠缠语音级语音表征的自监督神经因子分析链接：https://arxiv.org/abs/2305.08099
作者：Weiwei Lin,Chenhang He,Man-Wai Mak,Youzhi Tu机构：The Hong Kong Polytechnic University,  The Hong Kong PolytechnicUniversity备注：accepted by ICML 2023摘要：自监督学习（SSL）语音模型，如wav2vec和HuBERT，在自动语音识别（ASR）方面表现出了最先进的性能，并被证明在低标签资源设置中非常有用。然而，SSL模型的成功还没有转移到话语级任务，如说话人，情感和语言识别，这仍然需要监督微调的SSL模型，以获得良好的性能。我们认为，这个问题是由于缺乏解开的陈述和话语水平的学习目标，这些任务。受到HuBERT如何使用聚类来发现隐藏的声学单元的启发，我们制定了一个因子分析（FA）模型，该模型使用发现的隐藏的声学单元来对齐SSL特征。使用对齐特征上的概率推断，从语音内容中解开潜在的话语级表示。此外，从FA模型导出的变分下限提供了话语级目标，允许误差梯度被反向传播到Transformer层以学习高度区分的声学单元。当与HuBERT的掩蔽预测训练结合使用时，我们的模型在SUPERB基准测试中的所有话语级非语义任务上的表现优于当前最佳模型WavLM，只有20%的标记数据。摘要：Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition (ASR) and proved to be extremely useful in low label-resource settings. However, the success of SSL models has yet to transfer to utterance-level tasks such as speaker, emotion, and language recognition, which still require supervised fine-tuning of the SSL models to obtain good performance. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective for these tasks. Inspired by how HuBERT uses clustering to discover hidden acoustic units, we formulate a factor analysis (FA) model that uses the discovered hidden acoustic units to align the SSL features. The underlying utterance-level representations are disentangled from the content of speech using probabilistic inference on the aligned features. Furthermore, the variational lower bound derived from the FA model provides an utterance-level objective, allowing error gradients to be backpropagated to the Transformer layers to learn highly discriminative acoustic units. When used in conjunction with HuBERT's masked prediction training, our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.
【6】 Improving End-to-End SLU performance with Prosodic Attention and  Distillation标题：通过韵律注意和蒸馏提高端到端SLU性能链接：https://arxiv.org/abs/2305.08067
作者：Shangeth Rajaa机构：Skit.ai, Bengaluru, India备注：Submitted to InterSpeech 2023摘要：大多数端到端SLU方法依赖于用于意图预测的预训练ASR或语言模型特征。然而，语音中的其他重要信息，如韵律，往往被忽视。最近的研究表明，通过将韵律信息分类的对话行为的改进的结果。这些方法的改进幅度很小，因为神经模型忽略了韵律特征。在这项工作中，我们提出了韵律注意，它使用不同的韵律特征来生成跨时间帧的话语的注意力地图。然后，我们提出了韵律蒸馏显式学习的韵律信息的声学编码器，而不是连接的隐式韵律功能。这两种方法都提高了基线结果，韵律蒸馏方法在SLURP和STOP数据集上的意图分类准确率比韵律基线提高了8%和2%。摘要：Most End-to-End SLU methods depend on the pretrained ASR or language model features for intent prediction. However, other essential information in speech, such as prosody, is often ignored. Recent research has shown improved results in classifying dialogue acts by incorporating prosodic information. The margins of improvement in these methods are minimal as the neural models ignore prosodic features. In this work, we propose prosody-attention, which uses the prosodic features differently to generate attention maps across time frames of the utterance. Then we propose prosody-distillation to explicitly learn the prosodic information in the acoustic encoder rather than concatenating the implicit prosodic features. Both the proposed methods improve the baseline results, and the prosody-distillation method gives an intent classification accuracy improvement of 8 % and 2 % on SLURP and STOP datasets over the prosody baseline.
【7】 SongDriver2: Real-time Emotion-based Music Arrangement with Soft  Transition标题：SongDriver2：基于实时情感的软过渡音乐编曲链接：https://arxiv.org/abs/2305.08029
作者：Zihao Wang,Le Ma,Chen Zhang,Bo Han,Yikai Wang,Xinyi Chen,HaoRong Hong,Wenbo Liu,Xinda Wu,Kejun Zhang机构：Zhejiang University, China, Innovation Center of Yangtze River, Delta摘要：基于实时情感的音乐编排旨在实时地将给定的音乐作品转换成唤起与用户的特定情感共鸣的另一个音乐作品，在各种场景中具有重要的应用价值，例如，音乐疗法、电子游戏配乐和电影配乐。然而，由于目标情感的细粒度和可变性质，平衡情感实时拟合与软情感过渡是一个挑战。现有的研究主要集中在实现情感的实时拟合，而软过渡的问题仍然研究不足，影响整体的情感连贯的音乐。在本文中，我们提出了SongDriver2来解决这种平衡。具体来说，我们首先识别最后一个时间步的音乐情感，然后将其与当前时间步的目标输入情感融合。然后，融合的情感用作SongDriver2基于输入旋律数据生成即将到来的音乐的指导。为了灵活调整音乐相似度和情感实时拟合，我们对原始旋律进行下采样并将其馈送到生成模型中。此外，我们设计了四个音乐理论功能，以利用领域知识来增强情感信息，并采用半监督学习来减轻手动数据集注释引入的主观偏见。根据评估结果，SongDriver2在客观和主观指标上都超过了最先进的方法。这些结果表明，SongDriver2同时实现了实时适配和柔和过渡，增强了所生成音乐的连贯性。摘要：Real-time emotion-based music arrangement, which aims to transform a given music piece into another one that evokes specific emotional resonance with the user in real-time, holds significant application value in various scenarios, e.g., music therapy, video game soundtracks, and movie scores. However, balancing emotion real-time fit with soft emotion transition is a challenge due to the fine-grained and mutable nature of the target emotion. Existing studies mainly focus on achieving emotion real-time fit, while the issue of soft transition remains understudied, affecting the overall emotional coherence of the music. In this paper, we propose SongDriver2 to address this balance. Specifically, we first recognize the last timestep's music emotion and then fuse it with the current timestep's target input emotion. The fused emotion then serves as the guidance for SongDriver2 to generate the upcoming music based on the input melody data. To adjust music similarity and emotion real-time fit flexibly, we downsample the original melody and feed it into the generation model. Furthermore, we design four music theory features to leverage domain knowledge to enhance emotion information and employ semi-supervised learning to mitigate the subjective bias introduced by manual dataset annotation. According to the evaluation results, SongDriver2 surpasses the state-of-the-art methods in both objective and subjective metrics. These results demonstrate that SongDriver2 achieves real-time fit and soft transitions simultaneously, enhancing the coherence of the generated music.
【8】 Sound-to-Vibration Transformation for Sensorless Motor Health Monitoring标题：用于无位置传感器电机健康监测的声振转换链接：https://arxiv.org/abs/2305.07960
作者：Ozer Can Devecioglu,Serkan Kiranyaz,Amer Elhmes,Sadok Sassi,Turker Ince,Onur Avci,Mohammad Hesam Soleimani-Babakamali,Ertugrul Taciroglu,Moncef Gabbouj机构：M摘要：基于传感器的电机故障（如轴承故障）的自动检测对于各个行业的预测性维护至关重要。多年来已经开发了许多方法来检测轴承故障。尽管已经提出了许多不同的方法来诊断电机中的故障，基于振动的方法已经成为事实上的标准和最常用的技术。然而，获取可靠的振动信号，尤其是从旋转机械获取可靠的振动信号，有时可能由于具有挑战性的安装和操作条件（例如，振动传感器）而难以实现。电机主体上的加速度计位置的变化），这将不仅显著地改变信号模式，而且还可能引起严重的伪影。此外，传感器是昂贵的并且需要定期维护以维持可靠的信号采集。为了解决这些缺点，并避免振动传感器的需要，在这项研究中，我们提出了一种新的声音到振动的转换方法，可以合成现实的振动信号直接从声音测量，无论工作条件，故障类型和故障的严重程度。结果，使用该变换，由简单的声音记录器获取的数据，例如，移动电话可以被转换成振动信号，然后该振动信号可以通过预先训练的模型用于故障检测。所提出的方法进行了广泛评估的基准卡塔尔大学双机轴承故障基准数据集（QU-DMBF），其中封装的声音和振动数据从两个不同的机器在各种条件下运行。实验结果表明，这种新的方法可以合成这样的现实振动信号，可以直接用于可靠的和高精度的电机健康监测。摘要：Automatic sensor-based detection of motor failures such as bearing faults is crucial for predictive maintenance in various industries. Numerous methodologies have been developed over the years to detect bearing faults. Despite the appearance of numerous different approaches for diagnosing faults in motors have been proposed, vibration-based methods have become the de facto standard and the most commonly used techniques. However, acquiring reliable vibration signals, especially from rotating machinery, can sometimes be infeasibly difficult due to challenging installation and operational conditions (e.g., variations on accelerometer locations on the motor body), which will not only alter the signal patterns significantly but may also induce severe artifacts. Moreover, sensors are costly and require periodic maintenance to sustain a reliable signal acquisition. To address these drawbacks and void the need for vibration sensors, in this study, we propose a novel sound-to-vibration transformation method that can synthesize realistic vibration signals directly from the sound measurements regardless of the working conditions, fault type, and fault severity. As a result, using this transformation, the data acquired by a simple sound recorder, e.g., a mobile phone, can be transformed into the vibration signal, which can then be used for fault detection by a pre-trained model. The proposed method is extensively evaluated over the benchmark Qatar University Dual-Machine Bearing Fault Benchmark dataset (QU-DMBF), which encapsulates sound and vibration data from two different machines operating under various conditions. Experimental results show that this novel approach can synthesize such realistic vibration signals that can directly be used for reliable and highly accurate motor health monitoring.
【9】 APNet: An All-Frame-Level Neural Vocoder Incorporating Direct Prediction  of Amplitude and Phase Spectra标题：APNet：一种结合幅相谱直接预测的全帧级神经声码器链接：https://arxiv.org/abs/2305.07952
作者：Yang Ai,Zhen-Hua Ling备注：Accepted by IEEE/ACM Transactions on Audio, Speech, and Language Processing. Codes are available摘要：本文提出了一种新的神经声码器APNet，它通过直接预测幅度谱和相位谱来从声学特征重构语音波形。APNet声码器由幅度谱预测器（ASP）和相位谱预测器（PSP）组成。ASP是一个残差卷积网络，它从声学特征预测帧级对数幅度谱。PSP还采用了一个以声学特征为输入的残差卷积网络，然后将该网络的输出分别通过两个并行的线性卷积层，最后集成成一个相位计算公式，估计帧级相位谱。最后，ASP和PSP的输出相结合，重建语音波形的逆短时傅里叶变换（ISTFT）。ASP和PSP的所有操作都在帧级执行。我们共同训练ASP和PSP，并根据幅度均方误差，相位反包裹误差，短时谱不一致误差和时域重建误差定义多级损失函数。实验结果表明，由于全帧级操作，我们提出的APNet声码器在CPU上实现了比HiFi-GAN v1快约8倍的推理速度，而其合成语音质量与HiFi-GAN v1相当。APNet声码器的合成语音质量也优于几个同样有效的模型。消融实验也证实了所提出的并行相位估计架构是必要的相位建模和建议的损失函数是有助于提高合成语音质量。摘要：This paper presents a novel neural vocoder named APNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra directly. The APNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a residual convolution network which predicts frame-level log amplitude spectra from acoustic features. The PSP also adopts a residual convolution network using acoustic features as input, then passes the output of this network through two parallel linear convolution layers respectively, and finally integrates into a phase calculation formula to estimate frame-level phase spectra. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by inverse short-time Fourier transform (ISTFT). All operations of the ASP and PSP are performed at the frame level. We train the ASP and PSP jointly and define multilevel loss functions based on amplitude mean square error, phase anti-wrapping error, short-time spectral inconsistency error and time domain reconstruction error. Experimental results show that our proposed APNet vocoder achieves an approximately 8x faster inference speed than HiFi-GAN v1 on a CPU due to the all-frame-level operations, while its synthesized speech quality is comparable to HiFi-GAN v1. The synthesized speech quality of the APNet vocoder is also better than that of several equally efficient models. Ablation experiments also confirm that the proposed parallel phase estimation architecture is essential to phase modeling and the proposed loss functions are helpful for improving the synthesized speech quality.
【10】 Higher-Order Frequency Modulation Synthesis标题：高阶调频合成链接：https://arxiv.org/abs/2305.07909
作者：Victor Lazzarini,Joseph Timoney机构：Ireland, ARTICLE HISTORY备注：15 pages, 6 figures摘要：频率调制（FM）和相位调制（PM）是众所周知的合成方法，其已被广泛地部署在乐器中。最近，一些合成器已经实现了直接形式的FM（与PM相反），至少作为其设计的一部分，允许更高阶的调制拓扑。然而，这样的实施方式受到在频率调制中出现的公知困难的影响，这些困难通常通过使用PM来解决。在这篇文章中，我们分析了这些问题，并使用与PM的直接比较，我们提出了一个解决方案，直接应用FM在高阶调制安排。我们首先回顾了一阶FM的理论，将其与PM进行对比。然后，我们继续开发一个配方的二阶FM，这是相当于无问题的PM合成，并提出了一个封闭形式的表达式的二阶FM频谱的评价。然后，我们扩展到高阶拓扑结构的原则，通过推进FM运营商的概念，类似于PM仪器设计中使用的一个。从这一点上，我们表明，反馈调频也是一个实际的可能性。最后，我们补充的文件，在C++的参考实现。摘要：Frequency modulation (FM) and phase modulation (PM) are well-known synthesis methods, which have been deployed widely in musical instruments. More recently, some synthesisers have implemented direct forms of FM (as opposed to PM), allowing, at least as part of their design, for higher-order modulation topologies. However, such implementations are affected by well-known difficulties that arise in the modulation of frequency, which are normally solved by the use of PM. In this article, we analyse these problems and using a direct comparison with PM, we put forward a solution for the direct application of FM in higher-order modulation arrangements. We begin by reviewing the theory of first-order FM, contrasting it to PM. We then proceed to develop a formulation of second-order FM which is equivalent to the issue-free PM synthesis, and present a closed-form expression for the evaluation of the second-order FM spectrum. We then extend the principle to higher-order topologies, by advancing the concept of an FM operator, analogous to the one used in PM instrument designs. From this we demonstrate that feedback FM is also a practical possibility. Finally, we complement the paper by giving a reference implementation in C++.
【11】 Description and Discussion on DCASE 2023 Challenge Task 2: First-Shot  Unsupervised Anomalous Sound Detection for Machine Condition Monitoring标题：DCASE 2023挑战任务2：用于机器状态监测的首发非监督异常声音检测链接：https://arxiv.org/abs/2305.07828
作者：Kota Dohi,Keisuke Imoto,Noboru Harada,Daisuke Niizumi,Yuma Koizumi,Tomoya Nishida,Harsh Purohit,Ryo Tanabe,Takashi Endo,Yohei Kawaguchi机构：com 2 Doshisha University备注：anomaly detection, acoustic condition monitoring, domain shift, first-shot problem, DCASE Challenge. arXiv admin note: substantial text overlap with arXiv:2206.05876, arXiv:2106.04492摘要：我们提出了声学场景和事件的检测和分类（DCASE）2023挑战任务2的任务描述：“用于机器状态监控的首次无监督异常声音检测（ASD）”。其主要目标是使ASD系统的快速部署的新类型的机器使用只有几个正常的样本，而不需要超参数调整。在过去的ASD任务中，开发的方法为每种机器类型调整超参数，因为开发和评估数据集具有相同的机器类型。然而，收集正常和异常数据作为开发数据集在实践中可能是不可行的。在2023年的任务2中，我们专注于解决第一次射击问题，这是在几台全新机器类型的机器上训练模型的挑战。具体而言，（i）每种机器类型只有一个部分，以及（ii）开发和评估数据集中的机器类型完全不同。我们将在挑战提交截止日期后添加挑战结果和提交分析。摘要：We present the task description of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge Task 2: "First-shot unsupervised anomalous sound detection (ASD) for machine condition monitoring". The main goal is to enable rapid deployment of ASD systems for new kinds of machines using only a few normal samples, without the need for hyperparameter tuning. In the past ASD tasks, developed methods tuned hyperparameters for each machine type, as the development and evaluation datasets had the same machine types. However, collecting normal and anomalous data as the development dataset can be infeasible in practice. In 2023 Task 2, we focus on solving first-shot problem, which is the challenge of training a model on a few machines of a completely novel machine type. Specifically, (i) each machine type has only one section, and (ii) machine types in the development and evaluation datasets are completely different. We will add challenge results and analysis of the submissions after the challenge submission deadline.
【12】 Masked Audio Text Encoders are Effective Multi-Modal Rescorers标题：掩蔽音频文本编码器是一种有效的多模式识别器链接：https://arxiv.org/abs/2305.07677
作者：Jinglun Cai,Monica Sunkara,Xilai Li,Anshu Bhatia,Xiao Pan,Sravan Bodapati机构：AWS AI Labs摘要：掩蔽语言模型（Masked Language Models，MLMs）已被证明是自动语音识别（Automatic Speech Recognition，ASR）系统中二次评分的有效方法。在这项工作中，我们提出了掩蔽的音频文本编码器（MATE），多模态掩蔽语言模型rescoer，它将声学表示到MLM的输入空间。我们采用对比学习有效地对齐的方式，通过学习共享的表征。我们表明，使用多模态rescoorer是有益的ASR系统的域推广时，目标域数据不可用。与纯文本基线相比，MATE在域内数据集上将单词错误率（WER）降低了4%-16%，在域外数据集上降低了3%-7%。此外，在训练数据量非常有限（0.8小时）的情况下，MATE实现了比第一遍基线降低8%-23%的WER。摘要：Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours), MATE achieves a WER reduction of 8%-23% over the first-pass baseline.
【13】 Integrating Uncertainty into Neural Network-based Speech Enhancement标题：神经网络语音增强中的不确定性集成链接：https://arxiv.org/abs/2305.08744
作者：Huajian Fang,Dennis Becker,Stefan Wermter,Timo Gerkmann机构：Senior Member, IEEE备注：None摘要：时频域中的监督掩蔽方法旨在采用深度神经网络来估计乘法掩码以提取干净的语音。这导致对每个输入的单个估计，而没有任何可靠性的保证或措施。在本文中，我们研究了清洁语音估计建模的不确定性的好处。预测不确定性通常分为任意不确定性和认知不确定性。前者指的是数据中固有的随机性，而后者描述了模型参数的不确定性。在这项工作中，我们提出了一个框架，以联合模型任意和认知的不确定性，基于神经网络的语音增强。该方法通过估计语音后验分布的统计矩来捕获任意的不确定性，并明确地将不确定性估计纳入其中，以进一步提高干净的语音估计。对于认知不确定性，我们研究了两种贝叶斯深度学习方法：蒙特卡罗丢弃和深度集成来量化神经网络参数的不确定性。我们的分析表明，所提出的框架，促进捕获实际和可靠的不确定性，而结合不同来源的不确定性产生更可靠的预测不确定性估计。此外，我们通过评估不同数据集上的框架，展示了建模不确定性对语音增强性能的好处，与无法考虑不确定性的可比模型相比，表现出显着的改进。摘要：Supervised masking approaches in the time-frequency domain aim to employ deep neural networks to estimate a multiplicative mask to extract clean speech. This leads to a single estimate for each input without any guarantees or measures of reliability. In this paper, we study the benefits of modeling uncertainty in clean speech estimation. Prediction uncertainty is typically categorized into aleatoric uncertainty and epistemic uncertainty. The former refers to inherent randomness in data, while the latter describes uncertainty in the model parameters. In this work, we propose a framework to jointly model aleatoric and epistemic uncertainties in neural network-based speech enhancement. The proposed approach captures aleatoric uncertainty by estimating the statistical moments of the speech posterior distribution and explicitly incorporates the uncertainty estimate to further improve clean speech estimation. For epistemic uncertainty, we investigate two Bayesian deep learning approaches: Monte Carlo dropout and Deep ensembles to quantify the uncertainty of the neural network parameters. Our analyses show that the proposed framework promotes capturing practical and reliable uncertainty, while combining different sources of uncertainties yields more reliable predictive uncertainty estimates. Furthermore, we demonstrate the benefits of modeling uncertainty on speech enhancement performance by evaluating the framework on different datasets, exhibiting notable improvement over comparable models that fail to account for uncertainty.
【14】 DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement标题：DeepFilterNet：感知激励的实时语音增强链接：https://arxiv.org/abs/2305.08227
作者：Hendrik Schröter,Tobias Rosenkranz,Alberto N. Escalante-B.,Andreas Maier机构：Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Pattern Recognition Lab, WS Audiology, Research and Development, Erlangen, Germany备注：Accepted as show and tell demo to interspeech 2023摘要：用于单通道语音增强的多帧算法能够利用语音信号内的短时相关性。深度滤波（DF）被提出来直接在频域中估计复数滤波器以利用这些相关性。在这项工作中，我们提出了一个实时语音增强演示使用DeepFilterNet。DeepFilterNet的效率是通过利用语音产生和心理声学感知的领域知识来实现的。我们的模型是能够匹配国家的最先进的语音增强基准，同时实现一个单线程笔记本电脑CPU上的实时系数为0.19。该框架以及预训练的权重已在开源许可证下发布。摘要：Multi-frame algorithms for single-channel speech enhancement are able to take advantage from short-time correlations within the speech signal. Deep Filtering (DF) was proposed to directly estimate a complex filter in frequency domain to take advantage of these correlations. In this work, we present a real-time speech enhancement demo using DeepFilterNet. DeepFilterNet's efficiency is enabled by exploiting domain knowledge of speech production and psychoacoustic perception. Our model is able to match state-of-the-art speech enhancement benchmarks while achieving a real-time-factor of 0.19 on a single threaded notebook CPU. The framework as well as pretrained weights have been published under an open source license.
【15】 Vocal Style Factorization for Effective Speaker Recognition in Affective  Scenarios标题：情感场景下有效说话人识别的声调分解链接：https://arxiv.org/abs/2305.07997
作者：Morgan Sandler,Arun Ross机构：Michigan State University, USA摘要：自动说话人识别的准确性受到人的语音中的情绪变化的负面影响。在本文中，我们假设，扬声器的身份是由各种声乐风格的因素，可以从未标记的数据和重新组合使用的神经网络架构，以产生整体的扬声器身份表示的情感场景。在这方面，我们提出了E-Vector架构，由1-D CNN学习扬声器身份特征和声乐风格因子分解技术确定声乐风格。MSP-Podcast数据集上进行的实验表明，所提出的架构提高了国家的最先进的说话人识别的准确性，在情感领域的基线ECAPA-TDNN说话人识别模型。例如，在1%的错误匹配率下的真实匹配率从27.6%提高到46.2%。摘要：The accuracy of automated speaker recognition is negatively impacted by change in emotions in a person's speech. In this paper, we hypothesize that speaker identity is composed of various vocal style factors that may be learned from unlabeled data and re-combined using a neural network architecture to generate holistic speaker identity representations for affective scenarios. In this regard we propose the E-Vector architecture, composed of a 1-D CNN for learning speaker identity features and a vocal style factorization technique for determining vocal styles. Experiments conducted on the MSP-Podcast dataset demonstrate that the proposed architecture improves state-of-the-art speaker recognition accuracy in the affective domain over baseline ECAPA-TDNN speaker recognition models. For instance, the true match rate at a false match rate of 1% improves from 27.6% to 46.2%.
【16】 The Whole Is Greater than the Sum of Its Parts: Improving DNN-based  Music Source Separation标题：整体大于部分之和：改进基于DNN的音乐源分离链接：https://arxiv.org/abs/2305.07855
作者：Ryosuke Sawata,Naoya Takahashi,Stefan Uhlich,Shusuke Takahashi,Yuki Mitsufuji备注：Submitted to IEEE TASLP (under review), 11 pages, 8 figures摘要：本文提出了交叉方案（X方案），用于在不增加计算成本的情况下提高基于深度神经网络（DNN）的音乐源分离（MSS）的性能。它由三个部分组成：（i）多域损耗（MDL），（ii）桥接操作，其耦合各个仪器网络，以及（iii）组合损耗（CL）。MDL使得能够利用音频信号的频域和时域表示。我们修改目标网络，即，原始的基于DNN的MSS的网络架构，通过为每个输出仪器添加桥接路径来共享它们的信息。然后将MDL应用于输出源的组合以及每个独立源，因此我们称之为CL。MDL和CL可以很容易地应用于许多基于DNN的分离方法，因为它们仅仅是仅在训练期间使用的损失函数，并且不影响推理步骤。桥接操作不会增加网络中可学习参数的数量。实验结果表明，有效的开放-Unmix（UMX）和密集连接的扩张DenseNet（D3 Net）扩展与我们的X-计划，分别称为X-UMX和X-D3 Net，通过比较他们与原来的版本。我们还验证了X-计划在大规模数据制度的有效性，显示其一般性的数据大小。X-UMX Large（X-UMXL）是在大规模内部数据上训练并用于我们的实验中的，最近可在www.example.com上获得https: github.com asteroid-team asteroid tree master egs musdb18 X-UMX。摘要：This paper presents the crossing scheme (X-scheme) for improving the performance of deep neural network (DNN)-based music source separation (MSS) without increasing calculation cost. It consists of three components: (i) multi-domain loss (MDL), (ii) bridging operation, which couples the individual instrument networks, and (iii) combination loss (CL). MDL enables the taking advantage of the frequency- and time-domain representations of audio signals. We modify the target network, i.e., the network architecture of the original DNN-based MSS, by adding bridging paths for each output instrument to share their information. MDL is then applied to the combinations of the output sources as well as each independent source, hence we called it CL. MDL and CL can easily be applied to many DNN-based separation methods as they are merely loss functions that are only used during training and do not affect the inference step. Bridging operation does not increase the number of learnable parameters in the network. Experimental results showed that the validity of Open-Unmix (UMX) and densely connected dilated DenseNet (D3Net) extended with our X-scheme, respectively called X-UMX and X-D3Net, by comparing them with their original versions. We also verified the effectiveness of X-scheme in a large-scale data regime, showing its generality with respect to data size. X-UMX Large (X-UMXL), which was trained on large-scale internal data and used in our experiments, is newly available at https: github.com asteroid-team asteroid tree master egs musdb18 X-UMX.
eess.AS音频处理
【1】 Integrating Uncertainty into Neural Network-based Speech Enhancement标题：神经网络语音增强中的不确定性集成链接：https://arxiv.org/abs/2305.08744
*与cs.SD语音【13】为同一篇
作者：Huajian Fang,Dennis Becker,Stefan Wermter,Timo Gerkmann机构：Senior Member, IEEE备注：None摘要：时频域中的监督掩蔽方法旨在采用深度神经网络来估计乘法掩码以提取干净的语音。这导致对每个输入的单个估计，而没有任何可靠性的保证或措施。在本文中，我们研究了清洁语音估计建模的不确定性的好处。预测不确定性通常分为任意不确定性和认知不确定性。前者指的是数据中固有的随机性，而后者描述了模型参数的不确定性。在这项工作中，我们提出了一个框架，以联合模型任意和认知的不确定性，基于神经网络的语音增强。该方法通过估计语音后验分布的统计矩来捕获任意的不确定性，并明确地将不确定性估计纳入其中，以进一步提高干净的语音估计。对于认知不确定性，我们研究了两种贝叶斯深度学习方法：蒙特卡罗丢弃和深度集成来量化神经网络参数的不确定性。我们的分析表明，所提出的框架，促进捕获实际和可靠的不确定性，而结合不同来源的不确定性产生更可靠的预测不确定性估计。此外，我们通过评估不同数据集上的框架，展示了建模不确定性对语音增强性能的好处，与无法考虑不确定性的可比模型相比，表现出显着的改进。摘要：Supervised masking approaches in the time-frequency domain aim to employ deep neural networks to estimate a multiplicative mask to extract clean speech. This leads to a single estimate for each input without any guarantees or measures of reliability. In this paper, we study the benefits of modeling uncertainty in clean speech estimation. Prediction uncertainty is typically categorized into aleatoric uncertainty and epistemic uncertainty. The former refers to inherent randomness in data, while the latter describes uncertainty in the model parameters. In this work, we propose a framework to jointly model aleatoric and epistemic uncertainties in neural network-based speech enhancement. The proposed approach captures aleatoric uncertainty by estimating the statistical moments of the speech posterior distribution and explicitly incorporates the uncertainty estimate to further improve clean speech estimation. For epistemic uncertainty, we investigate two Bayesian deep learning approaches: Monte Carlo dropout and Deep ensembles to quantify the uncertainty of the neural network parameters. Our analyses show that the proposed framework promotes capturing practical and reliable uncertainty, while combining different sources of uncertainties yields more reliable predictive uncertainty estimates. Furthermore, we demonstrate the benefits of modeling uncertainty on speech enhancement performance by evaluating the framework on different datasets, exhibiting notable improvement over comparable models that fail to account for uncertainty.
【2】 DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement标题：DeepFilterNet：感知激励的实时语音增强链接：https://arxiv.org/abs/2305.08227
*与cs.SD语音【14】为同一篇
作者：Hendrik Schröter,Tobias Rosenkranz,Alberto N. Escalante-B.,Andreas Maier机构：Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Pattern Recognition Lab, WS Audiology, Research and Development, Erlangen, Germany备注：Accepted as show and tell demo to interspeech 2023摘要：用于单通道语音增强的多帧算法能够利用语音信号内的短时相关性。深度滤波（DF）被提出来直接在频域中估计复数滤波器以利用这些相关性。在这项工作中，我们提出了一个实时语音增强演示使用DeepFilterNet。DeepFilterNet的效率是通过利用语音产生和心理声学感知的领域知识来实现的。我们的模型是能够匹配国家的最先进的语音增强基准，同时实现一个单线程笔记本电脑CPU上的实时系数为0.19。该框架以及预训练的权重已在开源许可证下发布。摘要：Multi-frame algorithms for single-channel speech enhancement are able to take advantage from short-time correlations within the speech signal. Deep Filtering (DF) was proposed to directly estimate a complex filter in frequency domain to take advantage of these correlations. In this work, we present a real-time speech enhancement demo using DeepFilterNet. DeepFilterNet's efficiency is enabled by exploiting domain knowledge of speech production and psychoacoustic perception. Our model is able to match state-of-the-art speech enhancement benchmarks while achieving a real-time-factor of 0.19 on a single threaded notebook CPU. The framework as well as pretrained weights have been published under an open source license.
【3】 Deep Multi-Frame Filtering for Hearing Aids标题：深度多帧滤波在助听器中的应用链接：https://arxiv.org/abs/2305.08225
作者：Hendrik Schröter,Tobias Rosenkranz,Alberto N. Escalante-B.,Andreas Maier机构：Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg, Pattern Recognition Lab, WS Audiology, Research and Development, Erlangen, Germany备注：Submitted to Interspeech 2023摘要：用于单通道语音增强的多帧算法能够利用语音信号内的短时相关性。深度滤波（DF）最近展示了其用于低延迟场景的能力，例如助听器及其复杂的多帧（MF）滤波器。或者，可以经由MF最小方差无失真响应（MVDR）或MF维纳滤波器（WF）来估计复数滤波器。先前的研究已经表明，与经由DF的直接滤波器估计相比，使用MVDR滤波器并入算法域知识可能是有益的。在这项工作中，我们比较了使用各种多帧过滤器，如DF，MF-MVDR，或MF-WF的HA。我们评估不同的协方差估计方法MF-MVDR和MF-WF和客观地表现出改善的性能相比，直接DF估计，显着优于相关的工作，同时提高运行时的性能。摘要：Multi-frame algorithms for single-channel speech enhancement are able to take advantage from short-time correlations within the speech signal. Deep filtering (DF) recently demonstrated its capabilities for low-latency scenarios like hearing aids with its complex multi-frame (MF) filter. Alternatively, the complex filter can be estimated via an MF minimum variance distortionless response (MVDR), or MF Wiener filter (WF). Previous studies have shown that incorporating algorithm domain knowledge using an MVDR filter might be beneficial compared to the direct filter estimation via DF. In this work, we compare the usage of various multi-frame filters such as DF, MF-MVDR, or MF-WF for HAs. We assess different covariance estimation methods for both MF-MVDR and MF-WF and objectively demonstrate an improved performance compared to direct DF estimation, significantly outperforming related work while improving the runtime performance.
【4】 Vocal Style Factorization for Effective Speaker Recognition in Affective  Scenarios标题：情感场景下有效说话人识别的声调分解链接：https://arxiv.org/abs/2305.07997
*与cs.SD语音【15】为同一篇
作者：Morgan Sandler,Arun Ross机构：Michigan State University, USA摘要：自动说话人识别的准确性受到人的语音中的情绪变化的负面影响。在本文中，我们假设，扬声器的身份是由各种声乐风格的因素，可以从未标记的数据和重新组合使用的神经网络架构，以产生整体的扬声器身份表示的情感场景。在这方面，我们提出了E-Vector架构，由1-D CNN学习扬声器身份特征和声乐风格因子分解技术确定声乐风格。MSP-Podcast数据集上进行的实验表明，所提出的架构提高了国家的最先进的说话人识别的准确性，在情感领域的基线ECAPA-TDNN说话人识别模型。例如，在1%的错误匹配率下的真实匹配率从27.6%提高到46.2%。摘要：The accuracy of automated speaker recognition is negatively impacted by change in emotions in a person's speech. In this paper, we hypothesize that speaker identity is composed of various vocal style factors that may be learned from unlabeled data and re-combined using a neural network architecture to generate holistic speaker identity representations for affective scenarios. In this regard we propose the E-Vector architecture, composed of a 1-D CNN for learning speaker identity features and a vocal style factorization technique for determining vocal styles. Experiments conducted on the MSP-Podcast dataset demonstrate that the proposed architecture improves state-of-the-art speaker recognition accuracy in the affective domain over baseline ECAPA-TDNN speaker recognition models. For instance, the true match rate at a false match rate of 1% improves from 27.6% to 46.2%.
【5】 The Whole Is Greater than the Sum of Its Parts: Improving DNN-based  Music Source Separation标题：整体大于部分之和：改进基于DNN的音乐源分离链接：https://arxiv.org/abs/2305.07855
*与cs.SD语音【16】为同一篇
作者：Ryosuke Sawata,Naoya Takahashi,Stefan Uhlich,Shusuke Takahashi,Yuki Mitsufuji备注：Submitted to IEEE TASLP (under review), 11 pages, 8 figures摘要：本文提出了交叉方案（X方案），用于在不增加计算成本的情况下提高基于深度神经网络（DNN）的音乐源分离（MSS）的性能。它由三个部分组成：（i）多域损耗（MDL），（ii）桥接操作，其耦合各个仪器网络，以及（iii）组合损耗（CL）。MDL使得能够利用音频信号的频域和时域表示。我们修改目标网络，即，原始的基于DNN的MSS的网络架构，通过为每个输出仪器添加桥接路径来共享它们的信息。然后将MDL应用于输出源的组合以及每个独立源，因此我们称之为CL。MDL和CL可以很容易地应用于许多基于DNN的分离方法，因为它们仅仅是仅在训练期间使用的损失函数，并且不影响推理步骤。桥接操作不会增加网络中可学习参数的数量。实验结果表明，有效的开放-Unmix（UMX）和密集连接的扩张DenseNet（D3 Net）扩展与我们的X-计划，分别称为X-UMX和X-D3 Net，通过比较他们与原来的版本。我们还验证了X-计划在大规模数据制度的有效性，显示其一般性的数据大小。X-UMX Large（X-UMXL）是在大规模内部数据上训练并用于我们的实验中的，最近可在www.example.com上获得https: github.com asteroid-team asteroid tree master egs musdb18 X-UMX。摘要：This paper presents the crossing scheme (X-scheme) for improving the performance of deep neural network (DNN)-based music source separation (MSS) without increasing calculation cost. It consists of three components: (i) multi-domain loss (MDL), (ii) bridging operation, which couples the individual instrument networks, and (iii) combination loss (CL). MDL enables the taking advantage of the frequency- and time-domain representations of audio signals. We modify the target network, i.e., the network architecture of the original DNN-based MSS, by adding bridging paths for each output instrument to share their information. MDL is then applied to the combinations of the output sources as well as each independent source, hence we called it CL. MDL and CL can easily be applied to many DNN-based separation methods as they are merely loss functions that are only used during training and do not affect the inference step. Bridging operation does not increase the number of learnable parameters in the network. Experimental results showed that the validity of Open-Unmix (UMX) and densely connected dilated DenseNet (D3Net) extended with our X-scheme, respectively called X-UMX and X-D3Net, by comparing them with their original versions. We also verified the effectiveness of X-scheme in a large-scale data regime, showing its generality with respect to data size. X-UMX Large (X-UMXL), which was trained on large-scale internal data and used in our experiments, is newly available at https: github.com asteroid-team asteroid tree master egs musdb18 X-UMX.
【6】 Ripple sparse self-attention for monaural speech enhancement标题：用于单声道语音增强的波纹稀疏自我注意链接：https://arxiv.org/abs/2305.08541
*与cs.SD语音【3】为同一篇
作者：Qiquan Zhang,Hongxu Zhu,Qi Song,Xinyuan Qian,Zhaoheng Ni,Haizhou Li机构：School of Electrical Engineering and Telecommunications, University of New South Wales, Australia, Department of Electrical and Computer Engineering, National University of Singapore, Singapore, Alibaba Group, China, Meta, United States备注：5 pages, ICASSP 2023 published摘要：Transformer的使用代表了语音增强的最新成功。然而，作为其核心组成部分，自我注意遭受二次复杂性，这是计算上禁止长的语音记录。此外，它允许每个时间帧参加所有的时间帧，忽略了语音信号的强局部相关性。这项研究提出了一个简单而有效的稀疏的语音增强，称为纹波的注意力，同时进行细粒度和粗粒度建模的本地和全球的依赖关系，分别。具体而言，我们采用本地频带注意力，使每个帧参加其最近的邻居帧在一个窗口中的细粒度，而采用扩大注意力的窗口外的模型的全局依赖性在一个粗略的粒度。我们评估了我们的涟漪注意力的两个常用的训练目标的语音增强的功效。广泛的实验结果一致证实了卓越的性能的波纹注意设计超过标准的完全自我注意，块式注意，和双路径注意（Sep-Former）的语音质量和清晰度。摘要：The use of Transformer represents a recent success in speech enhancement. However, as its core component, self-attention suffers from quadratic complexity, which is computationally prohibited for long speech recordings. Moreover, it allows each time frame to attend to all time frames, neglecting the strong local correlations of speech signals. This study presents a simple yet effective sparse self-attention for speech enhancement, called ripple attention, which simultaneously performs fine- and coarse-grained modeling for local and global dependencies, respectively. Specifically, we employ local band attention to enable each frame to attend to its closest neighbor frames in a window at fine granularity, while employing dilated attention outside the window to model the global dependencies at a coarse granularity. We evaluate the efficacy of our ripple attention for speech enhancement on two commonly used training objectives. Extensive experimental results consistently confirm the superior performance of the ripple attention design over standard full self-attention, blockwise attention, and dual-path attention (Sep-Former) in terms of speech quality and intelligibility.
【7】 ForkNet: Simultaneous Time and Time-Frequency Domain Modeling for Speech  Enhancement标题：ForkNet：语音增强的同时时频域建模链接：https://arxiv.org/abs/2305.08292
*与cs.SD语音【4】为同一篇
作者：Feng Dang,Qi Hu,Pengyuan Zhang,Yonghong Yan机构：University of Chinese Academy of Sciences摘要：语音增强中的先前研究主要集中于单独建模时间或时间-频率域信息，很少考虑同时建模两个域的潜在益处。由于这些域包含互补信息，因此组合它们可以提高模型的性能。在这封信中，我们提出了一种新的方法，同时在一个单一的模型的时间和时间-频率域的信息模型。我们开始与DPT-FSNet（因果版本）模型作为基线和修改的编码器结构，取代原来的编码器与三个单独的编码器，每个专用于建模时域，实虚，和幅度信息，分别。此外，我们在双路径处理块之前和之后引入了特征融合模块，以更好地利用来自不同域的信息。我们的实验结果表明，所提出的方法相比，现有的国家的最先进的因果模型，同时保持一个相对紧凑的模型大小和较低的计算复杂度，实现了卓越的性能。摘要：Previous research in speech enhancement has mostly focused on modeling time or time-frequency domain information alone, with little consideration given to the potential benefits of simultaneously modeling both domains. Since these domains contain complementary information, combining them may improve the performance of the model. In this letter, we propose a new approach to simultaneously model time and time-frequency domain information in a single model. We begin with the DPT-FSNet (causal version) model as a baseline and modify the encoder structure by replacing the original encoder with three separate encoders, each dedicated to modeling time-domain, real-imaginary, and magnitude information, respectively. Additionally, we introduce a feature fusion module both before and after the dual-path processing blocks to better leverage information from the different domains. The outcomes of our experiments reveal that the proposed approach achieves superior performance compared to existing state-of-the-art causal models, while preserving a relatively compact model size and low computational complexity.
【8】 Self-supervised Neural Factor Analysis for Disentangling Utterance-level  Speech Representations标题：解纠缠语音级语音表征的自监督神经因子分析链接：https://arxiv.org/abs/2305.08099
*与cs.SD语音【5】为同一篇
作者：Weiwei Lin,Chenhang He,Man-Wai Mak,Youzhi Tu机构：The Hong Kong Polytechnic University,  The Hong Kong PolytechnicUniversity备注：accepted by ICML 2023摘要：自监督学习（SSL）语音模型，如wav2vec和HuBERT，在自动语音识别（ASR）方面表现出了最先进的性能，并被证明在低标签资源设置中非常有用。然而，SSL模型的成功还没有转移到话语级任务，如说话人，情感和语言识别，这仍然需要监督微调的SSL模型，以获得良好的性能。我们认为，这个问题是由于缺乏解开的陈述和话语水平的学习目标，这些任务。受到HuBERT如何使用聚类来发现隐藏的声学单元的启发，我们制定了一个因子分析（FA）模型，该模型使用发现的隐藏的声学单元来对齐SSL特征。使用对齐特征上的概率推断，从语音内容中解开潜在的话语级表示。此外，从FA模型导出的变分下限提供了话语级目标，允许误差梯度被反向传播到Transformer层以学习高度区分的声学单元。当与HuBERT的掩蔽预测训练结合使用时，我们的模型在SUPERB基准测试中的所有话语级非语义任务上的表现优于当前最佳模型WavLM，只有20%的标记数据。摘要：Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition (ASR) and proved to be extremely useful in low label-resource settings. However, the success of SSL models has yet to transfer to utterance-level tasks such as speaker, emotion, and language recognition, which still require supervised fine-tuning of the SSL models to obtain good performance. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective for these tasks. Inspired by how HuBERT uses clustering to discover hidden acoustic units, we formulate a factor analysis (FA) model that uses the discovered hidden acoustic units to align the SSL features. The underlying utterance-level representations are disentangled from the content of speech using probabilistic inference on the aligned features. Furthermore, the variational lower bound derived from the FA model provides an utterance-level objective, allowing error gradients to be backpropagated to the Transformer layers to learn highly discriminative acoustic units. When used in conjunction with HuBERT's masked prediction training, our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.
【9】 Improving End-to-End SLU performance with Prosodic Attention and  Distillation标题：通过韵律注意和蒸馏提高端到端SLU性能链接：https://arxiv.org/abs/2305.08067
*与cs.SD语音【6】为同一篇
作者：Shangeth Rajaa机构：Skit.ai, Bengaluru, India备注：Submitted to InterSpeech 2023摘要：大多数端到端SLU方法依赖于用于意图预测的预训练ASR或语言模型特征。然而，语音中的其他重要信息，如韵律，往往被忽视。最近的研究表明，通过将韵律信息分类的对话行为的改进的结果。这些方法的改进幅度很小，因为神经模型忽略了韵律特征。在这项工作中，我们提出了韵律注意，它使用不同的韵律特征来生成跨时间帧的话语的注意力地图。然后，我们提出了韵律蒸馏显式学习的韵律信息的声学编码器，而不是连接的隐式韵律功能。这两种方法都提高了基线结果，韵律蒸馏方法在SLURP和STOP数据集上的意图分类准确率比韵律基线提高了8%和2%。摘要：Most End-to-End SLU methods depend on the pretrained ASR or language model features for intent prediction. However, other essential information in speech, such as prosody, is often ignored. Recent research has shown improved results in classifying dialogue acts by incorporating prosodic information. The margins of improvement in these methods are minimal as the neural models ignore prosodic features. In this work, we propose prosody-attention, which uses the prosodic features differently to generate attention maps across time frames of the utterance. Then we propose prosody-distillation to explicitly learn the prosodic information in the acoustic encoder rather than concatenating the implicit prosodic features. Both the proposed methods improve the baseline results, and the prosody-distillation method gives an intent classification accuracy improvement of 8 % and 2 % on SLURP and STOP datasets over the prosody baseline.
【10】 SongDriver2: Real-time Emotion-based Music Arrangement with Soft  Transition标题：SongDriver2：基于实时情感的软过渡音乐编曲链接：https://arxiv.org/abs/2305.08029
*与cs.SD语音【7】为同一篇
作者：Zihao Wang,Le Ma,Chen Zhang,Bo Han,Yikai Wang,Xinyi Chen,HaoRong Hong,Wenbo Liu,Xinda Wu,Kejun Zhang机构：Zhejiang University, China, Innovation Center of Yangtze River, Delta摘要：基于实时情感的音乐编排旨在实时地将给定的音乐作品转换成唤起与用户的特定情感共鸣的另一个音乐作品，在各种场景中具有重要的应用价值，例如，音乐疗法、电子游戏配乐和电影配乐。然而，由于目标情感的细粒度和可变性质，平衡情感实时拟合与软情感过渡是一个挑战。现有的研究主要集中在实现情感的实时拟合，而软过渡的问题仍然研究不足，影响整体的情感连贯的音乐。在本文中，我们提出了SongDriver2来解决这种平衡。具体来说，我们首先识别最后一个时间步的音乐情感，然后将其与当前时间步的目标输入情感融合。然后，融合的情感用作SongDriver2基于输入旋律数据生成即将到来的音乐的指导。为了灵活调整音乐相似度和情感实时拟合，我们对原始旋律进行下采样并将其馈送到生成模型中。此外，我们设计了四个音乐理论功能，以利用领域知识来增强情感信息，并采用半监督学习来减轻手动数据集注释引入的主观偏见。根据评估结果，SongDriver2在客观和主观指标上都超过了最先进的方法。这些结果表明，SongDriver2同时实现了实时适配和柔和过渡，增强了所生成音乐的连贯性。摘要：Real-time emotion-based music arrangement, which aims to transform a given music piece into another one that evokes specific emotional resonance with the user in real-time, holds significant application value in various scenarios, e.g., music therapy, video game soundtracks, and movie scores. However, balancing emotion real-time fit with soft emotion transition is a challenge due to the fine-grained and mutable nature of the target emotion. Existing studies mainly focus on achieving emotion real-time fit, while the issue of soft transition remains understudied, affecting the overall emotional coherence of the music. In this paper, we propose SongDriver2 to address this balance. Specifically, we first recognize the last timestep's music emotion and then fuse it with the current timestep's target input emotion. The fused emotion then serves as the guidance for SongDriver2 to generate the upcoming music based on the input melody data. To adjust music similarity and emotion real-time fit flexibly, we downsample the original melody and feed it into the generation model. Furthermore, we design four music theory features to leverage domain knowledge to enhance emotion information and employ semi-supervised learning to mitigate the subjective bias introduced by manual dataset annotation. According to the evaluation results, SongDriver2 surpasses the state-of-the-art methods in both objective and subjective metrics. These results demonstrate that SongDriver2 achieves real-time fit and soft transitions simultaneously, enhancing the coherence of the generated music.
【11】 Sound-to-Vibration Transformation for Sensorless Motor Health Monitoring标题：用于无位置传感器电机健康监测的声振转换链接：https://arxiv.org/abs/2305.07960
*与cs.SD语音【8】为同一篇
作者：Ozer Can Devecioglu,Serkan Kiranyaz,Amer Elhmes,Sadok Sassi,Turker Ince,Onur Avci,Mohammad Hesam Soleimani-Babakamali,Ertugrul Taciroglu,Moncef Gabbouj机构：M摘要：基于传感器的电机故障（如轴承故障）的自动检测对于各个行业的预测性维护至关重要。多年来已经开发了许多方法来检测轴承故障。尽管已经提出了许多不同的方法来诊断电机中的故障，基于振动的方法已经成为事实上的标准和最常用的技术。然而，获取可靠的振动信号，尤其是从旋转机械获取可靠的振动信号，有时可能由于具有挑战性的安装和操作条件（例如，振动传感器）而难以实现。电机主体上的加速度计位置的变化），这将不仅显著地改变信号模式，而且还可能引起严重的伪影。此外，传感器是昂贵的并且需要定期维护以维持可靠的信号采集。为了解决这些缺点，并避免振动传感器的需要，在这项研究中，我们提出了一种新的声音到振动的转换方法，可以合成现实的振动信号直接从声音测量，无论工作条件，故障类型和故障的严重程度。结果，使用该变换，由简单的声音记录器获取的数据，例如，移动电话可以被转换成振动信号，然后该振动信号可以通过预先训练的模型用于故障检测。所提出的方法进行了广泛评估的基准卡塔尔大学双机轴承故障基准数据集（QU-DMBF），其中封装的声音和振动数据从两个不同的机器在各种条件下运行。实验结果表明，这种新的方法可以合成这样的现实振动信号，可以直接用于可靠的和高精度的电机健康监测。摘要：Automatic sensor-based detection of motor failures such as bearing faults is crucial for predictive maintenance in various industries. Numerous methodologies have been developed over the years to detect bearing faults. Despite the appearance of numerous different approaches for diagnosing faults in motors have been proposed, vibration-based methods have become the de facto standard and the most commonly used techniques. However, acquiring reliable vibration signals, especially from rotating machinery, can sometimes be infeasibly difficult due to challenging installation and operational conditions (e.g., variations on accelerometer locations on the motor body), which will not only alter the signal patterns significantly but may also induce severe artifacts. Moreover, sensors are costly and require periodic maintenance to sustain a reliable signal acquisition. To address these drawbacks and void the need for vibration sensors, in this study, we propose a novel sound-to-vibration transformation method that can synthesize realistic vibration signals directly from the sound measurements regardless of the working conditions, fault type, and fault severity. As a result, using this transformation, the data acquired by a simple sound recorder, e.g., a mobile phone, can be transformed into the vibration signal, which can then be used for fault detection by a pre-trained model. The proposed method is extensively evaluated over the benchmark Qatar University Dual-Machine Bearing Fault Benchmark dataset (QU-DMBF), which encapsulates sound and vibration data from two different machines operating under various conditions. Experimental results show that this novel approach can synthesize such realistic vibration signals that can directly be used for reliable and highly accurate motor health monitoring.
【12】 APNet: An All-Frame-Level Neural Vocoder Incorporating Direct Prediction  of Amplitude and Phase Spectra标题：APNet：一种结合幅相谱直接预测的全帧级神经声码器链接：https://arxiv.org/abs/2305.07952
*与cs.SD语音【9】为同一篇
作者：Yang Ai,Zhen-Hua Ling备注：Accepted by IEEE/ACM Transactions on Audio, Speech, and Language Processing. Codes are available摘要：本文提出了一种新的神经声码器APNet，它通过直接预测幅度谱和相位谱来从声学特征重构语音波形。APNet声码器由幅度谱预测器（ASP）和相位谱预测器（PSP）组成。ASP是一个残差卷积网络，它从声学特征预测帧级对数幅度谱。PSP还采用了一个以声学特征为输入的残差卷积网络，然后将该网络的输出分别通过两个并行的线性卷积层，最后集成成一个相位计算公式，估计帧级相位谱。最后，ASP和PSP的输出相结合，重建语音波形的逆短时傅里叶变换（ISTFT）。ASP和PSP的所有操作都在帧级执行。我们共同训练ASP和PSP，并根据幅度均方误差，相位反包裹误差，短时谱不一致误差和时域重建误差定义多级损失函数。实验结果表明，由于全帧级操作，我们提出的APNet声码器在CPU上实现了比HiFi-GAN v1快约8倍的推理速度，而其合成语音质量与HiFi-GAN v1相当。APNet声码器的合成语音质量也优于几个同样有效的模型。消融实验也证实了所提出的并行相位估计架构是必要的相位建模和建议的损失函数是有助于提高合成语音质量。摘要：This paper presents a novel neural vocoder named APNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra directly. The APNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a residual convolution network which predicts frame-level log amplitude spectra from acoustic features. The PSP also adopts a residual convolution network using acoustic features as input, then passes the output of this network through two parallel linear convolution layers respectively, and finally integrates into a phase calculation formula to estimate frame-level phase spectra. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by inverse short-time Fourier transform (ISTFT). All operations of the ASP and PSP are performed at the frame level. We train the ASP and PSP jointly and define multilevel loss functions based on amplitude mean square error, phase anti-wrapping error, short-time spectral inconsistency error and time domain reconstruction error. Experimental results show that our proposed APNet vocoder achieves an approximately 8x faster inference speed than HiFi-GAN v1 on a CPU due to the all-frame-level operations, while its synthesized speech quality is comparable to HiFi-GAN v1. The synthesized speech quality of the APNet vocoder is also better than that of several equally efficient models. Ablation experiments also confirm that the proposed parallel phase estimation architecture is essential to phase modeling and the proposed loss functions are helpful for improving the synthesized speech quality.
【13】 Higher-Order Frequency Modulation Synthesis标题：高阶调频合成链接：https://arxiv.org/abs/2305.07909
*与cs.SD语音【10】为同一篇
作者：Victor Lazzarini,Joseph Timoney机构：Ireland, ARTICLE HISTORY备注：15 pages, 6 figures摘要：频率调制（FM）和相位调制（PM）是众所周知的合成方法，其已被广泛地部署在乐器中。最近，一些合成器已经实现了直接形式的FM（与PM相反），至少作为其设计的一部分，允许更高阶的调制拓扑。然而，这样的实施方式受到在频率调制中出现的公知困难的影响，这些困难通常通过使用PM来解决。在这篇文章中，我们分析了这些问题，并使用与PM的直接比较，我们提出了一个解决方案，直接应用FM在高阶调制安排。我们首先回顾了一阶FM的理论，将其与PM进行对比。然后，我们继续开发一个配方的二阶FM，这是相当于无问题的PM合成，并提出了一个封闭形式的表达式的二阶FM频谱的评价。然后，我们扩展到高阶拓扑结构的原则，通过推进FM运营商的概念，类似于PM仪器设计中使用的一个。从这一点上，我们表明，反馈调频也是一个实际的可能性。最后，我们补充的文件，在C++的参考实现。摘要：Frequency modulation (FM) and phase modulation (PM) are well-known synthesis methods, which have been deployed widely in musical instruments. More recently, some synthesisers have implemented direct forms of FM (as opposed to PM), allowing, at least as part of their design, for higher-order modulation topologies. However, such implementations are affected by well-known difficulties that arise in the modulation of frequency, which are normally solved by the use of PM. In this article, we analyse these problems and using a direct comparison with PM, we put forward a solution for the direct application of FM in higher-order modulation arrangements. We begin by reviewing the theory of first-order FM, contrasting it to PM. We then proceed to develop a formulation of second-order FM which is equivalent to the issue-free PM synthesis, and present a closed-form expression for the evaluation of the second-order FM spectrum. We then extend the principle to higher-order topologies, by advancing the concept of an FM operator, analogous to the one used in PM instrument designs. From this we demonstrate that feedback FM is also a practical possibility. Finally, we complement the paper by giving a reference implementation in C++.
【14】 Description and Discussion on DCASE 2023 Challenge Task 2: First-Shot  Unsupervised Anomalous Sound Detection for Machine Condition Monitoring标题：DCASE 2023挑战任务2：用于机器状态监测的首发非监督异常声音检测链接：https://arxiv.org/abs/2305.07828
*与cs.SD语音【11】为同一篇
作者：Kota Dohi,Keisuke Imoto,Noboru Harada,Daisuke Niizumi,Yuma Koizumi,Tomoya Nishida,Harsh Purohit,Ryo Tanabe,Takashi Endo,Yohei Kawaguchi机构：com 2 Doshisha University备注：anomaly detection, acoustic condition monitoring, domain shift, first-shot problem, DCASE Challenge. arXiv admin note: substantial text overlap with arXiv:2206.05876, arXiv:2106.04492摘要：我们提出了声学场景和事件的检测和分类（DCASE）2023挑战任务2的任务描述：“用于机器状态监控的首次无监督异常声音检测（ASD）”。其主要目标是使ASD系统的快速部署的新类型的机器使用只有几个正常的样本，而不需要超参数调整。在过去的ASD任务中，开发的方法为每种机器类型调整超参数，因为开发和评估数据集具有相同的机器类型。然而，收集正常和异常数据作为开发数据集在实践中可能是不可行的。在2023年的任务2中，我们专注于解决第一次射击问题，这是在几台全新机器类型的机器上训练模型的挑战。具体而言，（i）每种机器类型只有一个部分，以及（ii）开发和评估数据集中的机器类型完全不同。我们将在挑战提交截止日期后添加挑战结果和提交分析。摘要：We present the task description of the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023 Challenge Task 2: "First-shot unsupervised anomalous sound detection (ASD) for machine condition monitoring". The main goal is to enable rapid deployment of ASD systems for new kinds of machines using only a few normal samples, without the need for hyperparameter tuning. In the past ASD tasks, developed methods tuned hyperparameters for each machine type, as the development and evaluation datasets had the same machine types. However, collecting normal and anomalous data as the development dataset can be infeasible in practice. In 2023 Task 2, we focus on solving first-shot problem, which is the challenge of training a model on a few machines of a completely novel machine type. Specifically, (i) each machine type has only one section, and (ii) machine types in the development and evaluation datasets are completely different. We will add challenge results and analysis of the submissions after the challenge submission deadline.
机器翻译，仅供参考