今日论文合集：cs.SD语音11篇，eess.AS音频处理14篇。
cs.SD语音

【1】Conditional Generation of Audio from Video via Foley Analogies标题：通过Foley类比从视频中条件生成音频链接：http://arxiv.org/pdf/2304.08490v1

作者：Yuexi Du,Ziyang Chen,Justin Salamon,Bryan Russell,Andrew Owens机构：University of Michigan, Yale University, Adobe Research, Conditional audio, Generated audio备注：CVPR 2023摘要：设计师添加到视频中的声音效果旨在传达特定的艺术效果，因此可能与场景的真实声音大不相同。灵感的挑战，创建一个不同于其真实声音的视频的配乐，但仍然匹配屏幕上发生的动作，我们提出了条件福利的问题。我们提出以下贡献来解决这个问题。首先，我们提出了一个借口任务，用于训练我们的模型，使用从同一源视频中的另一个时间采样的有条件的视听剪辑来预测输入视频剪辑的声音。其次，我们提出了一个模型，用于生成一个无声的输入视频的配乐，给定一个用户提供的例子，指定什么视频应该“听起来像”。我们通过人类研究和自动评估指标表明，我们的模型成功地从视频中生成声音，同时根据提供的示例的内容改变其输出。项目地点：https: xypb.github.io CondFoleyGen。摘要：The sound effects that designers add to videos are designed to convey a particular artistic effect and, thus, may be quite different from a scene's true sound. Inspired by the challenges of creating a soundtrack for a video that differs from its true sound, but that nonetheless matches the actions occurring on screen, we propose the problem of conditional Foley. We present the following contributions to address this problem. First, we propose a pretext task for training our model to predict sound for an input video clip using a conditional audio-visual clip sampled from another time within the same source video. Second, we propose a model for generating a soundtrack for a silent input video, given a user-supplied example that specifies what the video should "sound like". We show through human studies and automated evaluation metrics that our model successfully generates sound from video, while varying its output according to the content of a supplied example. Project site: https: xypb.github.io CondFoleyGen

【2】 Prak: An automatic phonetic alignment tool for Czech标题：PRAK：捷克语语音自动对齐工具链接：http://arxiv.org/pdf/2304.08431v1

作者：Václav Hanžl,Adléta Hanžlová机构：Freelance researcher,Institute of Phonetics, Charles University in Prague备注：Submitted for ICPhS 2023摘要：将语音标记到音素的身份和时间边界是语音研究的劳动密集型部分。为了简化这项工作，我们创建了一个免费的开源工具，从捷克语文本中生成电话序列，并将其与音频进行时间对齐。低架构复杂度使得语音学专业的学生易于使用该设计。使用PyTorch在小型CommonVoice数据上训练具有56k权重的声学模型ReLU NN。比对和变体选择解码器在Python中使用矩阵库实现。捷克语发音生成器由简单的基于规则的块组成，在可能的情况下捕获语言的逻辑，允许修改转录方法的细节。与迄今为止使用的工具相比，数据准备效率提高，该工具可在Mac，Linux和Windows上以Praat GUI或命令行使用，实现大多数正确的发音变体选择，包括声门停止检测，算法捕获大部分捷克语同化逻辑，具有教学性和实用性。摘要：Labeling speech down to the identity and time boundaries of phones is a labor-intensive part of phonetic research. To simplify this work, we created a free open-source tool generating phone sequences from Czech text and time-aligning them with audio. Low architecture complexity makes the design approachable for students of phonetics. Acoustic model ReLU NN with 56k weights was trained using PyTorch on small CommonVoice data. Alignment and variant selection decoder is implemented in Python with matrix library. A Czech pronunciation generator is composed of simple rule-based blocks capturing the logic of the language where possible, allowing modification of transcription approach details. Compared to tools used until now, data preparation efficiency improved, the tool is usable on Mac, Linux and Windows in Praat GUI or command line, achieves mostly correct pronunciation variant choice including glottal stop detection, algorithmically captures most of Czech assimilation logic and is both didactic and practical.

【3】 Physics-inspired Neuroacoustic Computing Based on Tunable Nonlinear Multiple-scattering标题：基于可调谐非线性多次散射的物理启发神经声学计算链接：http://arxiv.org/pdf/2304.08380v1

作者：Ali Momeni,Xinxin Guo,Herve Lissek,Romain Fleury机构：Laboratory of Wave Engineering, Ecole Polytechnique F´ed´erale de Lausanne (EPFL), CH-, Lausanne, Switzerland., Signal Processing Laboratory LTS, Ecole Polytechnique F´ed´erale de Lausanne (EPFL), CH-, Lausanne备注：28 pages摘要：波，例如光和声音，由于由我们周围的复杂物质物体引起的多次散射而固有地反弹和混合。这种散射过程严重扰乱了波所携带的信息，挑战了传统的通信系统、传感模式和基于波的计算方案。在这里，我们表明，而不是成为一个障碍，多重散射可以是有益的，使和增强模拟非线性信息映射，允许直接物理实现的计算范例，如水库计算和极端学习机。我们提出了一个物理启发的版本，这样的计算架构的语音和元音识别，直接在本机域的输入信号，即对真实的声音，没有任何数字预处理或编码转换和反向传播训练计算。我们首先实现它的概念验证原型，一个非线性混沌声腔包含多个可调和节能的非线性元散射。我们证明了效率的声学为基础的计算系统的元音识别任务具有较高的测试分类准确率（91.4%）。最后，我们展示了高性能的元音识别在混响室的自然环境中。我们的研究结果为高效的声学学习机器开辟了道路，这些机器直接对输入声音进行操作，并利用物理学来实现自然语言处理（NLP）。摘要：Waves, such as light and sound, inherently bounce and mix due to multiple scattering induced by the complex material objects that surround us. This scattering process severely scrambles the information carried by waves, challenging conventional communication systems, sensing paradigms, and wave-based computing schemes. Here, we show that instead of being a hindrance, multiple scattering can be beneficial to enable and enhance analog nonlinear information mapping, allowing for the direct physical implementation of computational paradigms such as reservoir computing and extreme learning machines. We propose a physics-inspired version of such computational architectures for speech and vowel recognition that operate directly in the native domain of the input signal, namely on real-sounds, without any digital pre-processing or encoding conversion and backpropagation training computation. We first implement it in a proof-of-concept prototype, a nonlinear chaotic acoustic cavity containing multiple tunable and power-efficient nonlinear meta-scatterers. We prove the efficiency of the acoustic-based computing system for vowel recognition tasks with high testing classification accuracy (91.4%). Finally, we demonstrate the high performance of vowel recognition in the natural environment of a reverberation room. Our results open the way for efficient acoustic learning machines that operate directly on the input sound, and leverage physics to enable Natural Language Processing (NLP).

【4】 Fast Random Approximation of Multi-channel Room Impulse Response标题：多通道房间冲击响应的快速随机逼近链接：http://arxiv.org/pdf/2304.08052v1

作者：Yi Luo,Rongzhi Gu摘要：现代的基于神经网络的语音处理系统通常需要对混响具有鲁棒性，并且这样的系统的训练因此需要大量的混响数据。在系统的训练过程中，实时仿真流水线现在是优选的，因为它允许模型在无限数量的数据样本上进行训练，而无需预先生成并将其保存在硬盘上。因此，RIR仿真方法不仅需要生成更真实的人工房间脉冲响应（RIR）滤波器，而且还需要以快速的方式生成它们以加速训练过程。现有的RIR仿真工具已被证明在广泛的语音处理任务和神经网络架构中是有效的，但由于它们的计算复杂性或所生成的RIR滤波器的质量，它们在实时仿真流水线中的使用仍然存在问题。在本文中，我们提出了FRAM-RIR，广泛使用的图像源方法（ISM）的快速随机逼近方法，有效地产生现实的多通道RIR滤波器。FRAM-RIR通过基于几个启发式假设对每个虚拟声源的反射的位置和数量进行随机采样来绕过基于ISM的算法中的声音传播路径的显式计算，同时仍然保持所有声源的准确到达方向（DOA）信息。oracle波束图和方向特征的可视化显示FRAM-RIR可以生成比现有的广泛使用的基于ISM的工具更真实的RIR滤波器。和实验结果的多通道噪声语音分离和去混响任务与广泛的神经网络架构表明，模型训练与FRAM-RIR还可以在真实RIR上实现与其他RIR仿真工具相当或更好的性能，并且训练过程显著加快。发布了FRAM-RIR的Python实现。摘要：Modern neural-network-based speech processing systems are typically required to be robust against reverberation, and the training of such systems thus needs a large amount of reverberant data. During the training of the systems, on-the-fly simulation pipeline is nowadays preferred as it allows the model to train on infinite number of data samples without pre-generating and saving them on harddisk. An RIR simulation method thus needs to not only generate more realistic artificial room impulse response (RIR) filters, but also generate them in a fast way to accelerate the training process. Existing RIR simulation tools have proven effective in a wide range of speech processing tasks and neural network architectures, but their usage in on-the-fly simulation pipeline remains questionable due to their computational complexity or the quality of the generated RIR filters. In this paper, we propose FRAM-RIR, a fast random approximation method of the widely-used image-source method (ISM), to efficiently generate realistic multi-channel RIR filters. FRAM-RIR bypasses the explicit calculation of sound propagation paths in ISM-based algorithms by randomly sampling the location and number of reflections of each virtual sound source based on several heuristic assumptions, while still maintains accurate direction-of-arrival (DOA) information of all sound sources. Visualization of oracle beampatterns and directional features shows that FRAM-RIR can generate more realistic RIR filters than existing widely-used ISM-based tools, and experiment results on multi-channel noisy speech separation and dereverberation tasks with a wide range of neural network architectures show that models trained with FRAM-RIR can also achieve on par or better performance on real RIRs compared to other RIR simulation tools with a significantly accelerated training procedure. A Python implementation of FRAM-RIR is released.

【5】 Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition标题：基于回归的情感识别中视听融合的递归联合注意链接：http://arxiv.org/pdf/2304.07958v1

作者：R Gnana Praveen,Eric Granger,Patrick Cardinal机构：Laboratoire d’imagerie, de vision et d’intelligence artificielle (LIVIA), ´Ecole de technologie sup´erieure, Montreal, Canada摘要：在基于视频的情感识别（ER）中，重要的是有效地利用音频（A）和视觉（V）模态之间的互补关系，同时保留个体模态的模态内特征。本文提出了一种递归联合注意模型以及长短期记忆（LSTM）模块，用于在基于回归的ER中融合语音和面部表情。具体而言，我们研究了利用A和V模态的互补性的可能性，使用联合交叉注意力模型，以递归的方式与LSTM捕获模态内的时间依赖性，在相同的方式，以及之间的A-V特征表示。通过将LSTM与递归联合交叉关注相结合，我们的模型可以有效地利用模态内和模态间的关系来融合A和V模态。在具有挑战性的Affwild 2和Fatigue（私有）数据集上进行的大量实验的结果表明，所提出的A-V融合模型可以显着优于最先进的方法。摘要：In video-based emotion recognition (ER), it is important to effectively leverage the complementary relationship among audio (A) and visual (V) modalities, while retaining the intra-modal characteristics of individual modalities. In this paper, a recursive joint attention model is proposed along with long short-term memory (LSTM) modules for the fusion of vocal and facial expressions in regression-based ER. Specifically, we investigated the possibility of exploiting the complementary nature of A and V modalities using a joint cross-attention model in a recursive fashion with LSTMs to capture the intra-modal temporal dependencies within the same modalities as well as among the A-V feature representations. By integrating LSTMs with recursive joint cross-attention, our model can efficiently leverage both intra- and inter-modal relationships for the fusion of A and V modalities. The results of extensive experiments performed on the challenging Affwild2 and Fatigue (private) datasets indicate that the proposed A-V fusion model can significantly outperform state-of-art-methods.

【6】 How does ChatGPT rate sound semantics?标题：ChatGPT Rate听起来语义如何？链接：http://arxiv.org/pdf/2304.07830v1

作者：Kai Siedenburg,Charalampos Saitis机构：Dept. of Medical Physics and Acoustics, University of Oldenburg, Centre for Digital Music, Queen Mary University of London, ∗equal contribution备注：12 pages, 3 figures摘要：声音的语义维度在理解听觉感官体验的本质以及感知、语言和意义之间更广泛的关系方面一直扮演着核心角色。因此，考虑到最近大型语言模型（LLM）的激增，我们在这里询问这些模型是否表现出与人类相似的感知语义组织。具体来说，我们促使ChatGPT（一个基于最先进的LLM的聊天机器人）在一组20个语义尺度上对乐器声音进行评级。我们在单独的聊天中引发了多个响应，类似于有多个人类评分员。ChatGPT生成的语义配置文件仅与人类评级部分相关，但在音乐声音的众所周知的心理物理维度（如亮度（亮-暗）和音高高度（深-高））上显示出强大的一致性。探索性因素分析表明，相同的维度，但不同的空间配置的一个潜在的因素空间之间的聊天机器人和人类的评级。出乎意料的是，聊天机器人显示出的内部变异程度与人类评级的程度相当。我们的工作突出了LLM捕捉人类感官体验的突出维度的潜力。摘要：Semantic dimensions of sound have been playing a central role in understanding the nature of auditory sensory experience as well as the broader relation between perception, language, and meaning. Accordingly, and given the recent proliferation of large language models (LLMs), here we asked whether such models exhibit an organisation of perceptual semantics similar to those observed in humans. Specifically, we prompted ChatGPT, a chatbot based on a state-of-the-art LLM, to rate musical instrument sounds on a set of 20 semantic scales. We elicited multiple responses in separate chats, analogous to having multiple human raters. ChatGPT generated semantic profiles that only partially correlated with human ratings, yet showed robust agreement along well-known psychophysical dimensions of musical sounds such as brightness (bright-dark) and pitch height (deep-high). Exploratory factor analysis suggested the same dimensionality but different spatial configuration of a latent factor space between the chatbot and human ratings. Unexpectedly, the chatbot showed degrees of internal variability that were comparable in magnitude to that of human ratings. Our work highlights the potential of LLMs to capture salient dimensions of human sensory experience.

【7】 Acoustic Beamforming for Object-relative Distance Estimation and Control in Unmanned Air Vehicles using Propulsion System Noise标题：基于推进系统噪声的无人机目标相对距离估计与控制声学波束形成链接：http://arxiv.org/pdf/2304.07596v1

作者：Alisha Sharma,Jason Geder,Joseph Lingevitch,Theodore Martin,Daniel Lofaro,Donald Sofge机构：USA 2University of Maryland备注：7 pages, 12 figures摘要：无人飞行器的推进系统经常产生显著的噪声。使用这种宽带信号作为辅助传感系统的“声照明”可以使车辆以最小的成本更加鲁棒。我们提出了一种基于声学波束形成的算法，估计对象的相对距离与一个小的两个麦克风阵列使用所产生的推进系统的车辆噪声。我们证明了这种方法在几个闭环距离反馈控制测试与安装在嘈杂的环境中的四旋翼飞行器，并显示准确的对象相对距离估计超过2倍，进一步比基线通道为基础的方法。我们的结论是，这种方法是强大的几个实际的车辆和噪声的情况下，并显示在更复杂的操作环境中使用的承诺。摘要：Unmanned air vehicles often produce significant noise from their propulsion systems. Using this broadband signal as "acoustic illumination" for an auxiliary sensing system could make vehicles more robust at a minimal cost. We present an acoustic beamforming-based algorithm that estimates object-relative distance with a small two-microphone array using the generated propulsion system noise of a vehicle. We demonstrate this approach in several closed-loop distance feedback control tests with a mounted quad-rotor vehicle in a noisy environment and show accurate object-relative distance estimates more than 2x further than the baseline channel-based approach. We conclude that this approach is robust to several practical vehicle and noise situations and shows promise for use in more complex operating environments.

【8】 Self-supervised Auxiliary Loss for Metric Learning in Music Similarity-based Retrieval and Auto-tagging标题：音乐相似性检索和自动标注中度量学习的自监督辅助损失链接：http://arxiv.org/pdf/2304.07449v1

作者：Taketo Akama,Hiroaki Kitano,Katsuhiro Takematsu,Yasushi Miyajima,Natalia Polouliakh机构：Sony Computer Science Laboratories, Inc, Tokyo, Japan, Koozyt, Inc, Tokyo, Japan备注：11 pages摘要：在音乐信息检索领域，基于相似性的检索和自动标注是必不可少的组成部分。鉴于人类监督信号的局限性和不可扩展性，模型从替代来源学习以提高其性能变得至关重要。完全依赖于从音乐音频数据导出的学习信号的自监督学习已经在自动标记的上下文中证明了其功效。在本研究中，我们提出了一个基于自监督学习方法的模型，通过引入我们的带有自监督辅助损失的度量学习方法来解决基于相似性的检索挑战。此外，与传统的自监督学习方法不同，我们发现了用自监督和监督信号同时训练模型的优点，而不会冻结预训练的模型。我们还发现，避免在微调阶段采用增强产生更好的结果。我们的实验结果证实了所提出的方法在两个不同的场景中提高了检索和标记性能指标：一种是人类注释的标签对于所有音乐曲目一致可用，另一种是这样的标签仅对于曲目的子集可访问。摘要：In the realm of music information retrieval, similarity-based retrieval and auto-tagging serve as essential components. Given the limitations and non-scalability of human supervision signals, it becomes crucial for models to learn from alternative sources to enhance their performance. Self-supervised learning, which exclusively relies on learning signals derived from music audio data, has demonstrated its efficacy in the context of auto-tagging. In this study, we propose a model that builds on the self-supervised learning approach to address the similarity-based retrieval challenge by introducing our method of metric learning with a self-supervised auxiliary loss. Furthermore, diverging from conventional self-supervised learning methodologies, we discovered the advantages of concurrently training the model with both self-supervision and supervision signals, without freezing pre-trained models. We also found that refraining from employing augmentation during the fine-tuning phase yields better results. Our experimental results confirm that the proposed methodology enhances retrieval and tagging performance metrics in two distinct scenarios: one where human-annotated tags are consistently available for all music tracks, and another where such tags are accessible only for a subset of tracks.

【9】 Novel features for the detection of bearing faults in railway vehicles标题：铁道车辆轴承故障检测的新特征链接：http://arxiv.org/pdf/2304.08249v1

作者：Matthias Kreuzer,Alexander Schmidt,Walter Kellermann机构：Multimedia Communications and Signal Processing, Friedrich-Alexander University Erlangen-Nürnberg, Cauerstr. , Erlangen, Germany备注：Accepted at Inter-Noise 2021摘要：{In摘要本文提出了从振动信号中检测轴承故障这一具有挑战性的问题。为此，过去已经提出了若干时域和频域特征。然而，这些功能通常是从相对简单的情况下，如果考虑更现实的情况下，可以观察到一个显着的性能损失的数据进行评估。为了克服这一点，我们引入梅尔频率倒谱系数（MFCC）和特征提取的幅度调制谱图（AMS）作为特征的轴承故障的检测。AMS和MFCC最初都是在音频信号处理的上下文中引入的，但是已经证明通过使用这些特征可以获得显著改善的分类性能。此外，为了解决在轴承故障检测的背景下的特征数据不平衡问题，即，通常更多的数据从健康的轴承比从损坏的轴承是可用的，我们建议训练一个类 ac{SVM}的数据从健康的轴承。轴承故障，然后分类的异常值的检测。我们的方法进行评估，在一个非常具有挑战性的情况下，包括一个国家的最先进的通勤铁路发动机，由工业电源转换器提供，并耦合到负载机的数据测量。摘要：{In this paper, we address the challenging problem of detecting bearing faults from vibration signals. For this, several time- and frequency-domain features have been proposed already in the past. However, these features are usually evaluated on data originating from relatively simple scenarios and a significant performance loss can be observed if more realistic scenarios are considered. To overcome this, we introduce Mel-Frequency Cepstral Coefficients (MFCCs) and features extracted from the Amplitude Modulation Spectrogram (AMS) as features for the detection of bearing faults. Both AMS and MFCCs were originally introduced in the context of audio signal processing but it is demonstrated that a significantly improved classification performance can be obtained by using these features. Furthermore, to tackle the characteristic data imbalance problem in the context of bearing fault detection, i.e., typically much more data from healthy bearings than from damaged bearings is available, we propose to train a One-class ac{SVM} with data from healthy bearings only. Bearing faults are then classified by the detection of outliers. Our approach is evaluated with data measured in a highly challenging scenario comprising a state-of-the-art commuter railway engine which is supplied by an industrial power converter and coupled to a load machine.

【10】 Airborne-Sound Analysis for the Detection of Bearing Faults in Railway Vehicles with Real-World Data标题：铁道车辆轴承故障检测的机载声学分析链接：http://arxiv.org/pdf/2304.07307v1

作者：Matthias Kreuzer,David Schmidt,Simon Wokusch,Walter Kellermann机构：Multimedia Communications and Signal Processing, FAU Erlangen-N¨urnberg, Germany, Siemens Mobility GmbH, N¨urnberg, Germany备注：Submitted to ICPHM 2023摘要：在本文中，我们解决了具有挑战性的问题，检测轴承故障的铁路车辆通过分析在正常运行过程中记录的声学信号。为此，我们引入梅尔频率倒谱系数（MFCC）的功能，形成一个简单的多层感知器分类器的输入。所提出的方法与现实世界的数据，获得了国家的最先进的通勤铁路车辆在测量活动进行评估。实验表明，与所选择的MFCC功能轴承故障可以可靠地检测，即使是轴承损坏，不包括在训练。摘要：In this paper, we address the challenging problem of detecting bearing faults in railway vehicles by analyzing acoustic signals recorded during regular operation. For this, we introduce Mel Frequency Cepstral Coefficients (MFCCs) as features, which form the input to a simple Multi-Layer Perceptron classifier. The proposed method is evaluated with real-world data that was obtained for state-of-the-art commuter railway vehicles in a measurement campaign. The experiments show that with the chosen MFCC features bearing faults can be reliably detected even for bearing damages that were not included in training.

【11】 1-D Residual Convolutional Neural Network coupled with Data Augmentation and Regularization Techniques for the ICPHM 2023 Data Challenge标题：结合数据增强和正则化技术的一维剩余卷积神经网络用于ICPHM 2023数据挑战赛链接：http://arxiv.org/pdf/2304.07305v1

作者：Matthias Kreuzer,Walter Kellermann机构：Multimedia Communications and Signal Processing, FAU Erlangen-N¨urnberg, Germany备注：Submitted to IEEE Conference on Prognostics and Health Management (ICPHM) 2023摘要：在这篇文章中，我们介绍了我们对ICPHM 2023年使用振动分析进行工业系统健康监测的数据挑战的贡献。对于分类太阳齿轮故障的任务，在齿轮箱中，我们提出了一个残余卷积神经网络，操作的原始三通道时域振动信号。结合数据增强和正则化技术，所提出的模型在具有真实世界数据的多类分类场景中产生非常好的结果，尽管其尺寸相对较小，即，少于30，000个可训练参数。即使当呈现从多个操作条件获得的数据时，网络仍然能够准确地预测被检查的齿轮箱的条件。摘要：In this article, we present our contribution to the ICPHM 2023 Data Challenge on Industrial Systems' Health Monitoring using Vibration Analysis. For the task of classifying sun gear faults in a gearbox, we propose a residual Convolutional Neural Network that operates on raw three-channel time-domain vibration signals. In conjunction with data augmentation and regularization techniques, the proposed model yields very good results in a multi-class classification scenario with real-world data despite its relatively small size, i.e., with less than 30,000 trainable parameters. Even when presented with data obtained from multiple operating conditions, the network is still capable to accurately predict the condition of the gearbox under inspection.
eess.AS音频处理

【1】 Novel features for the detection of bearing faults in railway vehicles标题：铁道车辆轴承故障检测的新特征链接：http://arxiv.org/pdf/2304.08249v1

*与cs.SD语音【9】为同一篇

作者：Matthias Kreuzer,Alexander Schmidt,Walter Kellermann机构：Multimedia Communications and Signal Processing, Friedrich-Alexander University Erlangen-Nürnberg, Cauerstr. , Erlangen, Germany备注：Accepted at Inter-Noise 2021摘要：{In摘要本文提出了从振动信号中检测轴承故障这一具有挑战性的问题。为此，过去已经提出了若干时域和频域特征。然而，这些功能通常是从相对简单的情况下，如果考虑更现实的情况下，可以观察到一个显着的性能损失的数据进行评估。为了克服这一点，我们引入梅尔频率倒谱系数（MFCC）和特征提取的幅度调制谱图（AMS）作为特征的轴承故障的检测。AMS和MFCC最初都是在音频信号处理的上下文中引入的，但是已经证明通过使用这些特征可以获得显著改善的分类性能。此外，为了解决在轴承故障检测的背景下的特征数据不平衡问题，即，通常更多的数据从健康的轴承比从损坏的轴承是可用的，我们建议训练一个类 ac{SVM}的数据从健康的轴承。轴承故障，然后分类的异常值的检测。我们的方法进行评估，在一个非常具有挑战性的情况下，包括一个国家的最先进的通勤铁路发动机，由工业电源转换器提供，并耦合到负载机的数据测量。摘要：{In this paper, we address the challenging problem of detecting bearing faults from vibration signals. For this, several time- and frequency-domain features have been proposed already in the past. However, these features are usually evaluated on data originating from relatively simple scenarios and a significant performance loss can be observed if more realistic scenarios are considered. To overcome this, we introduce Mel-Frequency Cepstral Coefficients (MFCCs) and features extracted from the Amplitude Modulation Spectrogram (AMS) as features for the detection of bearing faults. Both AMS and MFCCs were originally introduced in the context of audio signal processing but it is demonstrated that a significantly improved classification performance can be obtained by using these features. Furthermore, to tackle the characteristic data imbalance problem in the context of bearing fault detection, i.e., typically much more data from healthy bearings than from damaged bearings is available, we propose to train a One-class ac{SVM} with data from healthy bearings only. Bearing faults are then classified by the detection of outliers. Our approach is evaluated with data measured in a highly challenging scenario comprising a state-of-the-art commuter railway engine which is supplied by an industrial power converter and coupled to a load machine.

【2】 Neural TTS in French: Comparing Graphemic and Phonetic Inputs Using the SynPaFlex-Corpus and Tacotron2标题：法语中的神经TTS：使用SynPaFlex语料库和Taco tron2比较字形和语音输入链接：http://arxiv.org/pdf/2304.08209v1

作者：Samuel Delalez,Ludi Akue机构：Lunii备注：6 pages, 3 figures摘要：SynPaFlex-Corpus是一个面向TTS的公开数据集，它提供了由JTrans转录器自动生成的语音转录，音素错误率（PER）为6.1%。在本文中，我们分析了两个单扬声器Tacotron 2模型训练的字形和语音输入，提供的SynPaFlex语料库。通过三次主观听力测试，比较他们的发音准确度、音质和自然度。我们的结果显示，音素模型的发音准确度和韵律自然度明显更好，但在感知音质方面没有显著差异。他们表明，PER为6.1%，足以提高发音控制，使用语音成绩单，而不是字形与83小时的记录法语读讲话。他们认为SynPaFlex语料库适合于在单语者微调方法中预训练模型。摘要：The SynPaFlex-Corpus is a publicly available TTS-oriented dataset, which provides phonetic transcriptions automatically produced by the JTrans transcriber, with a Phoneme Error Rate (PER) of 6.1%. In this paper, we analyze two mono-speaker Tacotron2 models trained on graphemic and phonetic inputs, provided by the SynPaFlex-Corpus. Through three subjective listening tests, we compare their pronunciation accuracy, sound quality and naturalness. Our results show significantly better pronunciation accuracy and prosody naturalness for the phoneme-based model, but no significant difference in terms of perceived sound quality. They demonstrate that a PER of 6.1% is sufficient to enhance pronunciation control by using phonetic transcripts instead of graphemes with 83 hours of recorded French read speech. They suggest that the SynPaFlex-Corpus is suitable for pre-training a model in mono-speaker fine-tuning approaches.

【3】 Audio coding with unified noise shaping and phase contrast control标题：具有统一噪声整形和相位对比度控制的音频编码链接：http://arxiv.org/pdf/2304.08076v1

作者：Byeongho Jo,Seungkwon Beack,Taejin Lee机构：Electronics and Telecommunications Research Institute, Republic of Korea备注：Submitted and accepted in ICASSP (International Conference on Acoustics, Speech, and Signal Processing) 2023摘要：None摘要：Over the past decade, audio coding technology has seen standardization and the development of many frameworks incorporated with linear predictive coding (LPC). As LPC reduces information in the frequency domain, LP-based frequency-domain noise-shaping (FDNS) was previously proposed. To code transient signals effectively, FDNS with temporal noise shaping (TNS) has emerged. However, these mainly operated in the modified discrete cosine transform domain, which essentially accompanies time domain aliasing. In this paper, a unified noise-shaping (UNS) framework including FDNS and complex LPC-based TNS (CTNS) in the DFT domain is proposed to overcome the aliasing issues. Additionally, a modified polar quantizer with phase contrast control is proposed, which saves phase bits depending on the frequency envelope information. The core coding feasibility at low bit rates is verified through various objective metrics and subjective listening evaluations.

【4】 A Virtual Simulation-Pilot Agent for Training of Air Traffic Controllers标题：一种用于空中交通管制员训练的虚拟模拟飞行员代理链接：http://arxiv.org/pdf/2304.07842v1

作者：Juan Zuluaga-Gomez,Amrutha Prasad,Iuliia Nigmatulina,Petr Motlicek,Matthias Kleinert机构：Idiap Research Institute, Martigny, Switzerland, Ecole Polytechnique Fédérale de Lausanne, Switzerland, University of Zurich, Switzerland, Brno University of Technology, Brno, Czechia备注：Under review摘要：在本文中，我们提出了一种新的虚拟仿真飞行员引擎，通过集成不同的国家的最先进的人工智能（AI）为基础的工具，加快空中交通管制员（ATCo）的培训。虚拟仿真驾驶引擎接收ATCo学员的语音通信，并执行自动语音识别和理解。因此，它不仅仅是转录通信，还可以理解其含义。输出随后被发送到响应生成器系统，该系统类似于飞行员给ATCo受训者的口头回读。整个管道由以下子模块组成：（i）自动语音识别（ASR）系统，其将音频转换成单词序列;（ii）理解所转录的语音通信的高级空中交通管制（ATC）相关实体解析器;以及（iii）文本到语音子模块，其基于对话的情形生成类似于导频的口语话语。我们的系统采用了最先进的基于AI的工具，如Wav 2 Vec 2.0，Conformer，BERT和Tacotron模型。据我们所知，这是首个完全基于开源ATC资源和AI工具的作品。此外，我们已经开发了一个强大的模块化系统，可选的子模块，可以通过结合实时监控数据，与演习（如部门或跑道）相关的元数据，甚至引入故意回读错误来训练ATCo学员识别它们，以提高系统的性能。我们的ASR系统可以在高质量和低质量ATC音频上达到低至5.5%和15.9%的字错误率（WER）。我们还表明，将监视数据添加到ASR中可以产生超过96%的呼号检测精度。摘要：In this paper we propose a novel virtual simulation-pilot engine for speeding up air traffic controller (ATCo) training by integrating different state-of-the-art artificial intelligence (AI) based tools. The virtual simulation-pilot engine receives spoken communications from ATCo trainees, and it performs automatic speech recognition and understanding. Thus, it goes beyond only transcribing the communication and can also understand its meaning. The output is subsequently sent to a response generator system, which resembles the spoken read back that pilots give to the ATCo trainees. The overall pipeline is composed of the following submodules: (i) automatic speech recognition (ASR) system that transforms audio into a sequence of words; (ii) high-level air traffic control (ATC) related entity parser that understands the transcribed voice communication; and (iii) a text-to-speech submodule that generates a spoken utterance that resembles a pilot based on the situation of the dialogue. Our system employs state-of-the-art AI-based tools such as Wav2Vec 2.0, Conformer, BERT and Tacotron models. To the best of our knowledge, this is the first work fully based on open-source ATC resources and AI tools. In addition, we have developed a robust and modular system with optional submodules that can enhance the system's performance by incorporating real-time surveillance data, metadata related to exercises (such as sectors or runways), or even introducing a deliberate read-back error to train ATCo trainees to identify them. Our ASR system can reach as low as 5.5% and 15.9% word error rates (WER) on high and low-quality ATC audio. We also demonstrate that adding surveillance data into the ASR can yield callsign detection accuracy of more than 96%.

【5】 Soft Label Coding for End-to-end Sound Source Localization With Ad-hoc Microphone Arrays标题：自组织麦克风阵列端到端声源定位的软标签编码链接：http://arxiv.org/pdf/2304.07512v1

作者：Linfeng Feng,Yijun Gong,Xiao-Lei Zhang机构：School of Marine Science and Technology, Northwestern Polytechnical University, China, Research & Development Institute of Northwestern Polytechnical University in Shenzhen, China备注：4pages, 2figures, conference摘要：None摘要：Recently, an end-to-end two-dimensional sound source localization algorithm with ad-hoc microphone arrays formulates the sound source localization problem as a classification problem. The algorithm divides the target indoor space into a set of local areas, and predicts the local area where the speaker locates. However, the local areas are encoded by one-hot code, which may lose the connections between the local areas due to quantization errors. In this paper, we propose a new soft label coding method, named label smoothing, for the classification-based two-dimensional sound source location with ad-hoc microphone arrays. The core idea is to take the geometric connection between the classes into the label coding process.The first one is named static soft label coding (SSLC), which modifies the one-hot codes into soft codes based on the distances between the local areas. Because SSLC is handcrafted which may not be optimal, the second one, named dynamic soft label coding (DSLC), further rectifies SSLC, by learning the soft codes according to the statistics of the predictions produced by the classification-based localization model in the training stage. Experimental results show that the proposed methods can effectively improve the localization accuracy.

【6】 Airborne-Sound Analysis for the Detection of Bearing Faults in Railway Vehicles with Real-World Data标题：铁道车辆轴承故障检测的机载声学分析链接：http://arxiv.org/pdf/2304.07307v1

*与cs.SD语音【10】为同一篇

作者：Matthias Kreuzer,David Schmidt,Simon Wokusch,Walter Kellermann机构：Multimedia Communications and Signal Processing, FAU Erlangen-N¨urnberg, Germany, Siemens Mobility GmbH, N¨urnberg, Germany备注：Submitted to ICPHM 2023摘要：在本文中，我们解决了具有挑战性的问题，检测轴承故障的铁路车辆通过分析在正常运行过程中记录的声学信号。为此，我们引入梅尔频率倒谱系数（MFCC）的功能，形成一个简单的多层感知器分类器的输入。所提出的方法与现实世界的数据，获得了国家的最先进的通勤铁路车辆在测量活动进行评估。实验表明，与所选择的MFCC功能轴承故障可以可靠地检测，即使是轴承损坏，不包括在训练。摘要：In this paper, we address the challenging problem of detecting bearing faults in railway vehicles by analyzing acoustic signals recorded during regular operation. For this, we introduce Mel Frequency Cepstral Coefficients (MFCCs) as features, which form the input to a simple Multi-Layer Perceptron classifier. The proposed method is evaluated with real-world data that was obtained for state-of-the-art commuter railway vehicles in a measurement campaign. The experiments show that with the chosen MFCC features bearing faults can be reliably detected even for bearing damages that were not included in training.

【7】 1-D Residual Convolutional Neural Network coupled with Data Augmentation and Regularization Techniques for the ICPHM 2023 Data Challenge标题：结合数据增强和正则化技术的一维剩余卷积神经网络用于ICPHM 2023数据挑战赛链接：http://arxiv.org/pdf/2304.07305v1

*与cs.SD语音【11】为同一篇

作者：Matthias Kreuzer,Walter Kellermann机构：Multimedia Communications and Signal Processing, FAU Erlangen-N¨urnberg, Germany备注：Submitted to IEEE Conference on Prognostics and Health Management (ICPHM) 2023摘要：在这篇文章中，我们介绍了我们对ICPHM 2023年使用振动分析进行工业系统健康监测的数据挑战的贡献。对于分类太阳齿轮故障的任务，在齿轮箱中，我们提出了一个残余卷积神经网络，操作的原始三通道时域振动信号。结合数据增强和正则化技术，所提出的模型在具有真实世界数据的多类分类场景中产生非常好的结果，尽管其尺寸相对较小，即，少于30，000个可训练参数。即使当呈现从多个操作条件获得的数据时，网络仍然能够准确地预测被检查的齿轮箱的条件。摘要：In this article, we present our contribution to the ICPHM 2023 Data Challenge on Industrial Systems' Health Monitoring using Vibration Analysis. For the task of classifying sun gear faults in a gearbox, we propose a residual Convolutional Neural Network that operates on raw three-channel time-domain vibration signals. In conjunction with data augmentation and regularization techniques, the proposed model yields very good results in a multi-class classification scenario with real-world data despite its relatively small size, i.e., with less than 30,000 trainable parameters. Even when presented with data obtained from multiple operating conditions, the network is still capable to accurately predict the condition of the gearbox under inspection.

【8】 Prak: An automatic phonetic alignment tool for Czech标题：PRAK：捷克语语音自动对齐工具链接：http://arxiv.org/pdf/2304.08431v1

*与cs.SD语音【2】为同一篇

作者：Václav Hanžl,Adléta Hanžlová机构：Freelance researcher,Institute of Phonetics, Charles University in Prague备注：Submitted for ICPhS 2023摘要：将语音标记到音素的身份和时间边界是语音研究的劳动密集型部分。为了简化这项工作，我们创建了一个免费的开源工具，从捷克语文本中生成电话序列，并将其与音频进行时间对齐。低架构复杂度使得语音学专业的学生易于使用该设计。使用PyTorch在小型CommonVoice数据上训练具有56k权重的声学模型ReLU NN。比对和变体选择解码器在Python中使用矩阵库实现。捷克语发音生成器由简单的基于规则的块组成，在可能的情况下捕获语言的逻辑，允许修改转录方法的细节。与迄今为止使用的工具相比，数据准备效率提高，该工具可在Mac，Linux和Windows上以Praat GUI或命令行使用，实现大多数正确的发音变体选择，包括声门停止检测，算法捕获大部分捷克语同化逻辑，具有教学性和实用性。摘要：Labeling speech down to the identity and time boundaries of phones is a labor-intensive part of phonetic research. To simplify this work, we created a free open-source tool generating phone sequences from Czech text and time-aligning them with audio. Low architecture complexity makes the design approachable for students of phonetics. Acoustic model ReLU NN with 56k weights was trained using PyTorch on small CommonVoice data. Alignment and variant selection decoder is implemented in Python with matrix library. A Czech pronunciation generator is composed of simple rule-based blocks capturing the logic of the language where possible, allowing modification of transcription approach details. Compared to tools used until now, data preparation efficiency improved, the tool is usable on Mac, Linux and Windows in Praat GUI or command line, achieves mostly correct pronunciation variant choice including glottal stop detection, algorithmically captures most of Czech assimilation logic and is both didactic and practical.

【9】 VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset标题：VALOR：视觉-听觉-语言全方位感知预训练模型和数据集链接：http://arxiv.org/pdf/2304.08345v1

作者：Sihan Chen,Xingjian He,Longteng Guo,Xinxin Zhu,Weining Wang,Jinhui Tang,Jing Liu机构：Institute of Automation备注：Preprint version wo audio files embeded in PDF. Audio embeded version can be found on project page or github摘要：None摘要：In this paper, we propose a Vision-Audio-Language Omni-peRception pretraining model (VALOR) for multi-modal understanding and generation. Different from widely-studied vision-language pretraining models, VALOR jointly models relationships of vision, audio and language in an end-to-end manner. It contains three separate encoders for single modality representations, and a decoder for multimodal conditional text generation. We design two pretext tasks to pretrain VALOR model, including Multimodal Grouping Alignment (MGA) and Multimodal Grouping Captioning (MGC). MGA projects vision, language and audio to the same common space, building vision-language, audio-language and audiovisual-language alignment simultaneously. MGC learns how to generate text tokens in conditions of vision, audio or their both. To promote vision-audio-language pretraining research, we construct a large-scale high-quality tri-modality dataset named VALOR-1M, which contains 1M audiable videos with human annotated audiovisual captions. Extensive experiments show that VALOR can learn strong multimodal correlations and be generalized to various downstream tasks (e.g., retrieval, captioning and question answering), with different input modalities (e.g., vision-language, audio-language and audiovisual-language). VALOR achieves new state-of-the-art performances on series of public cross-modality benchmarks. Code and data are available at project page https: casia-iva-group.github.io projects VALOR.

【10】 Fast Random Approximation of Multi-channel Room Impulse Response标题：多通道房间冲击响应的快速随机逼近链接：http://arxiv.org/pdf/2304.08052v1

*与cs.SD语音【4】为同一篇

作者：Yi Luo,Rongzhi Gu摘要：现代的基于神经网络的语音处理系统通常需要对混响具有鲁棒性，并且这样的系统的训练因此需要大量的混响数据。在系统的训练过程中，实时仿真流水线现在是优选的，因为它允许模型在无限数量的数据样本上进行训练，而无需预先生成并将其保存在硬盘上。因此，RIR仿真方法不仅需要生成更真实的人工房间脉冲响应（RIR）滤波器，而且还需要以快速的方式生成它们以加速训练过程。现有的RIR仿真工具已被证明在广泛的语音处理任务和神经网络架构中是有效的，但由于它们的计算复杂性或所生成的RIR滤波器的质量，它们在实时仿真流水线中的使用仍然存在问题。在本文中，我们提出了FRAM-RIR，广泛使用的图像源方法（ISM）的快速随机逼近方法，有效地产生现实的多通道RIR滤波器。FRAM-RIR通过基于几个启发式假设对每个虚拟声源的反射的位置和数量进行随机采样来绕过基于ISM的算法中的声音传播路径的显式计算，同时仍然保持所有声源的准确到达方向（DOA）信息。oracle波束图和方向特征的可视化显示FRAM-RIR可以生成比现有的广泛使用的基于ISM的工具更真实的RIR滤波器。和实验结果的多通道噪声语音分离和去混响任务与广泛的神经网络架构表明，模型训练与FRAM-RIR还可以在真实RIR上实现与其他RIR仿真工具相当或更好的性能，并且训练过程显著加快。发布了FRAM-RIR的Python实现。摘要：Modern neural-network-based speech processing systems are typically required to be robust against reverberation, and the training of such systems thus needs a large amount of reverberant data. During the training of the systems, on-the-fly simulation pipeline is nowadays preferred as it allows the model to train on infinite number of data samples without pre-generating and saving them on harddisk. An RIR simulation method thus needs to not only generate more realistic artificial room impulse response (RIR) filters, but also generate them in a fast way to accelerate the training process. Existing RIR simulation tools have proven effective in a wide range of speech processing tasks and neural network architectures, but their usage in on-the-fly simulation pipeline remains questionable due to their computational complexity or the quality of the generated RIR filters. In this paper, we propose FRAM-RIR, a fast random approximation method of the widely-used image-source method (ISM), to efficiently generate realistic multi-channel RIR filters. FRAM-RIR bypasses the explicit calculation of sound propagation paths in ISM-based algorithms by randomly sampling the location and number of reflections of each virtual sound source based on several heuristic assumptions, while still maintains accurate direction-of-arrival (DOA) information of all sound sources. Visualization of oracle beampatterns and directional features shows that FRAM-RIR can generate more realistic RIR filters than existing widely-used ISM-based tools, and experiment results on multi-channel noisy speech separation and dereverberation tasks with a wide range of neural network architectures show that models trained with FRAM-RIR can also achieve on par or better performance on real RIRs compared to other RIR simulation tools with a significantly accelerated training procedure. A Python implementation of FRAM-RIR is released.

【11】 Recursive Joint Attention for Audio-Visual Fusion in Regression based Emotion Recognition标题：基于回归的情感识别中视听融合的递归联合注意链接：http://arxiv.org/pdf/2304.07958v1

*与cs.SD语音【5】为同一篇

作者：R Gnana Praveen,Eric Granger,Patrick Cardinal机构：Laboratoire d’imagerie, de vision et d’intelligence artificielle (LIVIA), ´Ecole de technologie sup´erieure, Montreal, Canada摘要：在基于视频的情感识别（ER）中，重要的是有效地利用音频（A）和视觉（V）模态之间的互补关系，同时保留个体模态的模态内特征。本文提出了一种递归联合注意模型以及长短期记忆（LSTM）模块，用于在基于回归的ER中融合语音和面部表情。具体而言，我们研究了利用A和V模态的互补性的可能性，使用联合交叉注意力模型，以递归的方式与LSTM捕获模态内的时间依赖性，在相同的方式，以及之间的A-V特征表示。通过将LSTM与递归联合交叉关注相结合，我们的模型可以有效地利用模态内和模态间的关系来融合A和V模态。在具有挑战性的Affwild 2和Fatigue（私有）数据集上进行的大量实验的结果表明，所提出的A-V融合模型可以显着优于最先进的方法。摘要：In video-based emotion recognition (ER), it is important to effectively leverage the complementary relationship among audio (A) and visual (V) modalities, while retaining the intra-modal characteristics of individual modalities. In this paper, a recursive joint attention model is proposed along with long short-term memory (LSTM) modules for the fusion of vocal and facial expressions in regression-based ER. Specifically, we investigated the possibility of exploiting the complementary nature of A and V modalities using a joint cross-attention model in a recursive fashion with LSTMs to capture the intra-modal temporal dependencies within the same modalities as well as among the A-V feature representations. By integrating LSTMs with recursive joint cross-attention, our model can efficiently leverage both intra- and inter-modal relationships for the fusion of A and V modalities. The results of extensive experiments performed on the challenging Affwild2 and Fatigue (private) datasets indicate that the proposed A-V fusion model can significantly outperform state-of-art-methods.

【12】 How does ChatGPT rate sound semantics?标题：ChatGPT Rate听起来语义如何？链接：http://arxiv.org/pdf/2304.07830v1

*与cs.SD语音【6】为同一篇

作者：Kai Siedenburg,Charalampos Saitis机构：Dept. of Medical Physics and Acoustics, University of Oldenburg, Centre for Digital Music, Queen Mary University of London, ∗equal contribution备注：12 pages, 3 figures摘要：声音的语义维度在理解听觉感官体验的本质以及感知、语言和意义之间更广泛的关系方面一直扮演着核心角色。因此，考虑到最近大型语言模型（LLM）的激增，我们在这里询问这些模型是否表现出与人类相似的感知语义组织。具体来说，我们促使ChatGPT（一个基于最先进的LLM的聊天机器人）在一组20个语义尺度上对乐器声音进行评级。我们在单独的聊天中引发了多个响应，类似于有多个人类评分员。ChatGPT生成的语义配置文件仅与人类评级部分相关，但在音乐声音的众所周知的心理物理维度（如亮度（亮-暗）和音高高度（深-高））上显示出强大的一致性。探索性因素分析表明，相同的维度，但不同的空间配置的一个潜在的因素空间之间的聊天机器人和人类的评级。出乎意料的是，聊天机器人显示出的内部变异程度与人类评级的程度相当。我们的工作突出了LLM捕捉人类感官体验的突出维度的潜力。摘要：Semantic dimensions of sound have been playing a central role in understanding the nature of auditory sensory experience as well as the broader relation between perception, language, and meaning. Accordingly, and given the recent proliferation of large language models (LLMs), here we asked whether such models exhibit an organisation of perceptual semantics similar to those observed in humans. Specifically, we prompted ChatGPT, a chatbot based on a state-of-the-art LLM, to rate musical instrument sounds on a set of 20 semantic scales. We elicited multiple responses in separate chats, analogous to having multiple human raters. ChatGPT generated semantic profiles that only partially correlated with human ratings, yet showed robust agreement along well-known psychophysical dimensions of musical sounds such as brightness (bright-dark) and pitch height (deep-high). Exploratory factor analysis suggested the same dimensionality but different spatial configuration of a latent factor space between the chatbot and human ratings. Unexpectedly, the chatbot showed degrees of internal variability that were comparable in magnitude to that of human ratings. Our work highlights the potential of LLMs to capture salient dimensions of human sensory experience.

【13】 A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition标题：一种基于CTC对齐的端到端自动语音识别非自回归转换器链接：http://arxiv.org/pdf/2304.07611v1

作者：Ruchao Fan,Wei Chu,Peng Chang,Abeer Alwan机构：Alwan are with the Department of Electrical and ComputerEngineering, University of California备注：Published in IEEE Transactions on Audio, Speech, and Language Processing摘要：近年来，端到端模型已被广泛应用于自动语音识别（ASR）系统中。其中最具代表性的两种方法是连接主义时间分类（CTC）和基于注意力的编码器-解码器（AED）模型。自回归Transformers（AED的变体）采用自回归机制来生成令牌，因此在推理期间相对较慢。在本文中，我们提出了一个全面的研究CTC对齐为基础的单步非自回归Transformer（CASS-NAT）的端到端ASR。在CASS-NAT中，自回归Transformer（AT）中的字嵌入被令牌级声学嵌入（TAE）所取代，令牌级声学嵌入（TAE）是从具有CTC对准所提供的声学边界信息的编码器输出中提取的。可以并行地获得TAE，从而导致输出令牌的并行生成。在训练期间，维特比对齐用于TAE生成，并且进一步探索多种训练策略以提高字错误率（WER）性能。在推理过程中，深入研究了一种基于误差的对齐采样方法，以减少训练和测试过程中的对齐失配。实验结果表明，CASS-NAT在各种ASR任务上具有接近AT的WER，同时提供~ 24倍的推理加速。在有和没有自监督学习的情况下，我们在几个数据集上实现了非自回归模型的最新结果。我们还分析了CASS-NAT解码器的行为，以解释为什么它可以执行类似于AT。我们发现，TAE有类似的功能，词嵌入的语法结构，这可能表明学习一些语义信息，从TAE没有语言模型的可能性。摘要：Recently, end-to-end models have been widely used in automatic speech recognition (ASR) systems. Two of the most representative approaches are connectionist temporal classification (CTC) and attention-based encoder-decoder (AED) models. Autoregressive transformers, variants of AED, adopt an autoregressive mechanism for token generation and thus are relatively slow during inference. In this paper, we present a comprehensive study of a CTC Alignment-based Single-Step Non-Autoregressive Transformer (CASS-NAT) for end-to-end ASR. In CASS-NAT, word embeddings in the autoregressive transformer (AT) are substituted with token-level acoustic embeddings (TAE) that are extracted from encoder outputs with the acoustical boundary information offered by the CTC alignment. TAE can be obtained in parallel, resulting in a parallel generation of output tokens. During training, Viterbi-alignment is used for TAE generation, and multiple training strategies are further explored to improve the word error rate (WER) performance. During inference, an error-based alignment sampling method is investigated in depth to reduce the alignment mismatch in the training and testing processes. Experimental results show that the CASS-NAT has a WER that is close to AT on various ASR tasks, while providing a ~24x inference speedup. With and without self-supervised learning, we achieve new state-of-the-art results for non-autoregressive models on several datasets. We also analyze the behavior of the CASS-NAT decoder to explain why it can perform similarly to AT. We find that TAEs have similar functionality to word embeddings for grammatical structures, which might indicate the possibility of learning some semantic information from TAEs without a language model.

【14】 Self-supervised Auxiliary Loss for Metric Learning in Music Similarity-based Retrieval and Auto-tagging标题：音乐相似性检索和自动标注中度量学习的自监督辅助损失链接：http://arxiv.org/pdf/2304.07449v1

*与cs.SD语音【8】为同一篇

作者：Taketo Akama,Hiroaki Kitano,Katsuhiro Takematsu,Yasushi Miyajima,Natalia Polouliakh机构：Sony Computer Science Laboratories, Inc, Tokyo, Japan, Koozyt, Inc, Tokyo, Japan备注：11 pages摘要：在音乐信息检索领域，基于相似性的检索和自动标注是必不可少的组成部分。鉴于人类监督信号的局限性和不可扩展性，模型从替代来源学习以提高其性能变得至关重要。完全依赖于从音乐音频数据导出的学习信号的自监督学习已经在自动标记的上下文中证明了其功效。在本研究中，我们提出了一个基于自监督学习方法的模型，通过引入我们的带有自监督辅助损失的度量学习方法来解决基于相似性的检索挑战。此外，与传统的自监督学习方法不同，我们发现了用自监督和监督信号同时训练模型的优点，而不会冻结预训练的模型。我们还发现，避免在微调阶段采用增强产生更好的结果。我们的实验结果证实了所提出的方法在两个不同的场景中提高了检索和标记性能指标：一种是人类注释的标签对于所有音乐曲目一致可用，另一种是这样的标签仅对于曲目的子集可访问。摘要：In the realm of music information retrieval, similarity-based retrieval and auto-tagging serve as essential components. Given the limitations and non-scalability of human supervision signals, it becomes crucial for models to learn from alternative sources to enhance their performance. Self-supervised learning, which exclusively relies on learning signals derived from music audio data, has demonstrated its efficacy in the context of auto-tagging. In this study, we propose a model that builds on the self-supervised learning approach to address the similarity-based retrieval challenge by introducing our method of metric learning with a self-supervised auxiliary loss. Furthermore, diverging from conventional self-supervised learning methodologies, we discovered the advantages of concurrently training the model with both self-supervision and supervision signals, without freezing pre-trained models. We also found that refraining from employing augmentation during the fine-tuning phase yields better results. Our experimental results confirm that the proposed methodology enhances retrieval and tagging performance metrics in two distinct scenarios: one where human-annotated tags are consistently available for all music tracks, and another where such tags are accessible only for a subset of tracks.

机器翻译，仅供参考