今日论文合集：cs.SD语音16篇，eess.AS音频处理19篇。
【1】 Introducing Semantics into Speech Encoders
标题：将语义学引入语音编码器
链接：https://arxiv.org/abs/2211.08402
作者：Derek Xu,Shuyan Dong,Changhan Wang,Suyoun Kim,Zhaojiang Lin,Akshat Shrivastava,Shang-Wen Li,Liang-Hsuan Tseng,Alexei Baevski,Guan-Ting Lin,Hung-yi Lee,Yizhou Sun,Wei Wang机构：University of California, Los Angeles,Meta AI,National Taiwan University摘要：最近的研究发现现有的自监督语音编码器主要包含声学信息而不是语义信息。结果，通过利用来自大语言模型（LLM）的丰富的语义表示，流水线化的有监督的自动语音识别（ASR）到大语言模型（LLM）系统在语义口语语言任务上实现了最先进的结果。这些系统是以标记音频转录为代价的，获得标记音频转录是昂贵且耗时的。我们提出了一种任务不可知的无监督方式，将来自LLM的语义信息合并到自监督语音编码器中，而无需标记音频转录。通过引入语义，在意图分类上，现有语音编码器的口语理解性能提高了10%以上，在命名实体解析和空位填充上有适度的提高，口语问答FF1得分提高了2%以上.我们的无监督方法实现了与在超过100小时的标记音频副本上训练的有监督方法类似的性能，证明了对现有语音编码器的无监督语义增强的可行性。摘要：Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding performance by over 10\% on intent classification, with modest gains in named entity resolution and slot filling, and spoken question answering FF1 score by over 2\%. Our unsupervised approach achieves similar performance as supervised methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.
【2】 Music Instrument Classification Reprogrammed
标题：乐器分类重新编程
链接：https://arxiv.org/abs/2211.08379
作者：Hsin-Hung Chen,Alexander Lerch机构：Music Informatics Group, Georgia Institute of Technology备注：Accepted at 29th International Conference on Multimedia Modeling (MMM23)摘要：乐器分类是音乐信息检索中的一个热门任务，其性能常常受到缺乏可用的标注数据的影响和限制。我们建议用“重新编程”来解决这个问题，这是一种利用预先训练的深度和复杂神经网络的技术，该网络最初通过修改和映射预先训练的模型的输入和输出来针对不同的任务。我们证明了重新编程可以有效地利用为不同任务学习的表征的能力，并且所得到的重新编程的系统可以在训练参数的一小部分上与现有技术的系统相当甚至优于现有技术的系统。因此，我们的结果表明，重新编程是一种有前途的技术，可能适用于其他因数据稀缺而受阻的任务。摘要：The performance of approaches to Music Instrument Classification, a popular task in Music Information Retrieval, is often impacted and limited by the lack of availability of annotated data for training. We propose to address this issue with "reprogramming," a technique that utilizes pre-trained deep and complex neural networks originally targeting a different task by modifying and mapping both the input and output of the pre-trained model. We demonstrate that reprogramming can effectively leverage the power of the representation learned for a different task and that the resulting reprogrammed system can perform on par or even outperform state-of-the-art systems at a fraction of training parameters. Our results, therefore, indicate that reprogramming is a promising technique potentially applicable to other tasks impeded by data scarcity.
【3】 FlowGrad: Using Motion for Visual Sound Source Localization
标题：FlowGrad：使用运动进行视觉声源定位
链接：https://arxiv.org/abs/2211.08367
作者：Rajsuryan Singh,Pablo Zinemanas,Xavier Serra,Juan Pablo Bello,Magdalena Fuentes机构：MTG, Universitat Pompeu Fabra, Barcelona, Spain, MARL, New York University, New York, USA, IDM, New York University, New York, USA备注：Submitted to ICASSP 2023摘要：视觉声源定位中的大多数最近工作依赖于以自监督方式学习的语义视听表示，并且通过设计排除了视频中存在的时间信息。虽然它被证明对广泛使用的基准数据集是有效的，但该方法在城市交通等具有挑战性的场景中却达不到要求。本文将时间背景引入到城市场景声源定位的最新方法中，使用光流作为编码运动信息的手段。通过分析这些方法的优缺点，有助于我们更好地理解视觉声源定位问题，并揭示视听场景理解面临的挑战。摘要：Most recent work in visual sound source localization relies on semantic audio-visual representations learned in a self-supervised manner, and by design excludes temporal information present in videos. While it proves to be effective for widely used benchmark datasets, the method falls short for challenging scenarios like urban traffic. This work introduces temporal context into the state-of-the-art methods for sound source localization in urban scenes using optical flow as a means to encode motion information. An analysis of the strengths and weaknesses of our methods helps us better understand the problem of visual sound source localization and sheds light on open challenges for audio-visual scene understanding.
【4】 Online Phase Reconstruction via DNN-based Phase Differences Estimation
标题：基于离散神经网络的相位差估计在线相位重构
链接：https://arxiv.org/abs/2211.08246
作者：Yoshiki Masuyama,Kohei Yatabe,Kento Nagatomo,Yasuhiro Oikawa机构：Tokyo University of Agriculture and Technology备注：Accepted to IEEE/ACM Trans. Audio, Speech, and Language Processing摘要：提出了一种基于因果深度神经网络的两阶段在线相位重构框架。相位重构是仅从相应幅度恢复短时傅里叶变换（STFT）系数的相位的任务。然而，相位对波形偏移很敏感，即使使用DNN也不容易从幅度估计。为了克服这个问题，我们提出使用DNN来估计相邻时间-频率仓之间的相位差。根据短时傅里叶变换相位和幅度偏导数之间的理论关系，证明了卷积神经网络适用于相位差估计。所估计的相位差用于通过以逐帧的方式求解加权最小二乘问题来重构相位。与现有的基于离散神经网络的相位重构方法相比，该方法具有因果关系，不需要迭代过程。实验结果表明，该方法优于现有的在线相位重构方法和基于DNN的相位重构方法。摘要：This paper presents a two-stage online phase reconstruction framework using causal deep neural networks (DNNs). Phase reconstruction is a task of recovering phase of the short-time Fourier transform (STFT) coefficients only from the corresponding magnitude. However, phase is sensitive to waveform shifts and not easy to estimate from the magnitude even with a DNN. To overcome this problem, we propose to use DNNs for estimating differences of phase between adjacent time-frequency bins. We show that convolutional neural networks are suitable for phase difference estimation, according to the theoretical relation between partial derivatives of STFT phase and magnitude. The estimated phase differences are used for reconstructing phase by solving a weighted least squares problem in a frame-by-frame manner. In contrast to existing DNN-based phase reconstruction methods, the proposed framework is causal and does not require any iterative procedure. The experiments showed that the proposed method outperforms existing online methods and a DNN-based method for phase reconstruction.
【5】 Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Search
标题：基于多门限机制和神经结构搜索的多语言语音情感识别
链接：https://arxiv.org/abs/2211.08237
作者：Zihan Wang,Qi Meng,HaiFeng Lan,XinRui Zhang,KeHao Guo,Akshat Gupta机构：Columbia University, JP Morgan AI Research, New York, USA摘要：语音情感识别（SER）将音频分类为情感类别，例如快乐、愤怒、恐惧、厌恶和中性。虽然语音情感识别（SER）是流行语言的常见应用，但是对于低资源语言，没有预先训练的语音到文本识别模型的语言。本文首先提出了一种基于语言的情感识别模型，该模型从多个预先训练好的语音模型中提取情感信息，然后设计了一种多领域模型，该模型可以同时对多种语言进行情感识别。我们的多领域模型采用多门机制为每种语言生成唯一的加权特征组合，并通过神经结构搜索模块为每种语言搜索特定的神经网络结构。此外，我们引入对比辅助损失，以建立更多可分离的音频数据表示。实验结果表明，该模型对德语和法语的准确率分别提高了3%和14.3%.摘要：Speech emotion recognition (SER) classifies audio into emotion categories such as Happy, Angry, Fear, Disgust and Neutral. While Speech Emotion Recognition (SER) is a common application for popular languages, it continues to be a problem for low-resourced languages, i.e., languages with no pretrained speech-to-text recognition models. This paper firstly proposes a language-specific model that extract emotional information from multiple pre-trained speech models, and then designs a multi-domain model that simultaneously performs SER for various languages. Our multidomain model employs a multi-gating mechanism to generate unique weighted feature combination for each language, and also searches for specific neural network structure for each language through a neural architecture search module. In addition, we introduce a contrastive auxiliary loss to build more separable representations for audio data. Our experiments show that our model raises the state-of-the-art accuracy by 3% for German and 14.3% for French.
【6】 Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition
标题：时态建模问题：一种新的语音情感识别时态情感建模方法
链接：https://arxiv.org/abs/2211.08233
作者：Jiaxin Ye,Xincheng Wen,Yujie Wei,Yong Xu,Kunhong Liu,Hongming Shan机构：Fudan University, Harbin Institute of Technology (Shenzhen), Fujian University of Technology, Xiamen University备注：Submitted to ICASSP 2023. 8 pages, 6 figures摘要：语音情感识别是从语音信号中推断人的情感和情感状态，从而改善人机交互的重要手段。鉴于最近的工作主要集中在挖掘时空信息从手工制作的特征，我们探索如何建模的时间模式的语音情感从动态时间尺度。为此，本文提出了一种新的时间情感建模方法--时间感知双向多尺度网络（TIM-Net），该方法从不同的时间尺度上学习多尺度的上下文情感表征。具体而言，TIM-Net首先利用时间感知块学习时间情感表征，然后整合过去和未来的互补信息丰富上下文表征，最后融合多个时间尺度特征以更好地适应情感变化.在6个基准SER数据集上的实验结果表明，TIM-Net的UAR和WAR平均值分别比第二好的TIM-Net提高了2.34%和2.61%。TIM-Net在跨语料库SER测试中的表现明显优于最新的领域自适应方法，具有较强的泛化能力。摘要：Speech emotion recognition (SER) plays a vital role in improving the interactions between humans and machines by inferring human emotion and affective states from speech signals. Whereas recent works primarily focus on mining spatiotemporal information from hand-crafted features, we explore how to model the temporal patterns of speech emotions from dynamic temporal scales. Towards that goal, we introduce a novel temporal emotional modeling approach for SER, termed Temporal-aware bI-direction Multi-scale Network (TIM-Net), which learns multi-scale contextual affective representations from various time scales. Specifically, TIM-Net first employs temporal-aware blocks to learn temporal affective representation, then integrates complementary information from the past and the future to enrich contextual representations, and finally, fuses multiple time scale features for better adaptation to the emotional variation. Extensive experimental results on six benchmark SER datasets demonstrate the superior performance of TIM-Net, gaining 2.34% and 2.61% improvements of the average UAR and WAR over the second-best on each corpus. Remarkably, TIM-Net outperforms the latest domain-adaptation method on the cross-corpus SER tasks, demonstrating strong generalizability.
【7】 SSM-Net: feature learning for Music Structure Analysis using a Self-Similarity-Matrix based loss
标题：SSM-Net：基于自相似矩阵损失的音乐结构特征学习
链接：https://arxiv.org/abs/2211.08141
作者：Geoffroy Peeters,Florian Angulo机构：LTCI, Télécom-Paris, IP-Paris备注：Extended Abstracts for the Late-Breaking Demo Session of the 23rd Int. Society for Music Information Retrieval Conf., Bengaluru, India, 2022摘要：本文提出了一种新的音频特征学习方法，用于音乐结构分析（MSA）。我们训练深度编码器来学习特征，使得从这些特征得到的自相似矩阵（SSM）近似于地面真实SSM。这通过最小化两个SSM之间的损耗来实现。由于这种损失是可微的。我们可以以直接的方式训练编码器。我们使用RWC-Pop数据集的曲线下面积ROC（AUC）成功证明了这种训练范式的使用。摘要：In this paper, we propose a new paradigm to learn audio features for Music Structure Analysis (MSA). We train a deep encoder to learn features such that the Self-Similarity-Matrix (SSM) resulting from those approximates a ground-truth SSM. This is done by minimizing a loss between both SSMs. Since this loss is differentiable w.r.t. its input features we can train the encoder in a straightforward way. We successfully demonstrate the use of this training paradigm using the Area Under the Curve ROC (AUC) on the RWC-Pop dataset.
【8】 Hierarchical Pronunciation Assessment with Multi-Aspect Attention
标题：多方面注意的分级发音测评
链接：https://arxiv.org/abs/2211.08102
作者：Heejin Do,Yunsu Kim,Gary Geunbae Lee机构：⋆Graduate School of AI, POSTECH, †Department of Computer Science and Engineering, POSTECH备注：Submitted to ICASSP 2023摘要：语音自动评测是计算机辅助语音训练系统的重要组成部分。为了提供深入的反馈，在各种粒度级别（如音素、单词和话语）上对发音进行评分，以及准确性、流畅性和完整性等不同方面是必不可少的。然而，现有的多方面多粒度方法在所有粒度级别上同时预测所有方面;因此，他们很难捕捉音素、词和话语的语言层次。这种局限性进一步导致忽略了同一语言单位中密切的跨体关系。本文提出了一种基于多方面注意的分层语音评估模型（HiPAMA），该模型通过分层表示语音的粒度层次，直接捕捉语音的语言结构，并引入多方面注意，反映同一层次上不同方面之间的关联，从而产生更多的内涵表征。通过从粒度和方面两个方面获取关系信息，HiPAMA能够充分利用多任务学习的优势。在speachocean 762数据集上的实验结果表明，该算法具有较好的鲁棒性，尤其是在难以评估的方面.摘要：Automatic pronunciation assessment is a major component of a computer-assisted pronunciation training system. To provide in-depth feedback, scoring pronunciation at various levels of granularity such as phoneme, word, and utterance, with diverse aspects such as accuracy, fluency, and completeness, is essential. However, existing multi-aspect multi-granularity methods simultaneously predict all aspects at all granularity levels; therefore, they have difficulty in capturing the linguistic hierarchy of phoneme, word, and utterance. This limitation further leads to neglecting intimate cross-aspect relations at the same linguistic unit. In this paper, we propose a Hierarchical Pronunciation Assessment with Multi-aspect Attention (HiPAMA) model, which hierarchically represents the granularity levels to directly capture their linguistic structures and introduces multi-aspect attention that reflects associations across aspects at the same level to create more connotative representations. By obtaining relational information from both the granularity- and aspect-side, HiPAMA can take full advantage of multi-task learning. Remarkable improvements in the experimental results on the speachocean762 datasets demonstrate the robustness of HiPAMA, particularly in the difficult-to-assess aspects.
【9】 Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio
标题：向我展示乐器：从混合音频中检索乐器
链接：https://arxiv.org/abs/2211.07951
作者：Kyungsu Kim,Minju Park,Haesun Joung,Yunkee Chae,Yeongbeom Hong,Seonghyeon Go,Kyogu Lee机构：Department of Intelligence and Information, Seoul National University, Interdisciplinary Program in Artificial Intelligence, Seoul National University, Artificial Intelligence Institute, Seoul National University备注：5 pages, 4 figures, submitted to ICASSP 2023摘要：随着数字音乐制作成为主流，选择合适的虚拟乐器在决定音乐质量方面起着至关重要的作用。为了搜索能够发出期望声音的乐器样本或虚拟乐器，音乐制作者使用他们的耳朵来聆听和比较他们的集合中的每个乐器样本，这是耗时且低效的。本文将此任务称为乐器检索，并提出了一种使用参考音乐混合作为查询来检索所需乐器的方法。该模型由单乐器编码器和多乐器编码器组成，均基于卷积神经网络。对单乐器编码器进行训练，对单声道音频中使用的乐器进行分类，并将其倒数第二层的激活作为乐器嵌入。多乐器编码器被训练为使用由单乐器编码器计算的乐器嵌入作为一组目标嵌入来估计多乐器嵌入。为了更广泛的训练和现实的评估，我们还提出了一个新的数据集称为Nlakh。实验结果表明，单乐器编码器能够学习未知乐器的音频信号到乐器嵌入空间的映射关系，多乐器编码器能够从混合音乐中提取多个嵌入，成功地检索出所需乐器。实验所用的代码和音频样本可从以下网址获得：https://github.com/minju0821/musical_instrument_retrieval摘要：As digital music production has become mainstream, the selection of appropriate virtual instruments plays a crucial role in determining the quality of music. To search the musical instrument samples or virtual instruments that make one's desired sound, music producers use their ears to listen and compare each instrument sample in their collection, which is time-consuming and inefficient. In this paper, we call this task as Musical Instrument Retrieval and propose a method for retrieving desired musical instruments using reference music mixture as a query. The proposed model consists of the Single-Instrument Encoder and the Multi-Instrument Encoder, both based on convolutional neural networks. The Single-Instrument Encoder is trained to classify the instruments used in single-track audio, and we take its penultimate layer's activation as the instrument embedding. The Multi-Instrument Encoder is trained to estimate multiple instrument embeddings using the instrument embeddings computed by the Single-Instrument Encoder as a set of target embeddings. For more generalized training and realistic evaluation, we also propose a new dataset called Nlakh. Experimental results showed that the Single-Instrument Encoder was able to learn the mapping from the audio signal of unseen instruments to the instrument embedding space and the Multi-Instrument Encoder was able to extract multiple embeddings from the mixture of music and retrieve the desired instruments successfully. The code used for the experiment and audio samples are available at: https://github.com/minju0821/musical_instrument_retrieval
【10】 Music Similarity Calculation of Individual Instrumental Sounds Using Metric Learning
标题：基于度量学习的器乐声音相似度计算
链接：https://arxiv.org/abs/2211.07863
作者：Yuka Hashizume,Li Li,Tomoki Toda机构：∗ Nagoya University, Nagoya, Japan, † Nagoya University, Nagoya, Japan, ‡ Nagoya University, Nagoya, Japan备注：APSIPA ASC 2022 (pp.33--38)摘要：音乐相似性度量标准对于开发灵活的音乐推荐系统具有重要意义。已经提出了一些数据驱动的方法来仅从音乐信号计算音乐相似度，诸如使用关于每个音乐片段的标签信息基于三重损失的度量学习。然而，得到的音乐相似性度量通常捕获整段音乐，即，各种乐器声源的混合限制了音乐推荐系统的能力，很难搜索包含类似鼓声的音乐作品。为了开发更灵活的音乐推荐系统，我们提出了一种音乐相似度计算方法，该方法关注音乐作品中的各个乐器声源。通过充分利用数据驱动方法的潜力，我们提出的方法在不使用任何标签信息的情况下，对单个乐器声源信号使用弱监督度量学习，其中三重音丢失中的正样本和负样本由它们是否来自同一音乐作品来定义。此外，假设每个乐器声源在实际中并不总是可用的，我们还研究了在所提出的方法中使用乐器声源分离来获得每个声源的效果。实验结果表明（1）可以为各个乐器声源学习唯一的相似性度量，（2）使用一些乐器声源学习的相似性度量可能导致比使用整个音乐作品学习的结果更准确的结果，（3）当用分离的乐器声音学习时，表现下降，以及（4）通过所提出的方法学习的相似性度量很好地产生了与人类感官的感知相对应的结果。摘要：The criteria for measuring music similarity are important for developing a flexible music recommendation system. Some data-driven methods have been proposed to calculate music similarity from only music signals, such as metric learning based on a triplet loss using tag information on each musical piece. However, the resulting music similarity metric usually captures the entire piece of music, i.e., the mixing of various instrumental sound sources, limiting the capability of the music recommendation system, e.g., it is difficult to search for a musical piece containing similar drum sounds. Towards the development of a more flexible music recommendation system, we propose a music similarity calculation method that focuses on individual instrumental sound sources in a musical piece. By fully exploiting the potential of data-driven methods for our proposed method, we employ weakly supervised metric learning to individual instrumental sound source signals without using any tag information, where positive and negative samples in a triplet loss are defined by whether or not they are from the same musical piece. Furthermore, assuming that each instrumental sound source is not always available in practice, we also investigate the effects of using instrumental sound source separation to obtain each source in the proposed method. Experimental results have shown that (1) unique similarity metrics can be learned for individual instrumental sound sources, (2) similarity metrics learned using some instrumental sound sources are possible to lead to more accurate results than that learned using the entire musical piece, (3) the performance degraded when learning with the separated instrumental sounds, and (4) similarity metrics learned by the proposed method well produced results that correspond to perception by human senses.
【11】 Improving Children's Speech Recognition by Fine-tuning Self-supervised Adult Speech Representations
标题：微调自监督成人语音表征提高儿童语音识别能力
链接：https://arxiv.org/abs/2211.07769
作者：Renee Lu,Mostafa Shahin,Beena Ahmed机构：School of Electrical and Computer Engineering, University of New South, Wales, Sydney, NSW, Australia备注：Under-review @ Speech Communication Journal摘要：儿童 语音 识别 是 一 个 至关 重要 的 领域 ， 但 在 构建 包容 性 语音 技术 时 ， 它 在 很 大 程度 上 被 忽视 。阻碍 该 领域 进展 的 主要 挑战 是 缺乏 足够 的 儿童 语音 语料 库 ;然而 ， 最近 在 自 监督 学习 方面 的 进展 已经 为 克服 数据 缺乏 的 问题 创造 了 新 的 机会 。本文 利用 自 监督 的 成人 语音 表示 方法 ， 利用 三 个 著名 的 儿童 语音 语料 库 建立 儿童 语音 识别 模型 。我们 评估 了 对 母语 和 非 母语 儿童 语音 的 微调 性能 ， 检查 了 跨 领域 儿童 语料 库 的 效果 ， 并 调查 了 微调 一 个 模型 所 需 的 最 小 儿童 语音 量 ， 该 模型 优于 最 先进 的 成人 模型 。我们 还 分析 了 不同 年龄 儿童 的 语音 识别 性能 。研究 结果 表明 ， 使用 跨 领域 儿童 语料 库 进行 微调 后 ， 母语 和 非 母语 儿童 语音 的 相对 改善 率 分别 达到 46.08% 和 45.53% ， 绝对 改善 率 分别 达到 14.70% 和 31.10% 。我们 还 表明 ， 用 少 至 5 小时 的 转录 儿童 语音 ， 就 有 可能 微调 儿童 语音 识别 系统 ， 使 其 优于 对 960 小时 的 成人 语音 进行 微调 的 最 先进 的 成人 模型 。摘要：Children's speech recognition is a vital, yet largely overlooked domain when building inclusive speech technologies. The major challenge impeding progress in this domain is the lack of adequate child speech corpora; however, recent advances in self-supervised learning have created a new opportunity for overcoming this problem of data scarcity. In this paper, we leverage self-supervised adult speech representations and use three well-known child speech corpora to build models for children's speech recognition. We assess the performance of fine-tuning on both native and non-native children's speech, examine the effect of cross-domain child corpora, and investigate the minimum amount of child speech required to fine-tune a model which outperforms a state-of-the-art adult model. We also analyze speech recognition performance across children's ages. Our results demonstrate that fine-tuning with cross-domain child corpora leads to relative improvements of up to 46.08% and 45.53% for native and non-native child speech respectively, and absolute improvements of 14.70% and 31.10%. We also show that with as little as 5 hours of transcribed children's speech, it is possible to fine-tune a children's speech recognition system that outperforms a state-of-the-art adult model fine-tuned on 960 hours of adult speech.
【12】 Describing emotions with acoustic property prompts for speech emotion recognition
标题：用于语音情感识别的声学特性提示描述情感
链接：https://arxiv.org/abs/2211.07737
作者：Hira Dhamyal,Benjamin Elizalde,Soham Deshmukh,Huaming Wang,Bhiksha Raj,Rita Singh机构：Carnegie Mellon University,Microsoft摘要：情感是一个广泛的连续体，将情感视为离散的类别限制了模型捕捉连续体中细微差别的能力。挑战在于如何描述情感的细微差别，以及如何让模特学会这些描述。在这项工作中，我们设计了一种方法，通过计算声学属性，如音高、响度、语速和发音速率，为给定的音频自动创建描述（或提示）。我们使用5个不同的情感数据集将提示与其对应的音频配对。我们使用这些音频-文本对训练神经网络模型。然后，我们使用另一个数据集评估模型。我们研究了该模型如何学习将音频与描述相关联，从而提高语音情感识别和语音音频检索的性能。我们希望我们的发现能激发描述情绪的广泛连续性的研究摘要：Emotions lie on a broad continuum and treating emotions as a discrete number of classes limits the ability of a model to capture the nuances in the continuum. The challenge is how to describe the nuances of emotions and how to enable a model to learn the descriptions. In this work, we devise a method to automatically create a description (or prompt) for a given audio by computing acoustic properties, such as pitch, loudness, speech rate, and articulation rate. We pair a prompt with its corresponding audio using 5 different emotion datasets. We trained a neural network model using these audio-text pairs. Then, we evaluate the model using one more dataset. We investigate how the model can learn to associate the audio with the descriptions, resulting in performance improvement of Speech Emotion Recognition and Speech Audio Retrieval. We expect our findings to motivate research describing the broad continuum of emotion
【13】 The Birds Need Attention Too: Analysing usage of Self Attention in identifying bird calls in soundscapes
标题：鸟类也需要注意：自我注意在声音场景中辨别鸟类叫声的使用分析
链接：https://arxiv.org/abs/2211.07722
作者：Chandra Kanth Nagesh,Abhishek Purushothama机构：Department of Computer Science, University of Colorado Boulder, Boulder, CO备注：12 pages, 9 tables and 7 figures摘要：Birds are vital parts of ecosystems across the world and are an excellent measure of the quality of life on earth. Many bird species are endangered while others are already extinct. Ecological efforts in understanding and monitoring bird populations are important to conserve their habitat and species, but this mostly relies on manual methods in rough terrains. Recent advances in Machine Learning and Deep Learning have made automatic bird recognition in diverse environments possible. Birdcall recognition till now has been performed using convolutional neural networks. In this work, we try and understand how self-attention can aid in this endeavor. With that we build an pre-trained Attention-based Spectrogram Transformer baseline for BirdCLEF 2022 and compare the results against the pre-trained Convolution-based baseline. Our results show that the transformer models outperformed the convolutional model and we further validate our results by building baselines and analyzing the results for the previous year BirdCLEF 2021 challenge. Source code available at https://github.com/ck090/BirdCLEF-22
【14】 End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English
标题：端到端语音到意图预测，以改进印地语和英语的电子商务客户支持Voicebot
链接：https://arxiv.org/abs/2211.07710
作者：Abhinav Goyal,Anupam Singh,Nikesh Garera备注：Accepted at EMNLP 2022 Industry Track摘要：自动化的电话客户支持在很大程度上依赖于准确和高效的语音到意图（S2I）系统。使用多组件管道构建这样的系统会带来各种挑战，因为它们需要大型注释数据集、具有较高的延迟并且具有复杂的部署。这些管道也容易出现复合误差。为了克服这些挑战，我们讨论了一个在双语环境中的客户支持语音机器人任务的端到端（E2E）S2I模型。我们展示了如何通过利用预先训练的自动语音识别（ASR）模型来解决E2E意图分类，该模型在小的注释数据集上进行了轻微的修改和微调。实验结果表明，我们的最佳E2E模型在F1得分上比传统管道的相对性能高约27%。摘要：Automation of on-call customer support relies heavily on accurate and efficient speech-to-intent (S2I) systems. Building such systems using multi-component pipelines can pose various challenges because they require large annotated datasets, have higher latency, and have complex deployment. These pipelines are also prone to compounding errors. To overcome these challenges, we discuss an end-to-end (E2E) S2I model for customer support voicebot task in a bilingual setting. We show how we can solve E2E intent classification by leveraging a pre-trained automatic speech recognition (ASR) model with slight modification and fine-tuning on small annotated datasets. Experimental results show that our best E2E model outperforms a conventional pipeline by a relative ~27% on the F1 score.
【15】 Reverberation as Supervision for Speech Separation
标题：混响作为语音分离的监督
链接：https://arxiv.org/abs/2211.08303
作者：Rohith Aralikatti,Christoph Boeddeker,Gordon Wichern,Aswin Shanmugam Subramanian,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, University of Maryland, College Park, MD, USA, Paderborn University, Paderborn, Germany备注：5 pages, 2 figures, 4 tables. Submitted to ICASSP 2023摘要：提出了一种新的无监督损失函数--混响监督（RAS），用于单通道混响语音分离。用于无监督分离的现有方法需要混合物的混合物的合成或假设教师模型的存在，使得它们难以被认为是解释动物听觉系统中分离能力的出现的潜在方法。我们假设在训练时间双通道混合的可用性，并且训练神经网络以在给定通道之一作为输入的情况下分离源，使得可以从分离的源预测另一个通道。由于每个信道的房间脉冲响应（RIR）之间的关系取决于源的位置，而源的位置对于网络是未知的，因此网络不能依赖于学习该关系。相反，我们提出的损失函数通过维纳滤波将每个分离的源拟合到目标通道中的混合，并将得到的混合与地面真实值进行比较。我们表明，最小化预测的右声道混合信号相对于真实信号的尺度不变信号失真比（SI-SDR）隐含地引导网络分离左声道源。基于WHAMR！的半监督混响语音分离任务数据集，使用训练数据，其中只有5%（分别，10%）的混合物用相关的分离源标记，我们实现了70%（分别，78%），而仅在标记数据上训练的模型获得43%（分别为，45%）。摘要：This paper proposes reverberation as supervision (RAS), a novel unsupervised loss function for single-channel reverberant speech separation. Prior methods for unsupervised separation required the synthesis of mixtures of mixtures or assumed the existence of a teacher model, making them difficult to consider as potential methods explaining the emergence of separation abilities in an animal's auditory system. We assume the availability of two-channel mixtures at training time, and train a neural network to separate the sources given one of the channels as input such that the other channel may be predicted from the separated sources. As the relationship between the room impulse responses (RIRs) of each channel depends on the locations of the sources, which are unknown to the network, the network cannot rely on learning that relationship. Instead, our proposed loss function fits each of the separated sources to the mixture in the target channel via Wiener filtering, and compares the resulting mixture to the ground-truth one. We show that minimizing the scale-invariant signal-to-distortion ratio (SI-SDR) of the predicted right-channel mixture with respect to the ground truth implicitly guides the network towards separating the left-channel sources. On a semi-supervised reverberant speech separation task based on the WHAMR! dataset, using training data where just 5% (resp., 10%) of the mixtures are labeled with associated isolated sources, we achieve 70% (resp., 78%) of the SI-SDR improvement obtained when training with supervision on the full training set, while a model trained only on the labeled data obtains 43% (resp., 45%).
【16】 Is Style All You Need? Dependencies Between Emotion and GST-based Speaker Recognition
标题：时尚是你所需要的全部吗？情绪与基于GST的说话人识别之间的依赖关系
链接：https://arxiv.org/abs/2211.08213
作者：Morgan Sandler,Arun Ross机构：Michigan State University备注：Submitted to ICASSP 2023摘要：在本研究中，我们研究了从语音样本中提取的说话人身份嵌入可以用于情感检测和分类的假设。特别地，我们表明通过使用1-D三元组卷积神经网络（CNN）和全局风格令牌（GST）方案（例如，DeepTalk网络）并重用经训练的说话人识别模型权重来生成情感分类域中的特征。利用VoxCeleb 1、VoxCeleb 2和Librispeech数据集训练自动说话人识别（ASR）网络，并使用说话人身份标签建立三元组训练损失函数.使用支持向量机（SVM）分类器，我们将说话人身份嵌入映射到来自CREMA-D、IEMOCAP和MSP-Podcast数据集的离散情感类别。在语音情感检测任务中，CREMA-D的动作情感样本的识别率为80.8%，IEMOCAP的半自然情感样本的识别率为81.2%，MSP-Podcast的自然情感样本的识别率为66.9%。我们还提出了一种新的两阶段层次分类器（HC）方法，该方法在CREMA-D情感样本上表现出+2%的ACC改善。通过这项工作，我们试图传达的重要性，整体建模内部用户变化的音频样本摘要：In this work, we study the hypothesis that speaker identity embeddings extracted from speech samples may be used for detection and classification of emotion. In particular, we show that emotions can be effectively identified by learning speaker identities by use of a 1-D Triplet Convolutional Neural Network (CNN) & Global Style Token (GST) scheme (e.g., DeepTalk Network) and reusing the trained speaker recognition model weights to generate features in the emotion classification domain. The automatic speaker recognition (ASR) network is trained with VoxCeleb1, VoxCeleb2, and Librispeech datasets with a triplet training loss function using speaker identity labels. Using an Support Vector Machine (SVM) classifier, we map speaker identity embeddings into discrete emotion categories from the CREMA-D, IEMOCAP, and MSP-Podcast datasets. On the task of speech emotion detection, we obtain 80.8% ACC with acted emotion samples from CREMA-D, 81.2% ACC with semi-natural emotion samples in IEMOCAP, and 66.9% ACC with natural emotion samples in MSP-Podcast. We also propose a novel two-stage hierarchical classifier (HC) approach which demonstrates +2% ACC improvement on CREMA-D emotion samples. Through this work, we seek to convey the importance of holistically modeling intra-user variation within audio samples
【1】 Reverberation as Supervision for Speech Separation
标题：混响作为语音分离的监督
链接：https://arxiv.org/abs/2211.08303
* 与cs.SD语音【15】为同一篇
作者：Rohith Aralikatti,Christoph Boeddeker,Gordon Wichern,Aswin Shanmugam Subramanian,Jonathan Le Roux机构：Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA, USA, University of Maryland, College Park, MD, USA, Paderborn University, Paderborn, Germany备注：5 pages, 2 figures, 4 tables. Submitted to ICASSP 2023摘要：提出了一种新的无监督损失函数--混响监督（RAS），用于单通道混响语音分离。用于无监督分离的现有方法需要混合物的混合物的合成或假设教师模型的存在，使得它们难以被认为是解释动物听觉系统中分离能力的出现的潜在方法。我们假设在训练时间双通道混合的可用性，并且训练神经网络以在给定通道之一作为输入的情况下分离源，使得可以从分离的源预测另一个通道。由于每个信道的房间脉冲响应（RIR）之间的关系取决于源的位置，而源的位置对于网络是未知的，因此网络不能依赖于学习该关系。相反，我们提出的损失函数通过维纳滤波将每个分离的源拟合到目标通道中的混合，并将得到的混合与地面真实值进行比较。我们表明，最小化预测的右声道混合信号相对于真实信号的尺度不变信号失真比（SI-SDR）隐含地引导网络分离左声道源。基于WHAMR！的半监督混响语音分离任务数据集，使用训练数据，其中只有5%（分别，10%）的混合物用相关的分离源标记，我们实现了70%（分别，78%），而仅在标记数据上训练的模型获得43%（分别为，45%）。摘要：This paper proposes reverberation as supervision (RAS), a novel unsupervised loss function for single-channel reverberant speech separation. Prior methods for unsupervised separation required the synthesis of mixtures of mixtures or assumed the existence of a teacher model, making them difficult to consider as potential methods explaining the emergence of separation abilities in an animal's auditory system. We assume the availability of two-channel mixtures at training time, and train a neural network to separate the sources given one of the channels as input such that the other channel may be predicted from the separated sources. As the relationship between the room impulse responses (RIRs) of each channel depends on the locations of the sources, which are unknown to the network, the network cannot rely on learning that relationship. Instead, our proposed loss function fits each of the separated sources to the mixture in the target channel via Wiener filtering, and compares the resulting mixture to the ground-truth one. We show that minimizing the scale-invariant signal-to-distortion ratio (SI-SDR) of the predicted right-channel mixture with respect to the ground truth implicitly guides the network towards separating the left-channel sources. On a semi-supervised reverberant speech separation task based on the WHAMR! dataset, using training data where just 5% (resp., 10%) of the mixtures are labeled with associated isolated sources, we achieve 70% (resp., 78%) of the SI-SDR improvement obtained when training with supervision on the full training set, while a model trained only on the labeled data obtains 43% (resp., 45%).
【2】 Is Style All You Need? Dependencies Between Emotion and GST-based Speaker Recognition
标题：时尚是你所需要的全部吗？情绪与基于GST的说话人识别之间的依赖关系
链接：https://arxiv.org/abs/2211.08213
* 与cs.SD语音【16】为同一篇
作者：Morgan Sandler,Arun Ross机构：Michigan State University备注：Submitted to ICASSP 2023摘要：在本研究中，我们研究了从语音样本中提取的说话人身份嵌入可以用于情感检测和分类的假设。特别地，我们表明通过使用1-D三元组卷积神经网络（CNN）和全局风格令牌（GST）方案（例如，DeepTalk网络）并重用经训练的说话人识别模型权重来生成情感分类域中的特征。利用VoxCeleb 1、VoxCeleb 2和Librispeech数据集训练自动说话人识别（ASR）网络，并使用说话人身份标签建立三元组训练损失函数.使用支持向量机（SVM）分类器，我们将说话人身份嵌入映射到来自CREMA-D、IEMOCAP和MSP-Podcast数据集的离散情感类别。在语音情感检测任务中，CREMA-D的动作情感样本的识别率为80.8%，IEMOCAP的半自然情感样本的识别率为81.2%，MSP-Podcast的自然情感样本的识别率为66.9%。我们还提出了一种新的两阶段层次分类器（HC）方法，该方法在CREMA-D情感样本上表现出+2%的ACC改善。通过这项工作，我们试图传达的重要性，整体建模内部用户变化的音频样本摘要：In this work, we study the hypothesis that speaker identity embeddings extracted from speech samples may be used for detection and classification of emotion. In particular, we show that emotions can be effectively identified by learning speaker identities by use of a 1-D Triplet Convolutional Neural Network (CNN) & Global Style Token (GST) scheme (e.g., DeepTalk Network) and reusing the trained speaker recognition model weights to generate features in the emotion classification domain. The automatic speaker recognition (ASR) network is trained with VoxCeleb1, VoxCeleb2, and Librispeech datasets with a triplet training loss function using speaker identity labels. Using an Support Vector Machine (SVM) classifier, we map speaker identity embeddings into discrete emotion categories from the CREMA-D, IEMOCAP, and MSP-Podcast datasets. On the task of speech emotion detection, we obtain 80.8% ACC with acted emotion samples from CREMA-D, 81.2% ACC with semi-natural emotion samples in IEMOCAP, and 66.9% ACC with natural emotion samples in MSP-Podcast. We also propose a novel two-stage hierarchical classifier (HC) approach which demonstrates +2% ACC improvement on CREMA-D emotion samples. Through this work, we seek to convey the importance of holistically modeling intra-user variation within audio samples
【3】 Improved disentangled speech representations using contrastive learning in factorized hierarchical variational autoencoder
标题：因子分解分层变分自动编码器中基于对比学习的解缠语音表示的改进
链接：https://arxiv.org/abs/2211.08191
作者：Yuying Xie,Thomas Arildsen,Zheng-Hua Tan机构：Department of Electronic Systems, Aalborg University, Denmark备注：submitted to ICASSP 2023摘要：利用说话人身份和内容在不同时间尺度上的变化，\acrlong{fhvae}（\acrshort{fhvae}）使用序列潜变量和段潜变量来符号化这两个属性。通过假设表示说话者和内容的潜在变量遵循序列相关和序列无关的先验来执行解纠缠。对于序列相关先验，\acrshort{fhvae}假设高斯分布具有话语尺度变化的平均值和固定的小方差。训练过程促使序列变量以较小的方差接近其先验均值。但是，这种约束相对较弱。因此，我们在\acrshort{fhvae}框架中引入对比学习。该方法的目的是使序列变量在代表同一说话人时聚类，同时尽可能远离其他说话人的序列变量。该方法不改变框架的结构，只改变训练过程，不增加测试成本。本文选择语音转换作为应用。潜在变量评估包括对序列潜在变量的说话者增加验证和识别，以及对分段潜在变量的语音识别。此外，语音转换性能的评估是基于说话人确认和语音识别实验。实验结果表明，与\acrshort{fhvae}方法相比，该方法在序列特征提取和分段特征提取方面都有一定的改进，语音转换性能也有一定程度的提高。摘要：By utilizing the fact that speaker identity and content vary on different time scales, \acrlong{fhvae} (\acrshort{fhvae}) uses a sequential latent variable and a segmental latent variable to symbolize these two attributes. Disentanglement is carried out by assuming the latent variables representing speaker and content follow sequence-dependent and sequence-independent priors. For the sequence-dependent prior, \acrshort{fhvae} assumes a Gaussian distribution with an utterance-scale varying mean and a fixed small variance. The training process promotes sequential variables getting close to the mean of its prior with small variance. However, this constraint is relatively weak. Therefore, we introduce contrastive learning in the \acrshort{fhvae} framework. The proposed method aims to make the sequential variables clustering when representing the same speaker, while distancing themselves as far as possible from those of other speakers. The structure of the framework has not been changed in the proposed method but only the training process, thus no more cost is needed during test. Voice conversion has been chosen as the application in this paper. Latent variable evaluations include speakerincrease verification and identification for the sequential latent variable, and speech recognition for the segmental latent variable. Furthermore, assessments of voice conversion performance are on the grounds of speaker verification and speech recognition experiments. Experiment results show that the proposed method improves both sequential and segmental feature extraction compared with \acrshort{fhvae}, and moderately improved voice conversion performance.
【4】 Exploring the Joint Use of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding
标题：探讨复述和知识提炼在口语理解继续学习中的联合运用
链接：https://arxiv.org/abs/2211.08161
作者：Umberto Cappellazzo,Daniele Falavigna,Alessio Brutti机构：University of Trento, Trento, Italy, Fondazione Bruno Kessler, Trento, Italy备注：Submitted to ICASSP 2023摘要：连续学习指的是一种动态框架，其中模型或代理随着时间的推移接收非静态数据流，并且必须在保留先前获得的知识的同时适应新数据。不幸的是，深度神经网络不能满足这两个要求，导致所谓的灾难性遗忘现象。虽然在计算机视觉领域已经提出了大量的策略来减少遗忘，但另一方面，对于语音相关的任务，却缺乏工作。本文主要探讨了在课堂递增学习情境下，复述和知识提炼（KD）方法在口语理解中的联合应用。我们报告了网络中不同级别的多个KD组合，表明组合特征级和预测级KD会产生最佳结果。最后，我们提供了一个关于复述记忆大小影响的消融研究，证实了我们的方法适用于低资源设备。摘要：Continual learning refers to a dynamical framework in which a model or agent receives a stream of non-stationary data over time and must adapt to new data while preserving previously acquired knowledge. Unfortunately, deep neural networks fail to meet these two desiderata, incurring the so-called catastrophic forgetting phenomenon. Whereas a vast array of strategies have been proposed to attenuate forgetting in the computer vision domain, for speech-related tasks, on the other hand, there is a dearth of works. In this paper, we turn our attention toward the joint use of rehearsal and knowledge distillation (KD) approaches for spoken language understanding under a class-incremental learning scenario. We report on multiple KD combinations at different levels in the network, showing that combining feature-level and predictions-level KDs leads to the best results. Finally, we provide an ablation study on the effect of the size of the rehearsal memory that corroborates the appropriateness of our approach for low-resource devices.
【5】 On Unsupervised Uncertainty-Driven Speech Pseudo-Label Filtering and Model Calibration
标题：无监督不确定性驱动的语音伪标签滤波及模型校正
链接：https://arxiv.org/abs/2211.07795
作者：Nauman Dawalatabad,Sameer Khurana,Antoine Laurent,James Glass机构：MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA, LIUM - Le Mans University, France摘要：伪标签（PL）滤波是自训练（ST）方法的一个重要组成部分。基于辍学的不确定性驱动的自训练（DUST）首先在源领域标记数据上训练教师模型。然后，教师模型被用于为未标记的目标域数据提供PL。最后，我们训练一个学生使用增广标记和伪标记数据。这个过程是迭代的，学生成为下一个DUST迭代的老师。在每个DUST迭代中，在学生模型训练之前的一个关键步骤是过滤掉可能导致学生模型误入歧途的噪声PL。在DUST中，我们提出了一种简单、有效、理论上合理的PL滤波策略，该策略基于教师模型对未标记语音话语预测的不确定性。我们通过计算从教师模型中抽取的多个样本之间的不一致性来估计模型的不确定性，在推断过程中通过丢弃注入噪声。在这项工作中，我们表明，DUST的PL滤波，如最初使用的，可能会失败的严重源和目标域不匹配。我们建议几种方法来消除或减轻这个问题。此外，我们从DUST的神经网络模型校准的研究中得到了启示，并表明良好校准的模型与DUST PL滤波步骤的积极结果强烈相关。摘要：Pseudo-label (PL) filtering forms a crucial part of Self-Training (ST) methods for unsupervised domain adaptation. Dropout-based Uncertainty-driven Self-Training (DUST) proceeds by first training a teacher model on source domain labeled data. Then, the teacher model is used to provide PLs for the unlabeled target domain data. Finally, we train a student on augmented labeled and pseudo-labeled data. The process is iterative, where the student becomes the teacher for the next DUST iteration. A crucial step that precedes the student model training in each DUST iteration is filtering out noisy PLs that could lead the student model astray. In DUST, we proposed a simple, effective, and theoretically sound PL filtering strategy based on the teacher model's uncertainty about its predictions on unlabeled speech utterances. We estimate the model's uncertainty by computing disagreement amongst multiple samples drawn from the teacher model during inference by injecting noise via dropout. In this work, we show that DUST's PL filtering, as initially used, may fail under severe source and target domain mismatch. We suggest several approaches to eliminate or alleviate this issue. Further, we bring insights from the research in neural network model calibration to DUST and show that a well-calibrated model correlates strongly with a positive outcome of the DUST PL filtering step.
【6】 Introducing Semantics into Speech Encoders
标题：将语义学引入语音编码器
链接：https://arxiv.org/abs/2211.08402
* 与cs.SD语音【1】为同一篇
作者：Derek Xu,Shuyan Dong,Changhan Wang,Suyoun Kim,Zhaojiang Lin,Akshat Shrivastava,Shang-Wen Li,Liang-Hsuan Tseng,Alexei Baevski,Guan-Ting Lin,Hung-yi Lee,Yizhou Sun,Wei Wang机构：University of California, Los Angeles,Meta AI,National Taiwan University摘要：最近的研究发现现有的自监督语音编码器主要包含声学信息而不是语义信息。结果，通过利用来自大语言模型（LLM）的丰富的语义表示，流水线化的有监督的自动语音识别（ASR）到大语言模型（LLM）系统在语义口语语言任务上实现了最先进的结果。这些系统是以标记音频转录为代价的，获得标记音频转录是昂贵且耗时的。我们提出了一种任务不可知的无监督方式，将来自LLM的语义信息合并到自监督语音编码器中，而无需标记音频转录。通过引入语义，在意图分类上，现有语音编码器的口语理解性能提高了10%以上，在命名实体解析和空位填充上有适度的提高，口语问答FF1得分提高了2%以上.我们的无监督方法实现了与在超过100小时的标记音频副本上训练的有监督方法类似的性能，证明了对现有语音编码器的无监督语义增强的可行性。摘要：Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recognition (ASR) to large language model (LLM) systems achieve state-of-the-art results on semantic spoken language tasks by utilizing rich semantic representations from the LLM. These systems come at the cost of labeled audio transcriptions, which is expensive and time-consuming to obtain. We propose a task-agnostic unsupervised way of incorporating semantic information from LLMs into self-supervised speech encoders without labeled audio transcriptions. By introducing semantics, we improve existing speech encoder spoken language understanding performance by over 10\% on intent classification, with modest gains in named entity resolution and slot filling, and spoken question answering FF1 score by over 2\%. Our unsupervised approach achieves similar performance as supervised methods trained on over 100 hours of labeled audio transcripts, demonstrating the feasibility of unsupervised semantic augmentations to existing speech encoders.
【7】 Music Instrument Classification Reprogrammed
标题：乐器分类重新编程
链接：https://arxiv.org/abs/2211.08379
* 与cs.SD语音【2】为同一篇
作者：Hsin-Hung Chen,Alexander Lerch机构：Music Informatics Group, Georgia Institute of Technology备注：Accepted at 29th International Conference on Multimedia Modeling (MMM23)摘要：乐器分类是音乐信息检索中的一个热门任务，其性能常常受到缺乏可用的标注数据的影响和限制。我们建议用“重新编程”来解决这个问题，这是一种利用预先训练的深度和复杂神经网络的技术，该网络最初通过修改和映射预先训练的模型的输入和输出来针对不同的任务。我们证明了重新编程可以有效地利用为不同任务学习的表征的能力，并且所得到的重新编程的系统可以在训练参数的一小部分上与现有技术的系统相当甚至优于现有技术的系统。因此，我们的结果表明，重新编程是一种有前途的技术，可能适用于其他因数据稀缺而受阻的任务。摘要：The performance of approaches to Music Instrument Classification, a popular task in Music Information Retrieval, is often impacted and limited by the lack of availability of annotated data for training. We propose to address this issue with "reprogramming," a technique that utilizes pre-trained deep and complex neural networks originally targeting a different task by modifying and mapping both the input and output of the pre-trained model. We demonstrate that reprogramming can effectively leverage the power of the representation learned for a different task and that the resulting reprogrammed system can perform on par or even outperform state-of-the-art systems at a fraction of training parameters. Our results, therefore, indicate that reprogramming is a promising technique potentially applicable to other tasks impeded by data scarcity.
【8】 FlowGrad: Using Motion for Visual Sound Source Localization
标题：FlowGrad：使用运动进行视觉声源定位
链接：https://arxiv.org/abs/2211.08367
* 与cs.SD语音【3】为同一篇
作者：Rajsuryan Singh,Pablo Zinemanas,Xavier Serra,Juan Pablo Bello,Magdalena Fuentes机构：MTG, Universitat Pompeu Fabra, Barcelona, Spain, MARL, New York University, New York, USA, IDM, New York University, New York, USA备注：Submitted to ICASSP 2023摘要：视觉声源定位中的大多数最近工作依赖于以自监督方式学习的语义视听表示，并且通过设计排除了视频中存在的时间信息。虽然它被证明对广泛使用的基准数据集是有效的，但该方法在城市交通等具有挑战性的场景中却达不到要求。本文将时间背景引入到城市场景声源定位的最新方法中，使用光流作为编码运动信息的手段。通过分析这些方法的优缺点，有助于我们更好地理解视觉声源定位问题，并揭示视听场景理解面临的挑战。摘要：Most recent work in visual sound source localization relies on semantic audio-visual representations learned in a self-supervised manner, and by design excludes temporal information present in videos. While it proves to be effective for widely used benchmark datasets, the method falls short for challenging scenarios like urban traffic. This work introduces temporal context into the state-of-the-art methods for sound source localization in urban scenes using optical flow as a means to encode motion information. An analysis of the strengths and weaknesses of our methods helps us better understand the problem of visual sound source localization and sheds light on open challenges for audio-visual scene understanding.
【9】 Online Phase Reconstruction via DNN-based Phase Differences Estimation
标题：基于离散神经网络的相位差估计在线相位重构
链接：https://arxiv.org/abs/2211.08246
* 与cs.SD语音【4】为同一篇
作者：Yoshiki Masuyama,Kohei Yatabe,Kento Nagatomo,Yasuhiro Oikawa机构：Tokyo University of Agriculture and Technology备注：Accepted to IEEE/ACM Trans. Audio, Speech, and Language Processing摘要：提出了一种基于因果深度神经网络的两阶段在线相位重构框架。相位重构是仅从相应幅度恢复短时傅里叶变换（STFT）系数的相位的任务。然而，相位对波形偏移很敏感，即使使用DNN也不容易从幅度估计。为了克服这个问题，我们提出使用DNN来估计相邻时间-频率仓之间的相位差。根据短时傅里叶变换相位和幅度偏导数之间的理论关系，证明了卷积神经网络适用于相位差估计。所估计的相位差用于通过以逐帧的方式求解加权最小二乘问题来重构相位。与现有的基于离散神经网络的相位重构方法相比，该方法具有因果关系，不需要迭代过程。实验结果表明，该方法优于现有的在线相位重构方法和基于DNN的相位重构方法。摘要：This paper presents a two-stage online phase reconstruction framework using causal deep neural networks (DNNs). Phase reconstruction is a task of recovering phase of the short-time Fourier transform (STFT) coefficients only from the corresponding magnitude. However, phase is sensitive to waveform shifts and not easy to estimate from the magnitude even with a DNN. To overcome this problem, we propose to use DNNs for estimating differences of phase between adjacent time-frequency bins. We show that convolutional neural networks are suitable for phase difference estimation, according to the theoretical relation between partial derivatives of STFT phase and magnitude. The estimated phase differences are used for reconstructing phase by solving a weighted least squares problem in a frame-by-frame manner. In contrast to existing DNN-based phase reconstruction methods, the proposed framework is causal and does not require any iterative procedure. The experiments showed that the proposed method outperforms existing online methods and a DNN-based method for phase reconstruction.
【10】 Multilingual Speech Emotion Recognition With Multi-Gating Mechanism and Neural Architecture Search
标题：基于多门限机制和神经结构搜索的多语言语音情感识别
链接：https://arxiv.org/abs/2211.08237
* 与cs.SD语音【5】为同一篇
作者：Zihan Wang,Qi Meng,HaiFeng Lan,XinRui Zhang,KeHao Guo,Akshat Gupta机构：Columbia University, JP Morgan AI Research, New York, USA摘要：语音情感识别（SER）将音频分类为情感类别，例如快乐、愤怒、恐惧、厌恶和中性。虽然语音情感识别（SER）是流行语言的常见应用，但是对于低资源语言，没有预先训练的语音到文本识别模型的语言。本文首先提出了一种基于语言的情感识别模型，该模型从多个预先训练好的语音模型中提取情感信息，然后设计了一种多领域模型，该模型可以同时对多种语言进行情感识别。我们的多领域模型采用多门机制为每种语言生成唯一的加权特征组合，并通过神经结构搜索模块为每种语言搜索特定的神经网络结构。此外，我们引入对比辅助损失，以建立更多可分离的音频数据表示。实验结果表明，该模型对德语和法语的准确率分别提高了3%和14.3%.摘要：Speech emotion recognition (SER) classifies audio into emotion categories such as Happy, Angry, Fear, Disgust and Neutral. While Speech Emotion Recognition (SER) is a common application for popular languages, it continues to be a problem for low-resourced languages, i.e., languages with no pretrained speech-to-text recognition models. This paper firstly proposes a language-specific model that extract emotional information from multiple pre-trained speech models, and then designs a multi-domain model that simultaneously performs SER for various languages. Our multidomain model employs a multi-gating mechanism to generate unique weighted feature combination for each language, and also searches for specific neural network structure for each language through a neural architecture search module. In addition, we introduce a contrastive auxiliary loss to build more separable representations for audio data. Our experiments show that our model raises the state-of-the-art accuracy by 3% for German and 14.3% for French.
【11】 Temporal Modeling Matters: A Novel Temporal Emotional Modeling Approach for Speech Emotion Recognition
标题：时态建模问题：一种新的语音情感识别时态情感建模方法
链接：https://arxiv.org/abs/2211.08233
* 与cs.SD语音【6】为同一篇
作者：Jiaxin Ye,Xincheng Wen,Yujie Wei,Yong Xu,Kunhong Liu,Hongming Shan机构：Fudan University, Harbin Institute of Technology (Shenzhen), Fujian University of Technology, Xiamen University备注：Submitted to ICASSP 2023. 8 pages, 6 figures摘要：语音情感识别是从语音信号中推断人的情感和情感状态，从而改善人机交互的重要手段。鉴于最近的工作主要集中在挖掘时空信息从手工制作的特征，我们探索如何建模的时间模式的语音情感从动态时间尺度。为此，本文提出了一种新的时间情感建模方法--时间感知双向多尺度网络（TIM-Net），该方法从不同的时间尺度上学习多尺度的上下文情感表征。具体而言，TIM-Net首先利用时间感知块学习时间情感表征，然后整合过去和未来的互补信息丰富上下文表征，最后融合多个时间尺度特征以更好地适应情感变化.在6个基准SER数据集上的实验结果表明，TIM-Net的UAR和WAR平均值分别比第二好的TIM-Net提高了2.34%和2.61%。TIM-Net在跨语料库SER测试中的表现明显优于最新的领域自适应方法，具有较强的泛化能力。摘要：Speech emotion recognition (SER) plays a vital role in improving the interactions between humans and machines by inferring human emotion and affective states from speech signals. Whereas recent works primarily focus on mining spatiotemporal information from hand-crafted features, we explore how to model the temporal patterns of speech emotions from dynamic temporal scales. Towards that goal, we introduce a novel temporal emotional modeling approach for SER, termed Temporal-aware bI-direction Multi-scale Network (TIM-Net), which learns multi-scale contextual affective representations from various time scales. Specifically, TIM-Net first employs temporal-aware blocks to learn temporal affective representation, then integrates complementary information from the past and the future to enrich contextual representations, and finally, fuses multiple time scale features for better adaptation to the emotional variation. Extensive experimental results on six benchmark SER datasets demonstrate the superior performance of TIM-Net, gaining 2.34% and 2.61% improvements of the average UAR and WAR over the second-best on each corpus. Remarkably, TIM-Net outperforms the latest domain-adaptation method on the cross-corpus SER tasks, demonstrating strong generalizability.
【12】 SSM-Net: feature learning for Music Structure Analysis using a Self-Similarity-Matrix based loss
标题：SSM-Net：基于自相似矩阵损失的音乐结构特征学习
链接：https://arxiv.org/abs/2211.08141
* 与cs.SD语音【7】为同一篇
作者：Geoffroy Peeters,Florian Angulo机构：LTCI, Télécom-Paris, IP-Paris备注：Extended Abstracts for the Late-Breaking Demo Session of the 23rd Int. Society for Music Information Retrieval Conf., Bengaluru, India, 2022摘要：本文提出了一种新的音频特征学习方法，用于音乐结构分析（MSA）。我们训练深度编码器来学习特征，使得从这些特征得到的自相似矩阵（SSM）近似于地面真实SSM。这通过最小化两个SSM之间的损耗来实现。由于这种损失是可微的。我们可以以直接的方式训练编码器。我们使用RWC-Pop数据集的曲线下面积ROC（AUC）成功证明了这种训练范式的使用。摘要：In this paper, we propose a new paradigm to learn audio features for Music Structure Analysis (MSA). We train a deep encoder to learn features such that the Self-Similarity-Matrix (SSM) resulting from those approximates a ground-truth SSM. This is done by minimizing a loss between both SSMs. Since this loss is differentiable w.r.t. its input features we can train the encoder in a straightforward way. We successfully demonstrate the use of this training paradigm using the Area Under the Curve ROC (AUC) on the RWC-Pop dataset.
【13】 Hierarchical Pronunciation Assessment with Multi-Aspect Attention
标题：多方面注意的分级发音测评
链接：https://arxiv.org/abs/2211.08102
* 与cs.SD语音【8】为同一篇
作者：Heejin Do,Yunsu Kim,Gary Geunbae Lee机构：⋆Graduate School of AI, POSTECH, †Department of Computer Science and Engineering, POSTECH备注：Submitted to ICASSP 2023摘要：语音自动评测是计算机辅助语音训练系统的重要组成部分。为了提供深入的反馈，在各种粒度级别（如音素、单词和话语）上对发音进行评分，以及准确性、流畅性和完整性等不同方面是必不可少的。然而，现有的多方面多粒度方法在所有粒度级别上同时预测所有方面;因此，他们很难捕捉音素、词和话语的语言层次。这种局限性进一步导致忽略了同一语言单位中密切的跨体关系。本文提出了一种基于多方面注意的分层语音评估模型（HiPAMA），该模型通过分层表示语音的粒度层次，直接捕捉语音的语言结构，并引入多方面注意，反映同一层次上不同方面之间的关联，从而产生更多的内涵表征。通过从粒度和方面两个方面获取关系信息，HiPAMA能够充分利用多任务学习的优势。在speachocean 762数据集上的实验结果表明，该算法具有较好的鲁棒性，尤其是在难以评估的方面.摘要：Automatic pronunciation assessment is a major component of a computer-assisted pronunciation training system. To provide in-depth feedback, scoring pronunciation at various levels of granularity such as phoneme, word, and utterance, with diverse aspects such as accuracy, fluency, and completeness, is essential. However, existing multi-aspect multi-granularity methods simultaneously predict all aspects at all granularity levels; therefore, they have difficulty in capturing the linguistic hierarchy of phoneme, word, and utterance. This limitation further leads to neglecting intimate cross-aspect relations at the same linguistic unit. In this paper, we propose a Hierarchical Pronunciation Assessment with Multi-aspect Attention (HiPAMA) model, which hierarchically represents the granularity levels to directly capture their linguistic structures and introduces multi-aspect attention that reflects associations across aspects at the same level to create more connotative representations. By obtaining relational information from both the granularity- and aspect-side, HiPAMA can take full advantage of multi-task learning. Remarkable improvements in the experimental results on the speachocean762 datasets demonstrate the robustness of HiPAMA, particularly in the difficult-to-assess aspects.
【14】 Show Me the Instruments: Musical Instrument Retrieval from Mixture Audio
标题：向我展示乐器：从混合音频中检索乐器
链接：https://arxiv.org/abs/2211.07951
* 与cs.SD语音【9】为同一篇
作者：Kyungsu Kim,Minju Park,Haesun Joung,Yunkee Chae,Yeongbeom Hong,Seonghyeon Go,Kyogu Lee机构：Department of Intelligence and Information, Seoul National University, Interdisciplinary Program in Artificial Intelligence, Seoul National University, Artificial Intelligence Institute, Seoul National University备注：5 pages, 4 figures, submitted to ICASSP 2023摘要：随着数字音乐制作成为主流，选择合适的虚拟乐器在决定音乐质量方面起着至关重要的作用。为了搜索能够发出期望声音的乐器样本或虚拟乐器，音乐制作者使用他们的耳朵来聆听和比较他们的集合中的每个乐器样本，这是耗时且低效的。本文将此任务称为乐器检索，并提出了一种使用参考音乐混合作为查询来检索所需乐器的方法。该模型由单乐器编码器和多乐器编码器组成，均基于卷积神经网络。对单乐器编码器进行训练，对单声道音频中使用的乐器进行分类，并将其倒数第二层的激活作为乐器嵌入。多乐器编码器被训练为使用由单乐器编码器计算的乐器嵌入作为一组目标嵌入来估计多乐器嵌入。为了更广泛的训练和现实的评估，我们还提出了一个新的数据集称为Nlakh。实验结果表明，单乐器编码器能够学习未知乐器的音频信号到乐器嵌入空间的映射关系，多乐器编码器能够从混合音乐中提取多个嵌入，成功地检索出所需乐器。实验所用的代码和音频样本可从以下网址获得：https://github.com/minju0821/musical_instrument_retrieval摘要：As digital music production has become mainstream, the selection of appropriate virtual instruments plays a crucial role in determining the quality of music. To search the musical instrument samples or virtual instruments that make one's desired sound, music producers use their ears to listen and compare each instrument sample in their collection, which is time-consuming and inefficient. In this paper, we call this task as Musical Instrument Retrieval and propose a method for retrieving desired musical instruments using reference music mixture as a query. The proposed model consists of the Single-Instrument Encoder and the Multi-Instrument Encoder, both based on convolutional neural networks. The Single-Instrument Encoder is trained to classify the instruments used in single-track audio, and we take its penultimate layer's activation as the instrument embedding. The Multi-Instrument Encoder is trained to estimate multiple instrument embeddings using the instrument embeddings computed by the Single-Instrument Encoder as a set of target embeddings. For more generalized training and realistic evaluation, we also propose a new dataset called Nlakh. Experimental results showed that the Single-Instrument Encoder was able to learn the mapping from the audio signal of unseen instruments to the instrument embedding space and the Multi-Instrument Encoder was able to extract multiple embeddings from the mixture of music and retrieve the desired instruments successfully. The code used for the experiment and audio samples are available at: https://github.com/minju0821/musical_instrument_retrieval
【15】 Music Similarity Calculation of Individual Instrumental Sounds Using Metric Learning
标题：基于度量学习的器乐声音相似度计算
链接：https://arxiv.org/abs/2211.07863
* 与cs.SD语音【10】为同一篇
作者：Yuka Hashizume,Li Li,Tomoki Toda机构：∗ Nagoya University, Nagoya, Japan, † Nagoya University, Nagoya, Japan, ‡ Nagoya University, Nagoya, Japan备注：APSIPA ASC 2022 (pp.33--38)摘要：音乐相似性度量标准对于开发灵活的音乐推荐系统具有重要意义。已经提出了一些数据驱动的方法来仅从音乐信号计算音乐相似度，诸如使用关于每个音乐片段的标签信息基于三重损失的度量学习。然而，得到的音乐相似性度量通常捕获整段音乐，即，各种乐器声源的混合限制了音乐推荐系统的能力，很难搜索包含类似鼓声的音乐作品。为了开发更灵活的音乐推荐系统，我们提出了一种音乐相似度计算方法，该方法关注音乐作品中的各个乐器声源。通过充分利用数据驱动方法的潜力，我们提出的方法在不使用任何标签信息的情况下，对单个乐器声源信号使用弱监督度量学习，其中三重音丢失中的正样本和负样本由它们是否来自同一音乐作品来定义。此外，假设每个乐器声源在实际中并不总是可用的，我们还研究了在所提出的方法中使用乐器声源分离来获得每个声源的效果。实验结果表明（1）可以为各个乐器声源学习唯一的相似性度量，（2）使用一些乐器声源学习的相似性度量可能导致比使用整个音乐作品学习的结果更准确的结果，（3）当用分离的乐器声音学习时，表现下降，以及（4）通过所提出的方法学习的相似性度量很好地产生了与人类感官的感知相对应的结果。摘要：The criteria for measuring music similarity are important for developing a flexible music recommendation system. Some data-driven methods have been proposed to calculate music similarity from only music signals, such as metric learning based on a triplet loss using tag information on each musical piece. However, the resulting music similarity metric usually captures the entire piece of music, i.e., the mixing of various instrumental sound sources, limiting the capability of the music recommendation system, e.g., it is difficult to search for a musical piece containing similar drum sounds. Towards the development of a more flexible music recommendation system, we propose a music similarity calculation method that focuses on individual instrumental sound sources in a musical piece. By fully exploiting the potential of data-driven methods for our proposed method, we employ weakly supervised metric learning to individual instrumental sound source signals without using any tag information, where positive and negative samples in a triplet loss are defined by whether or not they are from the same musical piece. Furthermore, assuming that each instrumental sound source is not always available in practice, we also investigate the effects of using instrumental sound source separation to obtain each source in the proposed method. Experimental results have shown that (1) unique similarity metrics can be learned for individual instrumental sound sources, (2) similarity metrics learned using some instrumental sound sources are possible to lead to more accurate results than that learned using the entire musical piece, (3) the performance degraded when learning with the separated instrumental sounds, and (4) similarity metrics learned by the proposed method well produced results that correspond to perception by human senses.
【16】 Improving Children's Speech Recognition by Fine-tuning Self-supervised Adult Speech Representations
标题：微调自监督成人语音表征提高儿童语音识别能力
链接：https://arxiv.org/abs/2211.07769
* 与cs.SD语音【11】为同一篇
作者：Renee Lu,Mostafa Shahin,Beena Ahmed机构：School of Electrical and Computer Engineering, University of New South, Wales, Sydney, NSW, Australia备注：Under-review @ Speech Communication Journal摘要：儿童语音识别是一个至关重要的领域，但在构建包容性语音技术时，它在很大程度上被忽视。阻碍该领域进展的主要挑战是缺乏足够的儿童语音语料库;然而，自监督学习的最新进展已经为克服数据缺乏的问题创造了新的机会。本文利用自监督的成人语音表示方法，利用三个著名的儿童语音语料库建立儿童语音识别模型。我们评估了对母语和非母语儿童语音的微调性能，检查了跨领域儿童语料库的效果，并调查了微调一个模型所需的最小儿童语音量，该模型优于最先进的成人模型。我们还分析了不同年龄儿童的语音识别性能。研究结果表明，使用跨领域儿童语料库进行微调后，母语和非母语儿童语音的相对改善率分别达到46.08%和45.53%，绝对改善率分别达到14.70%和31.10%。我们还表明，用少至5小时的转录儿童语音，就有可能微调儿童语音识别系统，使其优于对960小时的成人语音进行微调的最先进的成人模型。摘要：Children's speech recognition is a vital, yet largely overlooked domain when building inclusive speech technologies. The major challenge impeding progress in this domain is the lack of adequate child speech corpora; however, recent advances in self-supervised learning have created a new opportunity for overcoming this problem of data scarcity. In this paper, we leverage self-supervised adult speech representations and use three well-known child speech corpora to build models for children's speech recognition. We assess the performance of fine-tuning on both native and non-native children's speech, examine the effect of cross-domain child corpora, and investigate the minimum amount of child speech required to fine-tune a model which outperforms a state-of-the-art adult model. We also analyze speech recognition performance across children's ages. Our results demonstrate that fine-tuning with cross-domain child corpora leads to relative improvements of up to 46.08% and 45.53% for native and non-native child speech respectively, and absolute improvements of 14.70% and 31.10%. We also show that with as little as 5 hours of transcribed children's speech, it is possible to fine-tune a children's speech recognition system that outperforms a state-of-the-art adult model fine-tuned on 960 hours of adult speech.
【17】 Describing emotions with acoustic property prompts for speech emotion recognition
标题：用于语音情感识别的声学特性提示描述情感
链接：https://arxiv.org/abs/2211.07737
* 与cs.SD语音【12】为同一篇
作者：Hira Dhamyal,Benjamin Elizalde,Soham Deshmukh,Huaming Wang,Bhiksha Raj,Rita Singh机构：Carnegie Mellon University,Microsoft摘要：情感是一个广泛的连续体，将情感视为离散的类别限制了模型捕捉连续体中细微差别的能力。挑战在于如何描述情感的细微差别，以及如何让模特学会这些描述。在这项工作中，我们设计了一种方法，通过计算声学属性，如音高、响度、语速和发音速率，为给定的音频自动创建描述（或提示）。我们使用5个不同的情感数据集将提示与其对应的音频配对。我们使用这些音频-文本对训练神经网络模型。然后，我们使用另一个数据集评估模型。我们研究了该模型如何学习将音频与描述相关联，从而提高语音情感识别和语音音频检索的性能。我们希望我们的发现能激发描述情绪的广泛连续性的研究摘要：Emotions lie on a broad continuum and treating emotions as a discrete number of classes limits the ability of a model to capture the nuances in the continuum. The challenge is how to describe the nuances of emotions and how to enable a model to learn the descriptions. In this work, we devise a method to automatically create a description (or prompt) for a given audio by computing acoustic properties, such as pitch, loudness, speech rate, and articulation rate. We pair a prompt with its corresponding audio using 5 different emotion datasets. We trained a neural network model using these audio-text pairs. Then, we evaluate the model using one more dataset. We investigate how the model can learn to associate the audio with the descriptions, resulting in performance improvement of Speech Emotion Recognition and Speech Audio Retrieval. We expect our findings to motivate research describing the broad continuum of emotion
【18】 The Birds Need Attention Too: Analysing usage of Self Attention in identifying bird calls in soundscapes
标题：鸟类也需要注意：自我注意在声音场景中辨别鸟类叫声的使用分析
链接：https://arxiv.org/abs/2211.07722
* 与cs.SD语音【13】为同一篇
作者：Chandra Kanth Nagesh,Abhishek Purushothama机构：Department of Computer Science, University of Colorado Boulder, Boulder, CO备注：12 pages, 9 tables and 7 figures摘要：鸟类是全世界生态系统的重要组成部分，也是衡量地球生命质量的一个很好的指标。许多鸟类濒临灭绝，而另一些则已经灭绝。了解和监测鸟类种群的生态努力对保护它们的栖息地和物种很重要，但这主要依赖于在崎岖地形中的人工方法。机器学习和深度学习的最新进展使得在不同环境中自动识别鸟类成为可能。迄今为止，鸟鸣识别一直使用卷积神经网络来执行。在本书中，我们试图理解自我注意力是如何在这一过程中发挥作用的。据此，我们为BirdCLEF 2022构建了一个预训练的基于注意力的谱图Transformer基线，并将结果与预训练的基于卷积的基线进行比较。我们的结果表明，Transformer模型优于卷积模型，我们通过建立基线和分析上一年BirdCLEF 2021挑战赛的结果进一步验证了我们的结果。源代码可从www.example.com获得https://github.com/ck090/BirdCLEF-22摘要：Birds are vital parts of ecosystems across the world and are an excellent measure of the quality of life on earth. Many bird species are endangered while others are already extinct. Ecological efforts in understanding and monitoring bird populations are important to conserve their habitat and species, but this mostly relies on manual methods in rough terrains. Recent advances in Machine Learning and Deep Learning have made automatic bird recognition in diverse environments possible. Birdcall recognition till now has been performed using convolutional neural networks. In this work, we try and understand how self-attention can aid in this endeavor. With that we build an pre-trained Attention-based Spectrogram Transformer baseline for BirdCLEF 2022 and compare the results against the pre-trained Convolution-based baseline. Our results show that the transformer models outperformed the convolutional model and we further validate our results by building baselines and analyzing the results for the previous year BirdCLEF 2021 challenge. Source code available at https://github.com/ck090/BirdCLEF-22
【19】 End-to-End Speech to Intent Prediction to improve E-commerce Customer Support Voicebot in Hindi and English
标题：端到端语音到意图预测，以改进印地语和英语的电子商务客户支持Voicebot
链接：https://arxiv.org/abs/2211.07710
* 与cs.SD语音【14】为同一篇
作者：Abhinav Goyal,Anupam Singh,Nikesh Garera备注：Accepted at EMNLP 2022 Industry Track摘要：自动化的电话客户支持在很大程度上依赖于准确和高效的语音到意图（S2I）系统。使用多组件管道构建这样的系统会带来各种挑战，因为它们需要大型注释数据集、具有较高的延迟并且具有复杂的部署。这些管道也容易出现复合误差。为了克服这些挑战，我们讨论了一个在双语环境中的客户支持语音机器人任务的端到端（E2E）S2I模型。我们展示了如何通过利用预先训练的自动语音识别（ASR）模型来解决E2E意图分类，该模型在小的注释数据集上进行了轻微的修改和微调。实验结果表明，我们的最佳E2E模型在F1得分上比传统管道的相对性能高约27%。摘要：Automation of on-call customer support relies heavily on accurate and efficient speech-to-intent (S2I) systems. Building such systems using multi-component pipelines can pose various challenges because they require large annotated datasets, have higher latency, and have complex deployment. These pipelines are also prone to compounding errors. To overcome these challenges, we discuss an end-to-end (E2E) S2I model for customer support voicebot task in a bilingual setting. We show how we can solve E2E intent classification by leveraging a pre-trained automatic speech recognition (ASR) model with slight modification and fine-tuning on small annotated datasets. Experimental results show that our best E2E model outperforms a conventional pipeline by a relative ~27% on the F1 score.
机器翻译，仅供参考
每日语音速递