【1】 Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition
标题：替代梯度尖峰神经网络作为大词汇量连续语音识别的编码器
链接：https://arxiv.org/abs/2212.01187
作者：Alexandre Bittar,Philip N. Garner机构：Idiap Research Institute, Martigny, Switzerland, ´Ecole Polytechnique F´ed´erale de Lausanne, Switzerland备注：Submitted to ICASSP 2023摘要：与产生密集和实值响应的传统人工神经元相比，生物激发的尖峰神经元传输稀疏和二进制信息，这也可以导致节能实现。最近的研究表明，脉冲神经网络可以像标准的递归神经网络一样使用替代梯度方法进行训练。它们在语音命令识别任务中表现出了有希望的结果。使用相同的技术，我们表明它们可扩展到大词汇量的连续语音识别，其中它们能够取代编码器中的LSTM，而性能只有很小的损失。这表明它们可能适用于更多涉及的序列到序列的任务。此外，与它们的循环非尖峰对应物相比，它们在不需要使用门的情况下对爆炸梯度问题表现出鲁棒性。摘要：Compared to conventional artificial neurons that produce dense and real-valued responses, biologically-inspired spiking neurons transmit sparse and binary information, which can also lead to energy-efficient implementations. Recent research has shown that spiking neural networks can be trained like standard recurrent neural networks using the surrogate gradient method. They have shown promising results on speech command recognition tasks. Using the same technique, we show that they are scalable to large vocabulary continuous speech recognition, where they are capable of replacing LSTMs in the encoder with only minor loss of performance. This suggests that they may be applicable to more involved sequence-to-sequence tasks. Moreover, in contrast to their recurrent non-spiking counterparts, they show robustness to exploding gradient problems without the need to use gates.
【2】 Cross-Modal Mutual Learning for Cued Speech Recognition
标题：用于线索语音识别的跨模式交互学习
链接：https://arxiv.org/abs/2212.01083
机构：Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, China摘要：自动提示语音识别（ACSR）为视觉通信提供了一个智能的人机界面，其中提示语音（CS）系统利用嘴唇运动和手势为听力受损的人编码口语。以前的ACSR方法通常使用直接特征连接作为主要的融合范例。但是，异步模态（\textit{即}、嘴唇、手形和手位置）可能对特征连接造成干扰。针对这一问题，提出了一种基于Transformer的跨模态交互学习框架，以促进多模态交互。与传统的自我注意相比，我们的模型迫使不同模态的模态特定信息通过一个模态不变码本，整理每种模态的标记的语言表示。然后利用共享的语言知识对多模态序列进行重新同步。此外，我们还建立了一个新的大规模汉语多说话人CS数据集。据我们所知，这是第一部针对汉语普通话的ACSR研究。针对不同语言（\textit{即}、中文、法文和英式英语）。实验结果表明，该模型的识别性能明显优于现有的识别方法.摘要：Automatic Cued Speech Recognition (ACSR) provides an intelligent human-machine interface for visual communications, where the Cued Speech (CS) system utilizes lip movements and hand gestures to code spoken language for hearing-impaired people. Previous ACSR approaches often utilize direct feature concatenation as the main fusion paradigm. However, the asynchronous modalities (\textit{i.e.}, lip, hand shape and hand position) in CS may cause interference for feature concatenation. To address this challenge, we propose a transformer based cross-modal mutual learning framework to prompt multi-modal interaction. Compared with the vanilla self-attention, our model forces modality-specific information of different modalities to pass through a modality-invariant codebook, collating linguistic representations for tokens of each modality. Then the shared linguistic knowledge is used to re-synchronize multi-modal sequences. Moreover, we establish a novel large-scale multi-speaker CS dataset for Mandarin Chinese. To our knowledge, this is the first work on ACSR for Mandarin Chinese. Extensive experiments are conducted for different languages (\textit{i.e.}, Chinese, French, and British English). Results demonstrate that our model exhibits superior recognition performance to the state-of-the-art by a large margin.
【3】 AccEar: Accelerometer Acoustic Eavesdropping with Unconstrained Vocabulary
标题：AccEar：无限制词汇的加速度计声学窃听
链接：https://arxiv.org/abs/2212.01042
作者：Pengfei Hu,Hui Zhuang,Panneer Selvam Santhalingamy,Riccardo Spolaor,Parth Pathaky,Guoming Zhang,Xiuzhen Cheng机构：∗ Shandong University, China, † George Mason University, USA摘要：With the increasing popularity of voice-based applications, acoustic eavesdropping has become a serious threat to users' privacy. While on smartphones the access to microphones needs an explicit user permission, acoustic eavesdropping attacks can rely on motion sensors (such as accelerometer and gyroscope), which access is unrestricted. However, previous instances of such attacks can only recognize a limited set of pre-trained words or phrases. In this paper, we present AccEar, an accelerometerbased acoustic eavesdropping attack that can reconstruct any audio played on the smartphone's loudspeaker with unconstrained vocabulary. We show that an attacker can employ a conditional Generative Adversarial Network (cGAN) to reconstruct highfidelity audio from low-frequency accelerometer signals. The presented cGAN model learns to recreate high-frequency components of the user's voice from low-frequency accelerometer signals through spectrogram enhancement. We assess the feasibility and effectiveness of AccEar attack in a thorough set of experiments using audio from 16 public personalities. As shown by the results in both objective and subjective evaluations, AccEar successfully reconstructs user speeches from accelerometer signals in different scenarios including varying sampling rate, audio volume, device model, etc.
【4】 Role of Audio in Audio-Visual Video Summarization
标题：音频在视听视频摘要中的作用
链接：https://arxiv.org/abs/2212.01040
作者：Ibrahim Shoer,Berkay Kopru,Engin Erzin机构：KUIS-AI Laboratory, Multimedia, Vision and Graphics Group, College of Engineering, Koc¸ University, Istanbul, Turkey摘要：视频摘要吸引了人们对有效的视频表示、检索和浏览以缓解容量和流量激增问题的关注。虽然视频摘要主要使用视觉通道进行压缩，但视听建模的好处在最近的文献中出现。来自音频信道的信息可以是视频内容中的视听相关性的结果。在本研究中，我们提出了一个新的视听视频摘要框架，整合了四种视听信息融合方式，以及基于群组和基于注意力的网络。此外，我们研究了一种新的解释性方法，使用视听典型相关分析（CCA）来更好地理解和解释音频在视频摘要任务中的作用。在TVSum数据集上的实验结果表明，该方法对视频摘要的F1评分和Kendall-tau评分都有一定的提高.此外，基于视听CCA将TVSum和COGNIMUSE数据集上的视频内容分割为正相关和负相关视频，相对于正相关视频，对于仅音频和视听视频摘要产生了强有力的性能改进。摘要：Video summarization attracts attention for efficient video representation, retrieval, and browsing to ease volume and traffic surge problems. Although video summarization mostly uses the visual channel for compaction, the benefits of audio-visual modeling appeared in recent literature. The information coming from the audio channel can be a result of audio-visual correlation in the video content. In this study, we propose a new audio-visual video summarization framework integrating four ways of audio-visual information fusion with GRU-based and attention-based networks. Furthermore, we investigate a new explainability methodology using audio-visual canonical correlation analysis (CCA) to better understand and explain the role of audio in the video summarization task. Experimental evaluations on the TVSum dataset attain F1 score and Kendall-tau score improvements for the audio-visual video summarization. Furthermore, splitting video content on TVSum and COGNIMUSE datasets based on audio-visual CCA as positively and negatively correlated videos yields a strong performance improvement over the positively correlated videos for audio-only and audio-visual video summarization.
【5】 Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptations
标题：索纳斯·特克西尔！使用电影改编为图书自动构建密集配乐
链接：https://arxiv.org/abs/2212.01033
作者：Jaidev Shriram,Makarand Tapaswi,Vinoo Alluri机构：International Institute of Information Technology, Hyderabad备注：Accepted to ISMIR 2022. Project page: this https URL摘要：阅读，就像听音乐一样，是一种身临其境的体验，在带读者踏上情感之旅的同时，也在运送他们。听补充性的音乐有可能增强阅读体验，尤其是当音乐在风格上有凝聚力，在情感上有关联时。本文提出了第一种全自动的方法来构建密集的书籍配乐，可以在整个阅读过程中播放高质量的器乐。我们的作品采用了独特的文本处理和音乐编织管道，决定了一章中场景的上下文和情感构成。这允许我们的方法识别和播放来自书的电影改编的音轨的相关摘录。凭借电影作曲家的精湛技艺，我们的图书配乐包括专家制作的主题和其他特定场景的音乐特征。我们通过感知研究验证了我们方法的设计决策。我们的读者注意到，由于通过不间断和风格一致的音乐赋予的高度沉浸感，以及通过高精度的情感和场景上下文识别获得的高度情绪状态，书籍配乐大大增强了他们的阅读体验。摘要：Reading, much like music listening, is an immersive experience that transports readers while taking them on an emotional journey. Listening to complementary music has the potential to amplify the reading experience, especially when the music is stylistically cohesive and emotionally relevant. In this paper, we propose the first fully automatic method to build a dense soundtrack for books, which can play high-quality instrumental music for the entirety of the reading duration. Our work employs a unique text processing and music weaving pipeline that determines the context and emotional composition of scenes in a chapter. This allows our method to identify and play relevant excerpts from the soundtrack of the book's movie adaptation. By relying on the movie composer's craftsmanship, our book soundtracks include expert-made motifs and other scene-specific musical characteristics. We validate the design decisions of our approach through a perceptual study. Our readers note that the book soundtrack greatly enhanced their reading experience, due to high immersiveness granted via uninterrupted and style-consistent music, and a heightened emotional state attained via high precision emotion and scene context recognition.
【6】 A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling
标题：基于领域知识的音乐嵌入空间与符号音乐造型注意机制
链接：https://arxiv.org/abs/2212.00973
作者：Z. Guo,J. Kang,D. Herremans机构：Singapore University of Technology and Design备注：This paper is accepted at AAAI 2023摘要：随着Transformer体系结构在自然语言领域的成功，类Transformer体系结构近年来被广泛应用于符号音乐领域。然而，象征音乐和文本是两种不同的形态。符号音乐包含多个属性，既有绝对属性（如：音调）和相对属性（例如，音调间隔）。这些相对属性塑造了人类对音乐主题的感知。然而，这些重要的相对属性在现有的符号音乐建模方法中大多被忽略，主要原因是缺乏一个音乐意义的嵌入空间，在该空间中符号音乐标记的绝对和相对嵌入都可以被有效地表示。本文提出了一种符号音乐的基本音乐嵌入（FME）算法，该算法基于一种偏移调整的正弦编码，在该编码中可以嵌入绝对和相对属性，平移不变性）被明确地保持。在此基础上，提出了一种基于相对索引、音高和首音嵌入的注意机制（RIPO attention），充分利用音乐领域知识进行符号音乐建模.实验结果表明：利用FME和RIPO注意力的RIPO Transformer的性能优于现有技术的Transformers（即，音乐Transformer、线性变换器）。此外，在下游音乐生成任务中使用RIPO Transformer，我们注意到臭名昭著的退化现象不再存在，并且由RIPO变换器生成的音乐在主观和客观评价方面都优于由最先进的变换器模型生成的音乐。摘要：Following the success of the transformer architecture in the natural language domain, transformer-like architectures have been widely applied to the domain of symbolic music recently. Symbolic music and text, however, are two different modalities. Symbolic music contains multiple attributes, both absolute attributes (e.g., pitch) and relative attributes (e.g., pitch interval). These relative attributes shape human perception of musical motifs. These important relative attributes, however, are mostly ignored in existing symbolic music modeling methods with the main reason being the lack of a musically-meaningful embedding space where both the absolute and relative embeddings of the symbolic music tokens can be efficiently represented. In this paper, we propose the Fundamental Music Embedding (FME) for symbolic music based on a bias-adjusted sinusoidal encoding within which both the absolute and the relative attributes can be embedded and the fundamental musical properties (e.g., translational invariance) are explicitly preserved. Taking advantage of the proposed FME, we further propose a novel attention mechanism based on the relative index, pitch and onset embeddings (RIPO attention) such that the musical domain knowledge can be fully utilized for symbolic music modeling. Experiment results show that our proposed model: RIPO transformer which utilizes FME and RIPO attention outperforms the state-of-the-art transformers (i.e., music transformer, linear transformer) in a melody completion task. Moreover, using the RIPO transformer in a downstream music generation task, we notice that the notorious degeneration phenomenon no longer exists and the music generated by the RIPO transformer outperforms the music generated by state-of-the-art transformer models in both subjective and objective evaluations.
【7】 Relative Acoustic Features for Distance Estimation in Smart-Homes
标题：用于智能家居距离估计的相对声学特征
链接：https://arxiv.org/abs/2212.01306
作者：Francesco Nespoli,Daniel Barreda,Patrick A. Naylor机构：Nuance Communications UK, Imperial College London摘要：Any audio recording encapsulates the unique fingerprint of the associated acoustic environment, namely the background noise and reverberation. Considering the scenario of a room equipped with a fixed smart speaker device with one or more microphones and a wearable smart device (watch, glasses or smartphone), we employed the improved proportionate normalized least mean square adaptive filter to estimate the relative room impulse response mapping the audio recordings of the two devices. We performed inter-device distance estimation by exploiting a new set of features obtained extending the definition of some acoustic attributes of the room impulse response to its relative version. In combination with the sparseness measure of the estimated relative room impulse response, the relative features allow precise inter-device distance estimation which can be exploited for tasks such as best microphone selection or acoustic scene analysis. Experimental results from simulated rooms of different dimensions and reverberation times demonstrate the effectiveness of this computationally lightweight approach for smart home acoustic ranging applications
【8】 CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models
标题：第二章：开发用于自监督语音模型的卷积神经网络适配器
链接：https://arxiv.org/abs/2212.01282
作者：Zih-Ching Chen,Yu-Shun Sung,Hung-yi Lee机构：Graduate Institute of Communication Engineering, National Taiwan University, Taiwan摘要：Self-supervised learning (SSL) is a powerful technique for learning representations from unlabeled data. Transformer based models such as HuBERT, which consist a feature extractor and transformer layers, are leading the field in the speech domain. SSL models are fine-tuned on a wide range of downstream tasks, which involves re-training the majority of the model for each task. Previous studies have introduced applying adapters, which are small lightweight modules commonly used in Natural Language Processing (NLP) to adapt pre-trained models to new tasks. However, such efficient tuning techniques only provide adaptation at the transformer layer, but failed to perform adaptation at the feature extractor. In this paper, we propose CHAPTER, an efficient tuning method specifically designed for SSL speech model, by applying CNN adapters at the feature extractor. Using this method, we can only fine-tune fewer than 5% of parameters per task compared to fully fine-tuning and achieve better and more stable performance. We empirically found that adding CNN adapters to the feature extractor can help the adaptation on emotion and speaker tasks. For instance, the accuracy of SID is improved from 87.71 to 91.56, and the accuracy of ER is improved by 5%.
【9】 Preliminary Study on SSCF-derived Polar Coordinate for ASR
标题：基于SSCF的ASR极坐标初步研究
链接：https://arxiv.org/abs/2212.01245
作者：Sotheara Leang,Eric Castelli,Dominique Vaufreydaz,Sethserey Sam机构：Institute of Digital Research and Innovation, CADT, Phnom Penh, Cambodia, Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France摘要：定义了转换角来描述频谱子带质心声学空间中元音到元音的转换，发现它们在说话人和说话速率之间是相似的。本文提出了用极坐标代替角度来描述语音信号的方法，并将其应用于自动语音识别中。根据在BRAF 100数据集上评估的实验结果，极坐标在混合和跨性别语音识别中实现了比角度显著更高的准确度，表明这些表示在定义语音信号的声学轨迹方面是优越的。此外，当它们与其一阶和二阶导数（$\Delta$，$\Delta$$\Delta$）一起使用时，准确率显著提高，尤其是在跨女性识别中。然而，结果表明，它们并不比传统的梅尔频率倒谱系数（MFCC）更不依赖于性别。摘要：The transition angles are defined to describe the vowel-to-vowel transitions in the acoustic space of the Spectral Subband Centroids, and the findings show that they are similar among speakers and speaking rates. In this paper, we propose to investigate the usage of polar coordinates in favor of angles to describe a speech signal by characterizing its acoustic trajectory and using them in Automatic Speech Recognition. According to the experimental results evaluated on the BRAF100 dataset, the polar coordinates achieved significantly higher accuracy than the angles in the mixed and cross-gender speech recognitions, demonstrating that these representations are superior at defining the acoustic trajectory of the speech signal. Furthermore, the accuracy was significantly improved when they were utilized with their first and second-order derivatives ($\Delta$, $\Delta$$\Delta$), especially in cross-female recognition. However, the results showed they were not much more gender-independent than the conventional Mel-frequency Cepstral Coefficients (MFCCs).
【10】 Injecting Spatial Information for Monaural Speech Enhancement via Knowledge Distillation
标题：基于知识提取的空间信息注入用于单声道语音增强
链接：https://arxiv.org/abs/2212.01012
作者：Xinmeng Xu,Weiping Tu,Yuhong Yang机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Hubei Luojia Laboratory, China备注：Submitted to ICASSP 2023摘要：Monaural speech enhancement (SE) provides a versatile and cost-effective approach to SE tasks by utilizing recordings from a single microphone. However, the monaural SE lags performance behind multi-channel SE as the monaural SE methods are unable to extract spatial information from one-channel recordings, which greatly limits their application scenarios. To address this issue, we inject spatial information into the monaural SE model and propose a knowledge distillation strategy to enable the monaural SE model to learn binaural speech features from the binaural SE model, which makes monaural SE model possible to reconstruct higher intelligibility and quality enhanced speeches under low signal-to-noise ratio (SNR) conditions. Extensive experiments show that our proposed monaural SE model by injecting spatial information via knowledge distillation achieves favorable performance against other monaural SE models with fewer parameters.
【1】 Relative Acoustic Features for Distance Estimation in Smart-Homes
标题：用于智能家居距离估计的相对声学特征
链接：https://arxiv.org/abs/2212.01306
*与cs.SD语音【7】为同一篇
作者：Francesco Nespoli,Daniel Barreda,Patrick A. Naylor机构：Nuance Communications UK, Imperial College London摘要：Any audio recording encapsulates the unique fingerprint of the associated acoustic environment, namely the background noise and reverberation. Considering the scenario of a room equipped with a fixed smart speaker device with one or more microphones and a wearable smart device (watch, glasses or smartphone), we employed the improved proportionate normalized least mean square adaptive filter to estimate the relative room impulse response mapping the audio recordings of the two devices. We performed inter-device distance estimation by exploiting a new set of features obtained extending the definition of some acoustic attributes of the room impulse response to its relative version. In combination with the sparseness measure of the estimated relative room impulse response, the relative features allow precise inter-device distance estimation which can be exploited for tasks such as best microphone selection or acoustic scene analysis. Experimental results from simulated rooms of different dimensions and reverberation times demonstrate the effectiveness of this computationally lightweight approach for smart home acoustic ranging applications
【2】 CHAPTER: Exploiting Convolutional Neural Network Adapters for Self-supervised Speech Models
标题：第二章：开发用于自监督语音模型的卷积神经网络适配器
链接：https://arxiv.org/abs/2212.01282
* 与cs.SD语音【8】为同一篇
作者：Zih-Ching Chen,Yu-Shun Sung,Hung-yi Lee机构：Graduate Institute of Communication Engineering, National Taiwan University, Taiwan摘要：自监督学习（SSL）是一种用于从未标记数据学习表示的强大技术。基于Transformer的模型，如HuBERT，由特征提取器和变换器层组成，在语音领域处于领先地位。SSL模型在广泛的下游任务上进行了微调，这涉及到为每个任务重新训练模型的大部分。先前的研究已经引入了应用适配器，这是自然语言处理（NLP）中常用的小型轻量级模块，用于使预先训练好的模型适应新的任务。然而，这种有效的调整技术仅在Transformer层提供自适应，而不能在特征提取器执行自适应。本文通过在特征提取器中引入CNN适配器，提出了一种针对SSL语音模型的CHAPTER优化方法。与完全微调相比，使用这种方法，我们可以在每个任务中仅微调不到5%的参数，并获得更好和更稳定的性能。实验结果表明，在特征提取器中加入CNN适配器，可以有效地提高特征提取器对情感和说话人任务的适应能力。例如，SID的准确度从87.71提高到91.56，ER的准确度提高了5%。摘要：Self-supervised learning (SSL) is a powerful technique for learning representations from unlabeled data. Transformer based models such as HuBERT, which consist a feature extractor and transformer layers, are leading the field in the speech domain. SSL models are fine-tuned on a wide range of downstream tasks, which involves re-training the majority of the model for each task. Previous studies have introduced applying adapters, which are small lightweight modules commonly used in Natural Language Processing (NLP) to adapt pre-trained models to new tasks. However, such efficient tuning techniques only provide adaptation at the transformer layer, but failed to perform adaptation at the feature extractor. In this paper, we propose CHAPTER, an efficient tuning method specifically designed for SSL speech model, by applying CNN adapters at the feature extractor. Using this method, we can only fine-tune fewer than 5% of parameters per task compared to fully fine-tuning and achieve better and more stable performance. We empirically found that adding CNN adapters to the feature extractor can help the adaptation on emotion and speaker tasks. For instance, the accuracy of SID is improved from 87.71 to 91.56, and the accuracy of ER is improved by 5%.
【3】 Preliminary Study on SSCF-derived Polar Coordinate for ASR
标题：基于SSCF的ASR极坐标初步研究
链接：https://arxiv.org/abs/2212.01245
* 与cs.SD语音【9】为同一篇
作者：Sotheara Leang,Eric Castelli,Dominique Vaufreydaz,Sethserey Sam机构：Institute of Digital Research and Innovation, CADT, Phnom Penh, Cambodia, Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France摘要：定义了转换角来描述频谱子带质心声学空间中元音到元音的转换，发现它们在说话人和说话速率之间是相似的。本文提出了用极坐标代替角度来描述语音信号的方法，并将其应用于自动语音识别中。根据在BRAF 100数据集上评估的实验结果，极坐标在混合和跨性别语音识别中实现了比角度显著更高的准确度，表明这些表示在定义语音信号的声学轨迹方面是优越的。此外，当它们与其一阶和二阶导数（$\Delta$，$\Delta$$\Delta$）一起使用时，准确率显著提高，尤其是在跨女性识别中。然而，结果表明，它们并不比传统的梅尔频率倒谱系数（MFCC）更不依赖于性别。摘要：The transition angles are defined to describe the vowel-to-vowel transitions in the acoustic space of the Spectral Subband Centroids, and the findings show that they are similar among speakers and speaking rates. In this paper, we propose to investigate the usage of polar coordinates in favor of angles to describe a speech signal by characterizing its acoustic trajectory and using them in Automatic Speech Recognition. According to the experimental results evaluated on the BRAF100 dataset, the polar coordinates achieved significantly higher accuracy than the angles in the mixed and cross-gender speech recognitions, demonstrating that these representations are superior at defining the acoustic trajectory of the speech signal. Furthermore, the accuracy was significantly improved when they were utilized with their first and second-order derivatives ($\Delta$, $\Delta$$\Delta$), especially in cross-female recognition. However, the results showed they were not much more gender-independent than the conventional Mel-frequency Cepstral Coefficients (MFCCs).
【4】 ExARN: self-attending RNN for target speaker extraction
标题：ExARN：用于目标说话人提取的自适应RNN
链接：https://arxiv.org/abs/2212.01106
作者：Pengjie Shen,Shulin He,Xueliang Zhang机构：College of Computer Science, Inner Mongolia University, China备注：Submitted to ICASSP 2023, 5 pages with 3 figures摘要：目标说话人提取是指在有竞争说话人的环境中，提取出由注册语音所指定的目标说话人。因此，本课题需要同时解决说话人识别和分离两个问题。本文将自我注意与递归神经网络相结合。此外，我们还利用各种方法将不同的辅助信息与混合表示相结合。实验结果表明，该模型在目标说话人提取任务中取得了良好的性能。摘要：Target speaker extraction is to extract the target speaker, specified by enrollment utterance, in an environment with other competing speakers. Therefore, the task needs to solve two problems, speaker identification and separation, at the same time. In this paper, we combine self-attention and Recurrent Neural Networks (RNN). Further, we exploit various ways to combining different auxiliary information with mixed representations. Experimental results show that our proposed model achieves excellent performance on the task of target speaker extraction.
【5】 Injecting Spatial Information for Monaural Speech Enhancement via Knowledge Distillation
标题：基于知识提取的空间信息注入用于单声道语音增强
链接：https://arxiv.org/abs/2212.01012
* 与cs.SD语音【10】为同一篇
作者：Xinmeng Xu,Weiping Tu,Yuhong Yang机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Hubei Luojia Laboratory, China备注：Submitted to ICASSP 2023摘要：单耳语音增强（SE）通过利用来自单个麦克风的录音来提供SE任务的通用且成本有效的方法。然而，由于单声道SE方法无法从单声道录音中提取空间信息，因此单声道SE的性能落后于多声道SE，这极大地限制了其应用场景。针对这一问题，在单耳SE模型中注入空间信息，提出一种知识提取策略，使单耳SE模型能够从双耳SE模型中学习双耳语音特征，从而使单耳SE模型在低信噪比条件下重建出更高可懂度和质量的增强语音成为可能。实验结果表明，与其他参数较少的单耳SE模型相比，本文提出的单耳SE模型通过知识提取注入空间信息，具有较好的性能。摘要：Monaural speech enhancement (SE) provides a versatile and cost-effective approach to SE tasks by utilizing recordings from a single microphone. However, the monaural SE lags performance behind multi-channel SE as the monaural SE methods are unable to extract spatial information from one-channel recordings, which greatly limits their application scenarios. To address this issue, we inject spatial information into the monaural SE model and propose a knowledge distillation strategy to enable the monaural SE model to learn binaural speech features from the binaural SE model, which makes monaural SE model possible to reconstruct higher intelligibility and quality enhanced speeches under low signal-to-noise ratio (SNR) conditions. Extensive experiments show that our proposed monaural SE model by injecting spatial information via knowledge distillation achieves favorable performance against other monaural SE models with fewer parameters.
【6】 Surrogate Gradient Spiking Neural Networks as Encoders for Large Vocabulary Continuous Speech Recognition
标题：替代梯度尖峰神经网络作为大词汇量连续语音识别的编码器
链接：https://arxiv.org/abs/2212.01187
* 与cs.SD语音【1】为同一篇
作者：Alexandre Bittar,Philip N. Garner机构：Idiap Research Institute, Martigny, Switzerland, ´Ecole Polytechnique F´ed´erale de Lausanne, Switzerland备注：Submitted to ICASSP 2023摘要：与产生密集和实值响应的传统人工神经元相比，生物激发的尖峰神经元传输稀疏和二进制信息，这也可以导致节能实现。最近的研究表明，脉冲神经网络可以像标准的递归神经网络一样使用替代梯度方法进行训练。它们在语音命令识别任务中表现出了有希望的结果。使用相同的技术，我们表明它们可扩展到大词汇量的连续语音识别，其中它们能够取代编码器中的LSTM，而性能只有很小的损失。这表明它们可能适用于更多涉及的序列到序列的任务。此外，与它们的循环非尖峰对应物相比，它们在不需要使用门的情况下对爆炸梯度问题表现出鲁棒性。摘要：Compared to conventional artificial neurons that produce dense and real-valued responses, biologically-inspired spiking neurons transmit sparse and binary information, which can also lead to energy-efficient implementations. Recent research has shown that spiking neural networks can be trained like standard recurrent neural networks using the surrogate gradient method. They have shown promising results on speech command recognition tasks. Using the same technique, we show that they are scalable to large vocabulary continuous speech recognition, where they are capable of replacing LSTMs in the encoder with only minor loss of performance. This suggests that they may be applicable to more involved sequence-to-sequence tasks. Moreover, in contrast to their recurrent non-spiking counterparts, they show robustness to exploding gradient problems without the need to use gates.
【7】 Cross-Modal Mutual Learning for Cued Speech Recognition
标题：用于线索语音识别的跨模式交互学习
链接：https://arxiv.org/abs/2212.01083
* 与cs.SD语音【2】为同一篇
机构：Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, China摘要：自动提示语音识别（ACSR）为视觉通信提供了一个智能的人机界面，其中提示语音（CS）系统利用嘴唇运动和手势为听力受损的人编码口语。以前的ACSR方法通常使用直接特征连接作为主要的融合范例。但是，异步模态（\textit{即}、嘴唇、手形和手位置）可能对特征连接造成干扰。针对这一问题，提出了一种基于Transformer的跨模态交互学习框架，以促进多模态交互。与传统的自我注意相比，我们的模型迫使不同模态的模态特定信息通过一个模态不变码本，整理每种模态的标记的语言表示。然后利用共享的语言知识对多模态序列进行重新同步。此外，我们还建立了一个新的大规模汉语多说话人CS数据集。据我们所知，这是第一部针对汉语普通话的ACSR研究。针对不同语言（\textit{即}、中文、法文和英式英语）。实验结果表明，该模型的识别性能明显优于现有的识别方法.摘要：Automatic Cued Speech Recognition (ACSR) provides an intelligent human-machine interface for visual communications, where the Cued Speech (CS) system utilizes lip movements and hand gestures to code spoken language for hearing-impaired people. Previous ACSR approaches often utilize direct feature concatenation as the main fusion paradigm. However, the asynchronous modalities (\textit{i.e.}, lip, hand shape and hand position) in CS may cause interference for feature concatenation. To address this challenge, we propose a transformer based cross-modal mutual learning framework to prompt multi-modal interaction. Compared with the vanilla self-attention, our model forces modality-specific information of different modalities to pass through a modality-invariant codebook, collating linguistic representations for tokens of each modality. Then the shared linguistic knowledge is used to re-synchronize multi-modal sequences. Moreover, we establish a novel large-scale multi-speaker CS dataset for Mandarin Chinese. To our knowledge, this is the first work on ACSR for Mandarin Chinese. Extensive experiments are conducted for different languages (\textit{i.e.}, Chinese, French, and British English). Results demonstrate that our model exhibits superior recognition performance to the state-of-the-art by a large margin.
【8】 AccEar: Accelerometer Acoustic Eavesdropping with Unconstrained Vocabulary
标题：AccEar：无限制词汇的加速度计声学窃听
链接：https://arxiv.org/abs/2212.01042
* 与cs.SD语音【3】为同一篇
作者：Pengfei Hu,Hui Zhuang,Panneer Selvam Santhalingamy,Riccardo Spolaor,Parth Pathaky,Guoming Zhang,Xiuzhen Cheng机构：∗ Shandong University, China, † George Mason University, USA摘要：With the increasing popularity of voice-based applications, acoustic eavesdropping has become a serious threat to users' privacy. While on smartphones the access to microphones needs an explicit user permission, acoustic eavesdropping attacks can rely on motion sensors (such as accelerometer and gyroscope), which access is unrestricted. However, previous instances of such attacks can only recognize a limited set of pre-trained words or phrases. In this paper, we present AccEar, an accelerometerbased acoustic eavesdropping attack that can reconstruct any audio played on the smartphone's loudspeaker with unconstrained vocabulary. We show that an attacker can employ a conditional Generative Adversarial Network (cGAN) to reconstruct highfidelity audio from low-frequency accelerometer signals. The presented cGAN model learns to recreate high-frequency components of the user's voice from low-frequency accelerometer signals through spectrogram enhancement. We assess the feasibility and effectiveness of AccEar attack in a thorough set of experiments using audio from 16 public personalities. As shown by the results in both objective and subjective evaluations, AccEar successfully reconstructs user speeches from accelerometer signals in different scenarios including varying sampling rate, audio volume, device model, etc.
【9】 Role of Audio in Audio-Visual Video Summarization
标题：音频在视听视频摘要中的作用
链接：https://arxiv.org/abs/2212.01040
* 与cs.SD语音【4】为同一篇
作者：Ibrahim Shoer,Berkay Kopru,Engin Erzin机构：KUIS-AI Laboratory, Multimedia, Vision and Graphics Group, College of Engineering, Koc¸ University, Istanbul, Turkey摘要：Video summarization attracts attention for efficient video representation, retrieval, and browsing to ease volume and traffic surge problems. Although video summarization mostly uses the visual channel for compaction, the benefits of audio-visual modeling appeared in recent literature. The information coming from the audio channel can be a result of audio-visual correlation in the video content. In this study, we propose a new audio-visual video summarization framework integrating four ways of audio-visual information fusion with GRU-based and attention-based networks. Furthermore, we investigate a new explainability methodology using audio-visual canonical correlation analysis (CCA) to better understand and explain the role of audio in the video summarization task. Experimental evaluations on the TVSum dataset attain F1 score and Kendall-tau score improvements for the audio-visual video summarization. Furthermore, splitting video content on TVSum and COGNIMUSE datasets based on audio-visual CCA as positively and negatively correlated videos yields a strong performance improvement over the positively correlated videos for audio-only and audio-visual video summarization.
【10】 SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognition
标题：软校正：自动语音识别中的软检测纠错
链接：https://arxiv.org/abs/2212.01039
作者：Yichong Leng,Xu Tan,Wenjie Liu,Kaitao Song,Rui Wang,Xiang-Yang Li,Tao Qin,Edward Lin,Tie-Yan Liu机构：University of Science and Technology of China ,Microsoft Research Asia ,Microsoft Azure Speech摘要：自动语音识别（ASR）中的纠错是指对由ASR模型生成的句子中的错误单词进行纠错。由于最近的ASR模型通常具有低单词错误率（WER），为了避免影响最初正确的标记，纠错模型应当仅修改不正确的单词，因此检测不正确的单词对于纠错是重要的。以往的纠错工作要么通过目标-源注意或CTC（connectionist temporal classification）丢失隐式检测错误词，要么显式定位特定的删除/替换/插入错误。然而，隐式错误检测不提供关于哪些令牌不正确的明确信号，并且显式错误检测遭受低检测精度。本文提出了一种带有软错误检测机制的SoftCorrect算法，避免了显式和隐式错误检测的局限性。具体地说，首先通过专门设计的语言模型产生的概率来检测一个标记是否正确，然后设计一个仅复制检测到的错误标记的约束CTC损失，使解码器专注于错误标记的纠正.与CTC丢失的隐式错误检测相比，SoftCorrect提供了关于哪些字是错误的显式信号，因此不需要复制每个令牌而仅复制错误的令牌;与显式错误检测相比，SoftCorrect不检测特定的删除/替换/插入错误，而是将其留给CTC丢失。在AISHELL-1和Aidatatang数据集上的实验结果表明，SoftCorrect算法在保证并行生成速度的同时，CER分别减少了26.1%和9.4%，大大优于已有的算法.摘要：Error correction in automatic speech recognition (ASR) aims to correct those incorrect words in sentences generated by ASR models. Since recent ASR models usually have low word error rate (WER), to avoid affecting originally correct tokens, error correction models should only modify incorrect words, and therefore detecting incorrect words is important for error correction. Previous works on error correction either implicitly detect error words through target-source attention or CTC (connectionist temporal classification) loss, or explicitly locate specific deletion/substitution/insertion errors. However, implicit error detection does not provide clear signal about which tokens are incorrect and explicit error detection suffers from low detection accuracy. In this paper, we propose SoftCorrect with a soft error detection mechanism to avoid the limitations of both explicit and implicit error detection. Specifically, we first detect whether a token is correct or not through a probability produced by a dedicatedly designed language model, and then design a constrained CTC loss that only duplicates the detected incorrect tokens to let the decoder focus on the correction of error tokens. Compared with implicit error detection with CTC loss, SoftCorrect provides explicit signal about which words are incorrect and thus does not need to duplicate every token but only incorrect tokens; compared with explicit error detection, SoftCorrect does not detect specific deletion/substitution/insertion errors but just leaves it to CTC loss. Experiments on AISHELL-1 and Aidatatang datasets show that SoftCorrect achieves 26.1% and 9.4% CER reduction respectively, outperforming previous works by a large margin, while still enjoying fast speed of parallel generation.
【11】 Sonus Texere! Automated Dense Soundtrack Construction for Books using Movie Adaptations
标题：索纳斯·特克西尔！使用电影改编为图书自动构建密集配乐
链接：https://arxiv.org/abs/2212.01033
* 与cs.SD语音【5】为同一篇
作者：Jaidev Shriram,Makarand Tapaswi,Vinoo Alluri机构：International Institute of Information Technology, Hyderabad备注：Accepted to ISMIR 2022. Project page: this https URL摘要：阅读，就像听音乐一样，是一种身临其境的体验，在带读者踏上情感之旅的同时，也在运送他们。听补充性的音乐有可能增强阅读体验，尤其是当音乐在风格上有凝聚力，在情感上有关联时。本文提出了第一种全自动的方法来构建密集的书籍配乐，可以在整个阅读过程中播放高质量的器乐。我们的作品采用了独特的文本处理和音乐编织管道，决定了一章中场景的上下文和情感构成。这允许我们的方法识别和播放来自书的电影改编的音轨的相关摘录。凭借电影作曲家的精湛技艺，我们的图书配乐包括专家制作的主题和其他特定场景的音乐特征。我们通过感知研究验证了我们方法的设计决策。我们的读者注意到，由于通过不间断和风格一致的音乐赋予的高度沉浸感，以及通过高精度的情感和场景上下文识别获得的高度情绪状态，书籍配乐大大增强了他们的阅读体验。摘要：Reading, much like music listening, is an immersive experience that transports readers while taking them on an emotional journey. Listening to complementary music has the potential to amplify the reading experience, especially when the music is stylistically cohesive and emotionally relevant. In this paper, we propose the first fully automatic method to build a dense soundtrack for books, which can play high-quality instrumental music for the entirety of the reading duration. Our work employs a unique text processing and music weaving pipeline that determines the context and emotional composition of scenes in a chapter. This allows our method to identify and play relevant excerpts from the soundtrack of the book's movie adaptation. By relying on the movie composer's craftsmanship, our book soundtracks include expert-made motifs and other scene-specific musical characteristics. We validate the design decisions of our approach through a perceptual study. Our readers note that the book soundtrack greatly enhanced their reading experience, due to high immersiveness granted via uninterrupted and style-consistent music, and a heightened emotional state attained via high precision emotion and scene context recognition.
【12】 A Domain-Knowledge-Inspired Music Embedding Space and a Novel Attention Mechanism for Symbolic Music Modeling
标题：基于领域知识的音乐嵌入空间与符号音乐造型注意机制
链接：https://arxiv.org/abs/2212.00973
*与cs.SD语音【6】为同一篇
作者：Z. Guo,J. Kang,D. Herremans机构：Singapore University of Technology and Design备注：This paper is accepted at AAAI 2023摘要：Following the success of the transformer architecture in the natural language domain, transformer-like architectures have been widely applied to the domain of symbolic music recently. Symbolic music and text, however, are two different modalities. Symbolic music contains multiple attributes, both absolute attributes (e.g., pitch) and relative attributes (e.g., pitch interval). These relative attributes shape human perception of musical motifs. These important relative attributes, however, are mostly ignored in existing symbolic music modeling methods with the main reason being the lack of a musically-meaningful embedding space where both the absolute and relative embeddings of the symbolic music tokens can be efficiently represented. In this paper, we propose the Fundamental Music Embedding (FME) for symbolic music based on a bias-adjusted sinusoidal encoding within which both the absolute and the relative attributes can be embedded and the fundamental musical properties (e.g., translational invariance) are explicitly preserved. Taking advantage of the proposed FME, we further propose a novel attention mechanism based on the relative index, pitch and onset embeddings (RIPO attention) such that the musical domain knowledge can be fully utilized for symbolic music modeling. Experiment results show that our proposed model: RIPO transformer which utilizes FME and RIPO attention outperforms the state-of-the-art transformers (i.e., music transformer, linear transformer) in a melody completion task. Moreover, using the RIPO transformer in a downstream music generation task, we notice that the notorious degeneration phenomenon no longer exists and the music generated by the RIPO transformer outperforms the music generated by state-of-the-art transformer models in both subjective and objective evaluations.
机器翻译，仅供参考
每日语音速递