今日论文合集：cs.SD语音10篇，eess.AS音频处理12篇。
cs.SD语音

【1】 Egocentric Audio-Visual Object Localization标题：以自我为中心的视听对象定位链接：http://arxiv.org/pdf/2303.13471v1

作者：Chao Huang,Yapeng Tian,Anurag Kumar,Chenliang Xu机构：University of Rochester,Meta Reality Labs Research备注：Accepted by CVPR 2023摘要：人类自然地通过以第一人称视角统一声音和视觉来感知周围的场景。同样地，机器通过从自我中心的角度学习多感官输入来接近人类智能。在本文中，我们探讨了具有挑战性的自我中心的视听对象定位任务，并观察到：1）自我运动普遍存在于第一人称录音，即使在很短的时间内;2）当佩戴者转移他们的注意力时，可以创建视野外声音分量。为了解决第一个问题，我们提出了一个几何感知的时间聚合模块来处理的egomotion明确。通过估计时间几何变换并利用它来更新视觉表示来减轻自运动的影响。此外，我们提出了一个级联的特征增强模块来解决第二个问题。它通过解开视觉指示的音频表示来提高跨模态定位鲁棒性。在训练过程中，我们利用自然可用的视听时间同步作为“自由”的自我监督，以避免昂贵的标记。我们还为评估目的注释和创建Epic Sounding Object数据集。大量的实验表明，我们的方法在以自我为中心的视频中实现了最先进的定位性能，并可以推广到不同的视听场景。摘要：Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with multisensory inputs from an egocentric perspective. In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention. To address the first problem, we propose a geometry-aware temporal aggregation module to handle the egomotion explicitly. The effect of egomotion is mitigated by estimating the temporal geometry transformation and exploiting it to update visual representations. Moreover, we propose a cascaded feature enhancement module to tackle the second issue. It improves cross-modal localization robustness by disentangling visually-indicated audio representation. During training, we take advantage of the naturally available audio-visual temporal synchronization as the free'' self-supervision to avoid costly labeling. We also annotate and create the Epic Sounding Object dataset for evaluation purposes. Extensive experiments show that our method achieves state-of-the-art localization performance in egocentric videos and can be generalized to diverse audio-visual scenes.

【2】 Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI标题：语音合成中的音频扩散模型：产生式人工智能中文语转换与语音增强研究综述链接：http://arxiv.org/pdf/2303.13336v1

作者：Chenshuang Zhang,Chaoning Zhang,Sheng Zheng,Mengchun Zhang,Maryam Qamar,Sung-Ho Bae,In So Kweon机构：Kyung Hee University, Beijing Institute of Technology备注：18 pages摘要：生成式AI在各个领域都表现出了令人印象深刻的性能，其中语音合成是一个有趣的方向。随着扩散模型作为最流行的生成模型，许多工作已经尝试了两个积极的任务：文本到语音和语音增强。这项工作进行了调查音频扩散模型，这是补充现有的调查，要么缺乏基于扩散的语音合成的最新进展，或突出了在多个领域应用扩散模型的全貌。具体来说，本文首先简要介绍了音频和扩散模型的背景。对于文语转换任务，我们根据采用扩散模型的阶段将方法分为三类：声学模型、声码器和端到端框架。此外，我们分类的各种语音增强任务，无论是某些信号被删除或添加到输入语音。本综述还包括实验结果的比较和讨论。摘要：Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction. With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.

【3】 Frame-Level Multi-Label Playing Technique Detection Using Multi-Scale Network and Self-Attention Mechanism标题：基于多尺度网络和自关注机制的帧级多标签播放技术检测链接：http://arxiv.org/pdf/2303.13272v1

作者：Dichucheng Li,Mingjin Che,Wenwu Meng,Yulun Wu,Yi Yu,Fan Xia,Wei Li机构：School of Computer Science and Technology, Fudan University, Shanghai, China, College of Experimental Art, Sichuan Conservatory of Music, Sichuan, China, Digital Content and Media Sciences Research Division, National Institute of Informatics (NII), Japan备注：Accepted to ICASSP 2023摘要：乐器演奏技术是音乐表现的关键要素。然而，大多数用于IPT检测的现有工作仅涉及单声道音乐信号，而很少有人在具有重叠IPT或混合IPT的复调乐器独奏作品中检测IPT。在本文中，我们制定了一个框架级的多标签分类问题，并将其应用到古筝，中国弹拨乐器。我们创建了一个新的数据集Guzheng _Tech99，其中包含古筝录音以及每个音符的开始、偏移、音高和IPT注释。由于不同的IPT的长度有很大的不同，我们提出了一种新的方法来解决这个问题，使用多尺度网络和自我注意。多尺度网络从不同尺度提取特征，并且应用于最粗尺度处的特征图的自注意机制进一步增强了远程特征提取。我们的方法优于现有的作品由一个大的利润率，表明其有效性IPT检测。摘要：Instrument playing technique (IPT) is a key element of musical presentation. However, most of the existing works for IPT detection only concern monophonic music signals, yet little has been done to detect IPTs in polyphonic instrumental solo pieces with overlapping IPTs or mixed IPTs. In this paper, we formulate it as a frame-level multi-label classification problem and apply it to Guzheng, a Chinese plucked string instrument. We create a new dataset, Guzheng _Tech99, containing Guzheng recordings and onset, offset, pitch, IPT annotations of each note. Because different IPTs vary a lot in their lengths, we propose a new method to solve this problem using multi-scale network and self-attention. The multi-scale network extracts features from different scales, and the self-attention mechanism applied to the feature maps at the coarsest scale further enhances the long-range feature extraction. Our approach outperforms existing works by a large margin, indicating its effectiveness in IPT detection.

【4】 Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognit标题：超越通用转换器：在转换器中利用适配器实现块重用以实现自动语音识别链接：http://arxiv.org/pdf/2303.13072v1

作者：Haoyu Tang,Zhaoyi Liu,Chang Zeng,Xinfeng Li机构：Department of Electronic Systems, NTNU, Trondheim, Norway, Computer Science, KU Leuven, Leuven, Belgium ,National Institute of Informatics, Tokyo, Japan, Electrical Engineering, Zhejiang University, Hangzhou, China摘要：基于转换器的模型最近在端到端（E2 E）自动语音识别（ASR）的应用中取得了显著的成就。借助基于Transformer的模型，可以在智能设备上部署E2 E ASR系统。而这些模型仍然具有需要大量模型参数的缺点。为了克服通用Transformer模型在边缘设备上应用ASR的缺点，我们提出了一种解决方案，该解决方案可以重用Transformer模型中的块用于小占用空间ASR系统的场合，这满足了在不影响识别准确性的情况下适应资源限制的目标。具体而言，我们设计了一种新的块重用策略的语音Transformer（BRST），以提高参数的有效性，并提出了一个适配器模块（ADM），可以产生一个紧凑的和自适应的模型，只有几个额外的可训练参数伴随着每个重用块。在AISHELL-1公共语料库上进行实验，结果表明，该方法在不使用ADM和使用ADM的情况下，仅使用7.6M 8.3M参数，字符错误率（CER）分别为9.3% 6.63%.此外，我们还对ADM在一般的块重用方法中的作用进行了深入的分析。摘要：Transformer-based models have recently made significant achievements in the application of end-to-end (E2E) automatic speech recognition (ASR). It is possible to deploy the E2E ASR system on smart devices with the help of Transformer-based models. While these models still have the disadvantage of requiring a large number of model parameters. To overcome the drawback of universal Transformer models for the application of ASR on edge devices, we propose a solution that can reuse the block in Transformer models for the occasion of the small footprint ASR system, which meets the objective of accommodating resource limitations without compromising recognition accuracy. Specifically, we design a novel block-reusing strategy for speech Transformer (BRST) to enhance the effectiveness of parameters and propose an adapter module (ADM) that can produce a compact and adaptable model with only a few additional trainable parameters accompanying each reusing block. We conducted an experiment with the proposed method on the public AISHELL-1 corpus, and the results show that the proposed approach achieves the character error rate (CER) of 9.3% 6.63% with only 7.6M 8.3M parameters without and with the ADM, respectively. In addition, we also make a deeper analysis to show the effect of ADM in the general block-reusing method.

【5】 LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models标题：LMCodec：一种基于因果变换模型的低码率语音编解码器链接：http://arxiv.org/pdf/2303.12984v1

作者：Teerapat Jenrungrot,Michael Chinen,W. Bastiaan Kleijn,Jan Skoglund,Zalán Borsos,Neil Zeghidour,Marco Tagliasacchi机构：University of Washington, Seattle, Google, School of Engineering and Computer Science, Victoria University of Wellington备注：5 pages, accepted to ICASSP 2023, project page: this https URL摘要：我们介绍LMCodec，因果神经语音编解码器，提供高品质的音频在非常低的比特率。该系统的骨干是一个因果卷积编解码器，它使用残差矢量量化将音频编码成一个由粗到细的令牌层次。LMCodec训练Transformer语言模型，以生成方式从粗糙令牌预测精细令牌，从而允许传输更少的代码。第二Transformer预测给定过去传输的代码的下一个代码的不确定性，并且用于执行条件熵编码。MUSHRA主观测试进行，并表明，质量是可比的参考编解码器在更高的比特率。示例音频可在www.example.com上获得https: mjenrungrot.github.io chrome-media-audio-papers publications lmcodec。摘要：We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates. Example audio is available at https: mjenrungrot.github.io chrome-media-audio-papers publications lmcodec.

【6】 Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline标题：未剪辑视频中的密集本地化视听事件：大规模基准和基准链接：http://arxiv.org/pdf/2303.12930v1

作者：Tiantian Geng,Teng Wang,Jinming Duan,Runmin Cong,Feng Zheng机构：Southern University of Science and Technology ,University of Birmingham, The University of Hong Kong ,Shandong University ,Peng Cheng Laboratory备注：Accepted by CVPR2023摘要：现有的视听事件定位（AVE）处理手动修剪的视频，其中每个视频中只有单个实例。然而，这种设置是不现实的，因为自然视频通常包含具有不同类别的许多视听事件。为了更好地适应现实生活中的应用，在本文中，我们专注于密集本地化的视听事件，其目的是共同定位和识别所有的视听事件发生在一个未修剪的视频的任务。这个问题是具有挑战性的，因为它需要细粒度的视听场景和上下文理解。为了解决这个问题，我们引入了第一个Untrimmed Audio-Visual（UnAV-100）数据集，其中包含10 K未修剪的视频，其中包含超过30 K的视听事件。每个视频平均有2.8个视听事件，这些事件通常彼此相关，并且可能像在现实生活场景中一样共同发生。接下来，我们使用一个新的基于学习的框架来制定任务，该框架能够完全集成音频和视觉模式，以定位具有各种长度的视听事件，并在单次传递中捕获它们之间的依赖关系。大量的实验证明了我们的方法的有效性，以及多尺度跨模态感知和依赖建模这项任务的意义。摘要：Existing audio-visual event localization (AVE) handles manually trimmed videos with only a single instance in each of them. However, this setting is unrealistic as natural videos often contain numerous audio-visual events with different categories. To better adapt to real-life applications, in this paper we focus on the task of dense-localizing audio-visual events, which aims to jointly localize and recognize all audio-visual events occurring in an untrimmed video. The problem is challenging as it requires fine-grained audio-visual scene and context understanding. To tackle this problem, we introduce the first Untrimmed Audio-Visual (UnAV-100) dataset, which contains 10K untrimmed videos with over 30K audio-visual events. Each video has 2.8 audio-visual events on average, and the events are usually related to each other and might co-occur as in real-life scenes. Next, we formulate the task using a new learning-based framework, which is capable of fully integrating audio and visual modalities to localize audio-visual events with various lengths and capture dependencies between them in a single pass. Extensive experiments demonstrate the effectiveness of our method as well as the significance of multi-scale cross-modal perception and dependency modeling for this task.

【7】 Better Together: Dialogue Separation and Voice Activity Detection for Audio Personalization in TV标题：更好的结合：用于电视音频个性化的对话分离和语音活动检测链接：http://arxiv.org/pdf/2303.13453v1

作者：Matteo Torcoli,Emanuël A. P. Habets机构：International Audio Laboratories Erlangen∗, Am Wolfsmantel , Erlangen, Germany.备注：Paper accepted to the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), Rhodes, Greece摘要：在TV服务中，对话级别个性化是满足用户偏好和需求的关键。当对话和背景声音不能从制作阶段单独获得时，对话分离（DS）可以估计它们以实现个性化。DS被证明为最终用户提供了明显的好处。然而，估计的信号并不完美，并且可能引入一些泄漏。这是不希望的，尤其是在没有对话的段落中。我们建议结合DS和语音活动检测（VAD），最近提出的电视音频。当它们的组合表明对话不活动时，对话估计中泄漏的背景分量被重新分配给背景估计。无对话信号的音频质量的明显改善，而没有性能下降时，对话是活跃的。还生成具有改进的检测精度的后处理VAD估计。结果表明，DS和VAD可以相互促进，更好地结合使用。摘要：In TV services, dialogue level personalization is key to meeting user preferences and needs. When dialogue and background sounds are not separately available from the production stage, Dialogue Separation (DS) can estimate them to enable personalization. DS was shown to provide clear benefits for the end user. Still, the estimated signals are not perfect, and some leakage can be introduced. This is undesired, especially during passages without dialogue. We propose to combine DS and Voice Activity Detection (VAD), both recently proposed for TV audio. When their combination suggests dialogue inactivity, background components leaking in the dialogue estimate are reassigned to the background estimate. A clear improvement of the audio quality is shown for dialogue-free signals, without performance drops when dialogue is active. A post-processed VAD estimate with improved detection accuracy is also generated. It is concluded that DS and VAD can improve each other and are better used together.

【8】 Pyramid Multi-branch Fusion DCNN with Multi-Head Self-Attention for Mandarin Speech Recognition标题：金字塔多分支融合多头自关注DCNN用于汉语语音识别链接：http://arxiv.org/pdf/2303.13243v1

作者：Kai Liu,Hailiang Xiong,Gangqiang Yang,Zhengfeng Du,Yewen Cao,Danyal Shah摘要：作为自动语音识别的主要分支之一，基于注意力的模型极大地提高了模型的特征表示能力。特别地，在注意力中采用了多头机制，希望在不同的注意力子空间中学习更多方面的语音特征。对于复杂语言的语音识别，一方面，较小的头部尺寸将导致可学习方面的明显不足。另一方面，当我们增加头部的数量时，我们需要降低每个子空间的维度以保持整体特征空间的大小不变，这将显著削弱表示每个子空间的特征的能力。因此，本文探讨了如何使用一个小的注意力子空间来表示完整的语音特征，同时确保许多头部。在这项工作中，我们提出了一种新的神经网络架构，即金字塔多分支融合DCNN与多头自注意。该架构的灵感来自于DCNN，它使用DCNN的多个分支来提取不同感受野下的输入语音的特征。为了减少参数的数量，每两个分支合并，直到所有分支合并为一个。因此，它的形状就像旋转90度的金字塔。我们证明，在Aishell-1，一个广泛使用的普通话语音数据集，我们的模型实现了字符错误率（CER）的6.45%的测试集。摘要：As one of the major branches of automatic speech recognition, attention-based models greatly improves the feature representation ability of the model. In particular, the multi-head mechanism is employed in the attention, hoping to learn speech features of more aspects in different attention subspaces. For speech recognition of complex languages, on the one hand, a small head size will lead to an obvious shortage of learnable aspects. On the other hand, we need to reduce the dimension of each subspace to keep the size of the overall feature space unchanged when we increase the number of heads, which will significantly weaken the ability to represent the feature of each subspace. Therefore, this paper explores how to use a small attention subspace to represent complete speech features while ensuring many heads. In this work we propose a novel neural network architecture, namely, pyramid multi-branch fusion DCNN with multi-head self-attention. The proposed architecture is inspired by Dilated Convolution Neural Networks (DCNN), it uses multiple branches with DCNN to extract the feature of the input speech under different receptive fields. To reduce the number of parameters, every two branches are merged until all the branches are merged into one. Thus, its shape is like a pyramid rotated 90 degrees. We demonstrate that on Aishell-1, a widely used Mandarin speech dataset, our model achieves a character error rate (CER) of 6.45% on the test sets.

【9】 Weighted Pressure and Mode Matching for Sound Field Reproduction: Theoretical and Experimental Comparisons标题：声场重现的加权声压和模式匹配：理论和实验比较链接：http://arxiv.org/pdf/2303.13027v1

作者：Shoichi Koyama,Keisuke Kimura,Natsuki Ueno机构：Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan, Tokyo Metropolitan University, Tokyo, Japan备注：Accepted to Journal of Audio Engineering Society, Special Issue on Spatial Audio摘要：从理论和实验两个方面对两种声场再现方法--加权声压匹配法和加权模式匹配法进行了比较。加权压力和模式匹配分别是常规压力和模式匹配的推广。这两种方法都是通过在压力和模式匹配中引入加权矩阵而得到的。加权压力匹配中的加权矩阵基于来自控制点的离散集合处的压力的声场的核插值来定义。在加权模式匹配中，加权矩阵由球面波函数的区域积分定义。从理论上证明了加权压力匹配是由压力观测值估计膨胀系数的无限维调和分析的加权模式匹配的一种特殊情况。通过实验讨论了两种方法的区别。摘要：Two sound field reproduction methods, weighted pressure matching and weighted mode matching, are theoretically and experimentally compared. The weighted pressure and mode matching are a generalization of conventional pressure and mode matching, respectively. Both methods are derived by introducing a weighting matrix in the pressure and mode matching. The weighting matrix in the weighted pressure matching is defined on the basis of the kernel interpolation of the sound field from pressure at a discrete set of control points. In the weighted mode matching, the weighting matrix is defined by a regional integration of spherical wavefunctions. It is theoretically shown that the weighted pressure matching is a special case of the weighted mode matching by infinite-dimensional harmonic analysis for estimating expansion coefficients from pressure observations. The difference between the two methods are discussed through experiments.

【10】 Self-supervised Learning with Speech Modulation Dropout标题：语音调制丢失情况下的自监督学习链接：http://arxiv.org/pdf/2303.12908v1

作者：Samik Sadhu,Hynek Hermansky机构：Center for Language and Speech Processing, Johns Hopkins University, USA, Human Language Technology Center of Excellence, Johns Hopkins University, USA摘要：我们证明，训练一个基于多头自我注意的深度网络来预测在1.5秒的语音段中删除的、信息密集的2-8 Hz语音调制是一种有效的方法，可以让机器学习使用时域上下文信息来提取语音调制。我们的工作表明，一旦在大量未标记的数据上训练，自我注意层的输出随时间变化，调制峰值为4 Hz。这些预先训练的层可以用于初始化自动语音识别系统的部分，以大大减少其对标记语音数据的依赖。摘要：We show that training a multi-headed self-attention-based deep network to predict deleted, information-dense 2-8 Hz speech modulations over a 1.5-second section of a speech utterance is an effective way to make machines learn to extract speech modulations using time-domain contextual information. Our work exhibits that, once trained on large volumes of unlabelled data, the outputs of the self-attention layers vary in time with a modulation peak at 4 Hz. These pre-trained layers can be used to initialize parts of an Automatic Speech Recognition system to reduce its reliance on labeled speech data greatly.
eess.AS音频处理

【1】 Better Together: Dialogue Separation and Voice Activity Detection for Audio Personalization in TV标题：更好的结合：用于电视音频个性化的对话分离和语音活动检测链接：http://arxiv.org/pdf/2303.13453v1

*与cs.SD语音【7】为同一篇

作者：Matteo Torcoli,Emanuël A. P. Habets机构：International Audio Laboratories Erlangen∗, Am Wolfsmantel , Erlangen, Germany.备注：Paper accepted to the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023), Rhodes, Greece摘要：在TV服务中，对话级别个性化是满足用户偏好和需求的关键。当对话和背景声音不能从制作阶段单独获得时，对话分离（DS）可以估计它们以实现个性化。DS被证明为最终用户提供了明显的好处。然而，估计的信号并不完美，并且可能引入一些泄漏。这是不希望的，尤其是在没有对话的段落中。我们建议结合DS和语音活动检测（VAD），最近提出的电视音频。当它们的组合表明对话不活动时，对话估计中泄漏的背景分量被重新分配给背景估计。无对话信号的音频质量的明显改善，而没有性能下降时，对话是活跃的。还生成具有改进的检测精度的后处理VAD估计。结果表明，DS和VAD可以相互促进，更好地结合使用。摘要：In TV services, dialogue level personalization is key to meeting user preferences and needs. When dialogue and background sounds are not separately available from the production stage, Dialogue Separation (DS) can estimate them to enable personalization. DS was shown to provide clear benefits for the end user. Still, the estimated signals are not perfect, and some leakage can be introduced. This is undesired, especially during passages without dialogue. We propose to combine DS and Voice Activity Detection (VAD), both recently proposed for TV audio. When their combination suggests dialogue inactivity, background components leaking in the dialogue estimate are reassigned to the background estimate. A clear improvement of the audio quality is shown for dialogue-free signals, without performance drops when dialogue is active. A post-processed VAD estimate with improved detection accuracy is also generated. It is concluded that DS and VAD can improve each other and are better used together.

【2】 Adaptive Endpointing with Deep Contextual Multi-armed Bandits标题：基于深度上下文多臂环的自适应终点链接：http://arxiv.org/pdf/2303.13407v1

作者：Do June Min,Andreas Stolcke,Anirudh Raju,Colin Vaz,Di He,Venkatesh Ravichandran,Viet Anh Trinh机构：University of Michigan, Amazon Alexa AI, USA摘要：None摘要：Current endpointing (EP) solutions learn in a supervised framework, which does not allow the model to incorporate feedback and improve in an online setting. Also, it is a common practice to utilize costly grid-search to find the best configuration for an endpointing model. In this paper, we aim to provide a solution for adaptive endpointing by proposing an efficient method for choosing an optimal endpointing configuration given utterance-level audio features in an online setting, while avoiding hyperparameter grid-search. Our method does not require ground truth labels, and only uses online learning from reward signals without requiring annotated labels. Specifically, we propose a deep contextual multi-armed bandit-based approach, which combines the representational power of neural networks with the action exploration behavior of Thompson modeling algorithms. We compare our approach to several baselines, and show that our deep bandit models also succeed in reducing early cutoff errors while maintaining low latency.

【3】 Pyramid Multi-branch Fusion DCNN with Multi-Head Self-Attention for Mandarin Speech Recognition标题：金字塔多分支融合多头自关注DCNN用于汉语语音识别链接：http://arxiv.org/pdf/2303.13243v1

*与cs.SD语音【8】为同一篇

作者：Kai Liu,Hailiang Xiong,Gangqiang Yang,Zhengfeng Du,Yewen Cao,Danyal Shah摘要：作为自动语音识别的主要分支之一，基于注意力的模型极大地提高了模型的特征表示能力。特别地，在注意力中采用了多头机制，希望在不同的注意力子空间中学习更多方面的语音特征。对于复杂语言的语音识别，一方面，较小的头部尺寸将导致可学习方面的明显不足。另一方面，当我们增加头部的数量时，我们需要降低每个子空间的维度以保持整体特征空间的大小不变，这将显著削弱表示每个子空间的特征的能力。因此，本文探讨了如何使用一个小的注意力子空间来表示完整的语音特征，同时确保许多头部。在这项工作中，我们提出了一种新的神经网络架构，即金字塔多分支融合DCNN与多头自注意。该架构的灵感来自于DCNN，它使用DCNN的多个分支来提取不同感受野下的输入语音的特征。为了减少参数的数量，每两个分支合并，直到所有分支合并为一个。因此，它的形状就像旋转90度的金字塔。我们证明，在Aishell-1，一个广泛使用的普通话语音数据集，我们的模型实现了字符错误率（CER）的6.45%的测试集。摘要：As one of the major branches of automatic speech recognition, attention-based models greatly improves the feature representation ability of the model. In particular, the multi-head mechanism is employed in the attention, hoping to learn speech features of more aspects in different attention subspaces. For speech recognition of complex languages, on the one hand, a small head size will lead to an obvious shortage of learnable aspects. On the other hand, we need to reduce the dimension of each subspace to keep the size of the overall feature space unchanged when we increase the number of heads, which will significantly weaken the ability to represent the feature of each subspace. Therefore, this paper explores how to use a small attention subspace to represent complete speech features while ensuring many heads. In this work we propose a novel neural network architecture, namely, pyramid multi-branch fusion DCNN with multi-head self-attention. The proposed architecture is inspired by Dilated Convolution Neural Networks (DCNN), it uses multiple branches with DCNN to extract the feature of the input speech under different receptive fields. To reduce the number of parameters, every two branches are merged until all the branches are merged into one. Thus, its shape is like a pyramid rotated 90 degrees. We demonstrate that on Aishell-1, a widely used Mandarin speech dataset, our model achieves a character error rate (CER) of 6.45% on the test sets.

【4】 Weighted Pressure and Mode Matching for Sound Field Reproduction: Theoretical and Experimental Comparisons标题：声场重现的加权声压和模式匹配：理论和实验比较链接：http://arxiv.org/pdf/2303.13027v1

*与cs.SD语音【9】为同一篇

作者：Shoichi Koyama,Keisuke Kimura,Natsuki Ueno机构：Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan, Tokyo Metropolitan University, Tokyo, Japan备注：Accepted to Journal of Audio Engineering Society, Special Issue on Spatial Audio摘要：从理论和实验两个方面对两种声场再现方法--加权声压匹配法和加权模式匹配法进行了比较。加权压力和模式匹配分别是常规压力和模式匹配的推广。这两种方法都是通过在压力和模式匹配中引入加权矩阵而得到的。加权压力匹配中的加权矩阵基于来自控制点的离散集合处的压力的声场的核插值来定义。在加权模式匹配中，加权矩阵由球面波函数的区域积分定义。从理论上证明了加权压力匹配是由压力观测值估计膨胀系数的无限维调和分析的加权模式匹配的一种特殊情况。通过实验讨论了两种方法的区别。摘要：Two sound field reproduction methods, weighted pressure matching and weighted mode matching, are theoretically and experimentally compared. The weighted pressure and mode matching are a generalization of conventional pressure and mode matching, respectively. Both methods are derived by introducing a weighting matrix in the pressure and mode matching. The weighting matrix in the weighted pressure matching is defined on the basis of the kernel interpolation of the sound field from pressure at a discrete set of control points. In the weighted mode matching, the weighting matrix is defined by a regional integration of spherical wavefunctions. It is theoretically shown that the weighted pressure matching is a special case of the weighted mode matching by infinite-dimensional harmonic analysis for estimating expansion coefficients from pressure observations. The difference between the two methods are discussed through experiments.

【5】 Self-supervised Learning with Speech Modulation Dropout标题：语音调制丢失情况下的自监督学习链接：http://arxiv.org/pdf/2303.12908v1

*与cs.SD语音【10】为同一篇

作者：Samik Sadhu,Hynek Hermansky机构：Center for Language and Speech Processing, Johns Hopkins University, USA, Human Language Technology Center of Excellence, Johns Hopkins University, USA摘要：我们证明，训练一个基于多头自我注意的深度网络来预测在1.5秒的语音段中删除的、信息密集的2-8 Hz语音调制是一种有效的方法，可以让机器学习使用时域上下文信息来提取语音调制。我们的工作表明，一旦在大量未标记的数据上训练，自我注意层的输出随时间变化，调制峰值为4 Hz。这些预先训练的层可以用于初始化自动语音识别系统的部分，以大大减少其对标记语音数据的依赖。摘要：We show that training a multi-headed self-attention-based deep network to predict deleted, information-dense 2-8 Hz speech modulations over a 1.5-second section of a speech utterance is an effective way to make machines learn to extract speech modulations using time-domain contextual information. Our work exhibits that, once trained on large volumes of unlabelled data, the outputs of the self-attention layers vary in time with a modulation peak at 4 Hz. These pre-trained layers can be used to initialize parts of an Automatic Speech Recognition system to reduce its reliance on labeled speech data greatly.

【6】 Egocentric Audio-Visual Object Localization标题：以自我为中心的视听对象定位链接：http://arxiv.org/pdf/2303.13471v1

*与cs.SD语音【1】为同一篇

作者：Chao Huang,Yapeng Tian,Anurag Kumar,Chenliang Xu机构：University of Rochester,Meta Reality Labs Research备注：Accepted by CVPR 2023摘要：人类自然地通过以第一人称视角统一声音和视觉来感知周围的场景。同样地，机器通过从自我中心的角度学习多感官输入来接近人类智能。在本文中，我们探讨了具有挑战性的自我中心的视听对象定位任务，并观察到：1）自我运动普遍存在于第一人称录音，即使在很短的时间内;2）当佩戴者转移他们的注意力时，可以创建视野外声音分量。为了解决第一个问题，我们提出了一个几何感知的时间聚合模块来处理的egomotion明确。通过估计时间几何变换并利用它来更新视觉表示来减轻自运动的影响。此外，我们提出了一个级联的特征增强模块来解决第二个问题。它通过解开视觉指示的音频表示来提高跨模态定位鲁棒性。在训练过程中，我们利用自然可用的视听时间同步作为“自由”的自我监督，以避免昂贵的标记。我们还为评估目的注释和创建Epic Sounding Object数据集。大量的实验表明，我们的方法在以自我为中心的视频中实现了最先进的定位性能，并可以推广到不同的视听场景。摘要：Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with multisensory inputs from an egocentric perspective. In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention. To address the first problem, we propose a geometry-aware temporal aggregation module to handle the egomotion explicitly. The effect of egomotion is mitigated by estimating the temporal geometry transformation and exploiting it to update visual representations. Moreover, we propose a cascaded feature enhancement module to tackle the second issue. It improves cross-modal localization robustness by disentangling visually-indicated audio representation. During training, we take advantage of the naturally available audio-visual temporal synchronization as the free'' self-supervision to avoid costly labeling. We also annotate and create the Epic Sounding Object dataset for evaluation purposes. Extensive experiments show that our method achieves state-of-the-art localization performance in egocentric videos and can be generalized to diverse audio-visual scenes.                 

【7】 W2KPE: Keyphrase Extraction with Word-Word Relation标题：W2KPE：基于词-词关系的关键词抽取链接：http://arxiv.org/pdf/2303.13463v1

作者：Wen Cheng,Shichen Dong,Wei Wang机构：Nanjing University, Nanjing, China摘要：本文介绍了我们提交给ICASSP 2023 MUG Challenge Track 4，Keyphrase Extraction，旨在从会议材料中提取与会议主题最相关的关键词。我们将挑战建模为单类命名实体识别任务，并开发了在挑战中获得更好性能的技术：对于数据预处理，我们在分词之后对分裂的关键词进行编码。此外，我们通过将多个预处理的句子融合到一个片段中来增加模型一次可以接受的输入信息量。我们用多类焦点损失代替损失函数来解决关键词的稀疏性。此外，我们对关键词的每次出现进行评分，并添加一个额外的输出层来适应关键词排名的分数。执行详尽的评估以找到分词工具、预训练的嵌入模型和对应的超参数的最佳组合。有了这些建议，我们在最终测试集上获得了45.04分。摘要：This paper describes our submission to ICASSP 2023 MUG Challenge Track 4, Keyphrase Extraction, which aims to extract keyphrases most relevant to the conference theme from conference materials. We model the challenge as a single-class Named Entity Recognition task and developed techniques for better performance on the challenge: For the data preprocessing, we encode the split keyphrases after word segmentation. In addition, we increase the amount of input information that the model can accept at one time by fusing multiple preprocessed sentences into one segment. We replace the loss function with the multi-class focal loss to address the sparseness of keyphrases. Besides, we score each appearance of keyphrases and add an extra output layer to fit the score to rank keyphrases. Exhaustive evaluations are performed to find the best combination of the word segmentation tool, the pre-trained embedding model, and the corresponding hyperparameters. With these proposals, we scored 45.04 on the final test set.

【8】 Audio Diffusion Model for Speech Synthesis: A Survey on Text To Speech and Speech Enhancement in Generative AI标题：语音合成中的音频扩散模型：产生式人工智能中文语转换与语音增强研究综述链接：http://arxiv.org/pdf/2303.13336v1

*与cs.SD语音【2】为同一篇

作者：Chenshuang Zhang,Chaoning Zhang,Sheng Zheng,Mengchun Zhang,Maryam Qamar,Sung-Ho Bae,In So Kweon机构：Kyung Hee University, Beijing Institute of Technology备注：18 pages摘要：生成式AI在各个领域都表现出了令人印象深刻的性能，其中语音合成是一个有趣的方向。随着扩散模型作为最流行的生成模型，许多工作已经尝试了两个积极的任务：文本到语音和语音增强。这项工作进行了调查音频扩散模型，这是补充现有的调查，要么缺乏基于扩散的语音合成的最新进展，或突出了在多个领域应用扩散模型的全貌。具体来说，本文首先简要介绍了音频和扩散模型的背景。对于文语转换任务，我们根据采用扩散模型的阶段将方法分为三类：声学模型、声码器和端到端框架。此外，我们分类的各种语音增强任务，无论是某些信号被删除或添加到输入语音。本综述还包括实验结果的比较和讨论。摘要：Generative AI has demonstrated impressive performance in various fields, among which speech synthesis is an interesting direction. With the diffusion model as the most popular generative model, numerous works have attempted two active tasks: text to speech and speech enhancement. This work conducts a survey on audio diffusion model, which is complementary to existing surveys that either lack the recent progress of diffusion-based speech synthesis or highlight an overall picture of applying diffusion model in multiple fields. Specifically, this work first briefly introduces the background of audio and diffusion model. As for the text-to-speech task, we divide the methods into three categories based on the stage where diffusion model is adopted: acoustic model, vocoder and end-to-end framework. Moreover, we categorize various speech enhancement tasks by either certain signals are removed or added into the input speech. Comparisons of experimental results and discussions are also covered in this survey.

【9】 Frame-Level Multi-Label Playing Technique Detection Using Multi-Scale Network and Self-Attention Mechanism标题：基于多尺度网络和自关注机制的帧级多标签播放技术检测链接：http://arxiv.org/pdf/2303.13272v1

*与cs.SD语音【3】为同一篇

作者：Dichucheng Li,Mingjin Che,Wenwu Meng,Yulun Wu,Yi Yu,Fan Xia,Wei Li机构：School of Computer Science and Technology, Fudan University, Shanghai, China, College of Experimental Art, Sichuan Conservatory of Music, Sichuan, China, Digital Content and Media Sciences Research Division, National Institute of Informatics (NII), Japan备注：Accepted to ICASSP 2023摘要：乐器演奏技术是音乐表现的关键要素。然而，大多数用于IPT检测的现有工作仅涉及单声道音乐信号，而很少有人在具有重叠IPT或混合IPT的复调乐器独奏作品中检测IPT。在本文中，我们制定了一个框架级的多标签分类问题，并将其应用到古筝，中国弹拨乐器。我们创建了一个新的数据集Guzheng _Tech99，其中包含古筝录音以及每个音符的开始、偏移、音高和IPT注释。由于不同的IPT的长度有很大的不同，我们提出了一种新的方法来解决这个问题，使用多尺度网络和自我注意。多尺度网络从不同尺度提取特征，并且应用于最粗尺度处的特征图的自注意机制进一步增强了远程特征提取。我们的方法优于现有的作品由一个大的利润率，表明其有效性IPT检测。摘要：Instrument playing technique (IPT) is a key element of musical presentation. However, most of the existing works for IPT detection only concern monophonic music signals, yet little has been done to detect IPTs in polyphonic instrumental solo pieces with overlapping IPTs or mixed IPTs. In this paper, we formulate it as a frame-level multi-label classification problem and apply it to Guzheng, a Chinese plucked string instrument. We create a new dataset, Guzheng _Tech99, containing Guzheng recordings and onset, offset, pitch, IPT annotations of each note. Because different IPTs vary a lot in their lengths, we propose a new method to solve this problem using multi-scale network and self-attention. The multi-scale network extracts features from different scales, and the self-attention mechanism applied to the feature maps at the coarsest scale further enhances the long-range feature extraction. Our approach outperforms existing works by a large margin, indicating its effectiveness in IPT detection.

【10】 Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognit标题：超越通用转换器：在转换器中利用适配器实现块重用以实现自动语音识别链接：http://arxiv.org/pdf/2303.13072v1

*与cs.SD语音【4】为同一篇

作者：Haoyu Tang,Zhaoyi Liu,Chang Zeng,Xinfeng Li机构：Department of Electronic Systems, NTNU, Trondheim, Norway, Computer Science, KU Leuven, Leuven, Belgium ,National Institute of Informatics, Tokyo, Japan, Electrical Engineering, Zhejiang University, Hangzhou, China摘要：基于转换器的模型最近在端到端（E2 E）自动语音识别（ASR）的应用中取得了显著的成就。借助基于Transformer的模型，可以在智能设备上部署E2 E ASR系统。而这些模型仍然具有需要大量模型参数的缺点。为了克服通用Transformer模型在边缘设备上应用ASR的缺点，我们提出了一种解决方案，该解决方案可以重用Transformer模型中的块用于小占用空间ASR系统的场合，这满足了在不影响识别准确性的情况下适应资源限制的目标。具体而言，我们设计了一种新的块重用策略的语音Transformer（BRST），以提高参数的有效性，并提出了一个适配器模块（ADM），可以产生一个紧凑的和自适应的模型，只有几个额外的可训练参数伴随着每个重用块。在AISHELL-1公共语料库上进行实验，结果表明，该方法在不使用ADM和使用ADM的情况下，仅使用7.6M 8.3M参数，字符错误率（CER）分别为9.3% 6.63%.此外，我们还对ADM在一般的块重用方法中的作用进行了深入的分析。摘要：Transformer-based models have recently made significant achievements in the application of end-to-end (E2E) automatic speech recognition (ASR). It is possible to deploy the E2E ASR system on smart devices with the help of Transformer-based models. While these models still have the disadvantage of requiring a large number of model parameters. To overcome the drawback of universal Transformer models for the application of ASR on edge devices, we propose a solution that can reuse the block in Transformer models for the occasion of the small footprint ASR system, which meets the objective of accommodating resource limitations without compromising recognition accuracy. Specifically, we design a novel block-reusing strategy for speech Transformer (BRST) to enhance the effectiveness of parameters and propose an adapter module (ADM) that can produce a compact and adaptable model with only a few additional trainable parameters accompanying each reusing block. We conducted an experiment with the proposed method on the public AISHELL-1 corpus, and the results show that the proposed approach achieves the character error rate (CER) of 9.3% 6.63% with only 7.6M 8.3M parameters without and with the ADM, respectively. In addition, we also make a deeper analysis to show the effect of ADM in the general block-reusing method.

【11】 LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models标题：LMCodec：一种基于因果变换模型的低码率语音编解码器链接：http://arxiv.org/pdf/2303.12984v1

*与cs.SD语音【5】为同一篇

作者：Teerapat Jenrungrot,Michael Chinen,W. Bastiaan Kleijn,Jan Skoglund,Zalán Borsos,Neil Zeghidour,Marco Tagliasacchi机构：University of Washington, Seattle, Google, School of Engineering and Computer Science, Victoria University of Wellington备注：5 pages, accepted to ICASSP 2023, project page: this https URL摘要：我们介绍LMCodec，因果神经语音编解码器，提供高品质的音频在非常低的比特率。该系统的骨干是一个因果卷积编解码器，它使用残差矢量量化将音频编码成一个由粗到细的令牌层次。LMCodec训练Transformer语言模型，以生成方式从粗糙令牌预测精细令牌，从而允许传输更少的代码。第二Transformer预测给定过去传输的代码的下一个代码的不确定性，并且用于执行条件熵编码。MUSHRA主观测试进行，并表明，质量是可比的参考编解码器在更高的比特率。示例音频可在www.example.com上获得https: mjenrungrot.github.io chrome-media-audio-papers publications lmcodec。摘要：We introduce LMCodec, a causal neural speech codec that provides high quality audio at very low bitrates. The backbone of the system is a causal convolutional codec that encodes audio into a hierarchy of coarse-to-fine tokens using residual vector quantization. LMCodec trains a Transformer language model to predict the fine tokens from the coarse ones in a generative fashion, allowing for the transmission of fewer codes. A second Transformer predicts the uncertainty of the next codes given the past transmitted codes, and is used to perform conditional entropy coding. A MUSHRA subjective test was conducted and shows that the quality is comparable to reference codecs at higher bitrates. Example audio is available at https: mjenrungrot.github.io chrome-media-audio-papers publications lmcodec.

【12】 Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline标题：未剪辑视频中的密集本地化视听事件：大规模基准和基准链接：http://arxiv.org/pdf/2303.12930v1

*与cs.SD语音【6】为同一篇

作者：Tiantian Geng,Teng Wang,Jinming Duan,Runmin Cong,Feng Zheng机构：Southern University of Science and Technology ,University of Birmingham, The University of Hong Kong ,Shandong University ,Peng Cheng Laboratory备注：Accepted by CVPR2023摘要：现有的视听事件定位（AVE）处理手动修剪的视频，其中每个视频中只有单个实例。然而，这种设置是不现实的，因为自然视频通常包含具有不同类别的许多视听事件。为了更好地适应现实生活中的应用，在本文中，我们专注于密集本地化的视听事件，其目的是共同定位和识别所有的视听事件发生在一个未修剪的视频的任务。这个问题是具有挑战性的，因为它需要细粒度的视听场景和上下文理解。为了解决这个问题，我们引入了第一个Untrimmed Audio-Visual（UnAV-100）数据集，其中包含10 K未修剪的视频，其中包含超过30 K的视听事件。每个视频平均有2.8个视听事件，这些事件通常彼此相关，并且可能像在现实生活场景中一样共同发生。接下来，我们使用一个新的基于学习的框架来制定任务，该框架能够完全集成音频和视觉模式，以定位具有各种长度的视听事件，并在单次传递中捕获它们之间的依赖关系。大量的实验证明了我们的方法的有效性，以及多尺度跨模态感知和依赖建模这项任务的意义。摘要：Existing audio-visual event localization (AVE) handles manually trimmed videos with only a single instance in each of them. However, this setting is unrealistic as natural videos often contain numerous audio-visual events with different categories. To better adapt to real-life applications, in this paper we focus on the task of dense-localizing audio-visual events, which aims to jointly localize and recognize all audio-visual events occurring in an untrimmed video. The problem is challenging as it requires fine-grained audio-visual scene and context understanding. To tackle this problem, we introduce the first Untrimmed Audio-Visual (UnAV-100) dataset, which contains 10K untrimmed videos with over 30K audio-visual events. Each video has 2.8 audio-visual events on average, and the events are usually related to each other and might co-occur as in real-life scenes. Next, we formulate the task using a new learning-based framework, which is capable of fully integrating audio and visual modalities to localize audio-visual events with various lengths and capture dependencies between them in a single pass. Extensive experiments demonstrate the effectiveness of our method as well as the significance of multi-scale cross-modal perception and dependency modeling for this task.

机器翻译，仅供参考