今日论文合集：cs.SD语音8篇，eess.AS音频处理7篇。
【1】 ASiT: Audio Spectrogram vIsion Transformer for General Audio Representation
标题：ASIT：用于通用音频表示的音频谱图视觉转换器
链接：https://arxiv.org/abs/2211.13189
作者：Sara Atito,Muhammad Awais,Wenwu Wang,Mark D Plumbley,Josef Kittler机构：CVSSP, University of Surrey, Surrey, UK, !摘要：Vision Transformers最初是为自然语言处理而开发的，由于其在学习远程关系方面的灵活性，最近在计算机视觉和音频社区中产生了极大的兴趣。受Transformers的数据饥饿特性和有限的标记数据的约束，大多数基于变换器的音频任务模型是从ImageNet预训练模型微调的，尽管自然图像域和音频域之间存在巨大差距。这就激发了对音频Transformers的自监督预训练的研究，该预训练减少了对大量标记数据的依赖，并且集中于提取音频谱图的简明表示。本文提出了一种新颖的自监督音频Transformer ASiT，它采用分组掩蔽模型学习和自提炼的方法来捕获局部和全局上下文信息.我们在音频和语音分类任务上评估我们的预训练模型，包括音频事件分类、关键词识别和说话人识别。我们进一步进行了全面的消融研究，包括对不同预训练策略的评价。提出的ASiT框架显著提高了所有任务的性能，并在五个音频和语音分类任务上设置了新的最先进的性能，优于最近的方法，包括使用额外数据集进行预训练的方法。代码和预先训练的权重将向科学界公开。摘要：Vision transformers, which were originally developed for natural language processing, have recently generated significant interest in the computer vision and audio communities due to their flexibility in learning long-range relationships. Constrained by data hungry nature of transformers and limited labelled data most transformer-based models for audio tasks are finetuned from ImageNet pretrained models, despite the huge gap between the natural images domain and audio domain. This has motivated the research in self-supervised pretraining of audio transformers, which reduces the dependency on large amounts of labeled data and focuses on extracting concise representation of the audio spectrograms. In this paper, we propose ASiT, a novel self-supervised transformer for general audio representations that captures local and global contextual information employing group masked model learning and self-distillation. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, and speaker identification. We further conduct comprehensive ablation studies, including evaluations of different pretraining strategies. The proposed ASiT framework significantly boosts the performance on all tasks and sets a new state-of-the-art performance on five audio and speech classification tasks, outperforming recent methods, including the approaches that use additional datasets for pretraining. The code and pretrained weights will be made publicly available for the scientific community.
【2】 Average Token Delay: A Latency Metric for Simultaneous Translation
标题：平均令牌延迟：一种用于同声翻译的延迟度量
链接：https://arxiv.org/abs/2211.13173
作者：Yasumasa Kano,Katsuhito Sudoh,Satoshi Nakamura机构：Nara Institute of Science and Technology, Japan摘要：同声传译是一种在说话人说完之前就开始翻译的任务。在评价翻译质量时，除了考虑翻译质量外，还必须考虑翻译的延迟。等待时间优选地尽可能小，以便用户以小的延迟理解讲话者所说的内容。现有的延迟度量关注于翻译何时开始，而没有充分考虑翻译何时结束。这意味着这样的度量标准不会惩罚由长翻译输出导致的延迟，这实际上延迟了用户的理解。本文提出了一种新的时延评估指标--平均令牌时延（ATD），该指标关注于同声翻译中部分翻译的结束时间。我们通过模拟实例讨论了ATD的优势，并通过同声传译实验研究了ATD与平均滞后的差异。摘要：Simultaneous translation is a task in which translation begins before the speaker has finished speaking. In its evaluation, we have to consider the latency of the translation in addition to the quality. The latency is preferably as small as possible for users to comprehend what the speaker says with a small delay. Existing latency metrics focus on when the translation starts but do not consider adequately when the translation ends. This means such metrics do not penalize the latency caused by a long translation output, which actually delays users' comprehension. In this work, we propose a novel latency evaluation metric called Average Token Delay (ATD) that focuses on the end timings of partial translations in simultaneous translation. We discuss the advantage of ATD using simulated examples and also investigate the differences between ATD and Average Lagging with simultaneous translation experiments.
【3】 Introducing topography in convolutional neural networks
标题：在卷积神经网络中引入拓扑学
链接：https://arxiv.org/abs/2211.13152
作者：Maxime Poli,Emmanuel Dupoux,Rachid Riad机构：⋆ENS, PSL Research University, EHESS, CNRS, INRIA, †Meta AI Research备注：Submitted to ICASSP 2023摘要：负责感觉任务的大脑部分是按地形图组织的：附近的神经元响应于输入信号的相同属性。因此，在这项工作中，受神经科学文献的启发，我们提出了一个新的拓扑诱导偏差卷积神经网络（CNNs）。为了实现这一点，我们引入了一个新的拓扑损失和一个有效的实现来拓扑组织任何CNN的每个卷积层。我们在视觉和音频任务中的4个数据集和3个模型上对我们的新方法进行了基准测试，并显示出与所有基准测试相当的性能。此外，我们还展示了我们的地形损失的通用性，以及如何在CNNs中使用不同的地形组织。最后，我们证明了增加地形诱导偏压使CNNs更能抵抗修剪。我们的方法提供了一种新的途径来获得在保持更好的精度的同时更有效地利用存储器的模型。摘要：Parts of the brain that carry sensory tasks are organized topographically: nearby neurons are responsive to the same properties of input signals. Thus, in this work, inspired by the neuroscience literature, we proposed a new topographic inductive bias in Convolutional Neural Networks (CNNs). To achieve this, we introduced a new topographic loss and an efficient implementation to topographically organize each convolutional layer of any CNN. We benchmarked our new method on 4 datasets and 3 models in vision and audio tasks and showed equivalent performance to all benchmarks. Besides, we also showcased the generalizability of our topographic loss with how it can be used with different topographic organizations in CNNs. Finally, we demonstrated that adding the topographic inductive bias made CNNs more resistant to pruning. Our approach provides a new avenue to obtain models that are more memory efficient while maintaining better accuracy.
【4】 On the Typicality of Musical Sequences
标题：论音乐序列的典型性
链接：https://arxiv.org/abs/2211.13016
作者：Mathias Rose Bjare,Stefan Lattner机构：Computational Perception, Johannes Kepler University Linz, Sony Computer Science Laboratories备注：2 pages, 1 figure, Accepted at the Extended Abstracts for the Late-Breaking Demo Session of the 23rd Int. Society for Music Information Retrieval Conf., Bengaluru, India, 2022摘要：在最近的一份出版物中已经表明，人类产生的英语语言中的单词倾向于具有接近条件熵的信息含量。在这篇论文中，我们证明了人类产生的单声道音乐序列中的事件也是如此。我们还展示了“典型采样”如何影响单个事件和序列的熵周围的信息分布。摘要：It has been shown in a recent publication that words in human-produced English language tend to have an information content close to the conditional entropy. In this paper, we show that the same is true for events in human-produced monophonic musical sequences. We also show how "typical sampling" influences the distribution of information around the entropy for single events and sequences.
【5】 IMaSC -- ICFOSS Malayalam Speech Corpus
标题：IMaSC--ICFOSS马来亚语语音语料库
链接：https://arxiv.org/abs/2211.12796
作者：Deepa P Gopinath,Thennal D K,Vrinda V Nair,Swaraj K S,Sachin G机构：Dept. of Electronics and Commn. Engg, Government, Engineering College, Kozhikode, Kerala, India., Dept.of Computer Science, IIIT, Kottayam, Kerala, India., State Project Facilitation Unit, Trivandrum, Kerala, India.摘要：现代文本到语音（TTS）系统使用深度学习来合成越来越接近人类质量的语音，但它们需要高质量的音频-文本句子对的数据库来进行训练。马拉雅拉姆语是印度喀拉拉邦的官方语言，有3500多万人使用，就TTS系统的可用语料库而言，马拉雅拉姆语是一种低资源语言。本文介绍了一个包含约50小时语音记录的马拉雅拉姆语文本和语音语料库IMASC。拥有8个扬声器和总共34，473个文本音频对，IMaSC比其他所有公开可用的替代产品都要大。我们通过使用该数据库来基于现代深度学习架构为每个说话人训练TTS模型，从而对该数据库进行了评估。通过主观评价，我们发现我们的模型在自然度方面的表现明显优于以往的研究和公开可用的模型，平均意见得分为4.50，表明合成的语音接近人类的质量。摘要：Modern text-to-speech (TTS) systems use deep learning to synthesize speech increasingly approaching human quality, but they require a database of high quality audio-text sentence pairs for training. Malayalam, the official language of the Indian state of Kerala and spoken by 35+ million people, is a low resource language in terms of available corpora for TTS systems. In this paper, we present IMaSC, a Malayalam text and speech corpora containing approximately 50 hours of recorded speech. With 8 speakers and a total of 34,473 text-audio pairs, IMaSC is larger than every other publicly available alternative. We evaluated the database by using it to train TTS models for each speaker based on a modern deep learning architecture. Via subjective evaluation, we show that our models perform significantly better in terms of naturalness compared to previous studies and publicly available models, with an average mean opinion score of 4.50, indicating that the synthesized speech is close to human quality.
【6】 Complex-Valued Time-Frequency Self-Attention for Speech Dereverberation
标题：用于语音去混响的复值时频自注意
链接：https://arxiv.org/abs/2211.12632
作者：Vinay Kothapally,John H. L. Hansen机构：Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, TX, USA备注：Interspeech 2022: ISCA Best Student Paper Award Finalist摘要：当深度复杂神经网络（DCNN）与自注意（SA）网络耦合时，一些语音处理系统已经显示出相当大的性能改善。然而，大多数基于DCNN的语音去混响研究在计算注意力时没有明确考虑真实和虚构特征之间的相互依赖性。在本研究中，我们提出了一个复值T-F注意力（TFA）模型，该模型通过计算跨时间和频率维度的二维注意力图来建模频谱和时间依赖性。我们使用REVERB挑战语料库验证了我们提出的复值TFA模块与深度复杂卷积递归网络（DCCRN）的有效性。实验结果表明，与早期的自我注意方法相比，将我们的复杂TFA模块与DCCRN集成可以提高整体语音质量和后端语音应用（如自动语音识别）的性能。摘要：Several speech processing systems have demonstrated considerable performance improvements when deep complex neural networks (DCNN) are coupled with self-attention (SA) networks. However, the majority of DCNN-based studies on speech dereverberation that employ self-attention do not explicitly account for the inter-dependencies between real and imaginary features when computing attention. In this study, we propose a complex-valued T-F attention (TFA) module that models spectral and temporal dependencies by computing two-dimensional attention maps across time and frequency dimensions. We validate the effectiveness of our proposed complex-valued TFA module with the deep complex convolutional recurrent network (DCCRN) using the REVERB challenge corpus. Experimental findings indicate that integrating our complex-TFA module with DCCRN improves overall speech quality and performance of back-end speech applications, such as automatic speech recognition, compared to earlier approaches for self-attention.
【7】 SkipConvGAN: Monaural Speech Dereverberation using Generative Adversarial Networks via Complex Time-Frequency Masking
标题：SkipConvGAN：基于复杂时频掩蔽的生成对抗性网络单声道语音去混响
链接：https://arxiv.org/abs/2211.12623
作者：Vinay Kothapally,J. H. L. Hansen机构：University of Texas at Dallas备注：Published in: IEEE/ACM Transactions on Audio, Speech, and Language Processing ( Volume: 30)摘要：随着深度学习方法的进步，语音增强系统在存在背景噪声的情况下的性能已经显示出显著的改进。然而，提高系统对混响的鲁棒性仍然是一项正在进行的工作，因为混响往往由于时间和频率上的拖尾效应而导致共振峰结构的损失。广泛的基于深度学习的系统要么增强幅度响应并重用失真的相位，要么使用复杂的时频掩模来增强复杂的谱图。虽然这些方法已经证明了令人满意的性能，但是它们没有直接解决由混响引起的共振峰结构丢失。我们相信检索共振峰结构可以帮助提高现有系统的效率。在本研究中，我们提出了SkipConvGAN -我们先前工作SkipConvNet的延伸。所提出的系统的发生器网络试图估计有效的复数时间-频率掩模，而鉴别器网络帮助驱动发生器以恢复丢失的共振峰结构。我们在REVERB挑战语料库的单通道任务的混响语音的模拟和真实记录上评估了我们提出的系统的性能。与其他基于深度学习的生成式对抗性框架相比，所提出的系统在多个房间配置中表现出一致的改进。摘要：With the advancements in deep learning approaches, the performance of speech enhancing systems in the presence of background noise have shown significant improvements. However, improving the system's robustness against reverberation is still a work in progress, as reverberation tends to cause loss of formant structure due to smearing effects in time and frequency. A wide range of deep learning-based systems either enhance the magnitude response and reuse the distorted phase or enhance complex spectrogram using a complex time-frequency mask. Though these approaches have demonstrated satisfactory performance, they do not directly address the lost formant structure caused by reverberation. We believe that retrieving the formant structure can help improve the efficiency of existing systems. In this study, we propose SkipConvGAN - an extension of our prior work SkipConvNet. The proposed system's generator network tries to estimate an efficient complex time-frequency mask, while the discriminator network aids in driving the generator to restore the lost formant structure. We evaluate the performance of our proposed system on simulated and real recordings of reverberant speech from the single-channel task of the REVERB challenge corpus. The proposed system shows a consistent improvement across multiple room configurations over other deep learning-based generative adversarial frameworks.
【8】 Deep Neural Mel-Subband Beamformer for In-car Speech Separation
标题：用于车内语音分离的深度神经Mel子带波束形成器
链接：https://arxiv.org/abs/2211.12590
作者：Vinay Kothapally,Yong Xu,Meng Yu,Shi-Xiong Zhang,Dong Yu机构：Tencent AI Lab, Bellevue, WA, USA备注：Submitted to ICASSP 2023摘要：虽然当前基于深度学习（DL）的波束形成技术已被证明在语音分离中是有效的，但它们通常被设计成独立地处理窄带（NB）频率，这导致较高的计算成本和推理时间，使它们不适合于现实世界的使用。本文提出了一种基于DL的Mel子带空时波束形成器，用于汽车环境下的语音分离，降低了计算量和推理时间。与传统的子带（SB）方法相反，我们的框架使用基于梅尔尺度的子带选择策略，该策略确保对存在大多数语音共振峰结构的较低频率进行细粒度处理，而对较高频率进行粗粒度处理。以递归的方式，根据估计的子带语音和噪声协方差矩阵，为汽车中的每个扬声器位置/区域确定鲁棒的帧级波束形成权重。此外，所提出的框架还通过使用回声参考信号来估计和抑制来自扬声器的任何回声。在语音质量和识别度量方面，我们将我们提出的框架的性能与几种NB、SB和全带（FB）处理技术进行了比较。基于模拟和真实世界记录的实验评估，我们发现我们提出的框架实现了比所有SB和FB方法更好的分离性能，并且实现了更接近NB处理技术的性能，同时需要更低的计算成本。摘要：While current deep learning (DL)-based beamforming techniques have been proved effective in speech separation, they are often designed to process narrow-band (NB) frequencies independently which results in higher computational costs and inference times, making them unsuitable for real-world use. In this paper, we propose DL-based mel-subband spatio-temporal beamformer to perform speech separation in a car environment with reduced computation cost and inference time. As opposed to conventional subband (SB) approaches, our framework uses a mel-scale based subband selection strategy which ensures a fine-grained processing for lower frequencies where most speech formant structure is present, and coarse-grained processing for higher frequencies. In a recursive way, robust frame-level beamforming weights are determined for each speaker location/zone in a car from the estimated subband speech and noise covariance matrices. Furthermore, proposed framework also estimates and suppresses any echoes from the loudspeaker(s) by using the echo reference signals. We compare the performance of our proposed framework to several NB, SB, and full-band (FB) processing techniques in terms of speech quality and recognition metrics. Based on experimental evaluations on simulated and real-world recordings, we find that our proposed framework achieves better separation performance over all SB and FB approaches and achieves performance closer to NB processing techniques while requiring lower computing cost.
【1】 Complex-Valued Time-Frequency Self-Attention for Speech Dereverberation
标题：用于语音去混响的复值时频自注意
链接：https://arxiv.org/abs/2211.12632
* 与cs.SD语音【6】为同一篇
作者：Vinay Kothapally,John H. L. Hansen机构：Center for Robust Speech Systems (CRSS), The University of Texas at Dallas, TX, USA备注：Interspeech 2022: ISCA Best Student Paper Award Finalist摘要：当深度复杂神经网络（DCNN）与自注意（SA）网络耦合时，一些语音处理系统已经显示出相当大的性能改善。然而，大多数基于DCNN的语音去混响研究在计算注意力时没有明确考虑真实和虚构特征之间的相互依赖性。在本研究中，我们提出了一个复值T-F注意力（TFA）模型，该模型通过计算跨时间和频率维度的二维注意力图来建模频谱和时间依赖性。我们使用REVERB挑战语料库验证了我们提出的复值TFA模块与深度复杂卷积递归网络（DCCRN）的有效性。实验结果表明，与早期的自我注意方法相比，将我们的复杂TFA模块与DCCRN集成可以提高整体语音质量和后端语音应用（如自动语音识别）的性能。摘要：Several speech processing systems have demonstrated considerable performance improvements when deep complex neural networks (DCNN) are coupled with self-attention (SA) networks. However, the majority of DCNN-based studies on speech dereverberation that employ self-attention do not explicitly account for the inter-dependencies between real and imaginary features when computing attention. In this study, we propose a complex-valued T-F attention (TFA) module that models spectral and temporal dependencies by computing two-dimensional attention maps across time and frequency dimensions. We validate the effectiveness of our proposed complex-valued TFA module with the deep complex convolutional recurrent network (DCCRN) using the REVERB challenge corpus. Experimental findings indicate that integrating our complex-TFA module with DCCRN improves overall speech quality and performance of back-end speech applications, such as automatic speech recognition, compared to earlier approaches for self-attention.
【2】 SkipConvGAN: Monaural Speech Dereverberation using Generative Adversarial Networks via Complex Time-Frequency Masking
标题：SkipConvGAN：基于复杂时频掩蔽的生成对抗性网络单声道语音去混响
链接：https://arxiv.org/abs/2211.12623
* 与cs.SD语音【7】为同一篇
作者：Vinay Kothapally,J. H. L. Hansen机构：University of Texas at Dallas备注：Published in: IEEE/ACM Transactions on Audio, Speech, and Language Processing ( Volume: 30)摘要：随着深度学习方法的进步，语音增强系统在存在背景噪声的情况下的性能已经显示出显著的改进。然而，提高系统对混响的鲁棒性仍然是一项正在进行的工作，因为混响往往由于时间和频率上的拖尾效应而导致共振峰结构的损失。广泛的基于深度学习的系统要么增强幅度响应并重用失真的相位，要么使用复杂的时频掩模来增强复杂的谱图。虽然这些方法已经证明了令人满意的性能，但是它们没有直接解决由混响引起的共振峰结构丢失。我们相信检索共振峰结构可以帮助提高现有系统的效率。在本研究中，我们提出了SkipConvGAN-我们先前工作SkipConvNet的延伸。所提出的系统的发生器网络试图估计有效的复数时间-频率掩模，而鉴别器网络帮助驱动发生器以恢复丢失的共振峰结构。我们在REVERB挑战语料库的单通道任务的混响语音的模拟和真实记录上评估了我们提出的系统的性能。与其他基于深度学习的生成式对抗性框架相比，所提出的系统在多个房间配置中表现出一致的改进。摘要：With the advancements in deep learning approaches, the performance of speech enhancing systems in the presence of background noise have shown significant improvements. However, improving the system's robustness against reverberation is still a work in progress, as reverberation tends to cause loss of formant structure due to smearing effects in time and frequency. A wide range of deep learning-based systems either enhance the magnitude response and reuse the distorted phase or enhance complex spectrogram using a complex time-frequency mask. Though these approaches have demonstrated satisfactory performance, they do not directly address the lost formant structure caused by reverberation. We believe that retrieving the formant structure can help improve the efficiency of existing systems. In this study, we propose SkipConvGAN - an extension of our prior work SkipConvNet. The proposed system's generator network tries to estimate an efficient complex time-frequency mask, while the discriminator network aids in driving the generator to restore the lost formant structure. We evaluate the performance of our proposed system on simulated and real recordings of reverberant speech from the single-channel task of the REVERB challenge corpus. The proposed system shows a consistent improvement across multiple room configurations over other deep learning-based generative adversarial frameworks.
【3】 Deep Neural Mel-Subband Beamformer for In-car Speech Separation
标题：用于车内语音分离的深度神经Mel子带波束形成器
链接：https://arxiv.org/abs/2211.12590
* 与cs.SD语音【8】为同一篇
作者：Vinay Kothapally,Yong Xu,Meng Yu,Shi-Xiong Zhang,Dong Yu机构：Tencent AI Lab, Bellevue, WA, USA备注：Submitted to ICASSP 2023摘要：虽然当前基于深度学习（DL）的波束形成技术已被证明在语音分离中是有效的，但它们通常被设计成独立地处理窄带（NB）频率，这导致较高的计算成本和推理时间，使它们不适合于现实世界的使用。本文提出了一种基于DL的Mel子带空时波束形成器，用于汽车环境下的语音分离，降低了计算量和推理时间。与传统的子带（SB）方法相反，我们的框架使用基于梅尔尺度的子带选择策略，该策略确保对存在大多数语音共振峰结构的较低频率进行细粒度处理，而对较高频率进行粗粒度处理。以递归的方式，根据估计的子带语音和噪声协方差矩阵，为汽车中的每个扬声器位置/区域确定鲁棒的帧级波束形成权重。此外，所提出的框架还通过使用回声参考信号来估计和抑制来自扬声器的任何回声。在语音质量和识别度量方面，我们将我们提出的框架的性能与几种NB、SB和全带（FB）处理技术进行了比较。基于模拟和真实世界记录的实验评估，我们发现我们提出的框架实现了比所有SB和FB方法更好的分离性能，并且实现了更接近NB处理技术的性能，同时需要更低的计算成本。摘要：While current deep learning (DL)-based beamforming techniques have been proved effective in speech separation, they are often designed to process narrow-band (NB) frequencies independently which results in higher computational costs and inference times, making them unsuitable for real-world use. In this paper, we propose DL-based mel-subband spatio-temporal beamformer to perform speech separation in a car environment with reduced computation cost and inference time. As opposed to conventional subband (SB) approaches, our framework uses a mel-scale based subband selection strategy which ensures a fine-grained processing for lower frequencies where most speech formant structure is present, and coarse-grained processing for higher frequencies. In a recursive way, robust frame-level beamforming weights are determined for each speaker location/zone in a car from the estimated subband speech and noise covariance matrices. Furthermore, proposed framework also estimates and suppresses any echoes from the loudspeaker(s) by using the echo reference signals. We compare the performance of our proposed framework to several NB, SB, and full-band (FB) processing techniques in terms of speech quality and recognition metrics. Based on experimental evaluations on simulated and real-world recordings, we find that our proposed framework achieves better separation performance over all SB and FB approaches and achieves performance closer to NB processing techniques while requiring lower computing cost.
【4】 ASiT: Audio Spectrogram vIsion Transformer for General Audio Representation
标题：ASIT：用于通用音频表示的音频谱图视觉转换器
链接：https://arxiv.org/abs/2211.13189
* 与cs.SD语音【1】为同一篇
作者：Sara Atito,Muhammad Awais,Wenwu Wang,Mark D Plumbley,Josef Kittler机构：CVSSP, University of Surrey, Surrey, UK, !摘要：Vision Transformers最初是为自然语言处理而开发的，由于其在学习远程关系方面的灵活性，最近在计算机视觉和音频社区中产生了极大的兴趣。受Transformers的数据饥饿特性和有限的标记数据的约束，大多数基于变换器的音频任务模型是从ImageNet预训练模型微调的，尽管自然图像域和音频域之间存在巨大差距。这就激发了对音频Transformers的自监督预训练的研究，该预训练减少了对大量标记数据的依赖，并且集中于提取音频谱图的简明表示。本文提出了一种新颖的自监督音频Transformer ASiT，它采用分组掩蔽模型学习和自提炼的方法来捕获局部和全局上下文信息.我们在音频和语音分类任务上评估我们的预训练模型，包括音频事件分类、关键词识别和说话人识别。我们进一步进行了全面的消融研究，包括对不同预训练策略的评价。提出的ASiT框架显著提高了所有任务的性能，并在五个音频和语音分类任务上设置了新的最先进的性能，优于最近的方法，包括使用额外数据集进行预训练的方法。代码和预先训练的权重将向科学界公开。摘要：Vision transformers, which were originally developed for natural language processing, have recently generated significant interest in the computer vision and audio communities due to their flexibility in learning long-range relationships. Constrained by data hungry nature of transformers and limited labelled data most transformer-based models for audio tasks are finetuned from ImageNet pretrained models, despite the huge gap between the natural images domain and audio domain. This has motivated the research in self-supervised pretraining of audio transformers, which reduces the dependency on large amounts of labeled data and focuses on extracting concise representation of the audio spectrograms. In this paper, we propose ASiT, a novel self-supervised transformer for general audio representations that captures local and global contextual information employing group masked model learning and self-distillation. We evaluate our pretrained models on both audio and speech classification tasks including audio event classification, keyword spotting, and speaker identification. We further conduct comprehensive ablation studies, including evaluations of different pretraining strategies. The proposed ASiT framework significantly boosts the performance on all tasks and sets a new state-of-the-art performance on five audio and speech classification tasks, outperforming recent methods, including the approaches that use additional datasets for pretraining. The code and pretrained weights will be made publicly available for the scientific community.
【5】 Introducing topography in convolutional neural networks
标题：在卷积神经网络中引入拓扑学
链接：https://arxiv.org/abs/2211.13152
* 与cs.SD语音【3】为同一篇
作者：Maxime Poli,Emmanuel Dupoux,Rachid Riad机构：⋆ENS, PSL Research University, EHESS, CNRS, INRIA, †Meta AI Research备注：Submitted to ICASSP 2023摘要：负责感觉任务的大脑部分是按地形图组织的：附近的神经元响应于输入信号的相同属性。因此，在这项工作中，受神经科学文献的启发，我们提出了一个新的拓扑诱导偏差卷积神经网络（CNNs）。为了实现这一点，我们引入了一个新的拓扑损失和一个有效的实现来拓扑组织任何CNN的每个卷积层。我们在视觉和音频任务中的4个数据集和3个模型上对我们的新方法进行了基准测试，并显示出与所有基准测试相当的性能。此外，我们还展示了我们的地形损失的通用性，以及如何在CNNs中使用不同的地形组织。最后，我们证明了增加地形诱导偏压使CNNs更能抵抗修剪。我们的方法提供了一种新的途径来获得在保持更好的精度的同时更有效地利用存储器的模型。摘要：Parts of the brain that carry sensory tasks are organized topographically: nearby neurons are responsive to the same properties of input signals. Thus, in this work, inspired by the neuroscience literature, we proposed a new topographic inductive bias in Convolutional Neural Networks (CNNs). To achieve this, we introduced a new topographic loss and an efficient implementation to topographically organize each convolutional layer of any CNN. We benchmarked our new method on 4 datasets and 3 models in vision and audio tasks and showed equivalent performance to all benchmarks. Besides, we also showcased the generalizability of our topographic loss with how it can be used with different topographic organizations in CNNs. Finally, we demonstrated that adding the topographic inductive bias made CNNs more resistant to pruning. Our approach provides a new avenue to obtain models that are more memory efficient while maintaining better accuracy.
【6】 On the Typicality of Musical Sequences
标题：论音乐序列的典型性
链接：https://arxiv.org/abs/2211.13016
* 与cs.SD语音【4】为同一篇
作者：Mathias Rose Bjare,Stefan Lattner机构：Computational Perception, Johannes Kepler University Linz, Sony Computer Science Laboratories备注：2 pages, 1 figure, Accepted at the Extended Abstracts for the Late-Breaking Demo Session of the 23rd Int. Society for Music Information Retrieval Conf., Bengaluru, India, 2022摘要：在最近的一份出版物中已经表明，人类产生的英语语言中的单词倾向于具有接近条件熵的信息含量。在这篇论文中，我们证明了人类产生的单声道音乐序列中的事件也是如此。我们还展示了“典型采样”如何影响单个事件和序列的熵周围的信息分布。摘要：It has been shown in a recent publication that words in human-produced English language tend to have an information content close to the conditional entropy. In this paper, we show that the same is true for events in human-produced monophonic musical sequences. We also show how "typical sampling" influences the distribution of information around the entropy for single events and sequences.
【7】 IMaSC -- ICFOSS Malayalam Speech Corpus
标题：IMaSC--ICFOSS马来亚语语音语料库
链接：https://arxiv.org/abs/2211.12796
* 与cs.SD语音【5】为同一篇
作者：Deepa P Gopinath,Thennal D K,Vrinda V Nair,Swaraj K S,Sachin G机构：Dept. of Electronics and Commn. Engg, Government, Engineering College, Kozhikode, Kerala, India., Dept.of Computer Science, IIIT, Kottayam, Kerala, India., State Project Facilitation Unit, Trivandrum, Kerala, India.摘要：现代文本到语音（TTS）系统使用深度学习来合成越来越接近人类质量的语音，但它们需要高质量的音频-文本句子对的数据库来进行训练。马拉雅拉姆语是印度喀拉拉邦的官方语言，有3500多万人使用，就TTS系统的可用语料库而言，马拉雅拉姆语是一种低资源语言。本文介绍了一个包含约50小时语音记录的马拉雅拉姆语文本和语音语料库IMASC。拥有8个扬声器和总共34，473个文本音频对，IMaSC比其他所有公开可用的替代产品都要大。我们通过使用该数据库来基于现代深度学习架构为每个说话人训练TTS模型，从而对该数据库进行了评估。通过主观评价，我们发现我们的模型在自然度方面的表现明显优于以往的研究和公开可用的模型，平均意见得分为4.50，表明合成的语音接近人类的质量。摘要：Modern text-to-speech (TTS) systems use deep learning to synthesize speech increasingly approaching human quality, but they require a database of high quality audio-text sentence pairs for training. Malayalam, the official language of the Indian state of Kerala and spoken by 35+ million people, is a low resource language in terms of available corpora for TTS systems. In this paper, we present IMaSC, a Malayalam text and speech corpora containing approximately 50 hours of recorded speech. With 8 speakers and a total of 34,473 text-audio pairs, IMaSC is larger than every other publicly available alternative. We evaluated the database by using it to train TTS models for each speaker based on a modern deep learning architecture. Via subjective evaluation, we show that our models perform significantly better in terms of naturalness compared to previous studies and publicly available models, with an average mean opinion score of 4.50, indicating that the synthesized speech is close to human quality.
机器翻译，仅供参考
每日语音速递