今日论文合集：cs.SD语音16篇，eess.AS音频处理16篇。
cs.SD语音
【1】 Weakly-Supervised Multi-Task Learning for Audio-Visual Speaker  Verification标题：用于视听说话人确认的弱监督多任务学习链接：https://arxiv.org/abs/2309.07115
作者：Anith Selvakumar,Homa Fashandi机构：LG Electronics, Toronto AI Lab摘要：在本文中，我们提出了一种方法来实现强大的多模态的人表示优化开放集视听说话人验证。距离度量学习（DML）方法通常在这个问题空间中占主导地位，这是由于在新的和看不见的类上的强大性能。在我们的工作中，我们探索了多任务学习技术，以进一步提高DML方法的性能，并表明具有弱标签的辅助任务可以增加学习的说话人表示的紧凑性。我们还将广义端到端损失（GE 2 E）扩展到多模态输入，并证明它可以在视听空间中实现有竞争力的性能。最后，我们介绍了一个非同步的视听采样随机策略，在训练时间，已被证明可以提高泛化。我们的网络在说话人验证方面达到了最先进的性能，在VoxCeleb 1-O/E/H的三个官方试验列表上报告了0.244%，0.252%，0.441%的等错误率（EER），据我们所知，这是VoxCeleb 1-E和VoxCeleb 1-H上公布的最佳结果。摘要：In this paper, we present a methodology for achieving robust multimodal person representations optimized for open-set audio-visual speaker verification. Distance Metric Learning (DML) approaches have typically dominated this problem space, owing to strong performance on new and unseen classes. In our work, we explored multitask learning techniques to further boost performance of the DML approach and show that an auxiliary task with weak labels can increase the compactness of the learned speaker representation. We also extend the Generalized end-to-end loss (GE2E) to multimodal inputs and demonstrate that it can achieve competitive performance in an audio-visual space. Finally, we introduce a non-synchronous audio-visual sampling random strategy during training time that has shown to improve generalization. Our network achieves state of the art performance for speaker verification, reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the three official trial lists of VoxCeleb1-O/E/H, which is to our knowledge, the best published results on VoxCeleb1-E and VoxCeleb1-H.
【2】 MASTERKEY: Practical Backdoor Attack Against Speaker Verification  Systems标题：MasterKey：针对说话人验证系统的实用后门攻击链接：https://arxiv.org/abs/2309.06981
作者：Hanqing Guo,Xun Chen,Junfeng Guo,Li Xiao,Qiben Yan机构：Michigan State University, East Lansing, Michigan, USA, Samsung Research America, Mountain View, California, USA, UT Dallas, Dallas, Texas, USA备注：Accepted by Mobicom 2023摘要：说话人验证（Speaker Verification，SV）是一种广泛应用于移动通信系统中的语音识别技术。在这项工作中，我们提出了一个后门攻击MASTERKEY，妥协的SV模型。不同于以往的攻击，我们专注于现实世界的实际设置，攻击者不知道的预期受害者。为了设计MASTERKEY，我们调查了现有的中毒攻击对看不见的目标的限制。然后，我们优化了一个通用的后门，能够攻击任意目标。接下来，我们将说话人的特征和语义信息嵌入到后门中，使其不可感知。最后，我们估计信道失真，并将其集成到后门。我们验证我们的攻击6流行的SV模型。具体来说，我们总共毒害了53个模型，并使用我们的触发器攻击了16，430个注册扬声器，其中包括53个中毒模型中注册的310个目标扬声器。我们的攻击达到100%的攻击成功率和15%的中毒率。通过将中毒率降低到3%，攻击成功率保持在50%左右。我们验证我们的攻击在3个现实世界的情况下，并成功地证明了攻击通过空中和电话线的情况。摘要：Speaker Verification (SV) is widely deployed in mobile systems to authenticate legitimate users by using their voice traits. In this work, we propose a backdoor attack MASTERKEY, to compromise the SV models. Different from previous attacks, we focus on a real-world practical setting where the attacker possesses no knowledge of the intended victim. To design MASTERKEY, we investigate the limitation of existing poisoning attacks against unseen targets. Then, we optimize a universal backdoor that is capable of attacking arbitrary targets. Next, we embed the speaker's characteristics and semantics information into the backdoor, making it imperceptible. Finally, we estimate the channel distortion and integrate it into the backdoor. We validate our attack on 6 popular SV models. Specifically, we poison a total of 53 models and use our trigger to attack 16,430 enrolled speakers, composed of 310 target speakers enrolled in 53 poisoned models. Our attack achieves 100% attack success rate with a 15% poison rate. By decreasing the poison rate to 3%, the attack success rate remains around 50%. We validate our attack in 3 real-world scenarios and successfully demonstrate the attack through both over-the-air and over-the-telephony-line scenarios.
【3】 EMALG: An Enhanced Mandarin Lombard Grid Corpus with Meaningful  Sentences标题：EMALG：一个增强的具有有意义句子的普通话Lombard网格语料库链接：https://arxiv.org/abs/2309.06858
作者：Baifeng Li,Qingmu Liu,Yuhong Yang,Hongyang Chen,Weiping Tu,Song Lin机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China.,  Hubei Key Laboratory of Multimedia and Network Communication Engineering,  Guangdong OPPO Mobile Telecommunications Corp., China.摘要：本研究调查了隆巴德效应，即个体在嘈杂环境中调整自己的言语。我们介绍了一个增强的普通话伦巴底格（EMALG）语料库的有意义的句子，增强普通话伦巴底格（MALG）语料库。EMALG拥有34个扬声器，并改进了录音设置，解决了MALG面临的无意义句子的挑战。我们的研究结果表明，在普通话中，女性表现出比男性更明显的隆巴德效应，特别是在说出有意义的句子。此外，我们发现，无意义的句子负面影响隆巴德效应分析。此外，我们的研究结果再次证实了以前的研究中发现的英语和普通话之间的隆巴德效应比较的一致性。摘要：This study investigates the Lombard effect, where individuals adapt their speech in noisy environments. We introduce an enhanced Mandarin Lombard grid (EMALG) corpus with meaningful sentences , enhancing the Mandarin Lombard grid (MALG) corpus. EMALG features 34 speakers and improves recording setups, addressing challenges faced by MALG with nonsense sentences. Our findings reveal that in Mandarin, female exhibit a more pronounced Lombard effect than male, particularly when uttering meaningful sentences. Additionally, we uncover that nonsense sentences negatively impact Lombard effect analysis. Moreover, our results reaffirm the consistency in the Lombard effect comparison between English and Mandarin found in previous research.
【4】 DCTTS: Discrete Diffusion Model with Contrastive Learning for  Text-to-speech Generation标题：DCTTS：基于对比学习的文语转换离散扩散模型链接：https://arxiv.org/abs/2309.06787
作者：Zhichao Wu,Qiulin Li,Sixing Liu,Qun Yang机构：Nanjing University of Aeronautics and Astronautics,Nanjing,China备注：5 pages, submitted to ICASSP摘要：在文语转换（TTS）任务中，隐扩散模型具有良好的保真度和泛化能力，但其昂贵的资源消耗和缓慢的推理速度一直是一个挑战。本文提出了一种基于对比学习的离散扩散模型的文语转换生成算法。DCTTS做出了以下贡献：1）基于离散空间的TTS扩散模型，显著降低了扩散模型的计算量，提高了采样速度; 2）采用基于离散空间的对比学习方法，增强语音与文本的对齐连接，提高采样质量; 3）采用高效的文本编码器简化模型参数，提高计算效率。实验结果表明，本文提出的方法具有优异的语音合成质量和采样速度，同时显著降低了扩散模型的资源消耗。合成的样品可在https://github.com/lawtherWu/DCTTS获得。摘要：In the Text-to-speech(TTS) task, the latent diffusion model has excellent fidelity and generalization, but its expensive resource consumption and slow inference speed have always been a challenging. This paper proposes Discrete Diffusion Model with Contrastive Learning for Text-to-Speech Generation(DCTTS). The following contributions are made by DCTTS: 1) The TTS diffusion model based on discrete space significantly lowers the computational consumption of the diffusion model and improves sampling speed; 2) The contrastive learning method based on discrete space is used to enhance the alignment connection between speech and text and improve sampling quality; and 3) It uses an efficient text encoder to simplify the model's parameters and increase computational efficiency. The experimental results demonstrate that the approach proposed in this paper has outstanding speech synthesis quality and sampling speed while significantly reducing the resource consumption of diffusion model. The synthesized samples are available at https://github.com/lawtherWu/DCTTS.
【5】 Distinguishing Neural Speech Synthesis Models Through Fingerprints in  Speech Waveforms标题：利用语音波形指纹识别神经语音合成模型链接：https://arxiv.org/abs/2309.06780
作者：Chu Yuan Zhang,Jiangyan Yi,Jianhua Tao,Chenglong Wang,Xinrui Yan机构：⋆International College, University of Chinese Academy of Sciences, Beijing, China, †State Key Laboratory of Multi-modal Artificial Intelligence Systems, Institute of Automation, §Department of Automation, Tsinghua University, Beijing, China备注：Submitted to ICASSP 2024摘要：神经语音合成技术的最新进展虽然得到了广泛的应用，但也带来了一系列挑战，激发了人们对防范误用和滥用威胁的兴趣。值得注意的是，合成语音的源归属在取证和知识产权保护中具有价值，但是在该领域中的先前工作在范围上具有某些限制。为了解决的差距，我们提出了我们的研究结果，在本文中的合成语音的来源的识别。我们调查的语音合成模型指纹在生成的语音波形中的存在，重点是声学模型和声码器，并研究在整体语音波形的指纹上的每个组件的影响。我们的研究使用多说话人LibriTTS数据集进行，展示了两个关键见解：（1）声码器和声学模型在它们生成的波形上赋予不同的、模型特定的指纹，以及（2）声码器指纹是这两者中更占主导地位的，并且可以从声学模型中掩盖指纹。这些研究结果强烈表明，存在模型特定的指纹的声学模型和声码器，突出其在源识别应用的潜在效用。摘要：Recent strides in neural speech synthesis technologies, while enjoying widespread applications, have nonetheless introduced a series of challenges, spurring interest in the defence against the threat of misuse and abuse. Notably, source attribution of synthesized speech has value in forensics and intellectual property protection, but prior work in this area has certain limitations in scope. To address the gaps, we present our findings concerning the identification of the sources of synthesized speech in this paper. We investigate the existence of speech synthesis model fingerprints in the generated speech waveforms, with a focus on the acoustic model and the vocoder, and study the influence of each component on the fingerprint in the overall speech waveforms. Our research, conducted using the multi-speaker LibriTTS dataset, demonstrates two key insights: (1) vocoders and acoustic models impart distinct, model-specific fingerprints on the waveforms they generate, and (2) vocoder fingerprints are the more dominant of the two, and may mask the fingerprints from the acoustic model. These findings strongly suggest the existence of model-specific fingerprints for both the acoustic model and the vocoder, highlighting their potential utility in source identification applications.
【6】 Leveraging Foundation models for Unsupervised Audio-Visual Segmentation标题：利用基础模型实现无监督的视听分割链接：https://arxiv.org/abs/2309.06728
作者：Swapnil Bhosale,Haosen Yang,Diptesh Kanojia,Xiatian Zhu机构：University of Surrey, UK摘要：视听分割（AVS）的目的是在像素级上精确地勾勒出视觉场景中的可听对象。现有的AVS方法需要以监督学习方式对音频掩码对进行细粒度注释。这限制了它们的可扩展性，因为获取这种跨模态像素级标签是耗时且繁琐的。为了克服这一障碍，在这项工作中，我们引入了无监督的视听分割，不需要特定于任务的数据注释和模型训练。为了解决这个新提出的问题，我们制定了一种新的跨模态语义过滤（CMSF）方法，以通过利用现成的多模态基础模型（例如，检测[1]、开放世界分割[2]和多模态对齐[3]）。通过音频或视觉提示指导提案生成，我们设计了两种免训练变体：AT-GDINO-SAM和OWOD-BIND。在AVS-Bench数据集上进行的大量实验表明，与现有技术的有监督的方法相比，我们的无监督方法在具有多个听觉对象的复杂场景中可以表现良好。特别是，在现有的监督AVS方法与重叠的前景对象斗争的情况下，我们的模型仍然擅长准确地分割重叠的听觉对象。我们的代码将公开发布。摘要：Audio-Visual Segmentation (AVS) aims to precisely outline audible objects in a visual scene at the pixel level. Existing AVS methods require fine-grained annotations of audio-mask pairs in supervised learning fashion. This limits their scalability since it is time consuming and tedious to acquire such cross-modality pixel level labels. To overcome this obstacle, in this work we introduce unsupervised audio-visual segmentation with no need for task-specific data annotations and model training. For tackling this newly proposed problem, we formulate a novel Cross-Modality Semantic Filtering (CMSF) approach to accurately associate the underlying audio-mask pairs by leveraging the off-the-shelf multi-modal foundation models (e.g., detection [1], open-world segmentation [2] and multi-modal alignment [3]). Guiding the proposal generation by either audio or visual cues, we design two training-free variants: AT-GDINO-SAM and OWOD-BIND. Extensive experiments on the AVS-Bench dataset show that our unsupervised approach can perform well in comparison to prior art supervised counterparts across complex scenarios with multiple auditory objects. Particularly, in situations where existing supervised AVS methods struggle with overlapping foreground objects, our models still excel in accurately segmenting overlapped auditory objects. Our code will be publicly released.
【7】 PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network标题：Piave：一种姿态不变的视听说话人提取网络链接：https://arxiv.org/abs/2309.06723
作者：Qinghua Liu,Meng Ge,Zhizheng Wu,Haizhou Li机构：Shenzhen Research Institute of Big Data, Shenzhen, China, School of Data Science, The Chinese University of Hong Kong, Shenzhen, China,  Department of Electrical and Computer Engineering, National University of Singapore, Singapore备注：None摘要：在日常口语交流中，我们经常会看着说话者的转动的头部来听他/她的声音。人类看到说话者是为了更好地倾听，机器也是如此。然而，以往的研究视听说话人提取没有有效地处理不断变化的说话人脸。本文研究如何充分利用变化的说话人脸。我们提出了一个姿势不变的视听说话人提取网络（PIAVE），它采用了一个额外的姿势不变的观点，以提高视听说话人提取。具体地，我们从每个原始姿势取向生成姿势不变视图，这使得模型能够接收讲话者的一致的正面视图，而不管他/她的头部姿势，因此，形成用于讲话者的多视图视觉输入。在多视角MEAD和野外LRS 3数据集上的实验表明，PIAVE优于现有技术，并且对姿态变化更具鲁棒性。摘要：It is common in everyday spoken communication that we look at the turning head of a talker to listen to his/her voice. Humans see the talker to listen better, so do machines. However, previous studies on audio-visual speaker extraction have not effectively handled the varying talking face. This paper studies how to take full advantage of the varying talking face. We propose a Pose-Invariant Audio-Visual Speaker Extraction Network (PIAVE) that incorporates an additional pose-invariant view to improve audio-visual speaker extraction. Specifically, we generate the pose-invariant view from each original pose orientation, which enables the model to receive a consistent frontal view of the talker regardless of his/her head pose, therefore, forming a multi-view visual input for the speaker. Experiments on the multi-view MEAD and in-the-wild LRS3 dataset demonstrate that PIAVE outperforms the state-of-the-art and is more robust to pose variations.
【8】 Attention-based Encoder-Decoder End-to-End Neural Diarization with  Embedding Enhancer标题：嵌入增强器的基于注意力的编解码端到端神经元二值化链接：https://arxiv.org/abs/2309.06672
作者：Zhengyang Chen,Bing Han,Shuai Wang,Yanmin Qian备注：IEEE/ACM Transactions on Audio Speech and Language Processing Under Review摘要：基于深度神经网络的系统显着提高了说话人日记任务的性能。然而，端到端神经日志化（EEND）系统通常难以推广到具有不可见数量的说话者的场景，而目标说话者语音活动检测（TS-VAD）系统往往过于复杂。在本文中，我们提出了一个简单的基于注意力的编码器-解码器网络的端到端神经日记（AED-EEND）。在我们的训练过程中，我们引入了一个教师强迫策略来解决说话人排列问题，从而加快模型收敛。为了评估，我们提出了一种迭代解码方法，输出diarization结果为每个扬声器顺序。此外，我们提出了一个增强器模块，以增强帧级扬声器嵌入，使模型能够处理的情况下，看不见的扬声器数量。我们还探索用Conformer架构替换Transformer编码器，该架构可以更好地模拟本地信息。此外，我们发现，常用的模拟数据集的说话人日记有一个更高的重叠率相比，真实的数据。我们发现，使用与真实数据更一致的模拟训练数据可以实现一致性的改善。广泛的实验验证表明，我们提出的方法的有效性。我们最好的系统实现了一个新的国家的最先进的日记错误率（DER）的性能在所有的CALLHOME（10.08%），DIHARD II（24.64%），和AMI（13.00%）的评估基准，当没有使用oracle语音活动检测（VAD）。除了扬声器diarization，我们的AED-EEND系统也表现出显着的竞争力，作为一个语音类型检测模型。摘要：Deep neural network-based systems have significantly improved the performance of speaker diarization tasks. However, end-to-end neural diarization (EEND) systems often struggle to generalize to scenarios with an unseen number of speakers, while target speaker voice activity detection (TS-VAD) systems tend to be overly complex. In this paper, we propose a simple attention-based encoder-decoder network for end-to-end neural diarization (AED-EEND). In our training process, we introduce a teacher-forcing strategy to address the speaker permutation problem, leading to faster model convergence. For evaluation, we propose an iterative decoding method that outputs diarization results for each speaker sequentially. Additionally, we propose an Enhancer module to enhance the frame-level speaker embeddings, enabling the model to handle scenarios with an unseen number of speakers. We also explore replacing the transformer encoder with a Conformer architecture, which better models local information. Furthermore, we discovered that commonly used simulation datasets for speaker diarization have a much higher overlap ratio compared to real data. We found that using simulated training data that is more consistent with real data can achieve an improvement in consistency. Extensive experimental validation demonstrates the effectiveness of our proposed methodologies. Our best system achieved a new state-of-the-art diarization error rate (DER) performance on all the CALLHOME (10.08%), DIHARD II (24.64%), and AMI (13.00%) evaluation benchmarks, when no oracle voice activity detection (VAD) is used. Beyond speaker diarization, our AED-EEND system also shows remarkable competitiveness as a speech type detection model.
【9】 Differentiable Modelling of Percussive Audio with Transient and Spectral  Synthesis标题：基于暂态和频谱合成的敲击声可微建模链接：https://arxiv.org/abs/2309.06649
作者：Jordie Shier,Franco Caspe,Andrew Robertson,Mark Sandler,Charalampos Saitis,Andrew McPherson机构：Centre for Digital Music, Queen Mary University of London, United Kingdom,  Ableton AG, Berlin, Germany,  Dyson School of Design Engineering, Imperial College London, United Kingdom备注：To be published in The Proceedings of Forum Acusticum, Sep 2023, Turin, Italy摘要：可微分数字信号处理（DDSP）技术，包括用于音频合成的方法，近年来已经获得关注，并且使它们自身在参数空间中具有可解释性。然而，当前的可微分合成方法尚未明确地寻求对信号的瞬态部分进行建模，这对于打击声音是重要的。在这项工作中，我们提出了一个统一的合成框架，旨在解决瞬态生成和打击乐合成内的DDSP框架。为此，我们提出了一个打击合成模型，建立在正弦建模合成，并采用了调制的时间卷积网络的瞬态生成。我们使用修改后的正弦峰值拾取算法来生成时变非谐波正弦曲线，并将其与共同训练以重建鼓组声音的可微分噪声和瞬态编码器配对。我们使用声学和电子打击样本的大数据集计算一组重建度量，这表明我们的方法导致膜乐器打击乐器的改善的起始信号重建。摘要：Differentiable digital signal processing (DDSP) techniques, including methods for audio synthesis, have gained attention in recent years and lend themselves to interpretability in the parameter space. However, current differentiable synthesis methods have not explicitly sought to model the transient portion of signals, which is important for percussive sounds. In this work, we present a unified synthesis framework aiming to address transient generation and percussive synthesis within a DDSP framework. To this end, we propose a model for percussive synthesis that builds on sinusoidal modeling synthesis and incorporates a modulated temporal convolutional network for transient generation. We use a modified sinusoidal peak picking algorithm to generate time-varying non-harmonic sinusoids and pair it with differentiable noise and transient encoders that are jointly trained to reconstruct drumset sounds. We compute a set of reconstruction metrics using a large dataset of acoustic and electronic percussion samples that show that our method leads to improved onset signal reconstruction for membranophone percussion instruments.
【10】 Can Whisper perform speech-based in-context learning标题：Whisper可以进行基于语音的情景学习吗链接：https://arxiv.org/abs/2309.07081
作者：Siyin Wang,Chao-Han Huck Yang,Ji Wu,Chao Zhang机构：Tsinghua University, Beijing, China, Georgia Institute of Technology, Atlanta, U.S.A备注：Submitted to ICASSP 2024摘要：本文研究了OpenAI发布的Whisper自动语音识别（ASR）模型的上下文学习能力。提出了一种新的基于语音的上下文学习（SICL）方法，用于测试时自适应，该方法可以降低单词错误率（WERs），只有少量的标记语音样本，而无需梯度下降。使用汉语方言的语言水平的适应实验表明，当应用SICL孤立词ASR，一致的和相当大的相对WER减少可以实现使用任何大小的耳语模型上的两种方言，这是平均32.3%。可以应用基于k最近邻的上下文内示例选择技术来进一步提高SICL的效率，这可以将平均相对WER减少增加到36.4%。使用扬声器自适应或连续语音识别任务的研究结果进行了验证，都取得了相当大的相对WER减少。定量分析也提供了详细的SICL的适应性，以阐明语音的变化和方言特有的词汇细微差别。摘要：This paper investigates the in-context learning abilities of the Whisper automatic speech recognition (ASR) models released by OpenAI. A novel speech-based in-context learning (SICL) approach is proposed for test-time adaptation, which can reduce the word error rates (WERs) with only a small number of labelled speech samples without gradient descent. Language-level adaptation experiments using Chinese dialects showed that when applying SICL to isolated word ASR, consistent and considerable relative WER reductions can be achieved using Whisper models of any size on two dialects, which is on average 32.3%. A k-nearest-neighbours-based in-context example selection technique can be applied to further improve the efficiency of SICL, which can increase the average relative WER reduction to 36.4%. The findings are verified using speaker adaptation or continuous speech recognition tasks, and both achieved considerable relative WER reductions. Detailed quantitative analyses are also provided to shed light on SICL's adaptability to phonological variances and dialect-specific lexical nuances.
【11】 A Flexible Online Framework for Projection-Based STFT Phase Retrieval标题：一种灵活的基于投影的短时傅里叶变换相位在线检索框架链接：https://arxiv.org/abs/2309.07043
作者：Tal Peer,Simon Welker,Johannes Kolhoff,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany,  Center for Free-Electron Laser Science, DESY, Hamburg, Germany备注：Submitted to ICASSP 24摘要：最近在迭代STFT相位恢复领域的几个贡献已经表明，经典的Griffin-Lim方法的性能可以大大提高。通过使用与Griffin-Lim相同的投影算子，但以创新的方式组合它们，这些方法在重建质量和所需迭代次数方面实现了更好的结果，同时保留了每次迭代类似的计算复杂度。然而，像Griffin-Lim一样，这些算法以离线方式操作，因此需要整个频谱图作为输入，这对于许多现实世界的语音通信应用来说是不现实的要求。我们建议扩展RTISI -现有的在线（逐帧）的格里芬-林算法的变体-到一个灵活的框架，使任何算法的迭代投影的基础上直接在线实施。我们进一步采用这个框架来实现在线变体的快速Griffin-Lim算法，加速Griffin-Lim算法，和两个算法从光学领域。语音信号的评估结果表明，类似于离线情况，这些算法可以实现相当大的性能增益相比RTISI。摘要：Several recent contributions in the field of iterative STFT phase retrieval have demonstrated that the performance of the classical Griffin-Lim method can be considerably improved upon. By using the same projection operators as Griffin-Lim, but combining them in innovative ways, these approaches achieve better results in terms of both reconstruction quality and required number of iterations, while retaining a similar computational complexity per iteration. However, like Griffin-Lim, these algorithms operate in an offline manner and thus require an entire spectrogram as input, which is an unrealistic requirement for many real-world speech communication applications. We propose to extend RTISI -- an existing online (frame-by-frame) variant of the Griffin-Lim algorithm -- into a flexible framework that enables straightforward online implementation of any algorithm based on iterative projections. We further employ this framework to implement online variants of the fast Griffin-Lim algorithm, the accelerated Griffin-Lim algorithm, and two algorithms from the optics domain. Evaluation results on speech signals show that, similarly to the offline case, these algorithms can achieve a considerable performance gain compared to RTISI.
【12】 Reorganization of the auditory-perceptual space across the human vocal  range标题：人类发声范围内听觉-知觉空间的重组链接：https://arxiv.org/abs/2309.06946
作者：Daniel Friedrichs,Volker Dellwo机构：Department of Computational Linguistics, University of Zurich备注：None摘要：我们分析了听觉感知空间在很大一部分的人类的音域（220-1046赫兹）使用多维尺度分析的耳蜗标度的频谱从250毫秒元音段，最初研究在弗里德里希等人。（2017）J. Acoust.美国社会142 1025-1033。该数据集包括由三位德国本土女性说话者产生的元音/i y e {\o} {\epsilon} a o u/（N=240），涵盖了他们各自的语音频率范围的广泛范围。最初的研究表明，在一个封闭的识别任务，涉及21名听众，点元音/i a u/显着承认在基本频率（fo）接近1 kHz，而其他元音的识别在较高的音高下降。基于这些发现，我们的研究揭示了系统的频谱变化与元音高度和frontness作为fo增加，与一个显着的集群周围/i a u/以上523赫兹。这些观察强调了元音感知中频谱形状的关键作用，说明了在较高音高下对声学锚的依赖。此外，这项研究揭示了这些元音的量子性质及其对语言进化的潜在影响，为它们在世界语言中的广泛存在提供了一个合理的解释。摘要：We analyzed the auditory-perceptual space across a substantial portion of the human vocal range (220-1046 Hz) using multidimensional scaling analysis of cochlea-scaled spectra from 250-ms vowel segments, initially studied in Friedrichs et al. (2017) J. Acoust. Soc. Am. 142 1025-1033. The dataset comprised the vowels /i y e {\o} {\epsilon} a o u/ (N=240) produced by three native German female speakers, encompassing a broad range of their respective voice frequency ranges. The initial study demonstrated that, during a closed-set identification task involving 21 listeners, the point vowels /i a u/ were significantly recognized at fundamental frequencies (fo) nearing 1 kHz, whereas the recognition of other vowels decreased at higher pitches. Building on these findings, our study revealed systematic spectral shifts associated with vowel height and frontness as fo increased, with a notable clustering around /i a u/ above 523 Hz. These observations underscore the pivotal role of spectral shape in vowel perception, illustrating the reliance on acoustic anchors at higher pitches. Furthermore, this study sheds light on the quantal nature of these vowels and their potential impact on language evolution, offering a plausible explanation for their widespread presence in the world's languages.
【13】 VRDMG: Vocal Restoration via Diffusion Posterior Sampling with Multiple  Guidance标题：VRDMG：基于多引导扩散后验采样的发声恢复链接：https://arxiv.org/abs/2309.06934
作者：Carlos Hernandez-Olivan,Koichi Saito,Naoki Murata,Chieh-Hsin Lai,Marco A. Martínez-Ramirez,Wei-Hsiang Liao,Yuki Mitsufuji机构：Marco A. Mart´ınez-Ram´ırez, Dept. of Electronic Engineering and Communications, University of Zaragoza, Spain, Sony AI, Tokyo, Japan, Sony Group Corporation, Tokyo, Japan摘要：恢复退化的音乐信号对于增强下游音乐处理的音频质量是必不可少的。最近的基于扩散的音乐恢复方法已经表现出令人印象深刻的性能，其中，扩散后验采样（DPS）由于其固有的特性而脱颖而出，使其在各种恢复任务中具有通用性。在本文中，我们确定有潜在的问题，这将降低当前基于DPS的方法的性能，并介绍了如何减轻问题的启发，不同的扩散指导技术，包括重绘（RP）的战略和伪逆指导扩散模型（$\Pi$GDM）。我们展示了我们的方法，在各种水平的失真和截止频率下的声乐去唇和带宽扩展任务，分别。在这两项任务中，我们的方法优于当前基于DPS的音乐恢复基准。我们参考\url{http：//carlosholivan.github.io/demos/audio-restoration-2023.html}以获取恢复的音频样本的示例。摘要：Restoring degraded music signals is essential to enhance audio quality for downstream music manipulation. Recent diffusion-based music restoration methods have demonstrated impressive performance, and among them, diffusion posterior sampling (DPS) stands out given its intrinsic properties, making it versatile across various restoration tasks. In this paper, we identify that there are potential issues which will degrade current DPS-based methods' performance and introduce the way to mitigate the issues inspired by diverse diffusion guidance techniques including the RePaint (RP) strategy and the Pseudoinverse-Guided Diffusion Models ($\Pi$GDM). We demonstrate our methods for the vocal declipping and bandwidth extension tasks under various levels of distortion and cutoff frequency, respectively. In both tasks, our methods outperform the current DPS-based music restoration benchmarks. We refer to \url{http://carlosholivan.github.io/demos/audio-restoration-2023.html} for examples of the restored audio samples.
【14】 Sound field decomposition based on two-stage neural networks标题：基于两级神经网络的声场分解链接：https://arxiv.org/abs/2309.06661
作者：Ryo Matsuda,Makoto Otani机构：Department of Architecture and Architectural Engineering, Graduate School of Engineering, Kyoto University, Kyoto-daigaku-katsura, Nishikyo-ku, Kyoto,-, Japan备注：31 pages, 16 figures摘要：提出了一种基于神经网络的声场分解方法。该方法包括两个阶段：声场分离阶段和单源定位阶段。在第一阶段中，由多个声源合成的传声器处的声压被分离为由每个声源激发的声压。在第二阶段中，源位置被获得作为从麦克风处的声压的回归，由单个声源组成。估计的位置不受离散化的影响，因为第二阶段被设计为回归而不是分类。使用格林函数通过模拟生成数据集，并且针对每个频率训练神经网络。数值实验表明，与传统方法相比，该方法具有更高的声源定位精度和声场重建精度。摘要：A method for sound field decomposition based on neural networks is proposed. The method comprises two stages: a sound field separation stage and a single-source localization stage. In the first stage, the sound pressure at microphones synthesized by multiple sources is separated into one excited by each sound source. In the second stage, the source location is obtained as a regression from the sound pressure at microphones consisting of a single sound source. The estimated location is not affected by discretization because the second stage is designed as a regression rather than a classification. Datasets are generated by simulation using Green's function, and the neural network is trained for each frequency. Numerical experiments reveal that, compared with conventional methods, the proposed method can achieve higher source-localization accuracy and higher sound-field-reconstruction accuracy.
【15】 Addressing the Blind Spots in Spoken Language Processing标题：解决口语处理中的盲点链接：https://arxiv.org/abs/2309.06572
作者：Amit Moryossef机构：Bar-Ilan University, University of Zürich备注：5 pages摘要：本文探讨了非语言线索的关键，但往往被忽视的作用，包括共同言语的手势和面部表情，在人类交流和自然语言处理（NLP）的影响。我们认为，理解人类的沟通需要一个更全面的方法，超越文本或口语的话，包括非语言元素。借用手语处理的进展，我们提出了通用的自动手势分割和转录模型的发展，转录这些非语言线索到文本形式。这种方法旨在弥合口语理解中的盲点，增强NLP模型的范围和适用性。通过激励的例子，我们证明了仅仅依靠基于文本的模型的局限性。我们提出了一种计算高效和灵活的方法，将非语言线索，它可以无缝集成与现有的NLP管道。最后，我们呼吁研究界为通用转录方法的发展做出贡献，并验证其有效性，捕捉现实世界的复杂性，多模式相互作用。摘要：This paper explores the critical but often overlooked role of non-verbal cues, including co-speech gestures and facial expressions, in human communication and their implications for Natural Language Processing (NLP). We argue that understanding human communication requires a more holistic approach that goes beyond textual or spoken words to include non-verbal elements. Borrowing from advances in sign language processing, we propose the development of universal automatic gesture segmentation and transcription models to transcribe these non-verbal cues into textual form. Such a methodology aims to bridge the blind spots in spoken language understanding, enhancing the scope and applicability of NLP models. Through motivating examples, we demonstrate the limitations of relying solely on text-based models. We propose a computationally efficient and flexible approach for incorporating non-verbal cues, which can seamlessly integrate with existing NLP pipelines. We conclude by calling upon the research community to contribute to the development of universal transcription methods and to validate their effectiveness in capturing the complexities of real-world, multi-modal interactions.
【16】 ASPED: An Audio Dataset for Detecting Pedestrians标题：ASPED：一种用于行人检测的音频数据集链接：https://arxiv.org/abs/2309.06531
作者：Pavan Seshadri,Chaeyeon Han,Bon-Woo Koo,Noah Posner,Subhrajit Guhathakurta,Alexander Lerch机构：CSPAV, bMusic Informatics Group, and cIPaT, Georgia Institute of Technology, Toronto Metropolitan University备注：4+1 pages摘要：我们介绍了新的音频分析任务的行人检测，并提出了一个新的大规模的数据集，这项任务。虽然初步结果证明了使用音频方法进行行人检测的可行性，但它们也表明，使用标准方法无法轻松解决这一具有挑战性的任务。摘要：We introduce the new audio analysis task of pedestrian detection and present a new large-scale dataset for this task. While the preliminary results prove the viability of using audio approaches for pedestrian detection, they also show that this challenging task cannot be easily solved with standard approaches.
eess.AS音频处理
【1】 Can Whisper perform speech-based in-context learning标题：Whisper可以进行基于语音的情景学习吗链接：https://arxiv.org/abs/2309.07081
作者：Siyin Wang,Chao-Han Huck Yang,Ji Wu,Chao Zhang机构：Tsinghua University, Beijing, China, Georgia Institute of Technology, Atlanta, U.S.A备注：Submitted to ICASSP 2024摘要：本文研究了OpenAI发布的Whisper自动语音识别（ASR）模型的上下文学习能力。提出了一种新的基于语音的上下文学习（SICL）方法，用于测试时自适应，该方法可以降低单词错误率（WERs），只有少量的标记语音样本，而无需梯度下降。使用汉语方言的语言水平的适应实验表明，当应用SICL孤立词ASR，一致的和相当大的相对WER减少可以实现使用任何大小的耳语模型上的两种方言，这是平均32.3%。可以应用基于k最近邻的上下文内示例选择技术来进一步提高SICL的效率，这可以将平均相对WER减少增加到36.4%。使用扬声器自适应或连续语音识别任务的研究结果进行了验证，都取得了相当大的相对WER减少。定量分析也提供了详细的SICL的适应性，以阐明语音的变化和方言特有的词汇细微差别。摘要：This paper investigates the in-context learning abilities of the Whisper automatic speech recognition (ASR) models released by OpenAI. A novel speech-based in-context learning (SICL) approach is proposed for test-time adaptation, which can reduce the word error rates (WERs) with only a small number of labelled speech samples without gradient descent. Language-level adaptation experiments using Chinese dialects showed that when applying SICL to isolated word ASR, consistent and considerable relative WER reductions can be achieved using Whisper models of any size on two dialects, which is on average 32.3%. A k-nearest-neighbours-based in-context example selection technique can be applied to further improve the efficiency of SICL, which can increase the average relative WER reduction to 36.4%. The findings are verified using speaker adaptation or continuous speech recognition tasks, and both achieved considerable relative WER reductions. Detailed quantitative analyses are also provided to shed light on SICL's adaptability to phonological variances and dialect-specific lexical nuances.
【2】 A Flexible Online Framework for Projection-Based STFT Phase Retrieval标题：一种灵活的基于投影的短时傅里叶变换相位在线检索框架链接：https://arxiv.org/abs/2309.07043
作者：Tal Peer,Simon Welker,Johannes Kolhoff,Timo Gerkmann机构：Signal Processing (SP), Universit¨at Hamburg, Germany,  Center for Free-Electron Laser Science, DESY, Hamburg, Germany备注：Submitted to ICASSP 24摘要：最近在迭代STFT相位恢复领域的几个贡献已经表明，经典的Griffin-Lim方法的性能可以大大提高。通过使用与Griffin-Lim相同的投影算子，但以创新的方式组合它们，这些方法在重建质量和所需迭代次数方面实现了更好的结果，同时保留了每次迭代类似的计算复杂度。然而，像Griffin-Lim一样，这些算法以离线方式操作，因此需要整个频谱图作为输入，这对于许多现实世界的语音通信应用来说是不现实的要求。我们建议扩展RTISI -现有的在线（逐帧）的格里芬-林算法的变体-到一个灵活的框架，使任何算法的迭代投影的基础上直接在线实施。我们进一步采用这个框架来实现在线变体的快速Griffin-Lim算法，加速Griffin-Lim算法，和两个算法从光学领域。语音信号的评估结果表明，类似于离线情况，这些算法可以实现相当大的性能增益相比RTISI。摘要：Several recent contributions in the field of iterative STFT phase retrieval have demonstrated that the performance of the classical Griffin-Lim method can be considerably improved upon. By using the same projection operators as Griffin-Lim, but combining them in innovative ways, these approaches achieve better results in terms of both reconstruction quality and required number of iterations, while retaining a similar computational complexity per iteration. However, like Griffin-Lim, these algorithms operate in an offline manner and thus require an entire spectrogram as input, which is an unrealistic requirement for many real-world speech communication applications. We propose to extend RTISI -- an existing online (frame-by-frame) variant of the Griffin-Lim algorithm -- into a flexible framework that enables straightforward online implementation of any algorithm based on iterative projections. We further employ this framework to implement online variants of the fast Griffin-Lim algorithm, the accelerated Griffin-Lim algorithm, and two algorithms from the optics domain. Evaluation results on speech signals show that, similarly to the offline case, these algorithms can achieve a considerable performance gain compared to RTISI.
【3】 Reorganization of the auditory-perceptual space across the human vocal  range标题：人类发声范围内听觉-知觉空间的重组链接：https://arxiv.org/abs/2309.06946
作者：Daniel Friedrichs,Volker Dellwo机构：Department of Computational Linguistics, University of Zurich备注：None摘要：我们分析了听觉感知空间在很大一部分的人类的音域（220-1046赫兹）使用多维尺度分析的耳蜗标度的频谱从250毫秒元音段，最初研究在弗里德里希等人。（2017）J. Acoust.美国社会142 1025-1033。该数据集包括由三位德国本土女性说话者产生的元音/i y e {\o} {\epsilon} a o u/（N=240），涵盖了他们各自的语音频率范围的广泛范围。最初的研究表明，在一个封闭的识别任务，涉及21名听众，点元音/i a u/显着承认在基本频率（fo）接近1 kHz，而其他元音的识别在较高的音高下降。基于这些发现，我们的研究揭示了系统的频谱变化与元音高度和frontness作为fo增加，与一个显着的集群周围/i a u/以上523赫兹。这些观察强调了元音感知中频谱形状的关键作用，说明了在较高音高下对声学锚的依赖。此外，这项研究揭示了这些元音的量子性质及其对语言进化的潜在影响，为它们在世界语言中的广泛存在提供了一个合理的解释。摘要：We analyzed the auditory-perceptual space across a substantial portion of the human vocal range (220-1046 Hz) using multidimensional scaling analysis of cochlea-scaled spectra from 250-ms vowel segments, initially studied in Friedrichs et al. (2017) J. Acoust. Soc. Am. 142 1025-1033. The dataset comprised the vowels /i y e {\o} {\epsilon} a o u/ (N=240) produced by three native German female speakers, encompassing a broad range of their respective voice frequency ranges. The initial study demonstrated that, during a closed-set identification task involving 21 listeners, the point vowels /i a u/ were significantly recognized at fundamental frequencies (fo) nearing 1 kHz, whereas the recognition of other vowels decreased at higher pitches. Building on these findings, our study revealed systematic spectral shifts associated with vowel height and frontness as fo increased, with a notable clustering around /i a u/ above 523 Hz. These observations underscore the pivotal role of spectral shape in vowel perception, illustrating the reliance on acoustic anchors at higher pitches. Furthermore, this study sheds light on the quantal nature of these vowels and their potential impact on language evolution, offering a plausible explanation for their widespread presence in the world's languages.
【4】 VRDMG: Vocal Restoration via Diffusion Posterior Sampling with Multiple  Guidance标题：VRDMG：基于多引导扩散后验采样的发声恢复链接：https://arxiv.org/abs/2309.06934
作者：Carlos Hernandez-Olivan,Koichi Saito,Naoki Murata,Chieh-Hsin Lai,Marco A. Martínez-Ramirez,Wei-Hsiang Liao,Yuki Mitsufuji机构：Marco A. Mart´ınez-Ram´ırez, Dept. of Electronic Engineering and Communications, University of Zaragoza, Spain, Sony AI, Tokyo, Japan, Sony Group Corporation, Tokyo, Japan摘要：恢复退化的音乐信号对于增强下游音乐处理的音频质量是必不可少的。最近的基于扩散的音乐恢复方法已经表现出令人印象深刻的性能，其中，扩散后验采样（DPS）由于其固有的特性而脱颖而出，使其在各种恢复任务中具有通用性。在本文中，我们确定有潜在的问题，这将降低当前基于DPS的方法的性能，并介绍了如何减轻问题的启发，不同的扩散指导技术，包括重绘（RP）的战略和伪逆指导扩散模型（$\Pi$GDM）。我们展示了我们的方法，在各种水平的失真和截止频率下的声乐去唇和带宽扩展任务，分别。在这两项任务中，我们的方法优于当前基于DPS的音乐恢复基准。我们参考\url{http：//carlosholivan.github.io/demos/audio-restoration-2023.html}以获取恢复的音频样本的示例。摘要：Restoring degraded music signals is essential to enhance audio quality for downstream music manipulation. Recent diffusion-based music restoration methods have demonstrated impressive performance, and among them, diffusion posterior sampling (DPS) stands out given its intrinsic properties, making it versatile across various restoration tasks. In this paper, we identify that there are potential issues which will degrade current DPS-based methods' performance and introduce the way to mitigate the issues inspired by diverse diffusion guidance techniques including the RePaint (RP) strategy and the Pseudoinverse-Guided Diffusion Models ($\Pi$GDM). We demonstrate our methods for the vocal declipping and bandwidth extension tasks under various levels of distortion and cutoff frequency, respectively. In both tasks, our methods outperform the current DPS-based music restoration benchmarks. We refer to \url{http://carlosholivan.github.io/demos/audio-restoration-2023.html} for examples of the restored audio samples.
【5】 Sound field decomposition based on two-stage neural networks标题：基于两级神经网络的声场分解链接：https://arxiv.org/abs/2309.06661
作者：Ryo Matsuda,Makoto Otani机构：Department of Architecture and Architectural Engineering, Graduate School of Engineering, Kyoto University, Kyoto-daigaku-katsura, Nishikyo-ku, Kyoto,-, Japan备注：31 pages, 16 figures摘要：提出了一种基于神经网络的声场分解方法。该方法包括两个阶段：声场分离阶段和单源定位阶段。在第一阶段中，由多个声源合成的传声器处的声压被分离为由每个声源激发的声压。在第二阶段中，源位置被获得作为从麦克风处的声压的回归，由单个声源组成。估计的位置不受离散化的影响，因为第二阶段被设计为回归而不是分类。使用格林函数通过模拟生成数据集，并且针对每个频率训练神经网络。数值实验表明，与传统方法相比，该方法具有更高的声源定位精度和声场重建精度。摘要：A method for sound field decomposition based on neural networks is proposed. The method comprises two stages: a sound field separation stage and a single-source localization stage. In the first stage, the sound pressure at microphones synthesized by multiple sources is separated into one excited by each sound source. In the second stage, the source location is obtained as a regression from the sound pressure at microphones consisting of a single sound source. The estimated location is not affected by discretization because the second stage is designed as a regression rather than a classification. Datasets are generated by simulation using Green's function, and the neural network is trained for each frequency. Numerical experiments reveal that, compared with conventional methods, the proposed method can achieve higher source-localization accuracy and higher sound-field-reconstruction accuracy.
【6】 Addressing the Blind Spots in Spoken Language Processing标题：解决口语处理中的盲点链接：https://arxiv.org/abs/2309.06572
作者：Amit Moryossef机构：Bar-Ilan University, University of Zürich备注：5 pages摘要：本文探讨了非语言线索的关键，但往往被忽视的作用，包括共同言语的手势和面部表情，在人类交流和自然语言处理（NLP）的影响。我们认为，理解人类的沟通需要一个更全面的方法，超越文本或口语的话，包括非语言元素。借用手语处理的进展，我们提出了通用的自动手势分割和转录模型的发展，转录这些非语言线索到文本形式。这种方法旨在弥合口语理解中的盲点，增强NLP模型的范围和适用性。通过激励的例子，我们证明了仅仅依靠基于文本的模型的局限性。我们提出了一种计算高效和灵活的方法，将非语言线索，它可以无缝集成与现有的NLP管道。最后，我们呼吁研究界为通用转录方法的发展做出贡献，并验证其有效性，捕捉现实世界的复杂性，多模式相互作用。摘要：This paper explores the critical but often overlooked role of non-verbal cues, including co-speech gestures and facial expressions, in human communication and their implications for Natural Language Processing (NLP). We argue that understanding human communication requires a more holistic approach that goes beyond textual or spoken words to include non-verbal elements. Borrowing from advances in sign language processing, we propose the development of universal automatic gesture segmentation and transcription models to transcribe these non-verbal cues into textual form. Such a methodology aims to bridge the blind spots in spoken language understanding, enhancing the scope and applicability of NLP models. Through motivating examples, we demonstrate the limitations of relying solely on text-based models. We propose a computationally efficient and flexible approach for incorporating non-verbal cues, which can seamlessly integrate with existing NLP pipelines. We conclude by calling upon the research community to contribute to the development of universal transcription methods and to validate their effectiveness in capturing the complexities of real-world, multi-modal interactions.
【7】 ASPED: An Audio Dataset for Detecting Pedestrians标题：ASPED：一种用于行人检测的音频数据集链接：https://arxiv.org/abs/2309.06531
作者：Pavan Seshadri,Chaeyeon Han,Bon-Woo Koo,Noah Posner,Subhrajit Guhathakurta,Alexander Lerch机构：CSPAV, bMusic Informatics Group, and cIPaT, Georgia Institute of Technology, Toronto Metropolitan University备注：4+1 pages摘要：我们介绍了新的音频分析任务的行人检测，并提出了一个新的大规模的数据集，这项任务。虽然初步结果证明了使用音频方法进行行人检测的可行性，但它们也表明，使用标准方法无法轻松解决这一具有挑战性的任务。摘要：We introduce the new audio analysis task of pedestrian detection and present a new large-scale dataset for this task. While the preliminary results prove the viability of using audio approaches for pedestrian detection, they also show that this challenging task cannot be easily solved with standard approaches.
【8】 Weakly-Supervised Multi-Task Learning for Audio-Visual Speaker  Verification标题：用于视听说话人确认的弱监督多任务学习链接：https://arxiv.org/abs/2309.07115
作者：Anith Selvakumar,Homa Fashandi机构：LG Electronics, Toronto AI Lab摘要：在本文中，我们提出了一种方法来实现强大的多模态的人表示优化开放集视听说话人验证。距离度量学习（DML）方法通常在这个问题空间中占主导地位，这是由于在新的和看不见的类上的强大性能。在我们的工作中，我们探索了多任务学习技术，以进一步提高DML方法的性能，并表明具有弱标签的辅助任务可以增加学习的说话人表示的紧凑性。我们还将广义端到端损失（GE 2 E）扩展到多模态输入，并证明它可以在视听空间中实现有竞争力的性能。最后，我们介绍了一个非同步的视听采样随机策略，在训练时间，已被证明可以提高泛化。我们的网络在说话人验证方面达到了最先进的性能，在VoxCeleb 1-O/E/H的三个官方试验列表上报告了0.244%，0.252%，0.441%的等错误率（EER），据我们所知，这是VoxCeleb 1-E和VoxCeleb 1-H上公布的最佳结果。摘要：In this paper, we present a methodology for achieving robust multimodal person representations optimized for open-set audio-visual speaker verification. Distance Metric Learning (DML) approaches have typically dominated this problem space, owing to strong performance on new and unseen classes. In our work, we explored multitask learning techniques to further boost performance of the DML approach and show that an auxiliary task with weak labels can increase the compactness of the learned speaker representation. We also extend the Generalized end-to-end loss (GE2E) to multimodal inputs and demonstrate that it can achieve competitive performance in an audio-visual space. Finally, we introduce a non-synchronous audio-visual sampling random strategy during training time that has shown to improve generalization. Our network achieves state of the art performance for speaker verification, reporting 0.244%, 0.252%, 0.441% Equal Error Rate (EER) on the three official trial lists of VoxCeleb1-O/E/H, which is to our knowledge, the best published results on VoxCeleb1-E and VoxCeleb1-H.
【9】 MASTERKEY: Practical Backdoor Attack Against Speaker Verification  Systems标题：MasterKey：针对说话人验证系统的实用后门攻击链接：https://arxiv.org/abs/2309.06981
作者：Hanqing Guo,Xun Chen,Junfeng Guo,Li Xiao,Qiben Yan机构：Michigan State University, East Lansing, Michigan, USA, Samsung Research America, Mountain View, California, USA, UT Dallas, Dallas, Texas, USA备注：Accepted by Mobicom 2023摘要：说话人验证（Speaker Verification，SV）是一种广泛应用于移动通信系统中的语音识别技术。在这项工作中，我们提出了一个后门攻击MASTERKEY，妥协的SV模型。不同于以往的攻击，我们专注于现实世界的实际设置，攻击者不知道的预期受害者。为了设计MASTERKEY，我们调查了现有的中毒攻击对看不见的目标的限制。然后，我们优化了一个通用的后门，能够攻击任意目标。接下来，我们将说话人的特征和语义信息嵌入到后门中，使其不可感知。最后，我们估计信道失真，并将其集成到后门。我们验证我们的攻击6流行的SV模型。具体来说，我们总共毒害了53个模型，并使用我们的触发器攻击了16，430个注册扬声器，其中包括53个中毒模型中注册的310个目标扬声器。我们的攻击达到100%的攻击成功率和15%的中毒率。通过将中毒率降低到3%，攻击成功率保持在50%左右。我们验证我们的攻击在3个现实世界的情况下，并成功地证明了攻击通过空中和电话线的情况。摘要：Speaker Verification (SV) is widely deployed in mobile systems to authenticate legitimate users by using their voice traits. In this work, we propose a backdoor attack MASTERKEY, to compromise the SV models. Different from previous attacks, we focus on a real-world practical setting where the attacker possesses no knowledge of the intended victim. To design MASTERKEY, we investigate the limitation of existing poisoning attacks against unseen targets. Then, we optimize a universal backdoor that is capable of attacking arbitrary targets. Next, we embed the speaker's characteristics and semantics information into the backdoor, making it imperceptible. Finally, we estimate the channel distortion and integrate it into the backdoor. We validate our attack on 6 popular SV models. Specifically, we poison a total of 53 models and use our trigger to attack 16,430 enrolled speakers, composed of 310 target speakers enrolled in 53 poisoned models. Our attack achieves 100% attack success rate with a 15% poison rate. By decreasing the poison rate to 3%, the attack success rate remains around 50%. We validate our attack in 3 real-world scenarios and successfully demonstrate the attack through both over-the-air and over-the-telephony-line scenarios.
【10】 EMALG: An Enhanced Mandarin Lombard Grid Corpus with Meaningful  Sentences标题：EMALG：一个增强的具有有意义句子的普通话Lombard网格语料库链接：https://arxiv.org/abs/2309.06858
作者：Baifeng Li,Qingmu Liu,Yuhong Yang,Hongyang Chen,Weiping Tu,Song Lin机构：National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University, Wuhan, China.,  Hubei Key Laboratory of Multimedia and Network Communication Engineering,  Guangdong OPPO Mobile Telecommunications Corp., China.摘要：本研究调查了隆巴德效应，即个体在嘈杂环境中调整自己的言语。我们介绍了一个增强的普通话伦巴底格（EMALG）语料库的有意义的句子，增强普通话伦巴底格（MALG）语料库。EMALG拥有34个扬声器，并改进了录音设置，解决了MALG面临的无意义句子的挑战。我们的研究结果表明，在普通话中，女性表现出比男性更明显的隆巴德效应，特别是在说出有意义的句子。此外，我们发现，无意义的句子负面影响隆巴德效应分析。此外，我们的研究结果再次证实了以前的研究中发现的英语和普通话之间的隆巴德效应比较的一致性。摘要：This study investigates the Lombard effect, where individuals adapt their speech in noisy environments. We introduce an enhanced Mandarin Lombard grid (EMALG) corpus with meaningful sentences , enhancing the Mandarin Lombard grid (MALG) corpus. EMALG features 34 speakers and improves recording setups, addressing challenges faced by MALG with nonsense sentences. Our findings reveal that in Mandarin, female exhibit a more pronounced Lombard effect than male, particularly when uttering meaningful sentences. Additionally, we uncover that nonsense sentences negatively impact Lombard effect analysis. Moreover, our results reaffirm the consistency in the Lombard effect comparison between English and Mandarin found in previous research.
【11】 DCTTS: Discrete Diffusion Model with Contrastive Learning for  Text-to-speech Generation标题：DCTTS：基于对比学习的文语转换离散扩散模型链接：https://arxiv.org/abs/2309.06787
作者：Zhichao Wu,Qiulin Li,Sixing Liu,Qun Yang机构：Nanjing University of Aeronautics and Astronautics,Nanjing,China备注：5 pages, submitted to ICASSP摘要：在文语转换（TTS）任务中，隐扩散模型具有良好的保真度和泛化能力，但其昂贵的资源消耗和缓慢的推理速度一直是一个挑战。本文提出了一种基于对比学习的离散扩散模型的文语转换生成算法。DCTTS做出了以下贡献：1）基于离散空间的TTS扩散模型，显著降低了扩散模型的计算量，提高了采样速度; 2）采用基于离散空间的对比学习方法，增强语音与文本的对齐连接，提高采样质量; 3）采用高效的文本编码器简化模型参数，提高计算效率。实验结果表明，本文提出的方法具有优异的语音合成质量和采样速度，同时显著降低了扩散模型的资源消耗。合成的样品可在https://github.com/lawtherWu/DCTTS获得。摘要：In the Text-to-speech(TTS) task, the latent diffusion model has excellent fidelity and generalization, but its expensive resource consumption and slow inference speed have always been a challenging. This paper proposes Discrete Diffusion Model with Contrastive Learning for Text-to-Speech Generation(DCTTS). The following contributions are made by DCTTS: 1) The TTS diffusion model based on discrete space significantly lowers the computational consumption of the diffusion model and improves sampling speed; 2) The contrastive learning method based on discrete space is used to enhance the alignment connection between speech and text and improve sampling quality; and 3) It uses an efficient text encoder to simplify the model's parameters and increase computational efficiency. The experimental results demonstrate that the approach proposed in this paper has outstanding speech synthesis quality and sampling speed while significantly reducing the resource consumption of diffusion model. The synthesized samples are available at https://github.com/lawtherWu/DCTTS.
【12】 Distinguishing Neural Speech Synthesis Models Through Fingerprints in  Speech Waveforms标题：利用语音波形指纹识别神经语音合成模型链接：https://arxiv.org/abs/2309.06780
作者：Chu Yuan Zhang,Jiangyan Yi,Jianhua Tao,Chenglong Wang,Xinrui Yan机构：⋆International College, University of Chinese Academy of Sciences, Beijing, China, †State Key Laboratory of Multi-modal Artificial Intelligence Systems, Institute of Automation, §Department of Automation, Tsinghua University, Beijing, China备注：Submitted to ICASSP 2024摘要：神经语音合成技术的最新进展虽然得到了广泛的应用，但也带来了一系列挑战，激发了人们对防范误用和滥用威胁的兴趣。值得注意的是，合成语音的源归属在取证和知识产权保护中具有价值，但是在该领域中的先前工作在范围上具有某些限制。为了解决的差距，我们提出了我们的研究结果，在本文中的合成语音的来源的识别。我们调查的语音合成模型指纹在生成的语音波形中的存在，重点是声学模型和声码器，并研究在整体语音波形的指纹上的每个组件的影响。我们的研究使用多说话人LibriTTS数据集进行，展示了两个关键见解：（1）声码器和声学模型在它们生成的波形上赋予不同的、模型特定的指纹，以及（2）声码器指纹是这两者中更占主导地位的，并且可以从声学模型中掩盖指纹。这些研究结果强烈表明，存在模型特定的指纹的声学模型和声码器，突出其在源识别应用的潜在效用。摘要：Recent strides in neural speech synthesis technologies, while enjoying widespread applications, have nonetheless introduced a series of challenges, spurring interest in the defence against the threat of misuse and abuse. Notably, source attribution of synthesized speech has value in forensics and intellectual property protection, but prior work in this area has certain limitations in scope. To address the gaps, we present our findings concerning the identification of the sources of synthesized speech in this paper. We investigate the existence of speech synthesis model fingerprints in the generated speech waveforms, with a focus on the acoustic model and the vocoder, and study the influence of each component on the fingerprint in the overall speech waveforms. Our research, conducted using the multi-speaker LibriTTS dataset, demonstrates two key insights: (1) vocoders and acoustic models impart distinct, model-specific fingerprints on the waveforms they generate, and (2) vocoder fingerprints are the more dominant of the two, and may mask the fingerprints from the acoustic model. These findings strongly suggest the existence of model-specific fingerprints for both the acoustic model and the vocoder, highlighting their potential utility in source identification applications.
【13】 Leveraging Foundation models for Unsupervised Audio-Visual Segmentation标题：利用基础模型实现无监督的视听分割链接：https://arxiv.org/abs/2309.06728
作者：Swapnil Bhosale,Haosen Yang,Diptesh Kanojia,Xiatian Zhu机构：University of Surrey, UK摘要：视听分割（AVS）的目的是在像素级上精确地勾勒出视觉场景中的可听对象。现有的AVS方法需要以监督学习方式对音频掩码对进行细粒度注释。这限制了它们的可扩展性，因为获取这种跨模态像素级标签是耗时且繁琐的。为了克服这一障碍，在这项工作中，我们引入了无监督的视听分割，不需要特定于任务的数据注释和模型训练。为了解决这个新提出的问题，我们制定了一种新的跨模态语义过滤（CMSF）方法，以通过利用现成的多模态基础模型（例如，检测[1]、开放世界分割[2]和多模态对齐[3]）。通过音频或视觉提示指导提案生成，我们设计了两种免训练变体：AT-GDINO-SAM和OWOD-BIND。在AVS-Bench数据集上进行的大量实验表明，与现有技术的有监督的方法相比，我们的无监督方法在具有多个听觉对象的复杂场景中可以表现良好。特别是，在现有的监督AVS方法与重叠的前景对象斗争的情况下，我们的模型仍然擅长准确地分割重叠的听觉对象。我们的代码将公开发布。摘要：Audio-Visual Segmentation (AVS) aims to precisely outline audible objects in a visual scene at the pixel level. Existing AVS methods require fine-grained annotations of audio-mask pairs in supervised learning fashion. This limits their scalability since it is time consuming and tedious to acquire such cross-modality pixel level labels. To overcome this obstacle, in this work we introduce unsupervised audio-visual segmentation with no need for task-specific data annotations and model training. For tackling this newly proposed problem, we formulate a novel Cross-Modality Semantic Filtering (CMSF) approach to accurately associate the underlying audio-mask pairs by leveraging the off-the-shelf multi-modal foundation models (e.g., detection [1], open-world segmentation [2] and multi-modal alignment [3]). Guiding the proposal generation by either audio or visual cues, we design two training-free variants: AT-GDINO-SAM and OWOD-BIND. Extensive experiments on the AVS-Bench dataset show that our unsupervised approach can perform well in comparison to prior art supervised counterparts across complex scenarios with multiple auditory objects. Particularly, in situations where existing supervised AVS methods struggle with overlapping foreground objects, our models still excel in accurately segmenting overlapped auditory objects. Our code will be publicly released.
【14】 PIAVE: A Pose-Invariant Audio-Visual Speaker Extraction Network标题：Piave：一种姿态不变的视听说话人提取网络链接：https://arxiv.org/abs/2309.06723
作者：Qinghua Liu,Meng Ge,Zhizheng Wu,Haizhou Li机构：Shenzhen Research Institute of Big Data, Shenzhen, China, School of Data Science, The Chinese University of Hong Kong, Shenzhen, China,  Department of Electrical and Computer Engineering, National University of Singapore, Singapore备注：None摘要：在日常口语交流中，我们经常会看着说话者的转动的头部来听他/她的声音。人类看到说话者是为了更好地倾听，机器也是如此。然而，以往的研究视听说话人提取没有有效地处理不断变化的说话人脸。本文研究如何充分利用变化的说话人脸。我们提出了一个姿势不变的视听说话人提取网络（PIAVE），它采用了一个额外的姿势不变的观点，以提高视听说话人提取。具体地，我们从每个原始姿势取向生成姿势不变视图，这使得模型能够接收讲话者的一致的正面视图，而不管他/她的头部姿势，因此，形成用于讲话者的多视图视觉输入。在多视角MEAD和野外LRS 3数据集上的实验表明，PIAVE优于现有技术，并且对姿态变化更具鲁棒性。摘要：It is common in everyday spoken communication that we look at the turning head of a talker to listen to his/her voice. Humans see the talker to listen better, so do machines. However, previous studies on audio-visual speaker extraction have not effectively handled the varying talking face. This paper studies how to take full advantage of the varying talking face. We propose a Pose-Invariant Audio-Visual Speaker Extraction Network (PIAVE) that incorporates an additional pose-invariant view to improve audio-visual speaker extraction. Specifically, we generate the pose-invariant view from each original pose orientation, which enables the model to receive a consistent frontal view of the talker regardless of his/her head pose, therefore, forming a multi-view visual input for the speaker. Experiments on the multi-view MEAD and in-the-wild LRS3 dataset demonstrate that PIAVE outperforms the state-of-the-art and is more robust to pose variations.
【15】 Attention-based Encoder-Decoder End-to-End Neural Diarization with  Embedding Enhancer标题：嵌入增强器的基于注意力的编解码端到端神经元二值化链接：https://arxiv.org/abs/2309.06672
作者：Zhengyang Chen,Bing Han,Shuai Wang,Yanmin Qian备注：IEEE/ACM Transactions on Audio Speech and Language Processing Under Review摘要：基于深度神经网络的系统显着提高了说话人日记任务的性能。然而，端到端神经日志化（EEND）系统通常难以推广到具有不可见数量的说话者的场景，而目标说话者语音活动检测（TS-VAD）系统往往过于复杂。在本文中，我们提出了一个简单的基于注意力的编码器-解码器网络的端到端神经日记（AED-EEND）。在我们的训练过程中，我们引入了一个教师强迫策略来解决说话人排列问题，从而加快模型收敛。为了评估，我们提出了一种迭代解码方法，输出diarization结果为每个扬声器顺序。此外，我们提出了一个增强器模块，以增强帧级扬声器嵌入，使模型能够处理的情况下，看不见的扬声器数量。我们还探索用Conformer架构替换Transformer编码器，该架构可以更好地模拟本地信息。此外，我们发现，常用的模拟数据集的说话人日记有一个更高的重叠率相比，真实的数据。我们发现，使用与真实数据更一致的模拟训练数据可以实现一致性的改善。广泛的实验验证表明，我们提出的方法的有效性。我们最好的系统实现了一个新的国家的最先进的日记错误率（DER）的性能在所有的CALLHOME（10.08%），DIHARD II（24.64%），和AMI（13.00%）的评估基准，当没有使用oracle语音活动检测（VAD）。除了扬声器diarization，我们的AED-EEND系统也表现出显着的竞争力，作为一个语音类型检测模型。摘要：Deep neural network-based systems have significantly improved the performance of speaker diarization tasks. However, end-to-end neural diarization (EEND) systems often struggle to generalize to scenarios with an unseen number of speakers, while target speaker voice activity detection (TS-VAD) systems tend to be overly complex. In this paper, we propose a simple attention-based encoder-decoder network for end-to-end neural diarization (AED-EEND). In our training process, we introduce a teacher-forcing strategy to address the speaker permutation problem, leading to faster model convergence. For evaluation, we propose an iterative decoding method that outputs diarization results for each speaker sequentially. Additionally, we propose an Enhancer module to enhance the frame-level speaker embeddings, enabling the model to handle scenarios with an unseen number of speakers. We also explore replacing the transformer encoder with a Conformer architecture, which better models local information. Furthermore, we discovered that commonly used simulation datasets for speaker diarization have a much higher overlap ratio compared to real data. We found that using simulated training data that is more consistent with real data can achieve an improvement in consistency. Extensive experimental validation demonstrates the effectiveness of our proposed methodologies. Our best system achieved a new state-of-the-art diarization error rate (DER) performance on all the CALLHOME (10.08%), DIHARD II (24.64%), and AMI (13.00%) evaluation benchmarks, when no oracle voice activity detection (VAD) is used. Beyond speaker diarization, our AED-EEND system also shows remarkable competitiveness as a speech type detection model.
【16】 Differentiable Modelling of Percussive Audio with Transient and Spectral  Synthesis标题：基于暂态和频谱合成的敲击声可微建模链接：https://arxiv.org/abs/2309.06649
作者：Jordie Shier,Franco Caspe,Andrew Robertson,Mark Sandler,Charalampos Saitis,Andrew McPherson机构：Centre for Digital Music, Queen Mary University of London, United Kingdom,  Ableton AG, Berlin, Germany,  Dyson School of Design Engineering, Imperial College London, United Kingdom备注：To be published in The Proceedings of Forum Acusticum, Sep 2023, Turin, Italy摘要：可微分数字信号处理（DDSP）技术，包括用于音频合成的方法，近年来已经获得关注，并且使它们自身在参数空间中具有可解释性。然而，当前的可微分合成方法尚未明确地寻求对信号的瞬态部分进行建模，这对于打击声音是重要的。在这项工作中，我们提出了一个统一的合成框架，旨在解决瞬态生成和打击乐合成内的DDSP框架。为此，我们提出了一个打击合成模型，建立在正弦建模合成，并采用了调制的时间卷积网络的瞬态生成。我们使用修改后的正弦峰值拾取算法来生成时变非谐波正弦曲线，并将其与共同训练以重建鼓组声音的可微分噪声和瞬态编码器配对。我们使用声学和电子打击样本的大数据集计算一组重建度量，这表明我们的方法导致膜乐器打击乐器的改善的起始信号重建。摘要：Differentiable digital signal processing (DDSP) techniques, including methods for audio synthesis, have gained attention in recent years and lend themselves to interpretability in the parameter space. However, current differentiable synthesis methods have not explicitly sought to model the transient portion of signals, which is important for percussive sounds. In this work, we present a unified synthesis framework aiming to address transient generation and percussive synthesis within a DDSP framework. To this end, we propose a model for percussive synthesis that builds on sinusoidal modeling synthesis and incorporates a modulated temporal convolutional network for transient generation. We use a modified sinusoidal peak picking algorithm to generate time-varying non-harmonic sinusoids and pair it with differentiable noise and transient encoders that are jointly trained to reconstruct drumset sounds. We compute a set of reconstruction metrics using a large dataset of acoustic and electronic percussion samples that show that our method leads to improved onset signal reconstruction for membranophone percussion instruments.
机器翻译，仅供参考