今日论文合集：cs.SD语音7篇，eess.AS音频处理10篇。
cs.SD语音
【1】 All-for-One and One-For-All: Deep learning-based feature fusion for  Synthetic Speech Detection标题：All-For-One和One-For-All：基于深度学习的特征融合合成语音检测链接：https://arxiv.org/abs/2307.15555
作者：Daniele Mari,Davide Salvi,Paolo Bestagini,Simone Milani机构：University of Padova - Padova, Italy,  Politecnico di Milano - Milan, Italy备注：Accepted at ECML-PKDD 2023 Workshop "Deep Learning and Multimedia Forensics. Combating fake media and misinformation"摘要：深度学习和计算机视觉的最新进展使得多媒体内容的合成和伪造比以往任何时候都更容易获得，从而导致恶意用户可能带来的威胁和危险。在音频领域，我们正在见证语音深度伪造生成技术的发展，这些技术要求开发合成语音检测算法，以对抗可能的恶意使用，如欺诈或身份盗窃。在本文中，我们考虑了三个不同的特征集在文献中提出的合成语音检测任务，并提出了一个模型，融合它们，实现整体更好的性能相对于国家的最先进的解决方案。该系统在不同的场景和数据集上进行了测试，以证明其对反取证攻击的鲁棒性和泛化能力。摘要：Recent advances in deep learning and computer vision have made the synthesis and counterfeiting of multimedia content more accessible than ever, leading to possible threats and dangers from malicious users. In the audio field, we are witnessing the growth of speech deepfake generation techniques, which solicit the development of synthetic speech detection algorithms to counter possible mischievous uses such as frauds or identity thefts. In this paper, we consider three different feature sets proposed in the literature for the synthetic speech detection task and present a model that fuses them, achieving overall better performances with respect to the state-of-the-art solutions. The system was tested on different scenarios and datasets to prove its robustness to anti-forensic attacks and its generalization capabilities.
【2】 Automated approach for source location in shallow waters标题：浅海水源地自动定位方法链接：https://arxiv.org/abs/2307.15491
作者：Angèle Niclas,Josselin Garnier机构：CMAP, CNRS, ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France摘要：本文提出了一种全自动的方法恢复源的位置和介质参数在浅水中。该方案涉及在浅水环境中发射低频声波的未知源，以及记录信号的单个水听器。首先，引入理论工具来理解翘曲方法的鲁棒性，并提出和分析一种自动化的方式来分离所记录的信号的模态分量。其次，使用每个模态分量的频谱图，本文研究了恢复模态行程时间的最佳方法，并提供稳定性估计。最后，一个惩罚最小化算法恢复估计的源位置和介质参数。所提出的方法进行了测试的实验数据的露脊鲸枪击和燃烧声源，证明了其在现实世界中的场景的有效性。摘要：This paper proposes a fully automated method for recovering the location of a source and medium parameters in shallow waters. The scenario involves an unknown source emitting low-frequency sound waves in a shallow water environment, and a single hydrophone recording the signal. Firstly, theoretical tools are introduced to understand the robustness of the warping method and to propose and analyze an automated way to separate the modal components of the recorded signal. Secondly, using the spectrogram of each modal component, the paper investigates the best way to recover the modal travel times and provides stability estimates. Finally, a penalized minimization algorithm is presented to recover estimates of the source location and medium parameters. The proposed method is tested on experimental data of right whale gunshot and combustive sound sources, demonstrating its effectiveness in real-world scenarios.
【3】 Minimally-Supervised Speech Synthesis with Conditional Diffusion Model  and Language Model: A Comparative Study of Semantic Coding标题：基于条件扩散模型和语言模型的最小监督语音合成：语义编码的比较研究链接：https://arxiv.org/abs/2307.15484
作者：Chunyu Qiang,Hao Li,Hao Ni,He Qu,Ruibo Fu,Tao Wang,Longbiao Wang,Jianwu Dang机构：College of Intelligence and Computing, Tianjin University, Tianjin, China, Kuaishou Technology Co., Ltd, Beijing, China, Institute of Automation, Chinese Academy of Sciences, Beijing, China摘要：最近，人们对文本到语音（TTS）方法越来越感兴趣，该方法可以通过组合两种类型的离散语音表示并使用两个序列到序列任务来解耦TTS来以最小的监督进行训练。为了解决与离散表示中的高维数和波形失真相关的挑战，我们提出了Diff-LM-Speech，它基于扩散模型将语义嵌入到Mel频谱图中，并引入了基于变分自动编码器和韵律瓶颈的提示编码器结构，以提高提示表示能力。自回归语言模型经常遭受丢失和重复的单词，而非自回归框架由于持续时间预测模型而面临表达平均化问题。为了解决这些问题，我们提出了Tetra-Diff-Speech，它设计了一个持续时间扩散模型来实现不同的韵律表达。虽然我们期望语义编码的信息内容介于文本和声学编码之间，但现有模型提取的语义编码具有大量的冗余信息和维度爆炸。为了验证语义编码是不必要的，我们提出了三差语音。实验结果表明，我们提出的方法优于基线方法。我们提供了一个网站的音频样本。摘要：Recently, there has been a growing interest in text-to-speech (TTS) methods that can be trained with minimal supervision by combining two types of discrete speech representations and using two sequence-to-sequence tasks to decouple TTS. To address the challenges associated with high dimensionality and waveform distortion in discrete representations, we propose Diff-LM-Speech, which models semantic embeddings into mel-spectrogram based on diffusion models and introduces a prompt encoder structure based on variational autoencoders and prosody bottlenecks to improve prompt representation capabilities. Autoregressive language models often suffer from missing and repeated words, while non-autoregressive frameworks face expression averaging problems due to duration prediction models. To address these issues, we propose Tetra-Diff-Speech, which designs a duration diffusion model to achieve diverse prosodic expressions. While we expect the information content of semantic coding to be between that of text and acoustic coding, existing models extract semantic coding with a lot of redundant information and dimensionality explosion. To verify that semantic coding is not necessary, we propose Tri-Diff-Speech. Experimental results show that our proposed methods outperform baseline methods. We provide a website with audio samples.
【4】 The FlySpeech Audio-Visual Speaker Diarization System for MISP Challenge  2022标题：FlySpeech视听说话人对讲机2022挑战赛链接：https://arxiv.org/abs/2307.15400
作者：Li Zhang,Huan Zhao,Yue Li,Bowen Pang,Yannan Wang,Hongji Wang,Wei Rao,Qing Wang,Lei Xie机构：Northwestern Polytechnical University (NPU), Xi’an, China, Tencent Corporation, Shenzhen, China摘要：本文介绍了在ICASSP 2022举办的第二届基于\textbf{M}多模态\textbf{I}信息的\textbf{S}语音\textbf{P}处理~（\textbf{MISP}）挑战赛中提交的FlySpeech说话人日志化系统。我们开发了一个端到端的音视频说话人日记化系统，它包括一个唇编码器，一个说话人编码器，和一个音视频解码器。具体来说，为了减轻由单独训练引起的日记化性能的下降，我们联合训练扬声器编码器和视听解码器。此外，我们利用大数据预训练的扬声器提取器来初始化扬声器编码器。摘要：This paper describes the FlySpeech speaker diarization system submitted to the second \textbf{M}ultimodal \textbf{I}nformation Based \textbf{S}peech \textbf{P}rocessing~(\textbf{MISP}) Challenge held in ICASSP 2022. We develop an end-to-end audio-visual speaker diarization~(AVSD) system, which consists of a lip encoder, a speaker encoder, and an audio-visual decoder. Specifically, to mitigate the degradation of diarization performance caused by separate training, we jointly train the speaker encoder and the audio-visual decoder. In addition, we leverage the large-data pretrained speaker extractor to initialize the speaker encoder.
【5】 Improving Audio-Text Retrieval via Hierarchical Cross-Modal Interaction  and Auxiliary Captions标题：通过层次化跨通道交互和辅助字幕改进音文检索链接：https://arxiv.org/abs/2307.15344
作者：Yifei Xin,Yuexian Zou机构：School of ECE, Peking University, Shenzhen, China备注：Accepted by Interspeech2023摘要：大多数现有的音频文本检索（ATR）方法专注于在整个音频片段和完整的字幕句子之间构建对比对，而忽略了细粒度的跨模态关系，例如，短片段和短语或框架和单词。在本文中，我们介绍了一个层次的跨模态交互（HCI）的ATR方法，同时探索剪辑句子，段短语，框架词的关系，实现了全面的多模态语义比较。此外，我们还提出了一种新的ATR框架，利用辅助字幕（AC）生成的预训练字幕执行音频和生成的字幕之间的功能交互，这产生了增强的音频表示，并补充到原来的ATR匹配分支。音频和生成的字幕还可以形成新的音频-文本对作为用于训练的数据增强。实验表明，我们的HCI显着提高ATR性能。此外，我们的AC框架在多个数据集上也显示出稳定的性能增益。摘要：Most existing audio-text retrieval (ATR) methods focus on constructing contrastive pairs between whole audio clips and complete caption sentences, while ignoring fine-grained cross-modal relationships, e.g., short segments and phrases or frames and words. In this paper, we introduce a hierarchical cross-modal interaction (HCI) method for ATR by simultaneously exploring clip-sentence, segment-phrase, and frame-word relationships, achieving a comprehensive multi-modal semantic comparison. Besides, we also present a novel ATR framework that leverages auxiliary captions (AC) generated by a pretrained captioner to perform feature interaction between audio and generated captions, which yields enhanced audio representations and is complementary to the original ATR matching branch. The audio and generated captions can also form new audio-text pairs as data augmentation for training. Experiments show that our HCI significantly improves the ATR performance. Moreover, our AC framework also shows stable performance gains on multiple datasets.
【6】 PCNN: A Lightweight Parallel Conformer Neural Network for Efficient  Monaural Speech Enhancement标题：PCNN：一种高效单声道语音增强的轻量级并行构形神经网络链接：https://arxiv.org/abs/2307.15251
作者：Xinmeng Xu,Weiping Tu,Yuhong Yang机构：NERCMS, School of Computer Science, Wuhan University, China, Hubei Luojia Laboratory, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan备注：Accepted at INTERSPEECH 2023摘要：卷积神经网络（CNN）和Transformer在多媒体应用中取得了广泛的成功。然而，需要做出更多的努力来有效地协调这两种架构，以满足语音增强。本文的目的是统一这两种架构，并提出了一个并行的语音增强Conformer。特别是，CNN和Transformer中的自注意力（SA）被充分利用于局部格式模式和全局结构表示。基于CNN的小感受野尺寸和SA的高计算复杂度，我们专门设计了多分支扩张卷积（MBDC）和自通道时频注意（Self-CTFA）模块。MBDC包含三个卷积层，对于从局部到非局部处理的特征具有不同的膨胀率。实验结果表明，我们的方法比国家的最先进的方法在大多数评价标准，同时保持最低的模型参数。摘要：Convolutional neural networks (CNN) and Transformer have wildly succeeded in multimedia applications. However, more effort needs to be made to harmonize these two architectures effectively to satisfy speech enhancement. This paper aims to unify these two architectures and presents a Parallel Conformer for speech enhancement. In particular, the CNN and the self-attention (SA) in the Transformer are fully exploited for local format patterns and global structure representations. Based on the small receptive field size of CNN and the high computational complexity of SA, we specially designed a multi-branch dilated convolution (MBDC) and a self-channel-time-frequency attention (Self-CTFA) module. MBDC contains three convolutional layers with different dilation rates for the feature from local to non-local processing. Experimental results show that our method performs better than state-of-the-art methods in most evaluation criteria while maintaining the lowest model parameters.
【7】 Detection of Children Abuse by Voice and Audio Classification by  Short-Time Fourier Transform Machine Learning implemented on Nvidia Edge GPU  device标题：NVIDIA Edge GPU设备上实现的短时傅里叶变换机器学习的语音和音频分类链接：https://arxiv.org/abs/2307.15101
作者：Jiuqi Yan,Yingxian Chen,W. W. T. Fok机构：Electrical and Electronic Engineering ,The University of Hong Kong, Hong Kong备注：5 pages, 7 figures, PRAI 2023摘要：儿童之家中儿童的安全问题已成为社会日益关注的问题，本实验的目的是利用机器学习应用于儿童虐待场景的检测，以提高儿童的安全性。这个实验使用机器学习来分类和识别孩子的声音，并预测孩子当前发出的声音是哭、尖叫还是笑。如果发现孩子在哭或尖叫，会立即向相关人员发出警报，以便他们在监控盲区感知孩子可能正在经历的事情，并及时做出反应。与视频图像分类的混合使用一起，可以显著提高儿童虐待检测的准确性。这大大降低了儿童在托儿所受到暴力虐待的可能性，并允许工作人员及时阻止即将发生或初期的儿童虐待事件。本实验收集的数据集完全来自儿童之家现场录制的声音，包括哭泣、大笑、尖叫声和背景噪音。使用短时傅里叶变换将这些声音文件转换为频谱图，然后将这些图像数据导入CNN神经网络进行分类，最终训练的模型可以达到约92%的声音检测准确率。摘要：The safety of children in children home has become an increasing social concern, and the purpose of this experiment is to use machine learning applied to detect the scenarios of child abuse to increase the safety of children. This experiment uses machine learning to classify and recognize a child's voice and predict whether the current sound made by the child is crying, screaming or laughing. If a child is found to be crying or screaming, an alert is immediately sent to the relevant personnel so that they can perceive what the child may be experiencing in a surveillance blind spot and respond in a timely manner. Together with a hybrid use of video image classification, the accuracy of child abuse detection can be significantly increased. This greatly reduces the likelihood that a child will receive violent abuse in the nursery and allows personnel to stop an imminent or incipient child abuse incident in time. The datasets collected from this experiment is entirely from sounds recorded on site at the children home, including crying, laughing, screaming sound and background noises. These sound files are transformed into spectrograms using Short-Time Fourier Transform, and then these image data are imported into a CNN neural network for classification, and the final trained model can achieve an accuracy of about 92% for sound detection.
eess.AS音频处理
【1】 Efficient Acoustic Echo Suppression with Condition-Aware Training标题：基于状态感知训练的声学回波抑制链接：https://arxiv.org/abs/2307.15630
作者：Ernst Seidel,Pejman Mowlaee,Tim Fingscheidt机构：∗Institute for Communications Technology, Technische Universit¨at Braunschweig, Schleinitzstraße , Braunschweig, Germany, ◦GN Audio AS, Lautrupbjerg , Ballerup, Denmark备注：5 pages, accepted to WASPAA 2023摘要：近年来，深层声学回声控制（DAEC）的主题已经看到了许多具有各种模型拓扑结构的方法。卷积递归网络（CRN），包括一个卷积编码器和解码器，包括一个经常性的瓶颈，被反复采用，由于他们的能力，以保持nearend语音，即使在双说话（DT）的条件。然而，过去的架构要么计算复杂，要么在性能降低的情况下权衡较小的模型大小。我们提出了一种改进的CRN拓扑结构，这类架构的其他实现相比，不仅节省了参数和计算复杂度，但也显示出改善的性能在DT，优于基线架构FCRN和CRUSE。为了实现条件感知的训练，我们还证明了DAEC训练数据中高比例的双重谈话和仅接近语音的缺失值的重要性。最后，我们展示了如何控制积极的回声抑制和近端语音保存之间的权衡与条件感知组件损失函数的微调。摘要：The topic of deep acoustic echo control (DAEC) has seen many approaches with various model topologies in recent years. Convolutional recurrent networks (CRNs), consisting of a convolutional encoder and decoder encompassing a recurrent bottleneck, are repeatedly employed due to their ability to preserve nearend speech even in double-talk (DT) condition. However, past architectures are either computationally complex or trade off smaller model sizes with a decrease in performance. We propose an improved CRN topology which, compared to other realizations of this class of architectures, not only saves parameters and computational complexity, but also shows improved performance in DT, outperforming both baseline architectures FCRN and CRUSE. Striving for a condition-aware training, we also demonstrate the importance of a high proportion of double-talk and the missing value of nearend-only speech in DAEC training data. Finally, we show how to control the trade-off between aggressive echo suppression and near-end speech preservation by fine-tuning with condition-aware component loss functions.
【2】 A Time-Frequency Generative Adversarial based method for Audio Packet  Loss Concealment标题：一种基于时频生成对抗的音频丢包隐藏方法链接：https://arxiv.org/abs/2307.15611
作者：Carlo Aironi,Samuele Cornell,Luca Serafini,Stefano Squartini机构：Department of Information Engineering, Italy备注：Accepted at EUSIPCO - 31st European Signal Processing Conference, 2023摘要：分组丢失是VoIP传输中语音质量下降的主要原因，严重影响可懂度和用户体验。本文描述了一种基于生成对抗方法的系统，其目的是修复音频流传输过程中丢失的片段。受生成对抗网络（GANs）强大的图像到图像翻译能力的启发，我们提出了bin2bin，这是一种改进的pix2pix框架，用于实现从丢失数据包的音频帧的幅度谱图到未损坏的语音谱图的翻译任务。为了更好地保持谱图翻译后的结构信息，本文引入了两个基于STFT的损失函数的组合，与传统的GAN目标混合。此外，我们采用了修改后的PatchGAN结构作为鉴别器，我们降低了隐藏时间通过适当的初始化的相位重建算法。实验结果表明，该方法与现有方法相比，具有明显的优势，能够更好地处理高丢包率和大间隙的情况。摘要：Packet loss is a major cause of voice quality degradation in VoIP transmissions with serious impact on intelligibility and user experience. This paper describes a system based on a generative adversarial approach, which aims to repair the lost fragments during the transmission of audio streams. Inspired by the powerful image-to-image translation capability of Generative Adversarial Networks (GANs), we propose bin2bin, an improved pix2pix framework to achieve the translation task from magnitude spectrograms of audio frames with lost packets, to noncorrupted speech spectrograms. In order to better maintain the structural information after spectrogram translation, this paper introduces the combination of two STFT-based loss functions, mixed with the traditional GAN objective. Furthermore, we employ a modified PatchGAN structure as discriminator and we lower the concealment time by a proper initialization of the phase reconstruction algorithm. Experimental results show that the proposed method has obvious advantages when compared with the current state-of-the-art methods, as it can better handle both high packet loss rates and large gaps.
【3】 PCNN: A Lightweight Parallel Conformer Neural Network for Efficient  Monaural Speech Enhancement标题：PCNN：一种高效单声道语音增强的轻量级并行构形神经网络链接：https://arxiv.org/abs/2307.15251
作者：Xinmeng Xu,Weiping Tu,Yuhong Yang机构：NERCMS, School of Computer Science, Wuhan University, China, Hubei Luojia Laboratory, China, Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan备注：Accepted at INTERSPEECH 2023摘要：卷积神经网络（CNN）和Transformer在多媒体应用中取得了广泛的成功。然而，需要做出更多的努力来有效地协调这两种架构，以满足语音增强。本文的目的是统一这两种架构，并提出了一个并行的语音增强Conformer。特别是，CNN和Transformer中的自注意力（SA）被充分利用于局部格式模式和全局结构表示。基于CNN的小感受野尺寸和SA的高计算复杂度，我们专门设计了多分支扩张卷积（MBDC）和自通道时频注意（Self-CTFA）模块。MBDC包含三个卷积层，对于从局部到非局部处理的特征具有不同的膨胀率。实验结果表明，我们的方法比国家的最先进的方法在大多数评价标准，同时保持最低的模型参数。摘要：Convolutional neural networks (CNN) and Transformer have wildly succeeded in multimedia applications. However, more effort needs to be made to harmonize these two architectures effectively to satisfy speech enhancement. This paper aims to unify these two architectures and presents a Parallel Conformer for speech enhancement. In particular, the CNN and the self-attention (SA) in the Transformer are fully exploited for local format patterns and global structure representations. Based on the small receptive field size of CNN and the high computational complexity of SA, we specially designed a multi-branch dilated convolution (MBDC) and a self-channel-time-frequency attention (Self-CTFA) module. MBDC contains three convolutional layers with different dilation rates for the feature from local to non-local processing. Experimental results show that our method performs better than state-of-the-art methods in most evaluation criteria while maintaining the lowest model parameters.
【4】 Detection of Children Abuse by Voice and Audio Classification by  Short-Time Fourier Transform Machine Learning implemented on Nvidia Edge GPU  device标题：NVIDIA Edge GPU设备上实现的短时傅里叶变换机器学习的语音和音频分类链接：https://arxiv.org/abs/2307.15101
作者：Jiuqi Yan,Yingxian Chen,W. W. T. Fok机构：Electrical and Electronic Engineering ,The University of Hong Kong, Hong Kong备注：5 pages, 7 figures, PRAI 2023摘要：儿童之家中儿童的安全问题已成为社会日益关注的问题，本实验的目的是利用机器学习应用于儿童虐待场景的检测，以提高儿童的安全性。这个实验使用机器学习来分类和识别孩子的声音，并预测孩子当前发出的声音是哭、尖叫还是笑。如果发现孩子在哭或尖叫，会立即向相关人员发出警报，以便他们在监控盲区感知孩子可能正在经历的事情，并及时做出反应。与视频图像分类的混合使用一起，可以显著提高儿童虐待检测的准确性。这大大降低了儿童在托儿所受到暴力虐待的可能性，并允许工作人员及时阻止即将发生或初期的儿童虐待事件。本实验收集的数据集完全来自儿童之家现场录制的声音，包括哭泣、大笑、尖叫声和背景噪音。使用短时傅里叶变换将这些声音文件转换为频谱图，然后将这些图像数据导入CNN神经网络进行分类，最终训练的模型可以达到约92%的声音检测准确率。摘要：The safety of children in children home has become an increasing social concern, and the purpose of this experiment is to use machine learning applied to detect the scenarios of child abuse to increase the safety of children. This experiment uses machine learning to classify and recognize a child's voice and predict whether the current sound made by the child is crying, screaming or laughing. If a child is found to be crying or screaming, an alert is immediately sent to the relevant personnel so that they can perceive what the child may be experiencing in a surveillance blind spot and respond in a timely manner. Together with a hybrid use of video image classification, the accuracy of child abuse detection can be significantly increased. This greatly reduces the likelihood that a child will receive violent abuse in the nursery and allows personnel to stop an imminent or incipient child abuse incident in time. The datasets collected from this experiment is entirely from sounds recorded on site at the children home, including crying, laughing, screaming sound and background noises. These sound files are transformed into spectrograms using Short-Time Fourier Transform, and then these image data are imported into a CNN neural network for classification, and the final trained model can achieve an accuracy of about 92% for sound detection.
【5】 All-for-One and One-For-All: Deep learning-based feature fusion for  Synthetic Speech Detection标题：All-For-One和One-For-All：基于深度学习的特征融合合成语音检测链接：https://arxiv.org/abs/2307.15555
作者：Daniele Mari,Davide Salvi,Paolo Bestagini,Simone Milani机构：University of Padova - Padova, Italy,  Politecnico di Milano - Milan, Italy备注：Accepted at ECML-PKDD 2023 Workshop "Deep Learning and Multimedia Forensics. Combating fake media and misinformation"摘要：深度学习和计算机视觉的最新进展使得多媒体内容的合成和伪造比以往任何时候都更容易获得，从而导致恶意用户可能带来的威胁和危险。在音频领域，我们正在见证语音深度伪造生成技术的发展，这些技术要求开发合成语音检测算法，以对抗可能的恶意使用，如欺诈或身份盗窃。在本文中，我们考虑了三个不同的特征集在文献中提出的合成语音检测任务，并提出了一个模型，融合它们，实现整体更好的性能相对于国家的最先进的解决方案。该系统在不同的场景和数据集上进行了测试，以证明其对反取证攻击的鲁棒性和泛化能力。摘要：Recent advances in deep learning and computer vision have made the synthesis and counterfeiting of multimedia content more accessible than ever, leading to possible threats and dangers from malicious users. In the audio field, we are witnessing the growth of speech deepfake generation techniques, which solicit the development of synthetic speech detection algorithms to counter possible mischievous uses such as frauds or identity thefts. In this paper, we consider three different feature sets proposed in the literature for the synthetic speech detection task and present a model that fuses them, achieving overall better performances with respect to the state-of-the-art solutions. The system was tested on different scenarios and datasets to prove its robustness to anti-forensic attacks and its generalization capabilities.
【6】 Automated approach for source location in shallow waters标题：浅海水源地自动定位方法链接：https://arxiv.org/abs/2307.15491
作者：Angèle Niclas,Josselin Garnier机构：CMAP, CNRS, ´Ecole polytechnique, Institut Polytechnique de Paris, Palaiseau, France摘要：本文提出了一种全自动的方法恢复源的位置和介质参数在浅水中。该方案涉及在浅水环境中发射低频声波的未知源，以及记录信号的单个水听器。首先，引入理论工具来理解翘曲方法的鲁棒性，并提出和分析一种自动化的方式来分离所记录的信号的模态分量。其次，使用每个模态分量的频谱图，本文研究了恢复模态行程时间的最佳方法，并提供稳定性估计。最后，一个惩罚最小化算法恢复估计的源位置和介质参数。所提出的方法进行了测试的实验数据的露脊鲸枪击和燃烧声源，证明了其在现实世界中的场景的有效性。摘要：This paper proposes a fully automated method for recovering the location of a source and medium parameters in shallow waters. The scenario involves an unknown source emitting low-frequency sound waves in a shallow water environment, and a single hydrophone recording the signal. Firstly, theoretical tools are introduced to understand the robustness of the warping method and to propose and analyze an automated way to separate the modal components of the recorded signal. Secondly, using the spectrogram of each modal component, the paper investigates the best way to recover the modal travel times and provides stability estimates. Finally, a penalized minimization algorithm is presented to recover estimates of the source location and medium parameters. The proposed method is tested on experimental data of right whale gunshot and combustive sound sources, demonstrating its effectiveness in real-world scenarios.
【7】 Minimally-Supervised Speech Synthesis with Conditional Diffusion Model  and Language Model: A Comparative Study of Semantic Coding标题：基于条件扩散模型和语言模型的最小监督语音合成：语义编码的比较研究链接：https://arxiv.org/abs/2307.15484
作者：Chunyu Qiang,Hao Li,Hao Ni,He Qu,Ruibo Fu,Tao Wang,Longbiao Wang,Jianwu Dang机构：College of Intelligence and Computing, Tianjin University, Tianjin, China, Kuaishou Technology Co., Ltd, Beijing, China, Institute of Automation, Chinese Academy of Sciences, Beijing, China摘要：最近，人们对文本到语音（TTS）方法越来越感兴趣，该方法可以通过组合两种类型的离散语音表示并使用两个序列到序列任务来解耦TTS来以最小的监督进行训练。为了解决与离散表示中的高维数和波形失真相关的挑战，我们提出了Diff-LM-Speech，它基于扩散模型将语义嵌入到Mel频谱图中，并引入了基于变分自动编码器和韵律瓶颈的提示编码器结构，以提高提示表示能力。自回归语言模型经常遭受丢失和重复的单词，而非自回归框架由于持续时间预测模型而面临表达平均化问题。为了解决这些问题，我们提出了Tetra-Diff-Speech，它设计了一个持续时间扩散模型来实现不同的韵律表达。虽然我们期望语义编码的信息内容介于文本和声学编码之间，但现有模型提取的语义编码具有大量的冗余信息和维度爆炸。为了验证语义编码是不必要的，我们提出了三差语音。实验结果表明，我们提出的方法优于基线方法。我们提供了一个网站的音频样本。摘要：Recently, there has been a growing interest in text-to-speech (TTS) methods that can be trained with minimal supervision by combining two types of discrete speech representations and using two sequence-to-sequence tasks to decouple TTS. To address the challenges associated with high dimensionality and waveform distortion in discrete representations, we propose Diff-LM-Speech, which models semantic embeddings into mel-spectrogram based on diffusion models and introduces a prompt encoder structure based on variational autoencoders and prosody bottlenecks to improve prompt representation capabilities. Autoregressive language models often suffer from missing and repeated words, while non-autoregressive frameworks face expression averaging problems due to duration prediction models. To address these issues, we propose Tetra-Diff-Speech, which designs a duration diffusion model to achieve diverse prosodic expressions. While we expect the information content of semantic coding to be between that of text and acoustic coding, existing models extract semantic coding with a lot of redundant information and dimensionality explosion. To verify that semantic coding is not necessary, we propose Tri-Diff-Speech. Experimental results show that our proposed methods outperform baseline methods. We provide a website with audio samples.
【8】 The FlySpeech Audio-Visual Speaker Diarization System for MISP Challenge  2022标题：FlySpeech视听说话人对讲机2022挑战赛链接：https://arxiv.org/abs/2307.15400
作者：Li Zhang,Huan Zhao,Yue Li,Bowen Pang,Yannan Wang,Hongji Wang,Wei Rao,Qing Wang,Lei Xie机构：Northwestern Polytechnical University (NPU), Xi’an, China, Tencent Corporation, Shenzhen, China摘要：本文介绍了在ICASSP 2022举办的第二届基于\textbf{M}多模态\textbf{I}信息的\textbf{S}语音\textbf{P}处理~（\textbf{MISP}）挑战赛中提交的FlySpeech说话人日志化系统。我们开发了一个端到端的音视频说话人日记化系统，它包括一个唇编码器，一个说话人编码器，和一个音视频解码器。具体来说，为了减轻由单独训练引起的日记化性能的下降，我们联合训练扬声器编码器和视听解码器。此外，我们利用大数据预训练的扬声器提取器来初始化扬声器编码器。摘要：This paper describes the FlySpeech speaker diarization system submitted to the second \textbf{M}ultimodal \textbf{I}nformation Based \textbf{S}peech \textbf{P}rocessing~(\textbf{MISP}) Challenge held in ICASSP 2022. We develop an end-to-end audio-visual speaker diarization~(AVSD) system, which consists of a lip encoder, a speaker encoder, and an audio-visual decoder. Specifically, to mitigate the degradation of diarization performance caused by separate training, we jointly train the speaker encoder and the audio-visual decoder. In addition, we leverage the large-data pretrained speaker extractor to initialize the speaker encoder.
【9】 Improving Audio-Text Retrieval via Hierarchical Cross-Modal Interaction  and Auxiliary Captions标题：通过层次化跨通道交互和辅助字幕改进音文检索链接：https://arxiv.org/abs/2307.15344
作者：Yifei Xin,Yuexian Zou机构：School of ECE, Peking University, Shenzhen, China备注：Accepted by Interspeech2023摘要：大多数现有的音频文本检索（ATR）方法专注于在整个音频片段和完整的字幕句子之间构建对比对，而忽略了细粒度的跨模态关系，例如，短片段和短语或框架和单词。在本文中，我们介绍了一个层次的跨模态交互（HCI）的ATR方法，同时探索剪辑句子，段短语，框架词的关系，实现了全面的多模态语义比较。此外，我们还提出了一种新的ATR框架，利用辅助字幕（AC）生成的预训练字幕执行音频和生成的字幕之间的功能交互，这产生了增强的音频表示，并补充到原来的ATR匹配分支。音频和生成的字幕还可以形成新的音频-文本对作为用于训练的数据增强。实验表明，我们的HCI显着提高ATR性能。此外，我们的AC框架在多个数据集上也显示出稳定的性能增益。摘要：Most existing audio-text retrieval (ATR) methods focus on constructing contrastive pairs between whole audio clips and complete caption sentences, while ignoring fine-grained cross-modal relationships, e.g., short segments and phrases or frames and words. In this paper, we introduce a hierarchical cross-modal interaction (HCI) method for ATR by simultaneously exploring clip-sentence, segment-phrase, and frame-word relationships, achieving a comprehensive multi-modal semantic comparison. Besides, we also present a novel ATR framework that leverages auxiliary captions (AC) generated by a pretrained captioner to perform feature interaction between audio and generated captions, which yields enhanced audio representations and is complementary to the original ATR matching branch. The audio and generated captions can also form new audio-text pairs as data augmentation for training. Experiments show that our HCI significantly improves the ATR performance. Moreover, our AC framework also shows stable performance gains on multiple datasets.
【10】 Cascaded Cross-Modal Transformer for Request and Complaint Detection标题：用于请求和投诉检测的级联跨模Transformer链接：https://arxiv.org/abs/2307.15097
作者：Nicolae-Catalin Ristea,Radu Tudor Ionescu机构：University of Bucharest, Romania备注：Accepted at ACMMM 2023摘要：我们提出了一种新的级联跨模态Transformer（CCMT），结合语音和文本转录检测客户的要求和投诉，在电话交谈。我们的方法利用多模态范式转录的语音使用自动语音识别（ASR）模型和翻译成不同的语言的成绩单。随后，我们结合语言特定的BERT为基础的模型与Wav2Vec2.0音频功能在一个新的级联交叉注意Transformer模型。我们将我们的系统应用于ACM多媒体2023计算副语言挑战赛的请求子挑战赛，投诉和请求类的未加权平均召回率（UAR）分别为65.41%和85.87%。摘要：We propose a novel cascaded cross-modal transformer (CCMT) that combines speech and text transcripts to detect customer requests and complaints in phone conversations. Our approach leverages a multimodal paradigm by transcribing the speech using automatic speech recognition (ASR) models and translating the transcripts into different languages. Subsequently, we combine language-specific BERT-based models with Wav2Vec2.0 audio features in a novel cascaded cross-attention transformer model. We apply our system to the Requests Sub-Challenge of the ACM Multimedia 2023 Computational Paralinguistics Challenge, reaching unweighted average recalls (UAR) of 65.41% and 85.87% for the complaint and request classes, respectively.
机器翻译，仅供参考