今日论文合集：cs.SD语音6篇，eess.AS音频处理6篇。
cs.SD语音
【1】 CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency  Model标题：CoMoSpeech：基于一致性模型的一步语音和歌唱语音合成链接：https://arxiv.org/abs/2305.06908
作者：Zhen Ye,Wei Xue,Xu Tan,Jie Chen,Qifeng Liu,Yike Guo机构：Hong Kong Baptist University , Hong Kong University of Science and Technology,  Microsoft Research Asia,  Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences ∗摘要：去噪扩散概率模型（DDPMs）在语音合成中表现出良好的性能。然而，需要大量的迭代步骤来实现高的样本质量，这限制了推理速度。在提高采样速度的同时保持采样质量已成为一项具有挑战性的任务。在本文中，我们提出了一个“共”一致性“Mo”del为基础的“语音”合成方法，CoMoSpeech，通过一个单一的扩散采样步骤，同时实现高音质的语音合成。一致性约束被应用于从设计良好的基于扩散的教师模型提取一致性模型，其最终在提取的CoMoSpeech中产生优异的性能。我们的实验表明，通过单个采样步骤生成音频记录，CoMoSpeech的推理速度比单个NVIDIA A100 GPU上的实时速度快150倍以上，与FastSpeech2相当，使基于扩散采样的语音合成真正实用。同时，客观和主观评价的文本到语音和歌声合成表明，所提出的教师模型产生最好的音频质量，和一步采样的CoMoSpeech实现了最好的推理速度，更好的或可比的音频质量，其他传统的多步扩散模型基线。音频样本可在www.example.com上获得https://comospeech.github.io/。摘要：Denoising diffusion probabilistic models (DDPMs) have shown promising performance for speech synthesis. However, a large number of iterative steps are required to achieve high sample quality, which restricts the inference speed. Maintaining sample quality while increasing sampling speed has become a challenging task. In this paper, we propose a "Co"nsistency "Mo"del-based "Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a single diffusion sampling step while achieving high audio quality. The consistency constraint is applied to distill a consistency model from a well-designed diffusion-based teacher model, which ultimately yields superior performances in the distilled CoMoSpeech. Our experiments show that by generating audio recordings by a single sampling step, the CoMoSpeech achieves an inference speed more than 150 times faster than real-time on a single NVIDIA A100 GPU, which is comparable to FastSpeech2, making diffusion-sampling based speech synthesis truly practical. Meanwhile, objective and subjective evaluations on text-to-speech and singing voice synthesis show that the proposed teacher models yield the best audio quality, and the one-step sampling based CoMoSpeech achieves the best inference speed with better or comparable audio quality to other conventional multi-step diffusion model baselines. Audio samples are available at https://comospeech.github.io/.
【2】 HappyQuokka System for ICASSP 2023 Auditory EEG Challenge标题：ICASSP 2023听觉脑电挑战赛HappyQuokka系统链接：https://arxiv.org/abs/2305.06806
作者：Zhenyu Piao,Miseul Kim,Hyungchan Yoon,Hong-Goo Kang机构：Department of Electrical and Electronic Engineering, Yonsei University, Seoul, South Korea备注：First Place in Task 2 of Auditory EEG decoding Challenge, which is part of ICASSP Signal Processing Grand Challenge (SPGC) 2023摘要：本报告描述了我们在ICASSP 2023信号处理大挑战赛（SPGC）上提交的听觉EEG解码挑战赛任务2。任务2是回归问题，其集中于从EEG信号重建语音包络。对于该任务，我们提出了一个前层归一化前馈Transformer（FFT）架构。对于受试者内生成，我们还利用了一个辅助的全局调节器，它为我们的模型提供了关于所见个体的额外信息。实验结果表明，我们提出的方法优于VLAAI基线和所有其他提交的系统。值得注意的是，它在受试者内任务上表现出显着的改进，这可能要归功于我们使用了辅助全局调节器。在评估指标设置的挑战，我们获得皮尔逊相关系数值为0.1895 - 0.0869的受试者内代测试和0.0976 - 0.0444的举行了主题测试。我们在线发布模型的训练代码。摘要：This report describes our submission to Task 2 of the Auditory EEG Decoding Challenge at ICASSP 2023 Signal Processing Grand Challenge (SPGC). Task 2 is a regression problem that focuses on reconstructing a speech envelope from an EEG signal. For the task, we propose a pre-layer normalized feed-forward transformer (FFT) architecture. For within-subjects generation, we additionally utilize an auxiliary global conditioner which provides our model with additional information about seen individuals. Experimental results show that our proposed method outperforms the VLAAI baseline and all other submitted systems. Notably, it demonstrates significant improvements on the within-subjects task, likely thanks to our use of the auxiliary global conditioner. In terms of evaluation metrics set by the challenge, we obtain Pearson correlation values of 0.1895 0.0869 for the within-subjects generation test and 0.0976 0.0444 for the heldout-subjects test. We release the training code for our model online.
【3】 Extending Audio Masked Autoencoders Toward Audio Restoration标题：将音频屏蔽自动编码器扩展到音频恢复链接：https://arxiv.org/abs/2305.06701
作者：Zhi Zhong,Hao Shi,Masato Hirano,Kazuki Shimada,Kazuya Tateishi,Takashi Shibuya,Shusuke Takahashi,Yuki Mitsufuji机构：Sony Group Corporation, Japan,  Kyoto University, Japan,  Sony Research, Japan备注：Under review摘要：音频分类和恢复是音频信号处理中的主要下游任务之一。然而，与预训练模型在分类任务中的压倒性成功相比，恢复从预训练模型中获得的好处较少。由于这种不平衡的好处，人们越来越关注如何提高预训练模型的性能，以用于语音增强（SE）等恢复任务。先前的工作已经表明，由预训练的音频编码器提取的特征对于SE任务是有效的，但是这些语音特定的仅编码器模型通常需要额外的解码器才能与SE任务兼容，并且涉及复杂的预训练过程或复杂的数据增强。因此，在追求一个通用的音频模型，音频掩蔽自动编码器（MAE）的骨干是Vision Transformers（ViT-AE）的自动编码器，本文从音频分类扩展到恢复任务。ViT-AE自然地学习与预训练期间的恢复任务兼容的mel到mel映射。在许多恢复任务中，选择SE是由于其完善的评估指标和测试数据。我们提出了ViT-AE的变化，以提高SE性能，其中梅尔到梅尔的变化产生高分数的非侵入性指标和STFT导向的变化是有效的标准侵入性指标，如PESQ。可以根据场景使用不同的变型。综合评估和消融研究表明，MAE预训练也有利于SE任务，并帮助ViT-AE更好地推广到域外失真。我们进一步发现，一般音频源的大规模噪声数据，而不是干净的语音，对于预训练足够有效。摘要：Audio classification and restoration are among major downstream tasks in audio signal processing. However, restoration derives less of a benefit from pretrained models compared to the overwhelming success of pretrained models in classification tasks. Due to such unbalanced benefits, there has been rising interest in how to improve the performance of pretrained models for restoration tasks such as speech enhancement (SE). Previous works have shown that the features extracted by pretrained audio encoders are effective for SE tasks, but these speech-specific encoder-only models usually require extra decoders to become compatible with SE tasks, and involve complicated pretraining procedures or complex data augmentation. Therefore, in pursuit of a universal audio model, the audio masked autoencoder (MAE) whose backbone is the autoencoder of Vision Transformers (ViT-AE), is extended from audio classification toward restoration tasks in this paper. ViT-AE naturally learns mel-to-mel mapping that is compatible with restoration tasks during pretraining. Among many restoration tasks, SE is chosen due to its well-established evaluation metrics and test data. We propose variations of ViT-AE to improve the SE performance, where the mel-to-mel variations yield high scores for non-intrusive metrics and the STFT-oriented variation is effective at standard intrusive metrics such as PESQ. Different variations can be used in accordance with the scenarios. Comprehensive evaluations and ablation studies show that MAE pretraining is also beneficial to SE tasks and help the ViT-AE to better generalize to out-of-domain distortions. We further found that large-scale noisy data of general audio sources, rather than clean speech, is sufficiently effective for pretraining.
【4】 V2Meow: Meowing to the Visual Beat via Music Generation标题：V2喵：通过音乐生成随着视觉节拍喵叫链接：https://arxiv.org/abs/2305.06594
作者：Kun Su,Judith Yue Li,Qingqing Huang,Dima Kuzmin,Joonseok Lee,Chris Donahue,Fei Sha,Aren Jansen,Yu Wang,Mauro Verzetti,Timo I. Denk机构：Timo Denk‡摘要：生成补充视频的视觉内容的高质量音乐是一项具有挑战性的任务。大多数现有的视觉调节音乐生成系统生成符号音乐数据，例如MIDI文件，而不是原始音频波形。鉴于符号音乐数据的有限可用性，这样的方法只能为少数乐器或特定类型的视觉输入生成音乐。在本文中，我们提出了一种称为V2Meow的新方法，可以生成高质量的音乐音频，与各种视频输入类型的视觉语义保持一致。具体而言，所提出的音乐生成系统是一个多阶段的自回归模型，它是训练与视频帧配对的O（100K）的音乐音频剪辑，这是从野生音乐视频挖掘，没有并行的符号音乐数据。V2Meow能够仅根据从任意无声视频剪辑中提取的预训练视觉特征来合成高保真音乐音频波形，并且除了视频帧调节之外，还允许通过支持文本提示来对生成示例的音乐风格进行高级控制。通过定性和定量的评估，我们表明，我们的模型优于几个现有的音乐生成系统的视听对应和音频质量。摘要：Generating high quality music that complements the visual content of a video is a challenging task. Most existing visual conditioned music generation systems generate symbolic music data, such as MIDI files, instead of raw audio waveform. Given the limited availability of symbolic music data, such methods can only generate music for a few instruments or for specific types of visual input. In this paper, we propose a novel approach called V2Meow that can generate high-quality music audio that aligns well with the visual semantics of a diverse range of video input types. Specifically, the proposed music generation system is a multi-stage autoregressive model which is trained with a number of O(100K) music audio clips paired with video frames, which are mined from in-the-wild music videos, and no parallel symbolic music data is involved. V2Meow is able to synthesize high-fidelity music audio waveform solely conditioned on pre-trained visual features extracted from an arbitrary silent video clip, and it also allows high-level control over the music style of generation examples via supporting text prompts in addition to the video frames conditioning. Through both qualitative and quantitative evaluations, we demonstrate that our model outperforms several existing music generation systems in terms of both visual-audio correspondence and audio quality.
【5】 Mispronunciation Detection of Basic Quranic Recitation Rules using Deep  Learning标题：基于深度学习的《古兰经》基本背诵规则发音错误检测链接：https://arxiv.org/abs/2305.06429
作者：Ahmad Al Harere,Khloud Al Jallad机构：Department of Information and Communication Engineering, Arab International University, Daraa, Syria.摘要：在伊斯兰教中，读者必须应用一套称为Tajweed规则的发音规则来背诵古兰经，就像天使Jibrael教先知穆罕默德一样。学习正确应用这些规则的传统过程需要一个人，他必须有执照和丰富的经验来检测发音错误。由于世界各地穆斯林人数的增加，Tajweed教师的数量不足以满足每个穆斯林每天的背诵练习。因此，许多学者在Tajweed规则的误读检测方面做了大量的工作，以帮助读者以比传统学习方式更简单、更短的时间正确地背诵《古兰经》。所有以前的作品都有三个共同的问题。首先，他们中的大多数只关注机器学习算法。其次，他们使用的是没有基准可供比较的私人数据集。第三，虽然语音信号是时间序列的，但它们没有最佳地考虑输入数据的序列。为了克服这些问题，我们提出了一种解决方案，该解决方案由梅尔频率倒谱系数（MFCC）特征和长短期记忆（LSTM）神经网络组成，该神经网络使用时间序列来检测Tajweed规则中的错误发音。此外，我们的实验是在一个公共数据集，QDAT数据集，其中包含超过1500个声音的正确和不正确的背诵三Tajweed规则（单独拉伸，紧中午，和隐藏）。据我们所知，QDAT数据集尚未被任何研究论文使用。我们将所提出的LSTM模型的性能与SoTA中使用的传统机器学习算法进行了比较。具有时间序列的LSTM模型显示出明显优于传统机器学习的优势。LSTM在QDAT数据集上实现的准确率分别为96%、95%和96%，这三个规则（分开拉伸、紧中午和隐藏）。摘要：In Islam, readers must apply a set of pronunciation rules called Tajweed rules to recite the Quran in the same way that the angel Jibrael taught the Prophet, Muhammad. The traditional process of learning the correct application of these rules requires a human who must have a license and great experience to detect mispronunciation. Due to the increasing number of Muslims around the world, the number of Tajweed teachers is not enough nowadays for daily recitation practice for every Muslim. Therefore, lots of work has been done for automatic Tajweed rules' mispronunciation detection to help readers recite Quran correctly in an easier way and shorter time than traditional learning ways. All previous works have three common problems. First, most of them focused on machine learning algorithms only. Second, they used private datasets with no benchmark to compare with. Third, they did not take into consideration the sequence of input data optimally, although the speech signal is time series. To overcome these problems, we proposed a solution that consists of Mel-Frequency Cepstral Coefficient (MFCC) features with Long Short-Term Memory (LSTM) neural networks which use the time series, to detect mispronunciation in Tajweed rules. In addition, our experiments were performed on a public dataset, the QDAT dataset, which contains more than 1500 voices of the correct and incorrect recitation of three Tajweed rules (Separate stretching , Tight Noon , and Hide ). To the best of our knowledge, the QDAT dataset has not been used by any research paper yet. We compared the performance of the proposed LSTM model with traditional machine learning algorithms used in SoTA. The LSTM model with time series showed clear superiority over traditional machine learning. The accuracy achieved by LSTM on the QDAT dataset was 96%, 95%, and 96% for the three rules (Separate stretching, Tight Noon, and Hide), respectively.
【6】 Speaker Diaphragm Excursion Prediction: deep attention and online  adaptation标题：扬声器振膜漂移预测：深度关注和在线适应链接：https://arxiv.org/abs/2305.06640
作者：Yuwei Ren,Matt Zivney,Yin Huang,Eddie Choy,Chirag Patel,Hao Xu机构：Qualcomm AI Research备注：5 pages, 4 figures, ICASSP 2023摘要：扬声器保护算法是利用回放信号特性来防止过偏移，同时保持最大响度，特别是对于具有微小扬声器的移动电话。本文提出了高效的DL解决方案，以准确地建模和预测的非线性偏移，这是传统的解决方案的挑战。首先，我们建立了实验和预处理流水线，其中的反馈电流和电压采样作为输入，并使用激光测量偏移作为地面真值。其次，提出了一种FFTNet模型来探索占主导地位的低频和其他未知谐波，并与基线ConvNet模型进行比较。此外，设计了BN重估计，探索了在线自适应;和基于AI模型效率工具包的INT 8量化（AIMET\footnote{AIMET是高通创新中心公司的产品}）以进一步降低复杂度。在两个扬声器和3个典型部署场景中验证了所提出的算法，$>$99\%的残余DC小于0.1mm，远优于传统的解决方案。摘要：Speaker protection algorithm is to leverage the playback signal properties to prevent over excursion while maintaining maximum loudness, especially for the mobile phone with tiny loudspeakers. This paper proposes efficient DL solutions to accurately model and predict the nonlinear excursion, which is challenging for conventional solutions. Firstly, we build the experiment and pre-processing pipeline, where the feedback current and voltage are sampled as input, and laser is employed to measure the excursion as ground truth. Secondly, one FFTNet model is proposed to explore the dominant low-frequency and other unknown harmonics, and compares to a baseline ConvNet model. In addition, BN re-estimation is designed to explore the online adaptation; and INT8 quantization based on AI Model efficiency toolkit (AIMET\footnote{AIMET is a product of Qualcomm Innovation Center, Inc.}) is applied to further reduce the complexity. The proposed algorithm is verified in two speakers and 3 typical deployment scenarios, and $>$99\% residual DC is less than 0.1 mm, much better than traditional solutions.
eess.AS音频处理
【1】 Speaker Diaphragm Excursion Prediction: deep attention and online  adaptation标题：扬声器振膜漂移预测：深度关注和在线适应链接：https://arxiv.org/abs/2305.06640
*与cs.SD语音【6】为同一篇
作者：Yuwei Ren,Matt Zivney,Yin Huang,Eddie Choy,Chirag Patel,Hao Xu机构：Qualcomm AI Research备注：5 pages, 4 figures, ICASSP 2023摘要：扬声器保护算法是利用回放信号特性来防止过偏移，同时保持最大响度，特别是对于具有微小扬声器的移动电话。本文提出了高效的DL解决方案，以准确地建模和预测的非线性偏移，这是传统的解决方案的挑战。首先，我们建立了实验和预处理流水线，其中的反馈电流和电压采样作为输入，并使用激光测量偏移作为地面真值。其次，提出了一种FFTNet模型来探索占主导地位的低频和其他未知谐波，并与基线ConvNet模型进行比较。此外，设计了BN重估计，探索了在线自适应;和基于AI模型效率工具包的INT 8量化（AIMET\footnote{AIMET是高通创新中心公司的产品}）以进一步降低复杂度。在两个扬声器和3个典型部署场景中验证了所提出的算法，$>$99\%的残余DC小于0.1mm，远优于传统的解决方案。摘要：Speaker protection algorithm is to leverage the playback signal properties to prevent over excursion while maintaining maximum loudness, especially for the mobile phone with tiny loudspeakers. This paper proposes efficient DL solutions to accurately model and predict the nonlinear excursion, which is challenging for conventional solutions. Firstly, we build the experiment and pre-processing pipeline, where the feedback current and voltage are sampled as input, and laser is employed to measure the excursion as ground truth. Secondly, one FFTNet model is proposed to explore the dominant low-frequency and other unknown harmonics, and compares to a baseline ConvNet model. In addition, BN re-estimation is designed to explore the online adaptation; and INT8 quantization based on AI Model efficiency toolkit (AIMET\footnote{AIMET is a product of Qualcomm Innovation Center, Inc.}) is applied to further reduce the complexity. The proposed algorithm is verified in two speakers and 3 typical deployment scenarios, and $>$99\% residual DC is less than 0.1 mm, much better than traditional solutions.
【2】 CoMoSpeech: One-Step Speech and Singing Voice Synthesis via Consistency  Model标题：CoMoSpeech：基于一致性模型的一步语音和歌唱语音合成链接：https://arxiv.org/abs/2305.06908
*与cs.SD语音【1】为同一篇
作者：Zhen Ye,Wei Xue,Xu Tan,Jie Chen,Qifeng Liu,Yike Guo机构：Hong Kong Baptist University , Hong Kong University of Science and Technology,  Microsoft Research Asia,  Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences ∗摘要：去噪扩散概率模型（DDPMs）在语音合成中表现出良好的性能。然而，需要大量的迭代步骤来实现高的样本质量，这限制了推理速度。在提高采样速度的同时保持采样质量已成为一项具有挑战性的任务。在本文中，我们提出了一个“共”一致性“Mo”del为基础的“语音”合成方法，CoMoSpeech，通过一个单一的扩散采样步骤，同时实现高音质的语音合成。一致性约束被应用于从设计良好的基于扩散的教师模型提取一致性模型，其最终在提取的CoMoSpeech中产生优异的性能。我们的实验表明，通过单个采样步骤生成音频记录，CoMoSpeech的推理速度比单个NVIDIA A100 GPU上的实时速度快150倍以上，与FastSpeech2相当，使基于扩散采样的语音合成真正实用。同时，客观和主观评价的文本到语音和歌声合成表明，所提出的教师模型产生最好的音频质量，和一步采样的CoMoSpeech实现了最好的推理速度，更好的或可比的音频质量，其他传统的多步扩散模型基线。音频样本可在www.example.com上获得https://comospeech.github.io/。摘要：Denoising diffusion probabilistic models (DDPMs) have shown promising performance for speech synthesis. However, a large number of iterative steps are required to achieve high sample quality, which restricts the inference speed. Maintaining sample quality while increasing sampling speed has become a challenging task. In this paper, we propose a "Co"nsistency "Mo"del-based "Speech" synthesis method, CoMoSpeech, which achieve speech synthesis through a single diffusion sampling step while achieving high audio quality. The consistency constraint is applied to distill a consistency model from a well-designed diffusion-based teacher model, which ultimately yields superior performances in the distilled CoMoSpeech. Our experiments show that by generating audio recordings by a single sampling step, the CoMoSpeech achieves an inference speed more than 150 times faster than real-time on a single NVIDIA A100 GPU, which is comparable to FastSpeech2, making diffusion-sampling based speech synthesis truly practical. Meanwhile, objective and subjective evaluations on text-to-speech and singing voice synthesis show that the proposed teacher models yield the best audio quality, and the one-step sampling based CoMoSpeech achieves the best inference speed with better or comparable audio quality to other conventional multi-step diffusion model baselines. Audio samples are available at https://comospeech.github.io/.
【3】 HappyQuokka System for ICASSP 2023 Auditory EEG Challenge标题：ICASSP 2023听觉脑电挑战赛HappyQuokka系统链接：https://arxiv.org/abs/2305.06806
*与cs.SD语音【2】为同一篇
作者：Zhenyu Piao,Miseul Kim,Hyungchan Yoon,Hong-Goo Kang机构：Department of Electrical and Electronic Engineering, Yonsei University, Seoul, South Korea备注：First Place in Task 2 of Auditory EEG decoding Challenge, which is part of ICASSP Signal Processing Grand Challenge (SPGC) 2023摘要：本报告描述了我们在ICASSP 2023信号处理大挑战赛（SPGC）上提交的听觉EEG解码挑战赛任务2。任务2是回归问题，其集中于从EEG信号重建语音包络。对于该任务，我们提出了一个前层归一化前馈Transformer（FFT）架构。对于受试者内生成，我们还利用了一个辅助的全局调节器，它为我们的模型提供了关于所见个体的额外信息。实验结果表明，我们提出的方法优于VLAAI基线和所有其他提交的系统。值得注意的是，它在受试者内任务上表现出显着的改进，这可能要归功于我们使用了辅助全局调节器。在评估指标设置的挑战，我们获得皮尔逊相关系数值为0.1895 - 0.0869的受试者内代测试和0.0976 - 0.0444的举行了主题测试。我们在线发布模型的训练代码。摘要：This report describes our submission to Task 2 of the Auditory EEG Decoding Challenge at ICASSP 2023 Signal Processing Grand Challenge (SPGC). Task 2 is a regression problem that focuses on reconstructing a speech envelope from an EEG signal. For the task, we propose a pre-layer normalized feed-forward transformer (FFT) architecture. For within-subjects generation, we additionally utilize an auxiliary global conditioner which provides our model with additional information about seen individuals. Experimental results show that our proposed method outperforms the VLAAI baseline and all other submitted systems. Notably, it demonstrates significant improvements on the within-subjects task, likely thanks to our use of the auxiliary global conditioner. In terms of evaluation metrics set by the challenge, we obtain Pearson correlation values of 0.1895 0.0869 for the within-subjects generation test and 0.0976 0.0444 for the heldout-subjects test. We release the training code for our model online.
【4】 Extending Audio Masked Autoencoders Toward Audio Restoration标题：将音频屏蔽自动编码器扩展到音频恢复链接：https://arxiv.org/abs/2305.06701
*与cs.SD语音【3】为同一篇
作者：Zhi Zhong,Hao Shi,Masato Hirano,Kazuki Shimada,Kazuya Tateishi,Takashi Shibuya,Shusuke Takahashi,Yuki Mitsufuji机构：Sony Group Corporation, Japan,  Kyoto University, Japan,  Sony Research, Japan备注：Under review摘要：音频分类和恢复是音频信号处理中的主要下游任务之一。然而，与预训练模型在分类任务中的压倒性成功相比，恢复从预训练模型中获得的好处较少。由于这种不平衡的好处，人们越来越关注如何提高预训练模型的性能，以用于语音增强（SE）等恢复任务。先前的工作已经表明，由预训练的音频编码器提取的特征对于SE任务是有效的，但是这些语音特定的仅编码器模型通常需要额外的解码器才能与SE任务兼容，并且涉及复杂的预训练过程或复杂的数据增强。因此，在追求一个通用的音频模型，音频掩蔽自动编码器（MAE）的骨干是Vision Transformers（ViT-AE）的自动编码器，本文从音频分类扩展到恢复任务。ViT-AE自然地学习与预训练期间的恢复任务兼容的mel到mel映射。在许多恢复任务中，选择SE是由于其完善的评估指标和测试数据。我们提出了ViT-AE的变化，以提高SE性能，其中梅尔到梅尔的变化产生高分数的非侵入性指标和STFT导向的变化是有效的标准侵入性指标，如PESQ。可以根据场景使用不同的变型。综合评估和消融研究表明，MAE预训练也有利于SE任务，并帮助ViT-AE更好地推广到域外失真。我们进一步发现，一般音频源的大规模噪声数据，而不是干净的语音，对于预训练足够有效。摘要：Audio classification and restoration are among major downstream tasks in audio signal processing. However, restoration derives less of a benefit from pretrained models compared to the overwhelming success of pretrained models in classification tasks. Due to such unbalanced benefits, there has been rising interest in how to improve the performance of pretrained models for restoration tasks such as speech enhancement (SE). Previous works have shown that the features extracted by pretrained audio encoders are effective for SE tasks, but these speech-specific encoder-only models usually require extra decoders to become compatible with SE tasks, and involve complicated pretraining procedures or complex data augmentation. Therefore, in pursuit of a universal audio model, the audio masked autoencoder (MAE) whose backbone is the autoencoder of Vision Transformers (ViT-AE), is extended from audio classification toward restoration tasks in this paper. ViT-AE naturally learns mel-to-mel mapping that is compatible with restoration tasks during pretraining. Among many restoration tasks, SE is chosen due to its well-established evaluation metrics and test data. We propose variations of ViT-AE to improve the SE performance, where the mel-to-mel variations yield high scores for non-intrusive metrics and the STFT-oriented variation is effective at standard intrusive metrics such as PESQ. Different variations can be used in accordance with the scenarios. Comprehensive evaluations and ablation studies show that MAE pretraining is also beneficial to SE tasks and help the ViT-AE to better generalize to out-of-domain distortions. We further found that large-scale noisy data of general audio sources, rather than clean speech, is sufficiently effective for pretraining.
【5】 V2Meow: Meowing to the Visual Beat via Music Generation标题：V2喵：通过音乐生成随着视觉节拍喵叫链接：https://arxiv.org/abs/2305.06594
*与cs.SD语音【4】为同一篇
作者：Kun Su,Judith Yue Li,Qingqing Huang,Dima Kuzmin,Joonseok Lee,Chris Donahue,Fei Sha,Aren Jansen,Yu Wang,Mauro Verzetti,Timo I. Denk机构：Timo Denk‡摘要：生成补充视频的视觉内容的高质量音乐是一项具有挑战性的任务。大多数现有的视觉调节音乐生成系统生成符号音乐数据，例如MIDI文件，而不是原始音频波形。鉴于符号音乐数据的有限可用性，这样的方法只能为少数乐器或特定类型的视觉输入生成音乐。在本文中，我们提出了一种称为V2Meow的新方法，可以生成高质量的音乐音频，与各种视频输入类型的视觉语义保持一致。具体而言，所提出的音乐生成系统是一个多阶段的自回归模型，它是训练与视频帧配对的O（100K）的音乐音频剪辑，这是从野生音乐视频挖掘，没有并行的符号音乐数据。V2Meow能够仅根据从任意无声视频剪辑中提取的预训练视觉特征来合成高保真音乐音频波形，并且除了视频帧调节之外，还允许通过支持文本提示来对生成示例的音乐风格进行高级控制。通过定性和定量的评估，我们表明，我们的模型优于几个现有的音乐生成系统的视听对应和音频质量。摘要：Generating high quality music that complements the visual content of a video is a challenging task. Most existing visual conditioned music generation systems generate symbolic music data, such as MIDI files, instead of raw audio waveform. Given the limited availability of symbolic music data, such methods can only generate music for a few instruments or for specific types of visual input. In this paper, we propose a novel approach called V2Meow that can generate high-quality music audio that aligns well with the visual semantics of a diverse range of video input types. Specifically, the proposed music generation system is a multi-stage autoregressive model which is trained with a number of O(100K) music audio clips paired with video frames, which are mined from in-the-wild music videos, and no parallel symbolic music data is involved. V2Meow is able to synthesize high-fidelity music audio waveform solely conditioned on pre-trained visual features extracted from an arbitrary silent video clip, and it also allows high-level control over the music style of generation examples via supporting text prompts in addition to the video frames conditioning. Through both qualitative and quantitative evaluations, we demonstrate that our model outperforms several existing music generation systems in terms of both visual-audio correspondence and audio quality.
【6】 Mispronunciation Detection of Basic Quranic Recitation Rules using Deep  Learning标题：基于深度学习的《古兰经》基本背诵规则发音错误检测链接：https://arxiv.org/abs/2305.06429
*与cs.SD语音【5】为同一篇
作者：Ahmad Al Harere,Khloud Al Jallad机构：Department of Information and Communication Engineering, Arab International University, Daraa, Syria.摘要：在伊斯兰教中，读者必须应用一套称为Tajweed规则的发音规则来背诵古兰经，就像天使Jibrael教先知穆罕默德一样。学习正确应用这些规则的传统过程需要一个人，他必须有执照和丰富的经验来检测发音错误。由于世界各地穆斯林人数的增加，Tajweed教师的数量不足以满足每个穆斯林每天的背诵练习。因此，许多学者在Tajweed规则的误读检测方面做了大量的工作，以帮助读者以比传统学习方式更简单、更短的时间正确地背诵《古兰经》。所有以前的作品都有三个共同的问题。首先，他们中的大多数只关注机器学习算法。其次，他们使用的是没有基准可供比较的私人数据集。第三，虽然语音信号是时间序列的，但它们没有最佳地考虑输入数据的序列。为了克服这些问题，我们提出了一种解决方案，该解决方案由梅尔频率倒谱系数（MFCC）特征和长短期记忆（LSTM）神经网络组成，该神经网络使用时间序列来检测Tajweed规则中的错误发音。此外，我们的实验是在一个公共数据集，QDAT数据集，其中包含超过1500个声音的正确和不正确的背诵三Tajweed规则（单独拉伸，紧中午，和隐藏）。据我们所知，QDAT数据集尚未被任何研究论文使用。我们将所提出的LSTM模型的性能与SoTA中使用的传统机器学习算法进行了比较。具有时间序列的LSTM模型显示出明显优于传统机器学习的优势。LSTM在QDAT数据集上实现的准确率分别为96%、95%和96%，这三个规则（分开拉伸、紧中午和隐藏）。摘要：In Islam, readers must apply a set of pronunciation rules called Tajweed rules to recite the Quran in the same way that the angel Jibrael taught the Prophet, Muhammad. The traditional process of learning the correct application of these rules requires a human who must have a license and great experience to detect mispronunciation. Due to the increasing number of Muslims around the world, the number of Tajweed teachers is not enough nowadays for daily recitation practice for every Muslim. Therefore, lots of work has been done for automatic Tajweed rules' mispronunciation detection to help readers recite Quran correctly in an easier way and shorter time than traditional learning ways. All previous works have three common problems. First, most of them focused on machine learning algorithms only. Second, they used private datasets with no benchmark to compare with. Third, they did not take into consideration the sequence of input data optimally, although the speech signal is time series. To overcome these problems, we proposed a solution that consists of Mel-Frequency Cepstral Coefficient (MFCC) features with Long Short-Term Memory (LSTM) neural networks which use the time series, to detect mispronunciation in Tajweed rules. In addition, our experiments were performed on a public dataset, the QDAT dataset, which contains more than 1500 voices of the correct and incorrect recitation of three Tajweed rules (Separate stretching , Tight Noon , and Hide ). To the best of our knowledge, the QDAT dataset has not been used by any research paper yet. We compared the performance of the proposed LSTM model with traditional machine learning algorithms used in SoTA. The LSTM model with time series showed clear superiority over traditional machine learning. The accuracy achieved by LSTM on the QDAT dataset was 96%, 95%, and 96% for the three rules (Separate stretching, Tight Noon, and Hide), respectively.
机器翻译，仅供参考