今日论文合集：cs.SD语音15篇，eess.AS音频处理15篇。
cs.SD语音
【1】 Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method  for Speech Emotion Recognition标题：时频变换：一种新的语音情感识别时频联合学习方法链接：https://arxiv.org/abs/2308.14568
作者：Yong Wang,Cheng Lu,Yuan Zong,Hailun Lian,Yan Zhao,Sunan Li机构：School of Information Science and Engineering, Southeast University, Nanjing , China,  Key Laboratory of Child Development and Learning Science of Ministry of Education,  School of Biological Science and Medical Engineering, Southeast University, Nanjing备注：Accepted by International Conference on Neural Information Processing (ICONIP2023)摘要：在本文中，我们提出了一种新的时频联合学习的语音情感识别方法，称为时频Transformer。其优点是时频Transformer可以在时域和频域分别对语音信号的局部情感相关性进行建模的同时，在时频域挖掘语音信号的全局情感模式。为此，我们首先设计了一个时间Transformer和频率变换器，分别捕获帧间和频带内的局部情感模式，以确保在时域和频域的情感信息建模的完整性。然后，提出了一种时频Transformer，通过局部时域和频域情感特征挖掘时频情感相关性，从而学习更具鉴别力的全局语音情感表征。整个过程是由一系列Transformer模型实现的时频联合学习过程。IEMOCAP和CASIA数据库上的实验表明，我们提出的方法优于国家的最先进的方法。摘要：In this paper, we propose a novel time-frequency joint learning method for speech emotion recognition, called Time-Frequency Transformer. Its advantage is that the Time-Frequency Transformer can excavate global emotion patterns in the time-frequency domain of speech signal while modeling the local emotional correlations in the time domain and frequency domain respectively. For the purpose, we first design a Time Transformer and Frequency Transformer to capture the local emotion patterns between frames and inside frequency bands respectively, so as to ensure the integrity of the emotion information modeling in both time and frequency domains. Then, a Time-Frequency Transformer is proposed to mine the time-frequency emotional correlations through the local time-domain and frequency-domain emotion features for learning more discriminative global speech emotion representation. The whole process is a time-frequency joint learning process implemented by a series of Transformer models. Experiments on IEMOCAP and CASIA databases indicate that our proposed method outdoes the state-of-the-art methods.
【2】 Spoken Language Intelligence of Large Language Models for Language  Learning标题：语言学习的大型语言模型的口语智能链接：https://arxiv.org/abs/2308.14536
作者：Linkai Peng,Baorian Nuchged,Yingming Gao机构：NetEase Youdao, Beijing , China, The University of Texas at Austin, Austin, TX , Beijing University of Posts and Telecommunications备注：28 pages, 7 figures, Preprint摘要：长期以来，人们一直希望有一个可以在现实生活中提供帮助的会话系统，而最近在大型语言模型（LLM）方面的进展正在使这一想法更接近现实。虽然LLM的性能通常令人印象深刻，但它们在需要专业知识的现实场景中的功效仍不清楚。LLM被认为在教育领域最具潜力和价值，特别是在开发能够促进语言学习的基于人工智能（AI）的虚拟教师方面。我们的重点是评估LLM在教育领域的功效，特别是在口语学习领域，包括语音学，音韵学和第二语言习得。我们引入了一个新的多项选择题数据集来评估LLM在上述场景中的有效性，包括对口语知识的理解和应用。此外，我们调查的影响，各种提示技术，如零和Few-Shot的方法（前置问题与问答范例），链的思想（CoT，认为一步一步），在域的exampler和外部工具（谷歌，维基百科）。我们使用这些方法对流行的LLM（20种不同的模型）进行了大规模评估。在实际问题推理中，我们与zero-shot基线相比取得了显著的成绩提升（GPT-3.5，49.1% -> 63.1%; LLaMA2-70B-Chat，42.2% -> 48.6%）。我们发现，不同规模的模型有很好的理解语音学，音韵学和第二语言习得的概念，但在现实世界的问题推理方面表现出局限性。此外，我们还探讨了会话沟通的初步研究结果。摘要：People have long hoped for a conversational system that can assist in real-life situations, and recent progress on large language models (LLMs) is bringing this idea closer to reality. While LLMs are often impressive in performance, their efficacy in real-world scenarios that demand expert knowledge remains unclear. LLMs are believed to hold the most potential and value in education, especially in the development of Artificial intelligence (AI) based virtual teachers capable of facilitating language learning. Our focus is centered on evaluating the efficacy of LLMs in the realm of education, specifically in the areas of spoken language learning which encompass phonetics, phonology, and second language acquisition. We introduce a new multiple-choice question dataset to evaluate the effectiveness of LLMs in the aforementioned scenarios, including understanding and application of spoken language knowledge. In addition, we investigate the influence of various prompting techniques such as zero- and few-shot method (prepending the question with question-answer exemplars), chain-of-thought (CoT, think step-by-step), in-domain exampler and external tools (Google, Wikipedia). We conducted large-scale evaluation on popular LLMs (20 distinct models) using these methods. We achieved significant performance improvements compared to the zero-shot baseline in the practical questions reasoning (GPT-3.5, 49.1% -> 63.1%; LLaMA2-70B-Chat, 42.2% -> 48.6%). We found that models of different sizes have good understanding of concepts in phonetics, phonology, and second language acquisition, but show limitations in reasoning for real-world problems. Additionally, we also explore preliminary findings on conversational communication.
【3】 InstructME: An Instruction Guided Music Edit And Remix Framework with  Latent Diffusion Models标题：InstructME：一个基于潜在扩散模型的教学制导音乐编辑与修复框架链接：https://arxiv.org/abs/2308.14360
作者：Bing Han,Junyu Dai,Xuchen Song,Weituo Hao,Xinyan He,Dong Guo,Jitong Chen,Yuxuan Wang,Yanmin Qian机构：X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, ByteDance备注：Demo samples are available at this https URL摘要：音乐编辑主要是指对乐器曲目进行修改或整体混音，通过一系列操作对原作品进行新颖的重新诠释。这些音乐处理方法在各种应用中具有巨大的潜力，但需要大量的专业知识。现有方法虽然对图像和音频修改有效，但当直接应用于音乐时会变得不稳定。这归因于音乐独特的数据性质，其中这样的方法可能无意中损害音乐的内在和谐和连贯性。在本文中，我们开发了InstructME，一个基于潜在扩散模型的教学指导音乐编辑和混音框架。我们的框架加强了U-Net与多尺度聚合，以保持编辑前后的一致性。此外，我们引入和弦进行矩阵作为条件信息，并将其纳入语义空间，以改善旋律和谐，而编辑。为了适应扩展的音乐作品，InstructME采用块Transformer，使其能够识别音乐序列中的长期时间依赖性。我们在仪器编辑，混音和多轮编辑中测试了InstructME。主观和客观的评价表明，我们所提出的方法显着超过以前的系统的音乐质量，文本相关性和和谐。演示示例可在https://musicedit.github.io/获得摘要：Music editing primarily entails the modification of instrument tracks or remixing in the whole, which offers a novel reinterpretation of the original piece through a series of operations. These music processing methods hold immense potential across various applications but demand substantial expertise. Prior methodologies, although effective for image and audio modifications, falter when directly applied to music. This is attributed to music's distinctive data nature, where such methods can inadvertently compromise the intrinsic harmony and coherence of music. In this paper, we develop InstructME, an Instruction guided Music Editing and remixing framework based on latent diffusion models. Our framework fortifies the U-Net with multi-scale aggregation in order to maintain consistency before and after editing. In addition, we introduce chord progression matrix as condition information and incorporate it in the semantic space to improve melodic harmony while editing. For accommodating extended musical pieces, InstructME employs a chunk transformer, enabling it to discern long-term temporal dependencies within music sequences. We tested InstructME in instrument-editing, remixing, and multi-round editing. Both subjective and objective evaluations indicate that our proposed method significantly surpasses preceding systems in music quality, text relevance and harmony. Demo samples are available at https://musicedit.github.io/
【4】 Voice Conversion with Denoising Diffusion Probabilistic GAN Models标题：基于去噪扩散概率GaN模型的语音转换链接：https://arxiv.org/abs/2308.14319
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by 19th International Conference on Advanced Data Mining and Applications. (ADMA 2023)摘要：语音转换是一种在保持语言信息完整性的前提下转换说话风格的方法。有许多研究人员使用深度生成模型进行语音转换任务。生成对抗网络（GANs）可以快速生成高质量的样本，但生成的样本缺乏多样性。去噪扩散概率模型（DDPMs）生成的样本在模式覆盖和样本多样性方面优于GANs。但DDPM算法计算量大，推理速度慢于GANs。为了使GANs和DDPMS更实用，我们提出了DiffGAN-VC，GANs和DDPMS的变体，以实现非并行多对多语音转换（VC）。我们使用大的步骤来实现去噪，并引入了一个多模态条件GANs模型去噪扩散生成对抗网络。根据客观和主观评价实验，DiffGAN-VC已被证明在非并行数据集上实现了高质量的语音。与CycleGAN-VC方法相比，DiffGAN-VC实现了说话人相似性、自然度和更高的音质。摘要：Voice conversion is a method that allows for the transformation of speaking style while maintaining the integrity of linguistic information. There are many researchers using deep generative models for voice conversion tasks. Generative Adversarial Networks (GANs) can quickly generate high-quality samples, but the generated samples lack diversity. The samples generated by the Denoising Diffusion Probabilistic Models (DDPMs) are better than GANs in terms of mode coverage and sample diversity. But the DDPMs have high computational costs and the inference speed is slower than GANs. In order to make GANs and DDPMs more practical we proposes DiffGAN-VC, a variant of GANs and DDPMS, to achieve non-parallel many-to-many voice conversion (VC). We use large steps to achieve denoising, and also introduce a multimodal conditional GANs to model the denoising diffusion generative adversarial network. According to both objective and subjective evaluation experiments, DiffGAN-VC has been shown to achieve high voice quality on non-parallel data sets. Compared with the CycleGAN-VC method, DiffGAN-VC achieves speaker similarity, naturalness and higher sound quality.
【5】 Symbolic & Acoustic: Multi-domain Music Emotion Modeling for  Instrumental Music标题：符号与声学：器乐的多领域音乐情感建模链接：https://arxiv.org/abs/2308.14317
作者：Kexin Zhu,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by 19th International Conference on Advanced Data Mining and Applications. (ADMA 2023)摘要：音乐情感识别涉及到音乐曲目中情感元素的自动识别，由于其在音乐信息检索领域的广泛应用，它已经获得了显着的关注。它还可以作为许多其他与人类相关的任务的上游任务，例如情感音乐生成和音乐推荐。根据现有的心理学研究，音乐情感是由音乐的音色、速度、结构等多个因素决定的。在MER中合并多个因子有助于实现更可解释和更细粒度的方法。然而，大多数先前的工作是单域和唤醒建模性能和效价建模性能之间的弱一致性。在此背景下，我们设计了一种结合符号分析和声学分析的多领域器乐情感建模方法。同时，由于音乐数据的稀有性和标注的困难性，我们的多领域方法可以充分利用有限的数据。我们的方法是使用公开的钢琴数据集EMOPIA实施和评估的，与我们的基线模型相比有了显着的改善，总体准确性提高了2.4%，建立了其最先进的性能。摘要：Music Emotion Recognition involves the automatic identification of emotional elements within music tracks, and it has garnered significant attention due to its broad applicability in the field of Music Information Retrieval. It can also be used as the upstream task of many other human-related tasks such as emotional music generation and music recommendation. Due to existing psychology research, music emotion is determined by multiple factors such as the Timbre, Velocity, and Structure of the music. Incorporating multiple factors in MER helps achieve more interpretable and finer-grained methods. However, most prior works were uni-domain and showed weak consistency between arousal modeling performance and valence modeling performance. Based on this background, we designed a multi-domain emotion modeling method for instrumental music that combines symbolic analysis and acoustic analysis. At the same time, because of the rarity of music data and the difficulty of labeling, our multi-domain approach can make full use of limited data. Our approach was implemented and assessed using the publicly available piano dataset EMOPIA, resulting in a notable improvement over our baseline model with a 2.4% increase in overall accuracy, establishing its state-of-the-art performance.
【6】 Anomalous Sound Detection Using Self-Attention-Based Frequency Pattern  Analysis of Machine Sounds标题：基于自注意的机器声音频率模式分析的异常声音检测链接：https://arxiv.org/abs/2308.14063
作者：Hejing Zhang,Jian Guan,Qiaoxi Zhu,Feiyang Xiao,Youde Liu机构：Group of Intelligent Signal Processing, Harbin Engineering University, China,  Centre for Audio, Acoustics and Vibration, University of Technology Sydney, Australia,  School of Computer Science and Technology, Harbin Institute of Technology, China备注：Accepted in INTERSPEECH 2023摘要：不同的机器可以在其发出的声音中表现出不同的频率模式。该特征最近已经在异常声音检测中被探索并且达到了最先进的性能。然而，现有方法依赖于通过观察训练数据中的有效频率范围来手动或经验地确定频率滤波器，这对于一般应用来说可能是不切实际的。提出了一种基于自注意的频率模式分析和谱时信息融合的异常声音检测方法。我们的实验表明，自我注意模块自动和自适应地分析的有效频率的机器声音，并增强的频谱特征表示的信息。通过频谱-时间信息融合，获得的音频特征最终提高了DCASE 2020挑战任务2数据集上的异常检测性能。摘要：Different machines can exhibit diverse frequency patterns in their emitted sound. This feature has been recently explored in anomaly sound detection and reached state-of-the-art performance. However, existing methods rely on the manual or empirical determination of the frequency filter by observing the effective frequency range in the training data, which may be impractical for general application. This paper proposes an anomalous sound detection method using self-attention-based frequency pattern analysis and spectral-temporal information fusion. Our experiments demonstrate that the self-attention module automatically and adaptively analyses the effective frequencies of a machine sound and enhances that information in the spectral feature representation. With spectral-temporal information fusion, the obtained audio feature eventually improves the anomaly detection performance on the DCASE 2020 Challenge Task 2 dataset.
【7】 Multi-Subdomain Adversarial Network for Cross-Subject EEG-based Emotion  Recognition标题：基于跨主体脑电信号的多子域对抗网络情绪识别链接：https://arxiv.org/abs/2308.14059
作者：Guang Lin,Jianhai Zhang机构：Hangzhou Dianzi University, Hangzhou, China摘要：在基于EEG的情绪识别中，受试者之间的个体差异是显著的，导致难以跨受试者共享模型。以前的研究使用域自适应算法来最小化全局域差异，而忽略了类信息，这可能会导致子域的不对齐，降低模型的性能。本文提出了一种多子域对抗网络（MSAN）的跨学科的基于EEG的情感识别。MSAN使用对抗性训练来建模全局域和子域中的差异，以减小类内距离并扩大类间距离。此外，MSAN通过预训练的自动编码器初始化参数，以确保模型的稳定性和可转换性。实验结果表明，在SEED数据集上，MSAN算法的准确率比非迁移算法提高了30.02%.摘要：The individual difference between subjects is significant in EEG-based emotion recognition, resulting in the difficulty of sharing the model across subjects. Previous studies use domain adaptation algorithms to minimize the global domain discrepancy while ignoring the class information, which may cause misalignment of subdomains and reduce model performance. This paper proposes a multi-subdomain adversarial network (MSAN) for cross-subject EEG-based emotion recognition. MSAN uses adversarial training to model the discrepancy in the global domain and subdomain to reduce the intra-class distance and enlarge the inter-class distance. In addition, MSAN initializes parameters through a pre-trained autoencoder to ensure the stability and convertibility of the model. The experimental results show that the accuracy of MSAN is improved by 30.02\% on the SEED dataset comparing with the nontransfer method.
【8】 A small vocabulary database of ultrasound image sequences of vocal tract  dynamics标题：声道动力学超声图像序列的小型词汇库链接：https://arxiv.org/abs/2308.13941
作者：Margareth Castillo,Felipe Rubio,Dagoberto Porras,Sonia H. Contreras-Ortiz,Alexander Sepúlveda机构：∗Escuela de Ingenier´ıas El´ectrica, Electr´o´onica y de Telecomunicaciones (E,T), Universidad Industrial de Santander, Cll , Cra , Bucaramanga, Colombia, †Electric and Electronic Engineering Programs, Universidad Tecnol´ogica de Bol´ıvar备注：None摘要：本文提出了一种新的数据库，包括并发的发音和声学语音数据。发音数据对应于声道动态的超声视频，其允许在语音产生过程期间舌头上部轮廓的可视化。声学数据由30个短句组成，这些短句是由定向心形麦克风采集的。该数据库包括来自哥伦比亚桑坦德地区的17名年轻受试者（8名男性和9名女性）的数据，他们报告没有任何言语病理学。摘要：This paper presents a new database consisting of concurrent articulatory and acoustic speech data. The articulatory data correspond to ultrasound videos of the vocal tract dynamics, which allow the visualization of the tongue upper contour during the speech production process. Acoustic data is composed of 30 short sentences that were acquired by a directional cardioid microphone. This database includes data from 17 young subjects (8 male and 9 female) from the Santander region in Colombia, who reported not having any speech pathology.
【9】 A Comprehensive Survey for Evaluation Methodologies of AI-Generated  Music标题：人工智能音乐评价方法综述链接：https://arxiv.org/abs/2308.13736
作者：Zeyu Xiong,Weitao Wang,Jing Yu,Yue Lin,Ziyan Wang机构：Chilli Chicky Band, Computational Media and Arts Thrust, The Hong Kong University of, Science and Technology (Guangzhou), Guangzhou, China, Earth, Ocean and Atmosphere Thrust, Carbon Neutrality and, Climate Change Thrust摘要：近年来，人工智能生成的音乐取得了重大进展，有几种模型在多模态和复杂的音乐类型和场景中表现良好。虽然客观指标可以用来评估生成音乐，但它们往往缺乏对音乐评估的可解释性。因此，研究人员经常诉诸主观用户研究来评估所生成的作品的质量，这可能是资源密集型的，并且比客观指标更难再现。本研究旨在全面评估评估人工智能生成音乐的主观，客观和组合方法，突出每种方法的优点和缺点。最终，本研究为音乐评价领域的生成式人工智能的统一提供了有价值的参考。摘要：In recent years, AI-generated music has made significant progress, with several models performing well in multimodal and complex musical genres and scenes. While objective metrics can be used to evaluate generative music, they often lack interpretability for musical evaluation. Therefore, researchers often resort to subjective user studies to assess the quality of the generated works, which can be resource-intensive and less reproducible than objective metrics. This study aims to comprehensively evaluate the subjective, objective, and combined methodologies for assessing AI-generated music, highlighting the advantages and disadvantages of each approach. Ultimately, this study provides a valuable reference for unifying generative AI in the field of music evaluation.
【10】 The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge标题：应对CHINE-7 DASR挑战的USTC-NERCSLIP系统链接：https://arxiv.org/abs/2308.14638
作者：Ruoyu Wang,Maokui He,Jun Du,Hengshun Zhou,Shutong Niu,Hang Chen,Yanyan Yue,Gaobin Yang,Shilong Wu,Lei Sun,Yanhui Tu,Haitao Tang,Shuangqing Qian,Tian Gao,Mengzhi Wang,Genshun Wan,Jia Pan,Jianqing Gao,Chin-Hui Lee机构：University of Science and Technology of China, China ,iFlytek Research, China, Georgia Institute of Technology, USA备注：Accepted by 2023 CHiME Workshop, Oral摘要：本技术报告详细介绍了我们的提交系统CHiME-7 DASR挑战赛，重点是在复杂的多说话人设置下的说话人日记和语音识别。此外，它还评估了系统处理不同阵列设备的效率。为了解决这些问题，我们实现了一个端到端的发言人日记系统，并介绍了一种矫正策略的基础上，多通道的空间信息。这种方法大大降低了字错误率（WER）。在识别方面，我们利用公开的预训练模型作为基础模型来训练我们的端到端语音识别模型。我们的系统在CHiME-7开发集上获得了22.4\%的宏观平均diarization-attributed WER（DA-WER），这意味着比官方基线系统相对提高了52.5\%。摘要：This technical report details our submission system to the CHiME-7 DASR Challenge, which focuses on speaker diarization and speech recognition under complex multi-speaker settings. Additionally, it also evaluates the efficiency of systems in handling diverse array devices. To address these issues, we implemented an end-to-end speaker diarization system and introduced a rectification strategy based on multi-channel spatial information. This approach significantly diminished the word error rates (WER). In terms of recognition, we utilized publicly available pre-trained models as the foundational models to train our end-to-end speech recognition models. Our system attained a macro-averaged diarization-attributed WER (DA-WER) of 22.4\% on the CHiME-7 development set, which signifies a relative improvement of 52.5\% over the official baseline system.
【11】 Data-driven 3D Room Geometry Inference with a Linear Loudspeaker Array  and a Single Microphone标题：利用线性扬声器阵列和单麦克风进行数据驱动的三维房间几何推断链接：https://arxiv.org/abs/2308.14611
作者：Cagdas Tuna,Altan Akat,H. Nazim Bicer,Andreas Walther,Emanuël A. P. Habets机构：Emanu¨el A. P. Habets,  Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany,  International Audio Laboratories Erlangen†, Germany备注：Accepted for publication in Forum Acusticum 2023摘要：知道房间几何形状对于许多音频应用可能是非常有益的，包括声音再现、声学场景分析和声源定位。房间几何推断（RGI）处理的反射器定位（RL）的基础上的一组房间脉冲响应（RIR）。受市售条形音箱日益普及的影响，本文提出了一种数据驱动的3D RGI方法，该方法使用从线性扬声器阵列到单个麦克风测量的RIR。卷积递归神经网络（CRNN）的训练使用模拟RIR在监督的方式为RL。Radon变换，这是等效的延迟和求和波束成形，被应用到多通道RIR，和所得的时域声学波束成形图馈送到CRNN。从由网络估计的麦克风位置和反射器位置推断房间几何形状。使用测量的RIR所获得的结果表明，所提出的数据驱动的方法泛化以及看不见的RIR，并实现了一个准确度水平，与基线模型驱动的RGI方法，涉及中间半监督步骤，从而提供了一个统一的和完全自动化的RGI框架。摘要：Knowing the room geometry may be very beneficial for many audio applications, including sound reproduction, acoustic scene analysis, and sound source localization. Room geometry inference (RGI) deals with the problem of reflector localization (RL) based on a set of room impulse responses (RIRs). Motivated by the increasing popularity of commercially available soundbars, this article presents a data-driven 3D RGI method using RIRs measured from a linear loudspeaker array to a single microphone. A convolutional recurrent neural network (CRNN) is trained using simulated RIRs in a supervised fashion for RL. The Radon transform, which is equivalent to delay-and-sum beamforming, is applied to multi-channel RIRs, and the resulting time-domain acoustic beamforming map is fed into the CRNN. The room geometry is inferred from the microphone position and the reflector locations estimated by the network. The results obtained using measured RIRs show that the proposed data-driven approach generalizes well to unseen RIRs and achieves an accuracy level comparable to a baseline model-driven RGI method that involves intermediate semi-supervised steps, thereby offering a unified and fully automated RGI framework.
【12】 Rep2wav: Noise Robust text-to-speech Using self-supervised  representations标题：REP2wav：基于自监督表示的抗噪文语转换链接：https://arxiv.org/abs/2308.14553
作者：Qiushi Zhu,Yu Gu,Chao Weng,Yuchen Hu,Lirong Dai,Jie Zhang机构：Tencent AI LAB, NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China, Nanyang Technological University, Singapore备注：5 pages,2 figures摘要：受益于深度学习的发展，使用干净语音的文本到语音（TTS）技术已经实现了显著的性能改进。从真实场景中采集的数据往往含有噪声，通常需要通过语音增强模型进行去噪。噪声鲁棒的TTS模型通常使用增强的语音来训练，从而遭受影响合成语音的质量的语音失真和背景噪声。同时，它表明，自我监督的预训练模型在许多语音任务中表现出良好的噪声鲁棒性，这意味着学习的表示具有更好的噪声扰动的容忍度。因此，在这项工作中，我们探索预训练模型，以提高TTS模型的噪声鲁棒性。基于HIFI-GAN，我们首先提出了一种表示到波形声码器，其目的是学习将预训练模型的表示映射到波形。然后，我们提出了一个文本到表示的Fastspeech 2模型，其目的是学习将文本映射到预训练的模型表示。在LJSpeech和LibriTTS数据集上的实验结果表明，该方法在主观和客观指标上都优于使用语音增强方法的方法。音频样本可从以下网址获得：https://zqs01.github.io/rep2wav/。摘要：Benefiting from the development of deep learning, text-to-speech (TTS) techniques using clean speech have achieved significant performance improvements. The data collected from real scenes often contain noise and generally needs to be denoised by speech enhancement models. Noise-robust TTS models are often trained using the enhanced speech, which thus suffer from speech distortion and background noise that affect the quality of the synthesized speech. Meanwhile, it was shown that self-supervised pre-trained models exhibit excellent noise robustness on many speech tasks, implying that the learned representation has a better tolerance for noise perturbations. In this work, we therefore explore pre-trained models to improve the noise robustness of TTS models. Based on HIFI-GAN we first propose a representation-to-waveform vocoder, which aims to learn to map the representation of pre-trained models to the waveform. We then propose a text-to-representation Fastspeech2 model, which aims to learn to map text to pre-trained model representations. Experimental results on the LJSpeech and LibriTTS datasets show that our method outperforms those using speech enhancement methods in both subjective and objective metrics. Audio samples are available at: https://zqs01.github.io/rep2wav/.
【13】 Speech Self-Supervised Representations Benchmarking: a Case for Larger  Probing Heads标题：语音自我监督表征基准：较大探测头的案例链接：https://arxiv.org/abs/2308.14456
作者：Salah Zaiem,Youcef Kemiche,Titouan Parcollet,Slim Essid,Mirco Ravanelli机构：Concordia University,  Mila-Quebec AI Institute备注：11 Pages摘要：自监督学习（SSL）利用未标记语音的大型数据集，以减少注释数据的数量来达到令人印象深刻的性能。大量提出的方法促进了全面基准的出现，评估其性能的一组下游任务，探索语音信号的各个方面。然而，虽然所考虑的任务的数量一直在增长，但大多数提案依赖于将冻结的SSL表示映射到任务标签的单个下游体系结构。本研究探讨了如何基准测试结果的影响，在探测头架构的变化。有趣的是，我们发现，改变下游的架构结构导致显着的波动，在性能排名的评估模型。针对语音SSL基准测试中的常见做法，我们评估更大容量的探测头，显示其性能，推理成本，泛化和多级功能利用的影响。摘要：Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of comprehensive benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, while the number of considered tasks has been growing, most proposals rely upon a single downstream architecture that maps the frozen SSL representations to the task labels. This study examines how benchmarking results are affected by changes in the probing head architecture. Interestingly, we found that altering the downstream architecture structure leads to significant fluctuations in the performance ranking of the evaluated models. Against common practices in speech SSL benchmarking, we evaluate larger-capacity probing heads, showing their impact on performance, inference costs, generalization and multi-level feature exploitation.
【14】 TextrolSpeech: A Text Style Control Speech Corpus With Codec Language  Text-to-Speech Models标题：基于编解码器语言文语转换模型的文体控制语料库链接：https://arxiv.org/abs/2308.14430
作者：Shengpeng Ji,Jialong Zuo,Minghui Fang,Ziyue Jiang,Feiyang Chen,Xinyu Duan,Baoxing Huai,Zhou Zhao机构：Zhejiang University, Huawei Cloud摘要：近年来，人们对可控文本到语音（TTS）领域的兴趣越来越大。虽然以前的研究依赖于用户提供特定的风格因素值的声学知识或选择符合某些要求的参考语音，生成语音仅从自然文本提示已经成为研究人员的新挑战。由于缺乏具有自然文本风格提示的高质量语音数据集和缺乏先进的文本可控TTS模型，这一挑战出现。鉴于此，1）我们提出TextrolSpeech，这是第一个用富文本属性注释的大规模语音情感数据集。该数据集包括236，220对自然文本描述中的风格提示，具有五个风格因子和相应的语音样本。通过迭代实验，我们引入了一个多阶段的提示编程方法，有效地利用GPT模型生成自然的风格描述，在大量。2)此外，为了解决需要产生更大的风格多样性的音频，我们提出了一个高效的架构称为Salle。该架构将文本可控TTS作为语言模型任务，利用音频编解码器代码作为中间表示来取代传统的梅尔频谱图。最后，我们成功地证明了所提出的模型的能力，在可控的TTS任务中显示出相当的性能。音频样本可在https://sall-e.github.io/上获得摘要：Recently, there has been a growing interest in the field of controllable Text-to-Speech (TTS). While previous studies have relied on users providing specific style factor values based on acoustic knowledge or selecting reference speeches that meet certain requirements, generating speech solely from natural text prompts has emerged as a new challenge for researchers. This challenge arises due to the scarcity of high-quality speech datasets with natural text style prompt and the absence of advanced text-controllable TTS models. In light of this, 1) we propose TextrolSpeech, which is the first large-scale speech emotion dataset annotated with rich text attributes. The dataset comprises 236,220 pairs of style prompt in natural text descriptions with five style factors and corresponding speech samples. Through iterative experimentation, we introduce a multi-stage prompt programming approach that effectively utilizes the GPT model for generating natural style descriptions in large volumes. 2) Furthermore, to address the need for generating audio with greater style diversity, we propose an efficient architecture called Salle. This architecture treats text controllable TTS as a language model task, utilizing audio codec codes as an intermediate representation to replace the conventional mel-spectrogram. Finally, we successfully demonstrate the ability of the proposed model by showing a comparable performance in the controllable TTS task. Audio samples are available at https://sall-e.github.io/
【15】 Fairness and Privacy in Voice Biometrics:A Study of Gender Influences  Using wav2vec 2.0标题：语音生物识别中的公平性和隐私性：基于Wav2vec 2.0的性别影响研究链接：https://arxiv.org/abs/2308.14049
作者：Oubaida Chouchane,Michele Panariello,Chiara Galdi,Massimiliano Todisco,Nicholas Evans机构：EURECOM, Sophia Antipolis, France, firstname [dot] lastname [at] eurecom [dot] fr备注：7 pages摘要：本研究调查性别信息对语音生物识别系统的效用，隐私和公平性的影响，指导下的通用数据保护条例（GDPR），强调需要尽量减少私人和敏感数据的处理和存储，并确保自动化决策系统的公平性。我们采用的方法，涉及微调的wav2vec 2.0模型的扬声器验证任务，评估潜在的性别相关的隐私漏洞的过程中。在微调过程中的性别影响，以提高公平性和隐私，以强调或模糊的性别信息内的发言者的嵌入。来自VoxCeleb数据集的结果表明，我们的对抗模型增加了针对不知情攻击的隐私，但与非对抗模型相比，说话人验证性能略有下降。然而，该模型的有效性降低对知情的攻击。系统性能进行了分析，以确定潜在的性别偏见，从而强调需要进一步的研究，以了解和改善实用程序，隐私和公平的语音生物识别系统之间的微妙的相互作用。摘要：This study investigates the impact of gender information on utility, privacy, and fairness in voice biometric systems, guided by the General Data Protection Regulation (GDPR) mandates, which underscore the need for minimizing the processing and storage of private and sensitive data, and ensuring fairness in automated decision-making systems. We adopt an approach that involves the fine-tuning of the wav2vec 2.0 model for speaker verification tasks, evaluating potential gender-related privacy vulnerabilities in the process. Gender influences during the fine-tuning process were employed to enhance fairness and privacy in order to emphasise or obscure gender information within the speakers' embeddings. Results from VoxCeleb datasets indicate our adversarial model increases privacy against uninformed attacks, yet slightly diminishes speaker verification performance compared to the non-adversarial model. However, the model's efficacy reduces against informed attacks. Analysis of system performance was conducted to identify potential gender biases, thus highlighting the need for further research to understand and improve the delicate interplay between utility, privacy, and equity in voice biometric systems.
eess.AS音频处理
【1】 The USTC-NERCSLIP Systems for the CHiME-7 DASR Challenge标题：应对CHINE-7 DASR挑战的USTC-NERCSLIP系统链接：https://arxiv.org/abs/2308.14638
作者：Ruoyu Wang,Maokui He,Jun Du,Hengshun Zhou,Shutong Niu,Hang Chen,Yanyan Yue,Gaobin Yang,Shilong Wu,Lei Sun,Yanhui Tu,Haitao Tang,Shuangqing Qian,Tian Gao,Mengzhi Wang,Genshun Wan,Jia Pan,Jianqing Gao,Chin-Hui Lee机构：University of Science and Technology of China, China ,iFlytek Research, China, Georgia Institute of Technology, USA备注：Accepted by 2023 CHiME Workshop, Oral摘要：本技术报告详细介绍了我们的提交系统CHiME-7 DASR挑战赛，重点是在复杂的多说话人设置下的说话人日记和语音识别。此外，它还评估了系统处理不同阵列设备的效率。为了解决这些问题，我们实现了一个端到端的发言人日记系统，并介绍了一种矫正策略的基础上，多通道的空间信息。这种方法大大降低了字错误率（WER）。在识别方面，我们利用公开的预训练模型作为基础模型来训练我们的端到端语音识别模型。我们的系统在CHiME-7开发集上获得了22.4\%的宏观平均diarization-attributed WER（DA-WER），这意味着比官方基线系统相对提高了52.5\%。摘要：This technical report details our submission system to the CHiME-7 DASR Challenge, which focuses on speaker diarization and speech recognition under complex multi-speaker settings. Additionally, it also evaluates the efficiency of systems in handling diverse array devices. To address these issues, we implemented an end-to-end speaker diarization system and introduced a rectification strategy based on multi-channel spatial information. This approach significantly diminished the word error rates (WER). In terms of recognition, we utilized publicly available pre-trained models as the foundational models to train our end-to-end speech recognition models. Our system attained a macro-averaged diarization-attributed WER (DA-WER) of 22.4\% on the CHiME-7 development set, which signifies a relative improvement of 52.5\% over the official baseline system.
【2】 Data-driven 3D Room Geometry Inference with a Linear Loudspeaker Array  and a Single Microphone标题：利用线性扬声器阵列和单麦克风进行数据驱动的三维房间几何推断链接：https://arxiv.org/abs/2308.14611
作者：Cagdas Tuna,Altan Akat,H. Nazim Bicer,Andreas Walther,Emanuël A. P. Habets机构：Emanu¨el A. P. Habets,  Fraunhofer Institute for Integrated Circuits IIS, Erlangen, Germany,  International Audio Laboratories Erlangen†, Germany备注：Accepted for publication in Forum Acusticum 2023摘要：知道房间几何形状对于许多音频应用可能是非常有益的，包括声音再现、声学场景分析和声源定位。房间几何推断（RGI）处理的反射器定位（RL）的基础上的一组房间脉冲响应（RIR）。受市售条形音箱日益普及的影响，本文提出了一种数据驱动的3D RGI方法，该方法使用从线性扬声器阵列到单个麦克风测量的RIR。卷积递归神经网络（CRNN）的训练使用模拟RIR在监督的方式为RL。Radon变换，这是等效的延迟和求和波束成形，被应用到多通道RIR，和所得的时域声学波束成形图馈送到CRNN。从由网络估计的麦克风位置和反射器位置推断房间几何形状。使用测量的RIR所获得的结果表明，所提出的数据驱动的方法泛化以及看不见的RIR，并实现了一个准确度水平，与基线模型驱动的RGI方法，涉及中间半监督步骤，从而提供了一个统一的和完全自动化的RGI框架。摘要：Knowing the room geometry may be very beneficial for many audio applications, including sound reproduction, acoustic scene analysis, and sound source localization. Room geometry inference (RGI) deals with the problem of reflector localization (RL) based on a set of room impulse responses (RIRs). Motivated by the increasing popularity of commercially available soundbars, this article presents a data-driven 3D RGI method using RIRs measured from a linear loudspeaker array to a single microphone. A convolutional recurrent neural network (CRNN) is trained using simulated RIRs in a supervised fashion for RL. The Radon transform, which is equivalent to delay-and-sum beamforming, is applied to multi-channel RIRs, and the resulting time-domain acoustic beamforming map is fed into the CRNN. The room geometry is inferred from the microphone position and the reflector locations estimated by the network. The results obtained using measured RIRs show that the proposed data-driven approach generalizes well to unseen RIRs and achieves an accuracy level comparable to a baseline model-driven RGI method that involves intermediate semi-supervised steps, thereby offering a unified and fully automated RGI framework.
【3】 Rep2wav: Noise Robust text-to-speech Using self-supervised  representations标题：REP2wav：基于自监督表示的抗噪文语转换链接：https://arxiv.org/abs/2308.14553
作者：Qiushi Zhu,Yu Gu,Chao Weng,Yuchen Hu,Lirong Dai,Jie Zhang机构：Tencent AI LAB, NERC-SLIP, University of Science and Technology of China (USTC), Hefei, China, Nanyang Technological University, Singapore备注：5 pages,2 figures摘要：受益于深度学习的发展，使用干净语音的文本到语音（TTS）技术已经实现了显著的性能改进。从真实场景中采集的数据往往含有噪声，通常需要通过语音增强模型进行去噪。噪声鲁棒的TTS模型通常使用增强的语音来训练，从而遭受影响合成语音的质量的语音失真和背景噪声。同时，它表明，自我监督的预训练模型在许多语音任务中表现出良好的噪声鲁棒性，这意味着学习的表示具有更好的噪声扰动的容忍度。因此，在这项工作中，我们探索预训练模型，以提高TTS模型的噪声鲁棒性。基于HIFI-GAN，我们首先提出了一种表示到波形声码器，其目的是学习将预训练模型的表示映射到波形。然后，我们提出了一个文本到表示的Fastspeech 2模型，其目的是学习将文本映射到预训练的模型表示。在LJSpeech和LibriTTS数据集上的实验结果表明，该方法在主观和客观指标上都优于使用语音增强方法的方法。音频样本可从以下网址获得：https://zqs01.github.io/rep2wav/。摘要：Benefiting from the development of deep learning, text-to-speech (TTS) techniques using clean speech have achieved significant performance improvements. The data collected from real scenes often contain noise and generally needs to be denoised by speech enhancement models. Noise-robust TTS models are often trained using the enhanced speech, which thus suffer from speech distortion and background noise that affect the quality of the synthesized speech. Meanwhile, it was shown that self-supervised pre-trained models exhibit excellent noise robustness on many speech tasks, implying that the learned representation has a better tolerance for noise perturbations. In this work, we therefore explore pre-trained models to improve the noise robustness of TTS models. Based on HIFI-GAN we first propose a representation-to-waveform vocoder, which aims to learn to map the representation of pre-trained models to the waveform. We then propose a text-to-representation Fastspeech2 model, which aims to learn to map text to pre-trained model representations. Experimental results on the LJSpeech and LibriTTS datasets show that our method outperforms those using speech enhancement methods in both subjective and objective metrics. Audio samples are available at: https://zqs01.github.io/rep2wav/.
【4】 Speech Self-Supervised Representations Benchmarking: a Case for Larger  Probing Heads标题：语音自我监督表征基准：较大探测头的案例链接：https://arxiv.org/abs/2308.14456
作者：Salah Zaiem,Youcef Kemiche,Titouan Parcollet,Slim Essid,Mirco Ravanelli机构：Concordia University,  Mila-Quebec AI Institute备注：11 Pages摘要：自监督学习（SSL）利用未标记语音的大型数据集，以减少注释数据的数量来达到令人印象深刻的性能。大量提出的方法促进了全面基准的出现，评估其性能的一组下游任务，探索语音信号的各个方面。然而，虽然所考虑的任务的数量一直在增长，但大多数提案依赖于将冻结的SSL表示映射到任务标签的单个下游体系结构。本研究探讨了如何基准测试结果的影响，在探测头架构的变化。有趣的是，我们发现，改变下游的架构结构导致显着的波动，在性能排名的评估模型。针对语音SSL基准测试中的常见做法，我们评估更大容量的探测头，显示其性能，推理成本，泛化和多级功能利用的影响。摘要：Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of comprehensive benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, while the number of considered tasks has been growing, most proposals rely upon a single downstream architecture that maps the frozen SSL representations to the task labels. This study examines how benchmarking results are affected by changes in the probing head architecture. Interestingly, we found that altering the downstream architecture structure leads to significant fluctuations in the performance ranking of the evaluated models. Against common practices in speech SSL benchmarking, we evaluate larger-capacity probing heads, showing their impact on performance, inference costs, generalization and multi-level feature exploitation.
【5】 TextrolSpeech: A Text Style Control Speech Corpus With Codec Language  Text-to-Speech Models标题：基于编解码器语言文语转换模型的文体控制语料库链接：https://arxiv.org/abs/2308.14430
作者：Shengpeng Ji,Jialong Zuo,Minghui Fang,Ziyue Jiang,Feiyang Chen,Xinyu Duan,Baoxing Huai,Zhou Zhao机构：Zhejiang University, Huawei Cloud摘要：近年来，人们对可控文本到语音（TTS）领域的兴趣越来越大。虽然以前的研究依赖于用户提供特定的风格因素值的声学知识或选择符合某些要求的参考语音，生成语音仅从自然文本提示已经成为研究人员的新挑战。由于缺乏具有自然文本风格提示的高质量语音数据集和缺乏先进的文本可控TTS模型，这一挑战出现。鉴于此，1）我们提出TextrolSpeech，这是第一个用富文本属性注释的大规模语音情感数据集。该数据集包括236，220对自然文本描述中的风格提示，具有五个风格因子和相应的语音样本。通过迭代实验，我们引入了一个多阶段的提示编程方法，有效地利用GPT模型生成自然的风格描述，在大量。2)此外，为了解决需要产生更大的风格多样性的音频，我们提出了一个高效的架构称为Salle。该架构将文本可控TTS作为语言模型任务，利用音频编解码器代码作为中间表示来取代传统的梅尔频谱图。最后，我们成功地证明了所提出的模型的能力，在可控的TTS任务中显示出相当的性能。音频样本可在https://sall-e.github.io/上获得摘要：Recently, there has been a growing interest in the field of controllable Text-to-Speech (TTS). While previous studies have relied on users providing specific style factor values based on acoustic knowledge or selecting reference speeches that meet certain requirements, generating speech solely from natural text prompts has emerged as a new challenge for researchers. This challenge arises due to the scarcity of high-quality speech datasets with natural text style prompt and the absence of advanced text-controllable TTS models. In light of this, 1) we propose TextrolSpeech, which is the first large-scale speech emotion dataset annotated with rich text attributes. The dataset comprises 236,220 pairs of style prompt in natural text descriptions with five style factors and corresponding speech samples. Through iterative experimentation, we introduce a multi-stage prompt programming approach that effectively utilizes the GPT model for generating natural style descriptions in large volumes. 2) Furthermore, to address the need for generating audio with greater style diversity, we propose an efficient architecture called Salle. This architecture treats text controllable TTS as a language model task, utilizing audio codec codes as an intermediate representation to replace the conventional mel-spectrogram. Finally, we successfully demonstrate the ability of the proposed model by showing a comparable performance in the controllable TTS task. Audio samples are available at https://sall-e.github.io/
【6】 Fairness and Privacy in Voice Biometrics:A Study of Gender Influences  Using wav2vec 2.0标题：语音生物识别中的公平性和隐私性：基于Wav2vec 2.0的性别影响研究链接：https://arxiv.org/abs/2308.14049
作者：Oubaida Chouchane,Michele Panariello,Chiara Galdi,Massimiliano Todisco,Nicholas Evans机构：EURECOM, Sophia Antipolis, France, firstname [dot] lastname [at] eurecom [dot] fr备注：7 pages摘要：本研究调查性别信息对语音生物识别系统的效用，隐私和公平性的影响，指导下的通用数据保护条例（GDPR），强调需要尽量减少私人和敏感数据的处理和存储，并确保自动化决策系统的公平性。我们采用的方法，涉及微调的wav2vec 2.0模型的扬声器验证任务，评估潜在的性别相关的隐私漏洞的过程中。在微调过程中的性别影响，以提高公平性和隐私，以强调或模糊的性别信息内的发言者的嵌入。来自VoxCeleb数据集的结果表明，我们的对抗模型增加了针对不知情攻击的隐私，但与非对抗模型相比，说话人验证性能略有下降。然而，该模型的有效性降低对知情的攻击。系统性能进行了分析，以确定潜在的性别偏见，从而强调需要进一步的研究，以了解和改善实用程序，隐私和公平的语音生物识别系统之间的微妙的相互作用。摘要：This study investigates the impact of gender information on utility, privacy, and fairness in voice biometric systems, guided by the General Data Protection Regulation (GDPR) mandates, which underscore the need for minimizing the processing and storage of private and sensitive data, and ensuring fairness in automated decision-making systems. We adopt an approach that involves the fine-tuning of the wav2vec 2.0 model for speaker verification tasks, evaluating potential gender-related privacy vulnerabilities in the process. Gender influences during the fine-tuning process were employed to enhance fairness and privacy in order to emphasise or obscure gender information within the speakers' embeddings. Results from VoxCeleb datasets indicate our adversarial model increases privacy against uninformed attacks, yet slightly diminishes speaker verification performance compared to the non-adversarial model. However, the model's efficacy reduces against informed attacks. Analysis of system performance was conducted to identify potential gender biases, thus highlighting the need for further research to understand and improve the delicate interplay between utility, privacy, and equity in voice biometric systems.
【7】 Time-Frequency Transformer: A Novel Time Frequency Joint Learning Method  for Speech Emotion Recognition标题：时频变换：一种新的语音情感识别时频联合学习方法链接：https://arxiv.org/abs/2308.14568
作者：Yong Wang,Cheng Lu,Yuan Zong,Hailun Lian,Yan Zhao,Sunan Li机构：School of Information Science and Engineering, Southeast University, Nanjing , China,  Key Laboratory of Child Development and Learning Science of Ministry of Education,  School of Biological Science and Medical Engineering, Southeast University, Nanjing备注：Accepted by International Conference on Neural Information Processing (ICONIP2023)摘要：在本文中，我们提出了一种新的时频联合学习的语音情感识别方法，称为时频Transformer。其优点是时频Transformer可以在时域和频域分别对语音信号的局部情感相关性进行建模的同时，在时频域挖掘语音信号的全局情感模式。为此，我们首先设计了一个时间Transformer和频率变换器，分别捕获帧间和频带内的局部情感模式，以确保在时域和频域的情感信息建模的完整性。然后，提出了一种时频Transformer，通过局部时域和频域情感特征挖掘时频情感相关性，从而学习更具鉴别力的全局语音情感表征。整个过程是由一系列Transformer模型实现的时频联合学习过程。IEMOCAP和CASIA数据库上的实验表明，我们提出的方法优于国家的最先进的方法。摘要：In this paper, we propose a novel time-frequency joint learning method for speech emotion recognition, called Time-Frequency Transformer. Its advantage is that the Time-Frequency Transformer can excavate global emotion patterns in the time-frequency domain of speech signal while modeling the local emotional correlations in the time domain and frequency domain respectively. For the purpose, we first design a Time Transformer and Frequency Transformer to capture the local emotion patterns between frames and inside frequency bands respectively, so as to ensure the integrity of the emotion information modeling in both time and frequency domains. Then, a Time-Frequency Transformer is proposed to mine the time-frequency emotional correlations through the local time-domain and frequency-domain emotion features for learning more discriminative global speech emotion representation. The whole process is a time-frequency joint learning process implemented by a series of Transformer models. Experiments on IEMOCAP and CASIA databases indicate that our proposed method outdoes the state-of-the-art methods.
【8】 Spoken Language Intelligence of Large Language Models for Language  Learning标题：语言学习的大型语言模型的口语智能链接：https://arxiv.org/abs/2308.14536
作者：Linkai Peng,Baorian Nuchged,Yingming Gao机构：NetEase Youdao, Beijing , China, The University of Texas at Austin, Austin, TX , Beijing University of Posts and Telecommunications备注：28 pages, 7 figures, Preprint摘要：长期以来，人们一直希望有一个可以在现实生活中提供帮助的会话系统，而最近在大型语言模型（LLM）方面的进展正在使这一想法更接近现实。虽然LLM的性能通常令人印象深刻，但它们在需要专业知识的现实场景中的功效仍不清楚。LLM被认为在教育领域最具潜力和价值，特别是在开发能够促进语言学习的基于人工智能（AI）的虚拟教师方面。我们的重点是评估LLM在教育领域的功效，特别是在口语学习领域，包括语音学，音韵学和第二语言习得。我们引入了一个新的多项选择题数据集来评估LLM在上述场景中的有效性，包括对口语知识的理解和应用。此外，我们调查的影响，各种提示技术，如零和Few-Shot的方法（前置问题与问答范例），链的思想（CoT，认为一步一步），在域的exampler和外部工具（谷歌，维基百科）。我们使用这些方法对流行的LLM（20种不同的模型）进行了大规模评估。在实际问题推理中，我们与zero-shot基线相比取得了显著的成绩提升（GPT-3.5，49.1% -> 63.1%; LLaMA2-70B-Chat，42.2% -> 48.6%）。我们发现，不同规模的模型有很好的理解语音学，音韵学和第二语言习得的概念，但在现实世界的问题推理方面表现出局限性。此外，我们还探讨了会话沟通的初步研究结果。摘要：People have long hoped for a conversational system that can assist in real-life situations, and recent progress on large language models (LLMs) is bringing this idea closer to reality. While LLMs are often impressive in performance, their efficacy in real-world scenarios that demand expert knowledge remains unclear. LLMs are believed to hold the most potential and value in education, especially in the development of Artificial intelligence (AI) based virtual teachers capable of facilitating language learning. Our focus is centered on evaluating the efficacy of LLMs in the realm of education, specifically in the areas of spoken language learning which encompass phonetics, phonology, and second language acquisition. We introduce a new multiple-choice question dataset to evaluate the effectiveness of LLMs in the aforementioned scenarios, including understanding and application of spoken language knowledge. In addition, we investigate the influence of various prompting techniques such as zero- and few-shot method (prepending the question with question-answer exemplars), chain-of-thought (CoT, think step-by-step), in-domain exampler and external tools (Google, Wikipedia). We conducted large-scale evaluation on popular LLMs (20 distinct models) using these methods. We achieved significant performance improvements compared to the zero-shot baseline in the practical questions reasoning (GPT-3.5, 49.1% -> 63.1%; LLaMA2-70B-Chat, 42.2% -> 48.6%). We found that models of different sizes have good understanding of concepts in phonetics, phonology, and second language acquisition, but show limitations in reasoning for real-world problems. Additionally, we also explore preliminary findings on conversational communication.
【9】 InstructME: An Instruction Guided Music Edit And Remix Framework with  Latent Diffusion Models标题：InstructME：一个基于潜在扩散模型的教学制导音乐编辑与修复框架链接：https://arxiv.org/abs/2308.14360
作者：Bing Han,Junyu Dai,Xuchen Song,Weituo Hao,Xinyan He,Dong Guo,Jitong Chen,Yuxuan Wang,Yanmin Qian机构：X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, ByteDance备注：Demo samples are available at this https URL摘要：音乐编辑主要是指对乐器曲目进行修改或整体混音，通过一系列操作对原作品进行新颖的重新诠释。这些音乐处理方法在各种应用中具有巨大的潜力，但需要大量的专业知识。现有方法虽然对图像和音频修改有效，但当直接应用于音乐时会变得不稳定。这归因于音乐独特的数据性质，其中这样的方法可能无意中损害音乐的内在和谐和连贯性。在本文中，我们开发了InstructME，一个基于潜在扩散模型的教学指导音乐编辑和混音框架。我们的框架加强了U-Net与多尺度聚合，以保持编辑前后的一致性。此外，我们引入和弦进行矩阵作为条件信息，并将其纳入语义空间，以改善旋律和谐，而编辑。为了适应扩展的音乐作品，InstructME采用块Transformer，使其能够识别音乐序列中的长期时间依赖性。我们在仪器编辑，混音和多轮编辑中测试了InstructME。主观和客观的评价表明，我们所提出的方法显着超过以前的系统的音乐质量，文本相关性和和谐。演示示例可在https://musicedit.github.io/获得摘要：Music editing primarily entails the modification of instrument tracks or remixing in the whole, which offers a novel reinterpretation of the original piece through a series of operations. These music processing methods hold immense potential across various applications but demand substantial expertise. Prior methodologies, although effective for image and audio modifications, falter when directly applied to music. This is attributed to music's distinctive data nature, where such methods can inadvertently compromise the intrinsic harmony and coherence of music. In this paper, we develop InstructME, an Instruction guided Music Editing and remixing framework based on latent diffusion models. Our framework fortifies the U-Net with multi-scale aggregation in order to maintain consistency before and after editing. In addition, we introduce chord progression matrix as condition information and incorporate it in the semantic space to improve melodic harmony while editing. For accommodating extended musical pieces, InstructME employs a chunk transformer, enabling it to discern long-term temporal dependencies within music sequences. We tested InstructME in instrument-editing, remixing, and multi-round editing. Both subjective and objective evaluations indicate that our proposed method significantly surpasses preceding systems in music quality, text relevance and harmony. Demo samples are available at https://musicedit.github.io/
【10】 Voice Conversion with Denoising Diffusion Probabilistic GAN Models标题：基于去噪扩散概率GaN模型的语音转换链接：https://arxiv.org/abs/2308.14319
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by 19th International Conference on Advanced Data Mining and Applications. (ADMA 2023)摘要：语音转换是一种在保持语言信息完整性的前提下转换说话风格的方法。有许多研究人员使用深度生成模型进行语音转换任务。生成对抗网络（GANs）可以快速生成高质量的样本，但生成的样本缺乏多样性。去噪扩散概率模型（DDPMs）生成的样本在模式覆盖和样本多样性方面优于GANs。但DDPM算法计算量大，推理速度慢于GANs。为了使GANs和DDPMS更实用，我们提出了DiffGAN-VC，GANs和DDPMS的变体，以实现非并行多对多语音转换（VC）。我们使用大的步骤来实现去噪，并引入了一个多模态条件GANs模型去噪扩散生成对抗网络。根据客观和主观评价实验，DiffGAN-VC已被证明在非并行数据集上实现了高质量的语音。与CycleGAN-VC方法相比，DiffGAN-VC实现了说话人相似性、自然度和更高的音质。摘要：Voice conversion is a method that allows for the transformation of speaking style while maintaining the integrity of linguistic information. There are many researchers using deep generative models for voice conversion tasks. Generative Adversarial Networks (GANs) can quickly generate high-quality samples, but the generated samples lack diversity. The samples generated by the Denoising Diffusion Probabilistic Models (DDPMs) are better than GANs in terms of mode coverage and sample diversity. But the DDPMs have high computational costs and the inference speed is slower than GANs. In order to make GANs and DDPMs more practical we proposes DiffGAN-VC, a variant of GANs and DDPMS, to achieve non-parallel many-to-many voice conversion (VC). We use large steps to achieve denoising, and also introduce a multimodal conditional GANs to model the denoising diffusion generative adversarial network. According to both objective and subjective evaluation experiments, DiffGAN-VC has been shown to achieve high voice quality on non-parallel data sets. Compared with the CycleGAN-VC method, DiffGAN-VC achieves speaker similarity, naturalness and higher sound quality.
【11】 Symbolic & Acoustic: Multi-domain Music Emotion Modeling for  Instrumental Music标题：符号与声学：器乐的多领域音乐情感建模链接：https://arxiv.org/abs/2308.14317
作者：Kexin Zhu,Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by 19th International Conference on Advanced Data Mining and Applications. (ADMA 2023)摘要：音乐情感识别涉及到音乐曲目中情感元素的自动识别，由于其在音乐信息检索领域的广泛应用，它已经获得了显着的关注。它还可以作为许多其他与人类相关的任务的上游任务，例如情感音乐生成和音乐推荐。根据现有的心理学研究，音乐情感是由音乐的音色、速度、结构等多个因素决定的。在MER中合并多个因子有助于实现更可解释和更细粒度的方法。然而，大多数先前的工作是单域和唤醒建模性能和效价建模性能之间的弱一致性。在此背景下，我们设计了一种结合符号分析和声学分析的多领域器乐情感建模方法。同时，由于音乐数据的稀有性和标注的困难性，我们的多领域方法可以充分利用有限的数据。我们的方法是使用公开的钢琴数据集EMOPIA实施和评估的，与我们的基线模型相比有了显着的改善，总体准确性提高了2.4%，建立了其最先进的性能。摘要：Music Emotion Recognition involves the automatic identification of emotional elements within music tracks, and it has garnered significant attention due to its broad applicability in the field of Music Information Retrieval. It can also be used as the upstream task of many other human-related tasks such as emotional music generation and music recommendation. Due to existing psychology research, music emotion is determined by multiple factors such as the Timbre, Velocity, and Structure of the music. Incorporating multiple factors in MER helps achieve more interpretable and finer-grained methods. However, most prior works were uni-domain and showed weak consistency between arousal modeling performance and valence modeling performance. Based on this background, we designed a multi-domain emotion modeling method for instrumental music that combines symbolic analysis and acoustic analysis. At the same time, because of the rarity of music data and the difficulty of labeling, our multi-domain approach can make full use of limited data. Our approach was implemented and assessed using the publicly available piano dataset EMOPIA, resulting in a notable improvement over our baseline model with a 2.4% increase in overall accuracy, establishing its state-of-the-art performance.
【12】 Anomalous Sound Detection Using Self-Attention-Based Frequency Pattern  Analysis of Machine Sounds标题：基于自注意的机器声音频率模式分析的异常声音检测链接：https://arxiv.org/abs/2308.14063
作者：Hejing Zhang,Jian Guan,Qiaoxi Zhu,Feiyang Xiao,Youde Liu机构：Group of Intelligent Signal Processing, Harbin Engineering University, China,  Centre for Audio, Acoustics and Vibration, University of Technology Sydney, Australia,  School of Computer Science and Technology, Harbin Institute of Technology, China备注：Accepted in INTERSPEECH 2023摘要：不同的机器可以在其发出的声音中表现出不同的频率模式。该特征最近已经在异常声音检测中被探索并且达到了最先进的性能。然而，现有方法依赖于通过观察训练数据中的有效频率范围来手动或经验地确定频率滤波器，这对于一般应用来说可能是不切实际的。提出了一种基于自注意的频率模式分析和谱时信息融合的异常声音检测方法。我们的实验表明，自我注意模块自动和自适应地分析的有效频率的机器声音，并增强的频谱特征表示的信息。通过频谱-时间信息融合，获得的音频特征最终提高了DCASE 2020挑战任务2数据集上的异常检测性能。摘要：Different machines can exhibit diverse frequency patterns in their emitted sound. This feature has been recently explored in anomaly sound detection and reached state-of-the-art performance. However, existing methods rely on the manual or empirical determination of the frequency filter by observing the effective frequency range in the training data, which may be impractical for general application. This paper proposes an anomalous sound detection method using self-attention-based frequency pattern analysis and spectral-temporal information fusion. Our experiments demonstrate that the self-attention module automatically and adaptively analyses the effective frequencies of a machine sound and enhances that information in the spectral feature representation. With spectral-temporal information fusion, the obtained audio feature eventually improves the anomaly detection performance on the DCASE 2020 Challenge Task 2 dataset.
【13】 Multi-Subdomain Adversarial Network for Cross-Subject EEG-based Emotion  Recognition标题：基于跨主体脑电信号的多子域对抗网络情绪识别链接：https://arxiv.org/abs/2308.14059
作者：Guang Lin,Jianhai Zhang机构：Hangzhou Dianzi University, Hangzhou, China摘要：在基于EEG的情绪识别中，受试者之间的个体差异是显著的，导致难以跨受试者共享模型。以前的研究使用域自适应算法来最小化全局域差异，而忽略了类信息，这可能会导致子域的不对齐，降低模型的性能。本文提出了一种多子域对抗网络（MSAN）的跨学科的基于EEG的情感识别。MSAN使用对抗性训练来建模全局域和子域中的差异，以减小类内距离并扩大类间距离。此外，MSAN通过预训练的自动编码器初始化参数，以确保模型的稳定性和可转换性。实验结果表明，在SEED数据集上，MSAN算法的准确率比非迁移算法提高了30.02%.摘要：The individual difference between subjects is significant in EEG-based emotion recognition, resulting in the difficulty of sharing the model across subjects. Previous studies use domain adaptation algorithms to minimize the global domain discrepancy while ignoring the class information, which may cause misalignment of subdomains and reduce model performance. This paper proposes a multi-subdomain adversarial network (MSAN) for cross-subject EEG-based emotion recognition. MSAN uses adversarial training to model the discrepancy in the global domain and subdomain to reduce the intra-class distance and enlarge the inter-class distance. In addition, MSAN initializes parameters through a pre-trained autoencoder to ensure the stability and convertibility of the model. The experimental results show that the accuracy of MSAN is improved by 30.02\% on the SEED dataset comparing with the nontransfer method.
【14】 A small vocabulary database of ultrasound image sequences of vocal tract  dynamics标题：声道动力学超声图像序列的小型词汇库链接：https://arxiv.org/abs/2308.13941
作者：Margareth Castillo,Felipe Rubio,Dagoberto Porras,Sonia H. Contreras-Ortiz,Alexander Sepúlveda机构：∗Escuela de Ingenier´ıas El´ectrica, Electr´o´onica y de Telecomunicaciones (E,T), Universidad Industrial de Santander, Cll , Cra , Bucaramanga, Colombia, †Electric and Electronic Engineering Programs, Universidad Tecnol´ogica de Bol´ıvar备注：None摘要：本文提出了一种新的数据库，包括并发的发音和声学语音数据。发音数据对应于声道动态的超声视频，其允许在语音产生过程期间舌头上部轮廓的可视化。声学数据由30个短句组成，这些短句是由定向心形麦克风采集的。该数据库包括来自哥伦比亚桑坦德地区的17名年轻受试者（8名男性和9名女性）的数据，他们报告没有任何言语病理学。摘要：This paper presents a new database consisting of concurrent articulatory and acoustic speech data. The articulatory data correspond to ultrasound videos of the vocal tract dynamics, which allow the visualization of the tongue upper contour during the speech production process. Acoustic data is composed of 30 short sentences that were acquired by a directional cardioid microphone. This database includes data from 17 young subjects (8 male and 9 female) from the Santander region in Colombia, who reported not having any speech pathology.
【15】 A Comprehensive Survey for Evaluation Methodologies of AI-Generated  Music标题：人工智能音乐评价方法综述链接：https://arxiv.org/abs/2308.13736
作者：Zeyu Xiong,Weitao Wang,Jing Yu,Yue Lin,Ziyan Wang机构：Chilli Chicky Band, Computational Media and Arts Thrust, The Hong Kong University of, Science and Technology (Guangzhou), Guangzhou, China, Earth, Ocean and Atmosphere Thrust, Carbon Neutrality and, Climate Change Thrust摘要：近年来，人工智能生成的音乐取得了重大进展，有几种模型在多模态和复杂的音乐类型和场景中表现良好。虽然客观指标可以用来评估生成音乐，但它们往往缺乏对音乐评估的可解释性。因此，研究人员经常诉诸主观用户研究来评估所生成的作品的质量，这可能是资源密集型的，并且比客观指标更难再现。本研究旨在全面评估评估人工智能生成音乐的主观，客观和组合方法，突出每种方法的优点和缺点。最终，本研究为音乐评价领域的生成式人工智能的统一提供了有价值的参考。摘要：In recent years, AI-generated music has made significant progress, with several models performing well in multimodal and complex musical genres and scenes. While objective metrics can be used to evaluate generative music, they often lack interpretability for musical evaluation. Therefore, researchers often resort to subjective user studies to assess the quality of the generated works, which can be resource-intensive and less reproducible than objective metrics. This study aims to comprehensively evaluate the subjective, objective, and combined methodologies for assessing AI-generated music, highlighting the advantages and disadvantages of each approach. Ultimately, this study provides a valuable reference for unifying generative AI in the field of music evaluation.
机器翻译，仅供参考