今日论文合集：cs.SD语音14篇，eess.AS音频处理15篇。
cs.SD语音
【1】 Towards Improving the Expressiveness of Singing Voice Synthesis with  BERT Derived Semantic Information标题：利用BERT导出的语义信息提高歌唱声音合成的表现力链接：https://arxiv.org/abs/2308.16836
作者：Shaohuan Zhou,Shun Lei,Weiya You,Deyi Tuo,Yuren You,Zhiyong Wu,Shiyin Kang,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China,  Huya Inc., Guangzhou, China,  The Chinese University of Hong Kong, Hong Kong SAR, China,  XVerse Inc., Shenzhen, China摘要：本文提出了一种端到端的高质量歌唱语音合成系统，该系统使用双向编码器表示从Transformers（BERT）派生的语义嵌入，以提高合成歌唱语音的表现力。基于最近提出的VISinger的主要架构，我们提出了几个具体的设计，以表现力的歌唱声音合成。首先，与以前的SVS模型不同，我们使用从预先训练的BERT提取的歌词的文本表示作为模型的额外输入。该表示包含歌词语义信息，有助于SVS系统产生更有表现力和自然的声音。其次，我们进一步引入了一个能量预测器来稳定合成的声音和模型的更广泛的能量变化，也有助于歌唱的声音的表现力。最后但并非最不重要的是，为了减弱跑调问题，重新设计音高预测器以预测真实音符音高比。客观和主观实验结果表明，所提出的SVS系统可以产生更高质量的歌唱声音比VISinger。摘要：This paper presents an end-to-end high-quality singing voice synthesis (SVS) system that uses bidirectional encoder representation from Transformers (BERT) derived semantic embeddings to improve the expressiveness of the synthesized singing voice. Based on the main architecture of recently proposed VISinger, we put forward several specific designs for expressive singing voice synthesis. First, different from the previous SVS models, we use text representation of lyrics extracted from pre-trained BERT as additional input to the model. The representation contains information about semantics of the lyrics, which could help SVS system produce more expressive and natural voice. Second, we further introduce an energy predictor to stabilize the synthesized voice and model the wider range of energy variations that also contribute to the expressiveness of singing voice. Last but not the least, to attenuate the off-key issues, the pitch predictor is re-designed to predict the real to note pitch ratio. Both objective and subjective experimental results indicate that the proposed SVS system can produce singing voice with higher-quality outperforming VISinger.
【2】 SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language  Models标题：SpeechTokenizer：适用于语音大语言模型的统一语音标记器链接：https://arxiv.org/abs/2308.16692
作者：Xin Zhang,Dong Zhang,Shimin Li,Yaqian Zhou,Xipeng Qiu机构：School of Computer Science, Fudan University, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University备注：SpeechTokenizer project page is this https URL摘要：当前的语音大语言模型建立在离散的语音表示上，其可以被分类为语义令牌和声学令牌。然而，现有的语音令牌不是专门为语音语言建模而设计的。为了评估语音标记用于构建语音语言模型的适用性，我们建立了第一个基准测试SLMTokBench。我们的研究结果表明，无论是语义还是声学令牌是理想的为此目的。因此，我们提出了SpeechTokenizer，一个统一的语音标记的语音大语言模型。SpeechTokenizer采用具有残差矢量量化（RVQ）的编码器-解码器架构。统一语义和声学令牌，SpeechTokenizer在不同的RVQ层上分层地解开语音信息的不同方面。此外，我们构建了一个统一的语音语言模型（USLM）利用SpeechTokenizer。实验表明，SpeechTokenizer在语音重建方面与EnCodec表现相当，并在SLMTokBench基准测试中表现出强大的性能。此外，USLM优于VALL-E在zero-shot文本到语音的任务。代码和型号可在https://github.com/ZhangXInFD/SpeechTokenizer/获得。摘要：Current speech large language models build upon discrete speech representations, which can be categorized into semantic tokens and acoustic tokens. However, existing speech tokens are not specifically designed for speech language modeling. To assess the suitability of speech tokens for building speech language models, we established the first benchmark, SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified speech tokenizer for speech large language models. SpeechTokenizer adopts the Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of speech information hierarchically across different RVQ layers. Furthermore, We construct a Unified Speech Language Model (USLM) leveraging SpeechTokenizer. Experiments show that SpeechTokenizer performs comparably to EnCodec in speech reconstruction and demonstrates strong performance on the SLMTokBench benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks. Code and models are available at https://github.com/ZhangXInFD/SpeechTokenizer/.
【3】 Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting标题：动态nsNet2：提前退出的高效深度噪声抑制链接：https://arxiv.org/abs/2308.16678
作者：Riccardo Miccini,Alaa Zniber,Clément Laroche,Tobias Piechowiak,Martin Schoeberl,Luca Pezzarossa,Ouassim Karrakchou,Jens Sparsø,Mounir Ghogho机构：Cl´ement Laroche⋆, ⋆ GN Audio, † Technical University of Denmark, § International University of Rabat备注：Accepted at the MLSP 2023摘要：尽管深度学习在深度噪声抑制领域取得了长足的进步，但在资源受限的设备上利用深度架构仍然具有挑战性。因此，我们提出了一个基于nsNet2的早期退出模型，该模型通过在不同阶段停止计算，提供了几个级别的准确性和资源节约。此外，我们适应原来的架构，通过分裂的信息流，以考虑到注入的动态。我们表现出的性能和计算复杂性之间的权衡建立的指标的基础上。摘要：Although deep learning has made strides in the field of deep noise suppression, leveraging deep architectures on resource-constrained devices still proved challenging. Therefore, we present an early-exiting model based on nsNet2 that provides several levels of accuracy and resource savings by halting computations at different stages. Moreover, we adapt the original architecture by splitting the information flow to take into account the injected dynamism. We show the trade-offs between performance and computational complexity based on established metrics.
【4】 Towards Spontaneous Style Modeling with Semi-supervised Pre-training for  Conversational Text-to-Speech Synthesis标题：会话文语合成中半监督预训练的自发风格建模链接：https://arxiv.org/abs/2308.16593
作者：Weiqin Li,Shun Lei,Qiaochu Huang,Yixuan Zhou,Zhiyong Wu,Shiyin Kang,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, XVerse Inc., Shenzhen, China, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by INTERSPEECH 2023摘要：对话中经常出现的自发行为使演讲比阅读方式更人性化。然而，由于缺乏高质量的自发数据集和标记自发行为的高成本，合成自发风格的语音具有挑战性。在本文中，我们提出了一种半监督预训练方法，以增加自发风格的语音和自发行为标签的数量。在半监督学习的过程中，文本和语音信息被认为是用于检测语音中的自发行为标签。此外，语言感知编码器被用来模拟会话中每个句子之间的关系。实验结果表明，我们所提出的方法实现了卓越的表达语音合成性能的能力，自发行为的自发风格的语音模型和预测合理的自发行为从文本。摘要：The spontaneous behavior that often occurs in conversations makes speech more human-like compared to reading-style. However, synthesizing spontaneous-style speech is challenging due to the lack of high-quality spontaneous datasets and the high cost of labeling spontaneous behavior. In this paper, we propose a semi-supervised pre-training method to increase the amount of spontaneous-style speech and spontaneous behavioral labels. In the process of semi-supervised learning, both text and speech information are considered for detecting spontaneous behaviors labels in speech. Moreover, a linguistic-aware encoder is used to model the relationship between each sentence in the conversation. Experimental results indicate that our proposed method achieves superior expressive speech synthesis performance with the ability to model spontaneous behavior in spontaneous-style speech and predict reasonable spontaneous behavior from text.
【5】 Improving Mandarin Prosodic Structure Prediction with Multi-level  Contextual Information标题：利用多层上下文信息改进汉语韵律结构预测链接：https://arxiv.org/abs/2308.16577
作者：Jie Chen,Changhe Song,Deyi Tuo,Xixin Wu,Shiyin Kang,Zhiyong Wu,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China,  XVerse Inc., Shenzhen, China,  Huya Inc., Guangzhou, China,  The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by Interspeech2022摘要：在文本-语音转换（TTS）合成中，韵律结构预测（PSP）在生成自然、可理解的语音中起着重要作用。虽然语间语言信息会影响目标话语的语音理解，但以往的PSP研究主要集中在利用当前话语的语内语言信息。本工作提出利用话语间的语言信息来提高PSP的性能。分层编码器从输入文本的字符层、话语层和话语层提取多层次的语境信息，包括话语间和话语内的语言信息。然后，多任务学习（MTL）解码器预测韵律边界的多级上下文信息。两个数据集的客观评价结果表明，我们的方法取得了更好的预测韵律词（PW），韵律短语（PPH）和语调短语（IPH）的F1得分。它证明了使用多级上下文信息的PSP的有效性。主观偏好测试也表明合成语音的自然度得到了改善。摘要：For text-to-speech (TTS) synthesis, prosodic structure prediction (PSP) plays an important role in producing natural and intelligible speech. Although inter-utterance linguistic information can influence the speech interpretation of the target utterance, previous works on PSP mainly focus on utilizing intrautterance linguistic information of the current utterance only. This work proposes to use inter-utterance linguistic information to improve the performance of PSP. Multi-level contextual information, which includes both inter-utterance and intrautterance linguistic information, is extracted by a hierarchical encoder from character level, utterance level and discourse level of the input text. Then a multi-task learning (MTL) decoder predicts prosodic boundaries from multi-level contextual information. Objective evaluation results on two datasets show that our method achieves better F1 scores in predicting prosodic word (PW), prosodic phrase (PPH) and intonational phrase (IPH). It demonstrates the effectiveness of using multi-level contextual information for PSP. Subjective preference tests also indicate the naturalness of synthesized speeches are improved.
【6】 LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech标题：LightGrad：轻量级文语转换扩散概率模型链接：https://arxiv.org/abs/2308.16569
作者：Jie Chen,Xingchen Song,Zhendong Peng,Binbin Zhang,Fuping Pan,Zhiyong Wu机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China,  Horizon Robotics, Beijing, China,  WeNet Open Source Community,  The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by ICASSP 2023摘要：神经文本到语音（TTS）模型的最新进展将成千上万的TTS应用带入日常生活中，其中模型被部署在云中以向海关提供服务。在这些模型中，扩散概率模型（DPMs），它可以稳定地训练，并且与其他生成模型相比，参数效率更高。由于在海关和云之间传输数据会带来高延迟和暴露私有数据的风险，因此在边缘设备上部署TTS模型是首选。当在边缘设备上实现DPM时，存在两个实际问题。首先，当前的DPM对于资源受限的设备来说不够轻量。其次，DPM在推理中需要许多去噪步骤，这增加了延迟。在这项工作中，我们提出了LightGrad，一个轻量级的DPM TTS。LightGrad配备了轻量级的U-Net扩散解码器和免训练快速采样技术，减少了模型参数和推理延迟。流式推理也在LightGrad中实现，以进一步减少延迟。与Grad-TTS相比，LightGrad在参数上减少了62.2%，延迟减少了65.7%，同时在4个去噪步骤中保持了汉语普通话和英语相当的语音质量。摘要：Recent advances in neural text-to-speech (TTS) models bring thousands of TTS applications into daily life, where models are deployed in cloud to provide services for customs. Among these models are diffusion probabilistic models (DPMs), which can be stably trained and are more parameter-efficient compared with other generative models. As transmitting data between customs and the cloud introduces high latency and the risk of exposing private data, deploying TTS models on edge devices is preferred. When implementing DPMs onto edge devices, there are two practical problems. First, current DPMs are not lightweight enough for resource-constrained devices. Second, DPMs require many denoising steps in inference, which increases latency. In this work, we present LightGrad, a lightweight DPM for TTS. LightGrad is equipped with a lightweight U-Net diffusion decoder and a training-free fast sampling technique, reducing both model parameters and inference latency. Streaming inference is also implemented in LightGrad to reduce latency further. Compared with Grad-TTS, LightGrad achieves 62.2% reduction in paramters, 65.7% reduction in latency, while preserving comparable speech quality on both Chinese Mandarin and English in 4 denoising steps.
【7】 Sequential Pitch Distributions for Raga Detection标题：序列基音分布在RAGA检测中的应用链接：https://arxiv.org/abs/2308.16421
作者：Vishwaas Narasinh,Senthil Raja G机构：Published on: Aug备注：None摘要：Raga是印度艺术音乐（IAM）中的一个基本旋律概念。它的特点是复杂的模式。所有的表演和组成都是基于raga框架。在音乐信息检索领域，拉加和声调检测一直是一个长期的研究课题。在本文中，我们试图使用一种新的功能，从音频样本中提取序列或时间信息来检测的拉加。我们称这些为顺序音高分布（SPD），其是随时间在两个给定音高值之间的音高值上取得的分布。我们还实现了国家的最先进的结果，在印度斯坦语和卡尔纳蒂克音乐拉加数据集的准确性分别为99%和88.13%。SPD提供了一个很大的提高准确性超过一个标准的间距分布。然而，本文的主要目标是提出一种替代方法来建模的旋律的时间方面，从而推导出的拉加。摘要：Raga is a fundamental melodic concept in Indian Art Music (IAM). It is characterized by complex patterns. All performances and compositions are based on the raga framework. Raga and tonic detection have been a long-standing research problem in the field of Music Information Retrieval. In this paper, we attempt to detect the raga using a novel feature to extract sequential or temporal information from an audio sample. We call these Sequential Pitch Distributions (SPD), which are distributions taken over pitch values between two given pitch values over time. We also achieve state-of-the-art results on both Hindustani and Carnatic music raga data sets with an accuracy of 99% and 88.13%, respectively. SPD gives a great boost in accuracy over a standard pitch distribution. The main goal of this paper, however, is to present an alternative approach to modeling the temporal aspects of the melody and thereby deducing the raga.
【8】 The Biased Journey of MSD_AUDIO.ZIP标题：MSD_AUDIO.ZIP的偏颇之旅链接：https://arxiv.org/abs/2308.16389
作者：Haven Kim,Keunwoo Choi,Mateusz Modrzejewski,Cynthia C. S. Liem机构：Graduate School of Culture Technology, KAIST, South Korea,  Gaudio Lab, Inc., Seoul, South Korea  Prescient Design, New York, USA,  Institute of Computer Science, Warsaw University of Technology, Poland摘要：学术数据的公平分配对于确保平等的研究机会以及最终取得进一步进展至关重要。然而，由于将API用于对应于百万歌曲数据集的音频数据的复杂性以及其误报（2016年之前）和该API的中断（2016年之后），对该数据的访问已被限制在某些关联中的点对点连接的那些人。在本文中，我们深入探讨了这个问题，从22个人的经验中得出见解，他们要么试图访问数据，要么在其创建中发挥了作用。通过这一点，我们希望在MIR社区中就访问特权发起更关键的对话和更深思熟虑的考虑。摘要：The equitable distribution of academic data is crucial for ensuring equal research opportunities, and ultimately further progress. Yet, due to the complexity of using the API for audio data that corresponds to the Million Song Dataset along with its misreporting (before 2016) and the discontinuation of this API (after 2016), access to this data has become restricted to those within certain affiliations that are connected peer-to-peer. In this paper, we delve into this issue, drawing insights from the experiences of 22 individuals who either attempted to access the data or played a role in its creation. With this, we hope to initiate more critical dialogue and more thoughtful consideration with regard to access privilege in the MIR community.
【9】 ReZero: Region-customizable Sound Extraction标题：ReZero：区域可定制的声音提取链接：https://arxiv.org/abs/2308.16892
作者：Rongzhi Gu,Yi Luo备注：13 pages, 11 figures摘要：我们介绍了区域可定制的声音提取（ReZero），一个通用的和灵活的框架多通道区域声音提取（R-SE）的任务。R-SE任务旨在提取所有活动目标声音（例如，人类语音），这不同于传统的和现有的任务，在传统的和现有的任务中，通常假设盲分离或固定的、预定义的空间区域。空间区域可以被定义为角度窗口、球体、圆锥体或其他几何图案。作为R-SE任务的解决方案，所提出的ReZero框架包括（1）不同类型的空间区域的定义，（2）用于区域特征提取和聚合的方法，以及（3）针对R-SE任务指定的带分裂RNN（BSRNN）模型的多通道扩展。我们设计不同的麦克风阵列的几何形状，不同类型的空间区域，并在不同的系统配置的全面消融研究的实验。模拟和真实记录的数据上的实验结果证明了ReZero的有效性。演示可在https://innerselfm.github.io/rezero/上查阅。摘要：We introduce region-customizable sound extraction (ReZero), a general and flexible framework for the multi-channel region-wise sound extraction (R-SE) task. R-SE task aims at extracting all active target sounds (e.g., human speech) within a specific, user-defined spatial region, which is different from conventional and existing tasks where a blind separation or a fixed, predefined spatial region are typically assumed. The spatial region can be defined as an angular window, a sphere, a cone, or other geometric patterns. Being a solution to the R-SE task, the proposed ReZero framework includes (1) definitions of different types of spatial regions, (2) methods for region feature extraction and aggregation, and (3) a multi-channel extension of the band-split RNN (BSRNN) model specified for the R-SE task. We design experiments for different microphone array geometries, different types of spatial regions, and comprehensive ablation studies on different system configurations. Experimental results on both simulated and real-recorded data demonstrate the effectiveness of ReZero. Demos are available at https://innerselfm.github.io/rezero/.
【10】 Time-Varying Quasi-Closed-Phase Analysis for Accurate Formant Tracking  in Speech Signals标题：语音信号中准确跟踪共振峰的时变准闭相分析链接：https://arxiv.org/abs/2308.16540
作者：Dhananjaya Gowda,Sudarsana Reddy Kadiri,Brad Story,Paavo Alku备注：None摘要：在本文中，我们提出了一种新的方法，准确估计和跟踪语音信号中的共振峰时变准闭相（TVQCP）分析。传统的共振峰跟踪方法通常采用两阶段估计和跟踪策略，其中使用短时分析（例如，10- 50 ms），随后是基于动态规划或线性状态空间模型的跟踪阶段。这些方法的主要缺点之一是跟踪阶段，无论它可能多么好，都不能改善第一阶段的共振峰估计精度。建议的TVQCP方法提供了一个单级共振峰跟踪相结合的估计和跟踪阶段成一个。TVQCP分析结合了三种方法来改进共振峰估计和跟踪：（1）它使用时间加权准闭相分析来导出声道的闭相估计，其中来自激励源的干扰减少，（2）它通过使用L_1优化来增加残余稀疏度，以及（3）它使用长时间窗口上的时变线性预测分析（例如，100- 200 ms），以在声道模型上并因此在共振峰轨迹上施加连续性约束。对各种合成和自然语音信号的共振峰跟踪实验表明，所提出的TVQCP方法的性能优于传统和流行的共振峰跟踪工具，例如Wavesurfer和Praat（基于动态规划），KARMA算法（基于卡尔曼滤波）和DeepFormants（基于以监督方式训练的深度神经网络）。所提出的方法的Matlab脚本可以在以下位置找到：https://github.com/njaygowda/ftrack摘要：In this paper, we propose a new method for the accurate estimation and tracking of formants in speech signals using time-varying quasi-closed-phase (TVQCP) analysis. Conventional formant tracking methods typically adopt a two-stage estimate-and-track strategy wherein an initial set of formant candidates are estimated using short-time analysis (e.g., 10--50 ms), followed by a tracking stage based on dynamic programming or a linear state-space model. One of the main disadvantages of these approaches is that the tracking stage, however good it may be, cannot improve upon the formant estimation accuracy of the first stage. The proposed TVQCP method provides a single-stage formant tracking that combines the estimation and tracking stages into one. TVQCP analysis combines three approaches to improve formant estimation and tracking: (1) it uses temporally weighted quasi-closed-phase analysis to derive closed-phase estimates of the vocal tract with reduced interference from the excitation source, (2) it increases the residual sparsity by using the $L_1$ optimization and (3) it uses time-varying linear prediction analysis over long time windows (e.g., 100--200 ms) to impose a continuity constraint on the vocal tract model and hence on the formant trajectories. Formant tracking experiments with a wide variety of synthetic and natural speech signals show that the proposed TVQCP method performs better than conventional and popular formant tracking tools, such as Wavesurfer and Praat (based on dynamic programming), the KARMA algorithm (based on Kalman filtering), and DeepFormants (based on deep neural networks trained in a supervised manner). Matlab scripts for the proposed method can be found at: https://github.com/njaygowda/ftrack
【11】 PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined  Keywords标题：PhonMatchNet：音素制导的用户自定义关键词零射关键词识别链接：https://arxiv.org/abs/2308.16511
作者：Yong-Hyeok Lee,Namhyun Cho机构：Speech AI Lab., NCSOFT Corporation, South Korea备注：None摘要：本研究提出了一种新的zero-shot用户自定义关键词识别模型，利用关键词的音频-音素关系，以提高性能。与以前的方法，估计在话语水平，我们使用话语和音素水平的信息。我们提出的方法包括一个双流语音编码器架构，基于自我注意的模式提取器，音素级检测损失的高性能在各种发音环境。实验结果的基础上，我们提出的模型优于基线模型，并实现了竞争力的性能相比，全镜头关键词发现模型。我们提出的模型显著提高了所有数据集的EER和AUC，包括熟悉的单词，专有名词和难以区分的发音，平均相对改善67%和80%，分别。我们提出的模型的实现代码可在https://github.com/ncsoft/PhonMatchNet获得。摘要：This study presents a novel zero-shot user-defined keyword spotting model that utilizes the audio-phoneme relationship of the keyword to improve performance. Unlike the previous approach that estimates at utterance level, we use both utterance and phoneme level information. Our proposed method comprises a two-stream speech encoder architecture, self-attention-based pattern extractor, and phoneme-level detection loss for high performance in various pronunciation environments. Based on experimental results, our proposed model outperforms the baseline model and achieves competitive performance compared with full-shot keyword spotting models. Our proposed model significantly improves the EER and AUC across all datasets, including familiar words, proper nouns, and indistinguishable pronunciations, with an average relative improvement of 67% and 80%, respectively. The implementation code of our proposed model is available at https://github.com/ncsoft/PhonMatchNet.
【12】 RAMP: Retrieval-Augmented MOS Prediction via Confidence-based Dynamic  Weighting标题：RAMP：基于置信度的动态加权检索增强MOS预测链接：https://arxiv.org/abs/2308.16488
作者：Hui Wang,Shiwan Zhao,Xiguang Zheng,Yong Qin机构：Nankai University, China, Kuaishou Technology, China备注：None摘要：自动平均意见得分（MOS）预测是评价合成语音的感知质量的关键。虽然最近使用预训练的自监督学习（SSL）模型的方法已经显示出有希望的结果，但它们仅部分地解决了特征提取器的数据稀缺问题。这使得解码器的数据稀缺问题未得到解决，并导致次优性能。为了解决这个挑战，我们提出了一个检索增强MOS预测方法，称为{\bf RAMP}，以提高解码器的能力，对数据稀缺的问题。融合网络也提出了动态调整检索范围为每个实例和融合权重的预测置信度的基础上。实验结果表明，我们提出的方法优于现有的方法在多种情况下。摘要：Automatic Mean Opinion Score (MOS) prediction is crucial to evaluate the perceptual quality of the synthetic speech. While recent approaches using pre-trained self-supervised learning (SSL) models have shown promising results, they only partly address the data scarcity issue for the feature extractor. This leaves the data scarcity issue for the decoder unresolved and leading to suboptimal performance. To address this challenge, we propose a retrieval-augmented MOS prediction method, dubbed {\bf RAMP}, to enhance the decoder's ability against the data scarcity issue. A fusing network is also proposed to dynamically adjust the retrieval scope for each instance and the fusion weights based on the predictive confidence. Experimental results show that our proposed method outperforms the existing methods in multiple scenarios.
【13】 Supervised Contrastive Learning with Nearest Neighbor Search for Speech  Emotion Recognition标题：最近邻搜索的有监督对比学习用于语音情感识别链接：https://arxiv.org/abs/2308.16485
作者：Xuechen Wang,Shiwan Zhao,Yong Qin机构：Nankai University, Tianjin, China备注：None摘要：语音情感识别是一项具有挑战性的任务，由于数据有限和某些情感的边界模糊。在本文中，我们提出了一个全面的方法来提高SER在整个模型生命周期中的性能，包括预训练，微调和推理阶段。为了解决数据稀缺问题，我们使用了一个预先训练的模型，wav2vec2.0。在微调过程中，我们提出了一种新的损失函数，结合交叉熵损失和监督对比学习损失，以提高模型的判别能力。这种方法增加了类间距离，减少了类内距离，减轻了模糊边界的问题。最后，为了利用改进的距离，我们提出了一种插值方法，在推理阶段，该方法结合了模型预测与k近邻模型的输出。IEMOCAP实验表明，我们提出的方法优于目前的最先进的结果。摘要：Speech Emotion Recognition (SER) is a challenging task due to limited data and blurred boundaries of certain emotions. In this paper, we present a comprehensive approach to improve the SER performance throughout the model lifecycle, including pre-training, fine-tuning, and inference stages. To address the data scarcity issue, we utilize a pre-trained model, wav2vec2.0. During fine-tuning, we propose a novel loss function that combines cross-entropy loss with supervised contrastive learning loss to improve the model's discriminative ability. This approach increases the inter-class distances and decreases the intra-class distances, mitigating the issue of blurred boundaries. Finally, to leverage the improved distances, we propose an interpolation method at the inference stage that combines the model prediction with the output from a k-nearest neighbors model. Our experiments on IEMOCAP demonstrate that our proposed methods outperform current state-of-the-art results.
【14】 OLISIA: a Cascade System for Spoken Dialogue State Tracking标题：OLISIA：一种级联语音对话状态跟踪系统链接：https://arxiv.org/abs/2304.11073
作者：Léo Jacqmin,Lucas Druart,Yannick Estève,Benoît Favre,Lina Maria Rojas-Barahona,Valentin Vielzeuf机构：Benoit Favre , Lina M. Rojas-Barahona , Orange Innovation, France, LIS - Aix-Marseille Université, France, LIA - Avignon Université, France摘要：虽然对话状态跟踪（DST）是一个核心组成部分，口语对话系统，最近的工作主要是处理聊天语料库，忽略口语和书面语之间的差异，在本文中，我们提出了OLISIA，级联系统，集成了自动语音识别（ASR）模型和DST模型。我们在ASR和DST模块中引入了一些调整，以提高口语对话的集成性和鲁棒性。通过这些调整，我们的系统在DSTC11 Track 3中排名第一，这是一个评估口语DST的基准。我们对结果进行了深入分析，发现通过数据增强来规范ASR输出和调整DST输入，以及增加预训练模型的大小，都在减少书面和口头对话之间的性能差异方面发挥了重要作用。摘要：Though Dialogue State Tracking (DST) is a core component of spoken dialogue systems, recent work on this task mostly deals with chat corpora, disregarding the discrepancies between spoken and written language.In this paper, we propose OLISIA, a cascade system which integrates an Automatic Speech Recognition (ASR) model and a DST model. We introduce several adaptations in the ASR and DST modules to improve integration and robustness to spoken conversations.With these adaptations, our system ranked first in DSTC11 Track 3, a benchmark to evaluate spoken DST. We conduct an in-depth analysis of the results and find that normalizing the ASR outputs and adapting the DST inputs through data augmentation, along with increasing the pre-trained models size all play an important role in reducing the performance discrepancy between written and spoken conversations.
eess.AS音频处理
【1】 ReZero: Region-customizable Sound Extraction标题：ReZero：区域可定制的声音提取链接：https://arxiv.org/abs/2308.16892
作者：Rongzhi Gu,Yi Luo备注：13 pages, 11 figures摘要：我们介绍了区域可定制的声音提取（ReZero），一个通用的和灵活的框架多通道区域声音提取（R-SE）的任务。R-SE任务旨在提取所有活动目标声音（例如，人类语音），这不同于传统的和现有的任务，在传统的和现有的任务中，通常假设盲分离或固定的、预定义的空间区域。空间区域可以被定义为角度窗口、球体、圆锥体或其他几何图案。作为R-SE任务的解决方案，所提出的ReZero框架包括（1）不同类型的空间区域的定义，（2）用于区域特征提取和聚合的方法，以及（3）针对R-SE任务指定的带分裂RNN（BSRNN）模型的多通道扩展。我们设计不同的麦克风阵列的几何形状，不同类型的空间区域，并在不同的系统配置的全面消融研究的实验。模拟和真实记录的数据上的实验结果证明了ReZero的有效性。演示可在https://innerselfm.github.io/rezero/上查阅。摘要：We introduce region-customizable sound extraction (ReZero), a general and flexible framework for the multi-channel region-wise sound extraction (R-SE) task. R-SE task aims at extracting all active target sounds (e.g., human speech) within a specific, user-defined spatial region, which is different from conventional and existing tasks where a blind separation or a fixed, predefined spatial region are typically assumed. The spatial region can be defined as an angular window, a sphere, a cone, or other geometric patterns. Being a solution to the R-SE task, the proposed ReZero framework includes (1) definitions of different types of spatial regions, (2) methods for region feature extraction and aggregation, and (3) a multi-channel extension of the band-split RNN (BSRNN) model specified for the R-SE task. We design experiments for different microphone array geometries, different types of spatial regions, and comprehensive ablation studies on different system configurations. Experimental results on both simulated and real-recorded data demonstrate the effectiveness of ReZero. Demos are available at https://innerselfm.github.io/rezero/.
【2】 Time-Varying Quasi-Closed-Phase Analysis for Accurate Formant Tracking  in Speech Signals标题：语音信号中准确跟踪共振峰的时变准闭相分析链接：https://arxiv.org/abs/2308.16540
作者：Dhananjaya Gowda,Sudarsana Reddy Kadiri,Brad Story,Paavo Alku备注：None摘要：在本文中，我们提出了一种新的方法，准确估计和跟踪语音信号中的共振峰时变准闭相（TVQCP）分析。传统的共振峰跟踪方法通常采用两阶段估计和跟踪策略，其中使用短时分析（例如，10- 50 ms），随后是基于动态规划或线性状态空间模型的跟踪阶段。这些方法的主要缺点之一是跟踪阶段，无论它可能多么好，都不能改善第一阶段的共振峰估计精度。建议的TVQCP方法提供了一个单级共振峰跟踪相结合的估计和跟踪阶段成一个。TVQCP分析结合了三种方法来改进共振峰估计和跟踪：（1）它使用时间加权准闭相分析来导出声道的闭相估计，其中来自激励源的干扰减少，（2）它通过使用L_1优化来增加残余稀疏度，以及（3）它使用长时间窗口上的时变线性预测分析（例如，100- 200 ms），以在声道模型上并因此在共振峰轨迹上施加连续性约束。对各种合成和自然语音信号的共振峰跟踪实验表明，所提出的TVQCP方法的性能优于传统和流行的共振峰跟踪工具，例如Wavesurfer和Praat（基于动态规划），KARMA算法（基于卡尔曼滤波）和DeepFormants（基于以监督方式训练的深度神经网络）。所提出的方法的Matlab脚本可以在以下位置找到：https://github.com/njaygowda/ftrack摘要：In this paper, we propose a new method for the accurate estimation and tracking of formants in speech signals using time-varying quasi-closed-phase (TVQCP) analysis. Conventional formant tracking methods typically adopt a two-stage estimate-and-track strategy wherein an initial set of formant candidates are estimated using short-time analysis (e.g., 10--50 ms), followed by a tracking stage based on dynamic programming or a linear state-space model. One of the main disadvantages of these approaches is that the tracking stage, however good it may be, cannot improve upon the formant estimation accuracy of the first stage. The proposed TVQCP method provides a single-stage formant tracking that combines the estimation and tracking stages into one. TVQCP analysis combines three approaches to improve formant estimation and tracking: (1) it uses temporally weighted quasi-closed-phase analysis to derive closed-phase estimates of the vocal tract with reduced interference from the excitation source, (2) it increases the residual sparsity by using the $L_1$ optimization and (3) it uses time-varying linear prediction analysis over long time windows (e.g., 100--200 ms) to impose a continuity constraint on the vocal tract model and hence on the formant trajectories. Formant tracking experiments with a wide variety of synthetic and natural speech signals show that the proposed TVQCP method performs better than conventional and popular formant tracking tools, such as Wavesurfer and Praat (based on dynamic programming), the KARMA algorithm (based on Kalman filtering), and DeepFormants (based on deep neural networks trained in a supervised manner). Matlab scripts for the proposed method can be found at: https://github.com/njaygowda/ftrack
【3】 PhonMatchNet: Phoneme-Guided Zero-Shot Keyword Spotting for User-Defined  Keywords标题：PhonMatchNet：音素制导的用户自定义关键词零射关键词识别链接：https://arxiv.org/abs/2308.16511
作者：Yong-Hyeok Lee,Namhyun Cho机构：Speech AI Lab., NCSOFT Corporation, South Korea备注：None摘要：本研究提出了一种新的zero-shot用户自定义关键词识别模型，利用关键词的音频-音素关系，以提高性能。与以前的方法，估计在话语水平，我们使用话语和音素水平的信息。我们提出的方法包括一个双流语音编码器架构，基于自我注意的模式提取器，音素级检测损失的高性能在各种发音环境。实验结果的基础上，我们提出的模型优于基线模型，并实现了竞争力的性能相比，全镜头关键词发现模型。我们提出的模型显著提高了所有数据集的EER和AUC，包括熟悉的单词，专有名词和难以区分的发音，平均相对改善67%和80%，分别。我们提出的模型的实现代码可在https://github.com/ncsoft/PhonMatchNet获得。摘要：This study presents a novel zero-shot user-defined keyword spotting model that utilizes the audio-phoneme relationship of the keyword to improve performance. Unlike the previous approach that estimates at utterance level, we use both utterance and phoneme level information. Our proposed method comprises a two-stream speech encoder architecture, self-attention-based pattern extractor, and phoneme-level detection loss for high performance in various pronunciation environments. Based on experimental results, our proposed model outperforms the baseline model and achieves competitive performance compared with full-shot keyword spotting models. Our proposed model significantly improves the EER and AUC across all datasets, including familiar words, proper nouns, and indistinguishable pronunciations, with an average relative improvement of 67% and 80%, respectively. The implementation code of our proposed model is available at https://github.com/ncsoft/PhonMatchNet.
【4】 RAMP: Retrieval-Augmented MOS Prediction via Confidence-based Dynamic  Weighting标题：RAMP：基于置信度的动态加权检索增强MOS预测链接：https://arxiv.org/abs/2308.16488
作者：Hui Wang,Shiwan Zhao,Xiguang Zheng,Yong Qin机构：Nankai University, China, Kuaishou Technology, China备注：None摘要：自动平均意见得分（MOS）预测是评价合成语音的感知质量的关键。虽然最近使用预训练的自监督学习（SSL）模型的方法已经显示出有希望的结果，但它们仅部分地解决了特征提取器的数据稀缺问题。这使得解码器的数据稀缺问题未得到解决，并导致次优性能。为了解决这个挑战，我们提出了一个检索增强MOS预测方法，称为{\bf RAMP}，以提高解码器的能力，对数据稀缺的问题。融合网络也提出了动态调整检索范围为每个实例和融合权重的预测置信度的基础上。实验结果表明，我们提出的方法优于现有的方法在多种情况下。摘要：Automatic Mean Opinion Score (MOS) prediction is crucial to evaluate the perceptual quality of the synthetic speech. While recent approaches using pre-trained self-supervised learning (SSL) models have shown promising results, they only partly address the data scarcity issue for the feature extractor. This leaves the data scarcity issue for the decoder unresolved and leading to suboptimal performance. To address this challenge, we propose a retrieval-augmented MOS prediction method, dubbed {\bf RAMP}, to enhance the decoder's ability against the data scarcity issue. A fusing network is also proposed to dynamically adjust the retrieval scope for each instance and the fusion weights based on the predictive confidence. Experimental results show that our proposed method outperforms the existing methods in multiple scenarios.
【5】 Supervised Contrastive Learning with Nearest Neighbor Search for Speech  Emotion Recognition标题：最近邻搜索的有监督对比学习用于语音情感识别链接：https://arxiv.org/abs/2308.16485
作者：Xuechen Wang,Shiwan Zhao,Yong Qin机构：Nankai University, Tianjin, China备注：None摘要：语音情感识别是一项具有挑战性的任务，由于数据有限和某些情感的边界模糊。在本文中，我们提出了一个全面的方法来提高SER在整个模型生命周期中的性能，包括预训练，微调和推理阶段。为了解决数据稀缺问题，我们使用了一个预先训练的模型，wav2vec2.0。在微调过程中，我们提出了一种新的损失函数，结合交叉熵损失和监督对比学习损失，以提高模型的判别能力。这种方法增加了类间距离，减少了类内距离，减轻了模糊边界的问题。最后，为了利用改进的距离，我们提出了一种插值方法，在推理阶段，该方法结合了模型预测与k近邻模型的输出。IEMOCAP实验表明，我们提出的方法优于目前的最先进的结果。摘要：Speech Emotion Recognition (SER) is a challenging task due to limited data and blurred boundaries of certain emotions. In this paper, we present a comprehensive approach to improve the SER performance throughout the model lifecycle, including pre-training, fine-tuning, and inference stages. To address the data scarcity issue, we utilize a pre-trained model, wav2vec2.0. During fine-tuning, we propose a novel loss function that combines cross-entropy loss with supervised contrastive learning loss to improve the model's discriminative ability. This approach increases the inter-class distances and decreases the intra-class distances, mitigating the issue of blurred boundaries. Finally, to leverage the improved distances, we propose an interpolation method at the inference stage that combines the model prediction with the output from a k-nearest neighbors model. Our experiments on IEMOCAP demonstrate that our proposed methods outperform current state-of-the-art results.
【6】 Multi-Transfer Learning Techniques for Detecting Auditory Brainstem  Response标题：用于检测听觉脑干反应的多迁移学习技术链接：https://arxiv.org/abs/2308.16203
作者：Fatih Ozyurt,Jafar Majidpour,Tarik A. Rashid,Amir Majidpour,Canan Koc机构：Department of Software Engineering, Firat University, Elazig, Turkey, Department of Computer Science, University of Raparin, Rania, Iraq., Computer Science and Engineering Department, University of Kurdistan Hewlêr, Erbil, Iraq.备注：None摘要：通过听觉脑干反应（ABR）测试来评估经历听力损伤的个体中的外周听觉神经系统的健康状况。听力学家评估和记录ABR测试的结果。他们解释发现，并使用基于参考的标记物，如峰值潜伏期、波形形态、振幅和其他相关因素为其分配标签。ABR测试的不准确评估可能导致对听觉神经系统完整性的错误判断;因此，正确的听力损失（HL）诊断和分析是必要的。为了识别和评估ABR自动化，同时降低人为错误的可能性，机器学习方法，特别是深度学习，可能是一个合适的选择。为了解决这些问题，本研究提出了使用迁移学习（TL）方法的深度学习模型，从ABR测试中提取特征，并使用支持向量机（SVM）诊断HL。预先训练的卷积神经网络（CNN）架构，如AlexNet，DenseNet，GoogleNet，InceptionResNetV 2，InceptionV 3，MobileNetV 2，NASNetMobile，ResNet 18，ResNet 50，ResNet 101，ShuffleNet，和SqueezeNet被用于从所提出的模型中收集的ABR报告图像数据集提取特征。它已经决定使用六个措施的准确性，精度，召回，几何平均值（GM），标准差（SD），和面积下的ROC曲线来衡量所提出的模型的有效性。根据实验结果，ShuffleNet和ResNet 50模型的TL对于ABR使用SVM分类器诊断HL是有效的，当使用5重交叉验证方法时，具有95%的高准确率。摘要：The assessment of the well-being of the peripheral auditory nerve system in individuals experiencing hearing impairment is conducted through auditory brainstem response (ABR) testing. Audiologists assess and document the results of the ABR test. They interpret the findings and assign labels to them using reference-based markers like peak latency, waveform morphology, amplitude, and other relevant factors. Inaccurate assessment of ABR tests may lead to incorrect judgments regarding the integrity of the auditory nerve system; therefore, proper Hearing Loss (HL) diagnosis and analysis are essential. To identify and assess ABR automation while decreasing the possibility of human error, machine learning methods, notably deep learning, may be an appropriate option. To address these issues, this study proposed deep-learning models using the transfer-learning (TL) approach to extract features from ABR testing and diagnose HL using support vector machines (SVM). Pre-trained convolutional neural network (CNN) architectures like AlexNet, DenseNet, GoogleNet, InceptionResNetV2, InceptionV3, MobileNetV2, NASNetMobile, ResNet18, ResNet50, ResNet101, ShuffleNet, and SqueezeNet are used to extract features from the collected ABR reported images dataset in the proposed model. It has been decided to use six measures accuracy, precision, recall, geometric mean (GM), standard deviation (SD), and area under the ROC curve to measure the effectiveness of the proposed model. According to experimental findings, the ShuffleNet and ResNet50 models' TL is effective for ABR to diagnose HL using an SVM classifier, with a high accuracy rate of 95% when using the 5-fold cross-validation method.
【7】 Towards Improving the Expressiveness of Singing Voice Synthesis with  BERT Derived Semantic Information标题：利用BERT导出的语义信息提高歌唱声音合成的表现力链接：https://arxiv.org/abs/2308.16836
作者：Shaohuan Zhou,Shun Lei,Weiya You,Deyi Tuo,Yuren You,Zhiyong Wu,Shiyin Kang,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China,  Huya Inc., Guangzhou, China,  The Chinese University of Hong Kong, Hong Kong SAR, China,  XVerse Inc., Shenzhen, China摘要：本文提出了一种端到端的高质量歌唱语音合成系统，该系统使用双向编码器表示从Transformers（BERT）派生的语义嵌入，以提高合成歌唱语音的表现力。基于最近提出的VISinger的主要架构，我们提出了几个具体的设计，以表现力的歌唱声音合成。首先，与以前的SVS模型不同，我们使用从预先训练的BERT提取的歌词的文本表示作为模型的额外输入。该表示包含歌词语义信息，有助于SVS系统产生更有表现力和自然的声音。其次，我们进一步引入了一个能量预测器来稳定合成的声音和模型的更广泛的能量变化，也有助于歌唱的声音的表现力。最后但并非最不重要的是，为了减弱跑调问题，重新设计音高预测器以预测真实音符音高比。客观和主观实验结果表明，所提出的SVS系统可以产生更高质量的歌唱声音比VISinger。摘要：This paper presents an end-to-end high-quality singing voice synthesis (SVS) system that uses bidirectional encoder representation from Transformers (BERT) derived semantic embeddings to improve the expressiveness of the synthesized singing voice. Based on the main architecture of recently proposed VISinger, we put forward several specific designs for expressive singing voice synthesis. First, different from the previous SVS models, we use text representation of lyrics extracted from pre-trained BERT as additional input to the model. The representation contains information about semantics of the lyrics, which could help SVS system produce more expressive and natural voice. Second, we further introduce an energy predictor to stabilize the synthesized voice and model the wider range of energy variations that also contribute to the expressiveness of singing voice. Last but not the least, to attenuate the off-key issues, the pitch predictor is re-designed to predict the real to note pitch ratio. Both objective and subjective experimental results indicate that the proposed SVS system can produce singing voice with higher-quality outperforming VISinger.
【8】 SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language  Models标题：SpeechTokenizer：适用于语音大语言模型的统一语音标记器链接：https://arxiv.org/abs/2308.16692
作者：Xin Zhang,Dong Zhang,Shimin Li,Yaqian Zhou,Xipeng Qiu机构：School of Computer Science, Fudan University, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University备注：SpeechTokenizer project page is this https URL摘要：当前的语音大语言模型建立在离散的语音表示上，其可以被分类为语义令牌和声学令牌。然而，现有的语音令牌不是专门为语音语言建模而设计的。为了评估语音标记用于构建语音语言模型的适用性，我们建立了第一个基准测试SLMTokBench。我们的研究结果表明，无论是语义还是声学令牌是理想的为此目的。因此，我们提出了SpeechTokenizer，一个统一的语音标记的语音大语言模型。SpeechTokenizer采用具有残差矢量量化（RVQ）的编码器-解码器架构。统一语义和声学令牌，SpeechTokenizer在不同的RVQ层上分层地解开语音信息的不同方面。此外，我们构建了一个统一的语音语言模型（USLM）利用SpeechTokenizer。实验表明，SpeechTokenizer在语音重建方面与EnCodec表现相当，并在SLMTokBench基准测试中表现出强大的性能。此外，USLM优于VALL-E在zero-shot文本到语音的任务。代码和型号可在https://github.com/ZhangXInFD/SpeechTokenizer/获得。摘要：Current speech large language models build upon discrete speech representations, which can be categorized into semantic tokens and acoustic tokens. However, existing speech tokens are not specifically designed for speech language modeling. To assess the suitability of speech tokens for building speech language models, we established the first benchmark, SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are ideal for this purpose. Therefore, we propose SpeechTokenizer, a unified speech tokenizer for speech large language models. SpeechTokenizer adopts the Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of speech information hierarchically across different RVQ layers. Furthermore, We construct a Unified Speech Language Model (USLM) leveraging SpeechTokenizer. Experiments show that SpeechTokenizer performs comparably to EnCodec in speech reconstruction and demonstrates strong performance on the SLMTokBench benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks. Code and models are available at https://github.com/ZhangXInFD/SpeechTokenizer/.
【9】 Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting标题：动态nsNet2：提前退出的高效深度噪声抑制链接：https://arxiv.org/abs/2308.16678
作者：Riccardo Miccini,Alaa Zniber,Clément Laroche,Tobias Piechowiak,Martin Schoeberl,Luca Pezzarossa,Ouassim Karrakchou,Jens Sparsø,Mounir Ghogho机构：Cl´ement Laroche⋆, ⋆ GN Audio, † Technical University of Denmark, § International University of Rabat备注：Accepted at the MLSP 2023摘要：尽管深度学习在深度噪声抑制领域取得了长足的进步，但在资源受限的设备上利用深度架构仍然具有挑战性。因此，我们提出了一个基于nsNet2的早期退出模型，该模型通过在不同阶段停止计算，提供了几个级别的准确性和资源节约。此外，我们适应原来的架构，通过分裂的信息流，以考虑到注入的动态。我们表现出的性能和计算复杂性之间的权衡建立的指标的基础上。摘要：Although deep learning has made strides in the field of deep noise suppression, leveraging deep architectures on resource-constrained devices still proved challenging. Therefore, we present an early-exiting model based on nsNet2 that provides several levels of accuracy and resource savings by halting computations at different stages. Moreover, we adapt the original architecture by splitting the information flow to take into account the injected dynamism. We show the trade-offs between performance and computational complexity based on established metrics.
【10】 Towards Spontaneous Style Modeling with Semi-supervised Pre-training for  Conversational Text-to-Speech Synthesis标题：会话文语合成中半监督预训练的自发风格建模链接：https://arxiv.org/abs/2308.16593
作者：Weiqin Li,Shun Lei,Qiaochu Huang,Yixuan Zhou,Zhiyong Wu,Shiyin Kang,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, XVerse Inc., Shenzhen, China, The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by INTERSPEECH 2023摘要：对话中经常出现的自发行为使演讲比阅读方式更人性化。然而，由于缺乏高质量的自发数据集和标记自发行为的高成本，合成自发风格的语音具有挑战性。在本文中，我们提出了一种半监督预训练方法，以增加自发风格的语音和自发行为标签的数量。在半监督学习的过程中，文本和语音信息被认为是用于检测语音中的自发行为标签。此外，语言感知编码器被用来模拟会话中每个句子之间的关系。实验结果表明，我们所提出的方法实现了卓越的表达语音合成性能的能力，自发行为的自发风格的语音模型和预测合理的自发行为从文本。摘要：The spontaneous behavior that often occurs in conversations makes speech more human-like compared to reading-style. However, synthesizing spontaneous-style speech is challenging due to the lack of high-quality spontaneous datasets and the high cost of labeling spontaneous behavior. In this paper, we propose a semi-supervised pre-training method to increase the amount of spontaneous-style speech and spontaneous behavioral labels. In the process of semi-supervised learning, both text and speech information are considered for detecting spontaneous behaviors labels in speech. Moreover, a linguistic-aware encoder is used to model the relationship between each sentence in the conversation. Experimental results indicate that our proposed method achieves superior expressive speech synthesis performance with the ability to model spontaneous behavior in spontaneous-style speech and predict reasonable spontaneous behavior from text.
【11】 Improving Mandarin Prosodic Structure Prediction with Multi-level  Contextual Information标题：利用多层上下文信息改进汉语韵律结构预测链接：https://arxiv.org/abs/2308.16577
作者：Jie Chen,Changhe Song,Deyi Tuo,Xixin Wu,Shiyin Kang,Zhiyong Wu,Helen Meng机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China,  XVerse Inc., Shenzhen, China,  Huya Inc., Guangzhou, China,  The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by Interspeech2022摘要：在文本-语音转换（TTS）合成中，韵律结构预测（PSP）在生成自然、可理解的语音中起着重要作用。虽然语间语言信息会影响目标话语的语音理解，但以往的PSP研究主要集中在利用当前话语的语内语言信息。本工作提出利用话语间的语言信息来提高PSP的性能。分层编码器从输入文本的字符层、话语层和话语层提取多层次的语境信息，包括话语间和话语内的语言信息。然后，多任务学习（MTL）解码器预测韵律边界的多级上下文信息。两个数据集的客观评价结果表明，我们的方法取得了更好的预测韵律词（PW），韵律短语（PPH）和语调短语（IPH）的F1得分。它证明了使用多级上下文信息的PSP的有效性。主观偏好测试也表明合成语音的自然度得到了改善。摘要：For text-to-speech (TTS) synthesis, prosodic structure prediction (PSP) plays an important role in producing natural and intelligible speech. Although inter-utterance linguistic information can influence the speech interpretation of the target utterance, previous works on PSP mainly focus on utilizing intrautterance linguistic information of the current utterance only. This work proposes to use inter-utterance linguistic information to improve the performance of PSP. Multi-level contextual information, which includes both inter-utterance and intrautterance linguistic information, is extracted by a hierarchical encoder from character level, utterance level and discourse level of the input text. Then a multi-task learning (MTL) decoder predicts prosodic boundaries from multi-level contextual information. Objective evaluation results on two datasets show that our method achieves better F1 scores in predicting prosodic word (PW), prosodic phrase (PPH) and intonational phrase (IPH). It demonstrates the effectiveness of using multi-level contextual information for PSP. Subjective preference tests also indicate the naturalness of synthesized speeches are improved.
【12】 LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech标题：LightGrad：轻量级文语转换扩散概率模型链接：https://arxiv.org/abs/2308.16569
作者：Jie Chen,Xingchen Song,Zhendong Peng,Binbin Zhang,Fuping Pan,Zhiyong Wu机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China,  Horizon Robotics, Beijing, China,  WeNet Open Source Community,  The Chinese University of Hong Kong, Hong Kong SAR, China备注：Accepted by ICASSP 2023摘要：神经文本到语音（TTS）模型的最新进展将成千上万的TTS应用带入日常生活中，其中模型被部署在云中以向海关提供服务。在这些模型中，扩散概率模型（DPMs），它可以稳定地训练，并且与其他生成模型相比，参数效率更高。由于在海关和云之间传输数据会带来高延迟和暴露私有数据的风险，因此在边缘设备上部署TTS模型是首选。当在边缘设备上实现DPM时，存在两个实际问题。首先，当前的DPM对于资源受限的设备来说不够轻量。其次，DPM在推理中需要许多去噪步骤，这增加了延迟。在这项工作中，我们提出了LightGrad，一个轻量级的DPM TTS。LightGrad配备了轻量级的U-Net扩散解码器和免训练快速采样技术，减少了模型参数和推理延迟。流式推理也在LightGrad中实现，以进一步减少延迟。与Grad-TTS相比，LightGrad在参数上减少了62.2%，延迟减少了65.7%，同时在4个去噪步骤中保持了汉语普通话和英语相当的语音质量。摘要：Recent advances in neural text-to-speech (TTS) models bring thousands of TTS applications into daily life, where models are deployed in cloud to provide services for customs. Among these models are diffusion probabilistic models (DPMs), which can be stably trained and are more parameter-efficient compared with other generative models. As transmitting data between customs and the cloud introduces high latency and the risk of exposing private data, deploying TTS models on edge devices is preferred. When implementing DPMs onto edge devices, there are two practical problems. First, current DPMs are not lightweight enough for resource-constrained devices. Second, DPMs require many denoising steps in inference, which increases latency. In this work, we present LightGrad, a lightweight DPM for TTS. LightGrad is equipped with a lightweight U-Net diffusion decoder and a training-free fast sampling technique, reducing both model parameters and inference latency. Streaming inference is also implemented in LightGrad to reduce latency further. Compared with Grad-TTS, LightGrad achieves 62.2% reduction in paramters, 65.7% reduction in latency, while preserving comparable speech quality on both Chinese Mandarin and English in 4 denoising steps.
【13】 Sequential Pitch Distributions for Raga Detection标题：序列基音分布在RAGA检测中的应用链接：https://arxiv.org/abs/2308.16421
作者：Vishwaas Narasinh,Senthil Raja G机构：Published on: Aug备注：None摘要：Raga是印度艺术音乐（IAM）中的一个基本旋律概念。它的特点是复杂的模式。所有的表演和组成都是基于raga框架。在音乐信息检索领域，拉加和声调检测一直是一个长期的研究课题。在本文中，我们试图使用一种新的功能，从音频样本中提取序列或时间信息来检测的拉加。我们称这些为顺序音高分布（SPD），其是随时间在两个给定音高值之间的音高值上取得的分布。我们还实现了国家的最先进的结果，在印度斯坦语和卡尔纳蒂克音乐拉加数据集的准确性分别为99%和88.13%。SPD提供了一个很大的提高准确性超过一个标准的间距分布。然而，本文的主要目标是提出一种替代方法来建模的旋律的时间方面，从而推导出的拉加。摘要：Raga is a fundamental melodic concept in Indian Art Music (IAM). It is characterized by complex patterns. All performances and compositions are based on the raga framework. Raga and tonic detection have been a long-standing research problem in the field of Music Information Retrieval. In this paper, we attempt to detect the raga using a novel feature to extract sequential or temporal information from an audio sample. We call these Sequential Pitch Distributions (SPD), which are distributions taken over pitch values between two given pitch values over time. We also achieve state-of-the-art results on both Hindustani and Carnatic music raga data sets with an accuracy of 99% and 88.13%, respectively. SPD gives a great boost in accuracy over a standard pitch distribution. The main goal of this paper, however, is to present an alternative approach to modeling the temporal aspects of the melody and thereby deducing the raga.
【14】 Knowledge Distillation from Non-streaming to Streaming ASR Encoder using  Auxiliary Non-streaming Layer标题：基于辅助非流层的非流媒体到流媒体ASR编码器的知识提取链接：https://arxiv.org/abs/2308.16415
作者：Kyuhong Shim,Jinkyu Lee,Simyung Chang,Kyuwoong Hwang机构：Qualcomm AI Research†, Qualcomm Korea YH, Seoul, Republic of Korea备注：Accepted to Interspeech 2023摘要：流式自动语音识别（ASR）模型被限制访问未来上下文，这导致与非流式模型相比性能更差。为了提高流ASR的性能，从非流模型到流模型的知识蒸馏（KD）进行了研究，主要集中在对齐输出令牌概率。在本文中，我们提出了一个从教师编码器到学生编码器的层到层KD。为了确保使用相同的上下文提取特征，我们将辅助非流式分支插入到学生，并执行从非流式教师层到非流式辅助层的KD。我们设计了一个特殊的KD损失，利用自回归预测编码（APC）机制，以鼓励流模型预测看不见的未来上下文。实验结果表明，与以往的令牌概率提取方法相比，该方法能显著降低误词率。摘要：Streaming automatic speech recognition (ASR) models are restricted from accessing future context, which results in worse performance compared to the non-streaming models. To improve the performance of streaming ASR, knowledge distillation (KD) from the non-streaming to streaming model has been studied, mainly focusing on aligning the output token probabilities. In this paper, we propose a layer-to-layer KD from the teacher encoder to the student encoder. To ensure that features are extracted using the same context, we insert auxiliary non-streaming branches to the student and perform KD from the non-streaming teacher layer to the non-streaming auxiliary layer. We design a special KD loss that leverages the autoregressive predictive coding (APC) mechanism to encourage the streaming model to predict unseen future contexts. Experimental results show that the proposed method can significantly reduce the word error rate compared to previous token probability distillation methods.
【15】 The Biased Journey of MSD_AUDIO.ZIP标题：MSD_AUDIO.ZIP的偏颇之旅链接：https://arxiv.org/abs/2308.16389
作者：Haven Kim,Keunwoo Choi,Mateusz Modrzejewski,Cynthia C. S. Liem机构：Graduate School of Culture Technology, KAIST, South Korea,  Gaudio Lab, Inc., Seoul, South Korea  Prescient Design, New York, USA,  Institute of Computer Science, Warsaw University of Technology, Poland摘要：学术数据的公平分配对于确保平等的研究机会以及最终取得进一步进展至关重要。然而，由于将API用于对应于百万歌曲数据集的音频数据的复杂性以及其误报（2016年之前）和该API的中断（2016年之后），对该数据的访问已被限制在某些关联中的点对点连接的那些人。在本文中，我们深入探讨了这个问题，从22个人的经验中得出见解，他们要么试图访问数据，要么在其创建中发挥了作用。通过这一点，我们希望在MIR社区中就访问特权发起更关键的对话和更深思熟虑的考虑。摘要：The equitable distribution of academic data is crucial for ensuring equal research opportunities, and ultimately further progress. Yet, due to the complexity of using the API for audio data that corresponds to the Million Song Dataset along with its misreporting (before 2016) and the discontinuation of this API (after 2016), access to this data has become restricted to those within certain affiliations that are connected peer-to-peer. In this paper, we delve into this issue, drawing insights from the experiences of 22 individuals who either attempted to access the data or played a role in its creation. With this, we hope to initiate more critical dialogue and more thoughtful consideration with regard to access privilege in the MIR community.
机器翻译，仅供参考