今日论文合集：cs.SD语音8篇，eess.AS音频处理8篇。【1】Multichannel blind speech source separation with a disjoint constraint source model
标题：基于不相交约束源模型的多声道语音盲分离链接：https://arxiv.org/abs/2401.01763
作者：Jianyu Wang,Shanzheng Guan摘要：多通道卷积盲语音源分离是指在没有混合系统先验信息的情况下，从观测到的多通道混合信号中分离出不同的语音源。多通道非负矩阵分解（MNMF）已被证明是最强大的分离框架之一，其代表性算法如MNMF和独立低秩矩阵分析（ILRMA）已显示出良好的性能。然而，语音源信号的稀疏特性没有充分考虑在这样的框架。众所周知，语音信号是稀疏的，这是在这项工作中考虑，以提高分离性能。具体来说，我们利用宾厄姆和拉普拉斯分布制定一个不相交的约束正则化，随后纳入MNMF和ILRMA。然后，我们推导出优化最小化规则更新相关的源模型的参数，从而在两个增强算法的发展：S-MNMF和S-ILRMA。进行了全面的模拟，结果明确地证明了我们提出的方法的有效性。摘要：Multichannel convolutive blind speech source separation refers to the problem of separating different speech sources from the observed multichannel mixtures without much a priori information about the mixing system. Multichannel nonnegative matrix factorization (MNMF) has been proven to be one of the most powerful separation frameworks and the representative algorithms such as MNMF and the independent low-rank matrix analysis (ILRMA) have demonstrated great performance. However, the sparseness properties of speech source signals are not fully taken into account in such a framework. It is well known that speech signals are sparse in nature, which is considered in this work to improve the separation performance. Specifically, we utilize the Bingham and Laplace distributions to formulate a disjoint constraint regularizer, which is subsequently incorporated into both MNMF and ILRMA. We then derive majorization-minimization rules for updating parameters related to the source model, resulting in the development of two enhanced algorithms: s-MNMF and s-ILRMA. Comprehensive simulations are conducted, and the results unequivocally demonstrate the efficacy of our proposed methodologies.
【2】 Independent low-rank matrix analysis based on the Sinkhorn divergence source model for blind source separation标题：基于Sinkhorn散射源模型的独立低阶矩阵盲源分离链接：https://arxiv.org/abs/2401.01762作者：Jianyu Wang,Shanzheng Guan,Jingdong Chen,Jacob Benesty摘要：独立低秩矩阵分析（ILRMA）在解决语音信号盲分离问题上有着巨大的潜力。该方法假设不同频段的谱是独立的，且任意频段的谱系数服从高斯分布。然后采用Itakura-Saito散度估计源模型相关参数。然而，在现实中，来自不同频带的谱系数可能是相关的，这在现有的ILRMA算法中是没有考虑的。本文提出了一种改进的ILRMA算法，该算法考虑了不同频带的谱系数之间的相关性。然后利用Sinkhorn发散来优化源模型参数。由于使用了跨带信息，BSS性能得到了改善。但是要估计的参数的数量也显著增加，并且计算复杂度也显著增加。为了降低算法的复杂度，我们应用克罗内克积将建模矩阵分解为许多维数更小的矩阵的乘积。然后，一个有效的算法来实现基于Sinkhorn发散的BSS算法和复杂度降低了一个数量级。摘要：The so-called independent low-rank matrix analysis (ILRMA) has demonstrated a great potential for dealing with the problem of determined blind source separation (BSS) for audio and speech signals. This method assumes that the spectra from different frequency bands are independent and the spectral coefficients in any frequency band are Gaussian distributed. The Itakura-Saito divergence is then employed to estimate the source model related parameters. In reality, however, the spectral coefficients from different frequency bands may be dependent, which is not considered in the existing ILRMA algorithm. This paper presents an improved version of ILRMA, which considers the dependency between the spectral coefficients from different frequency bands. The Sinkhorn divergence is then exploited to optimize the source model parameters. As a result of using the cross-band information, the BSS performance is improved. But the number of parameters to be estimated also increases significantly, and so is the computational complexity. To reduce the algorithm complexity, we apply the Kronecker product to decompose the modeling matrix into the product of a number of matrices of much smaller dimensionality. An efficient algorithm is then developed to implement the Sinkhorn divergence based BSS algorithm and the complexity is reduced by an order of magnitude.【3】 Incremental FastPitch: Chunk-based High Quality Text to Speech标题：增量式快速音调：基于块的高质量文语转换链接：https://arxiv.org/abs/2401.01755作者：Muyang Du,Chuan Liu,Junjie Lai备注：5 pages, 4 figures, 1 table摘要：并行文语转换模型已被广泛应用于实时语音合成中，与传统的自回归模型相比，它们具有更好的可控性和更快的合成过程。虽然并行模型在许多方面具有优势，但由于其完全并行的体系结构（如Transformer），它们自然不适合增量综合。在这项工作中，我们提出了增量FastPitch，一种新的FastPitch变体，能够通过改进基于块的FFT块的架构，使用接收域约束的块注意力掩码进行训练，以及使用固定大小的过去模型状态进行推理，来增量地产生高质量的Mel块。实验结果表明，我们的建议可以产生的语音质量媲美并行FastPitch，具有显着较低的延迟，允许更低的响应时间为实时语音应用程序。摘要：Parallel text-to-speech models have been widely applied for real-time speech synthesis, and they offer more controllability and a much faster synthesis process compared with conventional auto-regressive models. Although parallel models have benefits in many aspects, they become naturally unfit for incremental synthesis due to their fully parallel architecture such as transformer. In this work, we propose Incremental FastPitch, a novel FastPitch variant capable of incrementally producing high-quality Mel chunks by improving the architecture with chunk-based FFT blocks, training with receptive-field constrained chunk attention masks, and inference with fixed size past model states. Experimental results show that our proposal can produce speech quality comparable to the parallel FastPitch, with a significant lower latency that allows even lower response time for real-time speech applications.【4】 Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models标题：神经自动语音识别中的幻觉：识别错误和幻觉模型链接：https://arxiv.org/abs/2401.01572作者：Rita Frieske,Bertram E. Shi摘要：幻觉是深度神经网络产生的一种输出错误。虽然这在自然语言处理中已经研究过，但在自动语音识别中以前没有研究过。在这里，我们将ASR中的幻觉定义为由与源话语语义无关但仍然流畅和连贯的模型生成的transmittance。幻觉与模型的可能自然语言输出的相似性产生了欺骗的危险，并影响了系统的可信度。我们发现，常用的指标，如单词错误率，不能区分幻觉和非幻觉模型。为了解决这个问题，我们提出了一种基于扰动的方法，用于评估自动语音识别（ASR）模型在测试时对幻觉的敏感性，该方法不需要访问训练数据集。我们证明，这种方法有助于区分具有相似基线单词错误率的幻觉和非幻觉模型。我们进一步探索ASR错误类型与数据集噪声类型之间的关系，以确定哪些类型的噪声最有可能产生幻觉输出。我们设计了一个框架，通过分析他们的语义连接与地面真相和他们的流畅性识别幻觉。最后，我们发现如何诱导幻觉与随机噪声注入的话语。摘要：Hallucinations are a type of output error produced by deep neural networks. While this has been studied in natural language processing, they have not been researched previously in automatic speech recognition. Here, we define hallucinations in ASR as transcriptions generated by a model that are semantically unrelated to the source utterance, yet still fluent and coherent. The similarity of hallucinations to probable natural language outputs of the model creates a danger of deception and impacts the credibility of the system. We show that commonly used metrics, such as word error rates, cannot differentiate between hallucinatory and non-hallucinatory models. To address this, we propose a perturbation-based method for assessing the susceptibility of an automatic speech recognition (ASR) model to hallucination at test time, which does not require access to the training dataset. We demonstrate that this method helps to distinguish between hallucinatory and non-hallucinatory models that have similar baseline word error rates. We further explore the relationship between the types of ASR errors and the types of dataset noise to determine what types of noise are most likely to create hallucinatory outputs. We devise a framework for identifying hallucinations by analysing their semantic connection with the ground truth and their fluency. Finally, we discover how to induce hallucinations with a random noise injection to the utterance.【5】 Exploring Multi-Modal Control in Music-Driven Dance Generation标题：音乐驱动的舞蹈生成中的多模式控制探索链接：https://arxiv.org/abs/2401.01382作者：Ronghui Li,Yuqin Dai,Yachao Zhang,Jun Li,Jian Yang,Jie Guo,Xiu Li摘要：现有的音乐驱动的三维舞蹈生成方法主要集中在高质量的舞蹈生成上，但在生成过程中缺乏足够的控制。为了解决这些问题，我们提出了一个统一的框架，能够生成高质量的舞蹈动作，并支持多模态控制，包括体裁控制，语义控制和空间控制。首先，我们将舞蹈生成网络与舞蹈控制网络解耦，从而避免了添加额外控制信息时舞蹈质量的下降。其次，针对不同的控制信息设计了具体的控制策略，并将其整合到一个统一的框架中。实验结果表明，所提出的舞蹈生成框架优于国家的最先进的方法在运动质量和可控性。摘要：Existing music-driven 3D dance generation methods mainly concentrate on high-quality dance generation, but lack sufficient control during the generation process. To address these issues, we propose a unified framework capable of generating high-quality dance movements and supporting multi-modal control, including genre control, semantic control, and spatial control. First, we decouple the dance generation network from the dance control network, thereby avoiding the degradation in dance quality when adding additional control information. Second, we design specific control strategies for different control information and integrate them into a unified framework. Experimental results show that the proposed dance generation framework outperforms state-of-the-art methods in terms of motion quality and controllability.【6】 CoMoSVC: Consistency Model-based Singing Voice Conversion标题：CoMoSVC：基于一致性模型的歌声转换链接：https://arxiv.org/abs/2401.01792作者：Yiwen Lu,Zhen Ye,Wei Xue,Xu Tan,Qifeng Liu,Yike Guo摘要：基于扩散的歌唱声转换（SVC）方法已经取得了显着的性能，产生与目标音色高度相似的自然音频。然而，迭代采样过程导致推理速度慢，因此加速变得至关重要。在本文中，我们提出了CoMoSVC，基于一致性模型的SVC方法，其目的是实现高质量的生成和高速采样。首先设计了一个基于扩散的教师模型，并在自洽性下进一步提取学生模型，以实现一步抽样。在单个NVIDIA GTX 4090 GPU上的实验表明，尽管CoMoSVC的推理速度明显快于最先进的（SOTA）基于扩散的SVC系统，但它仍然可以根据主观和客观指标实现相当或更高的转换性能。音频样本和代码可在https://comosvc.github.io/上获得。摘要：The diffusion-based Singing Voice Conversion (SVC) methods have achieved remarkable performances, producing natural audios with high similarity to the target timbre. However, the iterative sampling process results in slow inference speed, and acceleration thus becomes crucial. In this paper, we propose CoMoSVC, a consistency model-based SVC method, which aims to achieve both high-quality generation and high-speed sampling. A diffusion-based teacher model is first specially designed for SVC, and a student model is further distilled under self-consistency properties to achieve one-step sampling. Experiments on a single NVIDIA GTX4090 GPU reveal that although CoMoSVC has a significantly faster inference speed than the state-of-the-art (SOTA) diffusion-based SVC system, it still achieves comparable or superior conversion performance based on both subjective and objective metrics. Audio samples and codes are available at https://comosvc.github.io/.【7】 Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction标题：基于语义词预测的两阶段文语转换神经传感器链接：https://arxiv.org/abs/2401.01498作者：Minchan Kim,Myeonghun Jeong,Byoung Jin Choi,Semin Kim,Joun Yeop Lee,Nam Soo Kim备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：我们提出了一种新的文本到语音（TTS）框架围绕神经换能器。我们的方法将整个TTS管道分为语义级序列到序列（seq2seq）建模和细粒度声学建模阶段，利用从wav2vec2.0嵌入获得的离散语义令牌。对于一个强大的和有效的对齐建模，我们采用了一个神经换能器命名为令牌换能器的语义令牌预测，受益于其硬单调对齐约束。随后，非自回归（NAR）语音发生器有效地合成这些语义令牌的波形。此外，参考语音控制每个阶段的时间动态和声学条件。这种解耦的框架降低了TTS的训练复杂度，同时允许每个阶段专注于语义和声学建模。我们在zero-shot自适应TTS上的实验结果表明，我们的模型在语音质量和说话人相似度方面都超过了基线，无论是客观的还是主观的。我们还深入研究了我们的方法的推理速度和韵律控制能力，突出了TTS框架中神经换能器的潜力。摘要：We propose a novel text-to-speech (TTS) framework centered around a neural transducer. Our approach divides the whole TTS pipeline into semantic-level sequence-to-sequence (seq2seq) modeling and fine-grained acoustic modeling stages, utilizing discrete semantic tokens obtained from wav2vec2.0 embeddings. For a robust and efficient alignment modeling, we employ a neural transducer named token transducer for the semantic token prediction, benefiting from its hard monotonic alignment constraints. Subsequently, a non-autoregressive (NAR) speech generator efficiently synthesizes waveforms from these semantic tokens. Additionally, a reference speech controls temporal dynamics and acoustic conditions at each stage. This decoupled framework reduces the training complexity of TTS while allowing each stage to focus on semantic and acoustic modeling. Our experimental results on zero-shot adaptive TTS demonstrate that our model surpasses the baseline in terms of speech quality and speaker similarity, both objectively and subjectively. We also delve into the inference speed and prosody control capabilities of our approach, highlighting the potential of neural transducers in TTS frameworks.【8】 Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learning标题：基于自精馏和在线聚类的自监督反思性说话人表征学习链接：https://arxiv.org/abs/2401.01473作者：Danwei Cai,Zexin Cai,Ming Li摘要：说话人表征学习是现代语音识别系统的关键。虽然监督学习技术需要大量的标记数据，但无监督方法可以利用大量未标记的语料库，提供可扩展的解决方案。本文介绍了自监督反射学习（SSRL），一种新的范式，简化现有的迭代无监督框架。SSRL将自监督知识提取与在线聚类相结合，以改进伪标签并训练模型，而不会出现迭代瓶颈。具体而言，教师模型通过在线聚类不断细化伪标签，提供动态监督信号来训练学生模型。学生模型通过输入和模型噪声进行噪声学生训练，以提高其建模能力。教师模型通过学生的指数移动平均来更新，作为过去迭代的集合。此外，伪标签队列保留历史标签以保持一致性，并且噪声标签建模将学习引向干净的样本。在VoxCeleb上的实验表明，SSRL优于当前的迭代方法，在单个训练轮中超过了5轮方法的性能。消融研究验证了关键组件，如嘈杂的标签建模和伪标签队列的贡献。此外，伪标记和聚类计数的收敛性的持续改进证明了SSRL在破译未标记数据方面的有效性。这项工作标志着一个重要的进步，有效和准确的扬声器表征学习通过新的反思学习范式。摘要：Speaker representation learning is critical for modern voice recognition systems. While supervised learning techniques require extensive labeled data, unsupervised methodologies can leverage vast unlabeled corpora, offering a scalable solution. This paper introduces self-supervised reflective learning (SSRL), a novel paradigm that streamlines existing iterative unsupervised frameworks. SSRL integrates self-supervised knowledge distillation with online clustering to refine pseudo labels and train the model without iterative bottlenecks. Specifically, a teacher model continually refines pseudo labels through online clustering, providing dynamic supervision signals to train the student model. The student model undergoes noisy student training with input and model noise to boost its modeling capacity. The teacher model is updated via an exponential moving average of the student, acting as an ensemble of past iterations. Further, a pseudo label queue retains historical labels for consistency, and noisy label modeling directs learning towards clean samples. Experiments on VoxCeleb show SSRL's superiority over current iterative approaches, surpassing the performance of a 5-round method in just a single training round. Ablation studies validate the contributions of key components like noisy label modeling and pseudo label queues. Moreover, consistent improvements in pseudo labeling and the convergence of cluster counts demonstrate SSRL's effectiveness in deciphering unlabeled data. This work marks an important advancement in efficient and accurate speaker representation learning through the novel reflective learning paradigm.【1】 CoMoSVC: Consistency Model-based Singing Voice Conversion标题：CoMoSVC：基于一致性模型的歌声转换链接：https://arxiv.org/abs/2401.01792作者：Yiwen Lu,Zhen Ye,Wei Xue,Xu Tan,Qifeng Liu,Yike Guo摘要：基于扩散的歌唱声转换（SVC）方法已经取得了显着的性能，产生与目标音色高度相似的自然音频。然而，迭代采样过程导致推理速度慢，因此加速变得至关重要。在本文中，我们提出了CoMoSVC，基于一致性模型的SVC方法，其目的是实现高质量的生成和高速采样。首先设计了一个基于扩散的教师模型，并在自洽性下进一步提取学生模型，以实现一步抽样。在单个NVIDIA GTX 4090 GPU上的实验表明，尽管CoMoSVC的推理速度明显快于最先进的（SOTA）基于扩散的SVC系统，但它仍然可以根据主观和客观指标实现相当或更高的转换性能。音频样本和代码可在https://comosvc.github.io/上获得。摘要：The diffusion-based Singing Voice Conversion (SVC) methods have achieved remarkable performances, producing natural audios with high similarity to the target timbre. However, the iterative sampling process results in slow inference speed, and acceleration thus becomes crucial. In this paper, we propose CoMoSVC, a consistency model-based SVC method, which aims to achieve both high-quality generation and high-speed sampling. A diffusion-based teacher model is first specially designed for SVC, and a student model is further distilled under self-consistency properties to achieve one-step sampling. Experiments on a single NVIDIA GTX4090 GPU reveal that although CoMoSVC has a significantly faster inference speed than the state-of-the-art (SOTA) diffusion-based SVC system, it still achieves comparable or superior conversion performance based on both subjective and objective metrics. Audio samples and codes are available at https://comosvc.github.io/.【2】 Utilizing Neural Transducers for Two-Stage Text-to-Speech via Semantic Token Prediction标题：基于语义词预测的两阶段文语转换神经传感器链接：https://arxiv.org/abs/2401.01498作者：Minchan Kim,Myeonghun Jeong,Byoung Jin Choi,Semin Kim,Joun Yeop Lee,Nam Soo Kim备注：This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible摘要：我们提出了一种新的文本到语音（TTS）框架围绕神经换能器。我们的方法将整个TTS管道分为语义级序列到序列（seq2seq）建模和细粒度声学建模阶段，利用从wav2vec2.0嵌入获得的离散语义令牌。对于一个强大的和有效的对齐建模，我们采用了一个神经换能器命名为令牌换能器的语义令牌预测，受益于其硬单调对齐约束。随后，非自回归（NAR）语音发生器有效地合成这些语义令牌的波形。此外，参考语音控制每个阶段的时间动态和声学条件。这种解耦的框架降低了TTS的训练复杂度，同时允许每个阶段专注于语义和声学建模。我们在zero-shot自适应TTS上的实验结果表明，我们的模型在语音质量和说话人相似度方面都超过了基线，无论是客观的还是主观的。我们还深入研究了我们的方法的推理速度和韵律控制能力，突出了TTS框架中神经换能器的潜力。摘要：We propose a novel text-to-speech (TTS) framework centered around a neural transducer. Our approach divides the whole TTS pipeline into semantic-level sequence-to-sequence (seq2seq) modeling and fine-grained acoustic modeling stages, utilizing discrete semantic tokens obtained from wav2vec2.0 embeddings. For a robust and efficient alignment modeling, we employ a neural transducer named token transducer for the semantic token prediction, benefiting from its hard monotonic alignment constraints. Subsequently, a non-autoregressive (NAR) speech generator efficiently synthesizes waveforms from these semantic tokens. Additionally, a reference speech controls temporal dynamics and acoustic conditions at each stage. This decoupled framework reduces the training complexity of TTS while allowing each stage to focus on semantic and acoustic modeling. Our experimental results on zero-shot adaptive TTS demonstrate that our model surpasses the baseline in terms of speech quality and speaker similarity, both objectively and subjectively. We also delve into the inference speed and prosody control capabilities of our approach, highlighting the potential of neural transducers in TTS frameworks.【3】 Self-supervised Reflective Learning through Self-distillation and Online Clustering for Speaker Representation Learning标题：基于自蒸馏和在线聚类的自监督反射式说话人表征学习链接：https://arxiv.org/abs/2401.01473作者：Danwei Cai,Zexin Cai,Ming Li摘要：None摘要：Speaker representation learning is critical for modern voice recognition systems. While supervised learning techniques require extensive labeled data, unsupervised methodologies can leverage vast unlabeled corpora, offering a scalable solution. This paper introduces self-supervised reflective learning (SSRL), a novel paradigm that streamlines existing iterative unsupervised frameworks. SSRL integrates self-supervised knowledge distillation with online clustering to refine pseudo labels and train the model without iterative bottlenecks. Specifically, a teacher model continually refines pseudo labels through online clustering, providing dynamic supervision signals to train the student model. The student model undergoes noisy student training with input and model noise to boost its modeling capacity. The teacher model is updated via an exponential moving average of the student, acting as an ensemble of past iterations. Further, a pseudo label queue retains historical labels for consistency, and noisy label modeling directs learning towards clean samples. Experiments on VoxCeleb show SSRL's superiority over current iterative approaches, surpassing the performance of a 5-round method in just a single training round. Ablation studies validate the contributions of key components like noisy label modeling and pseudo label queues. Moreover, consistent improvements in pseudo labeling and the convergence of cluster counts demonstrate SSRL's effectiveness in deciphering unlabeled data. This work marks an important advancement in efficient and accurate speaker representation learning through the novel reflective learning paradigm.【4】 Multichannel blind speech source separation with a disjoint constraint source model标题：基于不相交约束源模型的多声道语音盲分离链接：https://arxiv.org/abs/2401.01763作者：Jianyu Wang,Shanzheng Guan摘要：多通道卷积盲语音源分离是指在没有混合系统先验信息的情况下，从观测到的多通道混合信号中分离出不同的语音源。多通道非负矩阵分解（MNMF）已被证明是最强大的分离框架之一，其代表性算法如MNMF和独立低秩矩阵分析（ILRMA）已显示出良好的性能。然而，语音源信号的稀疏特性没有充分考虑在这样的框架。众所周知，语音信号是稀疏的，这是在这项工作中考虑，以提高分离性能。具体来说，我们利用宾厄姆和拉普拉斯分布制定一个不相交的约束正则化，随后纳入MNMF和ILRMA。然后，我们推导出优化最小化规则更新相关的源模型的参数，从而在两个增强算法的发展：S-MNMF和S-ILRMA。进行了全面的模拟，结果明确地证明了我们提出的方法的有效性。摘要：Multichannel convolutive blind speech source separation refers to the problem of separating different speech sources from the observed multichannel mixtures without much a priori information about the mixing system. Multichannel nonnegative matrix factorization (MNMF) has been proven to be one of the most powerful separation frameworks and the representative algorithms such as MNMF and the independent low-rank matrix analysis (ILRMA) have demonstrated great performance. However, the sparseness properties of speech source signals are not fully taken into account in such a framework. It is well known that speech signals are sparse in nature, which is considered in this work to improve the separation performance. Specifically, we utilize the Bingham and Laplace distributions to formulate a disjoint constraint regularizer, which is subsequently incorporated into both MNMF and ILRMA. We then derive majorization-minimization rules for updating parameters related to the source model, resulting in the development of two enhanced algorithms: s-MNMF and s-ILRMA. Comprehensive simulations are conducted, and the results unequivocally demonstrate the efficacy of our proposed methodologies.【5】 Independent low-rank matrix analysis based on the Sinkhorn divergence source model for blind source separation标题：基于Sinkhorn散射源模型的独立低阶矩阵盲源分离链接：https://arxiv.org/abs/2401.01762作者：Jianyu Wang,Shanzheng Guan,Jingdong Chen,Jacob Benesty摘要：独立低秩矩阵分析（ILRMA）在解决语音信号盲分离问题上有着巨大的潜力。该方法假设不同频段的谱是独立的，且任意频段的谱系数服从高斯分布。然后采用Itakura-Saito散度估计源模型相关参数。然而，在现实中，来自不同频带的谱系数可能是相关的，这在现有的ILRMA算法中是没有考虑的。本文提出了一种改进的ILRMA算法，该算法考虑了不同频带的谱系数之间的相关性。然后利用Sinkhorn发散来优化源模型参数。由于使用了跨带信息，BSS性能得到了改善。但是要估计的参数的数量也显著增加，并且计算复杂度也显著增加。为了降低算法的复杂度，我们应用克罗内克积将建模矩阵分解为许多维数更小的矩阵的乘积。然后，一个有效的算法来实现基于Sinkhorn发散的BSS算法和复杂度降低了一个数量级。摘要：The so-called independent low-rank matrix analysis (ILRMA) has demonstrated a great potential for dealing with the problem of determined blind source separation (BSS) for audio and speech signals. This method assumes that the spectra from different frequency bands are independent and the spectral coefficients in any frequency band are Gaussian distributed. The Itakura-Saito divergence is then employed to estimate the source model related parameters. In reality, however, the spectral coefficients from different frequency bands may be dependent, which is not considered in the existing ILRMA algorithm. This paper presents an improved version of ILRMA, which considers the dependency between the spectral coefficients from different frequency bands. The Sinkhorn divergence is then exploited to optimize the source model parameters. As a result of using the cross-band information, the BSS performance is improved. But the number of parameters to be estimated also increases significantly, and so is the computational complexity. To reduce the algorithm complexity, we apply the Kronecker product to decompose the modeling matrix into the product of a number of matrices of much smaller dimensionality. An efficient algorithm is then developed to implement the Sinkhorn divergence based BSS algorithm and the complexity is reduced by an order of magnitude.【6】 Incremental FastPitch: Chunk-based High Quality Text to Speech标题：增量式快速音调：基于块的高质量文语转换链接：https://arxiv.org/abs/2401.01755作者：Muyang Du,Chuan Liu,Junjie Lai备注：5 pages, 4 figures, 1 table摘要：并行文语转换模型已被广泛应用于实时语音合成中，与传统的自回归模型相比，它们具有更好的可控性和更快的合成过程。虽然并行模型在许多方面具有优势，但由于其完全并行的体系结构（如Transformer），它们自然不适合增量综合。在这项工作中，我们提出了增量FastPitch，一种新的FastPitch变体，能够通过改进基于块的FFT块的架构，使用接收域约束的块注意力掩码进行训练，以及使用固定大小的过去模型状态进行推理，来增量地产生高质量的Mel块。实验结果表明，我们的建议可以产生的语音质量媲美并行FastPitch，具有显着较低的延迟，允许更低的响应时间为实时语音应用程序。摘要：Parallel text-to-speech models have been widely applied for real-time speech synthesis, and they offer more controllability and a much faster synthesis process compared with conventional auto-regressive models. Although parallel models have benefits in many aspects, they become naturally unfit for incremental synthesis due to their fully parallel architecture such as transformer. In this work, we propose Incremental FastPitch, a novel FastPitch variant capable of incrementally producing high-quality Mel chunks by improving the architecture with chunk-based FFT blocks, training with receptive-field constrained chunk attention masks, and inference with fixed size past model states. Experimental results show that our proposal can produce speech quality comparable to the parallel FastPitch, with a significant lower latency that allows even lower response time for real-time speech applications.【7】 Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models标题：神经自动语音识别中的幻觉：识别错误和幻觉模型链接：https://arxiv.org/abs/2401.01572作者：Rita Frieske,Bertram E. Shi摘要：幻觉是深度神经网络产生的一种输出错误。虽然这在自然语言处理中已经研究过，但在自动语音识别中以前没有研究过。在这里，我们将ASR中的幻觉定义为由与源话语语义无关但仍然流畅和连贯的模型生成的transmittance。幻觉与模型的可能自然语言输出的相似性产生了欺骗的危险，并影响了系统的可信度。我们发现，常用的指标，如单词错误率，不能区分幻觉和非幻觉模型。为了解决这个问题，我们提出了一种基于扰动的方法，用于评估自动语音识别（ASR）模型在测试时对幻觉的敏感性，该方法不需要访问训练数据集。我们证明，这种方法有助于区分具有相似基线单词错误率的幻觉和非幻觉模型。我们进一步探索ASR错误类型与数据集噪声类型之间的关系，以确定哪些类型的噪声最有可能产生幻觉输出。我们设计了一个框架，通过分析他们的语义连接与地面真相和他们的流畅性识别幻觉。最后，我们发现如何诱导幻觉与随机噪声注入的话语。摘要：Hallucinations are a type of output error produced by deep neural networks. While this has been studied in natural language processing, they have not been researched previously in automatic speech recognition. Here, we define hallucinations in ASR as transcriptions generated by a model that are semantically unrelated to the source utterance, yet still fluent and coherent. The similarity of hallucinations to probable natural language outputs of the model creates a danger of deception and impacts the credibility of the system. We show that commonly used metrics, such as word error rates, cannot differentiate between hallucinatory and non-hallucinatory models. To address this, we propose a perturbation-based method for assessing the susceptibility of an automatic speech recognition (ASR) model to hallucination at test time, which does not require access to the training dataset. We demonstrate that this method helps to distinguish between hallucinatory and non-hallucinatory models that have similar baseline word error rates. We further explore the relationship between the types of ASR errors and the types of dataset noise to determine what types of noise are most likely to create hallucinatory outputs. We devise a framework for identifying hallucinations by analysing their semantic connection with the ground truth and their fluency. Finally, we discover how to induce hallucinations with a random noise injection to the utterance.【8】 Exploring Multi-Modal Control in Music-Driven Dance Generation标题：多模态控制在音乐驱动舞蹈生成中的应用研究链接：https://arxiv.org/abs/2401.01382作者：Ronghui Li,Yuqin Dai,Yachao Zhang,Jun Li,Jian Yang,Jie Guo,Xiu Li摘要：现有的音乐驱动的三维舞蹈生成方法主要集中在高质量的舞蹈生成上，但在生成过程中缺乏足够的控制。为了解决这些问题，我们提出了一个统一的框架，能够生成高质量的舞蹈动作，并支持多模态控制，包括体裁控制，语义控制和空间控制。首先，我们将舞蹈生成网络与舞蹈控制网络解耦，从而避免了添加额外控制信息时舞蹈质量的下降。其次，针对不同的控制信息设计了具体的控制策略，并将其整合到一个统一的框架中。实验结果表明，所提出的舞蹈生成框架优于国家的最先进的方法在运动质量和可控性。摘要：Existing music-driven 3D dance generation methods mainly concentrate on high-quality dance generation, but lack sufficient control during the generation process. To address these issues, we propose a unified framework capable of generating high-quality dance movements and supporting multi-modal control, including genre control, semantic control, and spatial control. First, we decouple the dance generation network from the dance control network, thereby avoiding the degradation in dance quality when adding additional control information. Second, we design specific control strategies for different control information and integrate them into a unified framework. Experimental results show that the proposed dance generation framework outperforms state-of-the-art methods in terms of motion quality and controllability. 
每日语音速递