今日论文合集：cs.SD语音8篇，eess.AS音频处理8篇。
cs.SD语音
【1】 Impact of time and note duration tokenizations on deep learning symbolic  music modeling标题：时间和音长标记化对深度学习符号音乐建模的影响链接：https://arxiv.org/abs/2310.08497
作者：Nathan Fradet,Nicolas Gutowski,Fabien Chhel,Jean-Pierre Briot备注：ISMIR 2023摘要：符号音乐广泛用于各种深度学习任务，包括生成、转录、合成和音乐信息检索（MIR）。它主要用于离散模型，如Transformers，这需要将音乐标记化，即，格式化为称为标记的不同元素的序列。令牌化可以以不同的方式执行。由于Transformer在推理方面比较困难，但更容易捕获显式信息，因此研究这种模型的信息表示方式如何影响其性能是很重要的。在这项工作中，我们分析了常见的标记化方法，并与时间和注意持续时间表示实验。我们比较了这两个有影响力的标准在几个任务上的表现，包括作曲家和情感分类，音乐生成和序列表征学习。我们证明，明确的信息导致更好的结果取决于任务。摘要：Symbolic music is widely used in various deep learning tasks, including generation, transcription, synthesis, and Music Information Retrieval (MIR). It is mostly employed with discrete models like Transformers, which require music to be tokenized, i.e., formatted into sequences of distinct elements called tokens. Tokenization can be performed in different ways. As Transformer can struggle at reasoning, but capture more easily explicit information, it is important to study how the way the information is represented for such model impact their performances. In this work, we analyze the common tokenization methods and experiment with time and note duration representations. We compare the performances of these two impactful criteria on several tasks, including composer and emotion classification, music generation, and sequence representation learning. We demonstrate that explicit information leads to better results depending on the task.
【2】 Multimodal Variational Auto-encoder based Audio-Visual Segmentation标题：基于多模变分自动编码器的视听分割链接：https://arxiv.org/abs/2310.08303
作者：Yuxin Mao,Jing Zhang,Mochu Xiang,Yiran Zhong,Yuchao Dai备注：Accepted by ICCV2023,Project page(https://npucvr.github.io/MMVAE-AVS),Code(this https URL)摘要：提出了一种用于音视频分割（AVS）的显式条件多模态变分自动编码器（ECMVAE），旨在分割视频序列中的声源。现有的AVS方法集中于隐式特征融合策略，其中模型被训练以拟合数据集中的离散样本。对于有限且多样性较低的数据集，结果性能通常不令人满意。相比之下，我们解决这个问题，从一个有效的表征学习的角度来看，旨在明确建模的每一个模态的贡献。具体来说，我们发现，音频包含关键类别信息的声音生产者，和视觉数据提供候选声音生产者（S）。它们的共享信息对应于可视数据中所示的目标声音产生者。在这种情况下，跨模态共享表征学习对AVS尤为重要。为了实现这一点，我们的ECMVAE因式分解与模态共享表示和模态特定的表示每一个模态的表示。在共享表示和特定表示之间应用正交性约束，以保持因子分解的潜在代码的独占属性。此外，引入互信息最大化正则化器来实现对每个模态的广泛探索。AVSBench上的定量和定性评估证明了我们的方法的有效性，从而为AVS带来了新的最先进的技术，在具有挑战性的MS3子集上实现了3.84 mIOU的性能飞跃，用于多声源分割。摘要：We propose an Explicit Conditional Multimodal Variational Auto-Encoder (ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sources in the video sequence. Existing AVS methods focus on implicit feature fusion strategies, where models are trained to fit the discrete samples in the dataset. With a limited and less diverse dataset, the resulting performance is usually unsatisfactory. In contrast, we address this problem from an effective representation learning perspective, aiming to model the contribution of each modality explicitly. Specifically, we find that audio contains critical category information of the sound producers, and visual data provides candidate sound producer(s). Their shared information corresponds to the target sound producer(s) shown in the visual data. In this case, cross-modal shared representation learning is especially important for AVS. To achieve this, our ECMVAE factorizes the representations of each modality with a modality-shared representation and a modality-specific representation. An orthogonality constraint is applied between the shared and specific representations to maintain the exclusive attribute of the factorized latent code. Further, a mutual information maximization regularizer is introduced to achieve extensive exploration of each modality. Quantitative and qualitative evaluations on the AVSBench demonstrate the effectiveness of our approach, leading to a new state-of-the-art for AVS, with a 3.84 mIOU performance leap on the challenging MS3 subset for multiple sound source segmentation.
【3】 On the Relevance of Phoneme Duration Variability of Synthesized Training  Data for Automatic Speech Recognition标题：用于自动语音识别的合成训练数据的音素时长变异性相关性研究链接：https://arxiv.org/abs/2310.08132
作者：Nick Rossenbach,Benedikt Hilmes,Ralf Schlüter备注：To appear at ASRU 2023摘要：由文本到语音（TTS）系统生成的合成数据可以用于在低资源或域失配任务中改进自动语音识别（ASR）系统。它已被证明，TTS生成的输出仍然不具有相同的质量作为真正的数据。在这项工作中，我们专注于合成数据的时间结构及其与ASR训练的关系。通过使用一种新的Oracle设置，我们显示了在非自回归（NAR）TTS中持续时间建模对合成数据质量的影响。为了获得参考音素持续时间，我们使用两种常见的对齐方法，隐马尔可夫高斯混合模型（HMM-GMM）对齐器和神经连接主义时间分类（CTC）对齐器。使用基于随机游走的简单算法，我们将TTS系统的音素持续时间分布更接近真实持续时间，从而在半监督设置中使用合成数据改进ASR系统。摘要：Synthetic data generated by text-to-speech (TTS) systems can be used to improve automatic speech recognition (ASR) systems in low-resource or domain mismatch tasks. It has been shown that TTS-generated outputs still do not have the same qualities as real data. In this work we focus on the temporal structure of synthetic data and its relation to ASR training. By using a novel oracle setup we show how much the degradation of synthetic data quality is influenced by duration modeling in non-autoregressive (NAR) TTS. To get reference phoneme durations we use two common alignment methods, a hidden Markov Gaussian-mixture model (HMM-GMM) aligner and a neural connectionist temporal classification (CTC) aligner. Using a simple algorithm based on random walks we shift phoneme duration distributions of the TTS system closer to real durations, resulting in an improvement of an ASR system using synthetic data in a semi-supervised setting.
【4】 Crowdsourced and Automatic Speech Prominence Estimation标题：基于众包的语音显著度自动估计链接：https://arxiv.org/abs/2310.08464
作者：Max Morrison,Pranav Pawar,Nathan Pruyne,Jennifer Cole,Bryan Pardo备注：Submitted to ICASSP 2024摘要：一个口语单词的突出度是一个普通的母语听众认为这个单词相对于它的上下文是突出或强调的程度。语音显著性估计是对话语中的每个词的显著性分配数值的过程。这些显著性标签对于语言分析很有用，也可以训练自动化系统执行由显著性控制的文本到语音或情感识别。人工标注显著性是耗时且昂贵的，这激发了用于语音显著性估计的自动化方法的发展。然而，使用机器学习方法开发这样的自动化系统需要人工注释的训练数据。使用我们的系统来获取这样的人类注释，我们收集并开源LibriTTS数据集的一部分的众包注释。我们使用这些注释作为地面实况来训练神经语音显著性估计器，该估计器概括了看不见的说话者、数据集和说话风格。我们研究了神经显著性估计的设计决策，以及神经显著性估计如何根据注释成本的两个关键因素进行改进：数据集大小和每个话语的注释数量。摘要：The prominence of a spoken word is the degree to which an average native listener perceives the word as salient or emphasized relative to its context. Speech prominence estimation is the process of assigning a numeric value to the prominence of each word in an utterance. These prominence labels are useful for linguistic analysis, as well as training automated systems to perform emphasis-controlled text-to-speech or emotion recognition. Manually annotating prominence is time-consuming and expensive, which motivates the development of automated methods for speech prominence estimation. However, developing such an automated system using machine-learning methods requires human-annotated training data. Using our system for acquiring such human annotations, we collect and open-source crowdsourced annotations of a portion of the LibriTTS dataset. We use these annotations as ground truth to train a neural speech prominence estimator that generalizes to unseen speakers, datasets, and speaking styles. We investigate design decisions for neural prominence estimation as well as how neural prominence estimation improves as a function of two key factors of annotation cost: dataset size and the number of annotations per utterance.
【5】 A cry for help: Early detection of brain injury in newborns标题：呼救：新生儿脑损伤的早期发现链接：https://arxiv.org/abs/2310.08338
作者：Charles C. Onu,Samantha Latremouille,Arsenii Gorin,Junhao Wang,Uchenna Ekwochi,Peter O. Ubuane,Omolara A. Kehinde,Muhammad A. Salisu,Datonye Briggs,Yoshua Bengio,Doina Precup摘要：自20世纪60年代以来，新生儿临床医生已经知道，患有某些神经系统疾病的新生儿表现出改变的哭声模式，例如出生窒息时的高音哭声。尽管每年有150多万婴儿死亡和残疾，但早期发现新生儿窒息造成的脑损伤仍然是一项挑战，特别是在发展中国家，那里的大多数分娩都没有受过训练的医生参加。在这里，我们报告了第一个洲际临床研究，证明新生儿脑损伤可以通过使用我们称为Roseline的AI算法从记录的婴儿哭声中可靠地确定。以前和最近的工作一直受到缺乏大型，高质量的临床哭记录数据库的限制，限制了最先进的机器学习的应用。我们为基于音频的病理检测模型开发了一种新的训练方法，并在从地理位置不同的环境（3大洲的5家医院）获得的新生儿哭声的大型数据库上评估了该系统。我们的系统提取可解释的声学生物标志物，支持临床决策，并能够准确地检测新生儿哭声中的神经损伤，AUC为92.5%（88.7%的灵敏度和80%的特异性）。基于哭声的神经监测为低成本、易于使用、非侵入性和无接触的高危婴儿筛查打开了大门，特别是当集成到智能手机或新生儿ICU监护仪等简单设备中时。这将提供一个可靠的工具，在没有替代方案的情况下，但也减少了定期对新生儿进行身体疲惫或辐射暴露评估（如脑部CT扫描）的需要。这项工作为将婴儿哭声作为生命体征奠定了基础，并表明了人工智能驱动的声音监测对未来负担得起的医疗保健的潜力。摘要：Since the 1960s, neonatal clinicians have known that newborns suffering from certain neurological conditions exhibit altered crying patterns such as the high-pitched cry in birth asphyxia. Despite an annual burden of over 1.5 million infant deaths and disabilities, early detection of neonatal brain injuries due to asphyxia remains a challenge, particularly in developing countries where the majority of births are not attended by a trained physician. Here we report on the first inter-continental clinical study to demonstrate that neonatal brain injury can be reliably determined from recorded infant cries using an AI algorithm we call Roseline. Previous and recent work has been limited by the lack of a large, high-quality clinical database of cry recordings, constraining the application of state-of-the-art machine learning. We develop a new training methodology for audio-based pathology detection models and evaluate this system on a large database of newborn cry sounds acquired from geographically diverse settings -- 5 hospitals across 3 continents. Our system extracts interpretable acoustic biomarkers that support clinical decisions and is able to accurately detect neurological injury from newborns' cries with an AUC of 92.5% (88.7% sensitivity at 80% specificity). Cry-based neurological monitoring opens the door for low-cost, easy-to-use, non-invasive and contact-free screening of at-risk babies, especially when integrated into simple devices like smartphones or neonatal ICU monitors. This would provide a reliable tool where there are no alternatives, but also curtail the need to regularly exert newborns to physically-exhausting or radiation-exposing assessments such as brain CT scans. This work sets the stage for embracing the infant cry as a vital sign and indicates the potential of AI-driven sound monitoring for the future of affordable healthcare.
【6】 A Single Speech Enhancement Model Unifying Dereverberation, Denoising,  Speaker Counting, Separation, and Extraction标题：一种统一了去混响、去噪、说话人计数、分离和提取的语音增强模型链接：https://arxiv.org/abs/2310.08277
作者：Kohei Saijo,Wangyou Zhang,Zhong-Qiu Wang,Shinji Watanabe,Tetsunori Kobayashi,Tetsuji Ogawa备注：6 pages, 4 figures, 2 tables, accepted by ASRU2023摘要：我们提出了一个多任务通用语音增强（MUSE）模型，可以执行五个语音增强（SE）任务：去混响、去噪、语音分离（SS）、目标说话人提取（TSE）和说话人计数。这是通过将两个模块集成到SE模型中来实现的：1）内部分离模块，其进行说话者计数和分离;以及2）TSE模块，其使用目标说话者提示从内部分离输出中提取目标语音。该模型被训练为如果目标说话人提示被给出则执行TSE，否则执行SS。通过训练模型来去除噪声和混响，我们允许模型用一个模型来处理上述五个任务，这还没有完成。评估结果表明，所提出的MUSE模型可以成功地处理多个任务与一个单一的模型。摘要：We propose a multi-task universal speech enhancement (MUSE) model that can perform five speech enhancement (SE) tasks: dereverberation, denoising, speech separation (SS), target speaker extraction (TSE), and speaker counting. This is achieved by integrating two modules into an SE model: 1) an internal separation module that does both speaker counting and separation; and 2) a TSE module that extracts the target speech from the internal separation outputs using target speaker cues. The model is trained to perform TSE if the target speaker cue is given and SS otherwise. By training the model to remove noise and reverberation, we allow the model to tackle the five tasks mentioned above with a single model, which has not been accomplished yet. Evaluation results demonstrate that the proposed MUSE model can successfully handle multiple tasks with a single model.
【7】 Fast Word Error Rate Estimation Using Self-Supervised Representations  For Speech And Text标题：基于语音和文本自监督表示的快速误词率估计链接：https://arxiv.org/abs/2310.08225
作者：Chanho Park,Chengsong Lu,Mingjie Chen,Thomas Hain备注：5 pages摘要：自动语音识别（ASR）的质量通常由单词错误率（WER）来衡量。WER估计是针对给定语音话语和转录来预测ASR系统的WER的任务。当高级ASR系统在大量数据上进行训练时，这项任务已经得到了越来越多的关注。在这种情况下，WER估计在许多场景中变得必要，例如，选择具有未知转录质量的训练数据或估计没有地面真实转录的ASR系统的测试性能。面对大量的数据，WER估计器的计算效率在实际应用中变得至关重要。然而，以前的工作通常不认为这是一个优先事项。本文介绍了一种使用自监督学习表示（SSLR）的快速WER估计器（Fe-WER）。估计器是建立在平均池聚合的SSLR上的。结果表明，Fe-WER在Ted-Lium 3上的均方根误差和Pearson相关系数的两个评价指标中分别相对于e-WER 3基线高出19.69%和7.16%。此外，当目标为10.88%时，按持续时间加权的估计为10.43%。最后，推理速度在实时因素方面约为4倍。摘要：The quality of automatic speech recognition (ASR) is typically measured by word error rate (WER). WER estimation is a task aiming to predict the WER of an ASR system, given a speech utterance and a transcription. This task has gained increasing attention while advanced ASR systems are trained on large amounts of data. In this case, WER estimation becomes necessary in many scenarios, for example, selecting training data with unknown transcription quality or estimating the testing performance of an ASR system without ground truth transcriptions. Facing large amounts of data, the computation efficiency of a WER estimator becomes essential in practical applications. However, previous works usually did not consider it as a priority. In this paper, a Fast WER estimator (Fe-WER) using self-supervised learning representation (SSLR) is introduced. The estimator is built upon SSLR aggregated by average pooling. The results show that Fe-WER outperformed the e-WER3 baseline relatively by 19.69% and 7.16% on Ted-Lium3 in both evaluation metrics of root mean square error and Pearson correlation coefficient, respectively. Moreover, the estimation weighted by duration was 10.43% when the target was 10.88%. Lastly, the inference speed was about 4x in terms of a real-time factor.
【8】 Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and  Textually Described Voices标题：对结巴的语音、乐器、看不见的语言和文字描述的语音进行语音转换链接：https://arxiv.org/abs/2310.08104
作者：Matthew Baas,Herman Kamper备注：11 pages, 1 figure, 5 tables. Accepted at SACAIR 2023摘要：语音转换的目的是利用目标说话人的录音作为参考，将源语音转换为目标语音。更新的模型正在产生越来越真实的输出。但是，当模型被输入非标准数据时会发生什么，比如来自语言障碍用户的语音？我们研究了最近的语音转换模型如何执行非标准的下游语音转换任务。我们使用一种简单但强大的方法，称为k-最近邻语音转换（kNN-VC）。我们来看四个非标准应用：口吃语音转换、跨语言语音转换、乐器转换和文本到语音转换。后者涉及转换为通过文本描述指定的目标语音，例如，“一个声音尖锐的年轻人”与已建立的基线相比，我们发现kNN-VC在口吃和跨语言语音转换中保持了高性能。乐器和文本到语音转换任务的结果更为复杂。例如，在一个示例中，kNN-VC在一些乐器上工作得很好，比如鼓，但在其他乐器上则不行。然而，这表明语音转换模型-特别是kNN-VC-越来越适用于一系列非标准下游任务。但当样本距离训练分布很远时，仍然存在局限性。代码、样本、训练模型：https://rf5.github.io/sacair2023-knnvc-demo/。摘要：Voice conversion aims to convert source speech into a target voice using recordings of the target speaker as a reference. Newer models are producing increasingly realistic output. But what happens when models are fed with non-standard data, such as speech from a user with a speech impairment? We investigate how a recent voice conversion model performs on non-standard downstream voice conversion tasks. We use a simple but robust approach called k-nearest neighbors voice conversion (kNN-VC). We look at four non-standard applications: stuttered voice conversion, cross-lingual voice conversion, musical instrument conversion, and text-to-voice conversion. The latter involves converting to a target voice specified through a text description, e.g. "a young man with a high-pitched voice". Compared to an established baseline, we find that kNN-VC retains high performance in stuttered and cross-lingual voice conversion. Results are more mixed for the musical instrument and text-to-voice conversion tasks. E.g., kNN-VC works well on some instruments like drums but not on others. Nevertheless, this shows that voice conversion models - and kNN-VC in particular - are increasingly applicable in a range of non-standard downstream tasks. But there are still limitations when samples are very far from the training distribution. Code, samples, trained models: https://rf5.github.io/sacair2023-knnvc-demo/.
eess.AS音频处理
【1】 Crowdsourced and Automatic Speech Prominence Estimation标题：基于众包的语音显著度自动估计链接：https://arxiv.org/abs/2310.08464
作者：Max Morrison,Pranav Pawar,Nathan Pruyne,Jennifer Cole,Bryan Pardo备注：Submitted to ICASSP 2024摘要：一个口语单词的突出度是一个普通的母语听众认为这个单词相对于它的上下文是突出或强调的程度。语音显著性估计是对话语中的每个词的显著性分配数值的过程。这些显著性标签对于语言分析很有用，也可以训练自动化系统执行由显著性控制的文本到语音或情感识别。人工标注显著性是耗时且昂贵的，这激发了用于语音显著性估计的自动化方法的发展。然而，使用机器学习方法开发这样的自动化系统需要人工注释的训练数据。使用我们的系统来获取这样的人类注释，我们收集并开源LibriTTS数据集的一部分的众包注释。我们使用这些注释作为地面实况来训练神经语音显著性估计器，该估计器概括了看不见的说话者、数据集和说话风格。我们研究了神经显著性估计的设计决策，以及神经显著性估计如何根据注释成本的两个关键因素进行改进：数据集大小和每个话语的注释数量。摘要：The prominence of a spoken word is the degree to which an average native listener perceives the word as salient or emphasized relative to its context. Speech prominence estimation is the process of assigning a numeric value to the prominence of each word in an utterance. These prominence labels are useful for linguistic analysis, as well as training automated systems to perform emphasis-controlled text-to-speech or emotion recognition. Manually annotating prominence is time-consuming and expensive, which motivates the development of automated methods for speech prominence estimation. However, developing such an automated system using machine-learning methods requires human-annotated training data. Using our system for acquiring such human annotations, we collect and open-source crowdsourced annotations of a portion of the LibriTTS dataset. We use these annotations as ground truth to train a neural speech prominence estimator that generalizes to unseen speakers, datasets, and speaking styles. We investigate design decisions for neural prominence estimation as well as how neural prominence estimation improves as a function of two key factors of annotation cost: dataset size and the number of annotations per utterance.
【2】 A cry for help: Early detection of brain injury in newborns标题：呼救：新生儿脑损伤的早期发现链接：https://arxiv.org/abs/2310.08338
作者：Charles C. Onu,Samantha Latremouille,Arsenii Gorin,Junhao Wang,Uchenna Ekwochi,Peter O. Ubuane,Omolara A. Kehinde,Muhammad A. Salisu,Datonye Briggs,Yoshua Bengio,Doina Precup摘要：自20世纪60年代以来，新生儿临床医生已经知道，患有某些神经系统疾病的新生儿表现出改变的哭声模式，例如出生窒息时的高音哭声。尽管每年有150多万婴儿死亡和残疾，但早期发现新生儿窒息造成的脑损伤仍然是一项挑战，特别是在发展中国家，那里的大多数分娩都没有受过训练的医生参加。在这里，我们报告了第一个洲际临床研究，证明新生儿脑损伤可以通过使用我们称为Roseline的AI算法从记录的婴儿哭声中可靠地确定。以前和最近的工作一直受到缺乏大型，高质量的临床哭记录数据库的限制，限制了最先进的机器学习的应用。我们为基于音频的病理检测模型开发了一种新的训练方法，并在从地理位置不同的环境（3大洲的5家医院）获得的新生儿哭声的大型数据库上评估了该系统。我们的系统提取可解释的声学生物标志物，支持临床决策，并能够准确地检测新生儿哭声中的神经损伤，AUC为92.5%（88.7%的灵敏度和80%的特异性）。基于哭声的神经监测为低成本、易于使用、非侵入性和无接触的高危婴儿筛查打开了大门，特别是当集成到智能手机或新生儿ICU监护仪等简单设备中时。这将提供一个可靠的工具，在没有替代方案的情况下，但也减少了定期对新生儿进行身体疲惫或辐射暴露评估（如脑部CT扫描）的需要。这项工作为将婴儿哭声作为生命体征奠定了基础，并表明了人工智能驱动的声音监测对未来负担得起的医疗保健的潜力。摘要：Since the 1960s, neonatal clinicians have known that newborns suffering from certain neurological conditions exhibit altered crying patterns such as the high-pitched cry in birth asphyxia. Despite an annual burden of over 1.5 million infant deaths and disabilities, early detection of neonatal brain injuries due to asphyxia remains a challenge, particularly in developing countries where the majority of births are not attended by a trained physician. Here we report on the first inter-continental clinical study to demonstrate that neonatal brain injury can be reliably determined from recorded infant cries using an AI algorithm we call Roseline. Previous and recent work has been limited by the lack of a large, high-quality clinical database of cry recordings, constraining the application of state-of-the-art machine learning. We develop a new training methodology for audio-based pathology detection models and evaluate this system on a large database of newborn cry sounds acquired from geographically diverse settings -- 5 hospitals across 3 continents. Our system extracts interpretable acoustic biomarkers that support clinical decisions and is able to accurately detect neurological injury from newborns' cries with an AUC of 92.5% (88.7% sensitivity at 80% specificity). Cry-based neurological monitoring opens the door for low-cost, easy-to-use, non-invasive and contact-free screening of at-risk babies, especially when integrated into simple devices like smartphones or neonatal ICU monitors. This would provide a reliable tool where there are no alternatives, but also curtail the need to regularly exert newborns to physically-exhausting or radiation-exposing assessments such as brain CT scans. This work sets the stage for embracing the infant cry as a vital sign and indicates the potential of AI-driven sound monitoring for the future of affordable healthcare.
【3】 A Single Speech Enhancement Model Unifying Dereverberation, Denoising,  Speaker Counting, Separation, and Extraction标题：一种统一了去混响、去噪、说话人计数、分离和提取的语音增强模型链接：https://arxiv.org/abs/2310.08277
作者：Kohei Saijo,Wangyou Zhang,Zhong-Qiu Wang,Shinji Watanabe,Tetsunori Kobayashi,Tetsuji Ogawa备注：6 pages, 4 figures, 2 tables, accepted by ASRU2023摘要：我们提出了一个多任务通用语音增强（MUSE）模型，可以执行五个语音增强（SE）任务：去混响、去噪、语音分离（SS）、目标说话人提取（TSE）和说话人计数。这是通过将两个模块集成到SE模型中来实现的：1）内部分离模块，其进行说话者计数和分离;以及2）TSE模块，其使用目标说话者提示从内部分离输出中提取目标语音。该模型被训练为如果目标说话人提示被给出则执行TSE，否则执行SS。通过训练模型来去除噪声和混响，我们允许模型用一个模型来处理上述五个任务，这还没有完成。评估结果表明，所提出的MUSE模型可以成功地处理多个任务与一个单一的模型。摘要：We propose a multi-task universal speech enhancement (MUSE) model that can perform five speech enhancement (SE) tasks: dereverberation, denoising, speech separation (SS), target speaker extraction (TSE), and speaker counting. This is achieved by integrating two modules into an SE model: 1) an internal separation module that does both speaker counting and separation; and 2) a TSE module that extracts the target speech from the internal separation outputs using target speaker cues. The model is trained to perform TSE if the target speaker cue is given and SS otherwise. By training the model to remove noise and reverberation, we allow the model to tackle the five tasks mentioned above with a single model, which has not been accomplished yet. Evaluation results demonstrate that the proposed MUSE model can successfully handle multiple tasks with a single model.
【4】 Fast Word Error Rate Estimation Using Self-Supervised Representations  For Speech And Text标题：基于语音和文本自监督表示的快速误词率估计链接：https://arxiv.org/abs/2310.08225
作者：Chanho Park,Chengsong Lu,Mingjie Chen,Thomas Hain备注：5 pages摘要：自动语音识别（ASR）的质量通常由单词错误率（WER）来衡量。WER估计是针对给定语音话语和转录来预测ASR系统的WER的任务。当高级ASR系统在大量数据上进行训练时，这项任务已经得到了越来越多的关注。在这种情况下，WER估计在许多场景中变得必要，例如，选择具有未知转录质量的训练数据或估计没有地面真实转录的ASR系统的测试性能。面对大量的数据，WER估计器的计算效率在实际应用中变得至关重要。然而，以前的工作通常不认为这是一个优先事项。本文介绍了一种使用自监督学习表示（SSLR）的快速WER估计器（Fe-WER）。估计器是建立在平均池聚合的SSLR上的。结果表明，Fe-WER在Ted-Lium 3上的均方根误差和Pearson相关系数的两个评价指标中分别相对于e-WER 3基线高出19.69%和7.16%。此外，当目标为10.88%时，按持续时间加权的估计为10.43%。最后，推理速度在实时因素方面约为4倍。摘要：The quality of automatic speech recognition (ASR) is typically measured by word error rate (WER). WER estimation is a task aiming to predict the WER of an ASR system, given a speech utterance and a transcription. This task has gained increasing attention while advanced ASR systems are trained on large amounts of data. In this case, WER estimation becomes necessary in many scenarios, for example, selecting training data with unknown transcription quality or estimating the testing performance of an ASR system without ground truth transcriptions. Facing large amounts of data, the computation efficiency of a WER estimator becomes essential in practical applications. However, previous works usually did not consider it as a priority. In this paper, a Fast WER estimator (Fe-WER) using self-supervised learning representation (SSLR) is introduced. The estimator is built upon SSLR aggregated by average pooling. The results show that Fe-WER outperformed the e-WER3 baseline relatively by 19.69% and 7.16% on Ted-Lium3 in both evaluation metrics of root mean square error and Pearson correlation coefficient, respectively. Moreover, the estimation weighted by duration was 10.43% when the target was 10.88%. Lastly, the inference speed was about 4x in terms of a real-time factor.
【5】 Voice Conversion for Stuttered Speech, Instruments, Unseen Languages and  Textually Described Voices标题：对结巴的语音、乐器、看不见的语言和文字描述的语音进行语音转换链接：https://arxiv.org/abs/2310.08104
作者：Matthew Baas,Herman Kamper备注：11 pages, 1 figure, 5 tables. Accepted at SACAIR 2023摘要：语音转换的目的是利用目标说话人的录音作为参考，将源语音转换为目标语音。更新的模型正在产生越来越真实的输出。但是，当模型被输入非标准数据时会发生什么，比如来自语言障碍用户的语音？我们研究了最近的语音转换模型如何执行非标准的下游语音转换任务。我们使用一种简单但强大的方法，称为k-最近邻语音转换（kNN-VC）。我们来看四个非标准应用：口吃语音转换、跨语言语音转换、乐器转换和文本到语音转换。后者涉及转换为通过文本描述指定的目标语音，例如，“一个声音尖锐的年轻人”与已建立的基线相比，我们发现kNN-VC在口吃和跨语言语音转换中保持了高性能。乐器和文本到语音转换任务的结果更为复杂。例如，在一个示例中，kNN-VC在一些乐器上工作得很好，比如鼓，但在其他乐器上则不行。然而，这表明语音转换模型-特别是kNN-VC-越来越适用于一系列非标准下游任务。但当样本距离训练分布很远时，仍然存在局限性。代码、样本、训练模型：https://rf5.github.io/sacair2023-knnvc-demo/。摘要：Voice conversion aims to convert source speech into a target voice using recordings of the target speaker as a reference. Newer models are producing increasingly realistic output. But what happens when models are fed with non-standard data, such as speech from a user with a speech impairment? We investigate how a recent voice conversion model performs on non-standard downstream voice conversion tasks. We use a simple but robust approach called k-nearest neighbors voice conversion (kNN-VC). We look at four non-standard applications: stuttered voice conversion, cross-lingual voice conversion, musical instrument conversion, and text-to-voice conversion. The latter involves converting to a target voice specified through a text description, e.g. "a young man with a high-pitched voice". Compared to an established baseline, we find that kNN-VC retains high performance in stuttered and cross-lingual voice conversion. Results are more mixed for the musical instrument and text-to-voice conversion tasks. E.g., kNN-VC works well on some instruments like drums but not on others. Nevertheless, this shows that voice conversion models - and kNN-VC in particular - are increasingly applicable in a range of non-standard downstream tasks. But there are still limitations when samples are very far from the training distribution. Code, samples, trained models: https://rf5.github.io/sacair2023-knnvc-demo/.
【6】 Impact of time and note duration tokenizations on deep learning symbolic  music modeling标题：时间和音长标记化对深度学习符号音乐建模的影响链接：https://arxiv.org/abs/2310.08497
作者：Nathan Fradet,Nicolas Gutowski,Fabien Chhel,Jean-Pierre Briot备注：ISMIR 2023摘要：符号音乐广泛用于各种深度学习任务，包括生成、转录、合成和音乐信息检索（MIR）。它主要用于离散模型，如Transformers，这需要将音乐标记化，即，格式化为称为标记的不同元素的序列。令牌化可以以不同的方式执行。由于Transformer在推理方面比较困难，但更容易捕获显式信息，因此研究这种模型的信息表示方式如何影响其性能是很重要的。在这项工作中，我们分析了常见的标记化方法，并与时间和注意持续时间表示实验。我们比较了这两个有影响力的标准在几个任务上的表现，包括作曲家和情感分类，音乐生成和序列表征学习。我们证明，明确的信息导致更好的结果取决于任务。摘要：Symbolic music is widely used in various deep learning tasks, including generation, transcription, synthesis, and Music Information Retrieval (MIR). It is mostly employed with discrete models like Transformers, which require music to be tokenized, i.e., formatted into sequences of distinct elements called tokens. Tokenization can be performed in different ways. As Transformer can struggle at reasoning, but capture more easily explicit information, it is important to study how the way the information is represented for such model impact their performances. In this work, we analyze the common tokenization methods and experiment with time and note duration representations. We compare the performances of these two impactful criteria on several tasks, including composer and emotion classification, music generation, and sequence representation learning. We demonstrate that explicit information leads to better results depending on the task.
【7】 Multimodal Variational Auto-encoder based Audio-Visual Segmentation标题：基于多模变分自动编码器的视听分割链接：https://arxiv.org/abs/2310.08303
作者：Yuxin Mao,Jing Zhang,Mochu Xiang,Yiran Zhong,Yuchao Dai备注：Accepted by ICCV2023,Project page(https://npucvr.github.io/MMVAE-AVS),Code(this https URL)摘要：提出了一种用于音视频分割（AVS）的显式条件多模态变分自动编码器（ECMVAE），旨在分割视频序列中的声源。现有的AVS方法集中于隐式特征融合策略，其中模型被训练以拟合数据集中的离散样本。对于有限且多样性较低的数据集，结果性能通常不令人满意。相比之下，我们解决这个问题，从一个有效的表征学习的角度来看，旨在明确建模的每一个模态的贡献。具体来说，我们发现，音频包含关键类别信息的声音生产者，和视觉数据提供候选声音生产者（S）。它们的共享信息对应于可视数据中所示的目标声音产生者。在这种情况下，跨模态共享表征学习对AVS尤为重要。为了实现这一点，我们的ECMVAE因式分解与模态共享表示和模态特定的表示每一个模态的表示。在共享表示和特定表示之间应用正交性约束，以保持因子分解的潜在代码的独占属性。此外，引入互信息最大化正则化器来实现对每个模态的广泛探索。AVSBench上的定量和定性评估证明了我们的方法的有效性，从而为AVS带来了新的最先进的技术，在具有挑战性的MS3子集上实现了3.84 mIOU的性能飞跃，用于多声源分割。摘要：We propose an Explicit Conditional Multimodal Variational Auto-Encoder (ECMVAE) for audio-visual segmentation (AVS), aiming to segment sound sources in the video sequence. Existing AVS methods focus on implicit feature fusion strategies, where models are trained to fit the discrete samples in the dataset. With a limited and less diverse dataset, the resulting performance is usually unsatisfactory. In contrast, we address this problem from an effective representation learning perspective, aiming to model the contribution of each modality explicitly. Specifically, we find that audio contains critical category information of the sound producers, and visual data provides candidate sound producer(s). Their shared information corresponds to the target sound producer(s) shown in the visual data. In this case, cross-modal shared representation learning is especially important for AVS. To achieve this, our ECMVAE factorizes the representations of each modality with a modality-shared representation and a modality-specific representation. An orthogonality constraint is applied between the shared and specific representations to maintain the exclusive attribute of the factorized latent code. Further, a mutual information maximization regularizer is introduced to achieve extensive exploration of each modality. Quantitative and qualitative evaluations on the AVSBench demonstrate the effectiveness of our approach, leading to a new state-of-the-art for AVS, with a 3.84 mIOU performance leap on the challenging MS3 subset for multiple sound source segmentation.
【8】 On the Relevance of Phoneme Duration Variability of Synthesized Training  Data for Automatic Speech Recognition标题：用于自动语音识别的合成训练数据的音素时长变异性相关性研究链接：https://arxiv.org/abs/2310.08132
作者：Nick Rossenbach,Benedikt Hilmes,Ralf Schlüter备注：To appear at ASRU 2023摘要：由文本到语音（TTS）系统生成的合成数据可以用于在低资源或域失配任务中改进自动语音识别（ASR）系统。它已被证明，TTS生成的输出仍然不具有相同的质量作为真正的数据。在这项工作中，我们专注于合成数据的时间结构及其与ASR训练的关系。通过使用一种新的Oracle设置，我们显示了在非自回归（NAR）TTS中持续时间建模对合成数据质量的影响。为了获得参考音素持续时间，我们使用两种常见的对齐方法，隐马尔可夫高斯混合模型（HMM-GMM）对齐器和神经连接主义时间分类（CTC）对齐器。使用基于随机游走的简单算法，我们将TTS系统的音素持续时间分布更接近真实持续时间，从而在半监督设置中使用合成数据改进ASR系统。摘要：Synthetic data generated by text-to-speech (TTS) systems can be used to improve automatic speech recognition (ASR) systems in low-resource or domain mismatch tasks. It has been shown that TTS-generated outputs still do not have the same qualities as real data. In this work we focus on the temporal structure of synthetic data and its relation to ASR training. By using a novel oracle setup we show how much the degradation of synthetic data quality is influenced by duration modeling in non-autoregressive (NAR) TTS. To get reference phoneme durations we use two common alignment methods, a hidden Markov Gaussian-mixture model (HMM-GMM) aligner and a neural connectionist temporal classification (CTC) aligner. Using a simple algorithm based on random walks we shift phoneme duration distributions of the TTS system closer to real durations, resulting in an improvement of an ASR system using synthetic data in a semi-supervised setting.
机器翻译，仅供参考