今日论文合集：cs.SD语音7篇，eess.AS音频处理10篇。
cs.SD语音
【1】 Style Description based Text-to-Speech with Conditional Prosodic Layer  Normalization based Diffusion GAN标题：基于扩散GAN的条件韵律层归一化基于风格描述的文语转换链接：https://arxiv.org/abs/2310.18169
作者：Neeraj Kumar,Ankur Narang,Brejesh Lall摘要：在本文中，我们提出了一种基于扩散GAN的方法（韵律Diff-TTS），以生成相应的高保真语音的基础上的风格描述和内容文本作为输入，生成语音样本，只有4个去噪步骤。它利用新颖的条件韵律层规范化将风格嵌入到基于多头注意的音素编码器和基于梅尔频谱图解码器的生成器架构中以生成语音。风格嵌入是通过对预训练的BERT模型在诸如音高、语速、情感、性别分类等辅助任务上进行微调来生成的。我们证明了我们提出的架构对多扬声器LibriTTS和LibritSpeech数据集的有效性，使用多个定量指标来衡量生成的准确性和MOS。摘要：In this paper, we present a Diffusion GAN based approach (Prosodic Diff-TTS) to generate the corresponding high-fidelity speech based on the style description and content text as an input to generate speech samples within only 4 denoising steps. It leverages the novel conditional prosodic layer normalization to incorporate the style embeddings into the multi head attention based phoneme encoder and mel spectrogram decoder based generator architecture to generate the speech. The style embedding is generated by fine tuning the pretrained BERT model on auxiliary tasks such as pitch, speaking speed, emotion,gender classifications. We demonstrate the efficacy of our proposed architecture on multi-speaker LibriTTS and PromptSpeech datasets, using multiple quantitative metrics that measure generated accuracy and MOS.
【2】 Enabling Acoustic Audience Feedback in Large Virtual Events标题：在大型虚拟活动中启用声音观众反馈链接：https://arxiv.org/abs/2310.18099
作者：Tamay Aykut,Markus Hofbauer,Christopher Kuhn,Eckehard Steinbach,Bernd Girod备注：4 pages, 2 figures摘要：COVID-19疫情将我们日常生活中的许多事件转移到虚拟领域。虽然虚拟会议系统提供了物理会议的替代方案，但大型活动需要静音观众，以避免背景噪音和失真音频的积累。然而，表演艺术家非常依赖观众的反馈。我们提出了一个虚拟观众框架的概念，它支持所有参与者与真实观众的氛围。观众反馈是在本地收集的，允许用户通过选择鼓掌、吹口哨、嘘声和笑声等方式来表达热情或不满。该反馈作为抽象信息被发送到虚拟观众服务器。我们将组合的虚拟观众反馈信息广播给所有参与者，客户端可以将其合成为单个声学反馈。合成可以通过将集体观众反馈转化为提示来完成，该提示被馈送到最先进的模型，如AudioGen。这样，每个用户听到整个虚拟事件的单个声学反馈声音，而不需要取消静音或冒着听到失真的、不同步的反馈的风险。摘要：The COVID-19 pandemic shifted many events in our daily lives into the virtual domain. While virtual conference systems provide an alternative to physical meetings, larger events require a muted audience to avoid an accumulation of background noise and distorted audio. However, performing artists strongly rely on the feedback of their audience. We propose a concept for a virtual audience framework which supports all participants with the ambience of a real audience. Audience feedback is collected locally, allowing users to express enthusiasm or discontent by selecting means such as clapping, whistling, booing, and laughter. This feedback is sent as abstract information to a virtual audience server. We broadcast the combined virtual audience feedback information to all participants, which can be synthesized as a single acoustic feedback by the client. The synthesis can be done by turning the collective audience feedback into a prompt that is fed to state-of-the-art models such as AudioGen. This way, each user hears a single acoustic feedback sound of the entire virtual event, without requiring to unmute or risk hearing distorted, unsynchronized feedback.
【3】 Whisper-MCE: Whisper Model Finetuned for Better Performance with Mixed  Languages标题：Whisper-MCE：Whisper模型经过优化，可在混合语言中获得更好的性能链接：https://arxiv.org/abs/2310.17953
作者：Peng Xie,XingYuan Liu,ZiWei Chen,Kani Chen,Yang Wang摘要：最近Whisper在英语自动语音识别（ASR）中已经接近人类水平的鲁棒性和准确性，而在小语种和混合语言语音识别中，仍然迫切需要进一步改进。在这项工作中，我们展示了Whisper-MCE的令人印象深刻的结果，我们的微调Whisper模型，它是使用我们自己收集的数据集，混合粤语和英语音频数据集（MCE）进行训练的。同时，考虑到词错误率（WER）提出的挑战，当涉及到评估其有效性在小语种和混合语言的情况下，我们提出了一种新的评级机制。通过将我们的模型与基线whisper-large-v2模型进行比较，我们证明了其准确捕获原始音频内容的卓越能力，实现了更高的识别精度，并表现出更快的识别速度。值得注意的是，我们的模型在识别混合语言的特定任务中优于其他现有模型。摘要：Recently Whisper has approached human-level robustness and accuracy in English automatic speech recognition (ASR), while in minor language and mixed language speech recognition, there remains a compelling need for further improvement. In this work, we present the impressive results of Whisper-MCE, our finetuned Whisper model, which was trained using our self-collected dataset, Mixed Cantonese and English audio dataset (MCE). Meanwhile, considering word error rate (WER) poses challenges when it comes to evaluating its effectiveness in minor language and mixed-language contexts, we present a novel rating mechanism. By comparing our model to the baseline whisper-large-v2 model, we demonstrate its superior ability to accurately capture the content of the original audio, achieve higher recognition accuracy, and exhibit faster recognition speed. Notably, our model outperforms other existing models in the specific task of recognizing mixed language.
【4】 Unified Segment-to-Segment Framework for Simultaneous Sequence  Generation标题：用于同时序列生成的统一段到段框架链接：https://arxiv.org/abs/2310.17940
作者：Shaolei Zhang,Yang Feng备注：Accepted at NeurIPS 2023摘要：同步序列生成是流媒体语音识别、同步机器翻译和同步语音翻译等实时场景中的一项关键任务，其中目标序列在接收源序列的同时生成。实现低延迟高质量生成的关键在于识别生成的最佳时刻，通过学习源序列和目标序列之间的映射来完成。然而，现有的方法往往依赖于特定于任务的算法，不同的序列类型，限制了模型的能力，自适应学习的源-目标映射，并阻碍了探索多任务学习的各种并发任务。在本文中，我们提出了一个统一的段到段的框架（Seg 2Seg）的同时序列生成，学习映射的自适应和统一的方式。在同时生成的过程中，模型在等待源段和生成目标段之间交替，使段充当源和目标之间的自然桥梁。为了实现这一点，Seg 2Seg引入了一个潜在段作为源到目标之间的枢纽，并通过所提出的期望训练来探索所有潜在的源-目标映射，从而学习生成的最佳时刻。多个同时生成任务的实验表明，Seg 2Seg实现了最先进的性能，并表现出更好的通用性在各种任务。摘要：Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.
【5】 TorchAudio 2.1: Advancing speech recognition, self-supervised learning,  and audio processing components for PyTorch标题：TorchAudio 2.1：针对PyTorch的高级语音识别、自我监督学习和音频处理组件链接：https://arxiv.org/abs/2310.17864
作者：Jeff Hwang,Moto Hira,Caroline Chen,Xiaohui Zhang,Zhaoheng Ni,Guangzhi Sun,Pingchuan Ma,Ruizhe Huang,Vineel Pratap,Yuekai Zhang,Anurag Kumar,Chin-Yun Yu,Chuang Zhu,Chunxi Liu,Jacob Kahn,Mirco Ravanelli,Peng Sun,Shinji Watanabe,Yangyang Shi,Yumeng Tao,Robin Scheibler,Samuele Cornell,Sean Kim,Stavros Petridis摘要：TorchAudio是为PyTorch构建的开源音频和语音处理库。它旨在通过提供精心设计、易于使用和高性能的PyTorch组件来加速音频和语音技术的研究和开发。它的贡献者经常与用户互动，了解他们的需求，并通过开发有影响力的功能来满足他们。在这里，我们概述了TorchAudio的开发原则和内容，并强调了我们在其最新版本（2.1）中包含的关键功能：自监督学习预训练管道和训练配方，高性能CTC解码器，语音识别模型和训练配方，高级媒体I/O功能，以及用于执行强制对齐，多通道语音增强和无参考语音评估的工具。对于这些功能的选择，通过实证研究，我们证明了它们的功效，并表明它们实现了竞争力或最先进的性能。摘要：TorchAudio is an open-source audio and speech processing library built for PyTorch. It aims to accelerate the research and development of audio and speech technologies by providing well-designed, easy-to-use, and performant PyTorch components. Its contributors routinely engage with users to understand their needs and fulfill them by developing impactful features. Here, we survey TorchAudio's development principles and contents and highlight key features we include in its latest version (2.1): self-supervised learning pre-trained pipelines and training recipes, high-performance CTC decoders, speech recognition models and training recipes, advanced media I/O capabilities, and tools for performing forced alignment, multi-channel speech enhancement, and reference-less speech assessment. For a selection of these features, through empirical studies, we demonstrate their efficacy and show that they achieve competitive or state-of-the-art performance.
【6】 Early Detection of Tuberculosis with Machine Learning Cough Audio  Analysis: Towards More Accessible Global Triaging Usage标题：利用机器学习咳嗽音频分析早期发现结核病：迈向更易获得的全球分诊使用链接：https://arxiv.org/abs/2310.17675
作者：Chandra Suda摘要：结核病（TB）是一种主要影响肺部的细菌性疾病，是全世界死亡的主要传染性原因之一。要防止结核病在体内扩散，导致危及生命的并发症，及时有效的抗结核治疗至关重要。咳嗽是结核病的一种客观生物标志物，是一种监测治疗反应和成功治疗后消退的分诊工具。目前结核病诊断的黄金标准进展缓慢或难以实现，特别是在结核病最流行的农村地区。此外，目前的机器学习（ML）诊断研究，如利用胸片，是无效的，不能监测治疗进展。为了实现有效的诊断，开发了一种集成模型，该模型使用新型ML架构分析来自智能手机麦克风的咳嗽声流行病学以检测TB。该架构包括一个2D-CNN和XGBoost，它在来自7个国家的724，964个咳嗽音频样本和人口统计数据上进行了训练。经过特征提取（Mel光谱图）和数据增强（IR卷积），该模型实现了88%的AUROC（接收操作员特征下的面积），超过了世卫组织对筛查测试的要求。结果可在15秒内获得，并可通过移动应用程序轻松访问。这项研究有助于通过有前途的准确，快速和可访问的分类工具来改善结核病诊断。摘要：Tuberculosis (TB), a bacterial disease mainly affecting the lungs, is one of the leading infectious causes of mortality worldwide. To prevent TB from spreading within the body, which causes life-threatening complications, timely and effective anti-TB treatment is crucial. Cough, an objective biomarker for TB, is a triage tool that monitors treatment response and regresses with successful therapy. Current gold standards for TB diagnosis are slow or inaccessible, especially in rural areas where TB is most prevalent. In addition, current machine learning (ML) diagnosis research, like utilizing chest radiographs, is ineffective and does not monitor treatment progression. To enable effective diagnosis, an ensemble model was developed that analyzes, using a novel ML architecture, coughs' acoustic epidemiologies from smartphones' microphones to detect TB. The architecture includes a 2D-CNN and XGBoost that was trained on 724,964 cough audio samples and demographics from 7 countries. After feature extraction (Mel-spectrograms) and data augmentation (IR-convolution), the model achieved AUROC (area under the receiving operator characteristic) of 88%, surpassing WHO's requirements for screening tests. The results are available within 15 seconds and can easily be accessible via a mobile app. This research helps to improve TB diagnosis through a promising accurate, quick, and accessible triaging tool.
【7】 Music Recommendation Based on Audio Fingerprint标题：基于音频指纹的音乐推荐链接：https://arxiv.org/abs/2310.17655
作者：Diego Saldaña Ulloa摘要：这项工作结合了不同的音频特征，以获得一个更强大的指纹，用于音乐推荐过程。这些方法的组合产生了高维向量。为了减少值的数量，PCA被应用到所得到的指纹集，选择对应于95\%$的解释方差的主成分的数量。最后，利用这些PCA指纹，计算每个指纹与整个数据集的相似度矩阵。该过程应用于个人音乐库中的200首歌曲;这些歌曲被标记为艺术家的相应流派。如果推荐歌曲的类型与目标歌曲的类型匹配，则推荐（具有最接近相似性的歌曲的指纹）被评为成功。有了这个程序，就有可能获得89%的准确性（成功的建议，总的建议请求）。摘要：This work combined different audio features to obtain a more robust fingerprint to be used in a music recommendation process. The combination of these methods resulted in a high-dimensional vector. To reduce the number of values, PCA was applied to the set of resulting fingerprints, selecting the number of principal components that corresponded to an explained variance of $95\%$. Finally, with these PCA-fingerprints, the similarity matrix of each fingerprint with the entire data set was calculated. The process was applied to 200 songs from a personal music library; the songs were tagged with the artists' corresponding genres. The recommendations (fingerprints of songs with the closest similarity) were rated successful if the recommended songs' genre matched the target songs' genre. With this procedure, it was possible to obtain an accuracy of $89\%$ (successful recommendations out of total recommendation requests).
eess.AS音频处理
【1】 Relative Transfer Function Vector Estimation for Acoustic Sensor  Networks Exploiting Covariance Matrix Structure标题：基于协方差矩阵结构的水声传感器网络相对传递函数矢量估计链接：https://arxiv.org/abs/2310.18199
作者：Wiebke Middelberg,Henri Gode,Simon Doclo备注：Proc. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), New Paltz NY, USA, Oct. 2023摘要：在许多用于降噪的多麦克风算法中，需要估计目标说话人的相对传递函数（RTF）向量。现有技术的协方差白化（CW）方法估计RTF向量作为白化的噪声协方差矩阵的主特征向量，其中使用噪声协方差矩阵的估计来执行白化。在本文中，我们考虑一个由多个麦克风节点组成的声传感器网络。假设不相关的噪声之间的节点，但不是在节点内，我们提出了两个RTF向量估计方法，利用块对角结构的噪声协方差矩阵。第一种方法通过仅考虑估计的噪声协方差矩阵的对角块来修改CW方法。相比之下，第二种方法只考虑噪声协方差矩阵的非对角块，但不能使用简单的特征值分解来解决。当在最小方差无失真响应波束形成器中应用估计的RTF向量时，在具有多个噪声源的混响环境中的真实世界记录的仿真结果表明，在SNR改善方面，修改的CW方法比CW方法执行得稍好，而非对角选择方法优于作为噪声协方差矩阵的主特征向量获得的有偏RTF向量估计。摘要：In many multi-microphone algorithms for noise reduction, an estimate of the relative transfer function (RTF) vector of the target speaker is required. The state-of-the-art covariance whitening (CW) method estimates the RTF vector as the principal eigenvector of the whitened noisy covariance matrix, where whitening is performed using an estimate of the noise covariance matrix. In this paper, we consider an acoustic sensor network consisting of multiple microphone nodes. Assuming uncorrelated noise between the nodes but not within the nodes, we propose two RTF vector estimation methods that leverage the block-diagonal structure of the noise covariance matrix. The first method modifies the CW method by considering only the diagonal blocks of the estimated noise covariance matrix. In contrast, the second method only considers the off-diagonal blocks of the noisy covariance matrix, but cannot be solved using a simple eigenvalue decomposition. When applying the estimated RTF vector in a minimum variance distortionless response beamformer, simulation results for real-world recordings in a reverberant environment with multiple noise sources show that the modified CW method performs slightly better than the CW method in terms of SNR improvement, while the off-diagonal selection method outperforms a biased RTF vector estimate obtained as the principal eigenvector of the noisy covariance matrix.
【2】 TorchAudio 2.1: Advancing speech recognition, self-supervised learning,  and audio processing components for PyTorch标题：TorchAudio 2.1：针对PyTorch的高级语音识别、自我监督学习和音频处理组件链接：https://arxiv.org/abs/2310.17864
作者：Jeff Hwang,Moto Hira,Caroline Chen,Xiaohui Zhang,Zhaoheng Ni,Guangzhi Sun,Pingchuan Ma,Ruizhe Huang,Vineel Pratap,Yuekai Zhang,Anurag Kumar,Chin-Yun Yu,Chuang Zhu,Chunxi Liu,Jacob Kahn,Mirco Ravanelli,Peng Sun,Shinji Watanabe,Yangyang Shi,Yumeng Tao,Robin Scheibler,Samuele Cornell,Sean Kim,Stavros Petridis摘要：TorchAudio是为PyTorch构建的开源音频和语音处理库。它旨在通过提供精心设计、易于使用和高性能的PyTorch组件来加速音频和语音技术的研究和开发。它的贡献者经常与用户互动，了解他们的需求，并通过开发有影响力的功能来满足他们。在这里，我们概述了TorchAudio的开发原则和内容，并强调了我们在其最新版本（2.1）中包含的关键功能：自监督学习预训练管道和训练配方，高性能CTC解码器，语音识别模型和训练配方，高级媒体I/O功能，以及用于执行强制对齐，多通道语音增强和无参考语音评估的工具。对于这些功能的选择，通过实证研究，我们证明了它们的功效，并表明它们实现了竞争力或最先进的性能。摘要：TorchAudio is an open-source audio and speech processing library built for PyTorch. It aims to accelerate the research and development of audio and speech technologies by providing well-designed, easy-to-use, and performant PyTorch components. Its contributors routinely engage with users to understand their needs and fulfill them by developing impactful features. Here, we survey TorchAudio's development principles and contents and highlight key features we include in its latest version (2.1): self-supervised learning pre-trained pipelines and training recipes, high-performance CTC decoders, speech recognition models and training recipes, advanced media I/O capabilities, and tools for performing forced alignment, multi-channel speech enhancement, and reference-less speech assessment. For a selection of these features, through empirical studies, we demonstrate their efficacy and show that they achieve competitive or state-of-the-art performance.
【3】 BERT-PIN: A BERT-based Framework for Recovering Missing Data Segments in  Time-series Load Profiles标题：BERT-PIN：一种基于BERT的时序负荷剖面缺失数据段恢复框架链接：https://arxiv.org/abs/2310.17742
作者：Yi Hu,Kai Ye,Hyeonjin Kim,Ning Lu摘要：受Transformer模型在自然语言处理和计算机视觉中的成功启发，本文介绍了BERT-PIN，一种基于Transformers（BERT）的轮廓修复网络。BERT-PIN使用负载和温度时间序列曲线作为输入来恢复多个丢失的数据段（MDS）。为了采用标准的Transformer模型结构进行轮廓修复，我们将负载和温度轮廓分割成线段，将每个线段视为一个单词，将整个轮廓视为一个句子。我们在BERT-PIN中引入了一个顶级候选人选择过程，使其能够产生一系列概率分布，基于此，用户可以生成多个合理的估算数据集，每个数据集都反映了不同的置信水平。我们开发和评估BERT-PIN使用现实世界的数据集为两个应用程序：多个MDS恢复和需求响应基线估计。仿真结果表明，BERT-PIN算法在准确性上优于现有方法，同时能够在较长的窗口内恢复多个MDS。BERT-PIN作为一个预先训练的模型，可以进行微调，以执行许多下游任务，如分类和超分辨率。摘要：Inspired by the success of the Transformer model in natural language processing and computer vision, this paper introduces BERT-PIN, a Bidirectional Encoder Representations from Transformers (BERT) powered Profile Inpainting Network. BERT-PIN recovers multiple missing data segments (MDSs) using load and temperature time-series profiles as inputs. To adopt a standard Transformer model structure for profile inpainting, we segment the load and temperature profiles into line segments, treating each segment as a word and the entire profile as a sentence. We incorporate a top candidates selection process in BERT-PIN, enabling it to produce a sequence of probability distributions, based on which users can generate multiple plausible imputed data sets, each reflecting different confidence levels. We develop and evaluate BERT-PIN using real-world dataset for two applications: multiple MDSs recovery and demand response baseline estimation. Simulation results show that BERT-PIN outperforms the existing methods in accuracy while is capable of restoring multiple MDSs within a longer window. BERT-PIN, served as a pre-trained model, can be fine-tuned for conducting many downstream tasks, such as classification and super resolution.
【4】 Early Detection of Tuberculosis with Machine Learning Cough Audio  Analysis: Towards More Accessible Global Triaging Usage标题：利用机器学习咳嗽音频分析早期发现结核病：迈向更易获得的全球分诊使用链接：https://arxiv.org/abs/2310.17675
作者：Chandra Suda摘要：结核病（TB）是一种主要影响肺部的细菌性疾病，是全世界死亡的主要传染性原因之一。要防止结核病在体内扩散，导致危及生命的并发症，及时有效的抗结核治疗至关重要。咳嗽是结核病的一种客观生物标志物，是一种监测治疗反应和成功治疗后消退的分诊工具。目前结核病诊断的黄金标准进展缓慢或难以实现，特别是在结核病最流行的农村地区。此外，目前的机器学习（ML）诊断研究，如利用胸片，是无效的，不能监测治疗进展。为了实现有效的诊断，开发了一种集成模型，该模型使用新型ML架构分析来自智能手机麦克风的咳嗽声流行病学以检测TB。该架构包括一个2D-CNN和XGBoost，它在来自7个国家的724，964个咳嗽音频样本和人口统计数据上进行了训练。经过特征提取（Mel光谱图）和数据增强（IR卷积），该模型实现了88%的AUROC（接收操作员特征下的面积），超过了世卫组织对筛查测试的要求。结果可在15秒内获得，并可通过移动应用程序轻松访问。这项研究有助于通过有前途的准确，快速和可访问的分类工具来改善结核病诊断。摘要：Tuberculosis (TB), a bacterial disease mainly affecting the lungs, is one of the leading infectious causes of mortality worldwide. To prevent TB from spreading within the body, which causes life-threatening complications, timely and effective anti-TB treatment is crucial. Cough, an objective biomarker for TB, is a triage tool that monitors treatment response and regresses with successful therapy. Current gold standards for TB diagnosis are slow or inaccessible, especially in rural areas where TB is most prevalent. In addition, current machine learning (ML) diagnosis research, like utilizing chest radiographs, is ineffective and does not monitor treatment progression. To enable effective diagnosis, an ensemble model was developed that analyzes, using a novel ML architecture, coughs' acoustic epidemiologies from smartphones' microphones to detect TB. The architecture includes a 2D-CNN and XGBoost that was trained on 724,964 cough audio samples and demographics from 7 countries. After feature extraction (Mel-spectrograms) and data augmentation (IR-convolution), the model achieved AUROC (area under the receiving operator characteristic) of 88%, surpassing WHO's requirements for screening tests. The results are available within 15 seconds and can easily be accessible via a mobile app. This research helps to improve TB diagnosis through a promising accurate, quick, and accessible triaging tool.
【5】 Music Recommendation Based on Audio Fingerprint标题：基于音频指纹的音乐推荐链接：https://arxiv.org/abs/2310.17655
作者：Diego Saldaña Ulloa摘要：这项工作结合了不同的音频特征，以获得一个更强大的指纹，用于音乐推荐过程。这些方法的组合产生了高维向量。为了减少值的数量，PCA被应用到所得到的指纹集，选择对应于95\%$的解释方差的主成分的数量。最后，利用这些PCA指纹，计算每个指纹与整个数据集的相似度矩阵。该过程应用于个人音乐库中的200首歌曲;这些歌曲被标记为艺术家的相应流派。如果推荐歌曲的类型与目标歌曲的类型匹配，则推荐（具有最接近相似性的歌曲的指纹）被评为成功。有了这个程序，就有可能获得89%的准确性（成功的建议，总的建议请求）。摘要：This work combined different audio features to obtain a more robust fingerprint to be used in a music recommendation process. The combination of these methods resulted in a high-dimensional vector. To reduce the number of values, PCA was applied to the set of resulting fingerprints, selecting the number of principal components that corresponded to an explained variance of $95\%$. Finally, with these PCA-fingerprints, the similarity matrix of each fingerprint with the entire data set was calculated. The process was applied to 200 songs from a personal music library; the songs were tagged with the artists' corresponding genres. The recommendations (fingerprints of songs with the closest similarity) were rated successful if the recommended songs' genre matched the target songs' genre. With this procedure, it was possible to obtain an accuracy of $89\%$ (successful recommendations out of total recommendation requests).
【6】 Style Description based Text-to-Speech with Conditional Prosodic Layer  Normalization based Diffusion GAN标题：基于扩散GAN的条件韵律层归一化基于风格描述的文语转换链接：https://arxiv.org/abs/2310.18169
作者：Neeraj Kumar,Ankur Narang,Brejesh Lall摘要：在本文中，我们提出了一种基于扩散GAN的方法（韵律Diff-TTS），以生成相应的高保真语音的基础上的风格描述和内容文本作为输入，生成语音样本，只有4个去噪步骤。它利用新颖的条件韵律层规范化将风格嵌入到基于多头注意的音素编码器和基于梅尔频谱图解码器的生成器架构中以生成语音。风格嵌入是通过对预训练的BERT模型在诸如音高、语速、情感、性别分类等辅助任务上进行微调来生成的。我们证明了我们提出的架构对多扬声器LibriTTS和LibritSpeech数据集的有效性，使用多个定量指标来衡量生成的准确性和MOS。摘要：In this paper, we present a Diffusion GAN based approach (Prosodic Diff-TTS) to generate the corresponding high-fidelity speech based on the style description and content text as an input to generate speech samples within only 4 denoising steps. It leverages the novel conditional prosodic layer normalization to incorporate the style embeddings into the multi head attention based phoneme encoder and mel spectrogram decoder based generator architecture to generate the speech. The style embedding is generated by fine tuning the pretrained BERT model on auxiliary tasks such as pitch, speaking speed, emotion,gender classifications. We demonstrate the efficacy of our proposed architecture on multi-speaker LibriTTS and PromptSpeech datasets, using multiple quantitative metrics that measure generated accuracy and MOS.
【7】 Enabling Acoustic Audience Feedback in Large Virtual Events标题：在大型虚拟活动中启用声音观众反馈链接：https://arxiv.org/abs/2310.18099
作者：Tamay Aykut,Markus Hofbauer,Christopher Kuhn,Eckehard Steinbach,Bernd Girod备注：4 pages, 2 figures摘要：COVID-19疫情将我们日常生活中的许多事件转移到虚拟领域。虽然虚拟会议系统提供了物理会议的替代方案，但大型活动需要静音观众，以避免背景噪音和失真音频的积累。然而，表演艺术家非常依赖观众的反馈。我们提出了一个虚拟观众框架的概念，它支持所有参与者与真实观众的氛围。观众反馈是在本地收集的，允许用户通过选择鼓掌、吹口哨、嘘声和笑声等方式来表达热情或不满。该反馈作为抽象信息被发送到虚拟观众服务器。我们将组合的虚拟观众反馈信息广播给所有参与者，客户端可以将其合成为单个声学反馈。合成可以通过将集体观众反馈转化为提示来完成，该提示被馈送到最先进的模型，如AudioGen。这样，每个用户听到整个虚拟事件的单个声学反馈声音，而不需要取消静音或冒着听到失真的、不同步的反馈的风险。摘要：The COVID-19 pandemic shifted many events in our daily lives into the virtual domain. While virtual conference systems provide an alternative to physical meetings, larger events require a muted audience to avoid an accumulation of background noise and distorted audio. However, performing artists strongly rely on the feedback of their audience. We propose a concept for a virtual audience framework which supports all participants with the ambience of a real audience. Audience feedback is collected locally, allowing users to express enthusiasm or discontent by selecting means such as clapping, whistling, booing, and laughter. This feedback is sent as abstract information to a virtual audience server. We broadcast the combined virtual audience feedback information to all participants, which can be synthesized as a single acoustic feedback by the client. The synthesis can be done by turning the collective audience feedback into a prompt that is fed to state-of-the-art models such as AudioGen. This way, each user hears a single acoustic feedback sound of the entire virtual event, without requiring to unmute or risk hearing distorted, unsynchronized feedback.
【8】 Whisper-MCE: Whisper Model Finetuned for Better Performance with Mixed  Languages标题：Whisper-MCE：Whisper模型微调，以提高混合语言的性能链接：https://arxiv.org/abs/2310.17953
作者：Peng Xie,XingYuan Liu,ZiWei Chen,Kani Chen,Yang Wang摘要：最近Whisper在英语自动语音识别（ASR）中已经接近人类水平的鲁棒性和准确性，而在小语种和混合语言语音识别中，仍然迫切需要进一步改进。在这项工作中，我们展示了Whisper-MCE的令人印象深刻的结果，我们的微调Whisper模型，它是使用我们自己收集的数据集，混合粤语和英语音频数据集（MCE）进行训练的。同时，考虑到词错误率（WER）提出的挑战，当涉及到评估其有效性在小语种和混合语言的情况下，我们提出了一种新的评级机制。通过将我们的模型与基线whisper-large-v2模型进行比较，我们证明了其准确捕获原始音频内容的卓越能力，实现了更高的识别精度，并表现出更快的识别速度。值得注意的是，我们的模型在识别混合语言的特定任务中优于其他现有模型。摘要：Recently Whisper has approached human-level robustness and accuracy in English automatic speech recognition (ASR), while in minor language and mixed language speech recognition, there remains a compelling need for further improvement. In this work, we present the impressive results of Whisper-MCE, our finetuned Whisper model, which was trained using our self-collected dataset, Mixed Cantonese and English audio dataset (MCE). Meanwhile, considering word error rate (WER) poses challenges when it comes to evaluating its effectiveness in minor language and mixed-language contexts, we present a novel rating mechanism. By comparing our model to the baseline whisper-large-v2 model, we demonstrate its superior ability to accurately capture the content of the original audio, achieve higher recognition accuracy, and exhibit faster recognition speed. Notably, our model outperforms other existing models in the specific task of recognizing mixed language.
【9】 Unified Segment-to-Segment Framework for Simultaneous Sequence  Generation标题：用于同时序列生成的统一段到段框架链接：https://arxiv.org/abs/2310.17940
作者：Shaolei Zhang,Yang Feng备注：Accepted at NeurIPS 2023摘要：同步序列生成是流媒体语音识别、同步机器翻译和同步语音翻译等实时场景中的一项关键任务，其中目标序列在接收源序列的同时生成。实现低延迟高质量生成的关键在于识别生成的最佳时刻，通过学习源序列和目标序列之间的映射来完成。然而，现有的方法往往依赖于特定于任务的算法，不同的序列类型，限制了模型的能力，自适应学习的源-目标映射，并阻碍了探索多任务学习的各种并发任务。在本文中，我们提出了一个统一的段到段的框架（Seg 2Seg）的同时序列生成，学习映射的自适应和统一的方式。在同时生成的过程中，模型在等待源段和生成目标段之间交替，使段充当源和目标之间的自然桥梁。为了实现这一点，Seg 2Seg引入了一个潜在段作为源到目标之间的枢纽，并通过所提出的期望训练来探索所有潜在的源-目标映射，从而学习生成的最佳时刻。多个同时生成任务的实验表明，Seg 2Seg实现了最先进的性能，并表现出更好的通用性在各种任务。摘要：Simultaneous sequence generation is a pivotal task for real-time scenarios, such as streaming speech recognition, simultaneous machine translation and simultaneous speech translation, where the target sequence is generated while receiving the source sequence. The crux of achieving high-quality generation with low latency lies in identifying the optimal moments for generating, accomplished by learning a mapping between the source and target sequences. However, existing methods often rely on task-specific heuristics for different sequence types, limiting the model's capacity to adaptively learn the source-target mapping and hindering the exploration of multi-task learning for various simultaneous tasks. In this paper, we propose a unified segment-to-segment framework (Seg2Seg) for simultaneous sequence generation, which learns the mapping in an adaptive and unified manner. During the process of simultaneous generation, the model alternates between waiting for a source segment and generating a target segment, making the segment serve as the natural bridge between the source and target. To accomplish this, Seg2Seg introduces a latent segment as the pivot between source to target and explores all potential source-target mappings via the proposed expectation training, thereby learning the optimal moments for generating. Experiments on multiple simultaneous generation tasks demonstrate that Seg2Seg achieves state-of-the-art performance and exhibits better generality across various tasks.
【10】 Cascaded Multi-task Adaptive Learning Based on Neural Architecture  Search标题：基于神经结构搜索的级联多任务自适应学习链接：https://arxiv.org/abs/2310.17664
作者：Yingying Gao,Shilei Zhang,Zihao Cui,Chao Deng,Junlan Feng摘要：级联多个预训练模型是组成端到端系统的有效方法。然而，微调完整的级联模型是参数和内存效率低下，我们的观察表明，仅应用适配器模块级联模型不能实现相当的性能作为微调。提出了一种基于神经结构搜索（NAS）框架的端到端级联多任务模型自动有效的自适应学习优化方法。每个特定模块上的候选自适应操作包括冻结、插入适配器和微调。我们还在损失上添加了一个惩罚项，以限制学习的结构，该结构考虑了可训练参数的数量。惩罚项成功地限制了搜索的结构，所提出的方法能够搜索类似的调整方案与手工，压缩优化参数到8.7%，对应于充分微调SLURP具有更好的性能。摘要：Cascading multiple pre-trained models is an effective way to compose an end-to-end system. However, fine-tuning the full cascaded model is parameter and memory inefficient and our observations reveal that only applying adapter modules on cascaded model can not achieve considerable performance as fine-tuning. We propose an automatic and effective adaptive learning method to optimize end-to-end cascaded multi-task models based on Neural Architecture Search (NAS) framework. The candidate adaptive operations on each specific module consist of frozen, inserting an adapter and fine-tuning. We further add a penalty item on the loss to limit the learned structure which takes the amount of trainable parameters into account. The penalty item successfully restrict the searched architecture and the proposed approach is able to search similar tuning scheme with hand-craft, compressing the optimizing parameters to 8.7% corresponding to full fine-tuning on SLURP with an even better performance.
机器翻译，仅供参考