今日论文合集：cs.SD语音12篇，eess.AS音频处理16篇。【1】 Blind Restoration of Real-World Audio by 1D Operational GANs
标题：一维操作GANS实现真实世界音频的盲恢复
链接：https://arxiv.org/abs/2212.14618
作者：Turker Ince,Serkan Kiranyaz,Ozer Can Devecioglu,Muhammad Salman Khan,Muhammad Chowdhury,Moncef Gabbouj摘要：目的：尽管文献中提出了许多关于音频恢复的研究，但是大多数研究集中于诸如去噪或去混响等孤立的恢复问题，而忽略了其他伪像。此外，假设具有有限数目的固定信号失真比（SDR）电平的噪声或混响环境是常见的做法。然而，真实世界的音频经常被诸如混响、传感器噪声和具有变化类型、严重性和持续时间的背景音频混合的伪像的混合破坏。在本研究中，我们提出了一种新的方法，通过运算生成式对抗网络（Op-GANs）对真实世界的音频信号进行盲恢复，该方法具有时间和频谱客观度量，以提高恢复的音频信号的质量，而不管破坏它的每个伪像的类型和严重程度。1D操作-GAN与针对任何损坏音频信号的盲恢复而优化的生成神经元模型一起使用。结果：该方法在TIMIT-RAR（语音）和GTZAN-RAR（非语音）数据集上进行了广泛的评估，这些数据集被随机混合的伪影污染，每个伪影都具有随机的严重程度，以模拟真实世界的音频信号。分别实现了超过7.2dB和4.9dB的平均SDR改善，当与基线方法相比时，这是实质性的。意义：这是盲音频恢复领域的先驱研究，具有直接（时域）恢复真实世界音频的独特能力，同时针对广泛的SDR范围和伪像类型实现前所未有的性能水平。结论：1D Op-GANs可以实现鲁棒的、计算有效的真实世界音频恢复，并且性能显著提高。源代码和生成的真实世界音频数据集将在专用的GitHub存储库1中与研究社区公开共享。摘要：Objective: Despite numerous studies proposed for audio restoration in the literature, most of them focus on an isolated restoration problem such as denoising or dereverberation, ignoring other artifacts. Moreover, assuming a noisy or reverberant environment with limited number of fixed signal-to-distortion ratio (SDR) levels is a common practice. However, real-world audio is often corrupted by a blend of artifacts such as reverberation, sensor noise, and background audio mixture with varying types, severities, and duration. In this study, we propose a novel approach for blind restoration of real-world audio signals by Operational Generative Adversarial Networks (Op-GANs) with temporal and spectral objective metrics to enhance the quality of restored audio signal regardless of the type and severity of each artifact corrupting it. Methods: 1D Operational-GANs are used with generative neuron model optimized for blind restoration of any corrupted audio signal. Results: The proposed approach has been evaluated extensively over the benchmark TIMIT-RAR (speech) and GTZAN-RAR (non-speech) datasets corrupted with a random blend of artifacts each with a random severity to mimic real-world audio signals. Average SDR improvements of over 7.2 dB and 4.9 dB are achieved, respectively, which are substantial when compared with the baseline methods. Significance: This is a pioneer study in blind audio restoration with the unique capability of direct (time-domain) restoration of real-world audio whilst achieving an unprecedented level of performance for a wide SDR range and artifact types. Conclusion: 1D Op-GANs can achieve robust and computationally effective real-world audio restoration with significantly improved performance. The source codes and the generated real-world audio datasets are shared publicly with the research community in a dedicated GitHub repository1.
【2】 Defense Against Adversarial Attacks on Audio DeepFake Detection
标题：音频DeepFake检测中的敌意攻击防御
链接：https://arxiv.org/abs/2212.14597
作者：Piotr Kawa,Marcin Plata,Piotr Syga机构：Department of Artificial Intelligence, Wrocław University of Science and Technology, Wrocław, Poland备注：Submitted to ICASSP 2023摘要：音频DeepFakes是使用深度学习方法创建的人工生成的话语，主要目的是愚弄听众，大多数这样的音频都具有高度的说服力。它们的质量足以在安全和隐私方面构成严重威胁，例如新闻的可靠性或诽谤。为了防止这些威胁，已经提出了多种基于神经网络的方法来检测生成的语音。在本文中，我们讨论了对抗性攻击的主题，它通过向输入数据添加表面（人类难以发现）更改来降低检测器的性能。我们的贡献包括评估3种检测结构在两种场景（白盒和使用可转移性机制）下抵抗对抗性攻击的鲁棒性，并随后通过使用由我们的新的自适应训练方法执行的对抗性训练来增强它。摘要：Audio DeepFakes are artificially generated utterances created using deep learning methods with the main aim to fool the listeners, most of such audio is highly convincing. Their quality is sufficient to pose a serious threat in terms of security and privacy, such as the reliability of news or defamation. To prevent the threats, multiple neural networks-based methods to detect generated speech have been proposed. In this work, we cover the topic of adversarial attacks, which decrease the performance of detectors by adding superficial (difficult to spot by a human) changes to input data. Our contribution contains evaluating the robustness of 3 detection architectures against adversarial attacks in two scenarios (white-box and using transferability mechanism) and enhancing it later by the use of adversarial training performed by our novel adaptive training method.
【3】 Multi-modal deep learning system for depression and anxiety detection
标题：用于抑郁和焦虑检测的多模式深度学习系统
链接：https://arxiv.org/abs/2212.14490
作者：Brian Diep,Marija Stanojevic,Jekaterina Novikova机构：Department of Computer Science, University of Toronto, Toronto, ON, Winterlight Labs备注：accepted to the PAI4MH workshop at NeurIPS 2022摘要：传统的焦虑和抑郁症筛查方法阻碍了对这些疾病的有效监测和治疗。然而，NLP和语音建模的最新进展使得基于文本、声学和手工语言的特征能够共同形成未来心理健康筛查和状况检测的基础。言语是洞察个体认知状态的丰富而现成的来源，通过利用言语的不同方面，我们可以开发新的抑郁和焦虑的数字生物标志物。为此，我们提出了一个多模态系统，用于筛选抑郁和焦虑的自我管理的言语任务。所提出的模型集成了来自音频和文本的深度学习特征，以及由临床验证的领域知识所告知的手工特征。我们发现，与单独的手工特征的基线相比，用深度学习特征增强手工特征提高了我们的总体分类F1得分，抑郁从0.58提高到0.63，焦虑从0.54提高到0.57。我们的研究结果表明，抑郁症和焦虑症的语音生物标志物在数字健康的未来有着重要的前景。摘要：Traditional screening practices for anxiety and depression pose an impediment to monitoring and treating these conditions effectively. However, recent advances in NLP and speech modelling allow textual, acoustic, and hand-crafted language-based features to jointly form the basis of future mental health screening and condition detection. Speech is a rich and readily available source of insight into an individual's cognitive state and by leveraging different aspects of speech, we can develop new digital biomarkers for depression and anxiety. To this end, we propose a multi-modal system for the screening of depression and anxiety from self-administered speech tasks. The proposed model integrates deep-learned features from audio and text, as well as hand-crafted features that are informed by clinically-validated domain knowledge. We find that augmenting hand-crafted features with deep-learned features improves our overall classification F1 score comparing to a baseline of hand-crafted features alone from 0.58 to 0.63 for depression and from 0.54 to 0.57 for anxiety. The findings of our work suggest that speech-based biomarkers for depression and anxiety hold significant promise in the future of digital health.
【4】 Macro-block dropout for improved regularization in training end-to-end speech recognition models
标题：用于端到端语音识别模型训练中改进正则化的宏块丢失
链接：https://arxiv.org/abs/2212.14149
作者：Chanwoo Kim,Sathish Indurti,Jinhwan Park,Wonyong Sung机构：Samsung Research, Seoul, South Korea, Seoul National University, Seoul, South Korea备注：Accepted for presentation at The 2022 IEEE Spoken Language Technology Workshop (SLT 2022)摘要：本文提出了一种新的正则化算法--宏块丢失算法。过拟合问题一直是大型神经网络模型训练中的难题。辍学技术已被证明是简单的，但非常有效的正则化，防止复杂的共同适应期间的训练。在我们的工作中，我们定义了一个宏块，它包含了从输入到递归神经网络（RNN）的大量单元。不是对每个单元应用丢弃，而是对每个宏块应用随机丢弃。该算法具有即使我们保持恒定的平均丢弃率也对每层应用不同的丢弃率的效果，这具有更好的正则化效果。在我们使用递归神经网络-转换器（RNN-T）的实验中，该算法在LibriSpeech测试-清除和测试-其他上的字错误率（WER）比传统的丢失率分别提高了4.30%和6.13%。在基于注意力的编解码器（AED）模型下，该算法在相同测试集上的丢包率分别比传统算法提高了4.36%和5.85%。摘要：This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large neural network models. The dropout technique has proven to be simple yet very effective for regularization by preventing complex co-adaptations during training. In our work, we define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN). Rather than applying dropout to each unit, we apply random dropout to each macro-block. This algorithm has the effect of applying different drop out rates for each layer even if we keep a constant average dropout rate, which has better regularization effects. In our experiments using Recurrent Neural Network-Transducer (RNN-T), this algorithm shows relatively 4.30 % and 6.13 % Word Error Rates (WERs) improvement over the conventional dropout on LibriSpeech test-clean and test-other. With an Attention-based Encoder-Decoder (AED) model, this algorithm shows relatively 4.36 % and 5.85 % WERs improvement over the conventional dropout on the same test sets.
【5】 ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
标题：ResGrad：文语转换的残差去噪扩散概率模型
链接：https://arxiv.org/abs/2212.14518
作者：Zehua Chen,Yihan Wu,Yichong Leng,Jiawei Chen,Haohe Liu,Xu Tan,Yang Cui,Ke Wang,Lei He,Sheng Zhao,Jiang Bian,Danilo Mandic机构：4Renmin University of China, 5Universityof Science and Technology of China, 6South China University of Technology, 7University of Surrey†Work done during an internship at Microsoft摘要：去噪扩散概率模型（DDPMs）由于其产生高保真样本的能力较强，在文语转换（TTS）合成中得到了广泛的应用。然而，由于其在高维数据空间中的迭代求精过程导致推理速度慢，限制了其在实时系统中的应用。先前的工作已经探索了通过最小化推理步骤的数量来加速，但是以样本质量为代价。在本文中，为了提高基于DDPM的TTS模型的推理速度，同时获得高样本质量，我们提出了ResGrad，这是一种轻量级扩散模型，它学习精炼现有TTS模型的输出谱图（例如，FastSpeech 2），通过预测模型输出和相应地面实况语音之间的残差。ResGrad有几个优点：1）与其他需要从头开始合成语音的DDPM加速方法相比，ResGrad通过将生成目标从地面实况梅尔谱图改为残差，降低了任务的复杂度，使得模型更加轻量级，从而降低了实时性。2)ResGrad以即插即用的方式应用于已有TTS模型的推理过程中，不需要重新训练该模型。我们在单说话者数据集LJSpeech和两个更具挑战性的多说话者（LibriTTS）和高采样率（VCTK）数据集上验证了ResGrad。实验结果表明，与其他DDPM加速方法相比：（1）ResGrad算法在保证实时性的前提下获得了更好的样本质量; 2）在语音质量相似的情况下，ResGrad合成语音的速度比基线方法快10倍以上。音频样本可从www.example.com获得https://resgrad1.github.io/。摘要：Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real-time systems. Previous works have explored speeding up by minimizing the number of inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to refine the output spectrogram of an existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground-truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad reduces the complexity of task by changing the generation target from ground-truth mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in a plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challenging datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods of DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at https://resgrad1.github.io/.
【6】 StyleTTS-VC: One-Shot Voice Conversion by Knowledge Transfer from Style-Based TTS Models
标题：StyleTTS-VC：通过基于风格的TTS模型的知识转移实现一次语音转换
链接：https://arxiv.org/abs/2212.14227
作者：Yinghao Aaron Li,Cong Han,Nima Mesgarani机构：Department of Electrical Engineering, Columbia University, USA摘要：单次语音转换（One-shot voice conversion，VC）的目的是利用来自目标说话人的仅几秒的参考语音将来自任意源说话人的语音转换成任意目标说话人。这在很大程度上取决于如何理清说话者的身份和说话内容，这一任务仍然具有挑战性。在此，我们提出了一种新的方法来学习去纠缠的语音表示，通过迁移学习从风格为基础的文本到语音（TTS）模型。通过循环一致性和对抗性训练，基于风格的TTS模型能够以高保真度和相似度执行转录引导的一次VC。通过教师-学生知识转移和新颖的数据扩充方案学习额外的梅尔谱图编码器，我们的方法在不需要输入文本的情况下产生解纠缠的语音表示。主观评价结果表明，该方法在自然度和相似度方面均优于现有的单次语音转换模型。摘要：One-shot voice conversion (VC) aims to convert speech from any source speaker to an arbitrary target speaker with only a few seconds of reference speech from the target speaker. This relies heavily on disentangling the speaker's identity and speech content, a task that still remains challenging. Here, we propose a novel approach to learning disentangled speech representation by transfer learning from style-based text-to-speech (TTS) models. With cycle consistent and adversarial training, the style-based TTS models can perform transcription-guided one-shot VC with high fidelity and similarity. By learning an additional mel-spectrogram encoder through a teacher-student knowledge transfer and novel data augmentation scheme, our approach results in disentangled speech representation without needing the input text. The subjective evaluation shows that our approach can significantly outperform the previous state-of-the-art one-shot voice conversion models in both naturalness and similarity.
【7】 Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling
标题：用于口语建模的离散自监督语音表示分析
链接：https://arxiv.org/abs/2301.00591
作者：Amitay Sicherman,Yossi Adi机构：School of Engineering and Computer Science, The Hebrew University of Jerusalem, Israel摘要：本文从生成式口语建模（GSLM）的角度深入分析了离散的自监督语音表示。根据这种分析的结果，我们对GSLM的分立单元提出了实际的改进。首先，我们从三个方面来分析这些单位：解释、可视化和再合成。我们的分析发现，语音单位与音素和音素族之间有很高的相关性，而与说话人和性别的相关性较弱。另外，我们发现在提取的单元中存在冗余，并声称其中一个原因可能是单元的上下文。在此基础上，我们提出了一种新的无监督度量单位冗余度的方法。最后，我们使用这个度量来开发新的方法，提高单元聚类的鲁棒性，并在考虑零资源语音度量（如ABX）时显示出显著的改进。代码和分析工具可通过以下链接获得。摘要：This work profoundly analyzes discrete self-supervised speech representations through the eyes of Generative Spoken Language Modeling (GSLM). Following the findings of such an analysis, we propose practical improvements to the discrete unit for the GSLM. First, we start comprehending these units by analyzing them in three axes: interpretation, visualization, and resynthesis. Our analysis finds a high correlation between the speech units to phonemes and phoneme families, while their correlation with speaker or gender is weaker. Additionally, we found redundancies in the extracted units and claim that one reason may be the units' context. Following this analysis, we propose a new, unsupervised metric to measure unit redundancies. Finally, we use this metric to develop new methods that improve the robustness of units clustering and show significant improvement considering zero-resource speech metrics such as ABX. Code and analysis tools are available under the following link.
【8】 EmoGator: A New Open Source Vocal Burst Dataset with Baseline Machine Learning Classification Methodologies
标题：EmoGator：一种新的具有基线机器学习分类方法的开源发声数据集
链接：https://arxiv.org/abs/2301.00508
备注：12 pages, 4 tables, 2 figures摘要：语音爆发--表达情感的短的非语音发声，例如笑、哭、叹息、呻吟和呻吟--是语音情感识别中经常被忽视的一个方面，但却是人类语音交流的一个重要方面。研究这些有趣的发声的一个障碍是缺乏大型数据集。我很高兴地介绍EmoGator数据集，它由来自365个扬声器的32,040个样本组成，16.91小时的音频;每个样本被说话者分类到30个不同情感类别中的一个。本文将讨论几种不同的构造分类器来识别情感类别的方法，并对未来的研究方向提出建议。数据集可从www.example.com下载https://github.com/fredbuhl/EmoGator。摘要：Vocal Bursts -- short, non-speech vocalizations that convey emotions, such as laughter, cries, sighs, moans, and groans -- are an often-overlooked aspect of speech emotion recognition, but an important aspect of human vocal communication. One barrier to study of these interesting vocalizations is a lack of large datasets. I am pleased to introduce the EmoGator dataset, which consists of 32,040 samples from 365 speakers, 16.91 hours of audio; each sample classified into one of 30 distinct emotion categories by the speaker. Several different approaches to construct classifiers to identify emotion categories will be discussed, and directions for future research will be suggested. Data set is available for download from https://github.com/fredbuhl/EmoGator.
【9】 Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek
标题：语音识别系统的样本高效无监督域自适应--以现代希腊语为例
链接：https://arxiv.org/abs/2301.00304
作者：Georgios Paraskevopoulos,Theodoros Kouzelis,Georgios Rouvalis,Athanasios Katsamanis,Vassilis Katsouros,Alexandros Potamianos机构：Katsourosare with the Institute for Speech and Language Processing摘要：现代语音识别系统在域移动下表现出快速的性能下降。这个问题在数据稀缺的环境中尤其普遍，例如资源匮乏的语言，在那里训练数据的多样性有限。在本文中，我们提出了M2 DS 2，一个简单的和样本有效的微调策略，为大型预训练语音模型，基于混合源和目标域自监督。我们发现，包括源域自我监督稳定训练和避免潜在表征的模式崩溃。为了评估，我们收集了HParl，一个120美元小时的希腊语演讲语料库，包括希腊议会的全体会议。我们将HParl与两个流行的希腊语语料库合并，创建了GREC-MD，这是一个用于希腊语ASR系统多领域评估的测试平台。在我们的实验中，我们发现，虽然其他无监督域自适应基线在这种资源受限的环境中失败，但M2 DS 2产生了跨域自适应的显著改进，即使当只有几个小时的域内音频可用时。当我们在弱监督设置中放松该问题时，我们发现使用M2 DS 2的音频和使用简单LM增强技术的语言的独立自适应特别有效，产生与完全监督基线相当的单词错误率。摘要：Modern speech recognition systems exhibits rapid performance degradation under domain shift. This issue is especially prevalent in data-scarce settings, such as low-resource languages, where diversity of training data is limited. In this work we propose M2DS2, a simple and sample-efficient finetuning strategy for large pretrained speech models, based on mixed source and target domain self-supervision. We find that including source domain self-supervision stabilizes training and avoids mode collapse of the latent representations. For evaluation, we collect HParl, a $120$ hour speech corpus for Greek, consisting of plenary sessions in the Greek Parliament. We merge HParl with two popular Greek corpora to create GREC-MD, a test-bed for multi-domain evaluation of Greek ASR systems. In our experiments we find that, while other Unsupervised Domain Adaptation baselines fail in this resource-constrained environment, M2DS2 yields significant improvements for cross-domain adaptation, even when a only a few hours of in-domain audio are available. When we relax the problem in a weakly supervised setting, we find that independent adaptation for audio using M2DS2 and language using simple LM augmentation techniques is particularly effective, yielding word error rates comparable to the fully supervised baselines.
【10】 Computational Charisma -- A Brick by Brick Blueprint for Building Charismatic Artificial Intelligence
标题：计算魅力--构建魅力人工智能的一砖一瓦蓝图
链接：https://arxiv.org/abs/2301.00142
作者：Björn W. Schuller,Shahin Amiriparian,Anton Batliner,Alexander Gebhard,Maurice Gerzcuk,Vincent Karas,Alexander Kathan,Lennart Seizer,Johanna Löchner机构：EIHW – Chair of Embedded Intelligence for Health Care and Wellbeing, University, of Augsburg, Augsburg, Germany, GLAM – Group on Language, Audio, & Music, Imperial College London, London, United Kingdom摘要：魅力被认为是一个人吸引和潜在影响他人的能力。显然，从人工智能（AI）的角度来看，为它提供这样的技能可能会有相当大的兴趣。除此之外，大量的用例为人类魅力的计算测量打开了大门，比如指导人类获得魅力，调解人与人之间的对话，或者在大社交数据中识别魅力个体。存在着许多基于不同维度的超凡魅力模型，通常遵循这样的观点：如果某人能够并且愿意帮助他人，那么超凡魅力就会被赋予。例子包括科学研究中的影响力（可能有帮助）和和蔼可亲（可能有帮助），或者权力（可能有帮助）、存在感和温暖（两者都有帮助）作为一个流行的概念。在这些维度上为类人机器人或虚拟代理人建立高水平的模型似乎是可以实现的。此外，随着情感计算和社会信号处理相关领域的最新进展，自动测量也显得相当可行。在这里，我们，因此提出了一个蓝图，建立机器，可以出现魅力，但也分析他人的魅力。为此，我们首先从心理学的角度，包括魅力的不同模型和行为线索，然后转向口语中的会话魅力，这是人与人和人机对话中必不可少的一种典型模态。然后，计算的视角处理人工智能对魅力行为的识别和生成。这包括对该领域的现状和上述蓝图的概述。然后，我们将列举计算魅力技能的典型用例，然后转到伦理方面，并总结这篇关于构建魅力型人工智能的概述和观点。摘要：Charisma is considered as one's ability to attract and potentially also influence others. Clearly, there can be considerable interest from an artificial intelligence's (AI) perspective to provide it with such skill. Beyond, a plethora of use cases opens up for computational measurement of human charisma, such as for tutoring humans in the acquisition of charisma, mediating human-to-human conversation, or identifying charismatic individuals in big social data. A number of models exist that base charisma on various dimensions, often following the idea that charisma is given if someone could and would help others. Examples include influence (could help) and affability (would help) in scientific studies or power (could help), presence, and warmth (both would help) as a popular concept. Modelling high levels in these dimensions for humanoid robots or virtual agents, seems accomplishable. Beyond, also automatic measurement appears quite feasible with the recent advances in the related fields of Affective Computing and Social Signal Processing. Here, we, thereforem present a blueprint for building machines that can appear charismatic, but also analyse the charisma of others. To this end, we first provide the psychological perspective including different models of charisma and behavioural cues of it. We then switch to conversational charisma in spoken language as an exemplary modality that is essential for human-human and human-computer conversations. The computational perspective then deals with the recognition and generation of charismatic behaviour by AI. This includes an overview of the state of play in the field and the aforementioned blueprint. We then name exemplary use cases of computational charismatic skills before switching to ethical aspects and concluding this overview and perspective on building charisma-enabled AI.
【11】 Addressing the Selection Bias in Voice Assistance: Training Voice Assistance Model in Python with Equal Data Selection
标题：解决语音辅助中的选择偏差：在等数据选择的Python中训练语音辅助模型
链接：https://arxiv.org/abs/2301.00646
作者：Kashav Piya,Srijal Shrestha,Cameran Frank,Estephanos Jebessa,Tauheed Khan Mohd摘要：近年来，语音助手已经成为我们日常生活的一部分，允许通过语音合成、语音识别和自然语言处理来进行信息检索。这些语音助手可以在许多现代设备中找到，如苹果、亚马逊、谷歌和三星。这个项目主要关注自然语言处理中的虚拟辅助。自然语言处理是人工智能的一种形式，它帮助机器理解人并创建反馈回路。这个项目将使用深度学习创建一个语音识别器，并使用Commonvoice和从当地社区收集的数据，通过谷歌协作室进行模型训练。AI助手在识别出一个命令后，将能够执行最合适的动作，然后给出响应。 这个项目的动机来自于许多虚拟助理中存在的种族和性别偏见。计算机行业主要由男性主导，正因为如此，生产的许多产品都不考虑女性。这种偏见对自然语言处理有影响。这个项目将利用各种开源项目来实现机器学习算法，并训练辅助算法来识别不同类型的语音、口音和方言。通过这个项目，目标是使用来自代表性不足群体的语音数据来构建一个语音助手，可以识别不分性别、种族或口音的语音。增加妇女在计算机行业的代表性对该行业的未来很重要。通过在语音助手的初步研究中代表妇女，可以表明女性在这一技术的发展中发挥了至关重要的作用。配合相关工作，本项目将利用大学生和中年人的第一手数据，训练语音助理，以对抗性别偏见。摘要：In recent times, voice assistants have become a part of our day-to-day lives, allowing information retrieval by voice synthesis, voice recognition, and natural language processing. These voice assistants can be found in many modern-day devices such as Apple, Amazon, Google, and Samsung. This project is primarily focused on Virtual Assistance in Natural Language Processing. Natural Language Processing is a form of AI that helps machines understand people and create feedback loops. This project will use deep learning to create a Voice Recognizer and use Commonvoice and data collected from the local community for model training using Google Colaboratory. After recognizing a command, the AI assistant will be able to perform the most suitable actions and then give a response. The motivation for this project comes from the race and gender bias that exists in many virtual assistants. The computer industry is primarily dominated by the male gender, and because of this, many of the products produced do not regard women. This bias has an impact on natural language processing. This project will be utilizing various open-source projects to implement machine learning algorithms and train the assistant algorithm to recognize different types of voices, accents, and dialects. Through this project, the goal to use voice data from underrepresented groups to build a voice assistant that can recognize voices regardless of gender, race, or accent. Increasing the representation of women in the computer industry is important for the future of the industry. By representing women in the initial study of voice assistants, it can be shown that females play a vital role in the development of this technology. In line with related work, this project will use first-hand data from the college population and middle-aged adults to train voice assistant to combat gender bias.
【12】 Unsupervised Acoustic Scene Mapping Based on Acoustic Features and Dimensionality Reduction
标题：基于声学特征和降维的无监督声场景映射
链接：https://arxiv.org/abs/2301.00448
作者：Idan Cohen,Ofir Lindenbaum,Sharon Gannot机构：Bar-Ilan University, Ramat-Gan, Israel摘要：经典的声场景映射方法需要估计麦克风之间的到达时间差（TDOA）。遗憾的是，TDOA估计对混响和加性噪声非常敏感。我们引入了一种无监督的数据驱动方法，该方法利用了数据的自然结构。我们的方法基于局部共形自动编码器（LOCA）--一种离线深度学习方案，用于从测量中学习标准化数据坐标。我们的实验装置包括一个麦克风阵列，用于测量整个隔音罩内多个位置的传输声源。我们证明了LOCA学习一个与麦克风的空间位置等距的表示。通过一系列仿真实验对该方法的性能进行了评估，并与其他降维方法进行了比较。我们进一步评估了混响对LOCA结果的影响，并表明它具有相当的鲁棒性。摘要：Classical methods for acoustic scene mapping require the estimation of time difference of arrival (TDOA) between microphones. Unfortunately, TDOA estimation is very sensitive to reverberation and additive noise. We introduce an unsupervised data-driven approach that exploits the natural structure of the data. Our method builds upon local conformal autoencoders (LOCA) - an offline deep learning scheme for learning standardized data coordinates from measurements. Our experimental setup includes a microphone array that measures the transmitted sound source at multiple locations across the acoustic enclosure. We demonstrate that LOCA learns a representation that is isometric to the spatial locations of the microphones. The performance of our method is evaluated using a series of realistic simulations and compared with other dimensionality-reduction schemes. We further assess the influence of reverberation on the results of LOCA and show that it demonstrates considerable robustness.【1】 ResGrad: Residual Denoising Diffusion Probabilistic Models for Text to Speech
标题：ResGrad：文语转换的残差去噪扩散概率模型
链接：https://arxiv.org/abs/2212.14518
*与cs.SD语音【5】为同一篇
作者：Zehua Chen,Yihan Wu,Yichong Leng,Jiawei Chen,Haohe Liu,Xu Tan,Yang Cui,Ke Wang,Lei He,Sheng Zhao,Jiang Bian,Danilo Mandic机构：4Renmin University of China, 5Universityof Science and Technology of China, 6South China University of Technology, 7University of Surrey†Work done during an internship at Microsoft摘要：去噪扩散概率模型（DDPMs）由于其产生高保真样本的能力较强，在文语转换（TTS）合成中得到了广泛的应用。然而，由于其在高维数据空间中的迭代求精过程导致推理速度慢，限制了其在实时系统中的应用。先前的工作已经探索了通过最小化推理步骤的数量来加速，但是以样本质量为代价。在本文中，为了提高基于DDPM的TTS模型的推理速度，同时获得高样本质量，我们提出了ResGrad，这是一种轻量级扩散模型，它学习精炼现有TTS模型的输出谱图（例如，FastSpeech 2），通过预测模型输出和相应地面实况语音之间的残差。ResGrad有几个优点：1）与其他需要从头开始合成语音的DDPM加速方法相比，ResGrad通过将生成目标从地面实况梅尔谱图改为残差，降低了任务的复杂度，使得模型更加轻量级，从而降低了实时性。2)ResGrad以即插即用的方式应用于已有TTS模型的推理过程中，不需要重新训练该模型。我们在单说话者数据集LJSpeech和两个更具挑战性的多说话者（LibriTTS）和高采样率（VCTK）数据集上验证了ResGrad。实验结果表明，与其他DDPM加速方法相比：（1）ResGrad算法在保证实时性的前提下获得了更好的样本质量; 2）在语音质量相似的情况下，ResGrad合成语音的速度比基线方法快10倍以上。音频样本可从www.example.com获得https://resgrad1.github.io/。摘要：Denoising Diffusion Probabilistic Models (DDPMs) are emerging in text-to-speech (TTS) synthesis because of their strong capability of generating high-fidelity samples. However, their iterative refinement process in high-dimensional data space results in slow inference speed, which restricts their application in real-time systems. Previous works have explored speeding up by minimizing the number of inference steps but at the cost of sample quality. In this work, to improve the inference speed for DDPM-based TTS model while achieving high sample quality, we propose ResGrad, a lightweight diffusion model which learns to refine the output spectrogram of an existing TTS model (e.g., FastSpeech 2) by predicting the residual between the model output and the corresponding ground-truth speech. ResGrad has several advantages: 1) Compare with other acceleration methods for DDPM which need to synthesize speech from scratch, ResGrad reduces the complexity of task by changing the generation target from ground-truth mel-spectrogram to the residual, resulting into a more lightweight model and thus a smaller real-time factor. 2) ResGrad is employed in the inference process of the existing TTS model in a plug-and-play way, without re-training this model. We verify ResGrad on the single-speaker dataset LJSpeech and two more challenging datasets with multiple speakers (LibriTTS) and high sampling rate (VCTK). Experimental results show that in comparison with other speed-up methods of DDPMs: 1) ResGrad achieves better sample quality with the same inference speed measured by real-time factor; 2) with similar speech quality, ResGrad synthesizes speech faster than baseline methods by more than 10 times. Audio samples are available at https://resgrad1.github.io/.
【2】 StyleTTS-VC: One-Shot Voice Conversion by Knowledge Transfer from Style-Based TTS Models
标题：StyleTTS-VC：通过基于风格的TTS模型的知识转移实现一次语音转换
链接：https://arxiv.org/abs/2212.14227
*与cs.SD语音【6】为同一篇
作者：Yinghao Aaron Li,Cong Han,Nima Mesgarani机构：Department of Electrical Engineering, Columbia University, USA摘要：单次语音转换（One-shot voice conversion，VC）的目的是利用来自目标说话人的仅几秒的参考语音将来自任意源说话人的语音转换成任意目标说话人。这在很大程度上取决于如何理清说话者的身份和说话内容，这一任务仍然具有挑战性。在此，我们提出了一种新的方法来学习去纠缠的语音表示，通过迁移学习从风格为基础的文本到语音（TTS）模型。通过循环一致性和对抗性训练，基于风格的TTS模型能够以高保真度和相似度执行转录引导的一次VC。通过教师-学生知识转移和新颖的数据扩充方案学习额外的梅尔谱图编码器，我们的方法在不需要输入文本的情况下产生解纠缠的语音表示。主观评价结果表明，该方法在自然度和相似度方面均优于现有的单次语音转换模型。摘要：One-shot voice conversion (VC) aims to convert speech from any source speaker to an arbitrary target speaker with only a few seconds of reference speech from the target speaker. This relies heavily on disentangling the speaker's identity and speech content, a task that still remains challenging. Here, we propose a novel approach to learning disentangled speech representation by transfer learning from style-based text-to-speech (TTS) models. With cycle consistent and adversarial training, the style-based TTS models can perform transcription-guided one-shot VC with high fidelity and similarity. By learning an additional mel-spectrogram encoder through a teacher-student knowledge transfer and novel data augmentation scheme, our approach results in disentangled speech representation without needing the input text. The subjective evaluation shows that our approach can significantly outperform the previous state-of-the-art one-shot voice conversion models in both naturalness and similarity.
【3】 Blind Restoration of Real-World Audio by 1D Operational GANs
标题：一维操作GANS实现真实世界音频的盲恢复
链接：https://arxiv.org/abs/2212.14618
*与cs.SD语音【1】为同一篇
作者：Turker Ince,Serkan Kiranyaz,Ozer Can Devecioglu,Muhammad Salman Khan,Muhammad Chowdhury,Moncef Gabbouj摘要：目的：尽管文献中提出了许多关于音频恢复的研究，但是大多数研究集中于诸如去噪或去混响等孤立的恢复问题，而忽略了其他伪像。此外，假设具有有限数目的固定信号失真比（SDR）电平的噪声或混响环境是常见的做法。然而，真实世界的音频经常被诸如混响、传感器噪声和具有变化类型、严重性和持续时间的背景音频混合的伪像的混合破坏。在本研究中，我们提出了一种新的方法，通过运算生成式对抗网络（Op-GANs）对真实世界的音频信号进行盲恢复，该方法具有时间和频谱客观度量，以提高恢复的音频信号的质量，而不管破坏它的每个伪像的类型和严重程度。1D操作-GAN与针对任何损坏音频信号的盲恢复而优化的生成神经元模型一起使用。结果：该方法在TIMIT-RAR（语音）和GTZAN-RAR（非语音）数据集上进行了广泛的评估，这些数据集被随机混合的伪影污染，每个伪影都具有随机的严重程度，以模拟真实世界的音频信号。分别实现了超过7.2dB和4.9dB的平均SDR改善，当与基线方法相比时，这是实质性的。意义：这是盲音频恢复领域的先驱研究，具有直接（时域）恢复真实世界音频的独特能力，同时针对广泛的SDR范围和伪像类型实现前所未有的性能水平。结论：1D Op-GANs可以实现鲁棒的、计算有效的真实世界音频恢复，并且性能显著提高。源代码和生成的真实世界音频数据集将在专用的GitHub存储库1中与研究社区公开共享。摘要：Objective: Despite numerous studies proposed for audio restoration in the literature, most of them focus on an isolated restoration problem such as denoising or dereverberation, ignoring other artifacts. Moreover, assuming a noisy or reverberant environment with limited number of fixed signal-to-distortion ratio (SDR) levels is a common practice. However, real-world audio is often corrupted by a blend of artifacts such as reverberation, sensor noise, and background audio mixture with varying types, severities, and duration. In this study, we propose a novel approach for blind restoration of real-world audio signals by Operational Generative Adversarial Networks (Op-GANs) with temporal and spectral objective metrics to enhance the quality of restored audio signal regardless of the type and severity of each artifact corrupting it. Methods: 1D Operational-GANs are used with generative neuron model optimized for blind restoration of any corrupted audio signal. Results: The proposed approach has been evaluated extensively over the benchmark TIMIT-RAR (speech) and GTZAN-RAR (non-speech) datasets corrupted with a random blend of artifacts each with a random severity to mimic real-world audio signals. Average SDR improvements of over 7.2 dB and 4.9 dB are achieved, respectively, which are substantial when compared with the baseline methods. Significance: This is a pioneer study in blind audio restoration with the unique capability of direct (time-domain) restoration of real-world audio whilst achieving an unprecedented level of performance for a wide SDR range and artifact types. Conclusion: 1D Op-GANs can achieve robust and computationally effective real-world audio restoration with significantly improved performance. The source codes and the generated real-world audio datasets are shared publicly with the research community in a dedicated GitHub repository1.
【4】 Defense Against Adversarial Attacks on Audio DeepFake Detection
标题：音频DeepFake检测中的敌意攻击防御
链接：https://arxiv.org/abs/2212.14597
*与cs.SD语音【2】为同一篇
作者：Piotr Kawa,Marcin Plata,Piotr Syga机构：Department of Artificial Intelligence, Wrocław University of Science and Technology, Wrocław, Poland备注：Submitted to ICASSP 2023摘要：音频DeepFakes是使用深度学习方法创建的人工生成的话语，主要目的是愚弄听众，大多数这样的音频都具有高度的说服力。它们的质量足以在安全和隐私方面构成严重威胁，例如新闻的可靠性或诽谤。为了防止这些威胁，已经提出了多种基于神经网络的方法来检测生成的语音。在本文中，我们讨论了对抗性攻击的主题，它通过向输入数据添加表面（人类难以发现）更改来降低检测器的性能。我们的贡献包括评估3种检测结构在两种场景（白盒和使用可转移性机制）下抵抗对抗性攻击的鲁棒性，并随后通过使用由我们的新的自适应训练方法执行的对抗性训练来增强它。摘要：Audio DeepFakes are artificially generated utterances created using deep learning methods with the main aim to fool the listeners, most of such audio is highly convincing. Their quality is sufficient to pose a serious threat in terms of security and privacy, such as the reliability of news or defamation. To prevent the threats, multiple neural networks-based methods to detect generated speech have been proposed. In this work, we cover the topic of adversarial attacks, which decrease the performance of detectors by adding superficial (difficult to spot by a human) changes to input data. Our contribution contains evaluating the robustness of 3 detection architectures against adversarial attacks in two scenarios (white-box and using transferability mechanism) and enhancing it later by the use of adversarial training performed by our novel adaptive training method.
【5】 Multi-modal deep learning system for depression and anxiety detection
标题：用于抑郁和焦虑检测的多模式深度学习系统
链接：https://arxiv.org/abs/2212.14490
*与cs.SD语音【3】为同一篇
作者：Brian Diep,Marija Stanojevic,Jekaterina Novikova机构：Department of Computer Science, University of Toronto, Toronto, ON, Winterlight Labs备注：accepted to the PAI4MH workshop at NeurIPS 2022摘要：传统的焦虑和抑郁症筛查方法阻碍了对这些疾病的有效监测和治疗。然而，NLP和语音建模的最新进展使得基于文本、声学和手工语言的特征能够共同形成未来心理健康筛查和状况检测的基础。言语是洞察个体认知状态的丰富而现成的来源，通过利用言语的不同方面，我们可以开发新的抑郁和焦虑的数字生物标志物。为此，我们提出了一个多模态系统，用于筛选抑郁和焦虑的自我管理的言语任务。所提出的模型集成了来自音频和文本的深度学习特征，以及由临床验证的领域知识所告知的手工特征。我们发现，与单独的手工特征的基线相比，用深度学习特征增强手工特征提高了我们的总体分类F1得分，抑郁从0.58提高到0.63，焦虑从0.54提高到0.57。我们的研究结果表明，抑郁症和焦虑症的语音生物标志物在数字健康的未来有着重要的前景。摘要：Traditional screening practices for anxiety and depression pose an impediment to monitoring and treating these conditions effectively. However, recent advances in NLP and speech modelling allow textual, acoustic, and hand-crafted language-based features to jointly form the basis of future mental health screening and condition detection. Speech is a rich and readily available source of insight into an individual's cognitive state and by leveraging different aspects of speech, we can develop new digital biomarkers for depression and anxiety. To this end, we propose a multi-modal system for the screening of depression and anxiety from self-administered speech tasks. The proposed model integrates deep-learned features from audio and text, as well as hand-crafted features that are informed by clinically-validated domain knowledge. We find that augmenting hand-crafted features with deep-learned features improves our overall classification F1 score comparing to a baseline of hand-crafted features alone from 0.58 to 0.63 for depression and from 0.54 to 0.57 for anxiety. The findings of our work suggest that speech-based biomarkers for depression and anxiety hold significant promise in the future of digital health.
【6】 Macro-block dropout for improved regularization in training end-to-end speech recognition models
标题：用于端到端语音识别模型训练中改进正则化的宏块丢失
链接：https://arxiv.org/abs/2212.14149
*与cs.SD语音【4】为同一篇
作者：Chanwoo Kim,Sathish Indurti,Jinhwan Park,Wonyong Sung机构：Samsung Research, Seoul, South Korea, Seoul National University, Seoul, South Korea备注：Accepted for presentation at The 2022 IEEE Spoken Language Technology Workshop (SLT 2022)摘要：本文提出了一种新的正则化算法--宏块丢失算法。过拟合问题一直是大型神经网络模型训练中的难题。辍学技术已被证明是简单的，但非常有效的正则化，防止复杂的共同适应期间的训练。在我们的工作中，我们定义了一个宏块，它包含了从输入到递归神经网络（RNN）的大量单元。不是对每个单元应用丢弃，而是对每个宏块应用随机丢弃。该算法具有即使我们保持恒定的平均丢弃率也对每层应用不同的丢弃率的效果，这具有更好的正则化效果。在我们使用递归神经网络-转换器（RNN-T）的实验中，该算法在LibriSpeech测试-清除和测试-其他上的字错误率（WER）比传统的丢失率分别提高了4.30%和6.13%。在基于注意力的编解码器（AED）模型下，该算法在相同测试集上的丢包率分别比传统算法提高了4.36%和5.85%。摘要：This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large neural network models. The dropout technique has proven to be simple yet very effective for regularization by preventing complex co-adaptations during training. In our work, we define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN). Rather than applying dropout to each unit, we apply random dropout to each macro-block. This algorithm has the effect of applying different drop out rates for each layer even if we keep a constant average dropout rate, which has better regularization effects. In our experiments using Recurrent Neural Network-Transducer (RNN-T), this algorithm shows relatively 4.30 % and 6.13 % Word Error Rates (WERs) improvement over the conventional dropout on LibriSpeech test-clean and test-other. With an Attention-based Encoder-Decoder (AED) model, this algorithm shows relatively 4.36 % and 5.85 % WERs improvement over the conventional dropout on the same test sets.
【7】 MnTTS2: An Open-Source Multi-Speaker Mongolian Text-to-Speech Synthesis Dataset
标题：MnTTS2：一个开源的多说话人蒙古语文语合成数据集
链接：https://arxiv.org/abs/2301.00657
作者：Kailin Liang,Bin Liu,Yifan Hu,Rui Liu,Feilong Bao,Guanglai Gao机构：Inner Mongolia University, Hohhot, China备注：Accepted by NCMMSC'2022 (this https URL)摘要：面向低资源语言的文语转换（TTS）合成是当前学术界和工业界研究的热点问题。蒙古语是内蒙古自治区的官方语言，也是世界上1000多万人使用的具有代表性的低资源语言。然而，蒙古语TTS的开源数据集相对缺乏。为此，我们公开了一个开源的蒙古语多人TTS数据集MnTTS 2，以供相关研究人员参考。在这项工作中，我们准备了不同主题的转录，并邀请了三位专业的蒙古语播音员组成一个三人TTS数据集，其中每位播音员用蒙古语录制了10个小时的演讲，总共30个小时。此外，我们建立了基于最先进的FastSpeech 2模型和HiFi-GAN声码器的基准系统。实验结果表明，所构建的MnTTS 2数据集足以构建鲁棒的多说话人TTS模型。MnTTS 2数据集、训练配方和预训练模型发布于：\链接{https：//github.com/ssmlkl/MnTTS2}摘要：Text-to-Speech (TTS) synthesis for low-resource languages is an attractive research issue in academia and industry nowadays. Mongolian is the official language of the Inner Mongolia Autonomous Region and a representative low-resource language spoken by over 10 million people worldwide. However, there is a relative lack of open-source datasets for Mongolian TTS. Therefore, we make public an open-source multi-speaker Mongolian TTS dataset, named MnTTS2, for the benefit of related researchers. In this work, we prepare the transcription from various topics and invite three professional Mongolian announcers to form a three-speaker TTS dataset, in which each announcer records 10 hours of speeches in Mongolian, resulting 30 hours in total. Furthermore, we build the baseline system based on the state-of-the-art FastSpeech2 model and HiFi-GAN vocoder. The experimental results suggest that the constructed MnTTS2 dataset is sufficient to build robust multi-speaker TTS models for real-world applications. The MnTTS2 dataset, training recipe, and pretrained models are released at: \url{https://github.com/ssmlkl/MnTTS2}
【8】 TriNet: stabilizing self-supervised learning from complete or slow collapse
标题：TriNet：从完全崩溃或缓慢崩溃中稳定自我监督学习
链接：https://arxiv.org/abs/2301.00656
作者：Lixin Cao,Jun Wang,Ben Yang,Dan Su,Dong Yu机构：Tencent AI Lab, China, Peking University, Tencent AI Lab, USA摘要：自监督学习（SSL）模型面临着信息突然崩溃或维度缓慢崩溃的挑战。我们提出TriNet，它引入了一种新颖的三分支结构来防止崩溃和稳定预训练。实验结果表明，该方法显著地稳定和加速了预训练过程，并且在一个下游基准ASR任务中，与现有的（SOTA）Data2vec相比，实现了5.32%的相对误字率降低.我们将在www.example.com上发布我们的代码https://github.com/tencent-ailab/。摘要：Self-supervised learning (SSL) models confront challenges of abrupt informational collapse or slow dimensional collapse. We propose TriNet, which introduces a novel triple-branch architecture for preventing collapse and stabilizing the pre-training. Our experimental results show that the proposed method notably stabilizes and accelerates pre-training and achieves a relative word error rate reduction (WERR) of 5.32% compared to the state-of-the-art (SOTA) Data2vec for a downstream benchmark ASR task. We will release our code at https://github.com/tencent-ailab/.
【9】 Efficient Speech Representation Learning with Low-Bit Quantization
标题：低位量化的高效语音表示学习
链接：https://arxiv.org/abs/2301.00652
作者：Ching-Feng Yeh,Wei-Ning Hsu,Paden Tomasello,Abdelrahman Mohamed摘要：随着机器学习硬件的发展，更新的模型往往以增加尺寸和计算复杂度为代价。为了提高这些模型的效率，我们应用和研究了最新的量化技术在语音表示学习模型上。在SUPERB基准上评估了量化技术。在ASR任务中，通过积极量化到1位，我们实现了86.32%的存储减少（184.42 -〉25.23），88%的估计运行时间减少（1.00 -〉0.12），字错误率增加（7.06 -〉15.96）。与同样针对模型压缩的DistillHuBERT相比，2位配置产生了略小的存储空间（35.84 vs. 46.98）、更好的字错误率（12.68 vs. 13.37）和更有效的估计运行时间（0.15 vs. 0.73）。摘要：With the development of hardware for machine learning, newer models often come at the cost of both increased sizes and computational complexity. In effort to improve the efficiency for these models, we apply and investigate recent quantization techniques on speech representation learning models. The quantization techniques were evaluated on the SUPERB benchmark. On the ASR task, with aggressive quantization to 1 bit, we achieved 86.32% storage reduction (184.42 -> 25.23), 88% estimated runtime reduction (1.00 -> 0.12) with increased word error rate (7.06 -> 15.96). In comparison with DistillHuBERT which also aims for model compression, the 2-bit configuration yielded slightly smaller storage (35.84 vs. 46.98), better word error rate (12.68 vs. 13.37) and more efficient estimated runtime (0.15 vs. 0.73).
【10】 Addressing the Selection Bias in Voice Assistance: Training Voice Assistance Model in Python with Equal Data Selection
标题：解决语音辅助中的选择偏差：在等数据选择的Python中训练语音辅助模型
链接：https://arxiv.org/abs/2301.00646
*与cs.SD语音【11】为同一篇
作者：Kashav Piya,Srijal Shrestha,Cameran Frank,Estephanos Jebessa,Tauheed Khan Mohd摘要：近年来，语音助手已经成为我们日常生活的一部分，允许通过语音合成、语音识别和自然语言处理来进行信息检索。这些语音助手可以在许多现代设备中找到，如苹果、亚马逊、谷歌和三星。这个项目主要关注自然语言处理中的虚拟辅助。自然语言处理是人工智能的一种形式，它帮助机器理解人并创建反馈回路。这个项目将使用深度学习创建一个语音识别器，并使用Commonvoice和从当地社区收集的数据，通过谷歌协作室进行模型训练。AI助手在识别出一个命令后，将能够执行最合适的动作，然后给出响应。 这个项目的动机来自于许多虚拟助理中存在的种族和性别偏见。计算机行业主要由男性主导，正因为如此，生产的许多产品都不考虑女性。这种偏见对自然语言处理有影响。这个项目将利用各种开源项目来实现机器学习算法，并训练辅助算法来识别不同类型的语音、口音和方言。通过这个项目，目标是使用来自代表性不足群体的语音数据来构建一个语音助手，可以识别不分性别、种族或口音的语音。增加妇女在计算机行业的代表性对该行业的未来很重要。通过在语音助手的初步研究中代表妇女，可以表明女性在这一技术的发展中发挥了至关重要的作用。配合相关工作，本项目将利用大学生和中年人的第一手数据，训练语音助理，以对抗性别偏见。摘要：In recent times, voice assistants have become a part of our day-to-day lives, allowing information retrieval by voice synthesis, voice recognition, and natural language processing. These voice assistants can be found in many modern-day devices such as Apple, Amazon, Google, and Samsung. This project is primarily focused on Virtual Assistance in Natural Language Processing. Natural Language Processing is a form of AI that helps machines understand people and create feedback loops. This project will use deep learning to create a Voice Recognizer and use Commonvoice and data collected from the local community for model training using Google Colaboratory. After recognizing a command, the AI assistant will be able to perform the most suitable actions and then give a response. The motivation for this project comes from the race and gender bias that exists in many virtual assistants. The computer industry is primarily dominated by the male gender, and because of this, many of the products produced do not regard women. This bias has an impact on natural language processing. This project will be utilizing various open-source projects to implement machine learning algorithms and train the assistant algorithm to recognize different types of voices, accents, and dialects. Through this project, the goal to use voice data from underrepresented groups to build a voice assistant that can recognize voices regardless of gender, race, or accent. Increasing the representation of women in the computer industry is important for the future of the industry. By representing women in the initial study of voice assistants, it can be shown that females play a vital role in the development of this technology. In line with related work, this project will use first-hand data from the college population and middle-aged adults to train voice assistant to combat gender bias.
【11】 Unsupervised Acoustic Scene Mapping Based on Acoustic Features and Dimensionality Reduction
标题：基于声学特征和降维的无监督声场景映射
链接：https://arxiv.org/abs/2301.00448
*与cs.SD语音【12】为同一篇
作者：Idan Cohen,Ofir Lindenbaum,Sharon Gannot机构：Bar-Ilan University, Ramat-Gan, Israel摘要：经典的声场景映射方法需要估计麦克风之间的到达时间差（TDOA）。遗憾的是，TDOA估计对混响和加性噪声非常敏感。我们引入了一种无监督的数据驱动方法，该方法利用了数据的自然结构。我们的方法基于局部共形自动编码器（LOCA）--一种离线深度学习方案，用于从测量中学习标准化数据坐标。我们的实验装置包括一个麦克风阵列，用于测量整个隔音罩内多个位置的传输声源。我们证明了LOCA学习一个与麦克风的空间位置等距的表示。通过一系列仿真实验对该方法的性能进行了评估，并与其他降维方法进行了比较。我们进一步评估了混响对LOCA结果的影响，并表明它具有相当的鲁棒性。摘要：Classical methods for acoustic scene mapping require the estimation of time difference of arrival (TDOA) between microphones. Unfortunately, TDOA estimation is very sensitive to reverberation and additive noise. We introduce an unsupervised data-driven approach that exploits the natural structure of the data. Our method builds upon local conformal autoencoders (LOCA) - an offline deep learning scheme for learning standardized data coordinates from measurements. Our experimental setup includes a microphone array that measures the transmitted sound source at multiple locations across the acoustic enclosure. We demonstrate that LOCA learns a representation that is isometric to the spatial locations of the microphones. The performance of our method is evaluated using a series of realistic simulations and compared with other dimensionality-reduction schemes. We further assess the influence of reverberation on the results of LOCA and show that it demonstrates considerable robustness.
【12】 Analysing Discrete Self Supervised Speech Representation for Spoken Language Modeling
标题：用于口语建模的离散自监督语音表示分析
链接：https://arxiv.org/abs/2301.00591
*与cs.SD语音【7】为同一篇
作者：Amitay Sicherman,Yossi Adi机构：School of Engineering and Computer Science, The Hebrew University of Jerusalem, Israel摘要：本文从生成式口语建模（GSLM）的角度深入分析了离散的自监督语音表示。根据这种分析的结果，我们对GSLM的分立单元提出了实际的改进。首先，我们从三个方面来分析这些单位：解释、可视化和再合成。我们的分析发现，语音单位与音素和音素族之间有很高的相关性，而与说话人和性别的相关性较弱。另外，我们发现在提取的单元中存在冗余，并声称其中一个原因可能是单元的上下文。在此基础上，我们提出了一种新的无监督度量单位冗余度的方法。最后，我们使用这个度量来开发新的方法，提高单元聚类的鲁棒性，并在考虑零资源语音度量（如ABX）时显示出显著的改进。代码和分析工具可通过以下链接获得。摘要：This work profoundly analyzes discrete self-supervised speech representations through the eyes of Generative Spoken Language Modeling (GSLM). Following the findings of such an analysis, we propose practical improvements to the discrete unit for the GSLM. First, we start comprehending these units by analyzing them in three axes: interpretation, visualization, and resynthesis. Our analysis finds a high correlation between the speech units to phonemes and phoneme families, while their correlation with speaker or gender is weaker. Additionally, we found redundancies in the extracted units and claim that one reason may be the units' context. Following this analysis, we propose a new, unsupervised metric to measure unit redundancies. Finally, we use this metric to develop new methods that improve the robustness of units clustering and show significant improvement considering zero-resource speech metrics such as ABX. Code and analysis tools are available under the following link.
【13】 EmoGator: A New Open Source Vocal Burst Dataset with Baseline Machine Learning Classification Methodologies
标题：EmoGator：一种新的具有基线机器学习分类方法的开源发声数据集
链接：https://arxiv.org/abs/2301.00508
*与cs.SD语音【8】为同一篇
备注：12 pages, 4 tables, 2 figures摘要：语音爆发--表达情感的短的非语音发声，例如笑、哭、叹息、呻吟和呻吟--是语音情感识别中经常被忽视的一个方面，但却是人类语音交流的一个重要方面。研究这些有趣的发声的一个障碍是缺乏大型数据集。我很高兴地介绍EmoGator数据集，它由来自365个扬声器的32,040个样本组成，16.91小时的音频;每个样本被说话者分类到30个不同情感类别中的一个。本文将讨论几种不同的构造分类器来识别情感类别的方法，并对未来的研究方向提出建议。数据集可从www.example.com下载https://github.com/fredbuhl/EmoGator。摘要：Vocal Bursts -- short, non-speech vocalizations that convey emotions, such as laughter, cries, sighs, moans, and groans -- are an often-overlooked aspect of speech emotion recognition, but an important aspect of human vocal communication. One barrier to study of these interesting vocalizations is a lack of large datasets. I am pleased to introduce the EmoGator dataset, which consists of 32,040 samples from 365 speakers, 16.91 hours of audio; each sample classified into one of 30 distinct emotion categories by the speaker. Several different approaches to construct classifiers to identify emotion categories will be discussed, and directions for future research will be suggested. Data set is available for download from https://github.com/fredbuhl/EmoGator.
【14】 Sample-Efficient Unsupervised Domain Adaptation of Speech Recognition Systems A case study for Modern Greek
标题：语音识别系统的样本高效无监督域自适应--以现代希腊语为例
链接：https://arxiv.org/abs/2301.00304
*与cs.SD语音【9】为同一篇
作者：Georgios Paraskevopoulos,Theodoros Kouzelis,Georgios Rouvalis,Athanasios Katsamanis,Vassilis Katsouros,Alexandros Potamianos机构：Katsourosare with the Institute for Speech and Language Processing摘要：现代语音识别系统在域移动下表现出快速的性能下降。这个问题在数据稀缺的环境中尤其普遍，例如资源匮乏的语言，在那里训练数据的多样性有限。在本文中，我们提出了M2 DS 2，一个简单的和样本有效的微调策略，为大型预训练语音模型，基于混合源和目标域自监督。我们发现，包括源域自我监督稳定训练和避免潜在表征的模式崩溃。为了评估，我们收集了HParl，一个120美元小时的希腊语演讲语料库，包括希腊议会的全体会议。我们将HParl与两个流行的希腊语语料库合并，创建了GREC-MD，这是一个用于希腊语ASR系统多领域评估的测试平台。在我们的实验中，我们发现，虽然其他无监督域自适应基线在这种资源受限的环境中失败，但M2 DS 2产生了跨域自适应的显著改进，即使当只有几个小时的域内音频可用时。当我们在弱监督设置中放松该问题时，我们发现使用M2 DS 2的音频和使用简单LM增强技术的语言的独立自适应特别有效，产生与完全监督基线相当的单词错误率。摘要：Modern speech recognition systems exhibits rapid performance degradation under domain shift. This issue is especially prevalent in data-scarce settings, such as low-resource languages, where diversity of training data is limited. In this work we propose M2DS2, a simple and sample-efficient finetuning strategy for large pretrained speech models, based on mixed source and target domain self-supervision. We find that including source domain self-supervision stabilizes training and avoids mode collapse of the latent representations. For evaluation, we collect HParl, a $120$ hour speech corpus for Greek, consisting of plenary sessions in the Greek Parliament. We merge HParl with two popular Greek corpora to create GREC-MD, a test-bed for multi-domain evaluation of Greek ASR systems. In our experiments we find that, while other Unsupervised Domain Adaptation baselines fail in this resource-constrained environment, M2DS2 yields significant improvements for cross-domain adaptation, even when a only a few hours of in-domain audio are available. When we relax the problem in a weakly supervised setting, we find that independent adaptation for audio using M2DS2 and language using simple LM augmentation techniques is particularly effective, yielding word error rates comparable to the fully supervised baselines.
【15】 Computational Charisma -- A Brick by Brick Blueprint for Building Charismatic Artificial Intelligence
标题：计算魅力--构建魅力人工智能的一砖一瓦蓝图
链接：https://arxiv.org/abs/2301.00142
*与cs.SD语音【10】为同一篇
作者：Björn W. Schuller,Shahin Amiriparian,Anton Batliner,Alexander Gebhard,Maurice Gerzcuk,Vincent Karas,Alexander Kathan,Lennart Seizer,Johanna Löchner机构：EIHW – Chair of Embedded Intelligence for Health Care and Wellbeing, University, of Augsburg, Augsburg, Germany, GLAM – Group on Language, Audio, & Music, Imperial College London, London, United Kingdom摘要：魅力被认为是一个人吸引和潜在影响他人的能力。显然，从人工智能（AI）的角度来看，为它提供这样的技能可能会有相当大的兴趣。除此之外，大量的用例为人类魅力的计算测量打开了大门，比如指导人类获得魅力，调解人与人之间的对话，或者在大社交数据中识别魅力个体。存在着许多基于不同维度的超凡魅力模型，通常遵循这样的观点：如果某人能够并且愿意帮助他人，那么超凡魅力就会被赋予。例子包括科学研究中的影响力（可能有帮助）和和蔼可亲（可能有帮助），或者权力（可能有帮助）、存在感和温暖（两者都有帮助）作为一个流行的概念。在这些维度上为类人机器人或虚拟代理人建立高水平的模型似乎是可以实现的。此外，随着情感计算和社会信号处理相关领域的最新进展，自动测量也显得相当可行。在这里，我们，因此提出了一个蓝图，建立机器，可以出现魅力，但也分析他人的魅力。为此，我们首先从心理学的角度，包括魅力的不同模型和行为线索，然后转向口语中的会话魅力，这是人与人和人机对话中必不可少的一种典型模态。然后，计算的视角处理人工智能对魅力行为的识别和生成。这包括对该领域的现状和上述蓝图的概述。然后，我们将列举计算魅力技能的典型用例，然后转到伦理方面，并总结这篇关于构建魅力型人工智能的概述和观点。摘要：Charisma is considered as one's ability to attract and potentially also influence others. Clearly, there can be considerable interest from an artificial intelligence's (AI) perspective to provide it with such skill. Beyond, a plethora of use cases opens up for computational measurement of human charisma, such as for tutoring humans in the acquisition of charisma, mediating human-to-human conversation, or identifying charismatic individuals in big social data. A number of models exist that base charisma on various dimensions, often following the idea that charisma is given if someone could and would help others. Examples include influence (could help) and affability (would help) in scientific studies or power (could help), presence, and warmth (both would help) as a popular concept. Modelling high levels in these dimensions for humanoid robots or virtual agents, seems accomplishable. Beyond, also automatic measurement appears quite feasible with the recent advances in the related fields of Affective Computing and Social Signal Processing. Here, we, thereforem present a blueprint for building machines that can appear charismatic, but also analyse the charisma of others. To this end, we first provide the psychological perspective including different models of charisma and behavioural cues of it. We then switch to conversational charisma in spoken language as an exemplary modality that is essential for human-human and human-computer conversations. The computational perspective then deals with the recognition and generation of charismatic behaviour by AI. This includes an overview of the state of play in the field and the aforementioned blueprint. We then name exemplary use cases of computational charismatic skills before switching to ethical aspects and concluding this overview and perspective on building charisma-enabled AI.
【16】 Memory Augmented Lookup Dictionary based Language Modeling for Automatic Speech Recognition
标题：基于记忆扩展查找词典的自动语音识别语言建模
链接：https://arxiv.org/abs/2301.00066
作者：Yukun Feng,Ming Tu,Rui Xia,Chuanzeng Huang,Yuxuan Wang机构：Johns Hopkins University, Speech and Music Intelligence (SAMI), ByteDance备注：Submitted to ICASSP 2023摘要：最近的研究表明，使用外部语言模型（LM）有利于端到端的自动语音识别（ASR）。然而，预测在训练集中出现频率较低的标记仍然相当具有挑战性。长尾预测问题已经在许多应用中被广泛研究，但是仅由ASR和LM的少数研究解决。本文提出了一种新的基于内存扩充查找字典的Transformer结构。新引入的查找字典包含了训练集中丰富的上下文信息，这对于正确预测长尾标记至关重要。在中英文数据集上的实验结果表明，该方法在单词/字符错误率和尾标记错误率上均优于Transformer LM.这在不影响解码效率的情况下实现。总之，我们证明了我们提出的方法在提高ASR解码性能方面的有效性，特别是对于长尾令牌。摘要：Recent studies have shown that using an external Language Model (LM) benefits the end-to-end Automatic Speech Recognition (ASR). However, predicting tokens that appear less frequently in the training set is still quite challenging. The long-tail prediction problems have been widely studied in many applications, but only been addressed by a few studies for ASR and LMs. In this paper, we propose a new memory augmented lookup dictionary based Transformer architecture for LM. The newly introduced lookup dictionary incorporates rich contextual information in training set, which is vital to correctly predict long-tail tokens. With intensive experiments on Chinese and English data sets, our proposed method is proved to outperform the baseline Transformer LM by a great margin on both word/character error rate and tail tokens error rate. This is achieved without impact on the decoding efficiency. Overall, we demonstrate the effectiveness of our proposed method in boosting the ASR decoding performance, especially for long-tail tokens.
机器翻译，仅供参考
每日语音速递