今日论文合集：cs.SD语音7篇，eess.AS音频处理8篇。
cs.SD语音
【1】 General Purpose Audio Effect Removal标题：通用音效去除链接：https://arxiv.org/abs/2308.16177
作者：Matthew Rice,Christian J. Steinmetz,George Fazekas,Joshua D. Reiss机构：Centre for Digital Music, Queen Mary University of London, UK备注：Preprint. Accepted to IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) 2023摘要：虽然音频效果的设计和应用是很好的理解，消除这些效果的逆问题显着更具挑战性和少得多的研究。最近，深度学习被应用到音频效果去除;然而，现有的方法集中于一次仅考虑一种效应或源类型的狭义公式。在现实场景中，多种效果应用于不同的源内容。这激发了更一般的任务，我们称之为通用音频效果去除。我们为这项任务开发了一个数据集，使用四个不同来源的五个音频效果，并使用它来训练和评估一组现有的架构。我们发现，没有一个单一的模型表现最佳的所有影响类型和来源。为了解决这个问题，我们引入了RemFX，一种旨在反映应用效果组合性的方法。我们首先训练了一组性能最佳的特定效果去除模型，然后利用音频效果分类模型在推理时动态构建模型的图。我们发现我们的方法优于单一模型基线，尽管存在许多影响的例子仍然具有挑战性。摘要：Although the design and application of audio effects is well understood, the inverse problem of removing these effects is significantly more challenging and far less studied. Recently, deep learning has been applied to audio effect removal; however, existing approaches have focused on narrow formulations considering only one effect or source type at a time. In realistic scenarios, multiple effects are applied with varying source content. This motivates a more general task, which we refer to as general purpose audio effect removal. We developed a dataset for this task using five audio effects across four different sources and used it to train and evaluate a set of existing architectures. We found that no single model performed optimally on all effect types and sources. To address this, we introduced RemFX, an approach designed to mirror the compositionality of applied effects. We first trained a set of the best-performing effect-specific removal models and then leveraged an audio effect classification model to dynamically construct a graph of our models at inference. We found our approach to outperform single model baselines, although examples with many effects present remain challenging.
【2】 CALM: Contrastive Cross-modal Speaking Style Modeling for Expressive  Text-to-Speech Synthesis标题：CAMLAM：用于表现力文语合成的对比跨通道语音风格建模链接：https://arxiv.org/abs/2308.16021
作者：Yi Meng,Xiang Li,Zhiyong Wu,Tingtian Li,Zixun Sun,Xinyu Xiao,Chi Sun,Hui Zhan,Helen Meng机构：Tsinghua-CUHK Joint Research Center for Media Sciences, Technologies and Systems, Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Department of Systems Engineering and Engineering Management备注：Accepted by InterSpeech 2022摘要：为了进一步改善合成语音的说话风格，当前的文本到语音（TTS）合成系统通常采用参考语音来风格化其输出而不是仅输入文本。这些参考语音是通过人工选择获得的，这是消耗资源的，或通过语义特征选择。然而，语义特征不仅包含与风格相关的信息，还包含与风格无关的信息。文本中与说话风格无关的信息会干扰参考音频的选择，导致说话风格不正确。为了提高参考文献的选择，我们提出了对比声学语言模块（CALM）提取的风格相关的文本特征（STF）的文本。CALM通过对比学习优化说话风格嵌入和提取的STF之间的相关性。因此，通过检索具有最高STF相似性的语音来选择用于输入文本的特定数量的最合适的参考语音。然后，根据STF相似度对风格嵌入进行加权总结，并将其用于合成语音的风格化。实验结果表明，我们所提出的方法的有效性，与客观评价和主观评价的合成语音说话风格优于基线方法与基于语义特征的参考选择。摘要：To further improve the speaking styles of synthesized speeches, current text-to-speech (TTS) synthesis systems commonly employ reference speeches to stylize their outputs instead of just the input texts. These reference speeches are obtained by manual selection which is resource-consuming, or selected by semantic features. However, semantic features contain not only style-related information, but also style irrelevant information. The information irrelevant to speaking style in the text could interfere the reference audio selection and result in improper speaking styles. To improve the reference selection, we propose Contrastive Acoustic-Linguistic Module (CALM) to extract the Style-related Text Feature (STF) from the text. CALM optimizes the correlation between the speaking style embedding and the extracted STF with contrastive learning. Thus, a certain number of the most appropriate reference speeches for the input text are selected by retrieving the speeches with the top STF similarities. Then the style embeddings are weighted summarized according to their STF similarities and used to stylize the synthesized speech of TTS. Experiment results demonstrate the effectiveness of our proposed approach, with both objective evaluations and subjective evaluations on the speaking styles of the synthesized speeches outperform a baseline approach with semantic-feature-based reference selection.
【3】 Dual-path Transformer Based Neural Beamformer for Target Speech  Extraction标题：基于双路径转换器的神经波束形成器用于目标语音提取链接：https://arxiv.org/abs/2308.15990
作者：Aoqi Guo,Sichong Qian,Baoxiang Li,Dazhi Gao机构：Ocean University of China, Qingdao, China, † SenseTime Research, Beijing, China摘要：神经波束形成器，集成了预分离和波束形成模块，在目标语音提取任务中显示出令人印象深刻的功效。然而，这些波束形成器的性能固有地受到预分离模块的预测精度的约束。在本文中，我们介绍了一个神经波束形成器支撑的双路径Transformer器。首先，我们利用时域中的交叉注意机制，提取关键的空间信息，波束形成的噪声协方差矩阵。随后，在频域中，自我注意机制，以加强该模型的能力，以处理特定频率的细节。通过设计，我们的模型避免了预分离模块的影响，以更全面的端到端方式提供性能。实验结果表明，我们的模型不仅超过当代领先的神经波束形成算法的分离性能，但也实现了这一点显着减少的参数计数。摘要：Neural beamformers, integrating both pre-separation and beamforming modules, have shown impressive efficacy in the target speech extraction task. Nevertheless, the performance of these beamformers is inherently constrained by the predictive accuracy of the pre-separation module. In this paper, we introduce a neural beamformer underpinned by a dual-path transformer. Initially, we harness the cross-attention mechanism in the time domain, extracting pivotal spatial information related to beamforming from the noisy covariance matrix. Subsequently, in the frequency domain, the self-attention mechanism is employed to bolster the model's capacity to process frequency-specific details. By design, our model circumvents the influence of pre-separation modules, delivering the performance in a more holistic end-to-end fashion. Experimental results reveal that our model not only surpasses contemporary leading neural beamforming algorithms in separation performance, but also achieves this with a notable reduction in parameter count.
【4】 LLaSM: Large Language and Speech Model标题：LLaSM：大型语言和语音模型链接：https://arxiv.org/abs/2308.15930
作者：Yu Shu,Siwei Dong,Guangyao Chen,Wenhao Huang,Ruihua Zhang,Daochen Shi,Qiqi Xiang,Yemin Shi机构：LinkSoul.AI,Peking University,.ai, † Equal contribution摘要：多模态大型语言模型最近引起了人们的极大兴趣。虽然，大多数的作品集中在视觉语言多模态模型提供强大的能力，遵循视觉和语言的指令。然而，我们认为，语音也是一个重要的方式，通过它人类与世界互动。因此，对于通用助理来说，能够遵循多模态语音和语言指令是至关重要的。在这项工作中，我们提出了大语言和语音模型（LLaSM）。LLaSM是一个端到端训练的大型多模态语音语言模型，具有跨模态会话能力，能够遵循语音和语言指令。我们的早期实验表明，LLaSM展示了人类与人工智能交互的更方便和自然的方式。具体来说，我们还发布了一个大型的语音指令跟随数据集LLaSM-Audio-Instructions。代码和演示可在https://github.com/LinkSoul-AI/LLaSM和https://huggingface.co/spaces/LinkSoul/LLaSM上获得。LLaSM-Audio-Instructions数据集可从https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions获得。摘要：Multi-modal large language models have garnered significant interest recently. Though, most of the works focus on vision-language multi-modal models providing strong capabilities in following vision-and-language instructions. However, we claim that speech is also an important modality through which humans interact with the world. Hence, it is crucial for a general-purpose assistant to be able to follow multi-modal speech-and-language instructions. In this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an end-to-end trained large multi-modal speech-language model with cross-modal conversational abilities, capable of following speech-and-language instructions. Our early experiments show that LLaSM demonstrates a more convenient and natural way for humans to interact with artificial intelligence. Specifically, we also release a large Speech Instruction Following dataset LLaSM-Audio-Instructions. Code and demo are available at https://github.com/LinkSoul-AI/LLaSM and https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions dataset is available at https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
【5】 ASTER: Automatic Speech Recognition System Accessibility Testing for  Stutterers标题：ASTER：口吃者的自动语音识别系统可访问性测试链接：https://arxiv.org/abs/2308.15742
作者：Yi Liu,Yuekang Li,Gelei Deng,Felix Juefei-Xu,Yao Du,Cen Zhang,Chengwei Liu,Yeting Li,Lei Ma,Yang Liu机构：∗Nanyang Technological University, †University of New South Wales, ‡Meta AI, §University of Southern California, ¶Institute of Information Engineering, Chinese Academy of Sciences;摘要：当今自动语音识别（ASR）系统的普及导致对提高其可访问性的需求日益增加。处理口吃语音是可访问ASR系统的重要特征。为了提高语音识别系统对口吃者的可及性，我们需要揭示和分析语音识别系统对口吃的失败。从口吃者记录的语音数据集不够多样化，无法暴露大多数故障。此外，这些数据集缺乏关于非口吃文本的地面真实信息，使得它们不适合作为综合测试套件。因此，需要一种用于生成口吃语音作为测试输入以测试和分析ASR系统的性能的方法。然而，在这种情况下，生成有效的测试输入是具有挑战性的。原因在于，尽管生成的测试输入应该模仿口吃者说话的方式，但它们也应该足够多样化，以触发更多的失败。为了解决这个挑战，我们提出了ASTER，自动测试ASR系统的可访问性的技术。ASTER可以通过注入五种不同类型的口吃来生成有效的测试用例。生成的测试用例既可以模拟真实的口吃语音，也可以暴露ASR系统中的故障。此外，ASTER可以进一步提高测试用例的质量与多目标优化为基础的种子更新算法。我们实现ASTER作为一个框架，并评估了四个开源ASR模型和三个商业ASR系统。我们进行了一个全面的评估ASTER，发现它显着增加字错误率，匹配错误率，和字信息损失评估ASR系统。此外，我们的用户研究表明，生成的口吃音频是无法区分的现实世界的口吃音频剪辑。摘要：The popularity of automatic speech recognition (ASR) systems nowadays leads to an increasing need for improving their accessibility. Handling stuttering speech is an important feature for accessible ASR systems. To improve the accessibility of ASR systems for stutterers, we need to expose and analyze the failures of ASR systems on stuttering speech. The speech datasets recorded from stutterers are not diverse enough to expose most of the failures. Furthermore, these datasets lack ground truth information about the non-stuttered text, rendering them unsuitable as comprehensive test suites. Therefore, a methodology for generating stuttering speech as test inputs to test and analyze the performance of ASR systems is needed. However, generating valid test inputs in this scenario is challenging. The reason is that although the generated test inputs should mimic how stutterers speak, they should also be diverse enough to trigger more failures. To address the challenge, we propose ASTER, a technique for automatically testing the accessibility of ASR systems. ASTER can generate valid test cases by injecting five different types of stuttering. The generated test cases can both simulate realistic stuttering speech and expose failures in ASR systems. Moreover, ASTER can further enhance the quality of the test cases with a multi-objective optimization-based seed updating algorithm. We implemented ASTER as a framework and evaluated it on four open-source ASR models and three commercial ASR systems. We conduct a comprehensive evaluation of ASTER and find that it significantly increases the word error rate, match error rate, and word information loss in the evaluated ASR systems. Additionally, our user study demonstrates that the generated stuttering audio is indistinguishable from real-world stuttering audio clips.
【6】 AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition标题：AGS：一种用于国内场景声事件识别的数据集和分类方法链接：https://arxiv.org/abs/2308.15726
作者：Nan Che,Chenrui Liu,Fei Yu机构：Department of Computing Science and Technology, Harbin University of Science and Technology, Harbin, China,  Zhejiang Lab, Hangzhou, China摘要：环境声场景和声事件识别对于室内外环境（如托儿所、智能家居、养老院等）可疑事件的识别具有重要意义并且是许多音频监视应用中涉及的基本任务。特别是，对于室内环境声场景的数据集，目前还没有公开的公共数据集用于声事件识别的研究领域。因此，本文提出了一个数据集（称为AGS）的家庭环境声音。该数据集考虑场景中各种类型的重叠音频、背景噪声。在此基础上，本文比较分析了各种先进的声音事件识别方法，说明了本文提出的数据集的可靠性，并研究了新数据集带来的挑战。我们建议的AGS和相应基线的源代码在https://github.com/taolunzu11/AGS。摘要：Environmental sound scene and sound event recognition is important for the recognition of suspicious events in indoor and outdoor environments (such as nurseries, smart homes, nursing homes, etc.) and is a fundamental task involved in many audio surveillance applications. In particular, there is no public common data set for the research field of sound event recognition for the data set of the indoor environmental sound scene. Therefore, this paper proposes a data set (called as AGS) for the home environment sound. This data set considers various types of overlapping audio in the scene, background noise. Moreover, based on the proposed data set, this paper compares and analyzes the advanced methods for sound event recognition, and then illustrates the reliability of the data set proposed in this paper, and studies the challenges raised by the new data set. Our proposed AGS and the source code of the corresponding baselines at https://github.com/taolunzu11/AGS .
【7】 The DeepZen Speech Synthesis System for Blizzard Challenge 2023标题：2023暴雪挑战赛DeepZen语音合成系统链接：https://arxiv.org/abs/2308.15945
作者：Christophe Veaux,Ranniery Maia,Spyridoula Papendreou机构：DeepZen, UK备注：Blizzard Challenge 2023摘要：本文介绍了2023年暴雪挑战赛的DeepZen文本到语音（TTS）系统。这项挑战的目标是从大型单说话人数据集（集线器任务）和通过说话人适应（辐条任务）从较小的数据集合成自然和高质量的法语语音。我们使用相同的模型架构参与了这两项任务。我们的方法是使用自回归模型，它保留了产生自然声音语音的优势，但以几种方式改善韵律控制。与非注意Tacotron类似，该模型在推理时使用持续时间预测器和高斯上采样，但具有更简单的无监督训练。我们还模型的说话风格在句子和单词的水平，通过提取全局和局部风格令牌从参考语音。在推理时，全局和局部样式标记是从在文本上运行的BERT模型预测的。该BERT模型也用于预测特定的发音特征，如schwa省略和可选的联系。最后，在大型公共数据集上训练并对目标语音进行微调的HifiGAN的修改版本用于生成语音波形。我们的团队在暴雪评估中被确定为O，MUSHRA测试结果显示，我们的系统在中心任务（中位数得分为0.75）和辐条任务（中位数得分为0.68）中分别在18和14名参与者中执行第二次公平。摘要：This paper describes the DeepZen text to speech (TTS) system for Blizzard Challenge 2023. The goal of this challenge is to synthesise natural and high-quality speech in French, from a large monospeaker dataset (hub task) and from a smaller dataset by speaker adaptation (spoke task). We participated to both tasks with the same model architecture. Our approach has been to use an auto-regressive model, which retains an advantage for generating natural sounding speech but to improve prosodic control in several ways. Similarly to non-attentive Tacotron, the model uses a duration predictor and gaussian upsampling at inference, but with a simpler unsupervised training. We also model the speaking style at both sentence and word levels by extracting global and local style tokens from the reference speech. At inference, the global and local style tokens are predicted from a BERT model run on text. This BERT model is also used to predict specific pronunciation features like schwa elision and optional liaisons. Finally, a modified version of HifiGAN trained on a large public dataset and fine-tuned on the target voices is used to generate speech waveform. Our team is identified as O in the the Blizzard evaluation and MUSHRA test results show that our system performs second ex aequo in both hub task (median score of 0.75) and spoke task (median score of 0.68), over 18 and 14 participants, respectively.
eess.AS音频处理
【1】 The DeepZen Speech Synthesis System for Blizzard Challenge 2023标题：2023暴雪挑战赛DeepZen语音合成系统链接：https://arxiv.org/abs/2308.15945
作者：Christophe Veaux,Ranniery Maia,Spyridoula Papendreou机构：DeepZen, UK备注：Blizzard Challenge 2023摘要：本文介绍了2023年暴雪挑战赛的DeepZen文本到语音（TTS）系统。这项挑战的目标是从大型单说话人数据集（集线器任务）和通过说话人适应（辐条任务）从较小的数据集合成自然和高质量的法语语音。我们使用相同的模型架构参与了这两项任务。我们的方法是使用自回归模型，它保留了产生自然声音语音的优势，但以几种方式改善韵律控制。与非注意Tacotron类似，该模型在推理时使用持续时间预测器和高斯上采样，但具有更简单的无监督训练。我们还模型的说话风格在句子和单词的水平，通过提取全局和局部风格令牌从参考语音。在推理时，全局和局部样式标记是从在文本上运行的BERT模型预测的。该BERT模型也用于预测特定的发音特征，如schwa省略和可选的联系。最后，在大型公共数据集上训练并对目标语音进行微调的HifiGAN的修改版本用于生成语音波形。我们的团队在暴雪评估中被确定为O，MUSHRA测试结果显示，我们的系统在中心任务（中位数得分为0.75）和辐条任务（中位数得分为0.68）中分别在18和14名参与者中执行第二次公平。摘要：This paper describes the DeepZen text to speech (TTS) system for Blizzard Challenge 2023. The goal of this challenge is to synthesise natural and high-quality speech in French, from a large monospeaker dataset (hub task) and from a smaller dataset by speaker adaptation (spoke task). We participated to both tasks with the same model architecture. Our approach has been to use an auto-regressive model, which retains an advantage for generating natural sounding speech but to improve prosodic control in several ways. Similarly to non-attentive Tacotron, the model uses a duration predictor and gaussian upsampling at inference, but with a simpler unsupervised training. We also model the speaking style at both sentence and word levels by extracting global and local style tokens from the reference speech. At inference, the global and local style tokens are predicted from a BERT model run on text. This BERT model is also used to predict specific pronunciation features like schwa elision and optional liaisons. Finally, a modified version of HifiGAN trained on a large public dataset and fine-tuned on the target voices is used to generate speech waveform. Our team is identified as O in the the Blizzard evaluation and MUSHRA test results show that our system performs second ex aequo in both hub task (median score of 0.75) and spoke task (median score of 0.68), over 18 and 14 participants, respectively.
【2】 DeFTAN-II: Efficient Multichannel Speech Enhancement with Subgroup  Processing标题：DeFTAN-II：对子群处理的高效多通道语音增强链接：https://arxiv.org/abs/2308.15777
作者：Dongheon Lee,Jung-Woo Choi备注：13 pages, 6 figures, submitted to IEEE/ACM Trans. Audio, Speech, Lang. Process摘要：在这项工作中，我们提出了DeFTAN-II，一个有效的多通道语音增强模型的基础上Transformer架构和子群处理。尽管Transformers在语音增强方面取得了成功，但它们在捕获本地关系、降低计算复杂度和降低内存使用方面面临着挑战。为了解决这些限制，我们引入了子群处理在我们的模型中，将局部强调的功能与其他子群包含原始功能相结合。子组处理在所提出的网络的几个块中实现。在提出的分裂密集块提取空间特征，一对子组顺序连接和处理的卷积层，以有效地降低计算复杂度和内存使用。对于提取时间和频谱关系的F-和T-变换器，我们引入子组之间的交叉注意，以确定局部强调和非强调功能之间的关系。然后，双路径前馈网络根据由扩张卷积处理的局部特征的门控来聚合出席的特征。通过与最先进的多通道语音增强模型进行广泛的比较，我们证明了具有子组处理的DeFTAN-II在显着降低计算复杂度的情况下优于现有方法。此外，我们评估了模型的泛化能力，对现实世界的数据没有微调，这进一步证明了它在实际情况下的有效性。摘要：In this work, we present DeFTAN-II, an efficient multichannel speech enhancement model based on transformer architecture and subgroup processing. Despite the success of transformers in speech enhancement, they face challenges in capturing local relations, reducing the high computational complexity, and lowering memory usage. To address these limitations, we introduce subgroup processing in our model, combining subgroups of locally emphasized features with other subgroups containing original features. The subgroup processing is implemented in several blocks of the proposed network. In the proposed split dense blocks extracting spatial features, a pair of subgroups is sequentially concatenated and processed by convolution layers to effectively reduce the computational complexity and memory usage. For the F- and T-transformers extracting temporal and spectral relations, we introduce cross-attention between subgroups to identify relationships between locally emphasized and non-emphasized features. The dual-path feedforward network then aggregates attended features in terms of the gating of local features processed by dilated convolutions. Through extensive comparisons with state-of-the-art multichannel speech enhancement models, we demonstrate that DeFTAN-II with subgroup processing outperforms existing methods at significantly lower computational complexity. Moreover, we evaluate the model's generalization capability on real-world data without fine-tuning, which further demonstrates its effectiveness in practical scenarios.
【3】 General Purpose Audio Effect Removal标题：通用音效去除链接：https://arxiv.org/abs/2308.16177
作者：Matthew Rice,Christian J. Steinmetz,George Fazekas,Joshua D. Reiss机构：Centre for Digital Music, Queen Mary University of London, UK备注：Preprint. Accepted to IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) 2023摘要：虽然音频效果的设计和应用是很好的理解，消除这些效果的逆问题显着更具挑战性和少得多的研究。最近，深度学习被应用到音频效果去除;然而，现有的方法集中于一次仅考虑一种效应或源类型的狭义公式。在现实场景中，多种效果应用于不同的源内容。这激发了更一般的任务，我们称之为通用音频效果去除。我们为这项任务开发了一个数据集，使用四个不同来源的五个音频效果，并使用它来训练和评估一组现有的架构。我们发现，没有一个单一的模型表现最佳的所有影响类型和来源。为了解决这个问题，我们引入了RemFX，一种旨在反映应用效果组合性的方法。我们首先训练了一组性能最佳的特定效果去除模型，然后利用音频效果分类模型在推理时动态构建模型的图。我们发现我们的方法优于单一模型基线，尽管存在许多影响的例子仍然具有挑战性。摘要：Although the design and application of audio effects is well understood, the inverse problem of removing these effects is significantly more challenging and far less studied. Recently, deep learning has been applied to audio effect removal; however, existing approaches have focused on narrow formulations considering only one effect or source type at a time. In realistic scenarios, multiple effects are applied with varying source content. This motivates a more general task, which we refer to as general purpose audio effect removal. We developed a dataset for this task using five audio effects across four different sources and used it to train and evaluate a set of existing architectures. We found that no single model performed optimally on all effect types and sources. To address this, we introduced RemFX, an approach designed to mirror the compositionality of applied effects. We first trained a set of the best-performing effect-specific removal models and then leveraged an audio effect classification model to dynamically construct a graph of our models at inference. We found our approach to outperform single model baselines, although examples with many effects present remain challenging.
【4】 CALM: Contrastive Cross-modal Speaking Style Modeling for Expressive  Text-to-Speech Synthesis标题：CAMLAM：用于表现力文语合成的对比跨通道语音风格建模链接：https://arxiv.org/abs/2308.16021
作者：Yi Meng,Xiang Li,Zhiyong Wu,Tingtian Li,Zixun Sun,Xinyu Xiao,Chi Sun,Hui Zhan,Helen Meng机构：Tsinghua-CUHK Joint Research Center for Media Sciences, Technologies and Systems, Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Department of Systems Engineering and Engineering Management备注：Accepted by InterSpeech 2022摘要：为了进一步改善合成语音的说话风格，当前的文本到语音（TTS）合成系统通常采用参考语音来风格化其输出而不是仅输入文本。这些参考语音是通过人工选择获得的，这是消耗资源的，或通过语义特征选择。然而，语义特征不仅包含与风格相关的信息，还包含与风格无关的信息。文本中与说话风格无关的信息会干扰参考音频的选择，导致说话风格不正确。为了提高参考文献的选择，我们提出了对比声学语言模块（CALM）提取的风格相关的文本特征（STF）的文本。CALM通过对比学习优化说话风格嵌入和提取的STF之间的相关性。因此，通过检索具有最高STF相似性的语音来选择用于输入文本的特定数量的最合适的参考语音。然后，根据STF相似度对风格嵌入进行加权总结，并将其用于合成语音的风格化。实验结果表明，我们所提出的方法的有效性，与客观评价和主观评价的合成语音说话风格优于基线方法与基于语义特征的参考选择。摘要：To further improve the speaking styles of synthesized speeches, current text-to-speech (TTS) synthesis systems commonly employ reference speeches to stylize their outputs instead of just the input texts. These reference speeches are obtained by manual selection which is resource-consuming, or selected by semantic features. However, semantic features contain not only style-related information, but also style irrelevant information. The information irrelevant to speaking style in the text could interfere the reference audio selection and result in improper speaking styles. To improve the reference selection, we propose Contrastive Acoustic-Linguistic Module (CALM) to extract the Style-related Text Feature (STF) from the text. CALM optimizes the correlation between the speaking style embedding and the extracted STF with contrastive learning. Thus, a certain number of the most appropriate reference speeches for the input text are selected by retrieving the speeches with the top STF similarities. Then the style embeddings are weighted summarized according to their STF similarities and used to stylize the synthesized speech of TTS. Experiment results demonstrate the effectiveness of our proposed approach, with both objective evaluations and subjective evaluations on the speaking styles of the synthesized speeches outperform a baseline approach with semantic-feature-based reference selection.
【5】 Dual-path Transformer Based Neural Beamformer for Target Speech  Extraction标题：基于双路径转换器的神经波束形成器用于目标语音提取链接：https://arxiv.org/abs/2308.15990
作者：Aoqi Guo,Sichong Qian,Baoxiang Li,Dazhi Gao机构：Ocean University of China, Qingdao, China, † SenseTime Research, Beijing, China摘要：神经波束形成器，集成了预分离和波束形成模块，在目标语音提取任务中显示出令人印象深刻的功效。然而，这些波束形成器的性能固有地受到预分离模块的预测精度的约束。在本文中，我们介绍了一个神经波束形成器支撑的双路径Transformer器。首先，我们利用时域中的交叉注意机制，提取关键的空间信息，波束形成的噪声协方差矩阵。随后，在频域中，自我注意机制，以加强该模型的能力，以处理特定频率的细节。通过设计，我们的模型避免了预分离模块的影响，以更全面的端到端方式提供性能。实验结果表明，我们的模型不仅超过当代领先的神经波束形成算法的分离性能，但也实现了这一点显着减少的参数计数。摘要：Neural beamformers, integrating both pre-separation and beamforming modules, have shown impressive efficacy in the target speech extraction task. Nevertheless, the performance of these beamformers is inherently constrained by the predictive accuracy of the pre-separation module. In this paper, we introduce a neural beamformer underpinned by a dual-path transformer. Initially, we harness the cross-attention mechanism in the time domain, extracting pivotal spatial information related to beamforming from the noisy covariance matrix. Subsequently, in the frequency domain, the self-attention mechanism is employed to bolster the model's capacity to process frequency-specific details. By design, our model circumvents the influence of pre-separation modules, delivering the performance in a more holistic end-to-end fashion. Experimental results reveal that our model not only surpasses contemporary leading neural beamforming algorithms in separation performance, but also achieves this with a notable reduction in parameter count.
【6】 LLaSM: Large Language and Speech Model标题：LLaSM：大型语言和语音模型链接：https://arxiv.org/abs/2308.15930
作者：Yu Shu,Siwei Dong,Guangyao Chen,Wenhao Huang,Ruihua Zhang,Daochen Shi,Qiqi Xiang,Yemin Shi机构：LinkSoul.AI,Peking University,.ai, † Equal contribution摘要：多模态大型语言模型最近引起了人们的极大兴趣。虽然，大多数的作品集中在视觉语言多模态模型提供强大的能力，遵循视觉和语言的指令。然而，我们认为，语音也是一个重要的方式，通过它人类与世界互动。因此，对于通用助理来说，能够遵循多模态语音和语言指令是至关重要的。在这项工作中，我们提出了大语言和语音模型（LLaSM）。LLaSM是一个端到端训练的大型多模态语音语言模型，具有跨模态会话能力，能够遵循语音和语言指令。我们的早期实验表明，LLaSM展示了人类与人工智能交互的更方便和自然的方式。具体来说，我们还发布了一个大型的语音指令跟随数据集LLaSM-Audio-Instructions。代码和演示可在https://github.com/LinkSoul-AI/LLaSM和https://huggingface.co/spaces/LinkSoul/LLaSM上获得。LLaSM-Audio-Instructions数据集可从https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions获得。摘要：Multi-modal large language models have garnered significant interest recently. Though, most of the works focus on vision-language multi-modal models providing strong capabilities in following vision-and-language instructions. However, we claim that speech is also an important modality through which humans interact with the world. Hence, it is crucial for a general-purpose assistant to be able to follow multi-modal speech-and-language instructions. In this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an end-to-end trained large multi-modal speech-language model with cross-modal conversational abilities, capable of following speech-and-language instructions. Our early experiments show that LLaSM demonstrates a more convenient and natural way for humans to interact with artificial intelligence. Specifically, we also release a large Speech Instruction Following dataset LLaSM-Audio-Instructions. Code and demo are available at https://github.com/LinkSoul-AI/LLaSM and https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions dataset is available at https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
【7】 ASTER: Automatic Speech Recognition System Accessibility Testing for  Stutterers标题：ASTER：口吃者的自动语音识别系统可访问性测试链接：https://arxiv.org/abs/2308.15742
作者：Yi Liu,Yuekang Li,Gelei Deng,Felix Juefei-Xu,Yao Du,Cen Zhang,Chengwei Liu,Yeting Li,Lei Ma,Yang Liu机构：∗Nanyang Technological University, †University of New South Wales, ‡Meta AI, §University of Southern California, ¶Institute of Information Engineering, Chinese Academy of Sciences;摘要：当今自动语音识别（ASR）系统的普及导致对提高其可访问性的需求日益增加。处理口吃语音是可访问ASR系统的重要特征。为了提高语音识别系统对口吃者的可及性，我们需要揭示和分析语音识别系统对口吃的失败。从口吃者记录的语音数据集不够多样化，无法暴露大多数故障。此外，这些数据集缺乏关于非口吃文本的地面真实信息，使得它们不适合作为综合测试套件。因此，需要一种用于生成口吃语音作为测试输入以测试和分析ASR系统的性能的方法。然而，在这种情况下，生成有效的测试输入是具有挑战性的。原因在于，尽管生成的测试输入应该模仿口吃者说话的方式，但它们也应该足够多样化，以触发更多的失败。为了解决这个挑战，我们提出了ASTER，自动测试ASR系统的可访问性的技术。ASTER可以通过注入五种不同类型的口吃来生成有效的测试用例。生成的测试用例既可以模拟真实的口吃语音，也可以暴露ASR系统中的故障。此外，ASTER可以进一步提高测试用例的质量与多目标优化为基础的种子更新算法。我们实现ASTER作为一个框架，并评估了四个开源ASR模型和三个商业ASR系统。我们进行了一个全面的评估ASTER，发现它显着增加字错误率，匹配错误率，和字信息损失评估ASR系统。此外，我们的用户研究表明，生成的口吃音频是无法区分的现实世界的口吃音频剪辑。摘要：The popularity of automatic speech recognition (ASR) systems nowadays leads to an increasing need for improving their accessibility. Handling stuttering speech is an important feature for accessible ASR systems. To improve the accessibility of ASR systems for stutterers, we need to expose and analyze the failures of ASR systems on stuttering speech. The speech datasets recorded from stutterers are not diverse enough to expose most of the failures. Furthermore, these datasets lack ground truth information about the non-stuttered text, rendering them unsuitable as comprehensive test suites. Therefore, a methodology for generating stuttering speech as test inputs to test and analyze the performance of ASR systems is needed. However, generating valid test inputs in this scenario is challenging. The reason is that although the generated test inputs should mimic how stutterers speak, they should also be diverse enough to trigger more failures. To address the challenge, we propose ASTER, a technique for automatically testing the accessibility of ASR systems. ASTER can generate valid test cases by injecting five different types of stuttering. The generated test cases can both simulate realistic stuttering speech and expose failures in ASR systems. Moreover, ASTER can further enhance the quality of the test cases with a multi-objective optimization-based seed updating algorithm. We implemented ASTER as a framework and evaluated it on four open-source ASR models and three commercial ASR systems. We conduct a comprehensive evaluation of ASTER and find that it significantly increases the word error rate, match error rate, and word information loss in the evaluated ASR systems. Additionally, our user study demonstrates that the generated stuttering audio is indistinguishable from real-world stuttering audio clips.
【8】 AGS: An Dataset and Taxonomy for Domestic Scene Sound Event Recognition标题：AGS：一种用于国内场景声事件识别的数据集和分类方法链接：https://arxiv.org/abs/2308.15726
作者：Nan Che,Chenrui Liu,Fei Yu机构：Department of Computing Science and Technology, Harbin University of Science and Technology, Harbin, China,  Zhejiang Lab, Hangzhou, China摘要：环境声场景和声事件识别对于室内外环境（如托儿所、智能家居、养老院等）可疑事件的识别具有重要意义并且是许多音频监视应用中涉及的基本任务。特别是，对于室内环境声场景的数据集，目前还没有公开的公共数据集用于声事件识别的研究领域。因此，本文提出了一个数据集（称为AGS）的家庭环境声音。该数据集考虑场景中各种类型的重叠音频、背景噪声。在此基础上，本文比较分析了各种先进的声音事件识别方法，说明了本文提出的数据集的可靠性，并研究了新数据集带来的挑战。我们建议的AGS和相应基线的源代码在https://github.com/taolunzu11/AGS。摘要：Environmental sound scene and sound event recognition is important for the recognition of suspicious events in indoor and outdoor environments (such as nurseries, smart homes, nursing homes, etc.) and is a fundamental task involved in many audio surveillance applications. In particular, there is no public common data set for the research field of sound event recognition for the data set of the indoor environmental sound scene. Therefore, this paper proposes a data set (called as AGS) for the home environment sound. This data set considers various types of overlapping audio in the scene, background noise. Moreover, based on the proposed data set, this paper compares and analyzes the advanced methods for sound event recognition, and then illustrates the reliability of the data set proposed in this paper, and studies the challenges raised by the new data set. Our proposed AGS and the source code of the corresponding baselines at https://github.com/taolunzu11/AGS .
机器翻译，仅供参考