今日论文合集：cs.SD语音9篇，eess.AS音频处理16篇。
cs.SD语音
【1】 Evaluation of Speech Representations for MOS prediction标题：用于MOS预测的语音表示的评估链接：https://arxiv.org/abs/2306.09979
作者：Frederico S. Oliveira,Edresson Casanova,Arnaldo Cândido Júnior,Lucas R. S. Gris,Anderson S. Soares,Arlindo R. Galvão Filho机构：UFG, Goi´as - GO - Brazil备注：12 pages, 4 figures, Accepted to the 26th International Conference of Text, Speech and Dialogue (TSD2023)摘要：在本文中，我们评估特征提取模型预测语音质量。我们还提出了一个模型架构来比较嵌入的监督学习和自我监督学习模型与嵌入的说话人验证模型，以预测度量MOS。我们的实验是在VCC 2018数据集和为这项工作创建的名为BRSpeechMOS的巴西-葡萄牙数据集上进行的。结果表明，Whisper模型适用于所有场景：使用VCC 2018和BRSpeech-MOS数据集。在使用BRSpeechMOS的监督和自监督学习模型中，Whisper-Small的线性相关性最好，为0.6980，说话人验证模型SpeakerNet的线性相关性为0.6963。使用VCC 2018，最佳监督和自我监督学习模型Whisper-Large实现了0.7274的线性相关性，最佳模型说话人验证TitaNet实现了0.6933的线性相关性。虽然说话人验证模型的结果略低，但SpeakerNet模型只有5 M个参数，适合实时应用，TitaNet模型产生的嵌入大小为192，是所有评估模型中最小的。实验结果是可重复的公开可用的源代码1。摘要：In this paper, we evaluate feature extraction models for predicting speech quality. We also propose a model architecture to compare embeddings of supervised learning and self-supervised learning models with embeddings of speaker verification models to predict the metric MOS. Our experiments were performed on the VCC2018 dataset and a Brazilian-Portuguese dataset called BRSpeechMOS, which was created for this work. The results show that the Whisper model is appropriate in all scenarios: with both the VCC2018 and BRSpeech- MOS datasets. Among the supervised and self-supervised learning models using BRSpeechMOS, Whisper-Small achieved the best linear correlation of 0.6980, and the speaker verification model, SpeakerNet, had linear correlation of 0.6963. Using VCC2018, the best supervised and self-supervised learning model, Whisper-Large, achieved linear correlation of 0.7274, and the best model speaker verification, TitaNet, achieved a linear correlation of 0.6933. Although the results of the speaker verification models are slightly lower, the SpeakerNet model has only 5M parameters, making it suitable for real-time applications, and the TitaNet model produces an embedding of size 192, the smallest among all the evaluated models. The experiment results are reproducible with publicly available source-code1 .
【2】 RealImpact: A Dataset of Impact Sound Fields for Real Objects标题：RealImpact：真实物体碰撞声场数据集链接：https://arxiv.org/abs/2306.09944
作者：Samuel Clarke,Ruohan Gao,Mason Wang,Mark Rau,Julia Xu,Jui-Hsien Wang,Doug L. James,Jiajun Wu机构：Stanford University, Adobe Research备注：CVPR 2023 (Highlight). Project page: this https URL摘要：物体在不同的扰动、环境条件和相对于听者的姿势下发出独特的声音。虽然先前的工作已经在模拟中对冲击声音和声音传播进行了建模，但我们缺乏真实对象的冲击声场的标准数据集，用于视听学习和模拟到真实差距的校准。我们提出了RealImpact，一个大规模的数据集，真实物体的冲击声音记录在受控条件下。RealImpact包含50个日常物体的150，000个撞击声录音，并附有详细的注释，包括其撞击位置，麦克风位置，接触力分布，材料标签和RGBD图像。我们初步尝试使用我们的数据集作为参考，目前的模拟方法估计对象的影响声音，符合现实世界。此外，我们证明了我们的数据集作为声学和视听学习的测试平台的有用性，通过两个基准任务的评估，包括听者位置分类和视觉声学匹配。摘要：Objects make unique sounds under different perturbations, environment conditions, and poses relative to the listener. While prior works have modeled impact sounds and sound propagation in simulation, we lack a standard dataset of impact sound fields of real objects for audio-visual learning and calibration of the sim-to-real gap. We present RealImpact, a large-scale dataset of real object impact sounds recorded under controlled conditions. RealImpact contains 150,000 recordings of impact sounds of 50 everyday objects with detailed annotations, including their impact locations, microphone locations, contact force profiles, material labels, and RGBD images. We make preliminary attempts to use our dataset as a reference to current simulation methods for estimating object impact sounds that match the real world. Moreover, we demonstrate the usefulness of our dataset as a testbed for acoustic and audio-visual learning via the evaluation of two benchmark tasks, including listener location classification and visual acoustic matching.
【3】 Correlation Clustering of Bird Sounds标题：鸟鸣的相关聚类链接：https://arxiv.org/abs/2306.09906
作者：David Stein,Bjoern Andres机构：TU Dresden备注：13 pages摘要：鸟类声音分类是将任何录音与录音中可以听到的鸟类种类相关联的任务。在这里，我们研究鸟类的声音聚类，决定任何一对声音记录是否同一种鸟类可以听到两者的任务。我们解决这个问题，首先学习，从一个训练集，以这种方式相关的记录对的概率，然后推断一个最大可能的分区的测试集的相关聚类。我们解决以下问题：与测试集的分类相比，这种聚类的准确性有多高？这样推断出的聚类如何与通过分类获得的聚类相关联？当应用于训练期间没有听到的鸟类的录音时，这种聚类的准确性如何？这种聚类在将训练期间未听到的环境噪声与鸟鸣分离方面的有效性如何？摘要：Bird sound classification is the task of relating any sound recording to those species of bird that can be heard in the recording. Here, we study bird sound clustering, the task of deciding for any pair of sound recordings whether the same species of bird can be heard in both. We address this problem by first learning, from a training set, probabilities of pairs of recordings being related in this way, and then inferring a maximally probable partition of a test set by correlation clustering. We address the following questions: How accurate is this clustering, compared to a classification of the test set? How do the clusters thus inferred relate to the clusters obtained by classification? How accurate is this clustering when applied to recordings of bird species not heard during training? How effective is this clustering in separating, from bird sounds, environmental noise not heard during training?
【4】 CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained  Language-Vision Models标题：CLIPSonic：无标签视频和预先训练的语言-视觉模型的文本到音频合成链接：https://arxiv.org/abs/2306.09635
作者：Hao-Wen Dong,Xiaoyu Liu,Jordi Pons,Gautam Bhattacharya,Santiago Pascual,Joan Serrà,Taylor Berg-Kirkpatrick,Julian McAuley机构：Joan Serra,  Dolby Laboratories,  University of California San Diego摘要：最近的工作已经研究了文本到音频合成使用大量的成对的文本音频数据。然而，具有高质量文本注释的音频记录可能难以获取。在这项工作中，我们使用未标记的视频和预训练的语言视觉模型来实现文本到音频的合成。我们建议学习所需的文本音频对应，利用视觉模态作为桥梁。我们训练一个条件扩散模型来生成视频的音轨，给定一个由预训练的对比语言图像预训练（CLIP）模型编码的视频帧。在测试时，我们首先探索执行zero-shot模态转移和条件的扩散模型与CLIP编码的文本查询。然而，我们观察到一个显着的性能下降，相对于图像查询。为了缩小这一差距，我们进一步采用预训练的扩散先验模型来生成CLIP图像嵌入，给定CLIP文本嵌入。我们的结果表明，所提出的方法的有效性，预训练的扩散先验可以减少模态转移间隙。虽然我们专注于文本到音频的合成，所提出的模型也可以从图像查询生成音频，它显示出竞争力的表现对国家的最先进的图像到音频的合成模型在主观听力测试。这项研究为文本到音频的合成提供了一个新的方向，它利用了视频中自然发生的视听对应关系和预训练语言视觉模型的功能。摘要：Recent work has studied text-to-audio synthesis using large amounts of paired text-audio data. However, audio recordings with high-quality text annotations can be difficult to acquire. In this work, we approach text-to-audio synthesis using unlabeled videos and pretrained language-vision models. We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining (CLIP) model. At test time, we first explore performing a zero-shot modality transfer and condition the diffusion model with a CLIP-encoded text query. However, we observe a noticeable performance drop with respect to image queries. To close this gap, we further adopt a pretrained diffusion prior model to generate a CLIP image embedding given a CLIP text embedding. Our results show the effectiveness of the proposed method, and that the pretrained diffusion prior can reduce the modality transfer gap. While we focus on text-to-audio synthesis, the proposed model can also generate audio from image queries, and it shows competitive performance against a state-of-the-art image-to-audio synthesis model in a subjective listening test. This study offers a new direction of approaching text-to-audio synthesis that leverages the naturally-occurring audio-visual correspondence in videos and the power of pretrained language-vision models.
【5】 Competitive and Resource Efficient Factored Hybrid HMM Systems are  Simpler Than You Think标题：竞争性和资源高效的混合HMM系统比你想象的要简单链接：https://arxiv.org/abs/2306.09517
作者：Tina Raissi,Christoph Lüscher,Moritz Gunz,Ralf Schlüter,Hermann Ney机构：Machine Learning and Human Language Technology, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：Accepted for presentation at InterSpeech 2023摘要：为自动语音识别（ASR）构建竞争性混合隐马尔可夫模型（HMM）系统需要一个复杂的多级流水线，该流水线由多个训练准则组成。最近的序列到序列模型提供了具有可以从头开始的更简单的管道的优点。我们提出了一个纯粹的基于神经的单级从头管道上下文相关的混合HMM，提供了类似的简单性。我们使用的对齐从一个完整的总和训练零阶后验HMM与BLSTM编码器。我们表明，与此对齐，我们可以建立一个构象因子的混合，表现甚至优于一个国家的最先进的经典的混合和一个因子的混合训练与更复杂的基于高斯混合系统的对齐。我们的发现证实了Switchboard 300h和LibriSpeech 960h任务与文献中的其他方法具有可比性的结果，并通过额外依赖于可用计算资源的负责任的选择。摘要：Building competitive hybrid hidden Markov model~(HMM) systems for automatic speech recognition~(ASR) requires a complex multi-stage pipeline consisting of several training criteria. The recent sequence-to-sequence models offer the advantage of having simpler pipelines that can start from-scratch. We propose a purely neural based single-stage from-scratch pipeline for a context-dependent hybrid HMM that offers similar simplicity. We use an alignment from a full-sum trained zero-order posterior HMM with a BLSTM encoder. We show that with this alignment we can build a Conformer factored hybrid that performs even better than both a state-of-the-art classic hybrid and a factored hybrid trained with alignments taken from more complex Gaussian mixture based systems. Our finding is confirmed on Switchboard 300h and LibriSpeech 960h tasks with comparable results to other approaches in the literature, and by additionally relying on a responsible choice of available computational resources.
【6】 Sound Demixing Challenge 2023 -- Music Demixing Track Technical Report标题：声音分离挑战赛2023--音乐分离轨道技术报告链接：https://arxiv.org/abs/2306.09382
作者：Minseok Kim,Jun Hyung Lee机构：Department of Computer Science, Korea University备注：4 pages, 3 tables摘要：在这份报告中，我们将为2023年的音乐解混合挑战赛（Music Demixing Track of Sound Demixing Challenge 2023）展示获奖解决方案。我们专注于为这一挑战而设计的两种方法：一个时间效率高的源分离网络，实现了MUSDB基准测试的最先进的结果和噪声鲁棒源分离的损失掩蔽方法。用于再现模型训练和最终提交的代码可在www.example.com上获得github.com/kuielab/sdx23。摘要：In this report, we present our award-winning solutions for the Music Demixing Track of Sound Demixing Challenge 2023. We focus on two methods designed for this challenge: a time-efficient source separation network that achieves state-of-the-art results on the MUSDB benchmark and a loss masking method for noise-robust source separation. Code for reproducing model training and final submissions is available at github.com/kuielab/sdx23.
【7】 Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances标题：基于文本的音频检索相关度的众包和评估链接：https://arxiv.org/abs/2306.09820
作者：Huang Xie,Khazar Khorrami,Okko Räsänen,Tuomas Virtanen机构：Unit of Computing Sciences, Tampere University, Finland摘要：本文探讨了基于文本的音频检索相关性与众包评估的分级。给定自由形式的文本（例如，字幕）作为查询，要求众包工作者使用数字分数（在0和100之间）来对音频剪辑进行分级，以指示他们对音频剪辑的声音内容与文本匹配的程度的判断，其中0指示完全没有内容匹配，并且100指示完美的内容匹配。我们将众包的相关性训练和评估基于文本的音频检索系统，并评估使用它们与二进制相关性从音频字幕的效果。常规地，这些二进制相关性由基于字幕的音频-字幕对定义，其中正指示字幕描述配对的音频，并且负适用于所有其他对。实验结果表明，当众包相关性被二值化用于对比学习时，将众包相关性与二值相关性合并并没有明显的益处。相反，结果表明，仅使用基于字幕的音频字幕对定义的二进制相关性是足够的对比学习。摘要：This paper explores grading text-based audio retrieval relevances with crowdsourcing assessments. Given a free-form text (e.g., a caption) as a query, crowdworkers are asked to grade audio clips using numeric scores (between 0 and 100) to indicate their judgements of how much the sound content of an audio clip matches the text, where 0 indicates no content match at all and 100 indicates perfect content match. We integrate the crowdsourced relevances into training and evaluating text-based audio retrieval systems, and evaluate the effect of using them together with binary relevances from audio captioning. Conventionally, these binary relevances are defined by captioning-based audio-caption pairs, where being positive indicates that the caption describes the paired audio, and being negative applies to all other pairs. Experimental results indicate that there is no clear benefit from incorporating crowdsourced relevances alongside binary relevances when the crowdsourced relevances are binarized for contrastive learning. Conversely, the results suggest that using only binary relevances defined by captioning-based audio-caption pairs is sufficient for contrastive learning.
【8】 FALL-E: A Foley Sound Synthesis Model and Strategies标题：Fall-E：一种Foley声音合成模型及策略链接：https://arxiv.org/abs/2306.09807
作者：Minsung Kang,Sangshin Oh,Hyeongi Moon,Kyungyun Lee,Ben Sangbae Chon备注：5 pages, 3 figures摘要：本文介绍了Foley综合系统FALL-E及其训练/推理策略。FALL-E模型采用包括低分辨率频谱图生成、频谱图超分辨率和声码器的级联方法。我们使用我们广泛的数据集从头开始训练每个与声音相关的模型，并使用预训练的语言模型。我们使用特定于数据集的文本来调节模型，使其能够根据文本输入学习音质和录制环境。此外，我们利用外部语言模型来改进数据集的文本描述，并对质量、一致性和多样性进行了快速工程。FALL-E通过客观测量以及DCASE 2023挑战任务7中的听力测试进行了评估。提交的作品平均获得了第二名，同时在多样性方面获得了最好的分数，音频质量获得了第二名，班级健身获得了第三名。摘要：This paper introduces FALL-E, a foley synthesis system and its training/inference strategies. The FALL-E model employs a cascaded approach comprising low-resolution spectrogram generation, spectrogram super-resolution, and a vocoder. We trained every sound-related model from scratch using our extensive datasets, and utilized a pre-trained language model. We conditioned the model with dataset-specific texts, enabling it to learn sound quality and recording environment based on text input. Moreover, we leveraged external language models to improve text descriptions of our datasets and performed prompt engineering for quality, coherence, and diversity. FALL-E was evaluated by an objective measure as well as listening tests in the DCASE 2023 challenge Task 7. The submission achieved the second place on average, while achieving the best score for diversity, second place for audio quality, and third place for class fitness.
【9】 MFAS: Emotion Recognition through Multiple Perspectives Fusion  Architecture Search Emulating Human Cognition标题：MFAs：模仿人类认知的多视角情感识别融合架构搜索链接：https://arxiv.org/abs/2306.09361
作者：Haiyang Sun,Fulin Zhang,Zheng Lian,Yingying Guo,Shilei Zhang机构：China Mobile Research Institute, Institute of Automation, Chinese Academy of Sciences摘要：语音情感识别旨在识别和分析目标语音中与人类相似的情感状态。完美的情感识别可以极大地有益于广泛的人机交互任务。受人类理解情感过程的启发，我们证明了与量化建模相比，从连续的角度理解语音内容，类似于人类的理解，使模型能够捕获更全面的情感信息。此外，考虑到人类调整其感知的情感词在文本语义的基础上，在语音中存在的某些线索，我们设计了一个新的搜索空间和搜索的最佳融合策略的两种类型的信息。实验结果进一步验证了这种感知调整的意义。基于这些观察，我们提出了一个新的框架，称为多视角融合架构搜索（MFAS）。具体来说，我们利用连续的知识来捕捉语音语义和量化的知识来学习文本语义。然后，我们寻找最佳的融合策略。实验结果表明，MFAS在全面捕捉语音情感信息方面优于现有模型，并能自动调整融合策略。摘要：Speech emotion recognition aims to identify and analyze emotional states in target speech similar to humans. Perfect emotion recognition can greatly benefit a wide range of human-machine interaction tasks. Inspired by the human process of understanding emotions, we demonstrate that compared to quantized modeling, understanding speech content from a continuous perspective, akin to human-like comprehension, enables the model to capture more comprehensive emotional information. Additionally, considering that humans adjust their perception of emotional words in textual semantic based on certain cues present in speech, we design a novel search space and search for the optimal fusion strategy for the two types of information. Experimental results further validate the significance of this perception adjustment. Building on these observations, we propose a novel framework called Multiple perspectives Fusion Architecture Search (MFAS). Specifically, we utilize continuous-based knowledge to capture speech semantic and quantization-based knowledge to learn textual semantic. Then, we search for the optimal fusion strategy for them. Experimental results demonstrate that MFAS surpasses existing models in comprehensively capturing speech emotion information and can automatically adjust fusion strategy.
eess.AS音频处理
【1】 Knowledge Distillation for Efficient Audio-Visual Video Captioning标题：面向高效视听字幕的知识提取链接：https://arxiv.org/abs/2306.09947
作者：Özkan Çaylı,Xubo Liu,Volkan Kılıç,Wenwu Wang机构：Electrical and Electronics Engineering, ˙Izmir Katip C¸ elebi University, T¨urkiye, Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey, UK备注：European Signal Processing Conference (EUSIPCO 2023)摘要：自动描述视听内容的文本，即视频字幕，已受到极大的关注，由于其潜在的应用在不同的领域。深度神经网络是主要的方法，提供最先进的性能。然而，由于模型参数的大尺寸，这些方法通常在低功率设备（如智能手机）中不可部署。在本文中，我们建议利用简单的池化前端和下采样算法与知识蒸馏的音频和视频属性，使用数量减少的视听帧。在教师模型知识提取的帮助下，我们提出的方法大大减少了音频视频流中的冗余信息，而不会丢失字幕生成的关键上下文。MSR-VTT数据集上的广泛实验评估表明，我们提出的方法显着减少了约80%的推理时间，在字幕准确性的小牺牲（小于0.02%）。摘要：Automatically describing audio-visual content with texts, namely video captioning, has received significant attention due to its potential applications across diverse fields. Deep neural networks are the dominant methods, offering state-of-the-art performance. However, these methods are often undeployable in low-power devices like smartphones due to the large size of the model parameters. In this paper, we propose to exploit simple pooling front-end and down-sampling algorithms with knowledge distillation for audio and visual attributes using a reduced number of audio-visual frames. With the help of knowledge distillation from the teacher model, our proposed method greatly reduces the redundant information in audio-visual streams without losing critical contexts for caption generation. Extensive experimental evaluations on the MSR-VTT dataset demonstrate that our proposed approach significantly reduces the inference time by about 80% with a small sacrifice (less than 0.02%) in captioning accuracy.
【2】 Crowdsourcing and Evaluating Text-Based Audio Retrieval Relevances标题：基于文本的音频检索相关度的众包和评估链接：https://arxiv.org/abs/2306.09820
作者：Huang Xie,Khazar Khorrami,Okko Räsänen,Tuomas Virtanen机构：Unit of Computing Sciences, Tampere University, Finland摘要：本文探讨了基于文本的音频检索相关性与众包评估的分级。给定自由形式的文本（例如，字幕）作为查询，要求众包工作者使用数字分数（在0和100之间）来对音频剪辑进行分级，以指示他们对音频剪辑的声音内容与文本匹配的程度的判断，其中0指示完全没有内容匹配，并且100指示完美的内容匹配。我们将众包的相关性训练和评估基于文本的音频检索系统，并评估使用它们与二进制相关性从音频字幕的效果。常规地，这些二进制相关性由基于字幕的音频-字幕对定义，其中正指示字幕描述配对的音频，并且负适用于所有其他对。实验结果表明，当众包相关性被二值化用于对比学习时，将众包相关性与二值相关性合并并没有明显的益处。相反，结果表明，仅使用基于字幕的音频字幕对定义的二进制相关性是足够的对比学习。摘要：This paper explores grading text-based audio retrieval relevances with crowdsourcing assessments. Given a free-form text (e.g., a caption) as a query, crowdworkers are asked to grade audio clips using numeric scores (between 0 and 100) to indicate their judgements of how much the sound content of an audio clip matches the text, where 0 indicates no content match at all and 100 indicates perfect content match. We integrate the crowdsourced relevances into training and evaluating text-based audio retrieval systems, and evaluate the effect of using them together with binary relevances from audio captioning. Conventionally, these binary relevances are defined by captioning-based audio-caption pairs, where being positive indicates that the caption describes the paired audio, and being negative applies to all other pairs. Experimental results indicate that there is no clear benefit from incorporating crowdsourced relevances alongside binary relevances when the crowdsourced relevances are binarized for contrastive learning. Conversely, the results suggest that using only binary relevances defined by captioning-based audio-caption pairs is sufficient for contrastive learning.
【3】 Investigating the Utility of Surprisal from Large Language Models for  Speech Synthesis Prosody标题：从大型语言模型中考察Surpral在语音合成韵律中的应用链接：https://arxiv.org/abs/2306.09814
作者：Sofoklis Kakouros,Juraj Šimko,Martti Vainio,Antti Suni机构：University of Helsinki, Finland备注：Accepted at SSW 2023摘要：本文探讨了使用单词surprisal，一个词在给定的上下文中的可预测性的措施，作为一个功能，以帮助语音合成韵律。我们探讨如何从大型语言模型（LLM）提取单词surprisal与单词突出，一个基于信号的措施，在一个给定的话语中的一个词的显着性。我们还研究了上下文长度和LLM大小如何影响结果，以及如何与基线系统相比，语音合成器的条件与surprisal值。为了评估这些因素，我们进行了实验，使用一个大型语料库的英语文本和LLM的不同大小。我们的研究结果表明，单词surprisal和单词的突出度是适度相关的，这表明他们捕捉相关的，但不同的语言使用方面。我们发现，长度的上下文和大小的LLM的影响的相关性，但不是在预期的方向，较长的上下文和较大的LLM一般underpredicting突出的话在一个接近线性的方式。我们证明，在这些研究结果，语音合成器条件与surprisal值提供了一个最小的改善，超过基线的结果表明，使用surprisal值引起适当的突出模式的影响有限。摘要：This paper investigates the use of word surprisal, a measure of the predictability of a word in a given context, as a feature to aid speech synthesis prosody. We explore how word surprisal extracted from large language models (LLMs) correlates with word prominence, a signal-based measure of the salience of a word in a given discourse. We also examine how context length and LLM size affect the results, and how a speech synthesizer conditioned with surprisal values compares with a baseline system. To evaluate these factors, we conducted experiments using a large corpus of English text and LLMs of varying sizes. Our results show that word surprisal and word prominence are moderately correlated, suggesting that they capture related but distinct aspects of language use. We find that length of context and size of the LLM impact the correlations, but not in the direction anticipated, with longer contexts and larger LLMs generally underpredicting prominent words in a nearly linear manner. We demonstrate that, in line with these findings, a speech synthesizer conditioned with surprisal values provides a minimal improvement over the baseline with the results suggesting a limited effect of using surprisal values for eliciting appropriate prominence patterns.
【4】 FALL-E: A Foley Sound Synthesis Model and Strategies标题：Fall-E：一种Foley声音合成模型及策略链接：https://arxiv.org/abs/2306.09807
作者：Minsung Kang,Sangshin Oh,Hyeongi Moon,Kyungyun Lee,Ben Sangbae Chon备注：5 pages, 3 figures摘要：本文介绍了Foley综合系统FALL-E及其训练/推理策略。FALL-E模型采用包括低分辨率频谱图生成、频谱图超分辨率和声码器的级联方法。我们使用我们广泛的数据集从头开始训练每个与声音相关的模型，并使用预训练的语言模型。我们使用特定于数据集的文本来调节模型，使其能够根据文本输入学习音质和录制环境。此外，我们利用外部语言模型来改进数据集的文本描述，并对质量、一致性和多样性进行了快速工程。FALL-E通过客观测量以及DCASE 2023挑战任务7中的听力测试进行了评估。提交的作品平均获得了第二名，同时在多样性方面获得了最好的分数，音频质量获得了第二名，班级健身获得了第三名。摘要：This paper introduces FALL-E, a foley synthesis system and its training/inference strategies. The FALL-E model employs a cascaded approach comprising low-resolution spectrogram generation, spectrogram super-resolution, and a vocoder. We trained every sound-related model from scratch using our extensive datasets, and utilized a pre-trained language model. We conditioned the model with dataset-specific texts, enabling it to learn sound quality and recording environment based on text input. Moreover, we leveraged external language models to improve text descriptions of our datasets and performed prompt engineering for quality, coherence, and diversity. FALL-E was evaluated by an objective measure as well as listening tests in the DCASE 2023 challenge Task 7. The submission achieved the second place on average, while achieving the best score for diversity, second place for audio quality, and third place for class fitness.
【5】 Use of a humanoid robot for auditory psychophysical testing标题：利用人形机器人进行听觉心理物理测试链接：https://arxiv.org/abs/2306.09714
作者：Luke Meyer,Laura Rachman,Gloria Araiza-Illan,Etienne Gaudrain,Deniz Başkent机构：Department of Otorhinolaryngology  Head and Neck Surgery, University Medical Center,  W.J. Kolff Institute for Biomedical Engineering and Materials Science, University Medical, Center Groningen, University of Groningen, the Netherlands备注：46 pages (double spaced), 7 figures, 55 references, submitted to PLOS ONE摘要：心理物理测试中的任务有时可能是重复的，并导致个体在测试期间失去参与。为了便于参与，我们建议使用一个人形NAO机器人作为一个替代接口进行心理物理测试。具体而言，我们的目标是评估的NAO作为听觉测试接口的性能，鉴于其潜在的限制和技术差异，在目前的笔记本电脑接口相比。我们研究的结果和持续时间的两个语音感知测试，语音提示敏感性和语音性别分类，从传统上使用的笔记本电脑接口和新的NAO接口。这两个测试调查的感知和使用两个说话者特定的语音线索，基频（F0）和声道长度（VTL），重要的特征语音性别。使用连接的鼠标在笔记本电脑上记录响应，并使用触觉传感器在NAO上记录响应。从两个接口的测试结果的比较显示接口之间的功能相似性，并复制了以前的研究结果与类似的测试。测试持续时间的比较显示NAO的测试时间更长，主要是由于与笔记本电脑相比处理时间更长，以及由于在机器人上实施测试而导致的其他设计限制。尽管NAO机器人的固有限制，如在音质，相对较长的处理和测试时间，以及不同的方法的响应记录，NAO接口似乎有利于收集类似的数据，目前的笔记本电脑接口，确认其潜在的替代心理物理测试接口的听觉感知测试。摘要：Tasks in psychophysical tests can at times be repetitive and cause individuals to lose engagement during the test. To facilitate engagement, we propose the use of a humanoid NAO robot as an alternative interface for conducting psychophysical tests. Specifically, we aim to evaluate the performance of the NAO as an auditory testing interface, given its potential limitations and technical differences, in comparison to the current laptop interface. We examine the results and durations of two voice perception tests, voice cue sensitivity and voice gender categorisation, obtained from both the conventionally used laptop interface and the new NAO interface. Both tests investigate the perception and use of two speaker-specific voice cues, fundamental frequency (F0) and vocal tract length (VTL), important for characterising voice gender. Responses are logged on the laptop using a connected mouse, and on the NAO using the tactile sensors. Comparison of test results from both interfaces shows functional similarity between the interfaces and replicates findings from previous studies with similar tests. Comparison of test durations shows longer testing times with NAO, primarily due to longer processing times in comparison to the laptop, as well as other design limitations due to the implementation of the test on the robot. Despite the inherent constraints of the NAO robot, such as in sound quality, relatively long processing and testing times, and different methods of response logging, the NAO interface appears to facilitate collecting similar data to the current laptop interface, confirming its potential as an alternative psychophysical test interface for auditory perception tests.
【6】 MF-PAM: Accurate Pitch Estimation through Periodicity Analysis and  Multi-level Feature Fusion标题：MF-PAM：周期分析和多层特征融合的精确基音估计链接：https://arxiv.org/abs/2306.09640
作者：Woo-Jin Chung,Doyeon Kim,Soo-Whan Chung,Hong-Goo Kang机构：Dept. of Electrical & Electronic Engineering, Yonsei University, South Korea, NAVER Cloud, South Korea备注：accepted at INTERSPEECH 2023摘要：我们引入了基于多级特征融合的周期性分析模型（MF-PAM），这是一种新型的基于深度学习的基音估计模型，可以在嘈杂和混响的声学环境中准确估计基音轨迹。我们的模型利用了音频信号的周期特性，并涉及两个关键步骤：使用周期性非周期性卷积（PNP-Conv）块来提取音调周期性，并且通过使用修改的双向特征金字塔网络（BiFPN）来聚合多级特征来估计音调。我们在语音和音乐数据集上评估了我们的模型，并在使用更少的模型参数的同时，实现了与最先进的基线相比更出色的音高估计性能。我们的模型在干净的音乐数据集上实现了99.20%的音高估计准确率。总体而言，我们提出的模型提供了一个有前途的解决方案，准确的音高估计在具有挑战性的声学环境中，并在音频信号处理中具有潜在的应用。摘要：We introduce Multi-level feature Fusion-based Periodicity Analysis Model (MF-PAM), a novel deep learning-based pitch estimation model that accurately estimates pitch trajectory in noisy and reverberant acoustic environments. Our model leverages the periodic characteristics of audio signals and involves two key steps: extracting pitch periodicity using periodic non-periodic convolution (PNP-Conv) blocks and estimating pitch by aggregating multi-level features using a modified bi-directional feature pyramid network (BiFPN). We evaluate our model on speech and music datasets and achieve superior pitch estimation performance compared to state-of-the-art baselines while using fewer model parameters. Our model achieves 99.20 % accuracy in pitch estimation on a clean musical dataset. Overall, our proposed model provides a promising solution for accurate pitch estimation in challenging acoustic environments and has potential applications in audio signal processing.
【7】 Distillation Strategies for Discriminative Speech Recognition Rescoring标题：用于区别性语音识别研究的提炼策略链接：https://arxiv.org/abs/2306.09452
作者：Prashanth Gurunath Shivakumar,Jari Kolehmainen,Yile Gu,Ankur Gandhe,Ariya Rastrow,Ivan Bulyko机构：Amazon Alexa AI, USA备注：Accepted at INTERSPEECH 2023摘要：第二遍重新评分在大多数最先进的语音识别系统中被采用。最近，基于BERT的模型已经获得了流行的重新排序n最佳假设，利用知识从掩蔽语言模型预训练。此外，利用诸如最小字错误率（MWER）的鉴别损失的微调已经示出比基于可能性的损失执行得更好。具有低延迟要求的流媒体应用程序对模型的大小施加了显着的约束，从而限制了字错误率（WER）的性能增益。在本文中，我们提出了有效的策略，从大型模型的区别性训练MWER的目标提取。我们实验Librispeech和生产规模内部数据集的语音助理。我们的研究结果表明，相对提高高达7%的WER训练学生模型与MWER。我们还表明，建议蒸馏可以减少学生和教师之间的WER差距62%到100%。摘要：Second-pass rescoring is employed in most state-of-the-art speech recognition systems. Recently, BERT based models have gained popularity for re-ranking the n-best hypothesis by exploiting the knowledge from masked language model pre-training. Further, fine-tuning with discriminative loss such as minimum word error rate (MWER) has shown to perform better than likelihood-based loss. Streaming applications with low latency requirements impose significant constraints on the size of the models, thereby limiting the word error rate (WER) performance gains. In this paper, we propose effective strategies for distilling from large models discriminatively trained with the MWER objective. We experiment on Librispeech and production scale internal dataset for voice-assistant. Our results demonstrate relative improvements of upto 7% WER over student models trained with MWER. We also show that the proposed distillation can reduce the WER gap between the student and the teacher by 62% upto 100%.
【8】 Diff-TTSG: Denoising probabilistic integrated speech and gesture  synthesis标题：DIFF-TTSG：去噪概率综合语音和手势合成链接：https://arxiv.org/abs/2306.09417
作者：Shivam Mehta,Siyang Wang,Simon Alexanderson,Jonas Beskow,Éva Székely,Gustav Eje Henter机构：Division of Speech, Music and Hearing, KTH Royal Institute of Technology, Sweden备注：7 pages, 2 figures, Accepted at Interspeech Speech Synthesis Workshop (SSW) 2023摘要：随着朗读语音合成达到高自然度分数，有越来越多的研究兴趣在合成自发语音。然而，人类自发的面对面对话具有口语和非语言方面（这里，共同语音手势）。直到最近，研究才开始探索在一个系统中联合合成这两种模式的好处。现有技术的先前状态使用非概率方法，其不能捕获人类语音和运动的可变性，并且有产生过度平滑伪像和次优合成质量的风险。我们提出了第一个基于扩散的概率模型，称为Diff-TTSG，共同学习合成语音和手势在一起。我们的方法可以在小数据集上从头开始训练。此外，我们描述了一组仔细的单模态和多模态的主观测试，用于评估集成的语音和手势合成系统，并使用它们来验证我们提出的方法。有关合成示例，请参见https://shivammehta25.github.io/Diff-TTSG摘要：With read-aloud speech synthesis achieving high naturalness scores, there is a growing research interest in synthesising spontaneous speech. However, human spontaneous face-to-face conversation has both spoken and non-verbal aspects (here, co-speech gestures). Only recently has research begun to explore the benefits of jointly synthesising these two modalities in a single system. The previous state of the art used non-probabilistic methods, which fail to capture the variability of human speech and motion, and risk producing oversmoothing artefacts and sub-optimal synthesis quality. We present the first diffusion-based probabilistic model, called Diff-TTSG, that jointly learns to synthesise speech and gestures together. Our method can be trained on small datasets from scratch. Furthermore, we describe a set of careful uni- and multi-modal subjective tests for evaluating integrated speech and gesture synthesis systems, and use them to validate our proposed approach. For synthesised examples please see https://shivammehta25.github.io/Diff-TTSG
【9】 MobileASR: A resource-aware on-device personalisation framework for  automatic speech recognition in mobile phones标题：MobileASR：一种用于手机自动语音识别的资源感知设备上个性化框架链接：https://arxiv.org/abs/2306.09384
作者：Zitha Sasindran,Harsha Yelchuri,Pooja Rao,T. V. Prabhakar机构：Department of Electronic Systems Engineering, Indian Institute of Science, Bengaluru, India,.摘要：我们描述了一个全面的方法开发用户语音个性化ASR模型，有效地训练模型在手机上，允许用户数据和模型存储和使用本地。为了实现这一目标，我们提出了一个资源感知的子模型为基础的训练方法，考虑的RAM，和手机的电池容量。我们还调查了可用的资源和训练时间之间的关系，强调了在这种情况下使用子模型的有效性。通过考虑到评估指标和手机的电池限制，我们能够执行有效的培训并相应地停止该过程。为了模拟真实用户，我们使用具有各种口音的扬声器。然后，整个设备上的培训和评估框架在不同品牌的各种手机上进行了测试。我们表明，微调模型和选择正确的超参数值是最低可实现的性能指标，设备上的训练时间和内存消耗之间的权衡。总的来说，我们的方法提供了一个全面的解决方案，开发个性化的ASR模型，同时利用移动电话的功能，并平衡对准确性的需求与资源限制。摘要：We describe a comprehensive methodology for developing user-voice personalised ASR models by effectively training models on mobile phones, allowing user data and models to be stored and used locally. To achieve this, we propose a resource-aware sub-model based training approach that considers the RAM, and battery capabilities of mobile phones. We also investigate the relationship between available resources and training time, highlighting the effectiveness of using sub-models in such scenarios. By taking into account the evaluation metric and battery constraints of the mobile phones, we are able to perform efficient training and halt the process accordingly. To simulate real users, we use speakers with various accents. The entire on-device training and evaluation framework was then tested on various mobile phones across brands. We show that fine-tuning the models and selecting the right hyperparameter values is a trade-off between the lowest achievable performance metric, on-device training time, and memory consumption. Overall, our methodology offers a comprehensive solution for developing personalized ASR models while leveraging the capabilities of mobile phones, and balancing the need for accuracy with resource constraints.
【10】 MFAS: Emotion Recognition through Multiple Perspectives Fusion  Architecture Search Emulating Human Cognition标题：MFAs：模仿人类认知的多视角情感识别融合架构搜索链接：https://arxiv.org/abs/2306.09361
作者：Haiyang Sun,Fulin Zhang,Zheng Lian,Yingying Guo,Shilei Zhang机构：China Mobile Research Institute, Institute of Automation, Chinese Academy of Sciences摘要：语音情感识别旨在识别和分析目标语音中与人类相似的情感状态。完美的情感识别可以极大地有益于广泛的人机交互任务。受人类理解情感过程的启发，我们证明了与量化建模相比，从连续的角度理解语音内容，类似于人类的理解，使模型能够捕获更全面的情感信息。此外，考虑到人类调整其感知的情感词在文本语义的基础上，在语音中存在的某些线索，我们设计了一个新的搜索空间和搜索的最佳融合策略的两种类型的信息。实验结果进一步验证了这种感知调整的意义。基于这些观察，我们提出了一个新的框架，称为多视角融合架构搜索（MFAS）。具体来说，我们利用连续的知识来捕捉语音语义和量化的知识来学习文本语义。然后，我们寻找最佳的融合策略。实验结果表明，MFAS在全面捕捉语音情感信息方面优于现有模型，并能自动调整融合策略。摘要：Speech emotion recognition aims to identify and analyze emotional states in target speech similar to humans. Perfect emotion recognition can greatly benefit a wide range of human-machine interaction tasks. Inspired by the human process of understanding emotions, we demonstrate that compared to quantized modeling, understanding speech content from a continuous perspective, akin to human-like comprehension, enables the model to capture more comprehensive emotional information. Additionally, considering that humans adjust their perception of emotional words in textual semantic based on certain cues present in speech, we design a novel search space and search for the optimal fusion strategy for the two types of information. Experimental results further validate the significance of this perception adjustment. Building on these observations, we propose a novel framework called Multiple perspectives Fusion Architecture Search (MFAS). Specifically, we utilize continuous-based knowledge to capture speech semantic and quantization-based knowledge to learn textual semantic. Then, we search for the optimal fusion strategy for them. Experimental results demonstrate that MFAS surpasses existing models in comprehensively capturing speech emotion information and can automatically adjust fusion strategy.
【11】 Evaluation of Speech Representations for MOS prediction标题：用于MOS预测的语音表示的评估链接：https://arxiv.org/abs/2306.09979
作者：Frederico S. Oliveira,Edresson Casanova,Arnaldo Cândido Júnior,Lucas R. S. Gris,Anderson S. Soares,Arlindo R. Galvão Filho机构：UFG, Goi´as - GO - Brazil备注：12 pages, 4 figures, Accepted to the 26th International Conference of Text, Speech and Dialogue (TSD2023)摘要：在本文中，我们评估特征提取模型预测语音质量。我们还提出了一个模型架构来比较嵌入的监督学习和自我监督学习模型与嵌入的说话人验证模型，以预测度量MOS。我们的实验是在VCC 2018数据集和为这项工作创建的名为BRSpeechMOS的巴西-葡萄牙数据集上进行的。结果表明，Whisper模型适用于所有场景：使用VCC 2018和BRSpeech-MOS数据集。在使用BRSpeechMOS的监督和自监督学习模型中，Whisper-Small的线性相关性最好，为0.6980，说话人验证模型SpeakerNet的线性相关性为0.6963。使用VCC 2018，最佳监督和自我监督学习模型Whisper-Large实现了0.7274的线性相关性，最佳模型说话人验证TitaNet实现了0.6933的线性相关性。虽然说话人验证模型的结果略低，但SpeakerNet模型只有5 M个参数，适合实时应用，TitaNet模型产生的嵌入大小为192，是所有评估模型中最小的。实验结果是可重复的公开可用的源代码1。摘要：In this paper, we evaluate feature extraction models for predicting speech quality. We also propose a model architecture to compare embeddings of supervised learning and self-supervised learning models with embeddings of speaker verification models to predict the metric MOS. Our experiments were performed on the VCC2018 dataset and a Brazilian-Portuguese dataset called BRSpeechMOS, which was created for this work. The results show that the Whisper model is appropriate in all scenarios: with both the VCC2018 and BRSpeech- MOS datasets. Among the supervised and self-supervised learning models using BRSpeechMOS, Whisper-Small achieved the best linear correlation of 0.6980, and the speaker verification model, SpeakerNet, had linear correlation of 0.6963. Using VCC2018, the best supervised and self-supervised learning model, Whisper-Large, achieved linear correlation of 0.7274, and the best model speaker verification, TitaNet, achieved a linear correlation of 0.6933. Although the results of the speaker verification models are slightly lower, the SpeakerNet model has only 5M parameters, making it suitable for real-time applications, and the TitaNet model produces an embedding of size 192, the smallest among all the evaluated models. The experiment results are reproducible with publicly available source-code1 .
【12】 RealImpact: A Dataset of Impact Sound Fields for Real Objects标题：RealImpact：真实物体碰撞声场数据集链接：https://arxiv.org/abs/2306.09944
作者：Samuel Clarke,Ruohan Gao,Mason Wang,Mark Rau,Julia Xu,Jui-Hsien Wang,Doug L. James,Jiajun Wu机构：Stanford University, Adobe Research备注：CVPR 2023 (Highlight). Project page: this https URL摘要：物体在不同的扰动、环境条件和相对于听者的姿势下发出独特的声音。虽然先前的工作已经在模拟中对冲击声音和声音传播进行了建模，但我们缺乏真实对象的冲击声场的标准数据集，用于视听学习和模拟到真实差距的校准。我们提出了RealImpact，一个大规模的数据集，真实物体的冲击声音记录在受控条件下。RealImpact包含50个日常物体的150，000个撞击声录音，并附有详细的注释，包括其撞击位置，麦克风位置，接触力分布，材料标签和RGBD图像。我们初步尝试使用我们的数据集作为参考，目前的模拟方法估计对象的影响声音，符合现实世界。此外，我们证明了我们的数据集作为声学和视听学习的测试平台的有用性，通过两个基准任务的评估，包括听者位置分类和视觉声学匹配。摘要：Objects make unique sounds under different perturbations, environment conditions, and poses relative to the listener. While prior works have modeled impact sounds and sound propagation in simulation, we lack a standard dataset of impact sound fields of real objects for audio-visual learning and calibration of the sim-to-real gap. We present RealImpact, a large-scale dataset of real object impact sounds recorded under controlled conditions. RealImpact contains 150,000 recordings of impact sounds of 50 everyday objects with detailed annotations, including their impact locations, microphone locations, contact force profiles, material labels, and RGBD images. We make preliminary attempts to use our dataset as a reference to current simulation methods for estimating object impact sounds that match the real world. Moreover, we demonstrate the usefulness of our dataset as a testbed for acoustic and audio-visual learning via the evaluation of two benchmark tasks, including listener location classification and visual acoustic matching.
【13】 Correlation Clustering of Bird Sounds标题：鸟鸣的相关聚类链接：https://arxiv.org/abs/2306.09906
作者：David Stein,Bjoern Andres机构：TU Dresden备注：13 pages摘要：鸟类声音分类是将任何录音与录音中可以听到的鸟类种类相关联的任务。在这里，我们研究鸟类的声音聚类，决定任何一对声音记录是否同一种鸟类可以听到两者的任务。我们解决这个问题，首先学习，从一个训练集，以这种方式相关的记录对的概率，然后推断一个最大可能的分区的测试集的相关聚类。我们解决以下问题：与测试集的分类相比，这种聚类的准确性有多高？这样推断出的聚类如何与通过分类获得的聚类相关联？当应用于训练期间没有听到的鸟类的录音时，这种聚类的准确性如何？这种聚类在将训练期间未听到的环境噪声与鸟鸣分离方面的有效性如何？摘要：Bird sound classification is the task of relating any sound recording to those species of bird that can be heard in the recording. Here, we study bird sound clustering, the task of deciding for any pair of sound recordings whether the same species of bird can be heard in both. We address this problem by first learning, from a training set, probabilities of pairs of recordings being related in this way, and then inferring a maximally probable partition of a test set by correlation clustering. We address the following questions: How accurate is this clustering, compared to a classification of the test set? How do the clusters thus inferred relate to the clusters obtained by classification? How accurate is this clustering when applied to recordings of bird species not heard during training? How effective is this clustering in separating, from bird sounds, environmental noise not heard during training?
【14】 CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained  Language-Vision Models标题：CLIPSonic：无标签视频和预先训练的语言-视觉模型的文本到音频合成链接：https://arxiv.org/abs/2306.09635
作者：Hao-Wen Dong,Xiaoyu Liu,Jordi Pons,Gautam Bhattacharya,Santiago Pascual,Joan Serrà,Taylor Berg-Kirkpatrick,Julian McAuley机构：Joan Serra,  Dolby Laboratories,  University of California San Diego摘要：最近的工作已经研究了文本到音频合成使用大量的成对的文本音频数据。然而，具有高质量文本注释的音频记录可能难以获取。在这项工作中，我们使用未标记的视频和预训练的语言视觉模型来实现文本到音频的合成。我们建议学习所需的文本音频对应，利用视觉模态作为桥梁。我们训练一个条件扩散模型来生成视频的音轨，给定一个由预训练的对比语言图像预训练（CLIP）模型编码的视频帧。在测试时，我们首先探索执行zero-shot模态转移和条件的扩散模型与CLIP编码的文本查询。然而，我们观察到一个显着的性能下降，相对于图像查询。为了缩小这一差距，我们进一步采用预训练的扩散先验模型来生成CLIP图像嵌入，给定CLIP文本嵌入。我们的结果表明，所提出的方法的有效性，预训练的扩散先验可以减少模态转移间隙。虽然我们专注于文本到音频的合成，所提出的模型也可以从图像查询生成音频，它显示出竞争力的表现对国家的最先进的图像到音频的合成模型在主观听力测试。这项研究为文本到音频的合成提供了一个新的方向，它利用了视频中自然发生的视听对应关系和预训练语言视觉模型的功能。摘要：Recent work has studied text-to-audio synthesis using large amounts of paired text-audio data. However, audio recordings with high-quality text annotations can be difficult to acquire. In this work, we approach text-to-audio synthesis using unlabeled videos and pretrained language-vision models. We propose to learn the desired text-audio correspondence by leveraging the visual modality as a bridge. We train a conditional diffusion model to generate the audio track of a video, given a video frame encoded by a pretrained contrastive language-image pretraining (CLIP) model. At test time, we first explore performing a zero-shot modality transfer and condition the diffusion model with a CLIP-encoded text query. However, we observe a noticeable performance drop with respect to image queries. To close this gap, we further adopt a pretrained diffusion prior model to generate a CLIP image embedding given a CLIP text embedding. Our results show the effectiveness of the proposed method, and that the pretrained diffusion prior can reduce the modality transfer gap. While we focus on text-to-audio synthesis, the proposed model can also generate audio from image queries, and it shows competitive performance against a state-of-the-art image-to-audio synthesis model in a subjective listening test. This study offers a new direction of approaching text-to-audio synthesis that leverages the naturally-occurring audio-visual correspondence in videos and the power of pretrained language-vision models.
【15】 Competitive and Resource Efficient Factored Hybrid HMM Systems are  Simpler Than You Think标题：竞争性和资源高效的混合HMM系统比你想象的要简单链接：https://arxiv.org/abs/2306.09517
作者：Tina Raissi,Christoph Lüscher,Moritz Gunz,Ralf Schlüter,Hermann Ney机构：Machine Learning and Human Language Technology, Computer Science Department, RWTH Aachen University, Aachen, Germany, AppTek GmbH, Aachen, Germany备注：Accepted for presentation at InterSpeech 2023摘要：为自动语音识别（ASR）构建竞争性混合隐马尔可夫模型（HMM）系统需要一个复杂的多级流水线，该流水线由多个训练准则组成。最近的序列到序列模型提供了具有可以从头开始的更简单的管道的优点。我们提出了一个纯粹的基于神经的单级从头管道上下文相关的混合HMM，提供了类似的简单性。我们使用的对齐从一个完整的总和训练零阶后验HMM与BLSTM编码器。我们表明，与此对齐，我们可以建立一个构象因子的混合，表现甚至优于一个国家的最先进的经典的混合和一个因子的混合训练与更复杂的基于高斯混合系统的对齐。我们的发现证实了Switchboard 300h和LibriSpeech 960h任务与文献中的其他方法具有可比性的结果，并通过额外依赖于可用计算资源的负责任的选择。摘要：Building competitive hybrid hidden Markov model~(HMM) systems for automatic speech recognition~(ASR) requires a complex multi-stage pipeline consisting of several training criteria. The recent sequence-to-sequence models offer the advantage of having simpler pipelines that can start from-scratch. We propose a purely neural based single-stage from-scratch pipeline for a context-dependent hybrid HMM that offers similar simplicity. We use an alignment from a full-sum trained zero-order posterior HMM with a BLSTM encoder. We show that with this alignment we can build a Conformer factored hybrid that performs even better than both a state-of-the-art classic hybrid and a factored hybrid trained with alignments taken from more complex Gaussian mixture based systems. Our finding is confirmed on Switchboard 300h and LibriSpeech 960h tasks with comparable results to other approaches in the literature, and by additionally relying on a responsible choice of available computational resources.
【16】 Sound Demixing Challenge 2023 -- Music Demixing Track Technical Report标题：声音分离挑战赛2023--音乐分离轨道技术报告链接：https://arxiv.org/abs/2306.09382
作者：Minseok Kim,Jun Hyung Lee机构：Department of Computer Science, Korea University备注：4 pages, 3 tables摘要：在这份报告中，我们将为2023年的音乐解混合挑战赛（Music Demixing Track of Sound Demixing Challenge 2023）展示获奖解决方案。我们专注于为这一挑战而设计的两种方法：一个时间效率高的源分离网络，实现了MUSDB基准测试的最先进的结果和噪声鲁棒源分离的损失掩蔽方法。用于再现模型训练和最终提交的代码可在www.example.com上获得github.com/kuielab/sdx23。摘要：In this report, we present our award-winning solutions for the Music Demixing Track of Sound Demixing Challenge 2023. We focus on two methods designed for this challenge: a time-efficient source separation network that achieves state-of-the-art results on the MUSDB benchmark and a loss masking method for noise-robust source separation. Code for reproducing model training and final submissions is available at github.com/kuielab/sdx23.
机器翻译，仅供参考