今日论文合集：cs.SD语音14篇，eess.AS音频处理12篇。
【1】 MooseNet: A trainable metric for synthesized speech with plda backend
标题：MooseNet：一种基于PLDA后端的可训练合成语音度量
链接：https://arxiv.org/abs/2301.07087
作者：Ondřej Plátek,Ondřej Dušek机构：A TRAINABLE METRIC FOR SYNTHESIZED SPEECH WITH PLDABACKENDOndˇrej PlátekandOndˇrej DušekCharles University, Faculty of Mathematics and PhysicsInstitute of Formal and Applied LinguisticsPrague摘要：我们提出了MooseNet，一个可训练的语音度量，预测听众的平均意见得分（MOS）。我们报告了使用易于使用的建模技术对挑战基线的改进，这些技术还可以扩展到更大的自监督学习（SSL）模型。我们提出了两种模型。第一个模型是神经网络（NN）。作为第二种模型，我们在第一种神经网络模型的上层提出了PLDA生成式模型，改进了纯神经网络模型。我们的两个型号的合奏在所有系统和发声级别指标上都达到了VoiceMOS排行榜的前3或前4名。摘要：We present MooseNet, a trainable speech metric that predicts listeners' Mean Opinion Score (MOS). We report improvements to the challenge baselines using easy-to-use modeling techniques, which also scales for larger self-supervised learning (SSL) model. We present two models. The first model is a Neural Network (NN). As a second model, we propose a PLDA generative model on the top layers of the first NN model, which improves the pure NN model. Ensembles from our two models achieve the top 3 or 4 VoiceMOS leaderboard places on all system and utterance level metrics.
【2】 Automated speech- and text-based classification of neuropsychiatric conditions in a multidiagnostic setting
标题：多诊断环境中神经精神疾病的基于语音和文本的自动分类
链接：https://arxiv.org/abs/2301.06916
作者：Lasse Hansen,Roberta Rocca,Arndis Simonsen,Alberto Parola,Vibeke Bliksted,Nicolai Ladegaard,Dan Bang,Kristian Tylén,Ethan Weed,Søren Dinesen Østergaard,Riccardo Fusaroli机构：Department of Affective Disorders, Aarhus University Hospital - Psychiatry, Aarhus, Denmark, Department of Clinical Medicine, Aarhus University, Aarhus, Denmark, Center for Humanities Computing, Aarhus University, Aarhus, Denmark摘要：言语模式已被确定为神经精神疾病的潜在诊断标志物。然而，大多数研究仅将单个临床组与健康对照组进行比较，而临床实践通常需要区分多个潜在诊断（多分类设置）。为了解决这个问题，我们收集了420名参与者（67名重度抑郁症患者、106名精神分裂症患者和46名自闭症患者，以及匹配的对照组）的重复录音数据集，并基于语音和文本特征测试了一系列传统机器学习模型和高级Transformer模型在二元和多元分类方面的性能。 二元模型的预测结果与以往的研究结果相当（孤独症谱系障碍F1评分在0.54-0.75之间;0.67-0.92用于重度抑郁症，MDD;精神分裂症为0.71-0.83）;当在多个诊断组之间进行区分时，表现显著下降（F1评分在0.35-0.44之间的ASD、0.57-0.75之间的MDD、0.15-0.66之间的精神分裂症和0.38-0.52宏F1）。结合语音和基于文本的模型产生了更高的性能，表明它们捕获了互补的诊断信息。 我们的结果表明，在二元分类上训练的模型可能学会依赖于临床和非临床人群之间的一般差异标志物，或跨疾病重叠的临床特征标志物，而不是识别个体疾病特异性标志物。我们对该领域未来的研究提出了建议，建议更多地关注开发更大的跨诊断数据集，包括更细粒度的临床特征，并可以支持更好地捕捉神经精神疾病和自然诊断评估的复杂性的模型开发。摘要：Speech patterns have been identified as potential diagnostic markers for neuropsychiatric conditions. However, most studies only compare a single clinical group to healthy controls, whereas clinical practice often requires differentiating between multiple potential diagnoses (multiclass settings). To address this, we assembled a dataset of repeated recordings from 420 participants (67 with major depressive disorder, 106 with schizophrenia and 46 with autism, as well as matched controls), and tested the performance of a range of conventional machine learning models and advanced Transformer models on both binary and multiclass classification, based on voice and text features. While binary models performed comparably to previous research (F1 scores between 0.54-0.75 for autism spectrum disorder, ASD; 0.67-0.92 for major depressive disorder, MDD; and 0.71-0.83 for schizophrenia); when differentiating between multiple diagnostic groups performance decreased markedly (F1 scores between 0.35-0.44 for ASD, 0.57-0.75 for MDD, 0.15-0.66 for schizophrenia, and 0.38-0.52 macro F1). Combining voice and text-based models yielded increased performance, suggesting that they capture complementary diagnostic information. Our results indicate that models trained on binary classification may learn to rely on markers of generic differences between clinical and non-clinical populations, or markers of clinical features that overlap across conditions, rather than identifying markers specific to individual conditions. We provide recommendations for future research in the field, suggesting increased focus on developing larger transdiagnostic datasets that include more fine-grained clinical features, and that can support the development of models that better capture the complexity of neuropsychiatric conditions and naturalistic diagnostic assessment.
【3】 Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducer
标题：两级上下文词过滤在统一流传输和非流传输传感器中的上下文偏向
链接：https://arxiv.org/abs/2301.06735
作者：Zhanheng Yang,Sining Sun,Xiong Wang,Yike Zhang,Long Ma,Lei Xie机构：Audio, Speech and Language Processing Group, School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Tencent Technology Co., Ltd, Beijing, China摘要：端到端（E2 E）ASR系统难以识别诸如在训练数据中不频繁出现的命名实体之类的词。缓解这一问题的广泛使用的方法是将上下文信息馈送到声学模型中。上下文单词列表是必要的，它列出了所有可能的上下文单词候选项。以前的工作已经证明，列表的大小和质量至关重要。一个紧凑而准确的列表可以显著提高性能。本文提出了一种高效的方法来获取高质量的上下文词表，用于统一的基于流和非流的Conformer-Transducer（C-T）模型。具体来说，我们利用音素级流输出来首先过滤预定义的上下文单词列表。在后续的非流式推理过程中，将过滤后的词汇列表作为上下文信息融合到非因果编码器和解码器中，生成最终的识别结果。该方法可以充分利用流识别假设，提高上下文ASR系统的准确率，加快推理过程。在两个数据集上的实验表明，与基线系统相比，该系统的相对字符错误率（CERR）降低了20%以上。同时，当上下文词表的大小超过6000时，系统的RTF可以稳定在0.15以内。摘要：It is difficult for an end-to-end (E2E) ASR system to recognize words such as named entities appearing infrequently in the training data. A widely used method to mitigate this issue is feeding contextual information into the acoustic model. A contextual word list is necessary, which lists all possible contextual word candidates. Previous works have proven that the size and quality of the list are crucial. A compact and accurate list can boost the performance significantly. In this paper, we propose an efficient approach to obtain a high quality contextual word list for a unified streaming and non-streaming based Conformer-Transducer (C-T) model. Specifically, we make use of the phone-level streaming output to first filter the predefined contextual word list. During the subsequent non-streaming inference, the words in the filtered list are regarded as contextual information fused into non-casual encoder and decoder to generate the final recognition results. Our approach can take advantage of streaming recognition hypothesis, improve the accuracy of the contextual ASR system and speed up the inference process as well. Experiments on two datasets demonstrates over 20% relative character error rate reduction (CERR) comparing to the baseline system. Meanwile, the RTF of our system can be stabilized within 0.15 when the size of the contextual word list grows over 6,000.
【4】 Using Kaldi for Automatic Speech Recognition of Conversational Austrian German
标题：使用Kaldi实现会话奥地利德语的自动语音识别
链接：https://arxiv.org/abs/2301.06475
作者：Julian Linke,Saskia Wepner,Gernot Kubin,Barbara Schuppler机构：Signal Processing and Speech Communication Laboratory, Graz University of Technology, Inffeldgasse ,c, Graz, Austria备注：10 pages, 2 figures, 4 tables摘要：随着对话系统越来越具有交互性和社会性，对会话语音进行准确的自动语音识别也变得越来越重要。这就把焦点从简短、自发、任务导向的对话转移到了更为复杂的随意的面对面对话上。然而，对这些会话的收集和注释是一个耗时的过程，并且针对这种特定说话风格的数据很少。本文以阅读和会话奥地利德语为被试进行了ASR实验。为了解决德语会话资源有限的问题，同时考虑到说话者之间在发音特征方面的巨大差异，我们通过引入（大的）基于知识的发音词典来改进基于Kaldi的ASR系统，同时探索不同的基于数据的方法来限制每个词条的发音变体的数量。我们在奥地利德语阅读演讲中达到了0.4%的最佳WER，在会话演讲中达到了48.5%的最佳平均WER。我们发现，通过使用我们的最佳发音词典，与通过将用于语言模型的数据的大小增加大约10%相比，可以实现类似的高性能。360%到760%。我们的研究结果表明，对于低资源的情况--尽管语音技术的总体趋势是只使用基于数据的方法--基于知识的方法是一种成功的、有效的方法。摘要：As dialogue systems are becoming more and more interactional and social, also the accurate automatic speech recognition (ASR) of conversational speech is of increasing importance. This shifts the focus from short, spontaneous, task-oriented dialogues to the much higher complexity of casual face-to-face conversations. However, the collection and annotation of such conversations is a time-consuming process and data is sparse for this specific speaking style. This paper presents ASR experiments with read and conversational Austrian German as target. In order to deal with having only limited resources available for conversational German and, at the same time, with a large variation among speakers with respect to pronunciation characteristics, we improve a Kaldi-based ASR system by incorporating a (large) knowledge-based pronunciation lexicon, while exploring different data-based methods to restrict the number of pronunciation variants for each lexical entry. We achieve best WER of 0.4% on Austrian German read speech and best average WER of 48.5% on conversational speech. We find that by using our best pronunciation lexicon a similarly high performance can be achieved than by increasing the size of the data used for the language model by approx. 360% to 760%. Our findings indicate that for low-resource scenarios -- despite the general trend in speech technology towards using data-based methods only -- knowledge-based approaches are a successful, efficient method.
【5】 Msanii: High Fidelity Music Synthesis on a Shoestring Budget
标题：Msanii：以极低的预算合成高保真音乐
链接：https://arxiv.org/abs/2301.06468
机构：Independent Researcher备注：15 pages, 8 figures, for demo see this https URL and for code, see this https URL, this paper is a work in progress摘要：在这篇文章中，我们提出了一个新的基于扩散的模型来有效地合成长上下文、高保真度的音乐。我们的模型结合了梅尔谱图的表现力，扩散模型的生成能力，以及神经声码器的声码能力。我们在不使用拼接合成、级联架构或压缩技术的情况下，以高采样率（44.1 kHz）合成数十秒（190秒）的立体声音乐，从而证明Msanii的有效性。据我们所知，这是首次成功地采用基于扩散的模型来以高采样率合成如此长的音乐样本。我们的演示可以在www.example.com上找到https://kinyugo.github.io/msanii-demo，我们的代码可以在https://github.com/Kinyugo/msanii。摘要：In this paper, we present Msanii, a novel diffusion-based model for synthesizing long-context, high-fidelity music efficiently. Our model combines the expressiveness of mel spectrograms, the generative capabilities of diffusion models, and the vocoding capabilities of neural vocoders. We demonstrate the effectiveness of Msanii by synthesizing tens of seconds (190 seconds) of stereo music at high sample rates (44.1 kHz) without the use of concatenative synthesis, cascading architectures, or compression techniques. To the best of our knowledge, this is the first work to successfully employ a diffusion-based model for synthesizing such long music samples at high sample rates. Our demo can be found https://kinyugo.github.io/msanii-demo and our code https://github.com/Kinyugo/msanii .
【6】 OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset
标题：OLKAVS：一个开放的大规模韩语视听语音数据集
链接：https://arxiv.org/abs/2301.06375
作者：Jeongkyun Park,Jung-Wook Hwang,Kwanghee Choi,Seung-Hyun Lee,Jun Hwan Ahn,Rae-Hong Park,Hyung-Min Park摘要：受人类以多模态方式理解语音的启发，已经构建了各种视听数据集。然而，大多数现有的数据集集中于英语，在数据集准备期间诱导与各种预测模型的依赖性，并且只有少量的多视角视频。为了减轻这些限制，我们最近开发了开放大规模韩语视听语音（OLKAVS）数据集，这是公开可用的视听语音数据集中最大的。该数据集包含1，107名韩语使用者在录音室中录制的1，150小时音频，设置有9个不同的视点和各种噪音情况。我们也提供了两个任务的预训练基线模型，视听语音识别和唇读。我们基于模型进行了实验，以验证多模态和多视角训练相对于单模态和仅正视训练的有效性。我们期望OLKAVS数据集能促进多模态研究在更广泛的领域，如韩国语语音识别，说话人识别，发音水平分类和嘴部运动分析。摘要：Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed. However, most existing datasets focus on English, induce dependencies with various prediction models during dataset preparation, and have only a small number of multi-view videos. To mitigate the limitations, we recently developed the Open Large-scale Korean Audio-Visual Speech (OLKAVS) dataset, which is the largest among publicly available audio-visual speech datasets. The dataset contains 1,150 hours of transcribed audio from 1,107 Korean speakers in a studio setup with nine different viewpoints and various noise situations. We also provide the pre-trained baseline models for two tasks, audio-visual speech recognition and lip reading. We conducted experiments based on the models to verify the effectiveness of multi-modal and multi-view training over uni-modal and frontal-view-only training. We expect the OLKAVS dataset to facilitate multi-modal research in broader areas such as Korean speech recognition, speaker recognition, pronunciation level classification, and mouth motion analysis.
【7】 Improving Target Speaker Extraction with Sparse LDA-transformed Speaker Embeddings
标题：稀疏LDA变换说话人嵌入改进目标说话人提取
链接：https://arxiv.org/abs/2301.06277
作者：Kai Liu,Xucheng Wan,Ziqing Du,Huan Zhou机构：AI Application Research Center, Huawei Technologies Co., Ltd. Shenzhen, China备注：ACCEPTED by NCMMSC 2022摘要：目标说话人提取（TSE）是一种实用的语音分离方法，它利用从说话人中提取的附加说话人线索来提取出期望说话人的语音。其主要挑战在于如何正确提取和利用说话人线索以提高提取的语音质量。现有的大多数TSE研究中采用的线索提取方法是直接利用区分性说话人嵌入，从预先训练好的模型中提取线索进行说话人确认。虽然高说话人区分度是说话人确认任务最理想的属性，但我们认为它对于TSE来说可能过于复杂。在本研究中，我们提出一个简化的说话人线索与明确的类别可分性可能是首选的TSE。为了验证我们的建议，我们引入了几种说话人线索的形式，包括原始说话人嵌入（如，x向量和xi向量）和由稀疏LDA变换产生的新说话人嵌入。通过将这些说话人线索与SepFormer（一种SOTA语音分离模型）相结合来建立相应的TSE模型。在基准WSJ 0 - 2 mix数据集上检查这些TSE模型的性能。实验结果验证了该方法的有效性和通用性，SI-SDRi相对提高了9.9%。此外，SI-SDRi为19.4 dB，PESQ为3.78，我们的最佳TSE系统显著优于当前SOTA系统，并提供了迄今为止WSJ 0 - 2 mix报告的最佳TSE结果。摘要：As a practical alternative of speech separation, target speaker extraction (TSE) aims to extract the speech from the desired speaker using additional speaker cue extracted from the speaker. Its main challenge lies in how to properly extract and leverage the speaker cue to benefit the extracted speech quality. The cue extraction method adopted in majority existing TSE studies is to directly utilize discriminative speaker embedding, which is extracted from the pre-trained models for speaker verification. Although the high speaker discriminability is a most desirable property for speaker verification task, we argue that it may be too sophisticated for TSE. In this study, we propose that a simplified speaker cue with clear class separability might be preferred for TSE. To verify our proposal, we introduce several forms of speaker cues, including naive speaker embedding (such as, x-vector and xi-vector) and new speaker embeddings produced from sparse LDA-transform. Corresponding TSE models are built by integrating these speaker cues with SepFormer (one SOTA speech separation model). Performances of these TSE models are examined on the benchmark WSJ0-2mix dataset. Experimental results validate the effectiveness and generalizability of our proposal, showing up to 9.9% relative improvement in SI-SDRi. Moreover, with SI-SDRi of 19.4 dB and PESQ of 3.78, our best TSE system significantly outperforms the current SOTA systems and offers the top TSE results reported till date on the WSJ0-2mix.
【8】 Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models
标题：多通道帮助单通道：基于多通道模型的跨通道短程学习
链接：https://arxiv.org/abs/2301.06267
作者：Zhiqiu Lin,Samuel Yu,Zhiyi Kuang,Deepak Pathak,Deva Ramana机构：Carnegie Mellon University备注：Project website: this https URL摘要：用最少的指令快速学习一个新任务的能力--称为Few-Shot学习--是智能代理的一个核心方面。经典的Few-Shot基准使用来自单个模态的少镜头样本，但是这样的样本可能不足以表征整个概念类。相反，人类使用跨模态信息来有效地学习新概念。在这项工作中，我们证明了人们确实可以通过${\bf read}$记录狗的信息和${\bf listen}$记录狗的叫声来构建一个更好的${\bf visual}$狗分类器。为了做到这一点，我们利用了这样一个事实，即最近的多模态基础模型，如CLIP，本质上是跨模态的，将不同的模态映射到同一个表示空间。具体地说，我们提出了一种简单的跨模态自适应方法，该方法从跨越不同模态的Few-Shot示例中学习。通过将类名称重新用作额外的一次性训练样本，我们用一个令人尴尬的简单线性分类器实现了SOTA结果，用于视觉语言适应。此外，我们还证明了我们的方法可以有益于现有的方法，如前缀调整，适配器，和分类器集成。最后，为了探索视觉和语言之外的其他模态，我们构建了第一个（据我们所知）视听Few-Shot基准，并使用跨模态训练来提高图像和音频分类的性能。摘要：The ability to quickly learn a new task with minimal instruction - known as few-shot learning - is a central aspect of intelligent agents. Classical few-shot benchmarks make use of few-shot samples from a single modality, but such samples may not be sufficient to characterize an entire concept class. In contrast, humans use cross-modal information to learn new concepts efficiently. In this work, we demonstrate that one can indeed build a better ${\bf visual}$ dog classifier by ${\bf read}$ing about dogs and ${\bf listen}$ing to them bark. To do so, we exploit the fact that recent multimodal foundation models such as CLIP are inherently cross-modal, mapping different modalities to the same representation space. Specifically, we propose a simple cross-modal adaptation approach that learns from few-shot examples spanning different modalities. By repurposing class names as additional one-shot training samples, we achieve SOTA results with an embarrassingly simple linear classifier for vision-language adaptation. Furthermore, we show that our approach can benefit existing methods such as prefix tuning, adapters, and classifier ensembling. Finally, to explore other modalities beyond vision and language, we construct the first (to our knowledge) audiovisual few-shot benchmark and use cross-modal training to improve the performance of both image and audio classification.
【9】 What artificial intelligence might teach us about the origin of human language
标题：关于人类语言的起源，人工智能可能会教给我们什么
链接：https://arxiv.org/abs/2301.06211
机构：Alexander Nagoya University of Business and Commerce This study has not undergone peer review备注：ICPHS2023 Conference Submission. 5 pages摘要：这项研究探索了一个有趣的模式，从研究中出现的结合人工智能与声音符号。在这些研究中，监督式机器学习算法被训练成基于所指名称的发音来对样本进行分类。机器学习算法是语音符号的高效学习者，但它们往往偏向于一个类别。模式如下：当类别可论证地表示更大的威胁时，算法倾向于对该类别进行过度预测。一个假设，框架的错误管理理论，提出这可能是适应的证据偏好谨慎行为。通过使用组成中文、日文和韩文口袋妖怪名称的声音构建极端梯度增强（XGBoost）模型并观察分类误差分布来检验该假设。摘要：This study explores an interesting pattern emerging from research that combines artificial intelligence with sound symbolism. In these studies, supervised machine learning algorithms are trained to classify samples based on the sounds of referent names. Machine learning algorithms are efficient learners of sound symbolism, but they tend to bias one category over the other. The pattern is this: when a category arguably represents greater threat, the algorithms tend to overpredict to that category. A hypothesis, framed by error management theory, is presented that proposes that this may be evidence of an adaptation to preference cautious behaviour. This hypothesis is tested by constructing extreme gradient boosted (XGBoost) models using the sounds that make up the names of Chinese, Japanese and Korean Pokemon and observing classification error distribution.
【10】 Training one model to detect heart and lung sound events from single point auscultations
标题：训练一个模型从单点听诊检测心音和肺音事件
链接：https://arxiv.org/abs/2301.06078
作者：Leander Melms,Robert R. Ilesan,Ulrich Köhler,Olaf Hildebrandt,Regina Conradt,Jens Eckstein,Cihan Atila,Sami Matrood,Bernhard Schieffer,Jürgen R. Schaefer,Tobias Müller,Julius Obergassel,Nadine Schlicker,Martin C. Hirsch机构：Institute of Artificial Intelligence, Philipps-University Marburg; ,Clinic of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel; ,Division of摘要：目的：提出一种半监督训练方法，仅用一个训练好的模型就能同时检测肺音和心音，且对听诊点不变性。方法：我们使用2016年Physionet/CinC挑战赛、2022年George Moody挑战赛和肺音数据库HF_V1的开放存取数据。我们首先使用来自不同听诊数据库的前景地面实况（GT）标签来训练专家单任务模型，以识别相应肺和心脏听诊数据库中的背景声音事件。以这种方式生成的伪标记在新的训练迭代中与地面真实标记组合，使得随后训练新的模型以检测前景和背景信号。基准测试确保了新训练的模型可以检测不同听诊部位的肺部和心音事件，而不会在原始任务上倒退。我们还为心肺听诊中的背景信号建立了人工验证的标签，以评估模型。结果：在这项工作中，我们首次报告了i）肺音事件的多类预测和ii）心脏和肺音事件的同时检测的结果，并仅使用一个模型获得了有竞争力的结果。组合的多任务模型在心音检测方面略有倒退，而在肺音检测准确度方面显著增加，在六个类别上的总体宏观F1评分为39.2%，表示比单任务基线模型提高了6.7%。结论/意义：据我们所知，这是迄今为止开发的第一种方法，用于测量对听诊位置和捕获设备都不变的心脏和肺音事件。因此，我们的模型能够从任何听诊位置执行肺和心音检测。摘要：Objective: This work proposes a semi-supervised training approach for detecting lung and heart sounds simultaneously with only one trained model and in invariance to the auscultation point. Methods: We use open-access data from the 2016 Physionet/CinC Challenge, the 2022 George Moody Challenge, and from the lung sound database HF_V1. We first train specialist single-task models using foreground ground truth (GT) labels from different auscultation databases to identify background sound events in the respective lung and heart auscultation databases. The pseudo-labels generated in this way were combined with the ground truth labels in a new training iteration, such that a new model was subsequently trained to detect foreground and background signals. Benchmark tests ensured that the newly trained model could detect both, lung, and heart sound events in different auscultation sites without regressing on the original task. We also established hand-validated labels for the respective background signal in heart and lung sound auscultations to evaluate the models. Results: In this work, we report for the first time results for i) a multi-class prediction for lung sound events and ii) for simultaneous detection of heart and lung sound events and achieve competitive results using only one model. The combined multi-task model regressed slightly in heart sound detection and gained significantly in lung sound detection accuracy with an overall macro F1 score of 39.2% over six classes, representing a 6.7% improvement over the single-task baseline models. Conclusion/Significance: To the best of our knowledge, this is the first approach developed to date for measuring heart and lung sound events invariant to both, the auscultation site and capturing device. Hence, our model is capable of performing lung and heart sound detection from any auscultation location.
【11】 An Order-Complexity Model for Aesthetic Quality Assessment of Symbolic Homophony Music Scores
标题：象征性同音乐谱审美质量评价的有序复杂性模型
链接：https://arxiv.org/abs/2301.05908
作者：Xin Jin,Wu Zhou,Jinyu Wang,Duo Xu,Yiqing Rong,Shuai Cui摘要：计算美学评价在视觉艺术领域已经取得了很大的成就，但在音乐领域的研究工作还有待探索。虽然现有的音乐生成工作非常丰富，但与人类作曲家创作的乐谱相比，人工智能生成的乐谱质量相对较差。AI创作的乐谱通常单调乏味，缺乏情感。基于Birkhoff的审美测度，提出了一种客观量化的主调乐谱审美质量评价方法。我们工作的主要贡献如下：首先，提出了一个客观评价乐谱质量的主调乐谱审美模型作为基准模型;其次，提出了八个基本音乐特征和四个音乐审美特征。摘要：Computational aesthetics evaluation has made great achievements in the field of visual arts, but the research work on music still needs to be explored. Although the existing work of music generation is very substantial, the quality of music score generated by AI is relatively poor compared with that created by human composers. The music scores created by AI are usually monotonous and devoid of emotion. Based on Birkhoff's aesthetic measure, this paper proposes an objective quantitative evaluation method for homophony music score aesthetic quality assessment. The main contributions of our work are as follows: first, we put forward a homophony music score aesthetic model to objectively evaluate the quality of music score as a baseline model; second, we put forward eight basic music features and four music aesthetic features.
【12】 Acoustic correlates of the syllabic rhythm of speech: Modulation spectrum or local features of the temporal envelope
标题：语音音节节奏的声学相关性：调制频谱还是时间包络的局部特征
链接：https://arxiv.org/abs/2301.05898
作者：Yuran Zhang,Jiajie Zou,Nai Ding机构：a College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China, Corresponding author:, Zhejiang University, Hangzhou, China摘要：音节在言语中是一个感知上显著的单位。由于音节及其声学相关，即，语音包络具有在4和8Hz之间的节奏性的优选范围，假设θ频带神经振荡在基于包络提取音节中起主要作用。然而，一项文献调查显示，关于语音包络和音节之间关系的证据并不一致，本研究通过分析大型语音语料库重新审视了这一问题。结果表明，只有当分析集中在几分钟的语音记录上时，以调制谱为特征的语音包络的中心频率才与音节速率可靠地相关。相反，在时域中，语音包络的分量被可靠地锁相到音节起始。基于说话者无关模型，音节起始的定时解释了语音包络的大约24%的方差。这些结果表明，语音包络中的局部特征，而不是调制谱，是音节的更可靠的声学相关性。摘要：The syllable is a perceptually salient unit in speech. Since both the syllable and its acoustic correlate, i.e., the speech envelope, have a preferred range of rhythmicity between 4 and 8 Hz, it is hypothesized that theta-band neural oscillations play a major role in extracting syllables based on the envelope. A literature survey, however, reveals inconsistent evidence about the relationship between speech envelope and syllables, and the current study revisits this question by analyzing large speech corpora. It is shown that the center frequency of speech envelope, characterized by the modulation spectrum, reliably correlates with the rate of syllables only when the analysis is pooled over minutes of speech recordings. In contrast, in the time domain, a component of the speech envelope is reliably phase-locked to syllable onsets. Based on a speaker-independent model, the timing of syllable onsets explains about 24% variance of the speech envelope. These results indicate that local features in the speech envelope, instead of the modulation spectrum, are a more reliable acoustic correlate of syllables.
【13】 Multi-resolution location-based training for multi-channel continuous speech separation
标题：用于多通道连续语音分离的多分辨率位置训练
链接：https://arxiv.org/abs/2301.06458
作者：Hassan Taherian,DeLiang Wang 机构：Department of Computer Science and Engineering, The Ohio State University, USA, Center for Cognitive and Brain Sciences, The Ohio State University, USA备注：Submitted to ICASSP 23摘要：当多人语音重叠时，自动语音识别（ASR）系统的性能会严重下降。在会议环境中，通常执行语音分离以提高ASR系统的鲁棒性。最近，基于位置的训练（LBT）被提出作为一种新的多通道说话人无关说话人分离训练准则。假设阵列几何形状固定，LBT在完全重叠的语音和匹配的混响条件下优于广泛使用的置换不变训练。本文将LBT扩展到会话式多声道说话人分离。我们引入多分辨率LBT来估计从低到高的时间和频率分辨率的复杂谱图。利用多分辨率LBT，基于物理空间中的说话者位置一致地分配卷积核。在LibriCSS语料库上的测试结果表明，多分辨率LBT的性能优于其他同类方法。摘要：The performance of automatic speech recognition (ASR) systems severely degrades when multi-talker speech overlap occurs. In meeting environments, speech separation is typically performed to improve the robustness of ASR systems. Recently, location-based training (LBT) was proposed as a new training criterion for multi-channel talker-independent speaker separation. Assuming fixed array geometry, LBT outperforms widely-used permutation-invariant training in fully overlapped utterances and matched reverberant conditions. This paper extends LBT to conversational multi-channel speaker separation. We introduce multi-resolution LBT to estimate the complex spectrograms from low to high time and frequency resolutions. With multi-resolution LBT, convolutional kernels are assigned consistently based on speaker locations in physical space. Evaluation results show that multi-resolution LBT consistently outperforms other competitive methods on the recorded LibriCSS corpus.
【14】 Modulation spectral features for speech emotion recognition using deep neural networks
标题：基于深度神经网络的语音情感识别调制谱特征
链接：https://arxiv.org/abs/2301.05868
作者：Premjeet Singh,Md Sahidullah,Goutam Saha机构：Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology, India-, Kharagpur, India, Université de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：本文研究了基于恒Q变换的调制谱特征（CQT-MSF）在语音情感识别（SER）中的应用。人类对声音的感知和分析包括两个重要的认知部分：早期听觉分析和基于皮层的处理。早期的听觉分析考虑基于声谱图的表征，而基于皮层的分析包括从声谱图中提取时间调制。这种谱图的时间调制表示称为调制谱特征（MSF）。由于恒定Q变换（CQT）在语音的情感显著低频区域提供了更高的分辨率，我们发现基于CQT的谱图连同其时间调制一起提供了富含情感特定信息的表示。我们认为，CQT-MSF与二维卷积网络结合使用时，可以为SER提供时移不变性和变形不敏感的表示.在Berlin EmoDB和RAVDESS两个流行的SER数据库上，我们的结果表明，CQT-MSF优于标准的基于mel尺度的谱图及其调制特征.我们还证明了我们提出的特征优于平移和变形不变的散射变换系数，因此，显示了联合手工和自学习特征提取的重要性，而不是完全依赖于手工特征。最后，我们执行Grad-CAM分析，以直观地检查恒定Q调制特征对SER的贡献。摘要：This work explores the use of constant-Q transform based modulation spectral features (CQT-MSF) for speech emotion recognition (SER). The human perception and analysis of sound comprise of two important cognitive parts: early auditory analysis and cortex-based processing. The early auditory analysis considers spectrogram-based representation whereas cortex-based analysis includes extraction of temporal modulations from the spectrogram. This temporal modulation representation of spectrogram is called modulation spectral feature (MSF). As the constant-Q transform (CQT) provides higher resolution at emotion salient low-frequency regions of speech, we find that CQT-based spectrogram, together with its temporal modulations, provides a representation enriched with emotion-specific information. We argue that CQT-MSF when used with a 2-dimensional convolutional network can provide a time-shift invariant and deformation insensitive representation for SER. Our results show that CQT-MSF outperforms standard mel-scale based spectrogram and its modulation features on two popular SER databases, Berlin EmoDB and RAVDESS. We also show that our proposed feature outperforms the shift and deformation invariant scattering transform coefficients, hence, showing the importance of joint hand-crafted and self-learned feature extraction instead of reliance on complete hand-crafted features. Finally, we perform Grad-CAM analysis to visually inspect the contribution of constant-Q modulation features over SER.【1】 Multi-resolution location-based training for multi-channel continuous speech separation
标题：用于多通道连续语音分离的多分辨率位置训练
链接：https://arxiv.org/abs/2301.06458
*与cs.SD语音【13】为同一篇
作者：Hassan Taherian,DeLiang Wang机构：Department of Computer Science and Engineering, The Ohio State University, USA, Center for Cognitive and Brain Sciences, The Ohio State University, USA备注：Submitted to ICASSP 23摘要：当多人语音重叠时，自动语音识别（ASR）系统的性能会严重下降。在会议环境中，通常执行语音分离以提高ASR系统的鲁棒性。最近，基于位置的训练（LBT）被提出作为一种新的多通道说话人无关说话人分离训练准则。假设阵列几何形状固定，LBT在完全重叠的语音和匹配的混响条件下优于广泛使用的置换不变训练。本文将LBT扩展到会话式多声道说话人分离。我们引入多分辨率LBT来估计从低到高的时间和频率分辨率的复杂谱图。利用多分辨率LBT，基于物理空间中的说话者位置一致地分配卷积核。在LibriCSS语料库上的测试结果表明，多分辨率LBT的性能优于其他同类方法。摘要：The performance of automatic speech recognition (ASR) systems severely degrades when multi-talker speech overlap occurs. In meeting environments, speech separation is typically performed to improve the robustness of ASR systems. Recently, location-based training (LBT) was proposed as a new training criterion for multi-channel talker-independent speaker separation. Assuming fixed array geometry, LBT outperforms widely-used permutation-invariant training in fully overlapped utterances and matched reverberant conditions. This paper extends LBT to conversational multi-channel speaker separation. We introduce multi-resolution LBT to estimate the complex spectrograms from low to high time and frequency resolutions. With multi-resolution LBT, convolutional kernels are assigned consistently based on speaker locations in physical space. Evaluation results show that multi-resolution LBT consistently outperforms other competitive methods on the recorded LibriCSS corpus.
【2】 Modulation spectral features for speech emotion recognition using deep neural networks
标题：基于深度神经网络的语音情感识别调制谱特征
链接：https://arxiv.org/abs/2301.05868
*与cs.SD语音【14】为同一篇
作者：Premjeet Singh,Md Sahidullah,Goutam Saha机构：Department of Electronics & Electrical Communication Engineering, Indian Institute of Technology, India-, Kharagpur, India, Université de Lorraine, CNRS, Inria, LORIA, F-, Nancy, France摘要：本文研究了基于恒Q变换的调制谱特征（CQT-MSF）在语音情感识别（SER）中的应用。人类对声音的感知和分析包括两个重要的认知部分：早期听觉分析和基于皮层的处理。早期的听觉分析考虑基于声谱图的表征，而基于皮层的分析包括从声谱图中提取时间调制。这种谱图的时间调制表示称为调制谱特征（MSF）。由于恒定Q变换（CQT）在语音的情感显著低频区域提供了更高的分辨率，我们发现基于CQT的谱图连同其时间调制一起提供了富含情感特定信息的表示。我们认为，CQT-MSF与二维卷积网络结合使用时，可以为SER提供时移不变性和变形不敏感的表示.在Berlin EmoDB和RAVDESS两个流行的SER数据库上，我们的结果表明，CQT-MSF优于标准的基于mel尺度的谱图及其调制特征.我们还证明了我们提出的特征优于平移和变形不变的散射变换系数，因此，显示了联合手工和自学习特征提取的重要性，而不是完全依赖于手工特征。最后，我们执行Grad-CAM分析，以直观地检查恒定Q调制特征对SER的贡献。摘要：This work explores the use of constant-Q transform based modulation spectral features (CQT-MSF) for speech emotion recognition (SER). The human perception and analysis of sound comprise of two important cognitive parts: early auditory analysis and cortex-based processing. The early auditory analysis considers spectrogram-based representation whereas cortex-based analysis includes extraction of temporal modulations from the spectrogram. This temporal modulation representation of spectrogram is called modulation spectral feature (MSF). As the constant-Q transform (CQT) provides higher resolution at emotion salient low-frequency regions of speech, we find that CQT-based spectrogram, together with its temporal modulations, provides a representation enriched with emotion-specific information. We argue that CQT-MSF when used with a 2-dimensional convolutional network can provide a time-shift invariant and deformation insensitive representation for SER. Our results show that CQT-MSF outperforms standard mel-scale based spectrogram and its modulation features on two popular SER databases, Berlin EmoDB and RAVDESS. We also show that our proposed feature outperforms the shift and deformation invariant scattering transform coefficients, hence, showing the importance of joint hand-crafted and self-learned feature extraction instead of reliance on complete hand-crafted features. Finally, we perform Grad-CAM analysis to visually inspect the contribution of constant-Q modulation features over SER.
【3】 MooseNet: A trainable metric for synthesized speech with plda backend
标题：MooseNet：一种基于PLDA后端的可训练合成语音度量
链接：https://arxiv.org/abs/2301.07087
*与cs.SD语音【1】为同一篇
作者：Ondřej Plátek,Ondřej Dušek机构：A TRAINABLE METRIC FOR SYNTHESIZED SPEECH WITH PLDABACKENDOndˇrej PlátekandOndˇrej DušekCharles University, Faculty of Mathematics and PhysicsInstitute of Formal and Applied LinguisticsPrague摘要：我们提出了MooseNet，一个可训练的语音度量，预测听众的平均意见得分（MOS）。我们报告了使用易于使用的建模技术对挑战基线的改进，这些技术还可以扩展到更大的自监督学习（SSL）模型。我们提出了两种模型。第一个模型是神经网络（NN）。作为第二种模型，我们在第一种神经网络模型的上层提出了PLDA生成式模型，改进了纯神经网络模型。我们的两个型号的合奏在所有系统和发声级别指标上都达到了VoiceMOS排行榜的前3或前4名。摘要：We present MooseNet, a trainable speech metric that predicts listeners' Mean Opinion Score (MOS). We report improvements to the challenge baselines using easy-to-use modeling techniques, which also scales for larger self-supervised learning (SSL) model. We present two models. The first model is a Neural Network (NN). As a second model, we propose a PLDA generative model on the top layers of the first NN model, which improves the pure NN model. Ensembles from our two models achieve the top 3 or 4 VoiceMOS leaderboard places on all system and utterance level metrics.
【4】 Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducer
标题：两级上下文词过滤在统一流传输和非流传输传感器中的上下文偏向
链接：https://arxiv.org/abs/2301.06735
*与cs.SD语音【3】为同一篇
作者：Zhanheng Yang,Sining Sun,Xiong Wang,Yike Zhang,Long Ma,Lei Xie机构：Audio, Speech and Language Processing Group, School of Computer Science, Northwestern Polytechnical University, Xi’an, China, Tencent Technology Co., Ltd, Beijing, China摘要：It is difficult for an end-to-end (E2E) ASR system to recognize words such as named entities appearing infrequently in the training data. A widely used method to mitigate this issue is feeding contextual information into the acoustic model. A contextual word list is necessary, which lists all possible contextual word candidates. Previous works have proven that the size and quality of the list are crucial. A compact and accurate list can boost the performance significantly. In this paper, we propose an efficient approach to obtain a high quality contextual word list for a unified streaming and non-streaming based Conformer-Transducer (C-T) model. Specifically, we make use of the phone-level streaming output to first filter the predefined contextual word list. During the subsequent non-streaming inference, the words in the filtered list are regarded as contextual information fused into non-casual encoder and decoder to generate the final recognition results. Our approach can take advantage of streaming recognition hypothesis, improve the accuracy of the contextual ASR system and speed up the inference process as well. Experiments on two datasets demonstrates over 20% relative character error rate reduction (CERR) comparing to the baseline system. Meanwile, the RTF of our system can be stabilized within 0.15 when the size of the contextual word list grows over 6,000.
【5】 Using Kaldi for Automatic Speech Recognition of Conversational Austrian German
标题：使用Kaldi实现会话奥地利德语的自动语音识别
链接：https://arxiv.org/abs/2301.06475
*与cs.SD语音【4】为同一篇
作者：Julian Linke,Saskia Wepner,Gernot Kubin,Barbara Schuppler机构：Signal Processing and Speech Communication Laboratory, Graz University of Technology, Inffeldgasse ,c, Graz, Austria备注：10 pages, 2 figures, 4 tables摘要：随着对话系统越来越具有交互性和社会性，对会话语音进行准确的自动语音识别也变得越来越重要。这就把焦点从简短、自发、任务导向的对话转移到了更为复杂的随意的面对面对话上。然而，对这些会话的收集和注释是一个耗时的过程，并且针对这种特定说话风格的数据很少。本文以阅读和会话奥地利德语为被试进行了ASR实验。为了解决德语会话资源有限的问题，同时考虑到说话者之间在发音特征方面的巨大差异，我们通过引入（大的）基于知识的发音词典来改进基于Kaldi的ASR系统，同时探索不同的基于数据的方法来限制每个词条的发音变体的数量。我们在奥地利德语阅读演讲中达到了0.4%的最佳WER，在会话演讲中达到了48.5%的最佳平均WER。我们发现，通过使用我们的最佳发音词典，与通过将用于语言模型的数据的大小增加大约10%相比，可以实现类似的高性能。360%到760%。我们的研究结果表明，对于低资源的情况--尽管语音技术的总体趋势是只使用基于数据的方法--基于知识的方法是一种成功的、有效的方法。摘要：As dialogue systems are becoming more and more interactional and social, also the accurate automatic speech recognition (ASR) of conversational speech is of increasing importance. This shifts the focus from short, spontaneous, task-oriented dialogues to the much higher complexity of casual face-to-face conversations. However, the collection and annotation of such conversations is a time-consuming process and data is sparse for this specific speaking style. This paper presents ASR experiments with read and conversational Austrian German as target. In order to deal with having only limited resources available for conversational German and, at the same time, with a large variation among speakers with respect to pronunciation characteristics, we improve a Kaldi-based ASR system by incorporating a (large) knowledge-based pronunciation lexicon, while exploring different data-based methods to restrict the number of pronunciation variants for each lexical entry. We achieve best WER of 0.4% on Austrian German read speech and best average WER of 48.5% on conversational speech. We find that by using our best pronunciation lexicon a similarly high performance can be achieved than by increasing the size of the data used for the language model by approx. 360% to 760%. Our findings indicate that for low-resource scenarios -- despite the general trend in speech technology towards using data-based methods only -- knowledge-based approaches are a successful, efficient method.
【6】 Msanii: High Fidelity Music Synthesis on a Shoestring Budget
标题：Msanii：以极低的预算合成高保真音乐
链接：https://arxiv.org/abs/2301.06468
*与cs.SD语音【5】为同一篇
机构：Independent Researcher备注：15 pages, 8 figures, for demo see this https URL and for code, see this https URL, this paper is a work in progress摘要：在这篇文章中，我们提出了一个新的基于扩散的模型来有效地合成长上下文、高保真度的音乐。我们的模型结合了梅尔谱图的表现力，扩散模型的生成能力，以及神经声码器的声码能力。我们在不使用拼接合成、级联架构或压缩技术的情况下，以高采样率（44.1 kHz）合成数十秒（190秒）的立体声音乐，从而证明Msanii的有效性。据我们所知，这是首次成功地采用基于扩散的模型来以高采样率合成如此长的音乐样本。我们的演示可以在www.example.com上找到https://kinyugo.github.io/msanii-demo，我们的代码可以在https://github.com/Kinyugo/msanii。摘要：In this paper, we present Msanii, a novel diffusion-based model for synthesizing long-context, high-fidelity music efficiently. Our model combines the expressiveness of mel spectrograms, the generative capabilities of diffusion models, and the vocoding capabilities of neural vocoders. We demonstrate the effectiveness of Msanii by synthesizing tens of seconds (190 seconds) of stereo music at high sample rates (44.1 kHz) without the use of concatenative synthesis, cascading architectures, or compression techniques. To the best of our knowledge, this is the first work to successfully employ a diffusion-based model for synthesizing such long music samples at high sample rates. Our demo can be found https://kinyugo.github.io/msanii-demo and our code https://github.com/Kinyugo/msanii .
【7】 Improving Target Speaker Extraction with Sparse LDA-transformed Speaker Embeddings
标题：稀疏LDA变换说话人嵌入改进目标说话人提取
链接：https://arxiv.org/abs/2301.06277
*与cs.SD语音【7】为同一篇
作者：Kai Liu,Xucheng Wan,Ziqing Du,Huan Zhou机构：AI Application Research Center, Huawei Technologies Co., Ltd. Shenzhen, China备注：ACCEPTED by NCMMSC 2022摘要：目标说话人提取（TSE）是一种实用的语音分离方法，它利用从说话人中提取的附加说话人线索来提取出期望说话人的语音。其主要挑战在于如何正确提取和利用说话人线索以提高提取的语音质量。现有的大多数TSE研究中采用的线索提取方法是直接利用区分性说话人嵌入，从预先训练好的模型中提取线索进行说话人确认。虽然高说话人区分度是说话人确认任务最理想的属性，但我们认为它对于TSE来说可能过于复杂。在本研究中，我们提出一个简化的说话人线索与明确的类别可分性可能是首选的TSE。为了验证我们的建议，我们引入了几种说话人线索的形式，包括原始说话人嵌入（如，x向量和xi向量）和由稀疏LDA变换产生的新说话人嵌入。通过将这些说话人线索与SepFormer（一种SOTA语音分离模型）相结合来建立相应的TSE模型。在基准WSJ 0 - 2 mix数据集上检查这些TSE模型的性能。实验结果验证了该方法的有效性和通用性，SI-SDRi相对提高了9.9%。此外，SI-SDRi为19.4 dB，PESQ为3.78，我们的最佳TSE系统显著优于当前SOTA系统，并提供了迄今为止WSJ 0 - 2 mix报告的最佳TSE结果。摘要：As a practical alternative of speech separation, target speaker extraction (TSE) aims to extract the speech from the desired speaker using additional speaker cue extracted from the speaker. Its main challenge lies in how to properly extract and leverage the speaker cue to benefit the extracted speech quality. The cue extraction method adopted in majority existing TSE studies is to directly utilize discriminative speaker embedding, which is extracted from the pre-trained models for speaker verification. Although the high speaker discriminability is a most desirable property for speaker verification task, we argue that it may be too sophisticated for TSE. In this study, we propose that a simplified speaker cue with clear class separability might be preferred for TSE. To verify our proposal, we introduce several forms of speaker cues, including naive speaker embedding (such as, x-vector and xi-vector) and new speaker embeddings produced from sparse LDA-transform. Corresponding TSE models are built by integrating these speaker cues with SepFormer (one SOTA speech separation model). Performances of these TSE models are examined on the benchmark WSJ0-2mix dataset. Experimental results validate the effectiveness and generalizability of our proposal, showing up to 9.9% relative improvement in SI-SDRi. Moreover, with SI-SDRi of 19.4 dB and PESQ of 3.78, our best TSE system significantly outperforms the current SOTA systems and offers the top TSE results reported till date on the WSJ0-2mix.
【8】 Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models
标题：多通道帮助单通道：基于多通道模型的跨通道短程学习
链接：https://arxiv.org/abs/2301.06267
*与cs.SD语音【8】为同一篇
作者：Zhiqiu Lin,Samuel Yu,Zhiyi Kuang,Deepak Pathak,Deva Ramana机构：Carnegie Mellon University备注：Project website: this https URL摘要：用最少的指令快速学习一个新任务的能力--称为Few-Shot学习--是智能代理的一个核心方面。经典的Few-Shot基准使用来自单个模态的少镜头样本，但是这样的样本可能不足以表征整个概念类。相反，人类使用跨模态信息来有效地学习新概念。在这项工作中，我们证明了人们确实可以通过${\bf read}$记录狗的信息和${\bf listen}$记录狗的叫声来构建一个更好的${\bf visual}$狗分类器。为了做到这一点，我们利用了这样一个事实，即最近的多模态基础模型，如CLIP，本质上是跨模态的，将不同的模态映射到同一个表示空间。具体地说，我们提出了一种简单的跨模态自适应方法，该方法从跨越不同模态的Few-Shot示例中学习。通过将类名称重新用作额外的一次性训练样本，我们用一个令人尴尬的简单线性分类器实现了SOTA结果，用于视觉语言适应。此外，我们还证明了我们的方法可以有益于现有的方法，如前缀调整，适配器，和分类器集成。最后，为了探索视觉和语言之外的其他模态，我们构建了第一个（据我们所知）视听Few-Shot基准，并使用跨模态训练来提高图像和音频分类的性能。摘要：The ability to quickly learn a new task with minimal instruction - known as few-shot learning - is a central aspect of intelligent agents. Classical few-shot benchmarks make use of few-shot samples from a single modality, but such samples may not be sufficient to characterize an entire concept class. In contrast, humans use cross-modal information to learn new concepts efficiently. In this work, we demonstrate that one can indeed build a better ${\bf visual}$ dog classifier by ${\bf read}$ing about dogs and ${\bf listen}$ing to them bark. To do so, we exploit the fact that recent multimodal foundation models such as CLIP are inherently cross-modal, mapping different modalities to the same representation space. Specifically, we propose a simple cross-modal adaptation approach that learns from few-shot examples spanning different modalities. By repurposing class names as additional one-shot training samples, we achieve SOTA results with an embarrassingly simple linear classifier for vision-language adaptation. Furthermore, we show that our approach can benefit existing methods such as prefix tuning, adapters, and classifier ensembling. Finally, to explore other modalities beyond vision and language, we construct the first (to our knowledge) audiovisual few-shot benchmark and use cross-modal training to improve the performance of both image and audio classification.
【9】 What artificial intelligence might teach us about the origin of human language
标题：关于人类语言的起源，人工智能可能会教给我们什么
链接：https://arxiv.org/abs/2301.06211
*与cs.SD语音【9】为同一篇
机构：Alexander Nagoya University of Business and Commerce This study has not undergone peer review备注：ICPHS2023 Conference Submission. 5 pages摘要：这项研究探索了一个有趣的模式，从研究中出现的结合人工智能与声音符号。在这些研究中，监督式机器学习算法被训练成基于所指名称的发音来对样本进行分类。机器学习算法是语音符号的高效学习者，但它们往往偏向于一个类别。模式如下：当类别可论证地表示更大的威胁时，算法倾向于对该类别进行过度预测。一个假设，框架的错误管理理论，提出这可能是适应的证据偏好谨慎行为。通过使用组成中文、日文和韩文口袋妖怪名称的声音构建极端梯度增强（XGBoost）模型并观察分类误差分布来检验该假设。摘要：This study explores an interesting pattern emerging from research that combines artificial intelligence with sound symbolism. In these studies, supervised machine learning algorithms are trained to classify samples based on the sounds of referent names. Machine learning algorithms are efficient learners of sound symbolism, but they tend to bias one category over the other. The pattern is this: when a category arguably represents greater threat, the algorithms tend to overpredict to that category. A hypothesis, framed by error management theory, is presented that proposes that this may be evidence of an adaptation to preference cautious behaviour. This hypothesis is tested by constructing extreme gradient boosted (XGBoost) models using the sounds that make up the names of Chinese, Japanese and Korean Pokemon and observing classification error distribution.
【10】 Training one model to detect heart and lung sound events from single point auscultations
标题：训练一个模型从单点听诊检测心音和肺音事件
链接：https://arxiv.org/abs/2301.06078
*与cs.SD语音【10】为同一篇
作者：Leander Melms,Robert R. Ilesan,Ulrich Köhler,Olaf Hildebrandt,Regina Conradt,Jens Eckstein,Cihan Atila,Sami Matrood,Bernhard Schieffer,Jürgen R. Schaefer,Tobias Müller,Julius Obergassel,Nadine Schlicker,Martin C. Hirsch机构：Institute of Artificial Intelligence, Philipps-University Marburg; ,Clinic of Oral and Cranio-Maxillofacial Surgery, University Hospital Basel; ,Division of摘要：目的：提出一种半监督训练方法，仅用一个训练好的模型就能同时检测肺音和心音，且对听诊点不变性。方法：我们使用2016年Physionet/CinC挑战赛、2022年George Moody挑战赛和肺音数据库HF_V1的开放存取数据。我们首先使用来自不同听诊数据库的前景地面实况（GT）标签来训练专家单任务模型，以识别相应肺和心脏听诊数据库中的背景声音事件。以这种方式生成的伪标记在新的训练迭代中与地面真实标记组合，使得随后训练新的模型以检测前景和背景信号。基准测试确保了新训练的模型可以检测不同听诊部位的肺部和心音事件，而不会在原始任务上倒退。我们还为心肺听诊中的背景信号建立了人工验证的标签，以评估模型。结果：在这项工作中，我们首次报告了i）肺音事件的多类预测和ii）心脏和肺音事件的同时检测的结果，并仅使用一个模型获得了有竞争力的结果。组合的多任务模型在心音检测方面略有倒退，而在肺音检测准确度方面显著增加，在六个类别上的总体宏观F1评分为39.2%，表示比单任务基线模型提高了6.7%。结论/意义：据我们所知，这是迄今为止开发的第一种方法，用于测量对听诊位置和捕获设备都不变的心脏和肺音事件。因此，我们的模型能够从任何听诊位置执行肺和心音检测。摘要：Objective: This work proposes a semi-supervised training approach for detecting lung and heart sounds simultaneously with only one trained model and in invariance to the auscultation point. Methods: We use open-access data from the 2016 Physionet/CinC Challenge, the 2022 George Moody Challenge, and from the lung sound database HF_V1. We first train specialist single-task models using foreground ground truth (GT) labels from different auscultation databases to identify background sound events in the respective lung and heart auscultation databases. The pseudo-labels generated in this way were combined with the ground truth labels in a new training iteration, such that a new model was subsequently trained to detect foreground and background signals. Benchmark tests ensured that the newly trained model could detect both, lung, and heart sound events in different auscultation sites without regressing on the original task. We also established hand-validated labels for the respective background signal in heart and lung sound auscultations to evaluate the models. Results: In this work, we report for the first time results for i) a multi-class prediction for lung sound events and ii) for simultaneous detection of heart and lung sound events and achieve competitive results using only one model. The combined multi-task model regressed slightly in heart sound detection and gained significantly in lung sound detection accuracy with an overall macro F1 score of 39.2% over six classes, representing a 6.7% improvement over the single-task baseline models. Conclusion/Significance: To the best of our knowledge, this is the first approach developed to date for measuring heart and lung sound events invariant to both, the auscultation site and capturing device. Hence, our model is capable of performing lung and heart sound detection from any auscultation location.
【11】 An Order-Complexity Model for Aesthetic Quality Assessment of Symbolic Homophony Music Scores
标题：象征性同音乐谱审美质量评价的有序复杂性模型
链接：https://arxiv.org/abs/2301.05908
*与cs.SD语音【11】为同一篇
作者：Xin Jin,Wu Zhou,Jinyu Wang,Duo Xu,Yiqing Rong,Shuai Cui摘要：计算美学评价在视觉艺术领域已经取得了很大的成就，但在音乐领域的研究工作还有待探索。虽然现有的音乐生成工作非常丰富，但与人类作曲家创作的乐谱相比，人工智能生成的乐谱质量相对较差。AI创作的乐谱通常单调乏味，缺乏情感。基于Birkhoff的审美测度，提出了一种客观量化的主调乐谱审美质量评价方法。我们工作的主要贡献如下：首先，提出了一个客观评价乐谱质量的主调乐谱审美模型作为基准模型;其次，提出了八个基本音乐特征和四个音乐审美特征。 摘要：Computational aesthetics evaluation has made great achievements in the field of visual arts, but the research work on music still needs to be explored. Although the existing work of music generation is very substantial, the quality of music score generated by AI is relatively poor compared with that created by human composers. The music scores created by AI are usually monotonous and devoid of emotion. Based on Birkhoff's aesthetic measure, this paper proposes an objective quantitative evaluation method for homophony music score aesthetic quality assessment. The main contributions of our work are as follows: first, we put forward a homophony music score aesthetic model to objectively evaluate the quality of music score as a baseline model; second, we put forward eight basic music features and four music aesthetic features.
【12】 Acoustic correlates of the syllabic rhythm of speech: Modulation spectrum or local features of the temporal envelope
标题：语音音节节奏的声学相关性：调制频谱还是时间包络的局部特征
链接：https://arxiv.org/abs/2301.05898
*与cs.SD语音【12】为同一篇
作者：Yuran Zhang,Jiajie Zou,Nai Ding机构：a College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China, Corresponding author:, Zhejiang University, Hangzhou, China摘要：音节在言语中是一个感知上显著的单位。由于音节及其声学相关，即，语音包络具有在4和8Hz之间的节奏性的优选范围，假设θ频带神经振荡在基于包络提取音节中起主要作用。然而，一项文献调查显示，关于语音包络和音节之间关系的证据并不一致，本研究通过分析大型语音语料库重新审视了这一问题。结果表明，只有当分析集中在几分钟的语音记录上时，以调制谱为特征的语音包络的中心频率才与音节速率可靠地相关。相反，在时域中，语音包络的分量被可靠地锁相到音节起始。基于说话者无关模型，音节起始的定时解释了语音包络的大约24%的方差。这些结果表明，语音包络中的局部特征，而不是调制谱，是音节的更可靠的声学相关性。摘要：The syllable is a perceptually salient unit in speech. Since both the syllable and its acoustic correlate, i.e., the speech envelope, have a preferred range of rhythmicity between 4 and 8 Hz, it is hypothesized that theta-band neural oscillations play a major role in extracting syllables based on the envelope. A literature survey, however, reveals inconsistent evidence about the relationship between speech envelope and syllables, and the current study revisits this question by analyzing large speech corpora. It is shown that the center frequency of speech envelope, characterized by the modulation spectrum, reliably correlates with the rate of syllables only when the analysis is pooled over minutes of speech recordings. In contrast, in the time domain, a component of the speech envelope is reliably phase-locked to syllable onsets. Based on a speaker-independent model, the timing of syllable onsets explains about 24% variance of the speech envelope. These results indicate that local features in the speech envelope, instead of the modulation spectrum, are a more reliable acoustic correlate of syllables.机器翻译，仅供参考
每日语音速递