今日论文合集：cs.SD语音2篇，eess.AS音频处理2篇。
【1】 Epic-Sounds: A Large-scale Dataset of Actions That Sound
标题：史诗-声音：一个大规模的行动数据集，听起来
链接：https://arxiv.org/abs/2302.00646
作者：Jaesung Huh,Jacob Chalk,Evangelos Kazakos,Dima Damen,Andrew Zisserman机构：Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, Department of Computer Science, University of Bristol, UK摘要：我们介绍EPIC-SOUNDS，一个大规模的音频标注数据集，捕获自我中心视频音频流中的时间范围和类标签。我们提出了一个注释管道，其中注释器暂时标记可区分的音频段，并描述可能导致这种声音的动作。我们通过将这些自由形式的音频描述分组到类中，来识别可以纯粹从音频中区分出来的动作。对于涉及物体碰撞的动作，我们收集这些物体的材料的人类注释（例如，玻璃物体被放置在木制表面上），我们从视觉标签中验证这些注释，丢弃歧义。总体而言，EPIC-SOUNDS包括78.4k个可闻事件和动作的分类段，分布在44个类别以及39.2k个未分类段。我们在我们的数据集上训练和评估了两个最先进的音频识别模型，强调了纯音频标签的重要性和当前模型识别声音动作的局限性。摘要：We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping these free-form descriptions of audio into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes as well as 39.2k non-categorised segments. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.
【2】 Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training
标题：Jointist：通过联合训练同步提高多乐器改编和乐源分离
链接：https://arxiv.org/abs/2302.00286
作者：Kin Wai Cheuk,Keunwoo Choi,Qiuqiang Kong,Bochen Li,Minz Won,Ju-Chiang Wang,Yun-Ning Hung Dorien Herremans机构：Yun-Ning Hung, Dorien Herremans, Singapore University of Technology and Design, Singapore, Gaudio Lab, South Korea, Bytedance, USA备注：arXiv admin note: text overlap with arXiv:2206.10805摘要：在本文中，我们介绍了Jointist，这是一个乐器感知的多乐器框架，能够转录、识别和分离音频片段中的多个乐器。Jointist包括一个仪器识别模块，用于调节其他两个模块：输出乐器专用钢琴卷的转录模块，以及利用乐器信息和转录结果的源分离模块。转录单元和源分离单元的联合培训有助于提高这两项任务的绩效。仪器模块是可选的，并且可以由人类用户直接控制。这使得Jointist成为一个灵活的用户可控框架。 鉴于现代流行音乐通常由多种乐器组成，我们富有挑战性的问题公式化使得该模型在现实世界中非常有用。然而，它的新颖性要求对如何评价这种模式采取新的观点。在实验中，我们从多个方面对所提出的模型进行了评估，为多仪器转录提供了一个新的评估视角。我们的主观聆听研究表明，Jointist在流行音乐上实现了最先进的表现，优于MT3等现有的多乐器转录模型。我们还认为，转录模型可以用作其他音乐分析任务的预处理模块。我们对几个下游任务进行了实验，发现所提出的方法提高了转录超过1个百分点（ppt.），5SDR源分离，1.8ppt的下拍检测，和弦识别为1.4ppt，并且密钥估计减少1.4ppt，当使用从Jointist获得的转录结果时。摘要：In this paper, we introduce Jointist, an instrument-aware multi-instrument framework that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip. Jointist consists of an instrument recognition module that conditions the other two modules: a transcription module that outputs instrument-specific piano rolls, and a source separation module that utilizes instrument information and transcription results. The joint training of the transcription and source separation modules serves to improve the performance of both tasks. The instrument module is optional and can be directly controlled by human users. This makes Jointist a flexible user-controllable framework. Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. Its novelty, however, necessitates a new perspective on how to evaluate such a model. In our experiments, we assess the proposed model from various aspects, providing a new evaluation perspective for multi-instrument transcription. Our subjective listening study shows that Jointist achieves state-of-the-art performance on popular music, outperforming existing multi-instrument transcription models such as MT3. %We also argue that transcription models can be used as a preprocessing module for other music analysis tasks. We conducted experiments on several downstream tasks and found that the proposed method improved transcription by more than 1 percentage points (ppt.), source separation by 5 SDR, downbeat detection by 1.8 ppt., chord recognition by 1.4 ppt., and key estimation by 1.4 ppt., when utilizing transcription results obtained from Jointist.
【1】 Epic-Sounds: A Large-scale Dataset of Actions That Sound
标题：史诗-声音：一个大规模的行动数据集，听起来
链接：https://arxiv.org/abs/2302.00646
*与cs.SD语音【1】为同一篇
作者：Jaesung Huh,Jacob Chalk,Evangelos Kazakos,Dima Damen,Andrew Zisserman机构：Visual Geometry Group, Department of Engineering Science, University of Oxford, UK, Department of Computer Science, University of Bristol, UK摘要：我们介绍EPIC-SOUNDS，一个大规模的音频标注数据集，捕获自我中心视频音频流中的时间范围和类标签。我们提出了一个注释管道，其中注释器暂时标记可区分的音频段，并描述可能导致这种声音的动作。我们通过将这些自由形式的音频描述分组到类中，来识别可以纯粹从音频中区分出来的动作。对于涉及物体碰撞的动作，我们收集这些物体的材料的人类注释（例如，玻璃物体被放置在木制表面上），我们从视觉标签中验证这些注释，丢弃歧义。总体而言，EPIC-SOUNDS包括78.4k个可闻事件和动作的分类段，分布在44个类别以及39.2k个未分类段。我们在我们的数据集上训练和评估了两个最先进的音频识别模型，强调了纯音频标签的重要性和当前模型识别声音动作的局限性。摘要：We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos. We propose an annotation pipeline where annotators temporally label distinguishable audio segments and describe the action that could have caused this sound. We identify actions that can be discriminated purely from audio, through grouping these free-form descriptions of audio into classes. For actions that involve objects colliding, we collect human annotations of the materials of these objects (e.g. a glass object being placed on a wooden surface), which we verify from visual labels, discarding ambiguities. Overall, EPIC-SOUNDS includes 78.4k categorised segments of audible events and actions, distributed across 44 classes as well as 39.2k non-categorised segments. We train and evaluate two state-of-the-art audio recognition models on our dataset, highlighting the importance of audio-only labels and the limitations of current models to recognise actions that sound.
【2】 Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training
标题：Jointist：通过联合训练同步提高多乐器改编和乐源分离
链接：https://arxiv.org/abs/2302.00286
*与cs.SD语音【2】为同一篇
作者：Kin Wai Cheuk,Keunwoo Choi,Qiuqiang Kong,Bochen Li,Minz Won,Ju-Chiang Wang,Yun-Ning Hung Dorien Herremans机构：Yun-Ning Hung, Dorien Herremans, Singapore University of Technology and Design, Singapore, Gaudio Lab, South Korea, Bytedance, USA备注：arXiv admin note: text overlap with arXiv:2206.10805摘要：In this paper, we introduce Jointist, an instrument-aware multi-instrument framework that is capable of transcribing, recognizing, and separating multiple musical instruments from an audio clip. Jointist consists of an instrument recognition module that conditions the other two modules: a transcription module that outputs instrument-specific piano rolls, and a source separation module that utilizes instrument information and transcription results. The joint training of the transcription and source separation modules serves to improve the performance of both tasks. The instrument module is optional and can be directly controlled by human users. This makes Jointist a flexible user-controllable framework. Our challenging problem formulation makes the model highly useful in the real world given that modern popular music typically consists of multiple instruments. Its novelty, however, necessitates a new perspective on how to evaluate such a model. In our experiments, we assess the proposed model from various aspects, providing a new evaluation perspective for multi-instrument transcription. Our subjective listening study shows that Jointist achieves state-of-the-art performance on popular music, outperforming existing multi-instrument transcription models such as MT3. %We also argue that transcription models can be used as a preprocessing module for other music analysis tasks. We conducted experiments on several downstream tasks and found that the proposed method improved transcription by more than 1 percentage points (ppt.), source separation by 5 SDR, downbeat detection by 1.8 ppt., chord recognition by 1.4 ppt., and key estimation by 1.4 ppt., when utilizing transcription results obtained from Jointist.
机器翻译，仅供参考
每日语音速递