今日论文合集：cs.SD语音10篇，eess.AS音频处理10篇。【1】PosCUDA: Position based Convolution for Unlearnable Audio Datasets
标题：PosCUDA：基于位置的不可学习音频数据卷积链接：https://arxiv.org/abs/2401.02135
作者：Vignesh Gokul,Shlomo Dubnov摘要：深度学习模型需要大量干净的数据来实现良好的性能。为了避免昂贵的数据采集成本，研究人员使用互联网上丰富的数据。这引发了对未经授权滥用个人数据进行模型训练的潜在隐私问题。最近的工作，如CUDA提出了解决这个问题的方法，通过添加类模糊来使数据集不可学习，即模型永远不能使用获取的数据集进行学习。然而，这些方法往往会降低数据的质量，使其无法用于实际应用。我们介绍PosCUDA，一种基于位置的卷积，用于创建不可学习的音频数据集。PosCUDA在小块音频上使用类卷积。补丁的位置基于每个类的私钥，因此模型学习位置模糊和标签之间的关系，而不能泛化。我们的经验表明，PosCUDA可以实现不可学习性，同时保持原始音频数据集的质量。我们提出的方法对不同的音频特征表示（如MFCC，原始音频）和不同的架构（如Transformers，卷积网络等）也具有鲁棒性。摘要：Deep learning models require large amounts of clean data to acheive good performance. To avoid the cost of expensive data acquisition, researchers use the abundant data available on the internet. This raises significant privacy concerns on the potential misuse of personal data for model training without authorisation. Recent works such as CUDA propose solutions to this problem by adding class-wise blurs to make datasets unlearnable, i.e a model can never use the acquired dataset for learning. However these methods often reduce the quality of the data making it useless for practical applications. We introduce PosCUDA, a position based convolution for creating unlearnable audio datasets. PosCUDA uses class-wise convolutions on small patches of audio. The location of the patches are based on a private key for each class, hence the model learns the relations between positional blurs and labels, while failing to generalize. We empirically show that PosCUDA can achieve unlearnability while maintaining the quality of the original audio datasets. Our proposed method is also robust to different audio feature representations such as MFCC, raw audio and different architectures such as transformers, convolutional networks etc.
【2】 PEFT for Speech: Unveiling Optimal Placement, Merging Strategies, and Ensemble Techniques标题：用于演讲的PEFT：揭示最佳位置、合并策略和集成技术链接：https://arxiv.org/abs/2401.02122作者：Tzu-Han Lin,How-Shing Wang,Hao-Yung Weng,Kuang-Chen Peng,Zih-Ching Chen,Hung-yi Lee备注：Submitted to ICASSP 2024 Self-supervision in Audio, Speech and Beyond workshop摘要：参数有效的语音微调（PEFT）是一种有效的语音处理方法。然而，PEFT方法的最佳方法和位置仍然没有定论。我们的研究进行了广泛的实验，比较不同的PEFT方法和他们的逐层放置适应微分架构搜索（DARTS）。我们还探讨了使用集成学习来利用不同的PEFT策略。结果表明，DARTS并没有优于基线方法，该方法涉及将相同的PEFT方法插入到自监督学习（SSL）模型的所有层中。相比之下，集成学习方法，特别是采用多数表决的方法，表现出优越的性能。我们的统计证据表明，不同的PEFT方法以不同的方式学习。这种变化可以解释为什么通过集成学习的各种PEFT方法的协同集成可以比单独的逐层优化更有效地利用其独特的学习能力。摘要：Parameter-Efficient Fine-Tuning (PEFT) is increasingly recognized as an effective method in speech processing. However, the optimal approach and the placement of PEFT methods remain inconclusive. Our study conducts extensive experiments to compare different PEFT methods and their layer-wise placement adapting Differentiable Architecture Search (DARTS). We also explore the use of ensemble learning to leverage diverse PEFT strategies. The results reveal that DARTS does not outperform the baseline approach, which involves inserting the same PEFT method into all layers of a Self-Supervised Learning (SSL) model. In contrast, an ensemble learning approach, particularly one employing majority voting, demonstrates superior performance. Our statistical evidence indicates that different PEFT methods learn in varied ways. This variation might explain why the synergistic integration of various PEFT methods through ensemble learning can harness their unique learning capabilities more effectively compared to individual layer-wise optimization.【3】 CLAPP: Contrastive Language-Audio Pre-training in Passive Underwater Vessel Classification标题：CLAPP：被动式水下航行器分类中的对比听觉预训练链接：https://arxiv.org/abs/2401.02099作者：Zeyu Li,Jingsheng Gao,Tong Yu,Suncheng Xiang,Jiacheng Ruan,Ting Liu,Yuzhuo Fu摘要：现有的音频分类研究在识别被动水下船只场景的属性方面面临挑战，并且由于数据隐私问题而缺乏良好注释的数据集。在这项研究中，我们介绍了CLAPP（对比存储-音频预训练在被动水下船只分类），一种新的模型。我们的目标是训练一个神经网络使用广泛的船舶音频和船舶状态文本对从海洋数据集。CLAPP能够直接从原始船舶音频数据中学习，并且在可用时，从精心策划的标签中学习，从而能够在被动水下船舶场景中改进船舶属性的识别。模型的zero-shot功能允许预测给定血管音频的最相关血管状态描述，而无需直接优化任务。我们的方法旨在解决两个挑战：船舶音频文本分类和被动水下船舶音频属性识别。该方法在Deepship和Shipsear公共数据集上都取得了新的最先进的结果，与zero-shot任务的现有方法相比，准确率约为7%-13%。摘要：Existing research on audio classification faces challenges in recognizing attributes of passive underwater vessel scenarios and lacks well-annotated datasets due to data privacy concerns. In this study, we introduce CLAPP (Contrastive Language-Audio Pre-training in Passive Underwater Vessel Classification), a novel model. Our aim is to train a neural network using a wide range of vessel audio and vessel state text pairs obtained from an oceanship dataset. CLAPP is capable of directly learning from raw vessel audio data and, when available, from carefully curated labels, enabling improved recognition of vessel attributes in passive underwater vessel scenarios. Model's zero-shot capability allows predicting the most relevant vessel state description for a given vessel audio, without directly optimizing for the task. Our approach aims to solve 2 challenges: vessel audio-text classification and passive underwater vessel audio attribute recognition. The proposed method achieves new state-of-the-art results on both Deepship and Shipsear public datasets, with a notable margin of about 7%-13% for accuracy compared to prior methods on zero-shot task.【4】 Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations标题：利用非说话人表示增强零激发多说话人TTS链接：https://arxiv.org/abs/2401.02014作者：Yejin Jeon,Yunsu Kim,Gary Geunbae Lee摘要：Zero-shot多人TTS的目标是合成语音与选定的目标说话人的声音没有任何微调。然而，主流方法在适应域外设置的新扬声器方面遇到限制，主要是由于扬声器解缠不足和内容泄漏。为了克服这些限制，我们提出了一个创新的否定特征学习范式，模型解耦扬声器属性的偏差，从完整的音频表示，通过利用减法运算。通过消除多余的内容信息的扬声器表示，我们的否定方案不仅减轻内容泄漏，从而提高合成的鲁棒性，而且还提高了扬声器保真度。此外，为了便于学习不同的扬声器属性，我们利用多流Transformers，它保留了多个假设，并煽动类似于集成学习的训练范式。为了统一这些假设，实现最终的说话人表示，我们采用注意池。最后，鉴于在所需的语音中生成目标文本话语的必要性，我们采用自适应层规范化来有效地将先前生成的扬声器表示与目标文本表示相融合，而不是仅仅将文本和音频模态连接起来。广泛的实验和验证证实了我们提出的方法在保存和利用扬声器特定的属性相对于-`一个-相对于替代基线模型的有效性。摘要：Zero-shot multi-speaker TTS aims to synthesize speech with the voice of a chosen target speaker without any fine-tuning. Prevailing methods, however, encounter limitations at adapting to new speakers of out-of-domain settings, primarily due to inadequate speaker disentanglement and content leakage. To overcome these constraints, we propose an innovative negation feature learning paradigm that models decoupled speaker attributes as deviations from the complete audio representation by utilizing the subtraction operation. By eliminating superfluous content information from the speaker representation, our negation scheme not only mitigates content leakage, thereby enhancing synthesis robustness, but also improves speaker fidelity. In addition, to facilitate the learning of diverse speaker attributes, we leverage multi-stream Transformers, which retain multiple hypotheses and instigate a training paradigm akin to ensemble learning. To unify these hypotheses and realize the final speaker representation, we employ attention pooling. Finally, in light of the imperative to generate target text utterances in the desired voice, we adopt adaptive layer normalizations to effectively fuse the previously generated speaker representation with the target text representations, as opposed to mere concatenation of the text and audio modalities. Extensive experiments and validations substantiate the efficacy of our proposed approach in preserving and harnessing speaker-specific attributes vis-`a-vis alternative baseline models.【5】 Generating Rhythm Game Music with Jukebox标题：用Juventure生成节奏游戏音乐链接：https://arxiv.org/abs/2401.01997作者：Nicholas Yan摘要：音乐一直被认为是一种“人类”的努力-当赞美一首音乐时，我们强调作曲家的创造力和音乐所唤起的情感。由于音乐也严重依赖于以重复旋律主题和和弦进行的形式出现的模式和重复，人工智能越来越能够以类似人类的方式复制音乐。这项研究调查了开源商业神经网络Juventure准确复制节奏游戏中常见的两种音乐类型（artcore和orchestral）的能力。一个Google Colab笔记本提供了必要的计算资源来采样和扩展两种流派的总共16个钢琴编曲。一项包含选定样本的调查被分发给当地的一个青年管弦乐队，以衡量人们对人工智能和人类创作音乐的音乐性的看法。虽然人类更喜欢人类创作的音乐，但Jupiter略高的评分表明它在模仿两种风格方面都有一定的能力。尽管Juventure仅使用原始音频和相对较小的样本量，但它显示了人工智能作为音乐制作协作工具的未来前景。摘要：Music has always been thought of as a "human" endeavor -- when praising a piece of music, we emphasize the composer's creativity and the emotions the music invokes. Because music also heavily relies on patterns and repetition in the form of recurring melodic themes and chord progressions, artificial intelligence has increasingly been able to replicate music in a human-like fashion. This research investigated the capabilities of Jukebox, an open-source commercially available neural network, to accurately replicate two genres of music often found in rhythm games, artcore and orchestral. A Google Colab notebook provided the computational resources necessary to sample and extend a total of sixteen piano arrangements of both genres. A survey containing selected samples was distributed to a local youth orchestra to gauge people's perceptions of the musicality of AI and human-generated music. Even though humans preferred human-generated music, Jukebox's slightly high rating showed that it was somewhat capable at mimicking the styles of both genres. Despite limitations of Jukebox only using raw audio and a relatively small sample size, it shows promise for the future of AI as a collaborative tool in music production.【6】 Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition标题：面向任务的对话作为自监督自动语音识别的催化剂链接：https://arxiv.org/abs/2401.02417作者：David M. Chan,Shalini Ghosh,Hitesh Tulsiani,Ariya Rastrow,Björn Hoffmeister备注：To appear in ICASSP 2024摘要：虽然自动语音识别（ASR）系统的单词错误率一直在下降，但建立在ASR系统之上的自然语言理解（NLU）应用程序仍然将大量失败归因于低质量的语音识别结果。现有的辅助系统收集大量这些不成功的交互，但是这些系统通常无法从这些交互中学习，即使是以离线方式。在这项工作中，我们介绍了CLC：对话的对比学习，这是一系列以自我监督的方式对模型进行对比微调的方法，利用了与助手不成功对话中容易检测到的工件。我们证明了我们的CLC系列方法可以将ASR模型在OD3上的性能提高高达19.2%，OD3是一个新的公共大规模半合成音频任务导向对话元数据集。这些收益也转移到现实世界的系统中，我们表明CLC可以帮助将性能提高6.7%。我们在https://github.com/amazon-science/amazon-od3上公开提供OD3。摘要：While word error rates of automatic speech recognition (ASR) systems have consistently fallen, natural language understanding (NLU) applications built on top of ASR systems still attribute significant numbers of failures to low-quality speech recognition results. Existing assistant systems collect large numbers of these unsuccessful interactions, but these systems usually fail to learn from these interactions, even in an offline fashion. In this work, we introduce CLC: Contrastive Learning for Conversations, a family of methods for contrastive fine-tuning of models in a self-supervised fashion, making use of easily detectable artifacts in unsuccessful conversations with assistants. We demonstrate that our CLC family of approaches can improve the performance of ASR models on OD3, a new public large-scale semi-synthetic meta-dataset of audio task-oriented dialogues, by up to 19.2%. These gains transfer to real-world systems as well, where we show that CLC can help to improve performance by up to 6.7% over baselines. We make OD3 publicly available at https://github.com/amazon-science/amazon-od3 .【7】 Direction of Arrival Estimation Using Microphone Array Processing for Moving Humanoid Robots标题：基于麦克风阵列处理的运动仿人机器人波达方向估计链接：https://arxiv.org/abs/2401.02386作者：Vladimir Tourbabin,Boaz Rafaely备注：None摘要：类人机器人的听觉系统近年来受到越来越多的关注。该系统通常借助于麦克风阵列来获取周围声场。然后使用各种方法处理由阵列获取的信号。其中一种广泛应用的方法是波达方向估计。传统的波达方向估计方法假设在估计过程中阵列固定在给定位置。然而，这对于安装在移动的人形机器人上的阵列不一定是正确的。如果不适当地考虑阵列运动，则可能在估计的到达方向中引入显著误差。目前的论文提出了一个信号模型，考虑到运动。基于该模型，提出了两种处理方法。第一个补偿机器人的运动。第二种方法适用于周期性信号，并利用运动，以提高性能的水平超过了一个固定的阵列。数值模拟和实验研究表明，运动补偿方法几乎消除了运动相关的误差。它还表明，通过使用基于运动的增强方法，它是可能的，以提高到达方向估计性能相比，当使用一个固定阵列。摘要：The auditory system of humanoid robots has gained increased attention in recent years. This system typically acquires the surrounding sound field by means of a microphone array. Signals acquired by the array are then processed using various methods. One of the widely applied methods is direction of arrival estimation. The conventional direction of arrival estimation methods assume that the array is fixed at a given position during the estimation. However, this is not necessarily true for an array installed on a moving humanoid robot. The array motion, if not accounted for appropriately, can introduce a significant error in the estimated direction of arrival. The current paper presents a signal model that takes the motion into account. Based on this model, two processing methods are proposed. The first one compensates for the motion of the robot. The second method is applicable to periodic signals and utilizes the motion in order to enhance the performance to a level beyond that of a stationary array. Numerical simulations and an experimental study are provided, demonstrating that the motion compensation method almost eliminates the motion-related error. It is also demonstrated that by using the motion-based enhancement method it is possible to improve the direction of arrival estimation performance, as compared to that obtained when using a stationary array.【8】 Optimal Real-Weighted Beamforming With Application to Linear and Spherical Arrays标题：最优实加权波束形成及其在线阵和球阵中的应用链接：https://arxiv.org/abs/2401.02285作者：V. Tourbabin,M. Agmon,B. Rafaely,J. Tabrikian备注：None摘要：传感器阵列的用途之一是用于空间滤波或波束形成。当前的数字信号处理方法促进了复加权波束形成，提供了阵列设计的灵活性。先前的研究提出了使用实值波束成形权重，虽然这降低了设计的灵活性，但可以提供一系列益处，例如，简化的波束形成器实现或有效的波束形成算法。本文提出了一种新的实值加权阵设计方法，使阵列方向性最大，并给出了加权阵的封闭解。线性和球形阵列的方法进行了研究，它表明，刚性球形阵列是特别适合于实重量的设计，因为它们不遭受栅瓣，在线性阵列与实重量的主导功能。线性和球形阵列的仿真研究，随着实验研究，验证理论的发展。摘要：One of the uses of sensor arrays is for spatial filtering or beamforming. Current digital signal processing methods facilitate complex-weighted beamforming, providing flexibility in array design. Previous studies proposed the use of real-valued beamforming weights, which although reduce flexibility in design, may provide a range of benefits, e.g., simplified beamformer implementation or efficient beamforming algorithms. This paper presents a new method for the design of arrays with real-valued weights, that achieve maximum directivity, providing closed-form solution to array weights. The method is studied for linear and spherical arrays, where it is shown that rigid spherical arrays are particularly suitable for real-weight designs as they do not suffer from grating lobes, a dominant feature in linear arrays with real weights. A simulation study is presented for linear and spherical arrays, along with an experimental investigation, validating the theoretical developments.【9】 Listening broadband physical model for microphones: a first step标题：麦克风收听宽带物理模型：第一步链接：https://arxiv.org/abs/2401.02164作者：Laurent Millot,Antoine Valette,Manuel Lopes,Gérard Pelé,Mohammed Elliq,Dominique Lambert备注：None摘要：我们将提出一个宽带物理模型的麦克风设计的第一步。在所提出的模型中，经典的方向性图案（全向，双向和心形家庭）被重新发现的极限情况下：单色激发，低频和远场近似。单声道音乐片段被用作模型的源，因此我们可以实时收听相关录制声场的模拟，这要归功于Max/MSP应用程序。听觉和子带分析表明，指向性是频率子带和声源位置的函数。该模型还表现出有趣的邻近效应。将提供音频演示。论文6638，在第120届音频工程学会大会上发表，巴黎，2006年摘要：We will present a first step in design of a broadband physical model for microphones. Within the proposed model, classical directivity patterns (omnidirectional, bidirectional and cardioids family) are refound as limit cases: monochromatic excitation, low frequency and far-field approximation. Monophonic pieces of music are used as sources for the model so we can listen the simulation of the associated recorded sound field in realtime thanks to a Max/MSP application. Listening and subbands analysis show that the directivity is a function of frequential subband and source location. This model also exhibits an interesting proximity effect. Audio demonstrations will be given.Paper 6638 presented at the 120th Convention of the Audio Engineering Society, Paris, 2006【10】 CTC Blank Triggered Dynamic Layer-Skipping for Efficient CTC-based Speech Recognition标题：基于CTC空白触发的动态跳层语音识别链接：https://arxiv.org/abs/2401.02046作者：Junfeng Hou,Peiyao Wang,Jincheng Zhang,Meng Yang,Minwei Feng,Jingcheng Yin备注：accepted by ASRU 2023摘要：尽管语音识别模型的性能令人印象深刻，但在有限的计算资源下部署端到端语音识别模型仍然具有挑战性。考虑到模型规模的逐渐增加和模型应用的广泛性，针对不同的输入选择性地执行模型组件以提高推理效率是非常有意义的。在本文中，我们提出了一个动态的跳层方法，利用CTC空白输出从中间层触发跳过最后几个编码器层的帧具有高空白概率。此外，我们因式分解CTC输出分布，并在中间层进行知识蒸馏，以减少计算量，提高识别精度。实验结果表明，通过利用CTC空白，编码器层深度可以动态调整，导致CTC模型推理速度提高29%，性能略有下降。摘要：Deploying end-to-end speech recognition models with limited computing resources remains challenging, despite their impressive performance. Given the gradual increase in model size and the wide range of model applications, selectively executing model components for different inputs to improve the inference efficiency is of great interest. In this paper, we propose a dynamic layer-skipping method that leverages the CTC blank output from intermediate layers to trigger the skipping of the last few encoder layers for frames with high blank probabilities. Furthermore, we factorize the CTC output distribution and perform knowledge distillation on intermediate layers to reduce computation and improve recognition accuracy. Experimental results show that by utilizing the CTC blank, the encoder layer depth can be adjusted dynamically, resulting in 29% acceleration of the CTC model inference with minor performance degradation.【1】 Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognition标题：面向任务的对话作为自监督自动语音识别的催化剂链接：https://arxiv.org/abs/2401.02417作者：David M. Chan,Shalini Ghosh,Hitesh Tulsiani,Ariya Rastrow,Björn Hoffmeister备注：To appear in ICASSP 2024摘要：None摘要：While word error rates of automatic speech recognition (ASR) systems have consistently fallen, natural language understanding (NLU) applications built on top of ASR systems still attribute significant numbers of failures to low-quality speech recognition results. Existing assistant systems collect large numbers of these unsuccessful interactions, but these systems usually fail to learn from these interactions, even in an offline fashion. In this work, we introduce CLC: Contrastive Learning for Conversations, a family of methods for contrastive fine-tuning of models in a self-supervised fashion, making use of easily detectable artifacts in unsuccessful conversations with assistants. We demonstrate that our CLC family of approaches can improve the performance of ASR models on OD3, a new public large-scale semi-synthetic meta-dataset of audio task-oriented dialogues, by up to 19.2%. These gains transfer to real-world systems as well, where we show that CLC can help to improve performance by up to 6.7% over baselines. We make OD3 publicly available at https://github.com/amazon-science/amazon-od3 .【2】 Direction of Arrival Estimation Using Microphone Array Processing for Moving Humanoid Robots标题：基于麦克风阵列处理的运动仿人机器人波达方向估计链接：https://arxiv.org/abs/2401.02386作者：Vladimir Tourbabin,Boaz Rafaely备注：None摘要：类人机器人的听觉系统近年来受到越来越多的关注。该系统通常借助于麦克风阵列来获取周围声场。然后使用各种方法处理由阵列获取的信号。其中一种广泛应用的方法是波达方向估计。传统的波达方向估计方法假设在估计过程中阵列固定在给定位置。然而，这对于安装在移动的人形机器人上的阵列不一定是正确的。如果不适当地考虑阵列运动，则可能在估计的到达方向中引入显著误差。目前的论文提出了一个信号模型，考虑到运动。基于该模型，提出了两种处理方法。第一个补偿机器人的运动。第二种方法适用于周期性信号，并利用运动，以提高性能的水平超过了一个固定的阵列。数值模拟和实验研究表明，运动补偿方法几乎消除了运动相关的误差。它还表明，通过使用基于运动的增强方法，它是可能的，以提高到达方向估计性能相比，当使用一个固定阵列。摘要：The auditory system of humanoid robots has gained increased attention in recent years. This system typically acquires the surrounding sound field by means of a microphone array. Signals acquired by the array are then processed using various methods. One of the widely applied methods is direction of arrival estimation. The conventional direction of arrival estimation methods assume that the array is fixed at a given position during the estimation. However, this is not necessarily true for an array installed on a moving humanoid robot. The array motion, if not accounted for appropriately, can introduce a significant error in the estimated direction of arrival. The current paper presents a signal model that takes the motion into account. Based on this model, two processing methods are proposed. The first one compensates for the motion of the robot. The second method is applicable to periodic signals and utilizes the motion in order to enhance the performance to a level beyond that of a stationary array. Numerical simulations and an experimental study are provided, demonstrating that the motion compensation method almost eliminates the motion-related error. It is also demonstrated that by using the motion-based enhancement method it is possible to improve the direction of arrival estimation performance, as compared to that obtained when using a stationary array.【3】 Optimal Real-Weighted Beamforming With Application to Linear and Spherical Arrays标题：最优实加权波束形成及其在线阵和球阵中的应用链接：https://arxiv.org/abs/2401.02285作者：V. Tourbabin,M. Agmon,B. Rafaely,J. Tabrikian备注：None摘要：传感器阵列的用途之一是用于空间滤波或波束形成。当前的数字信号处理方法促进了复加权波束形成，提供了阵列设计的灵活性。先前的研究提出了使用实值波束成形权重，虽然这降低了设计的灵活性，但可以提供一系列益处，例如，简化的波束形成器实现或有效的波束形成算法。本文提出了一种新的实值加权阵设计方法，使阵列方向性最大，并给出了加权阵的封闭解。线性和球形阵列的方法进行了研究，它表明，刚性球形阵列是特别适合于实重量的设计，因为它们不遭受栅瓣，在线性阵列与实重量的主导功能。线性和球形阵列的仿真研究，随着实验研究，验证理论的发展。摘要：One of the uses of sensor arrays is for spatial filtering or beamforming. Current digital signal processing methods facilitate complex-weighted beamforming, providing flexibility in array design. Previous studies proposed the use of real-valued beamforming weights, which although reduce flexibility in design, may provide a range of benefits, e.g., simplified beamformer implementation or efficient beamforming algorithms. This paper presents a new method for the design of arrays with real-valued weights, that achieve maximum directivity, providing closed-form solution to array weights. The method is studied for linear and spherical arrays, where it is shown that rigid spherical arrays are particularly suitable for real-weight designs as they do not suffer from grating lobes, a dominant feature in linear arrays with real weights. A simulation study is presented for linear and spherical arrays, along with an experimental investigation, validating the theoretical developments.【4】 Listening broadband physical model for microphones: a first step标题：麦克风收听宽带物理模型：第一步链接：https://arxiv.org/abs/2401.02164作者：Laurent Millot,Antoine Valette,Manuel Lopes,Gérard Pelé,Mohammed Elliq,Dominique Lambert备注：None摘要：None摘要：We will present a first step in design of a broadband physical model for microphones. Within the proposed model, classical directivity patterns (omnidirectional, bidirectional and cardioids family) are refound as limit cases: monochromatic excitation, low frequency and far-field approximation. Monophonic pieces of music are used as sources for the model so we can listen the simulation of the associated recorded sound field in realtime thanks to a Max/MSP application. Listening and subbands analysis show that the directivity is a function of frequential subband and source location. This model also exhibits an interesting proximity effect. Audio demonstrations will be given.Paper 6638 presented at the 120th Convention of the Audio Engineering Society, Paris, 2006【5】 CTC Blank Triggered Dynamic Layer-Skipping for Efficient CTC-based Speech Recognition标题：用于高效CTC语音识别的CTC空白触发动态跳层链接：https://arxiv.org/abs/2401.02046作者：Junfeng Hou,Peiyao Wang,Jincheng Zhang,Meng Yang,Minwei Feng,Jingcheng Yin备注：accepted by ASRU 2023摘要：None摘要：Deploying end-to-end speech recognition models with limited computing resources remains challenging, despite their impressive performance. Given the gradual increase in model size and the wide range of model applications, selectively executing model components for different inputs to improve the inference efficiency is of great interest. In this paper, we propose a dynamic layer-skipping method that leverages the CTC blank output from intermediate layers to trigger the skipping of the last few encoder layers for frames with high blank probabilities. Furthermore, we factorize the CTC output distribution and perform knowledge distillation on intermediate layers to reduce computation and improve recognition accuracy. Experimental results show that by utilizing the CTC blank, the encoder layer depth can be adjusted dynamically, resulting in 29% acceleration of the CTC model inference with minor performance degradation.【6】 PosCUDA: Position based Convolution for Unlearnable Audio Datasets标题：PosCUDA：基于位置的卷积用于不可学习的音频数据集链接：https://arxiv.org/abs/2401.02135作者：Vignesh Gokul,Shlomo Dubnov摘要：深度学习模型需要大量干净的数据来实现良好的性能。为了避免昂贵的数据采集成本，研究人员使用互联网上丰富的数据。这引发了对未经授权滥用个人数据进行模型训练的潜在隐私问题。最近的工作，如CUDA提出了解决这个问题的方法，通过添加类模糊来使数据集不可学习，即模型永远不能使用获取的数据集进行学习。然而，这些方法往往会降低数据的质量，使其无法用于实际应用。我们介绍PosCUDA，一种基于位置的卷积，用于创建不可学习的音频数据集。PosCUDA在小块音频上使用类卷积。补丁的位置基于每个类的私钥，因此模型学习位置模糊和标签之间的关系，而不能泛化。我们的经验表明，PosCUDA可以实现不可学习性，同时保持原始音频数据集的质量。我们提出的方法对不同的音频特征表示（如MFCC，原始音频）和不同的架构（如Transformers，卷积网络等）也具有鲁棒性。摘要：Deep learning models require large amounts of clean data to acheive good performance. To avoid the cost of expensive data acquisition, researchers use the abundant data available on the internet. This raises significant privacy concerns on the potential misuse of personal data for model training without authorisation. Recent works such as CUDA propose solutions to this problem by adding class-wise blurs to make datasets unlearnable, i.e a model can never use the acquired dataset for learning. However these methods often reduce the quality of the data making it useless for practical applications. We introduce PosCUDA, a position based convolution for creating unlearnable audio datasets. PosCUDA uses class-wise convolutions on small patches of audio. The location of the patches are based on a private key for each class, hence the model learns the relations between positional blurs and labels, while failing to generalize. We empirically show that PosCUDA can achieve unlearnability while maintaining the quality of the original audio datasets. Our proposed method is also robust to different audio feature representations such as MFCC, raw audio and different architectures such as transformers, convolutional networks etc.【7】 PEFT for Speech: Unveiling Optimal Placement, Merging Strategies, and Ensemble Techniques标题：语音PEFT：揭示最佳放置，合并策略和包围技术链接：https://arxiv.org/abs/2401.02122作者：Tzu-Han Lin,How-Shing Wang,Hao-Yung Weng,Kuang-Chen Peng,Zih-Ching Chen,Hung-yi Lee备注：Submitted to ICASSP 2024 Self-supervision in Audio, Speech and Beyond workshop摘要：参数有效的语音微调（PEFT）是一种有效的语音处理方法。然而，PEFT方法的最佳方法和位置仍然没有定论。我们的研究进行了广泛的实验，比较不同的PEFT方法和他们的逐层放置适应微分架构搜索（DARTS）。我们还探讨了使用集成学习来利用不同的PEFT策略。结果表明，DARTS并没有优于基线方法，该方法涉及将相同的PEFT方法插入到自监督学习（SSL）模型的所有层中。相比之下，集成学习方法，特别是采用多数表决的方法，表现出优越的性能。我们的统计证据表明，不同的PEFT方法以不同的方式学习。这种变化可以解释为什么通过集成学习的各种PEFT方法的协同集成可以比单独的逐层优化更有效地利用其独特的学习能力。摘要：Parameter-Efficient Fine-Tuning (PEFT) is increasingly recognized as an effective method in speech processing. However, the optimal approach and the placement of PEFT methods remain inconclusive. Our study conducts extensive experiments to compare different PEFT methods and their layer-wise placement adapting Differentiable Architecture Search (DARTS). We also explore the use of ensemble learning to leverage diverse PEFT strategies. The results reveal that DARTS does not outperform the baseline approach, which involves inserting the same PEFT method into all layers of a Self-Supervised Learning (SSL) model. In contrast, an ensemble learning approach, particularly one employing majority voting, demonstrates superior performance. Our statistical evidence indicates that different PEFT methods learn in varied ways. This variation might explain why the synergistic integration of various PEFT methods through ensemble learning can harness their unique learning capabilities more effectively compared to individual layer-wise optimization.【8】 CLAPP: Contrastive Language-Audio Pre-training in Passive Underwater Vessel Classification标题：CLAPP：被动水下舰船分类中的对比语言--音频预训练链接：https://arxiv.org/abs/2401.02099作者：Zeyu Li,Jingsheng Gao,Tong Yu,Suncheng Xiang,Jiacheng Ruan,Ting Liu,Yuzhuo Fu摘要：现有的音频分类研究在识别被动水下船只场景的属性方面面临挑战，并且由于数据隐私问题而缺乏良好注释的数据集。在这项研究中，我们介绍了CLAPP（对比存储-音频预训练在被动水下船只分类），一种新的模型。我们的目标是训练一个神经网络使用广泛的船舶音频和船舶状态文本对从海洋数据集。CLAPP能够直接从原始船舶音频数据中学习，并且在可用时，从精心策划的标签中学习，从而能够在被动水下船舶场景中改进船舶属性的识别。模型的zero-shot功能允许预测给定血管音频的最相关血管状态描述，而无需直接优化任务。我们的方法旨在解决两个挑战：船舶音频文本分类和被动水下船舶音频属性识别。该方法在Deepship和Shipsear公共数据集上都取得了新的最先进的结果，与zero-shot任务的现有方法相比，准确率约为7%-13%。摘要：Existing research on audio classification faces challenges in recognizing attributes of passive underwater vessel scenarios and lacks well-annotated datasets due to data privacy concerns. In this study, we introduce CLAPP (Contrastive Language-Audio Pre-training in Passive Underwater Vessel Classification), a novel model. Our aim is to train a neural network using a wide range of vessel audio and vessel state text pairs obtained from an oceanship dataset. CLAPP is capable of directly learning from raw vessel audio data and, when available, from carefully curated labels, enabling improved recognition of vessel attributes in passive underwater vessel scenarios. Model's zero-shot capability allows predicting the most relevant vessel state description for a given vessel audio, without directly optimizing for the task. Our approach aims to solve 2 challenges: vessel audio-text classification and passive underwater vessel audio attribute recognition. The proposed method achieves new state-of-the-art results on both Deepship and Shipsear public datasets, with a notable margin of about 7%-13% for accuracy compared to prior methods on zero-shot task.【9】 Enhancing Zero-Shot Multi-Speaker TTS with Negated Speaker Representations标题：利用非说话人表示增强零激发多说话人TTS链接：https://arxiv.org/abs/2401.02014作者：Yejin Jeon,Yunsu Kim,Gary Geunbae Lee摘要：Zero-shot多人TTS的目标是合成语音与选定的目标说话人的声音没有任何微调。然而，主流方法在适应域外设置的新扬声器方面遇到限制，主要是由于扬声器解缠不足和内容泄漏。为了克服这些限制，我们提出了一个创新的否定特征学习范式，模型解耦扬声器属性的偏差，从完整的音频表示，通过利用减法运算。通过消除多余的内容信息的扬声器表示，我们的否定方案不仅减轻内容泄漏，从而提高合成的鲁棒性，而且还提高了扬声器保真度。此外，为了便于学习不同的扬声器属性，我们利用多流Transformers，它保留了多个假设，并煽动类似于集成学习的训练范式。为了统一这些假设，实现最终的说话人表示，我们采用注意池。最后，鉴于在所需的语音中生成目标文本话语的必要性，我们采用自适应层规范化来有效地将先前生成的扬声器表示与目标文本表示相融合，而不是仅仅将文本和音频模态连接起来。广泛的实验和验证证实了我们提出的方法在保存和利用扬声器特定的属性相对于-`一个-相对于替代基线模型的有效性。摘要：Zero-shot multi-speaker TTS aims to synthesize speech with the voice of a chosen target speaker without any fine-tuning. Prevailing methods, however, encounter limitations at adapting to new speakers of out-of-domain settings, primarily due to inadequate speaker disentanglement and content leakage. To overcome these constraints, we propose an innovative negation feature learning paradigm that models decoupled speaker attributes as deviations from the complete audio representation by utilizing the subtraction operation. By eliminating superfluous content information from the speaker representation, our negation scheme not only mitigates content leakage, thereby enhancing synthesis robustness, but also improves speaker fidelity. In addition, to facilitate the learning of diverse speaker attributes, we leverage multi-stream Transformers, which retain multiple hypotheses and instigate a training paradigm akin to ensemble learning. To unify these hypotheses and realize the final speaker representation, we employ attention pooling. Finally, in light of the imperative to generate target text utterances in the desired voice, we adopt adaptive layer normalizations to effectively fuse the previously generated speaker representation with the target text representations, as opposed to mere concatenation of the text and audio modalities. Extensive experiments and validations substantiate the efficacy of our proposed approach in preserving and harnessing speaker-specific attributes vis-`a-vis alternative baseline models.【10】 Generating Rhythm Game Music with Jukebox标题：用自动点唱机生成节奏游戏音乐链接：https://arxiv.org/abs/2401.01997作者：Nicholas Yan摘要：音乐一直被认为是一种“人类”的努力-当赞美一首音乐时，我们强调作曲家的创造力和音乐所唤起的情感。由于音乐也严重依赖于以重复旋律主题和和弦进行的形式出现的模式和重复，人工智能越来越能够以类似人类的方式复制音乐。这项研究调查了开源商业神经网络Juventure准确复制节奏游戏中常见的两种音乐类型（artcore和orchestral）的能力。一个Google Colab笔记本提供了必要的计算资源来采样和扩展两种流派的总共16个钢琴编曲。一项包含选定样本的调查被分发给当地的一个青年管弦乐队，以衡量人们对人工智能和人类创作音乐的音乐性的看法。虽然人类更喜欢人类创作的音乐，但Jupiter略高的评分表明它在模仿两种风格方面都有一定的能力。尽管Juventure仅使用原始音频和相对较小的样本量，但它显示了人工智能作为音乐制作协作工具的未来前景。摘要：Music has always been thought of as a "human" endeavor -- when praising a piece of music, we emphasize the composer's creativity and the emotions the music invokes. Because music also heavily relies on patterns and repetition in the form of recurring melodic themes and chord progressions, artificial intelligence has increasingly been able to replicate music in a human-like fashion. This research investigated the capabilities of Jukebox, an open-source commercially available neural network, to accurately replicate two genres of music often found in rhythm games, artcore and orchestral. A Google Colab notebook provided the computational resources necessary to sample and extend a total of sixteen piano arrangements of both genres. A survey containing selected samples was distributed to a local youth orchestra to gauge people's perceptions of the musicality of AI and human-generated music. Even though humans preferred human-generated music, Jukebox's slightly high rating showed that it was somewhat capable at mimicking the styles of both genres. Despite limitations of Jukebox only using raw audio and a relatively small sample size, it shows promise for the future of AI as a collaborative tool in music production. 
每日语音速递