今日论文合集：cs.SD语音8篇，eess.AS音频处理10篇。
cs.SD语音
【1】 A Foundation Model for Music Informatics标题：音乐信息学的基础模型链接：https://arxiv.org/abs/2311.03318
作者：Minz Won,Yun-Ning Hung,Duc Le备注：5 pages摘要：本文研究了为音乐信息学量身定制的基础模型，这是一个目前面临标签数据稀缺和泛化问题的领域。为此，我们对各种基础模型变体进行了深入的比较研究，研究了模型架构、标记化方法、时间分辨率、数据和模型可扩展性等关键决定因素。本研究旨在通过阐明这些个体因素如何有助于音乐信息学基础模型的成功来弥合现有的知识差距。采用一个仔细的评估框架，我们评估这些模型在音乐信息检索的不同下游任务的性能，特别关注标记级和序列级分类。我们的研究结果表明，我们的模型表现出强大的性能，在特定的关键指标超过现有的模型。这些发现有助于理解音乐信息学中的自我监督学习，并为在该领域开发更有效和通用的基础模型铺平了道路。我们的模型的预训练版本是公开的，以促进可重复性和未来的研究。摘要：This paper investigates foundation models tailored for music informatics, a domain currently challenged by the scarcity of labeled data and generalization issues. To this end, we conduct an in-depth comparative study among various foundation model variants, examining key determinants such as model architectures, tokenization methods, temporal resolution, data, and model scalability. This research aims to bridge the existing knowledge gap by elucidating how these individual factors contribute to the success of foundation models in music informatics. Employing a careful evaluation framework, we assess the performance of these models across diverse downstream tasks in music information retrieval, with a particular focus on token-level and sequence-level classification. Our results reveal that our model demonstrates robust performance, surpassing existing models in specific key metrics. These findings contribute to the understanding of self-supervised learning in music informatics and pave the way for developing more effective and versatile foundation models in the field. A pretrained version of our model is publicly available to foster reproducibility and future research.
【2】 Attention or Convolution: Transformer Encoders in Audio Language Models  for Inference Efficiency标题：注意还是卷积：音频语言模型中用于提高推理效率的变换编码器链接：https://arxiv.org/abs/2311.02772
作者：Sungho Jeon,Ching-Feng Yeh,Hakan Inan,Wei-Ning Hsu,Rashi Rungta,Yashar Mehdad,Daniel Bikel备注：5 pages摘要：在本文中，我们表明，一个简单的自我监督的预训练的音频模型可以实现比较复杂的预训练模型与语音Transformer编码器的推理效率。这些语音Transformers依赖于将卷积模块与自注意模块混合。它们以最高的效率在ASR上实现了最先进的性能。我们首先表明，采用这些语音Transformers作为编码器也显着提高了预训练音频模型的效率。然而，我们的研究表明，我们可以单独使用高级自我注意力来实现相当的效率。我们证明，这种更简单的方法是特别有益的低比特权重量化技术的神经网络，以提高效率。我们假设，它可以防止传播不同的量化模块之间的错误相比，最近的语音Transformers混合量化卷积和量化的自我注意模块。摘要：In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules.
【3】 AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency  for Video Deepfake Detection标题：AV-Lip-Sync+：利用AV-Hubert利用多模式不一致性进行视频深伪检测链接：https://arxiv.org/abs/2311.02733
作者：Sahibzada Adil Shahzad,Ammarah Hashmi,Yan-Tsung Peng,Yu Tsao,Hsin-Min Wang摘要：多模态操作（也称为视听深度伪造）使得单模态深度伪造检测器难以检测多媒体内容中的伪造。要避免虚假宣传和假新闻的传播，及时发现至关重要。对任一模态的损害（即，视觉或音频）只能通过可以同时利用这两条信息的多模态模型来发现。以往的方法主要采用单模态视频取证和使用监督预训练的伪造检测。提出了一种基于多模态自监督学习（SSL）特征提取器的多模态视频伪造检测新方法。我们使用基于transformer的SSL预训练的Audio-Visual HuBERT（AV-HuBERT）模型作为视觉和声学特征提取器和多尺度时间卷积神经网络来捕获音频和视觉模态之间的时间相关性。由于AV-HuBERT仅从嘴唇区域提取视觉特征，因此我们还采用了另一种基于变换的视频模型来利用面部特征并捕获在deepfake生成过程中引起的空间和时间伪影。实验结果表明，我们的模型优于所有现有的模型，并在FakeAVCeleb和DeepfakeTIMIT数据集上实现了新的最先进的性能。摘要：Multimodal manipulations (also known as audio-visual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. To avoid the spread of false propaganda and fake news, timely detection is crucial. The damage to either modality (i.e., visual or audio) can only be discovered through multi-modal models that can exploit both pieces of information simultaneously. Previous methods mainly adopt uni-modal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multi-modal self-supervised-learning (SSL) feature extractor to exploit inconsistency between audio and visual modalities for multi-modal video forgery detection. We use the transformer-based SSL pre-trained Audio-Visual HuBERT (AV-HuBERT) model as a visual and acoustic feature extractor and a multi-scale temporal convolutional neural network to capture the temporal correlation between the audio and visual modalities. Since AV-HuBERT only extracts visual features from the lip region, we also adopt another transformer-based video model to exploit facial features and capture spatial and temporal artifacts caused during the deepfake generation process. Experimental results show that our model outperforms all existing models and achieves new state-of-the-art performance on the FakeAVCeleb and DeepfakeTIMIT datasets.
【4】 Yet Another Generative Model For Room Impulse Response Estimation标题：房间脉冲响应估计的另一种产生式模型链接：https://arxiv.org/abs/2311.02581
作者：Sungho Lee,Hyeong-Seok Choi,Kyogu Lee备注：WASPAA 2023摘要：最近的神经房间脉冲响应（RIR）估计器通常包括用于参考音频分析的编码器和用于RIR合成的生成器。特别是，它是发生器的性能，直接影响整体估计质量。在这种情况下，我们探索一种替代发电机架构，以提高性能。我们首先训练具有残差量化的自动编码器以学习离散潜在令牌空间，其中每个令牌表示RIR的小时频补丁。然后，我们将RIR估计问题转换为参考条件自回归令牌生成任务，采用跨频率，时间和量化深度轴操作的Transformer变体。通过这种方式，我们解决了标准的盲估计任务和额外的声学匹配问题，其目的是找到一个RIR，匹配源信号的目标信号的混响特性。实验结果表明，我们的系统是优于其他基线在各种评估指标。摘要：Recent neural room impulse response (RIR) estimators typically comprise an encoder for reference audio analysis and a generator for RIR synthesis. Especially, it is the performance of the generator that directly influences the overall estimation quality. In this context, we explore an alternate generator architecture for improved performance. We first train an autoencoder with residual quantization to learn a discrete latent token space, where each token represents a small time-frequency patch of the RIR. Then, we cast the RIR estimation problem as a reference-conditioned autoregressive token generation task, employing transformer variants that operate across frequency, time, and quantization depth axes. This way, we address the standard blind estimation task and additional acoustic matching problem, which aims to find an RIR that matches the source signal to the target signal's reverberation characteristics. Experimental results show that our system is preferable to other baselines across various evaluation metrics.
【5】 Generalized zero-shot audio-to-intent classification标题：广义零放炮音频到意图分类链接：https://arxiv.org/abs/2311.02482
作者：Veera Raghavendra Elluru,Devang Kulshreshtha,Rohit Paturi,Sravan Bodapati,Srikanth Ronanki摘要：使用纯音频数据的口语理解系统越来越受欢迎，但它们处理不可见意图的能力仍然有限。在这项研究中，我们提出了一个广义的zero-shot音频到意图的分类框架，每个意图只有几个样本文本句子。为了实现这一点，我们首先训练一个监督的音频意图分类器，利用自我监督的预训练模型。然后，我们利用神经音频合成器来创建样本文本话语的音频嵌入，并使用余弦相似性对看不见的意图执行广义zero-shot分类。我们还提出了一种多模态训练策略，将词汇信息纳入音频表示，以提高zero-shot性能。与仅音频训练相比，我们的多模态训练方法分别将SLURP和内部目标导向对话数据集的SLURP看不见意图的zero-shot意图分类的准确性提高了2.75%和18.2%。摘要：Spoken language understanding systems using audio-only data are gaining popularity, yet their ability to handle unseen intents remains limited. In this study, we propose a generalized zero-shot audio-to-intent classification framework with only a few sample text sentences per intent. To achieve this, we first train a supervised audio-to-intent classifier by making use of a self-supervised pre-trained model. We then leverage a neural audio synthesizer to create audio embeddings for sample text utterances and perform generalized zero-shot classification on unseen intents using cosine similarity. We also propose a multimodal training strategy that incorporates lexical information into the audio representation to improve zero-shot performance. Our multimodal training approach improves the accuracy of zero-shot intent classification on unseen intents of SLURP by 2.75% and 18.2% for the SLURP and internal goal-oriented dialog datasets, respectively, compared to audio-only training.
【6】 TACNET: Temporal Audio Source Counting Network标题：TACNET：时间音源计数网络链接：https://arxiv.org/abs/2311.02369
作者：Amirreza Ahmadnejad,Ahmad Mahmmodian Darviishani,Mohmmad Mehrdad Asadi,Sajjad Saffariyeh,Pedram Yousef,Emad Fatemizadeh摘要：在本文中，我们介绍了时间音频源计数网络（TaCNet），一个创新的架构，解决了音频源计数任务的限制。TaCNet直接对原始音频输入进行操作，消除了复杂的预处理步骤并简化了工作流程。值得注意的是，它在实时扬声器计数方面表现出色，即使在截断输入窗口的情况下也是如此。我们使用LibriCount数据集进行的广泛评估强调了TaCNet的卓越性能，将其定位为音频源计数任务的最先进解决方案。TaCNet在11个类别中的平均准确率为74.18%，在不同的场景中证明了其有效性，包括涉及中文和波斯语的应用程序。这种跨语言的适应性突出了它的多功能性和潜在的影响。摘要：In this paper, we introduce the Temporal Audio Source Counting Network (TaCNet), an innovative architecture that addresses limitations in audio source counting tasks. TaCNet operates directly on raw audio inputs, eliminating complex preprocessing steps and simplifying the workflow. Notably, it excels in real-time speaker counting, even with truncated input windows. Our extensive evaluation, conducted using the LibriCount dataset, underscores TaCNet's exceptional performance, positioning it as a state-of-the-art solution for audio source counting tasks. With an average accuracy of 74.18 percentage over 11 classes, TaCNet demonstrates its effectiveness across diverse scenarios, including applications involving Chinese and Persian languages. This cross-lingual adaptability highlights its versatility and potential impact.
【7】 OverHear: Headphone based Multi-sensor Keystroke Inference标题：窃听：基于耳机的多传感器按键推理链接：https://arxiv.org/abs/2311.02288
作者：Raveen Wijewickrama,Maryam Abbasihafshejani,Anindya Maiti,Murtuza Jadliwala摘要：耳机，传统上仅限于音频播放，已经发展到集成传感器，如高清麦克风和加速度计。虽然这些改进增强了用户体验，但它们也引入了潜在的窃听漏洞，其中窃听推断是我们在这项工作中关注的问题。为了验证这一威胁，我们开发了OverHear，这是一个利用耳机中的声学和加速度计数据的语音推断框架。加速度计数据虽然对于个体指纹识别不够详细，但有助于通过手的位置对按键进行聚类。同时，对声学数据进行分析以提取梅尔频率倒谱系数（MFCC），帮助区分不同的倒谱系数。这些特征输入到机器学习模型中进行单词预测，并通过基于词典的单词预测方法进一步优化结果。在我们的实验设置中，我们在不同的环境条件下测试了各种键盘类型。对于机械键盘，我们能够实现大约80%的前5个键预测准确率，对于薄膜键盘，我们能够实现大约60%的前100个单词预测准确率，对于所有键盘类型，我们都能够实现超过70%的前100个单词预测准确率。结果突出了我们的方法在现实世界的情况下的有效性和局限性。摘要：Headphones, traditionally limited to audio playback, have evolved to integrate sensors like high-definition microphones and accelerometers. While these advancements enhance user experience, they also introduce potential eavesdropping vulnerabilities, with keystroke inference being our concern in this work. To validate this threat, we developed OverHear, a keystroke inference framework that leverages both acoustic and accelerometer data from headphones. The accelerometer data, while not sufficiently detailed for individual keystroke identification, aids in clustering key presses by hand position. Concurrently, the acoustic data undergoes analysis to extract Mel Frequency Cepstral Coefficients (MFCC), aiding in distinguishing between different keystrokes. These features feed into machine learning models for keystroke prediction, with results further refined via dictionary-based word prediction methods. In our experimental setup, we tested various keyboard types under different environmental conditions. We were able to achieve top-5 key prediction accuracy of around 80% for mechanical keyboards and around 60% for membrane keyboards with top-100 word prediction accuracies over 70% for all keyboard types. The results highlight the effectiveness and limitations of our approach in the context of real-world scenarios.
【8】 Design Of Rubble Analyzer Probe Using ML For Earthquake标题：基于ML的地震碎石分析探头的设计链接：https://arxiv.org/abs/2311.02087
作者：Abhishek Sebastian,R Pragna,K Vishal Vythianathan,Dasaraju Sohan Sai,U Shiva Sri Hari Al,R Anirudh,Apurv Choudhary摘要：地震碎石分析仪使用机器学习通过环境声音检测人类存在，准确率达到97.45%。它还提供实时环境数据，帮助评估被困人员的生存前景，这对震后救援工作至关重要摘要：The earthquake rubble analyzer uses machine learning to detect human presence via ambient sounds, achieving 97.45% accuracy. It also provides real-time environmental data, aiding in assessing survival prospects for trapped individuals, crucial for post-earthquake rescue efforts
eess.AS音频处理
【1】 Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic  Token Prediction标题：Transduce and Speak：具有语义标记预测的文本到语音的神经转换器链接：https://arxiv.org/abs/2311.02898
作者：Minchan Kim,Myeonghun Jeong,Byoung Jin Choi,Dongjune Lee,Nam Soo Kim备注：Accepted at ASRU2023摘要：我们介绍了一个文本到语音（TTS）框架的基础上神经换能器。我们使用从wav2vec2.0嵌入获得的离散化语义令牌，这使得它很容易采用神经换能器的TTS框架享受其单调对齐约束。该模型首先使用神经换能器生成对齐的语义标记，然后使用非自回归（NAR）语音发生器从语义标记合成语音样本。这种解耦的框架简化了TTS的训练复杂度，并允许每个阶段分别专注于1）语言和对齐建模以及2）细粒度声学建模。在zero-shot自适应语音合成系统上的实验结果表明，该模型在语音质量和说话人相似度方面均优于基线。我们还研究了我们提出的模型的推理速度和韵律可控性，显示了TTS框架的神经换能器的潜力。摘要：We introduce a text-to-speech(TTS) framework based on a neural transducer. We use discretized semantic tokens acquired from wav2vec2.0 embeddings, which makes it easy to adopt a neural transducer for the TTS framework enjoying its monotonic alignment constraints. The proposed model first generates aligned semantic tokens using the neural transducer, then synthesizes a speech sample from the semantic tokens using a non-autoregressive(NAR) speech generator. This decoupled framework alleviates the training complexity of TTS and allows each stage to focus on 1) linguistic and alignment modeling and 2) fine-grained acoustic modeling, respectively. Experimental results on the zero-shot adaptive TTS show that the proposed model exceeds the baselines in speech quality and speaker similarity via objective and subjective measures. We also investigate the inference speed and prosody controllability of our proposed model, showing the potential of the neural transducer for TTS frameworks.
【2】 A Foundation Model for Music Informatics标题：音乐信息学的基础模型链接：https://arxiv.org/abs/2311.03318
作者：Minz Won,Yun-Ning Hung,Duc Le备注：5 pages摘要：本文研究了为音乐信息学量身定制的基础模型，这是一个目前面临标签数据稀缺和泛化问题的领域。为此，我们对各种基础模型变体进行了深入的比较研究，研究了模型架构、标记化方法、时间分辨率、数据和模型可扩展性等关键决定因素。本研究旨在通过阐明这些个体因素如何有助于音乐信息学基础模型的成功来弥合现有的知识差距。采用一个仔细的评估框架，我们评估这些模型在音乐信息检索的不同下游任务的性能，特别关注标记级和序列级分类。我们的研究结果表明，我们的模型表现出强大的性能，在特定的关键指标超过现有的模型。这些发现有助于理解音乐信息学中的自我监督学习，并为在该领域开发更有效和通用的基础模型铺平了道路。我们的模型的预训练版本是公开的，以促进可重复性和未来的研究。摘要：This paper investigates foundation models tailored for music informatics, a domain currently challenged by the scarcity of labeled data and generalization issues. To this end, we conduct an in-depth comparative study among various foundation model variants, examining key determinants such as model architectures, tokenization methods, temporal resolution, data, and model scalability. This research aims to bridge the existing knowledge gap by elucidating how these individual factors contribute to the success of foundation models in music informatics. Employing a careful evaluation framework, we assess the performance of these models across diverse downstream tasks in music information retrieval, with a particular focus on token-level and sequence-level classification. Our results reveal that our model demonstrates robust performance, surpassing existing models in specific key metrics. These findings contribute to the understanding of self-supervised learning in music informatics and pave the way for developing more effective and versatile foundation models in the field. A pretrained version of our model is publicly available to foster reproducibility and future research.
【3】 Attention or Convolution: Transformer Encoders in Audio Language Models  for Inference Efficiency标题：注意还是卷积：音频语言模型中用于提高推理效率的变换编码器链接：https://arxiv.org/abs/2311.02772
作者：Sungho Jeon,Ching-Feng Yeh,Hakan Inan,Wei-Ning Hsu,Rashi Rungta,Yashar Mehdad,Daniel Bikel备注：5 pages摘要：在本文中，我们表明，一个简单的自我监督的预训练的音频模型可以实现比较复杂的预训练模型与语音Transformer编码器的推理效率。这些语音Transformers依赖于将卷积模块与自注意模块混合。它们以最高的效率在ASR上实现了最先进的性能。我们首先表明，采用这些语音Transformers作为编码器也显着提高了预训练音频模型的效率。然而，我们的研究表明，我们可以单独使用高级自我注意力来实现相当的效率。我们证明，这种更简单的方法是特别有益的低比特权重量化技术的神经网络，以提高效率。我们假设，它可以防止传播不同的量化模块之间的错误相比，最近的语音Transformers混合量化卷积和量化的自我注意模块。摘要：In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules.
【4】 AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency  for Video Deepfake Detection标题：AV-Lip-Sync+：利用AV-Hubert利用多模式不一致性进行视频深伪检测链接：https://arxiv.org/abs/2311.02733
作者：Sahibzada Adil Shahzad,Ammarah Hashmi,Yan-Tsung Peng,Yu Tsao,Hsin-Min Wang摘要：多模态操作（也称为视听深度伪造）使得单模态深度伪造检测器难以检测多媒体内容中的伪造。要避免虚假宣传和假新闻的传播，及时发现至关重要。对任一模态的损害（即，视觉或音频）只能通过可以同时利用这两条信息的多模态模型来发现。以往的方法主要采用单模态视频取证和使用监督预训练的伪造检测。提出了一种基于多模态自监督学习（SSL）特征提取器的多模态视频伪造检测新方法。我们使用基于transformer的SSL预训练的Audio-Visual HuBERT（AV-HuBERT）模型作为视觉和声学特征提取器和多尺度时间卷积神经网络来捕获音频和视觉模态之间的时间相关性。由于AV-HuBERT仅从嘴唇区域提取视觉特征，因此我们还采用了另一种基于变换的视频模型来利用面部特征并捕获在deepfake生成过程中引起的空间和时间伪影。实验结果表明，我们的模型优于所有现有的模型，并在FakeAVCeleb和DeepfakeTIMIT数据集上实现了新的最先进的性能。摘要：Multimodal manipulations (also known as audio-visual deepfakes) make it difficult for unimodal deepfake detectors to detect forgeries in multimedia content. To avoid the spread of false propaganda and fake news, timely detection is crucial. The damage to either modality (i.e., visual or audio) can only be discovered through multi-modal models that can exploit both pieces of information simultaneously. Previous methods mainly adopt uni-modal video forensics and use supervised pre-training for forgery detection. This study proposes a new method based on a multi-modal self-supervised-learning (SSL) feature extractor to exploit inconsistency between audio and visual modalities for multi-modal video forgery detection. We use the transformer-based SSL pre-trained Audio-Visual HuBERT (AV-HuBERT) model as a visual and acoustic feature extractor and a multi-scale temporal convolutional neural network to capture the temporal correlation between the audio and visual modalities. Since AV-HuBERT only extracts visual features from the lip region, we also adopt another transformer-based video model to exploit facial features and capture spatial and temporal artifacts caused during the deepfake generation process. Experimental results show that our model outperforms all existing models and achieves new state-of-the-art performance on the FakeAVCeleb and DeepfakeTIMIT datasets.
【5】 Yet Another Generative Model For Room Impulse Response Estimation标题：房间脉冲响应估计的另一种产生式模型链接：https://arxiv.org/abs/2311.02581
作者：Sungho Lee,Hyeong-Seok Choi,Kyogu Lee备注：WASPAA 2023摘要：最近的神经房间脉冲响应（RIR）估计器通常包括用于参考音频分析的编码器和用于RIR合成的生成器。特别是，它是发生器的性能，直接影响整体估计质量。在这种情况下，我们探索一种替代发电机架构，以提高性能。我们首先训练具有残差量化的自动编码器以学习离散潜在令牌空间，其中每个令牌表示RIR的小时频补丁。然后，我们将RIR估计问题转换为参考条件自回归令牌生成任务，采用跨频率，时间和量化深度轴操作的Transformer变体。通过这种方式，我们解决了标准的盲估计任务和额外的声学匹配问题，其目的是找到一个RIR，匹配源信号的目标信号的混响特性。实验结果表明，我们的系统是优于其他基线在各种评估指标。摘要：Recent neural room impulse response (RIR) estimators typically comprise an encoder for reference audio analysis and a generator for RIR synthesis. Especially, it is the performance of the generator that directly influences the overall estimation quality. In this context, we explore an alternate generator architecture for improved performance. We first train an autoencoder with residual quantization to learn a discrete latent token space, where each token represents a small time-frequency patch of the RIR. Then, we cast the RIR estimation problem as a reference-conditioned autoregressive token generation task, employing transformer variants that operate across frequency, time, and quantization depth axes. This way, we address the standard blind estimation task and additional acoustic matching problem, which aims to find an RIR that matches the source signal to the target signal's reverberation characteristics. Experimental results show that our system is preferable to other baselines across various evaluation metrics.
【6】 Generalized zero-shot audio-to-intent classification标题：广义零放炮音频到意图分类链接：https://arxiv.org/abs/2311.02482
作者：Veera Raghavendra Elluru,Devang Kulshreshtha,Rohit Paturi,Sravan Bodapati,Srikanth Ronanki摘要：使用纯音频数据的口语理解系统越来越受欢迎，但它们处理不可见意图的能力仍然有限。在这项研究中，我们提出了一个广义的zero-shot音频到意图的分类框架，每个意图只有几个样本文本句子。为了实现这一点，我们首先训练一个监督的音频意图分类器，利用自我监督的预训练模型。然后，我们利用神经音频合成器来创建样本文本话语的音频嵌入，并使用余弦相似性对看不见的意图执行广义zero-shot分类。我们还提出了一种多模态训练策略，将词汇信息纳入音频表示，以提高zero-shot性能。与仅音频训练相比，我们的多模态训练方法分别将SLURP和内部目标导向对话数据集的SLURP看不见意图的zero-shot意图分类的准确性提高了2.75%和18.2%。摘要：Spoken language understanding systems using audio-only data are gaining popularity, yet their ability to handle unseen intents remains limited. In this study, we propose a generalized zero-shot audio-to-intent classification framework with only a few sample text sentences per intent. To achieve this, we first train a supervised audio-to-intent classifier by making use of a self-supervised pre-trained model. We then leverage a neural audio synthesizer to create audio embeddings for sample text utterances and perform generalized zero-shot classification on unseen intents using cosine similarity. We also propose a multimodal training strategy that incorporates lexical information into the audio representation to improve zero-shot performance. Our multimodal training approach improves the accuracy of zero-shot intent classification on unseen intents of SLURP by 2.75% and 18.2% for the SLURP and internal goal-oriented dialog datasets, respectively, compared to audio-only training.
【7】 TACNET: Temporal Audio Source Counting Network标题：TACNET：时间音源计数网络链接：https://arxiv.org/abs/2311.02369
作者：Amirreza Ahmadnejad,Ahmad Mahmmodian Darviishani,Mohmmad Mehrdad Asadi,Sajjad Saffariyeh,Pedram Yousef,Emad Fatemizadeh摘要：在本文中，我们介绍了时间音频源计数网络（TaCNet），一个创新的架构，解决了音频源计数任务的限制。TaCNet直接对原始音频输入进行操作，消除了复杂的预处理步骤并简化了工作流程。值得注意的是，它在实时扬声器计数方面表现出色，即使在截断输入窗口的情况下也是如此。我们使用LibriCount数据集进行的广泛评估强调了TaCNet的卓越性能，将其定位为音频源计数任务的最先进解决方案。TaCNet在11个类别中的平均准确率为74.18%，在不同的场景中证明了其有效性，包括涉及中文和波斯语的应用程序。这种跨语言的适应性突出了它的多功能性和潜在的影响。摘要：In this paper, we introduce the Temporal Audio Source Counting Network (TaCNet), an innovative architecture that addresses limitations in audio source counting tasks. TaCNet operates directly on raw audio inputs, eliminating complex preprocessing steps and simplifying the workflow. Notably, it excels in real-time speaker counting, even with truncated input windows. Our extensive evaluation, conducted using the LibriCount dataset, underscores TaCNet's exceptional performance, positioning it as a state-of-the-art solution for audio source counting tasks. With an average accuracy of 74.18 percentage over 11 classes, TaCNet demonstrates its effectiveness across diverse scenarios, including applications involving Chinese and Persian languages. This cross-lingual adaptability highlights its versatility and potential impact.
【8】 OverHear: Headphone based Multi-sensor Keystroke Inference标题：OverHear：基于耳机的多传感器击键推断链接：https://arxiv.org/abs/2311.02288
作者：Raveen Wijewickrama,Maryam Abbasihafshejani,Anindya Maiti,Murtuza Jadliwala摘要：耳机，传统上仅限于音频播放，已经发展到集成传感器，如高清麦克风和加速度计。虽然这些改进增强了用户体验，但它们也引入了潜在的窃听漏洞，其中窃听推断是我们在这项工作中关注的问题。为了验证这一威胁，我们开发了OverHear，这是一个利用耳机中的声学和加速度计数据的语音推断框架。加速度计数据虽然对于个体指纹识别不够详细，但有助于通过手的位置对按键进行聚类。同时，对声学数据进行分析以提取梅尔频率倒谱系数（MFCC），帮助区分不同的倒谱系数。这些特征输入到机器学习模型中进行单词预测，并通过基于词典的单词预测方法进一步优化结果。在我们的实验设置中，我们在不同的环境条件下测试了各种键盘类型。对于机械键盘，我们能够实现大约80%的前5个键预测准确率，对于薄膜键盘，我们能够实现大约60%的前100个单词预测准确率，对于所有键盘类型，我们都能够实现超过70%的前100个单词预测准确率。结果突出了我们的方法在现实世界的情况下的有效性和局限性。摘要：Headphones, traditionally limited to audio playback, have evolved to integrate sensors like high-definition microphones and accelerometers. While these advancements enhance user experience, they also introduce potential eavesdropping vulnerabilities, with keystroke inference being our concern in this work. To validate this threat, we developed OverHear, a keystroke inference framework that leverages both acoustic and accelerometer data from headphones. The accelerometer data, while not sufficiently detailed for individual keystroke identification, aids in clustering key presses by hand position. Concurrently, the acoustic data undergoes analysis to extract Mel Frequency Cepstral Coefficients (MFCC), aiding in distinguishing between different keystrokes. These features feed into machine learning models for keystroke prediction, with results further refined via dictionary-based word prediction methods. In our experimental setup, we tested various keyboard types under different environmental conditions. We were able to achieve top-5 key prediction accuracy of around 80% for mechanical keyboards and around 60% for membrane keyboards with top-100 word prediction accuracies over 70% for all keyboard types. The results highlight the effectiveness and limitations of our approach in the context of real-world scenarios.
【9】 COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning标题：COSMIC：语音上下文学习的数据高效教学调优链接：https://arxiv.org/abs/2311.02248
作者：Jing Pan,Jian Wu,Yashesh Gaur,Sunit Sivasankaran,Zhuo Chen,Shujie Liu,Jinyu Li摘要：我们提出了一个数据和成本效益的方式，将语音模态到一个大型语言模型（LLM）。由此产生的多模态LLM是具有指令遵循/上下文学习能力的上下文语音模型- COSMIC。语音理解测试的问题-答案（SQA）对使用GPT-3.5的基础上的语音transmittance生成的一部分，监督的指令调整。COSMIC拥有不到2000万个可训练参数和450小时的英语语音数据用于SQA生成，在语音到文本的任务中表现出紧急的解释跟随和上下文学习能力。该模型能够遵循给定的文本指令生成文本响应，即使在看不见的EN$\到$X语音到文本翻译（S2 TT）任务与zero-shot设置。我们通过各种任务，如EN$\到$X S2 TT和Few-Shot域自适应来评估模型的上下文学习。并通过一个上下文偏置基准评估预防跟踪能力。我们的研究结果表明，所提出的低成本的配方，建立一个语音LLM和新的预防调整数据的有效性。摘要：We present a data and cost efficient way of incorporating the speech modality into a large language model (LLM). The resulting multi-modal LLM is a COntextual Speech Model with Instruction-following/in-context-learning Capabilities - COSMIC. Speech comprehension test question-answer (SQA) pairs are generated using GPT-3.5 based on the speech transcriptions as a part of the supervision for the instruction tuning. With fewer than 20M trainable parameters and as little as 450 hours of English speech data for SQA generation, COSMIC exhibits emergent instruction-following and in-context learning capabilities in speech-to-text tasks. The model is able to follow the given text instructions to generate text response even on the unseen EN$\to$X speech-to-text translation (S2TT) task with zero-shot setting. We evaluate the model's in-context learning via various tasks such as EN$\to$X S2TT and few-shot domain adaptation. And instruction-following capabilities are evaluated through a contextual biasing benchmark. Our results demonstrate the efficacy of the proposed low cost recipe for building a speech LLM and that with the new instruction-tuning data.
【10】 Design Of Rubble Analyzer Probe Using ML For Earthquake标题：基于ML的地震碎石分析探头的设计链接：https://arxiv.org/abs/2311.02087
作者：Abhishek Sebastian,R Pragna,K Vishal Vythianathan,Dasaraju Sohan Sai,U Shiva Sri Hari Al,R Anirudh,Apurv Choudhary摘要：地震碎石分析仪使用机器学习通过环境声音检测人类存在，准确率达到97.45%。它还提供实时环境数据，帮助评估被困人员的生存前景，这对震后救援工作至关重要摘要：The earthquake rubble analyzer uses machine learning to detect human presence via ambient sounds, achieving 97.45% accuracy. It also provides real-time environmental data, aiding in assessing survival prospects for trapped individuals, crucial for post-earthquake rescue efforts
机器翻译，仅供参考