【1】 Audio MFCC-gram Transformers for respiratory insufficiency detection in COVID-19
标题：音频MFCC-GRAM在新冠肺炎呼吸功能不全检测中的应用
链接：https://arxiv.org/abs/2210.14085
作者：Marcelo Matheus Gauy,Marcelo Finger机构：Instituto de Matem´atica e Estat´ıstica – Universidade de S˜ao Paulo (USP)摘要：本研究探索了语音作为生物标志物，并通过分析语音样本研究了呼吸功能不全（RI）的检测。之前的工作\cite{spira 2021}构建了呼吸功能不全COVID-19患者话语的数据集，并通过卷积神经网络对其进行分析，获得了$87.04\%$的准确度，验证了可以通过语音检测RI的假设。在这里，我们研究了Transformer神经网络架构如何提高RI检测的性能。该方法使得能够构建声学模型。通过选择正确的预训练技术，我们生成了一个自监督声学模型，从而提高了Transformers的RI检测性能（$96.53 $）.摘要：This work explores speech as a biomarker and investigates the detection of respiratory insufficiency (RI) by analyzing speech samples. Previous work \cite{spira2021} constructed a dataset of respiratory insufficiency COVID-19 patient utterances and analyzed it by means of a convolutional neural network achieving an accuracy of $87.04\%$, validating the hypothesis that one can detect RI through speech. Here, we study how Transformer neural network architectures can improve the performance on RI detection. This approach enables construction of an acoustic model. By choosing the correct pretraining technique, we generate a self-supervised acoustic model, leading to improved performance ($96.53\%$) of Transformers for RI detection.
【2】 A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives
标题：人工智能在音乐生成中的研究进展：动因、领域和前景
链接：https://arxiv.org/abs/2210.13944
作者：Carlos Hernandez-Olivan,Javier Hernandez-Olivan,Jose R. Beltran机构：Department of Electronic Engineering and Communications, University of Zaragoza, Calle Mar´ıa de Luna , Zaragoza摘要：音乐是加德纳多元智能理论中的一种智能。人类如何感知和理解音乐仍在研究中，对于开发模仿这类过程的人工智能模型至关重要。人工智能音乐生成是近年来备受关注的一个新兴领域。本文通过比较音乐创作技术领域过去和最近的进展，描述了人类如何创作音乐，以及新的人工智能系统如何模仿这一过程。为了理解人工智能模型和算法如何生成音乐以及未来可能出现的潜在应用，我们探索、分析和描述了参与音乐生成过程的代理：数据集、模型、界面、用户和生成的音乐。我们提到了可能受益于该领域的可能应用，并提出了未来可探索的新趋势和未来研究方向。摘要：Music is one of the Gardner's intelligences in his theory of multiple intelligences. How humans perceive and understand music is still being studied and is crucial to develop artificial intelligence models that imitate such processes. Music generation with Artificial Intelligence is an emerging field that is gaining much attention in the recent years. In this paper, we describe how humans compose music and how new AI systems could imitate such process by comparing past and recent advances in the field with music composition techniques. To understand how AI models and algorithms generate music and the potential applications that might appear in the future, we explore, analyze and describe the agents that take part of the music generation process: the datasets, models, interfaces, the users and the generated music. We mention possible applications that might benefit from this field and we also propose new trends and future research directions that could be explored in the future.
【3】 CoLoC: Conditioned Localizer and Classifier for Sound Event Localization and Detection
标题：CoLoC：用于声音事件定位和检测的条件定位器和分类器
链接：https://arxiv.org/abs/2210.13932
作者：Sławomir Kapka,Jakub Tkaczuk机构：Samsung R&D Institute Poland, Warsaw, Poland摘要：在本文中，我们描述了条件定位器和分类器（CoLoC），这是一个新的解决方案，声音事件定位和检测（SELD）。解决方案包括两个阶段：首先进行定位，然后进行以定位器的输出为条件的分类。为了解决信源数目未知的问题，我们引入了序列集生成（SSG）的思想。这两个阶段的模型都是类似SELDnet的CRNN，但具有单一输出。引导推理表明，这两个单输出模型都适合SELD任务。在STARSS22数据集上，我们的解决方案在大多数指标上都优于基线系统。摘要：In this article, we describe Conditioned Localizer and Classifier (CoLoC) which is a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes of two stages: the localization is done first and is followed by classification conditioned by the output of the localizer. In order to resolve the problem of the unknown number of sources we incorporate the idea borrowed from Sequential Set Generation (SSG). Models from both stages are SELDnet-like CRNNs, but with single outputs. Conducted reasoning shows that such two single-output models are fit for SELD task. We show that our solution improves on the baseline system in most metrics on the STARSS22 Dataset.
【4】 MetaSpeech: Speech Effects Switch Along with Environment for Metaverse
标题：MetaSpeech：Metverse的语音效果随环境变化
链接：https://arxiv.org/abs/2210.13811
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by AI2OT2022, The Third International Workshop on Artificial Intelligence Applications in Internet of Things摘要：Metaverse将物理世界扩展到一个新的维度，物理环境和Metaverse环境可以直接连接和进入。语音是现实世界和元宇宙中不可或缺的交流媒介。语音与环境效果的融合对于Metaverse中的用户沉浸感是重要的。本文提出了利用基于语音转换的方法进行目标环境效果语音的转换。该方法引入了环境效应模块，包括效应提取器和效应编码器，效应提取器提取环境信息，效应编码器编码环境效应条件，并利用梯度反转层进行对抗性训练，在保持语音内容和说话人信息的同时去除环境效应。在LJSpeech公共数据集上的实验结果表明，该模型能够完成特定环境效果的转换，并且在语音转换任务中的性能优于基线方法。摘要：Metaverse expands the physical world to a new dimension, and the physical environment and Metaverse environment can be directly connected and entered. Voice is an indispensable communication medium in the real world and Metaverse. Fusion of the voice with environment effects is important for user immersion in Metaverse. In this paper, we proposed using the voice conversion based method for the conversion of target environment effect speech. The proposed method was named MetaSpeech, which introduces an environment effect module containing an effect extractor to extract the environment information and an effect encoder to encode the environment effect condition, in which gradient reversal layer was used for adversarial training to keep the speech content and speaker information while disentangling the environmental effects. From the experiment results on the public dataset of LJSpeech with four environment effects, the proposed model could complete the specific environment effect conversion and outperforms the baseline methods from the voice conversion task.
【5】 Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approach
标题：利用语音级和音素级掩蔽方法改进语音表征学习
链接：https://arxiv.org/abs/2210.13805
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Kexin Zhu,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., School of Software, Fudan University, Shanghai, China备注：Accepted by MSN2022, The 18th International Conference on Mobility, Sensing and Networking摘要：在语音表示学习中，恢复被屏蔽的语音帧是一个广泛应用的问题。然而，这些模型中的大多数在预训练中使用随机掩蔽。在本研究中，我们提出了两种掩蔽方法：（1）语音级掩蔽，使模型掩蔽比无声段更多的语音段，（2）音素级掩蔽，迫使模型掩蔽音素的整个帧，而不是音素段。我们通过这两种方法对模型进行预训练，并在两个下游任务--音素分类和说话人识别上进行评估。实验结果表明，所提出的掩蔽方法有利于提高语音表示的性能。摘要：Recovering the masked speech frames is widely applied in speech representation learning. However, most of these models use random masking in the pre-training. In this work, we proposed two kinds of masking approaches: (1) speech-level masking, making the model to mask more speech segments than silence segments, (2) phoneme-level masking, forcing the model to mask the whole frames of the phoneme, instead of phoneme pieces. We pre-trained the model via these two approaches, and evaluated on two downstream tasks, phoneme classification and speaker recognition. The experiments demonstrated that the proposed masking approaches are beneficial to improve the performance of speech representation.
【6】 Adapitch: Adaption Multi-Speaker Text-to-Speech Conditioned on Pitch Disentangling with Untranscribed Data
标题：Adapitch：基于未转录数据的基音解缠的自适应多说话人文语转换
链接：https://arxiv.org/abs/2210.13803
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by MSN2022, The 18th International Conference on Mobility, Sensing and Networking摘要：本文提出了一种多说话人TTS方法Adapitch，该方法利用未转录的数据对有监督的模块进行自适应。我们设计了两个自监督模块，分别用非转录数据训练文本编码器和MEL解码器，以增强文本和MEL的表示。为了更好地处理合成语音中的韵律信息，在对语音的基音、文本和说话人进行内容去纠缠的基础上，设计了一个有监督的TTS模块。训练阶段分为两个阶段，首先对文本编码器和梅尔解码器进行无监督预训练和固定，然后对TTS解纠缠进行有监督预训练。实验结果表明，Adaptich算法的图像质量明显优于基线算法。摘要：In this paper, we proposed Adapitch, a multi-speaker TTS method that makes adaptation of the supervised module with untranscribed data. We design two self supervised modules to train the text encoder and mel decoder separately with untranscribed data to enhance the representation of text and mel. To better handle the prosody information in a synthesized voice, a supervised TTS module is designed conditioned on content disentangling of pitch, text, and speaker. The training phase was separated into two parts, pretrained and fixed the text encoder and mel decoder with unsupervised mode, then the supervised mode on the disentanglement of TTS. Experiment results show that the Adaptich achieved much better quality than baseline methods.
【7】 Evaluating and Optimizing Hearing-Aid Self-Fitting Methods using Population Coverage
标题：利用总体覆盖率评价和优化助听器自适应方法
链接：https://arxiv.org/abs/2210.13732
作者：Dhruv Vyas,Erik Jorgensen,Yu-Hsiang Wu,Octav Chipara机构：Department of Computer Science, University of Iowa, USA, Department of Communication Sciences and Disorders, University of Wisconsin-Madison, USA, Department of Communication Sciences and Disorders, University of Iowa, USA摘要：轻度至中度听力损失的成年人可以使用非处方助听器来治疗他们的听力损失，而费用只是传统听力保健费用的一小部分。这些产品采用了自适应方法，允许最终用户在没有听力学家帮助的情况下配置他们的助听器。自适应方法帮助用户配置增益-频率响应，以控制传入声音的每个频带的放大。本文考虑如何设计有效的自适应方法，以及我们是否可以在不求助于昂贵的用户研究的情况下评价它们设计的某些方面。大多数现有的装配方法提供了各种用户界面，以允许用户从预定的一组预置中选择配置。我们提出一个新的度量来评估基于预置的方法的性能，通过计算它们的人口覆盖率。人口覆盖率估计可以找到其首选配置的用户比例。我们的方法的一个独特方面是概率模型，该模型捕捉用户的独特偏好与其他具有类似听力损失的用户的差异。接下来，我们开发用于确定预设以最大化人口覆盖率的方法。实验结果表明，与基于聚类的方法相比，所提出的算法能够有效地选择少量的预置点，从而提供更高的人口覆盖率。此外，我们可以使用我们的算法来配置基于滑块的方法的增量的数量。摘要：Adults with mild-to-moderate hearing loss can use over-the-counter hearing aids to treat their hearing loss at a fraction of traditional hearing care costs. These products incorporate self-fitting methods that allow end-users to configure their hearing aids without the help of an audiologist. A self-fitting method helps users configure the gain-frequency responses that control the amplification for each frequency band of the incoming sound. This paper considers how to design effective self-fitting methods and whether we may evaluate certain aspects of their design without resorting to expensive user studies. Most existing fitting methods provide various user interfaces to allow users to select a configuration from a predetermined set of presets. We propose a novel metric for evaluating the performance of preset-based approaches by computing their population coverage. The population coverage estimates the fraction of users for which it is possible to find a configuration they prefer. A unique aspect of our approach is a probabilistic model that captures how a user's unique preferences differ from other users with similar hearing loss. Next, we develop methods for determining presets to maximize population coverage. Exploratory results demonstrate that the proposed algorithms can effectively select a small number of presets that provide higher population coverage than clustering-based approaches. Moreover, we may use our algorithms to configure the number of increments for slider-based methods.
【8】 Spectral Clustering-aware Learning of Embeddings for Speaker Diarisation
标题：用于说话人识别的嵌入谱聚类学习
链接：https://arxiv.org/abs/2210.13576
作者：Evonne P. C. Lee,Guangzhi Sun,Chao Zhang,Philip C. Woodland机构：Cambridge University Engineering Dept., Trumpington St., Cambridge, CB,PZ U.K.备注：Submitted to ICASSP 2023, 5 pages摘要：在说话人识别中，说话人嵌入提取模型的训练损失函数与说话人聚类方法之间存在不匹配的问题。本文提出了一种基于谱聚类的嵌入学习方法（SCALE）来解决这种失配问题。具体地说，除了角原型cal（AP）损失之外，SCALE使用了一种新颖的亲和矩阵损失，其直接最小化从说话人嵌入估计的亲和矩阵与参考之间的误差。SCALE还包括p百分位阈值和高斯模糊作为训练中谱聚类的两个重要超参数。在AMI数据集上的实验表明，当与基于AP损失的说话人嵌入的强基线相比时，使用SCALE获得的说话人嵌入使用oracle分割实现了超过50%的相对说话人错误率降低，并且使用自动分割实现了超过30%的相对日记化错误率降低。摘要：In speaker diarisation, speaker embedding extraction models often suffer from the mismatch between their training loss functions and the speaker clustering method. In this paper, we propose the method of spectral clustering-aware learning of embeddings (SCALE) to address the mismatch. Specifically, besides an angular prototype cal (AP) loss, SCALE uses a novel affinity matrix loss which directly minimises the error between the affinity matrix estimated from speaker embeddings and the reference. SCALE also includes p-percentile thresholding and Gaussian blur as two important hyper-parameters for spectral clustering in training. Experiments on the AMI dataset showed that speaker embeddings obtained with SCALE achieved over 50% relative speaker error rate reductions using oracle segmentation, and over 30% relative diarisation error rate reductions using automatic segmentation when compared to a strong baseline with the AP-loss-based speaker embeddings.
【9】 I see what you hear: a vision-inspired method to localize words
标题：我看到你听到的：一种视觉启发的本地化单词的方法
链接：https://arxiv.org/abs/2210.13567
作者：Mohammad Samragh,Arnav Kundu,Ting-Yao Hu,Minsik Cho,Aman Chadha,Ashish Shrivastava,Oncel Tuzel,Devang Naik摘要：本文探讨了使用视觉对象检测技术在语音数据中进行单词定位的可能性。在当代文献中已经针对视觉数据对对象检测进行了深入研究。注意，音频可以被解释为1维图像，对象定位技术对于单词定位可以是基本上有用的。基于这种思想，我们提出了一种轻量级的单词检测和本地化解决方案。我们使用边界框回归进行单词定位，这使得我们的模型能够检测给定音频流中关键词的出现、偏移和持续时间。我们使用LibriSpeech进行实验，并训练一个模型来本地化1000个单词。与已有的方法相比，该方法使模型规模减少了94%，F1得分提高了6.5%.摘要：This paper explores the possibility of using visual object detection techniques for word localization in speech data. Object detection has been thoroughly studied in the contemporary literature for visual data. Noting that an audio can be interpreted as a 1-dimensional image, object localization techniques can be fundamentally useful for word localization. Building upon this idea, we propose a lightweight solution for word detection and localization. We use bounding box regression for word localization, which enables our model to detect the occurrence, offset, and duration of keywords in a given audio stream. We experiment with LibriSpeech and train a model to localize 1000 words. Compared to existing work, our method reduces model size by 94%, and improves the F1 score by 6.5\%.
【10】 EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient microphones
标题：EBEN：适用于用抗噪声麦克风捕获的语音信号的极带宽扩展网络
链接：https://arxiv.org/abs/2210.14090
作者：Julien Hauret,Thomas Joubaud,Véronique Zimpfer,Éric Bavu机构：V´eronique Zimpfer†, ´Eric Bavu⋆, ⋆ LMSSC, Conservatoire national des arts et m´etiers, Paris, France, HESAM Universit´e, † Department of Acoustics and Soldier Protection, French-German Research Institute of Saint-Louis (ISL)备注：5 pages, 5 figures, submitted to ICASSP 2023摘要：本文提出了一种生成式对抗网络,,极端带宽扩展网络（Extreme Bandwidth Extension Network，EBEN），它可以增强抗噪麦克风测量的音频信号。这种类型的捕获设备以语音带宽为代价来抑制环境噪声，从而需要信号增强技术来恢复宽带语音信号。EBEN利用原始捕获语音的多频带分解来减小数据时域维度，并且给出对全频带信号的更好控制。该多频带表示被馈送到类U-Net模型，该模型采用特征和对抗损失的组合来恢复增强的音频信号。我们还受益于所提出的鉴别器架构中的这种原始表示。我们的方法可以实现最先进的结果与轻量级的发生器和实时兼容的操作。摘要：In this paper, we present Extreme Bandwidth Extension Network (EBEN), a generative adversarial network (GAN) that enhances audio measured with noise-resilient microphones. This type of capture equipment suppresses ambient noise at the expense of speech bandwidth, thereby requiring signal enhancement techniques to recover the wideband speech signal. EBEN leverages a multiband decomposition of the raw captured speech to decrease the data time-domain dimensions, and give better control over the full-band signal. This multiband representation is fed to a U-Net-like model, which adopts a combination of feature and adversarial losses to recover an enhanced audio signal. We also benefit from this original representation in the proposed discriminator architecture. Our approach can achieve state-of-the-art results with a lightweight generator and real-time compatible operation.
【11】 Enhanced Fuzzy Decomposition of Sound Into Sines, Transients, and Noise
标题：声音的增强模糊分解为正弦、瞬变和噪声
链接：https://arxiv.org/abs/2210.14041
作者：Leonardo Fierro,Vesa Välimäki机构：Acoustics Lab, Department of Signal Processing and Acoustics, Aalto University, Espoo, Finland备注：Submitted for publication to the Journal of Audio Engineering Society on October 20th, 2022摘要：将声音分解为正弦、瞬变和噪声是音频处理中的一个长期研究问题。用于该三向分离的当前解决方案检测谱图中的水平和垂直结构或各向异性和取向，以识别每个谱仓的属性并将其分类为正弦、瞬态或噪声。提出了一种基于模糊逻辑的增强型三向分解方法，该方法在保持图像完美重构特性的同时实现了软掩蔽。所提出的方法允许每个谱仓同时属于正弦和噪声或瞬态和噪声两类。通过与其他三种方法的主观听觉测试，结果表明，所提出的分解方法产生了更好的或相当的质量。主要的改进表现在瞬态分离，其具有很少或没有能量损失或从其他组件泄漏，并且对于呈现强瞬态的测试信号表现良好。对于所有测试的方法，分离的音频质量被示出依赖于输入信号的复杂性。所提出的方法有助于提高各种音频处理应用的质量。最后给出了一个实例，说明了该方法的有效性。摘要：The decomposition of sounds into sines, transients, and noise is a long-standing research problem in audio processing. The current solutions for this three-way separation detect either horizontal and vertical structures or anisotropy and orientations in the spectrogram to identify the properties of each spectral bin and classify it as sinusoidal, transient, or noise. This paper proposes an enhanced three-way decomposition method based on fuzzy logic, enabling soft masking while preserving the perfect reconstruction property. The proposed method allows each spectral bin to simultaneously belong to two classes, sine and noise or transient and noise. Results of a subjective listening test against three other techniques are reported, showing that the proposed decomposition yields a better or comparable quality. The main improvement appears in transient separation, which enjoys little or no loss of energy or leakage from the other components and performs well for test signals presenting strong transients. The audio quality of the separation is shown to depend on the complexity of the input signal for all tested methods. The proposed method helps improve the quality of various audio processing applications. A successful implementation over a state-of-the-art time-scale modification method is reported as an example.
【12】 Disentangled Speech Representation Learning for One-Shot Cross-lingual Voice Conversion Using $β$-VAE
标题：基于$β$-vae的单次跨语言语音转换解纠缠语音表示学习
链接：https://arxiv.org/abs/2210.13771
作者：Hui Lu,Disong Wang,Xixin Wu,Zhiyong Wu,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Centre for Perceptual and Interactive Intelligence, CUHK, Hong Kong SAR, China, Tsinghua-CUHK Joint Research Center for Media Sciences, Technologies and Systems摘要：提出了一种无监督学习方法，将语音分解为内容表示和说话人身份表示。我们将此方法应用于具有挑战性的单次跨语言语音转换任务，以验证解纠缠的有效性。受$\beta$-VAE的启发，我们引入了一个学习目标，在内容和说话人表征所捕获的信息之间进行平衡。此外，来自体系结构设计和训练数据集的归纳偏差进一步促进了期望的解开。客观和主观评价均表明了该方法在语音解纠缠和单次跨语言语音转换中的有效性。摘要：We propose an unsupervised learning method to disentangle speech into content representation and speaker identity representation. We apply this method to the challenging one-shot cross-lingual voice conversion task to demonstrate the effectiveness of the disentanglement. Inspired by $\beta$-VAE, we introduce a learning objective that balances between the information captured by the content and speaker representations. In addition, the inductive biases from the architectural design and the training dataset further encourage the desired disentanglement. Both objective and subjective evaluations show the effectiveness of the proposed method in speech disentanglement and in one-shot cross-lingual voice conversion.
【13】 Streaming Parrotron for on-device speech-to-speech conversion
标题：用于设备上语音到语音转换的流Parrotron
链接：https://arxiv.org/abs/2210.13761
作者：Oleg Rybakov,Fadi Biadsy,Xia Zhang,Liyang Jiang,Phoenix Meadowlark,Shivani Agrawal摘要：本文提出了一种完全基于设备的流式语音到语音（STS）转换模型，该模型将给定的输入语音直接归一化为合成的输出语音（也称为合成语音）。Parrotron）的声音。在移动设备上本地部署这样的端到端模型在内存占用和计算要求方面提出了重大挑战。在本文中，我们提出了一种基于流的方法，与基于非流服务器的方法相比，该方法可以产生可接受的延迟，同时最大限度地降低语音转换质量。我们的方法包括首先在说话者说话的同时实时地流式传输编码器。然后，一旦说话者停止说话，我们就沿着流式声码器的一侧以流式模式运行谱图解码器，以实时生成输出语音。为了实现可接受的延迟质量折衷，我们研究了一种新颖的混合方法用于编码器中的前瞻，该方法将前瞻特征堆叠器与前瞻自注意相结合。我们还将该模型与int4量化感知训练和int8后训练量化进行了比较，结果表明，我们的流方法在Pixel4 CPU上比实时快2倍。摘要：We present a fully on-device and streaming Speech-To-Speech (STS) conversion model that normalizes a given input speech directly to synthesized output speech (a.k.a. Parrotron). Deploying such an end-to-end model locally on mobile devices pose significant challenges in terms of memory footprint and computation requirements. In this paper, we present a streaming-based approach to produce an acceptable delay, with minimal loss in speech conversion quality, when compared to a non-streaming server-based approach. Our approach consists of first streaming the encoder in real time while the speaker is speaking. Then, as soon as the speaker stops speaking, we run the spectrogram decoder in streaming mode along the side of a streaming vocoder to generate output speech in real time. To achieve an acceptable delay quality trade-off, we study a novel hybrid approach for look-ahead in the encoder which combines a look-ahead feature stacker with a look-ahead self-attention. We also compare the model with int4 quantization aware training and int8 post training quantization and show that our streaming approach is 2x faster than real time on the Pixel4 CPU.
【14】 Mixed Emotion Modelling for Emotional Voice Conversion
标题：用于情感声音转换的混合情感模型
链接：https://arxiv.org/abs/2210.13756
作者：Kun Zhou,Berrak Sisman,Carlos Busso,Haizhou Li机构：National University of Singapore, Singapore ,The University of Texas at Dallas, USA, The Chinese University of HongKong, Shenzhen, China备注：Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2208.05890摘要：情感语音转换（EVC）的目的是在保持语言内容和说话人身份的同时，将话语的情感状态从一种情感转换为另一种情感。目前的研究主要集中在对几种特定情绪类型之间的转换进行建模。综合情感的混合效果可以帮助我们更好地模仿人类的情感，促进更自然的人机交互。本研究首次提出并研究了EVC中混合情感合成的研究问题。我们将情绪风格视为一系列情绪属性，这些情绪属性是通过基于排序的支持向量机（SVM）学习得到的。每个属性度量属于不同情感类型的语音记录之间的相关度。然后，我们将这些属性合并到序列到序列（seq2seq）情感语音转换框架中。在训练过程中，该框架不仅学会了对输入的情绪风格进行表征，而且还量化了它与其他情绪类型的相关性。在运行时，可以通过手动定义属性来产生各种情绪混合。我们进行了客观和主观的评估，以验证我们的想法在混合情感合成方面。作为情感转换的一个应用，我们进一步构建了一个情感三角。代码和语音样本是公开可用的。摘要：Emotional voice conversion (EVC) aims to convert the emotional state of an utterance from one emotion to another while preserving the linguistic content and speaker identity. Current studies mostly focus on modelling the conversion between several specific emotion types. Synthesizing mixed effects of emotions could help us to better imitate human emotions, and facilitate more natural human-computer interaction. In this research, for the first time, we formulate and study the research problem of mixed emotion synthesis for EVC. We regard emotional styles as a series of emotion attributes that are learnt from a ranking-based support vector machine (SVM). Each attribute measures the degree of the relevance between the speech recordings belonging to different emotion types. We then incorporate those attributes into a sequence-to-sequence (seq2seq) emotional voice conversion framework. During the training, the framework not only learns to characterize the input emotional style, but also quantifies its relevance with other emotion types. At run-time, various emotional mixtures can be produced by manually defining the attributes. We conduct objective and subjective evaluations to validate our idea in terms of mixed emotion synthesis. We further build an emotion triangle as an application of emotion transition. Codes and speech samples are publicly available.
【15】 Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering
标题：高效的实时流和完全在设备上的说话人二元化和多级聚类
链接：https://arxiv.org/abs/2210.13690
作者：Quan Wang,Yiling Huang,Han Lu,Guanlong Zhao,Ignacio Lopez Moreno摘要：虽然最近在说话人diarization方面的研究进展主要集中在提高diarization结果的质量，但是对提高diarization系统的效率也有越来越多的兴趣。本文提出了一种多阶段聚类策略，即对不同长度的输入采用不同的聚类算法。具体地说，回退聚类器用于处理短格式输入;主聚类器用于处理中等长度的输入;并且预群集器用于在长格式输入被主群集器处理之前压缩长格式输入。主聚类器和预聚类器都可以被配置有计算复杂度的上限，以适应具有不同约束的设备。这种多级聚类策略对于CPU、存储器和电池预算紧张的流式设备上扬声器diarization系统是至关重要的。摘要：While recent research advances in speaker diarization mostly focus on improving the quality of diarization results, there is also an increasing interest in improving the efficiency of diarization systems. In this paper, we propose a multi-stage clustering strategy, that uses different clustering algorithms for input of different lengths. Specifically, a fallback clusterer is used to handle short-form inputs; a main clusterer is used to handle medium-length inputs; and a pre-clusterer is used to compress long-form inputs before they are processed by the main clusterer. Both the main clusterer and the pre-clusterer can be configured with an upper bound of the computational complexity to adapt to devices with different constraints. This multi-stage clustering strategy is critical for streaming on-device speaker diarization systems, where the budgets of CPU, memory and battery are tight.
【1】 EBEN: Extreme bandwidth extension network applied to speech signals captured with noise-resilient microphones
标题：EBEN：适用于用抗噪声麦克风捕获的语音信号的极带宽扩展网络
链接：https://arxiv.org/abs/2210.14090
* 与cs.SD语音【10】为同一篇
作者：Julien Hauret,Thomas Joubaud,Véronique Zimpfer,Éric Bavu机构：V´eronique Zimpfer†, ´Eric Bavu⋆, ⋆ LMSSC, Conservatoire national des arts et m´etiers, Paris, France, HESAM Universit´e, † Department of Acoustics and Soldier Protection, French-German Research Institute of Saint-Louis (ISL)备注：5 pages, 5 figures, submitted to ICASSP 2023摘要：本文提出了一种生成式对抗网络,,极端带宽扩展网络（Extreme Bandwidth Extension Network，EBEN），它可以增强抗噪麦克风测量的音频信号。这种类型的捕获设备以语音带宽为代价来抑制环境噪声，从而需要信号增强技术来恢复宽带语音信号。EBEN利用原始捕获语音的多频带分解来减小数据时域维度，并且给出对全频带信号的更好控制。该多频带表示被馈送到类U-Net模型，该模型采用特征和对抗损失的组合来恢复增强的音频信号。我们还受益于所提出的鉴别器架构中的这种原始表示。我们的方法可以实现最先进的结果与轻量级的发生器和实时兼容的操作。摘要：In this paper, we present Extreme Bandwidth Extension Network (EBEN), a generative adversarial network (GAN) that enhances audio measured with noise-resilient microphones. This type of capture equipment suppresses ambient noise at the expense of speech bandwidth, thereby requiring signal enhancement techniques to recover the wideband speech signal. EBEN leverages a multiband decomposition of the raw captured speech to decrease the data time-domain dimensions, and give better control over the full-band signal. This multiband representation is fed to a U-Net-like model, which adopts a combination of feature and adversarial losses to recover an enhanced audio signal. We also benefit from this original representation in the proposed discriminator architecture. Our approach can achieve state-of-the-art results with a lightweight generator and real-time compatible operation.
【2】 Enhanced Fuzzy Decomposition of Sound Into Sines, Transients, and Noise
标题：声音的增强模糊分解为正弦、瞬变和噪声
链接：https://arxiv.org/abs/2210.14041
* 与cs.SD语音【11】为同一篇
作者：Leonardo Fierro,Vesa Välimäki机构：Acoustics Lab, Department of Signal Processing and Acoustics, Aalto University, Espoo, Finland备注：Submitted for publication to the Journal of Audio Engineering Society on October 20th, 2022摘要：将声音分解为正弦、瞬变和噪声是音频处理中的一个长期研究问题。用于该三向分离的当前解决方案检测谱图中的水平和垂直结构或各向异性和取向，以识别每个谱仓的属性并将其分类为正弦、瞬态或噪声。提出了一种基于模糊逻辑的增强型三向分解方法，该方法在保持图像完美重构特性的同时实现了软掩蔽。所提出的方法允许每个谱仓同时属于正弦和噪声或瞬态和噪声两类。通过与其他三种方法的主观听觉测试，结果表明，所提出的分解方法产生了更好的或相当的质量。主要的改进表现在瞬态分离，其具有很少或没有能量损失或从其他组件泄漏，并且对于呈现强瞬态的测试信号表现良好。对于所有测试的方法，分离的音频质量被示出依赖于输入信号的复杂性。所提出的方法有助于提高各种音频处理应用的质量。最后给出了一个实例，说明了该方法的有效性。摘要：The decomposition of sounds into sines, transients, and noise is a long-standing research problem in audio processing. The current solutions for this three-way separation detect either horizontal and vertical structures or anisotropy and orientations in the spectrogram to identify the properties of each spectral bin and classify it as sinusoidal, transient, or noise. This paper proposes an enhanced three-way decomposition method based on fuzzy logic, enabling soft masking while preserving the perfect reconstruction property. The proposed method allows each spectral bin to simultaneously belong to two classes, sine and noise or transient and noise. Results of a subjective listening test against three other techniques are reported, showing that the proposed decomposition yields a better or comparable quality. The main improvement appears in transient separation, which enjoys little or no loss of energy or leakage from the other components and performs well for test signals presenting strong transients. The audio quality of the separation is shown to depend on the complexity of the input signal for all tested methods. The proposed method helps improve the quality of various audio processing applications. A successful implementation over a state-of-the-art time-scale modification method is reported as an example.
【3】 Disentangled Speech Representation Learning for One-Shot Cross-lingual Voice Conversion Using $β$-VAE
标题：基于$β$-vae的单次跨语言语音转换解纠缠语音表示学习
链接：https://arxiv.org/abs/2210.13771
* 与cs.SD语音【12】为同一篇
作者：Hui Lu,Disong Wang,Xixin Wu,Zhiyong Wu,Xunying Liu,Helen Meng机构：The Chinese University of Hong Kong, Hong Kong SAR, China, Centre for Perceptual and Interactive Intelligence, CUHK, Hong Kong SAR, China, Tsinghua-CUHK Joint Research Center for Media Sciences, Technologies and Systems摘要：提出了一种无监督学习方法，将语音分解为内容表示和说话人身份表示。我们将此方法应用于具有挑战性的单次跨语言语音转换任务，以验证解纠缠的有效性。受$\beta$-VAE的启发，我们引入了一个学习目标，在内容和说话人表征所捕获的信息之间进行平衡。此外，来自体系结构设计和训练数据集的归纳偏差进一步促进了期望的解开。客观和主观评价均表明了该方法在语音解纠缠和单次跨语言语音转换中的有效性。摘要：We propose an unsupervised learning method to disentangle speech into content representation and speaker identity representation. We apply this method to the challenging one-shot cross-lingual voice conversion task to demonstrate the effectiveness of the disentanglement. Inspired by $\beta$-VAE, we introduce a learning objective that balances between the information captured by the content and speaker representations. In addition, the inductive biases from the architectural design and the training dataset further encourage the desired disentanglement. Both objective and subjective evaluations show the effectiveness of the proposed method in speech disentanglement and in one-shot cross-lingual voice conversion.
【4】 Streaming Parrotron for on-device speech-to-speech conversion
标题：用于设备上语音到语音转换的流Parrotron
链接：https://arxiv.org/abs/2210.13761
* 与cs.SD语音【13】为同一篇
作者：Oleg Rybakov,Fadi Biadsy,Xia Zhang,Liyang Jiang,Phoenix Meadowlark,Shivani Agrawal摘要：本文提出了一种完全基于设备的流式语音到语音（STS）转换模型，该模型将给定的输入语音直接归一化为合成的输出语音（也称为合成语音）。Parrotron）的声音。在移动设备上本地部署这样的端到端模型在内存占用和计算要求方面提出了重大挑战。在本文中，我们提出了一种基于流的方法，与基于非流服务器的方法相比，该方法可以产生可接受的延迟，同时最大限度地降低语音转换质量。我们的方法包括首先在说话者说话的同时实时地流式传输编码器。然后，一旦说话者停止说话，我们就沿着流式声码器的一侧以流式模式运行谱图解码器，以实时生成输出语音。为了实现可接受的延迟质量折衷，我们研究了一种新颖的混合方法用于编码器中的前瞻，该方法将前瞻特征堆叠器与前瞻自注意相结合。我们还将该模型与int4量化感知训练和int8后训练量化进行了比较，结果表明，我们的流方法在Pixel4 CPU上比实时快2倍。摘要：We present a fully on-device and streaming Speech-To-Speech (STS) conversion model that normalizes a given input speech directly to synthesized output speech (a.k.a. Parrotron). Deploying such an end-to-end model locally on mobile devices pose significant challenges in terms of memory footprint and computation requirements. In this paper, we present a streaming-based approach to produce an acceptable delay, with minimal loss in speech conversion quality, when compared to a non-streaming server-based approach. Our approach consists of first streaming the encoder in real time while the speaker is speaking. Then, as soon as the speaker stops speaking, we run the spectrogram decoder in streaming mode along the side of a streaming vocoder to generate output speech in real time. To achieve an acceptable delay quality trade-off, we study a novel hybrid approach for look-ahead in the encoder which combines a look-ahead feature stacker with a look-ahead self-attention. We also compare the model with int4 quantization aware training and int8 post training quantization and show that our streaming approach is 2x faster than real time on the Pixel4 CPU.
【5】 Mixed Emotion Modelling for Emotional Voice Conversion
标题：用于情感声音转换的混合情感模型
链接：https://arxiv.org/abs/2210.13756
* 与cs.SD语音【14】为同一篇
作者：Kun Zhou,Berrak Sisman,Carlos Busso,Haizhou Li机构：National University of Singapore, Singapore ,The University of Texas at Dallas, USA, The Chinese University of HongKong, Shenzhen, China备注：Submitted to ICASSP 2023. arXiv admin note: text overlap with arXiv:2208.05890摘要：情感语音转换（EVC）的目的是在保持语言内容和说话人身份的同时，将话语的情感状态从一种情感转换为另一种情感。目前的研究主要集中在对几种特定情绪类型之间的转换进行建模。综合情感的混合效果可以帮助我们更好地模仿人类的情感，促进更自然的人机交互。本研究首次提出并研究了EVC中混合情感合成的研究问题。我们将情绪风格视为一系列情绪属性，这些情绪属性是通过基于排序的支持向量机（SVM）学习得到的。每个属性度量属于不同情感类型的语音记录之间的相关度。然后，我们将这些属性合并到序列到序列（seq2seq）情感语音转换框架中。在训练过程中，该框架不仅学会了对输入的情绪风格进行表征，而且还量化了它与其他情绪类型的相关性。在运行时，可以通过手动定义属性来产生各种情绪混合。我们进行了客观和主观的评估，以验证我们的想法在混合情感合成方面。作为情感转换的一个应用，我们进一步构建了一个情感三角。代码和语音样本是公开可用的。摘要：Emotional voice conversion (EVC) aims to convert the emotional state of an utterance from one emotion to another while preserving the linguistic content and speaker identity. Current studies mostly focus on modelling the conversion between several specific emotion types. Synthesizing mixed effects of emotions could help us to better imitate human emotions, and facilitate more natural human-computer interaction. In this research, for the first time, we formulate and study the research problem of mixed emotion synthesis for EVC. We regard emotional styles as a series of emotion attributes that are learnt from a ranking-based support vector machine (SVM). Each attribute measures the degree of the relevance between the speech recordings belonging to different emotion types. We then incorporate those attributes into a sequence-to-sequence (seq2seq) emotional voice conversion framework. During the training, the framework not only learns to characterize the input emotional style, but also quantifies its relevance with other emotion types. At run-time, various emotional mixtures can be produced by manually defining the attributes. We conduct objective and subjective evaluations to validate our idea in terms of mixed emotion synthesis. We further build an emotion triangle as an application of emotion transition. Codes and speech samples are publicly available.
【6】 Does Joint Training Really Help Cascaded Speech Translation?
标题：联合训练真的有助于级联语音翻译吗？
链接：https://arxiv.org/abs/2210.13700
作者：Viet Anh Khoa Tran,David Thulke,Yingbo Gao,Christian Herold,Hermann Ney机构：Human Language Technology and Pattern Recognition Group, Computer Science Department, RWTH Aachen University, D-, Aachen, Germany备注：Accepted to EMNLP 2022摘要：目前，在语音翻译中，直接的方法--将识别系统与翻译系统级联--提供了最先进的结果。然而，诸如来自自动语音识别系统的错误传播之类的基本挑战仍然存在。为了缓解这些问题，近年来人们将注意力转向直接数据，并提出了各种联合训练方法。在这项工作中，我们试图回答联合训练是否真的有助于级联语音翻译的问题。我们回顾了最近关于这个主题的论文，并通过边缘化转录后验概率研究了一种联合训练准则。我们的研究结果表明，一个强级联基线可以减少任何改善获得使用联合培训，我们建议替代联合培训。我们希望这项工作可以作为一个刷新当前的语音翻译景观，并推动研究找到更有效和创造性的方法来利用直接数据的语音翻译。摘要：Currently, in speech translation, the straightforward approach - cascading a recognition system with a translation system - delivers state-of-the-art results. However, fundamental challenges such as error propagation from the automatic speech recognition system still remain. To mitigate these problems, recently, people turn their attention to direct data and propose various joint training methods. In this work, we seek to answer the question of whether joint training really helps cascaded speech translation. We review recent papers on the topic and also investigate a joint training criterion by marginalizing the transcription posterior probabilities. Our findings show that a strong cascaded baseline can diminish any improvements obtained using joint training, and we suggest alternatives to joint training. We hope this work can serve as a refresher of the current speech translation landscape, and motivate research in finding more efficient and creative ways to utilize the direct data for speech translation.
【7】 Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clustering
标题：高效的实时流和完全在设备上的说话人二元化和多级聚类
链接：https://arxiv.org/abs/2210.13690
* 与cs.SD语音【15】为同一篇
作者：Quan Wang,Yiling Huang,Han Lu,Guanlong Zhao,Ignacio Lopez Moreno摘要：虽然最近在说话人diarization方面的研究进展主要集中在提高diarization结果的质量，但是对提高diarization系统的效率也有越来越多的兴趣。本文提出了一种多阶段聚类策略，即对不同长度的输入采用不同的聚类算法。具体地说，回退聚类器用于处理短格式输入;主聚类器用于处理中等长度的输入;并且预群集器用于在长格式输入被主群集器处理之前压缩长格式输入。主聚类器和预聚类器都可以被配置有计算复杂度的上限，以适应具有不同约束的设备。这种多级聚类策略对于CPU、存储器和电池预算紧张的流式设备上扬声器diarization系统是至关重要的。摘要：While recent research advances in speaker diarization mostly focus on improving the quality of diarization results, there is also an increasing interest in improving the efficiency of diarization systems. In this paper, we propose a multi-stage clustering strategy, that uses different clustering algorithms for input of different lengths. Specifically, a fallback clusterer is used to handle short-form inputs; a main clusterer is used to handle medium-length inputs; and a pre-clusterer is used to compress long-form inputs before they are processed by the main clusterer. Both the main clusterer and the pre-clusterer can be configured with an upper bound of the computational complexity to adapt to devices with different constraints. This multi-stage clustering strategy is critical for streaming on-device speaker diarization systems, where the budgets of CPU, memory and battery are tight.
【8】 Audio MFCC-gram Transformers for respiratory insufficiency detection in COVID-19
标题：音频MFCC-GRAM在新冠肺炎呼吸功能不全检测中的应用
链接：https://arxiv.org/abs/2210.14085
* 与cs.SD语音【1】为同一篇
作者：Marcelo Matheus Gauy,Marcelo Finger机构：Instituto de Matem´atica e Estat´ıstica – Universidade de S˜ao Paulo (USP)摘要：本研究探索了语音作为生物标志物，并通过分析语音样本研究了呼吸功能不全（RI）的检测。之前的工作\cite{spira 2021}构建了呼吸功能不全COVID-19患者话语的数据集，并通过卷积神经网络对其进行分析，获得了$87.04\%$的准确度，验证了可以通过语音检测RI的假设。在这里，我们研究了Transformer神经网络架构如何提高RI检测的性能。该方法使得能够构建声学模型。通过选择正确的预训练技术，我们生成了一个自监督声学模型，从而提高了Transformers的RI检测性能（$96.53 $）.摘要：This work explores speech as a biomarker and investigates the detection of respiratory insufficiency (RI) by analyzing speech samples. Previous work \cite{spira2021} constructed a dataset of respiratory insufficiency COVID-19 patient utterances and analyzed it by means of a convolutional neural network achieving an accuracy of $87.04\%$, validating the hypothesis that one can detect RI through speech. Here, we study how Transformer neural network architectures can improve the performance on RI detection. This approach enables construction of an acoustic model. By choosing the correct pretraining technique, we generate a self-supervised acoustic model, leading to improved performance ($96.53\%$) of Transformers for RI detection.
【9】 A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives
标题：人工智能在音乐生成中的研究进展：动因、领域和前景
链接：https://arxiv.org/abs/2210.13944
* 与cs.SD语音【2】为同一篇
作者：Carlos Hernandez-Olivan,Javier Hernandez-Olivan,Jose R. Beltran机构：Department of Electronic Engineering and Communications, University of Zaragoza, Calle Mar´ıa de Luna , Zaragoza摘要：音乐是加德纳多元智能理论中的一种智能。人类如何感知和理解音乐仍在研究中，对于开发模仿这类过程的人工智能模型至关重要。人工智能音乐生成是近年来备受关注的一个新兴领域。本文通过比较音乐创作技术领域过去和最近的进展，描述了人类如何创作音乐，以及新的人工智能系统如何模仿这一过程。为了理解人工智能模型和算法如何生成音乐以及未来可能出现的潜在应用，我们探索、分析和描述了参与音乐生成过程的代理：数据集、模型、界面、用户和生成的音乐。我们提到了可能受益于该领域的可能应用，并提出了未来可探索的新趋势和未来研究方向。摘要：Music is one of the Gardner's intelligences in his theory of multiple intelligences. How humans perceive and understand music is still being studied and is crucial to develop artificial intelligence models that imitate such processes. Music generation with Artificial Intelligence is an emerging field that is gaining much attention in the recent years. In this paper, we describe how humans compose music and how new AI systems could imitate such process by comparing past and recent advances in the field with music composition techniques. To understand how AI models and algorithms generate music and the potential applications that might appear in the future, we explore, analyze and describe the agents that take part of the music generation process: the datasets, models, interfaces, the users and the generated music. We mention possible applications that might benefit from this field and we also propose new trends and future research directions that could be explored in the future.
【10】 CoLoC: Conditioned Localizer and Classifier for Sound Event Localization and Detection
标题：CoLoC：用于声音事件定位和检测的条件定位器和分类器
链接：https://arxiv.org/abs/2210.13932
* 与cs.SD语音【3】为同一篇
作者：Sławomir Kapka,Jakub Tkaczuk机构：Samsung R&D Institute Poland, Warsaw, Poland摘要：在本文中，我们描述了条件定位器和分类器（CoLoC），这是一个新的解决方案，声音事件定位和检测（SELD）。解决方案包括两个阶段：首先进行定位，然后进行以定位器的输出为条件的分类。为了解决信源数目未知的问题，我们引入了序列集生成（SSG）的思想。这两个阶段的模型都是类似SELDnet的CRNN，但具有单一输出。引导推理表明，这两个单输出模型都适合SELD任务。在STARSS22数据集上，我们的解决方案在大多数指标上都优于基线系统。摘要：In this article, we describe Conditioned Localizer and Classifier (CoLoC) which is a novel solution for Sound Event Localization and Detection (SELD). The solution constitutes of two stages: the localization is done first and is followed by classification conditioned by the output of the localizer. In order to resolve the problem of the unknown number of sources we incorporate the idea borrowed from Sequential Set Generation (SSG). Models from both stages are SELDnet-like CRNNs, but with single outputs. Conducted reasoning shows that such two single-output models are fit for SELD task. We show that our solution improves on the baseline system in most metrics on the STARSS22 Dataset.
【11】 MetaSpeech: Speech Effects Switch Along with Environment for Metaverse
标题：MetaSpeech：Metverse的语音效果随环境变化
链接：https://arxiv.org/abs/2210.13811
* 与cs.SD语音【4】为同一篇
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by AI2OT2022, The Third International Workshop on Artificial Intelligence Applications in Internet of Things摘要：Metaverse将物理世界扩展到一个新的维度，物理环境和Metaverse环境可以直接连接和进入。语音是现实世界和元宇宙中不可或缺的交流媒介。语音与环境效果的融合对于Metaverse中的用户沉浸感是重要的。本文提出了利用基于语音转换的方法进行目标环境效果语音的转换。该方法引入了环境效应模块，包括效应提取器和效应编码器，效应提取器提取环境信息，效应编码器编码环境效应条件，并利用梯度反转层进行对抗性训练，在保持语音内容和说话人信息的同时去除环境效应。在LJSpeech公共数据集上的实验结果表明，该模型能够完成特定环境效果的转换，并且在语音转换任务中的性能优于基线方法。摘要：Metaverse expands the physical world to a new dimension, and the physical environment and Metaverse environment can be directly connected and entered. Voice is an indispensable communication medium in the real world and Metaverse. Fusion of the voice with environment effects is important for user immersion in Metaverse. In this paper, we proposed using the voice conversion based method for the conversion of target environment effect speech. The proposed method was named MetaSpeech, which introduces an environment effect module containing an effect extractor to extract the environment information and an effect encoder to encode the environment effect condition, in which gradient reversal layer was used for adversarial training to keep the speech content and speaker information while disentangling the environmental effects. From the experiment results on the public dataset of LJSpeech with four environment effects, the proposed model could complete the specific environment effect conversion and outperforms the baseline methods from the voice conversion task.
【12】 Improving Speech Representation Learning via Speech-level and Phoneme-level Masking Approach
标题：利用语音级和音素级掩蔽方法改进语音表征学习
链接：https://arxiv.org/abs/2210.13805
* 与cs.SD语音【5】为同一篇
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Kexin Zhu,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd., School of Software, Fudan University, Shanghai, China备注：Accepted by MSN2022, The 18th International Conference on Mobility, Sensing and Networking摘要：在语音表示学习中，恢复被屏蔽的语音帧是一个广泛应用的问题。然而，这些模型中的大多数在预训练中使用随机掩蔽。在本研究中，我们提出了两种掩蔽方法：（1）语音级掩蔽，使模型掩蔽比无声段更多的语音段，（2）音素级掩蔽，迫使模型掩蔽音素的整个帧，而不是音素段。我们通过这两种方法对模型进行预训练，并在两个下游任务--音素分类和说话人识别上进行评估。实验结果表明，所提出的掩蔽方法有利于提高语音表示的性能。摘要：Recovering the masked speech frames is widely applied in speech representation learning. However, most of these models use random masking in the pre-training. In this work, we proposed two kinds of masking approaches: (1) speech-level masking, making the model to mask more speech segments than silence segments, (2) phoneme-level masking, forcing the model to mask the whole frames of the phoneme, instead of phoneme pieces. We pre-trained the model via these two approaches, and evaluated on two downstream tasks, phoneme classification and speaker recognition. The experiments demonstrated that the proposed masking approaches are beneficial to improve the performance of speech representation.
【13】 Adapitch: Adaption Multi-Speaker Text-to-Speech Conditioned on Pitch Disentangling with Untranscribed Data
标题：Adapitch：基于未转录数据的基音解缠的自适应多说话人文语转换
链接：https://arxiv.org/abs/2210.13803
* 与cs.SD语音【6】为同一篇
作者：Xulong Zhang,Jianzong Wang,Ning Cheng,Jing Xiao机构：Ping An Technology (Shenzhen) Co., Ltd.备注：Accepted by MSN2022, The 18th International Conference on Mobility, Sensing and Networking摘要：本文提出了一种多说话人TTS方法Adapitch，该方法利用未转录的数据对有监督的模块进行自适应。我们设计了两个自监督模块，分别用非转录数据训练文本编码器和MEL解码器，以增强文本和MEL的表示。为了更好地处理合成语音中的韵律信息，在对语音的基音、文本和说话人进行内容去纠缠的基础上，设计了一个有监督的TTS模块。训练阶段分为两个阶段，首先对文本编码器和梅尔解码器进行无监督预训练和固定，然后对TTS解纠缠进行有监督预训练。实验结果表明，Adaptich算法的图像质量明显优于基线算法。摘要：In this paper, we proposed Adapitch, a multi-speaker TTS method that makes adaptation of the supervised module with untranscribed data. We design two self supervised modules to train the text encoder and mel decoder separately with untranscribed data to enhance the representation of text and mel. To better handle the prosody information in a synthesized voice, a supervised TTS module is designed conditioned on content disentangling of pitch, text, and speaker. The training phase was separated into two parts, pretrained and fixed the text encoder and mel decoder with unsupervised mode, then the supervised mode on the disentanglement of TTS. Experiment results show that the Adaptich achieved much better quality than baseline methods.
【14】 Evaluating and Optimizing Hearing-Aid Self-Fitting Methods using Population Coverage
标题：利用总体覆盖率评价和优化助听器自适应方法
链接：https://arxiv.org/abs/2210.13732
* 与cs.SD语音【7】为同一篇
作者：Dhruv Vyas,Erik Jorgensen,Yu-Hsiang Wu,Octav Chipara机构：Department of Computer Science, University of Iowa, USA, Department of Communication Sciences and Disorders, University of Wisconsin-Madison, USA, Department of Communication Sciences and Disorders, University of Iowa, USA摘要：轻度至中度听力损失的成年人可以使用非处方助听器来治疗他们的听力损失，而费用只是传统听力保健费用的一小部分。这些产品采用了自适应方法，允许最终用户在没有听力学家帮助的情况下配置他们的助听器。自适应方法帮助用户配置增益-频率响应，以控制传入声音的每个频带的放大。本文考虑如何设计有效的自适应方法，以及我们是否可以在不求助于昂贵的用户研究的情况下评价它们设计的某些方面。大多数现有的装配方法提供了各种用户界面，以允许用户从预定的一组预置中选择配置。我们提出一个新的度量来评估基于预置的方法的性能，通过计算它们的人口覆盖率。人口覆盖率估计可以找到其首选配置的用户比例。我们的方法的一个独特方面是概率模型，该模型捕捉用户的独特偏好与其他具有类似听力损失的用户的差异。接下来，我们开发用于确定预设以最大化人口覆盖率的方法。实验结果表明，与基于聚类的方法相比，所提出的算法能够有效地选择少量的预置点，从而提供更高的人口覆盖率。此外，我们可以使用我们的算法来配置基于滑块的方法的增量的数量。摘要：Adults with mild-to-moderate hearing loss can use over-the-counter hearing aids to treat their hearing loss at a fraction of traditional hearing care costs. These products incorporate self-fitting methods that allow end-users to configure their hearing aids without the help of an audiologist. A self-fitting method helps users configure the gain-frequency responses that control the amplification for each frequency band of the incoming sound. This paper considers how to design effective self-fitting methods and whether we may evaluate certain aspects of their design without resorting to expensive user studies. Most existing fitting methods provide various user interfaces to allow users to select a configuration from a predetermined set of presets. We propose a novel metric for evaluating the performance of preset-based approaches by computing their population coverage. The population coverage estimates the fraction of users for which it is possible to find a configuration they prefer. A unique aspect of our approach is a probabilistic model that captures how a user's unique preferences differ from other users with similar hearing loss. Next, we develop methods for determining presets to maximize population coverage. Exploratory results demonstrate that the proposed algorithms can effectively select a small number of presets that provide higher population coverage than clustering-based approaches. Moreover, we may use our algorithms to configure the number of increments for slider-based methods.
【15】 Spectral Clustering-aware Learning of Embeddings for Speaker Diarisation
标题：用于说话人识别的嵌入谱聚类学习
链接：https://arxiv.org/abs/2210.13576
* 与cs.SD语音【8】为同一篇
作者：Evonne P. C. Lee,Guangzhi Sun,Chao Zhang,Philip C. Woodland机构：Cambridge University Engineering Dept., Trumpington St., Cambridge, CB,PZ U.K.备注：Submitted to ICASSP 2023, 5 pages摘要：在说话人识别中，说话人嵌入提取模型的训练损失函数与说话人聚类方法之间存在不匹配的问题。本文提出了一种基于谱聚类的嵌入学习方法（SCALE）来解决这种失配问题。具体地说，除了角原型cal（AP）损失之外，SCALE使用了一种新颖的亲和矩阵损失，其直接最小化从说话人嵌入估计的亲和矩阵与参考之间的误差。SCALE还包括p百分位阈值和高斯模糊作为训练中谱聚类的两个重要超参数。在AMI数据集上的实验表明，当与基于AP损失的说话人嵌入的强基线相比时，使用SCALE获得的说话人嵌入使用oracle分割实现了超过50%的相对说话人错误率降低，并且使用自动分割实现了超过30%的相对日记化错误率降低。摘要：In speaker diarisation, speaker embedding extraction models often suffer from the mismatch between their training loss functions and the speaker clustering method. In this paper, we propose the method of spectral clustering-aware learning of embeddings (SCALE) to address the mismatch. Specifically, besides an angular prototype cal (AP) loss, SCALE uses a novel affinity matrix loss which directly minimises the error between the affinity matrix estimated from speaker embeddings and the reference. SCALE also includes p-percentile thresholding and Gaussian blur as two important hyper-parameters for spectral clustering in training. Experiments on the AMI dataset showed that speaker embeddings obtained with SCALE achieved over 50% relative speaker error rate reductions using oracle segmentation, and over 30% relative diarisation error rate reductions using automatic segmentation when compared to a strong baseline with the AP-loss-based speaker embeddings.
【16】 I see what you hear: a vision-inspired method to localize words
标题：我看到你听到的：一种视觉启发的本地化单词的方法
链接：https://arxiv.org/abs/2210.13567
* 与cs.SD语音【9】为同一篇
作者：Mohammad Samragh,Arnav Kundu,Ting-Yao Hu,Minsik Cho,Aman Chadha,Ashish Shrivastava,Oncel Tuzel,Devang Naik摘要：本文探讨了使用视觉对象检测技术在语音数据中进行单词定位的可能性。在当代文献中已经针对视觉数据对对象检测进行了深入研究。注意，音频可以被解释为1维图像，对象定位技术对于单词定位可以是基本上有用的。基于这种思想，我们提出了一种轻量级的单词检测和本地化解决方案。我们使用边界框回归进行单词定位，这使得我们的模型能够检测给定音频流中关键词的出现、偏移和持续时间。我们使用LibriSpeech进行实验，并训练一个模型来本地化1000个单词。与已有的方法相比，该方法使模型规模减少了94%，F1得分提高了6.5%.摘要：This paper explores the possibility of using visual object detection techniques for word localization in speech data. Object detection has been thoroughly studied in the contemporary literature for visual data. Noting that an audio can be interpreted as a 1-dimensional image, object localization techniques can be fundamentally useful for word localization. Building upon this idea, we propose a lightweight solution for word detection and localization. We use bounding box regression for word localization, which enables our model to detect the occurrence, offset, and duration of keywords in a given audio stream. We experiment with LibriSpeech and train a model to localize 1000 words. Compared to existing work, our method reduces model size by 94%, and improves the F1 score by 6.5\%.
机器翻译，仅供参考
每日语音速递