【1】 DeAR: A Deep-learning-based Audio Re-recording Resilient Watermarking
标题：亲爱的：一种基于深度学习的音频重录抗攻击水印算法
链接：https://arxiv.org/abs/2212.02339
作者：Chang Liu,Jie Zhang,Han Fang,Zehua Ma,Weiming Zhang,Nenghai Yu机构：University of Science and Technology of China, University of Waterloo,National University of Singapore摘要：音频水印被广泛应用于泄漏源追踪。水印的鲁棒性决定了算法的可追踪性。随着数字技术的发展，音频重录（AR）已成为一种高效、隐蔽的窃取机密的手段。AR处理在保留原始信息的同时，会极大地破坏水印信号。这对现阶段的音频水印提出了新的要求，即对AR失真具有鲁棒性。然而，由于AR过程的复杂性，现有的算法都不能有效抵抗AR攻击。针对这一局限性，提出了一种基于深度学习的音频抗重录水印算法DeAR。受基于DNN的图像水印技术的启发，提出了一种音频载体的深度学习框架，基于该框架可以有效地嵌入和提取水印信号。同时，为了抵抗AR攻击，我们对AR过程中产生的失真进行了细致的分析，并设计了相应的失真层来配合所提出的水印框架。实验结果表明，该算法不仅能抵抗常见的电子信道失真，而且能抵抗AR失真。在高质量嵌入（SNR=25.86dB）的前提下，在普通重录距离（20cm）的情况下，该算法可以有效地达到平均98.55%的比特恢复准确率。摘要：Audio watermarking is widely used for leaking source tracing. The robustness of the watermark determines the traceability of the algorithm. With the development of digital technology, audio re-recording (AR) has become an efficient and covert means to steal secrets. AR process could drastically destroy the watermark signal while preserving the original information. This puts forward a new requirement for audio watermarking at this stage, that is, to be robust to AR distortions. Unfortunately, none of the existing algorithms can effectively resist AR attacks due to the complexity of the AR process. To address this limitation, this paper proposes DeAR, a deep-learning-based audio re-recording resistant watermarking. Inspired by DNN-based image watermarking, we pioneer a deep learning framework for audio carriers, based on which the watermark signal can be effectively embedded and extracted. Meanwhile, in order to resist the AR attack, we delicately analyze the distortions that occurred in the AR process and design the corresponding distortion layer to cooperate with the proposed watermarking framework. Extensive experiments show that the proposed algorithm can resist not only common electronic channel distortions but also AR distortions. Under the premise of high-quality embedding (SNR=25.86dB), in the case of a common re-recording distance (20cm), the algorithm can effectively achieve an average bit recovery accuracy of 98.55%.
【2】 End-to-end Recording Device Identification Based on Deep Representation Learning
标题：基于深度表征学习的端到端录音设备识别
链接：https://arxiv.org/abs/2212.02084
作者：Chunyan Zeng,Dongliang Zhu,Zhifeng Wang,Minghu Wu,Wei Xiong,Nan Zhao机构：Correspondence:, Department of Digital Media, Technology, Central China Normal, University, Luoyu Road , Wuhan, China, Full list of author information is, available at the end of the article备注：20 pages, 5 figures, recording device identification摘要：深度学习技术已经在记录设备源识别方面取得了具体成果。记录设备源特征包括空间信息和某些时间信息。然而，基于深度学习的录音设备源识别方法大多仅利用录音设备源特征的空间表示学习，无法充分利用录音设备源信息。为此，为了充分挖掘录音设备来源的空间信息和时间信息，提出了一种基于空间特征信息和时间特征信息融合的端到端录音设备来源识别方法。从特征的角度出发，设计了两种网络来提取录音设备源的时空信息。然后，利用注意机制自适应地分配空间信息和时间信息的权重，得到融合特征。从模型的角度来看，该模型采用端到端的框架，从空间特征和时间特征学习深度表示，并使用深度和浅度损失进行训练，以联合优化网络。将该方法与我们以前的工作和基线系统进行了比较。实验结果表明，在一般情况下，该方法优于我们的前期工作和基线系统。摘要：Deep learning techniques have achieved specific results in recording device source identification. The recording device source features include spatial information and certain temporal information. However, most recording device source identification methods based on deep learning only use spatial representation learning from recording device source features, which cannot make full use of recording device source information. Therefore, in this paper, to fully explore the spatial information and temporal information of recording device source, we propose a new method for recording device source identification based on the fusion of spatial feature information and temporal feature information by using an end-to-end framework. From a feature perspective, we designed two kinds of networks to extract recording device source spatial and temporal information. Afterward, we use the attention mechanism to adaptively assign the weight of spatial information and temporal information to obtain fusion features. From a model perspective, our model uses an end-to-end framework to learn the deep representation from spatial feature and temporal feature and train using deep and shallow loss to joint optimize our network. This method is compared with our previous work and baseline system. The results show that the proposed method is better than our previous work and baseline system under general conditions.
【3】 NBC2: Multichannel Speech Separation with Revised Narrow-band Conformer
标题：NBC2：基于改进窄带整形器的多通道语音分离
链接：https://arxiv.org/abs/2212.02076
作者：Changsheng Quan,Xiaofei Li机构： time domainChangsheng Quan is with Zhejiang University, and Westlake University &Westlake Institute for Advanced Study摘要：This work proposes a multichannel narrow-band speech separation network. In the short-time Fourier transform (STFT) domain, the proposed network processes each frequency independently, and all frequencies use a shared network. For each frequency, the network performs end-to-end speech separation, namely taking as input the STFT coefficients of microphone signals, and predicting the separated STFT coefficients of multiple speakers. The proposed network learns to cluster the frame-wise spatial/steering vectors that belong to different speakers. It is mainly composed of three components. First, a self-attention network. Clustering of spatial vectors shares a similar principle with the self-attention mechanism in the sense of computing the similarity of vectors and then aggregating similar vectors. Second, a convolutional feed-forward network. The convolutional layers are employed for signal smoothing and reverberation processing. Third, a novel hidden-layer normalization method, i.e. group batch normalization (GBN), is especially designed for the proposed narrow-band network to maintain the distribution of hidden units over frequencies. Overall, the proposed network is named NBC2, as it is a revised version of our previous NBC (narrow-band conformer) network. Experiments show that 1) the proposed network outperforms other state-of-the-art methods by a large margin, 2) the proposed GBN improves the signal-to-distortion ratio by 3 dB, relative to other normalization methods, such as batch/layer/group normalization, 3) the proposed narrow-band network is spectrum-agnostic, as it does not learn spectral patterns, and 4) the proposed network is indeed performing frame clustering (demonstrated by the attention maps).
【4】 Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models
标题：用于端到端语音识别模型文本自适应的快速准确的分解神经换能器
链接：https://arxiv.org/abs/2212.01992
作者：Rui Zhao,Jian Xue,Partha Parthasarathy,Veljko Miljanic,Jinyu Li机构：Microsoft Speech Group, Redmond, WA, USA摘要：Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability. However, it is challenging to adapt it with text-only data. Factorized neural transducer (FNT) model was proposed to mitigate this problem. The improved adaptation ability of FNT on text-only adaptation data came at the cost of lowered accuracy compared to the standard neural transducer model. We propose several methods to improve the performance of the FNT model. They are: adding CTC criterion during training, adding KL divergence loss during adaptation, using a pre-trained language model to seed the vocabulary predictor, and an efficient adaptation approach by interpolating the vocabulary predictor with the n-gram language model. A combination of these approaches results in a relative word-error-rate reduction of 9.48\% from the standard FNT model. Furthermore, n-gram interpolation with the vocabulary predictor improves the adaptation speed hugely with satisfactory adaptation performance.
【5】 Speech MOS multi-task learning and rater bias correction
标题：语音MOS多任务学习与评分者偏差校正
链接：https://arxiv.org/abs/2212.01911
作者：Haleh Akrami,Hannes Gamper机构：University of Southern California, Department of Biomedical Engineering, Los Angeles, CA, USA, Audio and Acoustics Research Group, Microsoft Research, Redmond, WA, USA备注：Submitted to ICASSP 2023摘要：感知语音质量是电话会议应用的重要性能度量。平均意见得分（MOS）被标准化用于语音质量的感知评估，并且通过要求收听者对语音样本的质量进行评级来获得。近年来，开发用于盲估计MOS的模型的研究兴趣日益增加。本文提出了一种多任务框架，在训练中加入额外的标签和数据，以提高盲MOS估计模型的性能。实验结果表明，该模型可以通过训练两个不相交的数据集（一个只包含MOS标签，另一个只包含T60和C50标签）来联合估计MOS、混响时间（T60）和清晰度（C50）.此外，我们使用半监督框架来组合训练中的两个MOS数据集，一个仅包含MOS标签（根据ITU-T建议P.808），而另一个包含语音信号、背景噪声和整体质量的单独分数（根据ITU-T建议P.835）。最后，我们给出了解决MOS标签中的个体评价者偏差的初步结果。摘要：Perceptual speech quality is an important performance metric for teleconferencing applications. The mean opinion score (MOS) is standardized for the perceptual evaluation of speech quality and is obtained by asking listeners to rate the quality of a speech sample. Recently, there has been increasing research interest in developing models for estimating MOS blindly. Here we propose a multi-task framework to include additional labels and data in training to improve the performance of a blind MOS estimation model. Experimental results indicate that the proposed model can be trained to jointly estimate MOS, reverberation time (T60), and clarity (C50) by combining two disjoint data sets in training, one containing only MOS labels and the other containing only T60 and C50 labels. Furthermore, we use a semi-supervised framework to combine two MOS data sets in training, one containing only MOS labels (per ITU-T Recommendation P.808), and the other containing separate scores for speech signal, background noise, and overall quality (per ITU-T Recommendation P.835). Finally, we present preliminary results for addressing individual rater bias in the MOS labels.
【6】 Melody transcription via generative pre-training
标题：通过生成性预训练实现旋律转录
链接：https://arxiv.org/abs/2212.01884
作者：Chris Donahue,John Thickstun,Percy Liang备注：Published as a conference paper at ISMIR 2022摘要：尽管旋律在音乐感知中扮演着核心角色，但在音乐信息检索中可靠地检测存在于任意音乐记录中的旋律的音符仍然是一个开放的挑战。旋律转录的一个关键挑战是建立能够处理包含任意数量的乐器合奏和音乐风格的广泛音频的方法-现有策略对于某些旋律乐器或风格效果良好，但不是所有的。为了应对这一挑战，我们利用了Jukebox（Dhariwal等人，2020）的表示，这是一种广泛的音乐音频生成模型，从而相对于传统谱图特征，将旋律转录的性能提高了20 $%。旋律转录的另一个障碍是缺乏训练数据--我们从广泛音乐的众包注释中获得了一个新的数据集，其中包含50 $小时的旋律转录。生成式预训练和该任务的新数据集的组合导致旋律转录相对于最强可用基线的$77$%的更强性能。通过将我们新的旋律转录方法与节拍检测、基调估计和和弦识别解决方案相结合，我们构建了Sheet Sage，这是一个能够直接从音乐音频转录人类可读的前导音的系统。 音频示例可在www.example.com上找到https://chrisdonahue.com/sheetsage，代码可在www.example.com上找到https://github.com/chrisdonahue/sheetsage。摘要：Despite the central role that melody plays in music perception, it remains an open challenge in music information retrieval to reliably detect the notes of the melody present in an arbitrary music recording. A key challenge in melody transcription is building methods which can handle broad audio containing any number of instrument ensembles and musical styles - existing strategies work well for some melody instruments or styles but not all. To confront this challenge, we leverage representations from Jukebox (Dhariwal et al. 2020), a generative model of broad music audio, thereby improving performance on melody transcription by $20$% relative to conventional spectrogram features. Another obstacle in melody transcription is a lack of training data - we derive a new dataset containing $50$ hours of melody transcriptions from crowdsourced annotations of broad music. The combination of generative pre-training and a new dataset for this task results in $77$% stronger performance on melody transcription relative to the strongest available baseline. By pairing our new melody transcription approach with solutions for beat detection, key estimation, and chord recognition, we build Sheet Sage, a system capable of transcribing human-readable lead sheets directly from music audio. Audio examples can be found at https://chrisdonahue.com/sheetsage and code at https://github.com/chrisdonahue/sheetsage .
【7】 Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech
标题：用于改善耳语的自然度、可理解性和语音性的生成模型
链接：https://arxiv.org/abs/2212.01775
作者：Dominik Wagner,Sebastian P. Bayerl,Hector A. Cordourier Maruri,Tobias Bocklet机构：Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, Intel Labs摘要：本工作采用了两种最新的生成模型结构，并评估了它们在耳语音到正常语音转换中的有效性。我们将正常目标语音加入到矢量量化变分自动编码器（VQ-VAE）和MelGANs的训练准则中，从而调节系统从耳声输入中恢复浊音语音。客观和主观质量测量表明，VQ-VAE和MelGAN都可以被修改以执行转换任务。我们发现所提出的方法相对于DiscoGAN基线显著地将Mel倒谱失真（MCD）度量改善了至少25%。主观听力测试表明，与耳语输入语音相比，基于MelGAN的系统显著改善了自然度、可懂度和发声。基于潜在语音表示之间的差异的新的评估度量也表明我们的基于MelGAN的方法产生了相对于基线的改进。摘要：This work adapts two recent architectures of generative models and evaluates their effectiveness for the conversion of whispered speech to normal speech. We incorporate the normal target speech into the training criterion of vector-quantized variational autoencoders (VQ-VAEs) and MelGANs, thereby conditioning the systems to recover voiced speech from whispered inputs. Objective and subjective quality measures indicate that both VQ-VAEs and MelGANs can be modified to perform the conversion task. We find that the proposed approaches significantly improve the Mel cepstral distortion (MCD) metric by at least 25% relative to a DiscoGAN baseline. Subjective listening tests suggest that the MelGAN-based system significantly improves naturalness, intelligibility, and voicing compared to the whispered input speech. A novel evaluation measure based on differences between latent speech representations also indicates that our MelGAN-based approach yields improvements relative to the baseline.
【8】 A dataset for audio-video based vehicle speed estimation
标题：一种基于音视频的车速估计数据集
链接：https://arxiv.org/abs/2212.01651
作者：Slobodan Djukanović,Nikola Bulatović,Ivana Čavor机构：A dataset for audio-video based vehicle speedestimation 1st Slobodan Djukanovi´cFaculty of Electrical EngineeringUniversity of MontenegroPodgorica, me 2nd Nikola Bulatovi´cFaculty of Electrical EngineeringUniversity of MontenegroPodgorica备注：30th Telecommunications Forum TELFOR 2022, Belgrade, Serbia, November 15-16, 2022. 5 pages, 2 figures, 1 table摘要：道路车辆的准确速度估计由于若干原因而很重要。一个是速度限制执行，这是减少交通事故和死亡的一个关键工具。与其他研究领域相比，可用于车辆速度估计的数据集数量仍然非常有限。我们提出了一个数据集的道路音频视频记录的单一车辆通过一个摄像机在已知的速度，保持稳定的车载巡航控制。该数据集包含13辆汽车，选择尽可能多样化的制造商、生产年份、发动机类型、动力和变速器，从而产生总计400美元的注释音频视频记录。该数据集是完全可用的，并打算作为一个公共基准，以促进研究音频视频车辆速度估计。除了数据集之外，我们还提出了一种交叉验证策略，该策略可用于车辆速度估计的机器学习模型中。提出了两种训练-验证数据集分割方法。摘要：Accurate speed estimation of road vehicles is important for several reasons. One is speed limit enforcement, which represents a crucial tool in decreasing traffic accidents and fatalities. Compared with other research areas and domains, the number of available datasets for vehicle speed estimation is still very limited. We present a dataset of on-road audio-video recordings of single vehicles passing by a camera at known speeds, maintained stable by the on-board cruise control. The dataset contains thirteen vehicles, selected to be as diverse as possible in terms of manufacturer, production year, engine type, power and transmission, resulting in a total of $ 400 $ annotated audio-video recordings. The dataset is fully available and intended as a public benchmark to facilitate research in audio-video vehicle speed estimation. In addition to the dataset, we propose a cross-validation strategy which can be used in a machine learning model for vehicle speed estimation. Two approaches to training-validation split of the dataset are proposed.
【9】 UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis
标题：UniSyn：一种端到端的语音合成和歌唱语音合成统一模型
链接：https://arxiv.org/abs/2212.01546
作者：Yi Lei,Shan Yang,Xinsheng Wang,Qicong Xie,Jixun Yao,Lei Xie,Dan Su机构：Northwestern Polytechnical University, Xi’an, China, Tencent AI Lab, China摘要：文本到语音（TTS）和歌声合成（SVS）分别旨在根据文本输入和乐谱生成高质量的说话声和歌声。将TTS和SVS统一到一个系统中对于需要这两种技术的应用程序至关重要。现有的方法通常受到一些限制，这些方法要么依赖于来自同一个人的唱歌和说话数据，要么依赖于多个任务的级联模型。针对这些问题，本文提出了一种简化的TTS和SVS框架--UniSyn。它是一个端到端的统一模型，可以使一个声音说话和唱歌，只有唱歌或说话的数据，从这个人。具体而言，在UniSyn中提出了一种多条件变分自动编码器（MC-VAE），它利用说话人和风格相关（即说话或唱歌）的条件构造两个独立的潜在子空间，以实现灵活的控制。此外，利用有监督的引导VAE和具有Wasserstein距离约束的音色扰动进一步分离说话人音色和风格。在两个说话人和两个歌唱者身上的实验表明，UniSyn算法无需相应的训练数据就能生成自然的说话声和歌唱声。该方法优于现有的端到端语音生成方法，证明了UniSyn的有效性和优越性。摘要：Text-to-speech (TTS) and singing voice synthesis (SVS) aim at generating high-quality speaking and singing voice according to textual input and music scores, respectively. Unifying TTS and SVS into a single system is crucial to the applications requiring both of them. Existing methods usually suffer from some limitations, which rely on either both singing and speaking data from the same person or cascaded models of multiple tasks. To address these problems, a simplified elegant framework for TTS and SVS, named UniSyn, is proposed in this paper. It is an end-to-end unified model that can make a voice speak and sing with only singing or speaking data from this person. To be specific, a multi-conditional variational autoencoder (MC-VAE), which constructs two independent latent sub-spaces with the speaker- and style-related (i.e. speak or sing) conditions for flexible control, is proposed in UniSyn. Moreover, supervised guided-VAE and timbre perturbation with the Wasserstein distance constraint are leveraged to further disentangle the speaker timbre and style. Experiments conducted on two speakers and two singers demonstrate that UniSyn can generate natural speaking and singing voice without corresponding training data. The proposed approach outperforms the state-of-the-art end-to-end voice generation work, which proves the effectiveness and advantages of UniSyn.
【10】 NEAL: An open-source tool for audio annotation
标题：Neal：一个开源的音频标注工具
链接：https://arxiv.org/abs/2212.01457
作者：Anthony Gibbons,Ian Donohue,Courtney E. Gorman,Emma King,Andrew Parnell机构： Hamilton Institute, Department of Mathematics and Statistics, University, Kildare, Ireland, Zoology, School of Natural Sciences, Trinity College Dublin, Dublin, Ireland摘要：被动声学监测广泛应用于生态学、生物多样性和保护研究。通过声学监测收集的数据集通常非常大，并使用人工智能和机器学习模型自动处理，旨在复制领域专家的工作。这些模型是有监督的学习算法，需要在由专家产生的高质量注释上进行训练。由于专家通常是资源有限的，因此需要一种用于注释音频的成本有效的过程来最大限度地利用数据。提出了一个开放源代码的交互式音频数据标注工具NEAL（Nature+Energy Audio Labeller）.该工具使用R和相关的Shiny框架构建，提供了一个反应式环境，用户可以在其中快速注释音频文件，并调整自动更改用户界面相应元素的设置。该应用程序的设计目标是让专家观鸟者和公民科学家为声学注释项目做出贡献。R编程在生物声学领域的普及性和灵活性意味着，可以针对其他鸟类标签数据集，甚至通用音频标签任务修改Shiny应用程序。我们通过标记从爱尔兰各地风电场收集的数据来演示该应用程序。摘要：Passive acoustic monitoring is used widely in ecology, biodiversity, and conservation studies. Data sets collected via acoustic monitoring are often extremely large and built to be processed automatically using Artificial Intelligence and Machine learning models, which aim to replicate the work of domain experts. These models, being supervised learning algorithms, need to be trained on high quality annotations produced by experts. Since the experts are often resource-limited, a cost-effective process for annotating audio is needed to get maximal use out of the data. We present an open-source interactive audio data annotation tool, NEAL (Nature+Energy Audio Labeller). Built using R and the associated Shiny framework, the tool provides a reactive environment where users can quickly annotate audio files and adjust settings that automatically change the corresponding elements of the user interface. The app has been designed with the goal of having both expert birders and citizen scientists contribute to acoustic annotation projects. The popularity and flexibility of R programming in bioacoustics means that the Shiny app can be modified for other bird labelling data sets, or even to generic audio labelling tasks. We demonstrate the app by labelling data collected from wind farm sites across Ireland.
【11】 LMEC: Learnable Multiplicative Absolute Position Embedding Based Conformer for Speech Recognition
标题：LMEC：基于可学习乘性绝对位置嵌入的语音识别整形器
链接：https://arxiv.org/abs/2212.02099
作者：Yuguang Yang,Yu Pan,Jingjing Yin,Heng Lu机构：Ximalaya Inc., ShangHai, China, University of Alberta, Edmonton, Canada摘要：提出了一种基于可学习乘法绝对位置嵌入的构造器（LMEC）。它包含一个核化的线性注意力（LA）模块，称为LMLA，以解决长序列语音识别的耗时问题，以及FFN结构的替代方案。首先，采用ELU函数作为LA模块的核函数.其次，提出了一种新的基于可学习乘法绝对位置嵌入（LM-APE）的权重重分配机制，降低了softmax自注意的二次时空复杂度.第三，我们使用门控线性单元（GLU）来代替前馈网络（FFN）以获得更好的性能。在公开的LibriSpeech数据集上进行了大量的实验。与Conformer模型和cosFormer类型的线性注意相比，本文方法在测试-他者模型上的误词率提高了0.63%，在LA模型上的推理速度分别提高了13%（左积）和33%（右积）.摘要：This paper proposes a Learnable Multiplicative absolute position Embedding based Conformer (LMEC). It contains a kernelized linear attention (LA) module called LMLA to solve the time-consuming problem for long sequence speech recognition as well as an alternative to the FFN structure. First, the ELU function is adopted as the kernel function of our proposed LA module. Second, we propose a novel Learnable Multiplicative Absolute Position Embedding (LM-APE) based re-weighting mechanism that can reduce the well-known quadratic temporal-space complexity of softmax self-attention. Third, we use Gated Linear Units (GLU) to substitute the Feed Forward Network (FFN) for better performance. Extensive experiments have been conducted on the public LibriSpeech datasets. Compared to the Conformer model with cosFormer style linear attention, our proposed method can achieve up to 0.63% word-error-rate improvement on test-other and improve the inference speed by up to 13% (left product) and 33% (right product) on the LA module.
【12】 Towards Generating Diverse Audio Captions via Adversarial Training
标题：通过对抗性训练生成多样化的音频字幕
链接：https://arxiv.org/abs/2212.02033
作者：Xinhao Mei,Xubo Liu,Jianyuan Sun,Mark D. Plumbley,Wenwu Wang摘要：自动音频字幕是一个跨模态翻译任务，用于用自然语言句子描述音频剪辑的内容。这一任务日益受到重视，近年来取得了实质性进展。由现有模型生成的字幕通常忠实于音频剪辑的内容，然而，这些机器生成的字幕通常是确定性的（例如，为给定音频剪辑生成固定字幕）、简单（例如，使用普通单词和简单语法），和通用的（例如，为相似的音频剪辑生成相同的字幕）。当人们被要求描述音频剪辑的内容时，不同的人倾向于关注不同的声音事件，并且使用不同的词语和语法从各个方面对音频剪辑进行不同的描述。我们相信音频字幕系统应该能够为固定的音频剪辑或跨类似的音频剪辑生成不同的字幕。为此，提出一种基于条件生成对抗网络（C-GAN）的对抗训练框架，以提高音频字幕系统的多样性。字幕生成器和两个混合鉴别器竞争并被联合学习，其中字幕生成器可以是用于生成字幕的任何标准编码器-解码器字幕模型，并且混合鉴别器从不同的标准（诸如它们的自然性和语义）来评估所生成的字幕。我们在Clotho数据集上进行了实验。实验结果表明，与现有方法相比，本文提出的模型能够生成具有更好多样性的字幕。摘要：Automated audio captioning is a cross-modal translation task for describing the content of audio clips with natural language sentences. This task has attracted increasing attention and substantial progress has been made in recent years. Captions generated by existing models are generally faithful to the content of audio clips, however, these machine-generated captions are often deterministic (e.g., generating a fixed caption for a given audio clip), simple (e.g., using common words and simple grammar), and generic (e.g., generating the same caption for similar audio clips). When people are asked to describe the content of an audio clip, different people tend to focus on different sound events and describe an audio clip diversely from various aspects using distinct words and grammar. We believe that an audio captioning system should have the ability to generate diverse captions, either for a fixed audio clip, or across similar audio clips. To this end, we propose an adversarial training framework based on a conditional generative adversarial network (C-GAN) to improve diversity of audio captioning systems. A caption generator and two hybrid discriminators compete and are learned jointly, where the caption generator can be any standard encoder-decoder captioning model used to generate captions, and the hybrid discriminators assess the generated captions from different criteria, such as their naturalness and semantics. We conduct experiments on the Clotho dataset. The results show that our proposed model can generate captions with better diversity as compared to state-of-the-art methods.
【13】 Evince the artifacts of Spoof Speech by blending Vocal Tract and Voice Source Features
标题：融合声道和声源特征的假话伪影检测
链接：https://arxiv.org/abs/2212.02013
作者：Tadipatri Uday Kiran Reddy,Sahukari Chaitanya Varun,Kota Pranav Kumar Sankala Sreekanth,Kodukula Sri Rama Murty机构：Indian Institute of Technology Hyderabad, Telangana, India摘要：随着合成语音生成技术的快速发展，如何区分自然语音和欺骗语音成为研究领域的热点。这些合成信号的识别不仅对于前沿的分类模型来说是一项困难的任务，而且对于人类本身也是一项困难的任务。为了防止潜在的不利影响，检测欺骗信号变得至关重要。从取证的角度来看，预测生成它们的算法以识别伪造者也很重要。这需要了解欺骗信号的基本属性，欺骗信号用作合成器的签名。本研究通过使用声道系统（\textit{VTS}）和声源（\textit{VS}）特征，强调在识别语音信号真实性方面至关重要的语音信号片段。 本文提出了一种检测欺骗信号并识别相应语音生成算法的系统。算法分类准确率达到99.58%。通过实验，我们发现基于VS特征的系统更关注音素的转换，而基于VTS特征的系统更关注语音信号的平稳段。我们在基于VS和基于VTS的系统上执行模型融合技术，以利用互补信息来开发鲁棒的分类器。在分析混淆图时，我们发现WaveRNN的分类较差，描绘了更多的自然性。另一方面，我们确定了波形级联和神经源滤波器等合成器的分类精度最高。这项工作的实际意义可以帮助来自取证（利用伪影）和语音社区（减轻伪影）的研究人员。摘要：With the rapid advancement in synthetic speech generation technologies, great interest in differentiating spoof speech from the natural speech is emerging in the research community. The identification of these synthetic signals is a difficult task not only for the cutting-edge classification models but also for humans themselves. To prevent potential adverse effects, it becomes crucial to detect spoof signals. From a forensics perspective, it is also important to predict the algorithm which generated them to identify the forger. This needs an understanding of the underlying attributes of spoof signals which serve as a signature for the synthesizer. This study emphasizes the segments of speech signals critical in identifying their authenticity by utilizing the Vocal Tract System(\textit{VTS}) and Voice Source(\textit{VS}) features. In this paper, we propose a system that detects spoof signals as well as identifies the corresponding speech-generating algorithm. We achieve 99.58\% in algorithm classification accuracy. From experiments, we found that a VS feature-based system gives more attention to the transition of phonemes, while, a VTS feature-based system gives more attention to stationary segments of speech signals. We perform model fusion techniques on the VS-based and VTS-based systems to exploit the complementary information to develop a robust classifier. Upon analyzing the confusion plots we found that WaveRNN is poorly classified depicting more naturalness. On the other hand, we identified that synthesizer like Waveform Concatenation, and Neural Source Filter is classified with the highest accuracy. Practical implications of this work can aid researchers from both forensics (leverage artifacts) and the speech communities (mitigate artifacts).
【14】 Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research
标题：悲剧说话者：用于视听机器学习研究的莎士比亚声场和光场数据集
链接：https://arxiv.org/abs/2212.01892
作者：Davide Berghi,Marco Volino,Philip J. B. Jackson机构：CVSSP, University of Surrey, Guildford, U.K.摘要：3D视听制作旨在为消费者提供身临其境的互动体验。然而，真实地再现真实世界的3D场景仍然是一项具有挑战性的任务。这部分是由于缺乏可用于这方面视听研究的数据集。在现有的多视点数据集中，伴随的音频被忽略。同样，用于空间音频研究的数据集主要提供单峰内容，当包括视觉数据时，质量远远不能满足标准生产需求。我们介绍“悲剧的说话者”，一个视听数据集，包括从“罗密欧与朱丽叶”戏剧摘录捕捉麦克风阵列和多个共置摄像机的光场视频。Tragic Talkers为基于对象的媒体（OBM）制作提供了理想的内容。它旨在涵盖各种传统的谈话场景，如独白、两人对话以及具有相当大的移动和遮挡的互动，产生从总共22个不同视角和两个16元件麦克风阵列捕获的30个序列。此外，我们还提供了语音活动标签、每个摄像机视图的2D人脸边界框、2D姿势检测关键点、演员嘴部的3D跟踪数据以及对话转录。我们相信社区将受益于这一数据集，因为它可以协助多学科研究。讨论了数据集的可能用途。摘要：3D audio-visual production aims to deliver immersive and interactive experiences to the consumer. Yet, faithfully reproducing real-world 3D scenes remains a challenging task. This is partly due to the lack of available datasets enabling audio-visual research in this direction. In most of the existing multi-view datasets, the accompanying audio is neglected. Similarly, datasets for spatial audio research primarily offer unimodal content, and when visual data is included, the quality is far from meeting the standard production needs. We present "Tragic Talkers", an audio-visual dataset consisting of excerpts from the "Romeo and Juliet" drama captured with microphone arrays and multiple co-located cameras for light-field video. Tragic Talkers provides ideal content for object-based media (OBM) production. It is designed to cover various conventional talking scenarios, such as monologues, two-people conversations, and interactions with considerable movement and occlusion, yielding 30 sequences captured from a total of 22 different points of view and two 16-element microphone arrays. Additionally, we provide voice activity labels, 2D face bounding boxes for each camera view, 2D pose detection keypoints, 3D tracking data of the mouth of the actors, and dialogue transcriptions. We believe the community will benefit from this dataset as it can assist multidisciplinary research. Possible uses of the dataset are discussed.
【15】 Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data
标题：利用辅助语音和文本数据改进端到端语音翻译
链接：https://arxiv.org/abs/2212.01778
作者：Yuhao Zhang,Chen Xu,Bojie Hu,Chunliang Zhang,Tong Xiao,Jingbo Zhu机构： School of Computer Science and Engineering, Northeastern University, Shenyang, China, NiuTrans Research, Shenyang, China, Tencent Minority-Mandarin Translation, Beijing, China摘要：We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-language text). Thus, the speech translation model can learn from both unlabeled and labeled data, especially when the source-language text data is abundant. Beyond this, we present a denoising method to build a robust text encoder that can deal with both normal and noisy text data. Our system sets new state-of-the-arts on the MuST-C En-De, En-Fr, and LibriSpeech En-Fr tasks.
【16】 A subjective study of the perceptual acceptability of audio-video desynchronization in sports videos
标题：体育视频音视频去同步化知觉可接受性的主观研究
链接：https://arxiv.org/abs/2212.01686
备注：6 pages; for associated code see this https URL摘要：This paper presents the results of a study conducted on the perceptual acceptability of audio-video desynchronization for sports videos. The study was conducted with 45 videos generated by applying 8 audio-video offsets on 5 source contents. 20 subjects participated in the study. The results show that humans are more sensitive to audio-video offset errors for speech stimuli, and the complex events that occur in sports broadcasts have higher thresholds of acceptability. This suggests the tuning of audio-video synchronization requirements in broadcasting to the content of the broadcast.
【17】 Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models
标题：基于自监督语音模型的ASR无监督精调数据选择
链接：https://arxiv.org/abs/2212.01661
作者：Reem Gody,David Harwath机构：The University of Texas at Austin摘要：自监督学习（Self-supervised learning，SSL）能够在只有少量转录语音数据的情况下，利用未标记数据来提高自动语音识别（Automatic Speech Recognition，ASR）模型的性能。然而，这提出了可用的未标记数据的哪个子集应该被选择用于转录的问题。我们的工作研究了不同的无监督数据选择技术，用于在有限的转录预算下微调HuBERT模型。我们研究了说话人多样性、性别偏见和话题多样性对下游ASR性能的影响。我们还设计了两种用于无监督数据选择的新颖技术：基于预训练丢失的数据选择和字节对编码的聚类单元（PBPE）的复杂性，并且我们展示了这些技术与纯随机数据选择的比较。最后，我们分析了所选微调子集的固有特性之间的相关性以及这些特性如何与最终的误字率相关。我们证明了令牌多样性、说话者多样性和主题多样性在获得最佳WER性能方面的重要性。摘要：Self-supervised learning (SSL) has been able to leverage unlabeled data to boost the performance of automatic speech recognition (ASR) models when we have access to only a small amount of transcribed speech data. However, this raises the question of which subset of the available unlabeled data should be selected for transcription. Our work investigates different unsupervised data selection techniques for fine-tuning the HuBERT model under a limited transcription budget. We investigate the impact of speaker diversity, gender bias, and topic diversity on the downstream ASR performance. We also devise two novel techniques for unsupervised data selection: pre-training loss based data selection and the perplexity of byte pair encoded clustered units (PBPE) and we show how these techniques compare to pure random data selection. Finally, we analyze the correlations between the inherent characteristics of the selected fine-tuning subsets as well as how these characteristics correlate with the resultant word error rate. We demonstrate the importance of token diversity, speaker diversity, and topic diversity in achieving the best performance in terms of WER.
【18】 Can we still use PEAQ? A Performance Analysis of the ITU Standard for the Objective Assessment of Perceived Audio Quality
标题：我们还能用PEAQ吗？ITU感知音频质量客观评价标准的性能分析
链接：https://arxiv.org/abs/2212.01467
作者：Pablo M. Delgado,Jürgen Herre机构：International Audio Laboratories Erlangen ‡, Erlangen, Germany, J¨urgen Herre摘要：如国际电信联盟（ITU）建议ITU-R BS.1387中所描述的音频质量的感知评估（PEAQ）方法已广泛用于计算地估计感知编码音频信号的质量，而不需要广泛的主观收听测试。然而，在其标准化结束之后，许多报告强调了该方案的明显限制，特别是涉及利用诸如带宽扩展或参数多信道编码的较新技术编码的信号。到目前为止，ITU还没有标准化用于测量语音和音频信号两者的质量的其它方法。因此，对这些限制的原因进行进一步调查将有利于对所述方案进行可能的更新。我们的实验结果表明，PEAQ的干扰响度模型的性能仍然与其他最先进的客观测量一样好（有时甚至更好），尽管性能随退化信号内容的类型（即语音或音乐）而变化。这一发现证明了需要改进认知模型。此外，结果表明基于更新的训练数据的模型输出值（MOV）到PEAQ的失真指数（DI）的更新映射可以极大地提高性能。最后，通过与其他系统的比较，提出了改进PEAQ的建议。摘要：The Perceptual Evaluation of Audio Quality (PEAQ) method as described in the International Telecommunication Union (ITU) recommendation ITU-R BS.1387 has been widely used for computationally estimating the quality of perceptually coded audio signals without the need for extensive subjective listening tests. However, many reports have highlighted clear limitations of the scheme after the end of its standardization, particularly involving signals coded with newer technologies such as bandwidth extension or parametric multi-channel coding. Until now, no other method for measuring the quality of both speech and audio signals has been standardized by the ITU. Therefore, a further investigation of the causes for these limitations would be beneficial to a possible update of said scheme. Our experimental results indicate that the performance of PEAQ's model of disturbance loudness is still as good as (and sometimes superior to) other state-of-the-art objective measures, albeit with varying performance depending on the type of degraded signal content (i.e. speech or music). This finding evidences the need for an improved cognitive model. In addition, results indicate that an updated mapping of Model Output Values (MOVs) to PEAQ's Distortion Index (DI) based on newer training data can greatly improve performance. Finally, some suggestions for the improvement of PEAQ are provided based on the reported results and comparison to other systems.
【19】 Objective Assessment of Spatial Audio Quality using Directional Loudness Maps
标题：基于方向性响度图的空间音质客观评价
链接：https://arxiv.org/abs/2212.01451
作者：Pablo M. Delgado,Jürgen Herre机构：⋆ International Audio Laboratories Erlangen ‡, Am Wolfsmantel , Erlangen, Germany, † Fraunhofer IIS, Am Wolfsmantel , Erlangen, Germany摘要：本工作引入了从立体声/双耳音频信号中提取的特征，旨在表示处理后的空间听觉场景中感知质量退化的度量。特征提取技术基于简化的立体声信号模型，该模型考虑了使用幅度平移（AP）技术朝向立体声场中的给定方向定位的听觉事件。我们在短时傅立叶变换域中针对给定AP值将立体声信号分解成一组定向信号，并且计算它们的总响度以形成定向响度表示或映射。然后，我们比较参考信号和劣化版本的方向响度图，以导出旨在描述在听力测试中报告的相关感知劣化分数的失真度量。然后，在广泛的收听测试数据库上测试该度量，其中立体声信号由使用非波形保留技术（例如，带宽扩展和联合立体声编码）的现有感知音频编解码器处理，已知所述非波形保留技术对现有质量预测器提出挑战。结果表明，所导出的失真度量可以作为对现有自动感知质量评估算法的扩展而被并入，以用于改进对空间编码音频信号的预测。摘要：This work introduces a feature extracted from stereophonic/binaural audio signals aiming to represent a measure of perceived quality degradation in processed spatial auditory scenes. The feature extraction technique is based on a simplified stereo signal model considering auditory events positioned towards a given direction in the stereo field using amplitude panning (AP) techniques. We decompose the stereo signal into a set of directional signals for given AP values in the Short-Time Fourier Transform domain and calculate their overall loudness to form a directional loudness representation or maps. Then, we compare directional loudness maps of a reference signal and a deteriorated version to derive a distortion measure aiming to describe the associated perceived degradation scores reported in listening tests. The measure is then tested on an extensive listening test database with stereo signals processed by state-of-the-art perceptual audio codecs using non waveform-preserving techniques such as bandwidth extension and joint stereo coding, known for presenting a challenge to existing quality predictors. Results suggest that the derived distortion measure can be incorporated as an extension to existing automated perceptual quality assessment algorithms for improving prediction on spatially coded audio signals.
【20】 Investigations on the Influence of Combined Inter-Aural Cue Distortions on Overall Audio Quality
标题：组合耳间线索失真对整体音质影响的研究
链接：https://arxiv.org/abs/2212.01427
作者：Pablo M. Delgado,Jürgen Herre摘要：开发能够预测感知编码信号的音频质量以避免在开发时间期间的大量收听测试的成本的算法是相当有兴趣的。虽然有许多用于预测具有单声道（音色）失真的信号的感知质量的已建立算法可用（PEAQ、POLQA），但是预测立体声和多声道空间信号的质量降级仍然被认为是一个挑战。由空间失真引起的音频质量劣化通常根据公知的耳间线索失真度量来测量，诸如耳间电平差失真（ILDD）、耳间时间差失真（ITDD）和耳间互相关失真（IACCD）。然而，它们的相互作用在多大程度上影响了复杂信号中的整体音频质量下降，例如在多刺激测试中所表达的。我们提出了一种系统方法，其在一组代表性信号上引入空间失真的受控组合，并通过分析所述信号上的听力测试分数来评估它们对整体感知质量降级的影响。从这项研究中，我们得出了设计有意义的失真措施的指导方针，考虑到耳间线索失真的相互作用。摘要：There is a considerable interest in developing algorithms that can predict audio quality of perceptually coded signals to avoid the cost of extensive listening tests during development time. While many established algorithms for predicting the perceived quality of signals with monaural (timbral) distortions are available (PEAQ, POLQA), predicting the quality degradation of stereo and multi-channel spatial signals is still considered a challenge. Audio quality degradation arising from spatial distortions is usually measured in terms of well known inter-aural cue distortion measures such as Inter-aural Level Difference Distortions (ILDD), Inter-aural Time Difference Distortions (ITDD) and Inter-aural Cross Correlation Distortions (IACCD). However, the extent to which their interaction influences the overall audio quality degradation in complex signals as expressed - for example - in a multiple stimuli test is not yet thoroughly studied. We propose a systematic approach that introduces controlled combinations of spatial distortions on a representative set of signals and evaluates their influence on overall perceived quality degradation by analyzing listening test scores over said signals. From this study we derive guidelines for designing meaningful distortion measures that consider inter-aural cue distortion interactions.
【21】 Continual Learning for On-Device Speech Recognition using Disentangled Conformers
标题：基于解缠构象的设备上语音识别的连续学习
链接：https://arxiv.org/abs/2212.01393
作者：Anuj Diwan,Ching-Feng Yeh,Wei-Ning Hsu,Paden Tomasello,Eunsol Choi,David Harwath,Abdelrahman Mohamed机构： University of Texas at Austin, Meta Inc.备注：8 pages, 2 figures. Submitted to ICASSP 2023摘要：Automatic speech recognition research focuses on training and evaluating on static datasets. Yet, as speech models are increasingly deployed on personal devices, such models encounter user-specific distributional shifts. To simulate this real-world scenario, we introduce LibriContinual, a continual learning benchmark for speaker-specific domain adaptation derived from LibriVox audiobooks, with data corresponding to 118 individual speakers and 6 train splits per speaker of different sizes. Additionally, current speech recognition models and continual learning algorithms are not optimized to be compute-efficient. We adapt a general-purpose training algorithm NetAug for ASR and create a novel Conformer variant called the DisConformer (Disentangled Conformer). This algorithm produces ASR models consisting of a frozen 'core' network for general-purpose use and several tunable 'augment' networks for speaker-specific tuning. Using such models, we propose a novel compute-efficient continual learning algorithm called DisentangledCL. Our experiments show that the DisConformer models significantly outperform baselines on general ASR i.e. LibriSpeech (15.58% rel. WER on test-other). On speaker-specific LibriContinual they significantly outperform trainable-parameter-matched baselines (by 20.65% rel. WER on test) and even match fully finetuned baselines in some settings.
【1】 LMEC: Learnable Multiplicative Absolute Position Embedding Based Conformer for Speech Recognition
标题：LMEC：基于可学习乘性绝对位置嵌入的语音识别整形器
链接：https://arxiv.org/abs/2212.02099
*与cs.SD语音【11】为同一篇
作者：Yuguang Yang,Yu Pan,Jingjing Yin,Heng Lu机构：Ximalaya Inc., ShangHai, China, University of Alberta, Edmonton, Canada摘要：提出了一种基于可学习乘法绝对位置嵌入的构造器（LMEC）。它包含一个核化的线性注意力（LA）模块，称为LMLA，以解决长序列语音识别的耗时问题，以及FFN结构的替代方案。首先，采用ELU函数作为LA模块的核函数.其次，提出了一种新的基于可学习乘法绝对位置嵌入（LM-APE）的权重重分配机制，降低了softmax自注意的二次时空复杂度.第三，我们使用门控线性单元（GLU）来代替前馈网络（FFN）以获得更好的性能。在公开的LibriSpeech数据集上进行了大量的实验。与Conformer模型和cosFormer类型的线性注意相比，本文方法在测试-他者模型上的误词率提高了0.63%，在LA模型上的推理速度分别提高了13%（左积）和33%（右积）.摘要：This paper proposes a Learnable Multiplicative absolute position Embedding based Conformer (LMEC). It contains a kernelized linear attention (LA) module called LMLA to solve the time-consuming problem for long sequence speech recognition as well as an alternative to the FFN structure. First, the ELU function is adopted as the kernel function of our proposed LA module. Second, we propose a novel Learnable Multiplicative Absolute Position Embedding (LM-APE) based re-weighting mechanism that can reduce the well-known quadratic temporal-space complexity of softmax self-attention. Third, we use Gated Linear Units (GLU) to substitute the Feed Forward Network (FFN) for better performance. Extensive experiments have been conducted on the public LibriSpeech datasets. Compared to the Conformer model with cosFormer style linear attention, our proposed method can achieve up to 0.63% word-error-rate improvement on test-other and improve the inference speed by up to 13% (left product) and 33% (right product) on the LA module.
【2】 Towards Generating Diverse Audio Captions via Adversarial Training
标题：通过对抗性训练生成多样化的音频字幕
链接：https://arxiv.org/abs/2212.02033
*与cs.SD语音【12】为同一篇
作者：Xinhao Mei,Xubo Liu,Jianyuan Sun,Mark D. Plumbley,Wenwu Wang摘要：自动音频字幕是一个跨模态翻译任务，用于用自然语言句子描述音频剪辑的内容。这一任务日益受到重视，近年来取得了实质性进展。由现有模型生成的字幕通常忠实于音频剪辑的内容，然而，这些机器生成的字幕通常是确定性的（例如，为给定音频剪辑生成固定字幕）、简单（例如，使用普通单词和简单语法），和通用的（例如，为相似的音频剪辑生成相同的字幕）。当人们被要求描述音频剪辑的内容时，不同的人倾向于关注不同的声音事件，并且使用不同的词语和语法从各个方面对音频剪辑进行不同的描述。我们相信音频字幕系统应该能够为固定的音频剪辑或跨类似的音频剪辑生成不同的字幕。为此，提出一种基于条件生成对抗网络（C-GAN）的对抗训练框架，以提高音频字幕系统的多样性。字幕生成器和两个混合鉴别器竞争并被联合学习，其中字幕生成器可以是用于生成字幕的任何标准编码器-解码器字幕模型，并且混合鉴别器从不同的标准（诸如它们的自然性和语义）来评估所生成的字幕。我们在Clotho数据集上进行了实验。实验结果表明，与现有方法相比，本文提出的模型能够生成具有更好多样性的字幕。摘要：Automated audio captioning is a cross-modal translation task for describing the content of audio clips with natural language sentences. This task has attracted increasing attention and substantial progress has been made in recent years. Captions generated by existing models are generally faithful to the content of audio clips, however, these machine-generated captions are often deterministic (e.g., generating a fixed caption for a given audio clip), simple (e.g., using common words and simple grammar), and generic (e.g., generating the same caption for similar audio clips). When people are asked to describe the content of an audio clip, different people tend to focus on different sound events and describe an audio clip diversely from various aspects using distinct words and grammar. We believe that an audio captioning system should have the ability to generate diverse captions, either for a fixed audio clip, or across similar audio clips. To this end, we propose an adversarial training framework based on a conditional generative adversarial network (C-GAN) to improve diversity of audio captioning systems. A caption generator and two hybrid discriminators compete and are learned jointly, where the caption generator can be any standard encoder-decoder captioning model used to generate captions, and the hybrid discriminators assess the generated captions from different criteria, such as their naturalness and semantics. We conduct experiments on the Clotho dataset. The results show that our proposed model can generate captions with better diversity as compared to state-of-the-art methods.
【3】 Evince the artifacts of Spoof Speech by blending Vocal Tract and Voice Source Features
标题：融合声道和声源特征的假话伪影检测
链接：https://arxiv.org/abs/2212.02013
*与cs.SD语音【13】为同一篇
作者：Tadipatri Uday Kiran Reddy,Sahukari Chaitanya Varun,Kota Pranav Kumar Sankala Sreekanth,Kodukula Sri Rama Murty机构：Indian Institute of Technology Hyderabad, Telangana, India摘要：随着合成语音生成技术的快速发展，如何区分自然语音和欺骗语音成为研究领域的热点。这些合成信号的识别不仅对于前沿的分类模型来说是一项困难的任务，而且对于人类本身也是一项困难的任务。为了防止潜在的不利影响，检测欺骗信号变得至关重要。从取证的角度来看，预测生成它们的算法以识别伪造者也很重要。这需要了解欺骗信号的基本属性，欺骗信号用作合成器的签名。本研究通过使用声道系统（\textit{VTS}）和声源（\textit{VS}）特征，强调在识别语音信号真实性方面至关重要的语音信号片段。 本文提出了一种检测欺骗信号并识别相应语音生成算法的系统。算法分类准确率达到99.58%。通过实验，我们发现基于VS特征的系统更关注音素的转换，而基于VTS特征的系统更关注语音信号的平稳段。我们在基于VS和基于VTS的系统上执行模型融合技术，以利用互补信息来开发鲁棒的分类器。在分析混淆图时，我们发现WaveRNN的分类较差，描绘了更多的自然性。另一方面，我们确定了波形级联和神经源滤波器等合成器的分类精度最高。这项工作的实际意义可以帮助来自取证（利用伪影）和语音社区（减轻伪影）的研究人员。摘要：With the rapid advancement in synthetic speech generation technologies, great interest in differentiating spoof speech from the natural speech is emerging in the research community. The identification of these synthetic signals is a difficult task not only for the cutting-edge classification models but also for humans themselves. To prevent potential adverse effects, it becomes crucial to detect spoof signals. From a forensics perspective, it is also important to predict the algorithm which generated them to identify the forger. This needs an understanding of the underlying attributes of spoof signals which serve as a signature for the synthesizer. This study emphasizes the segments of speech signals critical in identifying their authenticity by utilizing the Vocal Tract System(\textit{VTS}) and Voice Source(\textit{VS}) features. In this paper, we propose a system that detects spoof signals as well as identifies the corresponding speech-generating algorithm. We achieve 99.58\% in algorithm classification accuracy. From experiments, we found that a VS feature-based system gives more attention to the transition of phonemes, while, a VTS feature-based system gives more attention to stationary segments of speech signals. We perform model fusion techniques on the VS-based and VTS-based systems to exploit the complementary information to develop a robust classifier. Upon analyzing the confusion plots we found that WaveRNN is poorly classified depicting more naturalness. On the other hand, we identified that synthesizer like Waveform Concatenation, and Neural Source Filter is classified with the highest accuracy. Practical implications of this work can aid researchers from both forensics (leverage artifacts) and the speech communities (mitigate artifacts).
【4】 Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research
标题：悲剧说话者：用于视听机器学习研究的莎士比亚声场和光场数据集
链接：https://arxiv.org/abs/2212.01892
*与cs.SD语音【14】为同一篇
作者：Davide Berghi,Marco Volino,Philip J. B. Jackson机构：CVSSP, University of Surrey, Guildford, U.K.摘要：3D视听制作旨在为消费者提供身临其境的互动体验。然而，真实地再现真实世界的3D场景仍然是一项具有挑战性的任务。这部分是由于缺乏可用于这方面视听研究的数据集。在现有的多视点数据集中，伴随的音频信息被忽略。同样，用于空间音频研究的数据集主要提供单峰内容，当包括视觉数据时，质量远远不能满足标准生产需求。我们介绍“悲剧的说话者”，一个视听数据集，包括从“罗密欧与朱丽叶”戏剧摘录捕捉麦克风阵列和多个共置摄像机的光场视频。Tragic Talkers为基于对象的媒体（OBM）制作提供了理想的内容。它旨在涵盖各种传统的谈话场景，如独白、两人对话以及具有相当大的移动和遮挡的互动，产生从总共22个不同视角和两个16元件麦克风阵列捕获的30个序列。此外，我们还提供了语音活动标签、每个摄像机视图的2D人脸边界框、2D姿势检测关键点、演员嘴部的3D跟踪数据以及对话转录。我们相信社区将受益于这一数据集，因为它可以协助多学科研究。讨论了数据集的可能用途。摘要：3D audio-visual production aims to deliver immersive and interactive experiences to the consumer. Yet, faithfully reproducing real-world 3D scenes remains a challenging task. This is partly due to the lack of available datasets enabling audio-visual research in this direction. In most of the existing multi-view datasets, the accompanying audio is neglected. Similarly, datasets for spatial audio research primarily offer unimodal content, and when visual data is included, the quality is far from meeting the standard production needs. We present "Tragic Talkers", an audio-visual dataset consisting of excerpts from the "Romeo and Juliet" drama captured with microphone arrays and multiple co-located cameras for light-field video. Tragic Talkers provides ideal content for object-based media (OBM) production. It is designed to cover various conventional talking scenarios, such as monologues, two-people conversations, and interactions with considerable movement and occlusion, yielding 30 sequences captured from a total of 22 different points of view and two 16-element microphone arrays. Additionally, we provide voice activity labels, 2D face bounding boxes for each camera view, 2D pose detection keypoints, 3D tracking data of the mouth of the actors, and dialogue transcriptions. We believe the community will benefit from this dataset as it can assist multidisciplinary research. Possible uses of the dataset are discussed.
【5】 Improving End-to-end Speech Translation by Leveraging Auxiliary Speech and Text Data
标题：利用辅助语音和文本数据改进端到端语音翻译
链接：https://arxiv.org/abs/2212.01778
*与cs.SD语音【15】为同一篇
作者：Yuhao Zhang,Chen Xu,Bojie Hu,Chunliang Zhang,Tong Xiao,Jingbo Zhu机构： School of Computer Science and Engineering, Northeastern University, Shenyang, China, NiuTrans Research, Shenyang, China, Tencent Minority-Mandarin Translation, Beijing, China摘要：我们提出了一种将文本编码器引入预先训练的端到端语音翻译系统的方法。它增强了适应一种模态的能力（即，源语言语音）到另一个（即，源语言文本）。因此，语音翻译模型可以从未标记和标记数据中学习，特别是当源语言文本数据丰富时。除此之外，我们提出了一种去噪方法，以建立一个稳健的文本编码器，可以处理正常和噪声文本数据。我们的系统在Must-C En-De、En-Fr和LibriSpeech En-Fr任务上设置了新的最先进技术。摘要：We present a method for introducing a text encoder into pre-trained end-to-end speech translation systems. It enhances the ability of adapting one modality (i.e., source-language speech) to another (i.e., source-language text). Thus, the speech translation model can learn from both unlabeled and labeled data, especially when the source-language text data is abundant. Beyond this, we present a denoising method to build a robust text encoder that can deal with both normal and noisy text data. Our system sets new state-of-the-arts on the MuST-C En-De, En-Fr, and LibriSpeech En-Fr tasks.
【6】 A subjective study of the perceptual acceptability of audio-video desynchronization in sports videos
标题：体育视频音视频去同步化知觉可接受性的主观研究
链接：https://arxiv.org/abs/2212.01686
*与cs.SD语音【16】为同一篇
备注：6 pages; for associated code see this https URL摘要：本文介绍了一项针对体育视频音视频去同步的感知可接受性研究的结果。该研究使用通过对5个源内容应用8个音视频偏移而生成的45个视频进行。20名受试者参与了研究。结果表明，人类对语音刺激的音视频偏移误差更为敏感，体育赛事中出现的复杂事件具有更高的可接受阈值。这建议将广播中的音频-视频同步要求调谐到广播的内容。摘要：This paper presents the results of a study conducted on the perceptual acceptability of audio-video desynchronization for sports videos. The study was conducted with 45 videos generated by applying 8 audio-video offsets on 5 source contents. 20 subjects participated in the study. The results show that humans are more sensitive to audio-video offset errors for speech stimuli, and the complex events that occur in sports broadcasts have higher thresholds of acceptability. This suggests the tuning of audio-video synchronization requirements in broadcasting to the content of the broadcast.
【7】 Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models
标题：基于自监督语音模型的ASR无监督精调数据选择
链接：https://arxiv.org/abs/2212.01661
*与cs.SD语音【17】为同一篇
作者：Reem Gody,David Harwath机构：The University of Texas at Austin摘要：自监督学习（Self-supervised learning，SSL）能够利用未标记的语音数据来提高自动语音识别（Automatic Speech Recognition，ASR）模型的性能。然而，这提出了可用的未标记数据的哪个子集应该被选择用于转录的问题。我们的工作研究了不同的无监督数据选择技术，用于在有限的转录预算下微调HuBERT模型。我们研究了说话人多样性、性别偏见和话题多样性对下游ASR性能的影响。我们还设计了两种用于无监督数据选择的新颖技术：基于预训练丢失的数据选择和字节对编码的聚类单元（PBPE）的复杂性，并且我们展示了这些技术与纯随机数据选择的比较。最后，我们分析了所选微调子集的固有特性之间的相关性以及这些特性如何与最终的误字率相关。我们证明了令牌多样性、说话者多样性和主题多样性在获得最佳WER性能方面的重要性。摘要：Self-supervised learning (SSL) has been able to leverage unlabeled data to boost the performance of automatic speech recognition (ASR) models when we have access to only a small amount of transcribed speech data. However, this raises the question of which subset of the available unlabeled data should be selected for transcription. Our work investigates different unsupervised data selection techniques for fine-tuning the HuBERT model under a limited transcription budget. We investigate the impact of speaker diversity, gender bias, and topic diversity on the downstream ASR performance. We also devise two novel techniques for unsupervised data selection: pre-training loss based data selection and the perplexity of byte pair encoded clustered units (PBPE) and we show how these techniques compare to pure random data selection. Finally, we analyze the correlations between the inherent characteristics of the selected fine-tuning subsets as well as how these characteristics correlate with the resultant word error rate. We demonstrate the importance of token diversity, speaker diversity, and topic diversity in achieving the best performance in terms of WER.
【8】 Can we still use PEAQ? A Performance Analysis of the ITU Standard for the Objective Assessment of Perceived Audio Quality
标题：我们还能用PEAQ吗？ITU感知音频质量客观评价标准的性能分析
链接：https://arxiv.org/abs/2212.01467
*与cs.SD语音【18】为同一篇
作者：Pablo M. Delgado,Jürgen Herre机构：International Audio Laboratories Erlangen ‡, Erlangen, Germany, J¨urgen Herre摘要：如国际电信联盟（ITU）建议ITU-R BS.1387中所描述的音频质量的感知评估（PEAQ）方法已广泛用于计算地估计感知编码音频信号的质量，而不需要广泛的主观收听测试。然而，在其标准化结束之后，许多报告强调了该方案的明显限制，特别是涉及利用诸如带宽扩展或参数多信道编码的较新技术编码的信号。到目前为止，ITU还没有标准化用于测量语音和音频信号两者的质量的其它方法。因此，对这些限制的原因进行进一步调查将有利于对所述方案进行可能的更新。我们的实验结果表明，PEAQ的干扰响度模型的性能仍然与其他最先进的客观测量一样好（有时甚至更好），尽管性能随退化信号内容的类型（即语音或音乐）而变化。这一发现证明了需要改进认知模型。此外，结果表明基于更新的训练数据的模型输出值（MOV）到PEAQ的失真指数（DI）的更新映射可以极大地提高性能。最后，通过与其他系统的比较，提出了改进PEAQ的建议。摘要：The Perceptual Evaluation of Audio Quality (PEAQ) method as described in the International Telecommunication Union (ITU) recommendation ITU-R BS.1387 has been widely used for computationally estimating the quality of perceptually coded audio signals without the need for extensive subjective listening tests. However, many reports have highlighted clear limitations of the scheme after the end of its standardization, particularly involving signals coded with newer technologies such as bandwidth extension or parametric multi-channel coding. Until now, no other method for measuring the quality of both speech and audio signals has been standardized by the ITU. Therefore, a further investigation of the causes for these limitations would be beneficial to a possible update of said scheme. Our experimental results indicate that the performance of PEAQ's model of disturbance loudness is still as good as (and sometimes superior to) other state-of-the-art objective measures, albeit with varying performance depending on the type of degraded signal content (i.e. speech or music). This finding evidences the need for an improved cognitive model. In addition, results indicate that an updated mapping of Model Output Values (MOVs) to PEAQ's Distortion Index (DI) based on newer training data can greatly improve performance. Finally, some suggestions for the improvement of PEAQ are provided based on the reported results and comparison to other systems.
【9】 Objective Assessment of Spatial Audio Quality using Directional Loudness Maps
标题：基于方向性响度图的空间音质客观评价
链接：https://arxiv.org/abs/2212.01451
*与cs.SD语音【19】为同一篇
作者：Pablo M. Delgado,Jürgen Herre机构：⋆ International Audio Laboratories Erlangen ‡, Am Wolfsmantel , Erlangen, Germany, † Fraunhofer IIS, Am Wolfsmantel , Erlangen, Germany摘要：本工作引入了从立体声/双耳音频信号中提取的特征，旨在表示处理后的空间听觉场景中感知质量退化的度量。特征提取技术基于简化的立体声信号模型，该模型考虑了使用幅度平移（AP）技术朝向立体声场中的给定方向定位的听觉事件。我们在短时傅立叶变换域中针对给定AP值将立体声信号分解成一组定向信号，并且计算它们的总响度以形成定向响度表示或映射。然后，我们比较参考信号和劣化版本的方向响度图，以导出旨在描述在听力测试中报告的相关感知劣化分数的失真度量。然后，在广泛的收听测试数据库上测试该度量，其中立体声信号由使用非波形保留技术（例如，带宽扩展和联合立体声编码）的现有感知音频编解码器处理，已知所述非波形保留技术对现有质量预测器提出挑战。结果表明，所导出的失真度量可以作为对现有自动感知质量评估算法的扩展而被并入，以用于改进对空间编码音频信号的预测。摘要：This work introduces a feature extracted from stereophonic/binaural audio signals aiming to represent a measure of perceived quality degradation in processed spatial auditory scenes. The feature extraction technique is based on a simplified stereo signal model considering auditory events positioned towards a given direction in the stereo field using amplitude panning (AP) techniques. We decompose the stereo signal into a set of directional signals for given AP values in the Short-Time Fourier Transform domain and calculate their overall loudness to form a directional loudness representation or maps. Then, we compare directional loudness maps of a reference signal and a deteriorated version to derive a distortion measure aiming to describe the associated perceived degradation scores reported in listening tests. The measure is then tested on an extensive listening test database with stereo signals processed by state-of-the-art perceptual audio codecs using non waveform-preserving techniques such as bandwidth extension and joint stereo coding, known for presenting a challenge to existing quality predictors. Results suggest that the derived distortion measure can be incorporated as an extension to existing automated perceptual quality assessment algorithms for improving prediction on spatially coded audio signals.
【10】 Investigations on the Influence of Combined Inter-Aural Cue Distortions on Overall Audio Quality
标题：组合耳间线索失真对整体音质影响的研究
链接：https://arxiv.org/abs/2212.01427
*与cs.SD语音【20】为同一篇
作者：Pablo M. Delgado,Jürgen Herre摘要：开发能够预测感知编码信号的音频质量以避免在开发时间期间的大量收听测试的成本的算法是相当有兴趣的。虽然有许多用于预测具有单声道（音色）失真的信号的感知质量的已建立算法可用（PEAQ、POLQA），但是预测立体声和多声道空间信号的质量降级仍然被认为是一个挑战。由空间失真引起的音频质量劣化通常根据公知的耳间线索失真度量来测量，诸如耳间电平差失真（ILDD）、耳间时间差失真（ITDD）和耳间互相关失真（IACCD）。然而，它们的相互作用在多大程度上影响了复杂信号中的整体音频质量下降，例如，在多刺激测试中所表达的。我们提出了一种系统方法，其在一组代表性信号上引入空间失真的受控组合，并通过分析所述信号上的听力测试分数来评估它们对整体感知质量降级的影响。从这项研究中，我们得出了设计有意义的失真措施的指导方针，考虑到耳间线索失真的相互作用。摘要：There is a considerable interest in developing algorithms that can predict audio quality of perceptually coded signals to avoid the cost of extensive listening tests during development time. While many established algorithms for predicting the perceived quality of signals with monaural (timbral) distortions are available (PEAQ, POLQA), predicting the quality degradation of stereo and multi-channel spatial signals is still considered a challenge. Audio quality degradation arising from spatial distortions is usually measured in terms of well known inter-aural cue distortion measures such as Inter-aural Level Difference Distortions (ILDD), Inter-aural Time Difference Distortions (ITDD) and Inter-aural Cross Correlation Distortions (IACCD). However, the extent to which their interaction influences the overall audio quality degradation in complex signals as expressed - for example - in a multiple stimuli test is not yet thoroughly studied. We propose a systematic approach that introduces controlled combinations of spatial distortions on a representative set of signals and evaluates their influence on overall perceived quality degradation by analyzing listening test scores over said signals. From this study we derive guidelines for designing meaningful distortion measures that consider inter-aural cue distortion interactions.
【11】 Continual Learning for On-Device Speech Recognition using Disentangled Conformers
标题：基于解缠构象的设备上语音识别的连续学习
链接：https://arxiv.org/abs/2212.01393
*与cs.SD语音【21】为同一篇
作者：Anuj Diwan,Ching-Feng Yeh,Wei-Ning Hsu,Paden Tomasello,Eunsol Choi,David Harwath,Abdelrahman Mohamed机构： University of Texas at Austin, Meta Inc.备注：8 pages, 2 figures. Submitted to ICASSP 2023摘要：自动语音识别的研究主要集中在静态数据集上的训练和评价。然而，随着语音模型越来越多地部署在个人设备上，这样的模型遇到用户特定的分布转变。为了模拟这一现实场景，我们引入了LibriContinual，这是一个针对特定说话者领域自适应的持续学习基准，源自LibriVox有声读物，数据对应于118个个体说话者，每个说话者有6个不同大小的训练分段。另外，当前的语音识别模型和连续学习算法没有被优化为计算高效的。我们采用了一种通用的ASR训练算法NetAug，并创建了一种新的Conformer变体，称为DisConformer（解纠缠Conformer）。该算法产生ASR模型，该模型由用于通用用途的冻结“核心”网络和用于说话者特定调谐的几个可调谐“增强”网络组成。利用这些模型，我们提出了一种新的计算高效的连续学习算法DisentangledCL.实验结果表明，DisConformer模型在普通语音识别系统（LibriSpeech）上的性能明显优于基准模型（15.58%相对误差）。WER在测试-其他）。在特定于说话者的LibriContinual上，它们显著优于可训练参数匹配的基线（相对于WER测试），甚至在某些设置中完全匹配微调基线。摘要：Automatic speech recognition research focuses on training and evaluating on static datasets. Yet, as speech models are increasingly deployed on personal devices, such models encounter user-specific distributional shifts. To simulate this real-world scenario, we introduce LibriContinual, a continual learning benchmark for speaker-specific domain adaptation derived from LibriVox audiobooks, with data corresponding to 118 individual speakers and 6 train splits per speaker of different sizes. Additionally, current speech recognition models and continual learning algorithms are not optimized to be compute-efficient. We adapt a general-purpose training algorithm NetAug for ASR and create a novel Conformer variant called the DisConformer (Disentangled Conformer). This algorithm produces ASR models consisting of a frozen 'core' network for general-purpose use and several tunable 'augment' networks for speaker-specific tuning. Using such models, we propose a novel compute-efficient continual learning algorithm called DisentangledCL. Our experiments show that the DisConformer models significantly outperform baselines on general ASR i.e. LibriSpeech (15.58% rel. WER on test-other). On speaker-specific LibriContinual they significantly outperform trainable-parameter-matched baselines (by 20.65% rel. WER on test) and even match fully finetuned baselines in some settings.
【12】 DeAR: A Deep-learning-based Audio Re-recording Resilient Watermarking
标题：亲爱的：一种基于深度学习的音频重录抗攻击水印算法
链接：https://arxiv.org/abs/2212.02339
*与cs.SD语音【1】为同一篇
作者：Chang Liu,Jie Zhang,Han Fang,Zehua Ma,Weiming Zhang,Nenghai Yu机构：University of Science and Technology of China, University of Waterloo,National University of Singapore摘要：Audio watermarking is widely used for leaking source tracing. The robustness of the watermark determines the traceability of the algorithm. With the development of digital technology, audio re-recording (AR) has become an efficient and covert means to steal secrets. AR process could drastically destroy the watermark signal while preserving the original information. This puts forward a new requirement for audio watermarking at this stage, that is, to be robust to AR distortions. Unfortunately, none of the existing algorithms can effectively resist AR attacks due to the complexity of the AR process. To address this limitation, this paper proposes DeAR, a deep-learning-based audio re-recording resistant watermarking. Inspired by DNN-based image watermarking, we pioneer a deep learning framework for audio carriers, based on which the watermark signal can be effectively embedded and extracted. Meanwhile, in order to resist the AR attack, we delicately analyze the distortions that occurred in the AR process and design the corresponding distortion layer to cooperate with the proposed watermarking framework. Extensive experiments show that the proposed algorithm can resist not only common electronic channel distortions but also AR distortions. Under the premise of high-quality embedding (SNR=25.86dB), in the case of a common re-recording distance (20cm), the algorithm can effectively achieve an average bit recovery accuracy of 98.55%.
【13】 End-to-end Recording Device Identification Based on Deep Representation Learning
标题：基于深度表征学习的端到端录音设备识别
链接：https://arxiv.org/abs/2212.02084
*与cs.SD语音【2】为同一篇
作者：Chunyan Zeng,Dongliang Zhu,Zhifeng Wang,Minghu Wu,Wei Xiong,Nan Zhao机构：Correspondence:, Department of Digital Media, Technology, Central China Normal, University, Luoyu Road , Wuhan, China, Full list of author information is, available at the end of the article备注：20 pages, 5 figures, recording device identification摘要：Deep learning techniques have achieved specific results in recording device source identification. The recording device source features include spatial information and certain temporal information. However, most recording device source identification methods based on deep learning only use spatial representation learning from recording device source features, which cannot make full use of recording device source information. Therefore, in this paper, to fully explore the spatial information and temporal information of recording device source, we propose a new method for recording device source identification based on the fusion of spatial feature information and temporal feature information by using an end-to-end framework. From a feature perspective, we designed two kinds of networks to extract recording device source spatial and temporal information. Afterward, we use the attention mechanism to adaptively assign the weight of spatial information and temporal information to obtain fusion features. From a model perspective, our model uses an end-to-end framework to learn the deep representation from spatial feature and temporal feature and train using deep and shallow loss to joint optimize our network. This method is compared with our previous work and baseline system. The results show that the proposed method is better than our previous work and baseline system under general conditions.
【14】 NBC2: Multichannel Speech Separation with Revised Narrow-band Conformer
标题：NBC2：基于改进窄带整形器的多通道语音分离
链接：https://arxiv.org/abs/2212.02076
*与cs.SD语音【3】为同一篇
作者：Changsheng Quan,Xiaofei Li机构： time domainChangsheng Quan is with Zhejiang University, and Westlake University &Westlake Institute for Advanced Study摘要：This work proposes a multichannel narrow-band speech separation network. In the short-time Fourier transform (STFT) domain, the proposed network processes each frequency independently, and all frequencies use a shared network. For each frequency, the network performs end-to-end speech separation, namely taking as input the STFT coefficients of microphone signals, and predicting the separated STFT coefficients of multiple speakers. The proposed network learns to cluster the frame-wise spatial/steering vectors that belong to different speakers. It is mainly composed of three components. First, a self-attention network. Clustering of spatial vectors shares a similar principle with the self-attention mechanism in the sense of computing the similarity of vectors and then aggregating similar vectors. Second, a convolutional feed-forward network. The convolutional layers are employed for signal smoothing and reverberation processing. Third, a novel hidden-layer normalization method, i.e. group batch normalization (GBN), is especially designed for the proposed narrow-band network to maintain the distribution of hidden units over frequencies. Overall, the proposed network is named NBC2, as it is a revised version of our previous NBC (narrow-band conformer) network. Experiments show that 1) the proposed network outperforms other state-of-the-art methods by a large margin, 2) the proposed GBN improves the signal-to-distortion ratio by 3 dB, relative to other normalization methods, such as batch/layer/group normalization, 3) the proposed narrow-band network is spectrum-agnostic, as it does not learn spectral patterns, and 4) the proposed network is indeed performing frame clustering (demonstrated by the attention maps).
【15】 Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models
标题：用于端到端语音识别模型文本自适应的快速准确的分解神经换能器
链接：https://arxiv.org/abs/2212.01992
*与cs.SD语音【4】为同一篇
作者：Rui Zhao,Jian Xue,Partha Parthasarathy,Veljko Miljanic,Jinyu Li机构：Microsoft Speech Group, Redmond, WA, USA摘要：神经元传感器由于具有自然流动的能力，是目前最流行的端到端语音识别模型。但是，将其用于纯文本数据是一项挑战。针对这一问题，提出了因子化神经传感器（FNT）模型。与标准神经传感器模型相比，FNT对纯文本自适应数据的自适应能力的提高是以降低精度为代价的。我们提出了几种改进FNT模型性能的方法。它们是：在训练过程中加入CTC准则，在自适应过程中加入KL散度损失，使用预先训练好的语言模型作为词汇预测器的种子，以及通过用n元语言模型插值词汇预测器来实现有效的自适应方法。这些方法的组合使得相对误字率比标准FNT模型降低了9.48%。此外，结合词汇预测器的n-gram插值极大地提高了自适应速度，并获得了满意的自适应性能。摘要：Neural transducer is now the most popular end-to-end model for speech recognition, due to its naturally streaming ability. However, it is challenging to adapt it with text-only data. Factorized neural transducer (FNT) model was proposed to mitigate this problem. The improved adaptation ability of FNT on text-only adaptation data came at the cost of lowered accuracy compared to the standard neural transducer model. We propose several methods to improve the performance of the FNT model. They are: adding CTC criterion during training, adding KL divergence loss during adaptation, using a pre-trained language model to seed the vocabulary predictor, and an efficient adaptation approach by interpolating the vocabulary predictor with the n-gram language model. A combination of these approaches results in a relative word-error-rate reduction of 9.48\% from the standard FNT model. Furthermore, n-gram interpolation with the vocabulary predictor improves the adaptation speed hugely with satisfactory adaptation performance.
【16】 Speech MOS multi-task learning and rater bias correction
标题：语音MOS多任务学习与评分者偏差校正
链接：https://arxiv.org/abs/2212.01911
*与cs.SD语音【5】为同一篇
作者：Haleh Akrami,Hannes Gamper机构：University of Southern California, Department of Biomedical Engineering, Los Angeles, CA, USA, Audio and Acoustics Research Group, Microsoft Research, Redmond, WA, USA备注：Submitted to ICASSP 2023摘要：感知语音质量是电话会议应用的重要性能度量。平均意见得分（MOS）被标准化用于语音质量的感知评估，并且通过要求收听者对语音样本的质量进行评级来获得。近年来，开发用于盲估计MOS的模型的研究兴趣日益增加。本文提出了一种多任务框架，在训练中加入额外的标签和数据，以提高盲MOS估计模型的性能。实验结果表明，该模型可以通过训练两个不相交的数据集（一个只包含MOS标签，另一个只包含T60和C50标签）来联合估计MOS、混响时间（T60）和清晰度（C50）.此外，我们使用半监督框架来组合训练中的两个MOS数据集，一个仅包含MOS标签（根据ITU-T建议P.808），而另一个包含语音信号、背景噪声和整体质量的单独分数（根据ITU-T建议P.835）。最后，我们提出了解决MOS标签中的个人评价者偏见的初步结果。摘要：Perceptual speech quality is an important performance metric for teleconferencing applications. The mean opinion score (MOS) is standardized for the perceptual evaluation of speech quality and is obtained by asking listeners to rate the quality of a speech sample. Recently, there has been increasing research interest in developing models for estimating MOS blindly. Here we propose a multi-task framework to include additional labels and data in training to improve the performance of a blind MOS estimation model. Experimental results indicate that the proposed model can be trained to jointly estimate MOS, reverberation time (T60), and clarity (C50) by combining two disjoint data sets in training, one containing only MOS labels and the other containing only T60 and C50 labels. Furthermore, we use a semi-supervised framework to combine two MOS data sets in training, one containing only MOS labels (per ITU-T Recommendation P.808), and the other containing separate scores for speech signal, background noise, and overall quality (per ITU-T Recommendation P.835). Finally, we present preliminary results for addressing individual rater bias in the MOS labels.
【17】 Melody transcription via generative pre-training
标题：通过生成性预训练实现旋律转录
链接：https://arxiv.org/abs/2212.01884
*与cs.SD语音【6】为同一篇
作者：Chris Donahue,John Thickstun,Percy Liang备注：Published as a conference paper at ISMIR 2022摘要：尽管旋律在音乐感知中扮演着核心角色，但在音乐信息检索中可靠地检测存在于任意音乐记录中的旋律的音符仍然是一个开放的挑战。旋律转录的一个关键挑战是建立能够处理包含任意数量的乐器合奏和音乐风格的广泛音频的方法-现有策略对于某些旋律乐器或风格效果良好，但不是所有的。为了应对这一挑战，我们利用了Jukebox（Dhariwal等人，2020）的表示，这是一种广泛的音乐音频生成模型，从而相对于传统谱图特征，将旋律转录的性能提高了20 $%。旋律转录的另一个障碍是缺乏训练数据--我们从广泛音乐的众包注释中获得了一个新的数据集，其中包含50 $小时的旋律转录。生成式预训练和该任务的新数据集的组合导致旋律转录相对于最强可用基线的$77$%更强的性能。通过将我们新的旋律转录方法与节拍检测、基调估计和和弦识别解决方案相结合，我们构建了Sheet Sage，这是一个能够直接从音乐音频转录人类可读的前导音的系统。 音频示例可在www.example.com上找到https://chrisdonahue.com/sheetsage，代码可在www.example.com上找到https://github.com/chrisdonahue/sheetsage。摘要：Despite the central role that melody plays in music perception, it remains an open challenge in music information retrieval to reliably detect the notes of the melody present in an arbitrary music recording. A key challenge in melody transcription is building methods which can handle broad audio containing any number of instrument ensembles and musical styles - existing strategies work well for some melody instruments or styles but not all. To confront this challenge, we leverage representations from Jukebox (Dhariwal et al. 2020), a generative model of broad music audio, thereby improving performance on melody transcription by $20$% relative to conventional spectrogram features. Another obstacle in melody transcription is a lack of training data - we derive a new dataset containing $50$ hours of melody transcriptions from crowdsourced annotations of broad music. The combination of generative pre-training and a new dataset for this task results in $77$% stronger performance on melody transcription relative to the strongest available baseline. By pairing our new melody transcription approach with solutions for beat detection, key estimation, and chord recognition, we build Sheet Sage, a system capable of transcribing human-readable lead sheets directly from music audio. Audio examples can be found at https://chrisdonahue.com/sheetsage and code at https://github.com/chrisdonahue/sheetsage .
【18】 Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech
标题：用于改善耳语的自然度、可理解性和语音性的生成模型
链接：https://arxiv.org/abs/2212.01775
*与cs.SD语音【7】为同一篇
作者：Dominik Wagner,Sebastian P. Bayerl,Hector A. Cordourier Maruri,Tobias Bocklet机构：Technische Hochschule N¨urnberg Georg Simon Ohm, Germany, Intel Labs摘要：本工作采用了两种最新的生成模型结构，并评估了它们在耳语音到正常语音转换中的有效性。我们将正常目标语音加入到矢量量化变分自动编码器（VQ-VAE）和MelGANs的训练准则中，从而调节系统从耳声输入中恢复浊音语音。客观和主观质量测量表明，VQ-VAE和MelGAN都可以被修改以执行转换任务。我们发现所提出的方法相对于DiscoGAN基线显著地将Mel倒谱失真（MCD）度量改善了至少25%。主观听力测试表明，与耳语输入语音相比，基于MelGAN的系统显著改善了自然度、可懂度和发声。基于潜在语音表示之间的差异的新的评估度量也表明我们的基于MelGAN的方法产生了相对于基线的改进。摘要：This work adapts two recent architectures of generative models and evaluates their effectiveness for the conversion of whispered speech to normal speech. We incorporate the normal target speech into the training criterion of vector-quantized variational autoencoders (VQ-VAEs) and MelGANs, thereby conditioning the systems to recover voiced speech from whispered inputs. Objective and subjective quality measures indicate that both VQ-VAEs and MelGANs can be modified to perform the conversion task. We find that the proposed approaches significantly improve the Mel cepstral distortion (MCD) metric by at least 25% relative to a DiscoGAN baseline. Subjective listening tests suggest that the MelGAN-based system significantly improves naturalness, intelligibility, and voicing compared to the whispered input speech. A novel evaluation measure based on differences between latent speech representations also indicates that our MelGAN-based approach yields improvements relative to the baseline.
【19】 A dataset for audio-video based vehicle speed estimation
标题：一种基于音视频的车速估计数据集
链接：https://arxiv.org/abs/2212.01651
*与cs.SD语音【8】为同一篇
作者：Slobodan Djukanović,Nikola Bulatović,Ivana Čavor机构：A dataset for audio-video based vehicle speedestimation 1st Slobodan Djukanovi´cFaculty of Electrical EngineeringUniversity of MontenegroPodgorica, me 2nd Nikola Bulatovi´cFaculty of Electrical EngineeringUniversity of MontenegroPodgorica备注：30th Telecommunications Forum TELFOR 2022, Belgrade, Serbia, November 15-16, 2022. 5 pages, 2 figures, 1 table摘要：道路车辆的准确速度估计由于若干原因而很重要。一个是速度限制执行，这是减少交通事故和死亡的一个关键工具。与其他研究领域相比，可用于车辆速度估计的数据集数量仍然非常有限。我们提出了一个数据集的道路音频视频记录的单一车辆通过一个摄像机在已知的速度，保持稳定的车载巡航控制。该数据集包含13辆汽车，选择尽可能多样化的制造商、生产年份、发动机类型、动力和变速器，从而产生总计400美元的带注释音频视频记录。该数据集是完全可用的，并打算作为一个公共基准，以促进研究音频视频车辆速度估计。除了数据集之外，我们还提出了一种交叉验证策略，该策略可用于车辆速度估计的机器学习模型。提出了两种训练-验证数据集分割方法。摘要：Accurate speed estimation of road vehicles is important for several reasons. One is speed limit enforcement, which represents a crucial tool in decreasing traffic accidents and fatalities. Compared with other research areas and domains, the number of available datasets for vehicle speed estimation is still very limited. We present a dataset of on-road audio-video recordings of single vehicles passing by a camera at known speeds, maintained stable by the on-board cruise control. The dataset contains thirteen vehicles, selected to be as diverse as possible in terms of manufacturer, production year, engine type, power and transmission, resulting in a total of $ 400 $ annotated audio-video recordings. The dataset is fully available and intended as a public benchmark to facilitate research in audio-video vehicle speed estimation. In addition to the dataset, we propose a cross-validation strategy which can be used in a machine learning model for vehicle speed estimation. Two approaches to training-validation split of the dataset are proposed.
【20】 UniSyn: An End-to-End Unified Model for Text-to-Speech and Singing Voice Synthesis
标题：UniSyn：一种端到端的语音合成和歌唱语音合成统一模型
链接：https://arxiv.org/abs/2212.01546
*与cs.SD语音【9】为同一篇
作者：Yi Lei,Shan Yang,Xinsheng Wang,Qicong Xie,Jixun Yao,Lei Xie,Dan Su机构：Northwestern Polytechnical University, Xi’an, China, Tencent AI Lab, China摘要：Text-to-speech (TTS) and singing voice synthesis (SVS) aim at generating high-quality speaking and singing voice according to textual input and music scores, respectively. Unifying TTS and SVS into a single system is crucial to the applications requiring both of them. Existing methods usually suffer from some limitations, which rely on either both singing and speaking data from the same person or cascaded models of multiple tasks. To address these problems, a simplified elegant framework for TTS and SVS, named UniSyn, is proposed in this paper. It is an end-to-end unified model that can make a voice speak and sing with only singing or speaking data from this person. To be specific, a multi-conditional variational autoencoder (MC-VAE), which constructs two independent latent sub-spaces with the speaker- and style-related (i.e. speak or sing) conditions for flexible control, is proposed in UniSyn. Moreover, supervised guided-VAE and timbre perturbation with the Wasserstein distance constraint are leveraged to further disentangle the speaker timbre and style. Experiments conducted on two speakers and two singers demonstrate that UniSyn can generate natural speaking and singing voice without corresponding training data. The proposed approach outperforms the state-of-the-art end-to-end voice generation work, which proves the effectiveness and advantages of UniSyn.
【21】 NEAL: An open-source tool for audio annotation
标题：Neal：一个开源的音频标注工具
链接：https://arxiv.org/abs/2212.01457
*与cs.SD语音【10】为同一篇
作者：Anthony Gibbons,Ian Donohue,Courtney E. Gorman,Emma King,Andrew Parnell机构： Hamilton Institute, Department of Mathematics and Statistics, University, Kildare, Ireland, Zoology, School of Natural Sciences, Trinity College Dublin, Dublin, Ireland摘要：被动声学监测广泛应用于生态学、生物多样性和保护研究。通过声学监测收集的数据集通常非常大，并使用人工智能和机器学习模型自动处理，旨在复制领域专家的工作。这些模型是有监督的学习算法，需要在由专家产生的高质量注释上进行训练。由于专家通常是资源有限的，因此需要一种用于注释音频的成本有效的过程来最大限度地利用数据。提出了一个开放源代码的交互式音频数据标注工具NEAL（Nature+Energy Audio Labeller）.该工具使用R和相关的Shiny框架构建，提供了一个反应式环境，用户可以在其中快速注释音频文件，并调整自动更改用户界面相应元素的设置。该应用程序的设计目标是让专家观鸟者和公民科学家为声学注释项目做出贡献。R编程在生物声学领域的普及性和灵活性意味着，可以针对其他鸟类标签数据集，甚至通用音频标签任务修改Shiny应用程序。我们通过标记从爱尔兰各地风电场收集的数据来演示该应用程序。摘要：Passive acoustic monitoring is used widely in ecology, biodiversity, and conservation studies. Data sets collected via acoustic monitoring are often extremely large and built to be processed automatically using Artificial Intelligence and Machine learning models, which aim to replicate the work of domain experts. These models, being supervised learning algorithms, need to be trained on high quality annotations produced by experts. Since the experts are often resource-limited, a cost-effective process for annotating audio is needed to get maximal use out of the data. We present an open-source interactive audio data annotation tool, NEAL (Nature+Energy Audio Labeller). Built using R and the associated Shiny framework, the tool provides a reactive environment where users can quickly annotate audio files and adjust settings that automatically change the corresponding elements of the user interface. The app has been designed with the goal of having both expert birders and citizen scientists contribute to acoustic annotation projects. The popularity and flexibility of R programming in bioacoustics means that the Shiny app can be modified for other bird labelling data sets, or even to generic audio labelling tasks. We demonstrate the app by labelling data collected from wind farm sites across Ireland.
机器翻译，仅供参考
每日语音速递