今天跟大家分享一篇语音相关的论文合集：cs.SD语音2篇，eess.AS音频处理2篇。
【1】 DDX7: Differentiable FM Synthesis of Musical Instrument Sounds
标题：DDX7：乐器声音的可微调频合成
链接：https://arxiv.org/abs/2208.06169
作者：Franco Caspe,Andrew McPherson,Mark Sandler机构：Queen Mary University of London备注：Accepted to ISMIR 2022. See online supplement at this https URL摘要：FM合成是一种众所周知的算法，用于从一组紧凑的设计基元中生成复杂的音色。通常具有MIDI接口，从音频源控制它通常是不切实际的。另一方面，微分数字信号处理（DDSP）通过深度神经网络实现了细致入微的音频渲染（DNN），其学习控制来自任意声音输入的可微分合成层。训练过程涉及用于监督的音频语料库和频谱重构损失函数。这种函数，虽然与频谱幅度匹配得很好，但缺乏音调方向，这会妨碍FM合成器参数的联合优化.本文采取步骤，使来自音频输入的完善的FM合成结构能够连续控制.首先，我们讨论了一组设计约束条件，这些约束条件通过标准重构损耗简化了可微分FM合成器的频谱优化。（DDX7）是一种轻量级架构，可根据一组紧凑的参数对乐器声音进行神经FM再合成。我们在从URMP数据集中提取的乐器样本上训练该模型，并定量证明其与选定基准的音频质量相当。摘要：FM Synthesis is a well-known algorithm used to generate complex timbre from a compact set of design primitives. Typically featuring a MIDI interface, it is usually impractical to control it from an audio source. On the other hand, Differentiable Digital Signal Processing (DDSP) has enabled nuanced audio rendering by Deep Neural Networks (DNNs) that learn to control differentiable synthesis layers from arbitrary sound inputs. The training process involves a corpus of audio for supervision, and spectral reconstruction loss functions. Such functions, while being great to match spectral amplitudes, present a lack of pitch direction which can hinder the joint optimization of the parameters of FM synthesizers. In this paper, we take steps towards enabling continuous control of a well-established FM synthesis architecture from an audio input. Firstly, we discuss a set of design constraints that ease spectral optimization of a differentiable FM synthesizer via a standard reconstruction loss. Next, we present Differentiable DX7 (DDX7), a lightweight architecture for neural FM resynthesis of musical instrument sounds in terms of a compact set of parameters. We train the model on instrument samples extracted from the URMP dataset, and quantitatively demonstrate its comparable audio quality against selected benchmarks.
【2】 An investigation on selecting audio pre-trained models for audio captioning
标题：音频字幕中音频预训练模型选择的研究
链接：https://arxiv.org/abs/2208.06127
作者：Peiran Yan,Shengchen Li机构：Xi’an Jiaotong-Liverpool University, Suzhou, China摘要：音频字幕是一个基于内容生成音频描述的任务.由于复杂性，预训练模型在音频字幕中被广泛使用.除非对一个全面的系统进行再训练，否则很难确定预训练模型对音频字幕系统的贡献程度.为了避免耗时耗力的再训练过程，提出一种预测模型的性能预测器是十分必要的，研究了一系列预先训练的模型，用于所提取的音频特征和音频字幕的性能之间的相关性。实验results.The结果表明，所提取得音频特征得峰度和偏度可以作为预测音频字幕系统性能得指标.这是由于音频特征的峰度和偏斜度与音频字幕系统的性能之间的高度相关性。摘要：Audio captioning is a task that generates description of audio based on content. Pre-trained models are widely used in audio captioning due to high complexity. Unless a comprehensive system is re-trained, it is hard to determine how well pre-trained models contribute to audio captioning system. To prevent the time consuming and energy consuming process of retraining, it is necessary to propose a preditor of performance for the pre-trained model in audio captioning. In this paper, a series of pre-trained models are investigated for the correlation between extracted audio features and the performance of audio captioning. A couple of predictor is proposed based on the experiment results.The result demonstrates that the kurtosis and skewness of audio features extracted may act as an indicator of the performance of audio captioning systems for pre-trained audio due to the high correlation between kurtosis and skewness of audio features and the performance of audio captioning systems.
【1】 DDX7: Differentiable FM Synthesis of Musical Instrument Sounds
标题：DDX7：乐器声音的可微调频合成
链接：https://arxiv.org/abs/2208.06169
* 与cs.SD语音【1】为同一篇
作者：Franco Caspe,Andrew McPherson,Mark Sandler机构：Queen Mary University of London备注：Accepted to ISMIR 2022. See online supplement at this https URL摘要：FM合成是一种众所周知的算法，用于从一组紧凑的设计基元中生成复杂的音色。通常具有MIDI接口，从音频源控制它通常是不切实际的。另一方面，微分数字信号处理（DDSP）通过深度神经网络实现了细致入微的音频渲染（DNN），其学习控制来自任意声音输入的可微分合成层。训练过程涉及用于监督的音频语料库和频谱重构损失函数。这种函数，虽然与频谱幅度匹配得很好，但缺乏音调方向，这会妨碍FM合成器参数的联合优化.本文采取步骤，使来自音频输入的完善的FM合成结构能够连续控制.首先，我们讨论了一组设计约束条件，这些约束条件通过标准重构损耗简化了可微分FM合成器的频谱优化。（DDX7）是一种轻量级架构，可根据一组紧凑的参数对乐器声音进行神经FM再合成。我们在从URMP数据集中提取的乐器样本上训练该模型，并定量证明其与选定基准的音频质量相当。摘要：FM Synthesis is a well-known algorithm used to generate complex timbre from a compact set of design primitives. Typically featuring a MIDI interface, it is usually impractical to control it from an audio source. On the other hand, Differentiable Digital Signal Processing (DDSP) has enabled nuanced audio rendering by Deep Neural Networks (DNNs) that learn to control differentiable synthesis layers from arbitrary sound inputs. The training process involves a corpus of audio for supervision, and spectral reconstruction loss functions. Such functions, while being great to match spectral amplitudes, present a lack of pitch direction which can hinder the joint optimization of the parameters of FM synthesizers. In this paper, we take steps towards enabling continuous control of a well-established FM synthesis architecture from an audio input. Firstly, we discuss a set of design constraints that ease spectral optimization of a differentiable FM synthesizer via a standard reconstruction loss. Next, we present Differentiable DX7 (DDX7), a lightweight architecture for neural FM resynthesis of musical instrument sounds in terms of a compact set of parameters. We train the model on instrument samples extracted from the URMP dataset, and quantitatively demonstrate its comparable audio quality against selected benchmarks.
【2】 An investigation on selecting audio pre-trained models for audio captioning
标题：音频字幕中音频预训练模型选择的研究
链接：https://arxiv.org/abs/2208.06127
* 与cs.SD语音【2】为同一篇
作者：Peiran Yan,Shengchen Li机构：Xi’an Jiaotong-Liverpool University, Suzhou, China摘要：音频字幕是一个基于内容生成音频描述的任务.由于复杂性，预训练模型在音频字幕中被广泛使用.除非对一个全面的系统进行再训练，否则很难确定预训练模型对音频字幕系统的贡献程度.为了避免耗时耗力的再训练过程，提出一种预测模型的性能预测器是十分必要的，研究了一系列预先训练的模型，用于所提取的音频特征和音频字幕的性能之间的相关性。实验results.The结果表明，所提取得音频特征得峰度和偏度可以作为预测音频字幕系统性能得指标.这是由于音频特征的峰度和偏斜度与音频字幕系统的性能之间的高度相关性。摘要：Audio captioning is a task that generates description of audio based on content. Pre-trained models are widely used in audio captioning due to high complexity. Unless a comprehensive system is re-trained, it is hard to determine how well pre-trained models contribute to audio captioning system. To prevent the time consuming and energy consuming process of retraining, it is necessary to propose a preditor of performance for the pre-trained model in audio captioning. In this paper, a series of pre-trained models are investigated for the correlation between extracted audio features and the performance of audio captioning. A couple of predictor is proposed based on the experiment results.The result demonstrates that the kurtosis and skewness of audio features extracted may act as an indicator of the performance of audio captioning systems for pre-trained audio due to the high correlation between kurtosis and skewness of audio features and the performance of audio captioning systems.
机器翻译，仅供参考
每日语音速递