今日论文合集：cs.SD语音4篇，eess.AS音频处理4篇。
cs.SD语音
【1】 LAVSS: Location-Guided Audio-Visual Spatial Audio Separation标题：LAVSS：位置引导的视听空间音频分离链接：https://arxiv.org/abs/2310.20446
作者：Yuxin Ye,Wenming Yang,Yapeng Tian备注：Accepted by WACV2024摘要：现有的机器学习研究在单声道视听分离（MAVS）方面取得了可喜的成果。然而，大多数MAVS方法纯粹考虑声源是什么，而不是声源所在的位置。这在VR/AR场景中可能是一个问题，其中听众需要能够区分位于不同方向的类似音频源。为了解决这一问题，我们将MAVS推广到空间音频分离，并提出了LAVSS：一个位置引导的视听空间音频分离器。LAVSS的灵感来自于空间音频和视觉位置之间的相关性。我们引入双耳音频作为空间线索进行的相位差，我们利用发声对象的位置表示作为额外的模态指导。我们还利用多层次的跨模态注意力来执行与音频功能的视觉位置协作。此外，我们采用了一个预先训练的单声道分离器，从丰富的单声道声音传递知识，以提高空间音频分离。这利用了单声道和双耳声道之间的相关性。在FAIR-Play数据集上的实验表明，所提出的LAVSS优于现有的视听分离基准。我们的项目页面：https://yyx666660.github.io/LAVSS/。摘要：Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation. Our project page: https://yyx666660.github.io/LAVSS/.
【2】 Musical Form Generation标题：曲式生成链接：https://arxiv.org/abs/2310.19842
作者：Lilac Atassi摘要：虽然最近的生成模型可以产生迷人的音乐，但它们的实用性有限。音乐中的变化往往是偶然的，导致作品缺乏结构。超过一分钟的片段可能会变得不连贯或重复。本文介绍了一种生成结构化的、任意长度的音乐作品的方法。这种方法的核心是使用条件生成模型创建音乐片段，并在这些片段之间进行过渡。生成确定高级合成的提示与创建更精细的低级细节是不同的。一个大的语言模型，然后用来建议音乐形式。摘要：While recent generative models can produce engaging music, their utility is limited. The variation in the music is often left to chance, resulting in compositions that lack structure. Pieces extending beyond a minute can become incoherent or repetitive. This paper introduces an approach for generating structured, arbitrarily long musical pieces. Central to this approach is the creation of musical segments using a conditional generative model, with transitions between these segments. The generation of prompts that determine the high-level composition is distinct from the creation of finer, lower-level details. A large language model is then used to suggest the musical form.
【3】 Study of speaker localization with binaural microphone array  incorporating auditory filters and lateral angle estimation标题：双耳麦克风阵列结合听觉滤波和侧向角度估计的说话人定位研究链接：https://arxiv.org/abs/2310.20238
作者：Yanir Maymon,Israel Nelken,Boaz Rafaely备注：None摘要：双耳麦克风阵列的说话人定位在语音通信、视频会议和机器人试听等应用中得到了广泛的研究。为此任务开发的许多方法，包括直接路径优势（DPD）测试，在它们的处理中共享共同的阶段，包括使用短时傅立叶变换（STFT）的变换，以及基于头部相关传递函数（HRTF）集的到达方向（DOA）搜索。在本文中，这些处理阶段的替代方案，由人类听觉的动机，提出了。这些措施包括将听觉滤波器组，以取代STFT，和一个新的DOA搜索的基础上变换的HRTF作为导向矢量。仿真研究和实验研究进行了验证，提出的替代方案，都被应用到两个双耳DOA估计方法，结果表明，所提出的方法与现有的方法相比，毫不逊色。摘要：Speaker localization for binaural microphone arrays has been widely studied for applications such as speech communication, video conferencing, and robot audition. Many methods developed for this task, including the direct path dominance (DPD) test, share common stages in their processing, which include transformation using the short-time Fourier transform (STFT), and a direction of arrival (DOA) search that is based on the head related transfer function (HRTF) set. In this paper, alternatives to these processing stages, motivated by human hearing, are proposed. These include incorporating an auditory filter bank to replace the STFT, and a new DOA search based on transformed HRTF as steering vectors. A simulation study and an experimental study are conducted to validate the proposed alternatives, and both are applied to two binaural DOA estimation methods; the results show that the proposed method compares favorably with current methods.
【4】 Intelligibility prediction with a pretrained noise-robust automatic  speech recognition model标题：利用预先训练的抗噪自动语音识别模型进行清晰度预测链接：https://arxiv.org/abs/2310.19817
作者：Zehai Tu,Ning Ma,Jon Barker摘要：本文介绍了两个可懂度预测系统来自预训练的噪声鲁棒自动语音识别（ASR）模型的第二清晰度预测挑战（CPC 2）。一个系统是侵入式的，并利用ASR模型的隐藏表示。另一个系统是非侵入性的，并与派生ASR的不确定性进行预测。ASR模型仅使用模拟的噪声语音语料库进行预训练，并且不利用CPC 2数据。出于这个原因，可懂度预测系统对于CPC 2评估的准确预测性能而言对看不见的场景是鲁棒的。摘要：This paper describes two intelligibility prediction systems derived from a pretrained noise-robust automatic speech recognition (ASR) model for the second Clarity Prediction Challenge (CPC2). One system is intrusive and leverages the hidden representations of the ASR model. The other system is non-intrusive and makes predictions with derived ASR uncertainty. The ASR model is only pretrained with a simulated noisy speech corpus and does not take advantage of the CPC2 data. For that reason, the intelligibility prediction systems are robust to unseen scenarios given the accurate prediction performance on the CPC2 evaluation.
eess.AS音频处理
【1】 Study of speaker localization with binaural microphone array  incorporating auditory filters and lateral angle estimation标题：双耳麦克风阵列结合听觉滤波和侧向角度估计的说话人定位研究链接：https://arxiv.org/abs/2310.20238
作者：Yanir Maymon,Israel Nelken,Boaz Rafaely备注：None摘要：双耳麦克风阵列的说话人定位在语音通信、视频会议和机器人试听等应用中得到了广泛的研究。为此任务开发的许多方法，包括直接路径优势（DPD）测试，在它们的处理中共享共同的阶段，包括使用短时傅立叶变换（STFT）的变换，以及基于头部相关传递函数（HRTF）集的到达方向（DOA）搜索。在本文中，这些处理阶段的替代方案，由人类听觉的动机，提出了。这些措施包括将听觉滤波器组，以取代STFT，和一个新的DOA搜索的基础上变换的HRTF作为导向矢量。仿真研究和实验研究进行了验证，提出的替代方案，都被应用到两个双耳DOA估计方法，结果表明，所提出的方法与现有的方法相比，毫不逊色。摘要：Speaker localization for binaural microphone arrays has been widely studied for applications such as speech communication, video conferencing, and robot audition. Many methods developed for this task, including the direct path dominance (DPD) test, share common stages in their processing, which include transformation using the short-time Fourier transform (STFT), and a direction of arrival (DOA) search that is based on the head related transfer function (HRTF) set. In this paper, alternatives to these processing stages, motivated by human hearing, are proposed. These include incorporating an auditory filter bank to replace the STFT, and a new DOA search based on transformed HRTF as steering vectors. A simulation study and an experimental study are conducted to validate the proposed alternatives, and both are applied to two binaural DOA estimation methods; the results show that the proposed method compares favorably with current methods.
【2】 Intelligibility prediction with a pretrained noise-robust automatic  speech recognition model标题：利用预先训练的抗噪自动语音识别模型进行清晰度预测链接：https://arxiv.org/abs/2310.19817
作者：Zehai Tu,Ning Ma,Jon Barker摘要：本文介绍了两个可懂度预测系统来自预训练的噪声鲁棒自动语音识别（ASR）模型的第二清晰度预测挑战（CPC 2）。一个系统是侵入式的，并利用ASR模型的隐藏表示。另一个系统是非侵入性的，并与派生ASR的不确定性进行预测。ASR模型仅使用模拟的噪声语音语料库进行预训练，并且不利用CPC 2数据。出于这个原因，可懂度预测系统对于CPC 2评估的准确预测性能而言对看不见的场景是鲁棒的。摘要：This paper describes two intelligibility prediction systems derived from a pretrained noise-robust automatic speech recognition (ASR) model for the second Clarity Prediction Challenge (CPC2). One system is intrusive and leverages the hidden representations of the ASR model. The other system is non-intrusive and makes predictions with derived ASR uncertainty. The ASR model is only pretrained with a simulated noisy speech corpus and does not take advantage of the CPC2 data. For that reason, the intelligibility prediction systems are robust to unseen scenarios given the accurate prediction performance on the CPC2 evaluation.
【3】 LAVSS: Location-Guided Audio-Visual Spatial Audio Separation标题：LAVSS：位置引导的视听空间音频分离链接：https://arxiv.org/abs/2310.20446
作者：Yuxin Ye,Wenming Yang,Yapeng Tian备注：Accepted by WACV2024摘要：现有的机器学习研究在单声道视听分离（MAVS）方面取得了可喜的成果。然而，大多数MAVS方法纯粹考虑声源是什么，而不是声源所在的位置。这在VR/AR场景中可能是一个问题，其中听众需要能够区分位于不同方向的类似音频源。为了解决这一问题，我们将MAVS推广到空间音频分离，并提出了LAVSS：一个位置引导的视听空间音频分离器。LAVSS的灵感来自于空间音频和视觉位置之间的相关性。我们引入双耳音频作为空间线索进行的相位差，我们利用发声对象的位置表示作为额外的模态指导。我们还利用多层次的跨模态注意力来执行与音频功能的视觉位置协作。此外，我们采用了一个预先训练的单声道分离器，从丰富的单声道声音传递知识，以提高空间音频分离。这利用了单声道和双耳声道之间的相关性。在FAIR-Play数据集上的实验表明，所提出的LAVSS优于现有的视听分离基准。我们的项目页面：https://yyx666660.github.io/LAVSS/。摘要：Existing machine learning research has achieved promising results in monaural audio-visual separation (MAVS). However, most MAVS methods purely consider what the sound source is, not where it is located. This can be a problem in VR/AR scenarios, where listeners need to be able to distinguish between similar audio sources located in different directions. To address this limitation, we have generalized MAVS to spatial audio separation and proposed LAVSS: a location-guided audio-visual spatial audio separator. LAVSS is inspired by the correlation between spatial audio and visual location. We introduce the phase difference carried by binaural audio as spatial cues, and we utilize positional representations of sounding objects as additional modality guidance. We also leverage multi-level cross-modal attention to perform visual-positional collaboration with audio features. In addition, we adopt a pre-trained monaural separator to transfer knowledge from rich mono sounds to boost spatial audio separation. This exploits the correlation between monaural and binaural channels. Experiments on the FAIR-Play dataset demonstrate the superiority of the proposed LAVSS over existing benchmarks of audio-visual separation. Our project page: https://yyx666660.github.io/LAVSS/.
【4】 Musical Form Generation标题：音乐形式生成链接：https://arxiv.org/abs/2310.19842
作者：Lilac Atassi摘要：虽然最近的生成模型可以产生迷人的音乐，但它们的实用性有限。音乐中的变化往往是偶然的，导致作品缺乏结构。超过一分钟的片段可能会变得不连贯或重复。本文介绍了一种生成结构化的、任意长度的音乐作品的方法。这种方法的核心是使用条件生成模型创建音乐片段，并在这些片段之间进行过渡。生成确定高级合成的提示与创建更精细的低级细节是不同的。一个大的语言模型，然后用来建议音乐形式。摘要：While recent generative models can produce engaging music, their utility is limited. The variation in the music is often left to chance, resulting in compositions that lack structure. Pieces extending beyond a minute can become incoherent or repetitive. This paper introduces an approach for generating structured, arbitrarily long musical pieces. Central to this approach is the creation of musical segments using a conditional generative model, with transitions between these segments. The generation of prompts that determine the high-level composition is distinct from the creation of finer, lower-level details. A large language model is then used to suggest the musical form.
机器翻译，仅供参考