【1】 Audio-visual scene classification via contrastive event-object alignment and semantic-based fusion
标题：基于对比事件—对象对齐和语义融合的视听场景分类
链接：https://arxiv.org/abs/2208.02086
作者：Yuanbo Hou,Bo Kang,Dick Botteldooren机构：WAVES Research Group, Ghent University, Gent, Belgium, IDLAB摘要：以往的场景分类研究主要基于音频或视频信号，而人类对环境场景的感知是通过多种感官进行的，近年来的视听场景分类研究分别对目标数据集上的大规模音频和图像预训练模型进行微调，然后融合音频模型和视觉模型的中间表示，或者在剪辑级融合两种模型的粗粒度决策，这样的方法忽略了视听场景中的细节音频事件和视觉对象（AVS）、而人类通常通过声音事件和视觉对象以及它们之间的一致性来识别不同的场景。对AVS中音频事件和视觉对象的粒度信息进行了细化，协调了音频事件和视觉对象之间的隐含关系，本文提出了一个带有对比事件—对象对齐的多分支模型（CEOA）和基于语义的融合CEOA的目标是通过比较视听事件—对象对之间的差异来对齐所学习的音频事件和视觉对象的嵌入。然后，与特定音频事件相关的视觉对象和与特定音频事件相关的视觉对象通过交叉注意被强调，并经过SF进行语义级融合。（1）CEOA和SF相结合的AVSC模型的性能优于单模态模型和单模态模型，即视听效果优于单模态模型。（2）CEOA将音频事件和相关视觉对象的嵌入在细粒度的水平上对齐，而SF有效地将两者集成在一起。3）与其他大规模集成系统相比，该模型在不使用额外数据集和数据扩充技巧的情况下，表现出了较好的性能。摘要：Previous works on scene classification are mainly based on audio or visual signals, while humans perceive the environmental scenes through multiple senses. Recent studies on audio-visual scene classification separately fine-tune the largescale audio and image pre-trained models on the target dataset, then either fuse the intermediate representations of the audio model and the visual model, or fuse the coarse-grained decision of both models at the clip level. Such methods ignore the detailed audio events and visual objects in audio-visual scenes (AVS), while humans often identify different scenes through audio events and visual objects within and the congruence between them. To exploit the fine-grained information of audio events and visual objects in AVS, and coordinate the implicit relationship between audio events and visual objects, this paper proposes a multibranch model equipped with contrastive event-object alignment (CEOA) and semantic-based fusion (SF) for AVSC. CEOA aims to align the learned embeddings of audio events and visual objects by comparing the difference between audio-visual event-object pairs. Then, visual objects associated with certain audio events and vice versa are accentuated by cross-attention and undergo SF for semantic-level fusion. Experiments show that: 1) the proposed AVSC model equipped with CEOA and SF outperforms the results of audio-only and visual-only models, i.e., the audio-visual results are better than the results from a single modality. 2) CEOA aligns the embeddings of audio events and related visual objects on a fine-grained level, and the SF effectively integrates both; 3) Compared with other large-scale integrated systems, the proposed model shows competitive performance, even without using additional datasets and data augmentation tricks.
【2】 The SJTU System for Short-duration Speaker Verification Challenge 2021
标题：2021年SJTU短时长说话人验证挑战赛系统
链接：https://arxiv.org/abs/2208.01933
作者：Bing Han,Zhengyang Chen,Zhikai Zhou,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Published by Interspeech 2021摘要：本文提出了一种用于短时说话人确认的SJTU系统，该系统同时支持文本相关和文本无关两种任务（SdSV）challenge 2021.在该挑战中，我们研究了不同的强嵌入提取器来提取鲁棒的说话人嵌入.对于文本无关的任务，我们研究了语言相关的自适应snrom来提高跨语言验证条件下的系统性能.对于文本相关的任务，重点研究了基于大规模域外数据预训练的模型的域内微调策略.为了提高不同说话人对同一短语的区分度，提出了几种新颖的短语感知微调策略和短语感知神经PLDA.这些策略进一步提高了系统性能.最后，我们融合了不同系统的得分，并且我们的融合系统在主要评估度量上在任务1中获得了0.0473（秩3），在任务2中获得了0.0581（秩8）。摘要：This paper presents the SJTU system for both text-dependent and text-independent tasks in short-duration speaker verification (SdSV) challenge 2021. In this challenge, we explored different strong embedding extractors to extract robust speaker embedding. For text-independent task, language-dependent adaptive snorm is explored to improve the system performance under the cross-lingual verification condition. For text-dependent task, we mainly focus on the in-domain fine-tuning strategies based on the model pre-trained on large-scale out-of-domain data. In order to improve the distinction between different speakers uttering the same phrase, we proposed several novel phrase-aware fine-tuning strategies and phrase-aware neural PLDA. With such strategies, the system performance is further improved. Finally, we fused the scores of different systems, and our fusion systems achieved 0.0473 in Task1 (rank 3) and 0.0581 in Task2 (rank 8) on the primary evaluation metric.
【3】 Self-Supervised Speaker Verification Using Dynamic Loss-Gate and Label Correction
标题：基于动态损失门和标记校正的自监督说话人确认
链接：https://arxiv.org/abs/2208.01928
作者：Bing Han,Zhengyang Chen,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by Interspeech 2022摘要：在自监督说话人确认系统中，由于存在大量不可靠的伪标签，伪标签的质量决定了系统的上界（DLG-LC）来减轻由于不可靠的估计标签而引起的性能下降。在DLG中，我们采用高斯混合模型（GMM）动态地对丢失分布建模并利用估计的GMM自动区分可靠标签和不可靠标签，为了更好地利用不可靠的数据而不是直接丢弃它们，我们利用模型预测来修正不可靠的标签.此外，我们在实验中应用了无负对的DINO框架来进一步改进.与最著名的基于自监督学习的说话人确认系统相比，我们提出的DLG-LC的收敛速度更快，达到了11.45%，Voxceleb1评价数据集的Vox-O、Vox-E和Vox-H试验相对改善18.35%和15.16%。摘要：For self-supervised speaker verification, the quality of pseudo labels decides the upper bound of the system due to the massive unreliable labels. In this work, we propose dynamic loss-gate and label correction (DLG-LC) to alleviate the performance degradation caused by unreliable estimated labels. In DLG, we adopt Gaussian Mixture Model (GMM) to dynamically model the loss distribution and use the estimated GMM to distinguish the reliable and unreliable labels automatically. Besides, to better utilize the unreliable data instead of dropping them directly, we correct the unreliable label with model predictions. Moreover, we apply the negative-pairs-free DINO framework in our experiments for further improvement. Compared to the best-known speaker verification system with self-supervised learning, our proposed DLG-LC converges faster and achieves 11.45%, 18.35% and 15.16% relative improvement on Vox-O, Vox-E and Vox-H trials of Voxceleb1 evaluation dataset.
【4】 Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding
标题：基于多模态风格编码对抗解纠缠的文本和语音驱动手势动画Zero-Shot风格转换
链接：https://arxiv.org/abs/2208.01917
作者：Mireille Fares,Michele Grimaldi,Catherine Pelachaud,Nicolas Obin摘要：虚拟主体的行为风格建模是实现虚拟主体交互个性化的一个重要因素.本文提出了一种高效的机器学习方法，用于合成由韵律特征和不同说话人风格的文本驱动的手势，包括训练过程中未看到的那些.我们的模型实现zero shot多模态风格转换，由来自包含不同说话人视频的PATS数据库的多模态数据驱动.我们认为风格在说话过程中是普遍存在的，它使交际行为具有表现力，而言语内容是由多模态信号和文本承载的，这种内容和风格的分离方案使我们能够直接推断说话者的风格嵌入，即使说话者的数据不是训练阶段的一部分，我们的模型的第一个目标是基于两种音频和文本模态的内容来生成源说话人的手势。第二个目标是在目标说话人的多模态行为风格嵌入上调节源说话人预测的姿势。第三个目标是在训练过程中不需要重新训练模型就可以实现扬声器的zero shot式传输。（1）说话人风格编码器网络，其学习以从目标说话人多模态数据生成固定维说话人嵌入风格，以及（2）序列到序列合成网络，其基于源说话者的输入模态的内容并以说话者风格嵌入为条件来合成姿势。我们评估我们的模型可以合成源说话人的手势，并将目标说话人风格变化的知识转移到zero shot设置中的手势生成任务。我们将2D手势转换成3D姿势，并制作3D动画，我们进行客观和主观的评估来验证我们的方法，并将其与基准进行比较。摘要：Modeling virtual agents with behavior style is one factor for personalizing human agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of speaker whose data are not part of the training phase, without requiring any further training or fine tuning. The first goal of our model is to generate the gestures of a source speaker based on the content of two audio and text modalities. The second goal is to condition the source speaker predicted gestures on the multimodal behavior style embedding of a target speaker. The third goal is to allow zero shot style transfer of speakers unseen during training without retraining the model. Our system consists of: (1) a speaker style encoder network that learns to generate a fixed dimensional speaker embedding style from a target speaker multimodal data and (2) a sequence to sequence synthesis network that synthesizes gestures based on the content of the input modalities of a source speaker and conditioned on the speaker style embedding. We evaluate that our model can synthesize gestures of a source speaker and transfer the knowledge of target speaker style variability to the gesture generation task in a zero shot setup. We convert the 2D gestures to 3D poses and produce 3D animations. We conduct objective and subjective evaluations to validate our approach and compare it with a baseline.
【5】 VQ-T: RNN Transducers using Vector-Quantized Prediction Network States
标题：VQ-T：使用矢量量化预测网络状态的RNN换能器
链接：https://arxiv.org/abs/2208.01818
作者：Jiatong Shi,George Saon,David Haws,Shinji Watanabe,Brian Kingsbury机构：Language Technologies Institute, Carnegie Mellon University, PA, USA, IBM Research AI, Yorktown Heights, NY, USA备注：Interspeech 2022 accepted paper摘要：波束搜索是端到端模型的主要ASR解码算法，它产生树型结构的假设，但是最近的研究表明，假设合并解码可以实现更有效的搜索，并且具有相当或更好的性能，但是，循环网络中的完整上下文与假设合并不兼容，我们建议使用向量量化的长短时记忆单元（VQ-LSTM）在RNN传感器预测网络中的应用，可以主动地合并假设以用于网格生成。我们在Switchboard语料库上的实验表明，所提出的VQ RNN转换器比具有规则预测网络的转换器提高了ASR性能，同时还产生了具有非常低的oracle字的更密集的格额外的语言模型重评分实验也证明了所提出的格点生成方案的有效性。摘要：Beam search, which is the dominant ASR decoding algorithm for end-to-end models, generates tree-structured hypotheses. However, recent studies have shown that decoding with hypothesis merging can achieve a more efficient search with comparable or better performance. But, the full context in recurrent networks is not compatible with hypothesis merging. We propose to use vector-quantized long short-term memory units (VQ-LSTM) in the prediction network of RNN transducers. By training the discrete representation jointly with the ASR network, hypotheses can be actively merged for lattice generation. Our experiments on the Switchboard corpus show that the proposed VQ RNN transducers improve ASR performance over transducers with regular prediction networks while also producing denser lattices with a very low oracle word error rate (WER) for the same beam size. Additional language model rescoring experiments also demonstrate the effectiveness of the proposed lattice generation scheme.
【6】 A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis
标题：粤语神经语音合成中升调建模的研究
链接：https://arxiv.org/abs/2208.02189
作者：Qibing Bai,Tom Ko,Yu Zhang机构：Department of Computer Science and Engineering, Southern University of Science and Technology, ByteDance AI Lab备注：Accepted by INTERSPEECH 2022摘要：在人类语言中，说话者的态度不能单靠文字内容来表达，而必须配合语调。陈述性问句是日常粤语会话中常用的问句，而且通常是用升调来表达的。（TTS）系统由于语义信息的丢失而不能合成这些句子的升调。虽然用额外的语言模型来补充系统已经变得越来越普遍，但是它们在模拟上升语调方面的性能却没有得到很好的研究，我们提出了一个基于BERT的语句/疑问句分类器来补充粤语TTS模型，设计了不同的训练策略并比较了它们的性能。在粤语语料CanTTS上的实验结果表明，分离训练方法具有最好的泛化性能和可行性。摘要：In human speech, the attitude of a speaker cannot be fully expressed only by the textual content. It has to come along with the intonation. Declarative questions are commonly used in daily Cantonese conversations, and they are usually uttered with rising intonation. Vanilla neural text-to-speech (TTS) systems are not capable of synthesizing rising intonation for these sentences due to the loss of semantic information. Though it has become more common to complement the systems with extra language models, their performance in modeling rising intonation is not well studied. In this paper, we propose to complement the Cantonese TTS model with a BERT-based statement/question classifier. We design different training strategies and compare their performance. We conduct our experiments on a Cantonese corpus named CanTTS. Empirical results show that the separate training approach obtains the best generalization performance and feasibility.
【1】 A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis
标题：粤语神经语音合成中升调建模的研究
链接：https://arxiv.org/abs/2208.02189
* 与cs.SD语音【6】为同一篇
作者：Qibing Bai,Tom Ko,Yu Zhang机构：Department of Computer Science and Engineering, Southern University of Science and Technology, ByteDance AI Lab备注：Accepted by INTERSPEECH 2022摘要：在人类语言中，说话者的态度不能单靠文字内容来表达，而必须配合语调。陈述性问句是日常粤语会话中常用的问句，而且通常是用升调来表达的。（TTS）系统由于语义信息的丢失而不能合成这些句子的升调。虽然用额外的语言模型来补充系统已经变得越来越普遍，但是它们在模拟上升语调方面的性能却没有得到很好的研究，我们提出了一个基于BERT的语句/疑问句分类器来补充粤语TTS模型，设计了不同的训练策略并比较了它们的性能。在粤语语料CanTTS上的实验结果表明，分离训练方法具有最好的泛化性能和可行性。摘要：In human speech, the attitude of a speaker cannot be fully expressed only by the textual content. It has to come along with the intonation. Declarative questions are commonly used in daily Cantonese conversations, and they are usually uttered with rising intonation. Vanilla neural text-to-speech (TTS) systems are not capable of synthesizing rising intonation for these sentences due to the loss of semantic information. Though it has become more common to complement the systems with extra language models, their performance in modeling rising intonation is not well studied. In this paper, we propose to complement the Cantonese TTS model with a BERT-based statement/question classifier. We design different training strategies and compare their performance. We conduct our experiments on a Cantonese corpus named CanTTS. Empirical results show that the separate training approach obtains the best generalization performance and feasibility.
【2】 Audio-visual scene classification via contrastive event-object alignment and semantic-based fusion
标题：基于对比事件—对象对齐和语义融合的视听场景分类
链接：https://arxiv.org/abs/2208.02086
* 与cs.SD语音【1】为同一篇
作者：Yuanbo Hou,Bo Kang,Dick Botteldooren机构：WAVES Research Group, Ghent University, Gent, Belgium, IDLAB摘要：以往的场景分类研究主要基于音频或视频信号，而人类对环境场景的感知是通过多种感官进行的，近年来的视听场景分类研究分别对目标数据集上的大规模音频和图像预训练模型进行微调，然后融合音频模型和视觉模型的中间表示，或者在剪辑级融合两种模型的粗粒度决策，这样的方法忽略了视听场景中的细节音频事件和视觉对象（AVS）、而人类通常通过声音事件和视觉对象以及它们之间的一致性来识别不同的场景。对AVS中音频事件和视觉对象的粒度信息进行了细化，协调了音频事件和视觉对象之间的隐含关系，本文提出了一个带有对比事件—对象对齐的多分支模型（CEOA）和基于语义的融合CEOA的目标是通过比较视听事件—对象对之间的差异来对齐所学习的音频事件和视觉对象的嵌入。然后，与特定音频事件相关的视觉对象和与特定音频事件相关的视觉对象通过交叉注意被强调，并经过SF进行语义级融合。（1）CEOA和SF相结合的AVSC模型的性能优于单模态模型和单模态模型，即视听效果优于单模态模型。（2）CEOA将音频事件和相关视觉对象的嵌入在细粒度的水平上对齐，而SF有效地将两者集成在一起。3）与其他大规模集成系统相比，该模型在不使用额外数据集和数据扩充技巧的情况下，表现出了较好的性能。摘要：Previous works on scene classification are mainly based on audio or visual signals, while humans perceive the environmental scenes through multiple senses. Recent studies on audio-visual scene classification separately fine-tune the largescale audio and image pre-trained models on the target dataset, then either fuse the intermediate representations of the audio model and the visual model, or fuse the coarse-grained decision of both models at the clip level. Such methods ignore the detailed audio events and visual objects in audio-visual scenes (AVS), while humans often identify different scenes through audio events and visual objects within and the congruence between them. To exploit the fine-grained information of audio events and visual objects in AVS, and coordinate the implicit relationship between audio events and visual objects, this paper proposes a multibranch model equipped with contrastive event-object alignment (CEOA) and semantic-based fusion (SF) for AVSC. CEOA aims to align the learned embeddings of audio events and visual objects by comparing the difference between audio-visual event-object pairs. Then, visual objects associated with certain audio events and vice versa are accentuated by cross-attention and undergo SF for semantic-level fusion. Experiments show that: 1) the proposed AVSC model equipped with CEOA and SF outperforms the results of audio-only and visual-only models, i.e., the audio-visual results are better than the results from a single modality. 2) CEOA aligns the embeddings of audio events and related visual objects on a fine-grained level, and the SF effectively integrates both; 3) Compared with other large-scale integrated systems, the proposed model shows competitive performance, even without using additional datasets and data augmentation tricks.
【3】 The SJTU System for Short-duration Speaker Verification Challenge 2021
标题：2021年SJTU短时长说话人验证挑战赛系统
链接：https://arxiv.org/abs/2208.01933
* 与cs.SD语音【2】为同一篇
作者：Bing Han,Zhengyang Chen,Zhikai Zhou,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Published by Interspeech 2021摘要：本文提出了一种用于短时说话人确认的SJTU系统，该系统同时支持文本相关和文本无关两种任务（SdSV）challenge 2021.在该挑战中，我们研究了不同的强嵌入提取器来提取鲁棒的说话人嵌入.对于文本无关的任务，我们研究了语言相关的自适应snrom来提高跨语言验证条件下的系统性能.对于文本相关的任务，重点研究了基于大规模域外数据预训练的模型的域内微调策略.为了提高不同说话人对同一短语的区分度，提出了几种新颖的短语感知微调策略和短语感知神经PLDA.这些策略进一步提高了系统性能.最后，我们融合了不同系统的得分，并且我们的融合系统在主要评估度量上在任务1中获得了0.0473（秩3），在任务2中获得了0.0581（秩8）。摘要：This paper presents the SJTU system for both text-dependent and text-independent tasks in short-duration speaker verification (SdSV) challenge 2021. In this challenge, we explored different strong embedding extractors to extract robust speaker embedding. For text-independent task, language-dependent adaptive snorm is explored to improve the system performance under the cross-lingual verification condition. For text-dependent task, we mainly focus on the in-domain fine-tuning strategies based on the model pre-trained on large-scale out-of-domain data. In order to improve the distinction between different speakers uttering the same phrase, we proposed several novel phrase-aware fine-tuning strategies and phrase-aware neural PLDA. With such strategies, the system performance is further improved. Finally, we fused the scores of different systems, and our fusion systems achieved 0.0473 in Task1 (rank 3) and 0.0581 in Task2 (rank 8) on the primary evaluation metric.
【4】 Self-Supervised Speaker Verification Using Dynamic Loss-Gate and Label Correction
标题：基于动态损失门和标记校正的自监督说话人确认
链接：https://arxiv.org/abs/2208.01928
* 与cs.SD语音【3】为同一篇
作者：Bing Han,Zhengyang Chen,Yanmin Qian机构：MoE Key Lab of Artificial Intelligence, AI Institute, X-LANCE Lab, Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China备注：Accepted by Interspeech 2022摘要：在自监督说话人确认系统中，由于存在大量不可靠的伪标签，伪标签的质量决定了系统的上界（DLG-LC）来减轻由于不可靠的估计标签而引起的性能下降。在DLG中，我们采用高斯混合模型（GMM）动态地对丢失分布建模并利用估计的GMM自动区分可靠标签和不可靠标签，为了更好地利用不可靠的数据而不是直接丢弃它们，我们利用模型预测来修正不可靠的标签.此外，我们在实验中应用了无负对的DINO框架来进一步改进.与最著名的基于自监督学习的说话人确认系统相比，我们提出的DLG-LC的收敛速度更快，达到了11.45%，Voxceleb1评价数据集的Vox-O、Vox-E和Vox-H试验相对改善18.35%和15.16%。摘要：For self-supervised speaker verification, the quality of pseudo labels decides the upper bound of the system due to the massive unreliable labels. In this work, we propose dynamic loss-gate and label correction (DLG-LC) to alleviate the performance degradation caused by unreliable estimated labels. In DLG, we adopt Gaussian Mixture Model (GMM) to dynamically model the loss distribution and use the estimated GMM to distinguish the reliable and unreliable labels automatically. Besides, to better utilize the unreliable data instead of dropping them directly, we correct the unreliable label with model predictions. Moreover, we apply the negative-pairs-free DINO framework in our experiments for further improvement. Compared to the best-known speaker verification system with self-supervised learning, our proposed DLG-LC converges faster and achieves 11.45%, 18.35% and 15.16% relative improvement on Vox-O, Vox-E and Vox-H trials of Voxceleb1 evaluation dataset.
【5】 Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encoding
标题：基于多模态风格编码对抗解纠缠的文本和语音驱动手势动画Zero-Shot风格转换
链接：https://arxiv.org/abs/2208.01917
* 与cs.SD语音【4】为同一篇
作者：Mireille Fares,Michele Grimaldi,Catherine Pelachaud,Nicolas Obin摘要：虚拟主体的行为风格建模是实现虚拟主体交互个性化的一个重要因素.本文提出了一种高效的机器学习方法，用于合成由韵律特征和不同说话人风格的文本驱动的手势，包括训练过程中未看到的那些.我们的模型实现zero shot多模态风格转换，由来自包含不同说话人视频的PATS数据库的多模态数据驱动.我们认为风格在说话过程中是普遍存在的，它使交际行为具有表现力，而言语内容是由多模态信号和文本承载的，这种内容和风格的分离方案使我们能够直接推断说话者的风格嵌入，即使说话者的数据不是训练阶段的一部分，我们的模型的第一个目标是基于两种音频和文本模态的内容来生成源说话人的手势。第二个目标是在目标说话人的多模态行为风格嵌入上调节源说话人预测的姿势。第三个目标是在训练过程中不需要重新训练模型就可以实现扬声器的zero shot式传输。（1）说话人风格编码器网络，其学习以从目标说话人多模态数据生成固定维说话人嵌入风格，以及（2）序列到序列合成网络，其基于源说话者的输入模态的内容并以说话者风格嵌入为条件来合成姿势。我们评估我们的模型可以合成源说话人的手势，并将目标说话人风格变化的知识转移到zero shot设置中的手势生成任务。我们将2D手势转换成3D姿势，并制作3D动画，我们进行客观和主观的评估来验证我们的方法，并将其与基准进行比较。摘要：Modeling virtual agents with behavior style is one factor for personalizing human agent interaction. We propose an efficient yet effective machine learning approach to synthesize gestures driven by prosodic features and text in the style of different speakers including those unseen during training. Our model performs zero shot multimodal style transfer driven by multimodal data from the PATS database containing videos of various speakers. We view style as being pervasive while speaking, it colors the communicative behaviors expressivity while speech content is carried by multimodal signals and text. This disentanglement scheme of content and style allows us to directly infer the style embedding even of speaker whose data are not part of the training phase, without requiring any further training or fine tuning. The first goal of our model is to generate the gestures of a source speaker based on the content of two audio and text modalities. The second goal is to condition the source speaker predicted gestures on the multimodal behavior style embedding of a target speaker. The third goal is to allow zero shot style transfer of speakers unseen during training without retraining the model. Our system consists of: (1) a speaker style encoder network that learns to generate a fixed dimensional speaker embedding style from a target speaker multimodal data and (2) a sequence to sequence synthesis network that synthesizes gestures based on the content of the input modalities of a source speaker and conditioned on the speaker style embedding. We evaluate that our model can synthesize gestures of a source speaker and transfer the knowledge of target speaker style variability to the gesture generation task in a zero shot setup. We convert the 2D gestures to 3D poses and produce 3D animations. We conduct objective and subjective evaluations to validate our approach and compare it with a baseline.
【6】 VQ-T: RNN Transducers using Vector-Quantized Prediction Network States
标题：VQ-T：使用矢量量化预测网络状态的RNN换能器
链接：https://arxiv.org/abs/2208.01818
* 与cs.SD语音【5】为同一篇
作者：Jiatong Shi,George Saon,David Haws,Shinji Watanabe,Brian Kingsbury机构：Language Technologies Institute, Carnegie Mellon University, PA, USA, IBM Research AI, Yorktown Heights, NY, USA备注：Interspeech 2022 accepted paper摘要：波束搜索是端到端模型的主要ASR解码算法，它产生树型结构的假设，但是最近的研究表明，假设合并解码可以实现更有效的搜索，并且具有相当或更好的性能，但是，循环网络中的完整上下文与假设合并不兼容，我们建议使用向量量化的长短时记忆单元（VQ-LSTM）在RNN传感器预测网络中的应用，可以主动地合并假设以用于网格生成。我们在Switchboard语料库上的实验表明，所提出的VQ RNN转换器比具有规则预测网络的转换器提高了ASR性能，同时还产生了具有非常低的oracle字的更密集的格额外的语言模型重评分实验也证明了所提出的格点生成方案的有效性。摘要：Beam search, which is the dominant ASR decoding algorithm for end-to-end models, generates tree-structured hypotheses. However, recent studies have shown that decoding with hypothesis merging can achieve a more efficient search with comparable or better performance. But, the full context in recurrent networks is not compatible with hypothesis merging. We propose to use vector-quantized long short-term memory units (VQ-LSTM) in the prediction network of RNN transducers. By training the discrete representation jointly with the ASR network, hypotheses can be actively merged for lattice generation. Our experiments on the Switchboard corpus show that the proposed VQ RNN transducers improve ASR performance over transducers with regular prediction networks while also producing denser lattices with a very low oracle word error rate (WER) for the same beam size. Additional language model rescoring experiments also demonstrate the effectiveness of the proposed lattice generation scheme.
机器翻译，仅供参考
每日语音速递