【1】MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition
链接：http://arxiv.org/pdf/2212.00500v1
作者：Xiaohuan Zhou,Jiaming Wang,Zeyu Cui,Shiliang Zhang,Zhijie Yan,Jingren Zhou,Chang Zhou机构：DAMO Academy, Alibaba Group, China备注：Submitted to ICASSP 2023摘要：In this paper, we propose a novel multi-modal multi-task encoder-decoder pre-training framework (MMSpeech) for Mandarin automatic speech recognition (ASR), which employs both unlabeled speech and text data. The main difficulty in speech-text joint pre-training comes from the significant difference between speech and text modalities, especially for Mandarin speech and text. Unlike English and other languages with an alphabetic writing system, Mandarin uses an ideographic writing system where character and sound are not tightly mapped to one another. Therefore, we propose to introduce the phoneme modality into pre-training, which can help capture modality-invariant information between Mandarin speech and text. Specifically, we employ a multi-task learning framework including five self-supervised and supervised tasks with speech and text data. For end-to-end pre-training, we introduce self-supervised speech-to-pseudo-codes (S2C) and phoneme-to-text (P2T) tasks utilizing unlabeled speech and text data, where speech-pseudo-codes pairs and phoneme-text pairs are a supplement to the supervised speech-text pairs. To train the encoder to learn better speech representation, we introduce self-supervised masked speech prediction (MSP) and supervised phoneme prediction (PP) tasks to learn to map speech into phonemes. Besides, we directly add the downstream supervised speech-to-text (S2T) task into the pre-training process, which can further improve the pre-training performance and achieve better recognition results even without fine-tuning. Experiments on AISHELL-1 show that our proposed method achieves state-of-the-art performance, with a more than 40% relative improvement compared with other pre-training methods.
【2】Deep neural network techniques for monaural speech enhancement: state of the art analysis
链接：http://arxiv.org/pdf/2212.00369v1
机构：Department of Computer Science, University of Cambridge摘要：Deep neural networks (DNN) techniques have become pervasive in domains such as natural language processing and computer vision. They have achieved great success in these domains in task such as machine translation and image generation. Due to their success, these data driven techniques have been applied in audio domain. More specifically, DNN models have been applied in speech enhancement domain to achieve denosing, dereverberation and multi-speaker separation in monaural speech enhancement. In this paper, we review some dominant DNN techniques being employed to achieve speech separation. The review looks at the whole pipeline of speech enhancement from feature extraction, how DNN based tools are modelling both global and local features of speech and model training (supervised and unsupervised). We also review the use of speech-enhancement pre-trained models to boost speech enhancement process. The review is geared towards covering the dominant trends with regards to DNN application in speech enhancement in speech obtained via a single speaker.
【3】Noisy Label Detection for Speaker Recognition
链接：http://arxiv.org/pdf/2212.00239v1
作者：Ruibin Yuan,Hanzhi Yin,Yi Wang,Yifan He,Yushi Ye,Lei Zhang机构：Carnegie Mellon University, Pittsburgh, PA , USA, Stardust.ai, Beijing, China摘要：The success of deep neural networks requires both high annotation quality and massive data. However, the size and the quality of a dataset are usually a trade-off in practice, as data collection and cleaning are expensive and time-consuming. Therefore, automatic noisy label detection (NLD) techniques are critical to real-world applications, especially those using crowdsourcing datasets. As this is an under-explored topic in automatic speaker verification (ASV), we present a simple but effective solution to the task. First, we compare the effectiveness of various commonly used metric learning loss functions under different noise settings. Then, we propose two ranking-based NLD methods, inter-class inconsistency and intra-class inconsistency ranking. They leverage the inconsistent nature of noisy labels and show high detection precision even under a high level of noise. Our solution gives rise to both efficient and effective cleaning of large-scale speaker recognition datasets.
【4】Advanced Audio Aid for Blind People
链接：http://arxiv.org/pdf/2212.00004v1
作者：Savera Sarwar,Muhammad Turab,Danish Channa,Aisha Chandio,M. Uzair Sohu,Vikram Kumar机构：Department of Electronics Engineering, Quaid-e-Awam University of Engineering, Science and Technology, Larkana, Pakistan, Department of Computer Systems Engineering, Mehran University of Engineering and Technology, Jamshoro, Pakistan, Department of Computer Science备注：Under revision. Submitted to International Conference On Emerging Technologies In Electronics, Computing And Communication (ICETECC) 2022摘要：One of the most important senses in human life is vision, without it life is totally filled with darkness. According to WHO globally millions of people are visually impaired estimated there are 285 million, of whom some millions are blind. Unfortunately, there are around 2.4 million people are blind in our beloved country Pakistan. Human are a crucial part of society and the blind community is a main part of society. The technologies are grown so far to make the life of humans easier more comfortable and more reliable for. However, this disability of the blind community would reduce their chance of using such innovative products. Therefore, the visually impaired community believe that they are burden to other societies and they do not capture in normal activities separates the blind people from society and because of this believe did not participate in the normally tasks of society . The visual impair people mainly face most of the problems in this real-time The aim of this work is to turn the real time world into an audio world by telling blind person about the objects in their way and can read printed text. This will enable blind persons to identify the things and read the text without any external help just by using the object detection and reading system in real time. Objective of this work: i) Object detection ii) Read printed text, using state-of-the-art (SOTA) technology.
【1】High Fidelity Speech Enhancement with Band-split RNN
链接：http://arxiv.org/pdf/2212.00406v1
作者：Jianwei Yu,Yi Luo,Hangting Chen,Rongzhi Gu,Chao Weng机构：Tencent AI Lab, Audio and Speech Signal Processing Oteam摘要：This report presents the development of our speech enhancement system, which includes the use of a recently proposed music separation model, the band-split recurrent neural network (BSRNN), and a MetricGAN-based training objective to improve non-differentiable quality metrics such as perceptual evaluation of speech quality (PESQ) score. Experiment conducted on Interspeech 2021 DNS challenge shows that our BSRNN system outperforms various top-ranking benchmark systems in previous deep noise suppression (DNS) challenges and achieves state-of-the-art (SOTA) result on the DNS-2020 non-blind test set in both offline and online scenarios.
【2】A Novel Speech Feature Fusion Algorithm for Text-Independent Speaker Recognition
链接：http://arxiv.org/pdf/2212.00329v1
作者：Biao Ma,Chengben Xu,Ye Zhang摘要：A novel speech feature fusion algorithm with independent vector analysis (IVA) and parallel convolutional neural network (PCNN) is proposed for text-independent speaker recognition. Firstly, some different feature types, such as the time domain (TD) features and the frequency domain (FD) features, can be extracted from a speaker's speech, and the TD and the FD features can be considered as the linear mixtures of independent feature components (IFCs) with an unknown mixing system. To estimate the IFCs, the TD and the FD features of the speaker's speech are concatenated to build the TD and the FD feature matrix, respectively. Then, a feature tensor of the speaker's speech is obtained by paralleling the TD and the FD feature matrix. To enhance the dependence on different feature types and remove the redundancies of the same feature type, the independent vector analysis (IVA) can be used to estimate the IFC matrices of TD and FD features with the feature tensor. The IFC matrices are utilized as the input of the PCNN to extract the deep features of the TD and FD features, respectively. The deep features can be integrated to obtain the fusion feature of the speaker's speech. Finally, the fusion feature of the speaker's speech is employed as the input of a deep convolutional neural network (DCNN) classifier for speaker recognition. The experimental results show the effectiveness and performances of the proposed speaker recognition system.
【3】MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition
链接：http://arxiv.org/pdf/2212.00500v1
* 与cs.SD语音【1】为同一篇
作者：Xiaohuan Zhou,Jiaming Wang,Zeyu Cui,Shiliang Zhang,Zhijie Yan,Jingren Zhou,Chang Zhou机构：DAMO Academy, Alibaba Group, China备注：Submitted to ICASSP 2023摘要：In this paper, we propose a novel multi-modal multi-task encoder-decoder pre-training framework (MMSpeech) for Mandarin automatic speech recognition (ASR), which employs both unlabeled speech and text data. The main difficulty in speech-text joint pre-training comes from the significant difference between speech and text modalities, especially for Mandarin speech and text. Unlike English and other languages with an alphabetic writing system, Mandarin uses an ideographic writing system where character and sound are not tightly mapped to one another. Therefore, we propose to introduce the phoneme modality into pre-training, which can help capture modality-invariant information between Mandarin speech and text. Specifically, we employ a multi-task learning framework including five self-supervised and supervised tasks with speech and text data. For end-to-end pre-training, we introduce self-supervised speech-to-pseudo-codes (S2C) and phoneme-to-text (P2T) tasks utilizing unlabeled speech and text data, where speech-pseudo-codes pairs and phoneme-text pairs are a supplement to the supervised speech-text pairs. To train the encoder to learn better speech representation, we introduce self-supervised masked speech prediction (MSP) and supervised phoneme prediction (PP) tasks to learn to map speech into phonemes. Besides, we directly add the downstream supervised speech-to-text (S2T) task into the pre-training process, which can further improve the pre-training performance and achieve better recognition results even without fine-tuning. Experiments on AISHELL-1 show that our proposed method achieves state-of-the-art performance, with a more than 40% relative improvement compared with other pre-training methods.
【4】Deep neural network techniques for monaural speech enhancement: state of the art analysis
链接：http://arxiv.org/pdf/2212.00369v1
* 与cs.SD语音【2】为同一篇
机构：Department of Computer Science, University of Cambridge摘要：Deep neural networks (DNN) techniques have become pervasive in domains such as natural language processing and computer vision. They have achieved great success in these domains in task such as machine translation and image generation. Due to their success, these data driven techniques have been applied in audio domain. More specifically, DNN models have been applied in speech enhancement domain to achieve denosing, dereverberation and multi-speaker separation in monaural speech enhancement. In this paper, we review some dominant DNN techniques being employed to achieve speech separation. The review looks at the whole pipeline of speech enhancement from feature extraction, how DNN based tools are modelling both global and local features of speech and model training (supervised and unsupervised). We also review the use of speech-enhancement pre-trained models to boost speech enhancement process. The review is geared towards covering the dominant trends with regards to DNN application in speech enhancement in speech obtained via a single speaker.
【5】 Noisy Label Detection for Speaker Recognition
链接：http://arxiv.org/pdf/2212.00239v1
* 与cs.SD语音【3】为同一篇
作者：Ruibin Yuan,Hanzhi Yin,Yi Wang,Yifan He,Yushi Ye,Lei Zhang机构：Carnegie Mellon University, Pittsburgh, PA , USA, Stardust.ai, Beijing, China摘要：The success of deep neural networks requires both high annotation quality and massive data. However, the size and the quality of a dataset are usually a trade-off in practice, as data collection and cleaning are expensive and time-consuming. Therefore, automatic noisy label detection (NLD) techniques are critical to real-world applications, especially those using crowdsourcing datasets. As this is an under-explored topic in automatic speaker verification (ASV), we present a simple but effective solution to the task. First, we compare the effectiveness of various commonly used metric learning loss functions under different noise settings. Then, we propose two ranking-based NLD methods, inter-class inconsistency and intra-class inconsistency ranking. They leverage the inconsistent nature of noisy labels and show high detection precision even under a high level of noise. Our solution gives rise to both efficient and effective cleaning of large-scale speaker recognition datasets.
【6】 Advanced Audio Aid for Blind People
链接：http://arxiv.org/pdf/2212.00004v1
* 与cs.SD语音【4】为同一篇
作者：Savera Sarwar,Muhammad Turab,Danish Channa,Aisha Chandio,M. Uzair Sohu,Vikram Kumar机构：Department of Electronics Engineering, Quaid-e-Awam University of Engineering, Science and Technology, Larkana, Pakistan, Department of Computer Systems Engineering, Mehran University of Engineering and Technology, Jamshoro, Pakistan, Department of Computer Science备注：Under revision. Submitted to International Conference On Emerging Technologies In Electronics, Computing And Communication (ICETECC) 2022摘要：One of the most important senses in human life is vision, without it life is totally filled with darkness. According to WHO globally millions of people are visually impaired estimated there are 285 million, of whom some millions are blind. Unfortunately, there are around 2.4 million people are blind in our beloved country Pakistan. Human are a crucial part of society and the blind community is a main part of society. The technologies are grown so far to make the life of humans easier more comfortable and more reliable for. However, this disability of the blind community would reduce their chance of using such innovative products. Therefore, the visually impaired community believe that they are burden to other societies and they do not capture in normal activities separates the blind people from society and because of this believe did not participate in the normally tasks of society . The visual impair people mainly face most of the problems in this real-time The aim of this work is to turn the real time world into an audio world by telling blind person about the objects in their way and can read printed text. This will enable blind persons to identify the things and read the text without any external help just by using the object detection and reading system in real time. Objective of this work: i) Object detection ii) Read printed text, using state-of-the-art (SOTA) technology.
每日语音速递