Contents

  • Evaluation Framework: The evaluation framework of SLT2022 SUPERB challenge.

  • Upstream Specification: The Upstream Specification about data, programming language, and interface.

  • Leaderboard Submission: The definitions of two leaderboards and how to make a submission.

  • Overall Metrics: Metrics definitions.


Abstract

This challenge

  • benchmarks generalizability of Self-Supervised Learning (SSL) speech model.

    • The similar evaluation framework introduced in SUPERB Benchmark

    • Frozen SSL model (termed Upstream)

    • Extract multiple frozen hidden states from upstream and trains a learnable weighted-sum over them along with the downstream model task-by-task

  • has several downstream tasks

    • Content: PR, ASR, QbE

    • Speaker: SID, ASV, SD

    • Paralinguistics: ER

    • Semantics: ST

    • Generation: SE, SS

  • has two leaderboards to submit:

    • We will use several different learning rates for each downstream task training while fixing all the other hyperparameters and present the best result on leaderboard

    • Fixed downstream model architectures

    • Frozen upstream model

    • Except above, no other limitations in downstream training procedure (e.g. hyperparameter, optimizer, etc.)

    • Can only evaluated towards a subset of tasks

    • Public-set: Dataset is publicly available, submit prediction file

    • Hidden-set: Dataset is totally inaccessible, submit model (model definition & pre-trained weights)

  • encourage innovative and aims to be less competitive:

    • Participants are free to choose whether present their submissions on leaderboard or not

    • In addition to model accuracy, we will have metrics such as number of parameters and operations to capture computation efficiency of proposed approach. We encourage algorithm improvement from diverse perspectives.


Participant Requirements

The following describes the requirements for a team to join the challenge.

Submit an upstream model to the hidden-set leaderboard

The public-set is for the upstream development purpose. You can pre-train your upstream and evaluate it with any method you like. You are required to submit at least one upstream model to the hidden-set leaderboard.


System description paper (Optional)

To verify the submitted upstream follows the challenge policy, we suggest to submit a system description paper in SLT submission format without the page limit. The paper should describe the method for your submissions, containing at least the following materials:

  • SSL objectives

  • Model architecture

  • Pre-training data

  • Parameter size for each submission

The submission should follow the challenge policy and the paper is expected to be well-written. The deadline for the system description paper is Nov 1, 2022. Review of these papers will be the responsibility of our SUPERB challenge organisers. Accepted system description papers will not be indexed by the IEEE, but authors of these papers will have the opportunity to present their work in a dedicated session at the Workshop.

The system description paper is for the challenge review only and is not considered as SLT workshop paper by default, but we encourage participants to submit their methods' papers to SLT workshop. If the method turns out to be similar to that used for the final selected submissions, the same paper can be used as the system description paper.

Invitee Announcement and Presentation

After reviewing the system description papers and comparing their performance with the hidden-set private scores, we will announce the final results on Dec 25, 2022. And the outstanding one may be invited to present their methods in SLT workshop.

Timeline

  • Mar 02, 2022: Challenge announcement

  • Mar 02, 2022: Leaderboard is online and accepts submissions

  • Jul 15, 2022: SLT paper submission (encouraged)

  • Sep 30, 2022: SLT paper notification

  • Nov 01, 2022: System description paper submission deadline

  • Dec 20, 2022: Result and invitee announcement

  • at least after the end of 2022: End of hidden-set submission

  • Jan 9 - 12, 2023: SLT workshop presentation


Organizers

Hung-yi Lee (NTU)

Tzu-Hsun Feng (NTU)

Tzu-Quan Lin (NTU)

Haibin Wu (NTU)

Shinji Watanabe (CMU)

Xuankai Chang (CMU)

Ching-Feng Yeh (Meta)

Annie Dong (Meta)

Zili Huang (JHU)


Contact

superb.announcement@gmail.com


website

https://superbbenchmark.org/challenge-slt2022/challenge_overview#Contents