Whistle is a multilingual and crosslingual ASR model pretrained with weak phonetic supervision using IPA transcriptions generated by LanguageNet G2P models. Unlike self-supervised or grapheme-based approaches, Whistle leverages phoneme-level representations to enable better data efficiency, crosslingual generalization, and reduced catastrophic forgetting. Trained and evaluated on the CommonVoice-based CV-Lang10 benchmark, Whistle demonstrates superior performance on both seen and unseen languages under limited-data conditions.
Whistle was proposed in the paper
Whistle: Data-Efficient Multilingual and Crosslingual Speech Recognition via Weakly Phonetic Supervision by Saierdaer Yusuyin et al from THU-SPMI. The original code repository can be found
here.
Whistle is a Conformer based encoder model, and trained using the CTC (Connectionist Temporal Classification) approach. It was trained on ~4k hours of labelled speech data sourced from the publicly available
CommonVoice_v11Whistle checkpoints come in three configurations of varying model sizes. Including small (90 MB), medium (218 MB) and large (543 MB). And subword-based and wav2vec-based model of small size are also trained for comprison. The multilingual ASR model are trained on CV-lang10 data and then is evaluated on test dataset of corresponding language whitout fine-tuneing. All of the pre-trained checkpoints are available on the
Hugging Face Hub. The checkpoints are summarised in the following table with links to the models on the Hub:
Results are reported in Phoneme Error Rate (PER%) and Word Error Rate (WER%).
More performance please ref to
benchmarkAll of our multilingual ASR model are trained with 10 languages of cv-lang10, which has been processed in
lang-process.
But for English wav2vec-base model and multilingul wav2vec-base model, only audio are used to train the model. The language ID and training hours of the ten languages are in the following table.
1@article{yusuyin2025whistle,
2 title={Whistle: Data-efficient multilingual and crosslingual speech recognition via weakly phonetic supervision},
3 author={Yusuyin, Saierdaer and Ma, Te and Huang, Hao and Zhao, Wenbo and Ou, Zhijian},
4 journal={IEEE Transactions on Audio, Speech and Language Processing},
5 year={2025},
6 publisher={IEEE}
7}
If you encounter problems in use, you can directly raise Issues on the
github page.
