NEST FastConformer — Self-Supervised Speech Encoder (LibriSpeech-100)
This is a self-supervised (SSL) speech encoder pretrained from scratch with the NEST objective using the NVIDIA NeMo framework (
https://github.com/NVIDIA-NeMo/Speech).
Important: This is a pretrained encoder, not a speech-recognition model. It cannot transcribe audio on its own; it has no text decoder and was never trained on transcripts. It is intended as an initialization / starting point for downstream fine-tuning (e.g. CTC or RNN-T ASR).
Model summary
- Type: Self-supervised speech encoder (masked-prediction pretraining)
- Objective: NEST; the model masks blocks of the input audio and learns to predict the masked content, learning general speech representations without any labels.
- Architecture: FastConformer encoder (ConformerEncoder)
- Encoder config: d_model=512, n_layers=17, n_heads=8, subsampling=dw_striding, subsampling_factor=8
- Parameters: ~108M (encoder)
- Framework: NVIDIA NeMo
- Language: English
- Format: .nemo
Intended use
A base model for downstream ASR fine-tuning. Typical workflow:
- Load this .nemo as the encoder initialization for a CTC or RNN-T ASR model.
- Fine-tune on a labeled (audio + transcript) dataset.
- Evaluate the fine-tuned model with Word Error Rate (WER).
It is not suitable for direct transcription, and it is a research/learning-scale checkpoint (see Limitations).
Loading (NeMo)
1import nemo.collections.asr as nemo_asr
2model = nemo_asr.models.ASRModel.restore_from("nest_encoder_best.nemo")
To use only the encoder weights for fine-tuning an ASR model, initialize your CTC/RNN-T model and load the encoder weights from this checkpoint via NeMo's init_from_nemo_model with an include: [encoder] filter, keeping the encoder architecture identical (n_layers=17, d_model=512).
Training details
- Pretraining method: NEST masked-prediction (self-supervised, no transcripts)
- Data: LibriSpeech train-clean-100 (~100 hours of English read speech), used as unlabeled audio (transcripts removed)
- Audio: 16 kHz mono
- Loss: MultiMLMLoss (masked-token prediction over a random-projection quantizer, 8192 codes)
- Optimizer: AdamW
- Schedule: Noam annealing
- Precision: bf16
- Best validation loss: ~5.76 (masked-prediction cross-entropy), reached at ~26 epochs (started from ~6.24 at epoch 0; random-guess baseline is about ln(8192), roughly 8.9)
The pretraining loss decreased steadily from ~8.9 (random) to 6.24 to 5.76, indicating the encoder learned meaningful speech structure from unlabeled audio.
Evaluation
Because this is a self-supervised encoder with no text output, standard ASR metrics such as WER do not apply directly. The only intrinsic training metric is the pretraining masked-prediction loss (best ~5.76, above).
The meaningful (extrinsic) evaluation of an SSL encoder is downstream: fine-tune it on a labeled task and measure WER, comparing against a randomly-initialized encoder of the same architecture. A downstream WER for this checkpoint has not been established and is left to the user.
Limitations
- Learning-scale checkpoint. Pretrained on only ~100 hours for a limited number of effective epochs. Production SSL encoders use thousands to tens of thousands of hours; this checkpoint will not match production-grade representations.
- Not a transcriber. No decoder, no text; it cannot recognize speech without fine-tuning.
- Under-trained encoders can underperform random initialization on some downstream fine-tunes. Users should A/B test against random init for their specific task.
- English read speech only. Pretrained on audiobook-style speech; representations may transfer poorly to other domains.
License and attribution
Released under CC-BY-4.0, inherited from the LibriSpeech dataset it was pretrained on. Attribution is required.
- Pretraining data: LibriSpeech (Panayotov et al., 2015), CC-BY-4.0 — https://www.openslr.org/12
- Framework: NVIDIA NeMo — https://github.com/NVIDIA-NeMo/Speech
- Pretraining method: NEST (Noise-robust Efficient Self-supervised pre-Training)
Citation
- NVIDIA NeMo — https://github.com/NVIDIA-NeMo/Speech
- LibriSpeech — V. Panayotov, G. Chen, D. Povey, S. Khudanpur, "Librispeech: An ASR corpus based on public domain audio books," ICASSP 2015.
