Kazakh ASR — omniASR CTC 300M v2 Fine-tuning
Ablation Study: Learning Rate
Task: Kazakh speech recognition (ASR)
Model: Meta omniASR_CTC_300M_v2 (1024-dim, 24 layers, 16 heads)
Tokenizer: omniASR_tokenizer_written_v2 (char-level, 10,288 vocab)
Data: KSC2 + FLEURS (clean, no augmentation)
Optim: AdamW, betas=[0.9, 0.98], weight_decay=0.0
Scheduler: Tri-stage LR
Experiment Table
| Run | LR | Steps | Final CTC Loss | Best CTC Loss |
|---|
| 2e-5, Run 1 | 2.0e-5 | 30,000 | -9.16 | -9.16 |
| 2e-5, Run 2 | 2.0e-5 | 21,000 | -10.24 | - |
| 1e-5, Run 1 | 1.0e-5 | 27,000 | -11.13 | -11.13 |
Checkpoints
| LR | Steps | Path | Size |
|---|
| 2e-5 (Run 1) | 30k | step_30000/model/ | 1.29 GB |
| 2e-5 (Run 2) | 21k | step_21000/model/ | 1 |
| 1e-5 | 27k | step_27000/model/ | 1.29 GB |
Architecture
- model_dim: 1024
- num_encoder_layers: 24
- num_encoder_attn_heads: 16
- ffn_inner_dim: 4096
- max_seq_len: 4096
- target_vocab_size: 10,288
- norm_order: PRE (Pre-LN)
- dropout: 0.0 (attn), 0.1 (FFN), 0.1 (layer drop)
Tri-Stage LR Schedule
| Stage | Ratio | LR |
|---|
| Warmup | 5% | 0 to LR |
| Constant | 35% | LR |
| Decay | 60% | LR to final_lr |
Notes
- Run 1 (2e-5) converged fastest, lowest loss at -9.16
- Run 2 (2e-5) stopped at 21k, loss still dropping
- 1e-5 more conservative, loss dropped slower (-11.13 at 27k)
- CTC loss: lower is better
Datasets
- KSC2: issai/Kazakh_Speech_Corpus_2
- FLEURS: google/fleurs (kk_kz)
Reference
- omnilingual-asr: facebookresearch/omnilingual-asr
- fairseq2: facebookresearch/fairseq2
