whisper-distant-voices
This model is a fine-tuned version of
openai/whisper-small on the
google/fleurs dataset.
Model description
whisper-distant-voices is a fine-tuned version of openai/whisper-small
for multilingual transcription of Swahili (sw), English (en),
French (fr), and Arabic (ar). It is designed for community voice
transcription in contexts such as elections, crisis response, and
civic reporting across East Africa and the broader Global South.
Intended uses & limitations
Intended uses:
- Transcribing audio from community reporters and field workers
- Supporting multilingual voice input in civic tech applications
- Automatic speech recognition across Swahili, English, French, and Arabic
Limitations:
- Fine-tuned on 500 samples per language — accuracy will be lower than
the base whisper-small on formal speech
- May struggle with heavy accents, background noise, or code-switching
between languages mid-sentence
- Training steps were limited (10 steps); a full run (2000 steps) will
significantly improve performance
- Not suitable for medical, legal, or safety-critical transcription
without further evaluation
Training and evaluation data
Fine-tuned on the
google/fleurs dataset:
sw_ke: 500 samples (Swahili, Kenya)en_us: 500 samples (English, US)fr_fr: 500 samples (French, France)ar_eg: 500 samples (Arabic, Egypt)
Total: ~2,000 samples, 90/10 train/eval split, shuffled with seed 42.
Training procedure
The model was fine-tuned using HuggingFace Seq2SeqTrainer on Google Colab
with a T4 GPU.
Data preparation
- Audio resampled to 16 kHz and converted to log-mel spectrograms
using
WhisperProcessor
- Transcriptions tokenized with a max length of 448 tokens
- Dataset shuffled with seed 42 and split 90/10 train/eval
Multilingual setup
No language token was forced during training — the model learns to predict
the language automatically per sample across all four languages
(Swahili, English, French, Arabic).
Compute
- Hardware: NVIDIA T4 GPU (Google Colab)
- Training time: ~45 minutes (10-step prototype run)
- Framework: Transformers 5.0.0, PyTorch 2.11.0+cu128
Intended next steps
- Full training run of 2000 steps on expanded data
- Evaluation on held-out community audio from East Africa
- Integration into the live demo Space: katoernest/distant-voices-transcription
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 16
- optimizer: AdamW (fused) with betas=(0.9, 0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 200
- training_steps: 10
- mixed_precision_training: Native AMP
Training results
| Training Steps | Training Loss | Validation Loss |
|---|
| 10 | - | - |
Note: This checkpoint is a prototype run (10 steps) used to validate
the training pipeline end-to-end. Loss values were not recorded before
the run completed. A full training run of 2000 steps is in progress and
results will be updated here upon completion.
What to expect after full training:
- Training loss should decrease steadily from ~3.0 toward ~0.5–1.0
- Validation WER (Word Error Rate) target: below 30% across all four languages
- Evaluation checkpoints saved every 500 steps
Framework versions
- Transformers 5.0.0
- PyTorch 2.11.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
