This is a fine-tuned version of
openai/whisper-large-v3 for
Kenyan Swahili non-standard speech. It is part of CDLI's effort to make speech technology work for people whose speech is underserved by mainstream ASR systems.
All CDLI models and datasets can be found on
CDLI's HuggingFace page.
Dataset
The model has been fine-tuned using
cdli/kenyan_swahili_nonstandard_speech_v1.0, a dataset of speech samples of people living with impaired speech across a range of impairment
severity levels and
etiologies.
Training
The train split was used for training, and the dev split for selecting the best checkpoint.
The encoder and output projection layer were fine-tuned (the decoder was kept frozen), with SpecAugment enabled.
Evaluation
This model was evaluated on the test split of the dataset. Utterances longer than 30 seconds were excluded:
- Examples evaluated: 849
- Speakers: 9
For decoding we ran Whisper with language=sw, task=transcribe,
greedy search (num_beams=1, do_sample=False).
Results are compared against the unadapted base model
openai/whisper-large-v3, evaluated identically, to show the effect of fine-tuning on non-standard speech.
We report two complementary word error rate (WER) metrics, both computed on text
normalized with Whisper's BasicTextNormalizer:
- Standard (corpus-level) WER — the usual error rate, pooling all reference
words and edit errors across the entire test set.
- Per-utterance averaged WER — WER computed separately for each utterance,
each capped at 1.0, then averaged across utterances.
The per-utterance averaged WER bounds each utterance to
[0, 1] and weights all utterances equally, so it reflects typical performance
without a few catastrophic utterances dominating — but it is not a true error
rate and isn't directly comparable to other published WER, hence we report the standard, corpus-level WER as well.
Results
Overall Results
- Adapted — this CDLI model, fine-tuned on non-standard speech.
- Unadapted — the base model it was fine-tuned from (here:
openai/whisper-large-v3).
| Model | Standard WER | Per-utterance averaged WER |
|---|
| Adapted | 0.30 | 0.30 |
| Unadapted | 0.85 | 0.58 |
| Relative improvement | 64% | 48% |
Detailed Analysis
Aggregated results can hide important underlying patterns, so we also break the WER down by subset: per speaker, and — where speaker severity is available — per impairment severity group.
Results by impairment severity
All WER values below are the per-utterance averaged WER, first averaged per
speaker and then averaged within each severity group. n_speakers and
n_utterances are the number of speakers and test utterances in each group.
| severity | n_speakers | n_utterances | Avg WER (unadapted model) | Avg WER (adapted model) | Rel. improvement |
|---|
| mild | 3 | 278 | 0.46 | 0.23 | 51% |
| moderate | 3 | 235 | 0.56 | 0.28 | 50% |
| severe | 3 | 336 | 0.69 | 0.37 | 46% |
Results by speaker
Per-utterance averaged WER per speaker. n_utterances is the number of test utterances for that speaker.
| speaker_id | severity | etiology | n_utterances | Avg WER (unadapted model) | Avg WER (adapted model) | Rel. improvement |
|---|
| KES013 | mild | Cerebral Palsy | 107 | 0.56 | 0.33 | 41% |
| KES021 | mild | Parkinson’s Disease | 68 | 0.43 | 0.2 | 53% |
| KES030 | mild | Neurodevelopmental disorder | 103 | 0.39 | 0.15 | 62% |
| KES012 | moderate | Neurodevelopmental disorder | 97 | 0.48 | 0.18 | 63% |
| KES028 | moderate | Cerebral Palsy | 44 | 0.71 | 0.32 | 55% |
| KES035 | moderate | Multiple Sclerosis (MS) | 94 | 0.5 | 0.34 | 32% |
| KES001 | severe | Cerebral Palsy | 140 | 0.8 | 0.46 | 43% |
| KES002 | severe | Cerebral Palsy | 82 | 0.67 | 0.34 | 48% |
| KES010 | severe | Neurodevelopmental disorder | 114 | 0.62 | 0.32 | 49% |
