A Yoruba–English code-switching ASR model that jointly transcribes speech and
labels the language of every word inline, in a single decoding pass:
<en> first aid </en> <yo> ni wọ́n ṣáré fún kí ó tó kú </yo>
It is openai/whisper-small (244M)
fine-tuned on the YECS Yoruba–English code-switching corpus, with four language-tag
tokens (<yo> </yo> <en> </en>) added to the tokenizer and the decoder embeddings
resized. Language-boundary prediction is learned as an auxiliary task at zero extra
inference cost — one model does transcription and per-word language ID.
Results (YECS full held-out test, 9,949 utterances)
Scored against the plain-fine-tuned control with identical normalization. Language tags
are stripped from both reference and hypothesis before computing WER/CER, so the
transcription comparison is apples-to-apples; the tags are scored separately as LID.
Metric
Tag-injection (this model)
Plain baseline
WER — tone-aware
16.79%
16.69%
WER — tone-insensitive
14.02%
13.83%
CER — tone-aware
6.24%
6.37%
CER — tone-insensitive
5.02%
5.09%
Per-word LID — accuracy
99.54%
—
Per-word LID — macro-F1
99.54%
—
Adding inline tags matches the plain model's WER (within run-to-run noise; CER is
actually a touch better) while delivering 99.54% per-word language identification for
free. For reference, the reported whisper-small-yoruba baseline is 20.76% WER; this
project's plain 5-epoch fine-tune reaches 16.69%.
The same result holds across three architectures
The tag-vs-plain A/B was run identically on three model families:
Model
WER tone-aware (plain → tag)
Per-word LID (tag)
Omnilingual CTC 300M
33.57 → 32.45
98.04%
Omnilingual LLM 1.63B
16.15 → 16.29
99.55%
Whisper-small (this)
16.69 → 16.79
99.54%
Inline language-tag injection costs ~nothing on WER (−1.1 to +0.1 absolute) and gives
98–99.5% free per-word LID on CTC, LLM, and Whisper alike.
How it was built
Target serialization — each transcript's per-word language_tags are grouped
into contiguous same-language spans and wrapped: <yo> … </yo> <en> … </en>.
Tokenizer — the four tags are added with add_tokens(..., special_tokens=False)
so they are ordinary vocabulary items that survive decode(skip_special_tokens=True)
(unlike control tokens, which would be dropped). Embeddings are resized to match.
Fine-tune — standard Seq2SeqTrainer, language conditioning yoruba,
task transcribe.
tone-aware keeps Yoruba tone diacritics (acute/grave/macron), so getting a word's
tone wrong counts as an error — the stricter, fuller score.
tone-insensitive additionally strips the three tonal combining marks, isolating
word/segment accuracy. The phonemic under-dot (ọ, ẹ, ṣ) is a distinct letter and is
always kept in both variants.
The gap between the two measures how well the model handles Yoruba tone marking.
Scored on the full held-out YECS test split (9,949 utts); test audio is the canonical
Mozilla/MDC release. Test reference labels are held privately by LyngualLabs (public
test_metadata.csv has targets stripped), so metrics are internal and not independently
reproducible.
License
Apache-2.0 (inherited from openai/whisper-small). YECS corpus terms apply to the
training data.
