facebook/mms-300m fine-tuned (character-level CTC) on the YECS Yoruba-English code-switching corpus to jointly transcribe and tag each word inline. Because MMS uses a character tokenizer, the four tags are encoded as four private-use-area sentinel codepoints (U+E000..U+E003) during training and mapped back to <yo> </yo> <en> </en> at decode.

Results (YECS test, 9,905 utts)

Metric	This (tag)	Plain baseline
WER tone-aware	24.20%	19.93%
WER tone-insensitive	21.08%	17.02%
CER tone-aware	7.79%	6.24%
Per-word LID accuracy	99.41%	—

Honest caveat: unlike the subword models in this project (Omni-CTC/LLM, Whisper), where tag injection is WER-neutral or better, MMS (character CTC) pays ~4.3 WER points for the inline tags — the char model over-emits tag sentinels and interleaving single-char tags disrupts alignment. It still yields 99.41% per-word LID. The lesson is about tokenization granularity: atomic subword tags slot cleanly between words; single-character tags do not.

Plain baseline: LyngualLabs/yecs-asr-mms-plain.

Usage

python

from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import torch, librosa
repo = "LyngualLabs/yecs-asr-mms-lid"
proc = Wav2Vec2Processor.from_pretrained(repo)
proc.tokenizer.set_target_lang("yor")           # MMS per-language vocab
model = Wav2Vec2ForCTC.from_pretrained(repo).to("cuda").eval()
a,_ = librosa.load("utt.wav", sr=16000)
iv = proc(a, sampling_rate=16000, return_tensors="pt").input_values.to("cuda")
ids = torch.argmax(model(iv).logits, dim=-1)
txt = proc.batch_decode(ids)[0]
for p,t in zip([chr(0xE000),chr(0xE001),chr(0xE002),chr(0xE003)], ["<yo>","</yo>","<en>","</en>"]):
    txt = txt.replace(p, " "+t+" ")
print(" ".join(txt.split()))

Training

facebook/mms-300m SSL base -> fresh CTC head; 5 epochs, lr 1e-4, fp16 (bf16 causes CTC blank collapse), dropout 0.05, warmup 500, batch 32, freeze feature encoder, 1xH200. W&B: afroscale_ai_cmu_africa/yecs-lid (run mms-tag). Code: https://github.com/osinkolu/yecs-asr-benchmark (tag-injection/).

Limitations

Research use, Yoruba-English read/prompted speech; two languages only. Char-CTC tag over-emission means the WER cost above; strip the tags for a plain transcript.