A Twi (Akan) language adaptation of openbmb/MiniCPM5-1B,
produced by tokenizer extension → continued pretraining (CPT) → supervised fine-tuning (SFT).
The goal of this v1 is fluent Twi generation — it speaks natural, idiomatic Twi with correct
Ghanaian orthography (ɛ/ɔ).
Base model: openbmb/MiniCPM5-1B (a 1.1B-param Llama-architecture model)
1. Tokenizer extension. The base tokenizer split Twi very inefficiently — the Ghanaian
vowels ɛ (U+025B) and ɔ (U+0254) each became two byte-level tokens, giving a Twi fertility of
2.90 tokens/word vs 1.29 for English. We trained an auxiliary byte-level BPE on clean Twi and
added the top 6,000 Twi word-pieces to the tokenizer (130,560 → 136,560), then resized the
model embeddings and warm-started each new token from the mean of its original subword
embeddings. This cut Twi fertility to ~1.26 tokens/word (−57%).
2. Continued pretraining (CPT). Full-parameter CPT for 1 epoch (~666M tokens) on a mix of
65% monolingual Twi / 20% EN–Twi parallel / 15% English replay (the English replay mitigates
catastrophic forgetting at this scale). Loss 6.29 → 1.90.
3. Supervised fine-tuning (SFT). LoRA SFT (rank 32) for 3 epochs on ~200k shuffled
instruction/QA examples — a blend of short factual QA and longer passage-grounded QA in Twi.
Loss 4.14 → 2.30.
Training data
All Twi text was Unicode-normalized to canonical Latin ɛ/ɔ before training.
Held-out Twi perplexity after CPT: 8.26. Qualitatively the model produces fluent, idiomatic
Twi with correct orthography. Descriptive/explanatory prompts work well.
Limitations
This is a v1 focused on fluency, and it has real weaknesses:
Factual accuracy is unreliable — output is fluent but can be wrong, and long answers
sometimes drift or include invented words toward the end.
Reasoning / math is weak — full-parameter CPT eroded some of the base model's
instruction-following/reasoning ("alignment tax"), and the v1 SFT data was plain QA.
Requires a repetition penalty (see above) — do not use greedy decoding.
SFT data included machine-translated content, which introduces some translationese.
Planned v2 improvements: cleaner/curated SFT data, instruction-format English replay during CPT,
and chain-of-thought-shaped examples (reason in English → answer in Twi).
