Fine-tuned facebook/nllb-200-distilled-600M for Russian → Abkhaz (apsua) translation. Abkhaz is a low-resource language not supported out of the box by NLLB-200, so the tokenizer was surgically extended and the model fully fine-tuned on real + back-translated data.

Built for the Yandex Data Dojo 2026 low-resource MT track.

Results

Metric	Value
Internal clean held-out (sentence-BLEU, beam=4)	15.15
Contest leaderboard (hidden test, mean `sacrebleu.sentence_bleu`)	9.98
Copy-paste floor	4.1

The hidden test is broader than the (religious-heavy) training corpus, hence the held-out → LB calibration shift of ≈0.66.

⚠️ Important: language-token leak — you MUST strip `abk_Cyrl`

The target language token abk_Cyrl was added to the tokenizer as a regular vocab token (id 256230), not a special token. Therefore skip_special_tokens=True does not remove it, and it leaks as a literal prefix into every output. You must strip it manually (skipping this costs ≈ −3.4 BLEU):

pred = pred.replace("abk_Cyrl", "").strip()

(rus_Cyrl on the source side IS a proper special token and needs no stripping.)

Usage

python

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

REPO = "audiosurffer0/nllb-600m-ru-ab-dojo26"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSeq2SeqLM.from_pretrained(REPO).to("cuda").eval()

tok.src_lang = "rus_Cyrl"
abk = tok.convert_tokens_to_ids("abk_Cyrl")        # = 256230

def translate(text):
    enc = tok(text, return_tensors="pt", truncation=True, max_length=128).to("cuda")
    out = model.generate(
        **enc,
        forced_bos_token_id=abk,
        max_new_tokens=128,
        num_beams=4,
        length_penalty=1.1,        # mild gain on longer test sentences
        do_sample=False,
    )
    pred = tok.decode(out[0], skip_special_tokens=True)
    return pred.replace("abk_Cyrl", "").strip()     # ← strip the leaked lang token

print(translate("Глава государства рассказал об экономике Абхазии"))

Training data

~185K real parallel RU↔AB pairs (domain is strongly religious/biblical — see limitations).
~150K clean back-translated pairs: monolingual Abkhaz → synthetic Russian produced by a dedicated AB→RU model, paired with the real Abkhaz target. The religious domain was filtered out of the back-translation to reduce domain over-fitting.

Training procedure

Tokenizer surgery (transformers 5.9, fast BPE): added abk_Cyrl + 26 missing Abkhaz characters and typographic marks directly into tokenizer.json model.vocab (vocab 256230 → 256231). resize_token_embeddings on the model. This drives Abkhaz unk from ~10% to 0%. (add_tokens() and editing sentencepiece.bpe.model do not work for NLLB-5.9.)
Full fine-tune (LoRA fails here: tied embeddings break under PEFT, so the new-token rows never train).
Real×1 + clean back-translation, weight_decay=0.01, optim=adafactor, fp16, 12000 steps.
Decoding tuned on held-out: beam=4 + length_penalty=1.1; no_repeat_ngram was found to hurt.

Key finding: in this low-resource setting the lever was back-translation (data augmentation), not model size — a 1.3B model without BT matched 600M without BT, while BT added +2.8 held-out / +1.2 LB on 600M.

Limitations & bias

Training corpus is heavily religious/biblical; outputs may skew toward that register and underperform on technical/colloquial text.
Overall quality is ~10 sentence-BLEU — usable for gisting, not production-grade.
Inherits NLLB-200 limitations and its CC-BY-NC-4.0 (non-commercial) license.

Companion model

The back-translation source was generated by the companion AB→RU model (audiosurffer0/Ab_ru_dojo26_7000check).

Citation / license

Base model © Meta AI (NLLB-200), CC-BY-NC-4.0. This derivative is released under the same non-commercial license. Educational/research use.

audiosurffer0/nllb-200-distilled-600M-ru-ab-fine-tune

NLLB-200-600M · Russian → Abkhaz (RU→AB)