NLLB-200-1.3B supervised fine-tuned jointly on four low-resource Indian language pairs. Three new language tokens were added to the tokenizer, since base NLLB supports none of Tulu or Gondi.

A LoRA adapter — load it on top of the base model facebook/nllb-200-1.3B.

Training data

Fine-tuned jointly on four low-resource pairs from randomDude26/mi_kh_tulu_dataset:

Pair	Train	Val	Test
`en-mizo` (English → Mizo, `lus`)	49,496	464	800
`en-tulu` (English → Tulu, `tcy`)	16,218	488	800
`hi-gondi` (Hindi → Gondi, `gon`)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
`eng→mizo`	800	33.43	59.26
`eng→tulu`	800	43.17	69.08
`hi→gondi`	737	7.83	33.53

Checkpoint used for these scores: nllb-sft-joint-upsample-20260525_084543/final_adapter

Against the zero/few-shot baselines (BLEU)

Direction	NLLB 0-shot	this model
`eng→mizo`	20.40	33.43
`eng→khasi`	4.50	5.76
`eng→tulu`	8.51	43.17
`hi→gondi`	2.22	7.83

Configuration

LoRA


rank `r`	64
`lora_alpha`	128
`lora_dropout`	0.1
target modules	`q_proj`, `k_proj`, `v_proj`, `out_proj`, `fc1`, `fc2`
`modules_to_save`	`embed_tokens`, `lm_head`

The embedding matrix and LM head are trained in full (modules_to_save) because three new language tokens — kha_Latn, tcy_Knda, gon_Deva — were added to the tokenizer. Base NLLB-200 has none of them; only Mizo (lus_Latn) is native.

Training


max steps	10,000
batch size	24 (per device), grad-accum 1
learning rate	3e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	256
precision	bf16
decoding	beam search, 4 beams

Trained jointly on all four pairs, with upsampling of the lower-resource pairs.

Usage

python

import torch
from transformers import AutoTokenizer
from transformers.models.m2m_100 import M2M100ForConditionalGeneration
from peft import PeftModel

REPO = "randomDude26/nllb_mktg_sft"

# the tokenizer in the adapter repo carries the 3 added language tokens
tokenizer = AutoTokenizer.from_pretrained(REPO)

base = M2M100ForConditionalGeneration.from_pretrained(
    "facebook/nllb-200-1.3B", dtype=torch.bfloat16
)
base.resize_token_embeddings(len(tokenizer), mean_resizing=False)

model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

tokenizer.src_lang = "eng_Latn"
inputs = tokenizer("The temple is very old.", return_tensors="pt").to("cuda")

out = model.generate(
    **inputs,
    forced_bos_token_id=tokenizer.convert_tokens_to_ids("tcy_Knda"),  # Tulu
    num_beams=4, max_new_tokens=256,
)
print(tokenizer.batch_decode(out, skip_special_tokens=True)[0])

Citation

Paper under anonymous review. Dataset: randomDude26/mi_kh_tulu_dataset.