Gemma-3-4B-IT with continued pretraining (CPT) on monolingual Mizo, Khasi, Hindi and Kannada text, then supervised fine-tuning jointly on four low-resource Indian language pairs.\n\nCPT helps most where the base model is weakest: it is the best model on Khasi and improves Gondi and Tulu over SFT alone.

A LoRA adapter — load it on top of the base model google/gemma-3-4b-it.

Training data

Fine-tuned jointly on four low-resource pairs from randomDude26/mi_kh_tulu_dataset:

Pair	Train	Val	Test
`en-mizo` (English → Mizo, `lus`)	49,496	464	800
`en-khasi` (English → Khasi, `kha`)	22,358	482	800
`en-tulu` (English → Tulu, `tcy`)	16,218	488	800
`hi-gondi` (Hindi → Gondi, `gon`)	15,500	389	737

The validation and test splits are decontaminated: no exact or near duplicate of any test/val sentence appears in training (exact pair, exact source, exact target, ≥0.90 character similarity, ≥0.80 4-gram containment, ≥0.80 token Jaccard). All scores below are on those decontaminated test sets.

Gondi has 737 test sentences rather than 800 — after decontamination only 737 clean sentences remain.

Results

Decontaminated test sets. BLEU and chrF++ via sacrebleu (BLEU(effective_order=True), CHRF(word_order=2)).

Direction	n	BLEU	chrF++
`eng→mizo`	800	30.89	56.86
`eng→khasi`	800	34.49	58.74
`eng→tulu`	800	35.28	63.33
`hi→gondi`	737	7.51	30.40

Checkpoint used for these scores: gemma-sft-joint-20260523_214614/checkpoint-3500

Against the zero/few-shot baselines (BLEU)

Direction	Gemma 0-shot	Gemma 3-shot	this model
`eng→mizo`	6.99	10.86	30.89
`eng→khasi`	3.16	3.69	34.49
`eng→tulu`	4.32	4.16	35.28
`hi→gondi`	2.44	2.64	7.51

Configuration

LoRA


rank `r`	64
`lora_alpha`	128
`lora_dropout`	0.05
target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`
`modules_to_save`	—

Training


batch size	16 (per device), grad-accum 2 → effective 32
learning rate	2e-4, cosine, 200 warmup steps
weight decay	0.01
max sequence length	512 (prompt + response)
precision	bf16, gradient checkpointing
decoding	greedy

Checkpoints are saved every 500 steps. Checkpoint 3500 is reported: validation loss bottoms out earlier (~step 2500), but validation BLEU keeps improving to 3500 — the two diverge, and checkpoints were selected on BLEU.

Usage

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

REPO = "randomDude26/gemma_mktg_cpt_sft"

tokenizer = AutoTokenizer.from_pretrained(REPO)
tokenizer.padding_side = "left"

base = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-4b-it", dtype=torch.bfloat16
)
model = PeftModel.from_pretrained(base, REPO).merge_and_unload().cuda().eval()

# the prompt MUST match the training format
prompt = (
    "<start_of_turn>user\n"
    "Translate the following English sentence into Tulu.\n"
    "Output only the translated sentence. Do not include explanations.\n\n"
    "The temple is very old.<end_of_turn>\n"
    "<start_of_turn>model\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Citation

Paper under anonymous review. Dataset: randomDude26/mi_kh_tulu_dataset.