QLoRA fine-tune of Qwen/Qwen2.5-7B-Instruct specialised on Urdu instruction-following. Trained on ~69,000 instruction-response pairs covering Urdu script and Roman Urdu, with explicit code-mixed (Urdu-English) examples, Pakistani context, and — new in v3 — RAG-format (query, context, grounded-answer) triples so the model can actually use retrieved context.

This repo ships the LoRA adapter only (~154 MB). Apply on top of the base model with PEFT to use. See the Usage section below.

Built as a public build by a first-time fine-tuner. Full training pipeline + eval scripts + write-ups on GitHub. v3 story: WRITEUP_V3.md.

Headline result

79.5% pairwise preference vs base Qwen 2.5 7B Instruct across two independent LLM judges (Claude + GPT-5.3), on a 100-prompt hand-curated Urdu evaluation set — up from v2's 66%. All three v2 regressions (summarization, grammar, reasoning) recovered.

(Gemini 3.1 Pro, the 3rd v2 judge, was dropped for v3: its free tier caps at 20 requests/day, too few for the 100-prompt set. v3 was judged by the two cross-model judges that matter most for a stress test.)

Per-category win rate vs base

Category	v3 win rate	vs v2
Creative writing	100%	↑
Translation (UR↔EN)	97%	↑
Summarization	82%	↑↑ (v2 regressed to 46%)
Grammar correction	82%	↑↑ (v2 regressed to 36%)
Question Answering	79%	↑
Code explanation in Urdu	75%	↑
Code-mixed (Urdu/English)	70%	~
Reasoning	53%	↑↑ (v2 was 31%) — still the soft spot

v3 vs v2, head-to-head: v3 wins only 43% of direct matchups — a rebalance, not a strict upgrade. It clearly wins summarization / grammar / code-mixed but trades away some translation / reasoning / QA. There is no free lunch in the data mix.

RAG (Urdu Wikipedia retrieval) — the honest result

v3 is trained on the exact {context}\n\nسوال:{query} surface the retrieval endpoint serves, which fixed RAG's structural failure:

Chinese leakage: base Qwen + Urdu RAG context leaked Chinese on 45/100 answers → v3 + RAG leaks Chinese on 0/100. Now deployable.
On factual prompts RAG corrects the model — e.g. Pakistan's largest province by area: plain v3 says "Punjab" (wrong), RAG says "Balochistan, 347,190 km²" from the article.
But across the full 100-prompt set RAG beats plain v3 only 15.5% of the time: ~79 of the prompts are creative / grammar / reasoning where retrieved Wikipedia is pure noise.

RAG is a safe, deployable grounding tool for factual queries — not a blanket upgrade.

Quick start (Usage)

Recommended: load via Unsloth (the adapter was trained against Unsloth's 4-bit base — see adapter_config.json). Unsloth gives identical-base loading and ~2× inference speed.

python

from unsloth import FastLanguageModel
from peft import PeftModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct",
    max_seq_length=4096,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "TayyabManan/qwen2.5-7b-urdu-v3")
FastLanguageModel.for_inference(model)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "پاکستان کا قومی ترانہ کس نے لکھا؟"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
import torch
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256, temperature=0.7, top_p=0.9,
                         repetition_penalty=1.1, do_sample=True)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Using retrieved context (RAG)

v3 was trained on the exact surface below — reproduce it verbatim for grounded answers. Concatenate your retrieved chunks (newline-separated), then the question:

python

context = "\n\n".join(retrieved_chunks)   # your retriever's top-k passages
user = f"{context}\n\nسوال: {question}"
system = ("You are a helpful assistant. Use the provided context to answer in Urdu. "
          "If the context is insufficient, answer briefly from general knowledge.")
# recommended RAG decoding: temperature 0.3, top_p 0.9, repetition_penalty 1.3

Alternative: vanilla Transformers + PEFT

Works, but the adapter was calibrated against Unsloth's 4-bit base — expect minor numerical drift (no catastrophic failure).

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(model, "TayyabManan/qwen2.5-7b-urdu-v3")
model.eval()

Recommended decoding

Use case	temp	top_p	rep_penalty	max_new_tokens
Creative / open-ended	0.7	0.9	1.1	512
Factual / short answer	0.3	0.9	1.1	256
RAG / grounded answer	0.3	0.9	1.3	512
Long Urdu generation (avoid repetition collapse)	0.5	0.9	1.3	512

Training details

Base model: Qwen/Qwen2.5-7B-Instruct (Apache 2.0)
Method: QLoRA (4-bit NF4 base weights + LoRA)
LoRA: r=16, alpha=32, targets = q_proj k_proj v_proj o_proj gate_proj up_proj down_proj
Effective batch: 16 (per-device 2 × grad_accum 8)
Optimiser: AdamW 8-bit, lr=2e-4, cosine schedule, 3% warmup
Epochs: 2 (same as v2 — 3 over-trains this data mix)
Max sequence length: 4096 tokens (v2 was 2048; raised so RAG context fits without truncation)
Precision: bfloat16 forward, 4-bit NF4 base
Hardware: Single NVIDIA H100 80GB on Modal
Wall-clock: 8,194 steps
Final train loss: 0.5738
Framework: Unsloth (2× throughput vs stock TRL on H100)
Adapter size: ~154 MB
Cost (single run): ~$5-6 on Modal H100

Training data (~69,000 examples total)

Source	Count	Type
v2 mix (translated Urdu-script + code-mixed + Roman Urdu)	63,322	Q→A, multi-domain
Grammar-correction pairs (v3)	~1.9k	corrupt→correct, gold = trusted human source
Summarization + reasoning (v3)	~1.9k	count-verified summaries; Python-owned arithmetic
RAG triples (v3)	~1.9k	`(query, context, grounded-answer)` in the retrieval surface

The v3 synthetic data was generated with GPT-4o-mini (~$0.63 total), deterministically checked (sentence counts, arithmetic digits, question presence), and human spot-checked (200 rows) before training — a review that caught two systemic generator bugs the automated tests missed (a 100%-broken reasoning template and confabulated RAG "noise" answers). Grammar gold answers are the original trusted sentence, never a model rewrite; reasoning answers are narrated over Python-computed numbers so the model never does math.

RAG-aware: unlike v2, the mix includes retrieval-format triples — a gold chunk, a gold chunk plus distractors, and "noise" (unrelated chunks where the correct move is to decline). This is what makes retrieved context in-distribution at inference time.

Limitations

Reasoning is still the soft spot (53% vs base) — recovered from v2's 31% but the weakest category. Multi-step numeric word problems remain unreliable.
RAG is not a blanket upgrade. It helps the minority of prompts that need a looked-up fact; on creative / grammar / reasoning prompts, prepended context is noise. Route retrieval at factual queries, not everything.
v3 is a rebalance vs v2, not a strict superset. v3 loses to v2 on some translation / reasoning / QA prompts. Pick the version that matches your workload.
English-leaning system-prompt sensitivity. Training used "You are a helpful assistant." (and the RAG system prompt above). Long or novel Urdu system prompts can destabilise generation.
Repetition collapse on long Urdu prompts. Mitigate with max_new_tokens ≤ 512, repetition_penalty 1.3, temperature 0.3–0.5.
Not for safety-critical use. Base-model refusal behaviours are preserved, not strengthened.
Eval is pairwise preference, not absolute capability. A 79.5% win vs base does not mean 79.5% of answers are factually correct — many wins are stylistic / fluency-based.

Evaluation methodology

Set: 100 hand-curated Urdu prompts across 8 categories (QA, summarization, translation, grammar, reasoning, creative, code explanation, code-mixed).
Decoding parity: base and fine-tuned generate with identical decoding parameters.
Judging: pairwise blinded preference, responses A/B-randomised per item to control position bias.
Judges: 2 independent cross-model LLM judges (Claude + GPT-5.3).
RAG eval: RAG-v3 vs plain-v3 judged over the same 100 prompts; plus a structural check for non-Urdu (Chinese) leakage.

Eval code: scripts/28_run_eval_v3.py

Acknowledgements

Base model: Qwen team at Alibaba (Qwen 2.5 paper)
Training framework: Unsloth by Daniel Han + team
Retrieval: BAAI bge-m3 embedder + bge-reranker-v2-m3, Qdrant, rank-bm25
Serverless compute: Modal
Evaluation infrastructure: Haystack by deepset

Author

Muhammad Tayyab — MS Artificial Intelligence Engineering, COMSATS University Islamabad.

This is my first fine-tuning project end-to-end. Total project cost to date: ~$60.

Citation

bibtex

@misc{tayyab2026qwen25urduv3,
  title  = {Qwen 2.5 7B Urdu (v3 LoRA adapter)},
  author = {Tayyab, Muhammad},
  year   = {2026},
  publisher = {Hugging Face},
  url    = {https://huggingface.co/TayyabManan/qwen2.5-7b-urdu-v3}
}

License

Apache 2.0. Inherits from Qwen 2.5 base.