QLoRA adapter for meta-llama/Llama-3.1-8B-Instruct, fine-tuned for open-domain question answering in Punjabi written in the Shahmukhi script — a language with 100M+ speakers and almost no NLP resources.

Trained on 15,000 QA pairs generated from Shahmukhi Wikipedia via a GPT-4o-assisted pipeline with cleaning, Unicode normalization, and coverage balancing. This is the final training checkpoint (step 2583, 3 epochs) used for all reported evaluations.

Results

LLM-as-judge scores (1–5), 578 test questions:

Model	Correctness	Relevance	Completeness	Overall
Qwen2.5-7B-Instruct (zero-shot)	1.93	2.31	1.85	2.03
Base LLaMA-3.1-8B-Instruct	2.08	2.57	2.14	2.26
Fine-tuned (this adapter)	3.64	4.04	3.75	3.81

Validated by an independent second judge (Pearson r = 0.88) and blinded native-speaker evaluation (4.22 vs 2.10). Surface metrics agree: chrF++ 46.55 vs 23.44 (base), BLEU 34.23 vs 5.20, ROUGE-L 57.35 vs 17.14.

Usage

Requires access to the gated base model (accept the Llama 3.1 license first).

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "meta-llama/Llama-3.1-8B-Instruct"
adapter = "mahnoornaeem47/Llama-3.1-8B-Punjabi-Shahmukhi-QA-adapter"

tok = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, adapter)

msgs = [{"role": "user", "content": "پنجاب دے مشہور صوفی شاعر کون نیں؟"}]
inputs = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Training

QLoRA on all attention + MLP projections and lm_head · r=64, α=128, dropout 0.10 · 3 epochs, lr 2e-4, AdamW · effective batch 16 · bfloat16 + Flash-Attention v2 · single NVIDIA A100 (Google Colab).

Limitations

Wikipedia-derived training data limits conversational variety; GPT-4o generation may introduce hallucination or bias; strongest on geography, history and culture.

Citation

Paper accepted at IBCAST 2026 (Artificial Intelligence and Software Technologies). Citation will be added when proceedings are published.