Views
No views yet
A bilingual (Hindi-English / Hinglish) fine-tune of Qwen2.5-7B-Instruct using QLoRA + Unsloth, designed to fluently converse in Hinglish — the natural code-mixed language spoken by 350M+ people across India.
| Metric | Baseline | Fine-tuned | Change |
|---|---|---|---|
| Eval Loss | 2.57 | 0.90 | ↓ 65% |
| Training Loss | 1.63 | 0.25 | — |
| Hinglish Response Rate | 10% | 80% | ↑ 8× |
| Baseline (Qwen2.5-7B) | Fine-tuned (Ours) | |
|---|---|---|
| "Yaar, Python seekhni hai" | Pure English bullet points, broken numbering | Natural Hinglish advice like a friend would give |
| "Ghar pe pizza banana sikhao" | Textbook English recipe format | Casual Hinglish with Indian context ("Domino's se bhi better!") |
| "Stock market invest karna chahiye?" | Generic English financial advice | Practical Hinglish with real talk ("Loan lekar mat karna!") |
| Field | Value |
|---|---|
| Base Model | Qwen2.5-7B-Instruct (4-bit NF4) |
| Method | QLoRA — LoRA adapters on quantized base |
| Framework | Unsloth + HuggingFace TRL SFTTrainer |
| Hardware | NVIDIA RTX 3060 (12GB VRAM) |
| Precision | BF16 mixed precision |
| LoRA Config | r=32, α=64, RSLoRA, 7 target modules |
| Trainable Params | 80.7M / 7.7B (1.05%) |
| Training Time | ~2 hours, 675 steps, 3 epochs |
| Parameter | Value |
|---|---|
| Epochs | 3 (675 total steps) |
| Effective Batch Size | 16 (1 × 16 grad accum) |
| Learning Rate | 2e-4 → cosine decay |
| Warmup | 50 steps |
| Optimizer | AdamW 8-bit |
| Eval Strategy | Every 100 steps |
| Best Checkpoint | Step 400 (eval loss 0.899) |
| Source | Samples |
|---|---|
| HuggingFace (HydraLM, NebulaByte, findnitai) | ~2,000 |
| Gemini Synthetic (5 categories) | ~1,000 |
| Alpaca → Hinglish Translation | ~1,000 |
1from unsloth import FastLanguageModel
2from peft import PeftModel
3
4model, tokenizer = FastLanguageModel.from_pretrained(
5 model_name="unsloth/Qwen2.5-7B-Instruct-bnb-4bit",
6 max_seq_length=2048,
7 load_in_4bit=True,
8)
9model = PeftModel.from_pretrained(model, "psyka-101/hinglish-qwen2.5-7b-instruct")
10FastLanguageModel.for_inference(model)
11
12messages = [
13 {"role": "system", "content": "Aap ek helpful AI assistant hain jo Hinglish mein baat kar sakta hai."},
14 {"role": "user", "content": "Yaar, mujhe Python seekhni hai. Kahan se start karun?"},
15]
16inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
17outputs = model.generate(inputs, max_new_tokens=256)
18print(tokenizer.decode(outputs[0], skip_special_tokens=True))