This model repository contains the LoRA adapter weights for Qwen2.5-0.5B-Instruct distilled from the teacher model Qwen2.5-3B-Instruct. The distillation objective integrates SFT cross-entropy loss, logit-level Kullback-Leibler (KL) divergence, and linear projection-based hidden state matching.

Base Student Model: Qwen/Qwen2.5-0.5B-Instruct
Teacher Model: Qwen/Qwen2.5-3B-Instruct
Merged Distilled Model: sarimahsan/distillation-hiddenstates-code

📊 Trial Run Evaluation Metrics

These metrics were recorded during a trial run on a single NVIDIA Tesla T4 (16GB) GPU for 1 epoch on a subset of Alpaca, Dolly, and Ultrachat:

Metric	Before Distillation	After Distillation	Change	Status
Validation Perplexity	5.0924	5.2620	+0.1696	✗
Teacher-Student KL Divergence	2.7913	1.9637	-0.8276	✓
Hidden State Cosine Similarity	0.0075	0.0054	-0.0021	✗

Analysis:

The Teacher-Student KL Divergence improved by 0.8276, showing that the student's output probabilities are starting to align with the teacher's logits.
Perplexity and Hidden similarity did not improve during the brief training run due to the restricted dataset size and quick epoch limits. A larger compute and training duration are recommended to achieve full convergence of the projection layers.

🚀 How to Use (LoRA Adapter)

To load and use this adapter, run:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "Qwen/Qwen2.5-0.5B-Instruct"
adapter_name = "sarimahsan101/Qwen2.5-0.5B-HiddenDistilled-LoRA"

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained(base_model_name, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_name)
model.eval()

# Inference example
messages = [{"role": "user", "content": "Explain gravity in one sentence."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=50)

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

🛠️ Training Configurations & Details

Framework: PyTorch & Hugging Face Transformers / Trainer
Quantization: 4-bit NF4 double quantization (bitsandbytes)
PEFT Method: LoRA (Rank = 32, Alpha = 64, Target modules = all linear layers)
Optimization: Paged AdamW 8bit with Cosine LR scheduler
Loss Weights: Cross-Entropy: 0.3, KL Divergence: 0.4, Hidden MSE: 0.3