You can deploy this model and start earning money today!

amaljoe88/
deepseek-r1-1.5b-indic-kl

Name: deepseek-r1-1.5b-indic-kl
Availability: InStock

safetensors

qwen2

Views

No views yet

DeepSeek-R1-1.5B Indic KL Distillation (Round 1 Pilot)

Fine-tuned from deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B using top-256 sparse KL distillation from Qwen/Qwen3-30B-A3B as teacher.

Method

Context distillation + KL distillation:
The teacher (Qwen3-30B-A3B) sees full cultural background context for each example. The student is trained without this context, learning to internalise South Asian cultural norms directly into its weights.

Loss:

Loss = α · CE(student, teacher_response)
     + (1 − α) · T² · KL(teacher_top256 ∥ student)

α = 0.5, T = 2.0

Training Data

Source	Examples	Task
NormAd (India/Pakistan/Bangladesh)	97	Cultural norm classification
MILU-en	15	Indian knowledge MCQ (English)
MILU-hi	15	Indian knowledge MCQ (Hindi)
BhED	12	Caste/religion stereotype reduction
GlobalOpinion	12	Indian survey opinion

Pilot run with small dataset (151 unique examples, 33 optimizer steps). See full report in the indic-alignment repo.

Results (Test set: Nepal + Sri Lanka)

Metric	Baseline	Post-SFT	This model
NormAd best accuracy	38.9%	30.6%	37.5%
BhED caste stereotype	58.2%	47.2%	49.1%

Outperforms standard CE fine-tuning on NormAd by +6.9pp
Reduces caste stereotype by −9.1pp vs. untuned baseline

Training Config

Student: DeepSeek-R1-Distill-Qwen-1.5B
Teacher: Qwen3-30B-A3B
Hardware: 2× A100 80GB
Epochs: 3 | LR: 2e-5 | Batch: 32 effective
Top-K logprobs: 256

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("amaljoe88/deepseek-r1-1.5b-indic-kl")
tokenizer = AutoTokenizer.from_pretrained("amaljoe88/deepseek-r1-1.5b-indic-kl")