Knowledge Distillation: Qwen2.5-7B Teacher to Qwen2.5-1.5B Student
Trains a 1.5B student model to imitate a fine-tuned 7B teacher on AIOps incident-response tasks, then measures how much of the teacher's quality survives the compression. Full code, scripts, and write-up:
namanadep/qwen2.5-7b-to-1.5b-aiops-distillation on GitHub.
Results (50-prompt frozen holdout)
| Metric | Student before | Teacher (AIOps-tuned 7B) | Student after distillation (v2) |
|---|
| Mean token F1 vs gold | 0.126 | 0.448 | 0.343 |
| Mean token recall vs gold | 0.220 | 0.419 | 0.338 |
| Holdout prompts improved vs before | - | - | 35 / 50 (70%) |
| Prompts where student matched or beat teacher | - | - | 18 / 50 (36%) |
| Eval-time peak VRAM | 3.98 GB | 9.64 GB | 3.0 GB |
| Eval-time inference speed | 48.4 tok/s | 18.47 tok/s | 16.47 tok/s |
Distilling from an untuned 7B teacher barely moved the student (F1 0.126 to 0.127). Fine-tuning the teacher on gold AIOps answers first, then distilling from it, closed 67.3% of the gap between the untuned student and the tuned teacher. The teacher has 7.616B parameters against the student's 1.544B (4.93x fewer), and the student adapter adds about 81 MB.
One number cuts against the student rather than for it: distilled inference is slower than the untuned baseline (16.47 vs 48.4 tok/s), because the distilled model now writes longer, more structured answers instead of short generic ones.
Full methodology, ablation study, and per-prompt scoreboard: see the
GitHub repository, particularly
docs/DISTILLATION_FULL_PROCESS_AND_COMPARISON.md and
docs/background/LIMITATIONS_AND_HONESTY.md.
Method
Offline response distillation: fine-tune the 7B teacher on gold AIOps answers (LoRA, BF16), have it generate answers for ~500 prompts, then train the 1.5B student by supervised fine-tuning on those teacher-written answers (LoRA, BF16, 3 epochs). Teacher and student are evaluated on the same 50-prompt holdout, frozen and never used in training or teacher generation. Trained on 2x NVIDIA H200 NVL GPUs.
Repository contents
adapters/qwen2.5_7b_aiops_teacher_adapter/: 7B teacher LoRA adapter (adapter_model.safetensors).adapters/qwen2.5_1.5b_distill_student_adapter/: 1.5B distilled student LoRA adapter (adapter_model.safetensors).QWEN_7B_DISTILLING_1.5B/: training and evaluation code, docs, and results (mirrored and kept current on GitHub).Distill_Qwen_7B_Teacher_to_1.5B_Student.pptx: 12-slide presentation deck.
