This repository contains a distilled 3B Llama model, trained using knowledge distillation from the larger Llama-3.1-8B-Instruct teacher model.
The goal of this project was to create a smaller, faster model while preserving strong instruction-following capabilities.

🔥 Overview

Teacher Model: meta-llama/Llama-3.1-8B-Instruct
Student Model: meta-llama/Llama-3.2-3B-Instruct
Objective: Compress the teacher’s behavior into a lighter 3B parameter model for improved latency and deployment efficiency.
Method: Logits-based distillation with temperature scaling and combined KD + student loss.

This model is ideal for:

Chat-based applications
Lightweight on-device or serverless inference
Fast prototyping with low compute resources

🧪 Distillation Process

Distillation was performed using a single epoch over a curated instruction-tuning dataset.

Training Command

bash

python distill.py \
  --teacher-model meta-llama/Llama-3.1-8B-Instruct \
  --student-model meta-llama/Llama-3.2-3B-Instruct \
  --train-file data/train.jsonl \
  --val-file data/test.jsonl \
  --output-dir llama3_3B_distilled \
  --batch-size 8 \
  --gradient-accumulation-steps 1 \
  --num-epochs 1 \
  --learning-rate 1e-5 \
  --warmup-steps 300 \
  --max-length 1024 \
  --temperature 2.0 \
  --alpha 0.5 \
  --save-steps 2000 \
  --eval-steps 200 \
  --logging-steps 5 \
  --bf16

SwashBuckler001/Llama-3.2-3B-distill-GLoRE

🦙 Llama 3.2 3B — Distilled from Llama 3.1 8B Instruct

🔥 Overview

🧪 Distillation Process

Training Command

SwashBuckler001/
Llama-3.2-3B-distill-GLoRE