You can deploy this model and start earning money today!

ishu-newaz/
llama32-1b-gkd-from-3b-GGUF

Name: llama32-1b-gkd-from-3b-GGUF
Availability: InStock

knowledge-distillation

unsloth/Llama-3.2-1B-Instruct

quantized

endpoints_compatible

conversational

Views

No views yet

llama32-1b-gkd-from-3b GGUF

This repo contains llama.cpp GGUF exports for a GKD-distilled unsloth/Llama-3.2-1B-Instruct student.

The LoRA adapter saved in llama32-1b-gkd-from-3b was merged into the base model before conversion, so these GGUF files are standalone and do not need the LoRA adapter at inference time.

Recommended Raspberry Pi file: llama32-1b-gkd-from-3b-q4_k_m.gguf

Example llama.cpp command:

llama-cli -hf ishu-newaz/llama32-1b-gkd-from-3b-GGUF --hf-file llama32-1b-gkd-from-3b-q4_k_m.gguf -p "Explain distillation in one sentence." -n 80 -c 512

For Raspberry Pi, keep the context small, for example -c 512 or -c 1024, to reduce RAM use and latency.