Available Quantizations

Quant	Size	Recommended Use	File
`Q2_K`	3.28 GB	Smallest, lowest quality — quick tests / very constrained devices	`qwen3-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf`
`Q3_K_M`	4.12 GB	Small, acceptable quality	`qwen3-8b-sft-claude-opus-reasoning-unsloth.q3_k_m.gguf`
`Q4_K_M`	5.03 GB	Recommended — best size/quality balance	`qwen3-8b-sft-claude-opus-reasoning-unsloth.q4_k_m.gguf`
`Q5_K_M`	5.85 GB	High quality, larger	`qwen3-8b-sft-claude-opus-reasoning-unsloth.q5_k_m.gguf`
`Q6_K`	6.73 GB	Very high quality, near-fp16	`qwen3-8b-sft-claude-opus-reasoning-unsloth.q6_k.gguf`
`Q8_0`	8.71 GB	Near-lossless, largest	`qwen3-8b-sft-claude-opus-reasoning-unsloth.q8_0.gguf`

Q4_K_M is the recommended default for most users.

Usage

Download a single quant

bash

pip install -U "huggingface_hub[cli]"
hf download ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF \
  --include "*q4_k_m*.gguf" --local-dir ./Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF

llama.cpp

bash

# build: https://github.com/ggerganov/llama.cpp
./llama-cli -m ./Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF/qwen3-8b-sft-claude-opus-reasoning-unsloth.q2_k.gguf \
  -p "Solve step by step: What is the sum of the first 10 prime numbers?" -n 512

Ollama

ollama run hf.co/ermiaazarkhalili/Qwen3-8B-SFT-Claude-Opus-Reasoning-Unsloth-GGUF:Q4_K_M "Solve step by step: What is the sum of the first 10 prime numbers?"

Training Outcome

Metric	Value
SLURM Job ID	`36885901`
Runtime	40m 30s
Final Training Loss	0.8753
Peak VRAM	14.23 GB
GPU	H100 80GB HBM3 (MIG 3g.40gb)

License

APACHE-2.0 — see the base model for full terms.

Acknowledgments

Unsloth for 2x faster fine-tuning
llama.cpp for GGUF quantization
Base model developers (unsloth)
Compute Canada / DRAC for HPC resources