Views
No views yet
| Quantization | Size | Use Case |
|---|---|---|
Q2_K | ~3.8 GB | Extreme compression, lowest quality |
Q3_K_S | ~4.3 GB | Small footprint |
Q3_K_M | ~4.6 GB | Small footprint, balanced |
Q3_K_L | ~4.9 GB | Small footprint, higher quality |
Q4_0 | ~5.3 GB | Good balance |
Q4_K_S | ~5.4 GB | Good balance |
Q4_K_M | ~5.7 GB | Recommended for most users |
Q5_0 | ~6.3 GB | High quality |
Q5_K_S | ~6.3 GB | High quality |
Q5_K_M | ~6.5 GB | High quality, balanced |
Q6_K | ~7.4 GB | Near-lossless |
Q8_0 | ~9.5 GB | Highest quality quantization |
BF16 | ~17.9 GB | Full precision |
1# Install llama.cpp
2brew install llama.cpp # macOS
3# or build from source: https://github.com/ggml-org/llama.cpp
4
5# Interactive chat
6llama-cli --hf-repo Tesslate/OmniCoder-9B-GGUF --hf-file omnicoder-9b-q4_k_m.gguf -p "Your prompt" -c 8192
7
8# Server mode (OpenAI-compatible API)
9llama-server --hf-repo Tesslate/OmniCoder-9B-GGUF --hf-file omnicoder-9b-q4_k_m.gguf -c 8192