Views
No views yet
🤖 Made Autonomously Using NEO — Your Autonomous AI Engineering Agent
Qwen/Qwen3.6-27B produced with llama.cpp.--ctx-size 512, --parallel 1) and llama-bench pp512 / tg128 throughput on a Tesla V100-SXM2-16GB. -ngl is the number of layers offloaded to GPU; smaller quants fit fully (-ngl 99), larger quants fall back to partial offload because the 16 GB VRAM cap doesn't hold the full model + KV cache.| Model | file_size_mb | perplexity | pp512 t/s | tg128 t/s | -ngl (perp / bench) |
|---|---|---|---|---|---|
| Qwen3.6-27B-Q2_K.gguf | 10215.44 | 6.8364 ± — | — | 37.03 | 99 / 99 |
| Qwen3.6-27B-Q4_K_M.gguf | 15780.83 | 5.9013 ± 0.160 | 360.89 | 4.88 | 50 / 50 |
| Qwen3.6-27B-Q5_K_S.gguf | 17814.27 | 5.7555 ± 0.154 | 402.79 | 4.98 | 42 / 53 |
| Qwen3.6-27B-Q8_0.gguf | 27271.04 | 5.7384 ± 0.153 | 133.71 | 1.98 | 28 / 35 |
Note: Q2_K's tg128 is high because it's the only quant that fits fully on the V100's 16 GB VRAM (-ngl 99). Q4_K_M / Q5_K_S / Q8_0 fall back to partial offload, so theirtok/sreflects the CPU spillover, not a fundamental quality-vs-speed regression. On a 24 GB+ GPU all four would land near Q2_K's headline number.
| Quant Type | Description | Use Case |
|---|---|---|
| Q2_K | 2-bit K-quants, aggressive compression | Edge / mobile, very limited RAM |
| Q4_K_M | 4-bit K-quants medium, balanced quality / size | General purpose, recommended |
| Q5_K_S | 5-bit K-quants small, better quality | Quality-critical applications |
| Q8_0 | 8-bit, near-lossless | Maximum fidelity, reference |
Qwen3.6-27B-Q2_K.ggufQwen3.6-27B-Q4_K_M.ggufQwen3.6-27B-Q5_K_S.ggufQwen3.6-27B-Q8_0.gguf1llama-server -m Qwen3.6-27B-Q2_K.gguf -c 4096 --host 0.0.0.0 --port 8080
2llama-server -m Qwen3.6-27B-Q4_K_M.gguf -c 4096 --host 0.0.0.0 --port 8080
3llama-server -m Qwen3.6-27B-Q5_K_S.gguf -c 4096 --host 0.0.0.0 --port 8080
4llama-server -m Qwen3.6-27B-Q8_0.gguf -c 4096 --host 0.0.0.0 --port 8080llama-cli -m Qwen3.6-27B-Q4_K_M.gguf -p "Hello, world!"