Views
No views yet
Q4_K_M GGUF quant of huihui-ai/Huihui-Qwable-3.6-27b-abliterated-MTP-GGUF.Q4_K_M_Q8 (19.1 GiB) and Q4_K_M_F16 (25.9 GiB). Both are a little too large to leave working room on a 22 GB card once you add the KV cache and compute buffers. This plain Q4_K_M is 15.7 GiB, so it keeps headroom on 22–24 GB GPUs (e.g. RTX 3090 / 4090, or a 22 GB-modded 2080 Ti) while keeping the baked MTP layer intact.| File | Size | BPW |
|---|---|---|
Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf | 15.7 GiB | 4.92 |
Q4_K_M, produced with llama.cpp (build b9190) from huihui's Q4_K_M_F16 release.nextn layer), so there is no separate draft file. Run it as self-speculation:1llama-server -m Huihui-Qwable-3.6-27b-abliterated-Q4_K_M-MTP.gguf \
2 -ngl 99 -fa on -c 32768 --parallel 1 --jinja \
3 --spec-type draft-mtp --spec-draft-n-max 3--spec-draft-n-max 3 works better than the larger value suggested upstream. At higher settings the draft head proposes too many tokens, acceptance drops, and throughput ends up lower. Measured on a GB10 (Grace-Blackwell):--spec-draft-n-max | decode | draft acceptance |
|---|---|---|
3 | ~18.7 tok/s | ~0.45 |
6 | slower | ~0.25 |
Q4, Q8, or F16) did not materially affect acceptance in testing.tools schema (it returns structured tool_calls) when thinking is disabled — e.g. pass "chat_template_kwargs": {"enable_thinking": false} in the request. With thinking enabled, the Fable traces can surface internal tool names in the reasoning output.