Bonsai-27B — Standard GGUF Ladder (Q3_K_M → Q8_0)
Imatrix GGUF quants of
prism-ml's Bonsai-27B (Qwen3.6-27B backbone, 262K context, hybrid attention, vision), covering the quality tiers between the official QAT releases and F16.
Which repo should you use?
If you have ≤8GB: use prism-ml's official QAT quants, not these.
- Q1_0 (3.8GB) — 89.5% of F16 quality at 1.125 bits
- Ternary (7.2GB) — 95% of F16
Those are quantization-aware-trained with custom kernels — at their sizes, they beat anything post-training quantization can produce, including anything in this repo.
This repo covers the gap above them: the standard ladder for 12–32GB setups where you want maximum quality per GB, quantized from prism-ml's own F16 GGUF with an importance matrix (63KB coding/reasoning calibration corpus).
Quants
| File | Size | Measured (GB10, 273GB/s) | + DSpark drafter |
|---|
| Q8_0 | 28.6 GB | 7.8 tok/s | — |
| Q6_K | 22.1 GB | 9.1 tok/s | — |
| Q5_K_M | 19.2 GB | 10.2 tok/s | — |
| Q4_K_M | 16.6 GB | 12.0 tok/s | 15.7 tok/s (+31%) |
| IQ4_XS | 15.1 GB | 13.7 tok/s | — |
| Q3_K_M | 13.3 GB | 13.5 tok/s | — |
All tiers individually smoke-tested (coherent code generation, chat template engages via --jinja). Q8_0 quantized without imatrix (unneeded at 8-bit); all others use the included Bonsai-27B.imatrix. Decode rates scale with your memory bandwidth — a 936GB/s GPU (RTX 3090-class) will run ~3× these numbers.
Both official mmproj files are included for vision (--mmproj Bonsai-27B-mmproj-Q8_0.gguf).
Speculative decoding with the official DSpark drafter (+31% measured)
prism-ml ships a
DSpark drafter (1.8GB) trained against this exact target model — and it
works with this repo's quants, verified:
| Config | tok/s | Draft acceptance |
|---|
| Q4_K_M, mainline llama.cpp | 12.0 | — |
| Q4_K_M + DSpark drafter, prism fork | 15.7 | 79% (242/305) |
Output at temperature 0 is byte-identical with and without the drafter (speculative decoding is verify-always — the drafter can only affect speed, never quality).
Requires PrismML-Eng's llama.cpp fork (branch
prism) — the
dspark drafter architecture is not in mainline llama.cpp (yet; their
pr/* branches suggest upstreaming is underway). Build and run:
1git clone --depth 1 --branch prism https://github.com/PrismML-Eng/llama.cpp
2cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server
3
4./build/bin/llama-server \
5 -m Bonsai-27B-Q4_K_M.gguf \
6 -md Bonsai-27B-dspark-Q4_1.gguf \
7 --spec-type draft-dspark --spec-draft-n-max 4 \
8 -ngl 999 -ngld 999 -fa on -c 16384 -np 1 --jinja
Gotchas (from prism-ml's
SPECULATIVE.md + our testing):
--spec-draft-n-max must be exactly 4 (the drafter's block size) — other values crash at the first draft round- Speculative mode disables cross-request prompt caching and concurrency (
-np 1) — best for single-user/agent use
- Use
-c 16384+ and a generous max_tokens — the model regularly thinks 1.5–2K tokens before the visible answer
- CUDA only for now (their Metal spec path is still being optimized)
- Expect bigger gains on higher-bandwidth GPUs (prism-ml measured ~1.9× on datacenter CUDA; our +31% is on a bandwidth-bound GB10)
Speed hunt: fastest Bonsai-27B config on a DGX Spark (GB10)
We benchmarked every acceleration path available for this model on GB10 (2026-07-14). Full matrix — 3-pass medians, coherence-gated, temp 0:
| Config | tok/s | Notes |
|---|
| prism-ml Q1_0 QAT (3.5 GiB), prism fork | 43.7 | 🏆 fastest — llama-bench: tg128 44.14, pp512 1003 |
| prism-ml Ternary Q2_0 QAT (7.2 GB) | 29.3 | higher quality (95% of F16) |
| This repo Q4_K_M + DSpark drafter | 15.7 | +31% over base, 79% acceptance |
| This repo IQ4_XS | 13.7 | fastest PTQ tier without the fork |
| This repo Q4_K_M | 12.0 | baseline |
Finding — 1-bit QAT makes speculative decoding counterproductive: against the Q1_0 target, every drafting scheme failed to beat base speed on this hardware: the official DSpark drafter halved throughput (21.6 tok/s at 78% acceptance), a Q2_K requant of it was no better, Bonsai-8B-Q1_0 as a draft-simple drafter was an exact wash (43.9), and ngram speculation got 6% acceptance. At 3.5 GiB of weights, decode is no longer bandwidth-bound — there's nothing left for a drafter to amortize. KV-cache quantization and batch tuning were also flat. prism-ml's experimental megakernel/rmsnorm-qmv-fuse branch measured parity (42.8).
Practical guidance: on unified-memory hardware (GB10, Apple Silicon, Strix Halo), run the QAT Q1_0/Q2_0 plain — save the drafter for the PTQ tiers in this repo, where it genuinely pays (+31% on Q4_K_M). On high-bandwidth discrete GPUs the tradeoff may differ — measure via timings.draft_n_accepted.
Provenance
- Source:
prism-ml/Bonsai-27B-gguf F16 (53.8GB) — quantized directly from their official F16 GGUF, no re-conversion
- llama.cpp: mainline, commit
cecbf5fb0 (quantization + baseline numbers); PrismML fork 62061f91 branch prism (drafter numbers)
- imatrix: 63KB ChatML coding/debugging/reasoning corpus, 4096 ctx
- LICENSE/NOTICE carried over from the upstream repo (Apache-2.0)
- Quantized on NVIDIA DGX Spark (GB10, aarch64) — day-zero release, ~2.5h after the upstream drop
Report issues in the community tab — smoke-test failures, incoherence, or numbers that don't reproduce. Community benchmark reports welcome (include hardware, backend, and full launch command).
