Views
No views yet
prompt_logprobs KL on a 12.3k-token held-out corpus
(disjoint from calibration), 8,176 positions, all arms served back-to-back on
one box against the same teacher capture — a strict relative comparison.| artifact | size | KL (all) | KL (confident half) | top-1 (conf) |
|---|---|---|---|---|
| this artifact (CB ladder, 5.5 bpp body) | 23.0 GB | 0.0049 | 0.00295 | 99.88% |
| Qwen3.6-27B-PrismaAURA-5.5bit (NVFP4+FP8) | 23 GB | 0.0211 | 0.0130 | 99.58% |
| Qwen3.6-27B-PrismaSCOUT 5p31 | 20.6 GB | 0.0344 | 0.0249 | 99.04% |
| section | format | notes |
|---|---|---|
| 504 body Linears | FP8-CB K36–K47 (426 units) + BF16 (69) | per-Linear rungs chosen by measured error under a byte budget; K47 (5.88 bpw) ×175, K44 ×109, K40 ×108 — the ⅛-bpw all-integer ladder in actual use |
| vision tower (110 Linears) | NVFP4 weight-only (W4A16) | quantized ViT, verified on image inputs |
| MTP draft block | BF16 | canon throughput selector: highest-fidelity rung that fits |
| embeddings + lm_head | BF16 | 248k vocab, untied |
allocation/. Codebooks (cb_codebooks.pqcb)
are shared per role family and total a few MB.| Plugin | gridbook — an out-of-tree vLLM quantization plugin. Stock vLLM, no fork, no core patches. |
| GPU | NVIDIA Blackwell, compute capability sm_120 / sm_121. Measured on GB10 / DGX Spark (sm_121). On older GPUs the plugin still loads but runs its Triton fallback kernels — correct, not fast, and not a production serving target. |
| Memory | 20.8 GiB weights + KV cache, measured on one 128 GB unified-memory GB10 (Model loading took 20.82 GiB). Before gridbook 0.1.0 the same artifact loaded at 35.86 GiB — dense codebook weights were resident twice (issue #1, fixed in 0.1.0); use 0.1.0 or newer. A 32 GB consumer Blackwell (RTX 5090, sm_120) should now fit weights plus a useful KV budget, and the issue reporter measured 20.54 GiB with 32k context using equivalent patches — but we have not verified 0.1.0 on a 32 GB card ourselves. |
| vLLM | Served here on the vllm-node image used for the 0.1.0 verification; also exercised on 0.25.1 and 0.23.1rc1.dev1060 by an external reporter (issue #1). The plugin surface is small but vLLM internals drift — pin a version you have tested. |
| Toolchain | CUDA toolkit with nvcc on PATH in the serving container — the plugin JIT-builds its kernels on first model load (~30 s, cached). nvcc 13.0 is the tested toolchain. |
| Parallelism | Single GPU (tp=1). Tensor parallelism is not implemented in the plugin. |
1pip install gridbook # JIT-builds kernels for your GPU (capability >= 8.0 floor)
2vllm serve rdtand/Qwen3.6-27B-prismaquant-gridbook-5.5bit-vllm \
3 --max-model-len 32768 --gpu-memory-utilization 0.90