The hybrid quant employs different quantization levels on a per layer basis to increased
flexibility of trading off performance vs file size. Less parameter bits are used at deep layers
and more bits at cortex layers to simultaneously optimize quantized size and model performance.
K quants are used in all the layers for faster CPU processing on partially offloaded models or
GPU processing on older GPUs.
The layer quants are as follows (refreshed on 4/22/2026):
To bypass thinking inject the think block delimiters following the assistant prompt template. The
model is strong with think blocked bypassed but less accurate on harder prompts. The model exhibits
strong "common sense", and exhibits correct reasoning on prompts smaller models mostly miss. It is a dense
32G parameter model which is not practical to run on CPU and must be offloaded into GPU by hook or crook
to get usable gen rates.
This model can run fully offloaded on a 24G VRAM. This 24G VRAM can be cobbled together with 2x4070
over RPC. Q8 KV cache can be used to expand context. The model can be efficiently speculated with Qwen3 0.6B.
Example configs and gen rates for 2x4070 (1 RPC) with optional Qwen3 0.6B speculation running with a custom
downstream speculator with fixed draft block size ND on llama.cpp:
