Credits
- Used calibration text from Bartowski's gist: https://gist.github.com/bartowski1182/82ae9b520227f57d79ba04add13d0d0d
- Used nvidia/Gemma-4-31B-IT-NVFP4 as base model
- Template is using google/gemma-4-31B-it latest template
Update
- [05-22-2026] Updated Q6_K-NVFP4 version: added the latest official template, tweaked some tensor precisions. The latest eval result is from this version.
- [05-22-2026] Updated custom template to completely preserve thinking. Experimental.
- [04-29-2026] Added a custom template that forces a new turn after tool call. From my own testing it stabilizes tool-calling loop,
basically eliminated "I'm editing this file now." and stop. Tested with latest llama.cpp build.
- [04-26-2026] Added a Q6_K version. Most attention weights are Q6_K, while full attention Wq is in q8_0 and Wk in bf16. BPW 5.43.
Eval (for smaller Q6_K-NVFP4 version)
- GPQA Diamond 84.3% Wilson Score [78.6%, 88.7%]. Official 84.3%
- AIME 2026 4 run average 90.8%, Aggregated 95% Wilson Score [84.3%, 94.8%]. Official 89.2% (Not better than Official!! It's within confidence interval)
Notes
Key modification include quantizing swa's Wq Wk Wv and attention output into q8_0, but kept Wk and global attention output bf16.
Since nvfp4 significantly compressed the FFN tensors, resulting BPW is conveniently just over 6.
Flow
- convert_hf_to_gguf with nvidia's nvfp4 checkpoint
- generate imatrix with bartowski's calibration data
- generate "keep" list to preserve unified Wk for global attention blocks (every 6 block)
- llama-quantize to q8_0 (for rest of the tensors: embedding, swa Wq Wk Wv, etc.)
NVFP4
NVFP4 related tensors are untouched: weight scales for proper fp4 scaling are kept unchanged.
Image
For image, get any of the mainstream mmproj-bf16 gguf.
