The dense FFN tensors are quantized to NVFP4 (NVIDIA's 4-bit float with E4M3 block scale), repacked from the official nvidia/Gemma-4-31B-IT-NVFP4 ModelOpt calibration. The remaining tensors (attention projections, embeddings, output) use a conventional GGUF quant — three variants are provided.
About LibertAI
LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.
If you want to put this model (or any other) to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.
Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the FFN matmul — the dominant compute cost during inference. On older GPUs the path falls back to dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.
Files
File
Size
FFN
Other tensors
When to pick
Gemma-4-31B-IT-NVFP4-Q4_K_M.gguf
17 GB
NVFP4
Q4_K_M
Recommended. Fastest serving throughput on Blackwell + smallest VRAM footprint
Gemma-4-31B-IT-NVFP4-Q8_0.gguf
21 GB
NVFP4
Q8_0
Higher precision attention/embeddings if you have the VRAM
Gemma-4-31B-IT-NVFP4-BF16.gguf
30 GB
NVFP4
BF16
Max quality (preserves source precision); slower in practice — only pick for bit-for-bit source fidelity
mmproj-Gemma-4-31B-IT-F16.gguf
1.2 GB
—
F16 vision + audio towers
Required for image/audio input — reusable with any Gemma-4-31B-IT GGUF
Performance
Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build c84e6d6db.
Batched serving (llama-batched-bench, 512 in / 128 out per request)
NVFP4-Q4_K_M vs stock Q4_K_M on RTX 5090
NVFP4-Q4_K_M beats stock Q4_K_M on total serving throughput at every parallel batch size we tested (+7% / +0% / +2% at 1 / 4 / 8 sequences), with consistent token-generation wins (+8% / +4% / +6%). It also uses less VRAM (16.6 vs 18.2 GiB), leaving more room for KV cache.
Gemma-4's unified-KV global attention has a large KV cache footprint — at parallel=16 with 8 K context the model + cache exceeds 32 GB on a 5090. If you serve high concurrency, the smaller weight footprint of the NVFP4 variant matters even more.
Gemma 4's mmproj bundles both vision and audio towers, so you can send image and audio content blocks — see the llama.cpp multimodal docs.
About the architecture
Gemma-4-31B-IT is a 30.7B-parameter dense decoder with a hybrid local-sliding + global attention scheme (unified KV in global layers, Proportional RoPE for long context). The NVIDIA NVFP4 source quantizes the 60 layers × 3 dense FFN projections (180 NVFP4 tensors total). The attention projections (attn_q/attn_output are particularly large in this architecture due to the unified-KV global attention) and embeddings stay at higher precision in the source — that's why the -BF16 variant is comparatively large at 30 GB.
