The dense FFN tensors (all 48 layers × 3 projections = 144 tensors) are quantized to NVFP4 — NVIDIA's 4-bit float with FP8-E4M3 block scale over 16-element groups. The remaining tensors (attention, embeddings, output) use a conventional GGUF quant; three variants are provided.

NVIDIA shipped official NVFP4 weights for Gemma-4-31B and the 26B-A4B MoE, but not the dense 12B — so we calibrated it ourselves with NVIDIA ModelOpt (cnn_dailymail, NVFP4 MLP-only, matching NVIDIA's 31B recipe). The ModelOpt source checkpoint is at LibertAIDAI/Gemma-4-12B-IT-NVFP4.

About LibertAI

LibertAI is a decentralized AI platform — private inference, an OpenAI-compatible API, and a chat UI, all running on community GPUs over Aleph Cloud instead of a single company's servers. No accounts required to chat, no logs sent home, and the same models you'd self-host are available behind a sovereign endpoint.

If you want to put this model to work as an autonomous agent without running your own infrastructure, check out LiberClaw — Hermes-style agents hosted on Aleph Cloud with LibertAI inference. Free tier: 2 agents, no credit card, 5 minutes to deploy. Open source.

Why NVFP4? On NVIDIA Blackwell GPUs (RTX 50-series, B100/B200), llama.cpp uses native NVFP4 tensor-core MMA kernels (added in llama.cpp #22196) for the FFN matmul — the dominant compute cost during inference. On older GPUs the path falls back to dp4a/MMQ kernels, where these GGUFs run but offer no perf advantage over standard K-quants.

Files

File	Size	FFN	Other tensors	When to pick
`Gemma-4-12B-IT-NVFP4-Q4_K_M.gguf`	6.5 GiB	NVFP4	Q4_K_M	Recommended. Fastest serving throughput on Blackwell + smallest VRAM footprint
`Gemma-4-12B-IT-NVFP4-Q8_0.gguf`	7.8 GiB	NVFP4	Q8_0	Higher-precision attention/embeddings if you have the VRAM
`Gemma-4-12B-IT-NVFP4-BF16.gguf`	10.8 GiB	NVFP4	BF16	Max quality (preserves source precision); slower in practice — only for bit-for-bit source fidelity

Text in / text out. This release is text-only. Gemma-4-12B-it is the unified (gemma4_unified) variant whose vision+audio towers use a transformer-less, soft-token design that llama.cpp's multimodal converter does not yet support, so no mmproj is shipped. The 144 NVFP4 FFN tensors and full text model are unaffected.

Performance

Measured on an NVIDIA RTX 5090 (32 GB, Blackwell, sm_120), llama.cpp build b9341-dbe9c0c8c.

Batched serving (`llama-batched-bench`, 512 in / 128 out per request)

NVFP4-Q4_K_M vs stock Q4_K_M on RTX 5090

NVFP4-Q4_K_M wins on total serving throughput across the board, with the largest gains at the moderate batch sizes that dominate real serving — +15% at 4 parallel, +21% at 8 parallel (and matching token-generation gains of +14% / +20%). At single stream the two are tied; at very high concurrency (16 parallel) total throughput still leads (+5%) while per-token generation converges (the regime becomes attention/KV-bound, where both keep attention in the same precision). It also uses less VRAM (6.5 vs 6.9 GiB), leaving more room for KV cache.

Variant comparison (single-stream, `llama-bench`, -r 3)

Variant	Size	PP512 (tok/s)	TG64 (tok/s)
`NVFP4-Q4_K_M`	6.52 GiB	9914	147.5
`NVFP4-Q8_0`	7.83 GiB	9587	129.3

Usage

llama-cli -m Gemma-4-12B-IT-NVFP4-Q4_K_M.gguf -ngl 999 -c 8192 -p "Your prompt here"

llama-server -m Gemma-4-12B-IT-NVFP4-Q4_K_M.gguf -ngl 999 -c 32768 --host 0.0.0.0 --port 8080

Gemma 4 is a reasoning model — it emits a thinking trace before its answer by default.

About the architecture

Gemma-4-12B-IT is an ~11.9B-parameter dense decoder (48 layers) with a hybrid local-sliding + global attention scheme — every 6th layer is a full-attention layer with a larger unified-KV head dim, the rest are sliding-window. The NVFP4 quantization covers the 48 × 3 dense FFN projections; attention projections, embeddings and lm_head stay at higher precision in the source (that's why the -BF16 variant is comparatively large).

Sources & credits

Base model: google/gemma-4-12B-it by Google — Gemma license
NVFP4 calibration: self-quantized with NVIDIA ModelOpt 0.44 on an RTX 5090 → LibertAIDAI/Gemma-4-12B-IT-NVFP4
Tooling: llama.cpp convert_hf_to_gguf.py and llama-quantize

License

Inherits the Gemma license from the upstream model.

LibertAIDAI/Gemma-4-12B-IT-NVFP4-GGUF

Gemma-4-12B-IT NVFP4 GGUF