Gemma 4 12B Instruct - QAT + NVFP4 Hybrid GGUF
QAT-optimized weights preserved at Q4_0, overhead tensors quantized to NVFP4.
What Makes This Different
This is a hybrid quantization of Google official QAT (Quantization-Aware Training) model. Instead of requantizing the Q4_0 weights (which breaks QAT benefits and vision quality), we:
- Kept all 329 weight tensors at Q4_0 - attention, FFN, embeddings - exactly as Google trained them
- Quantized only the 338 F32 norm/bias tensors to NVFP4 - these are the overhead tensors (layer norms, RMS norms, etc.)
- Used Google QAT mmproj (175 MB) - the vision projector trained alongside the QAT model
Why Standard NVFP4 from QAT Breaks Vision
Google QAT model was specifically trained to be resilient to Q4_0 quantization patterns. The weight values learned during QAT compensate for Q4_0 rounding. When you requantize Q4_0 -> F32 -> NVFP4:
- A second round of quantization error is introduced that QAT training did not account for
- NVFP4 uses different block sizes, rounding, and grouping than Q4_0
- Vision tokens flow through the same attention/FFN layers - precision loss disproportionately degrades vision
- The result: models that can not recognize text in images
How the Hybrid Approach Works
Using llama-quantize --tensor-type-file with --allow-requantize:
llama-quantize --allow-requantize --tensor-type-file keep_q4.txt input.gguf output.gguf NVFP4
The tensor-type-file lists all Q4_0/Q4_K_S tensors to keep at their current type. When the quantizer sees cur_type == new_type, it copies the tensor data as-is - zero precision loss. Only the remaining F32 tensors are quantized to NVFP4.
File Sizes
| Component | Original | Hybrid |
|---|
| Weight tensors (Q4_0) | 6.56 GB | 6.56 GB (unchanged) |
| Norm tensors (F32) | ~75 MB | ~15 MB (NVFP4) |
| Total GGUF | 6.65 GB | 6.39 GB |
| mmproj (QAT BF16) | 175 MB | 175 MB |
Why BPW Shows 4.50
The reported 4.50 BPW is misleading - it averages the Q4_0 weights (~4.34 BPW) with the NVFP4 norms (~4 BPW). The actual quality is determined by the Q4_0 weights, which are QAT-optimized.
Usage
`ash
llama.cpp
llama-server -m gemma-4-12b-it-qat-nvfp4.gguf --mmproj mmproj-gemma-4-12b-it-qat.gguf -ngl 99
`
Source
- Base model: google/gemma-4-12B-it-qat-q4_0-unquantized
- Quantized with: llama.cpp build 537 (commit d2c6795)
- Chat template: Native Gemma 4 (thinking enabled by default via <|think|> token)
- Vision: Full multimodal support via QAT mmproj
Files
| File | Size | Description |
|---|
| gemma-4-12b-it-qat-nvfp4.gguf | 6.39 GB | Q4_0 weights + NVFP4 norms |
| mmproj-gemma-4-12b-it-qat.gguf | 175 MB | QAT vision projector (BF16) |
License
Apache 2.0 (same as base model)
