gemma-4-12b-it-mxfp4.gguf — Text backbone (48 transformer layers, 3840 hidden dim, 262k context) quantized to MXFP4
mmproj-gemma-4-12b-it-f16.gguf — SigLIP vision encoder + projector at F16 precision (required for image input)

About MXFP4

MXFP4 is the OCP (Open Compute Project) microscaling FP4 standard — E2M1 4-bit values with E8M0 power-of-two block scaling (one scale per 32 elements). Portable across NVIDIA, AMD, and CPU backends.

Feature	MXFP4	NVFP4	Q4_K_M
Numeric format	E2M1 microscaling	E4M3 native FP4	INT4 block quantization
Block size	32 elements	32 elements	32 elements
Effective BPW	4.45	4.68	~4.50
Hardware portability	Universal (CUDA/AMD/CPU)	Blackwell only	Universal
Hardware acceleration	CUDA cores / CPU / AMD	Blackwell tensor cores	CUDA cores / CPU

When to use MXFP4: You want 4-bit quality on any hardware — NVIDIA (pre-Blackwell and Blackwell), AMD, or CPU. Universal portability with competitive compression.

When to use alternatives: You have a Blackwell GPU and want maximum throughput — use NVFP4 instead. You want maximum quality at 4-bit — use Q4_K_M.

Files

Filename	Type	Size	BPW	Description
`gemma-4-12b-it-mxfp4.gguf`	MXFP4 quantized (text)	6.18 GB	4.45	48-layer text backbone with hybrid attention
`mmproj-gemma-4-12b-it-f16.gguf`	Vision encoder (F16)	117 MB	16.0	SigLIP vision embedder + GEMMA4UV projector

Quantization Characteristics

Metric	Value
Input format	F16 GGUF (23.83 GB, 667 tensors)
Output format	MXFP4 GGUF (6.18 GB, 667 tensors)
Quantization type	`LLAMA_FTYPE_MOSTLY_MXFP4` (type 41)
Compression ratio	3.86× (23.83 GB → 6.18 GB)
Quantization time	~109 seconds on RTX 5060 Ti
1D tensors (norms, scales)	Kept at F32
Attention + FFN weights	Converted to MXFP4

Model Description

Gemma 4 12B is part of Google's fourth-generation Gemma family, featuring:

48 transformer layers with 3840 hidden dimensions and 15360 FFN intermediate size
Hybrid attention: 40 sliding-window layers (window 1024, kv_heads=8, head_dim=256) interleaved with 8 full-attention layers (kv_heads=2, head_dim=512) in a 5:1 pattern
Context window: up to 262,144 tokens
RoPE scaling: separate frequency bases for sliding window and full attention
Final logit softcapping: stabilizes large-vocabulary predictions
Vision: SigLIP-based embedder (not a full ViT — a lightweight patch embedder with learned positional encoding) enables native image understanding without a separate vision transformer
Instruction-tuned: optimized for chat and instruction-following with Gemma 4's structured turn format

The base model is google/gemma-4-12B-it under Apache 2.0 license.

Usage

llama.cpp CLI

bash

# Text-only inference
./llama-cli \
  -m gemma-4-12b-it-mxfp4.gguf \
  -p "Explain quantum computing in simple terms" \
  -n 512

# Vision inference (requires mmproj)
./llama-cli \
  -m gemma-4-12b-it-mxfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-f16.gguf \
  --image diagram.png \
  -p "Explain what this diagram shows" \
  -n 512

llama.cpp Server (OpenAI-compatible API)

bash

./llama-server \
  -m gemma-4-12b-it-mxfp4.gguf \
  --mmproj mmproj-gemma-4-12b-it-f16.gguf \
  --port 8080 \
  -ngl 99 \
  -c 8192

LM Studio

Download both .gguf files
Place them in the same model folder
In LM Studio, select the model and ensure the mmproj is auto-detected (same basename)
The chat template is embedded in the GGUF — no manual configuration needed

Python (llama-cpp-python)

python

from llama_cpp import Llama

llm = Llama(
    model_path="gemma-4-12b-it-mxfp4.gguf",
    mmproj="mmproj-gemma-4-12b-it-f16.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,
)

output = llm("What is the capital of France?", max_tokens=128)
print(output["choices"][0]["text"])

Thinking / Reasoning Behavior

Gemma 4 supports structured reasoning using <|channel>thought tags:

enable_thinking=false (default): Empty think tags suppress reasoning output.
enable_thinking=true: Model may generate internal reasoning tokens before the response.

Conversion Pipeline

google/gemma-4-12B-it (23.8 GB, safetensors, updated 2026-07-20)
  │
  ├─ convert_hf_to_gguf.py --outtype f16
  │     → gemma-4-12b-it-f16.gguf (23.83 GB, 667 tensors)
  │
  ├─ convert_hf_to_gguf.py --mmproj --outtype f16
  │     → mmproj-gemma-4-12b-it-F16.gguf (117 MB, 11 tensors)
  │
  └─ llama-quantize.exe MXFP4
        → gemma-4-12b-it-mxfp4.gguf (6.18 GB, 667 tensors, 4.45 BPW)

Built with latest llama.cpp (Gemma4 per_layer_config transformer 5.15 support)
Updated 2026-08-17: Rebuilt from latest google/gemma-4-12B-it (updated 2026-07-20)

Hardware Requirements

GPU	VRAM	Model + KV Cache (4k ctx)	Estimated Speed
RTX 5060 Ti 16GB	16 GB	~10 GB	~45 tok/s
RTX 4070 12GB	12 GB	~10 GB	~40 tok/s
CPU (6+ cores)	System RAM	~7 GB	~5-8 tok/s

Minimum VRAM: ~10 GB for the model (~6.2 GB at 4.45 BPW + ~2.5 GB KV cache at 8192 ctx).

License

Apache 2.0, as per Google's Gemma 4 license.

FreedomAISVR/Gemma-4-12B-it-MXFP4-GGUF

Gemma-4-12B-it-MXFP4-GGUF