You can deploy this model and start earning money today!

Zhongzhu/
OSCAR-LLAMACPP-Gemma-4-12B-it-INT2-KV

Name: OSCAR-LLAMACPP-Gemma-4-12B-it-INT2-KV
Availability: InStock

gguf

endpoints_compatible

Views

No views yet

Gemma 4 12B IT — OSCAR INT2 KV cache (GGUF)

Gemma-4-12B-it in GGUF, packaged for the OSCAR ~2-bit (INT2) KV cache fork of llama.cpp. Two independent axes are combined here:

weights: bf16 (full precision) or Q4_K_M (~4.5-bit) — affects model size/quality, not the KV cache.
OSCAR rotation: rot-kv = the per-layer calibrated rotation is baked into the GGUF, which is what makes the 2-bit KV cache keep near-f16 quality. base = plain model, no rotation.

The -rot-kv models need the OSCAR fork (-fa on + --cache-type-k/v q2_0 + the env vars below). The base models are standard GGUFs that run on stock llama.cpp with a normal f16 KV cache.

Folders

folder	weights	OSCAR rotation	size	what it's for
`q4km-rot-kv/` ⭐	Q4_K_M	✅ baked in	6.9 GB	Recommended. Small weights + OSCAR INT2 KV — the practical deployment model.
`bf16-rot-kv/`	bf16	✅ baked in	22 GB	Full-precision weights + OSCAR INT2 KV. Use to isolate the KV-cache effect (identical weights, compare f16-KV vs INT2-KV) or for max quality.
`base-q4km/`	Q4_K_M	❌ none	6.9 GB	Plain Gemma 4 12B, Q4_K_M weights. Runs on stock llama.cpp (f16 KV).
`base-bf16/`	bf16	❌ none	22 GB	Plain Gemma 4 12B, full precision. Stock llama.cpp, or use as the base to bake your own rotation.
`rotation/`	—	the matrices	2×18 MB	Raw OSCAR calibrated rotation (`k_/v_rotation_*.pt`, per-layer 512×512). Bake onto any Gemma 4 12B base GGUF.

"OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to Q/K (and V) so that the KV cache quantizes well at 2-bit. The -rot-kv GGUFs already contain it as blk.{i}.attn_{k,v}_rot.weight tensors; the base weights are copied through unchanged.

Run (the `-rot-kv` models)

Needs the OSCAR fork built with Metal. Gemma 4 also needs its chat template.

bash

LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m q4km-rot-kv/gemma-4-12b-it-rot-kv.gguf \
  -fa on -ngl 99 -c 16384 \
  --cache-type-k q2_0 --cache-type-v q2_0 \
  --chat-template-file models/templates/google-gemma-4-31B-it.jinja \
  --host 127.0.0.1 --port 8080

env var	meaning	value
`LLAMA_KV_FUSED_FA`	fused INT2+f16 flash-attention kernels (fast Metal path)	`1`
`LLAMA_KV_NO_HADAMARD`	rotation is in-graph, so skip the in-quant Hadamard	`1`
`LLAMA_KV_CLIP_RATIO`	per-row outlier clip before quant	`0.96`
`LLAMA_KV_HP_SINK` / `LLAMA_KV_HP_RECENT`	tokens kept high-precision (first/last)	`512` / `2048`

--cache-type-v f16 keeps V high-precision (a bit more quality, more memory). A base (non-rotated) model with INT2 flags falls back to degraded data-free INT2 — use a -rot-kv model for INT2.

Bake your own rotation