Uncensored gemma-4 12B coder for local, agentic tool use — GGUF quantizations for llama.cpp / Ollama.
Run it:llama-server -hf tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5-abliterated-GGUF:Q4_K_M --jinja (full commands below).
⚠️ Tool-calling needs the recovery shim. The model emits gemma-4's native tool markup, which llama.cpp --jinja under-parses — wrap your endpoint with the tool-shim (see Tool-calling below) to get standard tool_calls.
💡 Pick this to run locally with the best of both: SFT v5's tool-calling and an uncensored model. Our KL-guarded abliteration on top of SFT v5 — gate SHIM 8/8.
1# llama.cpp (server) — tool-calling needs the recovery shim, see below2llama-server -hf tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5-abliterated-GGUF:Q4_K_M --jinja --ctx-size 1638434# Ollama5ollama run hf.co/tpls/gemma-4-12B-coder-fable5-composer2.5-v1-sft-v5-abliterated-GGUF:Q4_K_M
Files
Sizes and a one-click loader are in the file browser / Quantizations widget above;
the note says which quant to reach for.
Quant
Notes
Q4_K_M
good default — fits 12 GB VRAM, best size/quality balance
Q5_K_M
higher quality, ~9.5 GB
Q6_K
near-bf16 quality, ~10.5 GB
Q8_0
highest GGUF quality, large
IQ4_XS
smallest usable — for <8 GB VRAM, slight quality cost
Q3_K_M
low-VRAM fallback, noticeable quality drop
Tool-calling
Tool-calling works — but llama.cpp --jinja doesn't recognise gemma-4's native
tool-call markup, so the bare parser under-reports calls. The model is fine; the
parser is blind to the format. Recover standard tool_calls with a small serve-side
post-processor (no weight change, no latency beyond a regex scan).
Ready-to-use → tpls/gemma4-tool-shim — a drop-in
callback for OpenAI-compatible proxies, a standalone (dependency-free) example, and the pure
parser, all Apache-2.0, with the full recovery algorithm documented. Point your
OpenAI-compatible endpoint through it.
You send tools the usual OpenAI way (tools=[…]); the model emits native markup; the
shim turns it into a standard tool_calls object:
# model completion (raw):
<|tool_call>get_weather{"city": "Paris", "units": "celsius"}
json
1// after the shim:2{"finish_reason":"tool_calls",3"message":{"role":"assistant","content":null,4"tool_calls":[{"id":"call_0","type":"function",5"function":{"name":"get_weather","arguments":"{\"city\": \"Paris\", \"units\": \"celsius\"}"}}]}}
Tool-calling gate
Measured 2026-06-26 on the tool-eval suite (7 tool cases + 1 abstain), the Q4_K_M quant served on llama.cpp --jinja with TOOLS_IN_PROMPT=1 at temp 0. raw = native parser; shim = the prod gemma_tool_parse recovery path. SHIM 8/8 = identical to clean SFT v5: abliteration preserved tool-calling (KL=0.0009). The low raw is gemma-4's known native-parser breakage (the shim is the prod path), not abliteration damage.
The rows are this model under two parse paths (raw and shim); the shim path is how it's served in production.
Measured on
Pass rate
this model — raw (--jinja)
0.125
this model — shim (prod path)
1.000
Intended use & limitations
Built for code generation and agentic tool use; serve locally via llama.cpp /
Ollama, or use as a base to fine-tune / merge / quantize. Outputs can be wrong or
fabricated — validate tool arguments before executing, and keep a human in the loop
for anything consequential.
⚠️ Uncensored. For this variant the refusal direction has been ablated from the weights — safety guardrails are
substantially removed and it will attempt requests a stock model would refuse. You
are responsible for what you generate and how it's used; not suitable where refusal
behaviour is itself a safety requirement.
llama.cpp quantization with an importance matrix (imatrix)
—
4
tool-shim
serve-side recovery of structured tool_calls from the model's native markup
—
1. sft-qlora
tools_mode: mixed
2. abliteration
weight ablation degrades the canonical <|tool_call> token — the model tends to leak calls as text markup, so the native llama.cpp parser may not fire. See the tool-call recovery note below to get structured calls back.
3. imatrix-quant
calibration: code + tool-call markup
embed/output: kept at f16 (protects tool-call logits)
eog_patch: tokens 105/106 → EOG (bounds the <|turn> runaway)
4. tool-shim
where: a thin pre/post wrapper on the OpenAI-compatible endpoint
format: re-parse <|tool_call>NAME{json-args} (and leaked <|tool>…) into tool_calls
serve-side only — does not modify the weights; recommended for abliterated variants.
Training data, hyperparameters & environment
The supervised fine-tune (sft-qlora step above) is inherited from
Gemma-4 12B Coder — SFT v5 (weights) — see that card
for the full training mix, exact hyperparameters, and pinned environment. The remaining
step(s) above are what this model adds on top; their measured effect is below.
Quantization environment
The GGUF bytes depend on the quantizer build, not just the weights — a different
llama.cpp release rounds tensors differently and can change the convert mapping. Pins
the toolchain these quants were produced with:
llama-imatrix over the calibration set (CPU forward pass)
quantize
llama-quantize --imatrix, token-embeddings + output tensor kept at f16
The image is the rolling :full tag, not a digest — for byte-exact reproduction pin
the image digest you build with. The imatrix-quant step above lists the calibration
set and the EOG patch this build applied.
