The near-lossless 8-bit FP8 quantization of our K=4 biprojection abliteration of google/gemma-4-12B-it. Matches BF16 capability (MMLU, HumanEval, IFEval all within noise) at ~half the size and 1.6× the throughput. Loads in vLLM with --quantization modelopt. ~13 GB.
This is the recommended variant when quality matters. For maximum speed/smallest size (with a measured reasoning trade-off) see the NVFP4 sibling.
Refusal behavior has been removed; the model responds to a wide range of prompts the base would decline. Operator-side safety is your responsibility — see the arbitration clause at the bottom.
🚀 QuickStart
Complete copy-paste recipe for DGX Spark / Blackwell — pull the container, download the model, serve. This is a plain-decode variant (no speculative drafter — MTP is net-neutral on the GB10), so there is no drafter to pull and no --speculative-config.
bash
1# 1. Pull the unified AEON vLLM container (vLLM 0.24.0, sm_121a)2docker pull ghcr.io/aeon-7/aeon-vllm-ultimate:latest
34# 2. Download this model (fresh)5huggingface-cli download AEON-7/Gemma-4-12B-it-AEON-Abliterated-K4-FP8 \6 --local-dir ./aeon-model
78# 3. No drafter — this is plain autoregressive decode (no --speculative-config)910# 4. Serve (OpenAI-compatible API on :8000)11docker run -d --name aeon-gemma12b --gpus all --ipc=host --shm-size=16g --net=host \12 -v $(pwd)/aeon-model:/model:ro \13 --entrypoint vllm \14 ghcr.io/aeon-7/aeon-vllm-ultimate:latest \15 serve /model \16 --served-model-name gemma12b \17 --quantization modelopt \18 --attention-backend triton_attn \19 --reasoning-parser gemma4 \20 --tool-call-parser gemma4 \21 --enable-auto-tool-choice \22 --kv-cache-dtype fp8_e4m3 \23 --max-model-len 8192\24 --max-num-seqs 16\25 --gpu-memory-utilization 0.70\26 --enable-prefix-caching \27 --enable-chunked-prefill \28 --trust-remote-code
Unified-memory gpu-util: On the DGX Spark's unified memory keep --gpu-memory-utilization at 0.6-0.7; above ~0.8 the shared CPU+GPU pool page-thrashes. Discrete-VRAM GPUs can run higher.
⚠️ Needs vLLM ≥ 0.23.0 (for the Gemma4UnifiedForConditionalGeneration loader; landed in 0.22.2, validated on 0.23.0) and an FP8-capable GPU (Hopper H100, or Blackwell GB10 / B100 / B200 / RTX 50-series). The AEON vLLM Ultimate container ships the loader pre-built for DGX Spark sm_121a on vLLM 0.24.0.
Measured on a single DGX Spark (GB10 / Blackwell sm_121a) running ghcr.io/aeon-7/aeon-vllm-ultimate:latest (vLLM v0.24.0, built from source for sm_121a). This is a dense 12B run on plain decode — no speculative drafter: MTP is net-neutral on the GB10 (per-step drafter forward + the 262k-vocab lm_head verify roughly cancels the acceptance win on this hardware), so the production config runs straight autoregressive decode. The strength of this variant is concurrency throughput plus the quant-vs-speed tradeoff shown in the family chart below.
Headline: ~15.8 tok/s single-stream and ~747 tok/s aggregate at c=64 — near-lossless FP8 (per-channel weight / per-token activation) that matches BF16 capability at ~13 GB.
Gemma-4-12B-it K4-FP8 aggregate throughput vs concurrency (1→64) on DGX Spark GB10, vLLM 0.24.0
Per-category single-stream (c=1)
Plain decode — no speculative drafter, so draft-acceptance is not applicable (0%).
Category
decode tok/s
TTFT (ms)
TPOT (ms)
prefill (tok/s)
DFlash accept
Coding
15.8
141
63.2
354
n/a (plain decode)
Math
15.7
145
63.5
457
n/a (plain decode)
Reasoning
15.8
140
63.3
371
n/a (plain decode)
Prose
15.8
141
63.5
285
n/a (plain decode)
Natural language
15.8
145
63.5
304
n/a (plain decode)
Extraction / JSON
16.2
142
61.7
401
n/a (plain decode)
Single-stream decode is memory-bandwidth-bound on the GB10's unified LPDDR5X and sits at ~15.7–16.2 tok/s across every prompt category — the dense 12B body has to stream all FP8 weights per token regardless of workload.
Aggregate throughput by concurrency
Aggregate throughput scales near-linearly with concurrency, peaking at c=64:
This is the real production lever for this variant: batch the GB10 and aggregate throughput climbs to ~747 tok/s at 64 concurrent requests, with no engine crash (see the high-concurrency fix below). Long-context draft acceptance is not reported — this is a plain-decode run with no drafter.
Quant family — speed vs precision
How the four K4 quant variants of this model trade single-stream speed, aggregate throughput, and precision on the same hardware:
Gemma-4-12B-it K4 quant family: single-stream and c=64 aggregate throughput for FP8 / NVFP4 / NVFP4-FP8 mixed / BF16 on DGX Spark GB10
FP8 (this variant) is the near-lossless point: it matches BF16 capability (see the eval table below) while running ~2× the single-stream speed of BF16 and reaching ~747 tok/s aggregate. NVFP4 and the NVFP4-FP8 mixed variant push higher aggregate throughput at the cost of measured reasoning quality; BF16 is the slowest. Pick FP8 when quality matters, the 4-bit siblings for max throughput / min size.
What we fixed for the DGX Spark
All AEON models now run on one unified container — ghcr.io/aeon-7/aeon-vllm-ultimate:latest (= :2026-07-01-v0.24.0; rollback :2026-06-18-v0.23.0-dflashfix): vLLM v0.24.0 built from source for sm_121a, merged with the AEON speculative-decoding stack and tuned end-to-end for the GB10's unified-memory Blackwell architecture.
Unified container. A single sm_121a-native image loads every correctly-quantized model in the fleet (no per-model image), with the SM120-family CUTLASS FP8/NVFP4 kernels the GB10 actually dispatches to.
DFlash high-concurrency fix. The speculative drafter previously crashed at ≥32 concurrent requests (padded-vs-unpadded KV block-table shape mismatch in FlashAttention); we now slice the block-table to the unpadded batch (block_table[:num_reqs]), so spec-decode models scale cleanly to c=64. (A port of upstream PR #43982, which fixed this for MTP but never for DFlash.) This variant runs plain decode and so isn't affected by the drafter, but it shares the same hardened container.
Stock baseline pending fresh vanilla re-bench. No stock / un-optimized vanilla-vLLM baseline exists for this variant yet; a fully-vanilla benchmark on the current version is pending, so all numbers above are from the optimized aeon-vllm-ultimate:latest (vLLM 0.24.0) build.
Why FP8 — measured capability vs BF16
All four axes evaluated through the vLLM serving path (the real thing you'll run), identical prompts/settings for every model.
Full-length eval — MMLU balanced across all 57 subjects (5 each = 285 Q), full 164-problem HumanEval, IFEval-50. All via the vLLM serving path, identical prompts/settings.
Model
MMLU (285)
HumanEval-syn (164)
HumanEval-fun (164)
IFEval (50)
google/gemma-4-12B-it (official base)
81.4%
99.4%
82.9%
90%
K4-BF16 (abliterated, full precision)
80.4%
99.4%
83.5%
90%
K4-FP8 (this)
80.4%
99.4%
85.4%
90%
K4-NVFP4 MLP-only (4-bit sibling)
76.8%
96.3%
76.2%
90%
FP8 is statistically identical to the BF16 model — same MMLU (80.4% / 229-of-285, exactly), same HumanEval-syntactic, +noise on HumanEval-functional, same IFEval. This is the defining property: 8-bit FP8 is imperceptible from BF16. (For reference, the abliteration itself is capability-neutral too — K4-BF16 is within ~1pp of Google's official base on every axis.)
Throughput (DGX Spark GB10, FP8 KV cache, greedy)
FP8 (this)
BF16
NVFP4
Concurrent ×16 aggregate
235 tok/s
144 tok/s
341 tok/s
Single-stream overall
15.8 tok/s
7.7 tok/s
23.6 tok/s
Single-stream TTFT median
143 ms
—
102 ms
Size
13 GB
24 GB
8.5 GB
FP8 is 1.6× faster than BF16 at half the memory. NVFP4 is faster still but trades the reasoning quality shown above. Pick FP8 for quality, NVFP4 for max throughput/min size.
vLLM --quantization modelopt via Gemma4UnifiedForConditionalGeneration
Kept at full BF16
lm_head, model.language_model.embed_tokens, model.embed_vision*, model.embed_audio*, model.vision_embedder*. All language-stack attention + MLP linears (48 layers) are FP8.
vLLM loader note (for reproducers)
Google's Gemma-4-12B is the encoder-free Gemma4UnifiedForConditionalGeneration. ModelOpt's HF export needs two touch-ups to load in vLLM: (1) rename the vision keys to vLLM's vision_embedder.* layout, and (2) add model.vision_embedder* to the quant ignore list so the patch embedder stays BF16. Both are scripted in make_vllm_ready.py (gemma4-nvfp4/). Requires vLLM ≥ 0.23.0 (loader landed in 0.22.2; validated on 0.23.0).
Abliteration methodology (inherited from the BF16 base)
K=4 multi-direction norm-preserving biprojection (extends TrevorJS's recipe). Basis layers L24/L37/L39/L26 (top-K by SNR), o_proj + mlp.down_proj edited on 24/48 layers, scale=1.0. See the BF16 card for the full biprojection math + capability comparison vs base.
Behavior
Benign prompts: indistinguishable from BF16 (capability table above confirms it numerically).
Previously-refused prompts: full responses, usually after a brief disclaimer paragraph.
Tool calling via --enable-auto-tool-choice --tool-call-parser gemma4.
By accessing, downloading, using, running inference on, fine-tuning, merging, quantizing, distributing, integrating, or otherwise interacting with this model, you acknowledge and agree to the following:
Sole Responsibility. You, the user, are solely and exclusively responsible for (a) every prompt you or your downstream system issue to this model, (b) every response this model produces in reply, (c) every downstream action taken by you, your systems, your agents, or your users in reliance on those responses, and (d) any harm — direct, indirect, consequential, foreseeable, or otherwise — that results from any of the above.
No Warranty. This model is provided strictly "AS IS", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, non-infringement, safety, alignment, factual accuracy, or legal compliance in any jurisdiction. No contributor, author, publisher, or hosting platform assumes liability of any kind for outputs or downstream use.
Legal Compliance. You are responsible for ensuring that your use of this model complies with all applicable laws, regulations, terms of service, industry codes of conduct, professional ethical standards, and organizational policies in every jurisdiction in which you operate or in which your outputs may be received. The unaligned nature of this model does not grant you any legal authorization you did not already have.
Operational Safety Layer. An uncensored model is not a toy. You are expected to implement appropriate downstream safety layers proportionate to your deployment context, including but not limited to: input validation, output filtering, content moderation, audit logging, rate limiting, access controls, and human-in-the-loop review for high-risk workflows. A production deployment of this model without such layers is unsafe by construction and is not a supported use case.
Heightened Duty of Care. The absence of internal refusal behavior means the duty of care that would ordinarily rest partly with the model rests entirely with you. You are expected to exercise greater — not lesser — caution, forethought, and ethical discipline when operating this model than you would operate a base aligned model. If you are uncertain whether your contemplated use is ethical, legal, or wise, the correct action is to not make the request.
No Endorsement of Outputs. The authors, contributors, and publishers of this model do not endorse, adopt, or take responsibility for any specific output this model produces. Outputs are a stochastic function of the prompt, the weights, and the sampler state — not a statement of position by any human.
Arbitration. Any dispute, claim, or controversy arising out of or relating to the use of this model, its outputs, or this clause shall be resolved through binding individual arbitration under the rules of a mutually agreed arbitration body (or, absent agreement, the American Arbitration Association's Consumer Arbitration Rules), waiving any right to a jury trial, class action, representative action, or consolidated proceeding. Venue shall be the jurisdiction of the disputing party bringing the claim. Costs and attorneys' fees shall be allocated per the applicable arbitration rules. This clause does not expand, and where legally prohibited does not establish, any liability in the other direction; it limits how the user may proceed when alleging harm tied to their own use of this model.
Indemnification. You agree to indemnify, defend, and hold harmless the authors, contributors, and publishers of this model from and against any claims, damages, losses, liabilities, costs, and expenses (including reasonable attorneys' fees) arising from or related to your use of the model or your breach of this clause.
Severability. If any provision of this clause is held unenforceable in a given jurisdiction, the remaining provisions remain in full force in that jurisdiction, and the unenforceable provision is replaced by the closest enforceable equivalent consistent with the original intent.
Acceptance. Your use of this model constitutes your acceptance of this clause in full. If you do not accept, do not use the model.
This model is a tool with no opinions of its own. You supply the opinions. You supply the judgement. You supply the ethics. The outputs carry your fingerprints, not the model's.
☕ Support the work
If this release has been useful, tips are deeply appreciated — they go directly toward more compute, more models, and more open releases.
