You can deploy this model and start earning money today!

Danny-Dasilva/
gemma-4-12B-it-W4A16-GPTQ-g32-DSpark

Name: gemma-4-12B-it-W4A16-GPTQ-g32-DSpark
Availability: InStock

Views

No views yet

gemma-4-12B-it — W4A16 GPTQ (group=32) · DSpark-verified

A post-training GPTQ int4 (W4A16, group_size=32, symmetric) quantization of google/gemma-4-12B-it, built specifically to be a DSpark speculative-decoding target (draft head: deepseek-ai/dspark_gemma4_12b_block7).

TL;DR: 198 tok/s single-stream on one RTX 5090 (32 GB) — the fastest gemma-4-12B configuration we measured on that card, at 7.8 GB of weights.

Why this exists — QAT breaks DSpark, post-training quant doesn't

The DSpark draft head was trained against the bf16 gemma-4-12B-it weights. Any retraining of the target — including the official QAT checkpoint google/gemma-4-12B-it-qat-w4a16-ct — drifts far enough that draft acceptance collapses and speculation makes the model slower:

target (same W4A16 format)	DSpark per-token accept	outcome
QAT w4a16-ct (retrained)	2.1%	slower than no speculation
this model (post-training GPTQ)	35.6%	1.73× speedup
fp8 (post-training, reference)	34.4%	1.9× over its own native
bf16 (reference)	39.5%	2.7× (eager)

Post-training rounding preserves draft alignment; QAT retraining destroys it. If you pair gemma-4-12B int4 with the DeepSeek DSpark draft, you need a post-training quant like this one.

Measured speed — RTX 5090 (32 GB, Blackwell sm_120), single stream (bs=1)

vLLM (recent main with gemma-4 DSpark support), FlashInfer target attention, greedy, 256-token generations over 4 chat prompts, CUDA graphs (piecewise):

config	tok/s	vs native int4
this model + DSpark k=7 + CUDA graphs + FlashInfer autotune	198.2 (peaks 239)	1.73×
this model + DSpark k=7 + CUDA graphs	193.8	1.69×
this model + DSpark k=14 (2 draft blocks)	178.6	don't — accept dies at block boundary
this model, native (no speculation), CUDA graphs	114.7	1.0×
fp8 (online) + DSpark k=7 (13 GB weights)	165	—
bf16 + DSpark k=7 (eager; graphs OOM on 32 GB)	144	—
official QAT w4a16-ct, native	118	speculation hurts this one
llama.cpp Q4_0 GGUF, native (reference)	156	its DSpark port is slower than native

Weights: 7.8 GB (vs 24 GB bf16, 13 GB fp8) — target + draft + KV fit comfortably in 32 GB with CUDA graphs, which bf16 cannot do.

Usage (vLLM + DSpark)

python

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

MODEL = "Danny-Dasilva/gemma-4-12B-it-W4A16-GPTQ-g32-DSpark"
DRAFT = "deepseek-ai/dspark_gemma4_12b_block7"

llm = LLM(
    model=MODEL,
    max_model_len=8192,
    attention_backend="FLASHINFER",       # gemma-4 full-attn layers have head_dim 512
    enable_flashinfer_autotune=True,      # +2% in our runs
    speculative_config={
        "method": "dspark",
        "model": DRAFT,
        "num_speculative_tokens": 7,      # = draft block size; k<7 or k=14 are both slower
        "attention_backend": "TRITON_ATTN",
    },
)

tok = AutoTokenizer.from_pretrained(MODEL)
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Explain transformers step by step."}],
    add_generation_prompt=True, tokenize=False,
)
print(llm.generate([prompt], SamplingParams(temperature=0.0, max_tokens=256))[0].outputs[0].text)

Notes:

Always apply the chat template — gemma-4-it uses <|turn>-style tokens and add_bos_token=False; raw prompts produce garbage on any backend.
num_speculative_tokens must be ≤ 7 or a multiple of 7 (draft block size). 7 is optimal.
Works without DSpark too, as a normal compressed-tensors W4A16 checkpoint (kernel path identical to the official QAT release: group=32, symmetric, pack-quantized).
kv_cache_dtype="fp8" measured slower (−3%) at short context on this card.

Quantization recipe

llm-compressor 0.12 GPTQ, one-shot:

256 calibration samples from HuggingFaceH4/ultrachat_200k (train_sft), chat template applied, max_seq_length 2048
scheme: int4, group_size=32, symmetric, weights-only (W4A16), pack-quantized format — mirrors the official QAT checkpoint's config so every vLLM kernel path is identical
sequential_targets=["Gemma4UnifiedTextDecoderLayer"]; lm_head and vision/audio embedder projections kept in bf16 (same 17-entry ignore list as the official QAT release, spelled so both HF and vLLM module names match)
processor_config.json included (vLLM's multimodal processor init requires it)

Provenance

Base: google/gemma-4-12B-it @ 5926caa4 (bf16, untouched — no finetuning)
Quantized 2026-07-04 on a single RTX 5090; outputs verified coherent against bf16 generations
Gemma is provided under and subject to the Gemma Terms of Use: https://ai.google.dev/gemma/terms