AWQ 4-bit quantization of cerebras/Gemma-4-21B-REAP-A4B-it — Cerebras's REAP-pruned variant of Gemma 4 26B (drops 32 of 128 experts → 96, ~21B total / 3B active) — calibrated to preserve thinking, vision, and video on consumer-class GPUs. Validated on NVIDIA RTX 3090 (Ampere sm_86, 24 GB) with SGLang v0.5.11 + 16 patches.

Model details


Base model	`cerebras/Gemma-4-21B-REAP-A4B-it` (REAP from `google/gemma-4-26b-a4b-it`)
Architecture	`Gemma4ForConditionalGeneration` — MoE (96 experts, top-8) + SigLIP vision tower + audio tower
Parameters	~21B total / ~3B active
Quantization	AWQ 4-bit, GPTQ-via-llmcompressor, group_size=128
Calibration recipe	`balanced_thinking_vision` (40% thinking + 60% non-thinking + 25% LLaVA images), regex `ignore` covering `vision_tower` / `embed_vision` / `multi_modal_projector`
Audit	0/11725 zero/NaN scales, 0/11725 zero/NaN qweight (`scripts/eval/check_awq_scales.py`)
File layout	`model-00001-of-00001.safetensors` (LM weights) + `model-vision.safetensors` (333 vision tensors retained)

Validation (2x RTX 3090, TP=1 / 4K, SGLang v0.5.11 + patches 023 + 028)

4/4 PASS in 12.0s via scripts/eval/validate_capabilities.py:

Check	Result
Basic	`finish=stop answer='paris'`
Thinking	structured `<\|channel>thought` channel + correct arithmetic
Vision	`'a solid red circle with a black outline on a white background.'` (content-aware)
Video	`'a red circle moves in a clockwise direction.'` (content-aware temporal)

Two SGLang patches are required for this checkpoint to load and serve correctly:

Patch 028 — gemma4_mm per-expert AWQ loader (maps model.layers.N.mlp.experts.E.{gate,up,down}_proj.{qweight,scales,qzeros} correctly when checkpoint stores per-expert keys).
Patch 023 — detects whether the dense MLP is quantized from quantization_config.ignore rather than hardcoding quant_config=None for dense MLP on MoE layers.

Without both, you'll see either KeyError on per-expert keys or BF16 placeholders that never match the AWQ checkpoint.

Recommended runtime

bash

# config.json carries architectures=Gemma4ForConditionalGeneration (multimodal route) by default.
git clone https://github.com/mattbucci/2x-3090-GA102-300-A1-sglang-inference
cd 2x-3090-GA102-300-A1-sglang-inference
./scripts/setup.sh                           # clones SGLang v0.5.11 + applies 16 patches
MODEL=mattbucci/gemma-4-21B-REAP-AWQ ./scripts/launch.sh gemma4

The gemma4 preset already bakes the working flag combo for Ampere sm_86 (head_dim=256 + FP8 KV incompat):

--attention-backend triton (FlashInfer rejects head_dim=256)
--kv-cache-dtype auto (FP8 E4M3 KV incompat with triton on sm_86)
--dtype bfloat16 (SigLIP vision tower NaNs in FP16)
--disable-cuda-graph --disable-piecewise-cuda-graph (trades cold-launch reliability for ~5-10% TPOT)
--enable-multimodal --reasoning-parser gemma4

Why REAP-21B over the 26B base

REAP (Cerebras) prunes the 32 lowest-impact experts from the 128-expert original, dropping the weight footprint from ~5.6 GB at AWQ-4bit (26B) to ~4.4 GB (21B) while retaining most of the multimodal capability. The smaller active-param fraction (3B vs 4B) also leaves more KV pool headroom on a 24 GB 3090 at long context.

For thinking + vision the 26B base remains slightly stronger; for tighter VRAM budgets or higher-throughput inference the 21B REAP variant is the right pick. Both are content-aware on the validator's red-circle / moving-red-circle probes.

Known limitations

Cross-stack RDNA4 cross-validation pending. R9700 (gfx1201, ROCm 7.2) port of patch 023 not yet landed; cross-validation will run after their v0.5.11 env upgrade. Until then this is Ampere-only verified.
v2 build (-thinking-vision-v2) is a calibration disaster — 164 all-zero scale tensors in the vision tower + MoE experts (silent calibration failure when the ignore list used bare strings instead of regex). The shipped checkpoint is the v3b corrected build; do not pull older -v2 directories from sister repos.
The 26B base supports an audio modality; this REAP-21B variant inherits whatever survived REAP pruning of the audio tower. Validated on text/image/video; audio not exercised in the 4/4 sweep.

Hardware

Validated on 2x NVIDIA RTX 3090 (GA102-300-A1, Ampere sm_86, 24 GB GDDR6X each) with CUDA 13.2 / driver 595.58 / PyTorch 2.11.0 + cu130 / SGLang v0.5.11 + 16 local patches. Fits comfortably on a single 3090 at TP=1 / 4K.

Citation

If you use this checkpoint, please cite the REAP authors and Google's Gemma 4 release:

Cerebras REAP — github.com/CerebrasResearch/reap
Gemma 4 — ai.google.dev/gemma/docs/gemma4