INT4 (W4A16, group_size=32) quantization of llmfan46/gemma-4-26B-A4B-it-ultra-uncensored-heretic produced with llm-compressor AWQ activation-aware scaling.

Loads in vLLM 0.20+ via the Marlin INT4 kernel (see compressed_tensors_moe_wna16_marlin).

Why AWQ?

The NVFP4 sibling model uses Blackwell's FP4 tensor cores (fastest at very high concurrency). AWQ uses Marlin INT4 kernels and is:

Faster at low-to-mid concurrency on RTX PRO 6000 / H100 / A100 / A6000 (~+39% throughput at u=8 vs NVFP4)
More portable: runs on any CUDA card from Ampere onwards (NVFP4 is Blackwell-only)
Slightly higher quality (W4A16 keeps activations in BF16; NVFP4 is W4A4)

Recipe

Method: AWQ activation-aware scaling + W4A16_ASYM quantization (AWQModifier from llmcompressor main, custom Gemma-4 mappings)
Group size: 32 (matches expert dims 704/2112)
Calibration: 512 samples × 2048 max_seq from neuralmagic/calibration (LLM split)
Ignored layers (kept BF16): dense mlp.{gate,up,down}_proj per layer, all router.proj, vision_tower.*, embed_vision.*, lm_head. Same recipe shape as cyankiwi/gemma-4-26B-A4B-it-AWQ-4bit applied to the heretic source.
Hardware: 1× RTX PRO 6000 Blackwell, ~3 hr quant time

Serving

bash

vllm serve louismuk/gemma-4-26B-A4B-heretic-AWQ \
  --max-model-len 131072 \
  --max-num-seqs 24 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --reasoning-parser gemma4 \
  --enable-auto-tool-choice --tool-call-parser gemma4 \
  --hf-overrides '{"architectures":["Gemma4ForCausalLM"]}'

License

Apache 2.0, matching the base model.