gemma-4-26B-A4B-it-qat-W4A16
Lossless Q4_0 GGUF → compressed-tensors pack-quantized W4A16 model for vLLM deployment.
What This Is
- Base model:
google/gemma-4-26B-A4B-it
- Source format: Q4_0 GGUF (QAT-trained, exported by llama.cpp)
- Target format: compressed-tensors
pack-quantized
- Quantization: 4-bit symmetric integer, group-wise (group_size=32)
- Scale dtype: bfloat16 (or float16 via
--dtype)
- Marlin padding: MoE and Dense MLP blocks padded to
min_thread_k=128 (Dense MLP skippable via --skip-dense-mlp)
- Vision tower: copied from
--unquantized-ref source (unquantized bfloat16)
- k_eq_v handling:
v_proj absent for full_attention layers (5, 11, 17, 23, 29)
Conversion
1python convert_gguf_q4_0_to_ct.py \
2 --gguf-path gemma-4-26B_q4_0-it.gguf \
3 --unquantized-ref gemma-4-26B-A4B-it-qat-q4_0-unquantized \
4 --output-dir gemma-4-26B-A4B-it-qat-W4A16
The conversion is lossless at the nibble level — every Q4_0 scale and quantized weight value is preserved bit-for-bit from the GGUF source.
Arguments
| Flag | Default | Description |
|---|
--gguf-path | (required) | Path to Q4_0 GGUF file |
--unquantized-ref | (required) | Unquantized bf16 model (config + tokenizer + non-Q4_0 weights) |
--output-dir | (required) | Output directory |
--dtype | bfloat16 | Dtype for unquantized tensors and scales (bfloat16 or float16) |
--group-size | 32 | CT quantization group size |
--no-marlin-pad | off | Skip Marlin K-dimension padding |
--skip-dense-mlp | off | Keep Dense MLP as unquantized bf16 (required for TP>1) |
Marlin Padding
The vLLM Marlin kernel requires in_features % 128 == 0. The script automatically pads affected FFN blocks with zero-fill rows/columns:
| Weight | Before | After |
|---|
mlp.gate_proj rows | 2112 | 2176 |
mlp.up_proj rows | 2112 | 2176 |
mlp.down_proj K | 2112 | 2176 |
experts.{e}.gate_proj rows | 704 | 768 |
experts.{e}.up_proj rows | 704 | 768 |
experts.{e}.down_proj K | 704 | 768 |
Padded regions use q=8 (q_signed=0 → zero weight). The config.json is updated with padded intermediate_size=2176 and moe_intermediate_size=768.
Tensor Parallel (TP > 1)
Dense MLP with Marlin fails with TP≥2 because 2176/2 = 1088 is still not divisible by min_thread_k=128. MoE experts work fine (768/2 = 384, 384/128 = 3).
Use --skip-dense-mlp to keep Dense MLP unquantized for TP>1 deployments (adds ~725 MiB VRAM):
1python convert_gguf_q4_0_to_ct.py \
2 --gguf-path gemma-4-26B_q4_0-it.gguf \
3 --unquantized-ref gemma-4-26B-A4B-it-qat-q4_0-unquantized \
4 --output-dir gemma-4-26B-A4B-it-qat-W4A16 \
5 --skip-dense-mlp
vLLM Serving
1vllm serve gemma-4-26B-A4B-it-qat-W4A16 \
2 --gpu-memory-utilization 0.85 \
3 --enforce-eager \
4 --trust-remote-code
Important: Use /v1/chat/completions (not /v1/completions) — Gemma4 is an instruct model that requires the chat template.
Tested with vLLM v0.22.1, CUDA compute capability 8.9 (RTX 4080 Super). Model loads at ~16.4 GiB GPU memory.
Key Design Decisions
k_eq_v (full_attention) Layers
vLLM duplicates k_proj as v_proj at runtime for full_attention layers. The checkpoint must NOT contain real v_proj tensors for these layers. The GGUF source already has no v_proj for full_attention layers (5, 11, 17, 23, 29), so this is handled naturally.
Vision Tower
GGUF does not contain vision tower weights. They are copied as unquantized bfloat16 from the --unquantized-ref source model (alongside all other non-Q4_0 tensors). Supports both single-file and sharded (index.json) models.
Non-Q4_0 Tensors
All non-quantized tensors — embed_tokens, layer norms, router weights, router.scale, layer_scalar, vision tower — are read directly from the unquantized bfloat16 source. This is simpler and more reliable than decoding F32/Q6_K tensors from GGUF.
Negative Q4_0 Scales
Q4_0 uses d = max/8 which can be negative when all values in a group are negative (~49% of groups). The CT Marlin kernel handles negative bfloat16/float16 scales correctly.
Bugs Fixed
The following issues were discovered and fixed during development:
-
F32 tensor data corruption: GGUFReader decodes F32 in HF layout, but old code did a bytes→reshape→transpose round-trip that scrambled data. Fixed by reading non-Q4_0 tensors directly from the unquantized bf16 source instead of GGUF passthrough.
-
Q6_K dequantization errors: Similar reshape issue for embed_tokens + Q6_K quantization loss (483M values differ from true bf16, max error 0.007). Fixed by reading embed_tokens from the unquantized bf16 source.
-
Config actorder: "static": Triggered Marlin g_idx permutation without providing weight_g_idx, scrambling weights at inference. Removed from config.
-
dtype consistency: Scale dtype, passthrough tensor dtype, and config dtype all follow --dtype uniformly.
Tensor Counts
| Category | Default | --skip-dense-mlp |
|---|
| Q4_0 2D (attention + Dense MLP) | 205 | 115 |
| Q4_0 MoE expert splits | 7,680 | 7,680 |
| Unquantized (embed, norms, router, vision) | 748 | 838 |
| Total | 35,923 | 35,743 |
