This is a calibrated W4A16 (INT4 weight-only) quantization of Mitchins/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic, a QAT/Heretic tuned Gemma 4 26B-A4B instruction checkpoint. It is not a stock, bit-identical conversion of the ordinary public Google IT checkpoint. The model remains a Gemma 4 26B-A4B instruction/multimodal model; useful-quants performed the W4A16 conversion and packaging, not the original model development or the parent tuning.
The language-model linear and MoE expert weights are packed with compressed-tensors W4A16/G64; activations remain BF16. The vision tower, routers, embeddings, and LM head remain BF16/unquantized. The packed payload is 14.58 GiB (model.safetensors); the full repository is approximately 14.61 GiB including tokenizer and processor files.
Text generation and real-image inference were validated on one RTX 3090 24 GB with vLLM, Ampere Marlin dense/MoE kernels, and CUDA graphs. A 131,072-token total sequence window (128K) was also validated on that reference system.
G64 is deliberate: the relevant Gemma 4 26B-A4B projection dimensions (intermediate_size=2112, moe_intermediate_size=704) divide cleanly by 64 but not by 128.
The immediate parent is Mitchins/gemma-4-26B-A4B-it-qat-q4_0-unquantized-uncensored-heretic, revision d62fa836581401beafd08fdd624927e31dcac6ee. That parent is an uncensored/abliterated Heretic derivative of the QAT-trained google/gemma-4-26B-A4B-it-qat-q4_0-unquantized lineage, itself based on Google Gemma 4 26B-A4B-IT. The local export was frozen before conversion, including source-file SHA256 hashes, tokenizer, processor, chat template, generation configuration, and architecture configuration. See provenance/calibration-and-source.json.
The parent model's Apache-2.0 license and Gemma 4 license link apply. This quantization does not change the parent model's intended use, safety guidance, or license terms.
What is quantized
The saved quantization_config is the source of truth:
Linear language-model and MoE expert weights: 4-bit signed integer weights, W4A16, group-wise G64, symmetric, static activation ordering, memoryless_minmax observer, pack-quantized storage.
Dense language-model linears and MoE experts are W4A16. The artifact audit found the expected packed expert tensors and finite weight-scale tensors.
The vision tower is retained BF16.
Routers and token/image embeddings are retained BF16/unquantized. This Gemma export has no separate lm_head tensor; output weights are tied through the BF16 token embedding.
No reduced-precision KV-cache profile is claimed or recommended here.
The GPTQ modifier also used a 128-token block size and dampening_frac=0.01; the complete public-safe recipe is in provenance/calibration-and-source.json.
Calibration
Calibration was multimodal rather than text-only so the conversion observed both normal language traffic and the image-processing/template path.
192 windows from Salesforce/wikitext, wikitext-103-raw-v1 train.
192 from HuggingFaceTB/smoltalk, openhermes-100k.
128 from HuggingFaceTB/smoltalk, apigen-80k.
Each source is deterministically shuffled; a valid 512-token span is sampled from each selected row. Conversational rows are rendered as role-labelled text before tokenization.
Images: 1,024 deterministically selected, SHA256-verified images from COCO 2017 validation. The source processor converts each to RGB and applies its native preprocessing.
Mixed layout: the first 512 calibration samples pair one frozen text window with one image; the remaining 512 use image-instruction prompts. Every selected COCO image and every frozen text window is used once.
Calibration sequence cap: 1,024 tokens, allowing the 512-token text windows and Gemma image soft tokens to coexist.
The exact datasets, sample counts, seed, selection method, image hash-manifest digest, exclusions, and software versions are in provenance/calibration-and-source.json. Calibration media are deliberately not included in this release.
Quality: held-out perplexity
Model
Perplexity
Delta vs. BF16
Tuned QAT/Heretic BF16 parent
388.150
—
W4A16-G64
432.318
+44.168 / +11.38%
The result uses held-out Salesforce/wikitextwikitext-2-raw-v1test text, which was not used for calibration. Both rows use the same frozen tokenizer, four contiguous non-empty 512-token test-corpus windows (2,044 scored next-token positions), next-token NLL arithmetic, and vLLM prompt_logprobs=1 scorer. BF16 is executed with vLLM CPU offload because the parent checkpoint exceeds a 24 GB GPU. See eval/evaluate_gemma4_ppl_vllm.py and benchmarks/perplexity.json.
This is a compact quantization sanity evaluation, not a replacement for task-specific evaluation of the tuned parent.
Verified functional gates
The final artifact was fresh-reloaded and passed:
deterministic short text generation;
real COCO image generation through the Gemma 4 processor/template path;
Marlin W4A16 dense-linear selection;
Marlin W4A16 MoE selection;
CUDA graph capture, without eager mode;
finite weight-scale and packed-tensor audit.
For images, use the Gemma 4 processor or OpenAI-compatible multimodal request format. The source chat template emits Gemma's actual <|image|> placeholder semantics; do not manually write a raw <image> token into a text prompt.
Gemma 4 supports video in its broader architecture, but video was neither required nor validated for this image-focused release profile. Unrestricted renderer warmup needlessly exercises a video path, so the validated image-serving profile explicitly uses:
--limit-mm-per-prompt '{"image":1,"video":0}'
This is a runtime warmup/configuration consideration, not a text or image generation failure.
Peak observed VRAM in the full-checkpoint performance test was 23,914 MiB; the benchmark conditions are below.
RTX 3090 24 GB performance
Reference setup: physical CUDA:1 RTX 3090, vLLM main commit 7ca49fbe4bab019e55d57cdc4b7fd3d55c67c1a6, CUDA graphs enabled, max_model_len=1024, max_num_seqs=32, max_num_batched_tokens=8192, prefix caching disabled, three repetitions per point, and fixed 128-token greedy ignore_eos outputs. Values are aggregate verified output tokens/s, not per-request rate.
Active requests
Text aggregate output tok/s
One image/request aggregate output tok/s
1
156.6
127.9
4
486.9
262.6
8
820.4
429.9
16
1,269.7
518.5
32
1,948.3
461.6
The text test is decode-dominant. The one-image/request test includes image processing and multimodal prefill, so it is not directly comparable to the text-only decode number. B32 maximized measured text aggregate throughput. B16 is the measured image-workload knee; B32 was slower for that workload because it exceeds the efficient 8,192-token multimodal batching envelope. vLLM profiling allocated encoder cache for up to 29 maximum-size images in this configuration. Raw public-safe measurements: benchmarks/rtx3090-performance.json.
Long-context validation
Gemma 4 advertises an architectural maximum of 262,144 tokens. That is not the same as what this 24 GB reference deployment can serve.
The validated RTX 3090 operational maximum is 131,072 total tokens (128K). A near-limit request with 130,816 input tokens plus 64 generated tokens completed with CUDA graphs, Marlin W4A16 dense/MoE paths, no OOM, and no allocator warning.
Validated fixed KV reservation: 6444425401 bytes.
Observed cache capacity: 135083 tokens.
Margin above the configured 128K window: 4011 tokens (about 3%).
A conventional 132K setting does not fit this fixed cache; 128K is the recommended validated ceiling for this reference configuration.
max_model_len includes generation: input_tokens + max_tokens <= 131072. For example, with a 2,048-token generation budget, the maximum input is 129,024 tokens. The exact fixed reservation is hardware- and runtime-specific; do not assume every 24 GB GPU has the same headroom.
Recommended serving configurations
A. Mixed agentic/search and batch service
This is the validated 128K RTX 3090 reference profile:
Do not combine this exact fixed-KV profile with --gpu-memory-utilization: explicit cache bytes are authoritative. The byte value is a validated RTX 3090 reference setting, not a universal default.
Admission policy matters as much as priority:
Set max_num_seqs=17.
Cap bulk ingress at 16 active requests, normally with priority: 100.
Submit interactive/Hermes/search traffic with priority: -10.
The free seventeenth slot lets the interactive request enter. Once admitted, priority scheduling can preempt/requeue lower-priority cache users if the long request requires cache. Priority alone should not be expected to create an admission slot when every sequence slot is occupied.
Example OpenAI-compatible high-priority request, reserving a 2K response budget:
For independent text-only jobs, use up to 32 active requests. The measured B32 point reached 1,948.3 aggregate verified output tok/s under the benchmark conditions above. This is a throughput profile, not the long-context mixed-service reservation.
C. Image-heavy batch profile
For one image per request, B16 is the measured knee (518.5 aggregate output tok/s). Retain the 8,192-token batching envelope and disable video unless it is explicitly required:
main commit 7ca49fbe4bab019e55d57cdc4b7fd3d55c67c1a6 (0.1.dev1+g7ca49fbe4)
Transformers
5.15.1
PyTorch
2.13.0+cu130
CUDA runtime
13.0
compressed-tensors runtime
0.17.0
Conversion artifact metadata
compressed-tensors 0.18.0
vLLM 0.27.1 did not contain the necessary handling for Transformers-5.15-era heterogeneous Gemma 4 attention metadata. The artifact was not the cause. Use the pinned commit above or a later vLLM release that explicitly includes the corresponding Gemma 4 heterogeneous-config support. Transformers-native inference is not claimed as validated for this packed artifact; vLLM is the supported runtime documented here.
Image usage
Use an OpenAI-compatible multimodal request and let vLLM/the model processor apply Gemma's chat template:
python
1from openai import OpenAI
23client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")4response = client.chat.completions.create(5 model="useful-quants/Gemma-4-26B-A4B-it-W4A16-G64-BF16Vision",6 messages=[{7"role":"user",8"content":[9{"type":"image_url","image_url":{"url":"https://example.org/image.jpg"}},10{"type":"text","text":"Describe this image accurately and concisely."},11],12}],13 max_tokens=256,14)15print(response.choices[0].message.content)
Do not hand-insert raw image-placeholder text; the Gemma 4 processor/template is responsible for the image placement.
Limitations and caveats
Performance and KV-cache figures are measurements from one RTX 3090 24 GB, not universal guarantees.
The 128K claim means a validated 131,072-token total window, including output; it does not make 262K feasible on this reference card.
Video was intentionally disabled and not validated for this image-focused profile.
Image throughput includes processor and prefill work; do not compare it directly with text decode throughput.
The long-context reference profile retained the quality-tested BF16/auto KV-cache behavior. Reduced-precision KV cache has not been validated here.
This is a quantization of the linked QAT/Heretic parent, not a stock-Google checkpoint conversion.
Reproducibility
Public-safe provenance, calibration, artifact-audit, benchmark, and perplexity files are included in this repository:
Please cite the QAT/Heretic parent model card, the original Gemma 4 model card, and technical report for their respective methods and architecture. Google DeepMind developed Gemma; useful-quants packaged this W4A16 artifact from the QAT/Heretic parent checkpoint.
This repository inherits the parent Apache-2.0 license; see LICENSE and the Gemma 4 license.
