AWQ 4-bit (W4A16) quantization of interpolators/gemma-4-12B-coder-fable5-composer2.5-v1-bf16 — a Composer-2.5 + Fable-5 verified-Python-CoT coding/reasoning fine-tune of Gemma-4-12B (gemma4_unified: a dense Gemma-4 text tower, sliding + full attention, with vestigial vision/audio towers that this fine-tune does not use).

Variant: AWQ W4A16 — 4-bit symmetric integer weights, group size 128, with activation-aware scaling. Activations stay BF16. Disk size: ~{SIZE} GB (vs ~24 GB BF16) Quantized by: sahilchachra Tooling: llm-compressor 0.12 (AWQModifier + QuantizationModifier) → compressed-tensors pack-quantized

What is quantized

Quantized to int4 (the language tower's standard linears, ~328 modules):

self_attn.{q,k,o}_proj on all 48 layers + self_attn.v_proj on the 40 non-KV-shared layers
mlp.{gate,up,down}_proj on all 48 layers

Kept in BF16: the vision/audio embedding projections (embed_vision.*, embed_audio.*), token embeddings, lm_head, and all norms (incl. q/k/v_norm).

Calibration

This is a Composer-2.5/Fable-5 verified Python chain-of-thought distillation, so it was calibrated on {N} sequences × {SEQ} tokens of code + reasoning transcripts (Claude-Code/ Fable agentic traces) rendered through the model's chat template — representative of its real coding/reasoning inference distribution.

Loading / serving

Important: gemma4_unified is a new architecture (built with transformers 5.12.1). Load with transformers ≥ 5.12.1 (trust_remote_code=True), or a vLLM build that supports gemma4 — stock vLLM 0.19 (transformers < 5) does not load it yet.

python

import torch
from transformers import AutoTokenizer, Gemma4UnifiedForConditionalGeneration
repo = "sahilchachra/gemma-4-12B-coder-fable5-composer2.5-AWQ"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = Gemma4UnifiedForConditionalGeneration.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True).eval()
msgs = [{"role": "user", "content": "Write a Python function to check if a number is prime."}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
print(tok.decode(model.generate(**enc, max_new_tokens=256)[0]))

Recommended sampling (from the base card): temperature 1.0, top_p 0.95, top_k 64; greedy for deterministic code. Gemma "thinking" channel is on by default (enable_thinking=true).

Loader note: compressed-tensors == 0.17.1 has a bug in its eager decompress_model path (mis-reads group_size) affecting all pack-quantized checkpoints — use a compressed-tensors version where this is fixed, or a runtime (vLLM) whose loader is unaffected. The weights/scales on disk are standard and correct.

Notes

Inherits the base model's coding/reasoning focus, reduced refusals (not safety-aligned), and 256K context. English-centric; general knowledge should be verified.
Effectively text-only (the vision/audio towers are vestigial and kept BF16).
Format compressed-tensors pack-quantized; symmetric int4 for broad loadability.

Original model

See interpolators/gemma-4-12B-coder-fable5-composer2.5-v1-bf16 and base google/gemma-4-12B-it for architecture, capabilities, intended use, and the Gemma Terms of Use license.