Size: 18 GB (BF16) → 8.1 GB on disk
Scheme: AWQ, W4A16 asymmetric, group size 128, compressed-tensors / pack-quantized
Calibration: 256 samples × 2048 tokens from ise-uiuc/Magicoder-Evol-Instruct-110K (code-domain calibration), via the model's chat template
Kept in BF16 (ignore): vision tower (model.visual.*), lm_head, MTP head (mtp.fc), linear-attention gating projections (linear_attn.in_proj_a/b) — following the layer recipe validated in cyankiwi/OmniCoder-9B-AWQ-4bit. The untied 248k-vocab embeddings and lm_head dominate the remaining BF16 weight.

The exact reproducible recipe is in recipe.yaml.

Usage (vLLM)

vllm serve Ar4ikov/OmniCoder-9B-AWQ-W4A16-ASYM

vLLM executes the packed INT4 weights natively: expect roughly 9–10 GB of weight VRAM plus KV cache — fits a single 24 GB GPU (RTX 3090/4090) with room for context.

Note: loading with plain transformers decompresses weights back to BF16 in memory; the memory savings materialize in vLLM (or other compressed-tensors-aware runtimes).

Quantization code

Produced with AWQModifier(targets=["Linear"], scheme="W4A16_ASYM", ignore=[...]) and model.save_pretrained(..., save_compressed=True); see recipe.yaml for the full config.