Quantization


Method	AWQ, `W4A16`, group size 128
Format	`compressed-tensors` (`pack-quantized`)
Ignored	`lm_head`, MoE router (`mlp.gate`)
Calibration	64 samples of `HuggingFaceH4/ultrachat_200k`, seq len 1024
MoE	`moe_calibrate_all_experts=True` — every calibration token is routed through all 128 experts, so each expert is calibrated

Qwen3-Coder-30B-A3B is a Mixture-of-Experts model (128 experts, 8 active per token) using standard GQA attention — so it serves on consumer Ada GPUs with the stock attention backends, no special kernels required.

Serving with vLLM

bash

vllm serve dark-side-of-the-code/Qwen3-Coder-30B-A3B-Instruct-AWQ \
    --max-model-len 65536 \
    --kv-cache-dtype fp8 \
    --gpu-memory-utilization 0.95 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20}'

Notes:

At --max-model-len 65536 on a 24 GB card the KV pool holds ~118K tokens (~1.8x concurrency). The base model's native context is 262K; 64K is the practical cap at 24 GB.
Tool calling works via vLLM's first-class qwen3_coder parser.
Non-thinking model — no reasoning parser needed.
Pass --override-generation-config if your client doesn't send sampling params, or vLLM will default to temperature=1.0, which is far too hot for coding.

Validation

Verified after quantization: correct code generation, working tool_calls through the qwen3_coder parser, and coherent prose output.

Selected as the local agentic coding model in a head-to-head evaluation driven by the Pi coding agent, where it reliably executed a read/edit/test tool loop.