GPTQ (error-compensated weight quantization) (W4A16) quantization of Qwen/Qwen3-Coder-Next, made with llm-compressor in the compressed-tensors format for efficient serving with vLLM and SGLang.

Chinese version: README_zh.md

Qwen publishes only an FP8 checkpoint for this model. This repo is the 4-bit (W4A16) build: ~40 GiB on disk versus ~159 GiB for the bf16 master (≈4× smaller) and about half the size of the FP8 release. That brings the 80B-A3B coder onto a single 24–48 GB GPU while the MoE routing is left untouched (see the protected layers below), so generation tracks the base model.

Weights only are quantized — activations stay 16-bit (W4A16), which is what keeps quality close to the original on a sparse MoE where the wrong rounding on a router would cascade.

GPTQ quantizes weights column-by-column and compensates the residual error into the not-yet-quantized columns using second-order (Hessian) information. It was calibrated on a 256-sample mixed code / English / Chinese corpus (2048 tokens each). On a 512-expert MoE the routing tensors are the fragile part, so they are kept at full precision:

Perplexity on wikitext-2-raw (test), n_ctx = 512, 12 chunks, measured with vLLM — the same recipe across all three, so the numbers are directly comparable. Lower is better; the bf16 row is the ceiling.

The quantization scheme is declared in config.json and detected automatically — no extra flags. Works with any engine that reads compressed-tensors (vLLM, SGLang). At ~40 GiB a single 48 GB GPU serves it comfortably; on 24 GB use tensor/pipeline parallel or offload.

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant