Intel/

Qwen3.8-27B-bpw2.8-AutoRound

This example demonstrates how to use AutoScheme in AutoRound to automatically generate a mixed-bit recipe.

This is a 2.8-bits-per-weight (averaged over quantized layers) mixed-bit weight-only quantized version of Qwen/Qwen3.8-27B, generated with AutoRound. The bits overhead of scale and zero points are included.

Please note that the 2-bit and 3-bit kernels are currently not as efficient as the 4-bit and 8-bit kernels.

This model requires vLLM PR #52890.

Until the PR is available in your installed vLLM release, install vLLM from the PR branch:

Serve the model:

Example request:

"We need answer user: "Summarize mixed-precision MoE quantization in one sentence." Need concise one sentence. Need think. Mixed-precision MoE quantization likely refers to quantizing mixture-of-experts models with different precision for different parts, e.g., weights, activations, experts, or different precision per layer/expert to balance accuracy and efficiency. Need one sentence. Could say: "Mixed-precision MoE quantization assigns different bit precisions to different parts of a mixture-of-experts model (e.g., weights, activations, or individual experts) to reduce memory and compute while preserving accuracy." That's one sentence. Need final only. Ensure no extra.\n\n\nMixed-precision MoE quantization assigns different bit precisions to different parts of a mixture-of-experts model—such as weights, activations, or individual experts—to reduce memory and compute while preserving

AutoRound>0.14.2 is required

You could remove AR_AUTO_SCHEME_NSAMPLES=128 AR_AUTO_SCHEME_SEQLEN=512 if it's too slow. The default values are 16/256

v2 version

v1 version

Quantization can change model behavior and may reduce accuracy on some tasks. This checkpoint is optimized for experiments with mixed INT2/INT4 MoE inference and requires a vLLM build with compressed-tensors WNA16 MoE Humming support. Users should run task-specific quality and safety evaluations before deployment.

The model may produce factually incorrect, biased, unsafe, or otherwise undesirable outputs. Downstream users are responsible for validating the model for their use case and for following the base model license and applicable policies.