Qwen3_5MoeForConditionalGeneration, Qwen3.6
family, with hybrid linear / full attention). Per the base model card it is text-only and aimed at
reasoning, agentic tool-use, and coding (see Capabilities).llm-compressor AWQ (oneshot) — activation-aware, calibrated on general instruct chat (UltraChat-200k)Note on what is quantized: only the linear weights that hold the bulk of the parameters are taken to int4 — the 256-way routed experts, the shared experts, and the full-attention projections. The linear/Gated-Delta-Net (mamba-style) layers, the MoE routers, embeddings,lm_head, the MTP head and all norms are kept in BF16 for stability. The architecture also carries a vision tower (Qwen3_5MoeForConditionalGeneration), which is likewise kept in BF16 — but the base model is documented as text-only, so this quantization neither adds nor validates any image capability. The headline variant name reflects the dominant (expert/attention) quantization; the on-disk size averages the int4 and BF16 halves of the model.
<think>…</think> chains-of-thought.<tool_use> XML blocks for file/shell operations (activates with
agent-style system prompts or prior <tool_result> turns).q/k/v/o) with the routers, linear-attention, vision, MTP and norms left in BF16. This
is a functional smoke test only — it is not a quality benchmark.| Quantized → int4 (AWQ W4A16) | Kept in BF16 |
|---|---|
Routed experts (mlp.experts.*.{gate,up,down}_proj, 40 layers × 256 experts) | Linear / Gated-Delta-Net layers (*.linear_attn.*) |
Shared experts (mlp.shared_expert.{gate,up,down}_proj) | MoE routers (mlp.gate), shared-expert gates |
Full-attention projections (self_attn.{q,k,v,o}_proj) | Embeddings, lm_head, MTP head, all norms |
Vision tower (model.visual.*) — present in the arch, unused for text |
1from vllm import LLM, SamplingParams
2
3llm = LLM(model="sahilchachra/Qwable-v1-AWQ", dtype="bfloat16", max_model_len=16384, trust_remote_code=True)
4out = llm.generate(["Hello!"], SamplingParams(temperature=0.7, top_p=0.9, max_tokens=128))
5print(out[0].outputs[0].text)compressed-tensors W4A16 support (vLLM unpacks the int4 weights for you).pack-quantized (compressed-tensors), per-expert layout — the standard layout vLLM consumes
for quantized MoE.compressed-tensors and a recent transformers (the qwen3_5_moe architecture).