GPTQ Int4 quantization of Jackrong/Qwopus3.5-9B-v3.5 — a multimodal tool-calling agent built on the qwen3_5 hybrid architecture (3:1 GatedDeltaNet : full attention + a 27-block vision tower).


📦 Size	8.45 GB (from 17.98 GB BF16, −53%)
🎯 Text stack	200/200 modules Int4 — zero exceptions
📉 Quant error	mean 4.2e-5, worst layer 1.9e-4
🧠 Context	262,144 positions (mrope)
🔧 Tools	XML `<tool_call>` format, vLLM `qwen3_xml` parser
👁 Vision	tower kept in BF16 — image understanding intact
⚖️ License	Apache-2.0

🚀 Serve it (vLLM)

bash

vllm serve malvavisc0/Qwopus3.5-9B-v3.5-gptq-int4 \
  --quantization gptq_marlin \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml

The repo ships the full v22 chat template (chat_template.jinja) — vLLM picks it up automatically. It supports reasoning_effort, preserve_thinking, and tool_call_format (XML default, JSON optional).

On 12–16 GB cards, bound the KV cache:

--max-model-len 8192 --max-num-seqs 10

🐍 Transformers

python

from transformers import AutoModelForMultimodalLM, AutoProcessor

model = AutoModelForMultimodalLM.from_pretrained(
    "malvavisc0/Qwopus3.5-9B-v3.5-gptq-int4", device_map="auto"
)
processor = AutoProcessor.from_pretrained(
    "malvavisc0/Qwopus3.5-9B-v3.5-gptq-int4"
)

🧪 How it was made

Quality comes from what you calibrate on — not from retraining:

Format — GPTQ Int4, group_size=128, desc_act (activation ordering), symmetric. Compatible with vLLM's gptq_marlin kernel.
Calibration — 128 agentic tool-use conversations from nvidia/Nemotron-Agentic-v1 (interactive_agent), each rendered through the shipped v22 chat template at 2048 tokens. Activation statistics match real serving traffic, not bare web text.
Tooling — aft + gptqmodel 7.3.4, transformers 5.15.0, built on NVIDIA GB10 (sm_121).

Full build record in aft_provenance.json; per-layer quantization error in quant_log.csv.

🗂 What's quantized (and what isn't)

Component	Precision	Why
Text stack — attention `q/k/v/o_proj`, MLP `gate/up/down_proj`, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`	Int4	the language brain
Vision tower (`model.visual.*`, 159 modules)	BF16	text-only calibration can't represent visual activations — kept lossless deliberately

📌 Notes

Source revision: dc2b00e1 (unchanged upstream since 2026-04).
Tool calling uses the v22 XML format: <tool_call><function=name><parameter=key>value</parameter></function></tool_call>
Reasoning renders in <think> blocks; effort is controllable via reasoning_effort (xhigh / medium / low).