Architecture: Qwen3_5MoeForConditionalGeneration (multimodal: text + vision; same architecture as Qwen3.5)
Total parameters: ~35B
Activated parameters: ~3B per token (8 of 256 experts selected per token)
Layers: 40 (30 linear attention + 10 full attention, repeating 3:1 pattern)
Experts: 256 per layer + 1 shared expert per layer
Context length: 262,144 tokens
Vision encoder: 27-block ViT (1152 hidden, 16x16 patches), BF16
MTP module: 1-layer speculative decoding head, BF16

Quantization Details

All 30,720 MoE expert modules (256 experts x 3 projections x 40 layers) are quantized to INT4 using GPTQ. Non-expert modules (including the full vision encoder and MTP module) remain at BF16/FP16 for quality preservation.

Component	Precision	Notes
MoE experts (`gate_proj`, `up_proj`, `down_proj`)	INT4 (GPTQ)	30,720 modules quantized
Full attention (`q_proj`, `k_proj`, `v_proj`, `o_proj`)	FP16	Every 4th layer
Linear attention (`in_proj_qkv`, `in_proj_z`, `out_proj`)	FP16	Full precision
Shared experts	FP16	Full precision
Vision encoder (`model.visual.*`)	BF16	333 tensors, full precision
MTP module (`mtp.*`)	BF16	785 tensors, full precision
Embeddings, LM head, norms	FP16	Full precision

GPTQ configuration:

Bits: 4
Group size: 32
Symmetric: Yes
desc_act: No
true_sequential: Yes
act_group_aware: Yes
Failsafe: RTN for poorly-calibrated rare experts (1,302 of 30,720 modules, ~4.24%)

Calibration

Dataset: Mixed - evol-codealpaca-v1 (code) + C4 (general text)
Samples: 2,048
Quantizer: GPTQModel v5.7.1

Model Size

Version	Size	Compression
BF16 (original)	67 GB	-
GPTQ 8-bit	40 GB	1.7x
GPTQ 4-bit	25 GB	2.7x

Perplexity

Evaluated on wikitext-2-raw-v1 (test set), seq_len=2048, stride=512:

Model	Perplexity	Degradation
BF16 (original)	6.0567	-
GPTQ 8-bit	6.0543	-0.04% (within noise; effectively lossless)
GPTQ 4-bit	6.1070	+0.83%

Usage

vLLM (Recommended for Serving)

bash

vllm serve btbtyler09/Qwen3.6-35B-A3B-GPTQ-4bit \
  --gpu-memory-utilization 0.95 \
  --max-model-len 256000 \
  --tensor-parallel-size 4 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --dtype float16 \
  --skip-mm-profiling \
  --limit-mm-per-prompt '{"image": 2}'

Parameter	Description
`--gpu-memory-utilization 0.95`	Use 95% of GPU VRAM for KV cache + weights
`--max-model-len 256000`	Full 256K context window support
`--tensor-parallel-size 4`	Shard across 4 GPUs (adjust to your setup)
`--reasoning-parser qwen3`	Enable thinking/reasoning token parsing
`--enable-auto-tool-choice --tool-call-parser qwen3_coder`	Enable tool/function calling
`--dtype float16`	Run in FP16 (required for ROCm GPTQ kernels)
`--skip-mm-profiling`	Skip multimodal memory profiling at startup
`--limit-mm-per-prompt '{"image": 2}'`	Allow up to 2 images per request

vLLM bug workaround (may apply): Up through at least vLLM 0.15.2, Qwen3_5MoeTextConfig defines ignore_keys_at_rope_validation as a list instead of a set, causing a TypeError during config parsing. Since Qwen3.6 reuses the same qwen3_5_moe model_type, the same fix applies. Apply this patch before serving if you hit the error:

python
1python3 -c " 2for f in [ 3 '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5_moe.py', 4 '/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5.py', 5]: 6 t = open(f).read() 7 t = t.replace( 8 'ignore_keys_at_rope_validation\"] = [\n \"mrope_section\",\n \"mrope_interleaved\",\n ]', 9 'ignore_keys_at_rope_validation\"] = {\n \"mrope_section\",\n \"mrope_interleaved\",\n }') 10 open(f,'w').write(t) 11 print('Patched', f) 12"

Vision Example (via OpenAI API)

python

import base64, requests

with open("image.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "btbtyler09/Qwen3.6-35B-A3B-GPTQ-4bit",
    "messages": [{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        {"type": "text", "text": "Describe what you see in this image."},
    ]}],
    "max_tokens": 1024,
})
print(response.json()["choices"][0]["message"]["content"])

GPTQModel / transformers

Note: Neither GPTQModel nor transformers can currently load this model directly. GPTQModel's Qwen3_5MoeGPTQ class expects the text-only weight prefix (model.layers.*) and does not support the multimodal architecture (model.language_model.layers.*). The transformers GPTQ path delegates to optimum, which does not handle the fused-expert architecture. Use vLLM for inference.

Technical Notes

Qwen3.6-35B-A3B is architecturally identical to Qwen3.5-35B-A3B (same Qwen3_5MoeForConditionalGeneration class, model_type=qwen3_5_moe, 256 experts/layer, 40 layers, hybrid linear+full attention pattern). The same GPTQModel definition and expert converter handle both checkpoints without modification.

MoE expert weights are stored as fused 3D nn.Parameter tensors in the source model rather than individual nn.Linear modules. During quantization, GPTQModel's MODULE_CONVERTER_MAP converts these to individual quantizable nn.Linear layers. This same conversion must also run during model loading for the quantized kernels to be applied correctly.

The vision encoder (27-block ViT) and MTP speculative decoding module are preserved at full BF16 precision from the original model. Only the text model's MoE expert weights are quantized.

Credits

Base Model: Qwen - Qwen3.6-35B-A3B
Quantization: GPTQ via GPTQModel v5.7.1
Expert Converter: convert_qwen3_5_moe_expert_converter for fused 3D expert weights (shared with Qwen3.5)
Quantized by: btbtyler09

License

This model inherits the Apache 2.0 license from the base model.