Extended Calibration (EC) INT4 AutoRound quantization of Qwen/Qwen3.6-35B-A3B, a 35B MoE (3B active, 128 experts) multimodal model. Drop-in replacement for Intel/Qwen3.6-35B-A3B-int4-AutoRound with wider calibration settings for improved quality on long-context and reasoning-heavy workloads.

Calibration — Extended vs Intel default

	Intel (v0.13.0)	EC (this model)
`iters`	200 (default)	400
`nsamples`	128 (default)	256
`seqlen`	512 (default)	4096
`batch_size`	8 (default)	8 (default)
`grad_accum`	1 (default)	1 (default)
`ignore_layers`	`mtp.fc`	`mtp.fc`
`bits / group`	4 / 128	4 / 128
`sym`	true	true
`packing_format`	auto_round:auto_gptq	auto_round:auto_gptq

Intel does not publish their iters/nsamples/seqlen in the released artifact or README. Our EC values are 2× Intel's CLI defaults on iters and nsamples, and 8× on seqlen — targeting better activation coverage for long-context and multimodal workloads.

Environment

Component	Version
auto-round	0.13.0 (merge commit `2dda035` — PR #1705)
transformers	5.5.1 (matches Intel's exact pin for this variant)
torch	2.6.0+cu124
safetensors	0.7.0
huggingface_hub	1.11.0
Hardware	RunPod H200 SXM (1× 141 GB HBM3e)

Reproducible invocation

bash

# Pin exact auto-round version used (merge SHA, never a branch name)
pip install 'setuptools>=76,<81' 'setuptools_scm<8' 'packaging>=24.2'
pip install -U 'torch==2.6.0' 'torchvision==0.21.0' --index-url https://download.pytorch.org/whl/cu124
pip install 'transformers==5.5.1' safetensors huggingface_hub
pip install 'git+https://github.com/intel/auto-round.git@2dda035b275a297464565ba8d4d2cc24ae6a07a9'

# Quantize (EC params)
auto-round "Qwen/Qwen3.6-35B-A3B" \
    --output_dir "./Qwen3.6-35B-A3B-int4-EC" \
    --ignore_layers mtp.fc \
    --iters 400 \
    --nsamples 256 \
    --seqlen 4096

No gptqmodel install needed — auto-round 0.13 has GPTQ packing built in.

Architecture

Layers: 40 transformer blocks (model.language_model.layers.0 through model.language_model.layers.39)
MoE: 128 experts, 3B active per token (model.language_model.layers.*.mlp.experts)
Attention: hybrid — layers [3, 7, 11, 15, 19, 23, 27, 31, 35, 39] are self_attn, rest are linear_attn (DeltaNet)
MTP: 1 multi-token-prediction head (mtp.layers.0, mtp.fc, mtp.norm)
Visual: 27 transformer blocks (model.visual.blocks.0 through model.visual.blocks.26) + patch embed + merger

Files

Path	What
`model-000{01..10}-of-00010.*`	Quantized language-model shards (INT4 GPTQ, w4g128)
`model_extra_tensors.safetensors`	Visual encoder + MTP + ignored-layer gates (WOQ INT4 / BF16 passthrough)
`config.json`	Multimodal config with embedded `quantization_config`
`quantization_config.json`	Standalone quant config (redundant with config.json)
`model.safetensors.index.json`	Weight-map → shard
`generation_config.json`	Default generation params
`preprocessor_config.json`	Vision preprocessor config
`processor_config.json`	Multimodal processor config
`chat_template.jinja`	Qwen3 chat template (reasoning-enabled)
`tokenizer.json` / `tokenizer_config.json`	Tokenizer

Total size: ~20.9 GB (10 shards ≈ 19.6 GB + extras ≈ 1.35 GB) — Intel-layout-matched (1.00× ratio vs Intel/Qwen3.6-35B-A3B-int4-AutoRound).

Layout note — Intel-layout-matched (after in-place fix)

Earlier builds of this repo had a ~21 GB model_extra_tensors.safetensors because auto-round 0.13 (PR #1705) auto-expanded block_name_to_quantize to include model.visual.blocks — the visual encoder ended up INT4'd inside extras, and several main-shard keys were duplicated into extras by missing_tensors.py. That broke vLLM's qwen3_vl.py loader, which expects visual weights to be BF16.

We rebuilt the repo in place (no re-quant): main shards untouched; model_extra_tensors.safetensors now contains BF16 visual tensors pulled from the base model (Qwen/Qwen3.6-35B-A3B, 167 tensors) plus the MTP head. Duplicated main-shard keys were dropped from the index. Net: ~20.9 GB total (1.00× Intel's reference size), no INT4 visual, vLLM loads cleanly.

MTP handling (project policy: RTN)

This repo's model_extra_tensors.safetensors contains INT4 MTP tensors (~2,335 per-expert weights under mtp.layers.0.*) produced via auto-round's native missing_tensors.py::copy_missing_tensors_from_source post-save path. The quantization method is deterministic symmetric RTN (group_size=128, no calibration data), which is the standard auto-round behavior for Qwen3.6 MoE models — and the same path Intel's published reference (Intel/Qwen3.6-35B-A3B-int4-AutoRound) uses.

Why RTN and not calibrated EC like our main LM blocks:

transformers>=5.5.1 loads Qwen3_5MoeForConditionalGeneration with _keys_to_ignore_on_load_unexpected = [r"^mtp.*"] at two locations (modeling_qwen3_5_moe.py L881, L1902), silently stripping MTP tensors during load. The instantiated model has no mtp submodules for auto-round's calibrated block iteration to discover.
We tried exposing MTP via a custom extension (roles/quant/code/qwen36_mtp_extension.py::attach_mtp) on 2026-04-23. The infrastructure worked (auto-round discovered mtp.layers.0 as a block family with 0 missing weights) but naive forward-hook wiring produced NaN calibration loss — mtp takes concat(embed(next_token), model.norm(hidden_states)) → mtp.fc → mtp.layers[0], not raw hidden_states, so the calibration search diverged.
Shipping calibrated MTP with a broken forward pipeline would be strictly worse than RTN (effectively random rounding). Proper EC requires implementing the full MTP forward; deferred as a future research task (see docs/quant-new-model-checklist.md §10.4 policy C).

Practical implication: the 40 main LM blocks use our EC calibration (iters=400, nsamples=256, seqlen=4096). The 1 MTP block uses RTN. For speculative-decoding workloads where MTP acceptance rate matters, this is the same baseline as Intel's reference. For non-spec-dec deployments (most vLLM serving), MTP is not loaded and this choice is invisible.

License

Apache 2.0 (inherits from Qwen/Qwen3.6-35B-A3B).

Shoutouts

Qwen team for the base Qwen3.6-35B-A3B model.
Intel for the reference AutoRound INT4 recipe and post-quant checkpoint layout we built on. Special thanks to @lvliang-intel and @wenhuach21 for PR #1705 which added Qwen3.6 support.
auto-round for the quantization tooling.

shieldstar/Qwen3.6-35B-A3B-int4-AutoRound-EC

Qwen3.6-35B-A3B-int4-AutoRound-EC