
Weights	INT4, per-group, group size 128, symmetric, static
Activations	not quantized (BF16)
Algorithm	AWQ (Activation-aware Weight Quantization)
Quark scheme	`int4_wo_128`
Expert path	unfused
Export	HF `safetensors`, `real_quantized`
Size	24 GB (from 67 GB BF16)

What is quantized

Quark's built-in qwen3_5_moe recipe quantizes the routed MoE experts only — 30,720 linears (40 layers x 256 experts x 3 projections). 462 modules are excluded and remain BF16:

lm_head, model.visual.* (vision tower), mtp.*, *mlp.gate, *shared_expert_gate*, *.linear_attn.*, *.self_attn.*, *.shared_expert.*

Because attention, the shared experts, the router and the vision tower all stay BF16, the size reduction is smaller than a whole-model quantization would give.

Reproduction

bash

python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
  --model_dir Qwen/Qwen3.6-35B-A3B \
  --device cuda --data_type bfloat16 --model_attn_implementation eager \
  --dataset pileval --num_calib_data 128 --seq_len 512 --batch_size 1 \
  --quant_scheme int4_wo_128 \
  --quant_algo awq --quant_algo_config_file awq <config.json> \
  --model_export hf_format --export_weight_format real_quantized \
  --output_dir <out> --skip_evaluation

Notes

Quantized on a single AMD Instinct MI355X (gfx950) with Quark 0.13+50dac6e159a.
Calibration: pileval (mit-han-lab/pile-val-backup), 128 samples, seq_len=512.
The vision tower is not quantized, so this remains a full vision-language model.
GGUF export is not possible for this architecture: Quark's GGUF writer maps only MODEL_ARCH.LLAMA and supports asymmetric INT4 per-group-32 weight-only on Llama2/Llama3, while this model is Qwen3_5MoeForConditionalGeneration. Safetensors is published instead.
No accuracy evaluation has been run on this checkpoint. Benchmark before use.

amd-satre/Qwen3.6-35B-A3B-w_int4_a_bf16-awq

Qwen3.6-35B-A3B-w_int4_a_bf16-awq

What is quantized

Reproduction

Notes

amd-satre/
Qwen3.6-35B-A3B-w_int4_a_bf16-awq