Model Architecture: Qwen2MoeForCausalLM
- Input: Text
- Output: Text
Supported Hardware Microarchitecture: AMD MI300 / MI350 / MI355
Inference Engine: vLLM
Model Optimizer: AMD-Quark (v0.11.2)
- Weight quantization: INT8, per-channel, Static, Symmetric
- Activation quantization: INT8, per-token, Dynamic, Symmetric
- Quantized layers: routed MoE experts only (attention, shared expert, and router/gate layers are kept in the original precision)

This is a W8A8-INT8 quantization of Qwen/Qwen1.5-MoE-A2.7B-Chat, used as vLLM CI test coverage for the Quark INT8 fused-MoE path (QuarkW8A8Int8MoEMethod).

Model Quantization

The model was quantized from Qwen/Qwen1.5-MoE-A2.7B-Chat using AMD-Quark. The routed expert weights are quantized to INT8 with per-channel static scales, and their activations are quantized to INT8 with per-token dynamic scales (symmetric).

Note: Quark quantizes nn.Linear modules. MoE experts are stored as individual nn.Linear layers in transformers ~4.57; quantize with that version so the routed experts are captured.

Quantization script (see tests/quantization/quantize_int8_moe_quark.py in vLLM):

bash

python tests/quantization/quantize_int8_moe_quark.py \
    --model Qwen/Qwen1.5-MoE-A2.7B-Chat \
    --output-dir ./Qwen1.5-MoE-A2.7B-Chat-w-int8-a-int8-sym \
    --exclude lm_head '*.self_attn.*' '*.shared_expert.*' \
              '*.shared_expert_gate' '*.mlp.gate'

Evaluation

The model was evaluated on gsm8k using the vLLM framework (the routed experts run through vLLM's Triton INT8 fused-MoE kernel).

Accuracy

Benchmark	amd/Qwen1.5-MoE-A2.7B-Chat-w-int8-a-int8-sym (this model)
gsm8k (5-shot, 1319 questions)	51.18

Reproduction

bash

lm_eval \
  --model vllm \
  --model_args pretrained=amd/Qwen1.5-MoE-A2.7B-Chat-w-int8-a-int8-sym,max_model_len=4096,gpu_memory_utilization=0.4,enforce_eager=True,trust_remote_code=True \
  --tasks gsm8k --num_fewshot 5 \
  --batch_size auto

License

This model inherits the Tongyi Qianwen license of the base model.