Model Architecture: Qwen3_5MoeForConditionalGeneration
- Input: Text, Image, Video
- Output: Text
Supported Hardware Microarchitecture: AMD MI350 / MI355
ROCm: 7.2.0
PyTorch: 2.9.1
Transformers: 5.3.0
Operating System(s): Linux
Inference Engine: SGLang
Model Optimizer: AMD-Quark (v0.12)
- Quantized layers: All MoE experts in the language model, including the shared expert (the shared expert is also fused into the MoE kernel for faster decode).
- Weight quantization: OCP MXFP4, Static
- Activation quantization: OCP MXFP4, Dynamic

This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert). Compared with keeping the shared expert in bf16, this further reduces the bf16 footprint and improves decode throughput, with no measurable accuracy loss on GSM8K (see Evaluation).

Model Quantization

The model was quantized from Qwen/Qwen3.5-397B-A17B-FP8 using AMD-Quark. Weights and activations are quantized to OCP MXFP4.

Quantization scripts:

import os
from quark.torch import LLMTemplate, ModelQuantizer

# Configuration
ckpt_path = "Qwen/Qwen3.5-397B-A17B-FP8"
output_dir = "amd/Qwen3.5-397B-A17B-MoE-MXFP4"
quant_scheme = "mxfp4"
# NOTE: shared expert is NOT excluded here, so it is quantized to MXFP4 as well.
exclude_layers = ["lm_head", "model.visual.*", "mtp.*", "*mlp.gate", "*shared_expert_gate*", "*.linear_attn.*", "*.self_attn.*"]

# Get quant config from template
template = LLMTemplate.get("qwen3_5_moe")
quant_config = template.get_config(scheme=quant_scheme, exclude_layers=exclude_layers)

# Quantize with file-to-file mode
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=ckpt_path,
    save_path=output_dir,
)

For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.

Evaluation

The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B-FP8 checkpoint, evaluated with the identical recipe on SGLang.

Accuracy

Benchmark	Qwen/Qwen3.5-397B-A17B-FP8	amd/Qwen3.5-397B-A17B-MoE-MXFP4 (this model)	Recovery
gsm8k (flexible-extract, 5-shot)	97.95	97.27	99.31%

GSM8K is essentially lossless after quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel.

Reproduction

The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe.

Serve the model with SGLang:

python3 -m sglang.launch_server \
    --model-path amd/Qwen3.5-397B-A17B-MoE-MXFP4 \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --attention-backend aiter \
    --mem-fraction-static 0.8 \
    --host 0.0.0.0 --port 30000

Run lm-evaluation-harness against the running server (chat-completions endpoint, chat template applied, 5-shot, greedy). This uses InferenceX's gsm8k.yaml, whose only change vs. the stock lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:

lm_eval --model local-chat-completions --apply_chat_template \
  --tasks gsm8k.yaml \
  --num_fewshot 5 \
  --model_args "model=amd/Qwen3.5-397B-A17B-MoE-MXFP4,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=64,tokenized_requests=False,max_length=16384" \
  --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"

amd/Qwen3.5-397B-A17B-MoE-MXFP4

Model Overview

Model Quantization

Evaluation

Accuracy

Reproduction

License

amd/
Qwen3.5-397B-A17B-MoE-MXFP4