You can deploy this model and start earning money today!

amd/
gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8

Name: gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8
Availability: InStock

Views

No views yet

amd/gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8

Introduction

This model was quantized from openai/gpt-oss-20b using AMD-Quark with calibration samples from the Pile dataset.
Quantization schemes
- Quantized Layers: All linear layers excluding *self_attn*, *router*, lm_head, i.e., only MoE MLP linear layers are quantized.
- Weight: quantized using OCP Microscaling (MX) FP4 scheme
- Activation: quantized using FP8 symmetric per-tensor scheme
- KV Cache: FP8 symmetric per-tensor
Quantization script

bash

python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
    --multi_gpu \
    --model_dir openai/gpt-oss-20b \
    --quant_scheme w_mxfp4_a_fp8 \
    --kv_cache_dtype fp8 \
    --exclude_layers "*self_attn*" "*router*" "lm_head" \
    --dataset pileval \
    --num_calib_data 128 \
    --output_dir amd/gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8 \
    --model_export hf_format \
    --skip_evaluation

Deployment

This model supports deployment through the vLLM backend. Please ensure PR#29008 has been correctly applied.

Evaluation

This model is evaluated on gpqa_diamond_generative_n_shot and gsm8k_platinum tasks using the lm-evaluation-harness framework with vllm backend.

Model name	Weight	Activation	KV cache	Exclude	gpqa_diamond_generative_n_shot (5)	gsm8k_platinum
TP1	TP2	TP4	TP8	TP1	TP2	TP4	TP8
openai/gpt-oss-20b	MXFP4	BF16	BF16	-	0.5606	0.5303	0.5657	0.5606	0.9016	0.9024	0.9032	0.8966
amd/gpt-oss-20b-WMXFP4-AFP8-KVFP8 (this one)	MXFP4	FP8	FP8	self_attn router lm_head	0.5303	0.5556	0.5152	0.5404	0.8999	0.8900	0.8958	0.9098

Disclaimer

This model is intentionally quantized for the vLLM's CI test usage (tests/models/quantization/test_gpt_oss.py). Model performances are not guaranteed to be optimal.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

amd/gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8

amd/gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8

Introduction

Quantization schemes

Quantization script

Deployment

Evaluation

Disclaimer

License

amd/
gpt-oss-20b-MoE-Quant-W-MXFP4-A-FP8-KV-FP8