You can deploy this model and start earning money today!

amd/
gpt-oss-20b-WFP8-AFP8-KVFP8

Name: gpt-oss-20b-WFP8-AFP8-KVFP8
Availability: InStock

Views

No views yet

gpt-oss-20b-WFP8-AFP8-KVFP8

Introduction

This model was quantized from openai/gpt-oss-20b using AMD-Quark with calibration samples from the Pile dataset.
Quantization schemes
- Quantized Layers: All linear (both attention linear and MoE linear) layers excluding lm_head
- Weight: quantized using FP8 symmetric per-tensor scheme
- Activation: quantized using FP8 symmetric per-tensor scheme
- KV Cache: FP8 symmetric per-tensor
Quantization script

bash

python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
    --multi_gpu \
    --model_dir openai/gpt-oss-20b \
    --quant_scheme w_fp8_a_fp8 \
    --kv_cache_dtype fp8 \
    --exclude_layers lm_head \
    --dataset pileval \
    --num_calib_data 128 \
    --output_dir amd/gpt-oss-20b-WFP8-AFP8-KVFP8 \
    --model_export hf_format \
    --skip_evaluation

Deployment

This model supports deployment through the vLLM backend. Please ensure PR#29008, PR#31962 have been correctly applied.

Evaluation

This model is evaluated on gpqa_diamond_generative_n_shot and gsm8k_platinum tasks using the lm-evaluation-harness framework with vllm backend.

Evaluation scores

Model name	Weight	Activation	KV cache	Exclude	gpqa_diamond_generative_n_shot (5)	gsm8k_platinum
TP1	TP2	TP4	TP8	TP1	TP2	TP4	TP8
openai/gpt-oss-20b	MXFP4	BF16	BF16	-	0.5606	0.5303	0.5657	0.5606	0.9016	0.9024	0.9032	0.8966
amd/gpt-oss-20b-WFP8-AFP8-KVFP8	FP8	FP8	FP8	*bias, lm_head	0.5505	0.5556	0.5253	0.5253	0.9024	0.9107	0.9024	0.8983

Disclaimer

This model is intentionally quantized for the vLLM's CI test usage (tests/models/quantization/test_gpt_oss.py). Model performances are not guaranteed to be optimal.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

amd/gpt-oss-20b-WFP8-AFP8-KVFP8