You can deploy this model and start earning money today!

amd/
Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Name: Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8
Availability: InStock

Views

No views yet

Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Introduction

This model was created by applying Quark with calibration samples from Pile dataset.
Quantization Stragegy
- Quantized Layers: All linear layers excluding "lm_head"
- Weight: Auto Mixed Precision quantized by Quark, each weight has either quantization scheme in candidates of
  - FP8 symmetric per-tensor
  - OCP Microscaling (MX) FP4
- Activation: Auto Mixed Precision quantized by Quark, each activation input has the same quantization scheme with weight, i.e., in candidates of
  - FP8 symmetric per-tensor
  - OCP Microscaling (MX) FP4
- KV Cache: FP8 symmetric per-tensor
Quick Start

Download and install Quark
[TODO] We will provide example script(s) to run auto mixed precision (AMP) quantizations later.

Deployment

The Quark quantized Auto Mixed Precision (AMP) models are now supported to be easily deployed in vLLM backend (vLLM-compatible).

Evaluation

The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

Evaluation scores

Quant scheme	arc challenge (↑) (acc)		gsm8k (↑) (strict-match)		mmlu (↑) (acc)		winogrande (↑) (acc)
	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate
BF16	0.5597	100.0%	0.8552	100.0%	0.7296	100.0%	0.6780	100.0%
FP8	0.5563	99.4%	0.8461	98.9%	0.7282	99.8%	0.6654	98.1%
AMP	0.5154	92.1%	0.8446	98.8%	0.7171	98.3%	0.6559	96.7%
MXFP4	0.4863	86.9%	0.8112	94.9%	0.6883	94.3%	0.6504	95.9%

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

amd/Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Introduction

Quantization Stragegy

Quick Start

Deployment

Evaluation

Evaluation scores

License

amd/
Qwen3-8B-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8