You can deploy this model and start earning money today!

amd/
Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Name: Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8
Availability: InStock

safetensors

mixtral

mistralai/Mixtral-8x7B-Instruct-v0.1

quantized

apache-2.0

quark

Views

No views yet

Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Introduction

This model was created by applying Quark with calibration samples from Pile dataset.
Quantization Stragegy
- Quantized Layers: All linear layers excluding lm_head, *.gate
- Weight: Auto Mixed Precision quantized by Quark, each weight has either quantization scheme in candidates of
  - FP8 symmetric per-tensor
  - OCP Microscaling (MX) FP4
- Activation: Auto Mixed Precision quantized by Quark, each activation input has the same quantization scheme with weight, i.e., in candidates of
  - FP8 symmetric per-tensor
  - OCP Microscaling (MX) FP4
- KV Cache: FP8 symmetric per-tensor
Quick Start

Download and install Quark
[TODO] We will provide example script(s) to run auto mixed precision (AMP) quantizations later.

Deployment

The Quark quantized Auto Mixed Precision (AMP) models are now supported to be easily deployed in vLLM backend (vLLM-compatible).

Evaluation

The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.

Evaluation scores

Quant scheme	arc challenge (↑) (acc)		gsm8k (↑) (strict-match)		mmlu (↑) (acc)		wikitext (↓) (word_perplexity)		winogrande (↑) (acc)
	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate	absolute value	recovery rate
BF16	0.6271	100.0%	0.6384	100.0%	0.6893	100.0%	5.7259	100.0%	0.7664	100.0%
FP8	0.6220	99.2%	0.6384	100.0%	0.6816	98.9%	5.8092	98.6%	0.7830	102.2%
AMP	0.6195	98.8%	0.6277	98.3%	0.6806	98.7%	6.2026	92.3%	0.7719	100.7%
MXFP4	0.5922	94.4%	0.4845	75.9%	0.6364	92.3%	7.0935	80.7%	0.7474	97.5%

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

amd/Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

Introduction

Quantization Stragegy

Quick Start

Deployment

Evaluation

Evaluation scores

License

amd/
Mixtral-8x7B-Instruct-v0.1-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8