Llama-2-70b-chat-hf-WMXFP4FP8-AMXFP4FP8-AMP-KVFP8

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Stragegy

  • Quantized Layers: All linear layers excluding "lm_head"
  • Weight: Auto Mixed Precision quantized by Quark, each weight has either quantization scheme in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • Activation: Auto Mixed Precision quantized by Quark, each activation input has the same quantization scheme with weight, i.e., in candidates of
    • FP8 symmetric per-tensor
    • OCP Microscaling (MX) FP4
  • KV Cache: FP8 symmetric per-tensor

• Quick Start

1. Download and install Quark
2. [TODO] We will provide example script(s) to run auto mixed precision (AMP) quantizations later.

Deployment

Evaluation

Evaluation scores

License