1export MODEL_DIR = [local model checkpoint folder] or hpcai-tech/grok-1
2
3python3 quantize_quark.py \
4 --model_dir $MODEL_DIR \
5 --output_dir Grok-1-FP8-KV \
6 --quant_scheme w_fp8_a_fp8 \
7 --kv_cache_dtype fp8 \
8 --seq_len 2048 \
9 --num_calib_data 128 \
10 --model_export quark_safetensors \
11 --multi_gpu \
12 --no_weight_matrix_merge \
13 --custom_mode fp8
Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).
Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py.
The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.
Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
