gemma-2-2b-awq-uint4-asym-g128-lmhead-g32-fp16-onnx

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Strategy

  • Quantized Layers: All linear layers
  • Weight: uint4 asymmetric per-group. group_size=32 for lm_head, and group_size=128 for the rest.

• Quick Start

1. Download and install Quark
2. Run the quantization script in the example folder using the following command line:
   sh

   1export MODEL_DIR = [local model checkpoint folder] or google/gemma-2-2b 2# single GPU 3python quantize_quark.py --model_dir $MODEL_DIR \ 4 --output_dir output_dir $MODEL_NAME-awq-uint4-asym-g128-lmhead-g32-fp16 \ 5 --quant_scheme w_uint4_per_group_asym \ 6 --num_calib_data 128 \ 7 --quant_algo awq \ 8 --dataset pileval_for_awq_benchmark \ 9 --model_export hf_format \ 10 --group_size 128 \ 11 --group_size_per_layer lm_head 32 \ 12 --data_type float16 \ 13 --exclude_layers 14# cpu 15python quantize_quark.py --model_dir $MODEL_DIR \ 16 --output_dir output_dir $MODEL_NAME-awq-uint4-asym-g128-lmhead-g32-fp16 \ 17 --quant_scheme w_uint4_per_group_asym \ 18 --num_calib_data 128 \ 19 --quant_algo awq \ 20 --dataset pileval_for_awq_benchmark \ 21 --model_export hf_format \ 22 --group_size 128 \ 23 --group_size_per_layer lm_head 32 \ 24 --data_type float16 \ 25 --exclude_layers \ 26 --device cpu

Deployment

Evaluation

Evaluation scores

License