Meta-Llama-3.2-1B-Instruct-awq-uint4-asym-g128-bf16

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Stragegy

  • Quantized Layers: All linear layers
  • Weight: int4 symmetric per-group, group_size=128
  • Pre-trained Model data type: BFloat16
  • Scale data type: BFloat16
  • Zero point data type: Int32

• Quick Start

1. Download and install Quark
2. Run the quantization script in the example folder using the following command line:
   sh

   1export MODEL_DIR = [local model checkpoint folder] or meta-llama/Meta-Llama-3.2-1B-Instruct-hf 2# single GPU 3python3 quantize_quark.py --model_dir $MODEL_DIR \ 4 --data_type bfloat16 \ 5 --quant_scheme w_uint4_per_group_asym \ 6 --num_calib_data 128 \ 7 --quant_algo awq \ 8 --dataset pileval_for_awq_benchmark \ 9 --seq_len 512 \ 10 --output_dir $MODEL_NAME-awq-uint4-asym-g128-bf16-lmhead \ 11 --exclude_layers \ 12 --model_export quark_safetensors \ 13 --custom_mode awq \ 14# cpu 15python3 quantize_quark.py --model_dir $MODEL_DIR \ 16 --data_type bfloat16 \ 17 --quant_scheme w_uint4_per_group_asym \ 18 --num_calib_data 128 \ 19 --quant_algo awq \ 20 --dataset pileval_for_awq_benchmark \ 21 --seq_len 512 \ 22 --output_dir $MODEL_NAME-awq-uint4-asym-g128-bf16-lmhead \ 23 --exclude_layers \ 24 --model_export quark_safetensors \ 25 --custom_mode awq \ 26 --device cpu

Deployment

Evaluation

Evaluation scores