DeepSeek-R1-Distill-Llama-8B-awq-asym-uint4-g128-lmhead-onnx-cpu

• Introduction

  This model was created by applying Quark with calibration samples from Pile dataset.

• Quantization Strategy

  • Quantized Layers: All linear layers
  • Weight: uint4 asymmetric per-group, group_size=128

• Quick Start

1. Download and install Quark
2. Run the quantization script in the example folder using the following command line:
   sh

   1export MODEL_DIR = [local model checkpoint folder] or DeepSeek-R1-Distill-Llama-8B 2# single GPU 3python quantize_quark.py --model_dir $MODEL_DIR \ 4 --output_dir output_dir $MODEL_NAME-awq-asym-uint4-g128-lmhead \ 5 --quant_scheme w_uint4_per_group_asym \ 6 --num_calib_data 128 \ 7 --quant_algo awq \ 8 --dataset pileval_for_awq_benchmark \ 9 --seq_len 512 \ 10 --model_export hf_format \ 11 --data_type bfloat16 \ 12 --exclude_layers 13# cpu 14python quantize_quark.py --model_dir $MODEL_DIR \ 15 --output_dir output_dir $MODEL_NAME-awq-asym-uint4-g128-lmhead \ 16 --quant_scheme w_uint4_per_group_asym \ 17 --num_calib_data 128 \ 18 --quant_algo awq \ 19 --dataset pileval_for_awq_benchmark \ 20 --seq_len 512 \ 21 --model_export hf_format \ 22 --data_type bfloat16 \ 23 --exclude_layers \ 24 --device cpu

Deployment

Evaluation

Evaluation scores

License