Alternatively, you can quantize 16-bit weights to 4-bit weights following the "4-bit Weight Quantization" section, and then perform inference as per the below instructions.

Take the 4-bit Llama-2-13B model from the model zoo as an example:

shell

git-lfs install
git clone https://huggingface.co/lmdeploy/llama2-chat-13b-w4

As demonstrated in the command below, first convert the model's layout using turbomind.deploy, and then you can interact with the AI assistant in the terminal

shell

## Convert the model's layout and store it in the default path, ./workspace.
python3 -m lmdeploy.serve.turbomind.deploy \
    --model-name llama2 \
    --model-path ./llama2-chat-13b-w4 \
    --model-format awq \
    --group-size 128

## inference
python3 -m lmdeploy.turbomind.chat ./workspace

Serve with gradio

If you wish to interact with the model via web ui, please initiate the gradio server as indicated below:

python3 -m lmdeploy.serve.turbomind ./workspace --server_name {ip_addr} ----server_port {port}

Subsequently, you can open the website http://{ip_addr}:{port} in your browser and interact with the model

Inference Performance

We benchmarked the Llama 2 7B and 13B with 4-bit quantization on NVIDIA GeForce RTX 4090 using profile_generation.py. And we measure the token generation throughput (tokens/s) by setting a single prompt token and generating 512 tokens. All the results are measured for single batch inference.

model	llm-awq	mlc-llm	turbomind
Llama 2 7B	112.9	159.4	206.4
Llama 2 13B	N/A	90.7	115.8

shell

python benchmark/profile_generation.py \
  ./workspace \
  --concurrency 1 --input_seqlen 1 --output_seqlen 512

4-bit Weight Quantization

It includes two steps:

generate quantization parameter
quantize model according to the parameter

Step 1: Generate Quantization Parameter

shell

python3 -m lmdeploy.lite.apis.calibrate \
  --model $HF_MODEL \
  --calib_dataset 'c4' \             # Calibration dataset, supports c4, ptb, wikitext2, pileval
  --calib_samples 128 \              # Number of samples in the calibration set, if memory is insufficient, you can appropriately reduce this
  --calib_seqlen 2048 \              # Length of a single piece of text, if memory is insufficient, you can appropriately reduce this
  --work_dir $WORK_DIR \             # Folder storing Pytorch format quantization statistics parameters and post-quantization weight

Step2: Quantize Weights

LMDeploy employs AWQ algorithm for model weight quantization.

shell

python3 -m lmdeploy.lite.apis.auto_awq \
  --model $HF_MODEL \
  --w_bits 4 \                       # Bit number for weight quantization
  --w_sym False \                    # Whether to use symmetric quantization for weights
  --w_group_size 128 \               # Group size for weight quantization statistics
  --work_dir $WORK_DIR \             # Directory saving quantization parameters from Step 1

After the quantization is complete, the quantized model is saved to $WORK_DIR. Then you can proceed with model inference according to the instructions in the "4-Bit Weight Model Inference" section.