AWQ-quantized base model derived from meta-llama/Meta-Llama-3-8B. Weights are W4A16 (4-bit weights, FP16 activations), group size 128, zero point enabled, and use the original MIT Han Lab llm-awq GEMM layout. This is a base model, not an instruction-tuned/chat model.

Load and run

An NVIDIA CUDA GPU is required by this llm-awq backend. Reproduce the pinned environment:

bash

git clone https://github.com/mit-han-lab/llm-awq.git
cd llm-awq
git checkout d6e797a42b9ef7778de8ee2352116e0f48a78d61
pip install -e .
cd awq/kernels
python setup.py install

The pinned project installs PyTorch 2.3.0, Transformers 4.46.0, and Accelerate 0.34.2. Then load directly from the Hub:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "Phuc2005/Llama-3-8B-AWQ-W4A16-g128"
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=False)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "The purpose of quantization in neural networks is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Quantization and validation

Source revision: 8cde5ca8380496c9a6cc7ef3a8b46a0372a1d920
AWQ implementation commit: d6e797a42b9ef7778de8ee2352116e0f48a78d61
Quantization: real W4A16, 4 bits, group size 128
Official llm-awq WikiText perplexity: 6.546638488769531

quantization_metadata.json contains the full environment and packaging record. The WikiText score validates the quantized checkpoint; it does not prove equal quality on every dataset or downstream task.

SHA-256

model-00001-of-00002.safetensors: 363c6b6b44dbec006ba5ed5c04b9f9ba7c60fdd990e4c4e5b1ab5360b7b7aeb5
model-00002-of-00002.safetensors: f3ecea694f172a2cb25e8123aa8626e30ff588237119149eba99a28bc47bf86e
source/llama3-8b-w4-g128-awq.pt: 6de86aecfc9eadead0966366004594704bb747c8acfde46fbf14cf16706e7617

License

This derivative is distributed under the Meta Llama 3 Community License. See LICENSE, NOTICE, and USE_POLICY.md. Built with Meta Llama 3.