Views
No views yet
llm-compressor and saved in the compressed-tensors format.| Field | Value |
|---|---|
| Base model | Qwen/Qwen2-0.5B-Instruct |
| Base revision | c540970f9e29518b1d8f06ab8b24cba66ad77b6d |
| Offline algorithm | RTN (round-to-nearest weight quantization) |
| Quantized targets | Linear |
| Weight format | Signed INT8, symmetric |
| Weight granularity | Per-output-channel (strategy=channel, group_size=null) |
| Activation/KV format | Floating point; not quantized by this checkpoint |
| Ignored module | lm_head |
| Resolved model dtype | BF16 |
| Serialization | compressed-tensors, pack-quantized |
| Export provider | llm-compressor==0.12.0 |
| Compression library | compressed-tensors==0.17.1 |
compressed-tensors format. The export environment used Transformers 5.10.1 and compressed-tensors==0.17.1.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_id = "<your-hf-account>/Qwen2-0.5B-Instruct-RTN-W8A16"
4
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(
7 model_id,
8 torch_dtype="auto",
9 device_map="auto",
10)
11
12messages = [
13 {"role": "system", "content": "You are a helpful assistant."},
14 {"role": "user", "content": "Explain INT8 weight-only quantization briefly."},
15]
16text = tokenizer.apply_chat_template(
17 messages,
18 tokenize=False,
19 add_generation_prompt=True,
20)
21inputs = tokenizer([text], return_tensors="pt").to(model.device)
22outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
23response = tokenizer.decode(
24 outputs[0][inputs.input_ids.shape[1]:],
25 skip_special_tokens=True,
26)
27print(response)config.json contains the effective compressed-tensors quantization configuration.provider_recipe.json records the requested and resolved provider parameters.recipe.yaml contains the effective llm-compressor recipe.model.safetensors contains the compressed model weights.not_checked. Before publishing results, evaluate this checkpoint and the BF16 base model with identical task versions, prompts, sample selections, chat templates, generation parameters, and random seeds.1lm-eval run \
2 --model hf \
3 --model_args pretrained=<your-hf-account>/Qwen2-0.5B-Instruct-RTN-W8A16,dtype=auto \
4 --tasks arc_challenge \
5 --num_fewshot 0 \
6 --apply_chat_template \
7 --batch_size 1 \
8 --log_samples \
9 --output_path runs/rtn_w8a16/arc_challengelm_head, activations, Attention operations, and KV Cache are not INT8-quantized by this checkpoint.1@article{qwen2,
2 title={Qwen2 Technical Report},
3 author={Yang, An and others},
4 journal={arXiv preprint arXiv:2407.10671},
5 year={2024}
6}