You can deploy this model and start earning money today!

blockblockblock/
Qwen2.5-0.5B-Instruct-exl3-4.5bpw

Name: Qwen2.5-0.5B-Instruct-exl3-4.5bpw
Availability: InStock

Qwen/Qwen2.5-0.5B-Instruct

quantized

apache-2.0

4-bit

Views

No views yet

Qwen2.5 · 0.5B · Instruct

_{EXL3 · 4.5 bpw · 0.6 GB · Dense · 24 layers}

base model

quantized by

collection

[!NOTE] An ExLlamaV3 build of Qwen/Qwen2.5-0.5B-Instruct at 4.5 bits per weight. See Quants for sibling repos at other bit‑widths or browse the collection.

Quants

BPW	Head bits	Calibration rows	Size	KL ÷ fp16	Status
3.0	8	250	0.5 GB	0.1082	link
4.0	8	250	0.6 GB	0.0277	link
4.5	8	250	0.6 GB	0.0158	`this repo`
5.0	8	250	0.6 GB	0.0074	link
6.0	8	250	0.7 GB	0.0020	link

_{KL ÷ fp16: mean KL-divergence from the fp16 source over wikitext rows — lower is closer to the original.}

Inference

Loader	Use it for
TabbyAPI	OpenAI‑compatible HTTP server. Drop‑in for OpenAI clients.
text‑generation‑webui	Local chat UI. Pick the ExLlamaV3 loader from the model dropdown.
ExLlamaV3	Direct Python API for embedding the model in your own code or pipeline.

Download

bash

pip install -U huggingface_hub

hf download \
  blockblockblock/Qwen2.5-0.5B-Instruct-exl3-4.5bpw \
  --local-dir ./Qwen2.5-0.5B-Instruct-exl3-4.5bpw

Quantization recipe _{(advanced, embedded in quantization_config.json)}

Setting	Value
Format	`EXL3`
Bits per weight	`4.5`
Head bits	`8`
Calibration rows	`250`
Codebook	`MCG`
Out‑scales	`always`
Parallel mode	`enabled`

Loaded automatically by every ExLlamaV3 loader; reproduced here for searchability.

License & use

[!IMPORTANT] Use and license follow the base model. Quantization adds no additional restrictions. Refer to the upstream repository for terms, citation, and safety documentation.

_{Quantized with BlockQuant · convention {org}/{model}-exl3-{bpw}bpw}