Quant	Size	VRAM est.	Notes
`IQ2_M`	0.33 GB	~0 GB	2-bit, iMatrix — smallest usable
`IQ3_M`	0.34 GB	~0 GB	3-bit, iMatrix — great quality/size tradeoff
`IQ4_XS`	0.35 GB	~0 GB	4-bit extra-small, iMatrix
`Q4_K_M`	0.40 GB	~0 GB	4-bit, iMatrix-calibrated (recommended)
`Q5_K_M`	0.42 GB	~0 GB	5-bit, iMatrix-calibrated
`Q6_K`	0.51 GB	~1 GB	6-bit, iMatrix-calibrated, near-lossless
`Q8_0`	0.53 GB	~1 GB	8-bit, essentially lossless

What is iMatrix?

Standard quantization treats all weights equally. iMatrix runs 128 calibration chunks through the full-precision model to find which weights matter most, then allocates more precision where it counts. At Q2/Q3/Q4 this means noticeably better coherence and instruction-following — same file size, better output.

Calibration: 2M tokens of WikiText-103.

Also see plain (non-iMatrix) quants: liodon-ai/Qwen2.5-0.5B-Instruct-GGUF

Source

Model: Qwen/Qwen2.5-0.5B-Instruct
License: other

Quantized by Liodon AI