All quants made using the imatrix option with a bilingual (English + Chinese) and code-heavy calibration dataset, so low-bit quants keep more of their Chinese and coding ability than a English-only calibration would.

Run them in LM Studio, Ollama, or directly with llama.cpp and any llama.cpp based project.

Prompt format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Download a file (not the whole branch) from below:

Filename	Quant type	File Size	Description
Qwen3-4B-Q8_0.gguf	Q8_0	4.28GB	Extremely high quality, generally unneeded but max available quant.
Qwen3-4B-Q6_K.gguf	Q6_K	3.31GB	Very high quality, near perfect, recommended.
Qwen3-4B-Q5_K_M.gguf	Q5_K_M	2.89GB	High quality, recommended.
Qwen3-4B-Q4_K_M.gguf	Q4_K_M	2.50GB	Good quality, default size for most use cases, recommended.
Qwen3-4B-IQ4_XS.gguf	IQ4_XS	2.27GB	Decent quality, smaller than Q4_K_S with similar performance, recommended.
Qwen3-4B-IQ3_M.gguf	IQ3_M	1.96GB	Medium-low quality, new method with decent performance comparable to Q3_K_M.
Qwen3-4B-Q2_K.gguf	Q2_K	1.67GB	Very low quality but surprisingly usable.

The .imatrix file used to produce these quants is included in this repo, so anyone can reproduce or extend the quant set with the exact same importance matrix.

Downloading using the hf CLI

Click to view download instructions

First make sure you have the CLI installed:

pip install -U "huggingface_hub[cli]"

Then target the specific file you want (do not clone the whole repo, it is large):

hf download qtum/Qwen3-4B-GGUF --include "Qwen3-4B-Q4_K_M.gguf" --local-dir ./

Which file should I choose?

Click here for details

A great write up with charts showing various performances is provided by Artefact2 here.

The first thing to figure out is how big a model you can run. For that you need to know how much RAM and/or VRAM you have.

If you want the model running as FAST as possible, fit the whole thing in your GPU's VRAM: pick a quant with a file size 1-2GB smaller than your total VRAM.

If you want maximum quality, add your system RAM and your GPU's VRAM together, then pick a quant 1-2GB smaller than that total.

Next, decide between an 'I-quant' and a 'K-quant'.

If you don't want to think about it, grab a K-quant, in the format QX_K_X such as Q5_K_M.

If you want to dig deeper, see the llama.cpp feature matrix.

In short: if you are targeting below Q4 and running cuBLAS (Nvidia) or rocBLAS (AMD), look at the I-quants, in the format IQX_X such as IQ3_M. They are newer and offer better quality for their size. I-quants also work on CPU, but are slower than their K-quant equivalent, so it is a speed-vs-quality tradeoff.

Credits

Quantization pipeline built on llama.cpp by ggml-org.

Thanks to bartowski and kalomaze for establishing the imatrix calibration practice this pipeline follows.