Two quant levels are provided so you can trade size against quality. For a model this small, Q8_0 is near-lossless and Q4_K_M is the smallest with still-solid quality.

File	Quant	Size	Notes
`Qwen2.5-0.5B-Instruct-Q4_K_M.gguf`	Q4_K_M	~379 MB	Smallest, good quality
`Qwen2.5-0.5B-Instruct-Q8_0.gguf`	Q8_0	~506 MB	Near-lossless

Usage

llama.cpp

llama-cli -hf andreolf/Qwen2.5-0.5B-Instruct-GGUF:Q8_0 -p "Hello!"

or with a local file:

bash

llama-cli -m Qwen2.5-0.5B-Instruct-Q8_0.gguf \
  -p "Explain what a GGUF file is in one sentence."

Ollama

ollama run hf.co/andreolf/Qwen2.5-0.5B-Instruct-GGUF:Q8_0

How it was made

Converted from the original safetensors to an F16 GGUF with convert_hf_to_gguf.py, then quantized with llama-quantize (Q4_K_M) and directly (Q8_0) from llama.cpp. No additional fine-tuning or modification was applied.

License & attribution

The original model is released by Qwen under Apache-2.0; this quantization inherits that license. All credit for the underlying model goes to the Qwen team. This repo only provides a quantized GGUF conversion for convenience.