qwen3-8b-Q4_K_M.gguf   # ~4.5 bpw, smallest, fastest, slight quality loss
qwen3-8b-Q5_K_M.gguf   # ~5.5 bpw, balanced
qwen3-8b-Q8_0.gguf     # ~8.5 bpw, near-lossless reference

Usage with llama-cpp-python

python

from llama_cpp import Llama
llm = Llama.from_pretrained(
    repo_id="ggfox00000/qwen3-8b-gguf",
    filename="qwen3-8b-Q4_K_M.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
)
out = llm.create_chat_completion(messages=[{"role":"user","content":"Bonjour"}])
print(out["choices"][0]["message"]["content"])

Quantization recipe

Quant	Method	Approximate quality vs FP16
Q4_K_M	k-quants, mixed bits per block	~95-97%
Q5_K_M	k-quants, mixed bits per block	~98-99%
Q8_0	block-wise 8-bit linear	~99.5%+

Built with llama.cpp.

Source

See Qwen/Qwen3-8B for the original model, license, benchmarks and capabilities.