qwen2.5-14b-instruct-Q4_K_M.gguf   # ~4.5 bpw, smallest, fastest, slight quality loss
qwen2.5-14b-instruct-Q5_K_M.gguf   # ~5.5 bpw, balanced
qwen2.5-14b-instruct-Q8_0.gguf     # ~8.5 bpw, near-lossless reference

Usage with llama-cpp-python

python

from llama_cpp import Llama
llm = Llama.from_pretrained(
    repo_id="ggfox00000/qwen2.5-14b-instruct-gguf",
    filename="qwen2.5-14b-instruct-Q4_K_M.gguf",
    n_ctx=32768,
    n_gpu_layers=-1,
)
out = llm.create_chat_completion(messages=[{"role":"user","content":"Bonjour"}])
print(out["choices"][0]["message"]["content"])

Quantization recipe

Quant	Method	Approximate quality vs FP16
Q4_K_M	k-quants, mixed bits per block	~95-97%
Q5_K_M	k-quants, mixed bits per block	~98-99%
Q8_0	block-wise 8-bit linear	~99.5%+

Built with llama.cpp.

Source

See Qwen/Qwen2.5-14B-Instruct for the original model, license, benchmarks and capabilities.