The purpose of this repository is to provide fast, easy-to-use local inference files for llama.cpp, Ollama, LM Studio, Jan, Open WebUI, and llama-cpp-python users.

Model Details

Base model: Qwen/Qwen2.5-Coder-7B-Instruct
Architecture: Qwen 2
Format: GGUF
Source license: apache-2.0
Conversion tool: convert_hf_to_gguf.py from llama.cpp
Quantization tool: llama-quantize
Recommended file: Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf

Quantized Files

Quant	Filename	Size	SHA256	Notes
FP16	`Qwen2.5-Coder-7B-Instruct-FP16.gguf`	~14.19 GiB	`274c0eb05fe4...`	Full precision converted GGUF baseline
Q2_K	`Qwen2.5-Coder-7B-Instruct-Q2_K.gguf`	~2.81 GiB	`3200d17f49c6...`	Smallest, lowest quality
Q3_K_M	`Qwen2.5-Coder-7B-Instruct-Q3_K_M.gguf`	~3.55 GiB	`30be23c2bc76...`	Small balanced version
Q4_0	`Qwen2.5-Coder-7B-Instruct-Q4_0.gguf`	~4.13 GiB	`339fbe41f5d4...`	Simple 4-bit quantization
Q4_K_M	`Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf`	~4.36 GiB	`46caa6175bbe...`	Recommended default for most users
Q5_K_M	`Qwen2.5-Coder-7B-Instruct-Q5_K_M.gguf`	~5.07 GiB	`38d6bd18220d...`	Better quality with moderate size
Q6_K	`Qwen2.5-Coder-7B-Instruct-Q6_K.gguf`	~5.82 GiB	`86b8e1701365...`	High quality
Q8_0	`Qwen2.5-Coder-7B-Instruct-Q8_0.gguf`	~7.54 GiB	`d4cdb65b1880...`	Near FP16 quality

Validation

Each file was tested with llama-cli for basic load + generation.

Quant	Filename	Status
FP16	`Qwen2.5-Coder-7B-Instruct-FP16.gguf`	✅ passed
Q2_K	`Qwen2.5-Coder-7B-Instruct-Q2_K.gguf`	✅ passed
Q3_K_M	`Qwen2.5-Coder-7B-Instruct-Q3_K_M.gguf`	✅ passed
Q4_0	`Qwen2.5-Coder-7B-Instruct-Q4_0.gguf`	✅ passed
Q4_K_M	`Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf`	✅ passed
Q5_K_M	`Qwen2.5-Coder-7B-Instruct-Q5_K_M.gguf`	✅ passed
Q6_K	`Qwen2.5-Coder-7B-Instruct-Q6_K.gguf`	✅ passed
Q8_0	`Qwen2.5-Coder-7B-Instruct-Q8_0.gguf`	✅ passed

Usage

llama.cpp

llama-cli -m Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

Older builds may use:

./main -m Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf -p "Hello! Introduce yourself briefly."

llama.cpp directly from Hugging Face

llama-cli -hf ShahzebKhoso/Qwen2.5-Coder-7B-Instruct:Q4_K_M -p "Hello! Introduce yourself briefly."

llama-cpp-python

python

from huggingface_hub import hf_hub_download
from llama_cpp import Llama

model_path = hf_hub_download(
    repo_id="ShahzebKhoso/Qwen2.5-Coder-7B-Instruct",
    filename="Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf",
)

llm = Llama(model_path=model_path)

out = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello! Introduce yourself briefly."},
    ],
    max_tokens=128,
)

print(out["choices"][0]["message"]["content"])

Which file should I use?

Use Q4_K_M for the best default balance.
Use Q5_K_M for better quality.
Use Q8_0 if you want near-original quality and have more memory.
Use Q2_K or Q3_K_M only when memory is very limited.

Provenance

This repository is a quantized derivative of:

Qwen/Qwen2.5-Coder-7B-Instruct

Base model metadata:

text

revision: c03e6d358207e414f1eca0bb1891e29f1db0e242
pipeline_tag: text-generation
tags: transformers, safetensors, qwen2, text-generation, code, codeqwen, chat, qwen, qwen-coder, conversational, en, arxiv:2409.12186, arxiv:2309.00071, arxiv:2407.10671, base_model:Qwen/Qwen2.5-Coder-7B, base_model:finetune:Qwen/Qwen2.5-Coder-7B, license:apache-2.0, text-generation-inference, endpoints_compatible, deploy:azure