A q4_K_M GGUF of Qwen2.5-14B-Instruct with the dolly LoRA merged in, built with llama.cpp. Part of a full open-model loop demo: fine-tune, merge, quantize, serve.

File: qwen2.5-14b-dolly-q4_K_M.gguf (8.4 GB, 4.87 bits per weight)
Base: Qwen/Qwen2.5-14B-Instruct with ArgusForge/qwen2.5-14b-dolly-lora merged in
Built with llama.cpp llama-quantize from an f16 GGUF

Why q4_K_M

On a quant-ladder benchmark of this model (H100 SXM 80GB), q4_K_M was the fastest at generation, about 126 tok/s on GPU and the fastest on CPU as well, while costing only about 3.5 percent perplexity over q8_0 on wikitext-2. Smallest, fastest, minimal quality loss. Full numbers and method: https://github.com/ArgusForge/open-model-ops

Run

llama-cli -m qwen2.5-14b-dolly-q4_K_M.gguf -ngl 99 -p "Explain why the sky is blue to a ten year old."

LoRA adapter: ArgusForge/qwen2.5-14b-dolly-lora
Benchmark and training code: https://github.com/ArgusForge/open-model-ops

ArgusForge/qwen2.5-14b-dolly-q4_K_M-GGUF

Qwen2.5-14B dolly, q4_K_M GGUF (demo)

Why q4_K_M

Run

Related

ArgusForge/
qwen2.5-14b-dolly-q4_K_M-GGUF