Quant	Use Case
Q2_K	Extreme compression
Q3_K_S / Q3_K_M / Q3_K_L	Small footprint
Q4_0 / Q4_K_S / Q4_K_M	Recommended
Q5_0 / Q5_K_S / Q5_K_M	High quality
Q6_K	Near-lossless
Q8_0	Highest quality quant
bf16	Full precision

Usage

bash

brew install llama.cpp
llama-cli --hf-repo smirki/tessa-coder-9b-sft-v0-GGUF --hf-file tessa-coder-9b-sft-v0-q4_k_m.gguf -p "Your prompt here"
llama-server --hf-repo smirki/tessa-coder-9b-sft-v0-GGUF --hf-file tessa-coder-9b-sft-v0-q4_k_m.gguf -c 8192