zephyr-q5_k_m.gguf — balanced quality vs size (≈ 4.8 GB). Good default for 16 GB RAM laptops.
zephyr-q8_0.gguf — higher fidelity (≈ 7.2 GB). Requires more RAM.

GGUF embeds tokenizer/vocab, so separate tokenizer files are not required for inference.

Prompt Format (Zephyr chat)

Use Zephyr chat tags for best results:

<|user|> YOUR_PROMPT_HERE <|assistant|>

Example

<|user|> List three ways Retrieval-Augmented Generation improves factuality. <|assistant|>

How to Run (llama.cpp)

Q5_K_M (fits most 16 GB RAM systems)

./llama-cli -m zephyr-q5_k_m.gguf \
  -p "<|user|>\nExplain RAG in 3 bullets.\n\n<|assistant|>\n" \
  -n 256 -c 2048 -ngl 0 -t $(nproc)

Q8_0 (higher quality; more RAM)

./llama-cli -m zephyr-q8_0.gguf \
  -p "<|user|>\nGive 5 note-taking tips.\n\n<|assistant|>\n" \
  -n 256 -c 2048 -ngl 0 -t $(nproc)

Flags

Some builds use ./main instead of ./llama-cli. Replace the binary name if needed.

Import the .gguf directly in:

Approximate RAM use at 2k context (CPU-only):

Actual usage varies with context length, batch size, and compile options.

Verify downloads:

sha256sum zephyr-q5_k_m.gguf
sha256sum zephyr-q8_0.gguf

(Add the resulting hashes here if you want to publish them.)

Quantized GGUF weights:

No training code or LoRA adapters are included here.