GGUF quants of darthcrawl/artifex-rp-orpheus-llama-3.1-8b for llama.cpp / Ollama / LM Studio / text-generation-webui. Built for CPU and mid-range-GPU inference on commodity desktops.

Part of the Artifex RP series — register-craft models tiered to common hardware. Curated small data, not scraped giant data. Anti-corporate-assistant by design.

Register: restrained-lyrical — vivid, intelligent prose; tension carries more weight than description.

Orpheus leading Eurydice from the underworld

"He meant the sentence differently than he knew how to say."

For the full model card see the base repo.

Pick a quant

These are register-calibrated (-register) — the importance matrix was built on Orpheus's own prose, not generic wikitext, so the low-bit quants preserve the voice, not just generic perplexity. kld is the mean KL-divergence vs the F16 model on held-out register text (lower = closer to F16). Pick the largest that fits your RAM/VRAM.

File	Quant	Size	KLD vs F16	For
`orpheus-Q8_0.gguf`	Q8_0	8.54 GB	0.00090	Highest fidelity. 12GB+ VRAM / 16GB+ RAM. Indistinguishable from F16.
`orpheus-Q6_K-register.gguf`	Q6_K	6.60 GB	0.00346	Excellent. 8GB GPU comfortable, 12GB+ RAM CPU.
`orpheus-Q5_K_M-register.gguf`	Q5_K_M	5.73 GB	0.00860	Very good. Sweet spot if you have room above 4-bit.
`orpheus-Q4_K_M-register.gguf`	Q4_K_M	4.92 GB	0.02501	Default. Best speed/quality. Fits 8GB RAM / 6GB VRAM with KV headroom.
`orpheus-IQ4_XS-register.gguf`	IQ4_XS	4.45 GB	0.03209	Smaller 4-bit (i-quant), close to Q4_K_M.
`orpheus-IQ3_M-register.gguf`	IQ3_M	3.78 GB	0.09266	Smallest. Tightest RAM/VRAM; visible quality drop, still coherent.

Sweet spot for most users is Q4_K_M. (Q8_0 needs no imatrix, so there's one file; everything else is register-calibrated.)

Why register-calibrated, and does it help?

The importance matrix (imatrix) that guides quantization was computed on Orpheus's own register (its training prose) rather than the usual generic wikitext. The quants are then scored by KL-divergence vs F16 on held-out register text — the metric that actually reflects use-case quality, where wikitext perplexity hides register degradation.

Each quant was built both ways and measured. Register-calibrated wins at every size, and by the largest margin at the lowest bit-rate, where quantization damage is worst:

Quant	Size	KLD generic	KLD register	Improvement
Q6_K	6.60 GB	0.003529	0.003457	−2.0%
Q5_K_M	5.73 GB	0.008928	0.008599	−3.7%
Q4_K_M	4.92 GB	0.025646	0.025011	−2.5%
IQ4_XS	4.45 GB	0.032220	0.032086	−0.4%
IQ3_M	3.78 GB	0.098040	0.092658	−5.5%

(Q8_0 anchor KLD 0.00090; F16 = reference. The raw per-quant measurements are in results.csv.)

Takeaway: at the bit-rates people on commodity hardware actually run, calibrating on the model's own voice measurably preserves more of it.

Run it

llama.cpp

bash

./llama-cli -m orpheus-Q4_K_M-register.gguf \
  --jinja \
  -p "She hadn't seen him in three years. He called on a Tuesday." \
  -n 400 --temp 0.85 --top-p 0.95 --repeat-penalty 1.1

Ollama

FROM ./orpheus-Q4_K_M-register.gguf

PARAMETER temperature 0.85
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1.1

SYSTEM """Write in vivid, restrained, intelligent prose. Concrete sensory detail. Never pretentious. Tension carries more weight than description. Match the user's energy and length."""

Then ollama create artifex -f Modelfile && ollama run artifex.

LM Studio / Jan / text-generation-webui

Drop the .gguf into the loader's model directory, pick Q4_K_M-register unless your RAM/VRAM dictates otherwise. Temp 0.85, top-p 0.95, repetition penalty 1.1. Llama-3 chat template (<|begin_of_text|> / <|eot_id|>).

System prompt

Write in vivid, restrained, intelligent prose. Concrete sensory detail.
Never pretentious. Tension carries more weight than description.
Match the user's energy and length.

License

Llama 3.1 Community License (inherited from base). See LICENSE.

darthcrawl/artifex-rp-orpheus-llama-3.1-8b-gguf

artifex-rp-orpheus-llama-3.1-8b — GGUF (register-calibrated)