GGUF quantizations of deepreinforce-ai/Ornith-1.0-9B,
produced with Intel AutoRound using its alg_ext
iterative calibration (~200 optimization steps per block) rather than plain round-to-nearest.
At a given size this recovers noticeably more quality than vanilla GGUF quants, with the largest
benefit at the low bit widths.
Ornith-1.0 is a self-improving, MIT-licensed family of open-source agentic-coding models from
Deep Reinforce; the 9B is the lightweight, single-GPU member (dense, 32 layers, 256K context). See
the original model card for capabilities
and coding benchmarks. This repo only re-packages those weights as GGUF.
Quality vs. size
Every quant was measured against the BF16 reference on wikitext-2 test: PPL and, more
importantly, KL-divergence of the output distribution (how far the quant drifts from full
precision; lower is better). Top-1 is the share of positions whose argmax token still matches
BF16. Reference BF16 PPL = 8.13.
Quality vs. size on wikitext-2: Mean KLD (top) and PPL (bottom) against bits-per-weight
Scheme
Size
bpw
PPL
ΔPPL
Mean KLD
Top-1 agree
Q2_K_S
3.7 GB
3.30
9.89
+21.7%
0.2960
76.8%
Q3_K_S
4.3 GB
3.81
8.53
+4.9%
0.1216
85.0%
Q3_K_M
4.5 GB
4.04
8.57
+5.4%
0.1214
84.9%
Q3_K_L
4.9 GB
4.40
8.60
+5.8%
0.1201
84.9%
Q4_K_S
5.4 GB
4.78
8.05
-1.0%
0.0425
91.0%
Q4_K_M ⭐
5.6 GB
5.03
8.08
-0.6%
0.0406
91.2%
Q6_K
7.4 GB
6.58
8.15
+0.3%
0.0040
96.6%
Q8_0
9.5 GB
8.51
8.14
+0.1%
0.0020
97.5%
Recommendations
Q4_K_M ⭐ is the best all-round pick: smaller than the Q5 tier yet closest of the mid-size
quants to BF16 (KLD 0.04, Top-1 91%). Start here.
Q6_K / Q8_0 are effectively lossless (KLD under 0.004) when you have the VRAM.
Q3_K_* are usable on tight memory. Reach for Q2_K_S only when you must, since 2-bit shows
noticeable drift.
Why no Q5 or Q2_K_MIXED? On the KLD benchmark the Q5_K quants were not closer to BF16 than
Q4_K_M despite being larger, so they add size without quality for this model and were omitted. The
Q2_K_MIXED variant was defective here (broken output distribution) and was also excluded. (This
base checkpoint ships no MTP/NextN head, so there is no speculative-decoding block to preserve.)
Multimodal
Ornith inherits a vision tower. Multimodal projectors are included for use via llama-mtmd-cli
(pair any LM quant above with one of these). For text-only / coding use you can ignore them.
All outputs were NaN-scanned and inference-checked. The comparison table and the Pareto plot above
(pareto.png, comparison.csv in this repo) come from llama-perplexity --kl-divergence against
the BF16 reference.
Support
These quantized models are made in my spare time using expensive hardware such as DGX Spark systems for quantization and validation. If you find these GGUFs useful for your projects, consider buying me a coffee to help cover hardware and compute costs. Every bit of support helps me keep producing high-quality quantized models for the community!
