Sizes below already include the 0.9 GB vision projector and about 1 GB of
runtime overhead at 8k context. Drop the projector if you only want text.
Your VRAM
Take
Why
24 GB and up
BF16
The reference itself. Nothing to lose.
16 GB
Q8_0
Stock, and stock is already optimal this high up.
12 GB
AD-Q8_0-Q6_K
Fills the gap between Q6_K and Q8_0.
8 GB
AD-Q5_K-Q4_K
The sweet spot. Smaller than stock Q5_K_M and more accurate.
8 GB, tight
AD-Q4_K-IQ4_XS
Same size as stock Q4_K_M, 31% lower divergence.
6 GB
AD-IQ3_S-IQ3_XXS
Usable, but top-1 agreement drops to 83%.
4 GB
AD-IQ2_S-IQ2_XS, text only
71% top-1. It runs. Read the warning below.
[!WARNING]
The two smallest files are published for completeness, not because we
recommend them. AD-IQ2_XXS-IQ1_M agrees with the reference on 53.7% of
tokens, which means roughly every second token is a different word than the
full model would have produced. That is not a small quality drop, it is a
different model. If a 9B does not fit, a smaller model at higher precision
will serve you better.
Files and measurements
Mean KL divergence against the BF16 reference, held-out eval_neutral corpus,
4096 context, single RTX 5090, llama.cpp b10505. Lower is better. Top-1 is the
share of tokens where the quant picks the same next token as BF16.
File
Size
Mean KLD
Top-1
Ornith-1.5-9B-BF16.gguf
17.9 GB
reference
100%
Ornith-1.5-9B-Q8_0.gguf
9.53 GB
0.002249
97.94%
Ornith-1.5-9B-AD-Q8_0-Q6_K.gguf
8.55 GB
0.003473
97.46%
Ornith-1.5-9B-Q6_K.gguf
7.36 GB
0.006045
96.54%
Ornith-1.5-9B-Q5_K_M.gguf
6.47 GB
0.029883
92.80%
Ornith-1.5-9B-AD-Q5_K-Q4_K.gguf
5.93 GB
0.025493
93.10%
Ornith-1.5-9B-AD-Q4_K-IQ4_XS.gguf
5.61 GB
0.034426
91.93%
Ornith-1.5-9B-AD-IQ4_XS.gguf
5.52 GB
0.037752
91.54%
Ornith-1.5-9B-IQ4_XS.gguf
5.20 GB
0.044856
90.76%
Ornith-1.5-9B-IQ3_M.gguf
4.42 GB
0.131672
84.18%
Ornith-1.5-9B-AD-IQ3_S-IQ3_XXS.gguf
4.29 GB
0.144132
83.44%
Ornith-1.5-9B-AD-IQ3_XXS-IQ2_S.gguf
3.84 GB
0.258554
77.98%
Ornith-1.5-9B-AD-IQ2_S-IQ2_XS.gguf
3.38 GB
0.441580
71.17%
Ornith-1.5-9B-AD-IQ2_XXS-IQ1_M.gguf
2.81 GB
1.122010
53.74%
mmproj-Ornith-1.5-9B-F16.gguf
0.92 GB
vision projector
AD means a per tensor bit layout tuned for this model. Files without the
prefix are stock llama.cpp presets built with the same importance matrix, so the
comparison isolates the layout and nothing else.
image_2026-08-19_18-00-17
Two results are worth reading off that chart directly.
AD-Q5_K-Q4_K is smaller than stock Q5_K_M and more accurate at the same
time, 5.93 GB against 6.47 GB and 0.0255 against 0.0299. No interpolation
needed to see it.
At the same size, the tuned layout cuts divergence by about a third.AD-Q4_K-IQ4_XS is 5.61 GB at 0.0344 against stock Q4_K_M at 5.63 GB and
0.0500.
[!NOTE]
The tuning only pays while the base type is coarse. Above six bits the stock
presets are already close to lossless, so lifting a small part of the model
buys nothing while paying for it out of a large part costs real accuracy. We
tested this and lost, so above 6 GB this repo ships stock presets rather than
a worse tuned file with our label on it.
[!IMPORTANT]
Pass --image-min-tokens 1024. Without it, dense images such as charts and
screenshots get too few visual tokens, and the model answers from what it
already knows instead of from what it was shown. llama.cpp prints a warning
about this at load time. We hit exactly this on our first attempt: the model
produced a confident, fluent, completely invented description of a chart.
[!IMPORTANT]
Upstream ships no generation_config.json, so llama.cpp falls back to its own
sampling defaults, which are not the ones this model was tuned for. Set them
yourself: --temp 0.6 --top-p 0.95 --top-k 20. Use --temp 1.0 to reproduce
the benchmark setup from the upstream model card.
Lines reading find_slot: non-consecutive token position are normal. That is
how this family numbers image patches, and they appear in successful runs too.
Speed
Single RTX 5090, llama.cpp b10505-ee4c505a4, CUDA 13.0, flash attention on.
Test
BF16
Prompt processing, 512 tokens
9047 t/s
Generation, 128 tokens
94.3 t/s
Generation is limited by memory bandwidth, not compute: 17.9 GB read per token
against 1792 GB/s puts the ceiling near 100 t/s, and BF16 reaches 94% of it.
Smaller files scale close to linearly, so a 5.9 GB build lands near 280 t/s on
the same card.
Vision
The model reads handwriting. Asked to describe a calligraphy piece it has not
seen before, it transcribed the full quote and picked out the small TB-09 mark
in the corner.
input image
model output
This image displays an elegant, calligraphic handwritten quote featuring a
large blue initial capital "P" and intricate botanical illustrations in the
corners. The text reads: "Peace of mind comes to me through making things
with my hands. An added bonus comes if my efforts inspire others to try the
creative process."
Image encoding takes 163 ms for this picture on one RTX 5090.
Reasoning
A short check that the thinking channel does real work rather than filling
space. Prompt: A gas station is 1 kilometer away. I want to wash my car. Should
I walk or drive?
thinking trace
The model answered "drive", then argued against itself in the open, noting that
some gas stations have car washes so walking there is not obviously wrong, then
closed the argument by observing that walking leaves the car at home either way:
You should drive. To wash your car, your car needs to be at the location
where it is being washed.
How the layout was chosen
The bit layout was not carried over from another model. We built eight candidate
layouts at one size class, measured all of them against the same reference on
the same box, and kept the winner. Every log below is in the metrics repo.
Layout
Size
Mean KLD
Top-1
What it changed
AB-G
5.58 GB
0.034841
91.85%
attn_gate and ssm_out up, MLP gate and up down
AB-Q
5.71 GB
0.044434
91.04%
split attn_q: full attention layers up, linear layers down
AB-E
5.64 GB
0.045923
90.93%
followed the energy bands from the importance matrix
AB-U
5.57 GB
0.048779
90.60%
control, flat across layers
AB-S
5.61 GB
0.049737
90.50%
ssm_in up
Q4_K_M
5.63 GB
0.049981
90.47%
stock preset, same importance matrix
AB-X
5.62 GB
0.051900
90.45%
edge layers lifted, a recipe that won on another model
AB-T
5.67 GB
0.066871
89.21%
embedding down hard, output head up hard
Three things came out of this.
The gates matter more than their size suggests.attn_gate and ssm_out
are 9% of the weights, and the importance matrix ranks attn_gate first in the
whole model by summed squared activation. Lifting those two and paying for them
out of the MLP gate and up projections, which are 36% of the weights, is the
single largest win available.
The embedding must not be squeezed. With a 248320 token vocabulary and
untied weights, the embedding and the output head are 22% of this model, which
makes them look like the obvious place to save. AB-T tested that and came last,
worse than the stock preset. A lifted output head does not pay for a crushed
embedding.
A layout does not transfer by layer position. AB-X applied a recipe that
works well on a 64 layer model by scaling the layer numbers, and it lost to the
flat control. What transfers is which tensor groups matter, read from the
importance matrix of this model.
Architecture
Read from the converted file, not from the config.
Parameters
8.95B language, 0.46B vision
Layers
32
Attention
hybrid: full attention on layers 3, 7, 11, 15, 19, 23, 27, 31, linear attention on the other 24
Hidden size
4096
FFN size
12288
Vocabulary
248320, untied embeddings
Context
262144
Architecture string
qwen35
Only 8 of the 32 layers keep a KV cache, the other 24 hold a fixed recurrent
state instead. Long context is unusually cheap here, roughly 32 KB per token, so
8k of context costs about 256 MB rather than the gigabytes this size of model
normally implies.
[!WARNING]
If you convert this model yourself, pass --no-nextn to
convert_hf_to_gguf.py. Upstream config.json declares
mtp_num_hidden_layers: 1, but the checkpoint contains no such tensors. The
converter believes the config, writes block_count = 33 for a 32 layer model,
and reports a successful export. The file then fails to load with
tensor 'blk.32.attn_norm.weight' not found. There is no speculative draft to
publish for this model.
Calibration
The importance matrix was computed on the BF16 weights, not on a quantized
stand-in, over 4,958,490 tokens in 9,686 chunks, fanned out across 20 GPUs and
merged. Vocabulary coverage of the calibration corpus is 99.5%, so almost no
embedding row was quantized without evidence.
Corpus composition: agentic tool traces 24.7%, code 17.8%, reasoning 14.8%,
multilingual 13.8%, long context 11.9%, vocabulary sweep 9.9%, structured data
3.9%, graphics 3.0%. The vocabulary sweep is regenerated per tokenizer, since a
sweep built for another model covers a different vocabulary and calibrates
nothing here.
Reproducing our numbers
Everything needed to check this card is public:
The BF16 file in this repo is the reference every measurement is taken against.
Ornith-1.5-9B-GGUF-metrics
holds the importance matrix, every raw KL divergence log including the losing
ablations, and the reference logits themselves, so you can measure your own
build against the same starting point instead of taking our word for it.
To measure a build of your own:
The corpus and the context have to match ours or the numbers are not comparable.
About the model
Ornith-1.5 is trained with a loop that generates its own tasks, builds its own
scaffolds, and rewards all three stages together rather than optimizing a policy
against a fixed human written harness. For the method and the benchmark results,
see the upstream model card
and the Ornith blog. Those benchmark numbers are the
authors' own and we have not independently reproduced them.
