A third of the size, level with BF16 on knowledge. Read the tool-calling note.
Mixed-precision quantization of ornith-ai/Ornith-1.5-35B-A3B to 22.6 GiB
(3.0× smaller than BF16) on stock vLLM, with the vision tower and the
in-checkpoint MTP block left intact.
Why this quant
🗜️ 3.0× smaller than BF16, 39% smaller than the official FP8. 22.6 GiB against 67.0 GiB and
36.7 GiB.
🎯 Closest to BF16 on knowledge of any quant of this model measured. 91.7 against BF16's 92.0
on a 1,170-item suite, against the official FP8's 91.2 and NVFP4's 90.9. The FP8 ties BF16 on this
suite too; this build is nearer.
🛠️ Tool calling is this build's weaker axis, and there is a sibling build for that. 71.8 over
six runs against BF16's 75.1 over four: real but marginal, and not distinguishable from either
official quant. Pooled over both suites this build scores 88.7, ahead of the official NVFP4
(88.3) and behind the FP8 (89.0). If tool calling is your workload, take
our agentic build:
same size, 74.0 on that suite with a third of the run-to-run spread.
⚡ 1.72× BF16 throughput at concurrency 32, 1.22× single-stream. 1,454 tok/s against 845.
👁️ The vision tower is untouched. Vision, the projector and norms stay BF16: a VLM loses its
visual grounding before its words.
🚀 The MTP block is preserved, so speculative decoding remains available. The tooling drops it
by default, and dropping it is silent.
🖥️ BF16 lm_head. It costs a little size and speed, and it is what keeps the Ampere
(Marlin W·A16) path loadable; a quantized head crashed that path on a sibling architecture.
Text and image inputs both work. For long-context serving, --kv-cache-dtype fp8 buys a large
increase in KV capacity at negligible throughput cost; it is left off here so the numbers below
describe the checkpoint rather than a serving flag.
Measured
1,370 items across fourteen public benchmarks. A 1,170-item knowledge suite and a 200-item
tool-calling suite, under one fixed protocol for every row: temperature 0.6 / top_p 0.95 /
top_k 20, thinking forced on, a 16,384-token budget, concurrency 32 plus a 60-item single-stream
subset, all rows back to back in one sitting on one RTX PRO 6000 Blackwell.
overall is one number over both suites: the 1,170 knowledge and 200 tool-calling items pooled
as 1,370, weighted 85.4% and 14.6% by item count. Repeat runs of one checkpoint moved it by
about a point, so treat a gap below 1.0 as a tie.
call is accuracy on the 160 rows that require a call; abstain is the 40 whose correct action is
to call nothing. Benchmarks: BFCL v4 (30, including irrelevance), xLAM/APIGen (45), ToolACE
(45), Glaive v2 (40), nvidia When2Call (40, the abstention rows). Tool schemas go in the system prompt
and the model answers with a JSON array of calls, the same way for every model. An item counts only
if every expected call is present with acceptable arguments and no call is invented.
Abstention is the weak axis for every model we have measured (52–82%), so a build can look strong
on overall and still over-call.
The last row is a drift control: the same checkpoint re-measured after two hours of other work came
back within 1.3% on throughput and 0.4 points on knowledge.
Every agentic figure above is a mean of repeated runs, and on this model that is essential.
Quantized builds of Ornith carry two to three times BF16's run-to-run spread on the 200-item
tool-calling suite: six runs of this checkpoint span 67.5 to 75.5 (sd 3.1), against BF16's 1.4
and the official quants' 1.2-1.7. An earlier version of this card reported a single run (69.0) as
though it were the number, and drew a conclusion from it that the data does not support. Corrected:
build
tool calling
n
sd
BF16
75.1
4
1.4
ornith-ai FP8
74.9
4
1.7
ornith-ai NVFP4
73.1
4
1.2
this repo
71.8
6
3.1
This build is 3.4 points below BF16 on tool calling (t=2.3, so real but marginal) and not
statistically distinguishable from either official quant -- the NVFP4 by 1.4 points at t=1.0. On the
pooled score it leads that NVFP4 (88.7 vs 88.3). Read differences under about 2 points here as noise.
🛠️ If tool calling is your workload, use
our agentic build
instead. Same size and format, calibrated for tool use: 74.0 on this suite, a tie with BF16's
75.1, with a third of this build's run-to-run spread and 0.2 points less general knowledge.
Read this honestly: the official NVFP4 build is faster and smaller than this one. 15.6% more
aggregate throughput, 27% more single-stream, and 0.75 GiB less on disk. It gets there partly by
quantizing the 508 M-parameter lm_head, which is hit on every decoded token; this build keeps that
head at BF16. So the trade is roughly: their build if you want maximum speed on Blackwell, this one
if you want the closest accuracy to BF16 and an output head that keeps the Ampere fallback path
loadable. All four accuracy figures sit within about a point of each other, which is this suite's
run-to-run spread, so treat that column as a tie and choose on the others.
Comparable with our other models
Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a
different temperature, or whether the model was allowed to reason at all. So every number in this
table, on this card and on our other cards, comes from one fixed protocol.
The same 1,370 items: a 1,170-item knowledge suite (MMLU-Pro, ARC-Challenge, HellaSwag, WinoGrande,
CommonsenseQA, BoolQ, OpenBookQA, GSM8K, MATH-500) and a 200-item tool-calling suite (BFCL v4,
xLAM/APIGen, ToolACE, Glaive v2, nvidia When2Call). temperature 0.6, top_p 0.95, top_k 20,
thinking forced on, a 16,384-token budget, no reasoning parser, scoring the last ANSWER: in the
reply. Concurrency 32 on one RTX PRO 6000 Blackwell, each model's rows in one sitting. Auto-scored,
no LLM judge. Both halves are means of at least three runs per build.
Read overall with finished.overall scores an answer that overran the token budget as wrong,
but it cannot say whether the model needed the room or failed to stop; finished and out separate
those. A gap under 1.0 is a tie. The tok/s column comes from each model's own sitting and drifts
a few percent between sittings, so read it as a bracket.
call and abstain are the tool-calling suite's two halves, reported separately.call is
accuracy on the 160 items that require a tool call; abstain is the 40 whose correct action is to
call nothing. They used to be pooled into one agentic number, and the pooling misled: a model with
ordinary call accuracy and unusual abstention discipline outscored models that are better at
actually making calls. Weight them by your own workload's mix.
lm_head, embeddings, router, shared_expert_gate, the linear-attention state path (in_proj_a/in_proj_b/conv1d/A_log/dt_bias), norms, vision tower, MTP block
BF16
Weights are quantized with GPTQ against a text-only calibration set; the vision tower is not
quantized, so image inputs are unaffected. Activations for the NVFP4 group are quantized dynamically
at runtime by vLLM.
