Single-GPU llama.cpp GGUF package for deepreinforce-ai/Ornith-1.0-35B.
The supported serving policy is tp=1 — one model copy per GPU. Multi-GPU
tensor-parallel serving is intentionally out of scope for this version.
This release ships six body quants (Q3_K_M → Q8_0) plus an integrated
IQ4_XS-MTP graft that adds a native multi-token-prediction (MTP) draft head for
low-concurrency speculative decode. Quant quality is measured against the upstream
BF16 GGUF with a native llama.cpp next-token top-64 KL-divergence probe over 32
coding prompts.
MTP graft note: active MTP is byte-for-byte identical to target-only at the
next-token (API-visible) level, and gives ~1.3× single-stream decode. It is
not bit-exact to target-only over long deterministic sequences — see
MTP. Use target-only for strict reproduction,
MTP for throughput on open-ended work.
Quantization & fidelity
Fidelity is KL(P_bf16 || P_candidate) over next-token top-64 distributions,
32 coding prompts, n_predict=1, temperature=-1, n_probs=64, keyed by token ID.
Lower KLD and higher greedy top-1 agreement are better.
Primary fidelity table
Artifact
Mean KLD (nats)
Greedy top-1 †
Q8_0
0.0108
31/32 (96.9%)
Q6_K
0.0165
32/32 (100.0%)
Q5_K_M
0.0354
30/32 (93.8%)
IQ4_XS-MTP (graft)
0.0731382442
29/32 (90.6%)
Q4_K_M
0.0864
29/32 (90.6%)
IQ4_XS
0.1426
27/32 (84.4%)
Q3_K_M
0.3620
27/32 (84.4%)
Next-token fidelity ladder: mean top-64 KLD vs BF16 for the six body quants plus the IQ4_XS-MTP graft, lower is better.
Fidelity ladder — the integrated IQ4_XS-MTP graft (purple) lands between Q5_K_M
and Q4_K_M, i.e. better next-token fidelity than its Q4_K_M neighbor.
† Top-1 vs mean KLD are different axes.Top-1 counts how often the single
most-likely (argmax) token matches BF16's, over the 32 prompts; mean KLD
measures divergence across the full top-64 distribution. Two quants can share a
top-1 count yet differ a lot in KLD — e.g. Q3_K_M and IQ4_XS are both
27/32 on top-1, but Q3_K_M's distribution diverges ~2.5× more (0.3620 vs
0.1426 nats). Neither value is a typo.
The integrated IQ4_XS-MTP graft is 48.70% lower mean KLD than the plain
IQ4_XS body (0.0731382442 vs 0.1425748206), and improves greedy top-1 from
27/32 to 29/32.
Active draft-mtp ≡ target-only at the next-token level: their top-64
next-token distributions are identical (KLD 0.0, 32/32) for the one-token
API-visible probe.
Body-quant distribution detail (secondary)
Full per-quant distribution from the corrected top-64 probe (same 32 prompts).
Mean / P50 / P95 / Max are explicit nats statistics of the per-prompt top-64
KLD; in this run the BF16 top-64 distribution captured mean probability mass
0.999965, so the top-64 approximation captured essentially all next-token mass.
Quant
GGUF GiB
KLD-probe VRAM GiB
Mean nats
P50 nats
P95 nats
Max nats
Top-1 †
Q3_K_M
15.61
16.32
0.3620
0.2548
1.1077
1.3730
27/32 (84.4%)
IQ4_XS
17.64
19.34
0.1426
0.0868
0.3195
0.6586
27/32 (84.4%)
Q4_K_M
19.71
20.35
0.0864
0.0379
0.2877
0.4503
29/32 (90.6%)
Q5_K_M
23.03
23.61
0.0354
0.0235
0.0943
0.2497
30/32 (93.8%)
Q6_K
26.56
27.07
0.0165
0.0092
0.0513
0.0586
32/32 (100.0%)
Q8_0
34.37
34.77
0.0108
0.0052
0.0440
0.0590
31/32 (96.9%)
The IQ4_XS-MTP graft uses the same probe on a different breakpoint context; its
distribution detail (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511) is
in the MTP section. Source:
benchmarks/kld-quant-vs-bf16-top64.md.
Serving performance
Aggregate decode throughput and p95 time-to-first-token across concurrency,
llama.cpp tp=1, from scripts/bench_openai.py --stream --max-tokens 256 (0
failed requests per row). Q4_K_M is the speed pick; Q5_K_M nearly ties it at c16.
Quant
c1 tok/s
c1 p95 TTFT ms
c4 tok/s
c4 p95 TTFT ms
c8 tok/s
c8 p95 TTFT ms
c16 tok/s
c16 p95 TTFT ms
Q3_K_M
240.5
77.9
422.0
170.7
464.4
344.9
493.0
493.7
IQ4_XS
234.1
75.1
297.7
159.7
411.5
330.2
476.0
541.5
Q4_K_M
243.3
76.3
458.3
192.3
615.0
361.8
655.6
650.0
Q5_K_M
236.7
75.1
311.0
198.8
439.0
383.6
638.6
620.4
Q6_K
225.9
76.8
295.8
194.2
409.6
394.8
603.3
657.4
Q8_0
208.5
76.9
281.5
190.8
405.8
389.1
601.4
725.8
Aggregate decode throughput vs concurrency (c1–c16) for the six quants; Q4_K_M leads at c16.
p95 time-to-first-token vs concurrency (c1–c16) for the six quants; lower is better.
Profile note: the table above is a short-context profile — CTX_SIZE=8192 PARALLEL=16 exposes n_ctx = 512/slot, with ~23-token prompts and 256-token
generations. Source: benchmarks/llamacpp-quant-benchmarks.md.
For how prefill latency scales with prompt length, see long-context TTFT below.
Long-context TTFT (single stream)
p95 time-to-first-token vs prompt/context length, single GPU tp=1, single stream
(CTX_SIZE=131072 PARALLEL=1, n_ctx/slot=131072, exact prompt token counts, 140
rows/quant, 0 failures).
Context tokens
Q4_K_M p50/p95 ms
Q4_K_M tok/s
IQ4_XS p50/p95 ms
IQ4_XS tok/s
MTP-graft p50/p95 ms
MTP-graft tok/s
512
91.8 / 94.1
188.3
87.1 / 88.7
177.4
86.0 / 87.8
184.0
1024
169.6 / 172.2
145.8
159.9 / 161.6
139.9
157.6 / 159.0
155.9
2048
341.8 / 346.1
97.9
316.1 / 318.7
112.0
307.6 / 308.9
113.9
4096
699.2 / 702.2
67.2
647.0 / 653.4
70.7
625.7 / 631.3
69.0
8192
1447.5 / 1458.0
33.7
1342.9 / 1348.5
39.8
1297.4 / 1304.0
40.9
16384
3009.7 / 3030.4
19.5
2793.2 / 2804.0
20.9
2709.8 / 2737.3
21.5
32768
6302.4 / 6313.0
9.7
5829.9 / 5853.1
10.5
5673.5 / 5696.6
10.7
Long-context p95 TTFT vs prompt length (512–32768 tokens) for Q4_K_M, IQ4_XS, and the MTP graft; single stream, tp=1.
p95 TTFT rises with prompt length (prefill cost) to ~6.3 s at 32k tokens for
Q4_K_M; the IQ4_XS body and IQ4_XS-MTP graft prefill slightly faster at every
length. Decode throughput falls from ~180–190 tok/s at 512 tokens to ~10 tok/s at
32k as the KV cache grows. Source: benchmarks/llamacpp-longctx-ttft.md.
Multi-token prediction (MTP)
The integrated IQ4_XS-MTP-graft-headQ6 artifact carries a native MTP draft head
on the IQ4_XS body. The release recommendation is adaptive: use MTP for
low-concurrency / single-user requests and keep LLAMA_SPEC_MAX_DRAFTING_SLOTS=1
so saturated batches fall back to target-only throughput. The MTP catalog
validates the published MTP profiles.
Next-token quality
Active MTP does not change the one-token, API-visible next-token
distribution. Mean BF16 KLD is 0.0731382442 nats over the 32-prompt top-64
probe (P50 0.0492811974 / P95 0.1593743642 / Max 0.3348258511), top-1 29/32. The
active draft-mtp vs target-only graft next-token KLD is 0.0 (32/32 identical).
Sequence-level behavior
A deterministic 8 prompt × 64 token probe (n_predict=64, temperature=-1, n_probs=64) compares the same GGUF running active native draft-mtp against
target-only.
Runtime
Client agg tok/s
Server decode tok/s
Draft acceptance
Exact 64-tok seqs
Token-position match
Target-only graft
172.57
~210 (per-request timing)
n/a
8/8 baseline
n/a
Fast active draft-mtp, n_max=2
233.81
325.70 (/metrics)
310/378 = 82.01%; per-position (0.884, 0.747)
6/8
478/512 = 93.36%
LLAMA_SPEC_VERIFY_SEQUENTIAL=1
164.34
204.88 (/metrics)
313/375 = 83.47%; per-position (0.884, 0.772)
5/8
455/512 = 88.87%
MTP single-stream decode speedup on the deterministic 8x64 probe: target-only vs active MTP graft, client and server decode.
Fast active MTP gives ~1.35× client throughput (233.81 vs 172.57) and
server decode 325.70 vs ~210, with 82.01% draft acceptance.
It matches 6/8 sequences exactly and 478/512 = 93.36% of token
positions; the two non-exact sequences first diverge at positions 60 and
34.
The LLAMA_SPEC_VERIFY_SEQUENTIAL=1 variant is both slower (164.34 /
204.88) and matches less (5/8, 88.87%; divergences at 25, 43, 60) — i.e. the
fast verifier is here both faster and more target-matching.
Note: MTP self-speculation is not bit-exact to target-only over long
deterministic generations. For workloads that require exact target
reproduction, run target-only; for throughput on open-ended work, MTP wins.
MTP is a single-user / low-concurrency win — it is not faster on saturated
batches, which is why the adaptive throttle falls back to target-only.
Sequence logprob comparability (API limitation)
llama.cpp omits candidate top_logprobs for most accepted speculative tokens, so
a naive "sequence KLD" looks enormous — that is an API/logprob-availability
artifact, not model quality. On the comparable subset where the emitted token
also matches, KLD is ≈ 0.001 nats (near zero):
Runtime
Paired positions
Comparable top-logprobs
Missing
Comparable same-token mean KLD (nats)
Max (nats)
Fast active draft-mtp vs target-only
512
12
500
0.0011215739
0.0040239103
Sequential verifier vs target-only
512
10
502
0.0012293172
0.0040239103
Full sequence KLD cannot be computed without changing the runtime to return
target verifier logprobs for accepted speculative tokens.
Draft-head distillation provenance
The meaningful KL-style metric for a standalone MTP draft head is teacher KL
against cached target hidden states, not standard base-model next-token KLD.
The draft head was trained and measurably improved across checkpoints —
teacher KL fell 1.92 → 1.50 → 0.76 and teacher argmax top-1 rose 0.54 → 0.69 →
0.89.
Draft checkpoint
Tokens
Windows
Teacher KL
Teacher argmax top-1
Teacher argmax top-5
Gold top-1
mtp-distill-kl-step500
22,637
128
1.9222025748
0.5360250917
0.7881786456
0.4425939833
snapshots-iq4-live-accepted-a/step-1000
58,754
413
1.5038724942
0.6876808388
0.8368451510
0.6764816013
snapshots-iq4-live-allrows-b/step-1000
32,264
2,048
0.7591610373
0.8908070915
0.9389722291
0.8076183982
MTP draft-head distillation progress: teacher KL falls (left) while top-1 agreement rises (right) across three checkpoints.
Teacher KL is measured against cached target hidden states (not base next-token
KLD). These training artifacts live under artifacts/mtp/ locally and are
referenced by name as provenance — they are not committed in this HF repo.
REASONING=off is the default because the model otherwise spends simple coding
prompts in reasoning_content before producing final content. See
benchmarks/llamacpp-q4-reasoning-off-fix.md.
Serving / quant / MTP profile catalogs:
configs/serving_profiles.yaml,
configs/quant_artifacts.yaml,
configs/mtp_profiles.yaml.
Benchmark environment
Hardware: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 97,887 MiB
VRAM, single GPU (tp=1). Driver 580.159.03, CUDA runtime reported by driver 13.0.
Dedicated MTP KLD eval config (2026-06-28): local llama.cpp CUDA server,
single RTX PRO 6000 Blackwell 96GB, tp=1, CTX_SIZE=8192, PARALLEL=1,
CACHE_RAM=0, REASONING=off.
llama.cpp build/commit:050ee92d04c2e1f639025786dea701c70e7d4204 (pinned for
the long-context TTFT run; earlier KLD/throughput sweeps were not separately pinned).
Reasoning mode: all benchmarks in this card ran with REASONING=off.
Provenance & reproducibility
Figures: regenerate with scripts/make_charts.py
→ assets/02_fidelity_ladder.png, assets/03_throughput_tps.png,
assets/04_ttft_p95.png, assets/05_mtp_draft_head_distill.png,
assets/06_mtp_tps.png.
Dedicated MTP KLD / sequence eval (2026-06-28):benchmarks/mtp-kld-eval-2026-06-28.md
and benchmarks/mtp-dedicated-kld-evaluation.md,
with raw evidence under benchmarks/raw/. Includes the
sequence-level probe and its logprob-availability limitation (full sequence KLD
is not computable from the current llama.cpp API response).
