✅ the first NVFP4 quantization of ornith-ai/Ornith-1.5-9B

Checked against Hugging Face repository metadata 2026-08-20: NVFP4 builds exist for the 35B-A3B and 397B members of this family, and none for the 9B — which is the family's most-downloaded size (54,860 downloads on the official GGUF repo). Repository-listing comparison only; no third-party build was benchmarked here.

An NVFP4 checkpoint of Ornith-1.5-9B for NVIDIA Blackwell, built with NVIDIA ModelOpt and served through vLLM. 200 Linear layers packed to NVFP4, the lm_head deliberately held at FP8, and the vision tower left in BF16.

Metric	Result
Quantization	NVFP4 `MIXED_PRECISION` (ModelOpt 0.46.0)
Model size	7,866,372,304 B (7.33 GiB, single `model.safetensors`)
vs BF16 source	18,819,720,984 B → 41.8%
Tested hardware	NVIDIA GB10 (DGX Spark class), 121 GiB unified
Compute capability	sm_121
Runtime	vLLM `nightly-aarch64` 0.26.1rc1.dev245+ge2fa28594
Decode	41.68 tok/s vs 12.77 for BF16 — 3.26×
Correctness	✅ 3/3 — 391 / Tokyo / 366, `finish_reason=stop`
Vision	✅ 4/4 on four-quadrant spatial placement
`lm_head`	F8_E4M3 — verified in the safetensors header

Why this build?

3.26× the decode of BF16 on the same box, same vLLM flags, exclusive GPU, same-prompt warm-ups
41.8% of the BF16 size — 7.33 GiB against 17.53 GiB
The output head is FP8, not 4-bit. With tie_word_embeddings: false this model has a real standalone lm_head; leaving it at 4 bits is a measurable quality risk, so it was excluded from the FP4 group and that exclusion was read back out of the file, not assumed
The vision tower is untouched (BF16) — this is a multimodal model and the projector is not quantization-damaged

What is actually quantized

Read from the model.safetensors header, 1362 tensors:

dtype	count	what
`U8`	200	NVFP4-packed Linear weights
`F8_E4M3`	201	FP8 tensors — the 200 activation scales plus `lm_head.weight`
`BF16`	559	vision tower, embeddings, norms
`F32`	402	scales

lm_head.weight                            F8_E4M3   [248320, 4096]
lm_head.weight_scale / input_scale        F32
model.language_model.embed_tokens.weight  BF16      [248320, 4096]

⚠️ embed_tokens is BF16 because it is an nn.Embedding — ModelOpt's recipe targets Linear modules, so an embedding is never a candidate. That is expected, not an omission.

Quick start

hf download kingjones777/Ornith-1.5-9B-Vision-NVFP4 --local-dir ornith9b-nvfp4

bash

docker run --rm --ipc=host --gpus all -p 8000:8000 \
  -v $PWD/ornith9b-nvfp4:/model \
  vllm/vllm-openai:nightly-aarch64 \
  --model /model \
  --max-model-len 8192 --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.85 --max-num-seqs 1

Requires a vLLM build with ModelOpt NVFP4 kernels and Qwen3_5ForConditionalGeneration. The nightly aarch64 image above has both; older releases do not.

Measured — not estimated

Idle GPU, identical flags across arms, warm-ups on the same prompt as the samples, median of 5.

arm	5 raw samples (tok/s)	median	spread
NVFP4 (this repo)	41.68, 41.66, 41.66, 41.90, 41.88	41.68	0.25
BF16 control	12.78, 12.77, 12.77, 12.76, 12.75	12.77	0.03

Correctness asserted on content + reasoning_content with finish_reason recorded, on every arm. Vision scored on spatial placement — a four-quadrant colour image where each quadrant is queried separately, so a fluent-but-misplaced answer fails.

⚖️ On calibration — what the evidence does and does not support

This checkpoint was produced with awq_clip on 384 real calibration samples (320 UltraChat conversations + 64 real Python files). An uncalibrated ablation (max amax over the same 384 samples, same FP8 head, same packing) was built and measured alongside it:

	calibrated (this repo)	plain ablation
decode median	41.68	41.58
correctness	3/3	3/3
vision	4/4	4/4
exact-match vs BF16 (10 prompts)	0/10	0/10
mean Jaccard vs BF16	0.516	0.570

They are tied on every gate run here. The speed and size win comes from NVFP4 plus the FP8 head, not from the clip search — and this repository does not claim otherwise.

⚠️ That gate is three factual needles, a vision check and a 10-prompt divergence probe. It is a regression guard, not a capability benchmark. It shows the clip search did not separate from plain on these tests; it does not establish that calibration is worthless. Perplexity, KL divergence or a real benchmark suite might separate them, and none of those were run.

Not yet measured

Test	Status
Perplexity / KL divergence vs BF16	❓ not measured
MMLU-Pro, GPQA, GSM8K, HumanEval+	❓ not run
Context beyond 8192	❓ only 8192 served
Vision beyond quadrant placement (OCR, charts, documents)	❓ not measured
Throughput at batch > 1	❓ `--max-num-seqs 1` only
Non-GB10 Blackwell (B100/B200/RTX 50)	❓ untested
Independent reproduction	❓ none yet

Known issues

Needs a recent vLLM. Older builds lack either the ModelOpt NVFP4 kernels or the Qwen3_5ForConditionalGeneration architecture.
Reasoning parser did not auto-initialise on the tested image — reasoning_content came back empty and the reasoning text (including </think>) appeared inside content. Parse accordingly, or expect literal think tags in output.
No speculative decoding. config.json declares mtp_num_hidden_layers: 1, but the published safetensors contain no MTP weights — there is no draft head to load.
Single-file checkpoint, no index.json. Expected for an unsharded export, not a fault.

Source

ornith-ai/Ornith-1.5-9B, revision c927ad73. Architecture Qwen3_5ForConditionalGeneration — a hybrid stack (24 gated-deltanet + 8 full-attention layers), 32 layers, hidden 4096, vocab 248,320, tie_word_embeddings: false.

Independent results

None yet. If you run this build, please open a discussion with hardware, driver, vLLM version, exact flags, context, prompt-processing and generation tok/s, and peak memory. Independent reproductions will be listed separately from author benchmarks and carry more weight.

Acknowledgements

NVIDIA TensorRT Model Optimizer — the NVFP4 quantization and the HF checkpoint exporter used to produce this file.

vLLM — the serving runtime and its ModelOpt NVFP4 kernels; every number here was measured through it.

Ornith — the base model and its licence are theirs. This repository contributes quantization and measurement only.