NVFP4 (W4A4) quantization of deepreinforce-ai/Ornith-1.0-9B, produced with llm-compressor and validated serving in production with vLLM on NVIDIA GB10 (Blackwell, sm_121).

Why this exists: at the time of quantization, no NVFP4 checkpoint of Ornith-1.0-9B on the Hub was both (a) unmodified — not "abliterated" — and (b) loadable by vLLM. Existing community quants either used a quantization scheme (NVFP4_AWQ with separate pre-quantization scale tensors) that vLLM's compressed-tensors path does not support, or were exported with a flattened text-only config, dropping the multimodal structure (text_config/vision_config and the vision tower weights) that vLLM requires to load this architecture — even for text-only use. This checkpoint keeps the full multimodal structure of the original.

What's inside

Language-model Linear layers quantized to NVFP4 (FP4 E2M1, W4A4, group size 16, tensor-group scales) in compressed-tensors nvfp4-pack-quantized format
Vision tower and lm_head kept unquantized (BF16) — same exclusions as the reference community recipes
Full multimodal config preserved (Qwen3_5ForConditionalGeneration, text_config + vision_config)
~8.3 GB on disk (vs ~19 GB BF16 original)

Quantization recipe

Reproduced from the recipe published by maci0/Ornith-1.0-9B-abliterated-NVFP4 (same target hardware), applied to the original, unmodified checkpoint:

Item	Value
Tool	llm-compressor 0.12.0, compressed-tensors 0.17.1, torch 2.11.0+cu130
Method	GPTQ, scheme `NVFP4`
Targets / ignore	`targets="Linear"`, `ignore=["lm_head", "re:.visual."]`
Calibration	512 samples of `HuggingFaceH4/ultrachat_200k`, max_seq_length 2048
Build hardware	NVIDIA GB10 (Grace Blackwell, `sm_121`, unified memory) — ~20 min end to end

Note: llm-compressor does not export the preprocessing files of the multimodal stack; preprocessor_config.json, video_preprocessor_config.json and vocab.json were copied from the original checkpoint.

Serving with vLLM

bash

vllm serve riclara/Ornith-1.0-9B-NVFP4 \
  --served-model-name ornith \
  --max-model-len 262144 \
  --gpu-memory-utilization 0.25 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3 \
  --trust-remote-code

Use qwen3_xml as the tool-call parser on vLLM (qwen3_coder is the SGLang parser; using the wrong one degrades tool calling)
Requires a vLLM build that supports the compressed-tensors nvfp4-pack-quantized format
NVFP4 is hardware-accelerated on Blackwell GPUs
KV sizing gotcha: at the full 262K context, vLLM needs ~8 GB of KV pool headroom just to accept one max-length request — with --gpu-memory-utilization below ~0.25 (on a 128 GB unified-memory machine) the server fails at startup with a KV-cache error. Lower --max-model-len if you want a smaller footprint.
Ornith is a reasoning model with thinking enabled by default: it spends ~250 tokens of thinking before answering. Clients that set a small max_tokens will get content: null responses. This build exposes thinking in the reasoning field.
Recommended for agentic use: a mild repetition_penalty (e.g. 1.05) and, if your vLLM build supports it, repetition_detection sampling params — small reasoning models can enter non-converging thinking loops without them.

Measured performance (GB10, single stream)

Checkpoint	Weights read/token	tok/s
FP8 (community)	~11 GB	16.1
This NVFP4	~8.3 GB	23.1

Single-stream decode on GB10 is memory-bandwidth-bound, so speed tracks bytes read per token. The unquantized lm_head (248k vocabulary ≈ 2 GB read per token) is the main remaining cost — kept in BF16 deliberately, matching the reference recipes.

Validation

Functional validation only (no perplexity/benchmark suite was run):

Loads and serves on vLLM without patches (structure check passes, quant config recognized)
Correct code generation on Spanish and English prompts
Structured tool calling verified end-to-end (finish_reason: tool_calls, well-formed arguments)
Ran as a production coding-assist backend behind a LiteLLM proxy from day one

If you run quantitative evals against the BF16 original, contributions to this card are welcome.

Credits

DeepReinforce for Ornith-1.0-9B (MIT)
maci0 for publishing the NVFP4 recipe details for this model on the same hardware
The vLLM and llm-compressor projects