Ornith-1.0-9B — MXFP4 + MTP (vision), for AMD RDNA4 / vLLM

• Trunk — Ornith-1.0-9B quantized to MXFP4 (compressed-tensors, group size 32, symmetric). lm_head, embed_tokens, norms, and the vision tower are kept BF16.
• Draft head — the KL-distilled MTP head from protoLabsAI/Ornith-1.0-9B-MTP (BF16), grafted into a separate model-mtp.safetensors shard and marked unquantized in the quant config so the mixed-precision model loads cleanly.
• Speculative decoding — vLLM native mtp method, num_speculative_tokens=3. Lossless: the target verifies every drafted token, so the output distribution is unchanged — the head only buys speed.

Measured on AMD RDNA4

• MTP draft acceptance (n=3): ≈66% average across a mixed code + 6k-context run — per-position 0.82 / 0.65 / 0.53, mean acceptance length ~3.0 (max 4) — rising to ~85% (length ~3.6) on cache-warm short context. Lossless throughout.
• Tool-calling (qwen3_xml) and vision confirmed working.
• Throughput (TP2, 256 output tokens, per-user / aggregate tok/s):

Run it on AMD RDNA4

The settings that actually matter on RDNA4

• --attention-backend TRITON_ATTN — required on gfx1201.
• --speculative-config '{"method":"mtp","num_speculative_tokens":3}' — enables the grafted MTP head. n=3 maximizes throughput; n=1–2 maximize per-token acceptance. Tune per workload.
• --tool-call-parser qwen3_xml --reasoning-parser qwen3 — Qwen3.5-family tool-calling + reasoning split.
• --trust-remote-code — the qwen3_5 vision architecture.
• Single GPU works too: --tensor-parallel-size 1 and pass one render node (e.g. --device=/dev/dri/renderD128).

How it was built (reproducible)

1. MXFP4 quantize Ornith-1.0-9B with compressed-tensors (4-bit float, group 32, symmetric; lm_head / embed_tokens / norms / vision tower left BF16).
2. Graft the 15 mtp.* head tensors from protoLabsAI/Ornith-1.0-9B-MTP into a new model-mtp.safetensors shard and patch model.safetensors.index.json.
3. Mark the head unquantized — add its Linear modules (mtp.fc, mtp.layers.0.self_attn.*, mtp.layers.0.mlp.*) to quantization_config.ignore, so vLLM's compressed-tensors loader keeps the BF16 head as-is instead of expecting MXFP4 weight-scales. This is the one mixed-precision gotcha.

Credits

• DeepReinforce — Ornith-1.0-9B, the base model (MIT).
• protoLabs — Ornith-1.0-9B-MTP, the KL-distilled MTP draft head and its recipe (MIT).
• Qwen / Alibaba — the Qwen3.5 architecture the MTP head derives from (the head was initialized from Qwen/Qwen3.5-9B's mtp.* tensors).
• vLLM and compressed-tensors — serving stack and quantization format.
• Rob Smith (tcclaviger) — the RDNA4 vLLM base image that made gfx1201 serving possible. The capicua25x/vllm-rocm-rdna4 image this model runs on is a forward-port of his work — without it, none of this runs.

License