A high-quality EXL3 quantization of ornith-ai/Ornith-1.5-397B, a 397B-parameter Mixture-of-Experts reasoning and agentic-coding model (MIT license) post-trained on top of Qwen 3.5 and Gemma 4.
This quant follows the same recipe as the earlier gigascake/Ornith-397B-EXL3-HQ-35bpw quant of Ornith-1.0 — same EXL3-HQ 3.5 bpw mixed-precision allocation, with all 512 experts preserved.
Quantized and benchmarked on 2 × NVIDIA RTX PRO 6000 (Blackwell Workstation, 96 GB each).
Quick facts
Base model
ornith-ai/Ornith-1.5-397B (MIT)
Format
EXL3 v1.4.2, --hq mixed precision
Average bitrate
3.51 bpw (excluding head)
On-disk size
~166 GiB (26 safetensors shards)
Max context
262,144 tokens
Architecture
Qwen3_5MoeForConditionalGeneration
Bit allocation
Component
Bits
Experts (up/gate/down_proj)
3 bpw
Linear attention (in_proj_qkv / in_proj_z / out_proj)
Generated directly from the full BF16 weights. This checkpoint was quantized from the
unquantized ornith-ai/Ornith-1.5-397B BF16 source (740 GB, 122 files) — not from a
previously-quantized or GGUF intermediate. The EXL3 --hq encoder calibrates against the
BF16 tensors and assigns each layer the bitrate that minimizes its quantization error, so
this is a single-generation lossy pass from the original precision rather than a re-quant.
The mixed-precision layout is the HQ encoder trading bits where they reduce error the most,
against the 192 GB target:
Experts at 3 bpw. The expert banks are ~99 % of the parameter count and are the most
redundant part of an MoE — any single token only routes through 10 of 512 experts. They
tolerate aggressive quantization with the smallest error-per-bit, so they absorb almost all
of the compression. This is what makes a 397B MoE fit in this envelope at all.
Attention, linear attention & shared expert at 5 bpw. These fire on every token and
carry the routing signal, context mixing, and language quality. They are the most
error-sensitive dense paths, so the HQ boost spends extra bits here to keep the per-token
quantization error low — the highest-leverage place to spend a few extra bits.
lm_head at 8 bpw. The head maps directly to the output logits over the full 248K
vocab, so any error there distorts the entire token distribution. Keeping it near-full
precision is the cheapest way to protect output quality.
Vision tower unquantized (BF16), embeddings & norms unquantized. Stored at source
precision, so zero quantization error on the vision path and the small normalization
tensors.
Why this fits 192 GB so cleanly: pushing the cheap expert bulk down to 3 bpw and keeping
only the error-sensitive dense paths (attention/router/head) at higher precision is what lets
the full model, plus a 1,048,576-token FP8 KV pool for four concurrent 262K contexts, land on
two 96 GB cards with ~1 GB + ~2 GB to spare. The bits follow the error.
What was measured here
No perplexity, KL divergence, or benchmark-vs-BF16 error was computed for this release.
Validation covered structural integrity only, and it passed cleanly: no NaN/Inf tensors,
all 60/60 layers present, all 512/512 experts present, tokenizer byte-for-byte identical to the
source, and key config values preserved. See the "How well does it work?" section above for
the (deliberately modest) quality framing.
VRAM footprint: the full checkpoint plus a 1,048,576-token FP8 KV pool (four 262K contexts) fits on this GPU pair with headroom to spare — roughly ~95 GB + ~94 GB in use, leaving about 1 GB free on one card and 2 GB free on the other.
Short answer: very well, subjectively — but this is not a benchmarked release.
This quant did well in a short, informal benchmark test I ran, but I have not run — and do not plan to run — a full benchmark suite (no SWE-Bench / Terminal-Bench / perplexity comparison against the unquantized weights is included here). The ~52 tok/s number above is a decode-throughput measurement only, not a quality score. Take the model on its own merits and evaluate it on your own tasks.
Notes
Reasoning model: assistant turns open with a <think> block before the final answer. Serve with a --reasoning_parser qwen3 so the chain-of-thought is returned in a separate reasoning_content field.
Tool calling: the model emits <tool_call> blocks that parse into the standard OpenAI-style tool_calls field — use --tool_call_parser qwen3_xml.
Vision: the vision tower is stored unquantized, so the checkpoint can still accept image/video tokens; for text-only use it simply sits idle.
Long context: YaRN is validated up to ~1M tokens (scaling factor 4.0) and is built into vLLM/SGLang.
Attribution & license
Quantization of ornith-ai/Ornith-1.5-397B, released by DeepReinforce AI under the MIT license. This EXL3 quant carries the same license forward. The mixed-precision HQ recipe matches the earlier Ornith-1.0 quant at gigascake/Ornith-397B-EXL3-HQ-35bpw.
