Ornith-1.5-35B-A3B — MQ4R & MQ4RP quants for hipfire
Pre-quantized Ornith-1.5-35B-A3B (MoE, 35.9B total / 3B activated) in hipfire's
.mq4r container, in two recipes — file names reflect the recipe:
| File | Recipe | Tensor mix | Size |
|---|
ornith-1.5-35b-a3b-mq4r.mq4r | MQ4R — uniform MQ4 everywhere | 20901 MQ4 / 1 Q8 / 191 F16 | 18.70 GB |
ornith-1.5-35b-a3b-mq4rp.mq4r | MQ4RP — uniform MQ4 + protected fixed tier | 20671 MQ4 / 231 Q8 / 191 F16 | 18.78 GB |
Both share: embed_tokens at Q8F16; norms, A_log, dt_bias at F16; routed
experts (gate_up_proj, down_proj) at uniform MQ4.
- Upstream model: ornith-ai/Ornith-1.5-35B-A3B (MIT)
- Engine: hipfire — a Rust-native LLM inference engine for AMD RDNA GPUs
Architecture (matches upstream config.json): 40 layers, 256 experts top-8,
hidden_size 2048, head_dim 256, 262144 context, vocab 248320. Loaded by hipfire's
arch_id=6 path; the .mq4r wire extension triggers the automatic Redline PM4
route on single-GPU gfx1100/1151/1201 loads.
The two recipes
MQ4R is the plain recipe: uniform 4.25-bit MQ4 across every weight class —
attention, router, shared experts, conv1d, lm_head, routed experts.
MQ4RP ("+Protected") additionally pins the precision-critical fixed tiers at
Q8F16 so every token passes through full-precision gating:
| Tier | Tensors | Precision |
|---|
Routed experts (gate_up_proj, down_proj) | 20671 | MQ4 uniform |
Router (mlp.gate.weight, mlp.shared_expert_gate.weight) + shared expert (mlp.shared_expert.{gate,up,down}_proj) + linear_attn.conv1d (every-token dense path) | 231 | Q8F16 (protected) |
The suffix parallels upstream hipfire's mq4p grammar. Trade-off: RP costs
~80 MB and ~10% decode throughput on this build (see Verification).
Verification
Both files were validated on RX 7900 XTX (gfx1100, RDNA3) with hipfire build
b97a175c (branch serve/ornith-mq4rp-multislot-pm4seed), served from two
dedicated containers — one per quant — via the retained-PM4 route with q8 KV.
Streaming decode throughput, measured identically for both containers: 12
requests each (4 long-form prompts × 3 rounds), 512 max tokens, thinking
disabled (chat_template_kwargs.enable_thinking=false), exact token counts
taken from server usage, first-token → last-token wall clock, warm-up run
discarded. All 24 requests completed (GPU busy 100%, TTFT ≈ 30 ms):
| Recipe | Decode tok/s (median) | Range | Runs completed |
|---|
MQ4R (ornith-1.5-35b-a3b-mq4r.mq4r) | 234.6 | 230.5 – 236.3 | 12 / 12 |
MQ4RP (ornith-1.5-35b-a3b-mq4rp.mq4r) | 211.8 | 210.2 – 212.6 | 12 / 12 |
MQ4RP decodes ~10% slower than MQ4R: its protected router / shared-expert /
conv1d tiers sit on the every-token dense path at Q8F16 and cost bandwidth.
Known issue (uniform-MQ4 routed experts on this model): when reasoning is
enabled, long structured generations can drive the think span into a
repetition attractor before </think>; if the token budget ends inside an
open span the daemon fails closed with open think span at end of generation
(observed even at small max_tokens). Disable thinking for throughput runs or
add client-side retries; do not add server-side caps (intentionally absent).
Usage
1hipfire serve ~/.hipfire/models/ornith-1.5-35b-a3b-mq4rp.mq4r --kv-mode q8
2hipfire serve ~/.hipfire/models/ornith-1.5-35b-a3b-mq4r.mq4r --kv-mode q8
Requires an AMD RDNA GPU (gfx1100/1151/1201 class) with ROCm; the .mq4r
extension selects the PM4 route automatically on single-GPU loads. With q8 KV
the full 200k+ context fits a 24 GB card (~22 GB VRAM class).
Quantization format
All hipfire MagnumQuant formats are FWHT-rotated (incoherence processing) with an
8-byte affine or fp16-codebook group header at group size 256. MQ4 = uniform
4.25-bits-per-weight scalar quantization. Reproduced with
hipfire-quantize --format mq4 --no-kmap --no-q8-conv1d (MQ4R adds
--no-q8-router; MQ4RP uses local HIPFIRE_Q8_PARTIAL=router,shared_expert
protection of the fixed tiers).
Attribution
- Base model: ornith-ai/Ornith-1.5-35B-A3B by ornith-ai — MIT license. All credit for the base model belongs to its authors; this repo redistributes only a lossy quantization of it.
- Quantization format & engine: hipfire — Rust-native, no-Python-hot-path inference engine for consumer + datacenter AMD GPUs.
- For reference SKUs of other models (including graded mixed-precision
+P builds and E8-lattice .mfp4), see hipfire-models/qwen3.6-35b-a3b.
Quantized locally on an AMD Ryzen 9 host from the upstream safetensors; validated on RDNA3 (gfx1100).
