Ornith-1.0-35B — UD-Q8_K_XL (mlx-node)
8-bit base mixed-precision quantization of
deepreinforce-ai/Ornith-1.0-35B for Apple Silicon, using the
Unsloth Dynamic per-tensor bit allocation (without imatrix AWQ) via
mlx-node.
Ornith-1.0 is a self-improving family of open-source
agentic coding models. The 35B member is a Qwen3.5-VL-MoE (hybrid Gated-DeltaNet + full attention, 256 experts, vision-language) post-train.
| Original (BF16) | This Model |
|---|
| Size | ~68 GB | 36 GB |
| Format | SafeTensors (sharded) | SafeTensors (sharded) |
| Precision | BF16 uniform | Mixed 8/8/8/8-bit affine + BF16 |
All Variants
Benchmarked on a cool Apple M5 Max: median decode throughput over three 512-token generations, with a 60-second idle GPU cooldown after every generation. (Sustained decode on Apple Silicon is thermally sensitive — back-to-back benchmarking on a hot chip can understate throughput by 20–30%, so every model here was measured from a comparable cool start.)
Performance
Steady-state decode: 91.5 tok/s (1.5x vs BF16) on Apple M5 Max. Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of 35.9B total), so the active-weight footprint streamed per token is what matters.
Output Quality
Decoded-text quality was verified against the BF16 reference with a multi-judge review of the actual generated output (not a heuristic): a 4-turn factual chat plus a Python is_balanced() bracket-matching task. This UD-Q8_K_XL build produced coherent prose, correct facts, and a correct implementation — no runaway generation, repetition loops, or stray tokens — on par with full precision. (The 2-bit tier is intentionally excluded from this collection: it was the only width that showed coherence breakdown.)
Per-Tensor Quantization
| Weight | Bits | Rationale |
|---|
embed_tokens | 8-bit affine | KLD ~0.15 — very low sensitivity |
lm_head | 8-bit affine | KLD ~0.05 — safest tensor |
self_attn.q/k/v_proj | 8-bit affine | KLD ~1.5–2.9 — attention-sensitive |
linear_attn.in_proj_qkv/z | 8-bit affine | KLD ~2.9 — SSM input gates |
self_attn.o_proj | 8-bit affine | KLD ~1.5; row-independent qmv for T=0 exactness |
linear_attn.out_proj | 8-bit affine | KLD ~6.0 — worst tensor; kept high |
linear_attn.in_proj_a/b | 8-bit affine | tiny low-rank GDN projections |
switch_mlp.down_proj | 8-bit affine | "slightly more sensitive" than other FFN |
switch_mlp.gate_proj/up_proj | 8-bit affine | bulk of the expert budget |
Router gates (mlp.gate, shared_expert_gate) | 8-bit affine | MoE routing accuracy |
GDN params (A_log, dt_bias) | bf16 | state-space dynamics |
vision_tower.* | bf16 | vision encoder kept full precision |
Quantization Strategy
Built on
Unsloth Dynamic 2.0 per-tensor KLD analysis: sensitive layers (attention/SSM inputs, down_proj, embeddings/head) get higher bits, while the bulk of FFN expert weights are quantized to the base width.
self_attn.o_proj,
linear_attn.out_proj, the split low-rank GDN projections (
in_proj_a/b) and the MoE router gates are pinned to 8-bit affine (group_size 64). GatedDeltaNet state-space parameters and the vision encoder stay bf16.
Note: These ornith quants apply the Unsloth bit allocation without imatrix AWQ pre-scaling — ornith has no published imatrix, so the attention/SSM channels are quantized directly. Expect a small quality gap versus an imatrix-calibrated build at the lowest bit widths.
Architecture
| Parameter | Value |
|---|
| Total parameters | 35.9B (~3B active per token) |
| Hidden size | 2,048 |
| Layers | 40 (30 linear GatedDeltaNet + 10 full attention) |
| Attention heads | 16 (2 KV heads, GQA 8:1) |
| Head dimension | 256 |
| Experts | 256 per MoE layer, top-8 routing |
| Vocab size | 248,320 |
| Vision | yes (Qwen3.5-VL vision tower, kept bf16) |
| Max context | 262,144 tokens |
Usage
1import { loadSession } from '@mlx-node/lm';
2
3const session = await loadSession('./Ornith-1.0-35B-UD-Q8_K_XL-mlx');
4
5for await (const event of session.sendStream('Write a Python function to merge two sorted lists.', {
6 config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
7})) {
8 if (!event.done) process.stdout.write(event.text);
9}
How It Was Made
1mlx convert \
2 -i Ornith-1.0-35B \
3 -o Ornith-1.0-35B-UD-Q8_K_XL-mlx \
4 -q --q-recipe unsloth --q-bits 8
The Unsloth recipe's per-tensor bit tiers were applied without imatrix AWQ (no native ornith imatrix). 7-bit tiers are snapped up to 8-bit (MLX affine supports 2/3/4/5/6/8-bit).
Acknowledgments
- Unsloth — Per-layer KLD bit-allocation strategy (Dynamic 2.0)
- DeepReinforce — For the Ornith-1.0 model family
- Qwen Team — For the Qwen3.5 base architecture
- Apple MLX — For the Metal-accelerated ML framework
License
MIT (inherited from base model).
