Ornith-1.5-35B-A3B — oQ6 / group_size 128 (vision + working MTP)
A 6-bit oQ quantization of shisa-ai/Ornith-1.5-35B-A3B-MTP
for oMLX on Apple Silicon, keeping both the vision
tower and a functioning MTP head, and quantized at group_size 128 rather than the
usual 64.
Two independent problems with the common Ornith MLX quants:
1. Stock Ornith's MTP head does nothing. Measured on an M2 Max under oMLX 0.6.2:
raw (MTP off)
MTP on
gain
Qwen3.6-35B-A3B oQ6
72.50 tok/s
80.11
+10.5%
Ornith-1.5 oQ5 (stock head)
72.68 tok/s
72.81
+0.2%
Raw autoregressive speed is identical; stock Ornith's MTP verify forward costs almost
exactly the 1.77× it produces, so speculative decoding is break-even. Shisa AI's source
fixes this by replacing the native head with a Qwen3.6-initialized one trained by
full-vocabulary KL distillation. This build starts from that fixed source.
Most published Ornith oQ builds list base_model: ornith-ai/Ornith-1.5-35B-A3B — i.e. the
stock head — so their MTP is largely inert.
2. group_size=64 costs ~9%. Against
pyros-vault/Ornith-1.5-35B-A3B-oQ6e-fixed-mtp,
which uses the same fixed head and is otherwise identical (6-bit affine, vision, 42 MTP
tensors, 6.89 vs 6.87 bpw) but group_size=64:
6 paired A/B rounds, server restarted before every measurement
per-pair delta: [+6.0, +11.5, +7.6, +9.9, +6.5, +10.9] %
mean +8.7% stdev 2.1% this build ahead 6/6
MTP acceptance was the same in both (78–84%, ~2.1 tok/cycle), so this is not better
drafting. At equal bits-per-weight, group_size=64 creates twice as many scale/bias arrays
and the dequantization path pays double the per-group overhead. The cost is kernel overhead,
not memory bandwidth — which is why reducing bpw (oQ5, target_bpw) did not help.
MLX supports group sizes 32/64/128 only, so 128 is the maximum.
Build recipe
python
1from omlx.oq import quantize_oq_streaming
2quantize_oq_streaming(3 model_path="shisa-ai/Ornith-1.5-35B-A3B-MTP",4 output_path="Ornith-1.5-fixedmtp-oQ6-lean",5 oq_level=6, group_size=128,# 128, not the default 646 dtype="float16",# M2/M1 have no native bf167 preserve_mtp=True, text_only=False,# keep the MTP head and the vision tower8 target_bpw=6.5, hard_cap_bpw=6.7,9 auto_proxy_sensitivity=True,# required: 70GB exceeds live calibration memory10)
Usage (oMLX)
MTP must be enabled explicitly — it is off by default and fails silently:
Verify it actually engaged with grep "MTP\[" ~/.omlx/logs/server.log — look for
accept= and tok/cycle=. Absence means it is not running.
Note mtp_enabled and vlm_mtp_enabled are mutually exclusive; setting both makes the
whole settings file fail validation and silently revert to defaults.
Measured behaviour
Tested against Jundot/Qwen3.6-35B-A3B-oQ6-fp16-mtp on the same machine:
Decode speed: statistically tied. 6 paired rounds: mean −3.1%, σ 8.7%, ahead in 3/6.
Not distinguishable.
Tool-calling reliability: better. On a multi-tool prompt requiring correct tool
selection plus typed arguments (days_ago int, extensions array), this build was
6/6; Qwen3.6 emitted no tool call on 4 of 6 runs (2/6).
Agentic suite (selection, typed args, multi-turn after a tool result, correct refusal
when no tool applies): 5/5.
Vision verified; instruction-following clean (finish=stop on a one-sentence constraint).
Limitations
Measured on one machine (M2 Max 64 GB, macOS 26.6) with one runtime (oMLX 0.6.2).
Not verified on other Apple Silicon or other MLX runtimes.
That machine shows ~±3% short-term benchmark noise and a slow ~20% decline in absolute
throughput over ~30 minutes of continuous load, so all comparisons above are paired and
interleaved. Treat single-run numbers from any source with suspicion.
Ornith's adaptive MTP depth controller ramps slowly: completions ≤128 tokens show roughly
10% less benefit than longer generations.
Quality checks here are behavioural (tool calls, instruction following, vision), not
perplexity or standardised benchmarks. No claim is made about quality relative to
higher-bit quants.
