A vision-enabled Ornith 1.0 35B model for Apple Silicon via MTPLX native
speculative decoding (MTP). The body is the
Shiftedx
mxfp4 + abliterated build; the MTP head has been re-quantized from its
original 5-bit to 4-bit affine (group 64) so the model loads on stock
MTPLX without a loader patch.
Why this derivative exists
The Shiftedx mxfp4-vision model has the best MTP acceptance of the
Ornith MTP variants I measured, but its MTP head is prequantized at
5-bit. Stock MTPLX (up to 2.1.0) miscomputes the 5-bit group size in
_infer_prequantized_group_size (integer-floor 32 // 5 = 6 instead of
6.4), infers group_size 60 instead of 64, and nn.quantize rejects
the layer at load:
ValueError: [quantize] The last dimension of the matrix needs to be
divisible by the quantization group size 60 ... shape (2048,4096)
That bug is filed as
youssofal/MTPLX#182
with a fix in
PR #183. This repo is the
model-side workaround: the MTP head is re-quantized to 4-bit affine
(group 64), which stock MTPLX loads natively (5-bit packing is the only
non-power-of-2 width that triggers the bug; 4-bit does not).
Transform
Per quantized MTP tensor (a weight / scales / biases triple):
dequantize at 5-bit, group 64 to bf16
re-quantize at 4-bit, group 64, affine
13 quantized triples transformed, 33 norm / layernorm tensors passed
through unchanged. The body (model shards) and vision tower are unchanged
from the Shiftedx source.
The 5-bit to 4-bit round-trip costs a small amount of draft-head fidelity:
mean abs error ~0.0006 per tensor against the original 5-bit dense, which
shows up as a modest acceptance drop at the deepest draft position (see
below).
Performance (measured)
The signal that matters for an MTP head is draft acceptance at the
chosen bit width: high acceptance at lower bits means the draft head
matches the target well without paying for extra precision. Wall-clock
tok/s is reported for completeness but varies with hardware state and is
not the figure of merit here.
Acceptance at depth 3, by MTP head bit width (same M5 Pro config)
Apple M5 Pro 64 GB, MTPLX 2.1.0 (stock, no patch), mtp_history_policy=committed
(CLI default), --draft-temperature 0, thinking OFF, warm.
This repo is the highest-acceptance Ornith MTP variant at 4-bit, and the
highest-acceptance variant overall that loads on stock MTPLX. At 4-bit it
reaches pos3=0.66, close to the 5-bit source's 0.76, while every other
4-bit variant is at 0.52 or collapses. The 5-bit source is marginally
higher but costs an extra bit and needs the MTPLX loader fix; this repo
gets most of the way there at lower precision with no patch.
The wang-yang source collapse (pos3=0.00) is not a quantization issue:
its shipped MTP draft head diverges from its own target body (264 of
1803 body leaf tensors differ from the vision variants; the MTP head is
bit-identical across variants). See the comparison repo
Jonandrop/Ornith-1.0-35B-MTPLX-Vision
README for the weight-diff evidence.
Wall-clock tok/s (this repo, for reference)
Depth
tok/s (e2e)
speedup vs AR
acceptance pos1/2/3
AR (--no-mtp)
64.7
1.00x
-
MTP depth 1
73.2
1.13x
0.893
MTP depth 2
68.5
1.06x
0.866 / 0.775
MTP depth 3
78.4
1.21x
0.899 / 0.768 / 0.661
tok/s (e2e) is generated_tokens / total_elapsed, wall-clock including
prefill. Varies run-to-run with thermal/contention state.
Why naive RTN, not AWQ or GPTQ
The 5-bit to 4-bit re-quantization here is plain round-to-nearest (RTN). I
tried two techniques that beat RTN for weight quality on standard LLMs
(AWQ, arXiv:2306.00978; GPTQ, arXiv:2210.17323). Both collapsed MTP
acceptance below RTN when applied to this draft head:
AWQ (activation-aware per-channel weight scaling): reduced MTP head
weight reconstruction error 13.6% vs RTN across all 10 standard Linears,
but acceptance collapsed to pos3 0.02 (vs RTN 0.66). Cause: AWQ scales
weights by s per input channel and must inverse-scale activations by
1/s at runtime. In a transformer draft head those rescaled activations
feed residual streams, RMSNorms, attention softmax, and the (RTN) MoE
experts, none of which are re-quantized with W*s, so the rescale leaks
and corrupts them. AWQ's rescale is only invisible when every consumer of
the rescaled activation uses the scaled weights.
GPTQ (sequential Hessian error compensation, no runtime rescale):
acceptance collapsed to pos3 0.00. Cause: a correct GPTQ needs the
per-column quantizer to exactly match MLX's group-64 affine grid; a
column-wise uniform quantizer misaligns the error compensation and
perturbs the final weights. Group-aware GPTQ is more work and the proven
margin at 4-bit is only ~0.1 to 0.2 perplexity (not acceptance), so the
expected upside is small.
The headline finding from this: weight reconstruction error is not
acceptance. A 13.6% weight-error win (AWQ) produced acceptance 30x worse.
RTN remains the best stock-loadable 4-bit MTP head here. The 5-bit source
(0.76 at pos3) stays marginally ahead but needs the MTPLX loader fix
(#182/#183).
OpenAI-compatible endpoint supporting text and image input via
POST /v1/chat/completions. The MTPLX CLI defaults to the
committed history policy this model requires.
