Tess-4-27B — MLX 8-bit, with a working MTP head
An MLX conversion of
migtissera/Tess-4-27B
(Qwen3.6-27B base, dense, vision-capable) quantized to 8-bit (group size 64,
affine),
with the native multi-token-prediction head restored as a BF16
sidecar — which, at the time of upload, no other public MLX conversion of the
Qwen3.6 family ships.
A 4-bit build with the same MTP sidecar is at
Tess-4-27B-MLX-Q4 (16.1 GB).
Why the MTP sidecar matters
mlx-lm's qwen3_5 sanitize() unconditionally drops mtp.* tensors during
conversion, so every community MLX checkpoint of this family carries
mtp_num_hidden_layers: 1 in its config while containing zero MTP weights.
Check your model.safetensors.index.json, not the config — the config will
tell you the head is there when it is not.
This repo re-extracts the head from the original BF16 release and ships it at
mtp/weights.safetensors (15 tensors, BF16 — quantizing the head collapses
acceptance to ~0%).
Measured with
vllm-mlx 0.4.0
(
--enable-mtp, greedy envelope) on an M5 Max:
78% draft acceptance,
+19.5% decode throughput with acceptance-verified speculative decoding — no
output change vs. non-speculative greedy.
An honest caveat. MTP is real and it works, but it is worth less than it
sounds. On the same model and workload,
omlx
decodes at 18.4 tok/s with
no speculative decoding at all, versus vllm-mlx's
19.4 tok/s
with MTP (and ~17.3 without). A better runtime recovered nearly the
whole gain on its own. Take the head because it is free; do not architect around it.
Benchmarks, and what you gain by choosing 8-bit over 4-bit
Same harness, same 3500-token budget, same machine (M5 Max). The
Q4 build is the same
conversion at 4 bits, so this is a clean read on what the extra 14 GB buys you:
| Axis | Q8 (this repo) | Q4 | Δ |
|---|
| HumanEval+ pass@1 (50-problem subset) | 0.90 | 0.82 | +0.08 |
| MMLU-Pro-style reasoning (40q, 10 options) | 0.80 | 0.85 | −0.05 |
| Tool-calling (20 scenarios) | 0.85 | 0.85 | — |
| Browser action-selection (20 simulated pages) | 0.95 | 0.95 | — |
| RULER-style long-context (8k/16k/32k) | 0.80 | 0.73 | +0.07 |
| Deterministic VQA (10 items) | 1.00 | 1.00 | — |
| My hard screenshot test (fine-print OCR) | 2/2 | 2/2 | — |
| Decode, single stream | 17.3 tok/s | 28.3 tok/s | −64% |
| On disk | 30.4 GB | 16.1 GB | +14.3 GB |
The honest summary: 8-bit buys you ~8 points of coding and ~7 of long-context.
It buys you nothing on reasoning, tool-use, browser action-selection or vision —
those survive 4-bit intact. And it costs you 40% of your decode speed and 14 GB.
Take Q8 if you lean on long-context recall or code generation, and you have the
memory.
Take Q4 if your
workload is agentic (tool calls, short-to-medium turns, vision), or you are on a
32 GB machine — there it is close to free.
Small subsets: treat as directional, not as leaderboard numbers. The reasoning axis
showing Q4 ahead is well within the noise of a 40-question set — read it as "no
loss", not "Q4 is smarter".
Usage
1# omlx (recommended — faster engine, and the only one I trust with large images)
2omlx serve --model-dir <dir-containing-this-model>
3
4# vllm-mlx (if you want the MTP head to actually engage)
5vllm-mlx serve <this-repo> --mllm --enable-mtp --mtp-num-draft-tokens 1
6# MTP engages only on exactly-greedy requests: temperature=0, top_p=1, top_k=0, min_p=0
Anything that loads standard MLX safetensors will run this model; engines without
MTP support simply ignore the mtp/ sidecar.
Known ecosystem traps this conversion accounts for
- Do not load MLX-format Qwen3.6 checkpoints with mlx-vlm 0.6.4 — it
re-applies a
+1.0 RMSNorm shift to already-converted weights and produces
deterministic garbage, with no error and no warning. Use ≥0.6.5 or ≤0.6.3.
- Do not serve large images through vllm-mlx 0.4.0's batched MLLM path — it
silently drops image context and answers anyway, confidently and with fabricated
detail (0/2 on a hard screenshot test while inventing numbers; omlx scored 2/2).
Serve vision without continuous batching, or via omlx.
- Do not benchmark this model with a short generation cap. It is a reasoning
model: a 768-token cap does not truncate its answer, it truncates its
thinking, so it emits nothing and scores near zero on problems it can solve.
Budget ≥3500 tokens.
Provenance & credits
- Base model: migtissera/Tess-4-27B
by Migel Tissera — post-trained on long-context agentic traces atop
Qwen/Qwen3.6-27B. All model
capabilities are theirs; this repo is packaging.
- Conversion:
mlx_vlm convert (8-bit, gs64) + BF16 MTP re-attachment.
- License: Apache-2.0, inherited from the base model.
Converted and benchmarked as part of a local inference stack for a personal
Automated Agentic Software Factory — something I'm building solo and will make
publicly available after its limited-alpha phase. Full benchmark data, harnesses and
the conversion recipe:
https://huggingface.co/datasets/studioburnside/mlx-local-inference-benchmarks.
Questions, results and corrections all welcome in Discussions.
