Tess-4-27B — MLX 4-bit, with a working MTP head
A 4-bit MLX conversion of
migtissera/Tess-4-27B
(Qwen3.6-27B base, dense, vision-capable) — group size 64, affine,
4.695 bits per
weight, 16.1 GB —
with the native multi-token-prediction head restored as a
BF16 sidecar.
Built from the original BF16 release, not by re-quantizing an 8-bit checkpoint,
so there is no compounded quantization error.
Sized for a 32 GB Mac. The 8-bit build is at
Tess-4-27B-MLX-Q8 (30.4 GB).
What 4-bit actually costs (measured, not guessed)
Same harness, same 3500-token budget, same machine (M5 Max):
| Axis | Q8 | Q4 | Δ |
|---|
| HumanEval+ pass@1 (50-problem subset) | 0.90 | 0.82 | −0.08 |
| MMLU-Pro-style reasoning (40q, 10 options) | 0.80 | 0.85 | +0.05 |
| Tool-calling (20 scenarios) | 0.85 | 0.85 | — |
| Browser action-selection (20 simulated pages) | 0.95 | 0.95 | — |
| RULER-style long-context (8k/16k/32k) | 0.80 | 0.73 | −0.07 |
| Deterministic VQA (10 items) | 1.00 | 1.00 | — |
| My hard screenshot test | 2/2 | 2/2 | — |
| Decode, single stream | 17.3 tok/s | 28.3 tok/s | +64% |
The honest summary: reasoning, tool-use, browser action-selection and vision
survive 4-bit intact. Coding loses ~8 points and long-context loses ~7 — the
latter is the consistent casualty of quantization in this family, because error
compounds across a long attention span and short tasks hide it entirely. In exchange
you get 1.64× the decode speed and half the memory.
If your workload is agentic (tool calls, short-to-medium turns, vision), Q4 is close
to free. If it leans on long-context recall, pay for Q8.
⚠️ A warning for anyone benchmarking this. My first Q4 run scored 0.36 on
code and looked like a catastrophic quantization cliff. It was a 768-token
generation cap. For a reasoning model that does not truncate the answer — it
truncates the thinking, so the model never emits code at all. The longest
generation ended mid-sentence: "Or simply: return string.swapcase() I'll".
Re-run with a 3500-token budget: 0.82. If a quantization result looks like a
cliff, check your token budget before you believe it.
The MTP sidecar
mlx-lm's qwen3_5 sanitize() drops mtp.* tensors during conversion, so every
community MLX checkpoint of this family declares mtp_num_hidden_layers: 1 while
containing zero MTP weights. Check model.safetensors.index.json, not the
config.
This repo re-extracts the head from the original BF16 release and ships it at
mtp/weights.safetensors (15 tensors, kept BF16 — quantizing the head collapses
draft acceptance to ~0%, so it stays BF16 even in a 4-bit build).
Acceptance on a 4-bit base is lower than on 8-bit (the head was fitted against BF16
weights), and MTP is worth less than it sounds regardless: on the same model, omlx
decodes at 18.4 tok/s with no speculative decoding versus vllm-mlx's 19.4 with
it. Take the head because it is free; do not architect around it.
Usage
1# omlx (recommended — faster engine, and the only one I trust with large images)
2omlx serve --model-dir <dir-containing-this-model>
3
4# vllm-mlx (if you want the MTP head to engage)
5vllm-mlx serve <this-repo> --mllm --enable-mtp --mtp-num-draft-tokens 1
6# MTP engages only on exactly-greedy requests: temperature=0, top_p=1, top_k=0, min_p=0
Known ecosystem traps
- Do not load MLX-format Qwen3.6 checkpoints with mlx-vlm 0.6.4 — it re-applies
a
+1.0 RMSNorm shift to already-converted weights and produces deterministic
garbage, silently. Use ≥0.6.5 or ≤0.6.3.
- Do not serve large images through vllm-mlx 0.4.0's batched MLLM path — it drops
image context and answers anyway, with fabricated detail. Use omlx, or disable
continuous batching.
- Do not benchmark with a short generation cap. See the warning above.
Provenance & credits
- Base model: migtissera/Tess-4-27B
by Migel Tissera, post-trained atop Qwen/Qwen3.6-27B.
All model capabilities are theirs; this repo is packaging.
- Conversion:
mlx_vlm convert (4-bit, gs64) from the BF16 source + BF16 MTP
re-attachment.
- License: Apache-2.0, inherited from the base model.
Converted and benchmarked as part of a local inference stack for a personal
Automated Agentic Software Factory — something I'm building solo and will make
publicly available after its limited-alpha phase. Full benchmark data, harnesses and
the conversion recipe:
https://huggingface.co/datasets/studioburnside/mlx-local-inference-benchmarks.
Questions, results and corrections all welcome in Discussions.
