A mixed-precision quant of Qwen/Qwen3.8-27B for Apple Silicon, built with mlx-optiq at 5.63 bits per weight. 4bit names the dominant precision, not the average.

Tip: the 5bit is smaller and measures better overall — German prose is the one span where this build stays ahead.

Highlights

🖼️ Image input on all three stacks: optiq serve, mlx-vlm, vllm-mlx. Ships the base model's processor configs, which quantization pipelines commonly drop — without them, images are silently ignored.
📏 Measured weights: per-layer {4, 8} bit from exact KL vs the bf16 base. No structural rules — every tensor's bit-width is bought by measured sensitivity. Results in the table below.
🧠 Measured KV cache: per-layer bit-widths, zero tool-call flips. Ships as optiq/kv_config.json — details below.
🔧 Hardened chat template, adopted from unsloth. Accepts developer, merges system messages, guards tool-call arguments; renders byte-identically to the original (verified).
⚡ MTP speculative decoding, engine-agnostic. Head at mtp/weights.safetensors — the default path optiq serve and vllm-mlx both search.

How it compares

	uniform 8bit ⁠	OptiQ 5bit ⁠	OptiQ 4bit ⁠	OptiQ 4bit ⁠	uniform 4bit ⁠
Publisher	p4ik ⁠	p4ik ⁠	p4ik (this repo) ⁠	mlx-community ⁠	p4ik ⁠
Weights (GiB)	26.62	17.67	18.06	18.09	14.09
BPW¹	8.50	5.64	5.77	5.78	4.50
Allocation	uniform²	measured (bf16)	measured (bf16)	measured (u4)	uniform²
Split 4/5/8	all @8	100/262/136	270/–/228	237/–/261	all @4
Measured KV	—	✓	✓	—	—
Vision³	✓	✓	✓	OptiQ only	✓
MTP head³	✓	✓	✓	OptiQ only	✓
Hardened template	✓	✓	✓	—	✓
ΔNLL overall⁴	0.000 (anchor)	+0.019 ± 0.019	+0.029 ± 0.015	+0.040 ± 0.030	+0.038 ± 0.045
— German prose⁵	0	+0.023 ± 0.003	+0.019 ± 0.002	+0.022 ± 0.002	+0.039 ± 0.003
— tool-call spans	0	−0.001 ± 0.013	−0.008 ± 0.017	+0.004 ± 0.005	+0.013 ± 0.009
— thinking spans	0	−0.001 ± 0.014	+0.004 ± 0.013	−0.005 ± 0.021	+0.008 ± 0.013
Flips per 10k⁶	—	216	503	566	772
Divergence⁷	(anchor)	10.3	10.3	9.7	7.9

¹ Bits per weight, file-based: shard bytes × 8 / parameters, same formula for every column.

² Our uniform reference builds — full packages (bf16 vision, MTP head, hardened template), deliberately without measured allocation or KV config.

³ ✓ = works on all three stacks (optiq serve, mlx-vlm, vllm-mlx). Vision needs the base model's processor configs, which quantization pipelines commonly drop; the MTP head needs the engine-neutral path mtp/weights.safetensors. "OptiQ only": runs solely under optiq serve — that repo lacks the processor configs, and its MTP head sits on optiq's internal path that other engines do not search.

⁴ Paired next-token NLL over a 196k-token corpus (agentic transcripts with tool calls and thinking, German prose, WikiText) against the uniform 8-bit anchor; corpus and method are ours.

⁵ All German-prose deltas lie beyond 2 SE; every other ΔNLL row is within noise.

⁶ Tokens the 8-bit anchor is near-certain about (NLL < 0.05) that jump above NLL 0.5 — the failure mode that breaks tool-call syntax. Lower is better.

⁷ Free-running greedy decoding, 32 tokens from 168 held-out prompt windows of the NLL corpus: mean position of the first token that departs from the anchor's trajectory (higher is better). Share of trajectories still identical after 8 tokens: 46 / 48 / 46 / 35 %.

Use

Everything at once — image input, measured KV cache, MTP speculation:

bash

pip install mlx-optiq
optiq serve --model p4ik/Qwen3.8-27B-MLX-OptiQ-4bit --mtp \
  --kv-config "$(hf download p4ik/Qwen3.8-27B-MLX-OptiQ-4bit optiq/kv_config.json)"

Image input with mlx-vlm:

python

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

model, processor = load("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")
config = load_config("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")

prompt = apply_chat_template(processor, config, "Describe this image.", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"]))

Text-only with mlx-lm:

python

from mlx_lm import load, generate

model, tokenizer = load("p4ik/Qwen3.8-27B-MLX-OptiQ-4bit")
messages = [{"role": "user", "content": "Write a merge sort in Python."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt))

Quantization

bash

optiq convert Qwen/Qwen3.8-27B \
    --method optiq --reference bf16 \
    --target-bpw 5.0 --candidate-bits 4,8 \
    --n-calibration 67 --calibration-mix optiq_erweitert.jsonl \
    --skip-baselines

Property	Value
Method	`optiq` (exact per-layer KL vs the bf16 base)
Target BPW	5.00 (achieved 5.13 before packing overhead)
Effective BPW	5.63
Weights	18.06 GiB, 270 tensors @ 4-bit, 228 @ 8-bit
Group size	64
KV cache	measured, 16 layers, 5.00 bits average
MTP	int4 prequantized (as shipped by optiq), 29 tensors, 300 MiB
Vision tower	bf16, 879 MiB

This build replaced an earlier static (rule-based) allocation: a paired corpus comparison put the two at parity, and the measured allocation ships because it is documented evidence rather than convention. The per-layer KL table is in optiq/sensitivity.json, the final assignment in optiq/metadata.json.

KV cache

8-bit on layers 51, 55, 59, 63; 4-bit on the other twelve (5.00 bits average; only the 16 full_attention layers carry a KV cache — the 48 linear_attention layers hold a fixed-size Gated DeltaNet state).

Measured with paired next-token NLL over 196k tokens (agentic transcripts with on-policy thinking and tool calls, German prose, WikiText), each layer probed individually. At this allocation, tool calls, thinking and English text are statistically indistinguishable from a full 8-bit cache; German prose retains ~+0.4% perplexity. A flip check over near-certain tokens found zero flips inside tool-call spans — the 4-bit cache does not break tool-call syntax.

Files

File	Purpose
`model-*.safetensors`	Mixed-precision weights
`preprocessor_config.json`	Image preprocessing for mlx-vlm / vllm-mlx
`optiq/kv_config.json`	Measured per-layer KV bit-widths
`mtp/weights.safetensors`	MTP head — default path for optiq serve and vllm-mlx
`optiq/optiq_vision.safetensors`	Vision tower, bf16
`optiq/sensitivity.json`	Measured per-layer KL table (4/8-bit vs bf16)
`optiq/metadata.json`	Final per-tensor bit assignment

Sampling

From the base model card, unchanged:

Parameter	Thinking	Instruct
`temperature`	1.0	0.7
`top_p`	0.95	0.80
`top_k`	20	20
`min_p`	0.0	0.0
`presence_penalty`	0.0	1.5
`repetition_penalty`	1.0	1.0

Thinking mode is on by default and can be disabled per request. Context is 262,144 tokens natively, extensible to 1M with YaRN.