MTPLX 4-bit conversion of barozp/Qwen3.8-27B-Opus-Distill-v2 — the bugfix release of Qwen3.8-27B-Opus-Distill ("v1") — for speculative decoding on Apple Silicon via mtplx.

Sibling formats: the bf16 safetensors source above, the GGUF collection for llama.cpp/Ollama/LM Studio, the 3-bit MTPLX repo, and MLX-4bit / MLX-8bit for non-speculative MLX.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

Same bugfix as the 3-bit MTPLX sibling: v1's training data included reconstructed reasoning traces that induced a deterministic self-verification loop on stacked format constraints ("no prose" + "no markdown" → 3000/3000 tokens, zero visible output). This release retrains on a rebuilt dataset — every row traced to a verified genuine Opus source, 30-way parallel LLM-judge scan, pathological rows dropped.

The 4-bit MTPLX point is the best quality/size balance in the family: ~16.9 GB on disk, vision intact, and the highest verified speculative speedup in this collection (2.39x at depth 3). If you have the headroom, pick this over the 3-bit.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

Test	v1	v2
"no prose" + "no markdown"	3000/3000 tokens, 0 visible output	87/4096 tokens, clean
"no prose" + "no comments"	3000/3000 tokens, 0 visible output	73/4096 tokens, clean
Generalization (different task)	--	43/4096 tokens, clean

All three converge quickly, </think> closes normally — no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1.

Task	Metric	Base	v2	Delta
wikitext	word perplexity (lower better)	8.4335	8.3788	-0.055
mmlu	acc	0.8494	0.8476	-0.002
hellaswag	acc_norm	0.7420	0.7500	+0.008
arc_challenge	acc_norm	0.5880	0.6220	+0.034
gpqa_diamond	acc_norm	0.2323	0.4697	+0.237

General knowledge and language modeling are essentially unchanged; reasoning shows a large same-protocol gain over base.

Available MTPLX conversions

Repo	Bits	Size (on disk)	Use case
barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-3bit	3	~13.6 GB	tight unified memory
this repo (4-bit)	4	~16.9 GB (15.7 GiB)	recommended all-rounder

Which one to pick:

Best quality + highest speculative throughput → this 4-bit MTPLX (2.39x at D3)
Tightest fit / 24 GB Mac with large images → 3-bit MTPLX (1.63x at D2)
No speculation → MLX-4bit; llama.cpp → GGUF Q4_K_M / Q6_K

Conversion notes

Source: barozp/Qwen3.8-27B-Opus-Distill-v2 (bf16, barozp--Qwen3.8-27B-Opus-Distill-v2 local cache), forge-local via mtplx==2.9.1
Recipe: body_bits=4, body_group_size=64, body_mode=affine, mtp_policy=keep_bf16, quantized trunk + bf16 MTP sidecar
MTP contract: base_hidden_variant=post_norm, hidden_variant=post_norm, concat_order=embedding_hidden, mtp_position_mode=local, mtp_quant_group_size=64, mtp_quant_mode=affine
Output: 3 safetensors shards + model-vision.safetensors (879 MB, 333 tensors, bf16) + mtp.safetensors (810 MB, bf16 sidecar) + model.safetensors.index.json
Forged at: 2026-08-24T14:12:22+03:00 on Apple M5 Pro (18 CPU / 20 GPU, 24 GB unified memory) — tuned for 24 GB Macs · macOS 27.0 arm64, mtplx_runtime.json ships in-repo as provenance

Quality-gate nuance: an earlier 2048-token-budget verify run for this 4-bit checkpoint was flagged quality_failed (strict judge, long-form drift). The current in-repo verification uses the standard 512-token budget — the same budget used for every other MTPLX card in this collection — and passes cleanly at all depths. No weights were changed between runs; only the verify budget was normalized. The table below is the normalized 512-token result.

Vision

The vision encoder is inside these weights (stored as model-vision.safetensors, auto-loaded) — no separate mmproj to fetch:

bash

mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --prompt "Describe this image." --image photo.jpg --depth 3

Text-only needs nothing extra. Cap max-pixels if you hit unified-memory pressure on very high-res images.

MTPLX usage

bash

# install
pip install -U mtplx

# text-only chat (verified optimum is --depth 3 for this 4-bit)
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --depth 3 --max-tokens 512

# multimodal
mtplx run --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --prompt "Describe this image." --image photo.jpg --depth 3

# OpenAI-compatible local server
mtplx serve --model barozp/Qwen3.8-27B-Opus-Distill-v2-MTPLX-4bit \
  --depth 3 --port 8080

Defaults verified: temperature=0.6, top_p=0.95, top_k=20, recommended_profile=sustained.

Verification results

Verified locally with mtplx forge verify (MLX backend, 512-token budget, quality gate enabled). Depth 0 is plain AR.

Depth	tok/s	vs AR	Acceptance by position	Quality
0 (AR)	17.4	1.00x	—	pass
1	29.0	1.67x	95.2%	pass
2	38.1	2.18x	89.4% / 76.7%	pass
3	41.6	2.39x	85.4% / 67.2% / 55.4%	pass

Recommended depth: 3 (mtp_depth_wins at D3)
Verdict: mtp_depth_wins; quality_rejected=[], failure_reasons=[]
Hardware: Apple M5 Pro (18 CPU / 20 GPU, 24 GB) · macOS 27.0 arm64 (Apple Silicon, MLX), mtplx 2.9.1, artifact sha256:06854b44...
All depths passed the quality gate; D3 is both the fastest and the deepest verified depth for this checkpoint.

Training details (source safetensors model)

Base: Qwen/Qwen3.8-27B — dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
Final validation loss: 0.4334
Vision + MTP: carried over byte-for-byte from the base checkpoint — never trained

Source chain

Qwen/Qwen3.8-27B (base) → barozp/Qwen3.8-27B-Opus-Distill-v2 (LoRA finetune, safetensors) → this repo (MTPLX 4-bit conversion)

Thanks to zxbc2023 for the bug report that made this release possible.