MLX 8-bit conversion of barozp/Qwen3.8-27B-Opus-Distill-v2 -- the bugfix release of Qwen3.8-27B-Opus-Distill ("v1") -- for Apple Silicon. Sibling formats: the bf16 safetensors source above, the GGUF collection for llama.cpp/Ollama/LM Studio, the FP8 build for CUDA serving stacks (sglang/vLLM), and the 4-bit MLX repo.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

A community member (zxbc2023) reported a clean, deterministic bug on v1: certain stacked output-format constraints (e.g. "no prose" + "no markdown") sent the model into a non-converging self-verification loop, burning the entire token budget with zero visible output. Root cause traced to part of v1's training data being sourced from a dataset with reconstructed (not verbatim) reasoning traces. This release retrains on a rebuilt dataset -- every row traced to a verified genuine Opus source, every row scanned (30-way parallel LLM-judge pass over the full set) and dropped if it showed the same pathological pattern.

Bug fix validation

Re-ran the reporter's exact repro (temp=0, seed=1234, fib(n) with stacked constraints):

Test	v1	v2
"no prose" + "no markdown"	3000/3000 tokens, 0 visible output	87/4096 tokens, clean
"no prose" + "no comments"	3000/3000 tokens, 0 visible output	73/4096 tokens, clean
Generalization (different task)	--	43/4096 tokens, clean

All three converge quickly, </think> closes normally -- no loop.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness: 0-shot, loglikelihood (multiple-choice), chat template OFF, QUICK mode (--limit 500). Same protocol as v1, so the delta column is the meaningful signal.

Task	Metric	Base	v2	Delta
wikitext	word perplexity (lower better)	8.4335	8.3788	-0.055
mmlu	acc	0.8494	0.8476	-0.002
hellaswag	acc_norm	0.7420	0.7500	+0.008
arc_challenge	acc_norm	0.5880	0.6220	+0.034
gpqa_diamond	acc_norm	0.2323	0.4697	+0.237

General knowledge (MMLU) and language modeling (wikitext) are essentially unaffected -- in the same noise band as v1. Reasoning (ARC, GPQA) shows a large, real gain over base, not comparable to Qwen's own published GPQA number (thinking-mode-on, different harness) -- this is a same-protocol base-vs-distill delta.

Available MLX conversions

Repo	Bits-per-weight	Size	Use case
this repo (8-bit)	8.627	27.49 GB	near-lossless
Qwen3.8-27B-Opus-Distill-v2-MLX-4bit	4.695	14.97 GB	recommended all-rounder

Which one to pick:

Best quality with headroom -> 8-bit
Best size/speed balance -> 4-bit (default recommendation)
24 GB Mac -> 4-bit (cap large images with --max-pixels, see Vision); 36 GB+ Mac -> either
On llama.cpp/Ollama/LM Studio workflows -> use the GGUF repo instead

Conversion notes

Command: python -m mlx_vlm.convert --hf-path <source> --mlx-path <out> --quantize --q-bits 8 --q-group-size 64
Tooling: mlx 0.32.1 · mlx-vlm 0.6.8 · huggingface_hub 1.27.0; source commit 75e188fcd327
Output: 2180 tensors, 6 safetensors shards, quantization mode affine; the full conversion log ships in this repo as conversion_log.txt
Environment: Google Colab G4 (RTX PRO 6000 Blackwell 96 GB), MLX CUDA backend
As with the GGUF builds, the native MTP head (blk.64, the nextn.* decoder layer) is not usable here either -- mlx-vlm's loader strips mtp.* weights during conversion, so these checkpoints do not offer native-MTP self-speculative decoding

Vision

Unlike the GGUF builds (where the tower ships separately as mmproj), the vision encoder is inside these weights -- image input works out of the box, no extra file to load:

bash

# multimodal generation
python -m mlx_vlm.generate \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit \
  --prompt "Describe this image." \
  --image photo.jpg

Text-only usage does not need anything extra and runs fine without --image. High-resolution photos expand into many visual tokens (~12k observed); on unified-memory Macs, cap resolution when needed with --max-pixels 1003520.

Quick start

bash

# install
pip install -U mlx-vlm

# text-only chat
python -m mlx_vlm.generate \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --max-tokens 512

# multimodal
python -m mlx_vlm.generate \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit \
  --prompt "Describe this image." \
  --image photo.jpg

# OpenAI-compatible local server
python -m mlx_vlm.server \
  --model barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit

Training details (source safetensors model)

Base: Qwen/Qwen3.8-27B -- dense 27B, hybrid Gated-DeltaNet / full-attention, 64 layers
Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
Data: barozp/opus-reasoning-distill-v2 (11,716 examples, 100% verified-genuine Opus traces)
Run: 1 epoch (696 steps), lr 1e-4 cosine + 3% warmup, effective batch 16, MAX_SEQ 4096, bf16, ~4h15m on A100 80GB
Final validation loss: 0.4334
Vision + MTP: carried over byte-for-byte from the base checkpoint -- never trained

Source chain

The full Qwen3.8-27B Opus Distill family:

Qwen/Qwen3.8-27B — base model
barozp/Qwen3.8-27B-Opus-Distill-v2-GGUF — v2, GGUF quants
barozp/Qwen3.8-27B-Opus-Distill-v2-FP8 — v2, FP8 build
barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-4bit — v2, MLX 4-bit
barozp/Qwen3.8-27B-Opus-Distill-v2-MLX-8bit — v2, MLX 8-bit
barozp/Qwen3.8-27B-Opus-Distill-v2 — v2, BF16 weights
barozp/Qwen3.8-27B-Opus-Distill-GGUF — v1, GGUF quants
barozp/Qwen3.8-27B-Opus-Distill — v1, BF16 weights

This release: v2, MLX 8-bit (the card you are reading).

Thanks to zxbc2023 for the bug report that made this release possible.