Mixed-precision quant of
trohrbaugh/Qwen3.8-27B-heretic-ara, produced with
oQ (oMLX v0.6.2). Vision tower intact — image + video input still works. MTP head preserved. Standard MLX safetensors — compatible with oMLX, mlx-lm, LM Studio, and any MLX-capable app.
Unlike uniform 3-bit quantization,
oQ is a data-driven mixed-precision quantizer that calibrates per-layer sensitivity and allocates bits where they matter most. Critical layers (embeddings, LM head, the most sensitive transformer layers) are automatically promoted to higher precision, while less sensitive layers stay at 3-bit.
With Lightning MTP enabled. For reference, with MTP disabled the same quant measures ~8.0 / 7.7 TG tok/s at 1k / 4k (13.4 / 14.9 GB) — on the lower-bit quants the MTP head mostly costs memory rather than adding throughput.
The original BF16 weights require ~55 GB. This oQ3e quant runs in ~13–15 GB on Apple Silicon while keeping the vision tower and MTP head intact — the lightest option in the oQe family.
1# oMLX
2omlx serve --model underlotus/Qwen3.8-27B-heretic-ara-oQ3e-mtp
1# mlx-lm
2from mlx_lm import load, generate
3
4model, tokenizer = load("underlotus/Qwen3.8-27B-heretic-ara-oQ3e-mtp")
5response = generate(model, tokenizer, prompt="Hello!", max_tokens=256)
6print(response)
Apache 2.0, inherited from base model.
