The circulating MLX quants of Hy3 drop the multi-token-prediction layer (layer 80) because mlx-lm's converter doesn't map it. Their configs still say num_nextn_predict_layers: 1, but the weights aren't there, so speculative decoding can't run.

Backbone: the 34 shards from XavierLocalAI/Hy3-4bit, unchanged. Checked before reuse: pass-through tensors are bit-identical to the official bf16 weights, and dequantization error sits at the expected bound for affine 4-bit.
Layer 80 (MTP head, in model-mtp-00001.safetensors): quantized from the official tencent/Hy3 bf16 weights with mlx.core.quantize, same recipe as the backbone (affine, group size 64, 4-bit, router gate 8-bit). 99% of elements reconstruct within half a quantization step of the originals.
Config: adds the layer 80 router gate override so the quant block matches the weights.

Serving

Runs on vLLM TPU (tpu-inference) with speculative decoding:

vllm serve enyoukai/Hy3-4bit-mtp-mlx \
  --trust-remote-code --enable-expert-parallel \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

Measured on a v6e-8: draft acceptance length about 1.5 at temperature 0.7 with batch 8, about 1.7 single stream. Same ballpark as the preview-era quants got from their own MTP head.

Credits

Base model: Tencent (tencent/Hy3, Apache-2.0)
Backbone quant: XavierLocalAI / bicVanYonk
MTP layer quant: this repo

enyoukai/Hy3-4bit-mtp-mlx

Hy3-4bit-mtp-mlx

Contents

Serving

Credits

enyoukai/
Hy3-4bit-mtp-mlx