8-bit MLX build of the MTP drafter (assistant) for Google's Gemma-4 26B (A4B-it). Pair with the matching 8-bit target: shdennlin/gemma-4-26B-A4B-it-8bit.

This is not a standalone chat model — it's a draft model used during speculative decoding to propose candidate tokens that the target model verifies.

Lineage

google/gemma-4-26B-A4B-it-assistant
  ↓
mlx-community/gemma-4-26B-A4B-it-assistant-bf16
  ↓
shdennlin/gemma-4-26B-A4B-it-assistant-8bit   ← this model

Quantization Details

Field	Value
Method	MLX affine PTQ
Bits	8
Group size	64
Mode	affine
Tool	mlx-vlm @ `cbbc56f97861` (≡ 0.5.0)
Hardware	Apple M4 Pro, 64GB unified memory

Why 8-bit drafter

When the target model is 8-bit, the drafter should also be 8-bit so MTP's hidden-state comparison stays consistent. A bf16 drafter paired with an 8-bit target risks zero or near-zero acceptance rate.

Usage (mlx-vlm speculative decoding)

bash

mlx_vlm.server \
  --model shdennlin/gemma-4-26B-A4B-it-8bit \
  --draft-model shdennlin/gemma-4-26B-A4B-it-assistant-8bit \
  --draft-kind mtp \
  --draft-block-size 6 \
  --port 8006

License

Gemma Terms of Use.