You can deploy this model and start earning money today!

dynamicagency/
gemma-4-31b-it-oQ8e-mtp

Name: gemma-4-31b-it-oQ8e-mtp
Availability: InStock

Views

No views yet

gemma-4-31b-it-oQ8e-mtp

Full-precision google/gemma-4-31B-it quantized to 8-bit with imatrix calibration ("oQ8e") using oMLX's own quantization tooling, converted from Google's official weights (not the already-MLX mlx-community mirror).

This supersedes our earlier gemma-4-31b-it-oQ8 release, which shipped without MTP support.

Quantization details

Model type: gemma4
Bits: 8 (enhanced / imatrix-calibrated)
Group size: 64
Format: MLX safetensors
Calibration dataset: oqe_code_multilingual

Multi-token prediction (MTP)

Gemma 4's MTP is implemented as a separate small "assistant"/drafter model (not tensors baked into the main checkpoint), per Google's own design. Pair this model with dynamicagency/gemma-4-31b-it-oQ8e-mtp-drafter for speculative decoding. In oMLX, set on this model:

json

{
  "vlm_mtp_enabled": true,
  "vlm_mtp_draft_model": "<local id of the drafter model>"
}

Benchmarks

Run on an Apple M3 Ultra (256GB), full pipeline: guidellm speed, long-context sweep, lm-evaluation-harness intelligence suite, promptfoo practical suite. MTP/speculative decoding was active for these runs (drafter attached and verified via server logs before benchmarking).

Speed (guidellm)

Concurrency	Output tok/s mean	TTFT mean
1	22.0	2.4 s
2	17.7	4.6 s

Context-length sweep

Prompt tokens	TTFT median	Decode tok/s
1,024	4.3 s	32.1
4,096	17.1 s	17.9
16,384	1.2 min	17.3
32,768	2.5 min	15.8
65,536	5.6 min	13.3
98,304	4.3 min	11.6
131,072	timed out	—

131K-token prefill exceeded our harness's 30-minute per-level budget on this model/hardware combination; treat long-context (>~100K token) use with that in mind.

Intelligence (lm-evaluation-harness)

Benchmark	Score
gsm8k (n=50)	98.0% exact match
ifeval (n=50)	92.0% prompt-strict
bbh causal judgement (n=30)	70.0% exact match

Practical (promptfoo, deterministic assertions)

6/7 pass (coding 2/2, reasoning 2/2, instruction-following 2/2, summarization 0/1).