The Gemma 4 31B it assistant converted to GGUF and is compatible with the
latest llama.cpp release (
b9549), which introduces support for Gemma 4 MTP (Multi-Token Prediction).
The GGUFs were produced using llama-quantize from the orginal Gemma 4 31b assistant
https://huggingface.co/google/gemma-4-31B-it-assistant1llama-server -m "gemma-4-31B-it-Q8_0.gguf" \
2--spec-draft-model "gemma4-31B-it-assistant-Q8_0.gguf" \
3--spec-type draft-mtp \
4--spec-draft-n-max 4
The results depend on your workload.
It works fine with multiple gpus as well and I've seen a 2x increase in inference.
The below example is for a 3 GPU setup.
1llama-server -m "gemma-4-31B-it-Q8_0.gguf" \
2--spec-draft-model "gemma4-31B-it-assistant-Q8_0.gguf" \
3--spec-type draft-mtp \
4--spec-draft-n-max 4 \
5--main-gpu 0 \
6--tensor-split 0.6,0.1,0.3
