This model is an unmodified GGUF quantization of
nvidia/Gemma-4-26B-A4B-NVFP4, made with the llama.cpp conversion tool.
Please refer to the official NVIDIA repository for quality metrics.
As of now, this repository contains NVFP4 GGUF in two variants: NVFP4 and NVFP4_FP8 of Gemma 4 26B A4B, and a projector file.
It is recommended to use llamacpp with docker. You can start inferencing this model with the command below:
1docker run --rm \
2 --runtime nvidia \
3 --gpus all \
4 -v ~/.cache/huggingface:/root/.cache/huggingface \
5 -e HF_HUB_CACHE=/root/.cache/huggingface/hub \
6 -p 8080:8080 \
7 ghcr.io/ggml-org/llama.cpp:server-cuda13 \
8 -hf catlilface/Gemma-4-26B-A4B-NVFP4-GGUF:NVFP4 \
9 -c 4000
Adjust llama.cpp parameters to better fit your hardware.
Special thanks to
ynankani for his contribution, which made this quantization possible.
