NVFP4 GGUF conversions derived from the Quantization-Aware Training (QAT) checkpoint of Gemma 4 26B A4B IT (google/gemma-4-26B-A4B-it-qat-q4_0-unquantized).

This repository provides multiple GGUF quantization levels, making the model usable with llama.cpp and other GGUF-compatible inference engines.

Model Overview

Gemma 4 26B A4B IT is a multimodal model built by Google DeepMind that handles text, image, and audio inputs and generates text output. It is designed for efficient on-device and server deployment.

Model Architecture

Property	Value
Architecture	Gemma4ForConditionalGeneration
Parameters	26B
Supported Modalities	Text, Image, Audio
Context Length	256K tokens (262144)
Vocabulary Size	262K (262144)

GGUF Files

File	Description
`gemma-4-26b-a4b-qat-nvfp4.gguf`	NVFP4 Quantization
`mmproj.gguf`	Multimodal projector (vision + audio)

A chat_template.jinja file is also provided for use with chat-based inference.

Usage

llama.cpp (CLI)

bash

# Run text-only inference
./llama-cli \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  -p "Explain quantum computing in simple terms." \
  --temp 1.0 --top-k 64 --top-p 0.95

llama-server (OpenAI-compatible API)

bash

# Text-only
./llama-server \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  --host 0.0.0.0 --port 8080

# Multimodal (image + audio)
./llama-server \
  -m gemma-4-26b-a4b-qat-nvfp4.gguf \
  --mmproj mmproj.gguf \
  --host 0.0.0.0 --port 8080

Multimodal (Image / Audio)

For image and audio inputs, use llama-server or llama-cli with the --mmproj flag pointing to mmproj.gguf. Refer to your inference engine's documentation for passing image/audio data alongside text prompts.

Modality order tip: For best results, place image content before text and audio content after text in your prompt.

Generation Parameters

Recommended parameters:

Parameter	Value
Temperature	1.0
Top-K	64
Top-P	0.95
BOS Token ID	2
EOS Token ID	1
Pad Token ID	0
Mask Token ID	4

Thinking Mode

Gemma 4 supports configurable thinking (reasoning) mode:

Enable: Include the <|think|> token at the start of the system prompt.
Output format: When thinking is enabled, the model outputs internal reasoning followed by the final answer:
```
<|channel>thought
[Internal reasoning]
<channel|>
[Final answer]
```
Disable: Omit the <|think|> token.

Key Features

Multimodal: Text, image, and audio understanding
Long Context: 256K token context window
Function Calling: Native support for structured tool use (agentic workflows)
Multilingual: Support for 140+ languages
Native System Prompt: Supports the system role for structured conversations

Acknowledgements

Original model: google/gemma-4-26B-A4B-it
QAT checkpoint: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
Technical report: Gemma 4 Technical Report (arXiv:2607.02770)

Citation

bibtex

@misc{gemmateam2026gemma4,
      title={Gemma 4 Technical Report},
      author={Gemma Team},
      year={2026},
      eprint={2607.02770},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.02770},
}

License

Apache License 2.0

WhiskyAKM/Gemma-4-26B-A4B-it-qat-NVFP4-GGUF

Gemma 4 26B A4B IT QAT — NVFP4 GGUF