File	Size	Min RAM	Recommended For
`gemma-4-E2B-it-Q4_0-imat.gguf`	3.4 Go	5 Go	Desktop / laptop
`gemma-4-E2B-it-Q4_0.gguf`	3.4 Go	5 Go	Desktop / laptop
`gemma-4-E2B-it-IQ4_XS-imat.gguf`	3.3 Go	4.5 Go	Best quality/size ratio ✅
`gemma-4-E2B-it-Q3_K_M-imat.gguf`	3.2 Go	4 Go	Good balance
`gemma-4-E2B-it-Q3_K_M.gguf`	3.2 Go	4 Go	Good balance
`gemma-4-E2B-it-Q3_K_S-imat.gguf`	3.1 Go	4 Go	Mobile mid-range
`gemma-4-E2B-it-IQ3_S-imat.gguf`	3.1 Go	4 Go	Mobile mid-range
`gemma-4-E2B-it-Q2_K-imat.gguf`	3.0 Go	3.5 Go	Low-end devices

Not sure which to pick? Start with IQ4_XS-imat on PC, and Q3_K_S-imat or Q2_K-imat on mobile.

What is Quantization?

Quantization reduces the precision of a model's numerical weights — for example from 16-bit floating point down to 3 or 4 bits per value. This dramatically shrinks the file size and RAM usage, making it possible to run large language models on consumer hardware.

The trade-off is a small loss in output quality, which is more noticeable at lower bit levels (Q2) and barely perceptible at higher ones (Q4).

imatrix (importance matrix) is a technique that analyzes which weights matter most for the model's behavior and preserves them more carefully during compression. The -imat variants in this repo use this method and consistently outperform standard quantizations of the same bit level.

Quick Start — Desktop / Laptop

1. Download llama.cpp

Pre-built binaries for Windows, macOS, and Linux are available on the llama.cpp releases page. Download the latest release for your platform.

2. Download a model file

Pick one file from the list above and download it from this repository.

3. Run

bash

./llama-cli \
  -m gemma-4-E2B-it-IQ4_XS-imat.gguf \
  -p "<start_of_turn>user\nHello, how are you?<end_of_turn>\n<start_of_turn>model\n" \
  -n 512 \
  --temp 1.0 --top-p 0.95 --top-k 64 \
  --no-display-prompt

Run as a local chat server (OpenAI-compatible API)

bash

./llama-server \
  -m gemma-4-E2B-it-IQ4_XS-imat.gguf \
  --port 8080 \
  --ctx-size 4096

Then open http://localhost:8080 in your browser for a built-in chat interface, or connect any OpenAI-compatible client to http://localhost:8080/v1.

Quick Start — Ollama (Easiest on Desktop)

Ollama is the simplest way to run GGUF models locally with a single command.

bash

# 1. Install Ollama (Linux/macOS)
curl -fsSL https://ollama.com/install.sh | sh

# 2. Create a Modelfile
cat > Modelfile << 'EOF'
FROM ./gemma-4-E2B-it-IQ4_XS-imat.gguf

TEMPLATE """<start_of_turn>system
{{ .System }}<end_of_turn>
<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
{{ .Response }}<end_of_turn>
"""

SYSTEM "You are a helpful assistant."
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER top_k 64
PARAMETER num_ctx 4096
EOF

# 3. Import and run
ollama create gemma4-e2b -f Modelfile
ollama run gemma4-e2b

Quick Start — Mobile (Android)

Several Android apps support GGUF models and run inference fully offline:

MLC Chat — open source, supports GGUF
ChatterUI — simple chat UI, llama.cpp backend
Pocketpal AI — polished interface, easy model import

General steps:

Transfer the GGUF file to your Android device (via USB or a file manager app)
Open the app and point it to the downloaded file
Start chatting — no internet connection required

Recommended files for mobile:

Device Type	Recommended File
Flagship (8+ Go RAM)	`IQ4_XS-imat` or `Q3_K_M-imat`
Mid-range (4–6 Go RAM)	`Q3_K_S-imat` or `IQ3_S-imat`
Low-end (3–4 Go RAM)	`Q2_K-imat`

Chat Template

Gemma 4 uses the following conversation format. Make sure your client applies it correctly:

<start_of_turn>system
Your system prompt here.<end_of_turn>
<start_of_turn>user
User message here.<end_of_turn>
<start_of_turn>model

Most modern clients (llama.cpp, Ollama, LM Studio, etc.) handle this automatically when loading a Gemma 4 GGUF file.

Recommended sampling parameters:

Parameter	Value
Temperature	1.0
Top-P	0.95
Top-K	64

Model Capabilities

Gemma 4 E2B is a 2.3B effective parameter multimodal model (text + image input, text output). Key strengths:

Multilingual — trained on 140+ languages, strong on French, English, Arabic, and more
Reasoning — configurable thinking mode for step-by-step problem solving
Coding — solid performance on code generation and completion
Long context — up to 128K token context window
Compact — designed from the ground up for on-device deployment

Note: Image input requires a multimodal-capable runtime. The GGUF files in this repo support text-only inference via llama.cpp.

Quantization Details

Step	Tool	Notes
HF → GGUF	`convert_hf_to_gguf.py`	F16 base file
Imatrix computation	`llama-imatrix`	Calibration dataset
Final quantization	`llama-quantize --imatrix`	Per variant

Base model: google/gemma-4-E2B-it (Apache 2.0)

License

These quantized files inherit the license of the original model: Apache 2.0. See the Gemma 4 license for full terms.

Shariyat/gemma-4-E2B-it-GGUF

Gemma 4 E2B Instruct — GGUF Quantizations

Available Files

What is Quantization?

Quick Start — Desktop / Laptop

1. Download llama.cpp

2. Download a model file

3. Run

Run as a local chat server (OpenAI-compatible API)

Quick Start — Ollama (Easiest on Desktop)

Quick Start — Mobile (Android)

Chat Template

Model Capabilities

Quantization Details

License

Shariyat/
gemma-4-E2B-it-GGUF