A distilled code-focused variant of Gemma-4 e2b, optimized for efficient local inference using GGUF format. This model is designed for coding assistance, reasoning, and structured generation tasks, with optional “thinking” mode enabled via chat templates.

Example usage:

For text only LLMs: llama-cli -hf nphearum/Gemma-4-e2b-CodeX-Distill-v1-GGUF --jinja
For multimodal models: llama-mtmd-cli -hf nphearum/Gemma-4-e2b-CodeX-Distill-v1-GGUF --jinja

📦 Available Model Files

gemma-4-e2b-it.Q8_0.gguf — Quantized model (Q8_0 for high quality)
gemma-4-e2b-it.BF16-mmproj.gguf — Multimodal projection (required for full functionality)

🚀 Features

Strong code generation & reasoning (CodeX-style distillation)
Long context support (tested up to 131k tokens)
Optimized for llama.cpp
Supports structured chat templates (Jinja-based)
Optional “thinking mode” for better reasoning traces

🖥️ Running with llama.cpp

Make sure you’re using a recent build of llama.cpp with:

Flash Attention enabled
Jinja/chat template support compiled

Start Server

bash

llama-server \
  -m gemma-4-e2b-it.Q8_0.gguf \
  --port 53281 \
  -c 131072 \
  --parallel 1 \
  --flash-attn on \
  --no-context-shift \
  -ngl -1 \
  --jinja \
  --chat-template-kwargs "{\"enable_thinking\": true}" \
  --mmproj gemma-4-e2b-it.BF16-mmproj.gguf

Key Flags Explained

-c 131072 → Enables long context (131k tokens)
--flash-attn on → Faster attention (requires compatible GPU)
-ngl -1 → Offload all layers to GPU
--jinja → Enables chat template rendering
--chat-template-kwargs → Activates thinking mode
--mmproj → Required for multimodal projection

Test Request

bash

curl http://localhost:53281/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {"role": "user", "content": "Write a Python function to reverse a linked list"}
    ]
  }'

🧠 Notes on Thinking Mode

When enable_thinking=true, the model may:

Produce intermediate reasoning steps
Improve structured problem solving
Slightly increase latency

Disable it if you need faster responses.

🦙 Running with Ollama

Important: ⚠️ Ollama Note for Vision Models, currently does not support separate mmproj files for vision models.

Create a Modelfile:

txt

FROM ./gemma-4-e2b-it.Q8_0.gguf

PARAMETER num_ctx 131072
PARAMETER num_gpu -1
PARAMETER stop "<end_of_turn>"

TEMPLATE """{{ if .System }}<start_of_turn>system
{{ .System }}<end_of_turn>
{{ end }}{{ if .Prompt }}<start_of_turn>user
{{ .Prompt }}<end_of_turn>
<start_of_turn>model
{{ end }}"""

# Optional: enable reasoning-style outputs
SYSTEM "You are a highly capable coding assistant with strong reasoning ability."

Build & Run

bash

ollama create gemma-4-codex -f Modelfile
ollama run gemma-4-codex

⚙️ Recommended Settings

Use Case	Context	GPU Layers	Notes
Coding assistant	32k–64k	Full (-1)	Best balance
Long reasoning	131k	Full	Needs high VRAM
Low VRAM setup	8k–16k	Partial	Disable flash-attn

⚠️ Limitations

Requires significant VRAM for full 131k context
Thinking mode increases latency
Multimodal projection file must match model variant

📜 License

Follow the original Gemma license and any additional terms from this distillation.

🙌 Credits

Base model: Google Gemma family
Distillation: Code-focused adaptation
Runtime: llama.cpp ecosystem

nphearum/Gemma-4-e2b-CodeX-Distill-v1-GGUF

Gemma-4-e2b-CodeX-Distill-v1-GGUF

📦 Available Model Files

🚀 Features

🖥️ Running with llama.cpp

Start Server

Key Flags Explained

Test Request

🧠 Notes on Thinking Mode

🦙 Running with Ollama

Build & Run

⚙️ Recommended Settings

⚠️ Limitations

📜 License

🙌 Credits

nphearum/
Gemma-4-e2b-CodeX-Distill-v1-GGUF