CQ-Gemma-E4B-IT (GGUF)
This repository contains quantized
GGUF formats of the
Google Gemma 4 E4B Instruction-Tuned model, published by
CQ Systems.
These files are optimized for local inference on consumer hardware, particularly Apple Silicon (M-series Macs) and standard GPUs, using tools like
llama.cpp, LM Studio, and Ollama.
Available Files
We provide three different versions of the model to suit different memory and performance requirements.
| Filename | Size | Quantization | Recommendation / Notes |
|---|
| CQ_Gemma4_E4B_f16_Q4_K.gguf | 5.0 GB | Q4_K_M | 🏆 Recommended for most users. Best balance of speed, size, and quality. Easily fits in 8GB+ unified memory (e.g., M1/M2/M3/M4 Macs). |
| cq_gemma4_e4b_q8.gguf | 7.5 GB | Q8_0 | High quality 8-bit integer quantization. Nearly indistinguishable from the F16 baseline. Requires ~10GB+ RAM/VRAM. |
| CQ_Gemma4_E4B_f16.gguf | 14.0 GB | F16 | Baseline 16-bit float. Highest fidelity, highest memory footprint. Useful for re-quantization or research purposes. |
How to Use
Using llama.cpp
You can run this model directly in your terminal using the compiled llama.cpp CLI tool.
# Basic chat launch with the recommended Q4_K model
./llama-cli -m CQ_Gemma4_E4B_f16_Q4_K.gguf \
-c 8192 \
-n 1024 \
-p "You are a helpful assistant.\n\nUser: Write a short joke about saving RAM.\nAssistant:"
Using LM Studio / Ollama
- Download the .gguf file of your choice (we recommend CQ_Gemma4_E4B_f16_Q4_K.gguf).
- LM Studio: Drag and drop the downloaded file into your LM Studio model folder, or use the local import feature.
- Ollama: Create a Modelfile with the line FROM ./CQ_Gemma4_E4B_f16_Q4_K.gguf, then run ollama create cq-gemma-e4b -f Modelfile.
About the Base Model (Gemma 4 E4B)
Gemma 4 is a family of open, multimodal models built by Google DeepMind. The E4B variant is explicitly optimized for on-device processing, making it incredibly powerful for local deployment on laptops and mobile devices.
Key Capabilities
- Thinking Mode: Features a built-in reasoning mode that lets the model think step-by-step before answering.
- Massive Context: Supports a context window of up to 128K tokens.
- Parameter Efficiency: "E4B" stands for 4.5B effective parameters (8B total with Per-Layer Embeddings). This allows it to perform well above its weight class while keeping active memory lookups fast.
- System Prompts: Gemma 4 uses standard system, assistant, and user roles natively.
(Note: While the base Gemma 4 E4B supports audio and image inputs natively, these GGUF files primarily support text generation out-of-the-box via llama.cpp. Multimodal capabilities in GGUF require an accompanying mmproj vision/audio encoder file).
Thinking Mode Configuration
To enable the built-in reasoning capabilities, include the <|think|> token at the start of your system prompt. The model will then output its internal reasoning inside <|channel>thought\n ... <channel|> tags before providing the final response.
License & Acknowledgements
- Base Model: Google DeepMind
- License: Gemma 4 License
- Quantization: CQ Systems using llama.cpp
For full benchmark details, evaluations, and ethical considerations, please refer to the original
Google Gemma 4 E4B documentation.
