Llamacpp imatrix Quantizations of gemma-4-E2B-it by san27079
This repository provides high-quality GGUF quantizations for Google Gemma 4 E2B-it (MoE architecture).
These quants were manually optimized using a custom Importance Matrix (imatrix) to ensure superior logic retention and grammatical precision, making them ideal for language learning and complex reasoning.
Deep Imatrix Calibration: The importance matrix was calculated over 4853 chunks (~2.5M tokens) using the wiki.train.raw dataset.
⚙️ Recommended Settings (LM Studio / llama.cpp)
- Flash Attention: ON (Mandatory for MoE stability and long-context performance).
- GPU Offload: Max (Offload all layers to VRAM for the fastest response).
- KV Cache: For MoE models, it is recommended to keep this at None/FP16. If you encounter Out-of-Memory (OOM) errors at 100k+ context, try reducing context length to 96k before enabling cache quantization.
- Imatrix File: The
my.imatrix file used for these quants is included in the repository for transparency and further experimentation.
🧬 Quantization & Testing Hardware
- CPU: AMD Ryzen 5 9600X (Zen 5)
- GPU (Testing): AMD Radeon RX 9070 XT 16GB
- RAM: 32GB (2x16GB) DDR5 6200MHz CL30
- Storage: 2TB Lexar NQ790 (NVMe PCIe 4.0)
- Base Weights: Converted from BF16 GGUF provided by
bartowski.
🚀 Performance Benchmarks
All tests were performed on AMD Radeon RX 9070 XT 16GB (ROCm/Windows) using a long sequence of 3603 tokens:
| Quantization | Avg. Speed (tokens/sec) |
|---|
| Q4_K_M | ~127.35 t/s |
| Q8_0 | ~123.29 t/s |
Note: High speed on 3600+ tokens demonstrates excellent MoE routing stability and optimized memory bandwidth on this hardware.
Maintained by san27079.
