Model Card for gemma-4-E2B-it (QAT, GGUF)
A GGUF repack of Google's quantization-aware-trained gemma-4-E2B-it build,
packaged to run on device. It carries the text, vision and audio towers in one
self-contained file: the tokenizer and chat template are embedded, so nothing
ships alongside it and nothing calls out to a server at inference time.
This repository holds the repacked weights only, not training data or the
original checkpoint.
File format
.ggxf is GGUF plus the tensors and metadata keys this project adds, so these
files load in Gadeon and are not expected to load in other GGUF readers.
Model Details
Model Description
gemma-4-E2B is a multimodal decoder with a per-layer-embedding (PLE) design:
alongside the usual token embedding, each layer receives its own 256-wide input
gathered from a large per-layer table, gated and projected into the hidden
stream. Attention alternates four sliding-window layers to one full-attention
layer, the two using different head widths and RoPE bases, and the upper twenty
layers share their neighbours' key/value state instead of computing their own.
A vision encoder and a Conformer audio encoder project into the same embedding
space.
The upstream checkpoint is quantization-aware TRAINED, not post-hoc quantized:
Google trained it while simulating low-bit error, which is why its two-bit
layers hold up where a post-hoc conversion of the same layers does not.
- Developed by: Google (base model); GGUF repack by leok7v
- Model type: Multimodal (text, image, audio) causal language model
- Language(s): the languages of the base model
- License: Apache 2.0, inherited from the base model; see the
Gemma 4 license
- Repacked from model:
google/gemma-4-E2B-it-qat-mobile-transformers
Model Sources
Uses
Direct Use
On-device assistants and chat in offline or privacy-sensitive settings where
prompts must not leave the device.
Out-of-Scope Use
Not for tasks requiring verified factual accuracy, and not a basis for legal,
medical, financial, or safety decisions. Not intended for high-stakes automated
decisions without human review.
Bias, Risks, and Limitations
A language model predicts plausible text; it does not verify facts and has no
understanding of truth. Output can be confident and still be wrong, incomplete,
or biased. The base model's biases and limitations carry over; see the base
model card.
Recommendations
Treat every response critically and check important output against a trusted
source.
How to Get Started with the Model
This file does not load in upstream llama.cpp. 170 of its tensors use
Q2_0, a block type that exists in the PrismML/Gadeon ggml lineage and not in
upstream ggml, so a tool that does not know Q2_0 will fail to read it. It is
published for runtimes that carry that type.
The tokenizer (262144 tokens, 514906 merges, byte fallback and metaspace) and
the chat template are embedded in the file, so a runtime reads both from the
GGUF rather than from sibling files.
Technical Specifications
Model Architecture and Objective
| Property | Value |
|---|
| Objective | Autoregressive next-token prediction |
| Hidden size | 1536 |
| Layers | 35 (28 sliding-window + 7 full attention) |
| Attention heads | 8 query / 1 key-value (multi-query) |
| Head width | 256 sliding, 512 full |
| Sliding window | 512 |
| Shared-KV layers | 20 |
| Vocabulary | 262144 |
| Logit softcap | 30 |
| Vision encoder | 16 layers, native resolution |
| Audio encoder | 12 Conformer layers |
| Activations | fp32 / fp16 |
Weight format
2.665 GB, 1024 tensors, GGUF v3, 16384-byte tensor alignment.
| Type | Tensors | Covers |
|---|
| Q4_0 | 158 | attention, layers 0-14 MLP, the per-layer embedding table |
| Q2_0 | 170 | layers 15-34 MLP, output projection, audio tower |
| Q8_0 | 182 | vision tower, per-layer gate and projection |
| BF16 | 361 | norms, layer scalars, the vision position table |
| F32 | 153 | the checkpoint's own unconverted modules |
The integer codes transfer from the QAT checkpoint exactly rather than being
re-quantized: its INT4 codebook and offset are Q4_0's, and its INT2 codes are
Q2_0's under a negation. The only numerical difference from the upstream
weights is a per-block fp32 to fp16 scale cast.
Citation
Please cite the upstream Gemma model. See the base model card at
google/gemma-4-E2B-it-qat-mobile-transformers
for the canonical reference and license.
Model Card Contact
leok7v on Hugging Face.
