36% closer to the original model than the community Q4_K_M — at the same file size. The most faithful Gemma-4-E2B quant we've measured, on every arm we tested. Every number reproducible. Receipts inside.

Gemma-4-E2B Q4_K_M GGUF built with calibration-aware quantization: −36.0% code/math KLD and −27.2% general KLD to the original bf16 model versus the pinned community Q4_K_M baseline, at the same file size (−0.03%). A second 2.86 GB phone artifact is 17.5% smaller and still −31.8%/−26.7% versus that same-size baseline. Every number on this page is measured, pinned, and reproducible from the provenance table below.

Model Overview

Description

This repository contains two GGUF quantizations of google/gemma-4-E2B-it produced with fraQtl's calibration-aware per-tensor quantization: the same llama.cpp Q4_K machinery everyone uses, with per-tensor precision decisions driven by a calibration analysis instead of fixed defaults. The claim is smarter bits, not more bits.

Artifact	Bytes	True bpw (all tensors)	KLD code/math (vs original bf16)	KLD general	Verdict vs baseline
`Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf` (iso-size)	3,461,675,328	5.93	0.05846 (3-run, drift 0.0)	0.09646 (3-run, drift 0.0)	−36.0% / −27.2%
`Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf`	2,856,122,688	4.89	0.06231	0.09716	−31.8% / −26.7% (and 17.5% smaller)
bartowski Q4_K_M (pinned baseline)	3,462,678,272	5.93	0.09139 (3-run)	0.13249 (3-run)	—

Fidelity metric: symmetric top-20 KL divergence on Q8 teacher support, mean over positions; teacher is a Q8_0 quantization of the ORIGINAL bf16 checkpoint. Lower is better; this measures how faithfully the quantized model reproduces the original model's token distributions.

On-device: what it does on an iPhone (airplane mode)

These are unedited screenshots from our demo assistant running this model family entirely on an iPhone — on-device speech recognition, this model for understanding, deterministic Swift code executing the actions. The airplane icon in every status bar is the point: zero network at runtime.

The proof pair: the app says it booked — the Calendar shows it did


Voice-in booking with receipt chip	The real event in Apple Calendar
Spoken request (note the waveform icons): "Find some time on Thursday for VC meeting." The model finds the free slot, the code layer writes the event, a receipt chip states exactly what was written — and the reply is read aloud by on-device TTS ("Speaking — tap to stop").	Apple Calendar, same phone, still in airplane mode: "VC meeting", Thursday Jul 30, 11:30–12:30 — the exact event the chip claimed. Note the "◀ fraQtl" breadcrumb: navigated straight from the app. The app cannot claim a write that did not happen.

Scope, receipts, and speed


States its own limits	Books from a typed request
Asked about email, it answers "I do not have access to your emails" with the context tag `no connectors`. Then a two-step instruction (move + alert) executes as two real calendar writes.	Typed request, same behavior: finds a free 90-minute block and writes the event, receipt chip naming calendar and time.
Live receipts panel	Benchmark gate pass
The receipts panel: model sha256-verified on load, live decode rate, resident memory, thermal state, network status.	In-app benchmark, 256 tokens greedy: 21.6 tok/s sustained* against a stated gate of ≥15 tok/s — the number ships with its pass condition.*

Dijkstra's algorithm in C++, generated offline at ~11 tok/s interactive.

It also has opinions. (Every answer still carries its context tag.)

Measured on an iPhone-16-class device: ≈516 MB resident · ~12 tok/s interactive (last turn, includes prefill) · 21.6 tok/s sustained on the 256-token greedy benchmark (gate ≥15: PASS) · sha256-verified artifact · llama.cpp b10075 · Metal · mmap · airplane mode throughout.

What it can do (all offline): voice input and spoken replies, calendar questions, find-a-slot-and-book, move/retitle events, chained reminders, contacts-aware event titles, reading documents and photos, general knowledge, code generation.

What it cannot do — by design, and labeled as such in the app: no email, no messages, no network calls of any kind (there are no connectors, and it says so when asked); no action without a receipt chip — if a write fails, no chip appears; and the model never announces an action itself — a deterministic code layer executes and reports.

Artifact note: the demo runs Gemma-4-E2B-it-fraQtl-SmartEdge-IQ3XXS.gguf — in this repository now, 2.45 GB, sha256 761c7002…816776, byte-identical to the sha shown in the receipts panel screenshot above. What you download is exactly what the phone is running. Fidelity receipts (KLD ladder) for this edge variant are the next lane item; until they land, its published evidence is the on-device behavior and benchmark shown here. The Q4_K_M and Phone artifacts carry full fidelity receipts today.

License/Terms of Use

Apache 2.0 for the quantization artifacts and receipts in this repository. The model weights derive from google/gemma-4-E2B-it and remain subject to Google's Gemma terms (license link). Nothing here modifies the base model's own usage terms.

Use Case

On-device and edge inference of Gemma-4-E2B-it via llama.cpp and llama-cpp-python where fidelity to the original model matters at 4-bit-class sizes: assistants, code/math workloads, and phone/laptop deployments (the Phone artifact targets memory-constrained devices at 2.86 GB).

Release Date

Hugging Face 07/2026 via this repository.

References

Serving lane (KV-cache compression for vLLM, same receipt discipline): Qwen3-4B kit · Mistral-7B kit
All fraQtl artifacts: huggingface.co/fraQtl

Model Architecture

Base model: google/gemma-4-E2B-it (instruction-tuned E2B). The GGUFs were converted from the pinned bf16 checkpoint with llama.cpp's convert_hf_to_gguf.py --outtype bf16, then quantized with llama-quantize using per-tensor --tensor-type overrides selected by fraQtl's calibration-aware analysis on a code+math calibration packet. How the per-tensor decisions are made is not published; the recipe hash, the imatrix, and every artifact hash are, so the result is verifiable end to end.

Phone artifact: identical transformer quantization; the per-layer-embedding (PLE) table is additionally demoted to Q4_K. Speed-neutral by construction: bandwidth-relevant bytes (everything except the two lookup tables) are byte-identical between the two artifacts (1,188,795,532 B).

True bits-per-weight: "Q4_K_M" for E2B is really 5.3–5.9 true bits/weight because the 2.35B-parameter per-layer-embedding table dominates bytes. We publish true average bits/weight over all tensors AND transformer-resident ("bandwidth-relevant") bytes — E2B decode reads ≈1.15 GB of transformer weights per token; the PLE table is a ≈7 KB/token lookup. Size comparisons on this page are total bytes; speed-relevant comparisons are bandwidth-relevant bytes.

Input / Output

Input: text (Gemma chat format)
Output: text
Context length: as the base model; evaluation on this page used ctx 2048 (calibration) and ctx 512 (perplexity sanity metric).

Calibration Dataset

Dataset: fraQtl Apex code+math packet (sha256 9bec84a28dcb0c940047e6084561a00857fd610b8a1e148cc38e27929e0a7e02), decontaminated against the eval slices.
Data collection method: Automated (curated public code/math text).
Properties: full packet, 212 chunks / 434,176 tokens; importance matrix computed with llama-imatrix at ctx 2048 on the BF16 substrate. The exact imatrix.dat ships in this repository (sha256 892e5f908dc43cef60be5bf02b3246bf5da8651d88d107c9fd00c20771d83e3c) so the quantization is reproducible.
Recorded deviation from the 26B build pattern: single full-packet imatrix; no imatrix-budget sweep (E1 lean scope).

Usage

Receipt-validated stack: llama.cpp @ 78d2f524682d9fee790a6460c93d018dafeb5229 (2026-07-05) / llama-cpp-python==0.3.33 (vendors that commit). Newer releases should work; the pin is what every number on this page was measured with.

bash

pip install llama-cpp-python==0.3.33
hf download fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf --local-dir .

python

from llama_cpp import Llama

llm = Llama(model_path="Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf", n_ctx=4096, seed=0)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Explain in two sentences why the sky is blue."}],
    temperature=0.0,
)
print(out["choices"][0]["message"]["content"])

Or with llama.cpp directly:

llama-cli -m Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf -cnv -p "You are a helpful assistant."

Chat template / turn-token note: the Gemma chat template is embedded in the GGUF — use create_chat_completion (or llama-cli -cnv) and let the runtime apply it. Gemma turns are delimited by <start_of_turn>user / <start_of_turn>model and terminated by <end_of_turn>, and the assistant role is named model, not assistant. If you hand-roll prompts, make sure <end_of_turn> is in your stop set: stacks that only stop on <eos> will run past the turn boundary and generate spurious extra turns.

Evaluation

All evaluation is fidelity to the ORIGINAL gemma-4-E2B-it (the axis a user migrating from the bf16 model experiences), on an A100-80GB with seed 0, deterministic settings, 3 independent runs where stated (run-to-run drift 0.0). Eval slices: code/math 400-record held-out (sha256 cce6860274ab715e3ab8493c9f817b1c29ff99c148f83a25de8944e027db1665); general chat/tool/wikitext 400-record (sha256 b10a79caf2c17cc10cd1edcae44d4655278baf5b09b6a867b4d3ade2f996b276).

Fidelity to the original model (KLD, lower is better)

Arm	bytes	true bpw	KLD code/math ↓	KLD general ↓	top-1 agree (gen)
bartowski Q4_K_M (community)	3,462,678,272	5.93	0.09139 (3-run)	0.13249 (3-run)	0.888
unsloth Q4_K_M (community)	3,106,738,272	5.32	0.10348	0.14767	0.882
google QAT q4_0	3,349,516,256	5.76	0.24730	0.35353	0.831
unsloth qat UD-Q4_K_XL	2,620,370,976	4.50	0.20666 (3-run)	0.30275 (3-run)	0.849
fraQtl Hi-Fi (iso-size)	3,461,675,328 (−0.03% vs baseline)	5.93	0.05846 (3-run, −36.0%)	0.09646 (3-run, −27.2%)	0.905
fraQtl Hi-Fi Phone (PLE-Q4_K)	2,856,122,688 (−17.5%)	4.89	0.06231 (−31.8%)	0.09716 (−26.7%)	—

QAT context (read before comparing QAT rows): the QAT GGUFs descend from google/gemma-4-E2B-it-qat-q4_0-unquantized — a retrained checkpoint. Unsloth's published E2B QAT mean KLD 0.00173 is measured against that retrained checkpoint, not against the original gemma-4-E2B-it. All rows above measure fidelity to the ORIGINAL model, which is the axis a user migrating from the bf16 model experiences. Both statements are true simultaneously; they answer different questions.

GSM8K

Metric	fraQtl Hi-Fi (iso-size)	bartowski Q4_K_M	Verdict
GSM8K, 0-shot instructed chain-of-thought, n=200, seed 0, greedy	86.0% (172/200)	79.0% (158/200)	diff +7.0pp, 95% tie band ±7.42pp → statistical tie

The +7.0pp difference is inside the ±7.42pp tie band (1.96·sqrt((p₁(1−p₁)+p₂(1−p₂))/n), recomputed for this model at n=200). We make no win claim on GSM8K. The protocol label is exact: 0-shot with a chain-of-thought instruction; no few-shot exemplars; not the full 1319-item set. Phone artifact: not run (lean scope).

Wikitext-2 perplexity (sanity flag — disclosed loss)

Metric	fraQtl Hi-Fi (iso-size)	bartowski Q4_K_M	Verdict
Wikitext-2 PPL, ctx 512	20009.7	18005.3	LOSS (flag)

One result runs against this artifact: it is worse on raw-web perplexity. Two caveats, neither of which removes the flag: (1) both absolute values are harness-inflated — ctx-512 chunked raw text on an instruction-tuned model is a sanity metric, not a capability measure; (2) the code/math-domain imatrix likely explains the relative gap (domain shift). Note the general-slice KLD — which includes a wikitext-anchor domain — still favors this artifact. PPL here is a sanity flag, not a gate.

Reproducibility

Deterministic eval rig: 3 independent KLD runs are byte-identical (drift 0.0). Verdicts are by mean; no confidence-interval claim is made on KLD. GSM8K uses the tie-band discipline above.

Provenance

Field	Value
Base model	`google/gemma-4-E2B-it` @ `179516f0c449474fdc46f08f30ead5b11e178497`
Base checkpoint sha256	`model.safetensors` = `2db5482b20d746879bb3ef79b5203e9075a2e2b98f54ec7c2f281c1477ddc550` (10,246,621,918 B)
Conversion + quantization	llama.cpp @ `78d2f524682d9fee790a6460c93d018dafeb5229` (2026-07-05); `convert_hf_to_gguf.py --outtype bf16` then `llama-quantize` with per-tensor `--tensor-type` overrides
Calibration	fraQtl Apex code+math packet (sha256 `9bec84a28dcb0c940047e6084561a00857fd610b8a1e148cc38e27929e0a7e02`), `llama-imatrix` ctx 2048, BF16 substrate, full packet (212 chunks / 434,176 tokens)
imatrix (ships in this repo)	`imatrix.dat` sha256 `892e5f908dc43cef60be5bf02b3246bf5da8651d88d107c9fd00c20771d83e3c`
Recipe sha256	`69f2151874412bcda4df1028c5b76743b1a60afa9f0a0abb2ff87c26a982bd37` (phone artifact differs only in the PLE embedding override)
Iso-size GGUF	`Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf` — 3,461,675,328 B, sha256 `d008ba9d736b1d51e55d1d1944196441c1dbfe4fe9eb4ebd9bfbda4c2f36e013`, true bpw 5.93, bandwidth-relevant bytes 1,188,795,532
Phone GGUF	`Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf` — 2,856,122,688 B, sha256 `d535787b323f79d8faf3867501c028dc05be4e0a3413a6ccdf54cb88bb737866`, true bpw 4.89, bandwidth-relevant bytes 1,188,795,532 (identical by construction — speed-neutral)
Eval runtime	`llama-cpp-python==0.3.33` (vendored llama.cpp `78d2f5246`), A100-80GB, seed 0, deterministic (3-run drift 0.0)
Fidelity metric	symmetric top-20 KLD on Q8 teacher support, mean over positions; teacher = `unsloth/gemma-4-E2B-it-GGUF` Q8_0 @ `0314792d7f1f7e229411f620751375812bb9faf2` (sha256 `605d3c2647d7c58c1e4b5375ccb5702acf94c2611b4c8d4877812f8fdd32d053`), quantized from the ORIGINAL bf16
Eval slices	code/math 400-record held-out (sha256 `cce6860274ab715e3ab8493c9f817b1c29ff99c148f83a25de8944e027db1665`); general chat/tool/wikitext 400-record (sha256 `b10a79caf2c17cc10cd1edcae44d4655278baf5b09b6a867b4d3ade2f996b276`)
Baseline (iso-size)	`bartowski/google_gemma-4-E2B-it-GGUF` Q4_K_M @ `b5e99bd964eaacc27ba484bb2eb3e9f6160b9143` — 3,462,678,272 B, sha256 `b5310340b3a23d31655d7119d100d5df1b2d8ee17b3ca8b0a23ad7e9eb5fa705`, true 5.93 bpw (all tensors)
Probe receipt	`gemma4_e2b_probe_2026-07-19.partial.json` sha256 `8511c8d6f1530a94e66f1f344e1e0bed57cc692819e36ca801946a71452f036a`
Drift	3-run KLD drift 0.0 (deterministic rig)

MTP speculative decoding (works with this quant)

Gemma 4 ships official MTP drafters (trained by DeepMind, distributed by Unsloth). The drafter pairs with this Hi-Fi quant out of the box — measured 74% draft acceptance at n-max=2 with byte-identical greedy output (llama.cpp b10280, llama-server, Apple Silicon; single tool-call prompt, greedy, warmup run first). Full numbers: receipts/MTP-DRAFTER-RECEIPT-2026-08-06.md.

Grab mtp-gemma-4-E2B-it.gguf (93 MB, sha256 9eba8199…191919) from unsloth/gemma-4-E2B-it-GGUF, then:

llama-server -m Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf \
  --spec-type draft-mtp --spec-draft-model mtp-gemma-4-E2B-it.gguf \
  --spec-draft-n-max 2

Don't expect a desktop speedup: on Apple Silicon it's a wash (51.0 → 52.5 tok/s in our test — at 51 tok/s the model isn't slow enough for drafting to pay). The case that benefits is slow-decode targets like phones — confirmed by the first independent report below. n-max=2 was optimal in our tests; deeper drafting (n-max 4/6) was net negative as acceptance decays (74% → 63% → 51%). Requires llama.cpp at or after the May 2026 MTP merge (PR #22673).

First independent result — Android (community-reported)

Reported by u/Sad-Enthusiastic on r/LLMDevs, 2026-08 (OnePlus 8T, Android ARM64, llama.cpp b9975 — their measurement, not ours):

Config	tok/s	Draft acceptance
Hi-Fi + MTP, n-max=4	~11.75	~55% (mean draft length ~2.0)
Hi-Fi + MTP, n-max=2	10.92	~55%

Reported as the fastest Gemma on that device — ahead of a curated Q2_K_XL default (~8–10 tok/s), with Q4-class output quality.

Tuning guidance from the two datapoints: pick n-max by your device's decode speed. On a slow target (~11 tok/s phone) the deeper draft paid even at ~55% acceptance (n-max=4 won); on a fast target (51 tok/s Mac) n-max=2 was optimal and deeper went negative. If you report numbers from other hardware, open a discussion — credited additions welcome.

Running it on Android (or any llama.cpp mobile build): the flags are identical — any llama.cpp-based runner at or past the MTP merge works, e.g. a Termux/NDK build or llama.cpp-based Android frontends. Checklist:

Main model: Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf (2.86 GB, this repo) — or the 2.45 GB SmartEdge-IQ3XXS build if disk is tighter.
Drafter: mtp-gemma-4-E2B-it.gguf from the Unsloth link above (93 MB — verify sha256 9eba8199…191919).
Same three flags: --spec-type draft-mtp --spec-draft-model mtp-gemma-4-E2B-it.gguf --spec-draft-n-max 2.
Memory: both files are mmap'd, so resident RAM stays well under the sum of file sizes (~516 MB resident for the main model in our iOS runs; the drafter adds a fraction of its 93 MB).
Reading your results: use llama-server and the /completion timings for tok/s and draft accept counts — llama-cli at default verbosity doesn't print draft stats, and a transient "failed to measure draft model memory" warning at init is known and recovers. If you run this on mobile hardware, we'd love your accept rate and tok/s numbers — open a discussion.

Model Limitations

Raw-web perplexity flag: wikitext-2 PPL (ctx 512) is worse than the community baseline (20009.7 vs 18005.3) — see the Evaluation section for the domain-shift note. If your workload is raw web-text continuation rather than chat/code/math, weigh this flag accordingly.
Calibration domain: the imatrix is code+math. Both eval slices (including the general chat/tool/wikitext slice) favor this artifact on KLD, but calibration is not domain-neutral by construction.
Lean eval scope (E1): GSM8K n=200 only; no MMLU, MATH-500, or long-context retrieval benchmarks were run for this model. The GSM8K result is a statistical tie, not a win.
Phone artifact: KLD-evaluated (single run) and smoke-tested; GSM8K/PPL were not run on it.
Quantization is lossy by construction; the evidence standard here is measured distributional fidelity (KLD) with pinned receipts, not bit-identity.
The base model's own limitations and biases are inherited unchanged.

Files

File	What	sha256
`Gemma-4-E2B-it-fraQtl-HiFi-Q4_K_M.gguf`	Iso-size artifact (3.46 GB, true bpw 5.93)	`d008ba9d736b1d51e55d1d1944196441c1dbfe4fe9eb4ebd9bfbda4c2f36e013`
`Gemma-4-E2B-it-fraQtl-HiFi-Phone.gguf`	Phone artifact (2.86 GB, true bpw 4.89, PLE at Q4_K)	`d535787b323f79d8faf3867501c028dc05be4e0a3413a6ccdf54cb88bb737866`
`imatrix.dat`	Calibration importance matrix (reproducibility)	`892e5f908dc43cef60be5bf02b3246bf5da8651d88d107c9fd00c20771d83e3c`

The fraQtl ladder

Tier	Job	Example
Hi-Fi	maximum fidelity at standard 4-bit size	this repo's Q4_K_M
Hi-Fi Phone	fits phone/laptop RAM, fidelity-first	E2B Phone, 2.86 GB
SmartEdge	smallest footprint that survives	E2B IQ3XXS, 2.45 GB

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

Weight quantization is the on-device lane. The serving lane — KV-cache compression sidecars for vLLM — just fit nine concurrent ~128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14): fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Also: fraQtl/qwen3-4b-instruct-2507-kv-sidecars and fraQtl/mistral-7b-instruct-v0.3-kv-sidecars. Same standard per artifact: pinned provenance, measured numbers, results reported in both directions. Org page: huggingface.co/fraQtl.