Google's Gemma 4 E4B IT, converted to a native Core ML stateful graph that runs on the GPU of an Apple Silicon Mac. One graph does decode, batched verify and prefill (variable sequence length on an MLState KV cache), so prompt-lookup lossless speculation needs no drafter model and no extra verify functions. int4 chunks + int8 sidecars: 6.5 GB on disk, 4.0 GB resident, 31 tok/s decode, up to ×3.5 on verbatim-copy spans.

The bundle also carries the vision encoder, so the model reads images: attach a photo and ask about it, or point a camera at the world and have it captioned cycle after cycle at 1.44 s per cycle.

Built with Gemma. A Core ML conversion of google/gemma-4-E4B-it — see License & attribution.

To my knowledge, this is the first open Core ML build of Gemma 4 E4B that ships working speculative decoding, and its lossless property is verified — the same prompts produce byte-identical output with speculation off and on. It is the second in a series: the E2B bundle targets the iPhone Neural Engine; this E4B bundle targets the Mac GPU. See Related work.

Who this is for

Yes — you want Gemma 4 E4B running fully offline on an Apple Silicon Mac through Core ML, with measured, reproducible numbers.
Yes — you are building a Core ML LLM runtime and want a reference for MLState KV caches, GQA / shared-KV layers, int8 embedding sidecars, and speculation verified token-exact.
Yes — you want image input on a stateful Core ML chain, where variable-width prefill feeds the 256 image rows in one pass rather than a fixed-width schedule.
No — iPhone: this graph needs the GPU + MLState, and 6.5 GB of weights exceeds a 6 GB phone's budget. For iPhone, use the E2B bundle.
No — long-context work: this is a 2048-token context build.

Requires: an Apple Silicon Mac on macOS 26+, the companion Swift runtime (below), and roughly 8 GB of free unified memory (measured footprint is 4.0 GB). Disk ~6.8 GB with the image encoder.

Quick start

hf download okayuji/Gemma-4-E4B-it-coreml-speculative --local-dir ./gemma-4-e4b-speculative

This repository is weights + config only. The stateful chain, the shared-KV wiring and the speculation loop are driven by the companion Swift runtime — the same library the CLI and the demo app link — at https://github.com/oka-yuji/coreml-llm-samples:

bash

git clone https://github.com/oka-yuji/coreml-llm-samples
cd coreml-llm-samples
swift run -c release corellm-chat --model ../gemma-4-e4b-speculative --stats \
  --prompt "List three fruits, one per line."

Speculation is opt-in: run the same command with CORELLM_MTP_PLD=1 in the environment to enable prompt-lookup speculation (draft length defaults to 4; CORELLM_MTP_DRAFT_LEN=6 favors verbatim-heavy work). Output is byte-identical either way — only the speed changes.

Sample output

Apple M4 Max (128 GB) / macOS 26.5.2, GPU, greedy (temperature 0), one process per condition, corellm-chat CLI (2026-07-28). Prompt (72 tokens with chat template):

Extract every person, place, and date from this note as JSON with keys people, places, dates: Maya flew from Osaka to Reykjavik on March 3, 2026, and met Dr. Chen and Lena Novak at the Harpa concert hall on March 5, 2026.

Output — the model wrapped its JSON in a code fence; shown verbatim, identical with speculation off and on (99 tokens to EOS):

```json
{
  "people": [
    "Maya",
    "Dr. Chen",
    "Lena Novak"
  ],
  "places": [
    "Osaka",
    "Reykjavik",
    "Harpa concert hall"
  ],
  "dates": [
    "March 3, 2026",
    "March 5, 2026"
  ]
}
```

Run	prompt → gen	Decode	Speculation
Speculation off	72 → 99 (eos)	32.0 tok/s	—
Speculation on (draft 6)	72 → 99 (eos)	29.8 tok/s	acceptance 0.38

(Across 5 interleaved off/on rounds the pair-ratio median is ×0.94–0.96.)

Two honest notes on this sample. First, on this build JSON extraction is slightly below break-even — the JSON scaffolding (\``json`, braces, key names) does not appear in the prompt, so prompt-lookup has little to draft from; speculation pays on verbatim-copy spans (next section). Second, this output is byte-identical (212 bytes, same SHA-256) to what the E2B iPhone bundle produces for the same prompt — a different model size, quantization, compute unit and runtime path landing on the same JSON.

Where speculation pays

Same machine and conditions, 5 interleaved off/on rounds, per-round pair-ratio median (2026-07-28). Rows run to EOS or a 48-token cap; the JSON row is the EOS-complete run (99 tokens):

Workload	Off tok/s	Speedup (draft 6)	Speedup (draft 4)	Acceptance (d6)
Verbatim copy (repeat a 10-item list)	31.4	×3.49	×3.13	0.86
ID enumeration (6 equipment IDs, answered correctly)	31.0	×0.77	×0.82	0.21
JSON extraction (99 tokens to EOS)	30.5	×0.96	×0.94	0.38
Free-form prose (one-sentence answer)	30.0	×0.98	×0.98	0.29

Verbatim copy is the one clear win; everything the model has to compose — even something as regular as an ID list — sits slightly below break-even, so leave speculation off for chat-style use. A cautionary tale from this bundle's own gating: an earlier internal build with a broken chat template measured ×2.37 on the enumeration row — the model was parroting the prompt, which turns any reply into a verbatim copy. A broken template can fake speculation gains; every number on this card is from the fixed, canonical-template bundle.

The verbatim-copy gate, shown in full — prompt (Japanese, "Copy this list exactly: apple, mandarin, grape, peach, pear, strawberry, melon, watermelon, kiwi, banana"):

次のリストをそのまま書き写してください。
りんご、みかん、ぶどう、もも、なし、いちご、メロン、すいか、キウイ、バナナ

Output (identical with speculation off and on, 30 tokens):

りんご、みかん、ぶどう、もも、なし、いちご、メロン、すいか、キウイ、バナナ

The CLI run above measured 32.6 tok/s off → 109.2 tok/s on (draft acceptance 0.86); across 5 interleaved rounds the pair-ratio median is ×3.49. The output ends cleanly at the end-of-turn token — 30 generated tokens, byte-identical off and on (the speculation run's --stats reports 31: it counts the end-of-turn token, which never appears in the text). Draft length 6 beats 4 on verbatim work (+11%); with mixed workloads, draft 4 is the safer default.

Key numbers

Ship bundle, Apple M4 Max (128 GB) / macOS 26.5.2, GPU, one process per condition, warmup-excluded medians (2026-07-28):

Metric	Value
Decode (speculation off)	31.1 tok/s (32.1 ms/tok)
Prefill (512-token prompt)	670 tok/s
TTFT, 28-token prompt	83 ms
TTFT, 512-token prompt	764 ms
Multi-turn TTFT, 2nd turn (35 of 53 tokens reused)	67 ms
Cold load (fresh process, to first token ready)	2.84 s
Warm load	2.50 s
Resident footprint	4.0 GB
Disk	6.5 GB

What's inside

4-chunk stateful Core ML graph (gemma4, 42 layers, hidden 2560, GQA 8 query / 2 KV heads; head_dim 256 on sliding layers, 512 on full-attention layers). Chunks 0–12 and 12–24 are stateful — their KV caches live in MLState, resident on the GPU, so the host never touches KV shapes. Layers 24–41 share the KV of layers 22/23: chunk 12–24 emits those two stores as outputs and the two stateless chunks 24–36 / 36–42 consume them as inputs.
One graph, three roles. Sequence length is a range dimension (S = 1…128): S=1 is decode, S=draft+1 is the batched speculation verify, larger S is prefill. There are no separate verify functions and no drafter model.
Prompt-lookup speculation drafts continuations by matching the last n-gram (n ≥ 3) against the prompt, then verifies on the main graph. Rejection rolls the position back; rejected rows are overwritten and masked, so state never diverges — that is what the lossless gates check.
int4 chunk weights (linear, per-block 32) with an int8 argmax lm_head (tanh soft-cap 30 baked in; emits token ids, not logits).
int8 sidecars for the token embedding and the per-layer embedding (PLE) with fp32 per-row scales — 3.5 GB smaller than fp16 sidecars, and verified output-identical to them.

Input format

The runtime reads the prompt template from manifest.json — transcribed verbatim: "promptPrefix": "<|turn>user\n", "promptSuffix": "<turn|>\n<|turn>model\n" — prepends BOS, and stops on the end-of-turn token <turn|> (id 106). A single user turn is therefore:

<bos><|turn>user
{user text}<turn|>
<|turn>model

<|turn> / <turn|> are Gemma 4's actual special tokens (ids 105 / 106); the Gemma-3-era <start_of_turn> / <end_of_turn> strings are not in this vocabulary and tokenize as plain text — a mismatched template silently degrades output (it costs ~17–18 tokens per turn and leaks a literal marker string into replies). chat_template.jinja is included for HF-ecosystem tooling.

Images and Live Camera

The bundle carries vision_fp16.mlmodelc next to the language chunks. The companion runtime looks for it inside the bundle directory, next to manifest.json, and enables image input only when it is there. An image is resized to 768x768 and becomes 256 soft tokens — 256 of the 2,048 context — whatever the source resolution.

The vision tower is the same weights as the E2B bundle's: the two vision_config blocks are byte-identical, 658 of 658 tensors are bit-identical, and the towers' activations agree at maxAbsDiff 0.000e+00. Only the projection into the language model differs — 2,560 wide here against E2B's 1,536 — which is why each bundle ships its own copy and the runtime refuses a mismatched encoder instead of producing garbage from a width mismatch.

Measured on an Apple M4 Max, one process per condition:

Metric	Value
Image question, vision encode	93–94 ms (GPU) / 313–320 ms (CPU)
Image question, TTFT	0.75 s
Image question, decode	29.8–34.3 tok/s
Prefill, 273-token prompt with 256 image rows	0.44–0.55 s
Live Camera, warm cycle (English)	1.44 s
Live Camera, warm cycle (Japanese)	1.63 s

The vision encoder runs on the GPU while the language chain keeps the full compute-unit set; the two were sharing one accelerator before that split, and separating them is where the encode went from hundreds of milliseconds to under a hundred.

This is where the stateful graph shows against the E2B bundle. On the same 10-cycle live captioning run, E4B takes 1.44 s per cycle against E2B's 2.48 s — not because it decodes faster in proportion (32.0 vs 23.5 tok/s) but because feeding the 256 image rows costs 0.36 s here against 1.21 s there. Variable-width prefill on an MLState cache absorbs the image span in one pass; a fixed-width scheduled prefill has to break it into chunks.

Files

File	Size	Role
`chunk_0_12.mlmodelc` / `chunk_12_24.mlmodelc`	0.65 GB each	stateful transformer chunks (int4)
`chunk_24_36.mlmodelc` / `chunk_36_42.mlmodelc`	0.63 / 0.31 GB	stateless shared-KV chunks (int4)
`lmhead.mlmodelc`	0.71 GB	argmax head (int8, soft-cap baked in)
`embed_int8.bin` + `embed_scale.bin`	0.67 GB	token embedding sidecar (int8 + fp32 row scales)
`ple_int8.bin` + `ple_scale.bin`	2.82 GB	per-layer embedding sidecar (int8 + fp32 row scales)
`tokenizer.json`, `tokenizer_config.json`	32 MB	tokenizer
`chat_template.jinja`	19 KB	chat template as published by Google for `gemma-4-E4B-it`
`generation_config.json`	208 B	upstream generation defaults (EOS ids)
`manifest.json`, `convert_config_v2int4_int8.json`	15 KB	runtime wiring (shapes, stores, sidecar dtypes, prompt template)
`vision_fp16.mlmodelc`	0.31 GB	vision encoder — 768x768 image to 256 soft tokens (fp16)
`LICENSE`	11 KB	Apache License 2.0 full text

Delete vision_fp16.mlmodelc and the bundle is a 6.5 GB text model with image input switched off.

Operational notes & troubleshooting

Updating a bundle you already downloaded. The app has no in-place update path: open the Models screen, Delete the model, then download it again to obtain vision_fp16.mlmodelc. Incremental refresh of a downloaded bundle is not exposed in the UI.
First run of each new sequence shape is slower. The GPU specializes kernels per shape on first use — the first differential prefill of a session can take ~2× its steady latency, and cold load is 2.9 s. Steady-state numbers in this card are warmup-excluded medians.
Benchmark one process per condition. Core ML keeps memory mapped after a model unloads; measuring several conditions in one process contaminates the later ones.
Core ML compile caches grow. Repeated conversions / first-loads populate ~/Library/Caches/**/com.apple.e5rt.e5bundlecache (tens of GB over time across models). Safe to delete; the next first-load re-specializes.
The lm_head emits argmax token ids only — greedy decoding. Temperature sampling would need a logits-head variant, which this bundle does not include.

Limitations

Context is 2048 tokens (this build). Longer-context variants exist in the lab for the 12B sibling but are not part of this bundle.
Speculation helps only where the reply copies the prompt: verbatim copy and quoting (×3.1–3.5 measured). Anything the model composes — enumeration answered in its own format, JSON scaffolding, free-form prose — is slightly below break-even (×0.77–0.98) — leave it off for chat-style use.
Mac GPU only. The MLState KV design does not run on the Neural Engine, and the bundle does not fit 6 GB iPhones. CPU-only execution is not supported for this graph.
int4 fidelity: against the fp32 reference, greedy output diverges at the first generated token on one tested prompt — a benign style fork (1、2、3… vs はい、1から10まで…; fp32 top-2 margin 0.97), not degradation. On the measured verbatim / enumeration / JSON workloads, int4 output is identical to the fp16 build's. AWQ (p999) was tried and worsened fidelity (it introduced a second divergence), so raw int4 shipped.
Greedy only (argmax head).
No audio. There is no E4B audio encoder; transcription is available on the E2B bundle only.
Images do not reach iPhone here. The image path is macOS-only for the same reason the rest of the bundle is: E4B does not fit an iPhone's memory budget.
An image costs 256 of 2,048 tokens, and the pressure of repeated image turns inside that context was not measured.
int4 moves a small number of tokens on image prompts too. Against the fp32 reference the fp16 chain shows content divergence 0 of 209 teacher-forced positions; the shipped int4 bundle shows 11 of 209. That is quantization rather than image feeding: the same bundle on image-free text shows 8 of 240, a difference that is not statistically significant, and swapping the fp32 vision tower for the Core ML one produces the identical flip set.

Verification

Everything below was gated on this exact bundle (or its direct inputs) on 2026-07-28, one process per condition, Apple M4 Max / macOS 26.5.2:

Conversion fidelity (fp16): the fp16 chain matches HF transformers fp32 greedy for 32 tokens exactly (#mismatch = 0). The rebuilt forward was first validated against HF at fp32 (argmax 20/20, logits max |Δ| = 2.98e-4, cos_min 0.999997).
Lossless speculation: speculation on == off, token-exact, 10/10 gate combinations on the ship bundle (4 workloads × draft length 4/6, plus two EOS-complete long runs); 16/16 on the pre-ship build including fp16.
Runtime equivalence: the Swift runtime reproduces the python reference chain 64/64 tokens (fp16 and int4), and batched prefill matches sequential decode exactly.
Sidecar quantization: int8 embedding/PLE sidecars produce output identical to fp16 sidecars — a 64-token greedy probe and all four workloads, token-exact.
Cross-bundle: the JSON sample above is byte-identical (212 bytes, same SHA-256) to the E2B iPhone bundle's output for the same prompt.
Vision tower vs the reference: reconstructed in fp32, the Core ML vision tower matches the Hugging Face fp32 tower at cosine 1.000000 on all three test images (maxAbsDiff 3.17e-05 … 5.83e-05 against reference amplitudes of ~5.4, the same order as PyTorch's own re-association noise). Its per-layer-embedding sidecar row matches 10752 of 10752 values exactly, maxAbsDiff 0.000e+00.
Vision tower vs the E2B bundle's: vision_config byte-identical, 658 / 658 tensors bit-identical, activations maxAbsDiff 0.000e+00 — the same tower, differing only in the projection width.
Image input, end to end (fp16 chain): teacher-forced against HF fp32 across 209 positions, content divergence 0; 4 of 6 cases match the reference greedy 40/40 exactly. A text-heavy image was transcribed verbatim across all 12 lines. The Swift vision encoder's soft tokens are byte-identical to the Python reference on all three images.
Image feeding is block-size independent: prefilling the 256-row image span in blocks of 128, 32, 7 and 1 produces exactly the same tokens — block 7 deliberately straddles the image span — and segment-based prefill matches plain prefill 5/5 exactly.
The image row uses per-layer-embedding row 0, not the image token's row. An earlier record of this conversion claimed row 258880; that was wrong and is corrected here. Three independent checks agree: the reference source substitutes the pad id, a reference hook gives the image position id [0] with maxAbsDiff 0.0 against row 0 (against 16.26 for row 258880), and a Swift A/B gives 209/209 exact with row 0 against 119/209 with row 258880.
Template canonicalization: pre-release gating caught an earlier build using a broken chat template (Gemma-3-era markers, not in this vocabulary); the release manifest pins the canonical Gemma 4 turn markers, and every performance and speculation number on this card was re-measured on the fixed bundle (the conversion-fidelity gates above are template-independent and carry over; a 64/64 exact re-gate confirmed the weights were untouched).

Related work

okayuji/Gemma-4-E2B-it-coreml-speculative — the series' first bundle: E2B on the iPhone Neural Engine, chunked multifunction graph with host-side KV, dedicated verify functions, KV save/restore.
A 12B Core ML speculative bundle — lube8163/gemma-4-12b-coreml-iphone-practical-chat — appeared the same week as the E2B release (2026-07-25), pairing a width-4 verifier with an MTP drafter on a 12B model. Differences here, stated as fact, not evaluation: this bundle is draft-free (prompt lookup — nothing extra to load or keep resident), its off/on outputs are verified byte-identical, and it decodes at practical speed (30 tok/s class on a Mac GPU versus that card's reported ~0.2–0.5 tok/s speculative on an iPhone 14 — largely a model-size and platform difference). I have not reproduced their bundle; the comparison is taken from their published card.

License & attribution

Weights: derived from google/gemma-4-E4B-it, released by Google under Apache License 2.0. This repository redistributes a quantized Core ML conversion under the same license; the full license text is in LICENSE.
Gemma is a trademark of Google LLC. This is an independent conversion and is not an official Google release.
Conversion, runtime and verification by okayuji. The companion runtime and demo app live at github.com/oka-yuji/coreml-llm-samples.