google/gemma-4-E2B-it-qat-q4_0-unquantized converted to Core AI .aimodel bundles for
Apple silicon by visible-cx. These are derivative
artifacts: Google's QAT-trained weights rounded onto the int4 grid they were trained for
and re-expressed as a Core AI graph. They load through Core AI on macOS and are not usable
by PyTorch, GGUF or MLX.
Gemma 4 E2B uses Per-Layer Embeddings, so these bundles take a large embedding gather
table as a static graph input rather than carrying it in the graph. That table ships in
ios-frontend/ and the bundles do not load without it; a missing table produces a bare
input-arity error naming ple_table/ple_scale.
⚠️ Known issue — memory growth on long generations. The Core AI runtime caches a
graph specialization per input-shape signature, and this export makes the sequence
length load-bearing: every generated token adds one token to position_ids and thus
mints a new signature, retaining roughly 80 MB of GPU allocations per generated
token (both engines, guided and unguided alike, at equal context). Workloads whose
prompt/output lengths repeat — e.g. short constrained extraction rows — stay bounded,
because previously seen lengths replay from cache at no cost. Long free-form
generations grow without bound until the process exits; no in-process mitigation
exists (token caps only postpone it, and shape-length reuse is the only stable state).
A re-export onto a shape-stable contract that removes this growth is available in this
repository under stable/ — see its section below. It requires updated host support
(in progress); on current hosts, use the bundles above for guided generation only.
coreai-core 1.0.0b2 on every inner asset metadata.json
Weight format
int4, per-block-32 (int4lin, symmetric-with-clipping) — the ggml q4_0 grid the QAT checkpoint was trained on
Vocab
262,144
Export functions
main (S=1 decode) and, in _pf64 bundles, prefill (S=64 chunked prefill)
"QAT-unquantized" means QAT-trained, stored at full width; the int4 rounding happens at
export, onto the grid training already targeted.
_tbl = the PLE gather table is bound as a static graph input. _pf64 = a second
entrypoint, prefill, with a static query width of 64
(function_map: {"main": ["main", "prefill"]}).
Each bundle folder holds <name>.aimodel/ (main.mlirb ≈ 2.09 GB, main.hash, asset
metadata.json), a bundle-level metadata.json, and tokenizer/ (tokenizer.json
32,169,626 B, tokenizer_config.json, generation_config.json, chat_template.jinja
18,569 B). The w4a8 folder additionally ships its calibration_corpus.jsonl (35,045 B).
Stop token: every bundle declares eos_token = "<turn|>" (id 106), which is the
turn terminator Gemma 4 emits. generation_config.json independently lists
eos_token_id: [1, 106, 50]. A host that stops on the raw upstream <eos> instead will
overrun every reply.
The PLE gather-table sidecar — required, not optional
Path
Files
Bytes
ios-frontend/gemma4_qat_gather_raw/
7
2,808,612,071
File
Bytes
embed_per_layer.i8
2,348,810,240
embed_tokens.i8
402,653,184
proj.f32
55,050,240
embed_per_layer.scale.f32
1,048,576
embed_tokens.scale.f32
1,048,576
proj_norm.f32
1,024
meta.json
231
meta.json records the shape and the dequant convention:
V 262144, D 1536, PLD 8960, L 35, ld 256, embed_scale_pl 16.0 (= √256, which is what the
exporter assumes).
Every _tbl bundle binds ple_table (from embed_per_layer.i8, 2.35 GB) and ple_scale
(from embed_per_layer.scale.f32) as static graph inputs.
Requirements
Apple silicon Mac, Core AI runtime.
Engine contract: 4 inputs — input_ids, position_ids, plus static ple_table and
ple_scale. Two engines accept that:
Pipelined engine — binds the statics zero-copy over the caller's buffer, but does
not support logits, so no grammar-constrained decoding.
Sequential engine — the only logits-capable engine, and therefore the only path for
guided decoding. It accepts >= 2 inputs and binds everything beyond
input_ids/position_ids from EngineOptions.staticInputBuffers; an extra input with
no buffer is rejected by name. It pays a one-time copy of every constant input at
load — 2.19 GB for E2B, because its submission path otherwise materialises a foreign
buffer-backed view on every forward pass. Binding the table as a raw view instead costs
~0.30 s per pass (3.4 tok/s); the load-time copy takes the same bundle to 8.6 s/row.
States:keyCache / valueCacheFloat16, 15 × 1 × 1 × ? × 512. Dynamic sequence
dim → GrowingKVCache (initial 256, doubling), not a static allocation at the manifest
maximum.
KV cost: 30,720 bytes per token (fp16) — 126 MB at 4096, 252 MB at 8192, 503 MB at
16384.
Memory envelope, 16 GB Mac (Metal recommendedMaxWorkingSetSize ≈ 10.7 GB):
Weights (main.mlirb)
2.09 GB
PLE table bound as static input
2.35 GB
Base
4.44 GB
Affordable context, fp16 KV
~204k tokens → 131,072, the model cap, binds first
KV is not the ceiling at this tier; the PLE table is. Measured peak footprint on a guided
run: 8.26 GB, max RSS 5.40 GB. Minimum practical machine memory: 16 GB, for
guided decoding only. Free-form generation does not fit this envelope on any machine —
see the known issue above.
Sliding-window note: E2B interleaves 12 sliding and 3 full attention layers across 35
layers collapsed to 15 KV slots. The export models the window in the mask, not in
memory — sliding layers occupy full-length slots and zero-pad head_dim 256→512, so 80% of
KV bytes sit in windowed slots and 40% is dead padding.
The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1. Both engines read
the static query width off the graph.
Performance
Measured on a 16 GB Apple silicon Mac (M2 Pro, macOS 27 beta): 10-sample harness,
guided JSON-constrained decoding against a fixed schema, greedy, sequential engine with the
PLE tables bound as static inputs, reset() between samples, 128-token cap. Load excluded
from s/row; sample 1 excluded as a cache-warm outlier.
_tbl_pf64
Load
12.1 s
Guided JSON parse
10/10
Enum-clean
10/10
s/row (long samples)
6.12
s/row (short samples)
4.23
Decode
22.7–32.7 tok/s
TTFT
0.59–4.20 s
Peak footprint
8.26 GB
Max RSS
5.40 GB
Outcome
completed all ten samples
The S=64 prefill function carries the time-to-first-token. Measured head to head on
a 942-token prompt against a decode-only export of the same weights, exporting prefill
moved TTFT 35.74 s → 4.65 s and s/row 45.04 → 8.83, a 7.7×. Decode is untouched — the
prefill function only serves full-width chunks. The decode-only bundle is no longer
published; every bundle in this repo now carries prefill.
Enum conformance is the grammar's doing. Unguided, the model emits an off-schema enum
value on essentially every sample. Guided, all ten are correct, because an off-enum token
is unsamplable. Decode throughput is essentially unchanged by the constraint; the ~25%
extra cost of a guided sample is the sequential engine's step-synchronous prefill.
For reference, upstream measured E2B at 77.0/87.1 tok/s on an M4 Max, and an unguided
pipelined run on the 16 GB machine reached 44.3 tok/s. The 22.7–32.7 tok/s above is the
guided, sequential-engine figure.
Unguided workload — memory-capped
Free-form generation from a fixed prompt. Every generation length below is a memory cap,
not a model stop — see the known issue.
depth
prompt tokens
TTFT
decode
generated
wall
peak footprint
3.4k
3314
12.21 s
7.2 tok/s
128 (capped)
30.4 s
19.97 GB
8k
7972
54.01 s
2.1 tok/s
64 (capped)
84.3 s
36.71 GB
Output quality up to the cap is sound; the limit is memory, not capability.
Needle-in-haystack recall: 3/3 verbatim at 8k, within a 64-token cap (three distinctive
facts planted at 10/50/90% of the filler, strict scoring). 15k was not attempted: 8k
already cost 36.71 GB of footprint.
QUALIFIED FOR GUIDED GENERATION — measured: 10/10 parse, 10/10 enum-clean, 6.12/4.23 s/row, 22.7–32.7 tok/s, 8.26 GB peak footprint. Not qualified for free-form generation on the current reference engine implementation.
w4a8/gemma4_e2b_qat_decode_int4lin_a8_tbl_pf64
EXPERIMENTAL — built, unmeasured. Gate: a Mac-side oracle/parity check plus a device benchmark.
stable/…_tbl_pf64_stable_c16384
EXPERIMENTAL — shape-stable decode contract; removes the per-token memory growth by design; host support exists; measured E2B: memory fix confirmed but ~1 tok/s decode at capacity 16384 — awaiting a smaller-capacity export before any use.
stable-smoke/…_stable_c16384_l5
NOT A MODEL — 5-layer truncated proving asset for host development; produces low-quality text by design.
About the w4a8 bundle
Same QAT int4 per-block-32 weights and the same graph as
gpu-pipelined/gemma4_e2b_qat_decode_int4lin_tbl_pf64; the difference is an int8
quantize/dequantize pair on the inputs of every F.linear, calibrated on 128 synthetic
samples (the corpus ships in the folder). It was built as a prefill/TTFT lever.
The caveat comes from the compression toolchain's own documentation: the runtime fast path
is W_INT8 × A_INT8, and a float weight path executes in floating point regardless of
activation quantization. These weights use int4 scale-multiply dequant, not an int8 LUT, so
the expected TTFT gain may be zero or negative. This bundle exists to be measured, not
because a win is predicted.
No numerics gate has been run on any bundle in this repo. The 10/10 results are
behavioural (parse rate, enum conformance, clean stop); a decode oracle against an fp32
reference has not been run.
License
The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma
model and the Gemma terms apply downstream, which is why this repo declares
license: gemma. Use is subject to the
Gemma Terms of Use, the
Gemma Prohibited Use Policy and the
Gemma 4 license. Those obligations
travel with any redistribution of these bundles, including the gather-table sidecar, which
is derived from the same weights. The contribution here is the conversion, not the weights.
stable-smoke/ — truncated proving asset
⚠️ EXPERIMENTAL — not a usable model. A 5-layer truncation of the E2B decoder,
published only so the shape-stable engine contract below can be developed against a
small download. It carries real weights for the layers it keeps and produces
low-quality text; do not evaluate quality from it.
⚠️ EXPERIMENTAL — built, not yet measured on a Mac. A re-export of the same weights
and the same quantization onto a decode/prefill contract in which no input shape moves
between steps, which removes the per-generated-token memory growth described in the
known issue above. It requires a host that feeds position_ids as the absolute
positions of the S new tokens only (host support in progress); a host that feeds the
growing 0..N prefix will write the KV cache at the wrong offset. Measured on host support (16 GB M2 Pro): the memory defect is
confirmed fixed — 6.53 GB flat peak across a 664-token free-form generation
(+0.011 MB/token, versus ~81 MB/token and a killed process on the default bundle) and
guided output byte-identical to the default bundle — but decode runs at ~1.0 tok/s
versus ~27–31, because every step reads the full 16,384-slot cache. Not usable as
shipped; a smaller-capacity export would trade window for speed. Treat as a working
proof of the contract, not a deployable bundle.
Contract. Both entrypoints take four inputs and two states, all statically shaped:
main IN input_ids Int32 1x1 | position_ids Int32 1x1 | ple_table Int8 V x (L*ld) | ple_scale Float32 V
prefill IN input_ids Int32 1x64 | position_ids Int32 1x64 | (same statics)
ST keyCache / valueCache Float16 [slots, 1, n_kv, 16384, 512]
OUT logits Float16 1 x S x 262144
position_ids carries the absolute position of each of the S tokens in the call, and
position_ids[0,0] is also the cache slot the K/V for those tokens is written at. The
attention mask (causality, unwritten slots, and the 512-token sliding window) is derived
inside the graph from those positions, so there is no mask input to bind. The KV state's
sequence extent is a literal 16,384 rather than a dynamic dimension, so a host that
resolves its cache strategy from the state descriptor will allocate the full cache up
front (0.50 GB for E2B, 1.61 GB for E4B) instead of growing it — that fixed cost is the
trade for the unbounded growth it replaces.
The context ceiling here is encoded in the graph: a different window needs a different
export, not a manifest edit.
Verification. Before conversion, the stable graph was checked against the shipped
graph in eager torch on the same weights: greedy argmax identical on all tested decode
steps (including positions past the 512-token sliding window, with 384 cache slots
unwritten-and-masked), worst logits cosine 0.99998 (fp16), and the prefill path
bit-identical in fp32. On-device behavior is unmeasured until host support lands.
ring-smoke/ — truncated proving asset
⚠️ EXPERIMENTAL — not a usable model. A 5-layer truncation of the E2B decoder,
published only so the shape-stable engine contract can be developed against a small
download. It carries real weights for the layers it keeps and produces low-quality
text; do not evaluate quality from it.
⚠️ EXPERIMENTAL — built and gated in torch, on-device measurements pending. A
re-export of the same weights and the same quantization as stable/, onto the same
host contract, with the KV layout reworked. No throughput or footprint numbers have
been taken on a Mac yet; the structural facts below are read off the converted program,
not inferred.
What this is.stable/ removed the per-generated-token memory growth by making every
input shape a literal, and it worked — but it gave every layer a full 16,384-slot KV cache
and read all of it every step. ring/ keeps that contract byte-for-byte and changes what
sits behind it:
sliding layers read a ring buffer, 576 slots (the model's own 512-token sliding
window plus one 64-token prefill chunk) instead of 16,384. Ring slot r holds the
newest written position with pos % 576 == r, and the mask that selects the written,
causal, in-window slots is derived inside the graph from the absolute positions.
the grouped-query head expansion is folded into the query rather than materialised.
The stock lowering broadcasts an index tensor to the full [1, heads, K, head_dim] and
gathers K and V through it; reshaping the query so the head counts already match
produces the same dot products with no index tensor and no gathered copies.
both regions are packed into the same two states, so the host still binds exactly
two KV states, positionally, at whatever literal extent the descriptor declares.
Contract — unchanged from stable/. Both entrypoints take four inputs and two states,
all statically shaped:
main IN input_ids Int32 1x1 | position_ids Int32 1x1 | ple_table Int8 V x (L*ld) | ple_scale Float32 V
prefill IN input_ids Int32 1x64 | position_ids Int32 1x64 | (same statics)
ST keyCache / valueCache Float16 [1, 1, n_kv, TOTAL, 512]
OUT logits Float16 1 x S x 262144
position_ids carries the absolute position of each of the S tokens in the call, and
position_ids[0,0] is the write position. The mask is derived in the graph, so there is
no mask input to bind. The KV states have literal extents, so a host that resolves its
cache strategy from the state descriptor allocates them up front rather than growing them.
Structural facts (dumped from the converted program):
E2B
E4B
KV state shape
[1, 1, 1, 56064, 512] x2
[1, 1, 2, 77056, 512] x2
KV bytes, both states
114.8 MB
315.6 MB
same for stable/
503.3 MB
1.61 GB
cache slots read per decode step
56,064
77,056
same for stable/
245,760
393,216
sliding / full layers
12 x 576 + 3 x 16,384
20 x 576 + 4 x 16,384
dynamic dimensions anywhere
none
none
One new host precondition. A prefill call's first position must be a multiple of 64.
The ring write is a fixed-width store at p0 % 576, and 576 is nine 64-token chunks, so a
64-aligned chunk can never straddle the wrap; an unaligned one would write past the end of
its own ring region. Feeding whole 64-token chunks from a 64-aligned position — and the
remainder one token at a time through main — was already what the contract asked for.
stable/ tolerated an unaligned chunk; ring/ does not.
Unlike the ctx*/ folders, the context ceiling is encoded in the graph: a different
window needs a different export, not a manifest edit.
ring2/ — the same graph at three KV capacities
EXPERIMENTAL — diagnostic assets, not products. These are the ring/ recipe, byte
for byte, exported at a smaller full-attention capacity. They exist to measure how the
per-step cost of the shape-stable graph depends on that capacity. No throughput numbers
have been taken on device for either of them. Their context ceilings (2,048 and 8,192)
are smaller than the 16,384 of ring/ and are encoded in the graph, so neither is a
drop-in replacement for anything already published here.
Contract: identical to ring/. Same four inputs in the same order, same two states in
the same positional order, position_ids as the absolute positions of the S tokens in the
call, the mask derived in the graph, the same 64-alignment precondition on prefill. The
only thing that differs is the literal extent of the full-attention region, and a host that
reads its cache strategy from the state descriptor needs no change to drive them.
Equivalence. Both were gated in eager torch against the shipped graph before conversion,
at their own capacity: 1,600 prompt tokens as 25 chunks of 64 then 32 greedy decode steps,
final position 1,631, so the sliding ring wraps roughly three times. fp32, 57 comparisons
per run, 0 argmax mismatches and worst cosine 0.999999999987 for both. No dynamic
dimension appears in either entrypoint of either bundle.
ring2-smoke/ — truncated proving asset
EXPERIMENTAL — not a usable model. A 5-layer truncation of the E2B decoder at the
2,048 capacity ([1, 1, 1, 4352, 512] states, 4 x 576 + 1 x 2,048), published only so
host-side work can be developed against the contract over a smaller download. It carries
real weights for the layers it keeps and produces low-quality text; do not evaluate
quality from it.
ring3/ — per-layer-type SDPA form
EXPERIMENTAL — gated in torch, no on-device numbers yet. Same weights, same
quantization, same host contract as ring/. One thing changes inside the graph: the
grouped-query expansion is folded into the query only in the full-attention layers,
and the sliding layers keep the stock gathered form.
Why the split. Folding the GQA expansion into the query removes the gathered copies and
the broadcast index tensor, but it also turns G independent per-head matmuls into a single
batch-1 matmul with G times the rows. That trade is worth taking when the key length is long
(the full-attention layers read the whole capacity) and not worth taking when it is short
(the sliding layers read a fixed 576-slot ring, where the operation is short enough that
losing the per-head parallelism costs more than the bytes it saves). ring/ applied the fold
everywhere; ring3/ applies it only where the key length is long.
Both forms compute the same dot products in the same order. The equality gate below is run
against the shipped graph, not against ring/.
Contract: unchanged. Same four inputs in the same order, same two KV states in the same
positional order, position_ids carrying the absolute position of each of the S tokens in
the call, the mask derived inside the graph, the same requirement that a prefill call start
at a multiple of 64. A host driving ring/ drives these with no change.
Equality. Gated in eager torch against the shipped graph before conversion: 1,600 prompt
tokens as 25 chunks of 64 then 32 greedy decode steps, final position 1,631, so the sliding
ring wraps about three times. fp32, 57 comparisons per run, 0 argmax mismatches on every
bundle below. No dynamic dimension appears in either entrypoint of any of them.
E2B runs 28 sliding and 7 full attention layers; in these bundles the 28 keep the gathered
form and the 7 use the folded one.
ring3-smoke/ — truncated proving asset
EXPERIMENTAL — not a usable model. A 5-layer truncation of the E2B decoder at the 4,096
capacity ([1, 1, 1, 6400, 512] states, 4 x 576 + 1 x 4,096), published only so host-side
work can be developed against the contract over a smaller download. It carries real weights
for the layers it keeps and produces low-quality text; do not evaluate quality from it.
