google/gemma-4-12B-it-qat-q4_0-unquantized converted to Core AI .aimodel bundles for
Apple silicon by visible-cx. These are derivative
artifacts: Google's QAT-trained weights rounded onto the int4 (or int8) grid and re-expressed
as a Core AI graph. They load through Core AI on macOS and are not usable by PyTorch, GGUF or
MLX. Gemma 4 12B is a dense model — no Per-Layer Embeddings and no gather-table sidecar,
unlike the E2B and E4B bundles in this org.
The repo holds two generations of build. The three dense2in* folders are the original 2-input
dense exports. tiered/ is the current one: a single bundle carrying seven pinned
context capacities from 4,096 to 262,144, gated bit-for-bit against each other, for 4.54 MB
more than a single-capacity bundle.
tiered/ is STAGED, not qualified. No tier has ever been run on a Mac, the multifunction
bundle has never been loaded by the engine, and its compiled blob is a forecast. Every
per-rung requirement below inherits that forecast. See
The seven-rung ladder.
Known issue on the dense2in* bundles — memory growth on long generations. The Core AI
runtime retains one compiled specialization per sequence-length signature, and these decode
graphs make sequence length load-bearing, so a long free-form generation retains roughly
80 MB of GPU allocations per generated token until the process exits. Measured on the sibling
E2B/E4B exports (~81 MB/token at 3.3k of context, guided and unguided alike). The
tiered/ line does not have this defect — shape-stability was measured directly on its
wkv_stable predecessor: 6,962 forward passes, one shape signature, stable=true on every
one, with total process-footprint growth of 5.0 MB across the whole run.
v5/ — ring extent R = 2W: single-pass speculative verify (2026-09-02)
v5/gpu-pipelined/gemma4_12b_qat_decode_int4linsym_hsc_r2048_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k
(main.mlirb SHA-256 bd67ee6c89d07f79…), sidecar v5/ios-frontend/gemma4_12b_tied_vocab/head.f16
(byte-identical to v4's, fb21807f…).
What changed, and only this. The sliding-attention ring holds 2,048 slots instead of
1,024 (capacity_tier_notes.ring_slots = 2048, sliding_window still 1024). Every weight,
every rung, the head sidecar, prefill and the verify signature are v4's. The ring state
per rung grows from 335 MB to 671 MB (localKeyCache/localValueCache[40,1,8,2048,256]
fp16); the global cache and everything else are unchanged.
Why. With R = W (v4), a speculative verify block had to run twice — once with
accepted_count = 0 to adjudicate, once to commit the accepted prefix — because a
rejected draft's row, once written, would be admitted by later queries. With R = 2W the
slot a rejected row occupies has an implied position (p = q − ((q − j) mod R)) that
fails the window test (q − p) < W for every live query until the slot is legitimately
overwritten. So the host may pass accepted_count = K on every call and continue from
p + n: one verify pass per block. The two-pass protocol remains valid.
Evidence (torch level, 6-layer core on the conversion box, fp32).
gate
result
R = 2W vs R = W, shared stream over 2,336 positions (both ring wraps)
max abs 5.9e-4
commit-all (K = 8) then continue with 3 accepted, vs a clean core — R = 2W
max abs 2.4e-4
the same experiment on R = W (the hazard the wider ring removes)
max abs 9.49
Not yet verified on these bytes: an end-to-end greedy-stream check through the
compiled verify_c65536 on a 16 GB Mac (three attempts were floor-killed by memory on a
loaded machine, not by the bundle). The app pin remains v4 until the Swift engine wires
the verify entrypoint; nothing routes to v5 today.
Each folder holds <name>.aimodel/ (main.mlirb, main.hash, asset metadata.json), a
bundle-level metadata.json, and tokenizer/ (tokenizer.json 32,169,626 B,
tokenizer_config.json 3,090 B, chat_template.jinja 18,683 B, generation_config.json).
With the shared tokenizer the staged tiered/ bundle is 6.27 GiB on disk.
dense2in-pf64's weights are bit-identical to dense2in's; the multifunction build adds an
S=64 prefill entrypoint for under a megabyte, because the exporter deduplicates weights
across entrypoints.
Stop token: every bundle declares eos_token = "<turn|>" (id 106), the turn terminator
Gemma 4 emits. A host that stops on the raw upstream <eos> instead will overrun every reply.
Manifest context is metadata only — except on tiered/. On the dense2in* bundles
--max-ctx sets language.max_context_length and nothing else, so the numbers above can be
raised or lowered without a re-export (whether the result fits in memory is a separate
question). On tiered/ the capacity is a real, pinned graph extent per entrypoint, and
language.max_context_length reports the top rung — see the warning under
Selecting a rung.
coreai-core 1.0.0b2 on every inner asset metadata.json
Weight format
int4 per-block-32 symmetric absmax (--lin-sym) — the ggml q4_0 grid the QAT checkpoint was trained on; int8 per-block-32 for dense2in-int8
Vocab
262,144
"QAT-unquantized" means QAT-trained, stored full width; the int4 rounding happens at export
onto the grid training already targeted. That is why int4 is the primary recipe rather than a
quality compromise — the int8 variant is an anchor, not the default.
tiered/ — the seven-rung ladder
One bundle, seven main entrypoints, each with its own pinned KV capacity:
The top rung is the checkpoint's own text_config.max_position_embeddings, so the ladder spans
the architecture end to end.
It is nearly free, and that was measured before it was built. A pinned capacity costs no
compiled bytes: at c4096 and c16384 the same graph's main.mlirb differs by 26 bytes on a
6.7 GB file, because capacity is a runtime state allocation and nothing else.
main.mlirb
Δ vs 1 entrypoint
per extra entrypoint
1 entrypoint (c4096)
6,700,591,835 B
—
—
4 entrypoints (c4k…c128k)
6,702,885,614 B
+2,293,779 B
764,593 B
7 entrypoints (c4k…c256k)
6,705,129,126 B
+4,537,291 B
756,215 B
seven separate single-tier bundles
46,904,142,845 B
—
—
one tiered bundle instead
6,705,129,126 B
−37.44 GiB
Six extra capacities — including two that quadruple and octuple the previous top rung — cost
4.54 MB, 0.068% of the bundle, and dedup gets marginally better with more entrypoints:
the four-rung build's cost law predicted 6,705,179,393 B for seven and the file came in
50,267 B under, 0.0007%.
The one idea that makes it work: the tiers are the same quantized module, copy.copy-ed so
each carries its own capacity — a fresh __dict__ over the same _modules and _parameters,
so every weight tensor is the identical object. Quantizing per tier would produce numerically
identical but distinct tensors, and distinct tensors are what dedup cannot merge.
Build-time memory is the one place seven rungs are not free. Peak export RSS was
98.10 GB against 67.29 GB at four rungs and 67.28 GB at one. It is a build cost, not an
artifact cost, but a machine with less than ~128 GB should not attempt a seven-rung export.
The cross-tier gates — bitwise, not "same SNR class"
Bottom rung against each higher rung, on identical weights, one S=1 decode step at a time,
fp32, window 128, on the length-bounded Metal kernel's registered torch reference:
legs
layers
tokens
argmax mismatches
bitwise-identical
shape sigs
c4096 vs c16384, masked + bounded
48
220
0/220
220/220
1 / 1
c4096 vs c16384, masked + bounded
6
4,096
0/4096
4096/4096
1 / 1
c4096 vs c8192, bounded kernel
48
220
0/220
220/220
1 / 1
c4096 vs c65536, bounded kernel
48
220
0/220
220/220
1 / 1
c4096 vs c131072, bounded kernel
48
220
0/220
220/220
1 / 1
c4096 vs c262144, bounded kernel
48
220
0/220
220/220
1 / 1
max|Δlogit| is exactly 0.0 at every step of every pair, so the verdict is bitwise
identity rather than a tolerance. Wraparound is exercised by construction in all of them
(92 ring writes land on a used slot at 220 tokens against a 128 window), and each rung
carries exactly one shape signature on each side — the property the whole design rests on,
measured per rung rather than assumed.
The top pair is the one that mattered: c4096 against c262144 is a 64× capacity ratio and
4,227,858,432 B of KV the small side never allocates, and it is bit-for-bit identical anyway.
What these gates do not touch: they exercise the bounded kernel's torch reference, not
its MSL (there is no Metal on the export box), and no tier has been run on device.
The per-tier cost, and what a 16 GB Mac needs to hold it
pinned KV is exact arithmetic from the state shapes: a global pair [8,1,1,C,512] at fp16 is
16,384 B per token of capacity, and the sliding ring [40,1,8,1024,256] is 0.3125 GiB
and identical in all seven rungs — it does not scale with capacity at all.
rung
global KV
ring
pinned KV
steady wired
requirement (steady + 1.25 floor)
what a 16 GB Mac needs
32 GB
c4,096
0.0625
0.3125
0.3750
7.03
8.28
a good quiet window
untested
c8,192
0.1250
0.3125
0.4375
7.10
8.35
a good quiet window
untested
c16,384
0.2500
0.3125
0.5625
7.22(measured)
8.47
a good quiet window
untested
c32,768
0.5000
0.3125
0.8125
7.47
8.72
a strong window
untested
c65,536
1.0000
0.3125
1.3125
7.97
9.22
a fresh-boot-class window
untested
c131,072
2.0000
0.3125
2.3125
8.97
10.22
24/32 GB machines
untested
c262,144
4.0000
0.3125
4.3125
10.97
12.22
24/32 GB machines
untested
All GiB. The forecast is blob + pinned KV(C) + R, where the tiered blob is 6.241 GiB and
R = 0.419 GiB is the non-KV resident remainder, obtained by decomposing the measured
7.22 GiB steady plateau at c16384. The c16384 row therefore reproduces a measurement by
construction, and every other row is that residual carried across.
The right-hand column is a window class, not a yes/no. Comparing each requirement against a
machine's median afternoon would assert that the only window a machine ever offers is its
median one, and this project's own record says otherwise. What the 16 GB reference machine has
actually been measured to hold:
reading
GiB
what it is
ordinary reclaimable
4.5–7.3
the median afternoon
a real admission
7.62
an 8B-class bundle was admitted on a sustained 7.62 GiB over 33 s and produced tokens. Windows in the 7.6+ class occur on this machine
best-ever
9.97
five days old at the time of writing, never repeated — an outlier, and still not the top of the range
So c4k/c8k/c16k are the ordinary case for a 16 GB Mac left alone for a few minutes; c32k is
plausible on a quiet machine and not something to promise; c64k is reachable and not
routine; and c128k/c256k are honestly out of reach on 16 GB. The 32 GB column is
"untested", not "yes" — this project has no 32 GB machine, reclaimable is not a fixed
fraction of RAM, and writing "yes" there would be a guess wearing a measurement's clothes.
There is no separate prefill-workspace adder. The prompt-length term was measured directly
— +0.018 GiB per 1,000 prompt tokens, within 18% of the KV-per-token arithmetic the table
already charges, and 20× below the naive extrapolation that had been feared. So the
requirements above stand as published, including c131,072 at 10.22 GiB. With int8 KV the
top rung would fall to 9.07 GiB; that lever is real, arithmetic and not yet built.
64k is the rung the evidence asks for
A retrieval-side audit of eleven real report questions against the app's own graph, run at each
rung with no model in the loop:
window
input budget
retrieval depth
GRAPH-BOUNDED
ceiling-capped
8,192
4,864
608
0
11
32,768
28,416
3,552
0
11
65,536
61,184
7,648
8
3
131,072
126,720
15,840
8
3
262,144
257,792
32,224
8
3
65,536 is where the ladder changes state. Eight of the eleven questions want
39,786–40,021 tokens; a 64k window offers 61,184, so all eight become graph-bounded — their
entire scoped universe fits, and no larger window improves those answers. 128k and 256k add
nothing for them at 1.00 and 3.00 GiB more KV. On evidence per gibibyte, 64k dominates 128k.
The other three questions want ~1,052,400 tokens and are unsatisfiable at every rung; what
256k buys them is that their need is finally computed against the whole pool instead of a
truncated one.
Selecting a rung
Do not read the bundle's language.max_context_length to size a prompt. On a tiered
bundle it is the top rung (262,144). A session that acquired main_c4096 and budgeted
against 262,144 would fail silently: the engine's stop is
min(requested, max(0, maxContextLength − input.count)), so it returns 0, yields nil before
producing anything, and the stream finishes empty. This is the one place a tiered bundle can
go wrong quietly.
metadata.json carries language.capacity_tiers[] — per rung function,
context_window_tokens, global_kv_bytes, ring_kv_bytes, total_kv_bytes — plus
capacity_tier_notes.global_kv_bytes_per_token, so a host can price every rung without
introspecting the graph.
If that field is ever absent, the rungs are still recoverable from function_map's main
role because each entrypoint is named for its capacity — but the KV cost is not in the
name, and a host must not invent one. An unpriced ladder should be climbed from the
cheapest rung, not the largest.
A downgrade is a new session. Re-acquiring rebuilds the cache and therefore discards the
KV, so "between-batch downgrade" is honest and "mid-session downgrade" is not.
tiered/ is decode-only: its function_map is {"main": [...]} with no prefill
function, so prompts are prefilled one token at a time through the same main.
Requirements
Apple silicon Mac, Core AI runtime. Mac-only; this does not fit an iPhone or iPad.
Engine contract: 2 inputs — input_ids [1,1], position_ids [1,S] → logits, no per-step
mask. That is the contract both engines accept, and the logits-capable sequential engine is
what grammar-constrained decoding needs.
dense2in* states:keyCache / valueCacheFloat16, 48 × 1 × 8 × ? × 512, dynamic
sequence dim → GrowingKVCache (initial 256, doubling). KV cost 786,432 bytes per token
— 3.22 GB at 4096, 12.9 GB at 16384. This dominates the dense2in* envelope.
tiered/ states: a global pair [8,1,1,C,512] at the selected rung plus a
capacity-invariant sliding ring [40,1,8,1024,256]. See the per-tier table.
dense2in* memory envelope:
int4
int8
Weights (main.mlirb)
8.71 GB
14.67 GB
+ KV at 4096
~11.9 GB
~17.9 GB
+ KV at 16384
~21.6 GB
~27.6 GB
Against a 16 GB Mac's Metal recommendedMaxWorkingSetSize of ≈ 10.7 GB, the int4
bundle's affordable context is roughly 2,500 tokens, and the int8 bundle does not fit
that machine at all. Minimum practical machine memory for dense2in*: 32 GB.
The bundle manifest declares runtime_env COREAI_CHUNK_THRESHOLD=1.
Measurements
No bundle in this repo has been measured on Apple silicon. What follows is measured on a
sibling build — gemma4_12b_qat_decode_int4linsym_e4_msdpa_g8_wkv_stable_c16384, the
single-capacity predecessor of tiered/, which is not published here — on a 16 GB M2 Pro under
a watchdog. It is included because it is the only Gemma 4 12B evidence this project owns and
because the tiered/ line inherits its graph.
value
answered
1 / 11, and that one did not parse
killed by
the wired ceiling — 11.13 GiB against 11.08
wired peak / growth
11.13 / 8.41 GiB (baseline 2.72)
steady plateau (18 min, 2,316 samples)
9.94 wired, 7.22 growth = blob × 1.157
decode
5.37 tok/s
TTFT, 3,228-token prompt
629.93 s
swapped out
1.52 GiB
shape signatures across 6,962 passes
1, stable=true throughout
ANE co-tenant busy
1 / 652 samples — the 8.41 GiB is the model's own
Three things come out of it, and they point in different directions:
Shape-stability held, under memory pressure severe enough to kill the run, and process
footprint grew 5.0 MB across 6,962 passes. The wired memory is held driver-side, not in the
process's own footprint. Respecialisation and leaks are both excluded.
The 12B does not fit a 16 GB Mac at c16384. A steady 9.94 GiB wired against a machine
whose ordinary reclaimable is 4.5–7.3 GiB is not a "wait for a quieter moment" case.
Most of the 5.37 tok/s is thrash, not architecture. A dense 12B against an 8B-A1B
activating ~1.5B parameters is perhaps a 6–8× compute ratio; the observed prefill ratio was
32×, on a machine that swapped 1.52 GiB and held reclaimable under 2.6 GiB for eighteen
minutes. An unpressured estimate is prefill in the tens of tok/s — an inference from the
ratio, not a measurement.
The plateau law is per-architecture, and this is what established that. Wired ÷ compiled
blob: LFM2.5 8B-A1B MoE 1.092 peak (flat in prompt length); Gemma 4 E2B 1.101 against
blob + PLE tables; Gemma 4 12B dense 1.157 steady / 1.348 peak, and not flat in prompt
length. A single constant applied to every bundle under-charges the dense 12B by 0.33 GiB at
its plateau and 1.51 GiB at its peak.
The c4096 re-export was run, and it bought 0.1875 GiB — the whole prize, because the
sliding ring is five-sixths of the KV at c4096 and does not scale with capacity. Steady demand
7.22 → 7.03 GiB, requirement 8.28 GiB. It works, it is free, and it does not rescue the 12B on
a 16 GB machine.
For context on the dense2in* bundles specifically, the only figure that exists is upstream's,
on hardware not used here: approximately 33 tok/s decode on an M4 Max for the int4 12B.
Read published Gemma 4 Q4 figures of 40–50 tok/s with care — those use Gemma 4's MTP
speculative drafters, which Core AI does not implement. Compare plain-decode to plain-decode.
Usage
Swift Package Manager, via CoreAIKit — a community
package, not affiliated with Apple, requiring macOS 27 beta:
ModelID addresses a bundle as repo + path + revision; the subtree downloads from the Hub
on first use and is cached afterwards:
swift
1importCoreAIKit23let model =ModelID(4"visible-cx/Gemma-4-12B-CoreAI",5 path:"dense2in-pf64/gemma4_12b_qat_decode_int4linsym_pf64")67var config =ChatSession.Configuration()8config.engineVariant =.sequential // required for guided / grammar-constrained decoding9let chat =tryawaitChatSession(model: model, configuration: config)
Unlike the E2B/E4B bundles in this org, these are plain 2-input graphs: no PLE table sidecar,
no static input buffers, and either engine will load them.
tiered/ needs a host that can name which of its seven main entrypoints to load and can
size prompts against the selected rung rather than the bundle scalar. ChatSession takes one
function name for the role today; driving a specific rung means loading the bundle directly
and passing that entrypoint's name.
Taken independently on both ends of the transfer that moved it, and matching. Core AI
.aimodel bundles are not byte-reproducible — the exporter is not deterministic even against
itself — so integrity rests on digests of the exact published bytes rather than on a rebuild.
STAGED — built, gated bitwise across all seven rungs in torch, never run on device. Established: the ladder converts, every rung reads back its own pinned extents, cross-tier logits are bit-for-bit identical, and the marginal cost of a rung is 756,215 B. Not established: the compiled blob (6.241 GiB is a forecast carried from the c16384 graph constant, and a graph constant is not a monotone function of the IR); whether AIModel exposes seven function names to loadFunction(named:) on device; and whether the length-bounded Metal kernel holds its per-step cost as capacity grows — that kernel has never been timed on hardware, and it is the entire argument for the upper rungs.
dense2in/gemma4_12b_qat_decode_int4linsym
EXPERIMENTAL — built and fingerprinted, unmeasured. Gate: a Mac-side decode oracle against an fp32 reference, then a guided-decoding run on a real schema.
dense2in-pf64/…_pf64
EXPERIMENTAL — built and fingerprinted, unmeasured. Same weights as dense2in plus the S=64 prefill function, which was worth 7.7× on time-to-first-token when measured on the E2B sibling. Same gate.
dense2in-int8/…_int8lin_pf64
EXPERIMENTAL — built and fingerprinted, unmeasured. Same gate, plus a quality comparison against int4; it exists as an anchor for the case where int4 phrasing quality fails qualification, at 1.7× the resident weights.
No numerics gate has been run on device for any bundle in this repo. The producer
fingerprint proves the toolchain, not the numerics.
License
Google publishes the upstream QAT checkpoint under Apache-2.0 with a
license_link to the Gemma 4 license, and
this repo mirrors that declaration. Use is governed by those terms and by the
Gemma Prohibited Use Policy; the
obligations travel with any redistribution of these bundles. The contribution here is the
conversion and the qualification evidence, not the weights.
tiered-v3/ — an fp16 vocab path, seven rungs, prefill and verify
The Quality-tier build, and the vocab path carries no int4 at all.
The embedding is fp16 and bit-exact to the checkpoint. Measured on the real tensor: the
fp16 cast of the bf16 table is lossless — worst |delta| 0.000e+00 — because bf16's 8
mantissa bits fit inside fp16's 11 and the table maxes at 1.3906, nowhere near fp16's range
limits. Not an approximation of the checkpoint's values; the values themselves.
That replaces a 4-bit embedding whose cost had never been priced against the checkpoint. When
it was, both candidates came out around 15–17 dB end-to-end (the previously published _e4
build measures 14.794 dB / 254-of-256 top-1), which is why this revision spends the bytes.
The head is fp16 too, because Gemma ties embedding and head: leaving the head int4 would
keep a blockwise_shift_scale reconstructing a full [262144, 3840] fp16 table for the
matmul. With both fp16 no full-table reconstruction remains anywhere on the vocab path.
Entrypoints. Seven capacity rungs (4k, 8k, 16k, 32k, 64k, 128k, 256k), each with a
last-row-only prefill_c<N> at S=64, plus verify_c262144 at S=8 returning FULL
[1, 8, 262144] logits — the shape a prompt-lookup or speculative loop needs to check eight
drafted tokens at once. Every entrypoint has zero symbolic extents.
Not yet run on a Mac. No token generated, no decode speed claimed.
gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/main.hash 32 B 2d9b6fde48da0ec177fadaf5f90c53bf86d5dcb5cbaa35397713fa89a005e50b
gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/main.mlirb 8,158,295,378 B 57bf5d507ca70e001c58bf72ea20308b232cb56835bfe3e5fcde13ac017cb34d
gemma4_12b_qat_decode_int4linsym_hf16_msdpa_g8_wkv_stable_pf64_tiered_c4k_c8k_c16k_c32k_c64k_c128k_c256k.aimodel/metadata.json 105 B 93893e9d87ff83172d1ca84eec61e939233e4956b3a677b49f873463c0e2499b
metadata.json 4,174 B c782edb705134877ac58bac96c9ffe7e8010b24dbac3c929ed91cb4073a55fda
tier_report.json 17,968 B b33ebdeb508d561805930a85f34acc2148dff2d91f20cab6e1ff2e2619229c19
tokenizer/chat_template.jinja 18,683 B ae53464bf3be25802b3a5b37def7fd89667067d7577049b3b2d74c4d8de4c6d4
tokenizer/tokenizer.json 32,169,626 B cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f
tokenizer/tokenizer_config.json 3,090 B c3fd34e321798e184e53894aaa1d8c4b37d08daf1fb1826dbceb3dcda0c62d06
