Gemma-4-E2B-it — 8-bit and 4-bit quantization for the Qualcomm Hexagon NPU
google/gemma-4-E2B-it quantized two ways, both running on the Hexagon NPU:
8-bit quant — int8 weights with int16 activations, compiled to QNN context binaries
that run 100% on the Hexagon NPU with no CPU or GPU fallback. Needs the QAIRT SDK.
4-bit quant — q4_0 group-32 weights as a GGUF executed on the NPU through llama.cpp's
ggml-hexagon backend. No QAIRT SDK, smaller on disk, and faster on decode.
Most shipped mobile builds of this model use int8 activations. Keeping activations at 16 bits
costs some memory and holds onto accuracy: on a held-out 400-question MMLU slice the 8-bit
build is statistically indistinguishable from the unquantized model.
Read the verification table before using a binary. Not every target below has been run
on physical hardware, and this README states exactly which have. As of 2026-07-27 the 8-bit
prefill binary is verified on both v79 and v81 silicon; v81 decode is not verified.
8-bit — verification status per target
Everything below was measured on physical hardware — Snapdragon 8 Elite (SM8750, HTP v79)
via Qualcomm Device Cloud, device 41c5710f, 2026-07-26, and Snapdragon 8 Elite Gen 5
(SM8850, HTP v81) via Qualcomm AI Hub inference, 2026-07-27. Each table says which.
Decode — the generated text is token-for-token identical to the float ONNX reference:
prompt : "The capital of France is" (Gemma-4 chat template)
device : 'The capital of France is **Paris**.'
float : 'The capital of France is **Paris**.'
The/ capital/ of/ France/ is/ **/Paris/**., terminating correctly on <turn|>.
Two consecutive runs produced byte-identical output, so decoding is deterministic on device.
Prefill — 12 held-out chat prompts, each one forward pass, compared against float:
12/12 (100%) next-token top-1, hidden cosine mean 0.992 / min 0.986.
Speed ladder (v79, 8-bit, AI Hub profile on real hardware)
decode graph
ms/step
tok/s
speedup
naive full-KV
307.9
3.25
1.0×
+ windowed KV + broadcast-GQA
69.8
14.3
4.41×
+ int8-KV (full-attention slots)
65.3
15.3
4.71×
int8-KV cost no accuracy on the held-out check (32/32 content-token agreement, unchanged).
v81 (hexagon-v81 / SM8850 / 8 Elite Gen 5) — prefill verified, decode still untested
Measured on real Snapdragon 8 Elite Gen 5 silicon via AI Hub inference, 2026-07-27.
binary
role
on hardware
perf
gemma4_trunk_a16w8_v81.bin
prefill, fixed SEQ=128
✅ 11/12 next-token, cos mean 0.991 / min 0.981
—
gemma4_decode_wgqa_int8kv_a16w8_v81.bin
decode
⚠️ still unverified — see below
profiles at 57.4 ms/step (17.4 tok/s)
Prefill on v81 matches v79 bit-for-bit. The 12 held-out prompts were run through the v81
trunk binary, and the same harness was run against the v79 trunk binary as a control, so the
two are directly comparable rather than being compared across measurement paths:
trunk binary
device
prompts
next-token top-1
hidden cos (mean / min)
gemma4_trunk_a16w8_v81.bin
SM8850 (v81), AI Hub
12
11/12
0.99077 / 0.98050
gemma4_trunk_a16w8_v79.bin
SM8750 (v79), AI Hub
2 (control)
1/2 — same prompt flips, same cosines
0.99234, 0.98050
gemma4_trunk_a16w8_v79.bin
SM8750 (v79), adb / Device Cloud
12
12/12
0.992 / 0.986
The v79 rows are a control on the measurement path, not a second verification: the AI Hub
row deliberately re-ran only the harness-validation prompt and the one prompt v81 flipped.
The single v81 disagreement is "Name a planet with rings.", where float opens '**' and the
device opens 'The' at cos 0.9805 — a near-tie between two plausible sentence openings, not a
degradation. Re-running that exact prompt on the v79 binary reproduced the identical
mismatch at an identical cosine of 0.9804982542991638, so it is a property of the 8-bit
quantization, not of v81. On this evidence the two architectures are numerically
indistinguishable on the trunk.
Decode on v81 remains unverified. An earlier spot-check of the v81 decode binary showed
hidden cosine degrading 0.912 → 0.848 → 0.794 → 0.839 across prefill steps 0–3, with the
hardware norm about half of float at step 2, and was stopped before the token comparison. That
result has not been explained or reproduced, and the trunk result above does not clear it:
the trunk is a different graph. Two candidate explanations remain open — the decode harness
itself, and the fact that the v81 decode binary was compiled from the int8-KV export
(decode_wgqa_A16W8_int8kv_full) whereas the token-exact v79 verification used the plain
WGQA export. Treat gemma4_decode_wgqa_int8kv_a16w8_v81.bin as "failed a spot-check, cause
unknown" — not as broken, and not as usable.
Why decode is expensive to verify: Qualcomm Device Cloud provisions v79 parts only, so
there is no adb-attached v81 device, and AI Hub inference bills one farm job per decode step
(~23 jobs for one short sentence). The trunk is stateless — one forward per prompt — which is
why prefill could be verified for 12 jobs and decode was not.
If you have v81 hardware, the useful next step is decode: run the loop per How to run
and compare against float. If it diverges, re-verify against a v81 build of the non-int8-KV
export to isolate whether int8-KV interacts badly with v81, and check that your QAIRT install
ships a hexagon-v81 skel matching the compile.
What is not verified
Long-context / ring-wrap. The windowed graph uses 512-entry ring buffers on the sliding
layers. Prompts long enough to wrap the ring (>512 tokens) were not exercised on
hardware. Verified prompts were ~14–18 tokens.
Prefill beyond 128 tokens. The trunk graph is a fixed SEQ=128 window. Longer prompts are
not covered by it at all.
Generative benchmarks (GSM8K etc). Not measured. MMLU is one forward pass per question;
multi-step generative reasoning compounds error over hundreds of steps and is untested here.
Throughput is NPU inference time from an AI Hub profile job, not end-to-end tokens/s.
Host-side embedding lookup and lm_head are excluded; a real application adds those, and the
net-run harness used for correctness reloads the context each step so its wall-clock
(~3.6 s/step) is not a throughput number.
8-bit accuracy
Held-out MMLU, 0-shot, chat-formatted, 400 questions disjoint from all calibration data:
accuracy
base model (float, ≡ HF)
56.75% ± 2.48
8-bit (this build)
59.25% ± 2.46
delta
+2.50 pp
random baseline
25.00%
The +2.50 pp delta is about one standard error — not evidence that quantization improves
the model. The correct reading is that 8-bit costs no measurable MMLU accuracy. Note the two
models disagree on ~24% of individual questions; they match in aggregate, not per-question.
Two things you must get right
1. Use the chat template. The raw completion format makes this instruction-tuned model
degenerate. Verified on the unquantized model, so this is not a quantization artifact:
format
output
raw + greedy
' France is France is France is…'
raw + temperature / top-p
byte-identical degeneration
raw + repetition_penalty 1.2
byte-identical degeneration
chat template + plain greedy
'The capital of France is **Paris**.'
Token layout (verified byte-exact against transformers.apply_chat_template):
2. Match the mask constant. Attention masks use a finite NEG = -1e4, not -inf or
float32.min. -inf cannot survive int16 activation quantization — it blows out the range so
real scores round to zero. -1e4 still zeroes the softmax while leaving real scores resolved.
The host must use the same value the model was calibrated with.
Architecture
The graph is split so the >2 GB vocab tensors never enter it:
Host (CPU/ARM): token embedding lookup, per-layer embedding lookup, and the tied
lm_head with 30·tanh(x/30) softcap.
NPU: the transformer decode graph, with the KV cache resident on device.
Gemma-4-E2B is dense: 35 layers, hidden 1536, GQA 8 query heads → 1 KV head, head_dim 256,
262144-token vocab, and hybrid attention (28 sliding-window layers of window 512, interleaved
with 7 full-attention; KV shared across the last 20 layers, so only 15 layers store KV).
Why the fast graph is fast
Decode is KV-attention-bound, not weight-bound. The published binary uses two changes over
a naive full-KV decode graph:
Windowed KV — sliding-window layers use a 512-entry ring buffer instead of a full 4096
buffer; only the 3 full-attention layers keep 4096. KV traffic per step drops from ~288 MB
to ~63 MB. The ring index is computed inside the graph as cache_position % buf, so the
host just passes pos.
Broadcast GQA — the expand op that materialized 1 KV head into 8 copies is removed
(verified: 0 Expand nodes in the exported ONNX).
Net effect on v79: 307.9 ms → 69.8 ms per decode step (4.41×).
Credit: these two levers come from the tps/ work in
gemma-4-e2b-hexagon-npu — this repo contributes a corrected
quantization of that graph.
The three host-model tensors are ~5.2 GB and stay on the host by design — putting the
262144-token vocab in the graph blows past ONNX's 2 GB protobuf limit.
How to run the 8-bit build
What you need that is NOT in this repo
You cannot run this from this repo alone. One dependency is missing by necessity:
Qualcomm AI Engine Direct (QAIRT / QNN) SDK — supplies qnn-net-run and
libQnnHtp*.so plus the HTP Stub/Skel pair for your Hexagon version. These are
Qualcomm-licensed and not redistributable here, so you must install the SDK yourself
(free, from Qualcomm). Built and tested against QAIRT 2.45.
v81 needs the V81 equivalents. Check your SDK actually ships hexagon-v81;
older installs do not.
A device: Snapdragon 8 Elite (SM8750, v79) or 8 Elite Gen 5 (SM8850, v81), reachable
over adb. Qualcomm Device Cloud works — that is what this was verified on.
Host Python 3.9+ with numpy and tokenizers (pip install -r requirements.txt).
No torch, no transformers needed to run — only to reproduce the quantization.
~6 GB free on the host for the embedding tensors, ~4 GB free on the device
(/data/local/tmp) per pair of context binaries.
Never run adb kill-server against that tunnel — it kills the remote pod's adb server,
which you cannot restart without portal access.
Step 3 — stage onto the device
bash
1exportQAIRT_DIR=/path/to/qairt/2.45.0.xxxxxx # your SDK install2./runtime/stage_device.sh <serial> v79 # or: v81
This pushes qnn-net-run, the HTP libs/skels, the matching *_v79.bin context binaries, and
the on-device step scripts. Two 1.9 GB pushes over adb take a while; over a QDC tunnel a
single stream runs ~1 MB/s, so expect ~30 min unless you parallelise (see Slow adb below).
KV buffers are not pushed — run_gate.py creates them on device with dd.
Step 4 — run the autoregressive loop
bash
1python runtime/run_gate.py \2 --prompt "The capital of France is"\3 --ntokens 14\4 --adb-serial <serial>\5 --chat --wgqa \6 --script gate_ondevice_int8kv.sh
Expected output:
continuation: 'The capital of France is **Paris**.'
Flags that matter:
flag
why
--chat
required. Without it the -it model degenerates into ' France is France is …'
--wgqa
required for these binaries — selects 512-entry ring buffers and the 512-wide sliding mask
--script
pick the binary: gate_ondevice_int8kv.sh (recommended) or gate_ondevice_wgqa.sh
Drop --script to use the non-int8-KV binary.
Step 5 (optional) — check prefill
verify_trunk.py compares the trunk against a float reference. Producing that reference
needs torch + transformers on a host that knows the gemma4 architecture (transformers
≥ 5.12 — older versions raise KeyError: 'gemma4'), so it is a reproduction step rather than
part of normal use.
4-bit — q4_0 weights on the Hexagon NPU
The 4-bit path uses a different runtime. It is not a QNN context binary: it is a GGUF executed
on the Hexagon NPU through llama.cpp's ggml-hexagon backend, which drives the DSP directly
over FastRPC instead of through QNN. It needs no QAIRT SDK.
On the same physical v79 silicon it is 2.5× faster on decode than the 8-bit build, fits
entirely in one 2.44 GiB file with no host-side tensors, and costs ~0.4 pp of MMLU:
4-bit / ggml-hexagon
8-bit / QNN
model on disk
2.44 GiB total
3.54 GiB + 5.2 GB host tensors
decode, sustained on device
37.8 – 38.5 tok/s
15.3 tok/s
prefill, NPU
1145 tok/s
—
peak device RSS
3.84 GiB
not measured
held-out MMLU
57.76% ±0.99
59.25% ±2.46
vs unquantized base
−0.44 pp
+2.50 pp
GSM8K (5-shot, chat)
0.5200 / 0.6467
not measured
embeddings
inside the file
5.2 GB on host
QAIRT SDK required
no
yes
Granularity, not bit-width, is what decides accuracy here. This build is group-32; at the
same 4 bits a per-channel grid costs ~5.3 pp of MMLU, measured by quantizing the same parent
checkpoint both ways on the same harness:
grid, same parent checkpoint
MMLU (n=2280)
unquantized parent
57.68% ±0.99
group-32 (this build)
57.76% ±0.99
per-channel int4
52.46% ±1.01
Group-wise 4-bit was previously ruled out on this graph because LPBQ stores 4-bit values in an
int8 container and runs 9.3× slower. That conclusion still holds for QNN — it does not apply
here, because ggml-hexagon consumes q4_0 blocks natively with its own HTP kernels rather
than going through QNN's blockwise path.
4-bit verification status
Measured on physical Snapdragon 8 Elite (SM8750, HTP v79) via Qualcomm Device Cloud, device
87b3a4aa, 2026-08-05. Every run started from 42 °C.
Throughput
dev
prefill pp128
decode tg32
HTP0 (NPU)
1145.39 ± 7.89 tok/s
28.50 ± 0.06 tok/s
CPU (6 threads)
225.17 ± 0.46 tok/s
42.48 ± 0.08 tok/s
NPU prefill is 5.1× CPU; CPU decode is 1.5× NPU. Decode is weight-bandwidth-bound, so the
NPU's compute advantage does not help it — prefill is compute-bound, where it does. A hybrid
placement (NPU prefill, CPU decode) is the strongest configuration on this hardware.
Sustained decode — and a measurement discrepancy you should know about
Five consecutive 128-token requests through llama-server on HTP0:
run
tok/s
peak RSS
hottest thermal zone
1
38.47
4,029,912 kB
79.2 °C
2
38.27
4,031,796 kB
83.4 °C
3
38.13
4,031,796 kB
83.8 °C
4
38.06
4,031,796 kB
86.1 °C
5
37.79
4,031,796 kB
87.2 °C
Latency and memory
TTFT 61.19 ms for an 8-token prompt. Not comparable to context-512 TTFT figures.
Peak RSS 3.84 GiB (VmHWM 4,031,796 kB) for a 2.44 GiB file — ~1.4 GiB of host-side
buffers on top of the weights.
The ggml-hex log reports the DSP session mapping budget as vmem 3355443200 =
3.125 GiB. RSS exceeds it, so not everything is resident in the session. A model whose
weights alone approach 3.1 GiB will need multi-device layer splitting.
Thermals
42 °C → 87.2 °C peak across 5×128-token bursts, with 1.8% decode decay. Hot but not
throttling within that window. A multi-minute soak has not been run — do not assume this
holds for continuous generation.
Generation quality on device
Greedy, raw completion (no chat template):
prompt
output
"The capital of France is"
" Paris."
"Who wrote Hamlet?"
"The answer is William Shakespeare."
"What color is grass?"
(empty)
"Name a country in South America."
"Brazil"
The empty response on "What color is grass?" is a prompt artifact, not a quantization
defect: raw completion emits an immediate end-of-turn, and Google's own q4_0 GGUF returns the
identical empty string on that prompt. Use the chat template.
Accuracy — measured on CPU, not on device
benchmark
4-bit
parent (unquantized)
base fp32
MMLU, 0-shot, n=2280
57.76% ±0.99
57.68% ±0.99
58.20% ±0.99
GSM8K 5-shot chat, strict
0.5200 ±0.041
0.5200 ±0.041
—
GSM8K 5-shot chat, flexible
0.6467 ±0.039
0.6467 ±0.039
—
top-1 agreement vs parent
98.2%
—
—
KL divergence vs parent
0.00173
—
—
GSM8K is identical to the unquantized parent to the digit, standard errors included. MMLU is
+0.08 pp — noise. Token-level top-1 agreement is 98.2% and KL is 1.7×10⁻³, so the two models are
not bit-identical per token; they match on the benchmarks in aggregate.
How this was measured, and its limit: the GGUF was dequantized and its tensors substituted
into the parent architecture, then evaluated with lm-evaluation-harness on CPU. That measures
the weight grid, which is what quantization changes — it does not exercise the HTP
kernels. On-device accuracy at scale is unverified; the four coherence prompts above are the only
device-side quality evidence.
Two harness details that otherwise waste time: MMLU needs tokenizer.add_bos_token=True or
loglikelihood-MC scores at chance; GSM8K needs apply_chat_template=Trueandadd_bos_token left off, because the template already emits <bos> and forcing it
double-prefixes. GSM8K run 0-shot raw scores strict-match = 0.0 — the model never emits the
#### N format, so that is a broken measurement, not a bad model.
What is not verified
v81.libggml-htp-v81.so builds and is auto-selected by arch, but has never been run —
Device Cloud provisions v79 only. Qualcomm's published figures for a q4_0 GGUF on v81 show
NPU decode lower than v79 (17.61 vs 26.66 tok/s at ctx 512) with prefill 36% higher, so do
not assume v81 is an upgrade for decode.
Long context. Nothing beyond depth 4096 exercised. The tg32 @d4096 figure was itself
unstable across sessions (21.90 ±0.97 vs 27.47 previously), so it is not quoted here.
Sustained multi-minute load, batch > 1, and on-device accuracy at scale.
How to run the 4-bit build
What you need that is not in this repo
Only a device — no QAIRT/QNN SDK, and no build step. The llama.cpp/ directory in this repo
is a prebuilt arm64-Android runtime, so unlike the 8-bit path this one is self-contained:
A device: SM8750 (v79) or SM8850 (v81) over adb.
~3 GB free on the device, ~4 GB RAM headroom for the process.
What is in llama.cpp/
llama.cpp/bin/llama-cli llama-server llama-bench arm64 Android, the three tools below
llama.cpp/lib/*.so 22 MB 11 arm64 libs + the v79/v81 DSP libs
llama.cpp/LICENSE MIT, plus the DSP-library note
Built from upstream llama.cpp 0ef6e55 in ghcr.io/snapdragon-toolchain/arm64-android:v0.7
(NDK r28b, Hexagon SDK 6.6). Hexagon support and the gemma4 architecture both need a build from
2026-06 or later, so an older llama.cpp will not work. The commit is recorded inside
libllama-common.so if you want to check what you got.
The arm64 libraries are stripped of debug info (175 MB → 20 MB); every PT_LOAD segment and
every SHF_ALLOC section is byte-identical to the unstripped build, and the DSP libraries are
shipped as built. This is the exact dependency closure of the three tools — nothing spare, and
nothing missing.
libggml-htp-v79.so / -v81.so are the DSP-side kernels; the right one is selected
automatically from the detected arch, so unlike QNN there is no version pinning to get right.
They need no Qualcomm libraries at runtime — their qurt_*, compute_resource_* and
dspqueue_* symbols resolve on-device against the vendor DSP image. That is also why
ADSP_LIBRARY_PATH must include the vendor paths (Step 2). If you want v73/v75 as well, or a
build without the unused libggml-opencl.so dependency, build it yourself:
A full clone of this repo is ~17 GB, almost all of it the 8-bit binaries and their host tensors.
The 4-bit path needs 2.7 GB of it, so fetch only that:
adb push does not preserve the executable bit, hence the chmod. Over a QDC tunnel expect
2.6–7.6 MB/s, so 10–20 min for the model — the runtime itself is 22 MB and lands in seconds. The
Slow adb section below applies to the model.
Step 2 — the one non-obvious requirement
ADSP_LIBRARY_PATH is mandatory. Without it the DSP loader cannot find the skel, and the
error looks like a hardware or arch problem but is not:
ggml-hex: failed to open session 0 : error 0x80000406
1curl -s localhost:8099/completion -H 'Content-Type: application/json'\2 -d '{"prompt":"Write a paragraph about on-device inference.","n_predict":128,"temperature":0}'
The timings block carries predicted_per_second, prompt_ms (TTFT) and, with a drafter
loaded, draft_n / draft_n_accepted.
flag
why
-dev HTP0
run on the NPU. -dev none = CPU. -dev HTP0,none benchmarks both
-fa on
flash attention; required for the windowed-KV path
-t 6
matches the HTP's 6 HVX contexts
--parallel 1 --no-cont-batching
single-stream, matching the quoted figures
-c 4096
context; nothing beyond this is verified
Quantized KV (-ctk q8_0 -ctv q8_0) is not supported on this backend — it aborts with
GGML_ASSERT(offset == 0) in ggml-hexagon.cpp. On CPU it works but is slower than f16
(25.66 vs 34.38 tok/s at depth 4096).
Step 4 — speculative decoding with the MTP drafter
gemma4-e2b-w4-mtp.gguf is a Multi-Token Prediction drafter (~57 MB) that shares the target's
KV cache. It needs a llama.cpp build newer than 2026-06-08 (arch gemma4-assistant).
The drafter cannot be instantiated standalone (failed to create context with model — it
requires the target's context), and the startup warning [spec] failed to measure draft model memory is benign; the drafter loads afterwards and common_speculative_init_result confirms it.
4-bit troubleshooting
symptom
cause
Permission denied running ./bin/llama-cli
adb push drops the executable bit; chmod +x the three tools (Step 1)
library "libllama-common.so" not found
LD_LIBRARY_PATH not pointing at llama.cpp/lib (Step 2)
failed to open session 0 : error 0x80000406
ADSP_LIBRARY_PATH not set — the DSP cannot find the skel
GGML_ASSERT(offset == 0) in ggml-hexagon.cpp
quantized KV cache (-ctk/-ctv) unsupported on HTP
failed to create context with model 'gemma4-e2b-w4-mtp.gguf'
the drafter needs the target's context; do not load it alone
503 {"error":"Loading model"}
server not ready. /health returns 503 while loading, so poll for HTTP 200, not just any response
decode slower on NPU than CPU
expected — decode is bandwidth-bound. Use the NPU for prefill
llama-cli idles at a > prompt
it entered conversation mode; use llama-server, or check this build's single-turn flag
arch gemma4 / gemma4-assistant unknown
llama.cpp too old
4-bit limitations
Decode figure is unsettled — 28.50 (llama-bench) vs ~38 tok/s (llama-server), cause
unresolved. See above.
On-device accuracy is unverified at scale. MMLU/GSM8K were measured on CPU against the
weight grid, not through the HTP kernels.
v81 has never been run, and Qualcomm's own v81 figures suggest decode is worse there.
No compiled artifact. Unlike the 8-bit context binaries, there is no AOT compile
step — ggml-hexagon reads the GGUF at runtime. Converting this to a QNN context binary is
not a shortcut: QAIRT's gguf_builder dequantizes rather than preserving the group-32 grid,
which is the property doing the work.
Quantized KV cache unsupported on the HTP backend.
Beyond 4096 context, batch > 1, and multi-minute sustained load are all untested.
How the loop actually works
run_gate.py is the reference implementation, and deliberately simple:
Tokenize with the chat template (hostlib.encode_chat).
Zero the KV buffers on device.
For each position: look up embeddings on the host, write the six small per-step tensors,
run qnn-net-run once on device, pull back hidden (1536 floats).
Apply the tied lm_head + 30·tanh(x/30) softcap on the host, take the argmax.
The on-device script renames present_* → past_* so KV never crosses adb.
Stop on <turn|> (106) or <eos> (1).
This harness is for correctness, not speed. It re-loads the 1.9 GB context binary every
step, so its wall clock (~3.6 s/token) is ~50× worse than the NPU's actual 65 ms. A real
application loads the context once, keeps KV device-resident, and does embeddings +
lm_head in-process. Building that is left to you.
Slow adb
A single adb stream over a QDC tunnel is bandwidth-delay-product limited (~1 MB/s), not
bandwidth limited. Splitting the binary and pushing chunks over separate SSH tunnels (one
local port each) reached ~7 MB/s:
bash
1split -n 6 -d gemma4_decode_wgqa_int8kv_a16w8_v79.bin chunk.
2# ...one `ssh -L 503X:$HOST:5037` per chunk, then push each with its own3# ADB_SERVER_SOCKET=tcp:127.0.0.1:503X, then on device:4adb shell 'cd /data/local/tmp/gemma/artifacts && cat chunk.* > out.bin && rm chunk.*'
Verify the checksum afterwards (SHA256SUMS) — and wait for all pushes to finish before
concatenating, or you will silently assemble a truncated file.
Troubleshooting
symptom
cause
Could not create context from binary
HTP arch mismatch — a v81 binary will not load on a v79 device, or vice versa
Cannot assign data from unexpected type. Expected int32, got int64
binaries are built with --truncate_64bit_io, so position_ids/cache_position are int32
Output repeats ' France is France is …'
--chat missing
Fluent but wrong answer
binary/mask mismatch — the host NEG must be -1e4, matching calibration
Garbage tokens, hidden norm ≈ 0
wrong context binary, or KV buffers not zeroed before position 0
Different output across identical runs
a KV buffer was corrupted mid-push; re-seed (dd on device) and retry
Fluent garbage from a binary you compiled yourself
local qnn-context-binary-generator emits UFIXED_POINT_16 IO; the harness writes fp32. Use the AI Hub build, or check graphInputs[].dataType
Could not create context from binary on a binary that used to work
QAIRT runtime version does not match the version that compiled it
Reproducing the 8-bit quantization
AIMET QuantizationSimModel, param_type=int8, activation_type=int16,
quant_scheme=min_max, calibrated on real activations captured from chat-formatted decode
loops — not random noise, and not raw-format text.
Both of those details matter and each caused a distinct on-device failure:
Calibrating on np.random.randn produced a binary whose residual stream collapsed to zero
on hardware (final hidden norm 0.0000 → pure noise tokens), even though it compiled fine.
Calibrating on raw-format text and then running chat-format prompts produced fluent but
unfaithful output on hardware — the chat template's special tokens hit activation ranges the
quantizer never observed.
min_max outperformed tf_enhanced here: on int16 there are 65k levels, so range coverage
matters more than outlier clipping, and tf_enhanced mis-estimated the range badly enough to
inflate hidden norms ~10×.
8-bit limitations
Fixed 4096 context.
Prefill is not included in this repo; the decode graph can prefill token-by-token, which is
slow for long prompts.
Batch size 1 only.
--truncate_64bit_io at compile time means index inputs (position_ids, cache_position)
are int32 on the compiled binary, though the float ONNX takes int64.
