Every v79 binary here is verified end to end on Hexagon v79 (SM8750). Each was pushed to a
physical Snapdragon 8 Elite, loaded, and run to completion — coherence checked across multiple
prompts at temperature 0, with KPIs measured from those same runs.
The v81 binary is profiled on real v81 silicon but not executed end to end, because no
Snapdragon 8 Elite Gen 5 device is available to us for interactive runs. It is built from the same
checkpoint as the verified v79 build, with byte-identical KV quantization. What that does and does
not license is spelled out in its section below.
Qwen3-0.6B quantized for Snapdragon Hexagon NPUs, in two deployment paths:
4-bit — 4-bit weights / 16-bit activations, run through llama.cpp's Hexagon HTP
backend. Fully measured on real silicon, coherence verified across multiple prompts.
A16W8 — int8 weights / int16 activations / 16-bit KV cache, compiled to QAIRT Genie context
binaries. Statistically indistinguishable from fp32 on MMLU, with much faster prefill and init.
Published here for Hexagon v79, coherence verified on device.
Everything on this page was measured on a Snapdragon 8 Elite (SM8750, Hexagon v79) on Qualcomm
Device Cloud. Nothing is estimated from per-graph profiling unless explicitly labelled.
4-bit — on-device KPIs
Context 4096, all 87 layers resident on the NPU, no CPU or GPU fallback.
KPI
value
init (cold load)
~900 ms
TTFT (17-token prompt)
38 ms
decode
91.7 tok/s
prefill
3618 tok/s
peak RAM
466 MB host + 448 MB NPU-side KV
MMLU vs fp32
93.9%
Decode is quoted at an empty cache — the figure comparable to a short chat turn. It is not constant
with context:
KV depth
decode
prefill
0
91.7 tok/s
3618 tok/s
128
90.1
3278
256
80.3
3293
512
71.0
3046
768
60.1
2970
1024
55.5
2888
2048
43.1
2198
Peak RAM is two separate real numbers. Host resident memory is 466 MB; the KV cache is a further
448 MB allocated NPU-side, which does not appear in host RSS. Total device footprint is about 914 MB
at context 4096. The cache scales linearly with context, so at context 1024 it needs about 112 MB
instead of 448 MB. Reporting host RSS alone undercounts by the size of the cache.
One environment variable is worth 67% of decode*
Without it the backend waits on interrupts for batch completion; with it the backend polls.
configuration
decode @ empty cache
polling enabled
91.7 tok/s
Measured at an empty cache; 55.1 tok/s without it, hence the 67%.
* Polling is done on a CPU thread, so one CPU core will sit at 100%. Generally not
recommended when building an app.
Keep operator fusion on — disabling it costs 8% decode.
Do not quantize the KV cache. An 8-bit cache collapses prefill by up to 8.3x on these kernels
(3448 to 416 tok/s) and is slower at every realistic depth, because the HTP kernels have no
optimized quantized-KV path. It costs no measurable accuracy; the throughput penalty is what makes
it a bad trade.
Operator batch and queue depth are already tuned — raising either changed nothing measurable.
4-bit accuracy
Full MMLU, all 14,042 questions — the complete test set, not a subsample.
Evaluation protocol. All MMLU figures on this page are 0-shot, scored by loglikelihood
(the log-probability of each of "A"/"B"/"C"/"D" as a continuation, highest wins — nothing is
generated), over the complete 14,042-question test set. Quantized and baseline runs use the
identical protocol, so the difference between them isolates the quantization.
For reference, the published 0-shot MMLU for the unquantized Qwen3-0.6B is 0.4039, against the
0.4017 we measure for the same model in our own harness — a 0.0022 difference, well inside one
standard error. MMLU scores are only comparable when the prompting
protocol matches; figures quoted elsewhere under a different protocol are not comparable to these.
config
MMLU
±SE
above-chance capability retained*
fp32 parent (0-shot)
0.4017
0.0041
100.0%
4-bit (this build)
0.3924
0.0041
93.9%
* (acc - 0.25) / (fp32 - 0.25). MMLU is 4-way multiple choice, so a non-functional model still
scores about 0.25. This column is more informative than a raw percentage of fp32, which flatters
every model by roughly 62 points before it has learned anything.
The 4-bit QAIRT Genie build we previously published retained 50.8% on the same measure. This build
retains 93.9%.
KV-cache precision does not affect accuracy here — 16-bit scores 0.3924 and an 8-bit block-wise cache
scores 0.3915, a 0.0009 spread against a standard error of 0.0041. Keep the cache at 16-bit for
throughput, not accuracy.
Task dependence, stated plainly. A second 4-bit build scores higher on generative reasoning and
lower on MMLU: 0.376 vs 0.334 on 5-shot chain-of-thought GSM8K (500 questions), against 0.3551 vs
0.3924 on MMLU. There is no single "best 4-bit" answer — pick by workload. Note also that agreement
with the parent's next-token distribution does not predict task accuracy here: the build with
better top-1 agreement and lower KL divergence is the one that scores worse on MMLU.
4-bit coherence
Verified across multiple prompts at temperature 0.
prompt: What is the capital of France?
output: [thinking] ... France is a country in Europe, and the capital is Paris ...
The capital of France is **Paris**.
Correct, terminates cleanly, no repetition and no language drift. We test more than one prompt
deliberately — a single well-behaved sample is not evidence of coherence, and we have measured builds
of this model that answer one prompt cleanly and then degenerate on the next.
A16W8 — int8 weights, 16-bit KV cache
The higher-accuracy option: int8 weights, 16-bit activations, and a 16-bit KV cache.
Accuracy
Full MMLU, all 14,042 questions, with the complete deployed pipeline modelled — int8 weights, int16
activations, and the 16-bit KV cache. This is not a weight-only figure.
Same protocol as above: 0-shot, loglikelihood-scored, complete test set.
config
MMLU
±SE
above-chance capability retained*
fp32 parent (0-shot)
0.4017
0.0041
100.0%
A16W8, 16-bit KV
0.4036
0.0041
101.2%
* (acc - 0.25) / (fp32 - 0.25).
The difference from fp32 is +0.0019, about a third of one standard error, so it is statistically
indistinguishable from zero. The honest phrasing is not "lossless" but "any loss is below what the
test set can resolve". The cache dtype has no measurable effect among 16-bit formats: 16-bit integer
scores 0.4036, and the two 16-bit float formats score 0.4042 and 0.4044 — a 0.0013 spread against a
standard error of 0.0041.
On-device KPIs
Context 1024, same device and methodology as the 4-bit numbers above.
KPI
value
init (GenieDialog_create)
358 ms (335–378)
prefill
6428 tok/s
decode
72.0 tok/s (71.3–72.3)
TTFT
~36 ms (17-token prompt)
peak RAM
754 MiB host resident (753–755)
Init, decode and peak RAM are measured directly: six profiled runs of a 17-token prompt producing
88 generated tokens, with resident memory sampled during each run. The decode figure is insensitive
to how prefill time is apportioned — it lands at 71–74 tok/s across any plausible prefill value.
Prefill is not reported directly by the runtime — its profile emits only create/query/free — so it
is solved by regression over four prompt lengths (17, 126, 369, 639 tokens):
query_ms = 15.6 + 0.1556·n_prompt + 13.824·n_gen. The regression predicts 72.3 tok/s decode
against the 72.0 tok/s measured here, which is the cross-check for the prefill figure resting on it.
A16W8 decode is flat with context depth (13.79–13.83 ms per token from depth 17 to 782), because
its graph is compiled for a fixed context and always pays full padded-attention cost. The 4-bit build
attends only over live KV, so it is faster on short contexts and degrades as the cache fills. The two
cross at roughly 475 tokens, and over a full 1024-token generation they land within 4% of each other.
A16W8's decisive advantages are prefill (1.75x) and init (2.5x), not decode.
Why the KV cache is 16-bit
An 8-bit KV cache with a single per-tensor scale does not work for this model. It costs most of the
model's above-chance capability and makes generation degenerate on device. The cause is scale
granularity, not bit width: one static scale cannot cover the K/V dynamic range regardless of how
the range is chosen, and clipping to the 99.9th percentile recovers almost nothing. A block-wise
8-bit cache would be fine, but QNN has no block-wise equivalent, so 16-bit is the correct choice here.
Measured cost of the 16-bit cache on device: decode ~72 vs 77.0 tok/s (−6%) and +56 MB of buffers.
Cheap for the accuracy it buys.
Binaries
Published for Hexagon v79 (SM8750), built from the 16-bit-KV checkpoint and verified on device.
Each part carries both graph families: prompt_ar128_cl1024_* (prefill, 128-token chunks) and
token_ar1_cl1024_* (autoregressive decode).
The A16W8 binaries that previously sat here carried an 8-bit KV cache and have been removed.
Host libraries and DSP skels must live in separate directories — the hexagon-v79/unsigned tree
ships a 32-bit libQnnSystem.so that overwrites the aarch64 host copy if both are pushed into one
directory, after which startup fails with a 32-bit/64-bit error.
Rename the two .bin files to match the ctx-bins entries in the config, or edit the config to
match. Verify what you downloaded with sha256sum -c SHA256SUMS.
Prompt format (required)
Prefill the opening <think> tag as part of the assistant turn:
This matches Qwen3's own chat template in thinking mode. It matters for this build: if the model is
left to emit <think> itself as the first generated token, a single spurious token can follow it and
derail the response, sometimes switching the reasoning into another language. With <think>
prefilled, generation is coherent across prompts, closes </think> correctly and terminates on EOS.
Prompts that do not use the chat template at all are unaffected.
Verified at temperature 0 on four prompts (capital of France, haiku, arithmetic, gravity):
prompt: What is the capital of France?
output: Okay, the user is asking for the capital of France. ... Just state the capital.
</think>
The capital of France is Paris.
A16W8 for Hexagon v81 (SM8850, 8 Elite Gen 5)
Same 16-bit-KV checkpoint as the verified v79 build, compiled for v81 with the same QAIRT 2.45.
The KV quantization is byte-identical (past_key_0_in scale 0.01514680590480566 in both), so this
is the same numerics on a different Hexagon version.
Measured on v81 silicon (per-graph)
graph
latency
peak memory
part2token_ar1_cl1024 (decode)
15.326 ms
77.9 – 88.3 MB
part2prompt_ar128_cl1024 (prefill)
22.531 ms
64.0 – 74.9 MB
part1token_ar1_cl1024 (embedding)
0.040 ms
~10 MB
Which gives decode ~65.1 tok/s and prefill ~5,681 tok/s (128 tokens per 22.531 ms chunk).
These are per-graph figures and are not comparable like-for-like with the end-to-end v79 numbers
above. Running the same per-graph method on v79 yields 65.8 tok/s, against 72.0 tok/s measured
end-to-end — so this method understates by roughly 9%. Applying that correction suggests v81 would
land near 71 tok/s in practice, but that is an inference, not a measurement.
The headline result is that v81 is not faster than v79 for this model: 15.326 ms vs 15.2 ms per
decode step, within a percent. Decode is DRAM-bandwidth-bound, and a newer Hexagon version does not
move that wall.
What is and is not established
Carries over from the v79 verification, because both are properties of the weights and the
quantization scheme: MMLU 0.4036, and the <think> prefill requirement documented above.
Not established: that this binary loads and runs on a real Gen 5 device, its init time, TTFT, host
peak RAM, and thermal behaviour. A successful compile is not evidence of a working binary — a
2.47-compiled build of this same checkpoint compiled cleanly and then failed to initialise on device.
If those read CPU, the model is on the CPU. A 0.6B 4-bit model prefills in the thousands of tokens
per second on the HTP and roughly an order of magnitude slower on the CPU, which is a second,
independent check.
Known gotchas
GGML_HEXAGON_NHVX=8 hangs. v79 has six HVX units; leave the variable unset.
llama-cli blocks on stdin even with -no-cnv. Pass -st and redirect stdin from /dev/null, or
it appears to hang forever and any memory sampled reads as a few MB from a process that never
loaded the model. llama-bench is the reliable timing tool.
llama-cli and llama-bench do not take the same flags: --mmap 0 and -fa 1 work for
llama-bench, while llama-cli needs --no-mmap and -fa on.
llama-cli prints no load log or buffer sizes without --verbose, which makes memory look
unmeasurable when it is not.
VTCM on v79 is 8 MB, so no model of this size is VTCM-resident. Weight-placement tuning does not
help decode on this backend; the polling flag does.
For the Genie path, host libraries and DSP skels must live in separate directories.
hexagon-v79/unsigned/ ships a 32-bit DSP libQnnSystem.so that silently overwrites the aarch64
host copy if both are pushed into one directory, and startup then fails with "is 32-bit instead of
64-bit".
Measurement methodology
Every figure was measured on physical silicon. Prefill and decode for the 4-bit build are
llama-bench with 3-5 repetitions and reported standard deviations. TTFT and init come from the
runtime's own timing output. Peak host memory is a resident high-water mark sampled during the run;
NPU-side memory is the runtime's per-backend buffer accounting. Accuracy is the full 14,042-question
MMLU test set, 0-shot and loglikelihood-scored. A16W8 init, decode and peak RAM are measured directly from profiled runs; only A16W8
prefill is derived by regression, because the Genie profile does not separate the phases.
Accuracy is a property of the weights, so it does not change with Hexagon architecture version. Only
speed and memory do.
