Views
No views yet
The GGUF Q4_0 path on the Hexagon NPU is effectively W4A16 compute:
Weights: Q4_0 — 4-bit, one FP16 scale per 32-element block, repacked on-chip into the HTP tiled format (32×32 tiles). Dequantized on the fly by the HTP (W_DEQUANT PMU events).
Activations: FP16 — the HMX (Hexagon Matrix eXtension) engine is a 16-bit FP matrix unit; inputs/activations flow as FP16.
Accumulation: FP32 — dot-products accumulate in 32-bit inside the matmul, then cast for the output tensor.
KV cache: Q8_0 (int8) — auto-enabled on the npu/HTP0 path (along with flash-attention).
Norms/scales: F32 — the small F32 tensors (RMSNorm γ, rope, layer scales) run as FP32.
So: Q4_0 weights × FP16 activations → FP32 accumulate, KV in Q8_0. That's the same w4a16 profile the QAIRT qairt bundles use — the difference is the GGUF path dequantizes Q4_0 blocks in hardware instead of using precompiled int4 context binaries.doesnt seem to be a difference between hybrid and npu. why?
Because on your 8 Gen 2 they collapse to the same thing:
hybrid = llama.cpp's per-tensor scheduler across all registered backend devices.
npu = pin everything to HTP0, unsupported ops fall back to CPU.
The scheduler's advantage only appears when multiple HTP devices exist (HTP0+HTP1…). But we found the source guard earlier: on Android with arch < 75 (v73), opt_ndev is forced to 1 — GenieX registers exactly one HTP session on your chip. So:
hybrid → puts HTP-supported ops on the only HTP device (HTP0), CPU for the rest.
npu → same thing, via pinning.
Identical placement → identical speed. There is one hidden difference: pinning HTP0 also enables flash-attn + Q8_0 KV cache (hybrid doesn't get that bonus) — but on your short 8-token test that's invisible. You'd only see it on long-prompt/long-context runs.
On an 8 Elite (v75+), GenieX registers 2–3 HTP sessions, and there hybrid genuinely beats npu (that's the ~30% bump Qualcomm documents). On v73, don't bother switching — they're the same.