Operator-fused variants of the vision and action_step components of
allenai/MolmoAct2-LIBERO, converted to
Qualcomm QNN (HTP) context binaries for Dragonwing IQ-9075 (QCS9075, Hexagon v73, soc_id 77).
This repo is a companion to
xpuenabler/molmoact2-libero_grid_sampler_qnn:
it ships only the fused artifacts (fused ONNX + newly-built HTP context binaries) for the two
modules where a projection-level fusion applies. The fused context binaries keep the same graph
names (vision, action_step) and the same I/O contract as the unfused ones, so they are
drop-in replacements in the resident runtime — the remaining components (llm_split0..3,
action_context) are unchanged and taken from the base repo.
Model variant / sequence length. These fused graphs are the original
allenai/MolmoAct2-LIBERO path: prompt seq = 488, vision emits 392 image tokens
(2 crops × 196). This differs from the grid-sampler (seq=128 / 32-token) golden shipped in the
base repo — see Verification for why parity is checked by ONNX-equivalence here.
What's in here
onnx/
vision_fused/ vision.onnx + vision_weights.bin (1.76 GB) + fusion_info.json
action_step_fused/ action_step.onnx + action_step_weights.bin (2.30 GB) + fusion_info.json
operator_fusion_applied.md what was fused + numerical-equivalence verification
operator_fusion_report.md fusion-opportunity survey across all modules
fusion_viz/ netron before/after screenshots of the fused subgraphs
ctx/ 7 HTP context binaries (fp16, soc_id 77 / Hexagon v73) — a complete resident set
vision_socid77_archv73.bin 931 MB — FUSED (this repo)
action_step_socid77_archv73.bin 1160 MB — FUSED (this repo)
llm_split0_socid77_archv73.bin 1820 MB — unchanged (from base repo)
llm_split1_socid77_archv73.bin 1820 MB — unchanged
llm_split2_socid77_archv73.bin 1820 MB — unchanged
llm_split3_socid77_archv73.bin 1820 MB — unchanged
action_context_socid77_archv73.bin 3.6 MB — unchanged
golden/
vision_io.npz action_step_io.npz reference I/O carried from the base repo
README.md
ctx/ is a complete 7-session resident set: the two fused binaries (vision,
action_step) built here, plus the unchangedllm_split0..3 and action_context carried
over from the base repo. All graph names are unchanged, so this drops straight into the base
repo's runtime/.
What was fused
Parallel projections sharing the same input activation are weight-concatenated into a single
MatMul followed by a Split that restores the original per-branch output tensor names —
so bias Add, reshape, and activation subgraphs are untouched and Split(concat(W)) ≡ [W_i]
holds bit-exactly. (Details + before/after node counts in onnx/operator_fusion_applied.md.)
module
fusion
targets
weight change
vision
QKV (wq+wk+wv → wqkv)
ViT resblocks ×25
[1152,1152]×3 → [1152,3456]
vision
gate_up (w1+w3 → gate_up, SwiGLU)
image_projector ×1
[1152,9728]×2 → [1152,19456]
vision
KV (wk+wv → kv)
image_pooling_2d ×1
[2304,1152]×2 → [2304,2304]
action_step
gate_up (gate_proj+up_proj → gate_up)
DiT blocks ×36
[768,3072]×2 → [768,6144]
Node-count deltas: vision MatMul 210→158 (Split 0→27); action_step MatMul 398→362 (Split 0→36).
The fused ONNX are numerically identical to the originals (max_abs_diff = 0.000e+00,
onnxruntime, ORT_DISABLE_ALL).
Verification
The base repo's golden tensors are the grid-sampler (seq=128) variant, whereas these fused
graphs are the original seq=488 path — so those golden tensors cannot serve as a PyTorch
parity reference here. Instead, each fused DLC is verified against its fused ONNX on identical
inputs (real golden pixel_values for vision; deterministic synthesized inputs at the ONNX's
declared shapes for action_step), run on the QNN CPU backend vs onnxruntime:
context
output
DLC ≡ ONNX cosine
within_tol
vision
image_embeds [392,2560]
1.00000000
1.0000
action_step
next_trajectory [1,10,32]
1.00000000
1.0000
Chain of trust: fused DLC ≡ fused ONNX (above) and fused ONNX ≡ original ONNX ≡ PyTorch
(upstream max_abs_diff = 0). fp16-on-HTP numerics + latency are validated on device by the base
repo's runtime/.
Build provenance
Base model: allenai/MolmoAct2-LIBERO (original, non-grid-sampler; prompt seq = 488).
