Qwen3-VL-2B-Instruct — ExecuTorch
A vision-language model as one .pte with three entry points. Ask it about a photograph on
the device; nothing leaves it.
- Source: Qwen/Qwen3-VL-2B-Instruct — 2.13B parameters, 28-layer decoder, 24-layer vision tower
- License: apache-2.0
- Input: a 512x512 RGB photograph and a question
- Output: text
Methods
| method | takes | returns |
|---|
vision_encoder | pixel_values [1024, 1536] | merged [256, 2048] and three deepstack maps [256, 2048] |
token_embeddings | ids [1, seq] | rows [1, seq, 2048] |
text_model | rows, positions [3, 1, seq], slots [1, seq], three deepstack maps [1, seq, 2048] | logits [1, seq, 151936] |
Three methods rather than three files because ExecuTorch copies a mutable buffer into every
method that names it, so a prefill method and a decode method would each get their own KV
cache and neither would see the other's writes. One graph serves a 277-token prompt and a
single token.
What the caller owns
Three things, and getting any of them wrong does not throw — it answers about the wrong
thing:
- Splice the 256 vision rows into the prompt embedding at the image token positions.
- Scatter the three deepstack maps to those same positions, zero everywhere else.
Qwen3-VL adds features from vision blocks 5, 11 and 17 into decoder layers 0, 1 and 2,
and only at image positions. Upstream writes that as
hidden_states[visual_pos_masks, :] += embeds, which does not export; here the maps
arrive already placed, so the graph just adds them.
- Pass Qwen's own 3-D positions from
get_rope_index, not arange. The rope is
interleaved M-RoPE with mrope_section [24, 20, 20] — time, height, width. A 1-D
position array gives every image patch the same angle and reads as a mild degradation
rather than an error.
convert/run_qwen3_vl.py does all three in 40 lines.
Verification (Mac arm64, executorch 1.4.0, torch 2.13.0)
Parity is the last-row logits of a real prompt against Qwen3VLForConditionalGeneration in
eager fp32, per question, because a model of this size can hold a scene and lose the small
print. Both questions are about the same photograph.
| build | file | size (MB) | corr, "name on the sign" | corr, "describe it" | vision (ms) | prefill (ms) | decode |
|---|
| 8da8w | qwen3_vl_2b_instruct_xnnpack_8da8w.pte | 2453.1 | 0.993435 | 0.992611 | 568.7 | 590.6 | 17.1 tok/s |
| 8da4w (not shipped) | qwen3_vl_2b_instruct_xnnpack_8da4w.pte | 1697.5 | 0.739964 | 0.948887 | 566.7 | 739.4 | 19.8 tok/s |
The prompt is 277 tokens, of which 256 are the picture. vision_encoder runs
once per image, text_model once per prompt and once per generated token.
What it says
Asked for the name on a pub sign, against the eager model's "Lamb & Flag":
- 8da8w — Based on the image provided, the name of the pub is The Lamb & Flag.
This name is visible on the sign above the entrance to the establishment. The sign also includes the address "
- 8da4w — Based on the text visible on the sign above the entrance of the establishment, the name of the pub is Pine Purpose.
The sign is located on the front of the building, just above
The rewrites, and what each was checked against
Neither this repo's generic VL bundle nor export_llm reaches Qwen3-VL. Five things had to
change, each measured against the module it replaced:
| what | why | checked |
|---|
| vision constants precomputed | position embeddings are bilinearly interpolated from grid_thw on every call | part of the 0.000e+00 below |
| variable-length attention → plain SDPA | one image is one sequence, so cu_seqlens is [0, 1024] | exact |
patch embedding Conv3d → F.linear | portable convolution takes 3-D or 4-D only: Expect input tensor to be 3-D or 4-D, but got, 5 | max_abs_diff 4.9e-04 |
| deepstack mask-assignment → add | boolean-mask assignment does not export | first-step logits corr 1.000000 |
| embedding table quantised by hand | see below | corr 0.999965 |
With the first two in place and no quantisation, the vision wrapper reproduces
Qwen3VLVisionModel at max_abs_diff 0.000e+00 across all four outputs, and the decoder
wrapper reproduces the full model's first-step logits at corr 1.000000.
Two traps worth knowing
The token embedding table escapes quantisation. It is an nn.Embedding, so a filter
written as isinstance(module, nn.Linear) skips it — and tie_word_embeddings does not
save it, because quantize_ swaps lm_head.weight for a new tensor while the embedding
keeps pointing at the old fp32 one. At 151,936 x 2048 that is 1.24 GB left in fp32: the
first export of this model came out at 2630 MB. torchao's own IntxWeightOnlyConfig does
reach an nn.Embedding but will not lower — Missing out variants: {'torchao::dequantize_affine'} — so the quantisation is written out by hand, one scale
per row.
512x512 is where small text lives or dies, and the resampling filter decides. Asked
for the name on the sign, the eager fp32 model answers:
| resize filter | eager fp32 |
|---|
| BICUBIC | "Lamb & Flag" — right |
| BILINEAR | "The Lamb & Flag" — right |
| LANCZOS | "The Pigeon House" — wrong |
| NEAREST | "Pompadour" — wrong |
That is before any quantisation. If fine print matters for your use, the preprocessing is
not a detail; convert/export_qwen3_vl.py puts the resize behind one helper for that reason.
Not shipped
8da4w — the same bundle with a 4-bit decoder — is 1697.5 MB against
2453.1 MB, and it does not hold: logits correlation 0.740 on the sign question
against 0.993, and it reads the sign as "Pine Purpose". Measured where the loss is,
one arm at a time: the vision tower at int8 costs corr 0.990 against fp32's 0.99982, the
hand-quantised table costs 0.99997, and the 4-bit decoder is the rest. Reported here rather
than dropped, because a 1.7 GB 2B vision-language model is worth wanting and this is what it
costs.
Conversion
1python convert/export_qwen3_vl.py # check both halves against eager
2python convert/export_qwen3_vl.py --export # write the bundle
3python convert/check_qwen3_vl.py 8da8w # parity and speed
4python convert/run_qwen3_vl.py photo.jpg "what is on the sign?"
(conversion scripts:
executorch-models)
