SmolVLM2-500M — ExecuTorch
smolvlm2_500m_video_instruct_xnnpack_8da8w.pte (518 MB)
- Source: HuggingFaceTB/SmolVLM2-500M-Video-Instruct — a SigLIP vision tower (768
wide, 12 layers) and a Llama decoder (960 wide, 32 layers)
- License: Apache-2.0
- Input: a 512×512 picture as
[1, 3, 512, 512], and token ids for the words around it
- Output: logits over the 49,280-token vocabulary
One file, three entry points — the shape ExecuTorch's multimodal runner asks for:
| method | in | out |
|---|
vision_encoder | [1, 3, 512, 512] | [1, 64, 960] rows in the decoder's embedding space |
token_embeddings | token ids | embeddings |
text_model | embeddings, positions | logits, and its own cache |
A decoder that only takes token ids cannot be told about a picture. Splitting a
vision-language model into a vision .pte and a text .pte runs aground there;
MultimodalPrefiller::load asks one module for token_embeddings and text_model and uses
vision_encoder if it finds it.
vision_encoder here takes a picture rather than a patch sequence, which is what that
runner hands it — so this file can be driven by the runner as well as method by method.
Verification (Mac arm64, 2026-08-21)
| check | result |
|---|
| vision half vs the untouched tower, 24 photographs | worst corr 0.99794 |
| first-step logits vs eager | corr 0.99536, same top-1 |
| greedy tokens vs eager, 12 steps | 12/12 |
The wrappers themselves are exact: run in eager without quantization they agree with the
model at corr 1.000000, so what these numbers measure is the weights, not the wiring.
Why the decoder is int8 and not int4
The vision tower is int8 per output channel and the decoder is int8 over groups of 32.
int4 on a decoder this size does not hold:
| int4 decoder | int8 decoder |
|---|
| size | 497 MB | 518 MB |
| first-step logits vs eager | corr 0.89878 | corr 0.99406 |
| greedy tokens vs eager | 6/12 | 12/12 |
int4 keeps the first word and then drifts. Only the int8 build is here. Those two columns
were measured before the embedding table was quantised, when this file was 657 MB.
Ask for one tile
The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph
becomes seventeen of them — 1088 image tokens, against the 64 this graph produces and the
512 of context it was built with. do_image_splitting = False on the processor (or its image
processor) gives the one tile that matches.
1VL_CKPT=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
2VL_PTE=smolvlm2_500m_video_instruct_xnnpack_8da8w.pte \
3python convert/run_vl.py <image> "What is in this picture?"
The driver builds the prompt with the model's own processor and writes the picture's rows
wherever the processor put an image token, which is what the model does internally.
It invents text on signs
Asked about a London street, this model answers with a shop name that is not there. That is
the model and not the conversion: eager, unquantized, invents a different one from the same
photograph. If reading signs matters, LFM2.5-VL reads them
(
450M).
Conversion
convert/export_vl_bundle.py. Two things needed re-authoring:
- The position embeddings are chosen with
torch.bucketize, so that a picture filling
part of the grid still lands on the right ones. There is no bucketize kernel in the
runtime, and a full square grid has one answer anyway: patch i takes position i. The
export checks that against the model's own code rather than assuming it.
- The cache has to live inside
text_model. ExecuTorch copies a mutable buffer into
each method that names it (Program::load_mutable_subsegment_into writes into the
method's own memory), so a prefill method and a decode method would each get their own and
neither would see the other's writes. One graph serves both. StaticCache does not
survive lowering, and neither does a cache held by any object outside the module tree —
its tensors get lifted a second time as constants, and run_decompositions then returns a
function where a GraphModule was expected.
The number that decides whether it runs on a phone
CONTEXT, the upper bound on the dynamic sequence dimension. The memory planner sizes its
arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and
iOS kills the process with signal 9 before the first method has finished loading. One
picture is 64 rows here, so 512 leaves room for a long question and a long answer.
(conversion scripts:
executorch-models ·
iOS sample:
executorch-samples)
The token embedding table
The table is an nn.Embedding, and the quantisation filter in convert/export_vl_bundle.py
was written as isinstance(module, nn.Linear), so it went out in fp32 — 189 MB of a
657 MB file. tie_word_embeddings does not save it either: quantize_ swaps
lm_head.weight for a new tensor and the embedding keeps pointing at the old fp32 one.
(LFM2.5-VL ties and SmolVLM2 does not; both leaked.)
It is now int8 over groups of 64 along each row, written out by hand because torchao's
IntxWeightOnlyConfig reaches an nn.Embedding but will not lower
(Missing out variants: {'torchao::dequantize_affine'}). On this model the first-step logits improve slightly — corr 0.99536 against 0.99406 for the fp32 table — while the twelfth greedy token differs where the fp32-table build matched all twelve. The file drops from 657 MB to 518 MB. Both numbers are here rather than the flattering one.
One scale per row is not enough for a narrow table: on SmolVLM2-256M, whose rows are 576
long, per-row scales moved the first-step logits from corr 0.99197 to 0.97811 and flipped
the top-1 token, so that model keeps its fp32 table. Groups of 64 are what make this one
free.
