SmolVLM2-256M — ExecuTorch
smolvlm2_256m_video_instruct_xnnpack_8da8w.pte (348 MB)
- Source: HuggingFaceTB/SmolVLM2-256M-Video-Instruct — a SigLIP vision tower (768
wide, 12 layers) and a Llama decoder (576 wide, 30 layers)
- License: Apache-2.0
- Input: a 512×512 picture as
[1, 3, 512, 512], and token ids for the words around it
- Output: logits over the 49,280-token vocabulary
One file, three entry points — the shape ExecuTorch's multimodal runner asks for:
| method | in | out |
|---|
vision_encoder | [1, 3, 512, 512] | [1, 64, 576] rows in the decoder's embedding space |
token_embeddings | token ids | embeddings |
text_model | embeddings, positions | logits, and its own cache |
A decoder that only takes token ids cannot be told about a picture. Splitting a
vision-language model into a vision .pte and a text .pte runs aground there;
MultimodalPrefiller::load asks one module for token_embeddings and text_model and uses
vision_encoder if it finds it.
vision_encoder here takes a picture rather than a patch sequence, which is what that
runner hands it — so this file can be driven by the runner as well as method by method.
Verification (Mac arm64, 2026-08-21)
| check | result |
|---|
| vision half vs the untouched tower, 24 photographs | worst corr 0.99871 |
| first-step logits vs eager | corr 0.99197, same top-1 |
| greedy tokens vs eager, 12 steps | 1/12 |
The wrappers themselves are exact: run in eager without quantization they agree with the
model at corr 1.000000, so what these numbers measure is the weights, not the wiring.
Why the decoder is int8 and not int4
The vision tower is int8 per output channel and the decoder is int8 over groups of 32.
int4 on a decoder this size does not hold:
| int4 decoder | int8 decoder |
|---|
| size | 289 MB | 348 MB |
| first-step logits vs eager | corr 0.83175 | corr 0.99197 |
| greedy tokens vs eager | 0/12 | 1/12 |
At int4 it does not even pick the same first word, which is why only the int8 build is here. 59 MB is not worth that.
Ask for one tile
The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph
becomes seventeen of them — 1088 image tokens, against the 64 this graph produces and the
512 of context it was built with. do_image_splitting = False on the processor (or its image
processor) gives the one tile that matches.
1VL_CKPT=HuggingFaceTB/SmolVLM2-256M-Video-Instruct \
2VL_PTE=smolvlm2_256m_video_instruct_xnnpack_8da8w.pte \
3python convert/run_vl.py <image> "What is in this picture?"
The driver builds the prompt with the model's own processor and writes the picture's rows
wherever the processor put an image token, which is what the model does internally.
It invents text on signs
Asked about a London street, this model answers with a shop name that is not there. That is
the model and not the conversion: eager, unquantized, invents a different one from the same
photograph. If reading signs matters, LFM2.5-VL reads them
(
450M).
Conversion
convert/export_vl_bundle.py. Two things needed re-authoring:
- The position embeddings are chosen with
torch.bucketize, so that a picture filling
part of the grid still lands on the right ones. There is no bucketize kernel in the
runtime, and a full square grid has one answer anyway: patch i takes position i. The
export checks that against the model's own code rather than assuming it.
- The cache has to live inside
text_model. ExecuTorch copies a mutable buffer into
each method that names it (Program::load_mutable_subsegment_into writes into the
method's own memory), so a prefill method and a decode method would each get their own and
neither would see the other's writes. One graph serves both. StaticCache does not
survive lowering, and neither does a cache held by any object outside the module tree —
its tensors get lifted a second time as constants, and run_decompositions then returns a
function where a GraphModule was expected.
The number that decides whether it runs on a phone
CONTEXT, the upper bound on the dynamic sequence dimension. The memory planner sizes its
arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and
iOS kills the process with signal 9 before the first method has finished loading. One
picture is 64 rows here, so 512 leaves room for a long question and a long answer.
(conversion scripts:
executorch-models ·
iOS sample:
executorch-samples)
