mlboydaisuke/SmolVLM2-500M-ExecuTorch

SmolVLM2-500M — ExecuTorch

smolvlm2_500m_video_instruct_xnnpack_8da8w.pte (657 MB)

• Source: HuggingFaceTB/SmolVLM2-500M-Video-Instruct — a SigLIP vision tower (768 wide, 12 layers) and a Llama decoder (960 wide, 32 layers)
• License: Apache-2.0
• Input: a 512×512 picture as [1, 3, 512, 512], and token ids for the words around it
• Output: logits over the 49,280-token vocabulary

One file, three entry points — the shape ExecuTorch's multimodal runner asks for:

method	in	out
vision_encoder	[1, 3, 512, 512]	[1, 64, 960] rows in the decoder's embedding space
token_embeddings	token ids	embeddings
text_model	embeddings, positions	logits, and its own cache

A decoder that only takes token ids cannot be told about a picture. Splitting a vision-language model into a vision .pte and a text .pte runs aground there; MultimodalPrefiller::load asks one module for token_embeddings and text_model and uses vision_encoder if it finds it.

vision_encoder here takes a picture rather than a patch sequence, which is what that runner hands it — so this file can be driven by the runner as well as method by method.

Verification (Mac arm64, 2026-08-21)

check	result
vision half vs the untouched tower, 24 photographs	worst corr 0.99794
first-step logits vs eager	corr 0.99406, same top-1
greedy tokens vs eager, 12 steps	12/12

The wrappers themselves are exact: run in eager without quantization they agree with the model at corr 1.000000, so what these numbers measure is the weights, not the wiring.

Why the decoder is int8 and not int4

The vision tower is int8 per output channel and the decoder is int8 over groups of 32. int4 on a decoder this size does not hold:

	int4 decoder	int8 decoder
size	497 MB	657 MB
first-step logits vs eager	corr 0.89878	corr 0.99406
greedy tokens vs eager	6/12	12/12

int4 keeps the first word and then drifts. Only the int8 build is here, because a build that reproduces eager exactly is worth 160 MB.

Ask for one tile

The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph becomes seventeen of them — 1088 image tokens, against the 64 this graph produces and the 512 of context it was built with. do_image_splitting = False on the processor (or its image processor) gives the one tile that matches.

VL_CKPT=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \
VL_PTE=smolvlm2_500m_video_instruct_xnnpack_8da8w.pte \
python convert/run_vl.py <image> "What is in this picture?"

The driver builds the prompt with the model's own processor and writes the picture's rows wherever the processor put an image token, which is what the model does internally.

It invents text on signs

Asked about a London street, this model answers with a shop name that is not there. That is the model and not the conversion: eager, unquantized, invents a different one from the same photograph. If reading signs matters, LFM2.5-VL reads them (450M).

Conversion

convert/export_vl_bundle.py. Two things needed re-authoring:

• The position embeddings are chosen with torch.bucketize, so that a picture filling part of the grid still lands on the right ones. There is no bucketize kernel in the runtime, and a full square grid has one answer anyway: patch i takes position i. The export checks that against the model's own code rather than assuming it.
• The cache has to live inside text_model. ExecuTorch copies a mutable buffer into each method that names it (Program::load_mutable_subsegment_into writes into the method's own memory), so a prefill method and a decode method would each get their own and neither would see the other's writes. One graph serves both. StaticCache does not survive lowering, and neither does a cache held by any object outside the module tree — its tensors get lifted a second time as constants, and run_decompositions then returns a function where a GraphModule was expected.

The number that decides whether it runs on a phone

CONTEXT, the upper bound on the dynamic sequence dimension. The memory planner sizes its arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and iOS kills the process with signal 9 before the first method has finished loading. One picture is 64 rows here, so 512 leaves room for a long question and a long answer.

(conversion scripts: executorch-models · iOS sample: executorch-samples)