| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - image-text-to-text |
| base_model: |
| - HuggingFaceTB/SmolVLM2-500M-Video-Instruct |
| --- |
| # SmolVLM2-500M — ExecuTorch |
|
|
| `smolvlm2_500m_video_instruct_xnnpack_8da8w.pte` (657 MB) |
|
|
| - **Source**: HuggingFaceTB/SmolVLM2-500M-Video-Instruct — a SigLIP vision tower (768 |
| wide, 12 layers) and a Llama decoder (960 wide, 32 layers) |
| - **License**: Apache-2.0 |
| - **Input**: a 512×512 picture as `[1, 3, 512, 512]`, and token ids for the words around it |
| - **Output**: logits over the 49,280-token vocabulary |
|
|
| One file, three entry points — the shape ExecuTorch's multimodal runner asks for: |
|
|
| | method | in | out | |
| |---|---|---| |
| | `vision_encoder` | `[1, 3, 512, 512]` | `[1, 64, 960]` rows in the decoder's embedding space | |
| | `token_embeddings` | token ids | embeddings | |
| | `text_model` | embeddings, positions | logits, and its own cache | |
|
|
| A decoder that only takes token ids cannot be told about a picture. Splitting a |
| vision-language model into a vision `.pte` and a text `.pte` runs aground there; |
| `MultimodalPrefiller::load` asks one module for `token_embeddings` and `text_model` and uses |
| `vision_encoder` if it finds it. |
|
|
| `vision_encoder` here takes a picture rather than a patch sequence, which is what that |
| runner hands it — so this file can be driven by the runner as well as method by method. |
|
|
| ## Verification (Mac arm64, 2026-08-21) |
|
|
| | check | result | |
| |---|---| |
| | vision half vs the untouched tower, 24 photographs | worst **corr 0.99794** | |
| | first-step logits vs eager | **corr 0.99406**, same top-1 | |
| | greedy tokens vs eager, 12 steps | **12/12** | |
|
|
| The wrappers themselves are exact: run in eager without quantization they agree with the |
| model at corr 1.000000, so what these numbers measure is the weights, not the wiring. |
|
|
| ## Why the decoder is int8 and not int4 |
|
|
| The vision tower is int8 per output channel and the decoder is int8 over groups of 32. |
| int4 on a decoder this size does not hold: |
|
|
| | | int4 decoder | int8 decoder | |
| |---|---|---| |
| | size | 497 MB | 657 MB | |
| | first-step logits vs eager | corr 0.89878 | corr 0.99406 | |
| | greedy tokens vs eager | 6/12 | 12/12 | |
|
|
| int4 keeps the first word and then drifts. Only the int8 build is here, because a build that reproduces eager exactly is worth 160 MB. |
|
|
| ## Ask for one tile |
|
|
| The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph |
| becomes **seventeen** of them — 1088 image tokens, against the 64 this graph produces and the |
| 512 of context it was built with. `do_image_splitting = False` on the processor (or its image |
| processor) gives the one tile that matches. |
|
|
| ```bash |
| VL_CKPT=HuggingFaceTB/SmolVLM2-500M-Video-Instruct \ |
| VL_PTE=smolvlm2_500m_video_instruct_xnnpack_8da8w.pte \ |
| python convert/run_vl.py <image> "What is in this picture?" |
| ``` |
|
|
| The driver builds the prompt with the model's own processor and writes the picture's rows |
| wherever the processor put an image token, which is what the model does internally. |
|
|
| ## It invents text on signs |
|
|
| Asked about a London street, this model answers with a shop name that is not there. That is |
| the model and not the conversion: eager, unquantized, invents a different one from the same |
| photograph. If reading signs matters, LFM2.5-VL reads them |
| ([450M](https://huggingface.co/mlboydaisuke/LFM2.5-VL-450M-ExecuTorch)). |
|
|
| ## Conversion |
|
|
| `convert/export_vl_bundle.py`. Two things needed re-authoring: |
|
|
| - **The position embeddings are chosen with `torch.bucketize`**, so that a picture filling |
| part of the grid still lands on the right ones. There is no `bucketize` kernel in the |
| runtime, and a full square grid has one answer anyway: patch *i* takes position *i*. The |
| export checks that against the model's own code rather than assuming it. |
| - **The cache has to live inside `text_model`.** ExecuTorch copies a mutable buffer into |
| each method that names it (`Program::load_mutable_subsegment_into` writes into the |
| method's own memory), so a prefill method and a decode method would each get their own and |
| neither would see the other's writes. One graph serves both. `StaticCache` does not |
| survive lowering, and neither does a cache held by any object outside the module tree — |
| its tensors get lifted a second time as constants, and `run_decompositions` then returns a |
| function where a GraphModule was expected. |
| |
| ## The number that decides whether it runs on a phone |
| |
| `CONTEXT`, the upper bound on the dynamic sequence dimension. The memory planner sizes its |
| arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and |
| iOS kills the process with signal 9 before the first method has finished loading. One |
| picture is 64 rows here, so 512 leaves room for a long question and a long answer. |
| |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) · |
| iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples)) |
| |
