README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- image-text-to-text
base_model:
- HuggingFaceTB/SmolVLM2-256M-Video-Instruct
---
# SmolVLM2-256M — ExecuTorch

`smolvlm2_256m_video_instruct_xnnpack_8da8w.pte` (348 MB)

- **Source**: HuggingFaceTB/SmolVLM2-256M-Video-Instruct — a SigLIP vision tower (768
  wide, 12 layers) and a Llama decoder (576 wide, 30 layers)
- **License**: Apache-2.0
- **Input**: a 512×512 picture as `[1, 3, 512, 512]`, and token ids for the words around it
- **Output**: logits over the 49,280-token vocabulary

One file, three entry points — the shape ExecuTorch's multimodal runner asks for:

| method | in | out |
|---|---|---|
| `vision_encoder` | `[1, 3, 512, 512]` | `[1, 64, 576]` rows in the decoder's embedding space |
| `token_embeddings` | token ids | embeddings |
| `text_model` | embeddings, positions | logits, and its own cache |

A decoder that only takes token ids cannot be told about a picture. Splitting a
vision-language model into a vision `.pte` and a text `.pte` runs aground there;
`MultimodalPrefiller::load` asks one module for `token_embeddings` and `text_model` and uses
`vision_encoder` if it finds it.

`vision_encoder` here takes a picture rather than a patch sequence, which is what that
runner hands it — so this file can be driven by the runner as well as method by method.

## Verification (Mac arm64, 2026-08-21)

| check | result |
|---|---|
| vision half vs the untouched tower, 24 photographs | worst **corr 0.99871** |
| first-step logits vs eager | **corr 0.99197**, same top-1 |
| greedy tokens vs eager, 12 steps | **1/12** |

The wrappers themselves are exact: run in eager without quantization they agree with the
model at corr 1.000000, so what these numbers measure is the weights, not the wiring.

## Why the decoder is int8 and not int4

The vision tower is int8 per output channel and the decoder is int8 over groups of 32.
int4 on a decoder this size does not hold:

| | int4 decoder | int8 decoder |
|---|---|---|
| size | 289 MB | 348 MB |
| first-step logits vs eager | corr 0.83175 | corr 0.99197 |
| greedy tokens vs eager | 0/12 | 1/12 |

At int4 it does not even pick the same first word, which is why only the int8 build is here. 59 MB is not worth that.

## Ask for one tile

The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph
becomes **seventeen** of them — 1088 image tokens, against the 64 this graph produces and the
512 of context it was built with. `do_image_splitting = False` on the processor (or its image
processor) gives the one tile that matches.

```bash
VL_CKPT=HuggingFaceTB/SmolVLM2-256M-Video-Instruct \
VL_PTE=smolvlm2_256m_video_instruct_xnnpack_8da8w.pte \
python convert/run_vl.py <image> "What is in this picture?"
```

The driver builds the prompt with the model's own processor and writes the picture's rows
wherever the processor put an image token, which is what the model does internally.

## It invents text on signs

Asked about a London street, this model answers with a shop name that is not there. That is
the model and not the conversion: eager, unquantized, invents a different one from the same
photograph. If reading signs matters, LFM2.5-VL reads them
([450M](https://huggingface.co/mlboydaisuke/LFM2.5-VL-450M-ExecuTorch)).

## Conversion

`convert/export_vl_bundle.py`. Two things needed re-authoring:

- **The position embeddings are chosen with `torch.bucketize`**, so that a picture filling
  part of the grid still lands on the right ones. There is no `bucketize` kernel in the
  runtime, and a full square grid has one answer anyway: patch *i* takes position *i*. The
  export checks that against the model's own code rather than assuming it.
- **The cache has to live inside `text_model`.** ExecuTorch copies a mutable buffer into
  each method that names it (`Program::load_mutable_subsegment_into` writes into the
  method's own memory), so a prefill method and a decode method would each get their own and
  neither would see the other's writes. One graph serves both. `StaticCache` does not
  survive lowering, and neither does a cache held by any object outside the module tree —
  its tensors get lifted a second time as constants, and `run_decompositions` then returns a
  function where a GraphModule was expected.

## The number that decides whether it runs on a phone

`CONTEXT`, the upper bound on the dynamic sequence dimension. The memory planner sizes its
arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and
iOS kills the process with signal 9 before the first method has finished loading. One
picture is 64 rows here, so 512 leaves room for a long question and a long answer.

(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))