Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,109 @@
+---
+license: apache-2.0
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- image-text-to-text
+base_model:
+- HuggingFaceTB/SmolVLM2-256M-Video-Instruct
+---
+# SmolVLM2-256M — ExecuTorch
+
+`smolvlm2_256m_video_instruct_xnnpack_8da8w.pte` (348 MB)
+
+- **Source**: HuggingFaceTB/SmolVLM2-256M-Video-Instruct — a SigLIP vision tower (768
+  wide, 12 layers) and a Llama decoder (576 wide, 30 layers)
+- **License**: Apache-2.0
+- **Input**: a 512×512 picture as `[1, 3, 512, 512]`, and token ids for the words around it
+- **Output**: logits over the 49,280-token vocabulary
+
+One file, three entry points — the shape ExecuTorch's multimodal runner asks for:
+
+| method | in | out |
+|---|---|---|
+| `vision_encoder` | `[1, 3, 512, 512]` | `[1, 64, 576]` rows in the decoder's embedding space |
+| `token_embeddings` | token ids | embeddings |
+| `text_model` | embeddings, positions | logits, and its own cache |
+
+A decoder that only takes token ids cannot be told about a picture. Splitting a
+vision-language model into a vision `.pte` and a text `.pte` runs aground there;
+`MultimodalPrefiller::load` asks one module for `token_embeddings` and `text_model` and uses
+`vision_encoder` if it finds it.
+
+`vision_encoder` here takes a picture rather than a patch sequence, which is what that
+runner hands it — so this file can be driven by the runner as well as method by method.
+
+## Verification (Mac arm64, 2026-08-21)
+
+| check | result |
+|---|---|
+| vision half vs the untouched tower, 24 photographs | worst **corr 0.99871** |
+| first-step logits vs eager | **corr 0.99197**, same top-1 |
+| greedy tokens vs eager, 12 steps | **1/12** |
+
+The wrappers themselves are exact: run in eager without quantization they agree with the
+model at corr 1.000000, so what these numbers measure is the weights, not the wiring.
+
+## Why the decoder is int8 and not int4
+
+The vision tower is int8 per output channel and the decoder is int8 over groups of 32.
+int4 on a decoder this size does not hold:
+
+| | int4 decoder | int8 decoder |
+|---|---|---|
+| size | 289 MB | 348 MB |
+| first-step logits vs eager | corr 0.83175 | corr 0.99197 |
+| greedy tokens vs eager | 0/12 | 1/12 |
+
+At int4 it does not even pick the same first word, which is why only the int8 build is here. 59 MB is not worth that.
+
+## Ask for one tile
+
+The processor cuts a picture into tiles by aspect ratio and by size, and a 512×512 photograph
+becomes **seventeen** of them — 1088 image tokens, against the 64 this graph produces and the
+512 of context it was built with. `do_image_splitting = False` on the processor (or its image
+processor) gives the one tile that matches.
+
+```bash
+VL_CKPT=HuggingFaceTB/SmolVLM2-256M-Video-Instruct \
+VL_PTE=smolvlm2_256m_video_instruct_xnnpack_8da8w.pte \
+python convert/run_vl.py <image> "What is in this picture?"
+```
+
+The driver builds the prompt with the model's own processor and writes the picture's rows
+wherever the processor put an image token, which is what the model does internally.
+
+## It invents text on signs
+
+Asked about a London street, this model answers with a shop name that is not there. That is
+the model and not the conversion: eager, unquantized, invents a different one from the same
+photograph. If reading signs matters, LFM2.5-VL reads them
+([450M](https://huggingface.co/mlboydaisuke/LFM2.5-VL-450M-ExecuTorch)).
+
+## Conversion
+
+`convert/export_vl_bundle.py`. Two things needed re-authoring:
+
+- **The position embeddings are chosen with `torch.bucketize`**, so that a picture filling
+  part of the grid still lands on the right ones. There is no `bucketize` kernel in the
+  runtime, and a full square grid has one answer anyway: patch *i* takes position *i*. The
+  export checks that against the model's own code rather than assuming it.
+- **The cache has to live inside `text_model`.** ExecuTorch copies a mutable buffer into
+  each method that names it (`Program::load_mutable_subsegment_into` writes into the
+  method's own memory), so a prefill method and a decode method would each get their own and
+  neither would see the other's writes. One graph serves both. `StaticCache` does not
+  survive lowering, and neither does a cache held by any object outside the module tree —
+  its tensors get lifted a second time as constants, and `run_decompositions` then returns a
+  function where a GraphModule was expected.
+
+## The number that decides whether it runs on a phone
+
+`CONTEXT`, the upper bound on the dynamic sequence dimension. The memory planner sizes its
+arena for the bound, not for what a picture costs: at 4096 that arena runs to gigabytes and
+iOS kills the process with signal 9 before the first method has finished loading. One
+picture is 64 rows here, so 512 leaves room for a long question and a long answer.
+
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
+iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))