Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,103 @@
+---
+license: other
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- image-text-to-text
+base_model:
+- LiquidAI/LFM2.5-VL-1.6B
+---
+# LFM2.5-VL-1.6B — ExecuTorch
+
+`lfm2_5_vl_1_6b_xnnpack_8da4w.pte` (2.03 GB)
+
+- **Source**: LiquidAI/LFM2.5-VL-1.6B — SigLIP 2 vision tower (1152 wide, 27 layers) and an
+  LFM2 decoder (2048 wide, 16 layers)
+- **License**: LFM Open License v1.0
+- **Input**: a 512×512 tile as `[1, 1024, 768]` patches plus `[1, 1024]` attention mask,
+  and token ids for the words around it
+- **Output**: logits over the 64,400-token vocabulary
+
+One file, three entry points — the shape ExecuTorch's multimodal runner asks for:
+
+| method | in | out |
+|---|---|---|
+| `vision_encoder` | patches, mask | `[1, 256, 2048]` rows in the decoder's embedding space |
+| `token_embeddings` | token ids | embeddings |
+| `text_model` | embeddings, positions | logits, and its own cache |
+
+A decoder that only takes token ids cannot be told about a picture. Splitting a
+vision-language model into a vision `.pte` and a text `.pte` runs aground there;
+`MultimodalPrefiller::load` asks one module for `token_embeddings` and `text_model` and uses
+`vision_encoder` if it finds it.
+
+## On a phone
+
+2.03 GB, and its arena is 176 MB — it fits where the fp32 export (6.93 GB) did not. Not
+measured on device; the rate quoted for
+[the 450M](https://huggingface.co/mlboydaisuke/LFM2.5-VL-450M-ExecuTorch) is that file's,
+not this one's.
+
+## Verification (Mac arm64, 2026-08-21)
+
+Greedy, through the three methods, on photographs from `convert/calib_images`:
+
+| picture | answer |
+|---|---|
+| a London street | "A group of people are walking on a sidewalk in front of a building that says Pizza Express." |
+| a man with a dog | "A black and white photograph shows a man standing next to a dog and a cart, with a large stack of wood in the background." |
+
+The shop sign is read correctly, which is the check that matters: a caption that fits any
+street would not tell you the vision half was wired up right.
+
+The vision half of this file agrees with the untouched model at **worst corr 0.98877** over
+24 photographs.
+
+## Square the picture first
+
+The processor picks a tile grid from the aspect ratio — a 768×477 photograph becomes one
+24×40 tile, a 1280×960 one becomes seven — and this graph takes 32×32. Centre-crop to a
+square before the processor sees it and every picture becomes exactly one tile. Feeding a
+stretched square instead is visible in the output: asked about a squashed street, the model
+called the scene "distorted and warped", which was a fair description of what it had been
+given.
+
+```bash
+python convert/run_lfm2_vl.py <image> "What is in this picture?"
+```
+
+## Conversion
+
+`convert/export_lfm2_vl_bundle.py`. Four things needed re-authoring:
+
+- **SigLIP 2 reads its grid out of a tensor** to size the position embeddings, which
+  `torch.export` cannot follow. The grid is fixed here, so the resize is computed once and
+  the constant handed to a replacement forward.
+- **The projector wants the grid back.** The tower returns a flat run of patches; the
+  projector's pixel-unshuffle trades resolution for channels and needs to know which patches
+  are neighbours.
+- **The quantization is not uniform.** The decoder is int4 over groups of 32; the vision
+  tower is int8 per output channel. At int4 throughout, the model reads the Pizza Express
+  sign as "Pocket Express" — the letters live in the tower, and four bits does not hold
+  them.
+- **The cache lives inside `text_model`.** ExecuTorch copies a mutable buffer into each
+  method that names it (`Program::load_mutable_subsegment_into` writes into the method's own
+  memory), so a prefill method and a decode method would each get their own and neither
+  would see the other's writes. One graph has to serve both, which rules out the two code
+  paths transformers keeps for LFM2's short convolution — a windowed convolution for a
+  prompt, a fused single-step update for a token, chosen in Python and therefore baked in by
+  tracing. Carrying the last `kernel - 1` columns and putting them in front of whatever
+  arrives is the same arithmetic in one branch-free path.
+
+## The number that decides whether it runs on a phone
+
+`CONTEXT`, the upper bound on the dynamic sequence dimension. The memory planner sizes its
+arena for the bound, not for what a picture costs: at 4096 that arena is **3.7 GB** and iOS
+kills the process with signal 9 before the first method has finished loading. One square
+photograph is 1024 patches, which the projector unshuffles to 256 rows, so 512 leaves room
+for a long question and a long answer and brings the arena to 176 MB.
+
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models) ·
+iOS sample: [executorch-samples](https://github.com/john-rocky/executorch-samples))