litert-community
/

North-Micro-Vision-Instruct

@@ -52,13 +52,13 @@ Pixel 8a (litert-lm v0.16.1 CLI, `--disable_cache`), wi8 bundle — the decoder
 Run the decoder on the GPU on 8 GB-class phones — the CPU backend pages against the 2.5 GB decoder (0.6 tok/s).
-iPhone 17 Pro (Metal decoder + Metal vision, cold): the 3.5 GB bundle loads (peak 3.46 GB resident); decode **24.9 tok/s** on a short vision turn / **14.6 tok/s** on a 50-token text turn; vision time-to-first-token 8.1–9.5 s. Same phone, same model through Apple's Core AI runtime (own measurement, 2026-08-14, int8 decoder + fp16 tower): **18.2 tok/s** decode, 21.5 tok/s prefill, image oracle 24/24 — the two runtimes land within ~35% of each other on decode, LiteRT-LM ahead on short vision turns and behind on longer text turns; the honest read is "same class", with the caveat that the LiteRT number for the final int8-vision composition is pending (the iPhone figures above are from the fp16-vision build of the same decoder).
 ## Quality
 - **9-case COCO suite** (3 images × 3 questions, 48-token greedy) against a fp32 PyTorch oracle running the same single-embedding / 1-D-position contract: content-correct and image-grounded on 9/9 (cats on a pink couch with remotes, the two kitchens, colour palettes, "where is this scene"); the int8 vision encoder shifts token choices, so token-exact is 1/9 (the desktop fp16-vision build keeps 5/9 token-exact).
 - **8-question text gate: 7/8**, non-degenerate. The one miss ("17 + 25" read as "1.7 + 2.5") reproduces token-for-token on the HF fp32 model — Cohere's per-digit pre-tokenizer, not a conversion artifact.
-- **Device**: Pixel 8a Ask-Image 2/2 grounded ("two cats lying on a pink surface, possibly a couch or a bed…", "warm brown wooden counter, black stove, white apron, hanging pots and pans"); iPhone vision probes track the image (no-text fractal → "No, this image…"; text probe → reads the text).
 - **int4 variant**: coherent and content-correct on all 9 suite cases but terser and further from the fp32 wording; prefer wi8 where storage allows.
 > **What the fast_vlm contract changes, and what it costs.** The released model injects three DeepStack vision embeddings after decoder layers 0/1/2 and uses interleaved M-RoPE. This bundle folds the DeepStack embeddings into the single image embedding (exactly representable; teacher-forced top-1 vs the released model 0.96 fold-only, 0.93 with the runtime's 1-D positions) and the runtime supplies plain sequential positions in place of M-RoPE. Measured effect on probe prompts: **describe / VQA / spatial relations / single-cell lookup preserved**; **2-D table cross-cell questions and digit-dense OCR degrade** (row count off-by-one, "$652,000" read as "$652,000,000", a duplicated word in a dense paragraph). Same class of trade as the Qwen2-VL-2B bundle. Use it for reading and describing; don't rely on it to rank table cells.

 Run the decoder on the GPU on 8 GB-class phones — the CPU backend pages against the 2.5 GB decoder (0.6 tok/s).
+iPhone 17 Pro (Metal decoder + Metal vision, cold, this exact 3.07 GB file): peak resident **2.9–3.0 GB** with the vision executor loaded (2.1 GB text-only); decode **13.6 tok/s** on a 39-token text turn; vision time-to-first-token 2.3–5.0 s cold (302-token image prompt, prefill 322–362 tok/s). Same phone, same model through Apple's Core AI runtime (own measurement, 2026-08-14, int8 decoder + fp16 tower): **18.2 tok/s** decode, 21.5 tok/s prefill, image oracle 24/24. Read it as "same class, Core AI ahead on decode (~1.3×), LiteRT-LM ahead on prefill and vision TTFT"; both numbers are cold single runs on the same device and the same day-class build, not a tuned benchmark.
 ## Quality
 - **9-case COCO suite** (3 images × 3 questions, 48-token greedy) against a fp32 PyTorch oracle running the same single-embedding / 1-D-position contract: content-correct and image-grounded on 9/9 (cats on a pink couch with remotes, the two kitchens, colour palettes, "where is this scene"); the int8 vision encoder shifts token choices, so token-exact is 1/9 (the desktop fp16-vision build keeps 5/9 token-exact).
 - **8-question text gate: 7/8**, non-degenerate. The one miss ("17 + 25" read as "1.7 + 2.5") reproduces token-for-token on the HF fp32 model — Cohere's per-digit pre-tokenizer, not a conversion artifact.
+- **Device**: Pixel 8a Ask-Image 2/2 grounded ("two cats lying on a pink surface, possibly a couch or a bed…", "warm brown wooden counter, black stove, white apron, hanging pots and pans"); iPhone 17 Pro vision-grounding probes 2/2 ("Does this image contain visible written text?" → **No** on the no-text fractal, **Yes** on the text probe), 8-question text gate 6/8 on-device (the same digit quirk plus one rhyme miss).
 - **int4 variant**: coherent and content-correct on all 9 suite cases but terser and further from the fp32 wording; prefer wi8 where storage allows.
 > **What the fast_vlm contract changes, and what it costs.** The released model injects three DeepStack vision embeddings after decoder layers 0/1/2 and uses interleaved M-RoPE. This bundle folds the DeepStack embeddings into the single image embedding (exactly representable; teacher-forced top-1 vs the released model 0.96 fold-only, 0.93 with the runtime's 1-D positions) and the runtime supplies plain sequential positions in place of M-RoPE. Measured effect on probe prompts: **describe / VQA / spatial relations / single-cell lookup preserved**; **2-D table cross-cell questions and digit-dense OCR degrade** (row count off-by-one, "$652,000" read as "$652,000,000", a duplicated word in a dense paragraph). Same class of trade as the Qwen2-VL-2B bundle. Use it for reading and describing; don't rely on it to rank table cells.