MiniCPM-V-4.6 Core AI card

README.md

@@ -0,0 +1,74 @@
+---
+license: apache-2.0
+base_model: openbmb/MiniCPM-V-4.6
+tags:
+- core-ai
+- coreai
+- vision-language
+- vlm
+- on-device
+- iphone
+- apple
+pipeline_tag: image-text-to-text
+language:
+- en
+- zh
+---
+
+# MiniCPM-V-4.6 — Core AI
+
+**On-device vision-language model for iPhone / Apple Silicon.** A Core AI port of
+[`openbmb/MiniCPM-V-4.6`](https://huggingface.co/openbmb/MiniCPM-V-4.6) — the strongest
+sub-2B open VLM — running fully local on the GPU via the Core AI **pipelined engine**:
+pick a photo, ask about it, stream the answer.
+
+Verified on **iPhone 17 Pro**: image → grounded answer at **~51.5 tok/s** decode, all local.
+
+## Architecture
+
+MiniCPM-V-4.6 (1.3B) = a **SigLIP So400m vision tower** (980px / patch 14 / 27 layers, with a
+window-attention insert-merger @ layer 6 + a downsample-MLP merger → ÷16 = 64 visual tokens per
+448px slice) + a **Qwen3.5-hybrid text backbone** (`qwen3_5_text`: 0.8B, 24 layers, GatedDeltaNet
+linear attention ×3 : full attention ×1, head_dim 256, vocab 248094, tied head). Connector =
+2×2 spatial merges + MLP, spliced into the text embeddings at `<image>` positions (`masked_scatter`).
+
+## Bundles
+
+| path | what | dtype | size |
+|---|---|---|---|
+| `gpu-pipelined/minicpmv46_vlm_decode_int8lin/` | VLM text decoder (`input_ids → logits` + a static `image_embeds[64,1024]` buffer; in-graph gather `ids ≥ V ? image_embeds[ids-V] : embed[ids]`) | int8 (per-block-32 linear) | ~1.0 GB |
+| `gpu-pipelined/minicpmv46_vision/` | fixed-grid SigLIP vision encoder (`pixel_values[1,3,448,448] → image_features[64,1024]`) | fp16 | ~1.0 GB |
+
+The decoder is a complete qwen3.5-hybrid text LLM when `image_embeds` is zero — same bundle, no image needed.
+
+## How a VLM rides the text-only engine
+
+The pipelined engine knows nothing about images. The whole multimodal state rides the
+**static-input hook** (`image_embeds` buffer) + an id-space trick — the graph stays `ids + positions → logits`:
+
+- The host runs the vision encoder **once per image** (resize 448, normalize `x/127.5−1`) and writes
+  `image_embeds [64,1024]` into one owned MTLBuffer the engine binds on every step.
+- The prompt's `<|image_pad|>` ids are rewritten to **extension ids** `V + slot` (slot 0..63).
+  In-graph: `embed = ids < V ? table[ids] : image_embeds[ids − V]`.
+- Positions are **plain 1D** (no M-RoPE / no rope-shift), the qwen3.5-hybrid KV + conv + recurrent
+  states are the engine's; nothing else changes.
+
+Simpler than the [Qwen3-VL port](https://huggingface.co/mlboydaisuke/Qwen3-VL-2B-CoreAI) (no deepstack, no M-RoPE).
+
+## Measured (iPhone 17 Pro, iOS 27 beta, release)
+
+- **decode ~51.5 tok/s** (text core 53.4), engine cold-spec ~3–5 s, ~1.5 GB resident (jetsam-safe).
+- **Numerics**: fp32-torch parity bit-exact (vision cos 1.000000, full overlay logits cos 1.00004);
+  Core AI engine ≡ python ≡ HF (text 24/24; image path reproduces the HF description modulo one int8
+  near-tie token, then reconverges).
+- Real-photo example (kakigōri): *"a bowl of shaved ice ... chunks of mango ... a dark blue saucer ...
+  a menu or a book, hinting at a café ... a wooden table"* — accurate, fully on-device.
+
+## Use it
+
+`apps/CoreAIChat` and the standalone `MiniCPMVLM` app have a **MiniCPM-V 4.6 mode with a photo picker**:
+pick an image, ask, stream. The vision tower runs once per image (~hundreds of ms); each turn re-prefills (S=1).
+
+Conversion + gates: see [coreai-model-zoo / minicpm-v-4.6](https://github.com/john-rocky/coreai-model-zoo/blob/main/zoo/minicpm-v-4.6.md).
+
+License: Apache-2.0 (inherited from `openbmb/MiniCPM-V-4.6`).