mlboydaisuke
/

WeVisOne-VL-2B-CoreAI

+---
+license: apache-2.0
+base_model: tencent/WeVisOne-VL-2B
+base_model_relation: quantized
+pipeline_tag: image-segmentation
+library_name: coreai
+tags: [coreai, apple, on-device, iphone, image-matting, referring-expression-segmentation, object-detection]
+---
+# WeVisOne-VL-2B — Core AI
+On an iPhone 18 Pro (iOS 27.2), cutting out what a few words name in a photo — each match's box and its alpha matte —
+took 0.99–1.00 s for one match and 4.1 s for seven with these graphs, measured 2026-10-25 in CoreAIKit's
+`Examples/Cutout`. This repository holds Tencent's [WeVisOne-VL-2B](https://huggingface.co/tencent/WeVisOne-VL-2B)
+(revision `3dad9ce8`) for Apple's Core AI as four graphs: the text decoder (int8, with its last hidden state as a
+second output), the vision tower with the pixel decoder's inputs (fp16), and the 9.5M-parameter pixel decoder in two
+parts (fp16).
+| Variant | Path | Size | Requires | Tested on |
+|---|---|---|---|---|
+| GPU, specialized on the device (`.aimodel`) | `ios/` | 3,298,811,031 B | iOS 27 and an app with `com.apple.developer.kernel.increased-memory-limit` | iPhone 18 Pro, iOS 27.2 (24B5099f), 2026-10-25 |
+| GPU, compiled ahead of time (`.h16c.aimodelc`) | `macos/` | 10,188,197,010 B | macOS 27 on a Mac16,x (the `h16c` architecture) | Mac M4 Max, macOS 27.0 (26A428), 2026-10-25 |
+Each folder is the model folder the Cutout example reads: name it `WeVisOne-VL-2B-CoreAI` and put it in the app's
+Documents (iPhone) or `~/Library/Application Support/Cutout/` (Mac).
+```sh
+hf download mlboydaisuke/WeVisOne-VL-2B-CoreAI --include "ios/*" --local-dir wevisone   # or "macos/*"
+```
+The graphs need a host that runs them in order; they are not a CoreAIKit catalog model. The Cutout example
+([CoreAIKit `Examples/Cutout`](https://github.com/john-rocky/coreai-kit/tree/main/Examples/Cutout)) is that host:
+pick or take a photo, type what to cut out, and save the cut-out as a PNG with the background removed.
+## What each folder holds
+| file | what | bytes (`ios/`) |
+|---|---|---:|
+| `wevisone_vl_2b_decode_int8hu_pf64/` | the text decoder: `main` (S = 1) and `prefill` (S = 64) sharing two KV states; outputs `logits` and `last_hidden` (the last decoder layer's output before the final norm, which the pixel decoder reads at each `<OBJ>`); linears int8 per block of 32 (body `symmetric_with_clipping`, the untied head absmax); `metadata.json` and `tokenizer/` (the source's `tokenizer.json`, `tokenizer_config.json` and `chat_template.jinja`, byte for byte) | 2,467,304,523 |
+| `wevisone_vl_2b_vision_taps_g14.aimodel` | the vision tower for a 448 × 448 image (196 rows): `image_embeds`, `deepstack_embeds` and `taps`, the outputs of ViT layers 5 / 11 / 17 / 23 the pixel decoder reads; fp16 | 811,114,774 |
+| `wv_mask_a_g28_n16_fp16_hf.aimodel` | pixel decoder part A: up to 16 matches per call, the box encoding (`box_pe`) given by the host; fp16 | 19,540,579 |
+| `wv_mask_b_fp16.aimodel` | pixel decoder part B: one call per match, the 1024 × 1024 matte logits; fp16 | 851,155 |
+`macos/` holds the same four graphs compiled for the Mac (`--platform macOS --architecture h16c`, the decoder with
+`--expect-frequent-reshapes`; the compile stores an fp16 copy of the decoder's int8 linears). Each `.aimodel` is its
+export with the MLIR debug locations removed (`strip_debug_info`; op counts and `coreai-build inspect` equal to the
+export's). `config.json` is the source checkpoint's, unchanged; `SHA256SUMS` lists every other file.
+## How a request runs
+The host does what the source package's `predict(task="segmentation")` does, with the photo resized to a fixed
+448 × 448 grid:
+1. Pillow's bicubic resize to 448 × 448 and to 1024 × 1024 (the Cutout host writes it in integer arithmetic: the same
+   bytes as Pillow on PNG input);
+2. the vision tower; the source's chat template with the package's segmentation prompt (its matting task uses the same
+   one) and the system line "You are a helpful assistant.";
+3. the decoder, greedy, no repetition penalty; the prompt in 64-token `prefill` calls with the tail padded into one
+   more; `last_hidden` kept at each `<OBJ>` it writes;
+4. the boxes by the package's `parse_seg_boxes` (four per-mille coordinates before each `<OBJ>`); their Fourier
+   encoding on the host in fp32 — it has to equal torch's bit for bit, including the 7 box widths where Darwin's
+   `logf` rounds differently from torch's `log`;
+5. part A once per 16 matches, part B once per match; sigmoid and a bilinear resize to the photo's size.
+The decoder runs on Core AI's low-level runtime, not the pipelined engine, because the engine returns token ids only
+and never the hidden state.
+## Decoder accuracy
+Against the source package in fp32 on the same fixed grid (33 rows), the int8 decoder gives the reference's argmax at
+the last four prompt positions on 33 of 33 rows and 16 greedy tokens that never leave the reference's except at its
+own near-ties (top-2 softmax gap under 0.1) on 33 of 33. Its logits are not within cosine 0.9999 of the reference at
+every one of 16 teacher-forced steps on any row (lowest 0.99581); no decoder with fp16 weights reaches that on every
+row either (fp16 31 of 33, fp16 weights with fp32 compute 32 of 33). On the task level it finds every one of the
+reference's boxes at IoU ≥ 0.75 and gives every count exactly.
+## Measured
+- **iPhone 18 Pro** (iOS 27.2, the Cutout app, graphs specialized on the phone, warm, thermal state nominal): the
+  lighthouse example (1024 × 683, one match, a 13-token answer) 0.988–0.997 s — tower 89 ms, the 237-token prompt
+  141 ms, the answer 233 ms, part A 75 ms, part B 403 ms; seven chess pieces (a 50-token answer) 4.119–4.138 s, 2.77 s
+  of it part B. The first launch after installing reached READY in 6.85 s and took 8.14 s for its first request;
+  later launches reached READY in 1.5–2.6 s. Peak memory footprint 2.00 GB.
+- **Mac M4 Max** (macOS 27.0, `macos/`): one match in a 3744 × 5616 NASA photo (decoded at 2731 × 4096) 0.545–0.554 s
+  (the PNG another 0.29 s); peak footprint 2.8 GB.
+## Accuracy
+Against the source package in fp32 (transformers 5.12.1, CPU), the photo resized to 448 × 448 first:
+- structured answers of the decoder: boxes found at IoU ≥ 0.75: 100.0 % (63); counts exact: 100.0 % (7); points
+  within 2 % of the side: 89.4 % (47); OCR character error rate 0.000;
+- the pixel decoder's two graphs on the reference's inputs: mattes within MAE 2.09e-4 and IoU(α > 0.5) ≥ 0.992 of the
+  package's fp32 decoder (46 matches);
+- the whole request on the iPhone 18 Pro, on the rows whose boxes equal the reference's (5 rows, 6 matches): mattes
+  within MAE 5.81e-4 and IoU ≥ 0.9959 of the package's.
+## Limits
+- **The fixed 448 × 448 grid.** The source package keeps the photo's aspect ratio (up to 2025 × 32² pixels); these
+  graphs stretch it. Against the package's own form, the fixed grid's fp32 answers find 85.5 % of its boxes at
+  IoU ≥ 0.75, 71.4 % of its counts and 81.8 % of its points.
+- **1,024 positions on iOS.** Core AI's pipelined engine stops a dynamic-KV decoder at 1,024 positions of prompt and
+  answer on iOS (apple/coreai-models#124); the Cutout host binds its cache at 1,024: 237 prompt tokens and up to 787
+  answer tokens.
+- **No repetition penalty.** The package's greedy decoding uses 1.05; in fp32 the two agree on 26 of 33 answers, with
+  every box and count the same.
+## Convert / verify
+The exporters, the gates and the records: the Core AI model zoo's
+[`models/wevisone-vl-2b`](https://github.com/john-rocky/coreai-model-zoo/tree/main/models/wevisone-vl-2b) card and
+[`conversion/wevisone/`](https://github.com/john-rocky/coreai-model-zoo/tree/main/conversion/wevisone), on
+coreai-torch 0.4.1 / coreai-core 1.0.0b2.
+## License
+Apache-2.0, as the source model. `LICENSE` is the source repository's file, unchanged: it names Qwen3-VL-2B-Instruct
+(Alibaba Cloud, Apache-2.0) as a component modified by Tencent.

SHA256SUMS ADDED Viewed

	@@ -0,0 +1,52 @@

+097f84bbe01f4d987e3342ef0810f98bbbe23bc9c9e53c4003afb74507804269  LICENSE
+99809a4eac54638a20d4056572b1cf86f5ae5a69cb82bf12d665ec0d339943b5  README.md
+c50561386446ad7a1b2d61b662d25adfa399acbda713dd78eca8a9696c7e1bea  config.json
+ea85c4914a09c34dffabde43c27adba296d3daa4a93f18df8a5614d633d56c2a  ios/wevisone_vl_2b_decode_int8hu_pf64/metadata.json
+3636d0f0bd6bef02654cdffdc447b79cb2cef8ab02cc75267345946291a489e4  ios/wevisone_vl_2b_decode_int8hu_pf64/tokenizer/chat_template.jinja
+3bb172429b952469b51c64783f52fa01c26d1b30f1f31a4693e385a43f9cfc5e  ios/wevisone_vl_2b_decode_int8hu_pf64/tokenizer/tokenizer.json
+064277e04934ac9c98e66f7325c4854cf225574b32a6d7bba21af2b06ed28ea5  ios/wevisone_vl_2b_decode_int8hu_pf64/tokenizer/tokenizer_config.json
+98a4b7d45613e18422b7790230c7fb449c438ff372dc1f90ccd87eb730d26e59  ios/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.aimodel/main.hash
+f62a162875ab9411a8d3c24d5173b91e2a76f7f244adf4d8c287026ea75bc94e  ios/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.aimodel/main.mlirb
+476071cb7f04bc0e6973f264bc097b67eac7fc0cbfb532ef0a8769d7180b656b  ios/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.aimodel/metadata.json
+9c4e575ce6ab06ad16cd8072a30bc2da418a7d2a4cca1ec180ee09d6cb3c4819  ios/wevisone_vl_2b_vision_taps_g14.aimodel/main.hash
+d8dd7a3f69c47cb95829f7f268d061704214548283eebbfb57acb5d914f56264  ios/wevisone_vl_2b_vision_taps_g14.aimodel/main.mlirb
+37da8d5e4af3e817f6897dd48d1a7b877bf8f236d2c4bfc0ecbc428d71631077  ios/wevisone_vl_2b_vision_taps_g14.aimodel/metadata.json
+1434d0b5184e8d7576723135726cc651e26bf3809e6569b5987bc4cfcc69b465  ios/wv_mask_a_g28_n16_fp16_hf.aimodel/main.hash
+ffcf2e67979851773849aa9d7d28cf2c1b006c45333375422e59ba7249288ac8  ios/wv_mask_a_g28_n16_fp16_hf.aimodel/main.mlirb
+63c23978a24fb97e4fb4bd4717d22ca7123a01898338de54217ef2619b938c72  ios/wv_mask_a_g28_n16_fp16_hf.aimodel/metadata.json
+b1d6d6f26d026eb5f0b9125662fa48973a74d68dfb184f21c777ba697ab389dd  ios/wv_mask_b_fp16.aimodel/main.hash
+13bd5dfef16b7967fb3378727731d7a3930dc058b429e504c607f467a96ac4b2  ios/wv_mask_b_fp16.aimodel/main.mlirb
+a594dae56cfeb974c0d09a957d70dd5a653c6eeb6b9de886dca6049f964ac2a8  ios/wv_mask_b_fp16.aimodel/metadata.json
+ea85c4914a09c34dffabde43c27adba296d3daa4a93f18df8a5614d633d56c2a  macos/wevisone_vl_2b_decode_int8hu_pf64/metadata.json
+3636d0f0bd6bef02654cdffdc447b79cb2cef8ab02cc75267345946291a489e4  macos/wevisone_vl_2b_decode_int8hu_pf64/tokenizer/chat_template.jinja
+3bb172429b952469b51c64783f52fa01c26d1b30f1f31a4693e385a43f9cfc5e  macos/wevisone_vl_2b_decode_int8hu_pf64/tokenizer/tokenizer.json
+064277e04934ac9c98e66f7325c4854cf225574b32a6d7bba21af2b06ed28ea5  macos/wevisone_vl_2b_decode_int8hu_pf64/tokenizer/tokenizer_config.json
+7d9c554e5b03cb2d34b96ba13ba6250dfebf1fe814871d40879d68ee56cf6132  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/manifest.plist
+14e21e5ab77df28feb7b6ae77322cce660c1e9f4c879ea74484dafc57663bed4  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/original_model_0.mpsgraph
+4327eb4a17b51317da76ee920a048cfecf8329f699ad7d53a878b62822065d39  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin
+79548dc9639f6c58f776322f5c7d91e04277978abd3b4381989a2d6cecf594d9  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_1.mpsgraph
+5717a2f2d183c58f20ed8f10c4627ea3500b9072975aceaee149798551543204  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/main-h16c.mlirb
+f5104d32699a284cd90e2b0567d13db42db261a84c31891cbe72cde8d2d34bfd  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/main.hash
+e599e40e5c369dcb3eb8c187de5db4e9192731176dbfddd06d059825fbd2ab32  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/metadata.json
+d36b36cff61256e7b5f5e4223ae04f4b21108f882049a8382419918a9ffab64b  macos/wevisone_vl_2b_decode_int8hu_pf64/wevisone_vl_2b_decode_int8hu_pf64.h16c.aimodelc/stats.json
+14e8b251f2f13da7ceeaa07352fb321c43721c2fe479e5e20e185ce6564cb4cb  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/manifest.plist
+29b64ba897384048ce8b42a4126a433bcd30d36e33e6bb11872bcda7137f4a6d  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin
+e3f463f01df9c1b218ef807f6b48b50b80e84b3944879f46a157153f40bb04f0  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_0.mpsgraph
+4721c693f56129b99283d45785eb32a4c8bbcd0feb1e929cc617c502b2d08b04  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/main-h16c.mlirb
+eb3c7353c36a71df43bfcd310223f22bbf4b646cd83aeb14f5ca79c90e1d2578  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/main.hash
+cc79c5dc0fec1a36fd2de4fc3206b3408e70d20ed3b8664318858af5a48f2539  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/metadata.json
+ec807a7ad31e70f33763d63d8be17affd64751e6184f038f590f56d0d36d9514  macos/wevisone_vl_2b_vision_taps_g14.h16c.aimodelc/stats.json
+99c32ea2b7d2b708885a76ca4e5f0f5cd45ac36672b54ebc69f59c5894c57cf6  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/manifest.plist
+7417e37165afefe97d2f182b4d6c8967c06bd31a099ae3c3454d2186013188ad  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin
+8801915b7ec1f4eaec4ed52b216a199a447c27b4e5951caa7bc5747e433e61da  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_0.mpsgraph
+2b77f9fa69e5be11218bcdc0e377c4ceab5476d69b4875a63fea674a7f611c9d  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/main-h16c.mlirb
+7f391e79d367c1711efec7c5d1537201de8f7b86d7b5eea31087a5d399ed43a9  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/main.hash
+3162d2637fedccabaa7371dd091ba10138cc04c8b6f38d624d273de4e71b9f93  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/metadata.json
+321ebec5f3fde508209c8c22964e64d552bbbd6428a70a98a8b986d0f5a7a63c  macos/wv_mask_a_g28_n16_fp16_hf.h16c.aimodelc/stats.json
+93da02ecb968da73f7c5c92a308d2f1cce1613be3ec2efbc4ef43f24cfaa950e  macos/wv_mask_b_fp16.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/manifest.plist
+84e852629c2898252c5411688b1bc0d881fb30138389ccffdf2a7d5262bb0320  macos/wv_mask_b_fp16.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/resources.bin
+e2bdb1dea7229be0f1dcd217c4d5ccfbfb76892ae0fd10d1e49234210b177d50  macos/wv_mask_b_fp16.h16c.aimodelc/main-h16c-delegates/MPSGraph/mpsExecutable.mpsgraphpackage/specialized_model_0.mpsgraph
+0f27b5c60c698744ce1ccbf24678b145bcfc6ba9467ba642ae2bbfa8deeab95d  macos/wv_mask_b_fp16.h16c.aimodelc/main-h16c.mlirb
+0ebb95fc924231f7bd2d4ab8da0bdc63da6128679d0ee65b9e5c3a918a2409a7  macos/wv_mask_b_fp16.h16c.aimodelc/main.hash
+5712779a178213fb880d090b7d968a111b7de23b21c6035ea53a6d3ffd102556  macos/wv_mask_b_fp16.h16c.aimodelc/metadata.json
+175bab4cfb57b6032d91d65d72bcd80bc2b27fe6434ddb97c0a253f312fb2649  macos/wv_mask_b_fp16.h16c.aimodelc/stats.json