| --- |
| license: apache-2.0 |
| base_model: Hcompany/Holo2-4B |
| pipeline_tag: image-text-to-text |
| tags: |
| - core-ai |
| - coreml |
| - on-device |
| - ios |
| - iphone |
| - vision-language |
| - gui-grounding |
| - computer-use |
| - qwen3-vl |
| language: |
| - en |
| --- |
| |
| # Holo2-4B — Core AI (on-device, iPhone) · GUI-grounding VLM |
|
|
| [Hcompany/Holo2-4B](https://huggingface.co/Hcompany/Holo2-4B) converted to Apple **Core AI** |
| for on-device inference, served by the **CoreAIChat** app. |
|
|
| Holo2 is H Company's **computer-use / GUI-grounding** vision-language model: given a screenshot |
| and an instruction ("click the submit button"), it predicts the **click coordinates / locates the |
| UI element**. Built on the **Qwen3-VL-4B** backbone, so it rides the Core AI zoo's existing |
| Qwen3-VL pipeline. The zoo's **first GUI-grounding / computer-use model**. |
|
|
| <!-- gen-cards:use-it begin id=holo2-4b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) --> |
| ## Use it |
|
|
| ▶️ **Run it (source)** — the [VLChat runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/VLChat) |
| (GUI + CLI, one app for every vision-language model in the catalog): |
|
|
| ```bash |
| git clone https://github.com/john-rocky/coreai-kit |
| open coreai-kit/Examples/VLChat/VLChat.xcodeproj |
| # → Run, then pick "Holo2 4B" in the model picker |
| |
| # agents / headless (macOS): |
| cd coreai-kit/Examples/VLChat |
| swift run vlchat-cli --model holo2-4b --image screenshot.png --prompt "Localize an element on the GUI image according to my instructions and output a click position as Click(x, y) with x num pixels from the left edge and y num pixels from the top edge. Instruction: click the Submit button." |
| ``` |
|
|
| 💻 **Build with it** — complete; the glue is kit API, copy-paste runs: |
|
|
| ```swift |
| import CoreAIKit |
| import FoundationModels |
| |
| let vlm = try await KitVisionModel(catalog: "holo2-4b") |
| let session = LanguageModelSession(model: vlm) |
| let image = try ImageFile.load(imageURL) // any image file → CGImage + EXIF orientation |
| let reply = try await session.respond(to: Prompt { |
| prompt |
| Attachment(image.cgImage, orientation: image.orientation) |
| }) |
| // reply.content: "Click(x, y)" in 0-1000-normalized coordinates for a grounding prompt, |
| // or a plain answer for a normal question - all generated on-device |
| ``` |
|
|
| The take-home is [`Examples/VLChat/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/VLChat/Sources/QuickStart.swift) |
| — this exact code as one typed function, no UI; the CLI is an argument shell over it, and |
| the GUI drives the same `KitVisionModel(catalog:)` behind a `LanguageModelSession`. |
| Holo2 is a GUI-grounding model: feed a screenshot and H Company's localization prompt |
| (see the card's grounding section) and it returns `Click(x, y)` in 0-1000-normalized |
| coordinates — multiply by `imageSize / 1000` for pixels. It also answers free-form |
| questions like its Qwen3-VL base. |
|
|
| **Integration checklist** |
|
|
| - SPM: `https://github.com/john-rocky/coreai-kit` → product **CoreAIKit** |
| - Info.plist: `NSPhotoLibraryUsageDescription` — only if you use PhotosPicker |
| - Entitlements (iOS): `com.apple.developer.kernel.increased-memory-limit` |
| - First run downloads the model — 5.5 GB (Mac) / 5.5 GB (iPhone) — then it loads from the |
| local cache (Application Support; progress via the `downloadProgress` callback) |
| - Measure in Release — Debug is ~3× slower on per-token host work |
| <!-- gen-cards:use-it end --> |
|
|
| ## Contents (`gpu-pipelined/`) |
| - `holo2_4b_decode_int8lin_s1/` — the **decode** bundle (static query=1, per-block-32 int8 linear |
| body; rides Apple's `coreai-pipelined` GPU engine, specializes on-device — no AOT needed). ~4.4 GB. |
| - `holo2_4b_vision/` — the fixed-grid **vision encoder** `.aimodel` (fp16): `patches [784,1536] |
| -> (image_embeds [196,2560], deepstack [3,196,2560])`. Run once per image. ~0.8 GB. |
| |
| ## Parity (vs fp32 HF oracle, Core AI GPU engine) |
| - **Vision:** image-embeds cos **0.999983**, deepstack cos **0.999989**. |
| - **Decoder (int8lin):** teacher-forced S=1 sweep **4/4**, **16/16** decode steps token-exact, |
| HF-seeded decode match. All PASS. |
| |
| ## Use |
| Install **CoreAIChat**, pick **Holo2 4B**, attach a screenshot, and ask where an element is / |
| what to click — it grounds the instruction to the image. |
| |
| ## License |
| Apache-2.0, inherited from the base model |
| [Hcompany/Holo2-4B](https://huggingface.co/Hcompany/Holo2-4B). See `LICENSE`. |
| |
