---
license: apache-2.0
base_model: HuggingFaceTB/SmolVLM-256M-Instruct
library_name: coreml
pipeline_tag: image-text-to-text
language:
  - en
tags:
  - coreml
  - core-ml
  - apple-silicon
  - on-device
  - vlm
  - smolvlm
  - image-to-text
---

# SmolVLM-256M — Core ML

Core ML (`.mlpackage`) builds of [HuggingFaceTB/SmolVLM-256M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct),
converted for on-device image captioning and VQA on Apple silicon.

> **Two of the four compute-unit settings load successfully and return garbage.** They do not
> crash, so nothing tells you they are wrong. Use `.all` or `.cpuAndGPU`, and compile the
> package before loading it. Details below.

Captions an image in **~2 s of model time on macOS 27.2** (~1.3 s on macOS 26.6.2), entirely on device. The companion Swift CLI
that drives these packages has **no Python and no third-party runtime dependencies**, and
its preprocessing is bit-exact with the HuggingFace reference — a libjpeg-compatible JPEG
decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified
against golden data on every run.

```
image ──► vision (fp16) ──► prefill (K/V as plain outputs)
                                │
text ──► BPE tokenizer ─────────┴─► decode (S=1, MLState) ──► tokens
```

## ⚠️ Compute units: pick `.all` or `.cpuAndGPU`, and compile first

Measured on macOS 26.5 / coremltools 9.0, loading the **precompiled `.mlmodelc`** from a
Swift app:

| setting | loads | correct | notes |
|---|---|---|---|
| `.all` | yes | **yes** | fastest measured — vision ~87 ms, decode ~44 tok/s |
| `.cpuAndGPU` | yes | **yes** | vision ~167 ms, decode ~43 tok/s |
| `.cpuAndNeuralEngine` | yes | **no** | emits `<end_of_utterance>` from the first token |
| `.cpuOnly` | yes | **no** | caption collapses to `This Is That Is.` |

Correctness needs the GPU in the mix. The two GPU-less settings **do not crash** — they
silently return wrong output, which is the harder failure to notice.

Separately, loading the `.mlpackage` **from Python** with `compute_units=ALL`
(`ct.models.MLModel(...)`) still crashes with SIGSEGV on this environment — re-confirmed on
macOS 27.2 with the real `smolvlm_decode_hybrid.mlpackage`, where it dies during the load. Compile to
`.mlmodelc` first, or stay on `.cpuAndGPU` in Python. On macOS 27.2 the CLI's `auto` probe still
selects `.all`; `.cpuAndNeuralEngine` and `.cpuOnly` were not re-tested there.

Because "it loaded" is not evidence of correctness, the companion CLI defaults to
`--compute-units auto`: it probes each candidate in a **subprocess** (these failures are
process-killing signals, not catchable errors) and accepts one only if its output token
matches `.cpuAndGPU`. The result is cached per OS build, costing about 0.7 s once.

## Companion repos

- **Verify with the dataset — [NagaYu/smolvlm-coreml-golden](https://huggingface.co/datasets/NagaYu/smolvlm-coreml-golden)** (~43 MB). It holds the golden token ids, pixel tensors, raw PIL decode and test image that `smolvlm --check --root .` compares against on every run. Download it next to these packages if you want the bit-exactness claims on this card proven on your machine rather than taken on faith.
- **Compare against the Space — `smolvlm-coreml-oracle` (currently not provided).** The Space is not published at the moment, so there is no link; its source is in [`space/`](https://github.com/NagaYu/smolvlm-coreml/tree/main/space) of the companion repo. It runs the original SmolVLM-256M in **PyTorch on CPU**; Hugging Face Spaces are Linux, so Core ML cannot execute there and this Space is deliberately *not* a demo of these packages. It is the OS-independent reference oracle — it emits the reference artifacts you diff your on-device Core ML run against, and its output does not move when your Mac's OS moves.

> **Arrived here after an Apple OS update?** Read [docs/os-upgrade.md](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/os-upgrade.md) first — these packages still load (built with `minimum_deployment_target = iOS18`, specificationVersion 9 = iOS18/macOS15), but the goldens and the compute-unit verdicts are stamped to macOS 26.5 / coremltools 9.0 (the timings below were re-measured on macOS 27.2), and that guide is the checklist for re-proving them.

## Files

| File | Role |
|---|---|
| `smolvlm_vision.mlpackage` | Vision encoder + connector (fp16) |
| `smolvlm_prefill_kv.mlpackage` | Batch prefill, P=128, K/V returned as plain outputs (fp16) |
| `smolvlm_decode_hybrid.mlpackage` | S=1 decode, state only for generated tokens (fp16) |
| `smolvlm_prefill_kv_1024.mlpackage` | Prefill for the 13-tile hi-res split path, P=1024 |
| `smolvlm_decode_hybrid_1024.mlpackage` | Decode for the split path |
| `*_int8.mlpackage` | INT8 per-channel **asymmetric** variants of the prefill/decode packages — roughly half the size, see the caveat below |

There is deliberately **no INT8 vision package**: quantizing the vision tower drops cosine
similarity against the torch reference from 1.00000 to 0.99407, which is enough to change the
caption wording (still accurate, but no longer matching the HuggingFace output). Vision stays fp16.

All packages were converted with **coremltools 9.0**. Other versions may lower the graph
differently; regenerate with `convert_all.py` if you need a different toolchain.

## Usage

The Swift CLI, the conversion scripts, and the bit-exactness verification all live in the
companion repository: **[NagaYu/smolvlm-coreml](https://github.com/NagaYu/smolvlm-coreml)**.

The CLI resolves both the `.mlpackage` files **and** `exports/` from a single `--root`
directory, so download the packages and runtime assets **into the cloned repo**. This repo
mirrors the clone's layout, so `--local-dir .` puts every file where `--root .` looks for it:

```bash
git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml

# download only the minimal fp16 set + runtime assets, into the repo root
hf download NagaYu/SmolVLM-256M-Instruct-CoreML --local-dir . \
  --include "smolvlm_vision.mlpackage/*" \
  --include "smolvlm_prefill_kv.mlpackage/*" \
  --include "smolvlm_decode_hybrid.mlpackage/*" \
  --include "exports/embeddings.bin" --include "exports/tok/*" --include "exports/specials.json"

cd SmolVLMSwift && swift build -c release && cd ..
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --question "How many cats?" --root .
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --split --root .   # 13-tile hi-res
./SmolVLMSwift/.build/release/smolvlm --check --root .                          # re-prove bit-exactness
```

Add `--include "*_1024*"` for the hi-res split path, or `--include "*_int8*"` for the
INT8 variants. The non-split goldens for `--check` are committed in the GitHub repo; the
split-path pixel golden is in the dataset repo. You can also regenerate every package
yourself with `python convert_all.py && python export_artifacts.py` instead of downloading them.

## Measured performance

Apple M2 MacBook Pro, macOS 27.2 (build 26B123), fp16, warm process, median of 10 runs after
2 warmups. The CLI default (`--compute-units auto`) still selects `.all` here; `.cpuAndGPU`
is shown as the conservative fallback. The `--split` column was not re-measured on 27.2 and
is the earlier macOS 26.6.2 measurement:

| Stage | `.all` (auto-selected) | `.cpuAndGPU` | `--split`, 13 tiles (P=1024, `.cpuAndGPU`) — 26.6.2 |
|---|---|---|---|
| Model load (warm) | 0.25 s | 0.2 s | 0.3 s |
| Vision | ~90 ms (1 tile) | ~168 ms | ~1.3 s (13 tiles) |
| Prefill + first logits | ~950 ms (81 tok) | ~960 ms | ~300 ms (876 tok) |
| Decode | ~41 tok/s | ~41 tok/s | ~34 tok/s |
| 40-token caption, model time | ~2.0 s | ~2.1 s | — |

On macOS 26.6.2 the same packages measured vision ~87 ms / ~167 ms, prefill + first logits
~200 ms / ~180 ms and decode ~43 / ~40 tok/s (`.all` / `.cpuAndGPU`), for a ~1.3 s caption.
The 27.2 prefill slowdown reproduces with both the original compile and a fresh 27.2 compile,
so it comes from the OS runtime, not the packages. Decode rate varies by a few tok/s between
sessions. The first run in a fresh process also pays a cold-start penalty on vision and
prefill; the numbers above are warm.

## Why the architecture looks like this

The obvious design — a multifunction package where `prefill` and `decode` share `MLState`,
as the WWDC-era examples suggest — does not survive on this runtime. It crashes in the
decode loop under every compute-unit combination tested, including `CPU_ONLY × CPU_ONLY`;
in the run that was instrumented step-by-step, the first cross-function predict succeeded
before the loop died. The same goes for `write_state` (appears to succeed, round-trips
cleanly through `read_state`, then dies dozens of predicts later) and for passing one
model's state into another model's `predict` (immediate crash).

So state never crosses a boundary here. **Prefill is stateless** and returns all 30 layers'
K/V as ordinary outputs (`[1, 90, P, 64]`); **decode takes them back as ordinary inputs**
and keeps `MLState` only for the tokens it generates itself. That design ported to Swift
without a single crash.

The full catalog of seven reproducible crashes, with the reasoning and repro steps:
**[the field report](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/field-report.md)**.

## Verified parity

The companion CLI's `--check` mode re-proves the tokenizer, JPEG decode and resize rows on
every run (re-confirmed on macOS 27.2 / Swift 6.4: 81/81, 876/876, 0/255, maxdiff 0.000000).
It does not run Core ML; the normalization, prefill and caption rows were measured against
the Python pipeline separately.

| Component | Match level |
|---|---|
| Tokenizer | Exact — 81/81 tokens non-split, 876/876 split |
| JPEG decode | Bit-exact vs libjpeg, maxdiff 0/255 |
| Resize | Bit-exact vs Pillow LANCZOS (down to its fixed-point arithmetic) |
| Normalization | Bit-exact, all 256 values identical in fp16 |
| Prefill embeddings / K/V | Bitwise identical to the Python Core ML pipeline |
| Caption output | Character-identical to the Python Core ML pipeline, deterministic across runs |

The non-split caption also matches HuggingFace `transformers` greedy output token-for-token.

## Limitations — please read

- **Environment-stamped.** The packages were built and the goldens exported on macOS 26.5 /
  coremltools 9.0 / Apple silicon; the timings, the `--check` results and the Python `ALL`
  crash were re-measured on macOS 27.2 (Apple M2, Swift 6.4) without rebuilding anything. A
  later OS may fix the crashes described here — or quietly change the numerics or the
  speed. All three have been observed.
- **Greedy output drifts across OS sessions.** With bit-identical inputs and artifacts,
  greedy decode produced different captions across OS sessions days apart, while remaining
  fully deterministic within a session. Likely last-ulp changes in GPU kernels between
  system updates — probably not a bug, but it means **never use a saved caption string as a
  regression baseline across OS updates.** Re-derive goldens against the current OS.
- **INT8 costs ~12 s of GPU shader compilation per process.** Core ML does not cache it
  across launches, so you trade a cold-start tax on every run for roughly half the size.
  Worth it for a long-lived process, not for a CLI — which is why fp16 is the default.
- **Split-path captions can differ from a torch-fp32-vision HuggingFace run in late
  tokens.** This is inherent to running the vision tower in Core ML fp16, not a porting
  bug: the *Python* Core ML pipeline diverges from torch-fp32 in exactly the same place.
- **This is a reference implementation for one model, not a general library.** Dimensions
  (576 hidden, 30 layers) are hardcoded where it keeps the code readable.
- The bundled JPEG decoder covers baseline 4:4:4 and grayscale — what libjpeg parity was
  verified against. Other JPEGs fall back to CoreGraphics (±4/255 decode variance).

## Licenses & attribution

- **Weights**: derivative works of [HuggingFaceTB/SmolVLM-256M-Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-256M-Instruct),
  Apache 2.0. All credit for the model itself goes to the SmolVLM authors at HuggingFaceTB.
- **Conversion scripts and Swift CLI**: MIT, in the [companion repository](https://github.com/NagaYu/smolvlm-coreml).

## Citation

If the crash catalog or the stateless-prefill design is useful in your own work, a link to
the [field report](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/field-report.md)
or the [repository](https://github.com/NagaYu/smolvlm-coreml) is appreciated. Reports of
these crashes reproducing — or *not* reproducing — on other OS versions are especially
welcome; the more independent reproductions exist, the faster they get looked at.