NagaYu/SmolVLM-256M-Instruct-CoreML

SmolVLM-256M — Core ML

Core ML (.mlpackage) builds of HuggingFaceTB/SmolVLM-256M-Instruct, converted for on-device image captioning and VQA on Apple silicon.

Two of the four compute-unit settings load successfully and return garbage. They do not crash, so nothing tells you they are wrong. Use .all or .cpuAndGPU, and compile the package before loading it. Details below.

Captions an image in ~2 s of model time on macOS 27.2 (~1.3 s on macOS 26.6.2), entirely on device. The companion Swift CLI that drives these packages has no Python and no third-party runtime dependencies, and its preprocessing is bit-exact with the HuggingFace reference — a libjpeg-compatible JPEG decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified against golden data on every run.

image ──► vision (fp16) ──► prefill (K/V as plain outputs)
                                │
text ──► BPE tokenizer ─────────┴─► decode (S=1, MLState) ──► tokens

⚠️ Compute units: pick .all or .cpuAndGPU, and compile first

Measured on macOS 26.5 / coremltools 9.0, loading the precompiled .mlmodelc from a Swift app:

setting	loads	correct	notes
.all	yes	yes	fastest measured — vision ~87 ms, decode ~44 tok/s
.cpuAndGPU	yes	yes	vision ~167 ms, decode ~43 tok/s
.cpuAndNeuralEngine	yes	no	emits <end_of_utterance> from the first token
.cpuOnly	yes	no	caption collapses to This Is That Is.

Correctness needs the GPU in the mix. The two GPU-less settings do not crash — they silently return wrong output, which is the harder failure to notice.

Separately, loading the .mlpackage from Python with compute_units=ALL (ct.models.MLModel(...)) still crashes with SIGSEGV on this environment — re-confirmed on macOS 27.2 with the real smolvlm_decode_hybrid.mlpackage, where it dies during the load. Compile to .mlmodelc first, or stay on .cpuAndGPU in Python. On macOS 27.2 the CLI's auto probe still selects .all; .cpuAndNeuralEngine and .cpuOnly were not re-tested there.

Because "it loaded" is not evidence of correctness, the companion CLI defaults to --compute-units auto: it probes each candidate in a subprocess (these failures are process-killing signals, not catchable errors) and accepts one only if its output token matches .cpuAndGPU. The result is cached per OS build, costing about 0.7 s once.

Companion repos

• Verify with the dataset — NagaYu/smolvlm-coreml-golden (~43 MB). It holds the golden token ids, pixel tensors, raw PIL decode and test image that smolvlm --check --root . compares against on every run. Download it next to these packages if you want the bit-exactness claims on this card proven on your machine rather than taken on faith.
• Compare against the Space — smolvlm-coreml-oracle (currently not provided). The Space is not published at the moment, so there is no link; its source is in space/ of the companion repo. It runs the original SmolVLM-256M in PyTorch on CPU; Hugging Face Spaces are Linux, so Core ML cannot execute there and this Space is deliberately not a demo of these packages. It is the OS-independent reference oracle — it emits the reference artifacts you diff your on-device Core ML run against, and its output does not move when your Mac's OS moves.

Arrived here after an Apple OS update? Read docs/os-upgrade.md first — these packages still load (built with minimum_deployment_target = iOS18, specificationVersion 9 = iOS18/macOS15), but the goldens and the compute-unit verdicts are stamped to macOS 26.5 / coremltools 9.0 (the timings below were re-measured on macOS 27.2), and that guide is the checklist for re-proving them.

Files

File	Role
smolvlm_vision.mlpackage	Vision encoder + connector (fp16)
smolvlm_prefill_kv.mlpackage	Batch prefill, P=128, K/V returned as plain outputs (fp16)
smolvlm_decode_hybrid.mlpackage	S=1 decode, state only for generated tokens (fp16)
smolvlm_prefill_kv_1024.mlpackage	Prefill for the 13-tile hi-res split path, P=1024
smolvlm_decode_hybrid_1024.mlpackage	Decode for the split path
*_int8.mlpackage	INT8 per-channel asymmetric variants of the prefill/decode packages — roughly half the size, see the caveat below

There is deliberately no INT8 vision package: quantizing the vision tower drops cosine similarity against the torch reference from 1.00000 to 0.99407, which is enough to change the caption wording (still accurate, but no longer matching the HuggingFace output). Vision stays fp16.

All packages were converted with coremltools 9.0. Other versions may lower the graph differently; regenerate with convert_all.py if you need a different toolchain.

Usage

The Swift CLI, the conversion scripts, and the bit-exactness verification all live in the companion repository: NagaYu/smolvlm-coreml.

The CLI resolves both the .mlpackage files and exports/ from a single --root directory, so download the packages and runtime assets into the cloned repo. This repo mirrors the clone's layout, so --local-dir . puts every file where --root . looks for it:

git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml

# download only the minimal fp16 set + runtime assets, into the repo root
hf download NagaYu/SmolVLM-256M-Instruct-CoreML --local-dir . \
  --include "smolvlm_vision.mlpackage/*" \
  --include "smolvlm_prefill_kv.mlpackage/*" \
  --include "smolvlm_decode_hybrid.mlpackage/*" \
  --include "exports/embeddings.bin" --include "exports/tok/*" --include "exports/specials.json"

cd SmolVLMSwift && swift build -c release && cd ..
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --question "How many cats?" --root .
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --split --root .   # 13-tile hi-res
./SmolVLMSwift/.build/release/smolvlm --check --root .                          # re-prove bit-exactness

Add --include "*_1024*" for the hi-res split path, or --include "*_int8*" for the INT8 variants. The non-split goldens for --check are committed in the GitHub repo; the split-path pixel golden is in the dataset repo. You can also regenerate every package yourself with python convert_all.py && python export_artifacts.py instead of downloading them.

Measured performance

Apple M2 MacBook Pro, macOS 27.2 (build 26B123), fp16, warm process, median of 10 runs after 2 warmups. The CLI default (--compute-units auto) still selects .all here; .cpuAndGPU is shown as the conservative fallback. The --split column was not re-measured on 27.2 and is the earlier macOS 26.6.2 measurement:

Stage	.all (auto-selected)	.cpuAndGPU	--split, 13 tiles (P=1024, .cpuAndGPU) — 26.6.2
Model load (warm)	0.25 s	0.2 s	0.3 s
Vision	~90 ms (1 tile)	~168 ms	~1.3 s (13 tiles)
Prefill + first logits	~950 ms (81 tok)	~960 ms	~300 ms (876 tok)
Decode	~41 tok/s	~41 tok/s	~34 tok/s
40-token caption, model time	~2.0 s	~2.1 s	—

On macOS 26.6.2 the same packages measured vision ~87 ms / ~167 ms, prefill + first logits ~200 ms / ~180 ms and decode ~43 / ~40 tok/s (.all / .cpuAndGPU), for a ~1.3 s caption. The 27.2 prefill slowdown reproduces with both the original compile and a fresh 27.2 compile, so it comes from the OS runtime, not the packages. Decode rate varies by a few tok/s between sessions. The first run in a fresh process also pays a cold-start penalty on vision and prefill; the numbers above are warm.

Why the architecture looks like this

The obvious design — a multifunction package where prefill and decode share MLState, as the WWDC-era examples suggest — does not survive on this runtime. It crashes in the decode loop under every compute-unit combination tested, including CPU_ONLY × CPU_ONLY; in the run that was instrumented step-by-step, the first cross-function predict succeeded before the loop died. The same goes for write_state (appears to succeed, round-trips cleanly through read_state, then dies dozens of predicts later) and for passing one model's state into another model's predict (immediate crash).

So state never crosses a boundary here. Prefill is stateless and returns all 30 layers' K/V as ordinary outputs ([1, 90, P, 64]); decode takes them back as ordinary inputs and keeps MLState only for the tokens it generates itself. That design ported to Swift without a single crash.

The full catalog of seven reproducible crashes, with the reasoning and repro steps: the field report.

Verified parity

The companion CLI's --check mode re-proves the tokenizer, JPEG decode and resize rows on every run (re-confirmed on macOS 27.2 / Swift 6.4: 81/81, 876/876, 0/255, maxdiff 0.000000). It does not run Core ML; the normalization, prefill and caption rows were measured against the Python pipeline separately.

Component	Match level
Tokenizer	Exact — 81/81 tokens non-split, 876/876 split
JPEG decode	Bit-exact vs libjpeg, maxdiff 0/255
Resize	Bit-exact vs Pillow LANCZOS (down to its fixed-point arithmetic)
Normalization	Bit-exact, all 256 values identical in fp16
Prefill embeddings / K/V	Bitwise identical to the Python Core ML pipeline
Caption output	Character-identical to the Python Core ML pipeline, deterministic across runs

The non-split caption also matches HuggingFace transformers greedy output token-for-token.

Limitations — please read

• Environment-stamped. The packages were built and the goldens exported on macOS 26.5 / coremltools 9.0 / Apple silicon; the timings, the --check results and the Python ALL crash were re-measured on macOS 27.2 (Apple M2, Swift 6.4) without rebuilding anything. A later OS may fix the crashes described here — or quietly change the numerics or the speed. All three have been observed.
• Greedy output drifts across OS sessions. With bit-identical inputs and artifacts, greedy decode produced different captions across OS sessions days apart, while remaining fully deterministic within a session. Likely last-ulp changes in GPU kernels between system updates — probably not a bug, but it means never use a saved caption string as a regression baseline across OS updates. Re-derive goldens against the current OS.
• INT8 costs ~12 s of GPU shader compilation per process. Core ML does not cache it across launches, so you trade a cold-start tax on every run for roughly half the size. Worth it for a long-lived process, not for a CLI — which is why fp16 is the default.
• Split-path captions can differ from a torch-fp32-vision HuggingFace run in late tokens. This is inherent to running the vision tower in Core ML fp16, not a porting bug: the Python Core ML pipeline diverges from torch-fp32 in exactly the same place.
• This is a reference implementation for one model, not a general library. Dimensions (576 hidden, 30 layers) are hardcoded where it keeps the code readable.
• The bundled JPEG decoder covers baseline 4:4:4 and grayscale — what libjpeg parity was verified against. Other JPEGs fall back to CoreGraphics (±4/255 decode variance).

Licenses & attribution

• Weights: derivative works of HuggingFaceTB/SmolVLM-256M-Instruct, Apache 2.0. All credit for the model itself goes to the SmolVLM authors at HuggingFaceTB.
• Conversion scripts and Swift CLI: MIT, in the companion repository.

Citation

If the crash catalog or the stateless-prefill design is useful in your own work, a link to the field report or the repository is appreciated. Reports of these crashes reproducing — or not reproducing — on other OS versions are especially welcome; the more independent reproductions exist, the faster they get looked at.