---
license: other
license_name: sam-license
license_link: https://github.com/facebookresearch/sam3/blob/main/LICENSE
library_name: coreml
pipeline_tag: mask-generation
base_model: facebook/sam3
tags:
  - coreml
  - sam3
  - sam3.1
  - segmentation
  - video-segmentation
  - object-tracking
  - object-multiplex
  - vision-backbone
  - stateful
  - fp16
  - apple-silicon
---

# SAM3.1 → CoreML (Apple Silicon)

Real SAM3.1 weights exported to CoreML for on-device video segmentation on Apple Silicon: a stateless per-frame vision backbone plus a stateful multiplex tracker (`track_step`), designed to run together.

- [`backbone/`](backbone) — per-frame image encoder (stateless)
- [`tracker/`](tracker) — per-frame `track_step` as a stateful CoreML package (multiplex_count=16)

## Contract

The backbone feeds the tracker:

```
image [1,3,1008,1008] fp32  (normalized to [-1,1])
  -> vis72  [5184,1,256]
  -> hires0 [1,32,288,288]   # post sam_mask_decoder.conv_s0
  -> hires1 [1,64,144,144]   # post conv_s1
```

## Performance

Backbone, per-frame encode, 16GB Apple Silicon (M-series laptop):

| Engine | Mean/frame |
|---|---|
| CoreML (`CPU_AND_GPU`) | ~2.05s |
| MLX | ~2.38s (published M3 Max ViT: ~0.8s — MLX scales better on faster silicon) |

Tracker, `track_step` steady-state (CoreML `CPU_AND_GPU`): ~0.7s/frame (~7s first call, JIT). CoreML fp16 vs eager PyTorch: mask sign-agree 1.0, worst relative error 3.1e-3 across 16 frames — parity holds.

## Tracker: `tracker/`

Stateful `track_step`, `multiplex_count=16`, real SAM3.1 weights (457 tensors, `tracker.model.` prefix stripped from `sam3.1_multiplex.pt`).

- `models/dense_sam3_trackstep.mlpackage` (fp16, 86M) — deploy this, `compute_units=CPU_AND_GPU`.
- `models/dense_sam3_trackstep_fp32.mlpackage` (fp32, 170M) — precision reference only; `CPU_ONLY` predict at mux=16 OOMs on 16GB.
- `scripts/` — export pipeline (`dense_wrapper.py`, `common.py`, `convert_fp16.py`, `export_coreml.py`) plus parity/repro scripts (`verify_coreml_lean.py`, `toy_*`, `ane_compile_test.py`).
- `patches/optimize_state.patch` — works around a coremltools 9.0 bug (`optimize_state.py::canonicalize_inplace_pattern` deletes ops mid-iteration; hits any graph with ≥3 rolling state buffers via shift+cat). Repro: `scripts/toy_bisect.py`.
- `triton_stub/` — stub so `sam3/model/edt.py`'s bare `import triton` doesn't fail on Mac; the real EDT kernel is CUDA-only and unused here.

**ANE:** stateful CoreML models fail ANE compile on macOS 26.5.2 / coremltools 9.0 (platform limitation, not this graph). Deploy on `CPU_AND_GPU`.

**Not wired:** only one conditioning frame is used; the model supports `max_cond_frames_in_attn=4`.

## Backbone: `backbone/`

Per-frame image encoder, stateless. Feeds the tracker with `vis72` / `hires0` / `hires1`.

- `models/sam3_vision_backbone.mlpackage` (~875MB fp16)

## Weights & license

Real SAM3.1 weights, pulled at export time from the community mirror `AEmotionStudio/sam3.1` (Meta gates the original on `facebook/sam3`; override with `SAM3_HF_REPO` / `SAM3_CHECKPOINT`). That mirror is unofficial — verify against Meta's release before trusting it.

Derivative of Meta's SAM Materials → subject to Meta's [SAM License](https://github.com/facebookresearch/sam3/blob/main/LICENSE) (share-alike, field-of-use, attribution). Not permissive — redistributing the weight binaries means honoring those terms.

## Env

Python 3.9, Torch 2.7.0, torchvision 0.22.0, coremltools 9.0 (with `patches/optimize_state.patch`) — this is a coremltools 9.0 ceiling; newer deps will drift the export path.