File size: 3,609 Bytes
47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 47d8ad6 9668c49 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
license: other
license_name: sam-license
license_link: https://github.com/facebookresearch/sam3/blob/main/LICENSE
library_name: coreml
pipeline_tag: mask-generation
base_model: facebook/sam3
tags:
- coreml
- sam3
- sam3.1
- segmentation
- video-segmentation
- object-tracking
- object-multiplex
- vision-backbone
- stateful
- fp16
- apple-silicon
---
# SAM3.1 → CoreML (Apple Silicon)
Real SAM3.1 weights exported to CoreML for on-device video segmentation on Apple Silicon: a stateless per-frame vision backbone plus a stateful multiplex tracker (`track_step`), designed to run together.
- [`backbone/`](backbone) — per-frame image encoder (stateless)
- [`tracker/`](tracker) — per-frame `track_step` as a stateful CoreML package (multiplex_count=16)
## Contract
The backbone feeds the tracker:
```
image [1,3,1008,1008] fp32 (normalized to [-1,1])
-> vis72 [5184,1,256]
-> hires0 [1,32,288,288] # post sam_mask_decoder.conv_s0
-> hires1 [1,64,144,144] # post conv_s1
```
## Performance
Backbone, per-frame encode, 16GB Apple Silicon (M-series laptop):
| Engine | Mean/frame |
|---|---|
| CoreML (`CPU_AND_GPU`) | ~2.05s |
| MLX | ~2.38s (published M3 Max ViT: ~0.8s — MLX scales better on faster silicon) |
Tracker, `track_step` steady-state (CoreML `CPU_AND_GPU`): ~0.7s/frame (~7s first call, JIT). CoreML fp16 vs eager PyTorch: mask sign-agree 1.0, worst relative error 3.1e-3 across 16 frames — parity holds.
## Tracker: `tracker/`
Stateful `track_step`, `multiplex_count=16`, real SAM3.1 weights (457 tensors, `tracker.model.` prefix stripped from `sam3.1_multiplex.pt`).
- `models/dense_sam3_trackstep.mlpackage` (fp16, 86M) — deploy this, `compute_units=CPU_AND_GPU`.
- `models/dense_sam3_trackstep_fp32.mlpackage` (fp32, 170M) — precision reference only; `CPU_ONLY` predict at mux=16 OOMs on 16GB.
- `scripts/` — export pipeline (`dense_wrapper.py`, `common.py`, `convert_fp16.py`, `export_coreml.py`) plus parity/repro scripts (`verify_coreml_lean.py`, `toy_*`, `ane_compile_test.py`).
- `patches/optimize_state.patch` — works around a coremltools 9.0 bug (`optimize_state.py::canonicalize_inplace_pattern` deletes ops mid-iteration; hits any graph with ≥3 rolling state buffers via shift+cat). Repro: `scripts/toy_bisect.py`.
- `triton_stub/` — stub so `sam3/model/edt.py`'s bare `import triton` doesn't fail on Mac; the real EDT kernel is CUDA-only and unused here.
**ANE:** stateful CoreML models fail ANE compile on macOS 26.5.2 / coremltools 9.0 (platform limitation, not this graph). Deploy on `CPU_AND_GPU`.
**Not wired:** only one conditioning frame is used; the model supports `max_cond_frames_in_attn=4`.
## Backbone: `backbone/`
Per-frame image encoder, stateless. Feeds the tracker with `vis72` / `hires0` / `hires1`.
- `models/sam3_vision_backbone.mlpackage` (~875MB fp16)
## Weights & license
Real SAM3.1 weights, pulled at export time from the community mirror `AEmotionStudio/sam3.1` (Meta gates the original on `facebook/sam3`; override with `SAM3_HF_REPO` / `SAM3_CHECKPOINT`). That mirror is unofficial — verify against Meta's release before trusting it.
Derivative of Meta's SAM Materials → subject to Meta's [SAM License](https://github.com/facebookresearch/sam3/blob/main/LICENSE) (share-alike, field-of-use, attribution). Not permissive — redistributing the weight binaries means honoring those terms.
## Env
Python 3.9, Torch 2.7.0, torchvision 0.22.0, coremltools 9.0 (with `patches/optimize_state.patch`) — this is a coremltools 9.0 ceiling; newer deps will drift the export path.
|