| --- |
| license: other |
| license_name: sam-license |
| license_link: https://github.com/facebookresearch/sam3/blob/main/LICENSE |
| library_name: coreml |
| pipeline_tag: mask-generation |
| base_model: facebook/sam3 |
| tags: |
| - coreml |
| - sam3 |
| - sam3.1 |
| - segmentation |
| - video-segmentation |
| - object-tracking |
| - object-multiplex |
| - vision-backbone |
| - stateful |
| - fp16 |
| - apple-silicon |
| --- |
| |
| # SAM3.1 → CoreML (Apple Silicon) |
|
|
| Real SAM3.1 weights exported to CoreML for on-device video segmentation on Apple Silicon: a stateless per-frame vision backbone plus a stateful multiplex tracker (`track_step`), designed to run together. |
|
|
| - [`backbone/`](backbone) — per-frame image encoder (stateless) |
| - [`tracker/`](tracker) — per-frame `track_step` as a stateful CoreML package (multiplex_count=16) |
| |
| ## Contract |
| |
| The backbone feeds the tracker: |
| |
| ``` |
| image [1,3,1008,1008] fp32 (normalized to [-1,1]) |
| -> vis72 [5184,1,256] |
| -> hires0 [1,32,288,288] # post sam_mask_decoder.conv_s0 |
| -> hires1 [1,64,144,144] # post conv_s1 |
| ``` |
| |
| ## Performance |
| |
| Backbone, per-frame encode, 16GB Apple Silicon (M-series laptop): |
| |
| | Engine | Mean/frame | |
| |---|---| |
| | CoreML (`CPU_AND_GPU`) | ~2.05s | |
| | MLX | ~2.38s (published M3 Max ViT: ~0.8s — MLX scales better on faster silicon) | |
| |
| Tracker, `track_step` steady-state (CoreML `CPU_AND_GPU`): ~0.7s/frame (~7s first call, JIT). CoreML fp16 vs eager PyTorch: mask sign-agree 1.0, worst relative error 3.1e-3 across 16 frames — parity holds. |
|
|
| ## Tracker: `tracker/` |
|
|
| Stateful `track_step`, `multiplex_count=16`, real SAM3.1 weights (457 tensors, `tracker.model.` prefix stripped from `sam3.1_multiplex.pt`). |
|
|
| - `models/dense_sam3_trackstep.mlpackage` (fp16, 86M) — deploy this, `compute_units=CPU_AND_GPU`. |
| - `models/dense_sam3_trackstep_fp32.mlpackage` (fp32, 170M) — precision reference only; `CPU_ONLY` predict at mux=16 OOMs on 16GB. |
| - `scripts/` — export pipeline (`dense_wrapper.py`, `common.py`, `convert_fp16.py`, `export_coreml.py`) plus parity/repro scripts (`verify_coreml_lean.py`, `toy_*`, `ane_compile_test.py`). |
| - `patches/optimize_state.patch` — works around a coremltools 9.0 bug (`optimize_state.py::canonicalize_inplace_pattern` deletes ops mid-iteration; hits any graph with ≥3 rolling state buffers via shift+cat). Repro: `scripts/toy_bisect.py`. |
| - `triton_stub/` — stub so `sam3/model/edt.py`'s bare `import triton` doesn't fail on Mac; the real EDT kernel is CUDA-only and unused here. |
|
|
| **ANE:** stateful CoreML models fail ANE compile on macOS 26.5.2 / coremltools 9.0 (platform limitation, not this graph). Deploy on `CPU_AND_GPU`. |
|
|
| **Not wired:** only one conditioning frame is used; the model supports `max_cond_frames_in_attn=4`. |
|
|
| ## Backbone: `backbone/` |
|
|
| Per-frame image encoder, stateless. Feeds the tracker with `vis72` / `hires0` / `hires1`. |
|
|
| - `models/sam3_vision_backbone.mlpackage` (~875MB fp16) |
|
|
| ## Weights & license |
|
|
| Real SAM3.1 weights, pulled at export time from the community mirror `AEmotionStudio/sam3.1` (Meta gates the original on `facebook/sam3`; override with `SAM3_HF_REPO` / `SAM3_CHECKPOINT`). That mirror is unofficial — verify against Meta's release before trusting it. |
|
|
| Derivative of Meta's SAM Materials → subject to Meta's [SAM License](https://github.com/facebookresearch/sam3/blob/main/LICENSE) (share-alike, field-of-use, attribution). Not permissive — redistributing the weight binaries means honoring those terms. |
|
|
| ## Env |
|
|
| Python 3.9, Torch 2.7.0, torchvision 0.22.0, coremltools 9.0 (with `patches/optimize_state.patch`) — this is a coremltools 9.0 ceiling; newer deps will drift the export path. |
|
|
