README.md

---
license: apache-2.0
base_model: facebook/sam2.1-hiera-tiny
tags:
  - segment-anything
  - sam2
  - interactive-segmentation
  - background-removal
  - core-ai
  - apple-silicon
  - macos
library_name: swift-object-cutout
pipeline_tag: mask-generation
---

# cutout-sam2-tiny: SAM 2.1 tiny for Core AI (macOS 27)

[SAM 2.1](https://github.com/facebookresearch/sam2) hiera tiny (Ravi et al., Meta, *SAM 2: Segment
Anything in Images and Videos*, 2024; code and weights Apache-2.0), exported to an Apple Core AI
`.aimodel`. Click a point on a picture and it returns the object under it: three candidate masks
and the network's own estimate of how good each one is.

[`cutout`](https://github.com/arraypress/swift-cutout-cli) uses it for `--at` (click to cut), through
[swift-object-cutout](https://github.com/arraypress/swift-object-cutout). It downloads this
repository by itself the first time it is needed, pinned to one commit, with every file checked
against its published hash.

## Files

| Path | What |
|---|---|
| `cutout-sam2-tiny-float16.aimodel/` | Two entry points (below). Float16 weights and arithmetic, float32 inputs and outputs, 63 MB. |
| `export_sam2.py` | The export script (`SAM2_BUILD_CUDA=0 uv run export_sam2.py --dtype float16`). |
| `LICENSE`, `NOTICE` | The authors' Apache-2.0 licence, and what the export changed. |

**`encode`**: `image` `[1,3,1024,1024]` float32, RGB in 0…1 → `image_embedding` `[1,256,64,64]`,
`feats_s0` `[1,32,256,256]`, `feats_s1` `[1,64,128,128]`. The host stretches the picture to
1024×1024 without keeping its aspect ratio, as SAM 2's own predictor does; the ImageNet
normalisation and SAM 2's no-memory embedding are inside.

**`decode`**: the three encoder outputs, `point` `[1,1,2]` (x then y, in pixels of the 1024² square)
and `label` `[1,1]` (1 = the object, 0 = not it) → `masks` `[1,3,256,256]` logits and `scores`
`[1,3]`, the predicted IoU of each. Take the best-scoring mask, threshold it at 0 and stretch it back
over the picture. One encoding serves any number of clicks.

Nothing in the network is rewritten: the two entry points are the official
`SAM2ImagePredictor.set_image` and `_predict` steps, traced as they are. In float32 on the CPU the
traced steps match the predictor to 8 × 10⁻⁵ in the logits.

## Fidelity

60 clicks (two per picture) on 30 generated pictures: 20 AI-generated room and house scenes at
1024², and 10 synthetic shape scenes at sizes from 640×480 to 1600×900. The same 1024² pixels went
to every runtime. The reference is the PyTorch original in float32 on the CPU; "chosen" is the
best-scoring mask, which is the one a click returns.

| Runtime | Chosen-mask IoU, mean | worst | Clicks ≥ 0.99 | Same mask chosen | Score error, worst |
|---|---|---|---|---|---|
| this export, float16 (this repo) | 0.9996 | 0.9959 | 60 of 60 | 60 of 60 | 0.006 |
| this export, float32 (not published) | 1.0000 | 1.0000 | 60 of 60 | 60 of 60 | < 10⁻⁴ |
| Apple's Core ML conversion (`apple/coreml-sam2.1-tiny`, FLOAT16) | 0.968 | 0.500 | 34 of 60 | 56 of 60 | 0.22 |

Apple's Core ML conversion is what `cutout` ran before. Its masks drift from the original on
textured scenes, where it sometimes picks a different mask (IoU 0.50 on one click). This export
agrees with the original on every click. On the synthetic scenes the two runtimes agree to
0.99 or better, end to end; on the generated rooms they differ by as much as Apple's conversion differs
from the original.

## Speed and memory

Apple M3 Max, macOS 27, GPU, measured through `cutout`:

- **First load:** compiles the model for the Mac in well under a second and keeps a 60 MB compiled
  copy in `~/Library/Caches/coreai-cache`.
- **Per click:** `encode` takes 0.15 s and `decode` 5 ms. One `cutout --at` run takes 0.6–1.0 s from
  start to finish, against 4.5 s for the Core ML conversion, most of which went on loading.
- **Batches:** a folder of 30 pictures took 0.51 s per picture (decode, model, edge, PNG), against
  0.42 s for the Core ML conversion, which ran on the Neural Engine.
- **Memory:** the folder run peaked at 1.2 GB, against 1.4 GB for the Core ML conversion.

## Use

```sh
cutout photo.jpg --at 0.32,0.6          # fetches this model on first use
cutout ./shots --at 0.5,0.45 --out cut  # a folder, the model loaded once
cutout model remove click               # the model and its compiled copy
```

Or by hand: `hf download arraypress/cutout-sam2-tiny --local-dir models`, then point
`$CUTOUT_CLICK_MODEL` at the `.aimodel`. Requires macOS 27 (Core AI) on Apple silicon. Converted
with coreai-torch 0.4.3, coreai-core 1.0.0b3, torch 2.9.0, hydra-core 1.3.2 and sam2 at 2b90b9f.

## Licence

Apache-2.0, as the original: see `LICENSE` (the authors' licence) and `NOTICE` (the copyright, and
what the export changed).