cutout-sam2-tiny: SAM 2.1 tiny for Core AI (macOS 27)

SAM 2.1 hiera tiny (Ravi et al., Meta, SAM 2: Segment Anything in Images and Videos, 2024; code and weights Apache-2.0), exported to an Apple Core AI .aimodel. Click a point on a picture and it returns the object under it: three candidate masks and the network's own estimate of how good each one is.

cutout uses it for --at (click to cut), through swift-object-cutout. It downloads this repository by itself the first time it is needed, pinned to one commit, with every file checked against its published hash.

Files

Path	What
cutout-sam2-tiny-float16.aimodel/	Two entry points (below). Float16 weights and arithmetic, float32 inputs and outputs, 63 MB.
export_sam2.py	The export script (SAM2_BUILD_CUDA=0 uv run export_sam2.py --dtype float16).
LICENSE, NOTICE	The authors' Apache-2.0 licence, and what the export changed.

encode: image [1,3,1024,1024] float32, RGB in 0…1 → image_embedding [1,256,64,64], feats_s0 [1,32,256,256], feats_s1 [1,64,128,128]. The host stretches the picture to 1024×1024 without keeping its aspect ratio, as SAM 2's own predictor does; the ImageNet normalisation and SAM 2's no-memory embedding are inside.

decode: the three encoder outputs, point [1,1,2] (x then y, in pixels of the 1024² square) and label [1,1] (1 = the object, 0 = not it) → masks [1,3,256,256] logits and scores [1,3], the predicted IoU of each. Take the best-scoring mask, threshold it at 0 and stretch it back over the picture. One encoding serves any number of clicks.

Nothing in the network is rewritten: the two entry points are the official SAM2ImagePredictor.set_image and _predict steps, traced as they are. In float32 on the CPU the traced steps match the predictor to 8 × 10⁻⁵ in the logits.

Fidelity

60 clicks (two per picture) on 30 generated pictures: 20 AI-generated room and house scenes at 1024², and 10 synthetic shape scenes at sizes from 640×480 to 1600×900. The same 1024² pixels went to every runtime. The reference is the PyTorch original in float32 on the CPU; "chosen" is the best-scoring mask, which is the one a click returns.

Runtime	Chosen-mask IoU, mean	worst	Clicks ≥ 0.99	Same mask chosen	Score error, worst
this export, float16 (this repo)	0.9996	0.9959	60 of 60	60 of 60	0.006
this export, float32 (not published)	1.0000	1.0000	60 of 60	60 of 60	< 10⁻⁴
Apple's Core ML conversion (apple/coreml-sam2.1-tiny, FLOAT16)	0.968	0.500	34 of 60	56 of 60	0.22

Apple's Core ML conversion is what cutout ran before. Its masks drift from the original on textured scenes, where it sometimes picks a different mask (IoU 0.50 on one click). This export agrees with the original on every click. On the synthetic scenes the two runtimes agree to 0.99 or better, end to end; on the generated rooms they differ by as much as Apple's conversion differs from the original.

Speed and memory

Apple M3 Max, macOS 27, GPU, measured through cutout:

• First load: compiles the model for the Mac in well under a second and keeps a 60 MB compiled copy in ~/Library/Caches/coreai-cache.
• Per click: encode takes 0.15 s and decode 5 ms. One cutout --at run takes 0.6–1.0 s from start to finish, against 4.5 s for the Core ML conversion, most of which went on loading.
• Batches: a folder of 30 pictures took 0.51 s per picture (decode, model, edge, PNG), against 0.42 s for the Core ML conversion, which ran on the Neural Engine.
• Memory: the folder run peaked at 1.2 GB, against 1.4 GB for the Core ML conversion.

Use

cutout photo.jpg --at 0.32,0.6          # fetches this model on first use
cutout ./shots --at 0.5,0.45 --out cut  # a folder, the model loaded once
cutout model remove click               # the model and its compiled copy

Or by hand: hf download arraypress/cutout-sam2-tiny --local-dir models, then point $CUTOUT_CLICK_MODEL at the .aimodel. Requires macOS 27 (Core AI) on Apple silicon. Converted with coreai-torch 0.4.3, coreai-core 1.0.0b3, torch 2.9.0, hydra-core 1.3.2 and sam2 at 2b90b9f.

Licence

Apache-2.0, as the original: see LICENSE (the authors' licence) and NOTICE (the copyright, and what the export changed).