README: add SAM 3 section (shapes, timings, SAM License)
Browse files
README.md
CHANGED
|
@@ -2,11 +2,13 @@
|
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
- hustvl/Moebius
|
|
|
|
| 5 |
tags:
|
| 6 |
- executorch
|
| 7 |
- react-native
|
| 8 |
- on-device
|
| 9 |
- inpainting
|
|
|
|
| 10 |
- coreml
|
| 11 |
- xnnpack
|
| 12 |
---
|
|
@@ -73,3 +75,37 @@ DA3-SMALL and DA3-BASE weights and code are Apache 2.0, and these exports inheri
|
|
| 73 |
year={2025}
|
| 74 |
}
|
| 75 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
- hustvl/Moebius
|
| 5 |
+
- facebook/sam3
|
| 6 |
tags:
|
| 7 |
- executorch
|
| 8 |
- react-native
|
| 9 |
- on-device
|
| 10 |
- inpainting
|
| 11 |
+
- segmentation
|
| 12 |
- coreml
|
| 13 |
- xnnpack
|
| 14 |
---
|
|
|
|
| 75 |
year={2025}
|
| 76 |
}
|
| 77 |
```
|
| 78 |
+
|
| 79 |
+
## sam3
|
| 80 |
+
|
| 81 |
+
[SAM 3](https://huggingface.co/facebook/sam3) (Meta, 848M parameters), text-prompted instance segmentation ("segment every X"), exported to ExecuTorch for the [on-device SAM 3 demo app](https://github.com/NorbertKlockiewicz/on-device-sam3). Type a word, every instance is segmented on the phone.
|
| 82 |
+
|
| 83 |
+
Layout: `sam3/coreml/` (iOS). All graphs are static at a 672x672 input (SAM 3 natively uses 1008; 672 finds the same instances on 6 of 7 test prompts and keeps the ViT on the Neural Engine). Core ML fp16.
|
| 84 |
+
|
| 85 |
+
| file | runs | size | iPhone 16 Pro |
|
| 86 |
+
|---|---|---|---|
|
| 87 |
+
| `sam3/coreml/sam3_vision0_672_coreml.pte` | per photo | 230 MB | ~370 ms |
|
| 88 |
+
| `sam3/coreml/sam3_vision1_672_coreml.pte` | per photo | 224 MB | ~330 ms |
|
| 89 |
+
| `sam3/coreml/sam3_vision2_672_coreml.pte` | per photo | 224 MB | ~330 to 500 ms |
|
| 90 |
+
| `sam3/coreml/sam3_vision3_672_coreml.pte` | per photo | 238 MB | ~330 ms |
|
| 91 |
+
| `sam3/coreml/sam3_head_672_coreml.pte` | per prompt | 105 MB | ~170 to 300 ms |
|
| 92 |
+
| `sam3/coreml/sam3_text_672_coreml.pte` | per prompt | 707 MB | ~11 to 20 ms |
|
| 93 |
+
| `sam3/coreml/sam3_tokenizer.json` | | 4 MB | |
|
| 94 |
+
|
| 95 |
+
- `sam3_vision0..3`: the 32-layer ViT in four 8-layer chunks (the ANE compiler rejects the whole ViT as one program). `vision0` takes `(1,3,672,672)` pixels in [-1,1] (`x/127.5 - 1`, photo stretched to square) and returns `(1,48,48,1024)`; `vision1` and `vision2` map that to the same shape; `vision3` adds the FPN neck and returns 3 maps `(1,256,192,192)`, `(1,256,96,96)`, `(1,256,48,48)`.
|
| 96 |
+
- `sam3_text`: CLIP text encoder, `input_ids` and `attention_mask` `(1,32)` int64 (CLIP tokenizer, padded with 49407) -> `(1,32,256)`.
|
| 97 |
+
- `sam3_head`: `(fpn0, fpn1, fpn2, text_features, attention_mask)` -> masks `(1,200,192,192)` logits, boxes `(1,200,4)` normalized cxcywh, logits `(1,200)`, presence `(1,1)`. Score = `sigmoid(logit) * sigmoid(presence)`. FPN position encodings are baked in.
|
| 98 |
+
|
| 99 |
+
Verified end to end against PyTorch on real photos: identical instances and scores, mask IoU 0.994 to 1.0.
|
| 100 |
+
|
| 101 |
+
These exports are derivatives of the SAM 3 weights and are distributed under Meta's SAM License, a copy of which is in `sam3/LICENSE`. They are not covered by the Apache 2.0 license of the other folders.
|
| 102 |
+
|
| 103 |
+
```
|
| 104 |
+
@misc{carion2025sam3segmentconcepts,
|
| 105 |
+
title={SAM 3: Segment Anything with Concepts},
|
| 106 |
+
author={Nicolas Carion and others},
|
| 107 |
+
year={2025},
|
| 108 |
+
eprint={2511.16719},
|
| 109 |
+
archivePrefix={arXiv}
|
| 110 |
+
}
|
| 111 |
+
```
|