nklockiewicz commited on
Commit
689f683
·
verified ·
1 Parent(s): 12fc2d3

README: add SAM 3 section (shapes, timings, SAM License)

Browse files
Files changed (1) hide show
  1. README.md +36 -0
README.md CHANGED
@@ -2,11 +2,13 @@
2
  license: apache-2.0
3
  base_model:
4
  - hustvl/Moebius
 
5
  tags:
6
  - executorch
7
  - react-native
8
  - on-device
9
  - inpainting
 
10
  - coreml
11
  - xnnpack
12
  ---
@@ -73,3 +75,37 @@ DA3-SMALL and DA3-BASE weights and code are Apache 2.0, and these exports inheri
73
  year={2025}
74
  }
75
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
3
  base_model:
4
  - hustvl/Moebius
5
+ - facebook/sam3
6
  tags:
7
  - executorch
8
  - react-native
9
  - on-device
10
  - inpainting
11
+ - segmentation
12
  - coreml
13
  - xnnpack
14
  ---
 
75
  year={2025}
76
  }
77
  ```
78
+
79
+ ## sam3
80
+
81
+ [SAM 3](https://huggingface.co/facebook/sam3) (Meta, 848M parameters), text-prompted instance segmentation ("segment every X"), exported to ExecuTorch for the [on-device SAM 3 demo app](https://github.com/NorbertKlockiewicz/on-device-sam3). Type a word, every instance is segmented on the phone.
82
+
83
+ Layout: `sam3/coreml/` (iOS). All graphs are static at a 672x672 input (SAM 3 natively uses 1008; 672 finds the same instances on 6 of 7 test prompts and keeps the ViT on the Neural Engine). Core ML fp16.
84
+
85
+ | file | runs | size | iPhone 16 Pro |
86
+ |---|---|---|---|
87
+ | `sam3/coreml/sam3_vision0_672_coreml.pte` | per photo | 230 MB | ~370 ms |
88
+ | `sam3/coreml/sam3_vision1_672_coreml.pte` | per photo | 224 MB | ~330 ms |
89
+ | `sam3/coreml/sam3_vision2_672_coreml.pte` | per photo | 224 MB | ~330 to 500 ms |
90
+ | `sam3/coreml/sam3_vision3_672_coreml.pte` | per photo | 238 MB | ~330 ms |
91
+ | `sam3/coreml/sam3_head_672_coreml.pte` | per prompt | 105 MB | ~170 to 300 ms |
92
+ | `sam3/coreml/sam3_text_672_coreml.pte` | per prompt | 707 MB | ~11 to 20 ms |
93
+ | `sam3/coreml/sam3_tokenizer.json` | | 4 MB | |
94
+
95
+ - `sam3_vision0..3`: the 32-layer ViT in four 8-layer chunks (the ANE compiler rejects the whole ViT as one program). `vision0` takes `(1,3,672,672)` pixels in [-1,1] (`x/127.5 - 1`, photo stretched to square) and returns `(1,48,48,1024)`; `vision1` and `vision2` map that to the same shape; `vision3` adds the FPN neck and returns 3 maps `(1,256,192,192)`, `(1,256,96,96)`, `(1,256,48,48)`.
96
+ - `sam3_text`: CLIP text encoder, `input_ids` and `attention_mask` `(1,32)` int64 (CLIP tokenizer, padded with 49407) -> `(1,32,256)`.
97
+ - `sam3_head`: `(fpn0, fpn1, fpn2, text_features, attention_mask)` -> masks `(1,200,192,192)` logits, boxes `(1,200,4)` normalized cxcywh, logits `(1,200)`, presence `(1,1)`. Score = `sigmoid(logit) * sigmoid(presence)`. FPN position encodings are baked in.
98
+
99
+ Verified end to end against PyTorch on real photos: identical instances and scores, mask IoU 0.994 to 1.0.
100
+
101
+ These exports are derivatives of the SAM 3 weights and are distributed under Meta's SAM License, a copy of which is in `sam3/LICENSE`. They are not covered by the Apache 2.0 license of the other folders.
102
+
103
+ ```
104
+ @misc{carion2025sam3segmentconcepts,
105
+ title={SAM 3: Segment Anything with Concepts},
106
+ author={Nicolas Carion and others},
107
+ year={2025},
108
+ eprint={2511.16719},
109
+ archivePrefix={arXiv}
110
+ }
111
+ ```