Hugging Face's logo

mlx-community
/
sam3.1-bf16

MLX
Safetensors
sam3.1_video
sam3
sam3.1
segmentation
detection
tracking
object-multiplex
Model card Files
xet
 Community
prince-canuma commited on
Commit
1f6152b
·
verified ·
1 Parent(s): 0794c1f

Upload SAM 3.1 bf16 weights converted from facebook/sam3.1

Browse files
Files changed (3) hide show
  1. README.md +150 -0
  2. config.json +894 -0
  3. model.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: mlx
3
+ base_model: facebook/sam3.1
4
+ tags:
5
+ - mlx
6
+ - sam3
7
+ - sam3.1
8
+ - segmentation
9
+ - detection
10
+ - tracking
11
+ - object-multiplex
12
+ ---
13
+
14
+ # sam3.1-bf16
15
+
16
+ [facebook/sam3.1](https://huggingface.co/facebook/sam3.1) converted to MLX (bfloat16, 3.3 GB).
17
+
18
+ Open-vocabulary **object detection**, **instance segmentation**, and **video tracking** with **Object Multiplex** on Apple Silicon (~873M parameters).
19
+
20
+ SAM 3.1 extends SAM 3 with:
21
+ - **MultiplexMaskDecoder**: processes 16 objects simultaneously (2.4-4x faster tracking)
22
+ - **TriViTDetNeck**: 3 parallel FPN heads (detection, interactive, propagation)
23
+ - **DecoupledMemoryAttention**: image cross-attention with RoPE
24
+ - Improved detection accuracy (0.90 vs 0.87 on cats benchmark)
25
+
26
+ ## Quick Start
27
+
28
+ ```bash
29
+ pip install mlx-vlm
30
+ ```
31
+
32
+ ```python
33
+ from PIL import Image
34
+ from mlx_vlm.utils import load_model, get_model_path
35
+ from mlx_vlm.models.sam3.generate import Sam3Predictor
36
+ from mlx_vlm.models.sam3_1.processing_sam3_1 import Sam31Processor
37
+
38
+ model_path = get_model_path("mlx-community/sam3.1-bf16")
39
+ model = load_model(model_path)
40
+ processor = Sam31Processor.from_pretrained(str(model_path))
41
+ predictor = Sam3Predictor(model, processor, score_threshold=0.3)
42
+ ```
43
+
44
+ ## Object Detection
45
+
46
+ ```python
47
+ image = Image.open("photo.jpg")
48
+ result = predictor.predict(image, text_prompt="a dog")
49
+
50
+ for i in range(len(result.scores)):
51
+ x1, y1, x2, y2 = result.boxes[i]
52
+ print(f"[{result.scores[i]:.2f}] box=({x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f})")
53
+ ```
54
+
55
+ ## Instance Segmentation
56
+
57
+ ```python
58
+ result = predictor.predict(image, text_prompt="a person")
59
+
60
+ # result.boxes -> (N, 4) xyxy bounding boxes
61
+ # result.masks -> (N, H, W) binary segmentation masks
62
+ # result.scores -> (N,) confidence scores
63
+
64
+ import numpy as np
65
+ overlay = np.array(image).copy()
66
+ W, H = image.size
67
+ for i in range(len(result.scores)):
68
+ mask = result.masks[i]
69
+ if mask.shape != (H, W):
70
+ mask = np.array(Image.fromarray(mask.astype(np.float32)).resize((W, H)))
71
+ binary = mask > 0
72
+ overlay[binary] = (overlay[binary] * 0.5 + np.array([255, 0, 0]) * 0.5).astype(np.uint8)
73
+ ```
74
+
75
+ ## Multi-Prompt Detection
76
+
77
+ ```python
78
+ from mlx_vlm.models.sam3_1.generate import predict_multi
79
+
80
+ result = predict_multi(predictor, image, ["a cat", "a remote control"])
81
+ for i in range(len(result.scores)):
82
+ x1, y1, x2, y2 = result.boxes[i]
83
+ print(f"[{result.scores[i]:.2f}] {result.labels[i]} box=({x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f})")
84
+ ```
85
+
86
+ ## Box-Guided Detection
87
+
88
+ ```python
89
+ import numpy as np
90
+ boxes = np.array([[100, 50, 400, 350]]) # xyxy pixel coords
91
+ result = predictor.predict(image, text_prompt="a cat", boxes=boxes)
92
+ ```
93
+
94
+ ## CLI
95
+
96
+ ```bash
97
+ # Object detection
98
+ python -m mlx_vlm.models.sam3_1.generate --task detect --image photo.jpg --prompt "a cat" --model mlx-community/sam3.1-bf16
99
+
100
+ # Instance segmentation
101
+ python -m mlx_vlm.models.sam3_1.generate --image photo.jpg --prompt "a cat" --model mlx-community/sam3.1-bf16
102
+
103
+ # Video tracking
104
+ python -m mlx_vlm.models.sam3_1.generate --task track --video input.mp4 --prompt "a car" --model mlx-community/sam3.1-bf16
105
+
106
+ # Real-time webcam (optimized: backbone caching + tracker propagation)
107
+ python -m mlx_vlm.models.sam3_1.generate --task realtime --prompt "a person" --model mlx-community/sam3.1-bf16 --resolution 224
108
+ ```
109
+
110
+ | Flag | Default | Description |
111
+ |------|---------|-------------|
112
+ | `--task` | `segment` | `detect`, `segment`, `track`, `realtime` |
113
+ | `--prompt` | *(required)* | Text prompt(s), supports multiple |
114
+ | `--resolution` | `1008` | Input resolution (224 for faster realtime) |
115
+ | `--detect-every` | `15` | Re-run full detection every N frames |
116
+ | `--backbone-every` | `30` | Re-run ViT backbone every N frames |
117
+
118
+ ## Benchmarks (M3 Max, bf16)
119
+
120
+ ### Detection Accuracy
121
+
122
+ | Prompt | SAM 3 | SAM 3.1 |
123
+ |--------|-------|---------|
124
+ | "a cat" (2 cats) | 0.87, 0.82 | **0.90, 0.86** |
125
+ | "a remote control" | 0.95, 0.94 | 0.94, 0.94 |
126
+
127
+ ### Tracker Multiplex Speed
128
+
129
+ | Objects | SAM 3 | SAM 3.1 | Speedup |
130
+ |---------|-------|---------|---------|
131
+ | 3 | 547ms/frame | 227ms/frame | **2.4x** |
132
+ | 4 | 608ms/frame | 203ms/frame | **3.0x** |
133
+ | 5 | 766ms/frame | 190ms/frame | **4.0x** |
134
+
135
+ ### Optimized Realtime (224px)
136
+
137
+ | Metric | Value |
138
+ |--------|-------|
139
+ | Cached frame | 38ms (26 FPS) |
140
+ | Sustained average | ~40ms (25 FPS) |
141
+ | Baseline (no optimization) | ~212ms (5 FPS) |
142
+ | **Total speedup** | **4.6x** |
143
+
144
+ ## Original Model
145
+
146
+ [facebook/sam3.1](https://huggingface.co/facebook/sam3.1) · [Code](https://github.com/facebookresearch/sam3)
147
+
148
+ ## License
149
+
150
+ The original SAM 3.1 model weights are released by Meta under the [SAM License](https://huggingface.co/facebook/sam3.1/blob/main/LICENSE), a custom permissive license for commercial and research use.
config.json ADDED
@@ -0,0 +1,894 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Sam3VideoModel"
4
+ ],
5
+ "assoc_iou_thresh": 0.1,
6
+ "decrease_trk_keep_alive_for_empty_masklets": false,
7
+ "det_nms_thresh": 0.1,
8
+ "detector_config": {
9
+ "detr_decoder_config": {
10
+ "_name_or_path": "",
11
+ "add_cross_attention": false,
12
+ "architectures": null,
13
+ "bad_words_ids": null,
14
+ "begin_suppress_tokens": null,
15
+ "bos_token_id": null,
16
+ "box_rpb_mode": "log",
17
+ "chunk_size_feed_forward": 0,
18
+ "cross_attention_hidden_size": null,
19
+ "decoder_start_token_id": null,
20
+ "diversity_penalty": 0.0,
21
+ "do_sample": false,
22
+ "dropout": 0.1,
23
+ "dtype": null,
24
+ "early_stopping": false,
25
+ "encoder_no_repeat_ngram_size": 0,
26
+ "eos_token_id": null,
27
+ "exponential_decay_length_penalty": null,
28
+ "finetuning_task": null,
29
+ "forced_bos_token_id": null,
30
+ "forced_eos_token_id": null,
31
+ "hidden_act": "relu",
32
+ "hidden_dropout": 0.0,
33
+ "hidden_size": 256,
34
+ "id2label": {
35
+ "0": "LABEL_0",
36
+ "1": "LABEL_1"
37
+ },
38
+ "initializer_range": 0.02,
39
+ "intermediate_size": 2048,
40
+ "is_decoder": false,
41
+ "is_encoder_decoder": false,
42
+ "label2id": {
43
+ "LABEL_0": 0,
44
+ "LABEL_1": 1
45
+ },
46
+ "layer_norm_eps": 1e-06,
47
+ "length_penalty": 1.0,
48
+ "max_length": 20,
49
+ "min_length": 0,
50
+ "model_type": "sam3_detr_decoder",
51
+ "no_repeat_ngram_size": 0,
52
+ "num_attention_heads": 8,
53
+ "num_beam_groups": 1,
54
+ "num_beams": 1,
55
+ "num_layers": 6,
56
+ "num_queries": 200,
57
+ "num_return_sequences": 1,
58
+ "output_attentions": false,
59
+ "output_hidden_states": false,
60
+ "output_scores": false,
61
+ "pad_token_id": null,
62
+ "prefix": null,
63
+ "problem_type": null,
64
+ "remove_invalid_values": false,
65
+ "repetition_penalty": 1.0,
66
+ "return_dict": true,
67
+ "return_dict_in_generate": false,
68
+ "sep_token_id": null,
69
+ "suppress_tokens": null,
70
+ "task_specific_params": null,
71
+ "temperature": 1.0,
72
+ "tie_encoder_decoder": false,
73
+ "tie_word_embeddings": true,
74
+ "tokenizer_class": null,
75
+ "top_k": 50,
76
+ "top_p": 1.0,
77
+ "typical_p": 1.0,
78
+ "use_presence_token": true
79
+ },
80
+ "detr_encoder_config": {
81
+ "_name_or_path": "",
82
+ "add_cross_attention": false,
83
+ "architectures": null,
84
+ "bad_words_ids": null,
85
+ "begin_suppress_tokens": null,
86
+ "bos_token_id": null,
87
+ "chunk_size_feed_forward": 0,
88
+ "cross_attention_hidden_size": null,
89
+ "decoder_start_token_id": null,
90
+ "diversity_penalty": 0.0,
91
+ "do_sample": false,
92
+ "dropout": 0.1,
93
+ "dtype": null,
94
+ "early_stopping": false,
95
+ "encoder_no_repeat_ngram_size": 0,
96
+ "eos_token_id": null,
97
+ "exponential_decay_length_penalty": null,
98
+ "finetuning_task": null,
99
+ "forced_bos_token_id": null,
100
+ "forced_eos_token_id": null,
101
+ "hidden_act": "relu",
102
+ "hidden_dropout": 0.0,
103
+ "hidden_size": 256,
104
+ "id2label": {
105
+ "0": "LABEL_0",
106
+ "1": "LABEL_1"
107
+ },
108
+ "initializer_range": 0.02,
109
+ "intermediate_size": 2048,
110
+ "is_decoder": false,
111
+ "is_encoder_decoder": false,
112
+ "label2id": {
113
+ "LABEL_0": 0,
114
+ "LABEL_1": 1
115
+ },
116
+ "layer_norm_eps": 1e-06,
117
+ "length_penalty": 1.0,
118
+ "max_length": 20,
119
+ "min_length": 0,
120
+ "model_type": "sam3_detr_encoder",
121
+ "no_repeat_ngram_size": 0,
122
+ "num_attention_heads": 8,
123
+ "num_beam_groups": 1,
124
+ "num_beams": 1,
125
+ "num_layers": 6,
126
+ "num_return_sequences": 1,
127
+ "output_attentions": false,
128
+ "output_hidden_states": false,
129
+ "output_scores": false,
130
+ "pad_token_id": null,
131
+ "prefix": null,
132
+ "problem_type": null,
133
+ "remove_invalid_values": false,
134
+ "repetition_penalty": 1.0,
135
+ "return_dict": true,
136
+ "return_dict_in_generate": false,
137
+ "sep_token_id": null,
138
+ "suppress_tokens": null,
139
+ "task_specific_params": null,
140
+ "temperature": 1.0,
141
+ "tie_encoder_decoder": false,
142
+ "tie_word_embeddings": true,
143
+ "tokenizer_class": null,
144
+ "top_k": 50,
145
+ "top_p": 1.0,
146
+ "typical_p": 1.0
147
+ },
148
+ "geometry_encoder_config": {
149
+ "_name_or_path": "",
150
+ "add_cross_attention": false,
151
+ "architectures": null,
152
+ "bad_words_ids": null,
153
+ "begin_suppress_tokens": null,
154
+ "bos_token_id": null,
155
+ "chunk_size_feed_forward": 0,
156
+ "cross_attention_hidden_size": null,
157
+ "decoder_start_token_id": null,
158
+ "diversity_penalty": 0.0,
159
+ "do_sample": false,
160
+ "dropout": 0.1,
161
+ "dtype": null,
162
+ "early_stopping": false,
163
+ "encoder_no_repeat_ngram_size": 0,
164
+ "eos_token_id": null,
165
+ "exponential_decay_length_penalty": null,
166
+ "finetuning_task": null,
167
+ "forced_bos_token_id": null,
168
+ "forced_eos_token_id": null,
169
+ "hidden_act": "relu",
170
+ "hidden_dropout": 0.0,
171
+ "hidden_size": 256,
172
+ "id2label": {
173
+ "0": "LABEL_0",
174
+ "1": "LABEL_1"
175
+ },
176
+ "initializer_range": 0.02,
177
+ "intermediate_size": 2048,
178
+ "is_decoder": false,
179
+ "is_encoder_decoder": false,
180
+ "label2id": {
181
+ "LABEL_0": 0,
182
+ "LABEL_1": 1
183
+ },
184
+ "layer_norm_eps": 1e-06,
185
+ "length_penalty": 1.0,
186
+ "max_length": 20,
187
+ "min_length": 0,
188
+ "model_type": "sam3_geometry_encoder",
189
+ "no_repeat_ngram_size": 0,
190
+ "num_attention_heads": 8,
191
+ "num_beam_groups": 1,
192
+ "num_beams": 1,
193
+ "num_layers": 3,
194
+ "num_return_sequences": 1,
195
+ "output_attentions": false,
196
+ "output_hidden_states": false,
197
+ "output_scores": false,
198
+ "pad_token_id": null,
199
+ "prefix": null,
200
+ "problem_type": null,
201
+ "remove_invalid_values": false,
202
+ "repetition_penalty": 1.0,
203
+ "return_dict": true,
204
+ "return_dict_in_generate": false,
205
+ "roi_size": 7,
206
+ "sep_token_id": null,
207
+ "suppress_tokens": null,
208
+ "task_specific_params": null,
209
+ "temperature": 1.0,
210
+ "tie_encoder_decoder": false,
211
+ "tie_word_embeddings": true,
212
+ "tokenizer_class": null,
213
+ "top_k": 50,
214
+ "top_p": 1.0,
215
+ "typical_p": 1.0
216
+ },
217
+ "initializer_range": 0.02,
218
+ "mask_decoder_config": {
219
+ "_name_or_path": "",
220
+ "add_cross_attention": false,
221
+ "architectures": null,
222
+ "bad_words_ids": null,
223
+ "begin_suppress_tokens": null,
224
+ "bos_token_id": null,
225
+ "chunk_size_feed_forward": 0,
226
+ "cross_attention_hidden_size": null,
227
+ "decoder_start_token_id": null,
228
+ "diversity_penalty": 0.0,
229
+ "do_sample": false,
230
+ "dropout": 0.0,
231
+ "dtype": null,
232
+ "early_stopping": false,
233
+ "encoder_no_repeat_ngram_size": 0,
234
+ "eos_token_id": null,
235
+ "exponential_decay_length_penalty": null,
236
+ "finetuning_task": null,
237
+ "forced_bos_token_id": null,
238
+ "forced_eos_token_id": null,
239
+ "hidden_size": 256,
240
+ "id2label": {
241
+ "0": "LABEL_0",
242
+ "1": "LABEL_1"
243
+ },
244
+ "initializer_range": 0.02,
245
+ "is_decoder": false,
246
+ "is_encoder_decoder": false,
247
+ "label2id": {
248
+ "LABEL_0": 0,
249
+ "LABEL_1": 1
250
+ },
251
+ "layer_norm_eps": 1e-06,
252
+ "length_penalty": 1.0,
253
+ "max_length": 20,
254
+ "min_length": 0,
255
+ "model_type": "sam3_mask_decoder",
256
+ "no_repeat_ngram_size": 0,
257
+ "num_attention_heads": 8,
258
+ "num_beam_groups": 1,
259
+ "num_beams": 1,
260
+ "num_return_sequences": 1,
261
+ "num_upsampling_stages": 3,
262
+ "output_attentions": false,
263
+ "output_hidden_states": false,
264
+ "output_scores": false,
265
+ "pad_token_id": null,
266
+ "prefix": null,
267
+ "problem_type": null,
268
+ "remove_invalid_values": false,
269
+ "repetition_penalty": 1.0,
270
+ "return_dict": true,
271
+ "return_dict_in_generate": false,
272
+ "sep_token_id": null,
273
+ "suppress_tokens": null,
274
+ "task_specific_params": null,
275
+ "temperature": 1.0,
276
+ "tie_encoder_decoder": false,
277
+ "tie_word_embeddings": true,
278
+ "tokenizer_class": null,
279
+ "top_k": 50,
280
+ "top_p": 1.0,
281
+ "typical_p": 1.0
282
+ },
283
+ "model_type": "sam3",
284
+ "text_config": {
285
+ "_name_or_path": "",
286
+ "add_cross_attention": false,
287
+ "architectures": null,
288
+ "attention_dropout": 0.0,
289
+ "bad_words_ids": null,
290
+ "begin_suppress_tokens": null,
291
+ "bos_token_id": 49406,
292
+ "chunk_size_feed_forward": 0,
293
+ "cross_attention_hidden_size": null,
294
+ "decoder_start_token_id": null,
295
+ "diversity_penalty": 0.0,
296
+ "do_sample": false,
297
+ "dtype": null,
298
+ "early_stopping": false,
299
+ "encoder_no_repeat_ngram_size": 0,
300
+ "eos_token_id": 49407,
301
+ "exponential_decay_length_penalty": null,
302
+ "finetuning_task": null,
303
+ "forced_bos_token_id": null,
304
+ "forced_eos_token_id": null,
305
+ "hidden_act": "gelu",
306
+ "hidden_size": 1024,
307
+ "id2label": {
308
+ "0": "LABEL_0",
309
+ "1": "LABEL_1"
310
+ },
311
+ "initializer_factor": 1.0,
312
+ "initializer_range": 0.02,
313
+ "intermediate_size": 4096,
314
+ "is_decoder": false,
315
+ "is_encoder_decoder": false,
316
+ "label2id": {
317
+ "LABEL_0": 0,
318
+ "LABEL_1": 1
319
+ },
320
+ "layer_norm_eps": 1e-05,
321
+ "length_penalty": 1.0,
322
+ "max_length": 20,
323
+ "max_position_embeddings": 32,
324
+ "min_length": 0,
325
+ "model_type": "clip_text_model",
326
+ "no_repeat_ngram_size": 0,
327
+ "num_attention_heads": 16,
328
+ "num_beam_groups": 1,
329
+ "num_beams": 1,
330
+ "num_hidden_layers": 24,
331
+ "num_return_sequences": 1,
332
+ "output_attentions": false,
333
+ "output_hidden_states": false,
334
+ "output_scores": false,
335
+ "pad_token_id": 1,
336
+ "prefix": null,
337
+ "problem_type": null,
338
+ "projection_dim": 512,
339
+ "remove_invalid_values": false,
340
+ "repetition_penalty": 1.0,
341
+ "return_dict": true,
342
+ "return_dict_in_generate": false,
343
+ "sep_token_id": null,
344
+ "suppress_tokens": null,
345
+ "task_specific_params": null,
346
+ "temperature": 1.0,
347
+ "tie_encoder_decoder": false,
348
+ "tie_word_embeddings": true,
349
+ "tokenizer_class": null,
350
+ "top_k": 50,
351
+ "top_p": 1.0,
352
+ "typical_p": 1.0,
353
+ "vocab_size": 49408
354
+ },
355
+ "vision_config": {
356
+ "_name_or_path": "",
357
+ "add_cross_attention": false,
358
+ "architectures": null,
359
+ "backbone_config": {
360
+ "_name_or_path": "",
361
+ "add_cross_attention": false,
362
+ "architectures": null,
363
+ "attention_dropout": 0.0,
364
+ "bad_words_ids": null,
365
+ "begin_suppress_tokens": null,
366
+ "bos_token_id": null,
367
+ "chunk_size_feed_forward": 0,
368
+ "cross_attention_hidden_size": null,
369
+ "decoder_start_token_id": null,
370
+ "diversity_penalty": 0.0,
371
+ "do_sample": false,
372
+ "dtype": null,
373
+ "early_stopping": false,
374
+ "encoder_no_repeat_ngram_size": 0,
375
+ "eos_token_id": null,
376
+ "exponential_decay_length_penalty": null,
377
+ "finetuning_task": null,
378
+ "forced_bos_token_id": null,
379
+ "forced_eos_token_id": null,
380
+ "global_attn_indexes": [
381
+ 7,
382
+ 15,
383
+ 23,
384
+ 31
385
+ ],
386
+ "hidden_act": "gelu",
387
+ "hidden_dropout": 0.0,
388
+ "hidden_size": 1024,
389
+ "id2label": {
390
+ "0": "LABEL_0",
391
+ "1": "LABEL_1"
392
+ },
393
+ "image_size": 1008,
394
+ "initializer_range": 0.02,
395
+ "intermediate_size": 4736,
396
+ "is_decoder": false,
397
+ "is_encoder_decoder": false,
398
+ "label2id": {
399
+ "LABEL_0": 0,
400
+ "LABEL_1": 1
401
+ },
402
+ "layer_norm_eps": 1e-06,
403
+ "layer_scale_init_value": null,
404
+ "length_penalty": 1.0,
405
+ "max_length": 20,
406
+ "min_length": 0,
407
+ "model_type": "sam3_vit_model",
408
+ "no_repeat_ngram_size": 0,
409
+ "num_attention_heads": 16,
410
+ "num_beam_groups": 1,
411
+ "num_beams": 1,
412
+ "num_channels": 3,
413
+ "num_hidden_layers": 32,
414
+ "num_return_sequences": 1,
415
+ "output_attentions": false,
416
+ "output_hidden_states": false,
417
+ "output_scores": false,
418
+ "pad_token_id": null,
419
+ "patch_size": 14,
420
+ "prefix": null,
421
+ "pretrain_image_size": 336,
422
+ "problem_type": null,
423
+ "qkv_bias": true,
424
+ "remove_invalid_values": false,
425
+ "repetition_penalty": 1.0,
426
+ "return_dict": true,
427
+ "return_dict_in_generate": false,
428
+ "rope_theta": 10000.0,
429
+ "sep_token_id": null,
430
+ "suppress_tokens": null,
431
+ "task_specific_params": null,
432
+ "temperature": 1.0,
433
+ "tie_encoder_decoder": false,
434
+ "tie_word_embeddings": true,
435
+ "tokenizer_class": null,
436
+ "top_k": 50,
437
+ "top_p": 1.0,
438
+ "typical_p": 1.0,
439
+ "window_size": 24
440
+ },
441
+ "backbone_feature_sizes": [
442
+ [
443
+ 288,
444
+ 288
445
+ ],
446
+ [
447
+ 144,
448
+ 144
449
+ ],
450
+ [
451
+ 72,
452
+ 72
453
+ ]
454
+ ],
455
+ "bad_words_ids": null,
456
+ "begin_suppress_tokens": null,
457
+ "bos_token_id": null,
458
+ "chunk_size_feed_forward": 0,
459
+ "cross_attention_hidden_size": null,
460
+ "decoder_start_token_id": null,
461
+ "diversity_penalty": 0.0,
462
+ "do_sample": false,
463
+ "dtype": null,
464
+ "early_stopping": false,
465
+ "encoder_no_repeat_ngram_size": 0,
466
+ "eos_token_id": null,
467
+ "exponential_decay_length_penalty": null,
468
+ "finetuning_task": null,
469
+ "forced_bos_token_id": null,
470
+ "forced_eos_token_id": null,
471
+ "fpn_hidden_size": 256,
472
+ "fpn_kernel_size": 2,
473
+ "fpn_stride": 2,
474
+ "hidden_act": "gelu",
475
+ "id2label": {
476
+ "0": "LABEL_0",
477
+ "1": "LABEL_1"
478
+ },
479
+ "initializer_range": 0.02,
480
+ "is_decoder": false,
481
+ "is_encoder_decoder": false,
482
+ "label2id": {
483
+ "LABEL_0": 0,
484
+ "LABEL_1": 1
485
+ },
486
+ "layer_norm_eps": 1e-06,
487
+ "length_penalty": 1.0,
488
+ "max_length": 20,
489
+ "min_length": 0,
490
+ "model_type": "sam3_vision_model",
491
+ "no_repeat_ngram_size": 0,
492
+ "num_beam_groups": 1,
493
+ "num_beams": 1,
494
+ "num_feature_levels": 3,
495
+ "num_return_sequences": 1,
496
+ "output_attentions": false,
497
+ "output_hidden_states": false,
498
+ "output_scores": false,
499
+ "pad_token_id": null,
500
+ "prefix": null,
501
+ "problem_type": null,
502
+ "remove_invalid_values": false,
503
+ "repetition_penalty": 1.0,
504
+ "return_dict": true,
505
+ "return_dict_in_generate": false,
506
+ "scale_factors": [
507
+ 4.0,
508
+ 2.0,
509
+ 1.0
510
+ ],
511
+ "sep_token_id": null,
512
+ "suppress_tokens": null,
513
+ "task_specific_params": null,
514
+ "temperature": 1.0,
515
+ "tie_encoder_decoder": false,
516
+ "tie_word_embeddings": true,
517
+ "tokenizer_class": null,
518
+ "top_k": 50,
519
+ "top_p": 1.0,
520
+ "typical_p": 1.0
521
+ }
522
+ },
523
+ "dtype": "float32",
524
+ "fill_hole_area": 16,
525
+ "high_conf_thresh": 0.8,
526
+ "high_iou_thresh": 0.8,
527
+ "hotstart_delay": 15,
528
+ "hotstart_dup_thresh": 8,
529
+ "hotstart_unmatch_thresh": 8,
530
+ "init_trk_keep_alive": 30,
531
+ "initializer_range": 0.02,
532
+ "low_res_mask_size": 288,
533
+ "max_num_objects": 10000,
534
+ "max_trk_keep_alive": 30,
535
+ "min_trk_keep_alive": -1,
536
+ "model_type": "sam3.1_video",
537
+ "new_det_thresh": 0.7,
538
+ "recondition_every_nth_frame": 16,
539
+ "recondition_on_trk_masks": false,
540
+ "score_threshold_detection": 0.5,
541
+ "suppress_overlapping_based_on_recent_occlusion_threshold": 0.7,
542
+ "suppress_unmatched_only_within_hotstart": true,
543
+ "tracker_config": {
544
+ "enable_occlusion_spatial_embedding": true,
545
+ "enable_temporal_pos_encoding_for_object_pointers": true,
546
+ "image_size": 1008,
547
+ "initializer_range": 0.02,
548
+ "mask_decoder_config": {
549
+ "_name_or_path": "",
550
+ "add_cross_attention": false,
551
+ "architectures": null,
552
+ "attention_downsample_rate": 2,
553
+ "bad_words_ids": null,
554
+ "begin_suppress_tokens": null,
555
+ "bos_token_id": null,
556
+ "chunk_size_feed_forward": 0,
557
+ "cross_attention_hidden_size": null,
558
+ "decoder_start_token_id": null,
559
+ "diversity_penalty": 0.0,
560
+ "do_sample": false,
561
+ "dtype": null,
562
+ "dynamic_multimask_stability_delta": 0.05,
563
+ "dynamic_multimask_stability_thresh": 0.98,
564
+ "dynamic_multimask_via_stability": true,
565
+ "early_stopping": false,
566
+ "encoder_no_repeat_ngram_size": 0,
567
+ "eos_token_id": null,
568
+ "exponential_decay_length_penalty": null,
569
+ "finetuning_task": null,
570
+ "forced_bos_token_id": null,
571
+ "forced_eos_token_id": null,
572
+ "hidden_act": "gelu",
573
+ "hidden_size": 256,
574
+ "id2label": {
575
+ "0": "LABEL_0",
576
+ "1": "LABEL_1"
577
+ },
578
+ "iou_head_depth": 3,
579
+ "iou_head_hidden_dim": 256,
580
+ "is_decoder": false,
581
+ "is_encoder_decoder": false,
582
+ "label2id": {
583
+ "LABEL_0": 0,
584
+ "LABEL_1": 1
585
+ },
586
+ "length_penalty": 1.0,
587
+ "max_length": 20,
588
+ "min_length": 0,
589
+ "mlp_dim": 2048,
590
+ "model_type": "",
591
+ "no_repeat_ngram_size": 0,
592
+ "num_attention_heads": 8,
593
+ "num_beam_groups": 1,
594
+ "num_beams": 1,
595
+ "num_hidden_layers": 2,
596
+ "num_multimask_outputs": 3,
597
+ "num_return_sequences": 1,
598
+ "output_attentions": false,
599
+ "output_hidden_states": false,
600
+ "output_scores": false,
601
+ "pad_token_id": null,
602
+ "prefix": null,
603
+ "problem_type": null,
604
+ "remove_invalid_values": false,
605
+ "repetition_penalty": 1.0,
606
+ "return_dict": true,
607
+ "return_dict_in_generate": false,
608
+ "sep_token_id": null,
609
+ "suppress_tokens": null,
610
+ "task_specific_params": null,
611
+ "temperature": 1.0,
612
+ "tie_encoder_decoder": false,
613
+ "tie_word_embeddings": true,
614
+ "tokenizer_class": null,
615
+ "top_k": 50,
616
+ "top_p": 1.0,
617
+ "typical_p": 1.0
618
+ },
619
+ "mask_downsampler_embed_dim": 256,
620
+ "mask_downsampler_hidden_act": "gelu",
621
+ "mask_downsampler_kernel_size": 3,
622
+ "mask_downsampler_padding": 1,
623
+ "mask_downsampler_stride": 2,
624
+ "mask_downsampler_total_stride": 16,
625
+ "max_cond_frame_num": 4,
626
+ "max_object_pointers_in_encoder": 16,
627
+ "memory_attention_downsample_rate": 1,
628
+ "memory_attention_dropout": 0.1,
629
+ "memory_attention_feed_forward_hidden_act": "relu",
630
+ "memory_attention_feed_forward_hidden_size": 2048,
631
+ "memory_attention_hidden_size": 256,
632
+ "memory_attention_num_attention_heads": 1,
633
+ "memory_attention_num_layers": 4,
634
+ "memory_attention_rope_dropout": 0.1,
635
+ "memory_attention_rope_feat_sizes": [
636
+ 72,
637
+ 72
638
+ ],
639
+ "memory_attention_rope_theta": 10000,
640
+ "memory_encoder_hidden_size": 256,
641
+ "memory_encoder_output_channels": 64,
642
+ "memory_fuser_embed_dim": 256,
643
+ "memory_fuser_hidden_act": "gelu",
644
+ "memory_fuser_intermediate_dim": 1024,
645
+ "memory_fuser_kernel_size": 7,
646
+ "memory_fuser_layer_scale_init_value": 1e-06,
647
+ "memory_fuser_num_layers": 2,
648
+ "memory_fuser_padding": 3,
649
+ "model_type": "sam3_tracker_video",
650
+ "multimask_max_pt_num": 1,
651
+ "multimask_min_pt_num": 0,
652
+ "multimask_output_for_tracking": true,
653
+ "multimask_output_in_sam": true,
654
+ "num_maskmem": 7,
655
+ "prompt_encoder_config": {
656
+ "_name_or_path": "",
657
+ "add_cross_attention": false,
658
+ "architectures": null,
659
+ "bad_words_ids": null,
660
+ "begin_suppress_tokens": null,
661
+ "bos_token_id": null,
662
+ "chunk_size_feed_forward": 0,
663
+ "cross_attention_hidden_size": null,
664
+ "decoder_start_token_id": null,
665
+ "diversity_penalty": 0.0,
666
+ "do_sample": false,
667
+ "dtype": null,
668
+ "early_stopping": false,
669
+ "encoder_no_repeat_ngram_size": 0,
670
+ "eos_token_id": null,
671
+ "exponential_decay_length_penalty": null,
672
+ "finetuning_task": null,
673
+ "forced_bos_token_id": null,
674
+ "forced_eos_token_id": null,
675
+ "hidden_act": "gelu",
676
+ "hidden_size": 256,
677
+ "id2label": {
678
+ "0": "LABEL_0",
679
+ "1": "LABEL_1"
680
+ },
681
+ "image_size": 1008,
682
+ "is_decoder": false,
683
+ "is_encoder_decoder": false,
684
+ "label2id": {
685
+ "LABEL_0": 0,
686
+ "LABEL_1": 1
687
+ },
688
+ "layer_norm_eps": 1e-06,
689
+ "length_penalty": 1.0,
690
+ "mask_input_channels": 16,
691
+ "max_length": 20,
692
+ "min_length": 0,
693
+ "model_type": "",
694
+ "no_repeat_ngram_size": 0,
695
+ "num_beam_groups": 1,
696
+ "num_beams": 1,
697
+ "num_point_embeddings": 4,
698
+ "num_return_sequences": 1,
699
+ "output_attentions": false,
700
+ "output_hidden_states": false,
701
+ "output_scores": false,
702
+ "pad_token_id": null,
703
+ "patch_size": 14,
704
+ "prefix": null,
705
+ "problem_type": null,
706
+ "remove_invalid_values": false,
707
+ "repetition_penalty": 1.0,
708
+ "return_dict": true,
709
+ "return_dict_in_generate": false,
710
+ "scale": 1,
711
+ "sep_token_id": null,
712
+ "suppress_tokens": null,
713
+ "task_specific_params": null,
714
+ "temperature": 1.0,
715
+ "tie_encoder_decoder": false,
716
+ "tie_word_embeddings": true,
717
+ "tokenizer_class": null,
718
+ "top_k": 50,
719
+ "top_p": 1.0,
720
+ "typical_p": 1.0
721
+ },
722
+ "sigmoid_bias_for_mem_enc": -10.0,
723
+ "sigmoid_scale_for_mem_enc": 20.0,
724
+ "vision_config": {
725
+ "_name_or_path": "",
726
+ "add_cross_attention": false,
727
+ "architectures": null,
728
+ "backbone_config": {
729
+ "_name_or_path": "",
730
+ "add_cross_attention": false,
731
+ "architectures": null,
732
+ "attention_dropout": 0.0,
733
+ "bad_words_ids": null,
734
+ "begin_suppress_tokens": null,
735
+ "bos_token_id": null,
736
+ "chunk_size_feed_forward": 0,
737
+ "cross_attention_hidden_size": null,
738
+ "decoder_start_token_id": null,
739
+ "diversity_penalty": 0.0,
740
+ "do_sample": false,
741
+ "dtype": null,
742
+ "early_stopping": false,
743
+ "encoder_no_repeat_ngram_size": 0,
744
+ "eos_token_id": null,
745
+ "exponential_decay_length_penalty": null,
746
+ "finetuning_task": null,
747
+ "forced_bos_token_id": null,
748
+ "forced_eos_token_id": null,
749
+ "global_attn_indexes": [
750
+ 7,
751
+ 15,
752
+ 23,
753
+ 31
754
+ ],
755
+ "hidden_act": "gelu",
756
+ "hidden_dropout": 0.0,
757
+ "hidden_size": 1024,
758
+ "id2label": {
759
+ "0": "LABEL_0",
760
+ "1": "LABEL_1"
761
+ },
762
+ "image_size": 1008,
763
+ "initializer_range": 0.02,
764
+ "intermediate_size": 4736,
765
+ "is_decoder": false,
766
+ "is_encoder_decoder": false,
767
+ "label2id": {
768
+ "LABEL_0": 0,
769
+ "LABEL_1": 1
770
+ },
771
+ "layer_norm_eps": 1e-06,
772
+ "layer_scale_init_value": null,
773
+ "length_penalty": 1.0,
774
+ "max_length": 20,
775
+ "min_length": 0,
776
+ "model_type": "sam3_vit_model",
777
+ "no_repeat_ngram_size": 0,
778
+ "num_attention_heads": 16,
779
+ "num_beam_groups": 1,
780
+ "num_beams": 1,
781
+ "num_channels": 3,
782
+ "num_hidden_layers": 32,
783
+ "num_return_sequences": 1,
784
+ "output_attentions": false,
785
+ "output_hidden_states": false,
786
+ "output_scores": false,
787
+ "pad_token_id": null,
788
+ "patch_size": 14,
789
+ "prefix": null,
790
+ "pretrain_image_size": 336,
791
+ "problem_type": null,
792
+ "qkv_bias": true,
793
+ "remove_invalid_values": false,
794
+ "repetition_penalty": 1.0,
795
+ "return_dict": true,
796
+ "return_dict_in_generate": false,
797
+ "rope_theta": 10000.0,
798
+ "sep_token_id": null,
799
+ "suppress_tokens": null,
800
+ "task_specific_params": null,
801
+ "temperature": 1.0,
802
+ "tie_encoder_decoder": false,
803
+ "tie_word_embeddings": true,
804
+ "tokenizer_class": null,
805
+ "top_k": 50,
806
+ "top_p": 1.0,
807
+ "typical_p": 1.0,
808
+ "window_size": 24
809
+ },
810
+ "backbone_feature_sizes": [
811
+ [
812
+ 288,
813
+ 288
814
+ ],
815
+ [
816
+ 144,
817
+ 144
818
+ ],
819
+ [
820
+ 72,
821
+ 72
822
+ ]
823
+ ],
824
+ "bad_words_ids": null,
825
+ "begin_suppress_tokens": null,
826
+ "bos_token_id": null,
827
+ "chunk_size_feed_forward": 0,
828
+ "cross_attention_hidden_size": null,
829
+ "decoder_start_token_id": null,
830
+ "diversity_penalty": 0.0,
831
+ "do_sample": false,
832
+ "dtype": null,
833
+ "early_stopping": false,
834
+ "encoder_no_repeat_ngram_size": 0,
835
+ "eos_token_id": null,
836
+ "exponential_decay_length_penalty": null,
837
+ "finetuning_task": null,
838
+ "forced_bos_token_id": null,
839
+ "forced_eos_token_id": null,
840
+ "fpn_hidden_size": 256,
841
+ "fpn_kernel_size": 2,
842
+ "fpn_stride": 2,
843
+ "hidden_act": "gelu",
844
+ "id2label": {
845
+ "0": "LABEL_0",
846
+ "1": "LABEL_1"
847
+ },
848
+ "initializer_range": 0.02,
849
+ "is_decoder": false,
850
+ "is_encoder_decoder": false,
851
+ "label2id": {
852
+ "LABEL_0": 0,
853
+ "LABEL_1": 1
854
+ },
855
+ "layer_norm_eps": 1e-06,
856
+ "length_penalty": 1.0,
857
+ "max_length": 20,
858
+ "min_length": 0,
859
+ "model_type": "sam3_vision_model",
860
+ "no_repeat_ngram_size": 0,
861
+ "num_beam_groups": 1,
862
+ "num_beams": 1,
863
+ "num_feature_levels": 3,
864
+ "num_return_sequences": 1,
865
+ "output_attentions": false,
866
+ "output_hidden_states": false,
867
+ "output_scores": false,
868
+ "pad_token_id": null,
869
+ "prefix": null,
870
+ "problem_type": null,
871
+ "remove_invalid_values": false,
872
+ "repetition_penalty": 1.0,
873
+ "return_dict": true,
874
+ "return_dict_in_generate": false,
875
+ "scale_factors": [
876
+ 4.0,
877
+ 2.0,
878
+ 1.0
879
+ ],
880
+ "sep_token_id": null,
881
+ "suppress_tokens": null,
882
+ "task_specific_params": null,
883
+ "temperature": 1.0,
884
+ "tie_encoder_decoder": false,
885
+ "tie_word_embeddings": true,
886
+ "tokenizer_class": null,
887
+ "top_k": 50,
888
+ "top_p": 1.0,
889
+ "typical_p": 1.0
890
+ }
891
+ },
892
+ "transformers_version": "5.0.0.dev0",
893
+ "trk_assoc_iou_thresh": 0.5
894
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a1b1c19dcc9bdd68438bcd74433fadc90740e73c37a1f386872672d134879c42
3
+ size 3493016552