| --- |
| license: other |
| license_name: sam-license |
| license_link: https://huggingface.co/facebook/sam3 |
| base_model: sanskar003/SAM3-Split-Encoder-Decoder |
| pipeline_tag: image-segmentation |
| library_name: onnxruntime |
| tags: |
| - sam3 |
| - segmentation |
| - onnx |
| - tensorrt |
| - text-prompted |
| - open-vocabulary |
| - instance-segmentation |
| - int8 |
| - fp16 |
| extra_gated_prompt: >- |
| This model is derived from facebook/sam3 and is released under the SAM License. By requesting |
| access you agree to the SAM License. |
| --- |
| |
| # SAM3 Split · Text-Free Decoder + Prompt Bank |
|
|
| Text-prompted SAM3 segmentation for apps that use a **fixed list of prompts**. The 353M-parameter |
| text encoder is taken off the GPU: it runs once, offline, on CPU, and its outputs are saved to a |
| small prompt bank. While SAM3 runs, the GPU holds only the vision encoder and a 26M-parameter decoder. |
|
|
| | | | |
| |---|---| |
| | **VRAM while SAM3 runs, 4K frames** | SAM3 base 4,028 MiB → **1,474 MiB** for the whole process; the model itself 3,766 → **1,262 MiB** | |
| | **On the GPU, the whole time** | `sam3_encoder` (455.2M params) + `sam3_decoder_notext` (26.0M params) | |
| | **Run once, offline, on CPU** | `sam3_text_encoder` (353.4M params) → `prompt_bank.npz` (~33 KB per prompt) | |
| | **Accuracy vs the FP32 model** | 1 of 72 detections differs (the previous split: 8 of 72) | |
|
|
| Visual walkthrough of the files and numbers: [SAM3 Model File Map (Space)](https://huggingface.co/spaces/sanskar003/SAM3-Model-File-Map) |
|
|
| ## The journey |
|
|
| ### 1 · SAM3 base: [facebook/sam3](https://huggingface.co/facebook/sam3) |
|
|
| Meta's PyTorch model, measured exactly as we served it in production: transformers `Sam3Model` in |
| FP16, 840.4M parameters. Every prompt sends the image and the text through the whole network. The |
| official post-processing also resizes every predicted mask to the full frame on the GPU, so its |
| memory grows with frame size: **2,428 MiB** on a 1006×679 image, **2,794 MiB** at 1080p and |
| **4,028 MiB** at 4K. |
|
|
| ### 2 · SAM3 Split Encoder–Decoder: [sanskar003/SAM3-Split-Encoder-Decoder](https://huggingface.co/sanskar003/SAM3-Split-Encoder-Decoder) |
|
|
| Built from an INT8/FP16 ONNX export of SAM3, which is one graph: image + token ids in, detections out. Stage 2 cuts that graph in two: |
|
|
| - `sam3_encoder.onnx`: image → 3 feature maps. Runs **once per image**. |
| - `sam3_decoder.onnx`: feature maps + token ids → detections. Runs **once per prompt**. |
|
|
| N prompts on one image now cost 1 encode + N decodes. Production VRAM: **1,872 MiB** (model memory 1,664 MiB). |
|
|
| What stage 2 left on the table: `sam3_decoder` is 93% text encoder (353.4M of its 379.4M |
| parameters), and it re-ran that text encoder on every decode although the prompts never change. |
|
|
| ### 3 · This repo: text-free decoder + prompt bank |
|
|
| The text encoder is cut out of `sam3_decoder.onnx` at the narrowest point of the graph, where only |
| two tensors connect it to the rest: |
|
|
| - `sam3_text_encoder.onnx`: token ids → `text_features` + `text_padding_mask`. Run once per prompt |
| list, on CPU, and saved to `prompt_bank.npz`. |
| - `sam3_decoder_notext.onnx`: feature maps + the two text tensors → detections. Runs once per prompt. |
|
|
| Production VRAM: **1,474 MiB** (model memory 1,262 MiB). |
|
|
| | Stage | Per image with 4 prompts | Params on the GPU | Production VRAM (whole process) | Model memory | |
| |---|---|---|---|---| |
| | 1 · SAM3 base | 4 × full model | 840.4M | 4,028 MiB | 3,766 MiB | |
| | 2 · Split encoder–decoder | 1 encode + 4 decodes, text encoder runs 4× | 834.6M | 1,872 MiB | 1,664 MiB | |
| | **3 · Text-free + prompt bank** | **1 encode + 4 decodes, text encoder runs 0×** | **481.2M** | **1,474 MiB** | **1,262 MiB** | |
|
|
| All three measured on an RTX 4070 (16 GB) with one 3840×2160 image and 4 prompts. Stage 1 runs the |
| official `facebook/sam3` weights in PyTorch 2.6 / transformers 5.7 (FP16); stages 2 and 3 run |
| TensorRT 10.3 engines built `--fp16 --int8`. "Whole process" is what `nvidia-smi` shows for the |
| running app. "Model memory" leaves out the CUDA context: for stage 1 it is what PyTorch's allocator |
| holds, for stages 2 and 3 the engines' weights, working memory and I/O buffers. |
|
|
| Frame size only matters for stage 1: |
|
|
| | Input frame | 1 · SAM3 base | 2 · Split | 3 · Text-free | |
| |---|---|---|---| |
| | 1006×679 | 2,428 MiB | 1,872 MiB | 1,474 MiB | |
| | 1920×1080 | 2,794 MiB | 1,872 MiB | 1,474 MiB | |
| | 3840×2160 | 4,028 MiB | 1,872 MiB | 1,474 MiB | |
|
|
| The split pipelines take a fixed 924×924 input and resize masks on the CPU, so their GPU memory is |
| the same for any frame size. |
|
|
| ## Files |
|
|
| | File | Params | Size | Where it runs | What it is | |
| |---|---|---|---|---| |
| | `sam3_encoder.onnx` | 455.2M | 1.82 GB | GPU, the whole time, once per image | Vision encoder | |
| | `sam3_decoder_notext.onnx` | 26.0M | 104 MB | GPU, the whole time, once per prompt | Fusion encoder, DETR decoder, mask head | |
| | `sam3_text_encoder.onnx` | 353.4M | 1.41 GB | CPU, once per prompt list | Text encoder with its 1024→256 projection | |
| | `sam3_decoder.onnx` | 379.4M | 1.52 GB | Not needed in production | Stage-2 decoder (text encoder inside). Kept for verification and rollback | |
|
|
| All graphs are ONNX opset 17 with INT8 Q/DQ nodes. Build TensorRT engines from them with `--fp16 --int8`. |
|
|
| ## Tensor interfaces |
|
|
| ```text |
| sam3_encoder |
| in image float32 [1, 3, 924, 924] RGB, resized to 924, (x/255 - 0.5) / 0.5 |
| out /Add_179_output_0 float32 [1, 4356, 256] |
| out /fused_229/Conv_output_0 float32 [1, 256, 264, 264] |
| out /fused_232/Conv_output_0 float32 [1, 256, 132, 132] |
| |
| sam3_text_encoder |
| in tokenized_text int64 [1, 32] facebook/sam3 tokenizer, zero-padded |
| out text_features float32 [32, 1, 256] |
| out text_padding_mask bool [1, 32] true where the token id is 0 |
| |
| sam3_decoder_notext |
| in the 3 encoder outputs (same names) |
| in text_features float32 [32, 1, 256] |
| in text_padding_mask bool [1, 32] |
| out pred_logits float32 [1, 200, 1] |
| out pred_boxes float32 [1, 200, 4] normalized cx, cy, w, h |
| out presence_logit_dec float32 [1, 1] |
| out pred_masks float32 [1, 200, 264, 264] |
| ``` |
|
|
| Every model's outputs carry the same names as the next model's inputs, so tensors pass straight through by name. |
|
|
| ## Usage |
|
|
| ### 1. Encode your prompts once (CPU, ONNX Runtime) |
|
|
| ```python |
| import numpy as np |
| import onnxruntime as ort |
| from transformers import AutoTokenizer |
| |
| prompts = ["person", "car", "truck", "bus"] # your fixed prompt list |
| |
| tok = AutoTokenizer.from_pretrained("facebook/sam3") # gated: accept the SAM License first |
| tok.pad_token_id = 0 # the model expects zero padding |
| text_encoder = ort.InferenceSession("sam3_text_encoder.onnx", providers=["CPUExecutionProvider"]) |
| |
| ids = tok(prompts, padding="max_length", max_length=32, truncation=True, |
| return_tensors="np")["input_ids"].astype(np.int64) |
| features, masks = zip(*(text_encoder.run(None, {"tokenized_text": ids[i:i + 1]}) |
| for i in range(len(prompts)))) |
| |
| np.savez("prompt_bank.npz", |
| prompts=np.array(prompts), |
| tokenized_text=ids[:, None], # [N, 1, 32] |
| text_features=np.stack(features), # [N, 32, 1, 256] float32 |
| text_padding_mask=np.stack(masks)) # [N, 1, 32] bool |
| ``` |
|
|
| After this step, the text encoder and the tokenizer are never loaded again. Run it again only when the prompt list changes. It takes seconds and needs no GPU. |
|
|
| ### 2. Segment (ONNX Runtime reference) |
|
|
| ```python |
| import cv2 |
| import numpy as np |
| import onnxruntime as ort |
| |
| bank = np.load("prompt_bank.npz") |
| row = {str(p): i for i, p in enumerate(bank["prompts"])} |
| encoder = ort.InferenceSession("sam3_encoder.onnx", providers=["CPUExecutionProvider"]) |
| decoder = ort.InferenceSession("sam3_decoder_notext.onnx", providers=["CPUExecutionProvider"]) |
| |
| |
| def sigmoid(v): |
| return 1 / (1 + np.exp(-np.clip(v, -50, 50))) |
| |
| |
| rgb = cv2.cvtColor(cv2.imread("image.jpg"), cv2.COLOR_BGR2RGB) |
| h, w = rgb.shape[:2] |
| x = cv2.resize(rgb, (924, 924)).astype(np.float32) / 255.0 |
| x = ((x - 0.5) / 0.5).transpose(2, 0, 1)[None] # [1, 3, 924, 924] |
| |
| names = [o.name for o in encoder.get_outputs()] |
| features = dict(zip(names, encoder.run(None, {"image": x}))) # once per image |
| |
| for prompt in ["person", "car"]: # once per prompt |
| i = row[prompt] |
| logits, boxes, presence, masks = decoder.run( |
| ["pred_logits", "pred_boxes", "presence_logit_dec", "pred_masks"], |
| {**features, |
| "text_features": bank["text_features"][i], |
| "text_padding_mask": bank["text_padding_mask"][i]}) |
| scores = sigmoid(logits[0, :, 0]) * sigmoid(presence[0]) |
| keep = scores > 0.25 |
| cx, cy, bw, bh = boxes[0][keep].T # normalized cx, cy, w, h |
| xyxy = np.stack([cx - bw / 2, cy - bh / 2, cx + bw / 2, cy + bh / 2], 1) * [w, h, w, h] |
| instance_masks = [sigmoid(cv2.resize(m, (w, h))) > 0.5 for m in masks[0][keep]] |
| print(prompt, len(xyxy), "detections") |
| ``` |
|
|
| ### 3. Production: TensorRT engines |
|
|
| ```bash |
| trtexec --onnx=sam3_encoder.onnx --fp16 --int8 --builderOptimizationLevel=5 \ |
| --memPoolSize=workspace:4G --saveEngine=sam3_encoder.engine |
| trtexec --onnx=sam3_decoder_notext.onnx --fp16 --int8 --builderOptimizationLevel=5 \ |
| --memPoolSize=workspace:4G --saveEngine=sam3_decoder_notext.engine |
| ``` |
|
|
| Engines only run on the GPU model they were built on, so build them on the production GPU. The text |
| encoder gets no engine. The engines keep the ONNX tensor names, so the code above maps one to one: |
| run the encoder engine once per image, then the decoder engine once per prompt with that prompt's |
| two tensors from `prompt_bank.npz`. Tested with TensorRT 10.3 on an RTX 4070. |
|
|
| ## Results |
|
|
| RTX 4070 (16 GB), TensorRT 10.3, `--fp16 --int8`. Stage 2 is `sam3_encoder` + `sam3_decoder`; stage 3 is `sam3_encoder` + `sam3_decoder_notext`. |
|
|
| **VRAM per engine, once loaded** |
|
|
| | Engine | Params | Weights | Working memory | I/O buffers | Total | |
| |---|---|---|---|---|---| |
| | `sam3_encoder` | 455.2M | 529 MiB | 140 MiB | 99 MiB | 768 MiB | |
| | `sam3_decoder` (stage 2) | 379.4M | 450 MiB | 303 MiB | 143 MiB | 896 MiB | |
| | `sam3_decoder_notext` (stage 3) | 26.0M | 49 MiB | 302 MiB | 143 MiB | 494 MiB | |
|
|
| The whole stage 2 → 3 saving is decoder weights (450 → 49 MiB). Working memory stays at about 300 MiB |
| because the mask head (200 queries at 264×264) dominates it. |
|
|
| **Accuracy against the FP32 model.** Both TensorRT pipelines were compared with `sam3_decoder.onnx` |
| run in FP32 by ONNX Runtime, on the same image features: 1 test image, 7 prompts, 72 reference |
| detections at score > 0.25. This is a sanity check, not a benchmark. |
|
|
| | | Stage 2 | Stage 3 | |
| |---|---|---| |
| | Detections that differ from FP32 | 8 of 72 | **1 of 72** | |
| | Mean score error on detections | 0.0135 | **0.0036** | |
| | Mean mask IoU vs FP32 | 0.9855 | **0.9898** | |
| | Decode, per prompt | 32.2 ms | 30.5 ms | |
| | 1 encode + 4 decodes, with CPU post-processing | 270 ms | 260 ms | |
|
|
| Stage 3 comes out *more* accurate because the text encoder was the noisiest part of the INT8/FP16 |
| engine: its output there was about 30% off FP32 (mean absolute error 0.24 on features of mean |
| magnitude 0.81). The prompt bank is computed once in FP32, so that error is gone. |
|
|
| ## How the split was made |
|
|
| 1. **Find the text-only part.** Every node reachable from `tokenized_text` but not from the image |
| features is text-only: 353.4M of the decoder's 379.4M parameters. |
| 2. **Find the narrowest cut.** A minimum vertex cut between `tokenized_text` and the image-dependent |
| nodes has exactly two tensors: the projected text features (internal name |
| `/fused_407/Add_output_0`) and the padding mask (`/Not_1_output_0`). They are renamed |
| `text_features` and `text_padding_mask`, so the prompt bank doesn't depend on internal names. |
| 3. **Extract both sides** with `onnx.utils.Extractor`, and check that the new decoder no longer |
| reads `tokenized_text`. |
| 4. **Verify.** In FP32 ONNX Runtime, text encoder → text-free decoder matches `sam3_decoder.onnx` |
| bit for bit (largest difference 0.0 on all 4 outputs, 4 prompts). |
|
|
| ## Limitations |
|
|
| - **Fixed prompt lists only.** A prompt has to be in the prompt bank. To add one, re-run step 1 with the full list; no engine rebuild is needed. |
| - Input is fixed at 924×924, batch 1. |
| - TensorRT engines are tied to the GPU model and TensorRT version they were built with. |
| - The accuracy and latency figures come from one GPU and one test image. |
|
|
| ## License and credits |
|
|
| - **SAM 3** by Meta: [SAM License](https://huggingface.co/facebook/sam3). Read the licence before use. |
| - Stage 2 split: [sanskar003/SAM3-Split-Encoder-Decoder](https://huggingface.co/sanskar003/SAM3-Split-Encoder-Decoder). |
|
|
