You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model is derived from facebook/sam3 and is released under the SAM License. By requesting access you agree to the SAM License.

SAM3 Split · Text-Free Decoder + Prompt Bank

Text-prompted SAM3 segmentation for apps that use a fixed list of prompts. The 353M-parameter text encoder is taken off the GPU: it runs once, offline, on CPU, and its outputs are saved to a small prompt bank. While SAM3 runs, the GPU holds only the vision encoder and a 26M-parameter decoder.


VRAM while SAM3 runs, 4K frames	SAM3 base 4,028 MiB → 1,474 MiB for the whole process; the model itself 3,766 → 1,262 MiB
On the GPU, the whole time	`sam3_encoder` (455.2M params) + `sam3_decoder_notext` (26.0M params)
Run once, offline, on CPU	`sam3_text_encoder` (353.4M params) → `prompt_bank.npz` (~33 KB per prompt)
Accuracy vs the FP32 model	1 of 72 detections differs (the previous split: 8 of 72)

Visual walkthrough of the files and numbers: SAM3 Model File Map (Space)

The journey

1 · SAM3 base: facebook/sam3

Meta's PyTorch model, measured exactly as we served it in production: transformers Sam3Model in FP16, 840.4M parameters. Every prompt sends the image and the text through the whole network. The official post-processing also resizes every predicted mask to the full frame on the GPU, so its memory grows with frame size: 2,428 MiB on a 1006×679 image, 2,794 MiB at 1080p and 4,028 MiB at 4K.

2 · SAM3 Split Encoder–Decoder: sanskar003/SAM3-Split-Encoder-Decoder

Built from an INT8/FP16 ONNX export of SAM3, which is one graph: image + token ids in, detections out. Stage 2 cuts that graph in two:

sam3_encoder.onnx: image → 3 feature maps. Runs once per image.
sam3_decoder.onnx: feature maps + token ids → detections. Runs once per prompt.

N prompts on one image now cost 1 encode + N decodes. Production VRAM: 1,872 MiB (model memory 1,664 MiB).

What stage 2 left on the table: sam3_decoder is 93% text encoder (353.4M of its 379.4M parameters), and it re-ran that text encoder on every decode although the prompts never change.

3 · This repo: text-free decoder + prompt bank

The text encoder is cut out of sam3_decoder.onnx at the narrowest point of the graph, where only two tensors connect it to the rest:

sam3_text_encoder.onnx: token ids → text_features + text_padding_mask. Run once per prompt list, on CPU, and saved to prompt_bank.npz.
sam3_decoder_notext.onnx: feature maps + the two text tensors → detections. Runs once per prompt.

Production VRAM: 1,474 MiB (model memory 1,262 MiB).

Stage	Per image with 4 prompts	Params on the GPU	Production VRAM (whole process)	Model memory
1 · SAM3 base	4 × full model	840.4M	4,028 MiB	3,766 MiB
2 · Split encoder–decoder	1 encode + 4 decodes, text encoder runs 4×	834.6M	1,872 MiB	1,664 MiB
3 · Text-free + prompt bank	1 encode + 4 decodes, text encoder runs 0×	481.2M	1,474 MiB	1,262 MiB

All three measured on an RTX 4070 (16 GB) with one 3840×2160 image and 4 prompts. Stage 1 runs the official facebook/sam3 weights in PyTorch 2.6 / transformers 5.7 (FP16); stages 2 and 3 run TensorRT 10.3 engines built --fp16 --int8. "Whole process" is what nvidia-smi shows for the running app. "Model memory" leaves out the CUDA context: for stage 1 it is what PyTorch's allocator holds, for stages 2 and 3 the engines' weights, working memory and I/O buffers.

Frame size only matters for stage 1:

Input frame	1 · SAM3 base	2 · Split	3 · Text-free
1006×679	2,428 MiB	1,872 MiB	1,474 MiB
1920×1080	2,794 MiB	1,872 MiB	1,474 MiB
3840×2160	4,028 MiB	1,872 MiB	1,474 MiB

The split pipelines take a fixed 924×924 input and resize masks on the CPU, so their GPU memory is the same for any frame size.

Files

File	Params	Size	Where it runs	What it is
`sam3_encoder.onnx`	455.2M	1.82 GB	GPU, the whole time, once per image	Vision encoder
`sam3_decoder_notext.onnx`	26.0M	104 MB	GPU, the whole time, once per prompt	Fusion encoder, DETR decoder, mask head
`sam3_text_encoder.onnx`	353.4M	1.41 GB	CPU, once per prompt list	Text encoder with its 1024→256 projection
`sam3_decoder.onnx`	379.4M	1.52 GB	Not needed in production	Stage-2 decoder (text encoder inside). Kept for verification and rollback

All graphs are ONNX opset 17 with INT8 Q/DQ nodes. Build TensorRT engines from them with --fp16 --int8.

Tensor interfaces

sam3_encoder
  in   image                     float32 [1, 3, 924, 924]   RGB, resized to 924, (x/255 - 0.5) / 0.5
  out  /Add_179_output_0         float32 [1, 4356, 256]
  out  /fused_229/Conv_output_0  float32 [1, 256, 264, 264]
  out  /fused_232/Conv_output_0  float32 [1, 256, 132, 132]

sam3_text_encoder
  in   tokenized_text            int64   [1, 32]            facebook/sam3 tokenizer, zero-padded
  out  text_features             float32 [32, 1, 256]
  out  text_padding_mask         bool    [1, 32]            true where the token id is 0

sam3_decoder_notext
  in   the 3 encoder outputs (same names)
  in   text_features             float32 [32, 1, 256]
  in   text_padding_mask         bool    [1, 32]
  out  pred_logits               float32 [1, 200, 1]
  out  pred_boxes                float32 [1, 200, 4]        normalized cx, cy, w, h
  out  presence_logit_dec        float32 [1, 1]
  out  pred_masks                float32 [1, 200, 264, 264]

Every model's outputs carry the same names as the next model's inputs, so tensors pass straight through by name.

Usage

1. Encode your prompts once (CPU, ONNX Runtime)

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

prompts = ["person", "car", "truck", "bus"]            # your fixed prompt list

tok = AutoTokenizer.from_pretrained("facebook/sam3")   # gated: accept the SAM License first
tok.pad_token_id = 0                                   # the model expects zero padding
text_encoder = ort.InferenceSession("sam3_text_encoder.onnx", providers=["CPUExecutionProvider"])

ids = tok(prompts, padding="max_length", max_length=32, truncation=True,
          return_tensors="np")["input_ids"].astype(np.int64)
features, masks = zip(*(text_encoder.run(None, {"tokenized_text": ids[i:i + 1]})
                        for i in range(len(prompts))))

np.savez("prompt_bank.npz",
         prompts=np.array(prompts),
         tokenized_text=ids[:, None],              # [N, 1, 32]
         text_features=np.stack(features),         # [N, 32, 1, 256] float32
         text_padding_mask=np.stack(masks))        # [N, 1, 32] bool

After this step, the text encoder and the tokenizer are never loaded again. Run it again only when the prompt list changes. It takes seconds and needs no GPU.

2. Segment (ONNX Runtime reference)

import cv2
import numpy as np
import onnxruntime as ort

bank = np.load("prompt_bank.npz")
row = {str(p): i for i, p in enumerate(bank["prompts"])}
encoder = ort.InferenceSession("sam3_encoder.onnx", providers=["CPUExecutionProvider"])
decoder = ort.InferenceSession("sam3_decoder_notext.onnx", providers=["CPUExecutionProvider"])


def sigmoid(v):
    return 1 / (1 + np.exp(-np.clip(v, -50, 50)))


rgb = cv2.cvtColor(cv2.imread("image.jpg"), cv2.COLOR_BGR2RGB)
h, w = rgb.shape[:2]
x = cv2.resize(rgb, (924, 924)).astype(np.float32) / 255.0
x = ((x - 0.5) / 0.5).transpose(2, 0, 1)[None]                   # [1, 3, 924, 924]

names = [o.name for o in encoder.get_outputs()]
features = dict(zip(names, encoder.run(None, {"image": x})))     # once per image

for prompt in ["person", "car"]:                                  # once per prompt
    i = row[prompt]
    logits, boxes, presence, masks = decoder.run(
        ["pred_logits", "pred_boxes", "presence_logit_dec", "pred_masks"],
        {**features,
         "text_features": bank["text_features"][i],
         "text_padding_mask": bank["text_padding_mask"][i]})
    scores = sigmoid(logits[0, :, 0]) * sigmoid(presence[0])
    keep = scores > 0.25
    cx, cy, bw, bh = boxes[0][keep].T                              # normalized cx, cy, w, h
    xyxy = np.stack([cx - bw / 2, cy - bh / 2, cx + bw / 2, cy + bh / 2], 1) * [w, h, w, h]
    instance_masks = [sigmoid(cv2.resize(m, (w, h))) > 0.5 for m in masks[0][keep]]
    print(prompt, len(xyxy), "detections")

3. Production: TensorRT engines

trtexec --onnx=sam3_encoder.onnx        --fp16 --int8 --builderOptimizationLevel=5 \
        --memPoolSize=workspace:4G --saveEngine=sam3_encoder.engine
trtexec --onnx=sam3_decoder_notext.onnx --fp16 --int8 --builderOptimizationLevel=5 \
        --memPoolSize=workspace:4G --saveEngine=sam3_decoder_notext.engine

Engines only run on the GPU model they were built on, so build them on the production GPU. The text encoder gets no engine. The engines keep the ONNX tensor names, so the code above maps one to one: run the encoder engine once per image, then the decoder engine once per prompt with that prompt's two tensors from prompt_bank.npz. Tested with TensorRT 10.3 on an RTX 4070.

Results

RTX 4070 (16 GB), TensorRT 10.3, --fp16 --int8. Stage 2 is sam3_encoder + sam3_decoder; stage 3 is sam3_encoder + sam3_decoder_notext.

VRAM per engine, once loaded

Engine	Params	Weights	Working memory	I/O buffers	Total
`sam3_encoder`	455.2M	529 MiB	140 MiB	99 MiB	768 MiB
`sam3_decoder` (stage 2)	379.4M	450 MiB	303 MiB	143 MiB	896 MiB
`sam3_decoder_notext` (stage 3)	26.0M	49 MiB	302 MiB	143 MiB	494 MiB

The whole stage 2 → 3 saving is decoder weights (450 → 49 MiB). Working memory stays at about 300 MiB because the mask head (200 queries at 264×264) dominates it.

Accuracy against the FP32 model. Both TensorRT pipelines were compared with sam3_decoder.onnx run in FP32 by ONNX Runtime, on the same image features: 1 test image, 7 prompts, 72 reference detections at score > 0.25. This is a sanity check, not a benchmark.

	Stage 2	Stage 3
Detections that differ from FP32	8 of 72	1 of 72
Mean score error on detections	0.0135	0.0036
Mean mask IoU vs FP32	0.9855	0.9898
Decode, per prompt	32.2 ms	30.5 ms
1 encode + 4 decodes, with CPU post-processing	270 ms	260 ms

Stage 3 comes out more accurate because the text encoder was the noisiest part of the INT8/FP16 engine: its output there was about 30% off FP32 (mean absolute error 0.24 on features of mean magnitude 0.81). The prompt bank is computed once in FP32, so that error is gone.

How the split was made

Find the text-only part. Every node reachable from tokenized_text but not from the image features is text-only: 353.4M of the decoder's 379.4M parameters.
Find the narrowest cut. A minimum vertex cut between tokenized_text and the image-dependent nodes has exactly two tensors: the projected text features (internal name /fused_407/Add_output_0) and the padding mask (/Not_1_output_0). They are renamed text_features and text_padding_mask, so the prompt bank doesn't depend on internal names.
Extract both sides with onnx.utils.Extractor, and check that the new decoder no longer reads tokenized_text.
Verify. In FP32 ONNX Runtime, text encoder → text-free decoder matches sam3_decoder.onnx bit for bit (largest difference 0.0 on all 4 outputs, 4 prompts).

Limitations

Fixed prompt lists only. A prompt has to be in the prompt bank. To add one, re-run step 1 with the full list; no engine rebuild is needed.
Input is fixed at 924×924, batch 1.
TensorRT engines are tied to the GPU model and TensorRT version they were built with.
The accuracy and latency figures come from one GPU and one test image.

License and credits

SAM 3 by Meta: SAM License. Read the licence before use.
Stage 2 split: sanskar003/SAM3-Split-Encoder-Decoder.

Downloads last month: -; Downloads are not tracked for this model. How to track

Model tree for sanskar003/SAM3-Split-TextFree

Base model

sanskar003/SAM3-Split-Encoder-Decoder

Quantized

(1)

this model

sanskar003
/

SAM3-Split-TextFree