CardAnvilModels/sam3-ONNX

SAM 3 (text prompts) — ONNX for WebGPU

ONNX exports of SAM 3 (Meta, 2025) for in-browser inference with ONNX Runtime Web on WebGPU. They cover SAM 3's text-prompted path, "promptable concept segmentation": a short phrase in, a mask for every instance of it out, plus a semantic mask. Used by Card Anvil's Describe command.

Licence: the SAM License, not an open-source licence. These files are a format conversion of Meta's facebook/sam3 weights, distributed under the SAM License as it requires. By downloading or using them you accept its terms, including its restrictions on use. The full text is in LICENSE.

The model is split into three graphs, each at a fixed shape: a 1008 × 1008 image, a 32-token phrase, batch 1. Two precisions share one repository.

Variant	Files	Download	For
Standard	vision_fp16, text_fp16, detector_fp16	1.68 GB	GPUs with plenty of video memory
Compact	vision_q8f16, text_q8f16, detector_fp16	0.94 GB	Integrated GPUs, less video memory

Files

File	Graph	Precision	Size
onnx/vision_fp16.onnx	Perception Encoder ViT (32 layers) and neck	fp16	915 MB
onnx/vision_q8f16.onnx	The same	MatMul weights 8-bit (MatMulNBits), fp16 activations	477 MB
onnx/text_fp16.onnx	CLIP text encoder (24 layers) and projection	fp16	709 MB
onnx/text_q8f16.onnx	The same	MatMul weights 8-bit, fp16 activations	411 MB
onnx/detector_fp16.onnx	DETR encoder and decoder, scoring, masks	fp16 (both variants)	54 MB

Every graph takes and returns float32 (int64 for token ids and masks); fp16 is internal.

The tokenizer (tokenizer.json, vocab.json, merges.txt and their configs) and processor_config.json are copied unchanged from facebook/sam3.

The graphs

Graph	Inputs	Outputs
vision	pixel_values [1, 3, 1008, 1008]	features_288 [1, 256, 288, 288], features_144 [1, 256, 144, 144], features_72 [1, 256, 72, 72]
text	input_ids [1, 32], attention_mask [1, 32]	text_features [1, 32, 256]
detector	the three feature levels, text_features, text_mask [1, 32] (the attention mask)	pred_logits [1, 200], pred_boxes [1, 200, 4] (x0, y0, x1, y1, 0–1), presence_logits [1, 1], pred_masks [1, 200, 288, 288], semantic_seg [1, 1, 288, 288]

Run vision once per image, and text then detector once per phrase. The feature levels can stay on the GPU between the two sessions (preferredOutputLocation: "gpu-buffer").

Using it

Image. Resize to exactly 1008 × 1008 (bilinear; SAM 3 stretches, it does not pad), scale to 0–1, then normalise with mean 0.5 and standard deviation 0.5 (so −1 to 1), channels first.

Phrase. SAM 3's CLIP tokenizer: lower-case byte-pair encoding, <|startoftext|> (49406) first, <|endoftext|> (49407) last and as padding, to 32 tokens. attention_mask is 1 for real tokens. One phrase per run: "dragon, sword" is one phrase to the model, so split on commas and run each.

Results.

• A detection's score is sigmoid(pred_logits[q]) × sigmoid(presence_logits). Keep those above 0.5, the reference threshold.
• Its mask is pred_masks[q] > 0 (logits) at 288 × 288, over the whole stretched image. Resize it to the original image's size.
• semantic_seg > 0 is the phrase's semantic mask. It covers stuff such as sky better than the instances do.

How it was made

export_sam3.py (here), with the graph split in sam3_graphs.py:

1. Load Sam3Model from facebook/sam3 at 3c879f39 (model.safetensors SHA-256 6d06f0a5f84e435071fe6603e61d0b4cc7b40e0d39d487cfd4d67d8cc11cc14a), with eager attention. Versions: Transformers 5.20.2, PyTorch 2.14.1, on the CPU.
2. Split it as Sam3Model.forward runs a text prompt with no box prompts. The 72² level's position encoding depends only on the fixed input size, so the detector holds it as a constant.
3. Export each graph with torch.onnx.export (the dynamo exporter), opset 18, fp32.
4. Convert to fp16 with onnxconverter-common (keep_io_types). The converter retypes a Cast from bool (the text mask) without changing its target type, so fix_casts points it at fp16; without that, the detector doesn't load.
5. Quantise the vision and text graphs' MatMul weights to 8 bits with ONNX Runtime's MatMulNBitsQuantizer (symmetric, block size 128). The detector stays fp16.

Why weight-only 8-bit. The usual dynamic "int8" export runs MatMulInteger and DynamicQuantizeLinear, which ONNX Runtime Web's WebGPU backend doesn't implement, so those nodes would fall back to the CPU. MatMulNBits keeps the weights at 8 bits and runs on WebGPU.

How it was checked

Against PyTorch fp32 (reference.py, then verify_onnx.py), on six images and ten phrases: Hugging Face's SAM car example, and Magic: The Gathering art crops. The art was used locally only and isn't included. Each check runs the whole chain (vision, text, detector) from the same inputs. It compares which detections pass 0.5, and the IoU, at 288 × 288, of the kept detections' union and of the semantic mask.

Run	Same detections	Worst union IoU	Worst semantic IoU
The split graphs in PyTorch	10 / 10	max |Δ| 2e-6
fp32 ONNX, ONNX Runtime 1.31 CPU	10 / 10	1.000	1.000
Standard, ONNX Runtime 1.31 CPU	10 / 10	0.999	0.999
Compact, ONNX Runtime 1.31 CPU	10 / 10	0.993	0.997
Standard, ONNX Runtime Web 1.31 WebGPU	10 / 10	0.975	0.978
Compact, ONNX Runtime Web 1.31 WebGPU	10 / 10	0.975	0.974

Speed on WebGPU, warm, in Microsoft Edge on an NVIDIA RTX 4070 Ti SUPER with ONNX Runtime Web 1.31 (ort.webgpu, single-threaded WebAssembly):

Variant	Vision (once per image)	Text (per phrase)	Detector (per phrase)
Standard	about 0.8 s	about 20 ms	about 0.25–0.33 s
Compact	about 2.2 s	about 15 ms	about 0.23–0.33 s

Every node that does arithmetic runs on WebGPU in both variants. ONNX Runtime keeps only shape-only nodes on the CPU, by design. Compact is smaller, not faster. On a fast discrete GPU, expanding 8-bit weights costs more than it saves, so Standard is quicker there. Compact is for machines short of video memory, where it hasn't been measured yet.

Licence

The SAM License (Meta, last updated November 19, 2025). These ONNX files are a derivative of the SAM Materials and are distributed under its terms. Section 1.b.i requires that redistribution carry the same terms and a copy of the licence. The scripts here (export_sam3.py, sam3_graphs.py, reference.py, verify_onnx.py) are Card Anvil's own. They run Hugging Face Transformers' SAM 3 implementation (Apache-2.0) and copy none of it.

Citation

@misc{carion2025sam3segmentconcepts,
      title={SAM 3: Segment Anything with Concepts},
      author={Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},
      year={2025},
      eprint={2511.16719},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.16719},
}