--- license: other license_name: sam-license license_link: LICENSE library_name: onnx pipeline_tag: mask-generation base_model: facebook/sam3 tags: - sam3 - segment-anything - text-prompted-segmentation - onnx - onnxruntime-web - webgpu --- # SAM 3 (text prompts) — ONNX for WebGPU ONNX exports of [SAM 3](https://github.com/facebookresearch/sam3) (Meta, 2025) for in-browser inference with [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/) on WebGPU. They cover SAM 3's **text-prompted** path, "promptable concept segmentation": a short phrase in, a mask for every instance of it out, plus a semantic mask. Used by [Card Anvil](https://github.com/Card-Anvil)'s Describe command. **Licence: the [SAM License](LICENSE), not an open-source licence.** These files are a format conversion of Meta's `facebook/sam3` weights, distributed under the SAM License as it requires. By downloading or using them you accept its terms, including its restrictions on use. The full text is in [`LICENSE`](LICENSE). The model is split into three graphs, each at a fixed shape: a 1008 × 1008 image, a 32-token phrase, batch 1. Two precisions share one repository. | Variant | Files | Download | For | | ------------ | -------------------------------------------------------------- | -------- | --------------------------------------- | | **Standard** | `vision_fp16`, `text_fp16`, `detector_fp16` | 1.68 GB | GPUs with plenty of video memory | | **Compact** | `vision_q8f16`, `text_q8f16`, `detector_fp16` | 0.94 GB | Integrated GPUs, less video memory | ## Files | File | Graph | Precision | Size | | -------------------------- | --------------------------------------------- | ------------------------------------------------------------ | ------ | | `onnx/vision_fp16.onnx` | Perception Encoder ViT (32 layers) and neck | fp16 | 915 MB | | `onnx/vision_q8f16.onnx` | The same | MatMul weights 8-bit (`MatMulNBits`), fp16 activations | 477 MB | | `onnx/text_fp16.onnx` | CLIP text encoder (24 layers) and projection | fp16 | 709 MB | | `onnx/text_q8f16.onnx` | The same | MatMul weights 8-bit, fp16 activations | 411 MB | | `onnx/detector_fp16.onnx` | DETR encoder and decoder, scoring, masks | fp16 (both variants) | 54 MB | Every graph takes and returns float32 (int64 for token ids and masks); fp16 is internal. The tokenizer (`tokenizer.json`, `vocab.json`, `merges.txt` and their configs) and `processor_config.json` are copied unchanged from `facebook/sam3`. ## The graphs | Graph | Inputs | Outputs | | ------------ | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **vision** | `pixel_values` [1, 3, 1008, 1008] | `features_288` [1, 256, 288, 288], `features_144` [1, 256, 144, 144], `features_72` [1, 256, 72, 72] | | **text** | `input_ids` [1, 32], `attention_mask` [1, 32] | `text_features` [1, 32, 256] | | **detector** | the three feature levels, `text_features`, `text_mask` [1, 32] (the attention mask) | `pred_logits` [1, 200], `pred_boxes` [1, 200, 4] (x0, y0, x1, y1, 0–1), `presence_logits` [1, 1], `pred_masks` [1, 200, 288, 288], `semantic_seg` [1, 1, 288, 288] | Run **vision** once per image, and **text** then **detector** once per phrase. The feature levels can stay on the GPU between the two sessions (`preferredOutputLocation: "gpu-buffer"`). ## Using it **Image.** Resize to exactly 1008 × 1008 (bilinear; SAM 3 stretches, it does not pad), scale to 0–1, then normalise with mean 0.5 and standard deviation 0.5 (so −1 to 1), channels first. **Phrase.** SAM 3's CLIP tokenizer: lower-case byte-pair encoding, `<|startoftext|>` (49406) first, `<|endoftext|>` (49407) last and as padding, to 32 tokens. `attention_mask` is 1 for real tokens. One phrase per run: `"dragon, sword"` is one phrase to the model, so split on commas and run each. **Results.** - A detection's score is `sigmoid(pred_logits[q]) × sigmoid(presence_logits)`. Keep those above 0.5, the reference threshold. - Its mask is `pred_masks[q] > 0` (logits) at 288 × 288, over the whole stretched image. Resize it to the original image's size. - `semantic_seg > 0` is the phrase's semantic mask. It covers stuff such as sky better than the instances do. ## How it was made `export_sam3.py` (here), with the graph split in `sam3_graphs.py`: 1. **Load** `Sam3Model` from `facebook/sam3` at [`3c879f39`](https://huggingface.co/facebook/sam3/tree/3c879f39826c281e95690f02c7821c4de09afae7) (`model.safetensors` SHA-256 `6d06f0a5f84e435071fe6603e61d0b4cc7b40e0d39d487cfd4d67d8cc11cc14a`), with eager attention. Versions: Transformers 5.20.2, PyTorch 2.14.1, on the CPU. 2. **Split it** as `Sam3Model.forward` runs a text prompt with no box prompts. The 72² level's position encoding depends only on the fixed input size, so the detector holds it as a constant. 3. **Export** each graph with `torch.onnx.export` (the dynamo exporter), opset 18, fp32. 4. **Convert to fp16** with `onnxconverter-common` (`keep_io_types`). The converter retypes a `Cast` from bool (the text mask) without changing its target type, so `fix_casts` points it at fp16; without that, the detector doesn't load. 5. **Quantise** the vision and text graphs' MatMul weights to 8 bits with ONNX Runtime's `MatMulNBitsQuantizer` (symmetric, block size 128). The detector stays fp16. **Why weight-only 8-bit.** The usual dynamic "int8" export runs `MatMulInteger` and `DynamicQuantizeLinear`, which ONNX Runtime Web's WebGPU backend doesn't implement, so those nodes would fall back to the CPU. `MatMulNBits` keeps the weights at 8 bits and runs on WebGPU. ## How it was checked Against PyTorch fp32 (`reference.py`, then `verify_onnx.py`), on six images and ten phrases: Hugging Face's SAM car example, and Magic: The Gathering art crops. The art was used locally only and isn't included. Each check runs the whole chain (vision, text, detector) from the same inputs. It compares which detections pass 0.5, and the IoU, at 288 × 288, of the kept detections' union and of the semantic mask. | Run | Same detections | Worst union IoU | Worst semantic IoU | | ------------------------------------------- | --------------- | --------------- | ------------------ | | The split graphs in PyTorch | 10 / 10 | max \|Δ\| 2e-6 | | | fp32 ONNX, ONNX Runtime 1.31 CPU | 10 / 10 | 1.000 | 1.000 | | Standard, ONNX Runtime 1.31 CPU | 10 / 10 | 0.999 | 0.999 | | Compact, ONNX Runtime 1.31 CPU | 10 / 10 | 0.993 | 0.997 | | Standard, ONNX Runtime Web 1.31 WebGPU | 10 / 10 | 0.975 | 0.978 | | Compact, ONNX Runtime Web 1.31 WebGPU | 10 / 10 | 0.975 | 0.974 | **Speed on WebGPU,** warm, in Microsoft Edge on an NVIDIA RTX 4070 Ti SUPER with ONNX Runtime Web 1.31 (`ort.webgpu`, single-threaded WebAssembly): | Variant | Vision (once per image) | Text (per phrase) | Detector (per phrase) | | -------- | ----------------------- | ----------------- | --------------------- | | Standard | about 0.8 s | about 20 ms | about 0.25–0.33 s | | Compact | about 2.2 s | about 15 ms | about 0.23–0.33 s | Every node that does arithmetic runs on WebGPU in both variants. ONNX Runtime keeps only shape-only nodes on the CPU, by design. **Compact is smaller, not faster.** On a fast discrete GPU, expanding 8-bit weights costs more than it saves, so Standard is quicker there. Compact is for machines short of video memory, where it hasn't been measured yet. ## Licence The [SAM License](LICENSE) (Meta, last updated November 19, 2025). These ONNX files are a derivative of the SAM Materials and are distributed under its terms. Section 1.b.i requires that redistribution carry the same terms and a copy of the licence. The scripts here (`export_sam3.py`, `sam3_graphs.py`, `reference.py`, `verify_onnx.py`) are Card Anvil's own. They run Hugging Face Transformers' SAM 3 implementation (Apache-2.0) and copy none of it. ## Citation ```bibtex @misc{carion2025sam3segmentconcepts, title={SAM 3: Segment Anything with Concepts}, author={Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer}, year={2025}, eprint={2511.16719}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2511.16719}, } ```