| --- |
| license: other |
| license_name: sam-license-passthrough |
| license_link: https://ai.meta.com/sam3/license/ |
| tags: |
| - segment-anything |
| - sam3 |
| - open-vocabulary-segmentation |
| - knowledge-distillation |
| - model-compression |
| extra_gated_prompt: >- |
| These weights are DERIVED from Meta's SAM3 (SAM License) and are provided for |
| research/evaluation. Redistribution requires SAM License pass-through. Commercial/ |
| production use requires a separate license from General Instinct, Inc. |
| extra_gated_fields: |
| Name: text |
| Affiliation: text |
| Intended use: text |
| --- |
| |
| # InstinctSAM — Compressed SAM3 (commercial-clean components) |
|
|
| Compressed components for **SAM3** open-vocabulary ("concept", text-prompted) instance |
| segmentation. This repo ships **only our own distilled weights** — a compressed **ViT-B vision |
| trunk** and commercial-clean **GIText text towers** — which you graft onto the (separately |
| licensed, Meta-gated) SAM3 heads. No Meta vision/head weights and **no Apple/MobileCLIP |
| (`apple-amlr`) weights** are redistributed here. |
|
|
| Metric: **cgF1 on SA-Co/Gold** = pmF1 × IL_MCC (official SAM3 leaderboard metric). |
| Teacher (SAM3, 840M): cgF1 **0.521** on our 6544-pair SA-1B/Gold eval. |
| |
| ## The compression↔accuracy frontier (6544-pair SA-Co/Gold cgF1) |
| |
| | variant | what's compressed | params | vs teacher | cgF1 | % teacher | commercial | |
| |---|---|---|---|---|---|---| |
| | teacher SAM3 | — | 840M | 1.0× | 0.521 | 100% | — | |
| | **LiteText (GIText-large)** | **text 354M→63M** | 551M | 1.5× | **0.493** | **94.6%** | ✅ clean | |
| | LiteText (GIText-base) | text 354M→44M | 530M | 1.6× | 0.489 | 93.9% | ✅ clean | |
| | **Hiera-L (vision-compressed)** | **vision 454M→150M (SAM2-Hiera-L)** | 537M | 1.6× | **0.431** | **82.7%** | ✅ clean | |
| | vision-compressed (ViT-B, earlier) | vision 454M→107M | 493M | 1.7× | 0.353 | 68% | ✅ clean | |
| | dual-compressed | vision + text | 182M | 4.6× | 0.246 | 47% | ✅ clean | |
| |
| **Key findings.** (1) The text encoder is the "free" thing to compress — distilling it costs |
| ~10% accuracy for a similar param saving, while vision compression costs ~32%. (2) Our |
| **commercial-clean GIText text tower** (a from-scratch CLIP-BPE transformer, our code, no Apple |
| dependency) **matches and beats** the `apple-amlr` MobileCLIP LiteText (0.493 vs 0.469) at 90–95% |
| of teacher, distilled from SAM3's PE-text on a broad open-vocab + RefCOCO prompt set. All variants |
| beat the released EfficientSAM3 (0.133 on this harness) by 1.9–3.7×. |
| |
| ## Files |
| - `gitext_large_v4.pt` — commercial-clean text tower, 63M (GIText-large; 0.493/94.6%). |
| - `gitext_base_v3.pt` — commercial-clean text tower, 44M (GIText-base, 0.489/93.9%). |
| - `hiera_large_concept_trunk.pt` — **compressed vision trunk, SAM2-Hiera-L 150M (concept-distilled, 0.431/82.7% teacher)** — the best vision-compression point; graft into `model.backbone.vision_backbone.trunk`. Preserves occlusion tracking (~teacher). Full pipeline (with the trunk TensorRT-compiled) measures **12.7 FPS on an idle A100** (vision+neck 29 ms / decode+mask 49 ms); the decode+mask head is the dominant floor, so ≥15 FPS at ≥80% needs a lighter decoder. Thor not yet measured — run `scripts/bench_thor.py` on-device. Reproduction recipe + edge-speed detail: `docs/HIERA_BACKBONE.md` in the GitHub repo. |
| - `concept_vitb_trunk_step6000.pt` — earlier ViT-B vision trunk (concept-distilled, 0.353/68%). |
| - `vit_base_stageA.pt` — ViT-B vision trunk (Stage-A feature distill). |
|
|
| ## Usage (sketch) |
| Build SAM3 (your own gated SAM3 checkpoint), swap in the GIText text tower for the recommended |
| "LiteText" variant. Full reproduction, training recipe, and eval harness: |
| **https://github.com/william-Dic/InstinctSAM** (see `docs/MODEL_CARD.md`, `src/train_text.py`, |
| `src/eval_saco_cgf1.py`). |
|
|
| ## ⚠️ Licensing |
| - These weights are **derived from SAM3** → **SAM License** with pass-through (research/eval OK; |
| redistribute under the same terms + include the license). |
| - The GIText text tower **architecture is our own** (standard CLIP-BPE transformer) — **no |
| `apple-amlr` / MobileCLIP dependency**, so it is commercially usable where SAM3's own license permits. |
| - **Commercial/production use** of the combined system requires a separate license from |
| **General Instinct, Inc.** — guanming@general-instinct.com. |
| - SA-1B (distillation data) — Meta research license; hold your own rights. |
|
|
