README.md

metadata

license: other
license_name: sam-license-passthrough
license_link: https://ai.meta.com/sam3/license/
tags:
  - segment-anything
  - sam3
  - open-vocabulary-segmentation
  - knowledge-distillation
  - model-compression
extra_gated_prompt: >-
  These weights are DERIVED from Meta's SAM3 (SAM License) and are provided for
  research/evaluation. Redistribution requires SAM License pass-through.
  Commercial/ production use requires a separate license from General Instinct,
  Inc.
extra_gated_fields:
  Name: text
  Affiliation: text
  Intended use: text

InstinctSAM — Compressed SAM3 (commercial-clean components)

Compressed components for SAM3 open-vocabulary ("concept", text-prompted) instance segmentation. This repo ships only our own distilled weights — a compressed ViT-B vision trunk and commercial-clean GIText text towers — which you graft onto the (separately licensed, Meta-gated) SAM3 heads. No Meta vision/head weights and no Apple/MobileCLIP (apple-amlr) weights are redistributed here.

Metric: cgF1 on SA-Co/Gold = pmF1 × IL_MCC (official SAM3 leaderboard metric). Teacher (SAM3, 840M): cgF1 0.521 on our 6544-pair SA-1B/Gold eval.

The compression↔accuracy frontier (6544-pair SA-Co/Gold cgF1)

variant	what's compressed	params	vs teacher	cgF1	% teacher	commercial
teacher SAM3	—	840M	1.0×	0.521	100%	—
LiteText (GIText-large)	text 354M→63M	551M	1.5×	0.493	94.6%	✅ clean
LiteText (GIText-base)	text 354M→44M	530M	1.6×	0.489	93.9%	✅ clean
Hiera-L (vision-compressed)	vision 454M→150M (SAM2-Hiera-L)	537M	1.6×	0.431	82.7%	✅ clean
vision-compressed (ViT-B, earlier)	vision 454M→107M	493M	1.7×	0.353	68%	✅ clean
dual-compressed	vision + text	182M	4.6×	0.246	47%	✅ clean

Key findings. (1) The text encoder is the "free" thing to compress — distilling it costs ~10% accuracy for a similar param saving, while vision compression costs ~32%. (2) Our commercial-clean GIText text tower (a from-scratch CLIP-BPE transformer, our code, no Apple dependency) matches and beats the apple-amlr MobileCLIP LiteText (0.493 vs 0.469) at 90–95% of teacher, distilled from SAM3's PE-text on a broad open-vocab + RefCOCO prompt set. All variants beat the released EfficientSAM3 (0.133 on this harness) by 1.9–3.7×.

Files

• gitext_large_v4.pt — commercial-clean text tower, 63M (GIText-large; 0.493/94.6%).
• gitext_base_v3.pt — commercial-clean text tower, 44M (GIText-base, 0.489/93.9%).
• hiera_large_concept_trunk.pt — compressed vision trunk, SAM2-Hiera-L 150M (concept-distilled, 0.431/82.7% teacher) — the best vision-compression point; graft into model.backbone.vision_backbone.trunk. Preserves occlusion tracking (~teacher). Full pipeline (with the trunk TensorRT-compiled) measures 12.7 FPS on an idle A100 (vision+neck 29 ms / decode+mask 49 ms); the decode+mask head is the dominant floor, so ≥15 FPS at ≥80% needs a lighter decoder. Thor not yet measured — run scripts/bench_thor.py on-device. Reproduction recipe + edge-speed detail: docs/HIERA_BACKBONE.md in the GitHub repo.
• concept_vitb_trunk_step6000.pt — earlier ViT-B vision trunk (concept-distilled, 0.353/68%).
• vit_base_stageA.pt — ViT-B vision trunk (Stage-A feature distill).

Usage (sketch)

Build SAM3 (your own gated SAM3 checkpoint), swap in the GIText text tower for the recommended "LiteText" variant. Full reproduction, training recipe, and eval harness: https://github.com/william-Dic/InstinctSAM (see docs/MODEL_CARD.md, src/train_text.py, src/eval_saco_cgf1.py).

⚠️ Licensing

• These weights are derived from SAM3 → SAM License with pass-through (research/eval OK; redistribute under the same terms + include the license).
• The GIText text tower architecture is our own (standard CLIP-BPE transformer) — no apple-amlr / MobileCLIP dependency, so it is commercially usable where SAM3's own license permits.
• Commercial/production use of the combined system requires a separate license from General Instinct, Inc. — guanming@general-instinct.com.
• SA-1B (distillation data) — Meta research license; hold your own rights.