pramodthe/sam3-construction-drawings

SAM 3 fine-tuned on construction detail drawings

A research fine-tune of Meta's SAM 3 for segmenting building elements in architectural and façade detail drawings from short text prompts. Out of the box, SAM 3 finds almost none of these elements in line drawings. After fine-tuning about 5% of its parameters on 536 labelled drawings, mask AP on held-out drawings rises from 0.002 to 0.617.

Results

Test split: 77 drawings, 383 labelled objects, never seen during training. COCO metrics with the category name as the text prompt.

Metric	Base SAM 3	Fine-tuned
Mask AP	0.002	0.617
Mask AP50	0.002	0.836
Mask AP75	0.002	0.703
Box AP	0.002	0.604
Box AP50	0.002	0.828

Fine-tuned mask AP by class:

Prompt	Mask AP
insulation	0.833
concrete	0.789
double glazed unit	0.653
anchor	0.612
single glazed unit	0.562
laminated glass	0.540
cast-in channel	0.333

Full metrics are in results/test_metrics.json.

What is in this repository

Path	Contents
sam3_drawings_finetuned.safetensors	Fine-tuned parameters only (40.5M, 162 MB). The frozen image and text encoders are loaded from facebook/sam3.
sam3_drawings/	Loading, prediction, COCO evaluation and overlay helpers.
configs/, scripts/	Training configs for Meta's SAM 3 trainer and dataset preparation scripts.
app.py	Streamlit demo comparing base and fine-tuned SAM 3.
notebooks/quickstart.ipynb	Minimal load-and-predict example.

Usage

The base model facebook/sam3 is gated: accept its licence on Hugging Face and log in (huggingface-cli login) before loading. A CUDA GPU is required; inference uses about 4 GB of GPU memory.

pip install torch torchvision  # pick the CUDA build for your system from pytorch.org
pip install -r requirements.txt

from PIL import Image
from sam3_drawings import load_model, make_processor, predict, overlay

model = load_model("hub")                    # base SAM 3 + fine-tuned weights from this repo
processor = make_processor(model, threshold=0.5)

image = Image.open("detail_drawing.png")
detections = predict(processor, image)       # default: the 7 trained prompts
for d in detections:
    print(d["prompt"], round(d["score"], 2), d["box"])

Image.fromarray(overlay(image, detections)[0]).save("overlay.png")

load_model also accepts a local weights path, or "base" for the original SAM 3.

Evaluate on any COCO-format split whose category names are the prompts:

python -m sam3_drawings.inference /path/to/test --weights hub --out metrics.json

Demo app (binds to localhost only):

DATASET_DIR=/path/to/test streamlit run app.py --server.address localhost

Training

Setting	Value
Base model	SAM 3 image model (840.5M parameters)
Trainable	40.5M (4.8%): detector encoder and decoder, vision neck, geometry encoder, scoring head, mask decoder
Frozen	Vision trunk, text encoder, semantic segmentation head
Data	536 train / 153 validation / 77 test drawings, 7 classes
Resolution	1008 px; train and validation images pre-downscaled to a 1008 px short side
Schedule	10 epochs, batch size 1, AdamW, learning rates 0.1× Meta's defaults, bf16 autocast
Hardware	One RTX 5070 Ti (16 GB), about 75 minutes, peak GPU memory about 10 GB

To reproduce:

pip install "sam3[train] @ git+https://github.com/facebookresearch/sam3.git"
# Export the dataset from Roboflow in COCO format, then:
python scripts/prepare_dataset.py /path/to/roboflow_export datasets/drawing_v1
python scripts/shrink_dataset.py datasets/drawing_v1 datasets/drawing_v1_1008
DATASET_ROOT=datasets/drawing_v1_1008 scripts/train.sh drawing_v1_full

Notes for machines with about 16 GB of RAM: the configs use num_train_workers: 0 because DataLoader worker processes grew host memory steadily over an epoch. scripts/memguard.sh stops training before the system runs out of memory. Triton's focal loss needs a C compiler, so the plain PyTorch version is used unless SAM3_TRITON_FOCAL_LOSS=1 is set.

Limitations

• Trained on one small dataset of façade and glazing details; expect lower accuracy on other drawing styles, scanned sheets, or hand drawings.
• Only the 7 trained classes are reliable. Other prompts (for example waterproofing membranes or flashings) were not trained and are detected poorly, if at all.
• cast-in channel is rare in the data and has the lowest accuracy.
• Research use only. Outputs must not be relied on for construction, compliance or safety decisions.

Data

Trained on "drawing" by my-workspace-jbnon on Roboflow Universe, licensed CC BY 4.0. Changes made: the empty supercategory was dropped, category names were rewritten as plain-English prompts, and train and validation images were downscaled. The dataset images are not redistributed here, except for the two example figures in results/figures/.

License and acknowledgement

This model is a derivative of SAM 3 and is distributed under the SAM License. It uses the Segment Anything Model (SAM) 3 by Meta. If you use it, please cite SAM 3:

@misc{carion2025sam3segmentconcepts,
  title         = {SAM 3: Segment Anything with Concepts},
  author        = {Nicolas Carion and Laura Gustafson and Yuan-Ting Hu and Shoubhik Debnath and Ronghang Hu and Didac Suris and Chaitanya Ryali and Kalyan Vasudev Alwala and Haitham Khedr and Andrew Huang and Jie Lei and Tengyu Ma and Baishan Guo and Arpit Kalla and Markus Marks and Joseph Greer and Meng Wang and Peize Sun and Roman Rädle and Triantafyllos Afouras and Effrosyni Mavroudi and Katherine Xu and Tsung-Han Wu and Yu Zhou and Liliane Momeni and Rishi Hazra and Shuangrui Ding and Sagar Vaze and Francois Porcher and Feng Li and Siyuan Li and Aishwarya Kamath and Ho Kei Cheng and Piotr Dollár and Nikhila Ravi and Kate Saenko and Pengchuan Zhang and Christoph Feichtenhofer},
  year          = {2025},
  eprint        = {2511.16719},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2511.16719}
}