README.md

---
license: other
license_name: non-commercial-research-only
license_link: https://huggingface.co/adopd/SAM3.1-segmentation-ADOPD/blob/main/USE_RESTRICTIONS.md
library_name: sam3
pipeline_tag: image-segmentation
base_model: facebook/sam3.1
datasets:
  - adopd/adopd2026
tags:
  - document-ai
  - sam3.1
  - entity-segmentation
---

# SAM3.1 Segmentation ADOPD

[Thinking with Anchors Project](https://sichenzhu.github.io/thinking-with-anchors/) | **ADOPD 2026 Paper:** *Thinking with Anchors: Grounded and Efficient Document Reasoning* | [ADOPD 2024 Paper](https://openreview.net/forum?id=x1ptaXpOYa) | [Dataset](https://huggingface.co/datasets/adopd/adopd2026) | [Code](https://github.com/SichenZhu/ADOPD2026)

## Use Restrictions

> **Non-commercial research use only.** The ADOPD fine-tuned checkpoint weights
> in this repository are provided solely for non-commercial research.
> Commercial use of these checkpoint weights is not permitted. Users must also
> comply with every applicable upstream license and acceptable-use term; see
> [USE_RESTRICTIONS.md](USE_RESTRICTIONS.md).

## Model Overview

- **Model developer:** Thinking with Anchors project contributors
- **Base architecture:** [SAM3](https://github.com/facebookresearch/sam3), initialized from the SAM3.1 checkpoint
- **Task:** text-prompted document entity segmentation
- **Fine-tuning dataset:** [adopd/adopd2026](https://huggingface.co/datasets/adopd/adopd2026)
- **Input:** one RGB document image and the text prompt `entity`
- **Output:** entity boxes, confidence scores, and instance masks

## Description

SAM3.1 Segmentation ADOPD uses the SAM3 image architecture initialized from the
SAM3.1 checkpoint and fine-tuned to segment visual entities in document pages.
The companion inference interface uses the fixed text prompt `entity` and
returns class-agnostic instance predictions.

## Training Data

The checkpoint was fine-tuned on the ADOPD Doc2Mask task. Public supervision is
stored in:

```text
human_annotated_masks[].polygons
```

The released adapter rasterizes all valid polygon components into
full-resolution binary masks and creates native SAM3 training datapoints.

## Checkpoint Format

`model.pt` contains the official `detector.*` image-model state used for
inference and weight initialization. It does not include optimizer, scheduler,
scaler, or trainer state and cannot exactly resume an interrupted training run.

## Quick Start

```bash
git clone https://github.com/SichenZhu/ADOPD2026.git
cd ADOPD2026/release_code

git clone https://github.com/facebookresearch/sam3.git upstream/sam3
git -C upstream/sam3 checkout 5dd401d1c5c1d5c3eedff06d41b77af824517619

python -m pip install -e 'upstream/sam3[train]'
python -m pip install -e model_zoo/common
python -m pip install -e model_zoo/sam3_1

hf download adopd/SAM3.1-segmentation-ADOPD \
  --local-dir checkpoints/sam3.1

adopd-sam31-infer \
  --checkpoint checkpoints/sam3.1/model.pt \
  --image document.jpg \
  --threshold 0.5 \
  --output prediction.json
```

`prediction.json` contains pixel-space boxes, confidence scores, and
uncompressed COCO-style RLE masks.

## Fine-Tuning And Evaluation

Use the data adapter, one-node DDP trainer, and sharded evaluation commands in
[`sam3_1`](https://github.com/SichenZhu/ADOPD2026/tree/main/release_code/model_zoo/sam3_1).

## Limitations

The model predicts a single document-entity class and relies on the fixed
prompt `entity`. Confidence and mask thresholds may require calibration for new
document domains. Performance can vary with page resolution, language, scan
quality, and visual style.

## License

The ADOPD fine-tuned checkpoint weights are subject to the non-commercial,
research-only restriction above. The included SAM License and the licenses of
the upstream SAM3/SAM3.1 source, base weights, and dependencies also apply. Use
is permitted only when all applicable terms are satisfied.

## Citation

Please cite the ADOPD 2026 and ADOPD 2024 papers.

```bibtex
@misc{zhu2026thinkingwithanchors,
  title={Thinking with Anchors: Grounded and Efficient Document Reasoning},
  author={Sichen Zhu and Yuchen Zhu and Wenzhuo Xu and Jason Kuen and Wanrong Zhu and Jing Shi and Xuan Shen and Quanyi Wang and Yiwei Wang and Yujun Cai and Bing Shuai and Qin Zhang and Yongxin Chen and Shilong Liu and Molei Tao and Jiuxiang Gu},
  year={2026}
}
```

```bibtex
@inproceedings{gu2024adopd,
  title={{ADOPD}: A Large-Scale Document Page Decomposition Dataset},
  author={Jiuxiang Gu and Xiangxi Shi and Jason Kuen and Lu Qi and Ruiyi Zhang and Anqi Liu and Ani Nenkova and Tong Sun},
  booktitle={The Twelfth International Conference on Learning Representations},
  year={2024},
  url={https://openreview.net/forum?id=x1ptaXpOYa}
}
```