| --- |
| license: apache-2.0 |
| tags: |
| - segmentation |
| - efficientsam |
| - aria-digital-twin |
| - reproducibility |
| --- |
| |
| # Checkpoints — Projection-Assisted Segmentation Reproduction |
|
|
| Model weights for reproducing projection-assisted gaze-prompted segmentation on |
| the Aria Digital Twin dataset. |
|
|
| Companion data: [`tianxia2/projseg-adt-seq144-subset`](https://huggingface.co/datasets/tianxia2/projseg-adt-seq144-subset). |
|
|
| ## Files |
|
|
| | File | Size | What | |
| |---|---|---| |
| | `efficient_sam_vitt.pt` | 40 MB | **EfficientSAM ViT-Tiny.** The anchor segmenter, prompted with a single gaze point. Unmodified upstream weights from [yformer/EfficientSAM](https://github.com/yformer/EfficientSAM) (Apache-2.0), redistributed for convenience. | |
| | `refinenetgaze.pth` | 5.9 MB | **RefineNetGaze.** Optional U-Net that refines a reused mask from RGB + prior mask + gaze heatmap (5-channel input, base=16). Trained by us on ADT. | |
|
|
| ## Usage |
|
|
| ```bash |
| huggingface-cli download tianxia2/projseg-checkpoints --local-dir checkpoints |
| ``` |
|
|
| The reproduction artifact expects them at `checkpoints/` (override with |
| `PROJSEG_CKPT_ROOT`). |
|
|
| ## Notes |
|
|
| - `refinenetgaze.pth` is stored as a **plain tensor state dict**, so it loads |
| under PyTorch ≥ 2.6's default `weights_only=True`. Construct the model as |
| `RefineNetGaze(base=16)`. |
| - Refinement is **optional and off by default**. On the reported EfficientSAM + |
| `dense_gt` configuration it *lowers* mIoU (0.3283 → 0.3111), because it was |
| trained against a different anchor-mask distribution than the one ESAM |
| anchors produce. It is shipped for completeness and ablation, not as part of |
| the headline result. |
| - EfficientSAM is class-agnostic. Prompted with one gaze point it segments |
| *some* coherent region around that point, which does not always coincide with |
| the annotated ADT instance — hence modest absolute mIoU on this benchmark. |
|
|
