| --- |
| license: other |
| license_name: mixed-see-card |
| library_name: numpy |
| tags: |
| - image-segmentation |
| - open-vocabulary-segmentation |
| - computational-pathology |
| - training-free |
| - arxiv:2611.05599 |
| pipeline_tag: image-segmentation |
| --- |
| |
| # CAPTOR pretrained tables |
|
|
| Term-by-word tables for **CAPTOR** (Caption-Anchored Patch-Term Occurrence |
| Representation), text-encoder-free grounding by counting caption co-occurrence |
| over a frozen image encoder's visual words. Each table holds a frozen encoder's |
| visual words (k-means centroids of its patch tokens) and one signature per term, |
| counted from a caption corpus. With the encoder, a table scores every term at every |
| patch of an image; no text encoder is involved. |
|
|
| Authors: Elad Yoshai and Natan T. Shaked, Tel Aviv University. |
| Paper: https://arxiv.org/abs/2611.05599 |
| Code: https://github.com/EladYoshai/captor |
|
|
| | file | encoder | K × vocabularies | caption corpus | terms | reference result | |
| |---|---|---|---|---|---| |
| | `coco_dinov2reg_k640.npz` | DINOv2 ViT-B/14 with registers, 196 px | 640 × 4, joint words | COCO train2017, 15,000 images | 78 COCO objects | COCO val2017 reporting half: mIoU 0.5654 | |
| | `pathcap_conch_k80.npz` | CONCH image trunk | 80 × 4 | PathCap, 32,463 figures | 25 tissue terms | BCSS, 62 held-out patients: mIoU 0.4966 | |
|
|
| `MANIFEST.json` gives each file's settings, size and SHA-256. |
|
|
| ## Use |
|
|
| ```bash |
| pip install git+https://github.com/EladYoshai/captor.git |
| ``` |
|
|
| ```python |
| from captor import CAPTOR |
| |
| model = CAPTOR.from_pretrained("coco_dinov2reg_k640") # fetched from this repository |
| seg = model.segment("photo.jpg", ["cat", "dog", "bed"]) |
| ``` |
|
|
| Each file is a plain `.npz` (no pickle): `centroids` (V, K, D), `signatures` |
| (V, K, T), `terms`, corpus moments, optional joint-word arrays, and a JSON `meta` |
| entry with the read-out settings. The format is documented in `captor/io.py`. |
|
|
| ## Licences |
|
|
| The tables contain no images. They are statistics counted from caption corpora over |
| frozen encoders' features, and are to be used under the terms of those sources: |
|
|
| - `coco_dinov2reg_k640`: COCO captions (CC BY 4.0) and DINOv2 (Apache 2.0). |
| - `pathcap_conch_k80`: PathCap captions (CC BY-NC 2.0; Sun et al., PathAsst, 2023) |
| over CONCH features (CONCH: CC BY-NC-ND 4.0, gated). Non-commercial research use only. |
|
|
| The encoders themselves are not included and must be obtained from their own |
| model pages under their own terms. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{yoshai2026captor, |
| title = {{CAPTOR}: Text-Encoder-Free Grounding by Counting Caption Co-occurrence |
| over the Visual Words of a Frozen Encoder}, |
| author = {Yoshai, Elad and Shaked, Natan T.}, |
| journal = {arXiv preprint arXiv:2611.05599}, |
| year = {2026} |
| } |
| ``` |
|
|
