--- license: cc-by-nc-4.0 library_name: pytorch pipeline_tag: image-segmentation base_model: depth-anything/DA3-LARGE tags: - depth-anything-3 - object-segmentation - 3d-reconstruction - bop - linemod-occluded - ycb-video --- # objeval3d segmentation heads Trained segmentation heads for the paper **Object-level evaluation of feed-forward multi-view reconstruction: from lightweight segmentation to anchored alignment**. Code, the result files of every experiment and the instructions to run them: . ## What a head is A segmentation head for one capture setup: the decoder of Depth Anything 3's own depth head (`depth_anything_3.model.dpt.DPT`) with a 1×1 final convolution of `num_classes` outputs. It reads the same four intermediate feature maps of the frozen backbone of **DA3-LARGE** as the depth head does and outputs one logit per object (and one for the background). 32.65 M parameters, about 7 % of the model; the backbone, the depth head and the camera head are not touched. It is attached to the unmodified upstream model from outside with a forward hook, and runs in full precision. It adds about 10–12 ms to the forward pass that the reconstruction needs anyway. It is trained on frames of the very capture it is used on (the same objects in the same scene setup), with the visible masks that BOP ships as labels, so no manual annotation was done. Within that capture, F@2 % and completeness of the object's reconstruction agree with those from the ground-truth masks to within 2 % on LM-O; on scenes it was not trained on the head does not transfer (see the third row of the table). ## Files | file | trained on | classes | held-out mean IoU | role | |---|---|---|---|---| | `lmo_seg_head.pt` | LM-O, scene 2, 732 frames | 8 objects + background | 0.817 | the LM-O head of all experiments | | `ycbv_b20_seg_head.pt` | YCB-V, scenes 48, 50, 53, 58, 684 frames | 13 objects + background | 0.942 | the YCB-V head of the within-capture experiments | | `ycbv_cross_scene_seg_head.pt` | YCB-V, scenes 49, 52, 54–57, 1,695 frames | 14 objects + background | 0.287 on scenes 48, 50, 53, 58 | trained on other scenes than it is tested on: the test of the transfer to an unfamiliar scene | | `variants/lmo_b20_seg_head.pt` | LM-O, 20 blocks | as above | 0.813 | the split check | | `variants/lmo_b5_seg_head.pt` | LM-O, 5 blocks | | 0.759 | the harder split: the held-out frames form one long segment | | `variants/ycbv_b5_seg_head.pt` | YCB-V, 5 blocks | | 0.811 | the same for YCB-V | | `variants/lmo_seg_head_n100.pt`, `…_n250.pt`, `…_n500.pt` | a random subset of 100, 250, 500 of the 732 frames | | | the study of how many annotated frames are needed | "Held-out mean IoU" is the mean over the objects of the intersection over union on the test frames that were not used for training or for choosing the epoch (`python -m objeval3d.seg.confusion`). The same list with the classes, the checksums, the validation scores and the file each score comes from is [`configs/heads.json`](https://github.com/abramov-de/Objeval3D-release/blob/main/configs/heads.json). The `SHA256SUMS` file in this repository lists the checksums of the files. A checkpoint is a PyTorch dictionary: `seg_head` (the state dict of the head), `num_classes`, `class_names`, `obj_ids` (BOP object ids, in the order of the classes), `dataset`, `epoch`, `val_loss`, `val_miou`, `image_size`, `hw`, `n_train_images`. ## How to use ```bash git clone https://github.com/abramov-de/Objeval3D-release && cd Objeval3D-release pip install -e ".[analysis,gpu]" # plus Depth Anything 3 at the pinned commit, see docs/ENVIRONMENTS.md hf download abramov-de/objeval3d-heads lmo_seg_head.pt --local-dir workspace/ckpt ``` ```python from pathlib import Path from objeval3d.recon.backends import DA3Predictor predictor = DA3Predictor( device="cuda", head_ckpt="workspace/ckpt/lmo_seg_head.pt" ) # DA3-LARGE at the pinned revision + the head frames = sorted(Path("my_frames").glob("*.png")) # 2-6 views of one static scene pred = predictor.inference(image=[str(f) for f in frames], infer_seg=True) labels = pred.seg_logits.argmax(axis=1) # (views, H, W): 0 = background, 1..K = predictor.ckpt["class_names"][1:] ``` The images are resized to 504 px on the long side (the long side of the processed frame). To reconstruct, clean and score an object with the protocol of the paper, see the README of the repository. ## Limitations, and what not to use it for * **Valid only on `depth-anything/DA3-LARGE` at revision `c54c26b16ec04d218e8d584ecf4bce082a9fcc20`.** The head reads the features of that backbone; on any other weights (the successor `DA3-LARGE-1.1` included) its output is meaningless. * **A head is for one capture.** It knows the objects and the scene setup of its training frames. On scenes of the same dataset that it was not trained on, the head `ycbv_cross_scene_seg_head.pt` reaches a mean IoU of 0.287. It is a tool for the evaluation protocol of the paper, where the capture is annotated by the dataset, and not a general object segmenter. * The labels are the visible masks of BOP: an object that is more than 70 % occluded or smaller than 400 px (800 px for YCB-V) was ignored in training, so the head says little about such objects. * Not evaluated, and not intended for: safety-relevant use, people, or any object that is not in the table's class lists. ## Training All heads: 20 epochs, batch size 2, AdamW with learning rate 1e-4 (cosine decay to 1 % of it) and weight decay 1e-4, gradient clipping at 1.0, images at 504 px on the long side, augmentation on, seed 42; loss is cross-entropy (ignored pixels skipped) plus soft Dice over the foreground classes present; the checkpoint is the epoch with the best validation mean IoU. Only the head is trained. A head trained on the 675–732 frames of one capture takes 17–18 minutes on one GPU (the LM-O heads on an NVIDIA GeForce RTX 5090, the YCB-V heads on an NVIDIA RTX PRO 6000 Blackwell); 100 frames take 6 minutes, and the cross-scene YCB-V head (1,695 frames, RTX 5090) 43 minutes. ```bash python -m objeval3d.seg.prepare_data --bop-root workspace/data/bop/lmo --dataset lmo --split test --scenes 2 \ --obj-ids 1,5,6,8,9,10,11,12 --split-by block --n-blocks 20 --min-visib-fract 0.3 --min-px 400 --out-dir workspace/seg_data/lmo python -m objeval3d.seg.train --data-dir workspace/seg_data/lmo --out-ckpt workspace/ckpt/lmo_seg_head.pt \ --epochs 20 --batch-size 2 --lr 1e-4 --image-size 504 --augment ``` The data come from the BOP datasets: LM-O (CC BY-SA 4.0) and YCB-Video (MIT). The training logs of these heads are in the `results/training` folder of the code repository. Training again gives about the same scores, not the same weights (GPU training is not bit-deterministic). ## Licence CC BY-NC 4.0 (non-commercial), because the heads are trained on the features of DA3-LARGE, which is CC BY-NC 4.0. The code that uses them is Apache-2.0. ## Paper The link to the paper will be added here.