File size: 2,672 Bytes
113ac0f 93129e2 113ac0f 93129e2 113ac0f b7d49dc 113ac0f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | ---
license: mit
tags:
- 3d-vision
- referring-expression-segmentation
- point-cloud
- scanrefer
- multi3drefer
library_name: generic
---
# ReDiCo: Relay-Mediated Visual Updating for 3D Referring Segmentation
Official pretrained checkpoints for **ReDiCo**, a 3D Referring Expression Segmentation (3D-RES / 3D-GRES) method.
[Code](https://github.com/songchuanle-1/ReDiCo)
## Model Description
3D Referring Expression Segmentation (3D-RES) aims to produce point-level masks for objects described by natural language in complex 3D scenes. Existing query-based methods usually keep visual tokens largely fixed during decoding, while dense visual self-attention can refresh them but at $O(N_{sp}^2)$ cost.
**ReDiCo** (**Re**lay attention with **Di**versity and **Co**verage regularization) reformulates visual token updating as *token-relay-token* communication: language-guided relay tokens first aggregate global scene information, then distribute the refined context back to the visual stream, reducing the dominant interaction from $O(N_{sp}^2)$ to $O(N_q N_{sp})$ with $N_q \ll N_{sp}$. Geometry-aware relay attention injects 3D spatial proximity into relay assignments, while coverage and diversity regularization on the stage-1 relay-to-visual assignment jointly prevent region omission and relay redundancy.
With 3D+2D inputs, ReDiCo achieves **62.8%** Acc@0.25 / **51.7%** mIoU on 3D-RES (ScanRefer) and **73.8%** Acc@0.25 / **53.7%** mIoU on 3D-GRES (Multi3DRefer), reducing FLOPs by **33.5%** vs. dense self-attention.
Two checkpoints:
| File | Task | Dataset |
|---|---|---|
| `redico_res.pth` | 3D-RES | ScanRefer |
| `redico_gres.pth` | 3D-GRES | Multi3DRefer |
## Metrics (val)
| Task | Dataset | mIoU | Acc@0.5 | Acc@0.25 |
|---|---|---|---|---|
| 3D-RES | ScanRefer | 51.7 | 57.1 | 62.8 |
| 3D-GRES | Multi3DRefer | 53.7 | 54.1 | 73.8 |
## Usage
```bash
# clone code
git clone https://github.com/songchuanle-1/ReDiCo.git
cd ReDiCo
# download checkpoints
wget https://huggingface.co/chuanle/ReDiCo/resolve/main/redico_res.pth
wget https://huggingface.co/chuanle/ReDiCo/resolve/main/redico_gres.pth
# inference — 3D-RES (ScanRefer)
python tools/test.py configs/redico.yaml --checkpoint redico_res.pth --gpu_ids 0 --out ./output
# inference — 3D-GRES (Multi3DRefer)
python tools/test.py configs/redico_gres.yaml --checkpoint redico_gres.pth --gpu_ids 0 --out ./output
```
Data and pretrained backbone (`sp_unet_backbone.pth`) download links are in the [GitHub repo README](https://github.com/songchuanle-1/ReDiCo) (Baidu Netdisk mirror).
|