Hugging Face's logo

InSAI-Lab
/
Trans4Map

Image Segmentation
PyTorch
vision-transformer
bird-eye-view
semantic-mapping
egocentric-vision
matterport3d
replica
wacv-2023
Model card Files
xet
 Community
Trans4Map
File size: 2,853 Bytes
df5af83
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
---
pipeline_tag: image-segmentation
library_name: pytorch
license: apache-2.0
tags:
- vision-transformer
- bird-eye-view
- semantic-mapping
- egocentric-vision
- matterport3d
- replica
- pytorch
- wacv-2023
---

# Trans4Map Model Zoo

Official model repository for **Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers** (WACV 2023).

- **GitHub Repository:** [InSAI-Lab/Trans4Map](https://github.com/InSAI-Lab/Trans4Map)
- **Paper:** [arXiv:2207.06205](https://arxiv.org/abs/2207.06205)
- **Dataset / Ground Truth:** [InSAI-Lab/Trans4Map (Dataset Hub)](https://huggingface.co/datasets/InSAI-Lab/Trans4Map)
- **Demo Video:** [Trans4Map Video Demo](https://yangkailun.com/videos/trans4map.mp4)

---

## Architecture

Trans4Map proposes an end-to-end one-stage Transformer-based framework for Bird's-Eye-View mapping:
1. **Transformer Encoder:** Efficient Transformer backbone (e.g. SegFormer MiT-B2, MiT-B4) extracts rich contextual features from egocentric image batches.
2. **Bidirectional Allocentric Memory (BAM):** Projects egocentric features into an allocentric memory space.
3. **Map Decoder:** Parses accumulated memory and predicts the complete top-down semantic segmentation map.

---

## Benchmark Results (Matterport3D)

| Method | Backbone | mIoU (%) | Checkpoint |
| :--- | :--- | :---: | :---: |
| ConvNeXt | ConvNeXt-T | 35.91 | - |
| ConvNeXt | ConvNeXt-S | 36.49 | - |
| FAN | FAN-T | 31.07 | - |
| FAN | FAN-S | 34.62 | - |
| Swin | Swin-T | 34.19 | - |
| Swin | Swin-S | 36.80 | - |
| **Trans4Map** | **MiT-B2** | **40.02** | [`Tran4Map_B2_20points.pkl`](https://huggingface.co/InSAI-Lab/Trans4Map/resolve/main/Tran4Map_B2_20points.pkl) |
| **Trans4Map** | **MiT-B4** | **40.88** | [`Tran4Map_B4_20points.pkl`](https://huggingface.co/InSAI-Lab/Trans4Map/resolve/main/Tran4Map_B4_20points.pkl) |

---

## Quick Start & Evaluation

### Download Checkpoints:

```bash
# Download MiT-B2 model
hf download InSAI-Lab/Trans4Map Tran4Map_B2_20points.pkl --local-dir ./checkpoints

# Download MiT-B4 model
hf download InSAI-Lab/Trans4Map Tran4Map_B4_20points.pkl --local-dir ./checkpoints
```

### Python Evaluation Example:

```bash
python build_test_data_feature.py
python test.py
python eval/eval.py
python eval/eval_bfscore.py
```

---

## Citation

```bibtex
@inproceedings{chen2023trans4map,
  title={Trans4Map: Revisiting Holistic Bird's-Eye-View Mapping from Egocentric Images to Allocentric Semantics with Vision Transformers},
  author={Chen, Chang and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Stiefelhagen, Rainer},
  booktitle={2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2023}
}
```

## License

This repository is licensed under the [Apache-2.0 License](https://www.apache.org/licenses/LICENSE-2.0).