| --- |
| license: apache-2.0 |
| library_name: pytorch |
| tags: |
| - 3d |
| - point-cloud |
| - scannet |
| - 3d-question-answering |
| - 3d-dense-captioning |
| --- |
| |
| # PerLA: Perceptive 3D Language Assistant (CVPR 2025) |
|
|
| Checkpoints for [PerLA](https://github.com/gfmei/PerLA), a 3D language assistant |
| for question answering and dense captioning in indoor scenes. |
| [Paper](https://arxiv.org/abs/2411.19774) · [Code](https://github.com/gfmei/PerLA) · |
| [Project page](https://gfmei.github.io/PerLA/) |
|
|
| Each checkpoint holds the trained PerLA weights (detector, merger, Q-Former, |
| prompt encoder and projections) and its model settings. The frozen OPT-1.3B |
| language model is loaded separately from `facebook/opt-1.3b`. |
|
|
| | File | Benchmark | CIDEr | BLEU-4 | METEOR | ROUGE-L | |
| | --- | --- | ---: | ---: | ---: | ---: | |
| | `perla_scanqa.pth` | ScanQA val | 77.06 | 13.91 | 15.46 | 37.00 | |
|
|
| The model is trained on ScanQA from the pretrained detector, without generalist |
| pretraining (single run, seed 0; best validation epoch). |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/gfmei/PerLA.git && cd PerLA |
| hf download RyanMei/PerLA perla_scanqa.pth --local-dir ckpts |
| bash scripts/eval.sh scanqa ckpts/perla_scanqa.pth |
| ``` |
|
|
| See the [code repository](https://github.com/gfmei/PerLA) for installation, |
| data preparation and training. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{mei2025PerLA, |
| title = {PerLA: Perceptive 3D Language Assistant}, |
| author = {Guofeng Mei and Wei Lin and Luigi Riz and Yujiao Wu and Fabio Poiesi and Yiming Wang}, |
| booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, |
| year = {2025} |
| } |
| ``` |
|
|
