Update README.md
Browse files
README.md
CHANGED
|
@@ -20,11 +20,10 @@ tags:
|
|
| 20 |
**SpaceFormer** performs **proposal-free, open-vocabulary 3D instance segmentation**.
|
| 21 |
A Mask2Former-style query decoder (learned queries + rotary position embeddings) runs
|
| 22 |
on top of the WarpConvNet [`SpaCeFormer`](https://github.com/NVlabs/WarpConvNet) sparse
|
| 23 |
-
|
| 24 |
query masks plus a per-query CLIP feature; each mask is labeled by comparing its CLIP
|
| 25 |
feature against text embeddings of **arbitrary class names** (SigLIP2 text encoder, with
|
| 26 |
-
prompt ensembling).
|
| 27 |
-
weights — so the model can be queried with any label set.
|
| 28 |
|
| 29 |
Project page: https://nvlabs.github.io/SpaCeFormer/
|
| 30 |
|
|
@@ -46,17 +45,17 @@ Project page: https://nvlabs.github.io/SpaCeFormer/
|
|
| 46 |
Test-set mAP with the released recipe (**prompt ensembling on, TTA off, default
|
| 47 |
proposal-free post-processing**):
|
| 48 |
|
| 49 |
-
| Benchmark | mAP |
|
| 50 |
|---|---:|---:|---:|
|
| 51 |
-
| ScanNet200 | **0.1265** |
|
| 52 |
-
| ScanNet++ | 0.2217 |
|
| 53 |
-
| Replica | 0.2644 |
|
| 54 |
|
| 55 |
## How to use
|
| 56 |
|
| 57 |
The model lives in WarpConvNet as `warpconvnet.models.spaceformer` (the backbone needs
|
| 58 |
WarpConvNet's compiled CUDA extension — install a pre-built wheel or build from source).
|
| 59 |
-
It returns **raw** predictions; open-vocab
|
| 60 |
demo repo / HuggingFace Space, not in WarpConvNet.
|
| 61 |
|
| 62 |
```python
|
|
@@ -86,7 +85,7 @@ e.g. its `inference.py` CLI or the Gradio `app.py`.
|
|
| 86 |
point clouds (ScanNet-like) against custom class vocabularies.
|
| 87 |
- **Open-vocab mAP is semantics-bottlenecked:** rare/fine-grained classes are weaker than
|
| 88 |
head classes; class-agnostic mask recall is higher than the open-vocab mAP.
|
| 89 |
-
- **Domain:** trained
|
| 90 |
outdoor or very different sensor domains are out of distribution.
|
| 91 |
- **Large scenes:** very large clouds can exceed memory in the eval forward; the
|
| 92 |
inference code skips such a scene (single-process) rather than crashing.
|
|
|
|
| 20 |
**SpaceFormer** performs **proposal-free, open-vocabulary 3D instance segmentation**.
|
| 21 |
A Mask2Former-style query decoder (learned queries + rotary position embeddings) runs
|
| 22 |
on top of the WarpConvNet [`SpaCeFormer`](https://github.com/NVlabs/WarpConvNet) sparse
|
| 23 |
+
convnet transformer backbone. A single forward pass over an RGB point cloud produces a fixed set of
|
| 24 |
query masks plus a per-query CLIP feature; each mask is labeled by comparing its CLIP
|
| 25 |
feature against text embeddings of **arbitrary class names** (SigLIP2 text encoder, with
|
| 26 |
+
prompt ensembling).
|
|
|
|
| 27 |
|
| 28 |
Project page: https://nvlabs.github.io/SpaCeFormer/
|
| 29 |
|
|
|
|
| 45 |
Test-set mAP with the released recipe (**prompt ensembling on, TTA off, default
|
| 46 |
proposal-free post-processing**):
|
| 47 |
|
| 48 |
+
| Benchmark | mAP |
|
| 49 |
|---|---:|---:|---:|
|
| 50 |
+
| ScanNet200 | **0.1265** |
|
| 51 |
+
| ScanNet++ | 0.2217 |
|
| 52 |
+
| Replica | 0.2644 |
|
| 53 |
|
| 54 |
## How to use
|
| 55 |
|
| 56 |
The model lives in WarpConvNet as `warpconvnet.models.spaceformer` (the backbone needs
|
| 57 |
WarpConvNet's compiled CUDA extension — install a pre-built wheel or build from source).
|
| 58 |
+
It returns **raw** predictions; open-vocab CLIP feature + mask post-processing live in the
|
| 59 |
demo repo / HuggingFace Space, not in WarpConvNet.
|
| 60 |
|
| 61 |
```python
|
|
|
|
| 85 |
point clouds (ScanNet-like) against custom class vocabularies.
|
| 86 |
- **Open-vocab mAP is semantics-bottlenecked:** rare/fine-grained classes are weaker than
|
| 87 |
head classes; class-agnostic mask recall is higher than the open-vocab mAP.
|
| 88 |
+
- **Domain:** trained on indoor scenes (ScanNet200 / ScanNet++ / ARKit);
|
| 89 |
outdoor or very different sensor domains are out of distribution.
|
| 90 |
- **Large scenes:** very large clouds can exceed memory in the eval forward; the
|
| 91 |
inference code skips such a scene (single-process) rather than crashing.
|