chrischoy commited on
Commit
4714423
·
verified ·
1 Parent(s): 3e61512

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -9
README.md CHANGED
@@ -20,11 +20,10 @@ tags:
20
  **SpaceFormer** performs **proposal-free, open-vocabulary 3D instance segmentation**.
21
  A Mask2Former-style query decoder (learned queries + rotary position embeddings) runs
22
  on top of the WarpConvNet [`SpaCeFormer`](https://github.com/NVlabs/WarpConvNet) sparse
23
- point backbone. A single forward pass over an RGB point cloud produces a fixed set of
24
  query masks plus a per-query CLIP feature; each mask is labeled by comparing its CLIP
25
  feature against text embeddings of **arbitrary class names** (SigLIP2 text encoder, with
26
- prompt ensembling). The vocabulary is chosen at inference time — it is not baked into the
27
- weights — so the model can be queried with any label set.
28
 
29
  Project page: https://nvlabs.github.io/SpaCeFormer/
30
 
@@ -46,17 +45,17 @@ Project page: https://nvlabs.github.io/SpaCeFormer/
46
  Test-set mAP with the released recipe (**prompt ensembling on, TTA off, default
47
  proposal-free post-processing**):
48
 
49
- | Benchmark | mAP | mAP50 | recall (class-agnostic) |
50
  |---|---:|---:|---:|
51
- | ScanNet200 | **0.1265** | 0.210 | 0.756 |
52
- | ScanNet++ | 0.2217 | — | — |
53
- | Replica | 0.2644 | — | — |
54
 
55
  ## How to use
56
 
57
  The model lives in WarpConvNet as `warpconvnet.models.spaceformer` (the backbone needs
58
  WarpConvNet's compiled CUDA extension — install a pre-built wheel or build from source).
59
- It returns **raw** predictions; open-vocab labeling + mask post-processing live in the
60
  demo repo / HuggingFace Space, not in WarpConvNet.
61
 
62
  ```python
@@ -86,7 +85,7 @@ e.g. its `inference.py` CLI or the Gradio `app.py`.
86
  point clouds (ScanNet-like) against custom class vocabularies.
87
  - **Open-vocab mAP is semantics-bottlenecked:** rare/fine-grained classes are weaker than
88
  head classes; class-agnostic mask recall is higher than the open-vocab mAP.
89
- - **Domain:** trained/evaluated on indoor scenes (ScanNet200 / ScanNet++ / Replica);
90
  outdoor or very different sensor domains are out of distribution.
91
  - **Large scenes:** very large clouds can exceed memory in the eval forward; the
92
  inference code skips such a scene (single-process) rather than crashing.
 
20
  **SpaceFormer** performs **proposal-free, open-vocabulary 3D instance segmentation**.
21
  A Mask2Former-style query decoder (learned queries + rotary position embeddings) runs
22
  on top of the WarpConvNet [`SpaCeFormer`](https://github.com/NVlabs/WarpConvNet) sparse
23
+ convnet transformer backbone. A single forward pass over an RGB point cloud produces a fixed set of
24
  query masks plus a per-query CLIP feature; each mask is labeled by comparing its CLIP
25
  feature against text embeddings of **arbitrary class names** (SigLIP2 text encoder, with
26
+ prompt ensembling).
 
27
 
28
  Project page: https://nvlabs.github.io/SpaCeFormer/
29
 
 
45
  Test-set mAP with the released recipe (**prompt ensembling on, TTA off, default
46
  proposal-free post-processing**):
47
 
48
+ | Benchmark | mAP |
49
  |---|---:|---:|---:|
50
+ | ScanNet200 | **0.1265** |
51
+ | ScanNet++ | 0.2217 |
52
+ | Replica | 0.2644 |
53
 
54
  ## How to use
55
 
56
  The model lives in WarpConvNet as `warpconvnet.models.spaceformer` (the backbone needs
57
  WarpConvNet's compiled CUDA extension — install a pre-built wheel or build from source).
58
+ It returns **raw** predictions; open-vocab CLIP feature + mask post-processing live in the
59
  demo repo / HuggingFace Space, not in WarpConvNet.
60
 
61
  ```python
 
85
  point clouds (ScanNet-like) against custom class vocabularies.
86
  - **Open-vocab mAP is semantics-bottlenecked:** rare/fine-grained classes are weaker than
87
  head classes; class-agnostic mask recall is higher than the open-vocab mAP.
88
+ - **Domain:** trained on indoor scenes (ScanNet200 / ScanNet++ / ARKit);
89
  outdoor or very different sensor domains are out of distribution.
90
  - **Large scenes:** very large clouds can exceed memory in the eval forward; the
91
  inference code skips such a scene (single-process) rather than crashing.