Add SpaceFormer weights (spaceformer_512_siglip2_ssccc) + model card

README.md

@@ -0,0 +1,103 @@
+---
+license: apache-2.0
+library_name: warpconvnet
+pipeline_tag: image-segmentation
+tags:
+  - 3d
+  - point-cloud
+  - instance-segmentation
+  - open-vocabulary
+  - scannet
+  - scannet200
+  - scannetpp
+  - replica
+  - spaceformer
+  - warpconvnet
+---
+
+# SpaceFormer — Open-Vocabulary 3D Instance Segmentation
+
+**SpaceFormer** performs **proposal-free, open-vocabulary 3D instance segmentation**.
+A Mask2Former-style query decoder (learned queries + rotary position embeddings) runs
+on top of the WarpConvNet [`SpaCeFormer`](https://github.com/NVlabs/WarpConvNet) sparse
+point backbone. A single forward pass over an RGB point cloud produces a fixed set of
+query masks plus a per-query CLIP feature; each mask is labeled by comparing its CLIP
+feature against text embeddings of **arbitrary class names** (SigLIP2 text encoder, with
+prompt ensembling). The vocabulary is chosen at inference time — it is not baked into the
+weights — so the model can be queried with any label set.
+
+Project page: https://nvlabs.github.io/SpaCeFormer/
+
+## Model details
+
+- **Task:** open-vocabulary 3D instance segmentation on RGB point clouds.
+- **Architecture:** WarpConvNet `SpaCeFormer` backbone (mixed space/curve sparse
+  attention U-Net, `ssccc` encoder) → proposal-free query decoder (hidden dim 512,
+  200 learned queries, RoPE cross/self-attention, 3 decoder iterations) → objectness +
+  per-point mask + per-query CLIP heads. ~85.8M parameters.
+- **CLIP/text embedding:** `google/siglip2-so400m-patch14-224` (1152-d), used only at
+  inference to embed class names; not stored in this checkpoint.
+- **Input:** point coordinates in meters + RGB; voxelized internally at 2 cm.
+- **Naming:** `spaceformer_512_siglip2_ssccc` = hidden dim 512 · SigLIP2 embedding ·
+  `ssccc` encoder attention (space, space, curve, curve, curve).
+
+## Evaluation
+
+Test-set mAP with the released recipe (**prompt ensembling on, TTA off, default
+proposal-free post-processing**):
+
+| Benchmark | mAP | mAP50 | recall (class-agnostic) |
+|---|---:|---:|---:|
+| ScanNet200 | **0.1265** | 0.210 | 0.756 |
+| ScanNet++ | 0.2217 | — | — |
+| Replica | 0.2644 | — | — |
+
+## How to use
+
+The model lives in WarpConvNet as `warpconvnet.models.spaceformer` (the backbone needs
+WarpConvNet's compiled CUDA extension — install a pre-built wheel or build from source).
+It returns **raw** predictions; open-vocab labeling + mask post-processing live in the
+demo repo / HuggingFace Space, not in WarpConvNet.
+
+```python
+import torch
+from warpconvnet.models.spaceformer import build_spaceformer, load_spaceformer_checkpoint
+from huggingface_hub import hf_hub_download
+
+device = torch.device("cuda")
+ckpt = hf_hub_download("chrischoy/SpaCeFormer", "spaceformer_512_siglip2_ssccc.ckpt")
+
+net = build_spaceformer(device=device)
+load_spaceformer_checkpoint(net, ckpt)          # 487 tensors, strict=False
+
+# coord [N,3] float meters; feat [N,3] RGB in [-1,1]; offset [0, N]
+out = net({"coord": coord, "feat": feat, "offset": offset})
+# raw outputs: {"logit":[B,Q,2], "mask":List[[N,Q]], "clip_feat":[B,Q,1152]}
+```
+
+To turn `clip_feat` into open-vocabulary labels (SigLIP2 text + prompt ensembling) and
+clean up masks (NMS/min-points), use the inference pipeline in the demo repo / Space
+(`pipeline.py`, `clip_eval.py`, `text_encoder.py`, `postprocessing.py`, `labels.py`) —
+e.g. its `inference.py` CLI or the Gradio `app.py`.
+
+## Intended use & limitations
+
+- **Intended:** research on open-vocabulary 3D scene understanding; segmenting indoor RGB
+  point clouds (ScanNet-like) against custom class vocabularies.
+- **Open-vocab mAP is semantics-bottlenecked:** rare/fine-grained classes are weaker than
+  head classes; class-agnostic mask recall is higher than the open-vocab mAP.
+- **Domain:** trained/evaluated on indoor scenes (ScanNet200 / ScanNet++ / Replica);
+  outdoor or very different sensor domains are out of distribution.
+- **Large scenes:** very large clouds can exceed memory in the eval forward; the
+  inference code skips such a scene (single-process) rather than crashing.
+
+## Files
+
+- `spaceformer_512_siglip2_ssccc.ckpt` — weights-only Lightning `state_dict` (487
+  tensors; `net.*` decoder/backbone + `caption_loss.logit_scale`). Load via
+  `load_spaceformer_checkpoint` (strips the `net.` prefix, `strict=False`).
+- `spaceformer_512_siglip2_ssccc.ckpt.provenance.json` — architecture, eval numbers, md5.
+
+## License
+
+Apache-2.0.

spaceformer_512_siglip2_ssccc.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ac20817f6bab055483d2374c146c31050ae147c452f74123e43ef29f5ff9eed5
+size 343397654

spaceformer_512_siglip2_ssccc.ckpt.provenance.json

@@ -0,0 +1,21 @@
+{
+  "name": "spaceformer_512_siglip2_ssccc",
+  "model": "SpaCeFormerInstSeg (warpconvnet.models.spaceformer)",
+  "architecture": {
+    "hidden_dim": 512,
+    "clip_embedding": "siglip2-so400m-patch14-224 (1152-d)",
+    "encoder_attn_types": "ssccc",
+    "num_queries": 200,
+    "voxel_size": 0.02
+  },
+  "num_tensors": 487,
+  "num_params_million": 85.8,
+  "weights_only": true,
+  "md5": "0ec6a123b422053ed3988fbf442aeaa1",
+  "eval": {
+    "recipe": "prompt ensembling on, TTA off, default proposal-free post-processing",
+    "scannet200_map": 0.1265,
+    "scannetpp_map": 0.2217,
+    "replica_map": 0.2644
+  }
+}