Add pipeline tag and paper link

README.md

@@ -1,8 +1,9 @@
 ---
-license: cc-by-nc-4.0
-library_name: transformers
 base_model:
-  - Qwen/Qwen2.5-0.5B-Instruct
+- Qwen/Qwen2.5-0.5B-Instruct
+library_name: transformers
+license: cc-by-nc-4.0
+pipeline_tag: image-to-3d
 ---
 
 # SpatialLM-Qwen-0.5B
@@ -26,15 +27,17 @@ base_model:
 </div>
 <div align="center" style="line-height: 1;">
     <a href="https://huggingface.co/manycore-research/SpatialLM-Llama-1B" target="_blank" style="margin: 2px;"><img alt="Hugging Face"
-    src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-SpatialLM%201B-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>
+    src="https://img.shields.io/badge/🤗%20Hugging%20Face-SpatialLM%201B-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>
     <a href="https://huggingface.co/datasets/manycore-research/SpatialLM-Testset" target="_blank" style="margin: 2px;"><img alt="Dataset"
-    src="https://img.shields.io/badge/%F0%9F%A4%97%20Dataset-SpatialLM-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>
+    src="https://img.shields.io/badge/🤗%20Dataset-SpatialLM-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/></a>
 </div>
 
 ## Introduction
 
 SpatialLM is a 3D large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object bounding boxes with their semantic categories. Unlike previous methods that require specialized equipment for data collection, SpatialLM can handle point clouds from diverse sources such as monocular video sequences, RGBD images, and LiDAR sensors. This multimodal architecture effectively bridges the gap between unstructured 3D geometric data and structured 3D representations, offering high-level semantic understanding. It enhances spatial reasoning capabilities for applications in embodied robotics, autonomous navigation, and other complex 3D scene analysis tasks.
 
+This repository contains the model described in [SpatialLM: Training Large Language Models for Structured Indoor Modeling](https://huggingface.co/papers/2506.07491).
+
 <div align="center">
   <video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/3bz_jNRCLD2L9uj11HPnP.mp4" poster="https://cdn-uploads.huggingface.co/production/uploads/63efbb1efc92a63ac81126d0/euo94dNx28qBNe51_oiB1.png"></video>
   <p><i>SpatialLM reconstructs 3D layout from a monocular RGB video with MASt3R-SLAM. Results aligned to video with GT cameras for visualization.</i></p>
@@ -44,10 +47,12 @@ SpatialLM is a 3D large language model designed to process 3D point cloud data a
 
 <div align="center">
 
-|      **Model**      | **Download**                                                                   |
-| :-----------------: | ------------------------------------------------------------------------------ |
-| SpatialLM-Llama-1B  | [🤗 HuggingFace](https://huggingface.co/manycore-research/SpatialLM-Llama-1B)  |
-| SpatialLM-Qwen-0.5B | [🤗 HuggingFace](https://huggingface.co/manycore-research/SpatialLM-Qwen-0.5B) |
+|       **Model**        | **Download**                                                                      |
+| :--------------------: | --------------------------------------------------------------------------------- |
+| SpatialLM1.1-Llama-1B  | [🤗 HuggingFace](https://huggingface.co/manycore-research/SpatialLM1.1-Llama-1B)  |
+| SpatialLM1.1-Qwen-0.5B | [🤗 HuggingFace](https://huggingface.co/manycore-research/SpatialLM1.1-Qwen-0.5B) |
+| SpatialLM1.0-Llama-1B  | [🤗 HuggingFace](https://huggingface.co/manycore-research/SpatialLM-Llama-1B)     |
+| SpatialLM1.0-Qwen-0.5B | [🤗 HuggingFace](https://huggingface.co/manycore-research/SpatialLM-Qwen-0.5B)    |
 
 </div>
 
@@ -91,7 +96,23 @@ huggingface-cli download manycore-research/SpatialLM-Testset pcd/scene0000_00.pl
 Run inference:
 
 ```bash
-python inference.py --point_cloud pcd/scene0000_00.ply --output scene0000_00.txt --model_path manycore-research/SpatialLM-Qwen-0.5B
+python inference.py --point_cloud pcd/scene0000_00.ply --output scene0000_00.txt --model_path manycore-research/SpatialLM1.1-Qwen-0.5B
+```
+
+### Detection with user-specified categories
+
+SpatialLM1.1 supports object detection conditioned on user-specified categories by leveraging the flexibility of LLMs.
+
+SpatialLM1.1 offers three variants of structured indoor modeling tasks:
+
+- **Structured Reconstruction**: Detect walls, doors, windows, boxes.
+- **Layout Estimation**: Detect walls, doors, windows.
+- **3D Object Detection**: Detect boxes.
+
+For tasks that include object box estimation, you can specify a subset of the 59 furniture categories, and the model will only predict objects within those specified categories. For example:
+
+```bash
+python inference.py --point_cloud pcd/scene0000_00.ply --output scene0000_00.txt --model_path manycore-research/SpatialLM1.1-Qwen-0.5B --detect_type object --category bed nightstand
 ```
 
 ### Visualization
@@ -119,13 +140,17 @@ huggingface-cli download manycore-research/SpatialLM-Testset --repo-type dataset
 Run evaluation:
 
 ```bash
-# Run inference on the PLY point clouds in folder SpatialLM-Testset/pcd with SpatialLM-Qwen-0.5B model
-python inference.py --point_cloud SpatialLM-Testset/pcd --output SpatialLM-Testset/pred --model_path manycore-research/SpatialLM-Qwen-0.5B
+# Run inference on the PLY point clouds in folder SpatialLM-Testset/pcd with SpatialLM1.1-Qwen-0.5B model
+python inference.py --point_cloud SpatialLM-Testset/pcd --output SpatialLM-Testset/pred --model_path manycore-research/SpatialLM1.1-Qwen-0.5B
 
 # Evaluate the predicted layouts
 python eval.py --metadata SpatialLM-Testset/test.csv --gt_dir SpatialLM-Testset/layout --pred_dir SpatialLM-Testset/pred --label_mapping SpatialLM-Testset/benchmark_categories.tsv
 ```
 
+### Example using a custom video
+
+We provide an example of how to use our model to estimate scene layout starting from a RGB video with the newly released [SLAM3R](https://github.com/PKU-VCL-3DV/SLAM3R) in [EXAMPLE.md](EXAMPLE.md). These steps work for MASt3R-SLAM, and other reconstruction methods as well.
+
 ## SpatialLM Testset
 
 We provide a test set of 107 preprocessed point clouds, reconstructed from RGB videos using [MASt3R-SLAM](https://github.com/rmurai0610/MASt3R-SLAM). SpatialLM-Testset is quite challenging compared to prior clean RGBD scans datasets due to the noises and occlusions in the point clouds reconstructed from monocular RGB videos.
@@ -140,38 +165,66 @@ We provide a test set of 107 preprocessed point clouds, reconstructed from RGB v
 
 ## Benchmark Results
 
-Benchmark results on the challenging SpatialLM-Testset are reported in the following table:
+### Layout Estimation
+
+Layout estimation focuses on predicting architectural elements, i.e., walls, doors, and windows, within an indoor scene. We evaluated this task on the [Structured3D](https://structured3d-dataset.org) dataset. For [RoomFormer](https://github.com/ywyue/RoomFormer), we directly downloaded the model checkpoint. SceneScript and SpatialLM were first trained on our dataset, and further fine-tuned on Structured3D.
 
 <div align="center">
 
-| **Method**       | **SpatialLM-Llama-1B** | **SpatialLM-Qwen-0.5B** |
-| ---------------- | ---------------------- | ----------------------- |
-| **Floorplan**    | **mean IoU**           |                         |
-| wall             | 78.62                  | 74.81                   |
-|                  |                        |                         |
-| **Objects**      | **F1 @.25 IoU (3D)**   |                         |
-| curtain          | 27.35                  | 28.59                   |
-| nightstand       | 57.47                  | 54.39                   |
-| chandelier       | 38.92                  | 40.12                   |
-| wardrobe         | 23.33                  | 30.60                   |
-| bed              | 95.24                  | 93.75                   |
-| sofa             | 65.50                  | 66.15                   |
-| chair            | 21.26                  | 14.94                   |
-| cabinet          | 8.47                   | 8.44                    |
-| dining table     | 54.26                  | 56.10                   |
-| plants           | 20.68                  | 26.46                   |
-| tv cabinet       | 33.33                  | 10.26                   |
-| coffee table     | 50.00                  | 55.56                   |
-| side table       | 7.60                   | 2.17                    |
-| air conditioner  | 20.00                  | 13.04                   |
-| dresser          | 46.67                  | 23.53                   |
-|                  |                        |                         |
-| **Thin Objects** | **F1 @.25 IoU (2D)**   |                         |
-| painting         | 50.04                  | 53.81                   |
-| carpet           | 31.76                  | 45.31                   |
-| tv               | 67.31                  | 52.29                   |
-| door             | 50.35                  | 42.15                   |
-| window           | 45.4                   | 45.9                    |
+|   **Method**    | **RoomFormer** | **SceneScript (finetuned)** | **SpatialLM1.1-Qwen-0.5B (finetuned)** |
+| :-------------: | :------------: | :-------------------------: | :------------------------------------: |
+| **F1 @.25 IoU** |      70.4      |            83.1             |                  86.5                  |
+| **F1 @.5 IoU**  |      67.2      |            80.8             |                  84.6                  |
+
+</div>
+
+### 3D Object Detection
+
+We evaluate 3D object detection on [ScanNet](http://www.scan-net.org) with annotations of 18 object categories. For [V-DETR](https://github.com/V-DETR/V-DETR), we directly download the model checkpoint. SceneScript and SpatialLM were first trained on our dataset, and further fine-tuned on ScanNet.
+
+<div align="center">
+
+|   **Method**    | **V-DETR** | **SceneScript (finetuned)** | **SpatialLM1.1-Qwen-0.5B (finetuned)** |
+| :-------------: | :--------: | :-------------------------: | :------------------------------------: |
+| **F1 @.25 IoU** |    65.1    |            49.1             |                  65.6                  |
+| **F1 @.5 IoU**  |    56.8    |            36.8             |                  52.6                  |
+
+</div>
+
+### Zero-shot Detection on Videos
+
+Zero-shot detection results on the challenging SpatialLM-Testset are reported in the following table:
+
+<div align="center">
+
+|   **Method**    | **SpatialLM1.1-Llama-1B** | **SpatialLM1.1-Qwen-0.5B** |
+| :-------------: | :-----------------------: | :------------------------: |
+|   **Layout**    |   **F1 @.25 IoU (2D)**    |    **F1 @.25 IoU (2D)**    |
+|      wall       |           68.9            |            68.2            |
+|      door       |           46.3            |            43.1            |
+|     window      |           43.8            |            47.4            |
+|                 |                           |                            |
+|   **Objects**   |   **F1 @.25 IoU (3D)**    |    **F1 @.25 IoU (2D)**    |
+|     curtain     |           34.9            |            37.0            |
+|   nightstand    |           62.8            |            67.0            |
+|   chandelier    |           53.5            |            36.8            |
+|    wardrobe     |           29.4            |            39.6            |
+|       bed       |           96.8            |            95.2            |
+|      sofa       |           66.9            |            69.1            |
+|      chair      |           20.8            |            32.3            |
+|     cabinet     |           15.2            |            11.2            |
+|  dining table   |           40.7            |            24.2            |
+|     plants      |           29.5            |            26.3            |
+|   tv cabinet    |           34.4            |            27.3            |
+|  coffee table   |           56.4            |            64.9            |
+|   side table    |           14.6            |            9.7             |
+| air conditioner |           16.7            |            24.0            |
+|     dresser     |           46.7            |            46.7            |
+|      stool      |           17.6            |            30.8            |
+|  refrigerator   |            0.0            |            16.7            |
+|    painting     |           34.9            |            38.2            |
+|     carpet      |           40.3            |            24.1            |
+|       tv        |           16.0            |            18.0            |
 
 </div>
 
@@ -180,18 +233,23 @@ Benchmark results on the challenging SpatialLM-Testset are reported in the follo
 SpatialLM-Llama-1B is derived from Llama3.2-1B-Instruct, which is licensed under the Llama3.2 license.
 SpatialLM-Qwen-0.5B is derived from the Qwen-2.5 series, originally licensed under the Apache 2.0 License.
 
-All models are built upon the SceneScript point cloud encoder, licensed under the CC-BY-NC-4.0 License. TorchSparse, utilized in this project, is licensed under the MIT License.
+SpatialLM1.0 are built upon the SceneScript point cloud encoder, licensed under the CC-BY-NC-4.0 License. TorchSparse, utilized in this project, is licensed under the MIT License.
+
+SpatialLM1.1 are built upon Sonata point cloud encoder, model weight is licensed under the CC-BY-NC-4.0 License. Code built on Pointcept is licensed under the Apache 2.0 License.
 
 ## Citation
 
 If you find this work useful, please consider citing:
 
 ```bibtex
-@misc{spatiallm,
-  title        = {SpatialLM: Large Language Model for Spatial Understanding},
-  author       = {ManyCore Research Team},
-  howpublished = {\url{https://github.com/manycore-research/SpatialLM}},
-  year         = {2025}
+@article{SpatialLM,
+    title         = {SpatialLM: Training Large Language Models for Structured Indoor Modeling},
+    author        = {Mao, Yongsen and Zhong, Junhao and Fang, Chuan and Zheng, Jia and Tang, Rui and Zhu, Hao and Tan, Ping and Zhou, Zihan},
+    journal       = {arXiv preprint},
+    year          = {2025},
+    eprint        = {2506.07491},
+    archivePrefix = {arXiv},
+    primaryClass  = {cs.CV}
 }
 ```
 
@@ -199,4 +257,4 @@ If you find this work useful, please consider citing:
 
 We would like to thank the following projects that made this work possible:
 
-[Llama3.2](https://github.com/meta-llama) | [Qwen2.5](https://github.com/QwenLM/Qwen2.5) | [Transformers](https://github.com/huggingface/transformers) | [SceneScript](https://github.com/facebookresearch/scenescript) | [TorchSparse](https://github.com/mit-han-lab/torchsparse)
+[Llama3.2](https://github.com/meta-llama) | [Qwen2.5](https://github.com/QwenLM/Qwen2.5) | [Transformers](https://github.com/huggingface/transformers) | [SceneScript](https://github.com/facebookresearch/scenescript) | [TorchSparse](https://github.com/mit-han-lab/torchsparse) | [Sonata](https://xywu.me/sonata/) | [Pointcept](https://github.com/Pointcept/Pointcept)