Improve model card: Add metadata, links, and usage for GS-Reasoner

README.md

@@ -1,3 +1,125 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+---
+
+# Reasoning in Space via Grounding in the World
+
+We present **Grounded-Spatial Reasoner (GS-Reasoner)**, the first 3D-LLM that bridges 3D visual grounding and spatial reasoning, as explored in the paper [Reasoning in Space via Grounding in the World](https://huggingface.co/papers/2510.13800).
+
+The goal of GS-Reasoner is to explore effective spatial representations that bridge the gap between 3D visual grounding and spatial reasoning. Existing 3D LLMs suffer from the absence of a unified 3D representation capable of jointly capturing semantic and geometric information, leading to either poor grounding performance or excessive reliance on external modules. GS-Reasoner addresses this by proposing a simple yet effective dual-path pooling mechanism that tightly aligns geometric features with both semantic and positional cues, constructing a unified image patch-based 3D representation. This enables GS-Reasoner to achieve autoregressive grounding entirely without external modules while delivering comparable performance to state-of-the-art models, establishing a unified and self-contained framework for 3D spatial reasoning.
+
+**Project Page**: [https://yiming-cc.github.io/gs-reasoner/](https://yiming-cc.github.io/gs-reasoner/)
+**Code**: [https://github.com/WU-CVGL/GS-Reasoner](https://github.com/WU-CVGL/GS-Reasoner)
+
+<div style="text-align: center;">
+    <img src="https://huggingface.co/spaces/ymccccc/GS-Reasoner/resolve/main/assets/teaser.png" width=100% >
+</div>
+
+## Model Weights
+We provide two pretrained model checkpoints:
+
+*   **[GS-Reasoner](https://huggingface.co/ymccccc/GS-Reasoner)** – the main model used in our paper, producing more deterministic chain-of-thought reasoning.
+*   **[GS-Reasoner-Diverse](https://huggingface.co/ymccccc/GS-Reasoner-Diverse)** – a variant that generates more diverse chain-of-thought outputs with only a minor performance drop (less than 1.0 on VSI-Bench).
+
+## Sample Usage
+
+This section provides instructions on how to inference our pre-trained grounding models. The model can be loaded using classes from the `pae` library (which can be installed from the [GitHub repository](https://github.com/WU-CVGL/GS-Reasoner)).
+
+**Notes:** Our models accept images of any size as input. The model outputs are normalized to relative coordinates within a 0-1000 range (either a center point or a bounding box defined by top-left and bottom-right coordinates). For visualization, please remember to convert these relative coordinates back to the original image dimensions.
+
+First, set up your environment as described in the [official GitHub repository's Setup section](https://github.com/WU-CVGL/GS-Reasoner#setup). This typically involves:
+```bash
+conda create -n gs-reasoner python=3.11 -y
+conda activate gs-reasoner
+git clone git@github.com:WU-CVGL/GS-Reasoner.git
+cd GS-Reasoner
+pip install -e .
+# Install submodules dependencies as well, e.g., for Sonata and VSI-Bench Evaluation
+cd llava/submodules/sonata && pip install -r requirements.txt && cd ../../..
+cd llava/submodules/lmms_eval && pip install -r requirements.txt && cd ../../..
+```
+
+Inference code example:
+```python
+import pae
+from pae.models import LlavaAgent, ClaudeAgent
+from accelerate import Accelerator
+import torch
+from tqdm import tqdm
+from types import SimpleNamespace
+from pae.environment.webgym import BatchedWebEnv
+import os
+from llava.model.language_model.llava_mistral import LlavaMistralForCausalLM
+
+# ============= Instanstiate the agent =============
+config_dict = {"use_lora": False,
+               "use_q4": False, # our 34b model is quantized to 4-bit, set it to True if you are using 34B model
+               "use_anyres": False,
+               "temperature": 1.0,
+               "max_new_tokens": 512,
+               "train_vision": False,
+               "num_beams": 1,}
+config = SimpleNamespace(**config_dict)
+
+accelerator = Accelerator()
+agent = LlavaAgent(policy_lm = "ymccccc/GS-Reasoner", # or "ymccccc/GS-Reasoner-Diverse"
+                            device = accelerator.device,
+                            accelerator = accelerator,
+                            config = config)
+
+# ============= Instanstiate the environment =============
+test_tasks = [{"web_name": "Google Map",
+               "id": "0",
+          "ques": "Locate a parking lot near the Brooklyn Bridge that open 24 hours. Review the user comments about it.",
+          "web": "https://www.google.com/maps/"}]
+save_path = "xxx" # Placeholder, adapt for your needs
+
+test_env = BatchedWebEnv(tasks = test_tasks,
+                        do_eval = False,
+                        download_dir=os.path.join(save_path, 'test_driver', 'download'),
+                        output_dir=os.path.join(save_path, 'test_driver', 'output'),
+                        batch_size=1,
+                        max_iter=10,)
+# for you to check the images and actions
+image_histories = [] # stores the history of the paths of images
+action_histories = [] # stores the history of actions
+
+results = test_env.reset()
+image_histories.append(results[0][0]["image"])
+
+observations = [r[0] for r in results]
+actions = agent.get_action(observations)
+action_histories.append(actions[0])
+dones = None
+
+for _ in tqdm(range(3)):
+    if dones is not None and all(dones):
+        break
+    results = test_env.step(actions)
+    image_histories.append(results[0][0]["image"])
+    observations = [r[0] for r in results]
+    actions = agent.get_action(observations)
+    action_histories.append(actions[0])
+    dones = [r[2] for r in results]
+
+print("Done!")
+print("image_histories: ", image_histories)
+print("action_histories: ", action_histories)
+```
+
+## Citation
+If you find our work helpful or inspiring, please feel free to cite it.
+
+```bibtex
+@misc{chen2025gs-reasoner,
+    title={Reasoning in Space via Grounding in the World},
+    author={Yiming Chen and Zekun Qi and Wenyao Zhang and Xin Jin and Li Zhang and Peidong Liu},
+    year={2025},
+    eprint={2510.13800},
+    archivePrefix={arXiv},
+    primaryClass={cs.CV},
+    url={https://arxiv.org/abs/2510.13800},
+}
+```