hunarbatra
/

SpatialThinker-7B

@@ -1,202 +1,164 @@
 ---
-library_name: transformers
-datasets:
-- OX-PIXL/STVQA-7K
-base_model:
-- Qwen/Qwen2.5-VL-7B-Instruct
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+license: apache-2.0
+language:
+- en
+tags:
+- spatial-reasoning
+- multimodal
+- vision-language
+- scene-graph
+- reinforcement-learning
+base_model: Qwen/Qwen2.5-VL-7B-Instruct
+pipeline_tag: image-text-to-text
 ---
+# SpatialThinker-7B
+<p align="center">
+  <a href="https://arxiv.org/abs/2511.07403">
+    <img src="https://img.shields.io/badge/arXiv-2511.07403-b31b1b.svg" alt="arXiv">
+  </a>
+  <a href="https://hunarbatra.com/SpatialThinker">
+    <img src="https://img.shields.io/badge/🌐%20Project%20Page-blue.svg" alt="Project Page">
+  </a>
+  <a href="https://github.com/hunarbatra/SpatialThinker">
+    <img src="https://img.shields.io/badge/GitHub-Repository-black.svg" alt="GitHub">
+  </a>
+</p>
+**SpatialThinker-7B** is a 3D-aware multimodal large language model (MLLM) trained with reinforcement learning to integrate structured spatial grounding with multi-step reasoning. The model simulates human-like spatial perception by constructing a scene graph of task-relevant objects and spatial relations, and reasoning towards an answer via dense spatial rewards.
+## Model Description
+- **Base Model**: Qwen2.5-VL-7B-Instruct
+- **Training**: GRPO (Group Relative Policy Optimization) with dense spatial rewards
+- **Training Data**: STVQA-7K (7,587 spatial VQA samples)
+- **Authors**: Hunar Batra, Haoqin Tu, Hardy Chen, Yuanze Lin, Cihang Xie, Ronald Clark
+- **Institutions**: University of Oxford, UC Santa Cruz
+## Key Features
+- **Structured Spatial Reasoning**: Constructs question-focused scene subgraphs with objects, bounding boxes, and relations
+- **Dense Spatial Rewards**: Multi-objective reward function enforcing format, count, accuracy, and spatial grounding
+- **9 Spatial Reasoning Categories**: Relations, reach, size, orientation, instance location, depth, distance, count, and existence
+- **Outperforms GPT-4o**: On spatial understanding benchmarks while using only 7K training samples
+## Inference Template
+Use the following template for inference:
+```
+You FIRST observe the image in <observe> </observe> tags, then visualise the relevant scene graph in <scene> </scene> tags, followed by thinking about the reasoning process as an internal monologue within <think> </think> tags and then provide the final answer. The final answer MUST BE put within <answer> </answer> tags, and only return the final choice including the correct option and answer within the answer tags, e.g., <answer> (A) cat </answer>.
+Image size: {Width} x {Height}
+```
+## Output Format
+The model generates structured output with four components:
+1. **`<observe>`**: Scene description covering relevant objects
+2. **`<scene>`**: JSON scene graph with objects (id, bbox) and relationships (subject, predicate, object)
+3. **`<think>`**: Step-by-step reasoning as internal monologue
+4. **`<answer>`**: Final answer with option letter and text
+### Example Output
+```
+<observe>
+The image shows a living room with a couch, a coffee table, and a cat sitting on the floor.
+</observe>
+<scene>
+{
+  "objects": [
+    {"id": "couch.1", "bbox": [50, 100, 400, 350]},
+    {"id": "cat.1", "bbox": [200, 300, 280, 400]},
+    {"id": "table.1", "bbox": [150, 250, 350, 320]}
+  ],
+  "relationships": [
+    {"subject": "cat.1", "predicate": "in front of", "object": "couch.1"},
+    {"subject": "cat.1", "predicate": "beside", "object": "table.1"}
+  ]
+}
+</scene>
+<think>
+Looking at the scene graph, the cat is positioned in front of the couch and beside the coffee table. The bounding box coordinates show the cat is at y=300-400 while the couch extends to y=350, confirming the cat is on the floor in front of the couch.
+</think>
+<answer> (B) in front of the couch </answer>
+```
+## Usage
+```python
+from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
+from PIL import Image
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    "OX-PIXL/SpatialThinker-7B",
+    torch_dtype="auto",
+    device_map="auto"
+)
+processor = AutoProcessor.from_pretrained("OX-PIXL/SpatialThinker-7B")
+# Load image
+image = Image.open("your_image.jpg")
+width, height = image.size
+# Prepare prompt with template
+template = f"""You FIRST observe the image in <observe> </observe> tags, then visualise the relevant scene graph in <scene> </scene> tags, followed by thinking about the reasoning process as an internal monologue within <think> </think> tags and then provide the final answer. The final answer MUST BE put within <answer> </answer> tags, and only return the final choice including the correct option and answer within the answer tags, e.g., <answer> (A) cat </answer>.
+Image size: {width} x {height}"""
+question = "Where is the cat relative to the couch? (A) on top of (B) in front of (C) behind (D) beside"
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": image},
+            {"type": "text", "text": template + "\n\n" + question},
+        ],
+    }
+]
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
+generated_ids = model.generate(**inputs, max_new_tokens=1024)
+output = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(output)
+```
+## Evaluation Results
+SpatialThinker-7B achieves state-of-the-art performance on spatial reasoning benchmarks:
+| Benchmark | SpatialThinker-7B |
+|-----------|------------------------|
+| CV-Bench (3D) | Strong performance |
+| BLINK-Spatial | Outperforms GPT-4o |
+| SpatialBench | SOTA results |
+| RealWorldQA | Competitive |
+See the [paper](https://arxiv.org/abs/2511.07403) for detailed results.
+## Citation
+```bibtex
+@misc{batra2025spatialthinkerreinforcing3dreasoning,
+  title={SpatialThinker: Reinforcing 3D Reasoning in Multimodal LLMs via Spatial Rewards},
+  author={Hunar Batra and Haoqin Tu and Hardy Chen and Yuanze Lin and Cihang Xie and Ronald Clark},
+  year={2025},
+  eprint={2511.07403},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV},
+  url={https://arxiv.org/abs/2511.07403},
+}
+```
+## Links
+- 📄 **Paper**: [arXiv:2511.07403](https://arxiv.org/abs/2511.07403)
+- 🌐 **Project Page**: [hunarbatra.com/SpatialThinker](https://hunarbatra.com/SpatialThinker)
+- 💻 **GitHub**: [github.com/hunarbatra/SpatialThinker](https://github.com/hunarbatra/SpatialThinker)
+- 🤗 **Dataset**: [OX-PIXL/STVQA-7K](https://huggingface.co/datasets/OX-PIXL/STVQA-7K)