---
base_model:
- Qwen/Qwen2.5-VL-7B-Instruct
datasets:
- ybrrraway/V2LO-28K
language:
- en
license: apache-2.0
metrics:
- accuracy
pipeline_tag: video-text-to-text
library_name: transformers
---

# Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning

**Video2Layout** is a framework for reconstructing metric-grounded spatial layouts from video, leveraging continuous object boundary coordinates to quantify inter-object physical distances and object sizes. This equips the model with quantitative spatial computation capabilities, effectively mitigating the inherent ambiguity in describing spatial relationships through natural language. The framework adopts a two-stage SFT-to-RL training paradigm, enhancing the model's spatial reasoning abilities.

Paper: [Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning](https://huggingface.co/papers/2511.16160)
Code: https://github.com/ybrrraway/Video2Layout

<p align="center">
  <img src="https://github.com/ybrrraway/Video2Layout/raw/main/figs/show.png" width=100%/>
</p>

## 🚀 Overview

🎯 **Key Benefits**:
-   **Metric-Grounded Cognitive Map** — an accurate bird's-eye view reflects the specific position of an object in the scene.
-   **Spatial reasoning computation** — rigorous mathematical calculations eliminate the fuzziness of traditional natural language COT description spatial relationship reasoning.
-   **Generalization of real scenes** — only the information of simulation data is needed, and there are no requirements for real scenarios.

**QVS-Bench** is a diagnostic benchmark for systematically evaluating how the quantity of image inputs impacts spatial reasoning accuracy. It maintains a substantially uniform proportional distribution across five input scale configurations (1, 4, 8, 12, and 16 frames), ensuring fair and unbiased analysis of the relevant mechanisms.

<p align="center">
  <img src="https://github.com/ybrrraway/Video2Layout/raw/main/figs/benchmark.png" width=100%/>
</p>

## 🛠️ Usage
### (Step 1) Install
```bash
conda create -n v2lo python=3.10 -y
conda activate v2lo
pip install -r requirements.txt
```

### (Step 2) Training
```bash
# SFT training
bash src/ms-swift/sft.sh
# Merge model
bash src/ms-swift/merge_lora.sh

# RL training
bash src/EasyR1/examples/rl.sh
# Merge model
cd src/EasyR1
python3 scripts/model_merger.py --local_dir checkpoints/easy_r1/exp_name/global_step_1/actor
```

## Citation
```bibtex
@misc{2511.16160,
Author = {Yibin Huang and Wang Xu and Wanyue Zhang and Helu Zhi and Jingjing Huang and Yangbin Xu and Yangang Sun and Conghui Zhu and Tiejun Zhao},
Title = {Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning},
Year = {2025},
Eprint = {arXiv:2511.16160},
}
```