ybrrraway/V2LO-7B

Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning

Video2Layout is a framework for reconstructing metric-grounded spatial layouts from video, leveraging continuous object boundary coordinates to quantify inter-object physical distances and object sizes. This equips the model with quantitative spatial computation capabilities, effectively mitigating the inherent ambiguity in describing spatial relationships through natural language. The framework adopts a two-stage SFT-to-RL training paradigm, enhancing the model's spatial reasoning abilities.

Paper: Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning Code: https://github.com/ybrrraway/Video2Layout

🚀 Overview

🎯 Key Benefits:

• Metric-Grounded Cognitive Map — an accurate bird's-eye view reflects the specific position of an object in the scene.
• Spatial reasoning computation — rigorous mathematical calculations eliminate the fuzziness of traditional natural language COT description spatial relationship reasoning.
• Generalization of real scenes — only the information of simulation data is needed, and there are no requirements for real scenarios.

QVS-Bench is a diagnostic benchmark for systematically evaluating how the quantity of image inputs impacts spatial reasoning accuracy. It maintains a substantially uniform proportional distribution across five input scale configurations (1, 4, 8, 12, and 16 frames), ensuring fair and unbiased analysis of the relevant mechanisms.

🛠️ Usage

(Step 1) Install

conda create -n v2lo python=3.10 -y
conda activate v2lo
pip install -r requirements.txt

(Step 2) Training

# SFT training
bash src/ms-swift/sft.sh
# Merge model
bash src/ms-swift/merge_lora.sh

# RL training
bash src/EasyR1/examples/rl.sh
# Merge model
cd src/EasyR1
python3 scripts/model_merger.py --local_dir checkpoints/easy_r1/exp_name/global_step_1/actor

Citation

@misc{2511.16160,
Author = {Yibin Huang and Wang Xu and Wanyue Zhang and Helu Zhi and Jingjing Huang and Yangbin Xu and Yangang Sun and Conghui Zhu and Tiejun Zhao},
Title = {Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning},
Year = {2025},
Eprint = {arXiv:2511.16160},
}