Image-Text-to-Text
PEFT
Safetensors
multimodal
vision-language
visual-planning
spatial-planning
rule-following
qwen2.5-vl
lora
Instructions to use Fish-03/RuleMaze with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Fish-03/RuleMaze with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
datasets:
|
| 3 |
+
- Fish-03/RuleMaze
|
| 4 |
+
base_model:
|
| 5 |
+
- Qwen/Qwen2.5-VL-3B-Instruct
|
| 6 |
+
library_name: peft
|
| 7 |
+
tags:
|
| 8 |
+
- multimodal
|
| 9 |
+
- vision-language
|
| 10 |
+
- visual-planning
|
| 11 |
+
- spatial-planning
|
| 12 |
+
- rule-following
|
| 13 |
+
- qwen2.5-vl
|
| 14 |
+
- lora
|
| 15 |
+
pipeline_tag: image-text-to-text
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
# RuleMaze
|
| 19 |
+
|
| 20 |
+
RuleMaze is a benchmark and training framework for **rule-compliant visual spatial planning** with Multimodal Large Language Models (MLLMs).
|
| 21 |
+
|
| 22 |
+
Given a visual maze and a set of natural-language rules, the model is required to understand the environment, follow the rules, and generate a valid multi-step trajectory.
|
| 23 |
+
|
| 24 |
+
This repository provides LoRA adapters fine-tuned from **Qwen2.5-VL-3B-Instruct** on the RuleMaze training data.
|
| 25 |
+
|
| 26 |
+
## Checkpoints
|
| 27 |
+
|
| 28 |
+
Two scene types are provided:
|
| 29 |
+
|
| 30 |
+
* `RuleMaze/regular/checkpoint`
|
| 31 |
+
* `RuleMaze/quest/checkpoint`
|
| 32 |
+
|
| 33 |
+
The checkpoints are PEFT/LoRA adapters and should be loaded together with the base model:
|
| 34 |
+
|
| 35 |
+
```text
|
| 36 |
+
Qwen/Qwen2.5-VL-3B-Instruct
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
## Resources
|
| 40 |
+
|
| 41 |
+
* **Code:** https://github.com/oceanflowlab/RuleMaze
|
| 42 |
+
* **Dataset:** https://huggingface.co/datasets/Fish-03/RuleMaze
|
| 43 |
+
* **Base Model:** https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct
|
| 44 |
+
|
| 45 |
+
## Training and Evaluation
|
| 46 |
+
|
| 47 |
+
The models are trained using the RuleMaze DMP training pipeline with LLaMA-Factory.
|
| 48 |
+
|
| 49 |
+
RuleMaze evaluates visual planning under both **seen-rule** and **unseen-rule** settings and different rule difficulties.
|
| 50 |
+
|
| 51 |
+
For training and evaluation details, please refer to the official code repository.
|
| 52 |
+
|
| 53 |
+
## Intended Use
|
| 54 |
+
|
| 55 |
+
The models are intended for research on:
|
| 56 |
+
|
| 57 |
+
* multimodal reasoning
|
| 58 |
+
* visual spatial planning
|
| 59 |
+
* rule following
|
| 60 |
+
* compositional generalization
|
| 61 |
+
|
| 62 |
+
## Citation
|
| 63 |
+
|
| 64 |
+
If you find RuleMaze useful, please cite:
|
| 65 |
+
|
| 66 |
+
```bibtex
|
| 67 |
+
@misc{rulemaze,
|
| 68 |
+
title = {Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models},
|
| 69 |
+
author = {Yu Chen, Ting Lei, Yaoyi Li, Jia Cai, Zhecen Wu and Yang Liu},
|
| 70 |
+
year = {2026},
|
| 71 |
+
note = {Code and dataset release}
|
| 72 |
+
}
|
| 73 |
+
```
|