README.md

---
base_model: Qwen/Qwen2.5-VL-7B-Instruct
library_name: peft
license: apache-2.0
tags:
  - peft
  - lora
  - vision-language
  - video
  - optical-flow
  - uav
  - drone
  - motion
  - embodied-ai
  - spatial-intelligence
---

# SIS-Motion

<p align="center">
  <img src="images/poster.jpeg" alt="SIS-Motion" width="900"/>
</p>
<p align="center">
  <a href="https://arxiv.org"><img src="https://img.shields.io/badge/arXiv-Paper-b31b1b?style=for-the-badge&logo=arxiv" alt="arXiv"></a>
  <a href="https://choucisan.github.io/publications/self-in-space"><img src="https://img.shields.io/badge/Website-Project_Page-blue?style=for-the-badge" alt="Website"></a>
  <a href="https://github.com/IntelliSensing/Self-in-Space"><img src="https://img.shields.io/badge/GitHub-Self--in--Space-181717?style=for-the-badge&logo=github" alt="GitHub"></a>
  <a href="https://huggingface.co/choucsan/SIS-Motion"><img src="https://img.shields.io/badge/%F0%9F%A4%97_HuggingFace-Model-yellow?style=for-the-badge" alt="Hugging Face"></a>
  <a href="https://choosealicense.com/licenses/apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-green?style=for-the-badge" alt="License"></a>
</p>


SIS-Motion is a motion-aware extension of a standard video MLLM, designed to investigate whether explicitly modeling self-related dynamics can improve the joint understanding of *space* and *self* in embodied UAV scenarios.

> **Important:** SIS-Motion is **NOT a state-of-the-art model.** It is a **controlled experiment** where the only difference from the visual-only backbone is the addition of a frozen optical-flow estimator, a motion-feature branch, and a lightweight fusion connector. All other settings — base model, training data, LoRA rank, learning rate, and optimization — are kept identical. This ensures that any observed gain can be cleanly attributed to motion-aware integration rather than differences in scale or recipe.

---

## Architecture

SIS-Motion extends Qwen2.5-VL with a motion-aware branch. Video frames are processed in parallel through a shared vision encoder — once for appearance and once for optical-flow-derived motion features — then fused via a lightweight additive connector before the language model.

<p align="center">
  <img src="images/sis-motion-framework.webp" alt="SIS-Motion Framework" width="900"/>
</p>

The forward path is:

```text
video frames
├── Qwen2.5-VL vision encoder (frozen) ───────────────> visual tokens
└── VideoFlow MOFNet (frozen, FP32)
    └── forward/backward flow [T, 4, H/8, W/8]
        └── forward flow (dx, dy)
            └── pseudo-images [magnitude, dx, dy]
                └── Qwen2.5-VL vision encoder (shared, frozen) -> motion tokens

motion tokens -> LayerNorm -> Linear(3584, 2048) -> GELU
              -> Linear(2048, 3584)
visual tokens + projected motion tokens -> Qwen2.5-VL language model with LoRA
```

### 1. Appearance branch

The standard Qwen2.5-VL video processor and frozen vision encoder produce the appearance tokens used by the base model.

### 2. Motion branch

VideoFlow MOFNet receives at least three RGB frames, resizes them to `320x480`, and predicts forward and backward optical flow at `1/8` resolution. SIS-Motion uses the forward `(dx, dy)` channels, clamps invalid/extreme values, and builds a three-channel pseudo-image `[magnitude, dx, dy]`. These pseudo-images are patchified with the Qwen2.5-VL video grid and passed through the same frozen Qwen vision encoder to produce token-aligned motion features.

### 3. Additive connector

The trained `visual_flow` connector applies LayerNorm and a two-layer MLP to each motion token, then adds the result to the corresponding appearance token:

```text
motion' = Linear(GELU(Linear(LayerNorm(motion))))
fused   = visual + motion'
```

The fused tokens replace the normal video-token embeddings before the language model forward pass. During inference, the LoRA adapter is merged into the base language model.

| Component | Parameters | Training state | Included here |
| --- | ---: | --- | --- |
| Qwen2.5-VL-7B base model and vision encoder | See base model | Frozen base weights; LoRA on LLM | No, downloaded separately |
| VideoFlow MOFNet | 13,453,240 | Frozen | Yes |
| `visual_flow` connector | 14,692,864 | Trained | Yes |
| LoRA adapter (`r=32`, `alpha=64`) | 20,185,088 | Trained | Yes |
| **Trainable parameters** | **34,877,952** | LoRA + connector | Yes |
| **Bundled model parameters** | **48,331,192** | Includes frozen MOFNet | Yes |

---

## Why a Controlled Experiment?

Current MLLMs show a consistent gap: they are substantially better at modeling *space* (objects, scenes, layouts) than *self* (motion, actions, agent state). To test whether explicit motion cues can alleviate this, we constructed SIS-Motion under strict controls:

- ✅ **Same base model** — Qwen2.5-VL-7B-Instruct
- ✅ **Same training data** — SIS-Motion-54K (perception + memory tasks only)
- ✅ **Same LoRA configuration** — `r=32`, `α=64`, identical target modules
- ✅ **Same optimization** — identical learning rate, schedule, and batch size
- ⬜ **Only difference** — the optical-flow-driven motion branch + additive connector

The visual-only SFT baseline serves as the direct comparison point. Any difference in results can be interpreted as the marginal contribution of motion-aware modeling.

---

## Training Configuration

| Setting          | Value                                          |
| ---------------- | ---------------------------------------------- |
| Base model       | `Qwen/Qwen2.5-VL-7B-Instruct`                  |
| Training data    | SIS-Motion-54K (MCQ + Open QA + Description)   |
| Data scope       | Perception & memory tasks (no reasoning tasks) |
| LoRA rank `r`    | 32                                             |
| LoRA alpha       | 64                                             |
| LoRA dropout     | 0.05                                           |
| LoRA targets     | `q_proj`, `k_proj`, `v_proj`, `o_proj`         |
| Optimizer        | AdamW                                          |
| Precision        | BF16                                           |
| Trainable params | 34,877,952 (20,185,088 LoRA + 14,692,864 connector) |
| Frozen params    | Visual encoder, VideoFlow optical-flow estimator, LLM base (~7B) |

---

## Results

### SIS-Bench

| Model                    | Spatial Avg | Self Avg | Overall  |
| ------------------------ | ----------- | -------- | -------- |
| ZeroShot (Qwen2.5-VL 3B) | 65.8        | 40.5     | 53.6     |
| SFT visual-only          | 72.0        | 60.3     | 66.4     |
| **SIS-Motion**           | **74.2**    | **63.7** | **69.1** |

Motion-aware modeling improves both spatial cognition (+2.2) and self-awareness (+3.4) over the visual-only SFT baseline. The gains are most pronounced on memory-level tasks, where temporal integration matters most.

### Downstream Transfer — Zero-Shot UAV Navigation

Evaluated on the [OpenUAV-QA](https://huggingface.co/datasets/choucsan/OpenUAV-QA) navigation benchmark without any task-specific fine-tuning:

| Model                    | Accuracy  |
| ------------------------ | --------- |
| Qwen2.5-VL 3B (backbone) | 71.2%     |
| **SIS-Motion**           | **92.2%** |

The +21.0 point gain demonstrates that motion-aware representations transfer beyond the original benchmark to practical UAV decision-making scenarios.

> **Note on reported results:** The results above are from the paper's **3B** backbone (Qwen2.5-VL-3B-Instruct). This repository hosts a **7B** variant (Qwen2.5-VL-7B-Instruct) with the same architecture and training protocol. The 3B model can be reproduced using the open-source [training code](https://github.com/IntelliSensing/Self-in-Space). We release the 7B version here as a stronger off-the-shelf checkpoint; the controlled-experiment conclusion (motion-aware gains over visual-only SFT) holds across both scales.

---

## Repository Files

```text
sis_motion_config.json       Architecture manifest (connector type, paths)
adapter_config.json          LoRA configuration (r=32, alpha=64)
adapter_model.safetensors    LoRA weights (20,185,088 params)
connector_weights.pt         Motion fusion MLP weights (14,692,864 params)
MOF_kitti.pth                Frozen VideoFlow MOFNet checkpoint (13,453,240 params)
images/poster.jpeg           Model poster
```

**Note:** The Qwen2.5-VL-7B-Instruct base weights are **NOT included** in this repository. They are downloaded automatically from HuggingFace at inference time.

---

## Usage

This model package is not a standalone Transformers architecture. It must be loaded through the SIS-Motion evaluator, which constructs the motion branch before applying the PEFT adapter.

### 1. Clone the codebase

```bash
git clone https://github.com/IntelliSensing/Self-in-Space.git
cd Self-in-Space
```

### 2. Create the motion environment

With Conda:

```bash
bash scripts/setup_conda.sh motion
conda activate sis-motion-motion
```

Or with uv:

```bash
bash scripts/setup_uv.sh motion
source .venv-motion/bin/activate
```

The motion environment uses Python 3.10, PyTorch 2.6 with CUDA 12.4, Transformers 4.57.1, PEFT 0.18.1, and FlashAttention 2.7.4.

### 3. Prepare SIS-Bench

Place SIS-Bench under the registered data root or override the paths directly:

```text
data/SIS-Bench/SIS-Bench.jsonl
data/SIS-Bench/frames/
```

### 4. Run inference

The evaluator automatically downloads this model from HuggingFace, reads `sis_motion_config.json`, loads the VideoFlow optical-flow estimator (`MOF_kitti.pth`), applies the LoRA adapter, loads `connector_weights.pt`, and runs the full appearance-plus-motion pipeline:

```bash
MODEL_PATH=choucsan/SIS-Motion \
bash scripts/eval_motion.sh
```

An already downloaded model directory is also accepted:

```bash
MODEL_PATH=/path/to/SIS-Motion \
DATA_FILE=/path/to/SIS-Bench.jsonl \
FRAMES_DIR=/path/to/SIS-Bench/frames \
bash scripts/eval_motion.sh
```

The Qwen2.5-VL-7B base model is fetched from the `base_model_name_or_path` stored in `adapter_config.json`; it is not duplicated in this repository.

> **Note on the optical flow estimator:** The default checkpoint (`MOF_kitti.pth`) is a VideoFlow MOFNet pretrained on KITTI. Replacing it with another optical-flow estimator (e.g., RAFT or GMFlow) requires adapting the optical-flow loading and output-conversion logic. The connector weights may not transfer to a different flow backbone and would likely require retraining.

---

## Limitations

- **Not a SOTA model.** SIS-Motion is a controlled research prototype built on a 7B backbone. Stronger base models would likely yield higher absolute scores under the same setup.
- **Optical flow estimation.** The VideoFlow optical-flow estimator runs on each input video, adding compute not required by the visual-only baseline.
- **Training scope.** SIS-Motion-54K covers perception and memory tasks. Reasoning-level improvements come from generalization, not direct supervision.
- **Inference cost.** The runtime adds 28,146,104 module parameters over the base path (13,453,240 frozen MOFNet + 14,692,864 connector). The 20,185,088 LoRA parameters are merged into the base model before generation.

---

## Contact

For questions or collaboration requests:

[choucisan@gmail.com](mailto:choucisan@gmail.com)