README.md

---
license: apache-2.0
language:
- en
base_model:
- Qwen/Qwen2.5-VL-3B-Instruct
tags:
- R2R
- VLN
- Room-to-Room
- LVLM
---

# Qwen2.5-VL-3B-R2R-panoramic

**Qwen2.5-VL-3B-R2R-panoramic** is a Vision-and-Language Navigation (VLN) model fine-tuned from [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) on the [Room-to-Room (R2R)](https://bringmeaspoon.org/) dataset using the Matterport3D (MP3D) simulator. The model is trained using a panoramic action space, where the model recives a preprocessed panoramic image and a set of candidate views which each point towards a node in a Matterport3D simualtor environment.  

Only the LLM component is fine-tuned — the vision encoder and cross-modal projector are kept frozen.


## 🧠 Model Summary

- **Base Model**: Qwen2.5-VL-3B-Instruct
- **Dataset**: Room-to-Room (R2R) via the Matterport3D simulator.
- **Image Resolution**: 320x240 for candidate images and 960×240 for panoramic images.
- **Action Space**: Panoramic.

## 🧪 Training Setup

- **Frozen Modules**: Vision encoder and cross-modal projector  
- **Fine-Tuned Module**: LLM decoder (Qwen2.5)  
- **Optimizer**: AdamW  
- **Batch Size**: `1` (with gradient accumulation over each episode)  
- **Learning Rate**: `1e-5`  
- **Weight Decay**: `0.1`  
- **Precision**: `bfloat16`  
- **LR Scheduler**:  Linear scheduler with warmup (first 10% of steps)  
- **Hardware**: Trained on a single NVIDIA A100 80GB GPU  

Training was done using supervised learning for next-action prediction. The model was conditioned at each step with a system prompt, panoramic RGB image observations (960×240) of current view, as well as variable amount of candidate RGB iamges (320x240), and cumulative episode history including previosu panoramas. The model was trained offline (not in the MP3D simulator) using teacher-forcing on a preprocessed R2R dataset.


## 📦 Usage 
```python
import torch
from torch.utils.data import Dataset, DataLoader
from datasets import Dataset as DT
from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from PIL import Image

lass CustomDataset(Dataset):
    def __init__(self, data):
        self.text = data["text"]
        self.panoramas = data["panoramas"]
        self.candidates = data["candidates"]
        
    def __len__(self):
        return len(self.text)
    
    def __getitem__(self, index):
        return self.text[index], self.panoramas[index], self.candidates[index]

# TODO: make the collatefunctor work with batches
class CollateFunctor:
    # No batch, therefore no max length
    def __init__(self, processor, width, height):
        self.processor = processor
        self.width = width
        self.height = height

    def __call__(self, batch):
        text, panoramas, candidates = batch[0]
        label_start = processor.tokenizer("<|im_start|>assistant\nCandidate: ", return_tensors="pt").input_ids

        images = [Image.open(img) for img in panoramas]
        candidate_images = [Image.open(img) for img in candidates]
        #candidate_images = [Image.open(img).resize((self.width, self.height), Image.Resampling.LANCZOS) for img in candidates]
        images.extend(candidate_images)
        
        processed = processor(text=text, images=[images], return_tensors="pt")

        prompt_input_ids = processed["input_ids"]
        input_ids = torch.cat([prompt_input_ids, label_start], dim=1)
        
        attention_mask = torch.ones(1, input_ids.shape[1])
        processed["input_ids"] = input_ids
        processed["attention_mask"] = attention_mask
        
        return processed


def format_prompt(images_path, path_id, route_instruction, step_id, distance_traveled, candidates, processor, system_prompt):
    # should be in the order: panorama_history, current_panorama, candidates views from left to right
    images = os.listdir(images_path)
    panoramas = [os.path.join(images_path, img) for img in images if img.startswith("pano")]
    panoramas = sorted(panoramas, key=lambda x: int(x.split("_")[-1].split(".")[-2]))

    # these are probably sorted by default, however you might need to check
    candidate_images = [os.path.join(images_path, img) for img in images if img.startswith("pano") == False]
    candidate_images = sorted(candidate_images, key=lambda x: int(x.split("_")[-1].split(".")[0]))
    
    current_panorama = panoramas.pop(-1)

    # route instruction, current step, cumulative distance
    content = [
        {
            "type" : "text",
            "text" : f"Route instruction: {route_instruction}\nCurrent step: {step_id}\nCumulative Distance Traveled: {distance_traveled} meters\n\nPanorama Images from Previous Steps:"
        }
    ]

    # panorama from previous steps
    for i, img in enumerate(panoramas):
        content.append({
            "type" : "text",
            "text" : f"\n\tPanorama at step: {i}: "
        })
        content.append({
            "type" : "image",
            "image" : img
        })

    if len(panoramas) == 0:
        content[0]["text"] += f"[]"

    # current panorama
    content.append({
        "type" : "text",
        "text" : f"\n\nCurrent Panorama Image:\n\t"
    })

    content.append({
        "type" : "image",
        "image" : current_panorama
    })

    # candidate directions
    content.append({
        "type" : "text",
        "text" : "\n\nCandidate Directions:"
    })

    for i, candidate in enumerate(candidates):
        relative_angle = round(candidate["relative_angle"], 0)
        distance = round(candidate["distance"], 2)
        direction = "Left" if relative_angle < 0 else "Right"
        
        content.append({
            "type" : "text",
            "text" : f"\n\tCandidate: {i}:\n\t\tRelative angle: {abs(relative_angle)} degrees to the {direction}\n\t\tDistance: {distance} meters\n\t\tview: "
        })

        content.append({
            "type" : "image",
            "image" : candidate_images[i]
        })


    # adds candidate STOP and the select cnadidate view 
    content.append({
        "type" : "text",
        "text" : "\n\tCandidate: Stop\n\nNow, analyze the route instruction, your current position, and the available candidate directions. Select the candidate that best matches the instruction and helps you continue along the correct path. Answer on the format: Candidate: (and then the number)"
    })

    messages = [
        {"role" : "system", "content" : [{"type" : "text", "text" : system_prompt}]},
        {"role" : "user", "content" : content},
    ]

    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)

    panoramas.extend([current_panorama])

    formatted_sample = {}
    formatted_sample["text"] = text
    formatted_sample["candidates"] = candidate_images
    formatted_sample["panoramas"] = panoramas

    formatted_data = [formatted_sample] 
    formatted_data = DT.from_list(formatted_data)
    return formatted_data


processor = AutoProcessor.from_pretrained("Vebbern/Qwen2.5-VL-3B-R2R-panoramic")
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Vebbern/Qwen2.5-VL-3B-R2R-panoramic",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="cuda"
)

# remember to set the correct image resolution (however a higher might still work as the vision encoder is not trained)
collate_fn = CollateFunctor(processor, 320, 240)

# Load mandatory system prompt
with open("system_prompt.txt", "r") as f:
    system_prompt = f.read()

path_id = 4332 # id for the R2R path
route_instruction = "Walk to the other end of the lobby and wait near the exit. "
images_path = f"./images/{path_id}"
step_id = 0
cumulative_distance = 0
candidates = {
    "0" : {
        "relative_angle" -60.62797609213225,
        "relative_direction": "Left",
        "distance": 2.3325929641723633
    },
    "1": {
            "relative_angle": -0.00397697185949581,
            "relative_direction": "Front",
            "distance": 4.637096405029297
        },
    "2": {
        "relative_angle": 25.24592108757226,
        "relative_direction": "Front",
        "distance": 3.3661904335021973
        }
}

prompt = format_prompt(images_path, path_id, route_instruction, step_id, cumulative_distance, candidates, processor, system_prompt)

dataset = CustomDataset(prompt)
data_loader = DataLoader(
    dataset,
    batch_size=1,
    collate_fn=collate_fn
)

# Run inference
for batch in data_loader:
    batch.to("cuda")
            
    outputs = model(**batch)
    argmax = torch.argmax(outputs.logits, dim=2)[0]
    model_prediction = processor.decode(argmax[-1]) # is -1 because it does not predict one more
    print(f"Predicted action: {model_prediction}")

```

> ⚠️ Sorry for the rough code — the goal here is to show how the system prompt and inputs should be structured for inference. The system prompt is included in the repo.


## 📊 Evaluation Results

The model was evaluated on the standard Room-to-Room (R2R) validation sets using the Matterport3D simulator. Performance is measured using the standard VLN (Vision-and-Language Navigation) metrics.

| Metric                  | Val Seen | Val Unseen | Test  |
|-------------------------|----------|------------|-------|
| Path Length (↓)         | 9.98     | 9.83      | 9.96 |
| Navigation Error (↓)    | 5.69     | 6.65       | 6.53  |
| Oracle Success Rate (↑) | 56%      | 46%        | 50%   |
| Success Rate (↑)        | 50%      | 38%        | 41%   |
| SPL (↑)                 | 47%      | 35%        | 38%   |

### 🧾 Metric Definitions
- **Navigation Error**: Mean distance from the goal when the agent stops.
- **Success Rate**: Percentage of episodes where the agent ends within 3 meters of the goal.
- **SPL (Success weighted by Path Length)**: Penalizes long or inefficient paths.
- **Oracle Success**: If the agent had stopped at its closest point to the goal.

### 📝 Remarks

This model performs far behind R2R State-of-the-art models, likely due to a combination of factors such as underlying model archtiecture, training strategy, and panoramic representation. 

## 🔁 Related Models
There also exists a low-level action space eqivalent of this model.
- **Low-Level Action Space Version**: [Qwen2.5-VL-3B-R2R-low-level](https://huggingface.co/Vebbern/Qwen2.5-VL-3B-R2R-low-level)

## 🪪 License

This model is licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).