Update README.md

README.md

@@ -1,3 +1,273 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- en
+base_model:
+- Qwen/Qwen2.5-VL-3B-Instruct
+tags:
+- R2R
+- VLN
+- Room-to-Room
+- LVLM
+---
+
+# Qwen2.5-VL-3B-R2R-panoramic
+
+**Qwen2.5-VL-3B-R2R-panoramic** is a Vision-and-Language Navigation (VLN) model fine-tuned from [Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) on the [Room-to-Room (R2R)](https://bringmeaspoon.org/) dataset using the Matterport3D (MP3D) simulator. The model is trained using a panoramic action space, where the model recives a preprocessed panoramic image and a set of candidate views which each point towards a node in a Matterport3D simualtor environment.  
+
+Only the LLM component is fine-tuned — the vision encoder and cross-modal projector are kept frozen.
+
+
+## 🧠 Model Summary
+
+- **Base Model**: Qwen2.5-VL-3B-Instruct
+- **Dataset**: Room-to-Room (R2R) via the Matterport3D simulator.
+- **Image Resolution**: 320x240 for candidate images and 960×240 for panoramic images.
+- **Action Space**: Panoramic.
+
+## 🧪 Training Setup
+
+- **Frozen Modules**: Vision encoder and cross-modal projector  
+- **Fine-Tuned Module**: LLM decoder (Qwen2.5)  
+- **Optimizer**: AdamW  
+- **Batch Size**: `1` (with gradient accumulation over each episode)  
+- **Learning Rate**: `1e-5`  
+- **Weight Decay**: `0.1`  
+- **Precision**: `bfloat16`  
+- **LR Scheduler**:  Linear scheduler with warmup (first 10% of steps)  
+- **Hardware**: Trained on a single NVIDIA A100 80GB GPU  
+
+Training was done using supervised learning for next-action prediction. The model was conditioned at each step with a system prompt, panoramic RGB image observations (960×240) of current view, as well as variable amount of candidate RGB iamges (320x240), and cumulative episode history including previosu panoramas. The model was trained offline (not in the MP3D simulator) using teacher-forcing on a preprocessed R2R dataset.
+
+
+## 📦 Usage 
+```python
+import torch
+from torch.utils.data import Dataset, DataLoader
+from datasets import Dataset as DT
+from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
+from PIL import Image
+
+lass CustomDataset(Dataset):
+    def __init__(self, data):
+        self.text = data["text"]
+        self.panoramas = data["panoramas"]
+        self.candidates = data["candidates"]
+        
+    def __len__(self):
+        return len(self.text)
+    
+    def __getitem__(self, index):
+        return self.text[index], self.panoramas[index], self.candidates[index]
+
+# TODO: make the collatefunctor work with batches
+class CollateFunctor:
+    # No batch, therefore no max length
+    def __init__(self, processor, width, height):
+        self.processor = processor
+        self.width = width
+        self.height = height
+
+    def __call__(self, batch):
+        text, panoramas, candidates = batch[0]
+        label_start = processor.tokenizer("<|im_start|>assistant\nCandidate: ", return_tensors="pt").input_ids
+
+        images = [Image.open(img) for img in panoramas]
+        candidate_images = [Image.open(img) for img in candidates]
+        #candidate_images = [Image.open(img).resize((self.width, self.height), Image.Resampling.LANCZOS) for img in candidates]
+        images.extend(candidate_images)
+        
+        processed = processor(text=text, images=[images], return_tensors="pt")
+
+        prompt_input_ids = processed["input_ids"]
+        input_ids = torch.cat([prompt_input_ids, label_start], dim=1)
+        
+        attention_mask = torch.ones(1, input_ids.shape[1])
+        processed["input_ids"] = input_ids
+        processed["attention_mask"] = attention_mask
+        
+        return processed
+
+
+def format_prompt(images_path, path_id, route_instruction, step_id, distance_traveled, candidates, processor, system_prompt):
+    # should be in the order: panorama_history, current_panorama, candidates views from left to right
+    images = os.listdir(images_path)
+    panoramas = [os.path.join(images_path, img) for img in images if img.startswith("pano")]
+    panoramas = sorted(panoramas, key=lambda x: int(x.split("_")[-1].split(".")[-2]))
+
+    # these are probably sorted by default, however you might need to check
+    candidate_images = [os.path.join(images_path, img) for img in images if img.startswith("pano") == False]
+    candidate_images = sorted(candidate_images, key=lambda x: int(x.split("_")[-1].split(".")[0]))
+    
+    current_panorama = panoramas.pop(-1)
+
+    # route instruction, current step, cumulative distance
+    content = [
+        {
+            "type" : "text",
+            "text" : f"Route instruction: {route_instruction}\nCurrent step: {step_id}\nCumulative Distance Traveled: {distance_traveled} meters\n\nPanorama Images from Previous Steps:"
+        }
+    ]
+
+    # panorama from previous steps
+    for i, img in enumerate(panoramas):
+        content.append({
+            "type" : "text",
+            "text" : f"\n\tPanorama at step: {i}: "
+        })
+        content.append({
+            "type" : "image",
+            "image" : img
+        })
+
+    if len(panoramas) == 0:
+        content[0]["text"] += f"[]"
+
+    # current panorama
+    content.append({
+        "type" : "text",
+        "text" : f"\n\nCurrent Panorama Image:\n\t"
+    })
+
+    content.append({
+        "type" : "image",
+        "image" : current_panorama
+    })
+
+    # candidate directions
+    content.append({
+        "type" : "text",
+        "text" : "\n\nCandidate Directions:"
+    })
+
+    for i, candidate in enumerate(candidates):
+        relative_angle = round(candidate["relative_angle"], 0)
+        distance = round(candidate["distance"], 2)
+        direction = "Left" if relative_angle < 0 else "Right"
+        
+        content.append({
+            "type" : "text",
+            "text" : f"\n\tCandidate: {i}:\n\t\tRelative angle: {abs(relative_angle)} degrees to the {direction}\n\t\tDistance: {distance} meters\n\t\tview: "
+        })
+
+        content.append({
+            "type" : "image",
+            "image" : candidate_images[i]
+        })
+
+
+    # adds candidate STOP and the select cnadidate view 
+    content.append({
+        "type" : "text",
+        "text" : "\n\tCandidate: Stop\n\nNow, analyze the route instruction, your current position, and the available candidate directions. Select the candidate that best matches the instruction and helps you continue along the correct path. Answer on the format: Candidate: (and then the number)"
+    })
+
+    messages = [
+        {"role" : "system", "content" : [{"type" : "text", "text" : system_prompt}]},
+        {"role" : "user", "content" : content},
+    ]
+
+    text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
+
+    panoramas.extend([current_panorama])
+
+    formatted_sample = {}
+    formatted_sample["text"] = text
+    formatted_sample["candidates"] = candidate_images
+    formatted_sample["panoramas"] = panoramas
+
+    formatted_data = [formatted_sample] 
+    formatted_data = DT.from_list(formatted_data)
+    return formatted_data
+
+
+processor = AutoProcessor.from_pretrained("Vebbern/Qwen2.5-VL-3B-R2R-panoramic")
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    "Vebbern/Qwen2.5-VL-3B-R2R-panoramic",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+    device_map="cuda"
+)
+
+# remember to set the correct image resolution (however a higher might still work as the vision encoder is not trained)
+collate_fn = CollateFunctor(processor, 320, 240)
+
+# Load mandatory system prompt
+with open("system_prompt.txt", "r") as f:
+    system_prompt = f.read()
+
+path_id = 4332 # id for the R2R path
+route_instruction = "Walk to the other end of the lobby and wait near the exit. "
+images_path = f"./images/{path_id}"
+step_id = 0
+cumulative_distance = 0
+candidates = {
+    "0" : {
+        "relative_angle" -60.62797609213225,
+        "relative_direction": "Left",
+        "distance": 2.3325929641723633
+    },
+    "1": {
+            "relative_angle": -0.00397697185949581,
+            "relative_direction": "Front",
+            "distance": 4.637096405029297
+        },
+    "2": {
+        "relative_angle": 25.24592108757226,
+        "relative_direction": "Front",
+        "distance": 3.3661904335021973
+        }
+}
+
+prompt = format_prompt(images_path, path_id, route_instruction, step_id, cumulative_distance, candidates, processor, system_prompt)
+
+dataset = CustomDataset(prompt)
+data_loader = DataLoader(
+    dataset,
+    batch_size=1,
+    collate_fn=collate_fn
+)
+
+# Run inference
+for batch in data_loader:
+    batch.to("cuda")
+            
+    outputs = model(**batch)
+    argmax = torch.argmax(outputs.logits, dim=2)[0]
+    model_prediction = processor.decode(argmax[-1]) # is -1 because it does not predict one more
+    print(f"Predicted action: {model_prediction}")
+
+```
+
+> ⚠️ Sorry for the rough code — the goal here is to show how the system prompt and inputs should be structured for inference. The system prompt is included in the repo.
+
+
+## 📊 Evaluation Results
+
+The model was evaluated on the standard Room-to-Room (R2R) validation sets using the Matterport3D simulator. Performance is measured using the standard VLN (Vision-and-Language Navigation) metrics.
+
+| Metric                  | Val Seen | Val Unseen | Test  |
+|-------------------------|----------|------------|-------|
+| Path Length (↓)         | 9.98     | 9.83      | 9.96 |
+| Navigation Error (↓)    | 5.69     | 6.65       | 6.53  |
+| Oracle Success Rate (↑) | 56%      | 46%        | 50%   |
+| Success Rate (↑)        | 50%      | 38%        | 41%   |
+| SPL (↑)                 | 47%      | 35%        | 38%   |
+
+### 🧾 Metric Definitions
+- **Navigation Error**: Mean distance from the goal when the agent stops.
+- **Success Rate**: Percentage of episodes where the agent ends within 3 meters of the goal.
+- **SPL (Success weighted by Path Length)**: Penalizes long or inefficient paths.
+- **Oracle Success**: If the agent had stopped at its closest point to the goal.
+
+### 📝 Remarks
+
+This model performs far behind R2R State-of-the-art models, likely due to a combination of factors such as underlying model archtiecture, training strategy, and panoramic representation. 
+
+## 🔁 Related Models
+There also exists a low-level action space eqivalent of this model.
+- **Low-Level Action Space Version**: [Qwen2.5-VL-3B-R2R-low-level](https://huggingface.co/Vebbern/Qwen2.5-VL-3B-R2R-low-level)
+
+## 🪪 License
+
+This model is licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).