Update README.md

README.md

@@ -5,17 +5,215 @@ tags:
 - transformers
 - unsloth
 - qwen2_5_vl_text
+- lora
 - trl
+- vision-language
+- autonomous-driving
+- robotics
+- multi-view spatial-reasoning
 license: apache-2.0
+datasets:
+  - nvidia/PhysicalAI-Autonomous-Vehicles
+pipeline_tag: image-text-to-text
 language:
 - en
 ---
 
-# Uploaded  model
 
-- **Developed by:** Thunderbird2410
-- **License:** apache-2.0
-- **Finetuned from model :** unsloth/Qwen2.5-VL-7B-Instruct
+
+
+<div align="center">
+
+# KAIØ-SIGHT
+
+**Multi-View Vision-Language Reasoning for Autonomous Robotics**
+
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-yellow)](https://huggingface.co/Thunderbird2410/KAIO-SIGHT)
+[![GitHub](https://img.shields.io/badge/GitHub-Repository-blue)](https://github.com/poornachandra24/KAIO-SIGHT)
+[![AMD ROCm](https://img.shields.io/badge/AMD-MI300X-orange)](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)
+[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](LICENSE)
+
+
+</div>
+
+## Model Description
+
+**KAIØ-SIGHT** is a fine-tuned Vision-Language Model (VLM) designed for **multi-view spatial-temporal reasoning** in autonomous robotics and driving scenarios. Built on top of `Qwen2.5-VL-7B-Instruct`, this model learns to fuse multi-camera video feeds into a coherent understanding of 360° environments.
+This repo contains only the fine-tuned Lora adapters. Please pull the base model directly.
+### Key Capabilities
+
+- 🎥 **Multi-View Fusion**: Processes synchronized feeds from up to 7 cameras (Front Wide, Front Tele, Cross Left/Right, Rear Left/Right, Rear Tele)
+- 🧠 **Spatial Reasoning**: Understands object positions, motion trajectories, and scene dynamics across camera views
+- 🚗 **Egomotion Prediction**: Predicts vehicle state including position, velocity, and rotation
+- ⏱️ **Temporal Context**: Analyzes 16-frame sliding windows to capture motion and causality
+
+## Training Details
+
+### Base Model
+- **Architecture**: Qwen2.5-VL-7B-Instruct
+- **Training Method**: LoRA (Low-Rank Adaptation) with Unsloth optimization
+- **Precision**: BFloat16
+
+### LoRA Configuration
+| Parameter | Value |
+|-----------|-------|
+| Rank | 128 |
+| Alpha | 256 |
+| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
+| Max Sequence Length | 65,536 tokens |
+
+### Training Hyperparameters
+| Parameter | Value |
+|-----------|-------|
+| Learning Rate | 1e-4 |
+| Optimizer | Paged AdamW 8-bit |
+| Effective Batch Size | 144 (48 × 3 gradient accumulation) |
+| Weight Decay | 0.01 |
+| LR Scheduler | Cosine with 10% warmup |
+| Epochs | 1 |
+
+### Hardware
+- **GPU**: AMD Instinct MI300X (192GB VRAM)
+- **Framework**: ROCm 6.4 with custom kernel optimizations
+
+## Dataset
+
+
+Trained on the [NVIDIA PhysicalAI Autonomous Vehicles](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles) dataset featuring:
+
+- Multi-camera video streams from 7 synchronized cameras
+- Egomotion labels (position, velocity, rotation)
+- High-quality urban driving scenarios
+
+### Camera Configuration (7-cam Setup)
+```
+┌─────────────┬─────────────┬─────────────┐
+│ Front Wide  │ Front Tele  │   (empty)   │
+│   120° FOV  │   30° FOV   │             │
+├─────────────┼─────────────┼─────────────┤
+│ Cross Left  │   (ego)     │ Cross Right │
+│   120° FOV  │             │   120° FOV  │
+├─────────────┼─────────────┼─────────────┤
+│ Rear Left   │ Rear Tele   │ Rear Right  │
+│   70° FOV   │   30° FOV   │   70° FOV   │
+└─────────────┴─────────────┴─────────────┘
+```
+
+## Intended Use
+
+### Primary Use Cases
+- 🤖 Autonomous robotics research and development
+- 🚙 Driving scenario understanding and prediction
+- 📊 Multi-view video understanding research
+- 🔬 Vision-language model experimentation
+
+### Out-of-Scope Uses
+- ⚠️ Production autonomous vehicle deployment (experimental research only)
+- ⚠️ Safety-critical applications without additional validation
+- ⚠️ Real-time inference without hardware-specific optimization
+
+## Usage
+
+### Quick Start with Transformers
+
+```python
+from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
+from peft import PeftModel
+import torch
+
+# Load base model
+base_model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
+    "Qwen/Qwen2.5-VL-7B-Instruct",
+    torch_dtype=torch.bfloat16,
+    device_map="auto"
+)
+
+# Load LoRA adapter
+model = PeftModel.from_pretrained(base_model, "Thunderbird2410/KAIO-SIGHT")
+processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
+
+# Prepare your multi-view image
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": "path/to/multi_view_image.jpg"},
+            {"type": "text", "text": "Analyze this multi-camera driving scene. Describe the surroundings and predict the vehicle's motion."}
+        ]
+    }
+]
+
+# Generate response
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=text, images=[image], return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=512)
+response = processor.decode(outputs[0], skip_special_tokens=True)
+```
+
+### With Unsloth (Recommended for Training)
+
+```python
+from unsloth import FastVisionModel
+
+model, tokenizer = FastVisionModel.from_pretrained(
+    "Thunderbird2410/KAIO-SIGHT",
+    max_seq_length=65536,
+    dtype=torch.bfloat16,
+    load_in_4bit=True  # Optional: for lower VRAM
+)
+```
+
+## Limitations
+
+- **Experimental Status**: This model is a research prototype and not production-ready
+- **Hardware Dependency**: Optimized for AMD MI300X; performance on other GPUs may vary
+- **Domain Specificity**: Trained primarily on urban driving scenarios
+- **Temporal Windows**: Best performance with 4-frame sequences matching training distribution to meet model's context window 
+
+## Model Architecture
+
+```mermaid
+graph LR
+    A[7-Camera Video] -->|Tile to Grid| B[3×3 Composite Frame]
+    B -->|16-Frame Window| C[Temporal Sequence]
+    C -->|Vision Encoder| D[Qwen2.5-VL-7B]
+    D -->|LoRA Adapters| E[Fine-tuned Model]
+    E -->|Generate| F[Egomotion + Reasoning]
+```
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{kaio-sight-2024,
+  author = {Poornachandra},
+  title = {KAIØ-SIGHT: Multi-View Vision-Language Reasoning for Autonomous Robotics},
+  year = {2024},
+  publisher = {Hugging Face},
+  url = {https://huggingface.co/Thunderbird2410/KAIO-SIGHT}
+}
+```
+
+## Acknowledgments
+
+- [Qwen Team](https://huggingface.co/Qwen) for the Qwen2.5-VL foundation model
+- [Unsloth](https://github.com/unslothai/unsloth) for efficient fine-tuning optimizations
+- [NVIDIA](https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles) for the PhysicalAI dataset
+- [AMD](https://rocm.docs.amd.com/) for ROCm and MI300X hardware support
+
+## License
+
+This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
+
+---
+
+<div align="center">
+
+**⚠️ Experimental Research Model - Use at Your Own Risk ⚠️**
+
+
+</div>
 
 This qwen2_5_vl_text model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth)