---
language: en
license: apache-2.0
library_name: transformers
pipeline_tag: image-to-text
tags:
  - llava
  - llava-1.5
  - vision-language-model
  - multimodal
  - peft
  - lora
  - transformers
  - image-to-text
  - scene-understanding
  - navigation-guidance
  - assistive-ai
  - visually-impaired
base_model: llava-hf/llava-1.5-7b-hf
---

# LLaVA-via: Fine-Tuned LoRA Adapter for Scene Narration and Navigation Guidance

## Overview

**LLaVA-via** is a **PEFT LoRA adapter** fine-tuned on top of **LLaVA-1.5-7B** to improve scene narration and navigation guidance for assistive vision-language applications.

The objective of this work is to generate contextually grounded, coherent, and navigation-aware responses that can support visually impaired assistance scenarios. Rather than answering general visual questions, the adapter is optimized to describe surrounding environments and provide guidance-oriented information from input images.

This repository contains **only the LoRA adapter** and associated tokenizer/processor configuration files. It does **not** contain the full LLaVA model weights.

---

# Base Model

This adapter must be loaded on top of the following Hugging Face model:

**Base Model**

```
llava-hf/llava-1.5-7b-hf
```

The base model is **not included** in this repository.

---

# Repository Contents

This repository includes:

* `adapter_model.safetensors` — Fine-tuned LoRA adapter weights
* `adapter_config.json` — PEFT configuration
* Tokenizer files
* Processor configuration
* Image preprocessor configuration
* Chat template
* Model card (README)

Training checkpoints, optimizer states, and scheduler files are intentionally excluded.

---

# Intended Use

LLaVA-via is intended for research and development involving assistive vision-language systems, including:

* Scene narration
* Environmental understanding
* Navigation guidance
* Assistive AI research
* Vision-language model benchmarking

Example applications include:

* Assistive mobile applications
* Smart glasses
* Vision-language research
* Accessibility-focused AI systems
* Human-centered AI

---

# Fine-Tuning Method

* **Architecture:** LLaVA-1.5-7B
* **Fine-tuning:** PEFT LoRA
* **Task:** Vision-Language Conditional Generation
* **LoRA Rank (r):** 16
* **LoRA Alpha:** 32
* **Target Modules:** `q_proj`, `v_proj`
* **LoRA Dropout:** 0.05

---

# Evaluation

The fine-tuned adapter was evaluated using multiple automatic metrics covering lexical similarity, semantic similarity, vision-language alignment, and hallucination analysis.

| Metric                             |          Score |
| ---------------------------------- | -------------: |
| BLEU-1                             |         0.3125 |
| BLEU-2                             |         0.1842 |
| BLEU-3                             |         0.1228 |
| BLEU-4                             |         0.0860 |
| METEOR                             |         0.3165 |
| ROUGE-L                            |         0.2755 |
| **BERTScore-F1**                   |     **0.8872** |
| **SBERTScore**                     |     **0.7116** |
| CLIPScore (ViT-B/32)               |         0.2839 |
| CLIPScore (ViT-B/16)               |         0.2329 |
| CLIPScore (ViT-L/14)               |         0.1828 |
| PickScore (CLIP-ViT-H-14)          |         0.2030 |
| **SPECS-VIA Score**                | **3.2770 / 5** |
| **PickScore-VIA**                  |     **0.8132** |
| **CHAIR-i (Semantically Refined)** |      **0.410** |

The evaluation includes lexical overlap metrics (BLEU, METEOR, ROUGE-L), semantic similarity metrics (BERTScore and SBERTScore), vision-language alignment metrics (CLIPScore and PickScore), a task-specific evaluation (SPECS-VIA and PickScore-VIA), and hallucination analysis using the semantically refined CHAIR-i metric.

---

# Loading the Adapter

The adapter can be loaded using the Hugging Face Transformers and PEFT libraries.

```python
from transformers import AutoProcessor, LlavaForConditionalGeneration
from peft import PeftModel

base_model = LlavaForConditionalGeneration.from_pretrained(
    "llava-hf/llava-1.5-7b-hf"
)

processor = AutoProcessor.from_pretrained(
    "llava-hf/llava-1.5-7b-hf"
)

model = PeftModel.from_pretrained(
    base_model,
    "SobanHM/LLaVA-via"
)
```

---

# Example Use Cases

The adapter is designed for prompts such as:

* Describe the surrounding environment.
* Identify important objects in the scene.
* Provide navigation guidance.
* Describe possible obstacles.
* Explain the scene for a visually impaired user.

---

# Limitations

This repository provides a fine-tuned PEFT LoRA adapter evaluated using offline automatic metrics.

The reported evaluation does **not** include:

* Real-time deployment on electronic white canes
* Smart glasses deployment
* Mobile-device latency evaluation
* Embedded hardware benchmarking
* User studies involving visually impaired participants

Accordingly, inference latency, energy consumption, real-time responsiveness, and human-subject usability remain outside the scope of the reported experiments.

---

# Citation

If you use this repository in your research, please cite this work and the original LLaVA and PEFT publications.

A BibTeX entry for this work may be added once the associated thesis or publication becomes available.

---

# Authors

**Soban Hussain**
Department of Computer Science
Sukkur IBA University

**Praih Alias Faiza**
Department of Computer Science
Sukkur IBA University

**Tasmia**
Department of Computer Science
Sukkur IBA University

---

# Acknowledgements

This work builds upon the following open-source projects:

* LLaVA
* Hugging Face Transformers
* PEFT
* PyTorch

We gratefully acknowledge the developers and research community whose contributions made this work possible.