Hugging Face's logo

rishii100
/
knowledge-guided-vqa-models

Visual Question Answering
PyTorch
English
vqa
knowledge-guided
vit
flan-t5
ok-vqa
multimodal
Model card Files
xet
 Community
knowledge-guided-vqa-models
File size: 2,855 Bytes
e8a7653
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
---
license: mit
language:
  - en
tags:
  - vqa
  - visual-question-answering
  - knowledge-guided
  - vit
  - flan-t5
  - ok-vqa
  - multimodal
  - pytorch
datasets:
  - ok-vqa
pipeline_tag: visual-question-answering
library_name: pytorch
---

# Knowledge-Guided VQA Models

This repository contains trained model weights for **Knowledge-Guided Visual Question Answering** on the [OK-VQA](https://okvqa.allenai.org/) dataset.

These models explore different strategies for injecting external LLM-derived knowledge (visual focus cues and semantic knowledge) into a frozen Vision Transformer (ViT) + Flan-T5 pipeline for open-ended VQA.

## 🏗️ Architecture

| Component | Model |
|:---|:---|
| Vision Encoder | `google/vit-base-patch16-224` |
| Text Encoder | CLIP Text (`openai/clip-vit-base-patch16`) |
| Decoder (Answer Generator) | `google/flan-t5-small` |
| Dataset | OK-VQA (train + val split) |

## 📦 Available Models

### 1. `best_baseline_model.pth` (309 MB)
- **Strategy:** No external knowledge. Standard ViT → CLS projection → T5 decoder.
- **Purpose:** Serves as the baseline for comparison.

### 2. `best_fusion_model.pth` (313 MB)
- **Strategy:** Late Fusion. Concatenates ViT CLS token with CLIP-encoded guidance vector, then projects to T5.
- **Knowledge Source:** LLM-generated visual focus + semantic knowledge via CLIP text encoder.

### 3. `best_attention_model.pth` (325 MB)
- **Strategy:** Attention Modification. Injects per-layer guidance projections into the top 6 ViT self-attention layers.
- **Knowledge Source:** LLM-generated visual focus + semantic knowledge via CLIP text encoder.

## 🔧 Usage

```python
import torch
from transformers import ViTModel, AutoTokenizer, AutoModelForSeq2SeqLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Load base models
t5_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")
t5 = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small").to(device)
vit = ViTModel.from_pretrained("google/vit-base-patch16-224").to(device)

# Define your model class (see notebook for full definitions)
# Then load weights:
model = YourModelClass().to(device)
model.load_state_dict(torch.load("best_baseline_model.pth", map_location=device))
```

## 📊 Training Details

- **Optimizer:** Adam (lr=3e-5)
- **Early Stopping:** Patience = 3-4 epochs
- **Max Epochs:** 15
- **Batch Size:** 8
- **Evaluation Metrics:** Exact Match, Edit Similarity, Token F1, BERTScore

## 📄 Citation

If you use these models in your research, please cite:

```bibtex
@misc{knowledge-guided-vqa-2026,
  title={Knowledge-Guided Visual Question Answering with Adaptive Vision-Language Integration},
  author={Aneerban},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/rishii100/knowledge-guided-vqa-models}
}
```

## 📜 License

MIT License