rishii100 commited on
Commit
feaea21
ยท
verified ยท
1 Parent(s): e8a7653

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +146 -50
README.md CHANGED
@@ -2,91 +2,187 @@
2
  license: mit
3
  language:
4
  - en
 
 
 
 
 
 
 
 
 
 
 
 
5
  tags:
 
6
  - vqa
7
  - visual-question-answering
8
  - knowledge-guided
 
 
 
9
  - vit
10
  - flan-t5
 
11
  - ok-vqa
12
  - multimodal
13
  - pytorch
14
- datasets:
15
- - ok-vqa
16
- pipeline_tag: visual-question-answering
17
- library_name: pytorch
18
  ---
19
 
20
- # Knowledge-Guided VQA Models
 
 
 
 
21
 
22
- This repository contains trained model weights for **Knowledge-Guided Visual Question Answering** on the [OK-VQA](https://okvqa.allenai.org/) dataset.
23
 
24
- These models explore different strategies for injecting external LLM-derived knowledge (visual focus cues and semantic knowledge) into a frozen Vision Transformer (ViT) + Flan-T5 pipeline for open-ended VQA.
25
 
26
- ## ๐Ÿ—๏ธ Architecture
 
 
 
 
 
 
 
 
27
 
28
  | Component | Model |
29
- |:---|:---|
30
- | Vision Encoder | `google/vit-base-patch16-224` |
31
- | Text Encoder | CLIP Text (`openai/clip-vit-base-patch16`) |
32
- | Decoder (Answer Generator) | `google/flan-t5-small` |
33
- | Dataset | OK-VQA (train + val split) |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
- ## ๐Ÿ“ฆ Available Models
36
 
37
- ### 1. `best_baseline_model.pth` (309 MB)
38
- - **Strategy:** No external knowledge. Standard ViT โ†’ CLS projection โ†’ T5 decoder.
39
- - **Purpose:** Serves as the baseline for comparison.
 
 
40
 
41
- ### 2. `best_fusion_model.pth` (313 MB)
42
- - **Strategy:** Late Fusion. Concatenates ViT CLS token with CLIP-encoded guidance vector, then projects to T5.
43
- - **Knowledge Source:** LLM-generated visual focus + semantic knowledge via CLIP text encoder.
44
 
45
- ### 3. `best_attention_model.pth` (325 MB)
46
- - **Strategy:** Attention Modification. Injects per-layer guidance projections into the top 6 ViT self-attention layers.
47
- - **Knowledge Source:** LLM-generated visual focus + semantic knowledge via CLIP text encoder.
48
 
49
- ## ๐Ÿ”ง Usage
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  ```python
52
  import torch
53
- from transformers import ViTModel, AutoTokenizer, AutoModelForSeq2SeqLM
54
 
 
 
55
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
 
 
 
 
 
 
 
 
 
 
56
 
57
- # Load base models
58
- t5_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")
59
- t5 = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small").to(device)
60
- vit = ViTModel.from_pretrained("google/vit-base-patch16-224").to(device)
61
 
62
- # Define your model class (see notebook for full definitions)
63
- # Then load weights:
64
- model = YourModelClass().to(device)
65
- model.load_state_dict(torch.load("best_baseline_model.pth", map_location=device))
66
- ```
 
 
 
 
 
 
67
 
68
- ## ๐Ÿ“Š Training Details
69
 
70
- - **Optimizer:** Adam (lr=3e-5)
71
- - **Early Stopping:** Patience = 3-4 epochs
72
- - **Max Epochs:** 15
73
- - **Batch Size:** 8
74
- - **Evaluation Metrics:** Exact Match, Edit Similarity, Token F1, BERTScore
75
 
76
- ## ๐Ÿ“„ Citation
77
 
78
- If you use these models in your research, please cite:
79
 
80
  ```bibtex
81
- @misc{knowledge-guided-vqa-2026,
82
- title={Knowledge-Guided Visual Question Answering with Adaptive Vision-Language Integration},
83
- author={Aneerban},
84
- year={2026},
85
- publisher={Hugging Face},
86
- url={https://huggingface.co/rishii100/knowledge-guided-vqa-models}
 
 
87
  }
88
  ```
89
 
90
- ## ๐Ÿ“œ License
91
 
92
- MIT License
 
2
  license: mit
3
  language:
4
  - en
5
+ base_model:
6
+ - google/vit-base-patch16-224
7
+ - openai/clip-vit-base-patch16
8
+ - google/flan-t5-small
9
+ datasets:
10
+ - ok-vqa
11
+ metrics:
12
+ - exact_match
13
+ - f1
14
+ - bertscore
15
+ pipeline_tag: visual-question-answering
16
+ library_name: pytorch
17
  tags:
18
+ - arxiv:2610.17079
19
  - vqa
20
  - visual-question-answering
21
  - knowledge-guided
22
+ - late-fusion
23
+ - attention-injection
24
+ - vision-transformer
25
  - vit
26
  - flan-t5
27
+ - clip
28
  - ok-vqa
29
  - multimodal
30
  - pytorch
 
 
 
 
31
  ---
32
 
33
+ # Knowledge-Guided Visual Question Answering Models
34
+
35
+ This repository contains trained model checkpoints for **Knowledge-Guided Visual Question Answering (VQA)** on an augmented subset of the OK-VQA dataset.
36
+
37
+ ๐Ÿ“„ **Research paper:** [Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?](https://arxiv.org/abs/2610.17079)
38
 
39
+ The paper investigates whether deeper integration of LLM-generated knowledge improves VQA performance under data-scarce fine-tuning conditions. We compare decision-level late fusion with attention-matrix injection using a shared ViT + FLAN-T5 framework.
40
 
41
+ ## Research Highlights
42
 
43
+ - Compare shallow, decision-level late fusion against deep attention injection.
44
+ - Generate structured guidance with Llama 3.3 70B.
45
+ - Encode guidance using a CLIP text encoder.
46
+ - Expand the filtered training set from 3,144 to approximately 36,000 samples through question paraphrasing.
47
+ - Evaluate the baseline, late fusion, and attention injection using a consistent experimental setup.
48
+
49
+ The results show that **late fusion performs better than attention injection in this data-scarce experimental setting**. This does not imply that deeper integration is always inferior.
50
+
51
+ ## Architecture
52
 
53
  | Component | Model |
54
+ |---|---|
55
+ | Vision encoder | `google/vit-base-patch16-224` |
56
+ | Guidance text encoder | `openai/clip-vit-base-patch16` |
57
+ | Answer generation model | `google/flan-t5-small` |
58
+ | Benchmark | OK-VQA |
59
+ | LLM used to generate guidance | Llama 3.3 70B |
60
+
61
+ The pipeline generates structured guidance containing visual focus concepts, distractor elements to ignore, and external knowledge statements. A CLIP text encoder represents this guidance as an embedding used by the downstream VQA model.
62
+
63
+ ### Integration strategies
64
+
65
+ **1. Baseline โ€” No Guidance**
66
+
67
+ Uses visual features and question representations without external LLM-generated guidance.
68
+
69
+ **2. Late Fusion โ€” Shallow Integration**
70
+
71
+ Combines the global ViT representation with the encoded guidance immediately before answer decoding. The visual feature extraction itself is left unchanged.
72
+
73
+ **3. Attention Injection โ€” Deep Integration**
74
+
75
+ Injects guidance into the query and key projections of the last six ViT layers, modifying visual attention.
76
 
77
+ ## Available Checkpoints
78
 
79
+ | Checkpoint | Strategy | Approximate file size |
80
+ |---|---|---:|
81
+ | `best_baseline_model.pth` | No guidance | 309 MB |
82
+ | `best_fusion_model.pth` | Late fusion | 313 MB |
83
+ | `best_attention_model.pth` | Attention injection | 325 MB |
84
 
85
+ These files contain trained checkpoint weights. Loading them requires the matching model architecture and compatible pretrained components.
 
 
86
 
87
+ ## Evaluation Results
 
 
88
 
89
+ The following scores are reported in the paper for the validation split of the augmented, top-500-filtered OK-VQA subset. **These are not official full-benchmark OK-VQA leaderboard scores.**
90
+
91
+ | Strategy | Exact Match (%) | Edit Similarity (%) | Token F1 (%) | BERTScore |
92
+ |---|---:|---:|---:|---:|
93
+ | Baseline โ€” No Guidance | 29.44 | 40.57 | 30.97 | 91.21 |
94
+ | Late Fusion | **39.44** | **48.97** | **41.34** | **92.14** |
95
+ | Attention Injection | 31.80 | 42.66 | 33.33 | 91.00 |
96
+
97
+ ### Main finding
98
+
99
+ Late fusion achieves **39.44% exact match**, a 10.00-percentage-point improvement over the baseline. Attention injection achieves 31.80%, a 2.36-point improvement over the baseline.
100
+
101
+ These results suggest that, with limited fine-tuning data and largely frozen pretrained components, simpler knowledge integration can be more effective than modifying visual attention.
102
+
103
+ ## Training Details
104
+
105
+ - **Dataset:** Filtered OK-VQA training data with the top 500 answer classes.
106
+ - **Data augmentation:** LLM-based question paraphrasing.
107
+ - **Original filtered training set:** 3,144 samples.
108
+ - **Augmented training set:** Approximately 36,000 samples.
109
+ - **Optimizer:** Adam.
110
+ - **Learning rate:** `3e-5`.
111
+ - **Batch size:** 8โ€“16.
112
+ - **Maximum epochs:** 15.
113
+ - **Early stopping:** Validation exact match, patience 3โ€“4.
114
+ - **Evaluation metrics:** Exact Match, Edit Similarity, Token F1, and BERTScore.
115
+
116
+ The ViT and CLIP encoders remain frozen except for the explicitly modified and trainable components of the attention-injection strategy.
117
+
118
+ ## Usage
119
+
120
+ You can download a checkpoint from this repository with `huggingface_hub`:
121
+
122
+ ```python
123
+ from huggingface_hub import hf_hub_download
124
+
125
+ checkpoint_path = hf_hub_download(
126
+ repo_id="rishii100/knowledge-guided-vqa-models",
127
+ filename="best_fusion_model.pth",
128
+ )
129
+
130
+ print("Checkpoint downloaded to:", checkpoint_path)
131
+ ```
132
+
133
+ To run inference, instantiate the matching model class and load the checkpoint:
134
 
135
  ```python
136
  import torch
 
137
 
138
+ # YourLateFusionModel is a placeholder for the exact model class
139
+ # used during training. It must match the checkpoint's architecture.
140
  device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
141
+ model = YourLateFusionModel().to(device)
142
+
143
+ state_dict = torch.load(
144
+ checkpoint_path,
145
+ map_location=device,
146
+ weights_only=True,
147
+ )
148
+ model.load_state_dict(state_dict)
149
+ model.eval()
150
+ ```
151
 
152
+ **Important:** `YourLateFusionModel` is a placeholder, not an included implementation. The model class, forward pass, tokenizer configuration, and pretrained components must match the training implementation. The checkpoint alone is not a complete, standalone inference pipeline.
 
 
 
153
 
154
+ ## Limitations
155
+
156
+ - Results are based on the experimental subset and validation protocol described in the paper.
157
+ - Performance may differ on the full official OK-VQA benchmark or under different training-data regimes.
158
+ - Attention injection can help some question categories while harming others.
159
+ - The checkpoint files do not, by themselves, provide the complete training pipeline or guarantee reproducibility without the corresponding implementation and configuration.
160
+ - The pretrained components retain their respective upstream licenses and terms.
161
+
162
+ ## Paper and Citation
163
+
164
+ **Title:** Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?
165
 
166
+ **Authors:** Aneerban Saha, Vijay Kumar, Sameer Madhav D., Priyam Srivastava, and Dhruv Bansal.
167
 
168
+ **Conference:** Accepted at ADCIS 2026.
 
 
 
 
169
 
170
+ **arXiv:** [2610.17079](https://arxiv.org/abs/2610.17079)
171
 
172
+ ### BibTeX
173
 
174
  ```bibtex
175
+ @misc{saha2026rethinkingvqa,
176
+ title = {Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?},
177
+ author = {Saha, Aneerban and Kumar, Vijay and Madhav D., Sameer and Srivastava, Priyam and Bansal, Dhruv},
178
+ year = {2026},
179
+ eprint = {2610.17079},
180
+ archivePrefix = {arXiv},
181
+ primaryClass = {cs.CV},
182
+ url = {https://arxiv.org/abs/2610.17079}
183
  }
184
  ```
185
 
186
+ ## License
187
 
188
+ This repository is marked as MIT licensed.