File size: 2,348 Bytes
8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 c8a4342 8367239 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 | ---
license: mit
tags:
- visual-question-answering
- factor-graph-attention
- multimodal
library_name: transformers
---
# Open-ended VQA on factor graph attention
VQA v1, open-ended: the question and the image are attended jointly by the
attention layer of [Factor Graph Attention](https://arxiv.org/abs/1904.05880)
(CVPR 2019), and the answer is chosen from a 3000-answer vocabulary.
Code: [github.com/idansc/fga](https://github.com/idansc/fga).
## Results
Trained on COCO train2014, evaluated on val2014, with 36 bottom-up region
features. Scored with the official metric — answer normalization, and the average
over the ten leave-one-annotator-out subsets.
| objective | VQA accuracy |
| --- | --- |
| **soft_ce** (this checkpoint) | **61.55** |
| bce | 60.71 |
| ce (single label) | 60.47 |
VQA is graded rather than single-label: ten annotators answer each question and an
answer earns `min(matches/3, 1)`. Supervising those scores rather than one
"correct" id is worth about a point. The sigmoid-and-binary-cross-entropy form
recommended by the [2017 challenge writeup](https://arxiv.org/abs/1708.02711) also
helps, but is 0.8 behind the softmax form here — with a 3000-way vocabulary read
out by an argmax, keeping the answers competing suits the evaluation better than
scoring them independently.
## Usage
```python
from fga.tasks.vqa import OpenEndedVQAModel
model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa")
out = model(question_input_ids=q, image_features=v)
answer_id = out.logits.argmax(-1)
```
## Note on the multiple-choice model
The repository also contains `HighOrderAttentionForVQA`, the port of
[High-Order Attention Models for VQA](https://arxiv.org/abs/1711.04323)
(NeurIPS 2017), which adds the candidate answers as a third modality and a ternary
factor over (region, word, answer) triples. It is kept for that factor and is not
published here: it scores 61.4, below this model, despite being handed eighteen
candidates to choose between. Something in it is wrong and has not been found.
## Citation
```bibtex
@inproceedings{schwartz2019factor,
title={Factor graph attention},
author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={2039--2048},
year={2019}
}
```
|