README.md

---
license: mit
tags:
- visual-question-answering
- factor-graph-attention
- multimodal
library_name: transformers
---

# Open-ended VQA on factor graph attention

VQA v1, open-ended: the question and the image are attended jointly by the
attention layer of [Factor Graph Attention](https://arxiv.org/abs/1904.05880)
(CVPR 2019), and the answer is chosen from a 3000-answer vocabulary.

Code: [github.com/idansc/fga](https://github.com/idansc/fga).

## Results

Trained on COCO train2014, evaluated on val2014, with 36 bottom-up region
features. Scored with the official metric — answer normalization, and the average
over the ten leave-one-annotator-out subsets.

| objective | VQA accuracy |
| --- | --- |
| **soft_ce** (this checkpoint) | **61.55** |
| bce | 60.71 |
| ce (single label) | 60.47 |

VQA is graded rather than single-label: ten annotators answer each question and an
answer earns `min(matches/3, 1)`. Supervising those scores rather than one
"correct" id is worth about a point. The sigmoid-and-binary-cross-entropy form
recommended by the [2017 challenge writeup](https://arxiv.org/abs/1708.02711) also
helps, but is 0.8 behind the softmax form here — with a 3000-way vocabulary read
out by an argmax, keeping the answers competing suits the evaluation better than
scoring them independently.

## Usage

```python
from fga.tasks.vqa import OpenEndedVQAModel

model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa")
out = model(question_input_ids=q, image_features=v)
answer_id = out.logits.argmax(-1)
```

## Note on the multiple-choice model

The repository also contains `HighOrderAttentionForVQA`, the port of
[High-Order Attention Models for VQA](https://arxiv.org/abs/1711.04323)
(NeurIPS 2017), which adds the candidate answers as a third modality and a ternary
factor over (region, word, answer) triples. It is kept for that factor and is not
published here: it scores 61.4, below this model, despite being handed eighteen
candidates to choose between. Something in it is wrong and has not been found.

## Citation

```bibtex
@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}
```