license: mit
tags:
- visual-question-answering
- factor-graph-attention
- multimodal
library_name: transformers
Open-ended VQA on factor graph attention
VQA v1, open-ended: the question and the image are attended jointly by the attention layer of Factor Graph Attention (CVPR 2019), and the answer is chosen from a 3000-answer vocabulary.
Code: github.com/idansc/fga.
Results
Trained on COCO train2014, evaluated on val2014, with 36 bottom-up region features. Scored with the official metric — answer normalization, and the average over the ten leave-one-annotator-out subsets.
| objective | VQA accuracy |
|---|---|
| soft_ce (this checkpoint) | 61.55 |
| bce | 60.71 |
| ce (single label) | 60.47 |
VQA is graded rather than single-label: ten annotators answer each question and an
answer earns min(matches/3, 1). Supervising those scores rather than one
"correct" id is worth about a point. The sigmoid-and-binary-cross-entropy form
recommended by the 2017 challenge writeup also
helps, but is 0.8 behind the softmax form here — with a 3000-way vocabulary read
out by an argmax, keeping the answers competing suits the evaluation better than
scoring them independently.
Usage
from fga.tasks.vqa import OpenEndedVQAModel
model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa")
out = model(question_input_ids=q, image_features=v)
answer_id = out.logits.argmax(-1)
Note on the multiple-choice model
The repository also contains HighOrderAttentionForVQA, the port of
High-Order Attention Models for VQA
(NeurIPS 2017), which adds the candidate answers as a third modality and a ternary
factor over (region, word, answer) triples. It is kept for that factor and is not
published here: it scores 61.4, below this model, despite being handed eighteen
candidates to choose between. Something in it is wrong and has not been found.
Citation
@inproceedings{schwartz2019factor,
title={Factor graph attention},
author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={2039--2048},
year={2019}
}