README.md

---
license: mit
tags:
- visual-question-answering
- factor-graph-attention
- multimodal
library_name: transformers
---

# Open-ended VQA, trained on train2014 + val2014

The protocol behind VQA v1's published test-dev numbers: train on both annotated
splits, evaluate on test-dev. Built on the attention layer of
[Factor Graph Attention](https://arxiv.org/abs/1904.05880) (CVPR 2019).

Code: [github.com/idansc/fga](https://github.com/idansc/fga).

## No score, by construction

VQA v1's test answers were never released —
[visualqa.org](https://visualqa.org/vqa_v1_download.html) distributes annotations
for train2014 and val2014 only — so this model has no number attached to it. One
exists only after predictions are submitted to the evaluation server.

For a model with a verifiable score, see
[Idan/fga-vqa](https://huggingface.co/Idan/fga-vqa): the same architecture trained
on train2014 alone, scoring **61.97** on val2014 under the official metric.

## Training

351,596 questions (230,084 train + 121,512 val), 36 bottom-up region features, 20
epochs, batch 512, `soft_ce` against the VQA score each answer earns from the ten
annotators — worth about a point over a single label, and about 0.8 over the
sigmoid-and-binary-cross-entropy form the 2017 challenge writeup recommends.

## Usage

```python
from fga.tasks.vqa import OpenEndedVQAModel

model = OpenEndedVQAModel.from_pretrained("Idan/fga-vqa-trainval")
out = model(question_input_ids=q, image_features=v)
```

To produce a submission file for the evaluation server:

```bash
python scripts/predict_vqa_test.py --model Idan/fga-vqa-trainval --vqa_dir vqa \
    --questions vqa/raw/OpenEnded_mscoco_test2015_questions.json \
    --features vqa/test_features.h5 --output test2015_results.json
```

That script tokenizes with the training vocabulary and applies the same per-region
feature normalization used during training; getting either wrong quietly corrupts
a submission.

## Citation

```bibtex
@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}

@inproceedings{schwartz2017high,
  title={High-Order Attention Models for Visual Question Answering},
  author={Schwartz, Idan and Schwing, Alexander G and Hazan, Tamir},
  booktitle={Advances in Neural Information Processing Systems},
  year={2017}
}
```