README.md

---
license: mit
tags:
- visual-dialog
- factor-graph-attention
- multimodal
library_name: transformers
---

# 5xFGA — Factor Graph Attention ensemble

The five members of the 5xFGA row of
[Factor Graph Attention](https://arxiv.org/abs/1904.05880) (CVPR 2019), trained
from different seeds on VisDial v1.0 with F-RCNN image features.

Code: [github.com/idansc/fga](https://github.com/idansc/fga).

## Results on VisDial v1.0 val

| | NDCG | MRR | R@1 | R@5 | R@10 | Mean rank |
| --- | --- | --- | --- | --- | --- | --- |
| best single member | 56.07 | 65.46 | 51.76 | 82.51 | 90.47 | 4.01 |
| **5xFGA, score-averaged** | 60.86 | **68.43** | **55.26** | 85.06 | 92.52 | 3.47 |
| 5xFGA, rank-averaged | 60.82 | 67.37 | 53.90 | 84.07 | 92.11 | 3.56 |

Published 5xFGA: MRR 69, R@1 56%.

Averaging scores works better than averaging ranks for members of one architecture,
whose scores already share a scale. Ranks help when the members disagree in
confidence — mixing these with a dense-finetuned model gains 0.7 NDCG that way.

Two things that did not help: selecting each seed's best-MRR checkpoint instead of
its last gave 68.27, and stacking all 26 checkpoints of the five runs gave 68.40.
The diversity has to come from the seeds.

## Usage

```python
from fga import FGAForVisualDialog

members = [FGAForVisualDialog.from_pretrained("Idan/fga-ensemble", subfolder=name)
           for name in ["frcnn", "seed1", "seed2", "seed3", "seed4"]]
```

Or evaluate the ensemble directly:

```bash
python scripts/ensemble_eval.py --models <member dirs> \
    --image_features_path data/frcnn_features_new.h5 --combine score rank
```

## Citation

```bibtex
@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}
```