File size: 1,945 Bytes
9ff950f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | ---
license: mit
tags:
- visual-dialog
- factor-graph-attention
- multimodal
library_name: transformers
---
# 5xFGA — Factor Graph Attention ensemble
The five members of the 5xFGA row of
[Factor Graph Attention](https://arxiv.org/abs/1904.05880) (CVPR 2019), trained
from different seeds on VisDial v1.0 with F-RCNN image features.
Code: [github.com/idansc/fga](https://github.com/idansc/fga).
## Results on VisDial v1.0 val
| | NDCG | MRR | R@1 | R@5 | R@10 | Mean rank |
| --- | --- | --- | --- | --- | --- | --- |
| best single member | 56.07 | 65.46 | 51.76 | 82.51 | 90.47 | 4.01 |
| **5xFGA, score-averaged** | 60.86 | **68.43** | **55.26** | 85.06 | 92.52 | 3.47 |
| 5xFGA, rank-averaged | 60.82 | 67.37 | 53.90 | 84.07 | 92.11 | 3.56 |
Published 5xFGA: MRR 69, R@1 56%.
Averaging scores works better than averaging ranks for members of one architecture,
whose scores already share a scale. Ranks help when the members disagree in
confidence — mixing these with a dense-finetuned model gains 0.7 NDCG that way.
Two things that did not help: selecting each seed's best-MRR checkpoint instead of
its last gave 68.27, and stacking all 26 checkpoints of the five runs gave 68.40.
The diversity has to come from the seeds.
## Usage
```python
from fga import FGAForVisualDialog
members = [FGAForVisualDialog.from_pretrained("Idan/fga-ensemble", subfolder=name)
for name in ["frcnn", "seed1", "seed2", "seed3", "seed4"]]
```
Or evaluate the ensemble directly:
```bash
python scripts/ensemble_eval.py --models <member dirs> \
--image_features_path data/frcnn_features_new.h5 --combine score rank
```
## Citation
```bibtex
@inproceedings{schwartz2019factor,
title={Factor graph attention},
author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={2039--2048},
year={2019}
}
```
|