metadata
license: mit
tags:
- visual-dialog
- factor-graph-attention
- multimodal
library_name: transformers
5xFGA — Factor Graph Attention ensemble
The five members of the 5xFGA row of Factor Graph Attention (CVPR 2019), trained from different seeds on VisDial v1.0 with F-RCNN image features.
Code: github.com/idansc/fga.
Results on VisDial v1.0 val
| NDCG | MRR | R@1 | R@5 | R@10 | Mean rank | |
|---|---|---|---|---|---|---|
| best single member | 56.07 | 65.46 | 51.76 | 82.51 | 90.47 | 4.01 |
| 5xFGA, score-averaged | 60.86 | 68.43 | 55.26 | 85.06 | 92.52 | 3.47 |
| 5xFGA, rank-averaged | 60.82 | 67.37 | 53.90 | 84.07 | 92.11 | 3.56 |
Published 5xFGA: MRR 69, R@1 56%.
Averaging scores works better than averaging ranks for members of one architecture, whose scores already share a scale. Ranks help when the members disagree in confidence — mixing these with a dense-finetuned model gains 0.7 NDCG that way.
Two things that did not help: selecting each seed's best-MRR checkpoint instead of its last gave 68.27, and stacking all 26 checkpoints of the five runs gave 68.40. The diversity has to come from the seeds.
Usage
from fga import FGAForVisualDialog
members = [FGAForVisualDialog.from_pretrained("Idan/fga-ensemble", subfolder=name)
for name in ["frcnn", "seed1", "seed2", "seed3", "seed4"]]
Or evaluate the ensemble directly:
python scripts/ensemble_eval.py --models <member dirs> \
--image_features_path data/frcnn_features_new.h5 --combine score rank
Citation
@inproceedings{schwartz2019factor,
title={Factor graph attention},
author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
pages={2039--2048},
year={2019}
}