	NDCG	MRR	R@1	R@5	R@10	Mean rank
best single member	56.07	65.46	51.76	82.51	90.47	4.01
5xFGA, score-averaged	60.86	68.43	55.26	85.06	92.52	3.47
5xFGA, rank-averaged	60.82	67.37	53.90	84.07	92.11	3.56

Published 5xFGA: MRR 69, R@1 56%.

Averaging scores works better than averaging ranks for members of one architecture, whose scores already share a scale. Ranks help when the members disagree in confidence — mixing these with a dense-finetuned model gains 0.7 NDCG that way.

Two things that did not help: selecting each seed's best-MRR checkpoint instead of its last gave 68.27, and stacking all 26 checkpoints of the five runs gave 68.40. The diversity has to come from the seeds.

Usage

python

from fga import FGAForVisualDialog

members = [FGAForVisualDialog.from_pretrained("Idan/fga-ensemble", subfolder=name)
           for name in ["frcnn", "seed1", "seed2", "seed3", "seed4"]]

Or evaluate the ensemble directly:

bash

python scripts/ensemble_eval.py --models <member dirs> \
    --image_features_path data/frcnn_features_new.h5 --combine score rank

Citation

bibtex

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}