A general multimodal attention approach inspired by probabilistic graphical models.
Achieves a state-of-the-art performance (MRR) on visual dialog task.

This is the official implementation of Factor Graph Attention. (Appeared in CVPR'19). Code: idansc/fga.

Part of 2020 visual dialog challenge winning submission (https://github.com/idansc/mrr-ndcg)

This checkpoint targets MRR. For the NDCG-oriented variant, finetuned on the dense relevance annotations, see Idan/fga-ndcg (NDCG 69.07 against this model's 56.46, at the cost of MRR).

Use cases of FGA:

Video dialog, spatial interactions between frames, can be found here (https://github.com/idansc/simple-avsd)
Spatial navigation, can be found here (https://github.com/barmayo/spatial_attention)
Video retrieval, between text query and clips, can be found here (https://github.com/AmeenAli/VideoMatch)

How it works

Every modality is a set of entities with an embedding each — the 100 candidate answers, the question words, the caption words, the image regions, and the question and answer of each history round. Attention over a modality is the softmax of a sum of learned potentials, exactly as in a factor graph:

unary — how salient an entity is on its own,
self — how an entity relates to the other entities of the same modality,
pairwise — how an entity relates to the entities of every other modality,
prior — an external bias, such as sentence-length cues.

The potentials are stacked and combined by a learned, bias-free Conv1d, so the model learns how much to weight each factor. The nine history rounds share one set of factor weights, which is what keeps the interaction affordable.

The model can easily run on a single GPU :) — 52.4M parameters.

Results

Evaluation is done on VisDialv1.0.

VisDial v1.0 contains 1 dialog with 10 question-answer pairs (starting from an image caption) on ~130k images from COCO-trainval and Flickr, totalling ~1.3 million question-answer pairs.

These are the measured numbers for the weights in this repository, on the v1.0 validation split:

Metric	R@1	R@5	R@10	MRR	Mean rank	NDCG
This checkpoint (epoch 5)	52.46	82.95	90.97	66.01	3.92	56.46
Paper	53	—	—	66	—	—
5×FGA ensemble (paper)	56	—	—	69	—	—

Trained for 10 epochs on 8×L40S, ~3 hours. MRR peaks at epoch 5 while NDCG keeps improving to epoch 10 (58.19) — the two metrics disagree, which is the tension the 2020 challenge submission addressed.

Two things differ from the original run and are folded into the small R@1 gap. The image features were re-extracted, because every published copy of the originals has gone offline; the detector is the same one the paper used (Faster R-CNN, ResNeXt-101, fine-tuned on Visual Genome, 36 proposals). And evaluation no longer applies dropout: the original called F.dropout without forwarding self.training, so scoring was mildly stochastic even under model.eval().

Usage

python

from fga import FGAForVisualDialog

model = FGAForVisualDialog.from_pretrained("Idan/fga")

outputs = model(
    question_input_ids=...,          # (batch, 21)
    option_input_ids=...,            # (batch, 100, 21)
    history_question_input_ids=...,  # (batch, 9, 21)
    history_answer_input_ids=...,    # (batch, 9, 21)
    caption_input_ids=...,           # (batch, 41)
    question_lengths=...,
    option_lengths=...,
    caption_lengths=...,
    image_features=...,              # (batch, 37, 2048)
)
ranking = outputs.logits.argsort(dim=-1, descending=True)

Install with pip install git+https://github.com/idansc/fga.git.

Pass output_attentions=True to get the per-modality attention distributions for visualization.

The attention block is the reusable part of the paper and knows nothing about Visual Dialog — it is an ordinary torch.nn layer over any set of modalities:

python

from fga import FactorGraphAttention, Modality

attention = FactorGraphAttention(embed_dims=[512, 2048], num_entities=[20, 36])
pooled_text, pooled_image = attention(text, image)

# or declared by name, with weight sharing stated separately
history = [Modality(f"history_{i}", dim=128, size=21, connected_to=("text", "image"))
           for i in range(1, 10)]
attention = FactorGraphAttention.from_modalities(
    [Modality("text", dim=512, size=20), Modality("image", dim=2048, size=36), *history],
    share_weights=[[m.name for m in history]],
    use_prior=True,
)

The same layer backs the paper's follow-ups in video dialog, spatial navigation and video retrieval; their argument names (util_e, sizes, high_order_utils, *_flag) are all still accepted.

Data

Preprocessed dialogs: Idan/visdial-fga-preprocessed.

Image features are not distributed with the model — it expects an h5 with {split}_features of shape (num_images, 37, 2048). See the original paper for performance differences. I recommend using the FRCNN features, mainly because it is finetuned on the relevant VisualGenome dataset. The repository has scripts to regenerate them from the images.

Citation

Please cite Factor Graph Attention if you use this work in your research:

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}