Built on the attention layer of Factor Graph Attention (CVPR 2019), following Video and Text Matching with Conditioned Embeddings (WACV 2022). A caption and a video are attended jointly and scored against each other, trained with a max-margin contrastive loss against the hardest negative in the batch.

Code: github.com/idansc/fga.

This is not YouCook2's official train/val division, and these numbers do not compare to published YouCook2 retrieval figures.

The feature dump available here covers YouCook2's validation videos only -- 436 of them -- so the data was divided over those by video, no video appearing on two sides:

Retrieval over 617 segments is an easier problem than retrieval over the official split's several thousand, and the numbers below should be read as an internal comparison only.

Held-out test slice, 617 segments. Recall at k in both directions:

The checkpoint is the one that scored best on the validation videos, not on the test slice. Selecting on the test slice reads 0.438 / 0.468 instead; the validation split exists so that choice is not made there.

R@10 here counts a hit at rank 10 or better. Worth stating because the reference implementation's evaluation.py computes its r10 as ranks < 50 in all four of its ranking functions -- a commented-out print in the same file labels that value "Ranks@50" -- so an R@10 reported by that code is an R@50 and is not comparable to the column above.

Hidden size 768, learning rate 2e-4, 2,000 epochs. Video features are the InternVideo MM-L14 dump; captions are encoded with GloVe 300d.