You can deploy this model and start earning money today!

malper/
learning-interactions-cc-ft

Views

No views yet

CLIPCap fine-tuned for Human-Human Interaction captioning (cc_ft)

This is the best-performing checkpoint from "Learning Human-Human Interactions in Images from Weak Textual Supervision" (ICCV 2023): a CLIPCap model fine-tuned on pseudo-labels (pHHI) for describing human-human interactions (HHI) in images.

Paper: arXiv:2304.14104
Code: github.com/tau-vailab/learning-interactions
Project page: https://learning-interactions.github.io/

Note: this checkpoint has been computed from scratch and may differ slightly from the results reported in the paper due to randomness during training.

Usage

This checkpoint is a PyTorch Lightning checkpoint (cc_ft.ckpt), intended for use with the inference code in the repo above (modeling/src/test.py):

bash

hf download malper/learning-interactions-cc-ft cc_ft.ckpt --local-dir .
python src/test.py -d <image_dir> -o <output.json> -c cc_ft.ckpt -cc <clipcap_pretrained_weights>

See the modeling documentation for full details, including the required CLIPCap pretrained MLP mapping network weights.

Context

This is research code from 2023, prior to the widespread availability of general-purpose vision-language models (VLMs). It is provided as-is for reproducibility of the paper's results.

Citation

bibtex

@InProceedings{alper2023learning,
    author    = {Morris Alper and Hadar Averbuch-Elor},
    title     = {Learning Human-Human Interactions in Images from Weak Textual Supervision},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    year      = {2023}
}