[GitHub] [Paper (ACL Anthology)]

This repository contains the official model weights for the EMNLP 2024 paper: "Interpretable Composition Attribution Enhancement for Visio-linguistic Compositional Understanding".

Contrastively trained vision-language models (e.g., CLIP) often struggle with compositional reasoning, failing to distinguish fine-grained relations or attributes. We identify the root cause as the composition attribution issue, where models assign significantly lower attribution scores to relation and attribute terms compared to object terms.

CAE (Composition Attribution Enhancement) is a generic framework designed to rectify text attribution by shifting the model's focus beyond isolated object terms toward essential compositional concepts. Our approach significantly enhances the model's ability to discern intricate visual and linguistic details across seven benchmarks.

We provide four model variants based on different attribution methods used during training. All models use CLIP-ViT-B/32 as the backbone.

You can load these checkpoints using the open_clip library.