This model was fine-tuned on the HierarCaps dataset to enhance hierarchical organization in CLIP's embedding space, improving hierarchical reasoning about specificity between images and texts.

Usage

Paper: <a href="https://arxiv.org/abs/2407.08521" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">https://arxiv.org/abs/2407.08521
Project page: <a href="https://hierarcaps.github.io/" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">https://hierarcaps.github.io/
Code: <a href="https://github.com/TAU-VAILab/hierarcaps" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">https://github.com/TAU-VAILab/hierarcaps
Base model: <a href="https://huggingface.co/openai/clip-vit-base-patch32" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">openai/clip-vit-base-patch32

python

from transformers import CLIPModel, AutoProcessor

model = CLIPModel.from_pretrained("malper/hierarcaps-clip-b")
processor = AutoProcessor.from_pretrained("malper/hierarcaps-clip-b")

See the official code repository for training and evaluation scripts used with this checkpoint.

Citation

bibtex

@InProceedings{alper2024hierarcaps,
  author    = {Morris Alper and Hadar Averbuch-Elor},
  title     = {Emergent Visual-Semantic Hierarchies in Image-Text Representations},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2024}
}