dtian09/

clip_llama

The Flickr30k image dataset (either 10% or 50% of the training set) was firstly encoded using the CLIP image encoder, then the LlaMA3 8B transformer was fine-tuned on the encoded images plus their captions to predict image captions.

code repository: https://github.com/dtian09/Fine-Tuning-Transformers