This model was lovingly named after the Google Colab notebook that made it. It is a finetune of Microsoft's
git-large-coco model on the 1k subset of
poloclub/diffusiondb.
It is supposed to read images and extract a stable diffusion prompt from it but, it might not do a good job at it. I wouldn't know I haven't extensivly tested it.
As the title suggests this is a checkpoint as I formerly intended to do it on the entire dataset but, I'm unsure if I want to now...
1# Load model directly
2from transformers import AutoProcessor, AutoModelForCausalLM
3
4processor = AutoProcessor.from_pretrained("SE6446/Untitled7-colab_checkpoint")
5model = AutoModelForCausalLM.from_pretrained("SE6446/Untitled7-colab_checkpoint")
6
7#################################################################
8# Use a pipeline as a high-level helper
9from transformers import pipeline
10
11pipe = pipeline("image-to-text", model="SE6446/Untitled7-colab_checkpoint")
Don't use this model to discriminate, alienate or in any other way harm/harass individuals. You guys know the drill...
This model does not produce accurate prompts, this is merely a bit of fun (and waste of funds). However it can suffer from bias present in the orginal git-large-coco model.
If you want to further finetune it then you should freeze the embedding and vision tranformer layers
