Developed by: LAION, SkunkworksAI & Ontocord
Model type: LLaVA is an open-source chatbot trained by fine-tuning Phi-2 on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture
Finetuned from model: Phi-2
License: MIT

Usage

Repository: <a href="https://github.com/SkunkworksAI/BakLLaVA" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">BakLLaVa
LLama.cpp: <a href="https://github.com/ggerganov/llama.cpp" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">GitHub

make & ./llava-cli -m ../ggml-model-f16.gguf --mmproj ../mmproj-model-f16.gguf --image /path/to/image.jpg

Evaluation

Model	Parameters	SQA	GQA	TextVQA	POPE
LLaVA-1.5	7.3B	68.0	62.0	58.3	85.3
MC-LLaVA-3B	3B	-	49.6	38.59	-
LLaVA-Phi	3B	68.4	-	48.6	85.0
moondream1	1.6B	-	56.3	39.8	-
llava-phi-2-3b	3B	69.0	51.2	47.0	86.0

Model	BLEU_1	BLEU_2	BLEU_3	BLEU_4	METEOR	ROUGE_L	CIDEr	SPICE
llava-1.5-7b	75.8	59.8	45	33.3	29.4	57.7	108.8	23.5
llava-phi-2-3b	67.7	50.5	35.7	24.2	27.0	52.4	85.0	20.7