LLaVA-RLHF Model Card
Model details
Model type:
LLaVA-RLHF represents a novel aligned end-to-end trained large multimodal model that combines a CLIP vision encoder and Vicuna for general-purpose visual and language understanding, achieving impressive visual reasoning and perception capabilities mimicking spirits of the multimodal GPT-4.
Via Factually Augmented RLHF, LLaVA-RLHF is presented to be more helpful and less hallucinated than LLaVA or other open-sourced LMMs.
Usage:
NOTE: The RLHFed model is trained with LoRA and the bfloat16 data type.
Users have to apply the PEFT-LoRA on the LLaVA-SFT+ model.
1dtype = torch.bfloat16
2
3model_path = "LLaVA-RLHF-13b-v1.5-336/sft_model"
4lora_path = "LLaVA-RLHF-13b-v1.5-336/rlhf_lora_adapter_model"
5
6model = LlavaLlamaForCausalLM.from_pretrained(
7 model_path,
8 device_map={"": "cuda:0"},
9 torch_dtype=dtype,
10)
11
12model = PeftModel.from_pretrained(
13 model,
14 lora_path,
15)
Model date:
LLaVA-RLHF was trained in Sept 2023.
Paper or resources for more information:
https://llava-rlhf.github.io/
License:
Apache License 2.0
Where to send questions or comments about the model:
https://github.com/llava-rlhf/LLaVA-RLHF/issues
Intended use
Primary intended uses:
The primary use of LLaVA-RLHF is research on large multimodal chatbots.
Primary intended users:
The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
Training dataset
595K filtered image-text pairs from CC3M.
150K GPT-generated multimodal instruction-following chat data.
83K VQA v2 instruction-following VQA data.
16K A-OKVQA instruction-following CoT-VQA data.
23K FLICKR instruction-following spotting captioning data.
10K LLaVA-based human preference data
