LLaVA Skin Disease Multimodal Assistant
Model Details
Model Description
This model is a parameter-efficient fine-tuned version of LLaVA v1.5 7B designed to generate detailed descriptions and multi-turn conversations about dermatological images.
The model was fine-tuned to improve the ability of vision-language models to describe skin conditions and generate structured medical-style explanations from images. The goal of this project was to explore how multimodal models can be adapted for medical reasoning and structured visual analysis using limited compute resources.
Training focused on improving the model’s ability to:
- Describe dermatological patterns in images
- Generate structured explanations of possible conditions
- Produce multi-turn conversational responses grounded in visual input
This project was developed as an experimental research project exploring efficient multimodal fine-tuning.
- Developed by: Abdulmateen Ashifa
- Model type: Vision-Language Model (VLM)
- Language(s): English
- License: Same as base model (LLaVA v1.5 license)
- Finetuned from model: liuhaotian/llava-v1.5-7b
Model Sources
Uses
Direct Use
The model can be used for research and experimentation with multimodal models that generate descriptions of dermatological images.
Example tasks include:
- Visual description of skin lesions
- Educational demonstrations of medical image analysis
- Multimodal dialogue generation
Downstream Use
The model may be integrated into systems that explore:
- AI-assisted dermatology education
- Synthetic training data generation
- Multimodal medical reasoning research
Out-of-Scope Use
This model should not be used for clinical diagnosis or medical decision making. The model is intended for research and experimentation only.
Bias, Risks, and Limitations
The model inherits limitations from both the base LLaVA architecture and the dataset used for fine-tuning.
Potential risks include:
- Incorrect medical descriptions
- Hallucinated conditions
- Bias from the training dataset
- Limited generalization to unseen skin conditions
Because of these limitations, outputs should always be verified by qualified professionals in real-world contexts.
How to Get Started with the Model
Example usage with HuggingFace Transformers and PEFT:
1from transformers import AutoModelForCausalLM
2from peft import PeftModel
3
4base_model = AutoModelForCausalLM.from_pretrained(
5 "liuhaotian/llava-v1.5-7b",
6 device_map="auto"
7)
8
9model = PeftModel.from_pretrained(base_model, "your-model-path")
