HuatuoGPT-Vision is a multimodal LLM for medical applications, built with the PubMedVision dataset. HuatuoGPT-Vision-7B is trained based on Qwen2-7B using the LLaVA-v1.5 architecture.

Quick Start

Get the model inference code from Github.

git clone https://github.com/FreedomIntelligence/HuatuoGPT-Vision.git

Model inference

python

query = 'What does the picture show?'
image_paths = ['image_path1']

from cli import HuatuoChatbot
bot = HuatuoChatbot(huatuogpt_vision_model_path) # loads the model 
output = bot.inference(query, image_paths) # generates
print(output)  # Prints the model output

Citation

@misc{chen2024huatuogptvisioninjectingmedicalvisual,
      title={HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale}, 
      author={Junying Chen and Ruyi Ouyang and Anningzhe Gao and Shunian Chen and Guiming Hardy Chen and Xidong Wang and Ruifei Zhang and Zhenyang Cai and Ke Ji and Guangjun Yu and Xiang Wan and Benyou Wang},
      year={2024},
      eprint={2406.19280},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2406.19280}, 
}

FreedomIntelligence/HuatuoGPT-Vision-7B

HuatuoGPT-Vision-7B

Introduction

Quick Start

Citation

FreedomIntelligence/
HuatuoGPT-Vision-7B