The model was trained on the MedPMC-11M dataset, a carefully curated collection of approximately 11 million image-caption pairs derived from biomedical literature. Across a wide range of evaluations, MedPMC-CLIP consistently outperforms existing baseline models, including zero-shot medical image classification on 26 public benchmarks and zero-shot image retrieval on an internal clinical dermatology dataset.

For additional details on model training and benchmark results, please refer to the paper and GitHub repository.

To reproduce the evaluation results reported in the paper, please refer to medpmc-clip-eval.

This repository provides the checkpoint in OpenCLIP format. Text inputs should be tokenized using the default OpenCLIP tokenizer for ViT-L-14.

tokenizer = open_clip.get_tokenizer("ViT-L-14")

Files

open_clip_model.safetensors: OpenCLIP-format model checkpoint
inference_example.py: example code for image-text similarity
requirements.txt: minimal dependencies

Usage

python

import torch
import open_clip
from huggingface_hub import hf_hub_download
from PIL import Image

model_name = "ViT-L-14"
device = "cuda" if torch.cuda.is_available() else "cpu"

repo_id = "Yale-BIDS-Chen/medpmc-clip-l-14_jun24_v1"

ckpt_path = hf_hub_download(
    repo_id=repo_id,
    filename="open_clip_model.safetensors",
)

model, _, preprocess = open_clip.create_model_and_transforms(
    model_name,
    pretrained=ckpt_path,
    device=device,
)

model.eval()
tokenizer = open_clip.get_tokenizer(model_name)

image = preprocess(
    Image.open("/path/to/example.jpg").convert("RGB")
).unsqueeze(0).to(device)

text = tokenizer([
    "fundus photograph",
    "chest radiograph",
    "histopathology image",
]).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

    image_features /= image_features.norm(dim=-1, keepdim=True)
    text_features /= text_features.norm(dim=-1, keepdim=True)

    similarity = image_features @ text_features.T

print(similarity)

License

The model is released for non-commercial research use under CC BY-NC-SA 4.0.

Citation

bibtex

@article{kim2026medpmc,
  title={MedPMC: A Systematic Framework for Scaling High-Fidelity Medical Multimodal Data for Foundation Models},
  author={Kim, Hyunjae and Kim, Dain and Xiao, Pan and Applebaum, Serina S and Chung, Younjoon and Ai, Xuguang and Yin, Yu and Jiang, Roy and Du, Yuexi and Wei, Yawen and others},
  journal={arXiv preprint arXiv:2607.07673},
  year={2026}
}

Questions?

For questions or feedback, please contact Hyunjae Kim at hyunjae.kim@yale.edu.