A
ViT-L/14 encoder + 6-layer causal decoder trained from scratch on
MIMIC-CXR
to generate full radiology reports from chest X-ray images.
Unlike the contrastive pretrain variants, the generative objective forces the encoder
to retain fine-grained spatial information sufficient for region-level text generation.
The encoder weights (
encoder_final.pt) serve as the strongest feature extractor
in the LAPVQA downstream tasks.
The captioning-pretrained encoder matches or exceeds the contrastive variants on both
classification benchmarks, and is the best-performing encoder on DiffVQA when used downstream.
1import torch
2from lapvqa.pretrain.model import CaptioningModel
3
4ckpt = torch.load("model_best.pt", map_location="cpu")
5model = CaptioningModel()
6model.load_state_dict(ckpt)
7model.eval()
8
9# To use only the encoder as a feature extractor:
10enc_weights = torch.load("encoder_final.pt", map_location="cpu")
11model.vision_encoder.load_state_dict(enc_weights)
12# vis_tokens = model.vision_encoder(images) # [B, 256, 1024]
1@article{johnson2019mimic,
2 title = {MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports},
3 author = {Johnson, Alistair EW and others},
4 journal = {Scientific data},
5 volume = {6}, pages = {317}, year = {2019}
6}
