MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment

Introduction:

Quick Start:

• Zero-Shot Classification:
  We give an example on CXR14 in Sample_Zero-Shot_Classification_CXR14. Change the data paths, and test our model by python test.py. We give an example on RSNA in Sample_Zero-Shot_Classification_RSNA. Change the data paths, and test our model by python test.py.
• Zero-Shot Grounding:
  We give an example on RSNA_Pneumonia in Sample_Zero-Shot_Grounding_RSNA. Change the data paths, and test our model by python test.py.
• Finetuning:
  We give segmentation and classification finetune code on SIIM_ACR dataset in Sample_Finetuning_SIIMACR. Change the data paths, and finetune our model by python I1_classification/train_res_ft.py or python I2_segementation/train_res_ft.py.

Pre-train:

Data Preparation

• Extracted triplets: landmark_observation_adj_mtx.npy
• Training list: train.json
• Validation list: valid.json
• Test list: test.json

Pre-training

• Check the PreTrain_MeDSLIP/data_file dir and download the files for data preparation.
• Change the data and preparation files paths as you desire in PreTrain_MeDSLIP/configs/Pretrain_MeDSLIP.yaml, and python PreTrain_MeDSLIP/train_MeDSLIP.py to pre-train.

Reference

@article{fan2024medslip,
  title={MeDSLIP: Medical Dual-Stream Language-Image Pre-training with Pathology-Anatomy Semantic Alignment},
  author={Fan, Wenrui and Suvon, Mohammod Naimul Islam and Zhou, Shuo and Liu, Xianyuan and Alabed, Samer and Osmani, Venet and Swift, Andrew and Chen, Chen and Lu, Haiping},
  journal={arXiv preprint arXiv:2403.10635},
  year={2024}
}

Contact