Views
No views yet
Recent advancements in multimodal models have significantly improved vision-language (VL) alignment in radiology. However, existing approaches struggle to effectively utilize complex radiology reports for learning and offer limited interpretability through attention probability visualizations. To address these challenges, we introduce RadZero, a novel framework for VL alignment in chest X-ray with zero-shot multi-task capability. A key component of our approach is VL-CABS (Vision-Language Cross-Attention Based on Similarity), which aligns text embeddings with local image features for interpretable, fine-grained VL reasoning. RadZero leverages large language models to extract concise semantic sentences from radiology reports and employs multi-positive contrastive training to effectively capture relationships between images and multiple relevant textual descriptions. It uses a pre-trained vision encoder with additional trainable Transformer layers, allowing efficient high-resolution image processing. By computing similarity between text embeddings and local image patch features, VL-CABS enables zero-shot inference with similarity probability for classification, and pixel-level VL similarity maps for grounding and segmentation. Experimental results on public chest radiograph benchmarks show that RadZero outperforms state-of-the-art methods in zero-shot classification, grounding, and segmentation. Furthermore, VL similarity map analysis highlights the potential of VL-CABS for improving explainability in VL alignment. Additionally, qualitative evaluation demonstrates RadZero's capability for open-vocabulary semantic segmentation, further validating its effectiveness in medical imaging.
pip install -r requirements.txt1# Deepnoid/RadZero/inference.py
2import warnings
3
4import torch
5from transformers import AutoImageProcessor, AutoModel, AutoTokenizer
6
7from utils import model_inference
8
9# Suppress specific warnings for cleaner logs
10warnings.filterwarnings("ignore", category=UserWarning)
11
12
13def load_model(device, dtype):
14
15 tokenizer = AutoTokenizer.from_pretrained("Deepnoid/RadZero")
16 image_processor = AutoImageProcessor.from_pretrained("Deepnoid/RadZero")
17
18 model = AutoModel.from_pretrained(
19 "Deepnoid/RadZero",
20 trust_remote_code=True,
21 torch_dtype=dtype,
22 device_map=device,
23 )
24
25 models = {
26 "tokenizer": tokenizer,
27 "image_processor": image_processor,
28 "model": model,
29 }
30 return models
31
32
33if __name__ == "__main__":
34 # Setup constant
35 device = torch.device("cuda")
36 dtype = torch.float32
37
38 # load models
39 models = load_model(device, dtype)
40
41 # load image
42 image_path = "cxr_image.jpg"
43
44 # inference
45 similarity_prob, similarity_map = model_inference(
46 image_path, "There is fibrosis", **models
47 )
48
49 print(similarity_prob)
50 print(similarity_map.min())
51 print(similarity_map.max())
52 print(similarity_map.shape)