Medical Report Generator (BLIP fine-tuned on ROCO)
Fine-tuned BLIP model for generating radiology reports from medical images, developed as part of a capstone project in medical image analysis.
This model generates descriptive captions for radiology images (e.g., X-rays, CT scans) using the Salesforce/blip-image-captioning-base as the base model, fine-tuned on the eltorio/ROCOv2-radiology dataset.
Model Details
Model Description
Developed by a Computer Science student focusing on ML for medical imaging.
Funded by: optional (e.g., academic project).
Shared by: Siddartha01.
Model type: BLIP (vision-language).
Languages: English (NLP for captions).
License: apache-2.0 (or specify your choice).
Finetuned from model: Salesforce/blip-image-captioning-base.
Model Sources
Uses
Direct Use
This model generates radiology-style captions from medical images without further fine-tuning. Suitable for research, education, and prototyping medical report generators. Example: Input chest X-ray → Output: "Chest X-ray shows normal lung fields."
Downstream Use
Integrate into web apps with Gradio/Hugging Face Spaces, RAG systems for reports, or LLMs for detailed analysis.
Out-of-Scope Use
Not for clinical diagnosis, real-world medical decisions, or non-radiology images. Misuse may lead to inaccurate reports.
Bias, Risks, and Limitations
Medical datasets like ROCO have biases toward common pathologies and English-language captions from publications. Model may underperform on rare conditions, diverse demographics, or low-quality images. Risks include over-reliance leading to misdiagnosis—use only for research.
Recommendations
Users should validate outputs with experts, disclose model limitations in deployments, and fine-tune further on domain-specific data. Monitor for hallucinations in generated reports.
How to Get Started with the Model
1from transformers import BlipProcessor, BlipForConditionalGeneration
2from PIL import Image
3import requests
4
5model = BlipForConditionalGeneration.from_pretrained("[YOUR HF USERNAME]/[YOUR REPO NAME]")
6processor = BlipProcessor.from_pretrained("[YOUR HF USERNAME]/[YOUR REPO NAME]")
7
8url = "https://example.com/radiology-image.jpg"
9image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
10
11inputs = processor(images=image, return_tensors="pt")
12outputs = model.generate(**inputs, max_length=50)
13caption = processor.decode(outputs, skip_special_tokens=True)
14
15print("Generated Report:", caption)
