You can deploy this model and start earning money today!

parthsalke/
medclip-glioma-lora-8bit

Name: medclip-glioma-lora-8bit
Availability: InStock

zero-shot-image-classification

parthsalke/vlm_glioma_dataset

endpoints_compatible

Views

No views yet

🧠 MedCLIP-Glioma-LoRA-8bit

A fine-tuned version of PubMedCLIP using LoRA adapters on the BRATS-derived Glioma dataset for highly precise brain tumor classification.

🔍 Model Overview

This model builds upon pre-trained PubMedCLIP — a CLIP variant pre-trained on radiology images from the ROCO dataset — and adapts it for Glioma tumor classification using supervised fine-tuning (SFT) with Low-Rank Adaptation (LoRA) and 8-bit quantization.

Base model: flaviagiammarino/pubmed-clip-vit-base-patch32
Fine-tuned on: BRATS-derived dataset from The Cancer Imaging Archive (TCIA)
Preprocessing: Max Intensity Projection (MIP)
Format: DICOM & NIfTI ➝ PNG (via 2D slicing)
Adaptation: LoRA applied to attention and MLP layers
Precision: Quantized from FP32 → 8-bit using bitsandbytes

📁 Dataset

Name: parthsalke/vlm_glioma_dataset
Modalities: T1, T1-CE, T2, and T2 FLAIR MRI
Format: Preprocessed from DICOM/NIFTI to RGB PNG
Pairs: Image + Text

🧪 Fine-Tuning Strategy

Component	Strategy
Vision Encoder	Frozen base, LoRA adapters (qkv)
Text Encoder	Frozen base, LoRA adapters
Training Method	Supervised Fine-Tuning (SFT)
Optimizer	AdamW
LoRA Params	r=8, alpha=32, dropout=0.1
Precision	8-bit (bnb + PEFT)

🧪 How to Use

python

from transformers import CLIPProcessor
from peft import PeftModel
from PIL import Image

# Load base + LoRA adapter
from transformers import CLIPModel
base = CLIPModel.from_pretrained("flaviagiammarino/pubmed-clip-vit-base-patch32")
model = PeftModel.from_pretrained(base, "parthsalke/biomedclip-glioma-lora-8bit")

# Load processor
processor = CLIPProcessor.from_pretrained("flaviagiammarino/pubmed-clip-vit-base-patch32")

# Inference example
image = Image.open("example.png").convert("RGB")
text = [" ", " "]  #Input labels here

inputs = processor(images=image, text=text, return_tensors="pt", padding=True)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1)

parthsalke/medclip-glioma-lora-8bit