from transformers import pipeline
from PIL import Image
import requests
import torch

pipe = pipeline(
    "image-text-to-text",
    model="gbalachandhiran/medgemma-Histopathology-lung-colon",
    torch_dtype=torch.bfloat16,
    device="cuda",
)
## Lung cancer sample
image_url = "https://cdnintech.com/media/chapter/59926/1512345123/media/F1.png"
image = Image.open(requests.get(image_url, headers={"User-Agent": "example"}, stream=True).raw)

TISSUE_CLASSES = [
    "A: colon_aca",
    "B: colon_n",
    "C: lung_aca",
    "D: lung_n",
    "E: lung_scc"
]

options = "\n".join(TISSUE_CLASSES)
PROMPT = f"What is the most likely tissue type shown in the histopathology image?\n{options}"''

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are an expert radiologist."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": PROMPT},
            {"type": "image", "image": image},
        ]
    }
]

output = pipe(text=messages, max_new_tokens=50)
print(output[0]["generated_text"][-1]["content"])

Training procedure

This is a finetuned model from original Medgemma 4B model

Fine tuning notebook :

Framework versions

TRL: 0.19.0
Transformers: 4.53.0
Pytorch: 2.6.0+cu124
Datasets: 3.6.0
Tokenizers: 0.21.2

Citations

Dataset: https://www.kaggle.com/datasets/andrewmvd/lung-and-colon-cancer-histopathological-images

Credited Authors of the Dataset : Borkowski AA, Bui MM, Thomas LB, Wilson CP, DeLand LA, Mastorides SM. Lung and Colon Cancer Histopathological Image Dataset (LC25000). arXiv:1912.12142v1 [eess.IV], 2019

Cite TRL as:

bibtex

@misc{vonwerra2022trl,
	title        = {{Fine Tuning Medgemma to identify Histopathology images of lungs and Colon}},
	author       = {Dr. Ramanan. R. K:dr.ramanan.r.k@c-riht.org,Dhanu Elavarasan : nreldhanu1713@gmail.com,Gowtham Balachadhiran:gowthambalachandhiran@gmail.com},
	year         = 2025,
	journal      = {[GitHub repository](https://github.com/gowthambalachandhiran/MedGemmaAUKBC)},
	publisher    = {GitHub},
	howpublished = {\url{gbalachandhiran/medgemma-Histopathology-lung-colon}}
}