Fine-tuned google/vit-large-patch16-224 on the HAM10000 dermoscopy dataset to classify 7 types of skin lesions. Trained on a free Google Colab T4 GPU in 37 minutes using HuggingFace Trainer with FP16 mixed precision.

This was a personal project to learn medical image fine-tuning — not a clinical tool.

How to use

python

from transformers import pipeline

classifier = pipeline(
    "image-classification",
    model="Kuldeepmishra3/vit-large-skin-cancer-ham10000",
)

results = classifier("skin_image.jpg", top_k=3)
for r in results:
    print(f"{r['label']}: {r['score']:.2%}")

Or with manual preprocessing:

python

from transformers import ViTForImageClassification, ViTImageProcessor
from PIL import Image
import torch

model = ViTForImageClassification.from_pretrained("Kuldeepmishra3/vit-large-skin-cancer-ham10000")
processor = ViTImageProcessor.from_pretrained("Kuldeepmishra3/vit-large-skin-cancer-ham10000")

image = Image.open("skin_image.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

predicted_class = logits.argmax(-1).item()
print(model.config.id2label[predicted_class])

Classes

Label	Full name
akiec	Actinic Keratoses
bcc	Basal Cell Carcinoma
bkl	Benign Keratosis
df	Dermatofibroma
mel	Melanoma
nv	Melanocytic Nevi
vasc	Vascular Lesions

Results

Evaluated on the HAM10000 validation split (2,492 images):

Metric	Score
Accuracy	92.74%
Weighted F1	92.60%

Per-class F1:

Class	F1
vasc	97.14%
nv	96.54%
df	92.86%
bcc	92.24%
bkl	84.37%
akiec	82.50%
mel	79.78%

Melanoma was the hardest — it frequently gets confused with Melanocytic Nevi, which is a known challenge in dermoscopy even for trained dermatologists.

Training details

Base model: google/vit-large-patch16-224 (303M params)
Dataset: HAM10000 via marmal88/skin_cancer on HuggingFace
Epochs: 5 (best checkpoint at epoch 4)
Batch size: 16
Learning rate: 2e-5 with cosine schedule
Warmup steps: 300
Precision: FP16
GPU: Tesla T4 (Google Colab free tier)
Training time: ~37 minutes

Augmentations used during training: random horizontal/vertical flip, rotation up to 30°, color jitter. No augmentation at validation time.

Limitations

Not a clinical diagnostic tool — do not use for real medical decisions
Performance drops on minority classes (akiec, df, vasc) due to class imbalance in HAM10000
Only trained on dermoscopy images — won't work well on regular phone camera skin photos
Dataset is mostly from European populations which may affect generalization

Dataset

HAM10000 (Human Against Machine with 10000 training images) is a large collection of multi-source dermatoscopic images of common pigmented skin lesions. The dataset has significant class imbalance — Melanocytic Nevi makes up around 67% of all samples.