You can deploy this model and start earning money today!

cesaraha/
ham10000-multimodal-skin-lesion

Name: ham10000-multimodal-skin-lesion
Availability: InStock

CrossAttentionFusionModel

Views

No views yet

HAM10000 Multimodal Skin Lesion Classification

Cross-attention fusion model for 7-class dermoscopic skin lesion classification, combining image features with patient metadata (age, sex, lesion localization). Trained from scratch on HAM10000 using PyTorch.

Model description

The model uses a CNN-B backbone (residual connections + batch normalisation) as the image encoder, paired with a cross-attention fusion module that allows patient metadata to selectively attend to spatial regions of the image feature map. This produces a metadata-conditioned image representation rather than a spatially-averaged one.

Performance

Metric	Value
Test macro AUROC	0.9385
Test accuracy	0.7063

Per-class AUROC

Class	AUROC
Melanoma	0.8829
Actinic keratosis	0.9744
Benign keratosis	0.8944
Basal cell carcinoma	0.9689
Dermatofibroma	0.9327
Nevi	0.9391
Vascular	0.9771

How to use

python

import torch
from PIL import Image
from torchvision import transforms

# Load checkpoint
state = torch.load("best.pt", map_location="cpu", weights_only=False)

# Initialise model (architecture definition required — see GitHub repo)
model = CrossAttentionFusionModel(meta_dim=35)
model.load_state_dict(state["model"])
model.eval()

# Image preprocessing
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225]),
])

# Metadata: [age_normalised, sex_onehot(3), localization_onehot(31)]
# See GitHub repo for full encode_metadata() function

Training

Trained on Google Colab free tier (T4 GPU) for 30 epochs. Full experimental methodology including architecture ablation, hyperparameter search, and fusion strategy comparison available in the GitHub repository.

Limitations

Trained on HAM10000 only — performance on other dermoscopy datasets not evaluated
Patient metadata required at inference time
Not validated for clinical use

Citation

Tschandl P., Rosendahl C. & Kittler H. The HAM10000 dataset, a large
collection of multi-source dermatoscopic images of common pigmented
skin lesions. Sci. Data 5, 180161 (2018).

cesaraha/ham10000-multimodal-skin-lesion