SkinMap is a dermatology image/text embedding model. It fuses 12 frozen teacher encoders (multi-modal and self-supervised models) through a trained projector into a single 1024-d shared image/text space, that can be used to infer missing metadata, solve downstream tasks, and get relational information about the samples.

Compute. This is the faithful full ensemble: 12 backbones load into memory at once and need roughly 16 GB of GPU VRAM. It runs on CPU but slowly.

Usage

python

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Digital-Dermatology/SkinMap",
    trust_remote_code=True,
    device="cuda",            # "cpu" also works
)

# 1) Encode an image into the 1024-d SkinMap space (L2-normalized)
img_emb = model.encode_image("lesion.jpg")            # np.ndarray, shape (1024,)

# 2) Encode a text query into the same space
txt_emb = model.encode_text("melanoma on the back")   # np.ndarray, shape (1024,)

# 3) Predict clinical metadata from the image
meta = model.predict_meta("lesion.jpg")
# {'fitzpatrick': 3, 'fitzpatrick_grouped': '3-4', 'age': 42.1,
#  'gender': 'female', 'origin': ..., 'body_region': ..., ...}

import numpy as np
sim = float(np.dot(img_emb, txt_emb))                 # cosine in the shared space

encode_image accepts a file path, raw bytes, or a PIL.Image. encode_text expands the query into clinical prompt templates and averages the results; pass templates=["{}"] to encode the raw text as-is.

Metadata prediction

predict_meta applies bundled linear probes to the SkinMap embedding to estimate Fitzpatrick skin type, age, sex, geographic origin, body region and more. By default it uses global probes, which reproduce the demographic estimates reported in the paper. If you know the imaging modality, pass it to switch the Fitzpatrick prediction to a modality-specific probe (more accurate when the modality is known); the other attributes stay global:

meta = model.predict_meta("lesion.jpg", modality="dermoscopy") # clinical | dermoscopy | TBP

predict_meta also accepts a precomputed 1024-d vector instead of an image, and a subset via attributes=[...].

Nearest-neighbor search (optional)

If this repo was built with an atlas index bundled, you can retrieve neighbors directly:

indices, distances = model.search("lesion.jpg", k=10)

Without a bundled index, search() raises a clear error. Encoding still works.

What's inside

Component	Role
9 multi-modal teachers (CLIP & SigLIP; ViT-L/14 & ViT-B/32)	image + text features
3 self-supervised teachers (DINO, iBOT, MAE; ViT-B/16)	dermatology image features
Per-teacher whitening (`whitening_stats.npz`)	decorrelate before fusion
Trained projector (`projector_model.pth`)	fuse to 1024-d image/text heads
Metadata probes (`probes/`)	linear probes for `predict_meta`

Licensing

SkinMap is released under the Creative Commons Attribution-NonCommercial 4.0 International license, for research use only. Not a medical device. Not for clinical decision-making.