Fine-tuning: Last 4 vision transformer layers unfrozen (~7% trainable parameters), with classification head. Trained with Fitzpatrick-balanced sampling for fairness across skin tones.

Training Data: 47,277 images from 5 dermatology datasets:

Dataset	Samples	Type	Skin Tone Coverage
HAM10000	10,015	Dermoscopic	Limited
DDI/Stanford	656	Clinical	Fitzpatrick I-VI
Fitzpatrick17k	16,518	Clinical	Fitzpatrick I-VI
PAD-UFES-20	2,298	Smartphone	Brazilian population
BCN20000	17,790	Dermoscopic	European population

Performance

Binary Triage (Benign vs Malignant)

Model	AUC	Accuracy	F1
Fine-tuned XGBoost	0.960	92.0%	0.866
Fine-tuned MLP	0.945	91.2%	0.856
Frozen XGBoost (baseline)	0.916	88.2%	0.794

Clinical Thresholds:

At 95% sensitivity: 82.5% specificity
At 90% sensitivity: 88.3% specificity

Condition Estimation (10-class)

10 categories: Melanoma, BCC, SCC, Actinic Keratosis, Melanocytic Nevus, Seborrheic Keratosis, Dermatofibroma, Vascular Lesion, Non-Neoplastic, Other/Unknown.

Fairness

Tested across Fitzpatrick skin types I-VI with <5% sensitivity gap. See evaluation_results.json for per-group metrics.

Versioning

Branch	Description	Model Format
`main`	Production (current) -- EndToEndSigLIP v1 format	config.json + model_state.pt + head_state.pt
`v1-original`	Snapshot of original upload for rollback	Same as main at time of branch creation

To load a specific version:

python

from huggingface_hub import snapshot_download
# Latest (main)
snapshot_download("skintaglabs/siglip-skin-lesion-classifier")
# Rollback to v1
snapshot_download("skintaglabs/siglip-skin-lesion-classifier", revision="v1-original")

Files

Root (Model Weights)

config.json -- Architecture configuration (model_name, hidden_dim, n_classes, dropout, unfreeze_layers)
head_state.pt -- Classification head weights (~1.2 MB)
model_state.pt -- Full fine-tuned model weights (~3.5 GB)

Misc/ (Pipeline Artifacts)

classifier.pkl, classifier_logistic.pkl, classifier_deep.pkl -- Binary classifiers (frozen embeddings)
classifier_deep_mlp.pkl -- Alias for classifier_deep.pkl (web app compatibility)
classifier_xgboost.pkl -- XGBoost binary classifier (frozen embeddings)
classifier_xgboost_finetuned.pkl -- XGBoost on fine-tuned embeddings (AUC 0.960)
xgboost_finetuned_binary.pkl, mlp_finetuned_binary.pkl -- Fine-tuned pipeline classifiers
xgboost_finetuned_condition.pkl -- 10-class condition classifier (fine-tuned)
classifier_condition.pkl, classifier_condition_deep.pkl, classifier_condition_logistic.pkl -- Condition classifiers
embeddings.pt -- Frozen SigLIP embeddings (~109 MB)
embeddings_finetuned_train.pt, embeddings_finetuned_test.pt -- Fine-tuned embeddings
metadata.csv, test_metadata.csv -- Dataset metadata with labels
evaluation_results.json, training_results.json -- Metrics

Usage

End-to-End Inference (Recommended)

python

from src.model.deep_classifier import EndToEndClassifier
model = EndToEndClassifier.load_for_inference("path/to/downloaded/model")
probs = model.predict_proba(images)  # Returns [benign_prob, malignant_prob]

Embedding + Classifier Head

python

from transformers import AutoModel, AutoImageProcessor
import pickle

# Extract embeddings with frozen SigLIP
processor = AutoImageProcessor.from_pretrained("google/siglip-so400m-patch14-384")
model = AutoModel.from_pretrained("google/siglip-so400m-patch14-384")
embeddings = model.vision_model(pixel_values=processed_images).pooler_output

# Classify with downloaded head
with open("Misc/classifier.pkl", "rb") as f:
    clf = pickle.load(f)
predictions = clf.predict_proba(embeddings.numpy())

Full Implementation

See the SkinTag repository for the complete pipeline including web app, mobile distillation, and evaluation.

Citation

bibtex

@misc{skintag2026,
  title={SkinTag: AI-Powered Skin Lesion Triage for Equitable Melanoma Detection},
  author={SkinTag Labs},
  year={2026},
  url={https://github.com/skintaglabs/main}
}

Base model (SigLIP):

bibtex

@misc{zhai2023sigmoid,
  title={Sigmoid Loss for Language Image Pre-Training},
  author={Xiaohua Zhai and Basil Mustafa and Alexander Kolesnikov and Lucas Beyer},
  year={2023},
  eprint={2303.15343},
  archivePrefix={arXiv},
  primaryClass={cs.CV}
}

Attribution

Base Model: SigLIP by Google (Apache 2.0)
Fine-tuning: SkinTag Labs (MIT)

Medical Disclaimer

This model is for research and triage screening only, not clinical diagnosis. Users should consult healthcare professionals for medical advice.

License

MIT License (fine-tuned weights). Base model (SigLIP) is licensed under Apache 2.0 by Google.