You can deploy this model and start earning money today!

RamonK/
DistillPath-IS16-UNI2h

Name: DistillPath-IS16-UNI2h
Availability: InStock

timm

safetensors

computational-pathology

knowledge-distillation

vision-transformer

medical-imaging

image-feature-extraction

TCGA

2608.17872

cc-by-nc-4.0

Views

No views yet

DistillPath-IS16-UNI2h

A 22M ViT-S/16 pathology tile encoder distilled from UNI2-h (681M ViT-H/14) into an ImageNet-21k pretrained ViT-S/16 student using backbone-token distillation on 6,000 public TCGA slides.

This is the ImageNet-initialized variant. For the kaiko-initialized variant, see DistillPath-KS16-UNI2h.

Paper: DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance Ramon Kaspar, Andrey Ignatov, Valentina Boeva. ETH Zurich. Published at the ECCV 2026 Workshop on Medical Foundation Models and Benchmarks (MedFM-Bench).

Model details

Property	Value
Architecture	ViT-S/16 (`vit_small_patch16_224` in timm)
Parameters	21.7M
Feature dimension	384
Input size	224 x 224
Normalization	mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)
Student initialization	ImageNet-21k ViT-S/16
Teacher	UNI2-h (681M, ViT-H/14, CC BY-NC-ND 4.0)
Training data	6,000 TCGA H&E whole-slide images, 32 cohorts
Training steps	50,000 (batch size 256)

Benchmark results

Benchmark	DistillPath-IS16-UNI2h	IN21K baseline	UNI2-h teacher
EVA mean (7 tasks)	0.765	0.729	0.806
HEST mean (9 tasks)	0.364	0.311	0.414
PLISM score	0.561	0.383	0.333

See the paper for per-task results.

Usage

Load directly from the Hub with timm:

python

import timm

model = timm.create_model(
    "hf_hub:RamonK/DistillPath-IS16-UNI2h",
    pretrained=True,
    num_classes=0,
)
model.eval()

Or load manually:

python

import timm
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

model = timm.create_model("vit_small_patch16_224", pretrained=False, num_classes=0)
path = hf_hub_download("RamonK/DistillPath-IS16-UNI2h", "model.safetensors")
state_dict = load_file(path)
model.load_state_dict(state_dict, strict=True)
model.eval()

This model uses ImageNet normalization: mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225).

python

from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

Distillation recipe

The recipe reads only the teacher's final class and patch tokens (no teacher pretraining heads required):

Class-token loss: cosine distance + RKD (relational knowledge distillation)
Patch-token loss: cosine distance after bicubic grid resizing (teacher 16x16 to student 14x14)
Optimizer: AdamW, lr=1e-4, weight decay 0.05, cosine decay, 500 warmup steps
Projector: DINO-style MLP (384 to 2048 to 2048 to 256 to d_t), discarded after training

Full details in the paper and the DistillPath repository.

License

This model is released under the Creative Commons Attribution-NonCommercial 4.0 International License. The UNI2-h teacher is released under CC BY-NC-ND 4.0. The distillation process used UNI2-h only to generate supervisory outputs; the released student contains no UNI2-h weights. This model is intended solely for non-commercial academic research.

Citation

bibtex

@inproceedings{kaspar2026distillpath,
  title     = {DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance},
  author    = {Kaspar, Ramon and Ignatov, Andrey and Boeva, Valentina},
  booktitle = {Medical Foundation Models and Benchmarks (MedFM-Bench), ECCV 2026},
  year      = {2026}
}