Headline metrics (validation set)

Metric	Value
ROC-AUC	0.869
PR-AUC	0.669
Accuracy at Youden op-point	0.760
Op-point threshold	0.145
ECE (raw)	0.0344
ECE (after temperature scaling)	0.0297
Positive prevalence	0.225
Validation set size	5337 patients

The Youden-J operating-point threshold is 0.145, not the default 0.5. The validation accuracy above is computed at this threshold.

How to load

python

import timm, torch
from safetensors.torch import load_file

model = timm.create_model(
    "vit_small_patch14_dinov2.lvd142m",
    pretrained=False,
    in_chans=1,
    num_classes=2,
    img_size=224,
)
model.load_state_dict(load_file("model.safetensors"))
model.eval()

How to apply temperature scaling

python

import json, torch.nn.functional as F

T = json.load(open("temperature_scaling.json"))["temperature"]  # 1.0571
logits = model(x)  # (B, 2)
calibrated_probs = F.softmax(logits / T, dim=-1)
positive_prob = calibrated_probs[:, 1]
prediction = (positive_prob > 0.145).long()

Reproducibility

val_indices.json lists the 5337 patient IDs used as the held-out validation set. Its sha256 is recorded in evaluation_metrics.json and training_config.json. All metrics above are computed on exactly these patients. We publish this list rather than relying on random_state=42 because sklearn's stratified split is not invariant across releases.

DICOM preprocessing

The training pipeline applies the DICOM WindowCenter / WindowWidth if present, inverts MONOCHROME1 images so air is black, resamples to 224x224 with bilinear interpolation, and feeds a single grayscale channel. See training_config.json for the full recipe.

What this is and is not

This model is a teaching artifact. The recipe (linear probe -> partial unfreeze of the last 2 blocks) is the canonical foundation-model adaptation pattern, and the model card publishes the artifacts a clinical deployment would need to audit it. State-of-the-art RSNA Pneumonia models reach ROC-AUC ~0.93 with ensembles and aggressive augmentation; this single-model recipe is several points below that ceiling by design.

This model is not clinically deployable. The RSNA dataset uses surrogate labels ("Lung Opacity") with known label noise, and no real deployment ships without prospective evaluation against an institutional standard.

Failure analysis

natural_failure_gallery.png shows the highest-confidence false positives and false negatives on the held-out validation set, with DINOv2 attention rollout overlays. augmentation_failure_gallery.png shows augmentation-induced flips: examples the model originally classified correctly but flipped under one of 8 clinically-named perturbations. The lateral_flip augmentation is the laterality-shortcut probe.

Citation

If you use this model, please cite the underlying foundation model:

@misc{oquab2024dinov2,
  title  = {DINOv2: Learning Robust Visual Features without Supervision},
  author = {Oquab, Maxime and Darcet, Timothee and Moutakanni, Theo and
            Vo, Huy and Szafraniec, Marc and Khalidov, Vasil and
            Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and
            El-Nouby, Alaaeldin and Howes, Russell and Huang, Po-Yao and
            Xu, Hu and Sharma, Vasu and Li, Shang-Wen and Galuba, Wojciech and
            Rabbat, Mike and Assran, Mido and Ballas, Nicolas and
            Synnaeve, Gabriel and Misra, Ishan and Jegou, Herve and
            Mairal, Julien and Labatut, Patrick and Joulin, Armand and
            Bojanowski, Piotr},
  year   = {2024},
  eprint = {2304.07193},
  archivePrefix = {arXiv},
}

and the RSNA Pneumonia Detection Challenge.

t22000t/dinov2-small-rsna-pneumonia

DINOv2-Small fine-tuned on RSNA Pneumonia