Contrastive (NT-Xent) pretraining on real handwritten digits with a large batch; a few-shot linear probe on the frozen features beats a random encoder on held-out digits.
Trained from scratch in Ropedia Academy — an interactive, bilingual course on embodied & spatial AI. Educational model: small and quick to train; the value is the method and a reproducible pipeline, not a leaderboard score. Try it live in the Ropedia demos Space.
At a glance
Base model
Trained from scratch (random initialization) — no pretrained base model.
Task
contrastive representation learning
Training objective
NT-Xent contrastive loss on two augmented views; a frozen linear probe measures the learned features.
Size / stats: 1,797 real 8×8 digit images, 10 classes; 256/batch contrastive (unlabeled); probe = 60 labels, averaged over 8 random subsets / 540 test
Split: 1,257 train / 540 test; few-shot linear probe
Source: scikit-learn load_digits (UCI Optical Recognition of Handwritten Digits)
Training config
Adam (lr 1.5e-3, cosine), 2500 steps, batch 256; NT-Xent τ=0.5; proj 64→128→64 + light cutout. Probe: 60 labels, averaged over 8 subsets (Adam lr 1e-2).
Evaluation results
metric
value
meaning
nt_xent (final)
5.412
probe_simclr
0.6826
few-shot linear-probe accuracy (60 labels, avg of 8 subsets) on held-out digits — higher = better
probe_random
0.5343
same probe on an untrained encoder — the baseline SimCLR must beat
figure
Robustness (mean ± std over 5 seeds)
Single-run numbers above are one seed; this is the distribution over independent re-trains (honest variance — no cherry-picking).
metric
mean ± std
probe_simclr
0.6705 ± 0.04
probe_random
0.5517 ± 0.031
seeds
Inference example
python
1import torch, torch.nn as nn
2enc = nn.Sequential(nn.Conv2d(1,32,3,1,1), nn.ReLU(), nn.MaxPool2d(2),3 nn.Conv2d(32,64,3,1,1), nn.ReLU(), nn.AdaptiveAvgPool2d(1), nn.Flatten())4enc.load_state_dict(torch.load("encoder.pt", map_location="cpu")); enc.eval()5# images: (N,1,8,8) float in [0,1] -> features = enc(images) # (N, 64)
Limitations
Educational scale. Trained quickly on CPU on small or synthetic data, so absolute numbers are not competitive with production systems — the value is the method and a reproducible pipeline. No large-scale data, no hyperparameter sweep, and no multi-seed variance is reported. Not for production use.
Features are learned on 8×8 digits — they will not transfer to natural images; needs a large batch to work.
Failure cases
Wrong augmentations (e.g. horizontal flips on digits) destroy the signal; collapses with too-weak/too-strong augmentation.
Reproduce / train your own
One click: open the notebook in Colab → Runtime → GPU → Run all, then run its Publish to the Hugging Face Hub cell.
