Patient-Timeline Transformer — Future Condition Forecasting (M31)
Decoder-only, nanoGPT-style transformer (~3.8M params; 4 layers / 4 heads / d_model 256,
context 1024) that predicts which of 40 target conditions are newly diagnosed in the
5 years after a patient's anchor, from their leak-free pre-anchor Synthea EHR timeline.
Two stages: self-supervised next-event pretraining then multi-label fine-tuning on
the [ANCHOR] hidden state (BCE and focal-loss variants).
- Code & full leak-free pipeline: https://github.com/Ziqi-Hao/m31-ehr-future-conditions
- Technical report:
ziqi_hao_results.pdf in that repo.
Results (held-out validation, 365 patients)
| Model | macro-AUROC | mAP |
|---|
| Transformer (BCE) | 0.759 | 0.229 |
| Transformer (focal, gamma=2) | 0.764 | 0.230 |
| Submitted 3-way ensemble (0.65 focal + 0.20 BCE + 0.15 LogReg) | 0.766 | 0.240 |
Honest 5-fold cross-validation of the submitted ensemble: macro-AUROC 0.768 +/- 0.004,
mAP 0.205 +/- 0.014 (prevalence-matched random-baseline mAP ~ 0.047).
Files
pretrained.pt — next-event pretrained backbonefinetuned.pt — BCE fine-tune (ensemble member)finetuned_focal.pt — focal-loss fine-tune (primary member, ensemble weight 0.65)vocab.json — tokenizer vocabulary + gap/age bucket sizes (needed to rebuild the model)
Reproduce
Clone the GitHub repo, drop the checkpoints in models/ and vocab.json in artifacts/,
then run python src/evaluate.py. Trained only on the provided training split; no external
data or pretrained weights.
