BERT-ESI-Triage v58 — BiomedBERT multi-head triage classifier
TL;DR
bert-esi-triage-v58 is a fine-tuned BiomedBERT classifier for the
Emergency Severity Index (ESI 1-5) triage decision. v58 is the
successor to v57 (epoch 3) on the same 18-eval suite. Headline
improvements vs v57 e3:
- MIMIC-IV-ED ESI 1 recall: +4.0pp ensemble (78.1% → 82.1%).
- MC-MED Stanford ESI 1 recall: +7.0pp (77.0% → 84.0%).
- MIETIC ensemble ESI 1 recall held at 96.7%.
- Lukina v3 ESI 1 recall held at 94.3%.
- Concept-density / frequent-flyer / polypharmacy / geriatric / multi-CC
subgroup slices all win ESI 1 recall by +1.6 to +6.7pp.
Trade-off: cal_exact regresses by 2-6pp on MIETIC, judgment_gap,
cardiac_arrest, and sepsis slices (a recall-for-safety shift).
Architecture differences vs v57
- No engineered-feature fusion. v57 trained with an 11-d MEWS / qSOFA
/ shock_index feature vector projected to 64-d and concatenated with
CLS into
esi_head Linear(832, 5). v58 drops the fusion path and uses
esi_head Linear(768, 5) directly on CLS. The encoder learns vital
patterns from raw text alone.
- Trainer. v58 uses
train_bert_v44_bf16.py (bf16 native on GB10);
v57 used train_bert_v44_v53_3.py (feature-fusion variant).
- Symptom head dim. v58 = 199, v57 = 203 (concept vocab pruning).
Inference
Load with PyTorch + transformers; CLS-pooled features feed each head:
esi_head: Linear(768, 5) — main output.- 19 auxiliary heads: symptom, flag, pain, arrival, gestalt,
disposition, resource, vitals, etc.
The companion Space vadimbelsky/esi-triage-demo ships the full
calibration stack (demographic normalizer → per-dialect temperature →
confidence-aware ESI 1 bias → deterministic engine min ensemble).
License
- Model weights: MIT.
- Training data: private (MIMIC-IV-ED + Stanford MC-MED + MIETIC +
Lukina v3 + curated synth).
⚠️ Not a medical device. Not for clinical use. This is a research
artifact distributed for reproducibility and benchmarking. Real
deployment requires institutional validation + IRB + local recalibration.
Performance — v58 epoch 3 vs v57 epoch 3
Per-slice ESI 1 recall (calibrated):
| Slice | n | v57 cal_ESI1_R | v58 cal_ESI1_R | Δ |
|---|
| mimic_holdout | 7,917 | 76.6% | 80.2% | +3.6pp |
| mcmed_clean_holdout | 1,000 | 77.0% | 84.0% | +7.0pp |
| mietic_clean_holdout | 200 | 90.0% | 86.7% | -3.3pp |
| lukina_v3 | 201 | 94.3% | 94.3% | 0pp |
| sepsis_eval_v1 | 93 | 95.0% | 95.0% | 0pp |
| stroke_eval_v1 | 97 | 95.0% | 92.5% | -2.5pp |
| anaphylaxis_eval_v1 | 90 | 97.1% | 97.1% | 0pp |
| cardiac_arrest_eval_v1 | 103 | 93.3% | 90.0% | -3.3pp |
| ob_emergency_eval_v1 | 98 | 95.0% | 90.0% | -5.0pp |
| pediatric_n200_v2 | 200 | 95.0% | 95.0% | 0pp |
| judgment_gap_v1 | 177 | 88.9% | 83.3% | -5.6pp |
| geriatric_n200_v1 | 200 | 78.8% | 80.0% | +1.3pp |
| polypharmacy_n200_v1 | 200 | 72.5% | 76.2% | +3.7pp |
| frequent_flyer_n200_v1 | 135 | 80.0% | 86.7% | +6.7pp |
| vital_completeness | 231 | 83.3% | 86.7% | +3.3pp |
| multi_cc_complexity | 304 | 76.2% | 77.8% | +1.6pp |
| concept_density | 260 | 80.0% | 86.7% | +6.7pp |
Verdict: 8 wins / 5 losses / 5 neutral on ESI 1 recall (mean +0.78pp).
MIMIC per-class recall (calibrated):
| Class | n | v57 R | v58 R | Δ |
|---|
| ESI 1 | 475 | 76.6% | 80.2% | +3.6pp |
| ESI 2 | 2,582 | 64.6% | 62.7% | -1.9pp |
| ESI 3 | 4,286 | 54.4% | 50.1% | -4.4pp |
| ESI 4 | 554 | 64.6% | 67.7% | +3.1pp |
| ESI 5 | 20 | 40.0% | 40.0% | 0pp |
Known gaps (carried to v59)
- SYMPTOM_LABELS index mismatch. v58 symptom_head outputs 199
logits; the demo's SYMPTOM_LABELS list (inherited from v47/v49) has
178 canonical entries plus runtime padding for the extra positions.
Specific concepts that grew between v57→v58 may be mislabeled in the
Space UI. Tracked for v59 (task #463).
- ESI 5 recall ≤ 40% across slices — sparse class, dialect-noisy
labels. v58 holds the v57 floor but does not improve it.
- MIETIC / OB / cardiac_arrest ESI 1 R regressions vs v57. Likely
driven by dropping the engineered-feature signal on narrative slices
where vitals are inline rather than parsed.
- Engineered features no longer in graph. Downstream stacks that
computed MEWS / qSOFA / shock_index and passed them to v57 must drop
that path for v58.
Companion Space
vadimbelsky/esi-triage-demo — 19-head dashboard with engine ensemble,
demographic normalizer, dialect detection, per-dialect calibration. The
Space loads this model and exposes the full calibrated pipeline.
