PsiDDx — Clinical Differential-Diagnosis Model (gemma-3-4b-it + LoRA)

This model card is a study in honest measurement. The scored result is real; the external benchmark below is reported transparently, including a tradeoff most fine-tunes never check for.

Results — read both metrics

• The win is on preference (74%) — the scored metric. The model produces a ranked, calibrated, common-first differential that the judge prefers 3:1 over base.
• On a deliberately rare-case external benchmark, it trades accuracy (11.3 / 36.7 vs base 14.7 / 48.7). By design it is anti-"zebra" — it leads with common explanations — so on a set built entirely from rare "zebras," it deprioritizes the rare answer and scores lower.
• Preference and rare-case accuracy are different axes. We report both, on purpose.
• We mapped the dial. A gentler sibling (v5.2, r=8) holds Top-1 at 14.0 ≈ base for a 67% win rate; this release (v5.3, r=16) is the higher-preference point (74%) at a Top-1 cost. The Top-5 tradeoff (~37 vs 49) is shared by both — a structural effect of common-first calibration, not recipe.

The story — why we trust measurement over a score

1. An early full fine-tune scored a flashy 75% win rate — but benchmarking the actual weights showed it had regressed on rare cases (11.3 / 31.3). Root cause: ~43% of training rows were multiple-choice items where the pipeline had scraped the answer (a drug, an organism, a lab value) and mislabeled it the "diagnosis." A high score hid a worse diagnostician.
2. We decontaminated the data (v5.2) and switched to a knowledge-preserving LoRA (r=16). Win rate 74%.
3. We then measured this model correctly on NEJM-CPC and found it still trades rare-case accuracy for its preferred common-first calibration. We report that here rather than bury it — it's the whole point: a preference win is not an accuracy guarantee.

Intended use & safety

• Not a medical device. Not medical advice. Not for live diagnosis. Because it trades rare-case recall, it must never be relied on to rule out a rare or dangerous condition.
• The companion app (Yara) uses this model as a released artifact behind a hard red-flag safety guard and routes to real care — never as a live diagnostician.
• A small (4B) model: it can hallucinate. English only.

Usage (LoRA adapter on the base)

Note: this adapter targets the language-model layers. On the multimodal gemma-3-4b-it, ensure the adapter attaches to language_model.* modules (verify base≠adapted output before trusting results).

Training

• AutoScientist (Adaption Labs) — LoRA r=16 / α=32, lr 1e-4, 1 epoch, SFT, all-linear targets.
• Base frozen; only the adapter is trained. Data: the decontaminated v5.2 set.

Citation