Clinical NLP Fairness Auditor — DistilBERT Medical Specialty Classifier
Author: Steven Hill Date: 2026-04-24 Base model: distilbert-base-uncased Task: Multi-class text classification (20 medical specialties) Project: Clinical NLP Fairness Auditor — AI governance portfolio project
Model Description
This model fine-tunes DistilBERT on the MTSamples clinical transcription dataset
to classify medical notes by specialty. It is developed as part of a portfolio
project demonstrating responsible AI practices including fairness auditing,
SHAP explainability, and NIST AI RMF governance documentation.
This model is NOT intended for clinical use. It is a research and
demonstration artifact only.
Intended Use
Intended uses
Research and educational demonstration of clinical NLP
Portfolio demonstration of AI fairness auditing methodology
Teaching tool for AI governance practices in healthcare contexts
Out-of-scope uses
Clinical diagnosis or treatment recommendations
Autonomous patient care decisions
Production deployment without further development and independent audit
Any use involving real patient data without appropriate IRB approval and de-identification
Training Data
Dataset: MTSamples (via galileo-ai/medical_transcription_40 on HuggingFace Hub)
Source: Public medical transcription examples from mtsamples.com
Size: 4,087 notes after filtering (≥50 examples per specialty)
Split: 70% train / 15% validation / 15% test (stratified)
Classes: 20 medical specialties:
Cardiovascular / Pulmonary
Consult - History and Phy.
Discharge Summary
ENT - Otolaryngology
Emergency Room Reports
Gastroenterology
General Medicine
Hematology - Oncology
Nephrology
Neurology
Neurosurgery
Obstetrics / Gynecology
Ophthalmology
Orthopedic
Pain Management
Pediatrics - Neonatal
Radiology
SOAP / Chart / Progress Notes
Surgery
Urology
Data limitations
Not drawn from live EHR systems — may not generalize to real clinical workflows
No patient demographic information — demographic fairness audit not possible
Class imbalance: Surgery accounts for ~24% of training data
20.7% of notes truncated at 512 tokens — longer notes systematically disadvantaged
Training Procedure
Base model: distilbert-base-uncased
Framework: HuggingFace Transformers, PyTorch
Epochs: 3
Batch size: 16
Learning rate: 2e-5 with warmup ratio 0.1
Imbalance mitigation: Inverse-frequency class weights applied to cross-entropy loss
Across specialty types (procedural/consultative/diagnostic)
0.3405
Across note length quartiles
0.323
Fairness and Bias Findings
Critical findings (DO NOT DEPLOY without remediation)
Zero-F1 specialties — model completely fails to identify these note types:
Pediatrics - Neonatal
Hematology - Oncology
High-risk findings (FNR ≥ 70% — model systematically misses these specialties)
Pediatrics - Neonatal
Hematology - Oncology
Consult - History and Phy.
Surgery
General Medicine
Root cause analysis
Class imbalance: Surgery dominates training data (24%), causing the model
to over-predict high-frequency specialties
Class weighting over-correction: Inverse-frequency weighting reduced Surgery
recall to 8.8% — a new disparity created by the mitigation attempt
Truncation: 20.7% of notes exceed 512 tokens; Q3 length quartile accuracy
(0.261) is 32 points below Q1 (0.584)
Vocabulary overlap: Surgery misclassifications scatter across 9 specialties,
indicating shared procedural vocabulary confuses the model
Explainability
SHAP PartitionExplainer was applied to 4 representative examples to generate
token-level attribution scores. Key finding: the model attends to clinically
meaningful vocabulary (procedure names, anatomical terms) for correct predictions,
but relies on shared procedural language when making misclassifications.
Explainability artifacts: token attribution plots for correct prediction,
incorrect prediction, zero-F1 class, and best-performing class.
Governance Documentation
This model was developed in alignment with the NIST AI Risk Management Framework
(AI RMF 1.0, January 2023). A full NIST AI RMF scorecard is available in the
project repository covering all four functions: GOVERN, MAP, MEASURE, and MANAGE.
Deployment recommendation: DO NOT DEPLOY in production without addressing
critical risk items identified in the bias audit. See nist_rmf_scorecard.md
for full findings and recommended mitigations.
Citation
@misc{hill2025clinicalnlp,
author = {Hill, Steven},
title = {Clinical NLP Fairness Auditor},
year = {2025},
note = {AI governance portfolio project. NIST AI RMF aligned.}
}
This model was built explicitly to demonstrate responsible AI development practices,
not to be deployed. The fairness audit intentionally surfaces failure modes rather
than optimizing headline accuracy. The governance documentation is intended to model
the kind of transparency that healthcare AI systems should provide before deployment.
The absence of demographic data in the training set is itself documented as a
governance gap — a model that cannot be evaluated for demographic fairness should
not be deployed in healthcare settings without remediation.
