The med-doc-classifier is a lightweight, encoder-only model (~96M parameters) built on the popular SigLIP2-base image backbone. In a single pass it sorts an image into one of 27 classes across two hierarchical levels, from broad group down to specific leaf, and answers two further questions: whether the subject matter is medical, and whether it was handwritten.

The model was built by training many specialised models, ensembling them, and distilling the ensemble into this one compact student.

Live demo · Launch blog

Results

Full internal test = 16,204 images (see benchmark_results.json).

Precision	Size	Accuracy (27-class)	Macro-F1
bf16	193 MB	96.45	0.965
int8	103 MB	96.43	0.965
int4	60 MB	95.22	0.955

int8 is effectively lossless; int4 trades about a point of accuracy for a model over 3× smaller that runs comfortably on a plain CPU. It matches or beats models several times its size, including a 428M SigLIP2-SO400M variant (96.24%) and a 256M generative VLM baseline (90.42%).

Taxonomy (5 L1 groups → 27 L2 leaves)

Document  (10 leaves)
    ├─ Letter                 (key: letter)
    ├─ Other document         (key: other_document)
    ├─ Scan interpretation    (key: scan_interpretation)
    ├─ Lab / diagnostic report (key: lab_diagnostic_report)
    ├─ OPD Consultation Record (key: opd_consultation_record)
    ├─ Discharge summary      (key: discharge_summary)
    ├─ Insurance document     (key: insurance_document)
    ├─ Form                   (key: form)
    ├─ Invoice / bill         (key: invoice_bill)
    ├─ Certificate            (key: certificate)
Card / credential  (7 leaves)
    ├─ ABHA card              (key: abha_card)
    ├─ Aadhaar card           (key: aadhaar_card)
    ├─ PAN card               (key: pan_card)
    ├─ Voter ID               (key: voter_id)
    ├─ PMJAY / Ayushman       (key: pmjay_ayushman)
    ├─ Insurance e-card       (key: insurance_e_card)
    ├─ Other card             (key: other_card)
Diagnostic imaging  (5 leaves)
    ├─ X-ray                  (key: x_ray)
    ├─ CT                     (key: ct)
    ├─ MRI                    (key: mri)
    ├─ Ultrasound             (key: ultrasound)
    ├─ Other (Diagnostic imaging) (key: other_diagnostic_imaging)
Body images  (3 leaves)
    ├─ Skin / wound           (key: skin_wound)
    ├─ Headshot               (key: headshot)
    ├─ Other (Body images)    (key: other_body_images)
Miscellaneous  (2 leaves)
    ├─ Medication image       (key: medication_image)
    ├─ Other (Miscellaneous)  (key: other_miscellaneous)

Usage

python

from transformers import AutoModel
from PIL import Image

model = AutoModel.from_pretrained("ekacare/med-doc-classifier", trust_remote_code=True).eval()
img = Image.open("doc.jpg")

model.classify(img)

jsonc

// example return value
{
  "l2":          { "source": "flat", "key": "lab_diagnostic_report",
                    "value": "Lab / diagnostic report", "confidence": 0.97 },
  "l1":          { "value": "Document", "source": "inferred_from_flat_l2" },
  "medical":     { "value": "medical", "confidence": 0.99, "p_positive": 0.99 },
  "handwritten": { "value": "printed", "confidence": 0.98, "p_positive": 0.02 }
}

By default L2 comes from a flat 27-way head and L1 is inferred from it. Other options:

python

model.classify(img, scope="l1")           # L1 group only (no L2)
model.classify(img, scope="hierarchical") # L1 head → that group's leaf L2 head
model.classify(img, l1="Document")        # fix L1 → that group's leaf L2 head
model.classify(img, top_k=3)              # top-3 candidates for the multi-class heads

medical and handwritten are independent binary heads — disable them with medical=False / handwritten=False.

Quantized loading (optimum-quanto — CPU & GPU)

Only the vision tower is quantized; the heads stay full precision.

python

# pip install optimum-quanto
from transformers.dynamic_module_utils import get_class_from_dynamic_module

load_classifier = get_class_from_dynamic_module(
    "modeling_siglip2_hier.load_classifier", "ekacare/med-doc-classifier")

model = load_classifier("ekacare/med-doc-classifier", quantization="int4")  # None | "int8" | "int4"
model.classify(img)

Working from a clone instead? from modeling_siglip2_hier import load_classifier inside the repo directory works too, and load_classifier(".", quantization="int8") loads straight from the checkout.

Citation

If you use this model, please cite:

bibtex

@software{med_doc_classifier,
  author = {{Eka Care}},
  title  = {Med Doc Classifier: hierarchical classification of health-app document uploads},
  year   = {2026},
  url    = {https://huggingface.co/ekacare/med-doc-classifier}
}