⚠️ Not a medical device. This is a research and educational project. It is not FDA/CE cleared, has not been clinically validated, and must not be used to diagnose, treat, or make any decision about a real patient. Outputs are frequently wrong. See Limitations — they are substantial and you should read them before using anything here.

What's in this repo

The X-ray and the symptoms go into one model, not two. The fusion model is a single network that consumes the radiograph and the symptom text together and emits one set of logits — fusion_full.onnx is one graph with three inputs (pixel_values, input_ids, attention_mask). There is no separate image classifier and text classifier whose outputs get merged afterwards; the two modalities are fused inside the model, before the classifier head. The BLIP captioner below is a separate, optional model that only writes a text description of the image — it takes no symptom input and plays no part in the diagnosis path.

Component	Path	Size	What it does
Fusion model (ONNX, end-to-end) — the main model	`checkpoints/onnx_full/fusion_full.onnx`	787 MB	X-ray and symptom text → diagnosis logits, in one graph. Runs with `onnxruntime` alone — no PyTorch.
Fusion classifier head (PyTorch)	`checkpoints/fusion_model.pth`	5.4 MB	Trained classifier head only; needs CLIP + Bio_ClinicalBERT at runtime.
Fusion classifier head (ONNX)	`checkpoints/onnx/fusion_classifier.onnx`	4.5 MB	Head-only ONNX; encoders still run in PyTorch.
BLIP X-ray captioner	`blip-xray-finetuned/`	896 MB	`Salesforce/blip-image-captioning-base` fine-tuned on IU-Xray reports → radiology-style caption.
Default/demo classifier	`models/default/fusion_classifier.onnx`	1.3 MB	15 NIH classes, random weights. Ships so the app runs before training. Not predictive.
Application code	`.py`, `launch.`, `config.json`	—	CLI, Gradio web UI, batch predictor, training and ONNX export scripts.

The training dataset is not included — see Data.

Architecture

Fusion (Symptom Check) — one multimodal classifier over both inputs. Both encoders feed a single shared head, so the prediction is a joint function of the image and the symptoms; neither modality is scored on its own:

image ──► CLIP ViT-B/32 vision tower ──► visual_projection ──► L2-normalize ──┐
                                                                              ├─► concat ──► MLP classifier ──► logits
symptom text ──► Bio_ClinicalBERT ──► mean-pool last_hidden_state ────────────┘

Encoders are frozen; only the MLP head is trained. fusion_full.onnx bakes the whole graph — encoders included — into one file, which is why it is 787 MB.

ONNX signature (opset 14, dynamic batch and sequence length):

	Name	Shape	dtype
in	`pixel_values`	`[batch, 3, 224, 224]`	float32
in	`input_ids`	`[batch, seq_len]`	int64
in	`attention_mask`	`[batch, seq_len]`	int64
out	`logits`	`[batch, 3018]`	float32

Preprocess with CLIPProcessor (openai/clip-vit-base-patch32) for the image and AutoTokenizer (emilyalsentzer/Bio_ClinicalBERT) for the text. Class names are in checkpoints/onnx_full/labels.json, index-aligned to the logits.

Vision (captioning) — a separate BLIP model, image-only, loadable with BlipForConditionalGeneration.from_pretrained. It does not see the symptoms and does not feed the fusion model; it exists to write a human-readable description alongside the diagnosis.

Usage

ONNX, no PyTorch

python

import json
import numpy as np
import onnxruntime as ort
from PIL import Image
from transformers import CLIPProcessor, AutoTokenizer
from huggingface_hub import hf_hub_download

repo = "GAD-Research-Lab/MedicalAI-Light-Weight"
onnx_path = hf_hub_download(repo, "checkpoints/onnx_full/fusion_full.onnx")
labels = json.load(open(hf_hub_download(repo, "checkpoints/onnx_full/labels.json")))

clip = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
tok = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

image = Image.open("xray.jpg").convert("RGB")
pixel_values = clip(images=image, return_tensors="np")["pixel_values"]
text = tok("cough and fever", return_tensors="np", padding="max_length",
           truncation=True, max_length=64)

sess = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])
logits = sess.run(["logits"], {
    "pixel_values": pixel_values.astype(np.float32),
    "input_ids": text["input_ids"].astype(np.int64),
    "attention_mask": text["attention_mask"].astype(np.int64),
})[0]

probs = np.exp(logits - logits.max()) / np.exp(logits - logits.max()).sum()
top = probs[0].argmax()
print(labels[top], float(probs[0][top]))

BLIP captioning

python

from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image

repo = "GAD-Research-Lab/MedicalAI-Light-Weight"
processor = BlipProcessor.from_pretrained(repo, subfolder="blip-xray-finetuned")
model = BlipForConditionalGeneration.from_pretrained(repo, subfolder="blip-xray-finetuned")

inputs = processor(Image.open("xray.jpg").convert("RGB"), return_tensors="pt")
print(processor.decode(model.generate(**inputs, max_new_tokens=64)[0],
                       skip_special_tokens=True))

Full application

bash

git clone https://huggingface.co/GAD-Research-Lab/MedicalAI-Light-Weight
cd MedicalAI-Light-Weight
pip install -r requirements.txt gradio
python web_ui.py          # http://127.0.0.1:7860

Or launch.ps1 (Windows) / launch.sh (Linux/macOS) to set up a venv and start the UI in one step. python run.py for the interactive CLI, python batch_predict.py <dir> -o out.csv for batch.

Training

Fusion head — cross-entropy over the label set, 85/15 random train/val split, AdamW, frozen encoders. python training.py --mode train --epochs 10 --batch_size 8.
BLIP — fine-tuned on IU-Xray image/report pairs. python xray_training.py --mode train --epochs 3 --batch_size 4 --max_samples 500.

Sources: IU-Xray (~6,687 rows), NIH Chest X-ray (~3,000 rows), plus 160 synthetic rare-finding rows from expand_dataset.py — ~9,847 total.

Limitations

Read these. They are not boilerplate.

The label space is degenerate. labels.json has 3,018 classes, and most are not diagnoses — they are raw, deduplicated report strings scraped from IU-Xray, e.g. "findings: . impression: 1. all lines and tubes in stable , xxxx position...". Only a handful (atelectasis, cardiomegaly, consolidation, edema, effusion, emphysema, fibrosis, …) are clean condition names. With ~9.8k training rows across 3,018 classes there are roughly 3 examples per class, and the reported "confidence" is a softmax over that space — it is not calibrated and should not be read as a probability of disease. Treat the classifier as a demonstration of the architecture, not as a working diagnostic.
No held-out evaluation is published. Training reports validation loss only. There is no accuracy, AUROC, sensitivity/specificity, or per-class breakdown in this repo, and no evaluation on an external site or scanner. Nothing here supports a claim about real-world performance.
Encoders are frozen and general-purpose. CLIP ViT-B/32 was pretrained on web images, not radiographs. Only a small MLP adapts it to this domain.
Narrow data. Two US datasets, adult chest radiographs, frontal views, English free-text reports. Expect degradation on pediatric films, lateral views, other modalities, other populations, and other equipment. Known demographic and label-noise problems in IU-Xray and NIH ChestX-ray14 are inherited wholesale — NIH labels were themselves NLP-mined from reports and are noisy.
BLIP captions are fluent, not faithful. The captioner will produce confident, plausible, well-formed radiology prose for an image it has no ability to read correctly, including for non-X-ray inputs. Fluency here carries no signal about correctness.
models/default/ is random weights by construction, so the app can start before training. Its predictions are noise.
Automation bias is the main risk. The most likely harm from this repo is a person believing a confident-looking output. Do not put it in front of patients or in any workflow where a wrong answer reaches one.

Data

The training data is not redistributed here — download it yourself with python training.py --mode prepare-data.

IU-Xray (Indiana University / Open-i, NLM) — de-identified, public.
NIH ChestX-ray14 (NIH Clinical Center) — de-identified, public.

Both are de-identified at source; no PHI is contained in this repo. The BLIP model was fine-tuned on IU-Xray report text and can emit dataset artifacts such as xxxx anonymization tokens. Check each dataset's own terms before redistributing derivatives.

License

Apache-2.0 for the code and the trained weights in this repo. Upstream components carry their own licenses — CLIP (MIT), BLIP (BSD-3-Clause), Bio_ClinicalBERT (MIT) — and the source datasets carry their own terms. The license permits use; it does not make the model safe or fit for clinical use.

Citation

bibtex

@software{medicalai_light_weight,
  title  = {MedicalAI — Light Weight: CPU-friendly chest X-ray analysis},
  author = {GAD Research Lab},
  year   = {2026},
  url    = {https://huggingface.co/GAD-Research-Lab/MedicalAI-Light-Weight}
}