Project: Digital Health Twin Prototype
Developed by: Nikhilesh Patil(Hugging Face: nikhilesh123)
This project is part of an initiative to explore the AI/ML healthcare sector.
AI/ML Digital Health Twin Prototype Application for Analyzing Medical Reports (PDF)
This application uses a Large Language Model (LLM) to analyze medical reports and images and generate text-based insights.
Digital Health Twin Prototype
A comprehensive AI-powered application that digitizes medical lab reports. This tool ingests raw medical PDFs, extracts structured data, provides AI-generated health insights, and generates a portable QR code for data interoperability.
Important Disclaimer
Please Read Before Use:
This application is a Proof of Concept (PoC) and an experimental prototype.
Accuracy: The PDF ingestion and OCR (Optical Character Recognition) mechanism may occasionally misinterpret data or structure. Always verify the extracted values against the original PDF.
Not Medical Advice: The summaries, precautions, and risk scores generated by the AI are for demonstration purposes only. They should not be used as a substitute for professional medical advice, diagnosis, or treatment.
Key Features:
-
PDF Ingestion: Upload any standard medical lab report (PDF). The system analyzes the document structure to extract key health metrics (e.g., Glucose levels).
-
AI Analysis: Uses advanced LLMs to generate: Easy-to-understand summaries of complex medical jargon.
Personalized precautions and lifestyle recommendations.
Routine follow-up suggestions.
-
Structured Data: Converts unstructured PDF text into clean JSON format for easy integration.
-
QR Code & Simulation: Generates a QR code containing the digitized health record. Includes a "Simulate QR Scan" feature to demonstrate how data can be retrieved instantly by healthcare providers.
Current Limitations & Future Roadmap
This project is a Proof of Concept (PoC) designed to demonstrate the feasibility of the architecture. Users should be aware of the following:
- OCR & Ingestion: The current extraction pipeline (Tesseract/Gemini) may struggle with:
Complex, multi-column table layouts in PDF reports.
Low-resolution or handwritten scans.
-
LLM Hallucinations: Occasionally, the AI summary may misinterpret misread numerical values from the OCR step ("Garbage In, Garbage Out").
-
Accuracy: The current extraction accuracy is estimated at ~60-70%.
Future Improvements:
-
Implement RAG (Retrieval-Augmented Generation) to ground the AI responses in verifiable medical text.
-
Fine-tune the vision model on a specific dataset of lab reports to improve table recognition.
-
Add a "Human-in-the-Loop" validation step where users can correct the extracted JSON before generation.
How It Works
- Upload: Drag and drop your medical lab report (PDF) into the application.
- Analyze: The system processes the file, extracting values (e.g., "Glucose: 120 mg/dL") and calculating risk scores (e.g., "Diabetes Risk").
- Review: Read the AI-generated summary and extracted JSON data.
- Connect: A QR code is generated. Click "Simulate QR Scan" to see how this data is retrieved and displayed in a structured format.
Tech Stack -
User Interface:
Streamlit For building the interactive web dashboard.
OCR & Text Extraction:
-
pytesseract (Tesseract OCR) - For extracting text from scanned images/PDFs.
-
pdf2image - Converts PDF pages into images for OCR processing.
-
PyPDF2 - For parsing and analyzing PDF file structures.
-
Image Processing: Pillow (PIL) - Handles image manipulation and analysis.
-
Utilities: qrcode - Generates the scannable QR code for the digital health twin.
-
LLM: For secure and private local inference, I employed the Meta Llama 3 8B Instruct LLM, quantized with Q4_K_M, running entirely on-device via llama.cpp.
-
Programming Language: Python
How to Run the Application
To run this application locally, you must first ensure all dependencies are installed and the model file is in the correct location.
-
Install the core libraries:
All required libraries are listed in the requirements.txt file. You can install them by running:
pip install -r requirements.txt
-
Install the Llama library:
The llama-cpp-python library needs to be installed separately to work with the .gguf model file.
pip install llama-cpp-python
-
Model File Location:
Make sure the model file, Meta-Llama-3-8B-Instruct-Q4_K_M.gguf, is located inside the application folder.
-
Run the Application:
Once all dependencies are installed, you can start the Streamlit application with the following command:
streamlit run app.py
Tips for Optimal Performance
For the best results, it is important to use clear and high-quality images. Blurred or low-quality images may affect the model's ability to accurately generate a response.
Here are examples of the kind of clear images that lead to optimal results:
⚖️ Architectural Comparison: Cloud vs. Edge
To address the critical trade-off between model capability and data privacy in healthcare, I engineered two distinct versions of this application.
Note on Accuracy: Both versions are experimental prototypes. While the Cloud version generally outperforms the Local version, both currently face challenges with low-resolution PDF ingestion.
1. Privacy-First Version (Current Repository)
- Architecture: Streamlit + Meta Llama 3 8B (Quantized) + llama.cpp.
- Focus: Prioritizes Zero-Trust Privacy. This version runs entirely offline on consumer hardware, ensuring patient data never leaves the device.
- Trade-off: Running an 8B parameter model locally forces a significant trade-off in reasoning capability compared to massive cloud models, resulting in more frequent hallucinations on complex data.
- Status: Source code and model implementation available here.
2. Cloud-Native Version (Alternative)
- Architecture: Gradio + Google Gemini (Multimodal).
- Focus: Leverages the Gemini 2.5 Flash model to analyze structured lab data. It transforms raw extracted values into clear, human-readable medical summaries and provides actionable next steps for patients.
- Status: Deployed as a live demo.
- [Link to Hugging Face Space] : https://huggingface.co/spaces/nikhilesh123/medical
