This repository contains the systematic evaluation code from our paper "Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR".

Author: Richard J. Young Affiliation: DeepNeuro.AI | University of Nevada, Las Vegas Status: Paper Under Review

The Negative Result

We evaluated seven masking strategies (V3-V9) across different architectural layers of DeepSeek-OCR using 100 synthetic medical billing statements from a corpus of 38,517 annotated documents.

What We Found

PHI Type	Reduction Rate
Patient Names	✅ 100%
Dates of Birth	✅ 100%
Physical Addresses	✅ 100%
SSN	❌ 0%
Medical Record Numbers	❌ 0%
Email Addresses	❌ 0%
Account Numbers	❌ 0%
Overall	⚠️ 42.9%

Why It Matters

All strategies converged to 42.9% regardless of architectural layer (V3-V9)
Spatial expansion didn't help - mask radius r=1,2,3 showed no improvement
Root cause identified - language model contextual inference reconstructs masked short identifiers from document context
Hybrid approach needed - simulation shows 88.6% reduction when combining vision masking + NLP post-processing

What's Included

✅ Synthetic Data Pipeline: Generates 38,517+ annotated medical PDFs using Synthea ✅ PHI Annotation Tools: Ground-truth labeling for all 18 HIPAA categories ✅ Seven Masking Strategies: V3-V9 implementations targeting different DeepSeek-OCR layers ✅ Evaluation Framework: Code for measuring PHI reduction by category ✅ Configuration Files: DeepSeek-OCR integration settings

Quick Start

Installation

bash

git clone https://github.com/ricyoung/Justitia-Selective_Vision_Token_Masking_for_PHI-Compliant_OCR.git
cd Justitia-Selective_Vision_Token_Masking_for_PHI-Compliant_OCR

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

pip install -r requirements.txt

Generate Synthetic Data

bash

# Setup Synthea
bash scripts/setup_synthea.sh

# Generate patient data
bash scripts/generate_synthea_data.sh

# Create annotated PDFs
python scripts/generate_clinical_notes.py --output-dir data/pdfs --num-documents 1000

Explore the Code

bash

# View masking strategies
cat src/training/lora_phi_detector.py

# Check PHI annotation pipeline
cat src/preprocessing/phi_annotator.py

Research Contributions

First systematic evaluation of vision-level token masking for PHI protection in VLMs
Establishes boundaries - identifies which PHI types work with vision masking vs requiring language-level redaction
Negative result - proves vision-only approaches are insufficient for HIPAA compliance
Redirects future work - toward hybrid architectures and decoder-level fine-tuning

Architecture

Input PDF → Vision Encoder → PHI Detection → Vision Token Masking → DeepSeek Decoder → Text Output
            (SAM + CLIP)    (Ground Truth)   (V3-V9 Strategies)      (3B-MoE)
                                                    ↓
                                              42.9% Reduction
                                              (Insufficient!)

Seven Masking Strategies

V3-V5: SAM encoder blocks at different depths
V6: Compression layer (4096→1024 tokens)
V7: Dual vision encoders (SAM + CLIP)
V8: Post-compression stage
V9: Projector fusion layer

Result: All strategies converged to identical 42.9% reduction

Implications

⚠️ Vision-only masking is insufficient for HIPAA compliance (requires 99%+ PHI reduction)

✅ Hybrid architectures are necessary - combine vision masking with NLP post-processing

🔮 Future directions - decoder-level fine-tuning, defense-in-depth approaches

Use Cases

This code is useful for:

Researchers exploring privacy-preserving VLMs
Healthcare AI teams evaluating PHI protection strategies
Benchmarking alternative redaction approaches
Understanding VLM architectural limitations for sensitive data

Citation

bibtex

@article{young2025visionmasking,
  title={Vision Token Masking Alone Cannot Prevent PHI Leakage in Medical Document OCR: A Systematic Evaluation},
  author={Young, Richard J.},
  institution={DeepNeuro.AI; University of Nevada, Las Vegas},
  journal={Under Review},
  year={2025},
  url={https://huggingface.co/richardyoung/vision-token-masking-phi}
}

License

MIT License - See LICENSE for details

Contact

Richard J. Young 🌐 deepneuro.ai/richard 🤗 @richardyoung 💻 GitHub

Disclaimer

⚠️ Research project only - NOT for production use with real PHI. Always consult legal and compliance teams before deploying PHI-related systems.

Note: This negative result establishes important boundaries for vision-level privacy interventions in VLMs and redirects the field toward more effective hybrid approaches.