Universal Diagram Structure Understanding Model (UDSUM)
Model Details
Model Description
The Universal Diagram Structure Understanding Model (UDSUM) is a hybrid computer vision and graph reasoning system designed to extract structured representations from diagram images such as flowcharts, architecture diagrams, ER diagrams, and data flow diagrams.
The model performs:
Object detection for diagram elements
Relationship inference between elements
Graph construction
Flow path extraction
Loop detection
Connected component analysis
It outputs a machine-readable JSON representation of the diagram structure.
Developed by: Anurag Singha
Model type: Vision Transformer + Detection + Graph Inference
Language(s): N/A (image-based)
License: CC-BY-NC 4.0
Framework: PyTorch
Parameters: ~29.7M
Intended Uses & Limitations
Intended Uses
Diagram understanding
Workflow extraction
Automated testcase generation
Visual knowledge graph construction
Documentation automation
Out-of-Scope Uses
Medical, legal, or financial decision making
Surveillance or biometric identification
Safety-critical system control
Limitations
Arrow detection performance is moderate
Hand-drawn diagrams are not supported
BPMN and UML semantics are not modeled
Overlapping or crossing connectors may be misinterpreted
Training
Training Data
Dataset: robo-flowchart (Roboflow)
Format: COCO-style annotations
Train size: 12,525 images
Validation size: 1,000 images
Labels: shapes, connectors
Training Procedure
Backbone: ResNet-50
Detection: RT-DETR-Large
Epochs: 25
Batch size: 4
Image size: 640×640
Loss: CE + L1 + GIoU
Optimizer: AdamW
Scheduler: Cosine Annealing
Training time: ~4.7 hours on Kaggle GPU
Evaluation
Metrics
Metric Value
Recall@0.5 0.684
Recall@0.75 0.305
Mean IoU 0.571
Structural performance (estimated):
Element detection: ~88%
Edge inference: ~78%
Flow path extraction: ~80%
How to Use
Installation
pip install torch torchvision albumentations opencv-python networkx
Loading the Model
from model import load_model, analyze_image
model = load_model("udsum.pt")
result = analyze_image("diagram.jpg", model)
Output
{
"metadata": {...},
"elements": [...],
"relationships": {...},
"flow_analysis": {...}
}
Bias, Risks, and Ethical Considerations
No personal or biometric data used
No demographic attributes inferred
Risk of misinterpretation in complex diagrams
Should not be used for high-stakes automation
Citation
@misc{singha2025udsum,
title={Universal Diagram Structure Understanding Model},
author={Singha, Anurag},
year={2025},
note={Unpublished research}
}
Contact
Author: Anurag Singha
github-
https://github.com/Asin007
