Layout-Aware Receipt Information Extraction
This repository contains the trained PyTorch weights for an end-to-end Visually Rich Document Understanding (VRDU) pipeline. The system is custom-built to extract key entities from receipts (based on the ICDAR 2019 SROIE dataset) by simultaneously analyzing text semantics and physical document layout.
Neural Network Architecture
- OCR Engine (CRNN): A Convolutional Recurrent Neural Network (CNN + 2-layer Bi-LSTM) trained with CTC Loss to accurately decode raw text from cropped image regions.
- Spatial Intelligence (GNN): A Multi-Head Graph Attention Network (GAT) featuring 128 hidden channels. It processes dynamic graphs where each node is a 392-D multi-modal vector (combining 384-D Sentence-Transformer semantic embeddings, 4-D spatial coordinates, and 4-D custom financial/date heuristics).
Quantitative Evaluation
The pipeline was rigorously evaluated on the unseen SROIE 2019 test split.
1. OCR Engine (CRNN) Performance
Evaluated over 33,618 cropped word instances to measure text extraction exactness.
| Metric | Score |
|---|
| Word Accuracy (Exact Match) | 94.35% |
| Character Error Rate (CER) | 0.92% |
2. Layout-Aware GNN Performance
Evaluated over 4,835 test sentences, prioritizing the F1-Score to counteract the severe class imbalance inherent in receipt data (where background noise vastly outweighs key entities).
| Metric | Score |
|---|
| Accuracy | 91.50% |
| Precision | 70.37% |
| Recall | 73.99% |
| F1-Score | 72.14% |
Intended Use
These weights (.pth files) are designed to be loaded into the custom PyTorch inference pipeline. For the complete training codebase, data preprocessing modules, and end-to-end inference scripts, please refer to the associated GitHub repository.
