Model Card for eynollah-main-regions-ensembled
This model is part of a suite of 13 models. The suite introduces an end-to-end pipeline to extract layout, text lines and reading order for historic documents, where the output can be used as an input for OCR engines.
Questions and comments about the models can be directed to Vahid Rezanezhad at
Vahid.Rezanezhad@sbb.spk-berlin.de.
Table of Contents
Model Details
Model Description
This suite of 13 models presents a document layout analysis (DLA) system for historical documents implemented by pixel-wise segmentation using convolutional neural networks. In addition, heuristic methods are applied to detect marginals and to determine the reading order of text regions.
The detection and classification of multiple classes of layout elements such as headings, images, tables etc. as part of DLA is required in order to extract and process them in subsequent steps. Altogether, the combination of image detection, classification and segmentation on the wide variety that can be found in over 400 years of printed cultural heritage makes this a very challenging task. Deep learning models are complemented with heuristics for the detection of text lines, marginals, and reading order. Furthermore, an optional image enhancement step was added in case of documents that either have insufficient pixel density and/or require scaling. Also, a column classifier for the analysis of multi-column documents was added. With these additions, DLA performance was improved, and a high accuracy in the prediction of the reading order is accomplished.
Two Arabic/Persian terms form the name of the model suite: عين الله, which can be transcribed as "ain'allah" or "eynollah"; it translates into English as "God's Eye" -- it sees (nearly) everything on the document image.
- Developed by: Vahid Rezanezhad
- Shared by: Staatsbibliothek zu Berlin / Berlin State Library
- Model type: Neural Network
- Language(s) (NLP): Irrelevant; works on all languages
- License: apache-2.0
- Resources for more information:
Uses
The intended use of the suite is performing document layout analysis (DLA) on image data. The system returns the results in
PAGE-XML format.
Direct Use
The system performs document layout analysis in a series of steps. First, the image is cropped and the number of columns determined. Then the pixels-per-inch (ppi) rate is detected, and when ppi is below 300, the image is re-scaled and enhanced. Now the main region types (text regions, images, separators, background) are detected as the early layout. Marginals are detected with a heuristic method, then -- optionally -- headers (or headings or floatings) and drop capitals. Next, textline segmentation is performed for text regions, and for each text region the slope of deskewing is calculated. Heuristics are used to determine bounding boxes (or contours in the case of curved lines) of textlines in each region after deskewing. After that, the reading order of text regions is detected based on separators, headers (or headings or floatings) and the coordinates of columns. Finally, all results are written to a PAGE-XML file.
Four models of the suite are focused on the implementation of early document layout detection techniques aimed at extracting essential attributes such as background, text regions, images, and separators from document images. To enhance the performance and adaptability of the models across diverse scenarios, a comprehensive training approach involving various augmentation procedures has been employed. By systematically experimenting with different augmentation techniques, efforts are made to achieve optimal results, ensuring that the models exhibit robustness and accuracy in effectively detecting the distinct components of documents in a wide range of real-world scenarios.
Within the suite, the model eynollah-main-regions-ensembled/2021-04-25/saved_model.pb is used for the task of early layout detection.
The robustness of this model is attained through an ensembling technique that combines the weights from various epochs. By employing this approach, the model achieves a high level of resilience and stability, effectively leveraging the strengths of multiple epochs to enhance its overall performance and deliver consistent and reliable results.
Downstream Use
The intended use of this suite of 13 models is conceived of as a system. Comparable systems are
dhSegment and
P2PaLA. However -- and as always with a system -- , every component can potentially used on its own. Each model might therefore be used or trained for other downstream purposes.
Out-of-Scope Use
This model suite does NOT perform any Optical Character Recognition (OCR), it is an image-to-PAGE-XML system only.
Bias, Risks, and Limitations
The pre-processing of digitised historical and contemporary texts is a task contributing to knowledge creation, both by developing models facilitating the necessary pre-processing steps of document layout analysis and ultimately by enabling better discoverability of information in the processed works. Since the content of the document images is not touched, ethical challenges cannot be identified. The endeavor of developing the model was not undertaken for profit. Though a commercial product based on this model may be developed in the future, it will always remain openly accessible without any commercial interest. The aim of the development of this model was to improve document layout analysis, an endeavour that is not for profit. As a technical limitation, it has to be noted that there is a lot of performance to gain for historical text by adding more historical Ground Truth data.
Recommendations
The application of machine learning models to convert a document image into PAGE-XML is a process which can still be improved. The suite of 13 models performs pixel-wise segmentation, which is done in patches; it therefore lacks a global understanding of documents and makes the document layout analysis system unable to detect some document subcategories like page numbers. The end-to-end system with different stages uses the output of each task for the next step. Therefore a poor prediction in early steps may cause a poor final document information extraction. Patch-wise segmentation can cause problems to segment pixels between text blocks, large scale drop capitals, headers and tables, since in each patch the model sees only a part of the document element and not all of it. Therefore, there is a lot to gain by improving the existing model suite.
Training Details
Training Data
For model training, Ground Truth in PAGE-XML format was sourced mainly from three datasets. The
IMPACT dataset of historical document images contains a representative sample of historical books and newspapers from Europe’s major libraries. Due to restrictions, only the open data from the National Library of the Netherlands (KB) and the Poznan Supercomputing and Networking Center (PSNC) were used. The
Europeana Newspapers Project (ENP) image and ground truth dataset of historical newspapers contains images representative of the newspaper digitisation projects of 12 national and major European libraries. The
OCR-D dataset of German prints between 1500 and 1900 is based on a selection from the holdings of the “German Text Archive” (DTA), the Digitized Collections of the Berlin State Library and the Wolfenbüttel Digital Library.
Training Procedure
All models, except for the column classifier, are designed for pixelwise segmentation. The training of these models follows a patchwise approach, wherein the documents are divided into smaller patches and fed into the models during training. To train these segmentation models, annotated labels are required. Each sub-element in the document is assigned a unique value for identification.
If you consider examining the training process, take a look at the repository which contains the source code for training an encoder model for document image segmentation
on GitHub.
Preprocessing
In order to use this suite of models for historical documents, no preprocessing is needed for the input images.
Speeds, Sizes, Times
The duration of training per epoch varies, typically lasting between 2 to 5 hours, depending on the specific use case, the datasets used, and the extent of applied data augmentation.
Our ResNet-50-U-Net model has 38.15M parameters where only parameters of the decoder part are fully trained (14.71M parameters).
In the case of the column classifier, we used a ResNet-50 with two dense connected layers at the top. The column classifier model has 25.6M parameters, where only parameters of dense layers are fully trained (2.16M parameters).
Training hyperparameters
Within the context of a constant model architecture, our hyperparameters encompass diverse augmentations, each characterized by its unique set of parameters. In addition to these, the model training process involves other hyperparameters, including the choice of the loss function, the number of batches utilized, the patch size of input images, and the number of epochs.
Training results
Training results are reported in
this paper.
Evaluation
Given the inadequacy of the prevailing segmentation metric in achieving optimal outcomes for document segmentation, particularly with respect to isolating regions as effectively as desired, we proceeded to evaluate our model using smaller batches of the three above-named datasets used for training. In pursuit of improved results, we employed an ensemble learning approach by aggregating the best epoch weights.
Testing Data, Factors and Metrics
Testing Data
Three new datasets were used for evaluation to ensure an unbiased comparison.
The PRImA Layout Analysis Dataset contains 478 pages of realistic documents, reflecting various challenges in layout analysis.
The German-Brazilian Newspapers (GBN) Dataset comprises 152 pages from historical newspapers. We used only 61 pages to keep the complexity of the documents similar. Finally, the (unreleased) Vlaamse Erfgoedbibliotheken (VEB) dataset comprises ground truth for 75 pages from historical Belgian newspapers, split across three sets.
Metrics
In addition to utilizing conventional performance metrics such as mean Intersection over Union (mIoU), precision, recall, and F1-score, we have incorporated the
Prima layout evaluation metrics, namely Merge, Split, Miss, and the overall success rate.
Model Examination
Examination results are reported in
this paper.
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: GeForce RTX 2070
- Hours used: 2 to 5 hours per epoch
- Cloud Provider: No cloud.
- Compute Region: Germany.
- Carbon Emitted: More information needed.
Technical Specifications
Model Architecture and Objective
Software
See the code published on
GitHub.
Citation
BibTeX:
1@inproceedings{10.1145/3604951.3605513,
2author = {Rezanezhad, Vahid and Baierer, Konstantin and Gerber, Mike and Labusch, Kai and Neudecker, Clemens},
3title = {Document Layout Analysis with Deep Learning and Heuristics},
4year = {2023},
5isbn = {9798400708411},
6publisher = {Association for Computing Machinery},
7address = {New York, NY, USA},
8url = {https://doi.org/10.1145/3604951.3605513},
9doi = {10.1145/3604951.3605513},
10abstract = {The automated yet highly accurate layout analysis (segmentation) of historical document images remains a key challenge for the improvement of Optical Character Recognition (OCR) results. But historical documents exhibit a wide array of features that disturb layout analysis, such as multiple columns, drop capitals and illustrations, skewed or curved text lines, noise, annotations, etc. We present a document layout analysis (DLA) system for historical documents implemented by pixel-wise segmentation using convolutional neural networks. In addition, heuristic methods are applied to detect marginals and to determine the reading order of text regions. Our system can detect more layout classes (e.g. initials, marginals) and achieves higher accuracy than competitive approaches. We describe the algorithm, the different models and how they were trained and discuss our results in comparison to the state-of-the-art on the basis of three historical document datasets.},
11booktitle = {Proceedings of the 7th International Workshop on Historical Document Imaging and Processing},
12pages = {73–78},
13numpages = {6},
14keywords = {Document layout analysis, Reading order detection, Segmentation},
15location = {San Jose, CA, USA},
16series = {HIP '23}
17}
APA:
(Rezanezhad et al., 2023)
More Information
SHA256 and MD5 hashes for the model eynollah-main-regions-ensembled/2021-04-25/saved_model.pb:
SHA256: 983b2fe2d76891acba3affe50ae45e712f9daa16b54b359022561d54b8cedd08
MD5: 70a33a786219a4eac4848301d812ecd9
Model Card Authors
Model Card Contact
Questions and comments about the model can be directed to
Vahid Rezanezhad, questions and comments about the model card can be directed to
Jörg Lehmann.
How to Get Started with the Model
How to get started with this model is explained in the ReadMe file of the GitHub repository
over here.
Model Card as of August 17th, 2023
