Mathematical Formula Recognition (MFR) model from Pix2Text (P2T).
Model Details / 模型细节
This MFR model utilizes the TrOCR architecture developed by Microsoft, starting with its initial values and retrained using a dataset of mathematical formula images.
The resulting MFR model can be used to convert images of mathematical formulas into LaTeX text representation. More detailed can be found: Pix2Text V1.0 New Release: The Best Open-Source Formula Recognition Model | Breezedeus.com.
Purpose: This model is a mathematical formula recognition model, capable of converting input images of mathematical formulas into LaTeX text representation.
Limitation: Since the model is trained on images of mathematical formulas, it may not work when recognizing other types of images.
This method requires the installation of pix2text, utilizing the Mathematical Formula Detection model (MFD) within Pix2Text. It is capable of recognizing not only pure formula images but also mixed images containing text.
The original images for the test data are derived from real data uploaded by users on the Pix2Text Online Service. Initially, real user data from a specific period is selected, and then the Mathematical Formula Detection model (MFD) within Pix2Text is used to detect the mathematical formulas in these images and crop the corresponding parts. A subset of these formula images is then randomly chosen for manual annotation to create the test dataset. The following image shows some sample pictures from the test dataset. It is evident that the images in the test dataset are quite diverse, including mathematical formulas of various lengths and complexities, from single letters to formula groups and even matrices. This test dataset includes 485 images.
Below are the Character Error Rates (CER, the lower, the better) of various models on this test dataset. For the true annotated results, as well as the output of each model, normalization was first performed to ensure that irrelevant factors such as spaces do not affect the test outcomes. For the recognition results of Texify, the leading and trailing symbols $ or $$ of the formula are removed first.
As can be seen from the figure above, the Pix2Text V1.0 MFR open-source free version model has significantly outperformed the previous versions of the paid model. Moreover, compared to the V1.0 MFR open-source free model, the precision of the Pix2Text V1.0 MFR paid model has been further improved.
Texify is more suited for recognizing images with standard formatting. It performs poorly in recognizing images containing single letters. This is the main reason why Texify's performance on this test dataset is inferior to that of Latex-OCR.
