With the recent advancements in computer vision and optical character recognition and using a convolutional neural network to cut out the product from a picture, it has now become possible to reliably extract ingredient lists from the back of a product using the Anthropic API. Open-weight or even only on-device optical character recognition lacks the quality to be used in a production environment, although the progress in development is promising. The Anthropic API is also currently not feasible due to the high cost of 1 Swiss Franc per 100 pictures.

The training code and data is available on GitHub. This repository just contains an inference example and the report.

This is an entry for the 2024 Swiss AI competition.

Read the full report here.

This repository consists of two models, one vision model and a large language model.

Custom convolutional neural network based on ResNet18. It detects the four corner points and the upper and lower limits of a product.

Converts the text from the optical character recognition engine which lies in-between the two models to JSON. It is fine-tuned from unsloth/Qwen2-0.5B-Instruct-bnb-4bit.

Clone the repository and install the dependencies on any debian-based system:

Note: not all requirements are needed for inference, as both training and inference requirements are listed.

Select the OCR engine in main.py by uncommenting one of the lines 20 to 22:

Note: Qwen-VL-Next is not an official qwen model. This is only to protect business secrets of a private model.

Run the inference script:

You will be asked to enter the file path to a PNG image.

If you want to use the Anthropic API, create a .env file with the following content:

Here is how to cite this paper in the bibtex format: