| | ---
|
| | language:
|
| | - de
|
| | pipeline_tag: image-to-text
|
| | ---
|
| | # Ingredient Scanner
|
| | ## Abstract
|
| |
|
| | With the recent advancements in computer vision and optical character recognition and using a convolutional neural network to cut out the product from a picture, it has now become possible to reliably extract ingredient lists from the back of a product using the Anthropic API. Open-weight or even only on-device optical character recognition lacks the quality to be used in a production environment, although the progress in development is promising. The Anthropic API is also currently not feasible due to the high cost of 1 Swiss Franc per 100 pictures.
|
| |
|
| | The training code and data is available on [GitHub](https://github.com/lenamerkli/ingredient-scanner/). This repository just contains an inference example and the [report](https://huggingface.co/lenamerkli/ingredient-scanner/blob/main/ingredient-scanner.pdf).
|
| |
|
| | This is an entry for the [2024 Swiss AI competition](https://www.ki-wettbewerb.ch/).
|
| |
|
| | ## Table of Contents
|
| |
|
| | 0. [Abstract](#abstract)
|
| | 1. [Report](#report)
|
| | 2. [Model Details](#model-details)
|
| | 3. [Usage](#usage)
|
| | 4. [Citation](#citation)
|
| |
|
| | ## Report
|
| | Read the full report [here](https://huggingface.co/lenamerkli/ingredient-scanner/blob/main/ingredient-scanner.pdf).
|
| |
|
| | ## Model Details
|
| | This repository consists of two models, one vision model and a large language model.
|
| |
|
| | ### Vision Model
|
| | Custom convolutional neural network based on [ResNet18](https://pytorch.org/hub/pytorch_vision_resnet/). It detects the four corner points and the upper and lower limits of a product.
|
| |
|
| | ### Language Model
|
| | Converts the text from the optical character recognition engine which lies in-between the two models to JSON. It is fine-tuned from [unsloth/Qwen2-0.5B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Qwen2-0.5B-Instruct-bnb-4bit).
|
| |
|
| | ## Usage
|
| | Clone the repository and install the dependencies on any debian-based system:
|
| | ```bash
|
| | git clone https://huggingface.co/lenamerkli/ingredient-scanner
|
| | cd ingredient-scanner
|
| | python3 -m venv .venv
|
| | source .venv/bin/activate
|
| | pip3 install -r requirements.txt
|
| | ```
|
| | Note: not all requirements are needed for inference, as both training and inference requirements are listed.
|
| |
|
| | Select the OCR engine in `main.py` by uncommenting one of the lines 20 to 22:
|
| | ```python
|
| | # ENGINE: list[str] = ['easyocr']
|
| | # ENGINE: list[str] = ['anthropic', 'claude-3-5-sonnet-20240620']
|
| | # ENGINE: list[str] = ['llama_cpp/v2/vision', 'qwen-vl-next_b2583']
|
| | ```
|
| | Note: Qwen-VL-Next is not an official qwen model. This is only to protect business secrets of a private model.
|
| |
|
| | Run the inference script:
|
| | ```bash
|
| | python3 main.py
|
| | ```
|
| |
|
| | You will be asked to enter the file path to a PNG image.
|
| |
|
| | ### Anthropic API
|
| |
|
| | If you want to use the Anthropic API, create a `.env` file with the following content:
|
| | ```
|
| | ANTHROPIC_API_KEY=YOUR_API_KEY
|
| | ```
|
| |
|
| | ## Citation
|
| | Here is how to cite this paper in the bibtex format:
|
| | ```bibtex
|
| | @misc{merkli2024ingriedient-scanner,
|
| | title={Ingredient Scanner: Automating Reading of Ingredient Labels with Computer Vision},
|
| | author={Lena Merkli and Sonja Merkli},
|
| | date={2024-07-16},
|
| | url={https://huggingface.co/lenamerkli/ingredient-scanner},
|
| | }
|
| | ```
|
| |
|
