README.md

---
title: SafeBite
emoji: 🥫
colorFrom: green
colorTo: yellow
sdk: gradio
sdk_version: 6.14.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: Scan an ingredient label, flag your allergens — on device.
tags:
  - track:backyard
  - sponsor:openbmb
  - achievement:offgrid
  - achievement:sharing
  - achievement:fieldnotes
---

# 🥫 SafeBite

**Scan a label. Know before you bite.** Snap a product's ingredient label, pick what you
avoid, and SafeBite reads it *on-device* on a ~1.3B vision model and flags what to skip —
including hidden aliases (whey = dairy, casein = dairy, semolina = gluten) and "may contain"
warnings. Works in the grocery aisle with **no signal**, and your health profile never leaves
the device.

▶️ **[Watch the demo](https://huggingface.co/spaces/build-small-hackathon/safebite/resolve/main/assets/demo/safebite-demo.mp4)** · **Model:** openbmb/MiniCPM-V-4.6 (1.3B) · runs on-device, no cloud APIs

> ⚠️ Not medical advice and not a substitute for reading the physical label. The model reads
> from a single photo and can miss or misread text. If you have a severe allergy, always verify
> on the package itself.

## 🎬 Demo

![SafeBite flagging peanuts and milk on a granola bar label](assets/demo/safebite-demo.gif)

Above: a granola-bar label scanned on-device in **~3 seconds** → 🔴 **AVOID**, with each flag
showing the matched word and where it was found, plus a downloadable agent trace. Try the
three built-in examples: a granola bar (peanuts + milk), a cookie **labelled "vegan" that
secretly contains whey**, and crackers with a *may-contain tree nuts* facility warning.

**🏷️ Tiny Titan:** the whole app runs on a single **~1.3B** model (openbmb/MiniCPM-V-4.6) — well under the 4B bar.

**🎥 Demo video:** [safebite-demo.mp4](https://huggingface.co/spaces/build-small-hackathon/safebite/resolve/main/assets/demo/safebite-demo.mp4) (screen capture of a live run)
**📣 Social post:** https://x.com/austin_seb/status/2066348577404703140
**📝 Field notes (blog):** https://huggingface.co/blog/SebAustin/build-small-hackathon
**🔁 Shared agent traces:** https://huggingface.co/datasets/build-small-hackathon/safebite-traces

## The problem

For someone with a food allergy, a label is a minefield: allergens hide behind names you don't
recognize (casein, albumin, semolina), "free-from" marketing contradicts the fine print, and
the print is tiny. Checking every label in a busy aisle — often with bad reception — is slow
and error-prone, and the stakes are high.

## The solution

Photograph the label, tick your allergens/diet, get one clear verdict:

- 🔴 **AVOID** — an allergen is in the ingredients or the "Contains" line
- 🟠 **CAUTION** — only a "may contain" / shared-equipment warning
- 🟢 **No flagged allergens found** — (still verify on the package)

Each flag shows **the exact matched word and where it was found**, so you can judge it yourself.

## Architecture — a visible agent loop

```
label photo + your allergen/diet profile
      │
      ▼
 ① extract (vision model call)  → product name, ingredients[], allergen_statements[] (verbatim)
      │
      ▼
 ② normalize (rules)            → split items; classify each statement as "Contains" vs
      │                            "May contain" by trigger phrase (NOT trusted to the model)
      ▼
 ③ match (alias dictionary)     → word-boundary match vs your profile, incl. hidden aliases
      │                            (whey/casein→dairy, semolina→gluten, albumin→egg, …)
      ▼
 ④ advise (rules)               → AVOID / CAUTION / clear tier
      │
      ▼
 ⑤ assemble                     → verdict card with matched word + source per flag
```

Each step logs its I/O to a `Trace` (exportable to a HF dataset for the Open Trace badge).
**Only the extract step calls the model** — a ~1.3B model reads labels well (OCR) but reasons
poorly, so allergen matching is a deterministic, auditable alias dictionary. That's both safer
(no hallucinated "safe to eat") and faster (well under the <10s target).

## Model

[**openbmb/MiniCPM-V-4.6**](https://huggingface.co/openbmb/MiniCPM-V-4.6) — ~1.3B params,
Apache-2.0, image-text-to-text. Loaded via `MiniCPMV4_6ForConditionalGeneration` +
`processor.apply_chat_template(...)` + `model.generate(...)`.

## Runs local / off the grid

All inference happens **in the Space** on the model above — **no cloud LLM APIs**, no network
round-trip per scan. The same `app.py` runs on any CUDA box (or a phone-class edge device with
a quantized build): point-and-scan in the aisle with zero connectivity, health data stays local.

## Setup (≤5 commands)

```bash
git clone https://huggingface.co/spaces/build-small-hackathon/safebite
cd safebite
pip install -r requirements.txt
python app.py            # needs a CUDA GPU (the Space uses ZeroGPU)
```

## Roadmap

- Export each run's `Trace` to a HF dataset (Open Trace badge)
- Barcode fallback + crowd-sourced allergen corrections
- Custom "avoid" terms (additives, FODMAPs, religious/halal/kosher checks)
- Multilingual labels; on-device quantized build for true phone-offline use
- Optional Modal endpoint for the inference core (Modal award)