README.md

metadata

title: SafeBite
emoji: 🥫
colorFrom: green
colorTo: yellow
sdk: gradio
sdk_version: 6.14.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: Scan an ingredient label, flag your allergens — on device.
tags:
  - track:backyard
  - sponsor:openbmb
  - achievement:offgrid
  - achievement:sharing
  - achievement:fieldnotes

🥫 SafeBite

Scan a label. Know before you bite. Snap a product's ingredient label, pick what you avoid, and SafeBite reads it on-device on a ~1.3B vision model and flags what to skip — including hidden aliases (whey = dairy, casein = dairy, semolina = gluten) and "may contain" warnings. Works in the grocery aisle with no signal, and your health profile never leaves the device.

▶️ Watch the demo · Model: openbmb/MiniCPM-V-4.6 (1.3B) · runs on-device, no cloud APIs

⚠️ Not medical advice and not a substitute for reading the physical label. The model reads from a single photo and can miss or misread text. If you have a severe allergy, always verify on the package itself.

🎬 Demo

Above: a granola-bar label scanned on-device in ~3 seconds → 🔴 AVOID, with each flag showing the matched word and where it was found, plus a downloadable agent trace. Try the three built-in examples: a granola bar (peanuts + milk), a cookie labelled "vegan" that secretly contains whey, and crackers with a may-contain tree nuts facility warning.

🏷️ Tiny Titan: the whole app runs on a single ~1.3B model (openbmb/MiniCPM-V-4.6) — well under the 4B bar.

🎥 Demo video: safebite-demo.mp4 (screen capture of a live run) 📣 Social post: https://x.com/austin_seb/status/2066348577404703140 📝 Field notes (blog): https://huggingface.co/blog/SebAustin/build-small-hackathon 🔁 Shared agent traces: https://huggingface.co/datasets/build-small-hackathon/safebite-traces

The problem

For someone with a food allergy, a label is a minefield: allergens hide behind names you don't recognize (casein, albumin, semolina), "free-from" marketing contradicts the fine print, and the print is tiny. Checking every label in a busy aisle — often with bad reception — is slow and error-prone, and the stakes are high.

The solution

Photograph the label, tick your allergens/diet, get one clear verdict:

• 🔴 AVOID — an allergen is in the ingredients or the "Contains" line
• 🟠 CAUTION — only a "may contain" / shared-equipment warning
• 🟢 No flagged allergens found — (still verify on the package)

Each flag shows the exact matched word and where it was found, so you can judge it yourself.

Architecture — a visible agent loop

label photo + your allergen/diet profile
      │
      ▼
 ① extract (vision model call)  → product name, ingredients[], allergen_statements[] (verbatim)
      │
      ▼
 ② normalize (rules)            → split items; classify each statement as "Contains" vs
      │                            "May contain" by trigger phrase (NOT trusted to the model)
      ▼
 ③ match (alias dictionary)     → word-boundary match vs your profile, incl. hidden aliases
      │                            (whey/casein→dairy, semolina→gluten, albumin→egg, …)
      ▼
 ④ advise (rules)               → AVOID / CAUTION / clear tier
      │
      ▼
 ⑤ assemble                     → verdict card with matched word + source per flag

Each step logs its I/O to a Trace (exportable to a HF dataset for the Open Trace badge). Only the extract step calls the model — a ~1.3B model reads labels well (OCR) but reasons poorly, so allergen matching is a deterministic, auditable alias dictionary. That's both safer (no hallucinated "safe to eat") and faster (well under the <10s target).

Model

openbmb/MiniCPM-V-4.6 — ~1.3B params, Apache-2.0, image-text-to-text. Loaded via MiniCPMV4_6ForConditionalGeneration + processor.apply_chat_template(...) + model.generate(...).

Runs local / off the grid

All inference happens in the Space on the model above — no cloud LLM APIs, no network round-trip per scan. The same app.py runs on any CUDA box (or a phone-class edge device with a quantized build): point-and-scan in the aisle with zero connectivity, health data stays local.

Setup (≤5 commands)

git clone https://huggingface.co/spaces/build-small-hackathon/safebite
cd safebite
pip install -r requirements.txt
python app.py            # needs a CUDA GPU (the Space uses ZeroGPU)

Roadmap

• Export each run's Trace to a HF dataset (Open Trace badge)
• Barcode fallback + crowd-sourced allergen corrections
• Custom "avoid" terms (additives, FODMAPs, religious/halal/kosher checks)
• Multilingual labels; on-device quantized build for true phone-offline use
• Optional Modal endpoint for the inference core (Modal award)