README.md

metadata

title: Pinch
emoji: 🤏
colorFrom: yellow
colorTo: green
sdk: gradio
sdk_version: 6.18.0
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - build-small-hackathon
  - track:backyard
  - sponsor:openbmb
  - sponsor:modal
  - off-brand
  - best-agent
  - best-demo
  - tiny-titan

🤏 Pinch — photograph your ingredients, get a grounded plan

Build Small Hackathon 2026 · 🏡 Backyard AI track

Snap a photo of what's in your kitchen — Pinch works out what to cook, plans the seasoning grounded in real flavour science, and computes the amounts. A vision model reads the ingredients from your photo; an agent reasons over the Epicure flavour model (1,790 ingredients distilled from ~4M recipes) to decide what to add, what to substitute for what you lack, and in what order; and it runs Python in a sandbox for the quantitative side (servings, ratios, salt %, timing). Optionally, FLUX turns your ingredients photo into a picture of the finished dish.

Small models, each load-bearing, ≈24B total — under the 32B cap: MiniCPM-V (OpenBMB) sees the ingredients, Mellum 2 (12B, JetBrains) reasons about flavour, and FLUX.2 klein (4B) renders the dish.

Why this needs an agent

The hard parts aren't a lookup. Flavour — given the messy set of things in my kitchen and this dish, what's worth adding, and in what order? — is combinatorial and grounded in the Epicure pairing model, not the LLM's opinion. Arithmetic — scaling to servings, ratios, salt % — is something LLMs get wrong in their heads, so the agent offloads it to real code in a sandbox. The model reasons and sequences; the tools provide the facts.

🎥 Demo video: https://www.youtube.com/watch?v=gpXQS35QWW0

📣 Social post: https://x.com/ssaacar/status/2066645089942708604

Architecture

  You: photo (or typed ingredients) + optional constraints
            │  MiniCPM-V reads the photo → ingredient list (you edit it)
            ▼   suggest_dish()  +  build_plan()
   ┌────────────────────────────┐  pairing_score  ┌────────────────────────────┐
   │  JetBrains Mellum 2 (12B)  │ ──────────────► │  Epicure MCP (Kaikaku)     │
   │  strict-JSON planner loop  │ ◄────────────── │  pairing scores, neighbours │
   │                            │   run_python    ┌────────────────────────────┐
   │                            │ ──────────────► │  sandbox (kitchen math:     │
   │                            │ ◄────────────── │  amounts, ratios, timing)   │
   └────────────┬───────────────┘   stdout        └────────────────────────────┘
                │
                ▼  staged plan + amounts  ──►  (optional) FLUX.2 klein → dish photo

• Models: Mellum 2 reasons + writes the math code (served on Modal); MiniCPM-V reads the photo (OpenBMB's free hosted API); FLUX.2 klein renders the dish (HF Inference / fal-ai). All remote — the Space itself needs no GPU.
• Flavour grounding: Epicure MCP — public, anonymous flavour server (find_pairings, pairing_score, neighbors, …) via a hand-rolled MCP client (epicure_client.py) with 404/429 retry + a disk cache. Each step's evidence is re-derived from a real pairing score, never the model's guess.
• Sandbox: the quantitative math runs as real Python (sandbox.py), tailored to the dish type (a salad gets no simmer time; a dessert gets no savoury salt). SANDBOX_BACKEND=local|modal|mock.
• Robustness: the model emits one JSON object per turn — an Epicure tool call, a run_python, or the finished plan (agent.py). If it doesn't converge, a grounded scripted planner (real Epicure scores + sandbox math, no LLM) takes over.
• Honesty note: pairing scores are grounded facts; the substitution hint uses Epicure neighbors (co-occurrence, not a functional swap), so it's surfaced as "closest in flavour space," not "use X for the sourness of Y."

Run it

In the browser: add a photo (or type ingredients) → Detect ingredients → tidy the list to what you actually have → Decide a dish & plan → optionally ✨ See the dish.

Local dev:

pip install -r requirements.txt
cp .env.example .env     # fill in MODAL_REASON_URL, HF_TOKEN, etc.
python app.py
# fully offline (no models, scripted planner + sample detection):
MOCK_LLM=1 python app.py

Where the models run (inference backend)

Each model is configured independently via env vars (see .env.example).

Vision (MiniCPM-V) — VISION_BACKEND: has a free hosted API, so it runs real with no GPU.

VISION_BACKEND	runs on	cost
openbmb	OpenBMB's free hosted API (api.modelbest.cn, OpenAI-compatible)	free
zerogpu	in-Space ZeroGPU	free, quota
modal	your Modal endpoint	credits

Reasoning (Mellum 2) — INFERENCE_BACKEND: Mellum has no hosted API anywhere, so it runs on a GPU you control.

INFERENCE_BACKEND	runs on	cost
modal	your Modal endpoint (no quota)	credits
zerogpu	in-Space ZeroGPU (40 min/day)	free
mock (MOCK_LLM=1)	scripted planner, no model	free (dev)

Dish image — FLUX.2 klein: the "✨ See the dish" button renders the finished plate via HF Inference (provider="fal-ai"), billed to your account. Set HF_TOKEN; without it the button just shows a hint.

Deploy the Modal reasoning backend (one-time; pip install modal + modal token new):

modal deploy modal_app.py     # serves Mellum 2 on A100-40GB; weights cached in a Volume

It prints a URL. Set on the Space (Settings → Variables & secrets):

INFERENCE_BACKEND=modal
MODAL_REASON_URL=<the printed url>
VISION_BACKEND=openbmb

min_containers=0 → idle costs nothing; cold start ~30–60s (weights cached).