README.md

metadata

title: Cook With A LLM
emoji: 🍲
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.15.2
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0

🍲 Cook With Me — Multimodal Sous-Chef

Snap your fridge. Pick a dish. Cook step by step. Check your progress with a photo.

A closed-loop multimodal cooking assistant built for the Hugging Face Small Models / Big Adventures Hackathon (June 2026).

---

How it works

📸 Fridge photo  ──▶  [Vision Agent]          identify ingredients
                            │
                            ▼
                      [Recipe Planner]         propose 3 dishes → full recipe JSON
                            │
                            ▼
                      [Nutrition Engine]       per-serving macros (lookup, no hallucination)
                            │
                            ▼
📸 Progress photo ──▶  [Progress Validator]    go / wait / fix verdict

1. Snap your fridge or pantry — the fine-tuned vision model identifies every ingredient.
2. Pick one of three AI-suggested dishes tailored to what you have.
3. Cook step by step with a generated recipe and per-serving nutrition info.
4. Check your progress by uploading a photo of your pan — the model tells you go, wait, or fix.

---

Models

Role	Model	Params	Runtime
Vision + Planner + Validator	openbmb/MiniCPM-V-4.6 (fine-tuned)	~4.6B	transformers / ZeroGPU

Total: ~4.6B parameters (≤ 32B cap ✓ — significant headroom)

The ingredient-identification model is fine-tuned on fridge/pantry photos for higher precision.

---

Badges targeted

Badge	Status	How
🎯 Well-Tuned	✓	Fine-tuned MiniCPM-V-4.6 for ingredient detection, published to Hub
🎨 Off-Brand	✓	Recipe-card UI with custom CSS — Lora serif, warm parchment palette
📡 Sharing is Caring	✓	Agent traces shared on Hub
📓 Field Notes	✓	Blog post: "Building a closed-loop visual cooking coach"

---

Architecture highlights

• Single model, three roles: MiniCPM-V-4.6 handles vision (ingredients + progress) and text planning (recipe JSON generation) — no redundant model downloads.
• Closed-loop visual validation: Flux generates step targets → user cooks → vision model compares — a real agent loop, not a wrapper.
• Hallucination-free nutrition: macros come from a lookup table, not LLM arithmetic.
• Robust JSON extraction: multi-strategy parser handles markdown fences, single quotes, and trailing commas so generation failures degrade gracefully.

---

Track

Chapter One — Backyard AI · "Build something for someone you actually know."

Submission for the Hugging Face Hackathon · June 5–15, 2026.