| --- |
| title: Cook With A LLM |
| emoji: 🍲 |
| colorFrom: red |
| colorTo: yellow |
| sdk: gradio |
| sdk_version: 6.15.2 |
| python_version: '3.12' |
| app_file: app.py |
| pinned: false |
| license: apache-2.0 |
| --- |
| |
| # 🍲 Cook With Me — Multimodal Sous-Chef |
|
|
| > *Snap your fridge. Pick a dish. Cook step by step. Check your progress with a photo.* |
|
|
| A closed-loop multimodal cooking assistant built for the **Hugging Face Small Models / Big Adventures Hackathon (June 2026)**. |
|
|
| --- |
|
|
| ## How it works |
|
|
| ``` |
| 📸 Fridge photo ──▶ [Vision Agent] identify ingredients |
| │ |
| ▼ |
| [Recipe Planner] propose 3 dishes → full recipe JSON |
| │ |
| ▼ |
| [Nutrition Engine] per-serving macros (lookup, no hallucination) |
| │ |
| ▼ |
| 📸 Progress photo ──▶ [Progress Validator] go / wait / fix verdict |
| ``` |
|
|
| 1. **Snap** your fridge or pantry — the fine-tuned vision model identifies every ingredient. |
| 2. **Pick** one of three AI-suggested dishes tailored to what you have. |
| 3. **Cook** step by step with a generated recipe and per-serving nutrition info. |
| 4. **Check** your progress by uploading a photo of your pan — the model tells you *go*, *wait*, or *fix*. |
|
|
| --- |
|
|
| ## Models |
|
|
| | Role | Model | Params | Runtime | |
| |---|---|---|---| |
| | Vision + Planner + Validator | `openbmb/MiniCPM-V-4.6` (fine-tuned) | ~4.6B | `transformers` / ZeroGPU | |
|
|
| **Total: ~4.6B parameters** (≤ 32B cap ✓ — significant headroom) |
|
|
| The ingredient-identification model is **fine-tuned** on fridge/pantry photos for higher precision. |
|
|
| --- |
|
|
| ## Badges targeted |
|
|
| | Badge | Status | How | |
| |---|---|---| |
| | 🎯 Well-Tuned | ✓ | Fine-tuned MiniCPM-V-4.6 for ingredient detection, published to Hub | |
| | 🎨 Off-Brand | ✓ | Recipe-card UI with custom CSS — Lora serif, warm parchment palette | |
| | 📡 Sharing is Caring | ✓ | Agent traces shared on Hub | |
| | 📓 Field Notes | ✓ | Blog post: "Building a closed-loop visual cooking coach" | |
|
|
| --- |
|
|
| ## Architecture highlights |
|
|
| - **Single model, three roles:** MiniCPM-V-4.6 handles vision (ingredients + progress) *and* text planning (recipe JSON generation) — no redundant model downloads. |
| - **Closed-loop visual validation:** Flux generates step targets → user cooks → vision model compares — a real agent loop, not a wrapper. |
| - **Hallucination-free nutrition:** macros come from a lookup table, not LLM arithmetic. |
| - **Robust JSON extraction:** multi-strategy parser handles markdown fences, single quotes, and trailing commas so generation failures degrade gracefully. |
|
|
| --- |
|
|
| ## Track |
|
|
| **Chapter One — Backyard AI** · "Build something for someone you actually know." |
|
|
| Submission for the Hugging Face Hackathon · June 5–15, 2026. |
|
|
