---
title: Cook With A LLM
emoji: 🍲
colorFrom: red
colorTo: yellow
sdk: gradio
sdk_version: 6.15.2
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
---

# 🍲 Cook With Me — Multimodal Sous-Chef

> *Snap your fridge. Pick a dish. Cook step by step. Check your progress with a photo.*

A closed-loop multimodal cooking assistant built for the **Hugging Face Small Models / Big Adventures Hackathon (June 2026)**.

---

## How it works

```
📸 Fridge photo  ──▶  [Vision Agent]          identify ingredients
                            │
                            ▼
                      [Recipe Planner]         propose 3 dishes → full recipe JSON
                            │
                            ▼
                      [Nutrition Engine]       per-serving macros (lookup, no hallucination)
                            │
                            ▼
📸 Progress photo ──▶  [Progress Validator]    go / wait / fix verdict
```

1. **Snap** your fridge or pantry — the fine-tuned vision model identifies every ingredient.
2. **Pick** one of three AI-suggested dishes tailored to what you have.
3. **Cook** step by step with a generated recipe and per-serving nutrition info.
4. **Check** your progress by uploading a photo of your pan — the model tells you *go*, *wait*, or *fix*.

---

## Models

| Role | Model | Params | Runtime |
|---|---|---|---|
| Vision + Planner + Validator | `openbmb/MiniCPM-V-4.6` (fine-tuned) | ~4.6B | `transformers` / ZeroGPU |

**Total: ~4.6B parameters** (≤ 32B cap ✓ — significant headroom)

The ingredient-identification model is **fine-tuned** on fridge/pantry photos for higher precision.

---

## Badges targeted

| Badge | Status | How |
|---|---|---|
| 🎯 Well-Tuned | ✓ | Fine-tuned MiniCPM-V-4.6 for ingredient detection, published to Hub |
| 🎨 Off-Brand | ✓ | Recipe-card UI with custom CSS — Lora serif, warm parchment palette |
| 📡 Sharing is Caring | ✓ | Agent traces shared on Hub |
| 📓 Field Notes | ✓ | Blog post: "Building a closed-loop visual cooking coach" |

---

## Architecture highlights

- **Single model, three roles:** MiniCPM-V-4.6 handles vision (ingredients + progress) *and* text planning (recipe JSON generation) — no redundant model downloads.
- **Closed-loop visual validation:** Flux generates step targets → user cooks → vision model compares — a real agent loop, not a wrapper.
- **Hallucination-free nutrition:** macros come from a lookup table, not LLM arithmetic.
- **Robust JSON extraction:** multi-strategy parser handles markdown fences, single quotes, and trailing commas so generation failures degrade gracefully.

---

## Track

**Chapter One — Backyard AI** · "Build something for someone you actually know."

Submission for the Hugging Face Hackathon · June 5–15, 2026.