| --- |
| title: BudgetBuddy |
| emoji: 🧾 |
| colorFrom: indigo |
| colorTo: green |
| sdk: gradio |
| sdk_version: 6.18.0 |
| app_file: app.py |
| pinned: false |
| short_description: Snap a bill, ask where your money went — small-model AI |
| tags: |
| - track:backyard |
| - sponsor:openbmb |
| - sponsor:modal |
| - achievement:offbrand |
| - achievement:sharing |
| - achievement:fieldnotes |
| - build-small-hackathon |
| - minicpm |
| - modal |
| - gradio |
| - agent |
| --- |
| |
| # 🧾 BudgetBuddy |
|
|
| **A spend tracker for real people — built entirely on small, open models.** |
| Snap a messy receipt or a UPI/card screenshot (or just type it), and BudgetBuddy |
| reads it, fixes the totals, categorises it, saves it privately, and lets you |
| **chat with a tool-using agent** about where your money went — all in a custom |
| dashboard UI. No third-party AI APIs. Two MiniCPM models do everything. |
|
|
| > Build Small hackathon · **Backyard AI** track. Built for the people around me |
| > (homemakers, parents, small-shop owners) who want to understand their spending |
| > without a spreadsheet — and without handing their receipts to a cloud AI API. |
|
|
| ## ▶️ Demo & links |
| - **Live app:** https://huggingface.co/spaces/build-small-hackathon/BudgetBuddy |
| - **Demo video:** https://youtu.be/QbgY6HDbrxE |
| - **Social post:** https://x.com/KrishnaIsCoding/status/2066565121464541191 |
| - **Build write-up (Field Notes):** https://huggingface.co/blog/KrishnaGarg/budget-buddy-field-notes ([repo copy](FIELD_NOTES.md)) |
| - **Open agent traces (Sharing is Caring):** [AGENT_TRACES.md](AGENT_TRACES.md) |
|
|
| ## Why it fits "Build Small" |
| - **Small, open models — no third-party AI API.** Vision/OCR is |
| [MiniCPM-V-4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6) (**1.3B**), running |
| on the Space's **ZeroGPU**. The reasoning + agent brain is |
| [MiniCPM4.1-8B](https://huggingface.co/openbmb/MiniCPM4.1-8B) (**8B**), running on |
| our own **[Modal](https://modal.com)** GPU. **~9.3B total — well under the 32B |
| cap.** We never call a hosted AI inference API (no OpenAI/Anthropic/Gemini) — only |
| open weights we run ourselves. |
| - **Real problem, honest fit.** Real bills are messy: missing totals, taxes, |
| service charges, round-offs, mixed items, weird date formats. BudgetBuddy |
| reasons about them and reconciles the math, so editing is the exception. |
|
|
| ## What it does |
| 1. **Capture, three ways** — a photographed **receipt**, a **payment screenshot** |
| (UPI / GPay / PhonePe / card), or a quick **manual** entry. |
| 2. **Read & reconcile** — the vision model extracts vendor, date (normalised to |
| `YYYY-MM-DD`), line items, taxes/service/tip/discount/round-off, and total; |
| computes a missing total; flags anything that doesn't add up. |
| 3. **Reason & categorise** — the 8B reviews the extraction, fixes obvious errors, |
| and assigns an overall + per-item category (fixed 23-category list). |
| 4. **Dashboard** — monthly spend, vs-last-month, top category, spend-by-category |
| donut, spend-over-time chart, a budget ring, a calendar heatmap, and a |
| filterable transaction list that shows every line item **and** every tax/charge. |
| 5. **Agent chat** — ask *"how much did I spend on Groceries last month?"* or |
| *"what's my biggest expense?"* and the agent answers with your real numbers, |
| showing which **tools** it used. |
|
|
| ## 🤖 The agent (Best Agent) |
| The assistant is a real tool-using agent over `core/analytics`, not a chatbot that |
| guesses. It exposes **11 tools** — `total_spend`, `category_spend`, `item_spend`, |
| `vendor_spend`, `top_categories`, `biggest_expense`, `average_spend`, |
| `count_transactions`, `budget_status`, `monthly_trend`, `recent` — each scoped by a |
| flexible **period** (`this_month`, `last_month`, `this_year`, a specific month like |
| `2026-07`, a year, or `all`). |
|
|
| The **8B plans every question**: it reads the question, decides which tool to call |
| (and with what period), reads the result, optionally **chains another tool**, then |
| answers — a real ReAct loop. The tools are deterministic Python, so the *numbers* |
| can never be hallucinated; an answer is only ever returned once it is **grounded by |
| an actual tool call** (ungrounded model output is rejected). A deterministic router |
| over the same tools acts as a reliability fallback if the model can't produce a |
| valid plan. Every reply shows the **trace of tools used**, so the reasoning is |
| auditable. |
|
|
| ## Custom UI (Off-Brand) |
| The frontend is a hand-built dark single-page app ([frontend/](frontend/)) served |
| by **`gradio.Server`** (Gradio 6): Python API endpoints on the Gradio backend |
| (queue + ZeroGPU), our own HTML/CSS/JS + Chart.js on top. The default Gradio shell |
| is gone entirely. |
|
|
| ## ⚡ Modal (Best Use of Modal) |
| MiniCPM4.1-8B's `trust_remote_code` targets transformers ~4.56 and breaks on the |
| 5.7 that MiniCPM-V-4.6 needs. Modal resolves the conflict cleanly: the 8B runs in |
| its own container/env on an A10G, loaded once into a **memory snapshot** for fast |
| cold starts and kept warm (`scaledown_window`), and the Space calls it through the |
| Modal SDK ([core/modal_backend.py](core/modal_backend.py), |
| [modal_app.py](modal_app.py)). That's what makes the agent quick. |
|
|
| ## Privacy |
| Sign in with a **username + PIN** (PIN stored salted+hashed, never in plaintext). |
| Each user's transactions live in their own file in a **private** HF Dataset; the |
| frontend holds a signed session token, so no one can read another user's data. |
|
|
| ## Badges / prizes targeted |
| - **Backyard AI** track — a practical, everyday-life spending app. |
| - **Best MiniCPM Build** — the whole app is two MiniCPM models (vision + 8B). |
| - **Best Use of Modal** — the 8B reasoning/agent model runs on Modal. |
| - **Off-Brand** (achievement) — a fully custom `gradio.Server` frontend. |
| - **Sharing is Caring** (achievement) — open agent traces on the Hub ([AGENT_TRACES.md](AGENT_TRACES.md)). |
| - **Field Notes** (achievement) — a build write-up / report ([FIELD_NOTES.md](FIELD_NOTES.md)). |
| - **Best Agent** · **Best Demo** · **Bonus Quest Champion** — judged across entries (multi-step agent, full demo package, most bonus criteria met). |
|
|
| ## Structure |
| ``` |
| core/extract.py # MiniCPM-V-4.6: receipt + payment extraction, reconcile, date-normalise |
| core/categorize.py # 8B: refine/repair + categorise (overall + per-item) |
| core/agent.py # tool-using spending agent (router + ReAct loop, 11 tools) |
| core/chat.py # grounded one-shot answer (agent fallback) |
| core/analytics.py # pure-Python aggregations (summary, by-category, over-time, calendar) |
| core/inference.py # one place that owns the models / routes vision + text generation |
| core/modal_backend.py # client for the Modal 8B service |
| core/storage.py # per-user transactions + budget in a HF Dataset |
| core/auth.py # username + PIN accounts, signed session tokens |
| core/hubio.py # low-level dataset JSONL IO |
| app.py # gradio.Server: API endpoints + serves the custom frontend |
| modal_app.py # Modal service hosting MiniCPM4.1-8B (deploy: modal deploy modal_app.py) |
| frontend/ # custom dark SPA (index.html + assets/app.js, Chart.js) |
| ``` |
|
|
| ## Run locally |
| ```bash |
| python -m venv .venv && . .venv/bin/activate # Windows: .venv\Scripts\activate |
| pip install -r requirements.txt |
| python app.py # open http://127.0.0.1:7860 |
| ``` |
| A GPU isn't required to try the UI (first run downloads the vision weights). On the |
| Space, set an `HF_TOKEN` secret (dataset persistence) and `MODAL_TOKEN_ID` / |
| `MODAL_TOKEN_SECRET` (the 8B backend), and `BB_INFERENCE=modal`. **Tip:** log in to |
| Hugging Face in your browser to use your own ZeroGPU quota for the vision model. |
|
|
