---
title: BudgetBuddy
emoji: 🧾
colorFrom: indigo
colorTo: green
sdk: gradio
sdk_version: 6.18.0
app_file: app.py
pinned: false
short_description: Snap a bill, ask where your money went — small-model AI
tags:
  - track:backyard
  - sponsor:openbmb
  - sponsor:modal
  - achievement:offbrand
  - achievement:sharing
  - achievement:fieldnotes
  - build-small-hackathon
  - minicpm
  - modal
  - gradio
  - agent
---

# 🧾 BudgetBuddy

**A spend tracker for real people — built entirely on small, open models.**
Snap a messy receipt or a UPI/card screenshot (or just type it), and BudgetBuddy
reads it, fixes the totals, categorises it, saves it privately, and lets you
**chat with a tool-using agent** about where your money went — all in a custom
dashboard UI. No third-party AI APIs. Two MiniCPM models do everything.

> Build Small hackathon · **Backyard AI** track. Built for the people around me
> (homemakers, parents, small-shop owners) who want to understand their spending
> without a spreadsheet — and without handing their receipts to a cloud AI API.

## ▶️ Demo & links
- **Live app:** https://huggingface.co/spaces/build-small-hackathon/BudgetBuddy
- **Demo video:** https://youtu.be/QbgY6HDbrxE
- **Social post:** https://x.com/KrishnaIsCoding/status/2066565121464541191
- **Build write-up (Field Notes):** https://huggingface.co/blog/KrishnaGarg/budget-buddy-field-notes ([repo copy](FIELD_NOTES.md))
- **Open agent traces (Sharing is Caring):** [AGENT_TRACES.md](AGENT_TRACES.md)

## Why it fits "Build Small"
- **Small, open models — no third-party AI API.** Vision/OCR is
  [MiniCPM-V-4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6) (**1.3B**), running
  on the Space's **ZeroGPU**. The reasoning + agent brain is
  [MiniCPM4.1-8B](https://huggingface.co/openbmb/MiniCPM4.1-8B) (**8B**), running on
  our own **[Modal](https://modal.com)** GPU. **~9.3B total — well under the 32B
  cap.** We never call a hosted AI inference API (no OpenAI/Anthropic/Gemini) — only
  open weights we run ourselves.
- **Real problem, honest fit.** Real bills are messy: missing totals, taxes,
  service charges, round-offs, mixed items, weird date formats. BudgetBuddy
  reasons about them and reconciles the math, so editing is the exception.

## What it does
1. **Capture, three ways** — a photographed **receipt**, a **payment screenshot**
   (UPI / GPay / PhonePe / card), or a quick **manual** entry.
2. **Read & reconcile** — the vision model extracts vendor, date (normalised to
   `YYYY-MM-DD`), line items, taxes/service/tip/discount/round-off, and total;
   computes a missing total; flags anything that doesn't add up.
3. **Reason & categorise** — the 8B reviews the extraction, fixes obvious errors,
   and assigns an overall + per-item category (fixed 23-category list).
4. **Dashboard** — monthly spend, vs-last-month, top category, spend-by-category
   donut, spend-over-time chart, a budget ring, a calendar heatmap, and a
   filterable transaction list that shows every line item **and** every tax/charge.
5. **Agent chat** — ask *"how much did I spend on Groceries last month?"* or
   *"what's my biggest expense?"* and the agent answers with your real numbers,
   showing which **tools** it used.

## 🤖 The agent (Best Agent)
The assistant is a real tool-using agent over `core/analytics`, not a chatbot that
guesses. It exposes **11 tools** — `total_spend`, `category_spend`, `item_spend`,
`vendor_spend`, `top_categories`, `biggest_expense`, `average_spend`,
`count_transactions`, `budget_status`, `monthly_trend`, `recent` — each scoped by a
flexible **period** (`this_month`, `last_month`, `this_year`, a specific month like
`2026-07`, a year, or `all`).

The **8B plans every question**: it reads the question, decides which tool to call
(and with what period), reads the result, optionally **chains another tool**, then
answers — a real ReAct loop. The tools are deterministic Python, so the *numbers*
can never be hallucinated; an answer is only ever returned once it is **grounded by
an actual tool call** (ungrounded model output is rejected). A deterministic router
over the same tools acts as a reliability fallback if the model can't produce a
valid plan. Every reply shows the **trace of tools used**, so the reasoning is
auditable.

## Custom UI (Off-Brand)
The frontend is a hand-built dark single-page app ([frontend/](frontend/)) served
by **`gradio.Server`** (Gradio 6): Python API endpoints on the Gradio backend
(queue + ZeroGPU), our own HTML/CSS/JS + Chart.js on top. The default Gradio shell
is gone entirely.

## ⚡ Modal (Best Use of Modal)
MiniCPM4.1-8B's `trust_remote_code` targets transformers ~4.56 and breaks on the
5.7 that MiniCPM-V-4.6 needs. Modal resolves the conflict cleanly: the 8B runs in
its own container/env on an A10G, loaded once into a **memory snapshot** for fast
cold starts and kept warm (`scaledown_window`), and the Space calls it through the
Modal SDK ([core/modal_backend.py](core/modal_backend.py),
[modal_app.py](modal_app.py)). That's what makes the agent quick.

## Privacy
Sign in with a **username + PIN** (PIN stored salted+hashed, never in plaintext).
Each user's transactions live in their own file in a **private** HF Dataset; the
frontend holds a signed session token, so no one can read another user's data.

## Badges / prizes targeted
- **Backyard AI** track — a practical, everyday-life spending app.
- **Best MiniCPM Build** — the whole app is two MiniCPM models (vision + 8B).
- **Best Use of Modal** — the 8B reasoning/agent model runs on Modal.
- **Off-Brand** (achievement) — a fully custom `gradio.Server` frontend.
- **Sharing is Caring** (achievement) — open agent traces on the Hub ([AGENT_TRACES.md](AGENT_TRACES.md)).
- **Field Notes** (achievement) — a build write-up / report ([FIELD_NOTES.md](FIELD_NOTES.md)).
- **Best Agent** · **Best Demo** · **Bonus Quest Champion** — judged across entries (multi-step agent, full demo package, most bonus criteria met).

## Structure
```
core/extract.py       # MiniCPM-V-4.6: receipt + payment extraction, reconcile, date-normalise
core/categorize.py    # 8B: refine/repair + categorise (overall + per-item)
core/agent.py         # tool-using spending agent (router + ReAct loop, 11 tools)
core/chat.py          # grounded one-shot answer (agent fallback)
core/analytics.py     # pure-Python aggregations (summary, by-category, over-time, calendar)
core/inference.py     # one place that owns the models / routes vision + text generation
core/modal_backend.py # client for the Modal 8B service
core/storage.py       # per-user transactions + budget in a HF Dataset
core/auth.py          # username + PIN accounts, signed session tokens
core/hubio.py         # low-level dataset JSONL IO
app.py                # gradio.Server: API endpoints + serves the custom frontend
modal_app.py          # Modal service hosting MiniCPM4.1-8B (deploy: modal deploy modal_app.py)
frontend/             # custom dark SPA (index.html + assets/app.js, Chart.js)
```

## Run locally
```bash
python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python app.py        # open http://127.0.0.1:7860
```
A GPU isn't required to try the UI (first run downloads the vision weights). On the
Space, set an `HF_TOKEN` secret (dataset persistence) and `MODAL_TOKEN_ID` /
`MODAL_TOKEN_SECRET` (the 8B backend), and `BB_INFERENCE=modal`. **Tip:** log in to
Hugging Face in your browser to use your own ZeroGPU quota for the vision model.