---
title: PillPal
emoji: 💊
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.16.0
python_version: '3.13'
app_file: server.py
pinned: false
license: mit
short_description: Photo a pill bottle, get a daily schedule + refill alert
models:
  - openbmb/MiniCPM-V-4.6
tags:
  - build-small-hackathon
  - backyard-ai
  - minicpm
  - vision
  - gradio
  - healthcare
---

# 💊 PillPal

Photograph a pill bottle and get a **clear daily schedule** — plus a heads-up
before you run out. A small **vision model** reads the label; everything else is
plain, testable Python.

Built for the **Build Small Hackathon · Backyard AI track** — a real tool for a
real person (a parent, neighbor, or anyone juggling medications).

# Demo video



[![Demo video](https://img.youtube.com/vi/Qx64AinZyKM/maxresdefault.jpg)](https://youtu.be/Qx64AinZyKM)


## Model

- **[MiniCPM-V 4.6](https://huggingface.co/openbmb/MiniCPM-V-4.6)** (OpenBMB) via
  the hosted **MiniCPM-V API** (`https://api.modelbest.cn/v1`, OpenAI-compatible).
  The model has exactly one job: read the label and return JSON.

Targets the **OpenBMB special award** (built on MiniCPM models).

## What we're targeting

**Track:** 🏡 Backyard AI — solve a real problem for someone you actually know.

| Target | Status | How |
|--------|--------|-----|
| 🐝 **OpenBMB special award** | 🎯 | the label reader runs on **MiniCPM-V 4.6** (OpenBMB) |
| 🎨 **Off-Brand** | ✅ | fully custom dashboard UI (upload → confirm → schedule), not default Gradio |
| 📡 Sharing is Caring | optional | could share a sample-label dataset / trace on the Hub |
| 🔌 Off the Grid | ✖️ n/a | uses the hosted MiniCPM-V API (switch `PILLPAL_BACKEND=llama` for a fully-offline run) |
| 🦙 Llama Champion | ✖️ n/a | not a Llama model (available via the `llama` backend if desired) |
| 🎯 Well-Tuned | ✖️ n/a | no fine-tuned model — uses MiniCPM-V as-is |

## Why it fits the track

Medication mix-ups are a real, everyday problem. PillPal does one helpful thing
well: turn a photo into a schedule and a refill warning. The AI is used narrowly
(label → structured JSON); all the trust-critical logic — scheduling, run-out
math — is deterministic Python you can unit-test, so it never invents dosing.

## How it works

```
photo ─► MiniCPM-V reads the label ─► JSON ─► you confirm/correct ─► schedule + refill date
```

1. **Read** — MiniCPM-V returns strict JSON: `drug_name, dose, frequency_text,
   quantity, refill_or_expiry_date`.
2. **Confirm** — you check/correct the fields in the form.
3. **Schedule** — `logic.py` turns the instruction into concrete times of day.
4. **Refill** — `logic.py` estimates the run-out date and warns if it's near.

- **server.py** — `gradio.Server`: endpoints `/read_label`, `/save_med`,
  `/delete_med`, `/clear_all`, `/state`; serves the dashboard.
- **index.html** — a custom dashboard UI (upload + read, confirm/correct form,
  today's schedule, light/dark theme).
- **logic.py** — `safe_parse`, `build_schedule`, `check_refill` (the real work).
- **vision.py** — label → JSON via the MiniCPM-V API (or local backends).
- **storage.py** — local JSON persistence (`meds.json`).

## Run locally

```bash
pip install -r requirements.txt
export PILLPAL_API_KEY=sk-...      # your MiniCPM-V API key (or use a Space secret)
python server.py
```

Without a key, the label reader shows a friendly "enter details manually" message
and the rest of the app still works. Use `PILLPAL_MOCK=1` for canned label data.

## Test

```bash
pytest -q     # logic tests, fully offline
```

## Config (env vars)

| Var | Default | Purpose |
|-----|---------|---------|
| `PILLPAL_API_KEY` | _(none — set it)_ | MiniCPM-V API key (provide via env / Space **secret**) |
| `PILLPAL_API_MODEL` | `MiniCPM-V-4.6-Instruct` | or `MiniCPM-V-4.6-Thinking` |
| `PILLPAL_API_URL` | `https://api.modelbest.cn/v1/chat/completions` | MiniCPM-V endpoint |
| `PILLPAL_BACKEND` | `api` | `transformers` (local/ZeroGPU) or `llama` (offline llama.cpp) |
| `PILLPAL_MOCK` | `0` | `1` = canned label data, skip the model |
| `PILLPAL_STORE` | `meds.json` | where the medication list is saved |


X post https://x.com/i/status/2065869334702907504