# Plan: Local-First iMessage → Calendar Agent (Gradio + llama.cpp + fine-tuned Gemma + an OpenBMB MiniCPM-planned agent)

## Who this is for

One named person: **a busy parent** whose kid's school/activity events are buried in a noisy class
group chat (picture day, the practice that moved, the RSVP). They read it once, mean to add it later,
and miss it. Success = *their* day measurably improves — events captured from the chat, conflicts
caught against their calendar, minutes saved — with **zero setup**: paste the thread or a screenshot
from a phone browser. The local-LLM / fine-tune work below is a **means** to better extraction, not
the point; the app must deliver value with no GPU (stub agent) first.

## Context

You want an agent that reads iMessage-style threads, understands the conversation, and turns them
into calendar events/reminders — exposed through a custom Gradio UI deployed as a Hugging Face Space.
Two local models share the work: our fine-tuned Gemma does the *reading* (thread → validated
ActionPlan), and an **OpenBMB MiniCPM** planner does the *orchestrating* — the brain behind **Run the agents**,
driving the Space's own MCP tools (`extract_events → check_conflicts → make_ics`) as a visible
multi-step agent. The build competes in the **Backyard AI** track (general and OpenBMB prizes are
awarded per track) and satisfies the quests secondary to the user story above: **Off the Grid** (no
cloud AI APIs, local-first), **Well-Tuned** (a fine-tuned model on HF), **Off-Brand** (custom UI),
and **Llama Champion** (both Gemma and MiniCPM are served through llama.cpp).

### Feasibility verdict: YES, with one re-architecture

The request as *literally* worded has two impossibilities, both solvable:

1. **No app or cloud can read iMessage on iOS.** Apple exposes no API for iMessage/SMS content.
   → **Solved:** you have a Mac. iMessages sync to `~/Library/Messages/chat.db`; a small local
   collector reads it. This is the *only* supported path and it keeps data local ("off the grid").
2. **A model cannot "run on your phone," and a HF Space runs in the cloud, not on-device.**
   → **Solved:** "on my phone" = *used from* your phone's browser. The Space does its own llama.cpp
   inference and calls no external AI service, so "hosted Space" and "off the grid" reconcile.

Confirmed decisions:
- **Ingestion:** Mac collector reading `chat.db`.
- **Calendar output:** local `.ics` files first (strictly off-grid), with an *optional* Google
  Calendar push toggle as a bonus.
- **Extraction model:** fine-tune Gemma, serve as GGUF via llama.cpp (production serves the
  **E4B** edge fine-tune, `build-small-hackathon/gemma-4-cal-gguf`).
- **Agent planner:** **OpenBMB MiniCPM** (`openbmb/MiniCPM4.1-8B-GGUF`, Q4; the 1B variant is a
  config switch) on a second llama-server — it plans, the MCP tools execute, every step visible.

---

## Architecture

```
┌────────── Your Mac (local) ──────────┐         ┌──────── Hugging Face Space (Docker) ────────┐
│  collector.py (Full Disk Access)      │  HTTPS  │  Gradio (custom theme/CSS)  ── Off-Brand     │
│  • polls chat.db for new messages     │ +token  │        │                                     │
│  • parses text / attributedBody       ├────────▶│  FastAPI /ingest  ──▶  extraction pipeline   │
│  • POSTs new msgs to Space /ingest    │         │        │                                     │
└───────────────────────────────────────┘         │  llama.cpp (llama-cpp-python) ── Llama Champ │
                                                   │   running YOUR fine-tuned gemma-4-31B GGUF   │
   View/approve from phone browser ───────────────▶│        │            ── Off the Grid (local) │
                                                   │  JSON events → pydantic validate            │
                                                   │        ├──▶ .ics file (download)            │
                                                   │        └──▶ optional Google Calendar push   │
                                                   └──────────────────────────────────────────────┘
```

Flow: messages → extraction prompt → model emits structured JSON of candidate events →
validated → shown in UI for review → user approves → `.ics` generated (and/or pushed to GCal).

**Run the agents** runs the same flow agentically: an **OpenBMB MiniCPM** planner (second local
llama-server, OpenAI-compatible) consumes the Space's own **MCP tool surface** —
`extract_events → check_conflicts → make_ics` — through smolagents, so the pipeline above is
demonstrated as multi-step tool use over the public tool contract, with the planner's trace on
screen (`server/orchestrator.py`). Stub/CI falls back to a scripted planner so the tab always works.

---

## Components

### 1. Mac-side iMessage collector  (`collector/collector.py`)
- **Reuse, don't reinvent the DB parsing.** Modern macOS stores message text in the
  `attributedBody` (NSAttributedString) blob, not always the `text` column. Use the battle-tested
  **`imessage-exporter`** (ReagentX, Rust) or the Python **`imessage_reader`** lib rather than
  hand-rolling SQL. If hand-querying: join `message` ⨝ `handle` ⨝ `chat_message_join` ⨝ `chat`,
  track last seen `ROWID`, poll on an interval.
- Requires **Full Disk Access** for the running process (System Settings → Privacy & Security).
- Sends only new messages to the Space `/ingest` endpoint over HTTPS with a shared bearer token.
- Config: which chats to watch, poll interval, Space URL, token (`.env`, never committed).

### 2. HF Space backend  (`app.py`, `server/`)
- **Docker SDK Space** (`README.md` frontmatter: `sdk: docker`, `app_port: 7860`).
- **llama.cpp** loads the fine-tuned GGUF and serves chat completions — satisfies
  *Llama Champion*; no external AI call satisfies *Off the Grid*.
- **Agent orchestrator** (`server/orchestrator.py`): the **OpenBMB MiniCPM** planner behind
  **Run the agents** (its own llama-server) drives the Space's MCP tools as a multi-step agent — the OpenBMB
  per-track prize case, and the same extraction pipeline exercised through the public tool
  contract rather than private imports.
- `/ingest` (FastAPI, mounted alongside Gradio) receives messages, runs the extraction prompt,
  returns candidate events; results surface in the Gradio UI for review.
- **Compute:** Q4_K_M GGUF of a 31B ≈ 18–20 GB → does **not** fit the free CPU tier (16 GB / 2 cores).
  Serve on a GPU: **ZeroGPU** (free, H200/70 GB — but cold GGUF load per acquisition; document the
  caveat) or a **paid GPU Space** (e.g. L4/L40S) for a smooth always-warm demo. See Fallback.

### 3. Fine-tuning pipeline  (`training/`)
- **Task:** conversation snippet → strict JSON list of events
  `{title, start, end, location, attendees, reminder_minutes, notes}`.
- **Data:** build a synthetic instruction dataset (~500–2000 examples) of realistic chat threads
  paired with the target JSON. Generation/augmentation for *training data* can use any tooling
  offline — the "no cloud API" rule applies to the *running app's inference*, not dataset prep.
  Include hard cases: relative dates ("next Thurs"), ranges, no-event chitchat (empty list),
  timezones, multiple events per thread.
- **Method:** QLoRA via **Unsloth** (Qwen3-0.6B GRPO experience applies), 4-bit, r=16,
  1–3 epochs. 31B QLoRA needs an A100/H100 80 GB (Colab Pro+/RunPod/Lambda, ~hours).
- **Export:** merge LoRA → `convert_hf_to_gguf.py` (llama.cpp) → `llama-quantize` to Q4_K_M →
  **publish GGUF to your HF repo** (satisfies *Well-Tuned*). Space downloads it at startup via
  `huggingface_hub`.

### 4. Custom Gradio UI  (`ui/`, `static/`)  — *Off-Brand*
- `gr.Blocks` with a custom `gr.themes.Base(...)` palette + injected `css=` (custom fonts, layout,
  cards) to push well past the default look.
- Screens: connection/status, incoming-message feed, **review queue** (edit candidate events
  inline, approve/reject), download `.ics`, optional "Push to Google Calendar" toggle, settings.

### 5. Calendar output  (`calendar_out/`)
- **`.ics` (default, off-grid):** generate with the `icalendar` lib; offer as a download in the UI.
- **Google Calendar (optional bonus):** `google-api-python-client` OAuth; behind a toggle so the
  off-grid demo path stays pure. Clearly labeled as the one optional cloud touchpoint.

---

## Hackathon requirement mapping

| Track | How it's satisfied |
|---|---|
| Off the Grid (local-first, no cloud AI APIs) | All inference is local llama.cpp in the Space; data originates on your Mac; `.ics` is the default output. |
| Well-Tuned (fine-tuned model on HF) | QLoRA fine-tune of `gemma-4-31B-it`, GGUF published to your HF repo. |
| Off-Brand (custom UI) | Custom Gradio theme + CSS, not the stock look. |
| Llama Champion (llama.cpp) | Inference via `llama-cpp-python`. |
| Gradio app on HF Space | Docker Space serving Gradio + FastAPI `/ingest`. |

---

## Build phases

1. **Hero path (no GPU):** Docker Space with custom-themed Gradio + the *stub* extractor → paste /
   "Try a sample" / screenshot → review → `.ics` download, working end-to-end on a phone browser.
   This is the parent's whole experience and must stand alone with no model.
2. **Measure impact:** persisted **This week** panel (events captured, conflicts caught, minutes
   saved) via `server/impact.py`, recorded when the parent exports. Proves *their* day got better.
3. **Accuracy upgrade (optional):** wire `llama-cpp-python` with a community `gemma-4-31B-it` GGUF on
   a GPU Space; swap the stub for the model + JSON-schema prompt + pydantic validation.
4. **Fine-tune (optional):** dataset → Unsloth QLoRA → GGUF → publish to HF → point the Space at it.
5. **Optional auto-feed:** Mac `collector.py` reading `chat.db` → POST `/ingest` (power users only).

---

## Verification

- **End-to-end (stub, phase 1):** open Space in phone browser → tap **Try a sample** (or paste a
  chat) → event appears in review queue → download `.ics` → import to a calendar, confirm date/time.
- **Impact (phase 2):** after exporting, **Activity → This week** shows events captured and time
  saved > 0; restart the app (same `IMPACT_PATH`) and confirm the weekly numbers persist while the
  live tiles reset. `minutes_saved` is a stated estimate (`IMPACT_MIN_PER_EVENT`=8,
  `IMPACT_MIN_PER_CONFLICT`=15, env-overridable), not a measurement.
- **Collector (phase 2):** send yourself a test iMessage ("lunch Tuesday 1pm") → confirm it reaches
  `/ingest` and surfaces in the feed.
- **Model (phase 3+):** curated eval set of chats with known expected events; measure JSON validity
  rate + field accuracy (esp. relative-date resolution); confirm empty-list on non-event chats.
- **llama.cpp:** confirm the Space logs show llama.cpp loading *your* GGUF, no external AI calls.

---

## Risks & fallbacks

- **31B serving cost/latency.** Q4 31B needs a GPU; ZeroGPU has cold-load + quota friction, paid GPU
  has cost. **Fallback:** fine-tune **Gemma 4 E4B** (edge variant) — runs on free CPU tier / fast on
  small GPU, far cheaper to fine-tune, and arguably *more* on-theme for "local-first." Keep 31B as
  the headline, E4B as the safety net for a reliable live demo.
- **`chat.db` schema / `attributedBody`.** Mitigated by using `imessage-exporter`/`imessage_reader`.
- **Full Disk Access** must be granted to the collector's process or reads return empty.
- **Privacy:** the autonomous Mac-collector path sends messages to the Space (token-gated); the
  hero phone-paste path keeps data client-side (calendar tokens live in the browser, nothing
  persists server-side). The Space now lives in the public **`build-small-hackathon`** submission
  org, so the *source* is public — but user data still never lands on the server.
- **Relative-date accuracy** is the main quality risk — pass the current datetime into the prompt
  and weight the dataset toward relative-date examples.