| # Plan: Local-First iMessage → Calendar Agent (Gradio + llama.cpp + fine-tuned Gemma + an OpenBMB MiniCPM-planned agent) |
|
|
| ## Who this is for |
|
|
| One named person: **a busy parent** whose kid's school/activity events are buried in a noisy class |
| group chat (picture day, the practice that moved, the RSVP). They read it once, mean to add it later, |
| and miss it. Success = *their* day measurably improves — events captured from the chat, conflicts |
| caught against their calendar, minutes saved — with **zero setup**: paste the thread or a screenshot |
| from a phone browser. The local-LLM / fine-tune work below is a **means** to better extraction, not |
| the point; the app must deliver value with no GPU (stub agent) first. |
|
|
| ## Context |
|
|
| You want an agent that reads iMessage-style threads, understands the conversation, and turns them |
| into calendar events/reminders — exposed through a custom Gradio UI deployed as a Hugging Face Space. |
| Two local models share the work: our fine-tuned Gemma does the *reading* (thread → validated |
| ActionPlan), and an **OpenBMB MiniCPM** planner does the *orchestrating* — the brain behind **Run the agents**, |
| driving the Space's own MCP tools (`extract_events → check_conflicts → make_ics`) as a visible |
| multi-step agent. The build competes in the **Backyard AI** track (general and OpenBMB prizes are |
| awarded per track) and satisfies the quests secondary to the user story above: **Off the Grid** (no |
| cloud AI APIs, local-first), **Well-Tuned** (a fine-tuned model on HF), **Off-Brand** (custom UI), |
| and **Llama Champion** (both Gemma and MiniCPM are served through llama.cpp). |
|
|
| ### Feasibility verdict: YES, with one re-architecture |
|
|
| The request as *literally* worded has two impossibilities, both solvable: |
|
|
| 1. **No app or cloud can read iMessage on iOS.** Apple exposes no API for iMessage/SMS content. |
| → **Solved:** you have a Mac. iMessages sync to `~/Library/Messages/chat.db`; a small local |
| collector reads it. This is the *only* supported path and it keeps data local ("off the grid"). |
| 2. **A model cannot "run on your phone," and a HF Space runs in the cloud, not on-device.** |
| → **Solved:** "on my phone" = *used from* your phone's browser. The Space does its own llama.cpp |
| inference and calls no external AI service, so "hosted Space" and "off the grid" reconcile. |
|
|
| Confirmed decisions: |
| - **Ingestion:** Mac collector reading `chat.db`. |
| - **Calendar output:** local `.ics` files first (strictly off-grid), with an *optional* Google |
| Calendar push toggle as a bonus. |
| - **Extraction model:** fine-tune Gemma, serve as GGUF via llama.cpp (production serves the |
| **E4B** edge fine-tune, `build-small-hackathon/gemma-4-cal-gguf`). |
| - **Agent planner:** **OpenBMB MiniCPM** (`openbmb/MiniCPM4.1-8B-GGUF`, Q4; the 1B variant is a |
| config switch) on a second llama-server — it plans, the MCP tools execute, every step visible. |
|
|
| --- |
|
|
| ## Architecture |
|
|
| ``` |
| ┌────────── Your Mac (local) ──────────┐ ┌──────── Hugging Face Space (Docker) ────────┐ |
| │ collector.py (Full Disk Access) │ HTTPS │ Gradio (custom theme/CSS) ── Off-Brand │ |
| │ • polls chat.db for new messages │ +token │ │ │ |
| │ • parses text / attributedBody ├────────▶│ FastAPI /ingest ──▶ extraction pipeline │ |
| │ • POSTs new msgs to Space /ingest │ │ │ │ |
| └───────────────────────────────────────┘ │ llama.cpp (llama-cpp-python) ── Llama Champ │ |
| │ running YOUR fine-tuned gemma-4-31B GGUF │ |
| View/approve from phone browser ───────────────▶│ │ ── Off the Grid (local) │ |
| │ JSON events → pydantic validate │ |
| │ ├──▶ .ics file (download) │ |
| │ └──▶ optional Google Calendar push │ |
| └──────────────────────────────────────────────┘ |
| ``` |
|
|
| Flow: messages → extraction prompt → model emits structured JSON of candidate events → |
| validated → shown in UI for review → user approves → `.ics` generated (and/or pushed to GCal). |
|
|
| **Run the agents** runs the same flow agentically: an **OpenBMB MiniCPM** planner (second local |
| llama-server, OpenAI-compatible) consumes the Space's own **MCP tool surface** — |
| `extract_events → check_conflicts → make_ics` — through smolagents, so the pipeline above is |
| demonstrated as multi-step tool use over the public tool contract, with the planner's trace on |
| screen (`server/orchestrator.py`). Stub/CI falls back to a scripted planner so the tab always works. |
|
|
| --- |
|
|
| ## Components |
|
|
| ### 1. Mac-side iMessage collector (`collector/collector.py`) |
| - **Reuse, don't reinvent the DB parsing.** Modern macOS stores message text in the |
| `attributedBody` (NSAttributedString) blob, not always the `text` column. Use the battle-tested |
| **`imessage-exporter`** (ReagentX, Rust) or the Python **`imessage_reader`** lib rather than |
| hand-rolling SQL. If hand-querying: join `message` ⨝ `handle` ⨝ `chat_message_join` ⨝ `chat`, |
| track last seen `ROWID`, poll on an interval. |
| - Requires **Full Disk Access** for the running process (System Settings → Privacy & Security). |
| - Sends only new messages to the Space `/ingest` endpoint over HTTPS with a shared bearer token. |
| - Config: which chats to watch, poll interval, Space URL, token (`.env`, never committed). |
| |
| ### 2. HF Space backend (`app.py`, `server/`) |
| - **Docker SDK Space** (`README.md` frontmatter: `sdk: docker`, `app_port: 7860`). |
| - **llama.cpp** loads the fine-tuned GGUF and serves chat completions — satisfies |
| *Llama Champion*; no external AI call satisfies *Off the Grid*. |
| - **Agent orchestrator** (`server/orchestrator.py`): the **OpenBMB MiniCPM** planner behind |
| **Run the agents** (its own llama-server) drives the Space's MCP tools as a multi-step agent — the OpenBMB |
| per-track prize case, and the same extraction pipeline exercised through the public tool |
| contract rather than private imports. |
| - `/ingest` (FastAPI, mounted alongside Gradio) receives messages, runs the extraction prompt, |
| returns candidate events; results surface in the Gradio UI for review. |
| - **Compute:** Q4_K_M GGUF of a 31B ≈ 18–20 GB → does **not** fit the free CPU tier (16 GB / 2 cores). |
| Serve on a GPU: **ZeroGPU** (free, H200/70 GB — but cold GGUF load per acquisition; document the |
| caveat) or a **paid GPU Space** (e.g. L4/L40S) for a smooth always-warm demo. See Fallback. |
|
|
| ### 3. Fine-tuning pipeline (`training/`) |
| - **Task:** conversation snippet → strict JSON list of events |
| `{title, start, end, location, attendees, reminder_minutes, notes}`. |
| - **Data:** build a synthetic instruction dataset (~500–2000 examples) of realistic chat threads |
| paired with the target JSON. Generation/augmentation for *training data* can use any tooling |
| offline — the "no cloud API" rule applies to the *running app's inference*, not dataset prep. |
| Include hard cases: relative dates ("next Thurs"), ranges, no-event chitchat (empty list), |
| timezones, multiple events per thread. |
| - **Method:** QLoRA via **Unsloth** (Qwen3-0.6B GRPO experience applies), 4-bit, r=16, |
| 1–3 epochs. 31B QLoRA needs an A100/H100 80 GB (Colab Pro+/RunPod/Lambda, ~hours). |
| - **Export:** merge LoRA → `convert_hf_to_gguf.py` (llama.cpp) → `llama-quantize` to Q4_K_M → |
| **publish GGUF to your HF repo** (satisfies *Well-Tuned*). Space downloads it at startup via |
| `huggingface_hub`. |
|
|
| ### 4. Custom Gradio UI (`ui/`, `static/`) — *Off-Brand* |
| - `gr.Blocks` with a custom `gr.themes.Base(...)` palette + injected `css=` (custom fonts, layout, |
| cards) to push well past the default look. |
| - Screens: connection/status, incoming-message feed, **review queue** (edit candidate events |
| inline, approve/reject), download `.ics`, optional "Push to Google Calendar" toggle, settings. |
|
|
| ### 5. Calendar output (`calendar_out/`) |
| - **`.ics` (default, off-grid):** generate with the `icalendar` lib; offer as a download in the UI. |
| - **Google Calendar (optional bonus):** `google-api-python-client` OAuth; behind a toggle so the |
| off-grid demo path stays pure. Clearly labeled as the one optional cloud touchpoint. |
| |
| --- |
| |
| ## Hackathon requirement mapping |
| |
| | Track | How it's satisfied | |
| |---|---| |
| | Off the Grid (local-first, no cloud AI APIs) | All inference is local llama.cpp in the Space; data originates on your Mac; `.ics` is the default output. | |
| | Well-Tuned (fine-tuned model on HF) | QLoRA fine-tune of `gemma-4-31B-it`, GGUF published to your HF repo. | |
| | Off-Brand (custom UI) | Custom Gradio theme + CSS, not the stock look. | |
| | Llama Champion (llama.cpp) | Inference via `llama-cpp-python`. | |
| | Gradio app on HF Space | Docker Space serving Gradio + FastAPI `/ingest`. | |
| |
| --- |
| |
| ## Build phases |
| |
| 1. **Hero path (no GPU):** Docker Space with custom-themed Gradio + the *stub* extractor → paste / |
| "Try a sample" / screenshot → review → `.ics` download, working end-to-end on a phone browser. |
| This is the parent's whole experience and must stand alone with no model. |
| 2. **Measure impact:** persisted **This week** panel (events captured, conflicts caught, minutes |
| saved) via `server/impact.py`, recorded when the parent exports. Proves *their* day got better. |
| 3. **Accuracy upgrade (optional):** wire `llama-cpp-python` with a community `gemma-4-31B-it` GGUF on |
| a GPU Space; swap the stub for the model + JSON-schema prompt + pydantic validation. |
| 4. **Fine-tune (optional):** dataset → Unsloth QLoRA → GGUF → publish to HF → point the Space at it. |
| 5. **Optional auto-feed:** Mac `collector.py` reading `chat.db` → POST `/ingest` (power users only). |
| |
| --- |
| |
| ## Verification |
| |
| - **End-to-end (stub, phase 1):** open Space in phone browser → tap **Try a sample** (or paste a |
| chat) → event appears in review queue → download `.ics` → import to a calendar, confirm date/time. |
| - **Impact (phase 2):** after exporting, **Activity → This week** shows events captured and time |
| saved > 0; restart the app (same `IMPACT_PATH`) and confirm the weekly numbers persist while the |
| live tiles reset. `minutes_saved` is a stated estimate (`IMPACT_MIN_PER_EVENT`=8, |
| `IMPACT_MIN_PER_CONFLICT`=15, env-overridable), not a measurement. |
| - **Collector (phase 2):** send yourself a test iMessage ("lunch Tuesday 1pm") → confirm it reaches |
| `/ingest` and surfaces in the feed. |
| - **Model (phase 3+):** curated eval set of chats with known expected events; measure JSON validity |
| rate + field accuracy (esp. relative-date resolution); confirm empty-list on non-event chats. |
| - **llama.cpp:** confirm the Space logs show llama.cpp loading *your* GGUF, no external AI calls. |
|
|
| --- |
|
|
| ## Risks & fallbacks |
|
|
| - **31B serving cost/latency.** Q4 31B needs a GPU; ZeroGPU has cold-load + quota friction, paid GPU |
| has cost. **Fallback:** fine-tune **Gemma 4 E4B** (edge variant) — runs on free CPU tier / fast on |
| small GPU, far cheaper to fine-tune, and arguably *more* on-theme for "local-first." Keep 31B as |
| the headline, E4B as the safety net for a reliable live demo. |
| - **`chat.db` schema / `attributedBody`.** Mitigated by using `imessage-exporter`/`imessage_reader`. |
| - **Full Disk Access** must be granted to the collector's process or reads return empty. |
| - **Privacy:** the autonomous Mac-collector path sends messages to the Space (token-gated); the |
| hero phone-paste path keeps data client-side (calendar tokens live in the browser, nothing |
| persists server-side). The Space now lives in the public **`build-small-hackathon`** submission |
| org, so the *source* is public — but user data still never lands on the server. |
| - **Relative-date accuracy** is the main quality risk — pass the current datetime into the prompt |
| and weight the dataset toward relative-date examples. |
|
|
