PLAN.md

Plan: Local-First iMessage → Calendar Agent (Gradio + llama.cpp + fine-tuned Gemma + an OpenBMB MiniCPM-planned agent)

Who this is for

One named person: a busy parent whose kid's school/activity events are buried in a noisy class group chat (picture day, the practice that moved, the RSVP). They read it once, mean to add it later, and miss it. Success = their day measurably improves — events captured from the chat, conflicts caught against their calendar, minutes saved — with zero setup: paste the thread or a screenshot from a phone browser. The local-LLM / fine-tune work below is a means to better extraction, not the point; the app must deliver value with no GPU (stub agent) first.

Context

You want an agent that reads iMessage-style threads, understands the conversation, and turns them into calendar events/reminders — exposed through a custom Gradio UI deployed as a Hugging Face Space. Two local models share the work: our fine-tuned Gemma does the reading (thread → validated ActionPlan), and an OpenBMB MiniCPM planner does the orchestrating — the brain behind Run the agents, driving the Space's own MCP tools (extract_events → check_conflicts → make_ics) as a visible multi-step agent. The build competes in the Backyard AI track (general and OpenBMB prizes are awarded per track) and satisfies the quests secondary to the user story above: Off the Grid (no cloud AI APIs, local-first), Well-Tuned (a fine-tuned model on HF), Off-Brand (custom UI), and Llama Champion (both Gemma and MiniCPM are served through llama.cpp).

Feasibility verdict: YES, with one re-architecture

The request as literally worded has two impossibilities, both solvable:

1. No app or cloud can read iMessage on iOS. Apple exposes no API for iMessage/SMS content. → Solved: you have a Mac. iMessages sync to ~/Library/Messages/chat.db; a small local collector reads it. This is the only supported path and it keeps data local ("off the grid").
2. A model cannot "run on your phone," and a HF Space runs in the cloud, not on-device. → Solved: "on my phone" = used from your phone's browser. The Space does its own llama.cpp inference and calls no external AI service, so "hosted Space" and "off the grid" reconcile.

Confirmed decisions:

• Ingestion: Mac collector reading chat.db.
• Calendar output: local .ics files first (strictly off-grid), with an optional Google Calendar push toggle as a bonus.
• Extraction model: fine-tune Gemma, serve as GGUF via llama.cpp (production serves the E4B edge fine-tune, build-small-hackathon/gemma-4-cal-gguf).
• Agent planner: OpenBMB MiniCPM (openbmb/MiniCPM4.1-8B-GGUF, Q4; the 1B variant is a config switch) on a second llama-server — it plans, the MCP tools execute, every step visible.

---

Architecture

┌────────── Your Mac (local) ──────────┐         ┌──────── Hugging Face Space (Docker) ────────┐
│  collector.py (Full Disk Access)      │  HTTPS  │  Gradio (custom theme/CSS)  ── Off-Brand     │
│  • polls chat.db for new messages     │ +token  │        │                                     │
│  • parses text / attributedBody       ├────────▶│  FastAPI /ingest  ──▶  extraction pipeline   │
│  • POSTs new msgs to Space /ingest    │         │        │                                     │
└───────────────────────────────────────┘         │  llama.cpp (llama-cpp-python) ── Llama Champ │
                                                   │   running YOUR fine-tuned gemma-4-31B GGUF   │
   View/approve from phone browser ───────────────▶│        │            ── Off the Grid (local) │
                                                   │  JSON events → pydantic validate            │
                                                   │        ├──▶ .ics file (download)            │
                                                   │        └──▶ optional Google Calendar push   │
                                                   └──────────────────────────────────────────────┘

Flow: messages → extraction prompt → model emits structured JSON of candidate events → validated → shown in UI for review → user approves → .ics generated (and/or pushed to GCal).

Run the agents runs the same flow agentically: an OpenBMB MiniCPM planner (second local llama-server, OpenAI-compatible) consumes the Space's own MCP tool surface — extract_events → check_conflicts → make_ics — through smolagents, so the pipeline above is demonstrated as multi-step tool use over the public tool contract, with the planner's trace on screen (server/orchestrator.py). Stub/CI falls back to a scripted planner so the tab always works.

---

Components

1. Mac-side iMessage collector (collector/collector.py)

• Reuse, don't reinvent the DB parsing. Modern macOS stores message text in the attributedBody (NSAttributedString) blob, not always the text column. Use the battle-tested imessage-exporter (ReagentX, Rust) or the Python imessage_reader lib rather than hand-rolling SQL. If hand-querying: join message ⨝ handle ⨝ chat_message_join ⨝ chat, track last seen ROWID, poll on an interval.
• Requires Full Disk Access for the running process (System Settings → Privacy & Security).
• Sends only new messages to the Space /ingest endpoint over HTTPS with a shared bearer token.
• Config: which chats to watch, poll interval, Space URL, token (.env, never committed).

2. HF Space backend (app.py, server/)

• Docker SDK Space (README.md frontmatter: sdk: docker, app_port: 7860).
• llama.cpp loads the fine-tuned GGUF and serves chat completions — satisfies Llama Champion; no external AI call satisfies Off the Grid.
• Agent orchestrator (server/orchestrator.py): the OpenBMB MiniCPM planner behind Run the agents (its own llama-server) drives the Space's MCP tools as a multi-step agent — the OpenBMB per-track prize case, and the same extraction pipeline exercised through the public tool contract rather than private imports.
• /ingest (FastAPI, mounted alongside Gradio) receives messages, runs the extraction prompt, returns candidate events; results surface in the Gradio UI for review.
• Compute: Q4_K_M GGUF of a 31B ≈ 18–20 GB → does not fit the free CPU tier (16 GB / 2 cores). Serve on a GPU: ZeroGPU (free, H200/70 GB — but cold GGUF load per acquisition; document the caveat) or a paid GPU Space (e.g. L4/L40S) for a smooth always-warm demo. See Fallback.

3. Fine-tuning pipeline (training/)

• Task: conversation snippet → strict JSON list of events {title, start, end, location, attendees, reminder_minutes, notes}.
• Data: build a synthetic instruction dataset (~500–2000 examples) of realistic chat threads paired with the target JSON. Generation/augmentation for training data can use any tooling offline — the "no cloud API" rule applies to the running app's inference, not dataset prep. Include hard cases: relative dates ("next Thurs"), ranges, no-event chitchat (empty list), timezones, multiple events per thread.
• Method: QLoRA via Unsloth (Qwen3-0.6B GRPO experience applies), 4-bit, r=16, 1–3 epochs. 31B QLoRA needs an A100/H100 80 GB (Colab Pro+/RunPod/Lambda, ~hours).
• Export: merge LoRA → convert_hf_to_gguf.py (llama.cpp) → llama-quantize to Q4_K_M → publish GGUF to your HF repo (satisfies Well-Tuned). Space downloads it at startup via huggingface_hub.

4. Custom Gradio UI (ui/, static/) — Off-Brand

• gr.Blocks with a custom gr.themes.Base(...) palette + injected css= (custom fonts, layout, cards) to push well past the default look.
• Screens: connection/status, incoming-message feed, review queue (edit candidate events inline, approve/reject), download .ics, optional "Push to Google Calendar" toggle, settings.

5. Calendar output (calendar_out/)

• .ics (default, off-grid): generate with the icalendar lib; offer as a download in the UI.
• Google Calendar (optional bonus): google-api-python-client OAuth; behind a toggle so the off-grid demo path stays pure. Clearly labeled as the one optional cloud touchpoint.

---

Hackathon requirement mapping

Track	How it's satisfied
Off the Grid (local-first, no cloud AI APIs)	All inference is local llama.cpp in the Space; data originates on your Mac; .ics is the default output.
Well-Tuned (fine-tuned model on HF)	QLoRA fine-tune of gemma-4-31B-it, GGUF published to your HF repo.
Off-Brand (custom UI)	Custom Gradio theme + CSS, not the stock look.
Llama Champion (llama.cpp)	Inference via llama-cpp-python.
Gradio app on HF Space	Docker Space serving Gradio + FastAPI /ingest.

---

Build phases

1. Hero path (no GPU): Docker Space with custom-themed Gradio + the stub extractor → paste / "Try a sample" / screenshot → review → .ics download, working end-to-end on a phone browser. This is the parent's whole experience and must stand alone with no model.
2. Measure impact: persisted This week panel (events captured, conflicts caught, minutes saved) via server/impact.py, recorded when the parent exports. Proves their day got better.
3. Accuracy upgrade (optional): wire llama-cpp-python with a community gemma-4-31B-it GGUF on a GPU Space; swap the stub for the model + JSON-schema prompt + pydantic validation.
4. Fine-tune (optional): dataset → Unsloth QLoRA → GGUF → publish to HF → point the Space at it.
5. Optional auto-feed: Mac collector.py reading chat.db → POST /ingest (power users only).

---

Verification

• End-to-end (stub, phase 1): open Space in phone browser → tap Try a sample (or paste a chat) → event appears in review queue → download .ics → import to a calendar, confirm date/time.
• Impact (phase 2): after exporting, Activity → This week shows events captured and time saved > 0; restart the app (same IMPACT_PATH) and confirm the weekly numbers persist while the live tiles reset. minutes_saved is a stated estimate (IMPACT_MIN_PER_EVENT=8, IMPACT_MIN_PER_CONFLICT=15, env-overridable), not a measurement.
• Collector (phase 2): send yourself a test iMessage ("lunch Tuesday 1pm") → confirm it reaches /ingest and surfaces in the feed.
• Model (phase 3+): curated eval set of chats with known expected events; measure JSON validity rate + field accuracy (esp. relative-date resolution); confirm empty-list on non-event chats.
• llama.cpp: confirm the Space logs show llama.cpp loading your GGUF, no external AI calls.

---

Risks & fallbacks

• 31B serving cost/latency. Q4 31B needs a GPU; ZeroGPU has cold-load + quota friction, paid GPU has cost. Fallback: fine-tune Gemma 4 E4B (edge variant) — runs on free CPU tier / fast on small GPU, far cheaper to fine-tune, and arguably more on-theme for "local-first." Keep 31B as the headline, E4B as the safety net for a reliable live demo.
• chat.db schema / attributedBody. Mitigated by using imessage-exporter/imessage_reader.
• Full Disk Access must be granted to the collector's process or reads return empty.
• Privacy: the autonomous Mac-collector path sends messages to the Space (token-gated); the hero phone-paste path keeps data client-side (calendar tokens live in the browser, nothing persists server-side). The Space now lives in the public build-small-hackathon submission org, so the source is public — but user data still never lands on the server.
• Relative-date accuracy is the main quality risk — pass the current datetime into the prompt and weight the dataset toward relative-date examples.