# Field Notes — building the iMessage → Calendar agent What I set out to build, where reality bent the plan, and what I'd do next. This is the "what I learned" companion to the product docs ([README](./README.md)) and the design doc ([PLAN](./PLAN.md)). ## The goal in one line Turn the calendar logistics buried in a chat thread — *"picture day moved to Thursday 9am", "soccer is Tuesday now"* — into reviewed calendar events, from a phone, with the data staying private. ## 1. "Read my iMessages" is impossible as literally asked — and that shaped everything iOS exposes **no API** for iMessage/SMS *content*. There is no on-device path. The only place the messages exist in a queryable form is a Mac, where they sync to `~/Library/Messages/chat.db`. So the architecture forked early: - A Mac-side collector ([`collector/collector.py`](./collector/collector.py)) reads `chat.db` (read-only, `mode=ro`) and POSTs new rows to the Space. - "On my phone" was reinterpreted honestly as **used from** a phone browser — the Space is hosted, the UI is mobile-friendly, but the model runs in the Space. The biggest adoption lesson came later: requiring a Mac collector + Full Disk Access is a wall for a non-technical user. The fix was to make **paste-from-phone** the hero path (the collector is now strictly optional) — no install, no DB, no permissions. Most of that capability already existed in the Schedule tab; it was just framed as secondary. ## 2. `attributedBody` is the iMessage parsing trap Modern Messages often stores the body in `attributedBody` (an `NSAttributedString` binary blob), **not** the `text` column. The collector reads `text` directly for simplicity and **skips messages that only have `attributedBody`** ([`collector/collector.py:88-94`](./collector/collector.py)) — a deliberate, called-out gap. The right move for production is to not hand-roll this: use `imessage-exporter` (ReagentX) or `imessage_reader`. Noting the limitation in code beat pretending the naive SQL was complete. ## 3. Relative dates are the real accuracy battleground The hard part isn't "is there an event" — it's *when*. "Next Thursday", "the 14th", "in two weeks" only resolve against a reference time. Two design responses: - The system prompt pins **"Current datetime"** into every request and instructs the model to resolve relative dates from it ([`server/agent.py:21-34`](./server/agent.py)). - **Conflict math is deterministic, not model-driven.** Overlap/adjacent/tight detection and alternative-time proposals live in [`calendar_out/freebusy.py`](./calendar_out/freebusy.py), because once you have ISO datetimes, interval math should never be left to an LLM. The model decides *what*; code decides *when-it-clashes*. The stub extractor's naive "match a time → 1h event tomorrow" ([`server/agent.py:152-175`](./server/agent.py)) is intentionally dumb — it exists to prove the pipeline, and its dumbness is a good reminder of exactly how much the fine-tune has to get right. ## 4. Stub-first was the best architectural call `USE_STUB_EXTRACTOR=1` swaps the model for a regex heuristic ([`server/agent.py:85,124`](./server/agent.py)), forced on in tests ([`tests/conftest.py`](./tests/conftest.py)). Payoffs: - The whole app — paste → events → conflicts → `.ics` download → impact panel — **works end-to-end with no GPU**, so a demo (and CI) never depends on a model load. - `llama_cpp` and the Google libs are **lazy-imported**, so `requirements-ci.txt` can exclude them and the test suite runs in seconds, offline. Lesson: make the expensive dependency optional from day one and the cheap path becomes your test harness, your demo, and your free tier all at once. ## 5. Reframing around one person changed the scope more than any feature The project started as a four-track hackathon checklist. Rewriting it around a single named person — a **busy parent** whose kid's events are buried in a class group chat — forced three concrete changes: phone-paste as the default, a one-tap **Try a sample** class-chat ([`ui/blocks.py`](./ui/blocks.py)), and a **"This week"** impact panel. On measurement: `minutes_saved` ([`server/impact.py`](./server/impact.py)) is a **configurable estimate, not a measurement** (default 8 min/event + 15 min/conflict). Saying that plainly — in the UI, the README, and here — matters more than a bigger-looking number. A capture is only counted when the parent *accepts* events by exporting them, so the metric tracks value taken, not previews shown. ## 6. Fine-tuning economics: Modal credits + honest scope QLoRA on a 31B needs an 80GB GPU. [`training/modal_train.py`](./training/modal_train.py) wraps the existing `train_qlora.py` + `export_gguf.sh` to run on a serverless A100/H100 and publish the GGUF to HF — roughly **$5–15 per run**, so ~$250 of credit is 15–40 iterations. The "Well-Tuned" track went the distance: the eval-gated **E4B** fine-tune is published and is what production serves — [`build-small-hackathon/gemma-4-cal-gguf`](https://huggingface.co/build-small-hackathon/gemma-4-cal-gguf) — after clearing the gate over six runs at zero quality cost vs. stock E4B. (Re-running the pipeline still spends your own Modal credits; the turnkey path is there whenever you want to retrain.) A small rule that paid off: training-data generation can use *any* offline tooling — the "no cloud AI API" rule applies only to the **running app's inference**, not to dataset prep. ## 7. Two models, not one — a 1B planner over the same tools What shipped is two small local models, not one. The fine-tuned **gemma-cal E4B** does the *reading* (thread → validated `ActionPlan`); a 1B **OpenBMB MiniCPM** does the *orchestrating*. Clicking **Run the agents** hands the job to MiniCPM, which drives the Space's own MCP tools — `extract_events → check_conflicts → make_ics` — as a visible multi-step agent ([`server/orchestrator.py`](./server/orchestrator.py)), consuming the *public* tool contract instead of calling internals. Two things I'd underline: keep the planner **optional** (a deterministic scripted plan is the fallback, so the agentic path never hard-depends on a second model load), and don't let "agent" become a separate destination — the same **Run the agents** action drives both the home workflow and the orchestrated trace, so it stays one engine, not a second UI to keep in sync. ## 8. The Off-the-Grid tension "No cloud AI APIs" and "serve a 31B" pull against each other: a Q4 31B GGUF is ~18–20GB and needs a GPU. Keeping inference **in the Space via `llama.cpp`** preserves the privacy story but costs GPU. The honest compromise is the **E4B edge variant** for the free tier, with the 31B as the headline. I deliberately did **not** offload inference to a third-party endpoint, because "your own Modal GPU" and "a cloud AI API" are easy to conflate and a purist judge would be right to dock it. The same principle drove the trace-sharing design (below): the hosted Space holds **no HF token** — it only offers a **local download**, and a separate local CLI does the upload with your own auth. ## 9. What I'd do next - **Durable trace/metrics store.** The activity bus is an 800-entry in-memory ring buffer ([`server/events.py`](./server/events.py)) — runs are lost on restart, so only recent runs are exportable. A small append-only store (the impact log already shows the pattern) would fix it. - **Decode `attributedBody`** (or adopt `imessage-exporter`) so text-less messages stop being dropped. - **A real eval set** from the expanded dataset — measure JSON validity + field accuracy, especially relative-date resolution and empty-list-on-chitchat. - **Trace redaction as a tested invariant.** Today it's an allowlist over current emit sites ([`server/trace.py`](./server/trace.py)); a lint/test that fails when a new `emit(...)` puts free text on a non-`ingest` stage would keep it honest as the code grows. ## Publishing these notes This file is linked from the README. It can also be pasted into the Space's README (the Space card renders Markdown) or posted to the model/dataset repo's **Community** tab on the Hub so others can learn from the build.