FIELD_NOTES.md

Field Notes — building the iMessage → Calendar agent

What I set out to build, where reality bent the plan, and what I'd do next. This is the "what I learned" companion to the product docs (README) and the design doc (PLAN).

The goal in one line

Turn the calendar logistics buried in a chat thread — "picture day moved to Thursday 9am", "soccer is Tuesday now" — into reviewed calendar events, from a phone, with the data staying private.

1. "Read my iMessages" is impossible as literally asked — and that shaped everything

iOS exposes no API for iMessage/SMS content. There is no on-device path. The only place the messages exist in a queryable form is a Mac, where they sync to ~/Library/Messages/chat.db. So the architecture forked early:

• A Mac-side collector (collector/collector.py) reads chat.db (read-only, mode=ro) and POSTs new rows to the Space.
• "On my phone" was reinterpreted honestly as used from a phone browser — the Space is hosted, the UI is mobile-friendly, but the model runs in the Space.

The biggest adoption lesson came later: requiring a Mac collector + Full Disk Access is a wall for a non-technical user. The fix was to make paste-from-phone the hero path (the collector is now strictly optional) — no install, no DB, no permissions. Most of that capability already existed in the Schedule tab; it was just framed as secondary.

2. attributedBody is the iMessage parsing trap

Modern Messages often stores the body in attributedBody (an NSAttributedString binary blob), not the text column. The collector reads text directly for simplicity and skips messages that only have attributedBody (collector/collector.py:88-94) — a deliberate, called-out gap. The right move for production is to not hand-roll this: use imessage-exporter (ReagentX) or imessage_reader. Noting the limitation in code beat pretending the naive SQL was complete.

3. Relative dates are the real accuracy battleground

The hard part isn't "is there an event" — it's when. "Next Thursday", "the 14th", "in two weeks" only resolve against a reference time. Two design responses:

• The system prompt pins "Current datetime" into every request and instructs the model to resolve relative dates from it (server/agent.py:21-34).
• Conflict math is deterministic, not model-driven. Overlap/adjacent/tight detection and alternative-time proposals live in calendar_out/freebusy.py, because once you have ISO datetimes, interval math should never be left to an LLM. The model decides what; code decides when-it-clashes.

The stub extractor's naive "match a time → 1h event tomorrow" (server/agent.py:152-175) is intentionally dumb — it exists to prove the pipeline, and its dumbness is a good reminder of exactly how much the fine-tune has to get right.

4. Stub-first was the best architectural call

USE_STUB_EXTRACTOR=1 swaps the model for a regex heuristic (server/agent.py:85,124), forced on in tests (tests/conftest.py). Payoffs:

• The whole app — paste → events → conflicts → .ics download → impact panel — works end-to-end with no GPU, so a demo (and CI) never depends on a model load.
• llama_cpp and the Google libs are lazy-imported, so requirements-ci.txt can exclude them and the test suite runs in seconds, offline.

Lesson: make the expensive dependency optional from day one and the cheap path becomes your test harness, your demo, and your free tier all at once.

5. Reframing around one person changed the scope more than any feature

The project started as a four-track hackathon checklist. Rewriting it around a single named person — a busy parent whose kid's events are buried in a class group chat — forced three concrete changes: phone-paste as the default, a one-tap Try a sample class-chat (ui/blocks.py), and a "This week" impact panel.

On measurement: minutes_saved (server/impact.py) is a configurable estimate, not a measurement (default 8 min/event + 15 min/conflict). Saying that plainly — in the UI, the README, and here — matters more than a bigger-looking number. A capture is only counted when the parent accepts events by exporting them, so the metric tracks value taken, not previews shown.

6. Fine-tuning economics: Modal credits + honest scope

QLoRA on a 31B needs an 80GB GPU. training/modal_train.py wraps the existing train_qlora.py + export_gguf.sh to run on a serverless A100/H100 and publish the GGUF to HF — roughly $5–15 per run, so ~$250 of credit is 15–40 iterations. The "Well-Tuned" track went the distance: the eval-gated E4B fine-tune is published and is what production serves — build-small-hackathon/gemma-4-cal-gguf — after clearing the gate over six runs at zero quality cost vs. stock E4B. (Re-running the pipeline still spends your own Modal credits; the turnkey path is there whenever you want to retrain.)

A small rule that paid off: training-data generation can use any offline tooling — the "no cloud AI API" rule applies only to the running app's inference, not to dataset prep.

7. Two models, not one — a 1B planner over the same tools

What shipped is two small local models, not one. The fine-tuned gemma-cal E4B does the reading (thread → validated ActionPlan); a 1B OpenBMB MiniCPM does the orchestrating. Clicking Run the agents hands the job to MiniCPM, which drives the Space's own MCP tools — extract_events → check_conflicts → make_ics — as a visible multi-step agent (server/orchestrator.py), consuming the public tool contract instead of calling internals. Two things I'd underline: keep the planner optional (a deterministic scripted plan is the fallback, so the agentic path never hard-depends on a second model load), and don't let "agent" become a separate destination — the same Run the agents action drives both the home workflow and the orchestrated trace, so it stays one engine, not a second UI to keep in sync.

8. The Off-the-Grid tension

"No cloud AI APIs" and "serve a 31B" pull against each other: a Q4 31B GGUF is ~18–20GB and needs a GPU. Keeping inference in the Space via llama.cpp preserves the privacy story but costs GPU. The honest compromise is the E4B edge variant for the free tier, with the 31B as the headline. I deliberately did not offload inference to a third-party endpoint, because "your own Modal GPU" and "a cloud AI API" are easy to conflate and a purist judge would be right to dock it.

The same principle drove the trace-sharing design (below): the hosted Space holds no HF token — it only offers a local download, and a separate local CLI does the upload with your own auth.

9. What I'd do next

• Durable trace/metrics store. The activity bus is an 800-entry in-memory ring buffer (server/events.py) — runs are lost on restart, so only recent runs are exportable. A small append-only store (the impact log already shows the pattern) would fix it.
• Decode attributedBody (or adopt imessage-exporter) so text-less messages stop being dropped.
• A real eval set from the expanded dataset — measure JSON validity + field accuracy, especially relative-date resolution and empty-list-on-chitchat.
• Trace redaction as a tested invariant. Today it's an allowlist over current emit sites (server/trace.py); a lint/test that fails when a new emit(...) puts free text on a non-ingest stage would keep it honest as the code grows.

Publishing these notes

This file is linked from the README. It can also be pasted into the Space's README (the Space card renders Markdown) or posted to the model/dataset repo's Community tab on the Hub so others can learn from the build.