docs/architecture.md

Architecture — workflows and the LLMs behind them

An AI-solution-architect view of the agentic system: every workflow through the platform, and exactly which model (if any) each one calls. The architectural signature: the extraction core is one grammar-constrained LLM call, the MiniCPM planner adds a visible multi-step loop over the platform's own public MCP tool contract, everything verifiable — conflict math, dedup, time proposals, eval gates — stays deterministic, and there are zero cloud-AI API calls anywhere, training included.

System workflow

flowchart TB
    subgraph ENTRY["1 · Entry points — four front-ends, one contract"]
        direction LR
        UIIN["🖥️ Gradio UI<br/>Schedule flow + Agent tab<br/>(paste thread, screenshots, .ics)"]
        SHORT["📱 iOS Shortcut /<br/>Android Tasker"]
        MAC["🍎 Mac collector<br/>polls iMessage chat.db<br/>(collector/collector.py)"]
        MCPC["🤖 MCP clients<br/>Claude Desktop, Cursor"]
    end

    subgraph API["2 · API & orchestration — app.py (FastAPI + Gradio, one port)"]
        AGENTEP["POST /agent<br/>bearer-token, stateless"]
        INGEST["POST /ingest → feed store<br/>AUTONOMOUS=1 triggers on<br/>your outgoing message (is_from_me)"]
        ROLL["threads.rolling_thread<br/>per-chat window (20 msgs / 12 h)"]
        MCPT["MCP tools — server/mcp_tools.py<br/>extract_events · make_ics · check_conflicts"]
    end

    subgraph ORCH["2a · Agentic orchestration — server/orchestrator.py"]
        SMOL["smolagents ToolCallingAgent<br/>planned by MiniCPM, ≤6 steps<br/>playbook: extract → check → render<br/>final ActionPlan re-derived deterministically"]
        SCRIPT["ScriptedPlanner — no LLM<br/>identical tool sequence + step events<br/>(stub mode, CI, planner failure)"]
    end

    subgraph CORE["3 · Agent core — server/pipeline.py → server/agent.py"]
        PROMPT["Prompt assembly:<br/>SYSTEM + memory recall block<br/>+ existing calendar + thread + images"]
        GEN["Grammar-constrained generation<br/>→ ActionPlan JSON (always parses)"]
        PROMPT --> GEN
    end

    subgraph LLMT["4 · LLM tier — ALL inference is local llama.cpp, zero cloud AI APIs"]
        GEMMA["⭐ gemma-cal E4B — fine-tuned Gemma 4<br/>ParetoOptimal/gemma-4-cal-gguf<br/>gemma-cal-e4b-Q4_K_M.gguf (~5 GB)<br/>+ mmproj-F16.gguf vision projector"]
        MODES["served either:<br/>· in-process llama-cpp-python (ZeroGPU lease)<br/>· remote llama-server via INFERENCE_BASE_URL<br/>(Space sidecar / Mac launchd / phone)"]
        MINICPM["🧭 MiniCPM planner — OpenBMB (sponsor)<br/>openbmb/MiniCPM4.1-8B-GGUF Q4 (~5 GB)<br/>≤4B option: openbmb/MiniCPM5-1B-GGUF (config switch)<br/>2nd llama-server :8081 — enabled via<br/>PLANNER_HF_REPO / PLANNER_FILE"]
        HERMES["(optional) Hermes-3-Llama-3.1-8B Q4_K_M<br/>HERMES_TOOLS=1 — tool-calling loop:<br/>calls remember() to write memory mid-run"]
        STUB["(no LLM) regex stub extractor<br/>USE_STUB_EXTRACTOR=1 — CI & free tier"]
        GEMMA --- MODES
    end

    subgraph DET["5 · Deterministic post-processing — no LLM"]
        CONF["freebusy.annotate_conflicts<br/>overlap / adjacent / tight<br/>+ propose_times free slots"]
        DEDUP["dedup.filter_new<br/>idempotency for autonomous runs"]
        MEMW["memory.observe_plan<br/>learns recurring contacts"]
    end

    subgraph OUT["6 · Outputs"]
        CARDS["Event cards + reply draft<br/>+ clarification question"]
        ICS["📥 .ics download<br/>(off-grid default)"]
        GCAL["📆 Google Calendar push<br/>(per-user OAuth web flow, opt-in)"]
        TRACE["Redacted trace export<br/>→ public HF dataset"]
    end

    UIIN -->|"run_orchestrator (step trace streams into the UI)"| SMOL
    SHORT --> AGENTEP
    MAC -->|"store-only"| INGEST
    MAC -->|"AGENT_MODE=1"| AGENTEP
    MCPC --> MCPT
    AGENTEP --> CORE
    INGEST --> ROLL --> CORE
    SMOL ==>|"planning loop, ≤6 steps"| MINICPM
    SMOL -->|"tool calls — the Space's OWN MCP<br/>endpoint (localhost SSE)"| MCPT
    SMOL -.->|"planner down / stub mode"| SCRIPT
    SCRIPT -->|"same tool sequence,<br/>deterministic"| MCPT
    MCPT -->|"extract_events → 1 LLM call"| CORE
    MCPT -.->|"make_ics / check_conflicts → 0 LLM calls"| DET

    GEN ==>|"default"| GEMMA
    GEN -.->|"opt-in autonomous brain"| HERMES
    GEN -.->|"tests / free demo"| STUB
    HERMES -->|"remember()"| MEMW

    LLMT --> DET --> OUT

Offline loop — eval-gated fine-tuning (produces the serving LLM)

flowchart LR
    SEEDS["Seed data — NO LLM<br/>139 hand-authored template examples<br/>(gen_new_seeds.py / make_dataset.py)"]
    SMC["SMCalFlow import — NO LLM<br/>deterministic LISP-program parse, ~2000 rows"]
    TRAIN["QLoRA fine-tune — Unsloth on Modal A100-80GB<br/>base: google/gemma-4-31B-it or gemma-4-E4B-it<br/>r=16, lr 5e-5, 2 epochs, responses-only loss"]
    GGUF["convert_hf_to_gguf + llama-quantize<br/>→ staging Q4_K_M GGUF"]
    EVAL["Eval — NO LLM judge, deterministic metrics<br/>60-example held-out set:<br/>schema validity · event F1 · start-exact recall"]
    GATE{"Gate<br/>validity ≥ 0.95<br/>F1 ≥ 0.81<br/>recall ≥ 0.773"}
    PROD["Promote → ParetoOptimal/gemma-4-cal-gguf<br/>(the model the Space serves)"]
    TRASH["Discard staging —<br/>production untouched"]

    SEEDS --> TRAIN
    SMC --> TRAIN
    TRAIN --> GGUF --> EVAL --> GATE
    GATE -->|pass| PROD
    GATE -->|fail| TRASH

See eval-roadmap.md and the eval-gated fine-tuning post-mortem for the gate's history and rationale; hermes.md for the optional tool-calling backend; build-small-submission.md for how the MiniCPM planner maps to the sponsor:openbmb track.

Which LLM each workflow calls

#	Workflow	Trigger	LLM call(s)	Where it runs
1	Agentic orchestration (Schedule flow + Agent tab)	User pastes thread / uploads screenshots, clicks Find the events / Run the agents	1× MiniCPM planning loop (MiniCPM4.1-8B, or MiniCPM5-1B ≤4B variant; ≤6 steps) driving the Space's own MCP tools, + 1× gemma-cal E4B per extract_events tool call (vision via mmproj); check_conflicts/make_ics are zero-LLM. Planner unconfigured or down → ScriptedPlanner runs the identical sequence, gemma-cal only	Two local llama-servers — gemma-cal on :8080, MiniCPM on :8081
2	API extraction (POST /agent)	iOS Shortcut, Android Tasker, or Mac collector in AGENT_MODE=1	1× gemma-cal E4B (same pipeline, same prompt)	Same
3	Autonomous ingest	Mac collector → /ingest; your outgoing message triggers a run over the chat's rolling thread	1× gemma-cal E4B per affected chat, then deterministic dedup + calendar delivery	Same
4	Memory-writing agent (optional)	HERMES_TOOLS=1 on the remote path	Hermes-3-Llama-3.1-8B in a tool loop (≤3 rounds): may call remember() then returns the ActionPlan	Remote llama-server (e.g. Mac launchd)
5	MCP tools for external agents	MCP client calls the Space	extract_events → 1× gemma-cal E4B; make_ics and check_conflicts → zero LLM calls	Same as #1
6	CI / free-tier demo	USE_STUB_EXTRACTOR=1	No LLM — regex heuristic	CPU anywhere
7	Training & eval (offline)	training/gated_retrain.py	No LLM at the inference-API level: data gen is template-based, eval is metric-based (no judge). The LLM here is the training target: QLoRA on google/gemma-4-31B-it / gemma-4-E4B-it	Modal A100/H100