CONTEXT.md

Quillwright

An on-device, human-supervised AI agent for tradespeople that turns a field capture (photos + voice note) into a finished, editable estimate or service report — the paperwork techs otherwise hand-write after a job.

Language

Capture: The raw multimodal input from a job — photos plus a voice note (and optional typed note) — normalized into one object the agent works from. Avoid: upload, input, submission

Observation: A single fact the perception model extracts from a photo — an item, part, damage, or piece of read text (e.g. a nameplate model number). Avoid: detection, finding, result

Document Capture: A second capture path: a document the tech or customer hands over — a spec sheet, supplier quote, or old written estimate — read by the Extraction Model Role (Nemotron Parse) into structured text + tables that feed the same Estimate pipeline. Distinct from a job-site photo (different input, different model); entered via its own control, never auto-classified. Avoid: scan, OCR (Document Capture is the agent-facing capability; OCR is one mechanism)

Proposed Line Item: A Line Item whose price came from a Document Capture, not the catalog — surfaced to the human as a proposal to confirm or edit before it enters the Estimate. The document is the source, but the price only becomes customer-facing once a human confirms it (an Agent Pause), preserving Facts-from-Tools. Avoid: draft line (a Line Item is already a draft; "Proposed" specifically means awaiting human confirmation of a document-read price)

Agent Brain: The orchestrator model that runs the plan → act → self-check loop and decides which Tools to call. There is exactly one. Avoid: orchestrator, controller, LLM

Tool: A callable capability the Agent Brain invokes. Tools are where facts (prices, math) come from; the Brain does routing and judgment, not arithmetic. The v1 set: perceive, search_past_jobs, lookup_price, compute, draft_line_item, flag_for_human, translate, update_profile. Avoid: function, plugin, skill

Facts-from-Tools: The correctness rule: any number that reaches the customer (price, quantity, markup, tax, total) must come from a Tool (lookup_price, compute) or user-confirmed data — never from the Agent Brain's free generation. This extends to Document Capture: a price read off a document by Nemotron Parse is a model output, so it is never used directly — it becomes a Proposed Line Item the human confirms (the document is the source; the human is the gate). Avoid: no-hallucination (too vague)

Line Item: One priced row of an Estimate — description, quantity, unit, rate, subtotal. Avoid: entry, row, charge

Estimate: The hero Deliverable — an itemized, priced quote for a job. Editable draft, not a system-of-record invoice. Avoid: quote, invoice, bill

Service Report: The secondary Deliverable — a written record of findings, work done, and captioned photos. (Stretch beyond the Estimate.) Avoid: inspection report (a regulated cousin we are deliberately not building)

Deliverable: The finished work product the agent produces — an Estimate or a Service Report — that the human edits and exports. Avoid: output, document, artifact

Mode: The execution tier the user selects, distinguished by model class and data path (both hosted on Modal). Private Stack resolves every Model Role to an open small model (≤32B, no third-party AI APIs — the contest's "no cloud APIs" ask). Best Stack resolves roles to larger hosted sponsor models for higher quality. The two modes differ ONLY in which model each role resolves to — same loop, tools, UI, and Deliverable. Avoid: Local/Connected (renamed — "Local" wrongly implied on-device inference; a hosted Gradio Space runs models server-side), offline/online, tier

Airplane-Mode Proof: A separately filmed demonstration of the Private Stack running on a real local machine (via llama.cpp) with the network actually off — the honesty anchor for "small models genuinely work offline," distinct from the Modal-hosted Space. Avoid: offline mode, local mode (those wrongly describe the hosted Space)

Model Role: A named slot in the pipeline (Perception, Audio, Agent Brain, Multilingual, Embedding, Visual) that resolves to a concrete model depending on Mode. Avoid: model (a Model Role is the slot; a model is what fills it)

Trace: The recorded sequence of the Agent Brain's steps (thought, Tool call, result, confidence) — shown live in the UI and exportable to the Hub. Avoid: log, history

Workspace: The main two-panel screen — Trace streaming on the left, Deliverable building on the right — where the human watches and supervises the run. Laptop-first, gracefully stacked on mobile. Avoid: dashboard, canvas, screen

Run: One execution of the Agent Brain over a single Capture, from plan to finished Deliverable. Avoid: session, job (a "job" is the real-world work; a Run is one agent execution over its Capture)

Agent Pause: The agent stopping mid-Run to ask the human for a low-confidence or missing-info decision, then resuming. Avoid: prompt, interrupt (reserve "interrupt" for the human-initiated kind)

Interrupt: A human-initiated change during a Run (edit a Line Item, correct an Observation, redirect). Applied to shared state and picked up by the agent at the start of its next step — no restart. Avoid: barge-in, cancel

Profile Memory: A small persistent record of one tech's defaults — trade, common Line Items, markup, report tone, language. Read at the start of a Run (resolves the trade, user-overridable) and updated after. Avoid: settings, preferences, config

Episodic Memory: An append-only store of past Runs (Capture summary + final Deliverable + key Line Items) the agent can recall to inform a new Run. Avoid: history, log, cache

Recall: The agent retrieving relevant past Runs from Episodic Memory via hybrid search — a keyword/structured pre-filter then a semantic re-rank — exposed as the search_past_jobs Tool. Avoid: lookup, query, retrieval (use "Recall" for this specific agent-facing capability)

Account: The single owner of all stored data — saved Estimates, their Refinement Threads, and Profile Memory. The product targets solo tradespeople, so an Account == the one Tech == the business; there is no organization layer and no multi-tech sharing. Multi-tenant-shaped (everything is keyed by account_id) but bound to one fixed demo Account (account_id = "demo") — no login, no auth. A real login that swaps the fixed key for a session lookup is a deliberate post-hackathon extension (ADR-0013). Avoid: User, organization, workspace (Workspace is the screen), Customer (the homeowner the Estimate is sent to — a different party)

Saved Estimate: An Estimate persisted to the Estimate Store under an Account so the Tech can reopen, edit, and re-export it across sessions — distinct from a Run (the lossy Episodic-Memory record that feeds Recall). Auto-saved when the forge finishes and explicitly Save-able mid-draft; updated in place on edit; deleted by Discard. No "finalized/locked" status (an Estimate is not an invoice). Avoid: invoice, quote (an Estimate stays an editable draft), Run (a Run is the agent's memory record, not a reopenable Deliverable)

Estimate Store: The per-Account store of Saved Estimates + their Refinement Threads — separate from Episodic Memory (which stays a pure, append-only Recall corpus). The two have different jobs: Episodic Memory serves the agent (gets smarter), the Estimate Store serves the user (file and reopen work). Avoid: database, history (Episodic Memory is the history-for-the-agent)

Refinement Thread: The persisted, ordered record of post-forge chat turns for one Saved Estimate (human message + the operation taken, e.g. "set labor to 2h → qty 2"). Stored sanitized — intents and operations only, never dollar figures — so resuming the conversation can never feed a stale number back to the model (Facts-from-Tools). A sibling of Trace, not part of it: Trace is the Agent Brain's forge steps; the Refinement Thread is the human-driven editing conversation that follows. Avoid: chat log, conversation history, Trace (Trace is the forge-step record)

Thread Compaction: Keeping a long Refinement Thread inside the small model's context window by folding the oldest turns into one mechanical "earlier in this estimate: …" line — done deterministically in code from the stored operations (the last K turns stay verbatim), never by asking a model to summarize. A second instance of Facts-from-Tools: even the conversation's own compression is code-owned, so the model never restates its own pricing history. Avoid: summarization (implies a model writes it — it does not), truncation (compaction folds, it doesn't drop facts)

Relationships

• A Capture is turned into Observations by the Perception Model Role.
• The Agent Brain consumes Observations + the voice transcript, calls Tools, and produces Line Items → a Deliverable.
• Every Model Role resolves to a concrete model based on the active Mode.
• The Agent Brain emits a Trace of its steps.
• A human supervises: confirms Observations, answers low-confidence prompts, edits the Deliverable before export.
• An Account owns its Saved Estimates, Profile Memory, and Episodic Memory; everything stored is keyed by account_id (one fixed demo Account in the hackathon).
• A Saved Estimate lives in the Estimate Store and has one Refinement Thread (its post-forge chat turns); the Thread is sanitized and Thread Compaction keeps it bounded.
• The Refinement Thread is replayed to the Agent Brain on resume for reference (pronoun) resolution; the model gets sanitized history for context but the current Line Items for numbers — never historical dollars (Facts-from-Tools).

Flagged ambiguities

• "voice handling" was unspecified — resolved: the Audio Model Role transcribes the voice note (Cohere Transcribe 2B in Private Stack; Nemotron 3 ASR / Nemotron Omni audio in Best Stack — see ADR-0009).
• "online/offline" was overloaded — resolved into the single term Mode (Private Stack vs Best Stack), an endpoint swap only. "Local Mode" was retired because a hosted Gradio Space runs models server-side (see ADR-0005); the true-offline claim lives in the Airplane-Mode Proof.
• "adapt prices" was ambiguous — resolved: deterministic learning from user-confirmed edits/prefs only; novel items are flagged, never LLM-guessed (Facts-from-Tools).
• "translate tool vs language toggle" overlapped — resolved: one underlying translate function, two entry points (human toggle + autonomous agent call).
• "Local/on-device" (the hero) vs hosting reality — resolved: real compute on Modal; "small/private" = open models + no third-party APIs; literal offline = the filmed Airplane-Mode Proof.
• "account" (previously listed as a term to avoid under Customer) — resolved: Account is now the data-owner term (the Tech/business), distinct from Customer (the homeowner an Estimate is sent to). For a solo-tradesperson product, Account == Tech == business; no organization layer.
• "remember the chat" was ambiguous (audit trail vs resumable) — resolved: the Refinement Thread is both persisted (reopen shows it) and resumable (replayed to the model on continue), but sanitized (no dollars) with numbers always taken live from current Line Items. Cross-session resume is bounded by deterministic Thread Compaction.
• "chat history vs Trace" overlapped — resolved: Trace = the Agent Brain's forge steps; the Refinement Thread = the human-driven post-forge editing conversation. Siblings, separately stored.

Scope note

The Irreducible Core (ADR-0007) is the must-ship, polished slice: Capture → supervised agent builds a correct Estimate live → one Agent Pause + one Interrupt → editable Estimate → PDF, Private Stack only. All other designed features (second Mode, memory, multilingual, fine-tune, service report, video, FLUX) are layered on after the core is flawless and are cut from the end under time pressure.

Implementation reality (as built)

• Frontend is a bespoke HTML/CSS/JS app served by gradio.Server (FastAPI), not Gradio components — the rule-compliant way to a smooth custom UI (the 🎨 Off-Brand path).
• Private Stack runs genuinely locally via Ollama (llama.cpp) on the dev machine — Perception = MiniCPM-V, Agent Brain = nemotron-3-nano:4b (NVIDIA Nemotron — NOT gpt-oss; the spec's gpt-oss mapping is superseded by ADR-0009) — gated by FF_REAL_MODELS=1; otherwise a deterministic/keyword stub. This realizes the "no cloud" claim directly on-device; Modal (ADR-0005) remains the option for hosted-Space compute.
• Model orchestra (ADR-0009): Perception = MiniCPM-V (protects the OpenBMB track; Nemotron Omni is a selectable Best-Stack alternative), Brain = Nemotron 3 Nano, Audio = Cohere Transcribe, Multilingual = Cohere Aya, Embedding = Llama-Nemotron-Embed-1B (un-tuned). The one shipped fine-tune is MiniCPM-V on CORD/SROIE (ADR-0006).
• The Agent Brain is LLM-driven tool-calling over a narrow surface (add_priced_item + finish); deterministic tools still own all numbers (Facts-from-Tools). Accuracy is tracked by an eval set (data/brain_evalset.json, scripts/run_brain_eval.py).