diff --git a/CONTEXT.md b/CONTEXT.md index 7fed2677512d49d4683323ea065f58d372d6e6c2..86d49577a4643e340f7d16359fcc15bc5e307058 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -12,6 +12,14 @@ _Avoid_: upload, input, submission A single fact the perception model extracts from a photo — an item, part, damage, or piece of read text (e.g. a nameplate model number). _Avoid_: detection, finding, result +**Document Capture**: +A second capture path: a document the tech or customer hands over — a spec sheet, supplier quote, or old written estimate — read by the Extraction Model Role (Nemotron Parse) into structured text + tables that feed the same Estimate pipeline. Distinct from a job-site photo (different input, different model); entered via its own control, never auto-classified. +_Avoid_: scan, OCR (Document Capture is the agent-facing capability; OCR is one mechanism) + +**Proposed Line Item**: +A Line Item whose price came from a Document Capture, not the catalog — surfaced to the human as a proposal to confirm or edit before it enters the Estimate. The document is the _source_, but the price only becomes customer-facing once a human confirms it (an Agent Pause), preserving Facts-from-Tools. +_Avoid_: draft line (a Line Item is already a draft; "Proposed" specifically means awaiting human confirmation of a document-read price) + **Agent Brain**: The orchestrator model that runs the plan → act → self-check loop and decides which Tools to call. There is exactly one. _Avoid_: orchestrator, controller, LLM @@ -21,7 +29,7 @@ A callable capability the Agent Brain invokes. Tools are where facts (prices, ma _Avoid_: function, plugin, skill **Facts-from-Tools**: -The correctness rule: any number that reaches the customer (price, quantity, markup, tax, total) must come from a Tool (`lookup_price`, `compute`) or user-confirmed data — never from the Agent Brain's free generation. +The correctness rule: any number that reaches the customer (price, quantity, markup, tax, total) must come from a Tool (`lookup_price`, `compute`) or user-confirmed data — never from the Agent Brain's free generation. This extends to **Document Capture**: a price read off a document by Nemotron Parse is a _model_ output, so it is never used directly — it becomes a Proposed Line Item the human confirms (the document is the source; the human is the gate). _Avoid_: no-hallucination (too vague) **Line Item**: @@ -84,6 +92,26 @@ _Avoid_: history, log, cache The agent retrieving relevant past Runs from Episodic Memory via hybrid search — a keyword/structured pre-filter then a semantic re-rank — exposed as the `search_past_jobs` Tool. _Avoid_: lookup, query, retrieval (use "Recall" for this specific agent-facing capability) +**Account**: +The single owner of all stored data — saved Estimates, their Refinement Threads, and Profile Memory. The product targets solo tradespeople, so an Account == the one Tech == the business; there is no organization layer and no multi-tech sharing. Multi-tenant-_shaped_ (everything is keyed by `account_id`) but bound to one fixed demo Account (`account_id = "demo"`) — no login, no auth. A real login that swaps the fixed key for a session lookup is a deliberate post-hackathon extension (ADR-0013). +_Avoid_: User, organization, workspace (Workspace is the screen), Customer (the homeowner the Estimate is _sent to_ — a different party) + +**Saved Estimate**: +An Estimate persisted to the Estimate Store under an Account so the Tech can reopen, edit, and re-export it across sessions — distinct from a Run (the lossy Episodic-Memory record that feeds Recall). Auto-saved when the forge finishes and explicitly Save-able mid-draft; updated in place on edit; deleted by Discard. No "finalized/locked" status (an Estimate is not an invoice). +_Avoid_: invoice, quote (an Estimate stays an editable draft), Run (a Run is the agent's memory record, not a reopenable Deliverable) + +**Estimate Store**: +The per-Account store of Saved Estimates + their Refinement Threads — separate from Episodic Memory (which stays a pure, append-only Recall corpus). The two have different jobs: Episodic Memory serves the _agent_ (gets smarter), the Estimate Store serves the _user_ (file and reopen work). +_Avoid_: database, history (Episodic Memory is the history-for-the-agent) + +**Refinement Thread**: +The persisted, ordered record of post-forge chat turns for one Saved Estimate (human message + the operation taken, e.g. "set labor to 2h → qty 2"). Stored _sanitized_ — intents and operations only, never dollar figures — so resuming the conversation can never feed a stale number back to the model (Facts-from-Tools). A sibling of Trace, not part of it: Trace is the Agent Brain's forge steps; the Refinement Thread is the human-driven editing conversation that follows. +_Avoid_: chat log, conversation history, Trace (Trace is the forge-step record) + +**Thread Compaction**: +Keeping a long Refinement Thread inside the small model's context window by folding the oldest turns into one mechanical "earlier in this estimate: …" line — done _deterministically in code_ from the stored operations (the last K turns stay verbatim), never by asking a model to summarize. A second instance of Facts-from-Tools: even the conversation's own compression is code-owned, so the model never restates its own pricing history. +_Avoid_: summarization (implies a model writes it — it does not), truncation (compaction folds, it doesn't drop facts) + ## Relationships - A **Capture** is turned into **Observations** by the Perception **Model Role**. @@ -91,6 +119,9 @@ _Avoid_: lookup, query, retrieval (use "Recall" for this specific agent-facing c - Every **Model Role** resolves to a concrete model based on the active **Mode**. - The **Agent Brain** emits a **Trace** of its steps. - A human supervises: confirms **Observations**, answers low-confidence prompts, edits the **Deliverable** before export. +- An **Account** owns its **Saved Estimates**, **Profile Memory**, and **Episodic Memory**; everything stored is keyed by `account_id` (one fixed demo Account in the hackathon). +- A **Saved Estimate** lives in the **Estimate Store** and has one **Refinement Thread** (its post-forge chat turns); the Thread is sanitized and **Thread Compaction** keeps it bounded. +- The **Refinement Thread** is replayed to the **Agent Brain** on resume for reference (pronoun) resolution; the model gets sanitized history for _context_ but the **current Line Items** for _numbers_ — never historical dollars (**Facts-from-Tools**). ## Flagged ambiguities @@ -99,6 +130,9 @@ _Avoid_: lookup, query, retrieval (use "Recall" for this specific agent-facing c - "adapt prices" was ambiguous — resolved: deterministic learning from user-confirmed edits/prefs only; novel items are flagged, never LLM-guessed (**Facts-from-Tools**). - "translate tool vs language toggle" overlapped — resolved: one underlying translate function, two entry points (human toggle + autonomous agent call). - "Local/on-device" (the hero) vs hosting reality — resolved: real compute on Modal; "small/private" = open models + no third-party APIs; literal offline = the filmed **Airplane-Mode Proof**. +- "account" (previously listed as a term to _avoid_ under **Customer**) — resolved: **Account** is now the data-owner term (the Tech/business), distinct from **Customer** (the homeowner an Estimate is sent to). For a solo-tradesperson product, Account == Tech == business; no organization layer. +- "remember the chat" was ambiguous (audit trail vs resumable) — resolved: the **Refinement Thread** is both _persisted_ (reopen shows it) and _resumable_ (replayed to the model on continue), but **sanitized** (no dollars) with numbers always taken live from current Line Items. Cross-session resume is bounded by deterministic **Thread Compaction**. +- "chat history vs Trace" overlapped — resolved: **Trace** = the Agent Brain's forge steps; the **Refinement Thread** = the human-driven post-forge editing conversation. Siblings, separately stored. ## Scope note diff --git a/Dockerfile b/Dockerfile index aa34baf44198d830e03327f7fecb35dc95675ba2..3e26f8cdc0aa1a26e76a7e577977c643ecd79499 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,4 @@ -# FieldForge HF Space — Docker SDK. +# Quillwright HF Space — Docker SDK. # Runs the bespoke gr.Server (FastAPI) app in STUB mode (no GPU, no Ollama): the contest # requires a Gradio Space underneath, and Docker SDK is sanctioned. Real models reach the # hosted Space via Modal later (ADR-0005); this image needs neither GPU nor model weights. diff --git a/README.md b/README.md index 8e426d599462c0ff66b996ac87352a474acaece9..c210b012b29d564ea7a6d276cf9a10e63c4c6b02 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,13 @@ tags: A human-supervised, small-model agent for tradespeople: snap a job photo + voice note → a team of **local** small models forges a finished, itemized **estimate**. No cloud, runs on your machine. Build Small Hackathon entry (Backyard AI track). +> **⏳ Cold start (please wait ~30–60s on first load).** This Space scales to zero when idle, +> so the **first** visit after a quiet period has to boot the container before the app +> responds — you may see Hugging Face's "Building / Starting" screen, then a moment where +> the page is warming up. **The app is not broken — it's waking up.** Once it's up it's +> instant (it runs in stub mode on CPU, so there's no model to load). Reload once if the +> first paint hangs; the UI shows a "waking up → ready" banner when it reconnects. + > **This hosted Space runs in stub mode** (CPU, no GPU): the agent flow, trace, editable estimate, and PDF all work, but the small models are stubbed. The real models (MiniCPM-V, Nemotron, Aya) run locally via Ollama — see the demo video / Airplane-Mode Proof for them in action. Live models reach the hosted Space via Modal (in progress). See `docs/superpowers/specs/` and `docs/adr/` for the design. @@ -28,6 +35,8 @@ See `docs/superpowers/specs/` and `docs/adr/` for the design. - **Brain** — Nemotron-3-Nano (NVIDIA) drives the tool-calling agent loop (which items, quantities, when done), locally via Ollama. Tuned to ~0.97 item-F1 on the eval set (`scripts/run_brain_eval.py`). - **Facts-from-Tools** — every price/total comes from the catalog + deterministic `compute`, never the LLM. Holds even for human edits. - **Human-in-the-loop** — the agent pauses to ask when a price is missing; you answer and it resumes. +- **Saved Estimates** — per-account persistence: auto-save on forge, reopen from "My Estimates", resume the (sanitized) refinement chat (ADR-0013). +- **Phone capture** — call a Twilio number (it forges a draft + texts the PDF) or scan a QR to capture a photo + voice note on your phone and forge live on the desktop. - **Frontend** — a bespoke web UI served by `gradio.Server` (FastAPI under the hood): streaming "Digital Apprentice" trace, editable estimate, PDF export. ## Run @@ -46,15 +55,142 @@ ollama pull nemotron-3-nano:4b FF_REAL_MODELS=1 python -m quillwright.server ``` +### Backend resolution — read this before "am I on Modal?" + +There is no single local/Modal switch. `FF_REAL_MODELS=1` is the master gate out of +stub mode; backends then resolve **per role** (see `quillwright/resolver.py`), and a +few roles can only go one way: + +| Role | Stub (default) | Local (Ollama) | Modal (hosted Best Stack) | +| ----------------------------------------- | -------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------- | +| **brain** (agent loop) | scripted | Nemotron-3-Nano **4B** | Nemotron-3-Nano **30B** — set `FF_BACKEND=modal` + `FF_MODAL_BRAIN_URL` | +| **perception** (vision) | scripted | MiniCPM-V | Nemotron **Omni** — additionally set `FF_MODAL_OMNI_URL` (else stays on Ollama) | +| **audio** (voice note) | scripted | Cohere Transcribe on-device (transformers) | Nemotron **Omni** (same deployment as perception) — `FF_MODAL_OMNI_URL` | +| **multilingual** | scripted | Aya | **Aya Expanse 8B** — additionally set `FF_MODAL_AYA_URL` (else stays on Ollama) | +| **embedding** | scripted | on-device (sentence-transformers) — same path for any non-stub backend | _same on-device path_ | +| **extraction** (Document Capture / Parse) | scripted | _no local path_ (>30GB RAM on Apple Silicon) | **Modal only**, always remote — needs `FF_MODAL_PARSE_URL` | + +How the switches compose: + +- **`FF_BACKEND=modal` by itself moves only the _brain_ to Modal** (its URL is then + required — missing `FF_MODAL_BRAIN_URL` fails loud, never silently downgrades). +- **Each other role opts in per-URL**: with `FF_BACKEND=modal` set, perception/audio + upgrade to the hosted Omni only when `FF_MODAL_OMNI_URL` is also set, multilingual to + Aya Expanse only when `FF_MODAL_AYA_URL` is set. Unset URLs keep the local/on-device + path working — deploying one GPU app never breaks the roles you didn't deploy. +- **Parse keys off its own `FF_MODAL_PARSE_URL`, independent of `FF_BACKEND`.** So you can + run a local Ollama brain _and_ hit Modal Parse at the same time — "am I on Modal?" is not + a single yes/no. This is intentional: Parse has no local serving path (ADR-0011), but it + means the offline ("Airplane-Mode") story only holds while every `FF_MODAL_*_URL` is unset. + +### Wiring the hosted Space to Modal (live real models) + +The Space can serve the real models on Modal GPUs — no tunnel involved (Modal apps are +public HTTPS endpoints; the Space just calls them). The apps are deployed and scale to zero; +wiring is purely Space **secrets** (Settings → Variables and secrets): + +| Secret | Value | Effect | +| -------------------- | ------------------------------- | ------------------------------------------------- | +| `FF_BACKEND` | `modal` | moves the **brain** to Modal (Nemotron 30B) | +| `FF_MODAL_BRAIN_URL` | `https://.modal.run` | **required** when `FF_BACKEND=modal` (fails loud) | +| `FF_MODAL_OMNI_URL` | `https://.modal.run` | upgrades **vision + audio** to hosted Omni | +| `FF_MODAL_AYA_URL` | `https://.modal.run` | upgrades **multilingual** to Aya Expanse | +| `FF_MODAL_PARSE_URL` | `https://.modal.run` | enables **Document Capture** (Parse; Modal-only) | + +Get the URLs from `modal app list` / each app's deployed endpoint. Each role opts in per-URL; +unset URLs keep that role on its non-Modal path. + +> **⏳ Model cold-start.** The first request to each Modal app pays a GPU cold-start — up to a +> **minute or two for the 30B brain**. The app is warming, not broken: the UI shows a "Waking +> the models" card on the first forge whenever real models are in play. **Warm the apps before +> a live demo** (hit each once). When `FF_BACKEND` is unset the Space runs in instant CPU stub +> mode (the default for the public submission link). + +> **💸 Cost.** A judge-clickable hosted GPU can spend over the whole judging window. The apps +> are set to **scale to zero** when idle; confirm that before leaving the Space Modal-wired, +> and don't leave apps you only warmed for the demo serving afterwards. + +> **🔒 Phone features are NOT served by the Space.** The Twilio call and QR phone-capture run +> on a **tunneled local machine** (`FF_PUBLIC_BASE_URL` = ngrok/cloudflared URL), because +> third-party send creds (Twilio) can't live on a public Space (ADR-0005). Modal serves the +> _models_; the phone _capture paths_ are the local-demo + video story. + +## Finalize & Send (S10) + +**Finalize & Send** delivers a finished estimate to the customer by **SMS** (Twilio MMS — +the PDF attached by URL) or **email** (SendGrid — the PDF attached inline). It has the same +honest, env-gated framing as the models: + +- **Real send is opt-in and local-only.** Set `FF_SEND_ENABLED=1` plus the provider creds + and the message is actually transmitted. The providers (`twilio`, `sendgrid`) are an + optional extra — `pip install -e ".[send]"` — deliberately **not** in the Space + `requirements.txt` (third-party API creds can't live on a public Space — ADR-0005). +- **The public Space drafts only.** With `FF_SEND_ENABLED` unset, `/api/send_estimate` + returns `{status: "drafted", transmitted: false}` and the UI shows a "Draft ready — + nothing was transmitted from this hosted demo" card. It never claims a send it didn't do. + +**Email has two backends — whichever is configured wins, Gmail first.** Gmail SMTP is the +simplest (stdlib `smtplib`, no extra dep, no sender-verification step — just a Google +[App Password](https://myaccount.google.com/apppasswords)); SendGrid is the fallback. + +Env vars for the real path: + +| Var | For | Purpose | +| ------------------------------------------ | --------- | ----------------------------------------------------- | +| `FF_SEND_ENABLED=1` | both | master gate out of draft-only mode | +| `TWILIO_ACCOUNT_SID` / `TWILIO_AUTH_TOKEN` | SMS | Twilio auth | +| `FF_SEND_FROM` | SMS | the Twilio sending number | +| `GMAIL_ADDRESS` / `GMAIL_APP_PASSWORD` | email (1) | Gmail SMTP — preferred; sends from your Gmail address | +| `SENDGRID_API_KEY` | email (2) | SendGrid auth (fallback if no Gmail creds) | +| `FF_SEND_FROM_EMAIL` | email (2) | the SendGrid verified sender address | + +Facts-from-Tools holds: send introduces no numbers — the PDF and the summary line both go +through `recalc_estimate`, the same server-authoritative totals the PDF/JSON already show. +SMS needs a public PDF URL (MMS attaches by URL); the server mints one at +`/api/estimate_pdf/{token}`. + +## Saved Estimates (ADR-0013) + +Estimates persist per **Account** (one fixed demo account, `account_id="demo"`, no auth). +A finished forge **auto-saves**; **Save** persists mid-draft; edits **update in place**; +**Discard** deletes. **My Estimates** lists them (newest first) and reopens a frozen +snapshot — existing lines never silently re-price; only newly-added lines hit the live +catalog. Each saved estimate carries a **Refinement Thread**: the post-forge chat turns, +stored **sanitized** (intents/operations, never dollar figures), so resuming the chat can +never feed a stale number back to the model — Facts-from-Tools holds on the resume path. +Long threads are kept in-context by **deterministic compaction** done in code, not by a +model summary. JSON-on-disk behind a swappable `EstimateStore`; durable locally, per-session +on the Space (`FF_ESTIMATE_STORE`). + +## Phone capture (two inbound paths) + +Both reuse the same pipeline + Facts-from-Tools; both are real on a tunneled local machine +(`FF_PUBLIC_BASE_URL` = the ngrok/cloudflared URL), honestly framed everywhere else. + +- **Call a number (S12)** — a Twilio Voice webhook. The caller describes the job; Quillwright + transcribes the recording (Audio role — same resolution as the mic button), forges an + estimate, saves it as a **draft** (a human approves later), reads the spoken total back on + the call, and texts the PDF via the same SMS path as Finalize & Send. Webhooks: + `POST /api/voice/incoming` (greeting + ``) → `POST /api/voice/recording`. +- **Scan a QR (phone capture)** — the desktop shows a QR (tunnel URL + a pairing code). The + phone opens a dedicated mobile capture page (`/m/`), takes a photo and/or a voice + note, and the **desktop forges it live on screen**. QR via the optional `[capture]` extra + (segno) — local/tunnel only, not on the Space. + ## Test ``` pytest -v -ruff check . && ruff format --check . # Python lint/format -npx prettier --check "quillwright/web/**/*" # web lint/format +ruff check . && ruff format --check . # Python lint/format +npx prettier --check --ignore-unknown "quillwright/web/**/*" # web lint/format +python scripts/check_deps_sync.py # pyproject ↔ requirements.txt # brain accuracy against the eval set (needs Ollama + FF_REAL_MODELS=1) FF_REAL_MODELS=1 PYTHONPATH=. python scripts/run_brain_eval.py ``` +CI (`.github/workflows/ci.yml`) runs the same gate — the dependency-sync check, +ruff lint/format, pytest, and the web prettier check. It is **manual-only** (to +conserve Actions minutes): trigger it from the Actions tab or `gh workflow run ci.yml`. + Models resolve per role via `quillwright/resolver.py` (stub ↔ Ollama). Pricing is clearly-labeled sample data. diff --git a/data/recall_evalset.json b/data/recall_evalset.json new file mode 100644 index 0000000000000000000000000000000000000000..7552b42685ed2a636dc6e959df710d3a7c7e8821 --- /dev/null +++ b/data/recall_evalset.json @@ -0,0 +1,133 @@ +{ + "_note": "Recall eval for ADR-0003. Corpus = seeded past HVAC runs; queries point at the one 'gold' run a tech would want recalled. Several queries use words that DON'T appear literally in the gold run (coolant->refrigerant, AC unit->condenser) — those are where keyword recall fails and semantic recall should win. SAMPLE data.", + "corpus": [ + { + "id": 1, + "transcript": "replaced the dual run capacitor on the rooftop unit", + "line_items": ["Dual run capacitor", "Labor"] + }, + { + "id": 2, + "transcript": "topped up the refrigerant after finding a slow leak", + "line_items": ["R-410A refrigerant", "Labor"] + }, + { + "id": 3, + "transcript": "swapped a burnt compressor contactor", + "line_items": ["Compressor contactor", "Labor"] + }, + { + "id": 4, + "transcript": "condenser fan motor seized, installed a new one", + "line_items": ["Condenser fan motor", "Labor"] + }, + { + "id": 5, + "transcript": "cleared a clogged condensate drain line", + "line_items": ["Drain cleaning", "Labor"] + }, + { + "id": 6, + "transcript": "annual maintenance, replaced the air filter and checked charge", + "line_items": ["Air filter", "Labor"] + }, + { + "id": 7, + "transcript": "thermostat reading wrong, installed a new programmable stat", + "line_items": ["Thermostat", "Labor"] + }, + { + "id": 8, + "transcript": "blower motor was noisy, replaced bearings and belt", + "line_items": ["Blower motor", "Labor"] + }, + { + "id": 9, + "transcript": "low on coolant, recovered and recharged the system", + "line_items": ["R-410A refrigerant", "Labor"] + }, + { + "id": 10, + "transcript": "hard start kit added to a struggling compressor", + "line_items": ["Hard start kit", "Labor"] + }, + { + "id": 11, + "transcript": "evaporator coil frozen over, thawed and fixed airflow", + "line_items": ["Coil cleaning", "Labor"] + }, + { + "id": 12, + "transcript": "replaced a failed start capacitor on the furnace", + "line_items": ["Start capacitor", "Labor"] + }, + { + "id": 13, + "transcript": "ductwork had a disconnected joint, resealed it", + "line_items": ["Duct sealing", "Labor"] + }, + { + "id": 14, + "transcript": "gas furnace igniter cracked, installed a new hot surface igniter", + "line_items": ["Hot surface igniter", "Labor"] + }, + { + "id": 15, + "transcript": "capacitor and contactor both replaced on an old AC unit", + "line_items": ["Dual run capacitor", "Compressor contactor", "Labor"] + }, + { + "id": 16, + "transcript": "heat pump reversing valve stuck, replaced the valve", + "line_items": ["Reversing valve", "Labor"] + }, + { + "id": 17, + "transcript": "outdoor disconnect was corroded, replaced the whip and breaker", + "line_items": ["Disconnect", "Labor"] + }, + { + "id": 18, + "transcript": "mini split indoor head leaking, cleared the drain and pump", + "line_items": ["Condensate pump", "Labor"] + } + ], + "queries": [ + { + "query": "coolant recharge", + "gold_id": 9, + "note": "synonym: 'coolant' never appears literally — gold run says 'refrigerant'. keyword should miss." + }, + { + "query": "AC unit fan not spinning", + "gold_id": 4, + "note": "synonym: 'fan motor' is 'condenser fan motor'; 'AC unit' not literal in gold." + }, + { + "query": "run capacitor replacement", + "gold_id": 1, + "note": "keyword-friendly: 'capacitor' is literal." + }, + { + "query": "contactor burnt out", + "gold_id": 3, + "note": "keyword-friendly: 'contactor' literal." + }, + { "query": "clogged drain", "gold_id": 5, "note": "keyword-friendly: 'drain' literal." }, + { + "query": "system low on charge", + "gold_id": 2, + "note": "semantic: 'charge'/'low' map to the refrigerant leak run, not literal." + }, + { + "query": "furnace won't ignite", + "gold_id": 14, + "note": "semantic: 'ignite' vs 'igniter'; furnace literal." + }, + { + "query": "frozen evaporator", + "gold_id": 11, + "note": "keyword-friendly: 'evaporator'/'frozen' literal." + } + ] +} diff --git a/data/sample_inventory.json b/data/sample_inventory.json new file mode 100644 index 0000000000000000000000000000000000000000..b658e760c74a3e4dccaf7703125cd73132cd6ec5 --- /dev/null +++ b/data/sample_inventory.json @@ -0,0 +1,69 @@ +{ + "_note": "SAMPLE inventory for the read-only Parts Catalog page (ADR-0010). Stock levels are illustrative; prices mirror data/sample_catalog.json. A part is 'low' when stock <= reorder_at. Live decrement on estimate finalize is an explicit stretch, NOT built.", + "parts": [ + { + "key": "capacitor", + "description": "Dual run capacitor", + "category": "HVAC", + "unit": "ea", + "stock": 42, + "reorder_at": 15 + }, + { + "key": "contactor", + "description": "Compressor contactor", + "category": "HVAC", + "unit": "ea", + "stock": 9, + "reorder_at": 12 + }, + { + "key": "refrigerant_r410a", + "description": "R-410A refrigerant", + "category": "HVAC", + "unit": "lb", + "stock": 6, + "reorder_at": 20 + }, + { + "key": "labor", + "description": "Labor", + "category": "Service", + "unit": "hr", + "stock": 0, + "reorder_at": 0 + }, + { + "key": "condenser_fan_motor", + "description": "Condenser fan motor", + "category": "HVAC", + "unit": "ea", + "stock": 5, + "reorder_at": 3 + }, + { + "key": "thermostat", + "description": "Programmable thermostat", + "category": "Controls", + "unit": "ea", + "stock": 18, + "reorder_at": 8 + }, + { + "key": "air_filter", + "description": "Pleated air filter", + "category": "HVAC", + "unit": "ea", + "stock": 120, + "reorder_at": 40 + }, + { + "key": "hard_start_kit", + "description": "Hard start kit", + "category": "HVAC", + "unit": "ea", + "stock": 4, + "reorder_at": 6 + } + ] +} diff --git a/package-lock.json b/package-lock.json index 04685e832e868f00a5c44883512f8c7b03d0ec51..2618597a278113c1160d7077c3e327ce5c42fe47 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,10 +1,10 @@ { - "name": "fieldforge-web", + "name": "quillwright-web", "lockfileVersion": 3, "requires": true, "packages": { "": { - "name": "fieldforge-web", + "name": "quillwright-web", "devDependencies": { "prettier": "^3.8.3" } diff --git a/package.json b/package.json index 2ef37d8c5871ccd60df4b17fa7912243134ff5c3..77212f43a26a16e4c99896ffa49ced71cc9e8712 100644 --- a/package.json +++ b/package.json @@ -1 +1,7 @@ -{"name":"fieldforge-web","private":true,"devDependencies":{"prettier":"^3.8.3"}} \ No newline at end of file +{ + "name": "quillwright-web", + "private": true, + "devDependencies": { + "prettier": "^3.8.3" + } +} diff --git a/pyproject.toml b/pyproject.toml index 43d9d3eb43af11abf0fe87ce096115bb4a63b932..6f2b8fadce80713f87b25a5e19e694cc47d512a0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,3 +1,7 @@ +[build-system] +requires = ["setuptools>=61"] +build-backend = "setuptools.build_meta" + [project] name = "quillwright" version = "0.1.0" @@ -9,10 +13,31 @@ dependencies = [ "pydantic>=2.7", "reportlab>=4.2", "uvicorn>=0.30", + "requests>=2.31", ] [project.optional-dependencies] dev = ["pytest>=8.0", "ruff>=0.15"] +# Semantic Recall embedder (ADR-0003): heavy (~2GB torch). Opt-in — NOT in the Space +# requirements.txt (stub mode has no torch). Install with: pip install -e ".[embed]" +embed = ["sentence-transformers>=3.0", "numpy>=1.26"] +# Spoken voice note via Cohere Transcribe (ADR-0009): on-device STT. Opt-in, heavy. +# Gated model — needs HF access + token. Install with: pip install -e ".[audio]" +audio = ["transformers>=5.4", "torch", "accelerate", "soundfile", "librosa", "sentencepiece", "protobuf"] +# Finalize & Send (S10): real SMS (Twilio) + email (SendGrid). Third-party APIs — +# deliberately NOT in the Space requirements.txt (the Space drafts only; creds can't +# live on a public Space — ADR-0005). Local/demo path only. Install: pip install -e ".[send]" +send = ["twilio>=9.0", "sendgrid>=6.11"] +# QR phone-capture (Tier 3): segno is a tiny pure-Python QR encoder, only needed on the +# local/tunnel demo path (the Space has no tunnel to pair against). Opt-in, lazy-imported, +# NOT in the Space requirements. Install with: pip install -e ".[capture]" +capture = ["segno>=1.6"] + +# Flat layout with sibling dirs (data/, node_modules/) — scope discovery to the +# quillwright package (and its subpackages) so setuptools auto-discovery doesn't +# error on "multiple top-level packages". +[tool.setuptools.packages.find] +include = ["quillwright*"] [tool.pytest.ini_options] testpaths = ["tests"] diff --git a/quillwright/agent.py b/quillwright/agent.py index 16d4c2eeb5e75b74f4ee6d97d270c77810d81070..0a91f2ae0602bb9287cd182f50edd267c9a4e51d 100644 --- a/quillwright/agent.py +++ b/quillwright/agent.py @@ -1,4 +1,5 @@ from typing import TypedDict, Optional +from langgraph.errors import GraphInterrupt from langgraph.graph import StateGraph, START, END from langgraph.types import interrupt from quillwright.models import Capture, Observation, LineItem, Estimate, TraceStep @@ -67,12 +68,30 @@ def build_agent(perception_model: Model, catalog: Catalog, checkpointer, brain_m obs_text = ", ".join(ob.text for ob in state["observations"]) priced_extra: list[LineItem] = [] while True: - items, brain_trace, pause = run_brain( - brain_model, - catalog, - observations_text=obs_text, - transcript=state["capture"].transcript, - ) + try: + items, brain_trace, pause = run_brain( + brain_model, + catalog, + observations_text=obs_text, + transcript=state["capture"].transcript, + ) + except GraphInterrupt: + raise # an Agent Pause is control flow, not a failure — let it propagate + except Exception as exc: # noqa: BLE001 — brain/model failure: degrade, don't crash + # The LLM brain is unavailable (e.g. Ollama 500). Fall back to the + # deterministic catalog pricer so the forge still produces an estimate + # instead of crashing the stream. Facts-from-Tools still holds. + print(f"[quillwright] brain failed ({exc}); falling back to deterministic pricing.") + out = deterministic_price(state) + out["trace"] = out["trace"] + [ + TraceStep( + action="price", + model="fallback", + detail="Brain unavailable — priced from the catalog directly.", + status="ok", + ) + ] + return out if pause is None: trace = state["trace"] + brain_trace return { diff --git a/quillwright/api/chat.py b/quillwright/api/chat.py new file mode 100644 index 0000000000000000000000000000000000000000..82bd38ec8cd4d9971aa8a3dd95de2ce9577321ea --- /dev/null +++ b/quillwright/api/chat.py @@ -0,0 +1,551 @@ +"""Conversational refinement of the current estimate — talk to the Digital +Apprentice about the draft ("add a contactor", "change labor to 3 hours", +"drop the refrigerant"). + +It is the SAME supervised agent, just conversational. Two paths share ONE set of +operations (add/remove/change), so Facts-from-Tools (ADR-0004) holds either way — +the catalog supplies every price; neither the keywords nor the model invent a number: + +- FF_REAL_MODELS=1 -> Nemotron (tool-calling) picks the operation + item, the + deterministic ops below execute it (catalog owns the price). +- otherwise -> a keyword intent parser picks the operation (so the hosted + stub Space + tests run with zero models). + +Totals are always recomputed server-authoritatively via recalc_estimate. +""" + +import os +import re + +from quillwright.api.recalc import recalc_estimate +from quillwright.catalog import Catalog +from quillwright.thread import append_turn, compact + +CATALOG = Catalog.from_file("data/sample_catalog.json") +REAL_MODELS = os.environ.get("FF_REAL_MODELS") == "1" + +_NUM_WORDS = { + "a": 1, + "an": 1, + "one": 1, + "two": 2, + "three": 3, + "four": 4, + "five": 5, + "six": 6, + "half": 0.5, + "both": 2, + "pair": 2, +} + + +def _to_qty(text: str) -> float | None: + """First number-like token in `text` -> a quantity, or None.""" + m = re.search(r"\d+(?:\.\d+)?", text) + if m: + return float(m.group()) + for word, val in _NUM_WORDS.items(): + if re.search(rf"\b{word}\b", text): + return val + return None + + +def _to_dollar_amount(text: str) -> float | None: + """A user-stated price -> float, or None. Matches an explicit `$30` OR a spoken + `30 dollars` / `30 bucks` (voice transcripts have no `$`). A bare number with no + money cue is NOT treated as a price (it stays a possible quantity).""" + m = re.search(r"\$\s*(\d+(?:\.\d+)?)", text) + if m: + return float(m.group(1)) + m = re.search(r"(\d+(?:\.\d+)?)\s*(?:dollars?|bucks)\b", text) + return float(m.group(1)) if m else None + + +def _finish( + rows: list[dict], + tax_rate: float, + reply: str, + needs_price: bool = False, + changed: str | None = None, + thread: list[dict] | None = None, + message: str = "", + op: str = "", + pending: dict | None = None, +) -> dict: + est = recalc_estimate(rows, job_title="Estimate", tax_rate=tax_rate) + # `changed` names the line whose rate just changed, so the UI can pulse that cell. + # The Refinement Thread (ADR-0013) records the human message + a dollar-free op, + # so resuming can never feed a stale number back to the model (Facts-from-Tools). + new_thread = append_turn(thread or [], message=message, op=op) if op else (thread or []) + # `pending` carries a rate change awaiting a scope answer ("this estimate"/"the catalog") + # to the next turn — the user's stated number, deferred one turn, not the model's. + return { + "estimate": est, + "reply": reply, + "needs_price": needs_price, + "changed": changed, + "thread": new_thread, + "pending": pending, + } + + +def _find_row(rows: list[dict], text: str) -> int | None: + """Index of the row whose description best matches words in `text`.""" + words = set(re.findall(r"[a-z0-9]+", text.lower())) + best_i, best_overlap = None, 0 + for i, r in enumerate(rows): + desc_words = set(re.findall(r"[a-z0-9]+", r["description"].lower())) + overlap = len(words & desc_words) + if overlap > best_overlap: + best_i, best_overlap = i, overlap + return best_i if best_overlap else None + + +# --- The shared operations. Each mutates `rows` in place and returns a reply dict +# fragment {"reply": str, "needs_price"?: bool}. Both paths call these, so the +# catalog-owns-the-price guarantee lives in exactly one place. --- + + +def _op_add(rows: list[dict], item: str, quantity: float | None) -> dict: + priced = CATALOG.lookup(item) + if not priced: + return { + "reply": ( + "I couldn't find that part in the catalog, so I won't guess a price. " + "Add it manually with a rate and I'll keep the math straight." + ), + "needs_price": True, + "op": "tried to add an unknown part", + } + qty = quantity if (quantity and quantity > 0) else 1 + rows.append( + { + "description": priced["description"], + "quantity": qty, + "unit": priced["unit"], + "rate": priced["rate"], # Facts-from-Tools: catalog price, never the model. + } + ) + return { + "reply": ( + f"Done — added {qty:g} × {priced['description']} at the catalog rate of " + f"${priced['rate']:.2f}. I've updated the total." + ), + "op": f"added {priced['description']}", + } + + +def _op_remove(rows: list[dict], item: str) -> dict: + i = _find_row(rows, item) + if i is None: + return { + "reply": "I couldn't tell which line to remove — which item did you mean?", + "op": "tried to remove an unmatched item", + } + removed = rows.pop(i) + return { + "reply": f"Got it — took {removed['description']} off the estimate and recalculated the total.", + "op": f"removed {removed['description']}", + } + + +def _op_change_qty(rows: list[dict], item: str, quantity: float | None) -> dict: + i = _find_row(rows, item) + if i is None or quantity is None: + return { + "reply": "Tell me which item and the new quantity — e.g. “change labor to 2 hours”.", + "op": "asked to change a quantity (unclear)", + } + rows[i]["quantity"] = quantity + return { + "reply": f"Sure — {rows[i]['description']} is now {quantity:g}. Total's updated.", + "op": f"set {rows[i]['description']} to {quantity:g}", + } + + +def _op_change_rate(rows: list[dict], item: str, rate: float | None, scope: str | None) -> dict: + """Set a line's rate to a USER-SUPPLIED number (Facts-from-Tools: the number is + the user's, never the model's). Asks estimate-vs-catalog before applying when the + scope is unspecified. + + scope="estimate" -> this row only (price_source="user"). + scope="catalog" -> also writes the in-session catalog so later adds use it. + """ + i = _find_row(rows, item) + if i is None or rate is None: + return { + "reply": "Tell me which line and the exact rate — e.g. “set the capacitor rate to $30”.", + "op": "asked to change a rate (unclear)", + } + if scope not in ("estimate", "catalog"): + # Numbers are user-confirmed, but we still ask WHERE it applies before changing. + # Stash the change as `pending` so the next turn's scope answer can apply it. + return { + "reply": ( + f"Should ${rate:.2f} for {rows[i]['description']} apply to just this " + "estimate, or update the catalog price for future jobs too? " + "Say “this estimate” or “the catalog”." + ), + "op": "asked where a rate applies", + "pending": {"item": rows[i]["description"], "rate": rate}, + } + rows[i]["rate"] = rate + rows[i]["price_source"] = "user" # a human-confirmed price, not catalog/computed + desc = rows[i]["description"] + if scope == "catalog": + # Update the in-session catalog so a later add of the same part picks it up. + existing = CATALOG.lookup(desc) or {} + CATALOG.add( + key=existing.get("key", desc.lower().replace(" ", "_")), + description=desc, + unit=rows[i].get("unit", existing.get("unit", "ea")), + rate=rate, + ) + where = "this estimate and the catalog" + else: + where = "this estimate" + return { + "reply": f"Done — {desc} is now ${rate:.2f} for {where}, and the total's updated.", + "changed": desc, + # Op is dollar-free by construction (Facts-from-Tools holds in the thread too). + "op": f"set the rate for {desc} ({where})", + } + + +def _answer_about_estimate(rows: list[dict], tax_rate: float) -> str: + """A spoken-friendly answer to 'what's the total / what's on it' — every number from + recalc (Facts-from-Tools), never free-generated.""" + est = recalc_estimate(rows, job_title="Estimate", tax_rate=tax_rate) + items = est["line_items"] + if not items: + return "The estimate is empty right now — tell me what to add." + n = len(items) + listed = ", ".join(f"{li['quantity']:g} {li['description'].lower()}" for li in items) + return ( + f"You've got {n} item{'s' if n != 1 else ''}: {listed}. " + f"The total comes to ${est['total']:.2f}." + ) + + +# --- LLM tool surface: the model only PICKS the operation + item (+ quantity); +# execution + pricing stay in the deterministic ops above. --- + +CHAT_TOOLS = [ + { + "type": "function", + "function": { + "name": "add_item", + "description": "Add a part or labor to the estimate. The catalog price is applied automatically.", + "parameters": { + "type": "object", + "properties": { + "item": {"type": "string", "description": "part or labor name"}, + "quantity": {"type": "number", "description": "units/hours (default 1)"}, + }, + "required": ["item"], + }, + }, + }, + { + "type": "function", + "function": { + "name": "remove_item", + "description": "Remove a line item from the estimate.", + "parameters": { + "type": "object", + "properties": {"item": {"type": "string", "description": "the item to remove"}}, + "required": ["item"], + }, + }, + }, + { + "type": "function", + "function": { + "name": "change_quantity", + "description": "Change the quantity (units or hours) of an existing line item.", + "parameters": { + "type": "object", + "properties": { + "item": {"type": "string", "description": "the item to adjust"}, + "quantity": {"type": "number", "description": "the new quantity"}, + }, + "required": ["item", "quantity"], + }, + }, + }, + { + "type": "function", + "function": { + "name": "change_rate", + "description": ( + "Set the rate (unit price) of an existing line item to a price the USER " + "EXPLICITLY STATED. Only call this when the user gave an exact number — " + "never choose or estimate a price yourself. `scope` says where it applies: " + "'estimate' (this estimate only) or 'catalog' (also the catalog, for future " + "jobs). If the user did not say which, OMIT scope — the assistant will ask." + ), + "parameters": { + "type": "object", + "properties": { + "item": {"type": "string", "description": "the item whose rate to set"}, + "rate": { + "type": "number", + "description": "the exact unit price the user stated (e.g. 30 for $30)", + }, + "scope": { + "type": "string", + "enum": ["estimate", "catalog"], + "description": "'estimate' = this estimate only; 'catalog' = also " + "the catalog. Omit if the user didn't specify.", + }, + }, + "required": ["item", "rate"], + }, + }, + }, +] + +_CHAT_SYSTEM = ( + "You are a field-service estimator's assistant. The user wants to refine the current " + "estimate. Decide the single edit they're asking for and call ONE tool: add_item, " + "remove_item, change_quantity, or change_rate. " + "ALWAYS prefer calling a tool over replying in plain text. The user's intent is often " + "phrased conversationally or buried mid-sentence — extract it and act. Map the request " + "to the closest tool even when the wording is indirect. Examples:\n" + "- 'it actually took more than one capacitor, could you make it 2?' → change_quantity(" + "item='capacitor', quantity=2)\n" + "- 'I ended up using two contactors' → change_quantity(item='contactor', quantity=2)\n" + "- 'throw in a refrigerant too' / 'I also needed refrigerant' → add_item(item='refrigerant')\n" + "- 'scrap the labor line' / 'we didn't end up doing labor' → remove_item(item='labor')\n" + "- 'bump labor to three hours' → change_quantity(item='labor', quantity=3)\n" + "Never invent prices. For add_item the catalog supplies the price. " + "change_rate is ONLY for a price the user STATED EXACTLY (e.g. “make it $30”): pass that " + "exact number as `rate`. If the user asks to change a price WITHOUT giving a number " + "(e.g. “make it cheaper”), do NOT call change_rate and do NOT pick a number — answer in " + "plain text asking what rate they want. When you do call change_rate, include `scope` " + "ONLY if the user said whether it applies to just this estimate or the catalog; if they " + "did not say, omit `scope` and the assistant will ask. " + "Only reply in plain text WITHOUT a tool when they're genuinely just asking a question " + "(e.g. 'what's the total?') or when you truly cannot map the request to any edit." +) + + +def _apply_model_call(name: str, args: dict, rows: list[dict]) -> dict: + item = str(args.get("item", "")).strip() + qty = args.get("quantity") + qty = float(qty) if isinstance(qty, (int, float)) else _to_qty(item) + if name == "add_item": + return _op_add(rows, item, qty) + if name == "remove_item": + return _op_remove(rows, item) + if name == "change_quantity": + return _op_change_qty(rows, item, qty) + if name == "change_rate": + rate = args.get("rate") + rate = float(rate) if isinstance(rate, (int, float)) else None + scope = args.get("scope") + return _op_change_rate(rows, item, rate, scope) + return {"reply": "I'm not sure how to do that — try add, remove, or change a quantity or rate."} + + +def _model_chat(message: str, rows: list[dict], tax_rate: float, model, thread: list[dict]) -> dict: + """Let the tool-calling model pick the edit; execute it through the shared ops. + + The compacted, sanitized thread (ops only, no dollars — ADR-0013) is replayed for + reference resolution ("make *it* 2 hours"); numbers always come from the current rows. + """ + rows_summary = ( + ", ".join(f"{r['description']} (qty {r['quantity']:g})" for r in rows) or "(empty)" + ) + history = compact(thread) + user = ( + f"Earlier edits:\n{history}\n\n" if history else "" + ) + f"Current estimate: {rows_summary}\nRequest: {message}" + messages = [ + {"role": "system", "content": _CHAT_SYSTEM}, + {"role": "user", "content": user}, + ] + msg = model.chat(messages, CHAT_TOOLS) + tool_calls = msg.get("tool_calls") or [] + if not tool_calls: + # No edit — the model answered a question. Estimate stays untouched. If it's a + # total/contents question, answer it deterministically (the number must come from + # recalc, never the model — Facts-from-Tools), else relay the model's plain text. + text = (msg.get("content") or "").strip() + if re.search( + r"\b(total|how much|what'?s on|what is on|whats on|breakdown)\b", message.lower() + ): + text = _answer_about_estimate(rows, tax_rate) + return _finish( + rows, + tax_rate, + text or "Let me know what you'd like to change.", + thread=thread, + message=message, + op="asked a question", + ) + + fn = tool_calls[0].get("function", {}) + result = _apply_model_call(fn.get("name", ""), fn.get("arguments", {}) or {}, rows) + return _finish( + rows, + tax_rate, + result["reply"], + needs_price=result.get("needs_price", False), + changed=result.get("changed"), + thread=thread, + message=message, + op=result.get("op", ""), + pending=result.get("pending"), + ) + + +def _keyword_chat(message: str, rows: list[dict], tax_rate: float, thread: list[dict]) -> dict: + """Zero-model fallback: a keyword intent parser drives the same shared ops.""" + msg = message.strip().lower() + if not msg: + return _finish( + rows, + tax_rate, + "Tell me what to change — add a part, drop one, or adjust a quantity.", + thread=thread, + ) + + # A read-only question about the estimate ("what's the total", "what's on it", + # "how much is it") — answer it instead of falling through to generic help. Checked + # before the edit verbs, but only when no edit verb is present so "add ..." still adds. + is_question = re.search(r"\b(total|how much|what'?s on|what is on|whats on|breakdown)\b", msg) + has_edit_verb = re.search(r"\b(add|remove|delete|drop|set|change|make|update|include)\b", msg) + if is_question and not has_edit_verb: + return _finish( + rows, + tax_rate, + _answer_about_estimate(rows, tax_rate), + thread=thread, + message=message, + op="asked about the estimate", + ) + + if re.search(r"\b(remove|delete|drop|take off|get rid of)\b", msg): + result = _op_remove(rows, msg) + return _finish( + rows, tax_rate, result["reply"], thread=thread, message=message, op=result.get("op", "") + ) + + # An explicit dollar amount ("set the capacitor rate to $30") is a user-confirmed + # rate change. Checked BEFORE the quantity branch so the "$30" isn't read as a qty. + # The keyword path can't hold a follow-up turn, so it takes the conservative + # estimate-only scope (the model path is the one that asks catalog-vs-estimate). + # _to_dollar_amount only returns a value when a money cue is present ($, "dollars", + # "bucks"), so its non-None result is itself the signal this is a rate, not a quantity. + rate_amount = _to_dollar_amount(msg) + if rate_amount is not None: + i = _find_row(rows, msg) + if i is not None: + result = _op_change_rate(rows, rows[i]["description"], rate_amount, scope="estimate") + return _finish( + rows, + tax_rate, + result["reply"], + changed=result.get("changed"), + thread=thread, + message=message, + op=result.get("op", ""), + ) + + if re.search(r"\b(change|set|make|update)\b", msg) or re.search( + r"\bto\b.*\b(hour|hr|unit|lb|pound)", msg + ): + i = _find_row(rows, msg) + qty = _to_qty(msg) + if i is not None and qty is not None: + result = _op_change_qty(rows, rows[i]["description"], qty) + return _finish( + rows, + tax_rate, + result["reply"], + thread=thread, + message=message, + op=result.get("op", ""), + ) + + if re.search(r"\b(add|include|put in|need|another|more)\b", msg): + result = _op_add(rows, msg, _to_qty(msg)) + return _finish( + rows, + tax_rate, + result["reply"], + needs_price=result.get("needs_price", False), + thread=thread, + message=message, + op=result.get("op", ""), + ) + + return _finish( + rows, + tax_rate, + "I can add a part, remove one, or change a quantity — e.g. “add a contactor” or " + "“change labor to 2 hours”. What would you like to adjust?", + thread=thread, + ) + + +def _resolve_brain(): + """Real tool-calling model when enabled (local Ollama or hosted Modal); else None.""" + if REAL_MODELS or os.environ.get("FF_BACKEND") == "modal": + from quillwright.resolver import brain_resolver + + return brain_resolver().for_role("brain") + return None + + +def _scope_answer(message: str) -> str | None: + """Map a scope reply to 'estimate'/'catalog', or None if it isn't one.""" + m = message.strip().lower() + if re.search(r"\bcatalog\b|\bboth\b|future job", m): + return "catalog" + if re.search(r"\b(this|just this|estimate only|only this|here|this one)\b", m): + return "estimate" + return None + + +def chat_about_estimate( + message: str, rows: list[dict], tax_rate: float = 0.13, model=None, thread=None, pending=None +) -> dict: + """Apply a conversational edit to the estimate. Returns + {estimate, reply, needs_price, changed, thread, pending}. + + `model` is injectable for tests; in production it's resolved from FF_REAL_MODELS. + `thread` is the Refinement Thread (ADR-0013): sanitized, dollar-free history. + `pending` carries a rate change awaiting a scope answer from the previous turn — if it + is set and this message answers "this estimate"/"the catalog", apply it directly (no + model), so the two-turn rate change doesn't lose context. + """ + rows = [dict(r) for r in rows] # don't mutate the caller's list + thread = list(thread or []) + + # Resolve a pending rate change first: "the catalog" / "this estimate" applies the + # number the user stated last turn (Facts-from-Tools — it's the user's, just deferred). + if pending and pending.get("item") and pending.get("rate") is not None: + scope = _scope_answer(message) + if scope is not None: + result = _op_change_rate(rows, pending["item"], float(pending["rate"]), scope=scope) + return _finish( + rows, + tax_rate, + result["reply"], + changed=result.get("changed"), + thread=thread, + message=message, + op=result.get("op", ""), + ) + # Not a scope answer — fall through to normal handling, dropping the pending change. + + brain = model if model is not None else _resolve_brain() + if brain is not None: + try: + return _model_chat(message, rows, tax_rate, brain, thread) + except Exception as exc: # noqa: BLE001 — model down (e.g. Ollama 500): degrade + # Fall back to the deterministic keyword path so a chat turn never 500s the UI. + print(f"[quillwright] chat brain failed ({exc}); using keyword fallback.") + return _keyword_chat(message, rows, tax_rate, thread) diff --git a/quillwright/api/document.py b/quillwright/api/document.py new file mode 100644 index 0000000000000000000000000000000000000000..f318dee5b088b10f048784043dd1ef03df8efc52 --- /dev/null +++ b/quillwright/api/document.py @@ -0,0 +1,69 @@ +"""Document Capture: a handed-over document -> Observations + Proposed Line Items (ADR-0011). + +Thin adapter over the Extraction role (Nemotron Parse on Modal): file path in, JSON +the frontend confirm card renders out. The model is injectable for tests. In +production it resolves to ParseModel when FF_MODAL_PARSE_URL is set; otherwise a +deterministic demo parse runs the REAL blocks_to_pipeline logic over a canned +supplier quote, so the stub Space demos the flow with zero models (the same +honest-scaffolding pattern as _stub_perception in api/estimate.py). +""" + +import os + +from quillwright.backends.parse import blocks_to_pipeline + +# The canned supplier quote the demo "reads" when no Modal Parse endpoint is +# configured. Mirrors the test fixture in test_parse_backend.py. +_DEMO_BLOCKS = [ + {"class": "Title", "bbox": [], "text": "ACME HVAC Supply — Quote #1042"}, + { + "class": "Table", + "bbox": [], + "text": ( + "| Item | Qty | Unit Price |\n" + "| --- | --- | --- |\n" + "| Dual run capacitor | 2 | $42.50 |\n" + "| Compressor contactor | 1 | $28.00 |\n" + "| R-410A refrigerant | 4 | $30.00 |\n" + ), + }, + {"class": "Text", "bbox": [], "text": "Net 30 terms. Prices valid 30 days."}, +] + + +def _resolve_extraction(): + """Real Nemotron Parse when its Modal URL is configured; else None (demo parse).""" + if os.environ.get("FF_MODAL_PARSE_URL"): + from quillwright.resolver import ModelResolver + + return ModelResolver(mode="best", backend="modal").for_role("extraction") + return None + + +def parse_document_capture(path: str, model=None) -> dict: + """Parse the document at `path`; return {model, observations, proposed_items}. + + Every price stays *proposed* — the human confirms it in the UI before it becomes + a LineItem with price_source="document" (Facts-from-Tools, ADR-0004/0011). + """ + parser = model if model is not None else _resolve_extraction() + if parser is None: + observations, proposed = blocks_to_pipeline(_DEMO_BLOCKS) + name = "parse-stub (demo quote)" + else: + observations, proposed = parser.parse_document(path) + name = parser.name + return { + "model": name, + "observations": [{"kind": o.kind, "text": o.text} for o in observations], + "proposed_items": [ + { + "description": p.description, + "quantity": p.quantity, + "unit": p.unit, + "rate": p.rate, + "source_text": p.source_text, + } + for p in proposed + ], + } diff --git a/quillwright/api/estimate.py b/quillwright/api/estimate.py index 1bc7e88a9cfc309aed05cc46f263c09c3ae68b4c..5ecc0cdc6607c5ef9653498ba419b75fcdb356db 100644 --- a/quillwright/api/estimate.py +++ b/quillwright/api/estimate.py @@ -12,6 +12,7 @@ from langgraph.types import Command from quillwright.agent import build_agent from quillwright.catalog import Catalog +from quillwright.estimate_store import EstimateStore from quillwright.memory import Memory from quillwright.models import Capture from quillwright.resolver import ModelResolver, StubModel @@ -36,6 +37,32 @@ def reset_memory() -> None: _MEMORY = None +# Per-Account Estimate Store (ADR-0013) — separate from Episodic Memory above. +# A singleton mirroring _memory(); re-reads its env path after reset_estimate_store(). +_ESTIMATE_STORE: EstimateStore | None = None + + +def estimate_store() -> EstimateStore: + global _ESTIMATE_STORE + if _ESTIMATE_STORE is None: + _ESTIMATE_STORE = EstimateStore() + return _ESTIMATE_STORE + + +def reset_estimate_store() -> None: + """Drop the in-process store (re-reads env path next use). For tests.""" + global _ESTIMATE_STORE + _ESTIMATE_STORE = None + + +def save_estimate_record(rows, job_title, tax_rate, thread, id=None) -> dict: + """Recalc to authoritative numbers (Facts-from-Tools), then persist the snapshot.""" + from quillwright.api.recalc import recalc_estimate + + est = recalc_estimate(rows, job_title=job_title, tax_rate=tax_rate) + return estimate_store().save(estimate=est, thread=thread, id=id) + + # FF_REAL_MODELS=1 uses real local models via Ollama; otherwise the demo stub. REAL_MODELS = os.environ.get("FF_REAL_MODELS") == "1" @@ -60,16 +87,29 @@ def _stub_perception(transcript: str) -> StubModel: def _perception(transcript: str, has_real_image: bool): - """Real MiniCPM-V via Ollama when enabled AND a real photo exists; else the stub.""" - if REAL_MODELS and has_real_image: - return ModelResolver(mode="private", backend="ollama").for_role("perception") + """The Perception role for a real photo: hosted Omni (Best Stack) when its + Modal URL is configured, MiniCPM-V via Ollama under FF_REAL_MODELS, else stub.""" + if has_real_image: + from quillwright.resolver import modal_resolver_if_configured + + modal = modal_resolver_if_configured("perception") + if modal is not None: + return modal.for_role("perception") + if REAL_MODELS: + return ModelResolver(mode="private", backend="ollama").for_role("perception") return _stub_perception(transcript) def _brain(): - """Real Ollama tool-calling brain when FF_REAL_MODELS=1; else None (deterministic path).""" - if REAL_MODELS: - return ModelResolver(mode="private", backend="ollama").for_role("brain") + """Real tool-calling brain when enabled; else None (deterministic path). + + Local Ollama (FF_REAL_MODELS=1) or hosted Modal Best-Stack (FF_BACKEND=modal); + brain_resolver() picks based on env. + """ + if REAL_MODELS or os.environ.get("FF_BACKEND") == "modal": + from quillwright.resolver import brain_resolver + + return brain_resolver().for_role("brain") return None @@ -116,10 +156,22 @@ def forge_estimate( {"configurable": {"thread_id": "ui"}}, ) est = out.get("estimate") - return { + payload = { "trace": _trace_payload(out["trace"]), "estimate": _estimate_payload(est) if est is not None else None, } + if payload["estimate"] is not None: + _autosave(payload["estimate"]) + return payload + + +def _autosave(estimate: dict) -> None: + """Auto-save a finished estimate so 'My Estimates' populates (ADR-0013 lifecycle). + Best-effort: persistence must never fail a forge.""" + try: + estimate_store().save(estimate=estimate, thread=[]) + except Exception: # noqa: BLE001 — persistence is best-effort + pass # Active runs by thread_id, so a paused run can be resumed with the same agent + checkpointer. @@ -172,11 +224,15 @@ def _drive(agent, payload, thread_id: str): _memory().record_run( run.get("transcript", ""), [li.description for li in estimate.line_items], + total=estimate.total, ) + est_payload = _estimate_payload(estimate) if estimate is not None else None + if est_payload is not None: + _autosave(est_payload) # ADR-0013: finished forge auto-saves to the store yield { "type": "estimate", - "estimate": _estimate_payload(estimate) if estimate is not None else None, + "estimate": est_payload, } _RUNS.pop(thread_id, None) diff --git a/quillwright/api/export.py b/quillwright/api/export.py new file mode 100644 index 0000000000000000000000000000000000000000..590e72a246523d49527797228076c96ba1513eea --- /dev/null +++ b/quillwright/api/export.py @@ -0,0 +1,18 @@ +"""Export an estimate as machine-readable JSON — the "no lock-in" counterpart to +the PDF. Totals go through the same server-authoritative recalc (Facts-from-Tools), +so the exported numbers match exactly what the customer-facing PDF shows. +""" + +from quillwright.api.recalc import recalc_estimate + +DISCLAIMER = "AI-generated draft — review before sending. Sample pricing." + + +def estimate_to_json_payload(rows: list[dict], job_title: str, tax_rate: float) -> dict: + """Return a self-describing JSON payload for the (possibly edited) estimate.""" + est = recalc_estimate(rows, job_title=job_title, tax_rate=tax_rate) + return { + "format": "quillwright.estimate.v1", + "disclaimer": DISCLAIMER, + **est, + } diff --git a/quillwright/api/pages.py b/quillwright/api/pages.py new file mode 100644 index 0000000000000000000000000000000000000000..a5719640213c86bb9c5a2519fd984d2487aad105 --- /dev/null +++ b/quillwright/api/pages.py @@ -0,0 +1,94 @@ +"""Read-models for the secondary pages (ADR-0010): Dashboard, Active Jobs, Inventory. + +Dashboard + Active Jobs aggregate the SAME on-device memory store the agent already +writes to — so every number on those pages is real, derived from past Runs (no +invented revenue/technicians/CRM fields). Inventory is a read-only view over a +seeded JSON joined with the real catalog price; low-stock flags are computed, not +hardcoded. Live decrement is an explicit stretch and is NOT implemented here. +""" + +import json + +from quillwright.catalog import Catalog +from quillwright.memory import Memory + +MEMORY_PATH = "/tmp/quillwright_memory.json" +INVENTORY_PATH = "data/sample_inventory.json" +CATALOG_PATH = "data/sample_catalog.json" + + +def _memory() -> Memory: + # A fresh handle each call so the page always reflects the latest recorded runs. + return Memory(MEMORY_PATH) + + +def _items_summary(line_items: list[str], limit: int = 3) -> str: + shown = line_items[:limit] + extra = len(line_items) - len(shown) + text = ", ".join(shown) + return f"{text} +{extra} more" if extra > 0 else text + + +def dashboard_data() -> dict: + """KPI cards + recent activity, aggregated over real past Runs.""" + mem = _memory() + prof = mem.profile() + recent = mem.recent(limit=6) + return { + "job_count": prof["job_count"], + "revenue_total": prof["revenue_total"], + "top_items": prof["common_items"][:5], + "recent": [ + { + "id": r["id"], + "transcript": r["transcript"], + "items": _items_summary(r["line_items"]), + "total": r["total"], + } + for r in recent + ], + } + + +def jobs_data() -> dict: + """Every past Run as a table row, newest first.""" + mem = _memory() + return { + "jobs": [ + { + "id": r["id"], + "transcript": r["transcript"], + "items": _items_summary(r["line_items"], limit=4), + "total": r["total"], + } + for r in mem.recent() + ] + } + + +def inventory_data() -> dict: + """Read-only stock view: seeded levels joined with the real catalog price.""" + with open(INVENTORY_PATH) as f: + seeded = json.load(f)["parts"] + catalog = Catalog.from_file(CATALOG_PATH) + parts = [] + for p in seeded: + cat = catalog.lookup(p["key"]) + rate = cat["rate"] if cat else None + low = p["reorder_at"] > 0 and p["stock"] <= p["reorder_at"] + parts.append( + { + "description": p["description"], + "category": p["category"], + "unit": p["unit"], + "stock": p["stock"], + "reorder_at": p["reorder_at"], + "rate": rate, + "low": low, + } + ) + return { + "parts": parts, + "total_skus": len(parts), + "low_stock_count": sum(1 for p in parts if p["low"]), + } diff --git a/quillwright/api/pdf_links.py b/quillwright/api/pdf_links.py new file mode 100644 index 0000000000000000000000000000000000000000..2b5050e0bf090457fba88a9adbaa781663bcf669 --- /dev/null +++ b/quillwright/api/pdf_links.py @@ -0,0 +1,40 @@ +"""Tokenized PDF link registry (S10). + +Twilio MMS attaches media by **URL**, not by file upload — so to text a customer +their estimate PDF we need a public URL Twilio can GET. This registry holds rendered +PDF bytes in process, keyed by a content-hash token, and the server exposes them at +``/api/estimate_pdf/{token}``. + +The token is a content hash (deterministic, offline-friendly — no uuid/wall-clock, +matching the rest of the codebase's id scheme), URL-safe, and unguessable enough for +a demo (the bytes are an AI-draft estimate, not a secret). In-process only: it lives +for the life of the server, which is all the MMS fetch needs. A durable/expiring +store is a post-hackathon swap behind this same tiny interface. +""" + +import hashlib + +# token -> pdf bytes. Process-local; fine for the local demo path. +_PDFS: dict[str, bytes] = {} + + +def register_pdf(pdf_bytes: bytes) -> str: + """Store the bytes under a content-hash token and return the token.""" + token = hashlib.sha256(pdf_bytes).hexdigest()[:16] + _PDFS[token] = pdf_bytes + return token + + +def get_pdf(token: str) -> bytes | None: + """Return the bytes for a token, or None if unknown.""" + return _PDFS.get(token) + + +def public_pdf_url(token: str, base_url: str) -> str: + """Build the public URL Twilio fetches the PDF from.""" + return f"{base_url.rstrip('/')}/api/estimate_pdf/{token}" + + +def reset() -> None: + """Drop all registered PDFs (tests).""" + _PDFS.clear() diff --git a/quillwright/api/qr.py b/quillwright/api/qr.py new file mode 100644 index 0000000000000000000000000000000000000000..9a4dd38c7611006f1fd2a7a96de251233bd775ee --- /dev/null +++ b/quillwright/api/qr.py @@ -0,0 +1,24 @@ +"""QR code as inline SVG for the phone-capture pairing (Tier 3). + +``segno`` is a tiny, pure-Python, zero-dependency QR encoder — but it's only needed on +the local/tunnel demo path (the hosted Space has no tunnel to pair against), so it lives +in the optional ``[capture]`` extra and is lazy-imported, the same pattern as the +``[send]``/``[embed]``/``[audio]`` extras (kept OUT of the Space requirements, ADR-0005). + +If segno isn't installed we degrade honestly: an empty string, and the UI shows the +pairing link as scannable text instead of claiming a QR it can't render. +""" + +import io + + +def qr_svg(data: str, scale: int = 5) -> str: + """Return an inline SVG QR for ``data``, or '' if the encoder isn't installed.""" + try: + import segno # noqa: PLC0415 — lazy: optional [capture] dep, not in the Space + except ImportError: + return "" + buf = io.BytesIO() + # xmldecl=False so the SVG embeds cleanly inline in the desktop HTML (no prolog). + segno.make(data, error="m").save(buf, kind="svg", scale=scale, border=2, xmldecl=False) + return buf.getvalue().decode() diff --git a/quillwright/api/recalc.py b/quillwright/api/recalc.py index 946e86f6f3359184c427757594f9bf36e7445f34..930bfd9314a9cdbbb2e088ca3d9cd5e12161c338 100644 --- a/quillwright/api/recalc.py +++ b/quillwright/api/recalc.py @@ -5,6 +5,11 @@ human edits; the UI never computes its own authoritative totals. from quillwright.models import Estimate, LineItem +# Provenances a client row may carry through a recalc. "document" marks a confirmed +# Document Capture price (ADR-0011); anything unrecognized falls back to "user" so +# arbitrary client strings never enter the model. +_CLIENT_SOURCES = {"document", "user", "catalog", "computed"} + def _num(value) -> float: try: @@ -13,6 +18,10 @@ def _num(value) -> float: return 0.0 +def _source(value) -> str: + return value if value in _CLIENT_SOURCES else "user" + + def recalc_estimate(rows: list[dict], job_title: str, tax_rate: float) -> dict: items = [ LineItem( @@ -20,7 +29,7 @@ def recalc_estimate(rows: list[dict], job_title: str, tax_rate: float) -> dict: quantity=_num(r.get("quantity", 1)), unit=str(r.get("unit", "ea")), rate=_num(r.get("rate", 0)), - price_source="user", + price_source=_source(r.get("price_source")), ) for r in rows ] @@ -34,6 +43,7 @@ def recalc_estimate(rows: list[dict], job_title: str, tax_rate: float) -> dict: "unit": li.unit, "rate": li.rate, "subtotal": li.subtotal, + "price_source": li.price_source, } for li in est.line_items ], diff --git a/quillwright/api/send.py b/quillwright/api/send.py new file mode 100644 index 0000000000000000000000000000000000000000..e27989d24a6d97deb99378c47e5247ea54d440d3 --- /dev/null +++ b/quillwright/api/send.py @@ -0,0 +1,285 @@ +"""Finalize & Send (S10) — deliver a finished estimate to the customer by SMS or +email, with the same honest framing as the rest of Quillwright (ADR-0005). + +Three-state resolution, mirroring ``FF_REAL_MODELS``: + + - **real** — ``FF_SEND_ENABLED=1`` AND provider creds present: the message is + actually transmitted (Twilio MMS / SendGrid email). This is the local/demo + path; the providers are heavy, optional, third-party deps that are NOT in the + Space requirements (lazy-imported, like the ``[embed]``/``[audio]`` extras). + - **mock** — default / public Space: the estimate is *drafted* and a confirmation + is returned, but nothing is transmitted (``transmitted=False``, + ``status="drafted"``). Twilio creds can't live on a public Space, so the Space + never sends — and it says so honestly rather than claiming a send it didn't do. + +Facts-from-Tools (ADR-0004) holds: send introduces no numbers. The PDF and the +summary line both come from ``recalc_estimate`` — the same server-authoritative +totals the customer-facing PDF/JSON already show. + +MMS cannot attach a local file, so the SMS path needs a *public* PDF URL +(``pdf_url``) to hand Twilio as the media URL; the server mints one via the +tokenized ``/api/estimate_pdf/{token}`` route. Email attaches the PDF bytes inline, +so it needs no public URL. +""" + +import os +from collections.abc import Callable + +from quillwright.api.recalc import recalc_estimate + +CHANNELS = ("sms", "email") + + +class SendError(Exception): + """Raised when a send is refused (bad input) or a provider fails. Loud by + design — we never silently downgrade a requested send to a no-op.""" + + +def resolve_send_mode() -> str: + """'real' only when explicitly enabled; 'mock' otherwise (the Space default).""" + return "real" if os.environ.get("FF_SEND_ENABLED") == "1" else "mock" + + +# Required creds per channel — checked up front in real mode so a missing var +# fails loud with a clean message (never leaking the raw key name to the client). +# Email has two backends; the first whose creds are fully present is used (Gmail SMTP +# preferred — no sender-verification step, no extra dep — else SendGrid). +_EMAIL_BACKENDS = { + "gmail": ("GMAIL_ADDRESS", "GMAIL_APP_PASSWORD"), + "sendgrid": ("SENDGRID_API_KEY", "FF_SEND_FROM_EMAIL"), +} +_REQUIRED_ENV = { + "sms": ("TWILIO_ACCOUNT_SID", "TWILIO_AUTH_TOKEN", "FF_SEND_FROM"), +} + + +def _email_backend() -> str | None: + """The first email backend whose creds are all present, or None if none configured.""" + for name, keys in _EMAIL_BACKENDS.items(): + if all(os.environ.get(k) for k in keys): + return name + return None + + +def _require_provider_config(channel: str) -> None: + """Raise a SendError naming the CHANNEL (not the missing env var) if real-mode + creds are absent — so an HTTP 400 reply never discloses internal config keys.""" + if channel == "email": + if _email_backend() is None: + raise SendError( + "email send is not configured on this machine. " + "Set FF_SEND_ENABLED + a Gmail or SendGrid email backend to enable it." + ) + return + missing = [k for k in _REQUIRED_ENV.get(channel, ()) if not os.environ.get(k)] + if missing: + raise SendError( + f"{channel} send is not configured on this machine. " + f"Set FF_SEND_ENABLED + the {channel} provider credentials to enable it." + ) + + +def estimate_summary_line(rows: list[dict], job_title: str, tax_rate: float) -> str: + """A one-line, customer-safe summary with the authoritative total + (Facts-from-Tools — the total comes from recalc, never from free text).""" + est = recalc_estimate(rows, job_title=job_title, tax_rate=tax_rate) + n = len(est["line_items"]) + items = "item" if n == 1 else "items" + return f"{job_title}: {n} {items}, total ${est['total']:.2f}" + + +def _render_pdf_bytes(rows: list[dict], job_title: str, tax_rate: float) -> bytes: + """Render the (edited) estimate to PDF bytes via the same path as /api/pdf.""" + import tempfile + + from quillwright.models import Estimate, LineItem + from quillwright.pdf import estimate_to_pdf + + est = Estimate( + job_title=job_title, + line_items=[ + LineItem( + description=str(r.get("description", "")), + quantity=float(r.get("quantity", 1) or 0), + unit=str(r.get("unit", "ea")), + rate=float(r.get("rate", 0) or 0), + price_source="user", + ) + for r in rows + ], + tax_rate=tax_rate, + ) + with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as tmp: + path = tmp.name + estimate_to_pdf(est, path) + with open(path, "rb") as f: + data = f.read() + os.unlink(path) + return data + + +def _validate(channel: str, recipient: str) -> str: + if channel not in CHANNELS: + raise SendError(f"Unknown channel {channel!r}; expected one of {CHANNELS}.") + recipient = (recipient or "").strip() + if not recipient: + raise SendError("Recipient is required.") + if channel == "email" and "@" not in recipient: + raise SendError(f"{recipient!r} is not a valid email address.") + return recipient + + +# --- default real providers (lazy third-party imports) --------------------- + + +def _default_sms_provider(*, recipient: str, body: str, media_url: str, summary: str) -> dict: + """Send an MMS via Twilio. Imported lazily so neither the Space nor stub mode + needs the ``twilio`` package installed. Creds from the standard Twilio env vars + (+ ``FF_SEND_FROM`` for the sending number).""" + from twilio.rest import Client # noqa: PLC0415 — lazy: optional dep, not in the Space + + account_sid = os.environ["TWILIO_ACCOUNT_SID"] + auth_token = os.environ["TWILIO_AUTH_TOKEN"] + from_number = os.environ["FF_SEND_FROM"] + client = Client(account_sid, auth_token) + msg = client.messages.create( + to=recipient, + from_=from_number, + body=body, + media_url=[media_url] if media_url else None, + ) + return {"sid": msg.sid} + + +def _default_email_provider( + *, recipient: str, subject: str, body: str, pdf_bytes: bytes, filename: str +) -> dict: + """Send an email with the PDF attached via SendGrid (Twilio's email product — + keeps everything on the one Twilio account). Lazy-imported optional dep.""" + import base64 # noqa: PLC0415 + + from sendgrid import SendGridAPIClient # noqa: PLC0415 — lazy: optional dep + from sendgrid.helpers.mail import ( # noqa: PLC0415 + Attachment, + Disposition, + FileContent, + FileName, + FileType, + Mail, + ) + + message = Mail( + from_email=os.environ["FF_SEND_FROM_EMAIL"], + to_emails=recipient, + subject=subject, + plain_text_content=body, + ) + message.attachment = Attachment( + FileContent(base64.b64encode(pdf_bytes).decode()), + FileName(filename), + FileType("application/pdf"), + Disposition("attachment"), + ) + resp = SendGridAPIClient(os.environ["SENDGRID_API_KEY"]).send(message) + return {"id": resp.headers.get("X-Message-Id", "sendgrid-accepted")} + + +def _gmail_email_provider( + *, recipient: str, subject: str, body: str, pdf_bytes: bytes, filename: str +) -> dict: + """Send an email with the PDF attached via Gmail SMTP. Uses only the stdlib + (``smtplib`` + ``email``) — no third-party dep, nothing in the Space requirements — + and an App Password (``GMAIL_APP_PASSWORD``), so there's no SendGrid sender- + verification step. The from-address is the Gmail account itself.""" + import smtplib # noqa: PLC0415 — stdlib, lazy to keep the import surface small + from email.message import EmailMessage # noqa: PLC0415 + + sender = os.environ["GMAIL_ADDRESS"] + msg = EmailMessage() + msg["From"] = sender + msg["To"] = recipient + msg["Subject"] = subject + msg.set_content(body) + msg.add_attachment(pdf_bytes, maintype="application", subtype="pdf", filename=filename) + with smtplib.SMTP_SSL("smtp.gmail.com", 465) as server: + server.login(sender, os.environ["GMAIL_APP_PASSWORD"]) + server.send_message(msg) + return {"id": f"gmail:{sender}"} + + +def _resolve_email_provider() -> Callable: + """The real email provider for the configured backend (Gmail SMTP preferred).""" + return _gmail_email_provider if _email_backend() == "gmail" else _default_email_provider + + +# --- the one entry point --------------------------------------------------- + + +def send_estimate( + *, + channel: str, + recipient: str, + rows: list[dict], + job_title: str = "Estimate", + tax_rate: float = 0.13, + mode: str | None = None, + pdf_url: str | None = None, + sms_provider: Callable | None = None, + email_provider: Callable | None = None, +) -> dict: + """Send (or, in mock mode, draft) the finished estimate to ``recipient``. + + Returns a structured result the UI renders directly:: + + {status: "sent"|"drafted", transmitted: bool, channel, recipient, + summary, provider_id} + + Providers are injectable for tests; the defaults lazily import Twilio/SendGrid. + """ + recipient = _validate(channel, recipient) + mode = mode or resolve_send_mode() + summary = estimate_summary_line(rows, job_title=job_title, tax_rate=tax_rate) + + base = { + "channel": channel, + "recipient": recipient, + "summary": summary, + "provider_id": None, + } + + if mode != "real": + # Space / disabled: draft only, transmit nothing — and say so. + return {**base, "status": "drafted", "transmitted": False} + + body = f"Here is your estimate — {summary}. (AI-generated draft; review before accepting.)" + try: + if channel == "sms": + if not pdf_url: + raise SendError( + "SMS/MMS send requires a public pdf_url to attach (Twilio cannot " + "attach a local file). None was provided." + ) + if sms_provider is None: # only the real default provider needs creds + _require_provider_config("sms") + provider = sms_provider or _default_sms_provider + result = provider(recipient=recipient, body=body, media_url=pdf_url, summary=summary) + provider_id = result.get("sid") + else: # email + if email_provider is None: + _require_provider_config("email") + provider = email_provider or _resolve_email_provider() + pdf_bytes = _render_pdf_bytes(rows, job_title=job_title, tax_rate=tax_rate) + result = provider( + recipient=recipient, + subject=f"Your estimate — {job_title}", + body=body, + pdf_bytes=pdf_bytes, + filename="estimate.pdf", + ) + provider_id = result.get("id") + except SendError: + raise + except Exception as exc: # noqa: BLE001 — any provider failure becomes a loud SendError + raise SendError(f"{channel} send failed: {exc}") from exc + + return {**base, "status": "sent", "transmitted": True, "provider_id": provider_id} diff --git a/quillwright/api/tools_api.py b/quillwright/api/tools_api.py new file mode 100644 index 0000000000000000000000000000000000000000..7c25bfb28533433e01fecaf3e3d6cc191f5d683c --- /dev/null +++ b/quillwright/api/tools_api.py @@ -0,0 +1,151 @@ +"""Quillwright tool endpoints for a voice agent (ElevenLabs Conversational AI). + +ElevenLabs owns the phone call, the speech-to-text, the dialogue, and the (great) TTS. +Quillwright stays the source of truth: the agent calls these small JSON tools to forge +and refine an estimate, and every customer-facing number comes from a tool response — +never the agent's free speech (Facts-from-Tools, ADR-0004). + +Each tool takes a ``session_id`` (the agent passes its conversation id) so refinement +turns edit the same estimate. State is in-process and demo-scoped, like the pairing store +and the Twilio call state — a durable backend is a swap behind this same interface. + +Tools: + - ``forge(session_id, description)`` → itemized estimate + total from a job description + - ``edit(session_id, request)`` → add / remove / change a line (catalog-priced) + - ``lookup_price(item)`` → a single catalog price (read-only) + - ``text_estimate(session_id, to)`` → SMS the estimate PDF to the caller +""" + +# session_id -> {"rows": list[dict], "job_title": str, "tax_rate": float} +_SESSIONS: dict[str, dict] = {} + +_JOB_TITLE = "Phone estimate" +_TAX_RATE = 0.13 + + +def reset_sessions() -> None: + """Drop all voice-agent session state (tests).""" + _SESSIONS.clear() + + +def _session(session_id: str) -> dict: + return _SESSIONS.setdefault( + session_id, {"rows": [], "job_title": _JOB_TITLE, "tax_rate": _TAX_RATE} + ) + + +def _rows_from_est(est: dict) -> list[dict]: + return [ + { + "description": li["description"], + "quantity": li["quantity"], + "unit": li["unit"], + "rate": li["rate"], + } + for li in est["line_items"] + ] + + +def _spoken_items(est: dict) -> list[dict]: + """A compact, speech-friendly view of the lines (no internal fields).""" + return [ + {"description": li["description"], "quantity": li["quantity"], "rate": li["rate"]} + for li in est["line_items"] + ] + + +def forge(session_id: str, description: str) -> dict: + """Forge an estimate from a spoken job description; store it on the session.""" + from quillwright.api.estimate import forge_estimate + + forged = forge_estimate(description or "", trade="hvac") + est = forged.get("estimate") + if est is None or not est.get("line_items"): + return { + "ok": False, + "message": "I couldn't build an estimate from that. " + "Try naming the parts and the labor.", + } + sess = _session(session_id) + sess["rows"] = _rows_from_est(est) + return { + "ok": True, + "items": _spoken_items(est), + "item_count": len(est["line_items"]), + "total": round(est["total"], 2), + } + + +def edit(session_id: str, request: str) -> dict: + """Apply a spoken edit (add / remove / change) to the session's estimate. The catalog + owns every price (Facts-from-Tools); returns the assistant's reply + the new total.""" + from quillwright.api.chat import chat_about_estimate + + sess = _session(session_id) + out = chat_about_estimate(request or "", sess["rows"], tax_rate=sess["tax_rate"]) + est = out["estimate"] + sess["rows"] = _rows_from_est(est) + return { + "ok": True, + "reply": out["reply"], + "items": _spoken_items(est), + "item_count": len(est["line_items"]), + "total": round(est["total"], 2), + } + + +def lookup_price(item: str) -> dict: + """A single catalog price (read-only) so the agent can answer 'how much is X?'.""" + from quillwright.api.estimate import CATALOG + + hit = CATALOG.lookup(item or "") + if not hit: + return {"found": False, "item": item} + return { + "found": True, + "description": hit["description"], + "rate": hit["rate"], + "unit": hit["unit"], + } + + +def text_estimate(session_id: str, to: str, base_url: str | None = None, sms=None) -> dict: + """SMS the estimate PDF to the caller. Reuses the S10 send path + tokenized PDF link. + ``sms`` is injectable for tests; otherwise the real Twilio MMS provider is used.""" + import os + + from quillwright.api.estimate import save_estimate_record + from quillwright.api.pdf_links import public_pdf_url, register_pdf + from quillwright.api.recalc import recalc_estimate + from quillwright.api.send import _default_sms_provider, _render_pdf_bytes + + sess = _session(session_id) + rows, job_title, tax_rate = sess["rows"], sess["job_title"], sess["tax_rate"] + if not rows: + return {"ok": False, "message": "There's no estimate to send yet."} + if not to: + return {"ok": False, "message": "I need a phone number to text it to."} + + base = (base_url if base_url is not None else os.environ.get("FF_PUBLIC_BASE_URL", "")).rstrip( + "/" + ) + est = recalc_estimate(rows, job_title=job_title, tax_rate=tax_rate) + save_estimate_record(rows, job_title, tax_rate, thread=[]) # persist the draft (ADR-0013) + n = len(est["line_items"]) + send = sms or _default_sms_provider + try: + pdf_bytes = _render_pdf_bytes(rows, job_title=job_title, tax_rate=tax_rate) + token = register_pdf(pdf_bytes) + media_url = public_pdf_url(token, base_url=base or "") + send( + recipient=to, + body=( + f"Your Quillwright estimate: {n} item{'s' if n != 1 else ''}, " + f"total ${est['total']:.2f}. AI-generated draft — review before accepting." + ), + media_url=media_url, + summary="", + ) + return {"ok": True, "sent": True, "total": round(est["total"], 2)} + except Exception as exc: # noqa: BLE001 — report a clean failure to the agent + return {"ok": False, "sent": False, "message": f"Couldn't text it: {exc}"} diff --git a/quillwright/api/transcribe.py b/quillwright/api/transcribe.py new file mode 100644 index 0000000000000000000000000000000000000000..eae771425610d6952a9cc680c204cbaa535108e4 --- /dev/null +++ b/quillwright/api/transcribe.py @@ -0,0 +1,35 @@ +"""Transcribe a spoken voice note into the text the agent works from (ADR-0009). + +Thin adapter over the Audio role: file path in, {transcript} out. The model is +injectable for tests; in production it resolves to Cohere Transcribe on-device. +""" + +import os + + +def _resolve_audio(): + """The Audio role, by env: hosted Omni (Best Stack) when its Modal URL is + configured, on-device STT when FF_REAL_MODELS=1, else None (typed note).""" + from quillwright.resolver import modal_resolver_if_configured + + modal = modal_resolver_if_configured("audio") + if modal is not None: + return modal.for_role("audio") + if os.environ.get("FF_REAL_MODELS") == "1": + from quillwright.resolver import ModelResolver + + return ModelResolver(mode="private", backend="ollama").for_role("audio") + return None + + +def transcribe_audio(path: str, model=None) -> dict: + """Transcribe the audio at `path`. Returns {"transcript": str}. + + `model` is injectable (tests); otherwise resolves the Audio role. The transcript + is whitespace-normalized so it drops cleanly into the note field. + """ + asr = model if model is not None else _resolve_audio() + if asr is None: + return {"transcript": ""} + text = (asr.transcribe(path) or "").strip() + return {"transcript": text} diff --git a/quillwright/api/voice.py b/quillwright/api/voice.py new file mode 100644 index 0000000000000000000000000000000000000000..9322de82c3ec7ae468139a839e99718aadaf4eba --- /dev/null +++ b/quillwright/api/voice.py @@ -0,0 +1,429 @@ +"""Inbound voice-call capture (S12) — call a number, the agent forges an estimate. + +Flow (all reuse; no new business logic): + + 1. Twilio routes an inbound call to ``POST /api/voice/incoming``. We answer with + TwiML: a short greeting then ````, which posts the ``RecordingUrl`` to + ``POST /api/voice/recording`` when the caller hangs up. + 2. The recording webhook downloads the ``.wav``/``.mp3``, transcribes it (Audio role + — Nemotron Omni on the Best Stack, Cohere Transcribe on the Private Stack, same + ``transcribe_audio`` resolution as the mic button), forges an estimate, and saves + it as a **DRAFT** under ``account_id="demo"`` (ADR-0013). It then ````s the + spoken total and ````s the caller's reply (Tier A — a conversation). + 3. Each reply hits ``POST /api/voice/refine`` with Twilio's own ``SpeechResult`` + transcript. "Done/no" → recalc, persist, text the PDF, end the call. Otherwise the + spoken edit runs through the SAME ``chat_about_estimate`` ops (add / remove / change) + the desktop chat uses, the new total is read back, and we ```` again. Per-call + state is held under the Twilio ``CallSid`` (in-process, demo-scoped). + +Honesty (ADR-0004, ADR-0013): the estimate's numbers all come from the catalog + +``recalc`` (Facts-from-Tools); the call produces a draft a human approves later. On a +call the agent runs to completion without the interactive Agent Pause (``forge_estimate`` +is the non-streaming path — a missing price is auto-flagged in the trace, never blocks). + +The public base URL is read from ``FF_PUBLIC_BASE_URL`` (the ngrok/cloudflared tunnel), +so Twilio can fetch the recording-action URL and the PDF media URL. SMS reuses the S10 +``send_estimate`` SMS provider and the tokenized PDF registry — nothing new is sent. +""" + +import os +from collections.abc import Callable +from xml.sax.saxutils import escape + +# Spoken voice for every . Amazon Polly Neural voices (rendered by Twilio at no extra +# cost beyond standard call rates) sound far more natural than the default. Override with +# FF_VOICE if you prefer another (e.g. Polly.Joanna-Neural, Polly.Stephen-Neural). +VOICE = os.environ.get("FF_VOICE", "Polly.Matthew-Neural") + + +def _say(message: str) -> str: + """A with the configured natural voice.""" + return f'{escape(message)}' + + +def public_base_url() -> str: + """The tunnel's public base URL (FF_PUBLIC_BASE_URL), trailing slash stripped, or ''.""" + return os.environ.get("FF_PUBLIC_BASE_URL", "").rstrip("/") + + +def _action_url(path: str, base_url: str | None = None) -> str: + """An absolute URL on the public base when known, else a relative path (Twilio + resolves a relative against the request host).""" + base = (base_url if base_url is not None else public_base_url()).rstrip("/") + return f"{base}{path}" if base else path + + +def greeting_twiml(base_url: str | None = None) -> str: + """Answer an inbound call: greet, then record the caller's job description. + + ```` posts the RecordingUrl to /api/voice/recording on hang-up (or after the + silence timeout). ``playBeep`` cues the caller; ``maxLength`` caps a runaway call. + """ + action = _action_url("/api/voice/recording", base_url) + return ( + '\n' + "" + + _say( + "Welcome to Quillwright. After the beep, describe the job — the parts you " + "used and the labor — then stop talking. I'll forge an estimate, read it back, " + "and you can tell me what to change." + ) + + f'' + + _say("I didn't catch a recording. Goodbye.") + + "" + ) + + +def _say_response(message: str) -> str: + """A bare spoken TwiML response (no recording).""" + return f'\n{_say(message)}' + + +# --- Conversational refine loop (Tier A): after forging, the agent reads the total and +# keeps the call open, asking "anything else?" via . Each reply +# is a new /api/voice/refine turn that runs the SAME chat_about_estimate ops (add / +# remove / change), so Facts-from-Tools holds — the agent only READS totals that recalc +# produced. State is held per Twilio CallSid (in-process, demo-scoped, like pairing). --- + +# call_sid -> {"rows": list[dict], "job_title": str, "tax_rate": float, "from_number": str} +_CALLS: dict[str, dict] = {} + +# Phrases that end the conversation (caller says they're done). +_DONE_WORDS = ( + "no", + "nope", + "nothing", + "that's it", + "thats it", + "done", + "all set", + "good", + "send it", +) + + +def _call_state(call_sid: str) -> dict | None: + return _CALLS.get(call_sid) + + +def reset_calls() -> None: + """Drop all in-flight call + job state (tests).""" + _CALLS.clear() + _JOBS.clear() + + +def _is_done(speech: str) -> bool: + s = (speech or "").strip().lower() + if not s: + return False + return any(w in s for w in _DONE_WORDS) + + +def _ask_twiml(message: str, base_url: str | None = None) -> str: + """Speak `message`, then the caller's spoken reply to /api/voice/refine. + If they stay silent, end politely (the Gather falls through to the closing Say).""" + action = _action_url("/api/voice/refine", base_url) + return ( + '\n' + "" + f'' + + _say(message) + + "" + + _say("I didn't catch that — I'll text you what I have. Goodbye.") + + "" + ) + + +def _download_recording(url: str) -> str: + """Fetch a Twilio RecordingUrl to a local temp file. Twilio serves the media at + ``.wav`` (a safer container for Omni than the browser's webm). Auth with the + standard Twilio creds when present (recordings on a real account are protected).""" + import tempfile + import time + + import requests # already a core dep + + media_url = url if url.endswith((".wav", ".mp3")) else f"{url}.wav" + auth = None + sid, token = os.environ.get("TWILIO_ACCOUNT_SID"), os.environ.get("TWILIO_AUTH_TOKEN") + if sid and token: + auth = (sid, token) + # Twilio posts the recording webhook the instant recording ends, but the media file + # is often not encoded/available for a beat — an immediate GET 404s/403s. Retry a few + # times with a short backoff so the not-ready race doesn't fail the call. + last = None + for attempt in range(5): + resp = requests.get(media_url, auth=auth, timeout=20) + if resp.status_code == 200 and resp.content: + with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: + tmp.write(resp.content) + return tmp.name + last = resp + if resp.status_code not in (404, 403, 401): + break # a different error won't fix itself by waiting + time.sleep(1.5) + if last is not None: + last.raise_for_status() + raise RuntimeError(f"could not fetch recording at {media_url}") + + +def _send_sms(*, to: str, body: str, media_url: str) -> dict: + """Text the caller via Twilio (lazy import — same optional [send] dep as S10).""" + from twilio.rest import Client # noqa: PLC0415 — lazy: optional dep, not in the Space + + client = Client(os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"]) + msg = client.messages.create( + to=to, + from_=os.environ["FF_SEND_FROM"], + body=body, + media_url=[media_url] if media_url else None, + ) + return {"sid": msg.sid} + + +def _rows_from_est(est: dict) -> list[dict]: + """The editable rows (no subtotal/source) the chat ops + PDF renderer expect.""" + return [ + { + "description": li["description"], + "quantity": li["quantity"], + "unit": li["unit"], + "rate": li["rate"], + } + for li in est["line_items"] + ] + + +def _summary(est: dict) -> str: + n = len(est["line_items"]) + items = "item" if n == 1 else "items" + return f"{n} {items}, totaling {est['total']:.2f} dollars" + + +def _text_pdf(*, rows, job_title, tax_rate, total, n, from_number, base, sms) -> bool: + """Render + register the PDF and text it to the caller. Best-effort: returns True on + a send, False on any failure (the draft is already saved either way).""" + from quillwright.api.pdf_links import public_pdf_url, register_pdf + from quillwright.api.send import _render_pdf_bytes + + if not from_number: + return False + try: + pdf_bytes = _render_pdf_bytes(rows, job_title=job_title, tax_rate=tax_rate) + token = register_pdf(pdf_bytes) + media_url = public_pdf_url(token, base_url=base or "") + items = "item" if n == 1 else "items" + sms( + to=from_number, + body=( + f"Your Quillwright estimate: {n} {items}, total ${total:.2f}. " + "AI-generated draft — review before accepting." + ), + media_url=media_url, + ) + return True + except Exception: # noqa: BLE001 — texting is best-effort; the draft is saved + return False + + +def handle_recording( + *, + recording_url: str, + from_number: str, + call_sid: str = "default", + download: Callable[[str], str] | None = None, + transcribe: Callable[[str], dict] | None = None, + sms: Callable | None = None, + base_url: str | None = None, +) -> dict: + """Transcribe the recording, forge + save a draft estimate, then ASK the caller if + they want to change anything (the conversational refine loop — Tier A). + + Returns ``{"estimate": , "twiml": , "transcript": str}``. + The PDF is NOT texted here — it goes out when the caller says they're done (see + ``handle_refine``). Per-call state is held under ``call_sid``. Side-effects + (download / SMS) are injectable so tests need no network or twilio. + """ + from quillwright.api.estimate import estimate_store, forge_estimate + from quillwright.api.transcribe import transcribe_audio + + download = download or _download_recording + transcribe = transcribe or (lambda path: transcribe_audio(path)) + base = (base_url if base_url is not None else public_base_url()).rstrip("/") + + path = download(recording_url) + transcript = (transcribe(path) or {}).get("transcript", "").strip() + if not transcript: + return { + "estimate": None, + "transcript": "", + "twiml": _say_response( + "Sorry, I couldn't make out the job from that recording. " + "Please call back and describe the parts and labor after the beep." + ), + } + + forged = forge_estimate(transcript, trade="hvac") + est = forged.get("estimate") + if est is None or not est.get("line_items"): + return { + "estimate": None, + "transcript": transcript, + "twiml": _say_response( + "I heard the job but couldn't build an estimate from it. " + "I've made a note — please call back with the parts and labor." + ), + } + + # Hold the rows for this call so refine turns edit the same estimate (forge_estimate + # already auto-saved a DRAFT — ADR-0013; this is the same store the desktop reads). + _CALLS[call_sid] = { + "rows": _rows_from_est(est), + "job_title": est["job_title"], + "tax_rate": est["tax_rate"], + "from_number": from_number, + } + estimate_store() # touch so a misconfigured store surfaces in logs + spoken = ( + f"Done. I forged an estimate with {_summary(est)}. " + "Want to add or change anything, or should I text it to you?" + ) + return {"estimate": est, "transcript": transcript, "twiml": _ask_twiml(spoken, base)} + + +# --- Async job pattern: forge+transcribe take ~tens of seconds (model load + brain), +# far over Twilio's ~15s webhook timeout. So the recording webhook kicks the work off +# on a background thread and returns a holding response immediately; Twilio is parked on +# a + to /api/voice/status, which polls until the job finishes. Each +# webhook response stays well under the timeout. --- + +# call_sid -> {"status": "working"|"done"|"error", "twiml": } +_JOBS: dict[str, dict] = {} + + +def _hold_twiml(message: str, base_url: str | None = None) -> str: + """Speak a short status line, pause, then redirect to /api/voice/status to poll again.""" + action = _action_url("/api/voice/status", base_url) + return ( + '\n' + "" + + _say(message) + + '' + + f'{escape(action)}' + + "" + ) + + +def start_recording_job( + *, recording_url: str, from_number: str, call_sid: str, base_url: str | None = None +) -> str: + """Kick the forge off on a background thread; return holding TwiML immediately. + The webhook never blocks on the slow work (model load + brain).""" + import threading + + base = (base_url if base_url is not None else public_base_url()).rstrip("/") + _JOBS[call_sid] = {"status": "working", "twiml": None} + + def _work(): + try: + result = handle_recording( + recording_url=recording_url, + from_number=from_number, + call_sid=call_sid, + base_url=base, + ) + _JOBS[call_sid] = {"status": "done", "twiml": result["twiml"]} + except Exception as exc: # noqa: BLE001 — surface as an error status, not a crash + print(f"[quillwright] voice forge job failed: {exc}", flush=True) + _JOBS[call_sid] = { + "status": "error", + "twiml": _say_response( + "Sorry, I couldn't build that estimate. Please call back and try again." + ), + } + + threading.Thread(target=_work, daemon=True).start() + return _hold_twiml("Got it. I'm forging your estimate now — this takes a moment.", base) + + +def handle_status(*, call_sid: str, base_url: str | None = None) -> str: + """Poll the background forge: still working → hold + redirect again; done → the ask + (or error) TwiML the job produced. Unknown call → polite fallback.""" + base = (base_url if base_url is not None else public_base_url()).rstrip("/") + job = _JOBS.get(call_sid) + if job is None: + return _say_response( + "Sorry, I lost track of that estimate. Please call back to start again." + ) + if job["status"] == "working": + return _hold_twiml("Still working on it — just a few more seconds.", base) + # done or error: hand back the prepared TwiML and clear the job marker. + _JOBS.pop(call_sid, None) + return job["twiml"] + + +def handle_refine( + *, + call_sid: str, + speech_result: str, + base_url: str | None = None, + sms: Callable | None = None, +) -> dict: + """One caller turn in the refine loop. If they're done, text the PDF and end; else + apply the spoken edit through the SAME chat_about_estimate ops (Facts-from-Tools — + the catalog owns every price) and ask again. + + Returns ``{"estimate": , "twiml": str}``. ``sms`` is injectable for tests. + """ + from quillwright.api.chat import chat_about_estimate + from quillwright.api.estimate import save_estimate_record + + sms = sms or _send_sms + base = (base_url if base_url is not None else public_base_url()).rstrip("/") + state = _CALLS.get(call_sid) + if state is None: + # Lost the thread (server restart / stale call) — fail politely, don't crash. + return { + "estimate": None, + "twiml": _say_response( + "Sorry, I lost track of that estimate. Please call back to start again." + ), + } + + rows = state["rows"] + job_title, tax_rate = state["job_title"], state["tax_rate"] + + # Caller signalled they're finished → recalc to authoritative numbers, persist the + # final draft, text the PDF, and end the call. + if _is_done(speech_result): + from quillwright.api.recalc import recalc_estimate + + est = recalc_estimate(rows, job_title=job_title, tax_rate=tax_rate) + save_estimate_record(rows, job_title, tax_rate, thread=[]) # update the saved draft + n = len(est["line_items"]) + sent = _text_pdf( + rows=rows, + job_title=job_title, + tax_rate=tax_rate, + total=est["total"], + n=n, + from_number=state.get("from_number", ""), + base=base, + sms=sms, + ) + _CALLS.pop(call_sid, None) + tail = ( + "I've texted you the PDF. It's a draft — review before sending it on. Goodbye." + if sent + else "It's saved as a draft on your dashboard. Goodbye." + ) + return {"estimate": est, "twiml": _say_response(f"Got it. {tail}")} + + # Otherwise it's an edit: run it through the shared chat ops (catalog owns the price). + out = chat_about_estimate(speech_result, rows, tax_rate=tax_rate) + est = out["estimate"] + state["rows"] = _rows_from_est(est) # carry the edit forward to the next turn + save_estimate_record(state["rows"], job_title, tax_rate, thread=[]) # keep the draft current + spoken = f"{out['reply']} That's now {est['total']:.2f} dollars. Anything else?" + return {"estimate": est, "twiml": _ask_twiml(spoken, base)} diff --git a/quillwright/backends/audio.py b/quillwright/backends/audio.py new file mode 100644 index 0000000000000000000000000000000000000000..757e3b6a5b8dde8a414ba12ed57c73d50f69491d --- /dev/null +++ b/quillwright/backends/audio.py @@ -0,0 +1,92 @@ +"""AudioModel: local speech-to-text for the spoken voice note (ADR-0009). + +Wraps CohereLabs/cohere-transcribe-03-2026 (2B, #1 WER, on-device) via transformers' +canonical path — AutoProcessor + CohereAsrForConditionalGeneration, the approach the +model card documents (the generic `pipeline()` API errors on this model). Verified +locally: a trade note transcribes cleanly. + +Heavy (torch + a 2B model), so the import + load are LAZY — importing this module +costs nothing; the model loads on first `.transcribe()`. Keeps the spoken note inside +the on-device Private Stack (🔌 Off the Grid). Gated repo: needs HF access + token. +""" + +DEFAULT_MODEL = "CohereLabs/cohere-transcribe-03-2026" + + +def to_wav_16k_mono(path: str) -> str: + """Return a path to a 16kHz mono WAV for `path`. + + Browser/phone MediaRecorder emits a **webm** container (Opus), which librosa / + transformers' `load_audio` cannot decode ("appears to be a video file"). Twilio call + recordings can be `.mp3` too. We normalize anything that isn't already a `.wav` to + 16kHz mono PCM WAV with ffmpeg first — the format the ASR model wants, and a safer + container all round. A `.wav` input is returned unchanged (no needless transcode). + + Raises RuntimeError with a clear message if ffmpeg isn't installed. + """ + import os + import shutil + import subprocess + import tempfile + + if path.lower().endswith(".wav"): + return path + if shutil.which("ffmpeg") is None: + raise RuntimeError( + "ffmpeg is required to transcode the voice note (browser/phone records webm, " + "which the ASR model can't read). Install it (e.g. `brew install ffmpeg`)." + ) + with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: + out = tmp.name + subprocess.run( + ["ffmpeg", "-y", "-i", path, "-ar", "16000", "-ac", "1", "-f", "wav", out], + check=True, + capture_output=True, + ) + if not os.path.getsize(out): + raise RuntimeError(f"ffmpeg produced an empty WAV transcoding {path!r}.") + return out + + +class AudioModel: + def __init__(self, model: str = DEFAULT_MODEL, language: str = "en"): + self.name = model + self._language = language + self._processor = None + self._model = None + + def _load(self): + if self._model is None: + from transformers import AutoProcessor, CohereAsrForConditionalGeneration + + self._processor = AutoProcessor.from_pretrained(self.name) + self._model = CohereAsrForConditionalGeneration.from_pretrained( + self.name, device_map="auto" + ) + return self._processor, self._model + + def transcribe(self, path: str) -> str: + import os + + from transformers.audio_utils import load_audio + + processor, model = self._load() + # Normalize webm/m4a/mp3 → 16kHz mono WAV (load_audio can't decode webm). Clean up + # any temp file we created (but never the caller's original .wav). + wav = to_wav_16k_mono(path) + try: + audio = load_audio(wav, sampling_rate=16000) + finally: + if wav != path: + try: + os.unlink(wav) + except OSError: + pass + inputs = processor(audio, sampling_rate=16000, return_tensors="pt", language=self._language) + inputs.to(model.device, dtype=model.dtype) + outputs = model.generate(**inputs, max_new_tokens=256) + decoded = processor.decode(outputs, skip_special_tokens=True) + # decode() returns a list (one string per batch item); we transcribe one clip. + if isinstance(decoded, list): + decoded = decoded[0] if decoded else "" + return decoded diff --git a/quillwright/backends/embedding.py b/quillwright/backends/embedding.py new file mode 100644 index 0000000000000000000000000000000000000000..0b8686b88f8149ca5e102d936f1b74110225abe3 --- /dev/null +++ b/quillwright/backends/embedding.py @@ -0,0 +1,30 @@ +"""EmbeddingModel: local text embeddings for semantic Recall (ADR-0003). + +Wraps `nvidia/llama-nemotron-embed-1b-v2` via sentence-transformers — the model +card's recommended local path (NOT Ollama; it has no embeddings API). Runs fully +offline → preserves 🔌 Off the Grid and adds NVIDIA breadth. + +sentence-transformers + torch are heavy (~2GB), so the import is LAZY: importing +this module costs nothing; the model loads on first `.encode()`. Per ADR-0003 the +hot path only embeds the QUERY (run vectors are cached at record time), so torch +stays out of recall-time latency. +""" + +DEFAULT_MODEL = "nvidia/llama-nemotron-embed-1b-v2" + + +class EmbeddingModel: + def __init__(self, model: str = DEFAULT_MODEL): + self.name = model + self._st = None # lazily-loaded SentenceTransformer + + def _model(self): + if self._st is None: + from sentence_transformers import SentenceTransformer + + self._st = SentenceTransformer(self.name, trust_remote_code=True) + return self._st + + def encode(self, text: str) -> list[float]: + vec = self._model().encode(text, normalize_embeddings=True) + return vec.tolist() diff --git a/quillwright/backends/modal.py b/quillwright/backends/modal.py new file mode 100644 index 0000000000000000000000000000000000000000..5775407995357533349a31ea353b034e3cd24549 --- /dev/null +++ b/quillwright/backends/modal.py @@ -0,0 +1,202 @@ +"""ModalModel: the Best-Stack hosted-compute client (ADR-0005 / ADR-0009). + +Same interface as OllamaModel (name + generate + chat, plus transcribe for the +Audio role) so the resolver can swap to it with `backend="modal"`. It calls the +vLLM OpenAI-compatible server deployed by the role's modal app and adapts the +response back to our internal contract: + + - chat() returns {"content": str, "tool_calls": [{"function": {"name", "arguments"}}]} + where `arguments` is a DICT (vLLM/OpenAI gives it as a JSON string — we parse it), + matching what brain_loop.py expects from OllamaModel.chat(). + - generate() takes an optional image_path (Best-Stack Perception via Omni) sent + as an OpenAI data-URL image_url content part. + - transcribe() sends a voice note as OpenAI input_audio (Best-Stack Audio via the + SAME Omni deployment — it is omnimodal, so one app serves two Model Roles). + +Each role reads its own base URL env (printed by `modal deploy` of its app): +brain -> modal_app.py, perception/audio -> modal_omni_app.py, +multilingual -> modal_aya_app.py. +""" + +import base64 +import json +import os +from pathlib import Path + +import requests + +# role -> (url env, served-model override env, default served model repo id). +# The deployed vLLM server pins the real repo; the env override exists for the +# day a variant swap shouldn't need a redeploy of this client. +ROLE_ENDPOINTS = { + "brain": ( + "FF_MODAL_BRAIN_URL", + "FF_MODAL_BRAIN_MODEL", + "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8", + ), + "perception": ( + "FF_MODAL_OMNI_URL", + "FF_MODAL_OMNI_MODEL", + "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8", + ), + "audio": ( + "FF_MODAL_OMNI_URL", + "FF_MODAL_OMNI_MODEL", + "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8", + ), + "multilingual": ( + "FF_MODAL_AYA_URL", + "FF_MODAL_AYA_MODEL", + "CohereLabs/aya-expanse-8b", + ), +} + + +class ModalModel: + def __init__( + self, + model: str, + role: str = "brain", + base_url: str | None = None, + timeout: float = 300.0, + ): + # `model` is the label/role tag; the deployed server already pins the real + # repo id, so we send a model field vLLM accepts (the served model name). + self.name = model + url_env, model_env, default_model = ROLE_ENDPOINTS[role] + self._base = (base_url or os.environ.get(url_env, "")).rstrip("/") + if not self._base: + raise RuntimeError( + f"{url_env} is not set — deploy the Modal app for role '{role}' and " + "export the URL it prints (see quillwright/backends/modal_*.py)." + ) + self._served_model = os.environ.get(model_env, default_model) + self._timeout = timeout + + def _post(self, path: str, body: dict) -> dict: + resp = requests.post(f"{self._base}{path}", json=body, timeout=self._timeout) + resp.raise_for_status() + return resp.json() + + def chat(self, messages: list[dict], tools: list[dict]) -> dict: + """Tool-calling chat via vLLM's OpenAI API; adapt to our message contract.""" + body = { + "model": self._served_model, + "messages": _to_openai_messages(messages), + "tools": tools, + "tool_choice": "auto", + "stream": False, + } + data = self._post("/v1/chat/completions", body) + msg = (data.get("choices") or [{}])[0].get("message", {}) or {} + return { + "content": msg.get("content") or "", + "tool_calls": [_adapt_tool_call(tc) for tc in (msg.get("tool_calls") or [])], + } + + def generate(self, prompt: str, image_path: str | None = None) -> str: + """Completion via the OpenAI chat API; an image (Best-Stack Perception via + Omni) rides as a data-URL image_url content part.""" + content: str | list = prompt + if image_path: + suffix = Path(image_path).suffix.lstrip(".").lower() or "png" + content = [ + {"type": "text", "text": prompt}, + { + "type": "image_url", + "image_url": {"url": f"data:image/{suffix};base64,{_b64(image_path)}"}, + }, + ] + body = { + "model": self._served_model, + "messages": [{"role": "user", "content": content}], + "stream": False, + } + data = self._post("/v1/chat/completions", body) + return (data.get("choices") or [{}])[0].get("message", {}).get("content", "") or "" + + def transcribe(self, audio_path: str) -> str: + """Transcribe a voice note (Best-Stack Audio via Omni): OpenAI input_audio + content in, plain transcript out. Same contract as AudioModel.transcribe.""" + fmt = Path(audio_path).suffix.lstrip(".").lower() or "wav" + body = { + "model": self._served_model, + "messages": [ + { + "role": "user", + "content": [ + { + "type": "input_audio", + "input_audio": {"data": _b64(audio_path), "format": fmt}, + }, + {"type": "text", "text": "Transcribe this voice note verbatim."}, + ], + } + ], + "stream": False, + } + data = self._post("/v1/chat/completions", body) + return (data.get("choices") or [{}])[0].get("message", {}).get("content", "") or "" + + +def _b64(path: str) -> str: + with open(path, "rb") as fh: + return base64.b64encode(fh.read()).decode("ascii") + + +def _adapt_tool_call(tc: dict) -> dict: + """OpenAI tool_call -> our shape. `arguments` arrives as a JSON string; parse it.""" + fn = tc.get("function", {}) or {} + args = fn.get("arguments", {}) + if isinstance(args, str): + try: + args = json.loads(args) if args.strip() else {} + except json.JSONDecodeError: + args = {} + return {"function": {"name": fn.get("name", ""), "arguments": args}} + + +def _to_openai_messages(messages: list[dict]) -> list[dict]: + """Sanitize our internal message list into valid OpenAI chat format. + + vLLM's OpenAI endpoint is stricter than Ollama, which our brain_loop targets: + - assistant tool_calls need a string `arguments` (we carry a dict) + an `id`; + - each `tool` reply needs a `tool_call_id` matching its assistant tool_call. + We assign deterministic ids and thread them to the following tool messages, so + brain_loop.py / the Ollama path stay untouched. + """ + out: list[dict] = [] + pending_ids: list[str] = [] # tool_call ids awaiting their tool replies, in order + counter = 0 + + for m in messages: + role = m.get("role") + # A message carrying tool_calls is an assistant turn — even if it has no + # `role` (our chat() return value is appended verbatim by brain_loop and + # lacks one). vLLM requires role + string args + ids; normalize all of it. + if m.get("tool_calls"): + calls = [] + for tc in m["tool_calls"]: + fn = tc.get("function", {}) or {} + args = fn.get("arguments", {}) + if not isinstance(args, str): + args = json.dumps(args) + tc_id = tc.get("id") or f"call_{counter}" + counter += 1 + pending_ids.append(tc_id) + calls.append( + { + "id": tc_id, + "type": "function", + "function": {"name": fn.get("name", ""), "arguments": args}, + } + ) + out.append( + {"role": "assistant", "content": m.get("content") or None, "tool_calls": calls} + ) + elif role == "tool": + tc_id = m.get("tool_call_id") or (pending_ids.pop(0) if pending_ids else "call_0") + out.append({"role": "tool", "tool_call_id": tc_id, "content": m.get("content", "")}) + else: + out.append(m) + return out diff --git a/quillwright/backends/modal_app.py b/quillwright/backends/modal_app.py new file mode 100644 index 0000000000000000000000000000000000000000..521c4e34402835706691451cacea74fc393bca9f --- /dev/null +++ b/quillwright/backends/modal_app.py @@ -0,0 +1,103 @@ +"""Modal deployment of the Quillwright Best-Stack brain (ADR-0009). + +ADR-0005: the hosted HF Space has no GPU, so real models reach it via an outbound +HTTPS call to Modal. ADR-0009 locks the **Best Stack** brain as **Nemotron 3 Nano +30B-A3B** (31.6B total / 3.2B active, MoE) — genuinely better than the local 4B and +too big for local Ollama, which is exactly why it lives on Modal. + +Served with **vLLM** using NVIDIA's documented FP8 + tool-calling recipe +(https://docs.vllm.ai/projects/recipes/en/latest/NVIDIA/Nemotron-3-Nano-30B-A3B.html). +vLLM exposes an OpenAI-compatible API, so the client (`backends/modal.py`) talks +/v1/chat/completions and adapts the tool_calls shape back to our contract. + +De-risk scope (ADR-0005): the BRAIN only. Vision (Omni) + multilingual copy this +pattern once it's proven. + +Deploy (you run these — they touch your Modal account + credits): + modal setup # one-time auth + modal deploy quillwright/backends/modal_app.py +Then point the app at the printed URL: + export FF_BACKEND=modal + export FF_MODAL_BRAIN_URL="https://<...>.modal.run" +""" + +import modal + +MODEL = "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8" # ADR-0009 Best-Stack brain, FP8 (~32GB). +VLLM_PORT = 8000 +image = ( + # CUDA *devel* base (includes nvcc): FlashInfer's FP8 MoE kernel JIT-compiles at + # runtime, so debian_slim (no nvcc) crashes engine init. This matches torch's cu12. + modal.Image.from_registry("nvidia/cuda:12.8.1-devel-ubuntu22.04", add_python="3.12") + .pip_install("vllm==0.12.0", "huggingface_hub", "flashinfer-python") + # Fetch NVIDIA's custom reasoning-parser plugin via huggingface_hub. + .run_commands( + 'python -c "' + "from huggingface_hub import hf_hub_download; import shutil; " + "p = hf_hub_download(" + "repo_id='nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16', " + "filename='nano_v3_reasoning_parser.py'); " + "shutil.copy(p, '/root/nano_v3_reasoning_parser.py')\"" + ) + .env( + { + # FP8 MoE acceleration (FP8 variant only), per NVIDIA's recipe. + "VLLM_USE_FLASHINFER_MOE_FP8": "1", + "VLLM_FLASHINFER_MOE_BACKEND": "throughput", + # Download weights into the mounted cache volume (NOT ~/.cache, which the + # build populates — Modal refuses to mount a volume over a non-empty dir). + "HF_HOME": "/cache", + } + ) +) + +# Cache the downloaded weights across cold starts (pull once, not every boot). +hf_cache = modal.Volume.from_name("quillwright-hf-cache", create_if_missing=True) + +app = modal.App("quillwright-brain") + + +@app.function( + image=image, + gpu="L40S", # FP8 needs ~32GB VRAM; L40S has 48GB (A10G's 24GB is too small). + volumes={"/cache": hf_cache}, # clean mount point; HF_HOME points here. + # HF_TOKEN for the weight download (harmless if ungated; required if the NVIDIA + # repo is gated). Same secret across all four apps. + secrets=[modal.Secret.from_name("huggingface-secret")], + timeout=1200, + scaledown_window=120, # warm 2 min after a request (masks cold starts; limits idle L40S burn). + min_containers=0, # true scale-to-zero: $0 when idle (open-ended judging window — never pre-warm-and-forget). +) +@modal.concurrent(max_inputs=8) +@modal.web_server(port=VLLM_PORT, startup_timeout=900) +def serve(): + """Launch vLLM's OpenAI-compatible server with NVIDIA's tool-calling recipe.""" + import subprocess + + cmd = [ + "vllm", + "serve", + MODEL, + "--trust-remote-code", + "--async-scheduling", + "--kv-cache-dtype", + "fp8", + "--tensor-parallel-size", + "1", + "--enable-auto-tool-choice", + "--tool-call-parser", + "qwen3_coder", # the parser NVIDIA specifies for this model. + "--reasoning-parser-plugin", + "/root/nano_v3_reasoning_parser.py", + "--reasoning-parser", + "nano_v3", + "--max-model-len", + "262144", + "--max-num-seqs", + "8", + "--port", + str(VLLM_PORT), + "--host", + "0.0.0.0", + ] + subprocess.Popen(" ".join(cmd), shell=True) diff --git a/quillwright/backends/modal_aya_app.py b/quillwright/backends/modal_aya_app.py new file mode 100644 index 0000000000000000000000000000000000000000..c7124bb861a7cbcb62853532ac982acf80636e9d --- /dev/null +++ b/quillwright/backends/modal_aya_app.py @@ -0,0 +1,73 @@ +"""Modal deployment of the Quillwright Best-Stack Multilingual: Aya Expanse 8B. + +ADR-0009 lists Best-Stack Multilingual as "Aya larger / Command". The pick here is +CohereLabs/aya-expanse-8b in full BF16: a *newer generation* than the local Aya 23 +(q4 via Ollama) — better multilingual quality at full precision, and small enough +for a cheap A10G (24 GB). Deliberately NOT aya-expanse-32b: ADR-0009 records the +contest rule as STRICTLY under 32B per model ("verified at kickoff"), which a +32B-named model fails. If that reading is overturned (PROGRESS/CONTEXT say "<=32B"), +upgrading is this file's MODEL constant + FF_MODAL_AYA_MODEL — one line. + +Translation only needs .generate() (descriptions in, descriptions out — numbers +never pass through a model), so this is the simplest of the three vLLM apps: no +tool parsing, no reasoning parser, no FP8 MoE kernels (plain dense 8B). + +NOT deployed yet (build-only; deploys touch your Modal account + credits): + modal deploy quillwright/backends/modal_aya_app.py +Then point the app at the printed URL (per-role opt-in): + export FF_BACKEND=modal + export FF_MODAL_AYA_URL="https://<...>.modal.run" +""" + +import modal + +MODEL = "CohereLabs/aya-expanse-8b" # ADR-0009 Best-Stack Multilingual (see above). +VLLM_PORT = 8000 +image = ( + # No FP8-MoE JIT here (dense BF16 model) — the slim base is enough. + modal.Image.debian_slim(python_version="3.12") + .pip_install("vllm==0.12.0", "huggingface_hub") + .env({"HF_HOME": "/cache"}) +) + +# Shared with the other apps: pull weights once, not every cold start. +hf_cache = modal.Volume.from_name("quillwright-hf-cache", create_if_missing=True) + +app = modal.App("quillwright-aya") + + +@app.function( + image=image, + gpu="A10G", # 8B BF16 ~16GB; A10G's 24GB fits with short-context headroom. + volumes={"/cache": hf_cache}, + # aya-expanse-8b is a GATED HF repo — vLLM 401s without a token. HF_TOKEN from + # this secret authenticates the weight download. Accept the license on the repo + # page first: huggingface.co/CohereLabs/aya-expanse-8b + secrets=[modal.Secret.from_name("huggingface-secret")], + timeout=1200, + scaledown_window=300, # stay warm 5 min after a request to mask cold starts (cheap A10G — idle burn is low). + min_containers=0, # true scale-to-zero: $0 when idle (open-ended judging window — never pre-warm-and-forget). +) +@modal.concurrent(max_inputs=8) +@modal.web_server(port=VLLM_PORT, startup_timeout=900) +def serve(): + """Launch vLLM's OpenAI-compatible server for translation calls.""" + import subprocess + + cmd = [ + "vllm", + "serve", + MODEL, + # Estimate descriptions are short; a small context keeps VRAM comfortable. + "--max-model-len", + "8192", + "--max-num-seqs", + "8", + "--tensor-parallel-size", + "1", + "--port", + str(VLLM_PORT), + "--host", + "0.0.0.0", + ] + subprocess.Popen(" ".join(cmd), shell=True) diff --git a/quillwright/backends/modal_omni_app.py b/quillwright/backends/modal_omni_app.py new file mode 100644 index 0000000000000000000000000000000000000000..66aee60a4eab04d7203d1a7eeae118652d290338 --- /dev/null +++ b/quillwright/backends/modal_omni_app.py @@ -0,0 +1,101 @@ +"""Modal deployment of the Quillwright Best-Stack Perception + Audio: Nemotron Omni. + +ADR-0009: Omni (nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning, 31B total / 3B active +MoE) is the selectable Best-Stack *alternative* for Perception — MiniCPM-V stays the +Private-Stack default (protects the OpenBMB track). Because Omni is omnimodal +(image + audio + text), this ONE deployment also serves the Best-Stack Audio role: +the client (`backends/modal.py`) sends photos as image_url parts and voice notes as +input_audio parts to the same /v1/chat/completions endpoint. + +Served with vLLM (>=0.20 per NVIDIA's Omni recipe, +https://recipes.vllm.ai/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) in the +FP8 variant (32.8 GB — same L40S class the brain app proved; encoders stay BF16). + +NOT deployed yet (build-only; deploys touch your Modal account + credits): + modal deploy quillwright/backends/modal_omni_app.py +Then point the app at the printed URL (per-role opt-in — the brain URL stays separate): + export FF_BACKEND=modal + export FF_MODAL_OMNI_URL="https://<...>.modal.run" + +Deploy-time caveats (verify on first run, cheaply, ONE request at a time): + - VRAM: 32.8 GB weights + BF16 encoders on a 48 GB L40S is tighter than the brain; + --max-model-len is kept small (32K) for headroom. If engine init OOMs, bump to + gpu="A100-80GB" for the verification run only. + - The browser records voice notes as webm; Omni's recipe lists wav/mp3. The local + transformers path handles webm today — if Omni rejects it, transcode to wav in + /api/transcribe before the Modal call (do not silently drop audio). +""" + +import modal + +MODEL = "nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8" # ADR-0009 Best-Stack Omni. +VLLM_PORT = 8000 +image = ( + # CUDA devel base (nvcc) for FlashInfer's FP8 MoE JIT — same backbone/arch lesson + # as the brain app (debian_slim crashes engine init). + modal.Image.from_registry("nvidia/cuda:12.8.1-devel-ubuntu22.04", add_python="3.12") + # vllm[audio] pulls the audio decoders the Omni recipe requires. + .pip_install("vllm[audio]==0.20.0", "huggingface_hub", "flashinfer-python") + .env( + { + "VLLM_USE_FLASHINFER_MOE_FP8": "1", + "VLLM_FLASHINFER_MOE_BACKEND": "throughput", + # Weights go to the mounted cache volume (NOT ~/.cache — Modal refuses to + # mount a volume over a non-empty dir). + "HF_HOME": "/cache", + } + ) +) + +# Shared with the brain app: pull weights once, not every cold start. +hf_cache = modal.Volume.from_name("quillwright-hf-cache", create_if_missing=True) + +app = modal.App("quillwright-omni") + + +@app.function( + image=image, + gpu="L40S", # FP8 weights 32.8GB; encoders BF16. 48GB with a small context fits. + volumes={"/cache": hf_cache}, + # HF_TOKEN for the weight download (harmless if ungated; required if the NVIDIA + # repo is gated). Same secret across all four apps. + secrets=[modal.Secret.from_name("huggingface-secret")], + timeout=1200, + scaledown_window=120, # warm 2 min after a request (masks cold starts; limits idle L40S burn). + min_containers=0, # true scale-to-zero: $0 when idle (open-ended judging window — never pre-warm-and-forget). +) +@modal.concurrent(max_inputs=8) +@modal.web_server(port=VLLM_PORT, startup_timeout=900) +def serve(): + """Launch vLLM's OpenAI-compatible server with NVIDIA's Omni recipe.""" + import subprocess + + cmd = [ + "vllm", + "serve", + MODEL, + "--trust-remote-code", + # One photo or one short voice note per request — not the recipe's video load. + "--max-model-len", + "32768", + "--max-num-seqs", + "8", + # The cmd is joined into ONE shell string (shell=True, like the brain app): + # single-quote the JSON and keep it space-free so it stays one shell token. + "--limit-mm-per-prompt", + '\'{"image":4,"audio":1,"video":0}\'', + "--kv-cache-dtype", + "fp8", + "--tensor-parallel-size", + "1", + "--enable-auto-tool-choice", + "--tool-call-parser", + "qwen3_coder", + "--reasoning-parser", + "nemotron_v3", # built into vLLM >=0.20 (no plugin file, unlike the brain app). + "--port", + str(VLLM_PORT), + "--host", + "0.0.0.0", + ] + subprocess.Popen(" ".join(cmd), shell=True) diff --git a/quillwright/backends/modal_parse_app.py b/quillwright/backends/modal_parse_app.py new file mode 100644 index 0000000000000000000000000000000000000000..00a43376c60270cb9845db16feb90837f496271b --- /dev/null +++ b/quillwright/backends/modal_parse_app.py @@ -0,0 +1,139 @@ +"""Modal deployment of Nemotron Parse — Document Capture extraction (ADR-0011). + +Nemotron Parse is the Extraction Model Role (ADR-0009): a document image → structured +text + tables. Local de-risk on Apple Silicon FAILED (>30GB RAM, 5+ min/doc — the +C-RADIO encoder + mBART decoder thrash without a GPU), so it runs on Modal instead. + +Unlike the brain (text, vLLM OpenAI API), Parse is VISUAL — there is no standard +"parse image → structured output" OpenAI route, so this exposes a CUSTOM endpoint: +POST a base64 image, get back the parsed blocks. + +The model's raw output is a token-encoded string of (bbox, text, class) triples, e.g. +`Dual run capacitor`. Two repo-shipped files +turn that into structured blocks: `postprocessing.py` (extract_classes_bboxes, +transform_bbox_to_original, postprocess_text) and `latex2html.py` (table conversion). +They are NOT loaded by trust_remote_code — we download them with hf_hub_download and +import them. (Verified against the v1.2 model card, 2026-06-11; see ADR-0011.) + +This endpoint runs the full postprocessing server-side and returns clean blocks +[{class, bbox, text}], so the on-device client (backends/parse.py) stays light. + +Cost stance (ADR-0011): proven as a demo capability — NOT wired to the live Space +(no continuous spend). Parse is ~1GB, so a cheap T4 is plenty (no L40S needed). + +Deploy: + modal deploy quillwright/backends/modal_parse_app.py + export FF_MODAL_PARSE_URL="https://<...>.modal.run" +""" + +import modal + +MODEL = "nvidia/NVIDIA-Nemotron-Parse-v1.2" + +# Task prompt from the v1.2 model card — the FULL four-token form. v1.1's three-token +# prompt produces "significantly degraded" results on v1.2, per the card. +TASK_PROMPT = "" + +# Repo-shipped postprocessing files (standalone modules, NOT trust_remote_code). +POSTPROC_FILES = ("postprocessing.py", "latex2html.py") + +image = ( + modal.Image.debian_slim(python_version="3.12") + .pip_install( + # Pins from the v1.2 model card — looser versions risk the C-RADIO/mBART + # custom code breaking on a transformers API change. + "torch", + "transformers==5.6.1", + "accelerate==1.12.0", + "timm==1.0.22", + "albumentations==2.0.8", + # latex2html.py needs BeautifulSoup for table HTML conversion. + "beautifulsoup4", + "huggingface_hub", + "pillow", + "fastapi[standard]", + ) + .env({"HF_HOME": "/cache"}) +) + +hf_cache = modal.Volume.from_name("quillwright-hf-cache", create_if_missing=True) + +app = modal.App("quillwright-parse") + + +@app.cls( + image=image, + gpu="T4", # Parse is ~1GB; a T4 is plenty (and the cheapest GPU). + volumes={"/cache": hf_cache}, + # HF_TOKEN for the weight download (harmless if ungated; required if the NVIDIA + # repo is gated). Same secret across all four apps. + secrets=[modal.Secret.from_name("huggingface-secret")], + timeout=600, + scaledown_window=240, # cheap T4 — modest idle burn, kept warm a bit longer for multi-doc capture. + min_containers=0, # true scale-to-zero: $0 when idle (open-ended judging window — never pre-warm-and-forget). +) +@modal.concurrent(max_inputs=4) +class Parser: + @modal.enter() + def load(self): + """Load the model + its repo-shipped postprocessing once per container.""" + import importlib.util + import sys + + import torch + from huggingface_hub import hf_hub_download + from transformers import AutoModel, AutoProcessor, GenerationConfig + + # Pull postprocessing.py + latex2html.py from the model repo and put their dir + # on sys.path (postprocessing imports latex2html by name, so both must resolve). + for fname in POSTPROC_FILES: + module_dir = hf_hub_download(MODEL, fname).rsplit("/", 1)[0] + if module_dir not in sys.path: + sys.path.insert(0, module_dir) + spec = importlib.util.spec_from_file_location( + "nemotron_postprocessing", hf_hub_download(MODEL, "postprocessing.py") + ) + self.pp = importlib.util.module_from_spec(spec) + spec.loader.exec_module(self.pp) + + self.model = ( + AutoModel.from_pretrained(MODEL, trust_remote_code=True, dtype=torch.bfloat16) + .to("cuda") + .eval() + ) + self.processor = AutoProcessor.from_pretrained(MODEL, trust_remote_code=True) + self.gen_config = GenerationConfig.from_pretrained(MODEL, trust_remote_code=True) + + @modal.fastapi_endpoint(method="POST") + def parse(self, payload: dict): + """POST {image: base64} -> {blocks: [{class, bbox, text}], raw: }. + + Runs the full repo postprocessing server-side: decode → extract triples → + rescale bboxes to the original image → format text (tables as markdown). + """ + import base64 + import io + + from PIL import Image + + data = payload.get("image", "") + if "," in data and data.strip().startswith("data:"): + data = data.split(",", 1)[1] + img = Image.open(io.BytesIO(base64.b64decode(data))).convert("RGB") + + inputs = self.processor( + images=[img], text=TASK_PROMPT, return_tensors="pt", add_special_tokens=False + ).to("cuda") + outputs = self.model.generate(**inputs, generation_config=self.gen_config) + raw = self.processor.batch_decode(outputs, skip_special_tokens=True)[0] + + classes, bboxes, texts = self.pp.extract_classes_bboxes(raw) + bboxes = [self.pp.transform_bbox_to_original(b, img.width, img.height) for b in bboxes] + texts = [ + self.pp.postprocess_text(t, cls=c, table_format="markdown", text_format="markdown") + for t, c in zip(texts, classes) + ] + blocks = [ + {"class": c, "bbox": list(b), "text": t} for c, b, t in zip(classes, bboxes, texts) + ] + return {"blocks": blocks, "raw": raw} diff --git a/quillwright/backends/parse.py b/quillwright/backends/parse.py new file mode 100644 index 0000000000000000000000000000000000000000..83ee1aa68aa023d05eaa7b774135745c49bb397b --- /dev/null +++ b/quillwright/backends/parse.py @@ -0,0 +1,156 @@ +"""ParseModel: the Document Capture client (ADR-0011), backed by Nemotron Parse on Modal. + +Parse is the Extraction Model Role (ADR-0009). It is VISUAL, so it does not fit the +vLLM OpenAI route the brain uses — it has its own Modal endpoint (modal_parse_app.py) +that POSTs a base64 image and returns structured blocks [{class, bbox, text}], having +already run the model's repo-shipped postprocessing server-side. + +This client is the on-device half: it turns those blocks into the two things the +Estimate pipeline understands, per ADR-0011 decision C — + + - a priced table row -> ProposedLineItem (human confirms the price via Agent Pause) + - everything else -> Observation(kind="text") (flows straight through) + +The document is the *source*, but any price it read is *proposed*, never a fact: the +human gates every customer-facing number (Facts-from-Tools, ADR-0004). + +The base URL (printed by `modal deploy modal_parse_app.py`) comes from FF_MODAL_PARSE_URL. +""" + +import base64 +import os +import re + +import requests + +from quillwright.models import Observation, ProposedLineItem + +# Parse's table content arrives as markdown. A money cell looks like "$42.50", +# "$1,250.00", or "42.50" — capture the numeric value, tolerating $ and thousands commas. +_MONEY = re.compile(r"\$?\s*([\d,]+\.\d{1,2}|\d[\d,]*)") +# A leading integer/decimal in a cell is the quantity ("2", "4", "1.5"). +_QTY = re.compile(r"^\s*(\d+(?:\.\d+)?)\s*$") + + +class ParseModel: + name = "nemotron-parse-v1.2" + + def __init__(self, base_url: str | None = None, timeout: float = 180.0): + self._base = (base_url or os.environ.get("FF_MODAL_PARSE_URL", "")).rstrip("/") + if not self._base: + raise RuntimeError( + "FF_MODAL_PARSE_URL is not set — deploy modal_parse_app.py and export " + "the URL it prints (see backends/modal_parse_app.py)." + ) + self._timeout = timeout + + def parse_document(self, image_path: str) -> tuple[list[Observation], list[ProposedLineItem]]: + """Read a document image; return (observations, proposed_line_items).""" + with open(image_path, "rb") as fh: + b64 = base64.b64encode(fh.read()).decode("ascii") + resp = requests.post(f"{self._base}/parse", json={"image": b64}, timeout=self._timeout) + resp.raise_for_status() + blocks = resp.json().get("blocks", []) + return blocks_to_pipeline(blocks) + + +def blocks_to_pipeline( + blocks: list[dict], +) -> tuple[list[Observation], list[ProposedLineItem]]: + """Split Parse blocks into Observations + Proposed Line Items (ADR-0011). + + Pure function (no network) so it can be unit-tested against real Parse output. + Table blocks become priced ProposedLineItems where a price is present; their + non-priced rows and every non-table block become text Observations. + """ + observations: list[Observation] = [] + proposed: list[ProposedLineItem] = [] + + for block in blocks: + cls = block.get("class", "") + text = (block.get("text") or "").strip() + if not text: + continue + if cls == "Table": + rows_obs, rows_items = _table_to_items(text) + observations.extend(rows_obs) + proposed.extend(rows_items) + else: + observations.append(Observation(kind="text", text=text)) + return observations, proposed + + +def _table_to_items( + table_md: str, +) -> tuple[list[Observation], list[ProposedLineItem]]: + """Parse a markdown table into priced line items, gating on a real price. + + A row with a money cell -> ProposedLineItem (description = its longest text + cell, quantity from a bare-number cell if present, rate from the price). + A row with no price falls back to an Observation so nothing is silently dropped. + """ + observations: list[Observation] = [] + proposed: list[ProposedLineItem] = [] + + for line in table_md.splitlines(): + line = line.strip() + if not line or set(line) <= {"|", "-", " ", ":"}: + continue # blank or the header separator row (|---|---|) + cells = [c.strip() for c in line.strip("|").split("|")] + cells = [c for c in cells if c != ""] + if not cells: + continue + + price = _row_price(cells) + if price is None: + observations.append(Observation(kind="text", text=" ".join(cells))) + continue + + # Skip a header row that happens to contain the literal word "price" but no + # numeric description (e.g. "| Item | Qty | Price |" has no money cell, so it + # already fell through above — this guards a row that is only labels). + description = _row_description(cells) + if not description: + observations.append(Observation(kind="text", text=" ".join(cells))) + continue + + proposed.append( + ProposedLineItem( + description=description, + quantity=_row_quantity(cells), + rate=price, + source_text=" ".join(cells), + ) + ) + return observations, proposed + + +def _row_price(cells: list[str]) -> float | None: + """The price of a row = the money value in its last cell that has one. + + Scanning right-to-left picks the line *total* / unit price over an earlier + quantity that also matches the number pattern. + """ + for cell in reversed(cells): + if "$" in cell or "." in cell: + m = _MONEY.search(cell) + if m: + return float(m.group(1).replace(",", "")) + return None + + +def _row_quantity(cells: list[str]) -> float: + """A standalone integer/decimal cell is the quantity; default 1.""" + for cell in cells: + m = _QTY.match(cell) + if m: + return float(m.group(1)) + return 1.0 + + +def _row_description(cells: list[str]) -> str: + """The description is the longest cell that is neither a price nor a bare number.""" + candidates = [ + c for c in cells if not _QTY.match(c) and "$" not in c and not _MONEY.fullmatch(c) + ] + return max(candidates, key=len) if candidates else "" diff --git a/quillwright/brain_loop.py b/quillwright/brain_loop.py index ddd4f77b9bb7acf8765c1cdf82d8df9aced43966..e418d81faa4d145952cb92ae0b1f015ce1474cf3 100644 --- a/quillwright/brain_loop.py +++ b/quillwright/brain_loop.py @@ -38,6 +38,9 @@ def run_brain( """Drive the model to build line items. Returns (line_items, trace, pause-or-None).""" line_items: list[LineItem] = [] trace: list[TraceStep] = [] + # The actual model name (e.g. "nemotron-3-nano:4b" or "StubModel") so the trace + # truthfully shows which model answered — no guessing whether a model was hit. + brain_name = getattr(model, "name", "brain") messages = [ {"role": "system", "content": SYSTEM}, { @@ -65,7 +68,9 @@ def run_brain( return line_items, trace, {"item": result["item"]} if result["status"] == "done": done = True - trace.append(TraceStep(action="finish", model="brain", detail="estimate complete")) + trace.append( + TraceStep(action="finish", model=brain_name, detail="estimate complete") + ) break if result["status"] == "added": line_items.append(result["line_item"]) @@ -73,7 +78,7 @@ def run_brain( trace.append( TraceStep( action="add_priced_item", - model="brain", + model=brain_name, detail=f"{li.quantity:g} x {li.description} -> {li.subtotal}", ) ) diff --git a/quillwright/estimate_store.py b/quillwright/estimate_store.py new file mode 100644 index 0000000000000000000000000000000000000000..c8fd6bcd556f417132571e2a2e475d38d54f1492 --- /dev/null +++ b/quillwright/estimate_store.py @@ -0,0 +1,91 @@ +"""EstimateStore: per-Account persistence of Saved Estimates + Refinement Threads +(ADR-0013). + +JSON-on-disk behind a small, swappable interface (save / list / load / delete), +keyed by `account_id`. Separate from Episodic Memory (memory.py), which stays a +pure append-only Recall corpus. One file per estimate at +`//.json`. Ids are zero-padded sequence numbers (deterministic +and offline-friendly, like Memory's sequence ids — no uuid/wall-clock). + +Durable locally; on the hosted Space `path` points at a per-session temp dir so +visitors never see each other's data (the Space is one container / one account). +""" + +import json +import os + +ACCOUNT_ID = os.environ.get("FF_ACCOUNT_ID", "demo") +STORE_PATH = os.environ.get("FF_ESTIMATE_STORE", "/tmp/quillwright_estimates") + + +class EstimateStore: + def __init__(self, path: str | None = None, account_id: str | None = None): + # Read the env at construction time (not import time) so a test/launch that + # sets FF_ESTIMATE_STORE / FF_ACCOUNT_ID before building the store wins. + path = path or os.environ.get("FF_ESTIMATE_STORE", "/tmp/quillwright_estimates") + account_id = account_id or os.environ.get("FF_ACCOUNT_ID", "demo") + self._dir = os.path.join(path, account_id) + self._account_id = account_id + + def _ensure_dir(self) -> None: + os.makedirs(self._dir, exist_ok=True) + + def _path(self, id: str) -> str: + return os.path.join(self._dir, f"{id}.json") + + def _next_id(self) -> str: + if not os.path.isdir(self._dir): + return "0001" + existing = [f[:-5] for f in os.listdir(self._dir) if f.endswith(".json")] + nums = [int(e) for e in existing if e.isdigit()] + return f"{(max(nums) + 1 if nums else 1):04d}" + + def save(self, estimate: dict, thread: list[dict], id: str | None = None) -> dict: + """Create (id=None) or update-in-place (id given) a Saved Estimate.""" + self._ensure_dir() + if id is None: + id = self._next_id() + rec = { + "id": id, + "account_id": self._account_id, + "estimate": estimate, + "thread": thread, + "saved_seq": int(id), + } + with open(self._path(id), "w") as f: + json.dump(rec, f, indent=2) + return rec + + def load(self, id: str) -> dict | None: + path = self._path(id) + if not os.path.isfile(path): + return None + with open(path) as f: + return json.load(f) + + def list_estimates(self) -> list[dict]: + """Saved estimates newest-first, each a list-row summary (id + title + total).""" + if not os.path.isdir(self._dir): + return [] + recs = [] + for fname in os.listdir(self._dir): + if not fname.endswith(".json"): + continue + with open(os.path.join(self._dir, fname)) as f: + rec = json.load(f) + est = rec.get("estimate", {}) + recs.append( + { + "id": rec["id"], + "job_title": est.get("job_title", "Estimate"), + "total": est.get("total"), + "saved_seq": rec.get("saved_seq", 0), + } + ) + recs.sort(key=lambda r: r["saved_seq"], reverse=True) + return recs + + def delete(self, id: str) -> None: + path = self._path(id) + if os.path.isfile(path): + os.remove(path) diff --git a/quillwright/memory.py b/quillwright/memory.py index 4a548be503c618f6957e255a3c7e49cd7a51bab9..e61823d2952c69cf3ef2e80a9a6a309bdd354d75 100644 --- a/quillwright/memory.py +++ b/quillwright/memory.py @@ -10,8 +10,12 @@ from collections import Counter class Memory: - def __init__(self, path: str): + def __init__(self, path: str, embedder=None): + # `embedder` (anything with .encode(text)->vector) turns Recall semantic: + # run vectors are cached at record time, only the query is embedded at recall + # time (ADR-0003). Without one, Recall stays keyword-only (unchanged default). self._path = path + self._embedder = embedder self._runs: list[dict] = [] self._load() @@ -20,22 +24,51 @@ class Memory: with open(self._path) as f: self._runs = json.load(f).get("runs", []) - def record_run(self, transcript: str, line_items: list[str]) -> None: - self._runs.append({"transcript": transcript, "line_items": list(line_items)}) + def record_run( + self, transcript: str, line_items: list[str], total: float | None = None + ) -> None: + run = {"transcript": transcript, "line_items": list(line_items), "total": total} + if self._embedder is not None: + # Cache the run's embedding now so recall only embeds the query. + run["embedding"] = list(self._embedder.encode(self._haystack(run))) + self._runs.append(run) self._save() + def recent(self, limit: int | None = None) -> list[dict]: + """Past runs newest-first, each tagged with a 1-based sequence id. + + The id is the record order (not a wall-clock time) so it is deterministic + and offline-friendly. `total` is None for runs recorded before totals existed. + """ + tagged = [{"id": i + 1, "total": r.get("total"), **r} for i, r in enumerate(self._runs)] + tagged.reverse() # newest first + return tagged[:limit] if limit is not None else tagged + def _save(self) -> None: os.makedirs(os.path.dirname(self._path) or ".", exist_ok=True) with open(self._path, "w") as f: json.dump({"runs": self._runs}, f, indent=2) def recall(self, query: str) -> list[dict]: + if self._embedder is not None: + return self._semantic_recall(query) q = query.strip().lower() scored = [(self._haystack(r).count(q), r) for r in self._runs] matches = [(score, r) for score, r in scored if score > 0] matches.sort(key=lambda sr: sr[0], reverse=True) return [r for _score, r in matches] + def _semantic_recall(self, query: str) -> list[dict]: + """Rank past runs by embedding cosine similarity to the query (ADR-0003). + + Reuses recall_eval's ranker so there is one cosine implementation. Runs + without a cached embedding (recorded before the embedder) fall to the bottom. + """ + from quillwright.recall_eval import semantic_ranker + + scored = [r for r in self._runs if r.get("embedding")] + return semantic_ranker(query, scored, self._embedder) + @staticmethod def _haystack(run: dict) -> str: return (run["transcript"] + " " + " ".join(run["line_items"])).lower() @@ -44,4 +77,9 @@ class Memory: """Learned per-tech defaults derived from recorded runs.""" counts = Counter(item for r in self._runs for item in r["line_items"]) common = [item for item, _ in counts.most_common()] - return {"common_items": common, "job_count": len(self._runs)} + revenue_total = round(sum(r["total"] for r in self._runs if r.get("total")), 2) + return { + "common_items": common, + "job_count": len(self._runs), + "revenue_total": revenue_total, + } diff --git a/quillwright/models.py b/quillwright/models.py index d2c4f9b88bff9f9a4b38ef871e110224b0559b05..ec032cb1700150d62e2da863d5727a293986c5e1 100644 --- a/quillwright/models.py +++ b/quillwright/models.py @@ -19,7 +19,11 @@ class LineItem(BaseModel): quantity: float unit: str rate: float - price_source: Literal["catalog", "user", "computed"] = "catalog" + # "document" = a price Parse read from a Document Capture that the human has + # confirmed verbatim through the Agent Pause (ADR-0011). The document is the + # source, but the number is still user-gated — it never enters an Estimate + # straight from the model. + price_source: Literal["catalog", "user", "computed", "document"] = "catalog" @computed_field @property @@ -27,6 +31,23 @@ class LineItem(BaseModel): return round(self.quantity * self.rate, 2) +class ProposedLineItem(BaseModel): + """A priced row Parse read from a Document Capture, awaiting human confirmation. + + Not an Estimate line yet: per Facts-from-Tools (ADR-0004) + ADR-0011, a price + a model read off a document is *proposed*, not a fact. The human confirms or + edits it via the Agent Pause; on confirm it becomes a LineItem with + price_source="document". `source_text` is the raw cell the price came from, so + the human can spot an OCR slip (e.g. "$42.50" misread as "$425.0"). + """ + + description: str + quantity: float = 1.0 + unit: str = "ea" + rate: float + source_text: str = "" + + class Estimate(BaseModel): job_title: str line_items: list[LineItem] = [] diff --git a/quillwright/pairing.py b/quillwright/pairing.py new file mode 100644 index 0000000000000000000000000000000000000000..21cf81718843b1fe3c3da06757ec1f54c5ec2fb8 --- /dev/null +++ b/quillwright/pairing.py @@ -0,0 +1,63 @@ +"""Phone-capture pairing channel (Tier 3). + +The desktop creates a pairing → gets a short ``code`` → renders a QR of +``/m/``. The phone opens that mobile capture page, sends a capture +(photo server-path(s) +/or a transcript) to the pairing, and the desktop — which polls +``/api/pair/`` — picks it up and forges live on screen. + +In-process and demo-scoped: a dict of ``code -> pending capture``. The capture is +delivered exactly once (the desktop forges it a single time), then cleared. A durable / +multi-session transport is a post-hackathon swap behind this same tiny interface — the +same shape as ``pdf_links`` and the ``EstimateStore``. + +Codes come from ``os.urandom`` (URL-safe, unguessable enough for a demo; not +``random``/wall-clock, so nothing here depends on the global RNG or the clock). +""" + +import base64 +import os + +# code -> {"capture": }. Process-local; lives for the server's lifetime. +_PAIRINGS: dict[str, dict] = {} + + +def _new_code() -> str: + """A short, URL-safe pairing code (6 chars, ~36 bits).""" + return base64.urlsafe_b64encode(os.urandom(5)).decode().rstrip("=")[:6] + + +def create() -> str: + """Open a new pairing and return its code (desktop side).""" + code = _new_code() + while code in _PAIRINGS: # vanishingly unlikely; keep codes unique anyway + code = _new_code() + _PAIRINGS[code] = {"capture": None} + return code + + +def is_valid(code: str) -> bool: + """Whether ``code`` is a live pairing (the mobile page checks before capturing).""" + return code in _PAIRINGS + + +def submit(code: str, capture: dict) -> bool: + """Phone side: hand a capture to the paired desktop. False if the code is unknown.""" + if code not in _PAIRINGS: + return False + _PAIRINGS[code]["capture"] = capture + return True + + +def poll(code: str) -> dict | None: + """Desktop side: take the pending capture (delivered once), or None if none waiting.""" + pairing = _PAIRINGS.get(code) + if not pairing or pairing["capture"] is None: + return None + capture = pairing["capture"] + pairing["capture"] = None # consume: the desktop forges it exactly once + return capture + + +def reset() -> None: + """Drop all pairings (tests).""" + _PAIRINGS.clear() diff --git a/quillwright/recall_eval.py b/quillwright/recall_eval.py new file mode 100644 index 0000000000000000000000000000000000000000..063ed834d9bab1cc1296aff744c4cb1cdabdf49b --- /dev/null +++ b/quillwright/recall_eval.py @@ -0,0 +1,98 @@ +"""Score Recall: given a query, does the ranker put the right past run first? + +The viability question for semantic Recall (ADR-0003) is "does meaning-based +re-ranking beat keyword matching on queries where the words differ but the intent +matches" (e.g. query "coolant" should find a run that says "refrigerant"). This +module measures recall@1 for any ranker, plus a keyword baseline, so we can report +a measured keyword-vs-semantic delta for the Field Notes write-up. + +A `ranker` is `fn(query, runs) -> runs_ranked_best_first`. The keyword baseline +ranks by literal token overlap; the semantic ranker (embedding cosine) drops in +with the same signature once the embedder lands — no change here. +""" + +import json + + +def load_recall_cases(path: str) -> dict: + with open(path) as f: + return json.load(f) + + +def _haystack(run: dict) -> str: + return (run["transcript"] + " " + " ".join(run["line_items"])).lower() + + +def keyword_overlap(query: str, run: dict) -> int: + """How many query tokens appear literally in the run (the keyword signal).""" + hay = _haystack(run) + return sum(1 for tok in query.lower().split() if tok in hay) + + +def keyword_ranker(query: str, runs: list[dict]) -> list[dict]: + """Baseline: rank by literal token overlap, ties keep corpus order (stable).""" + return sorted(runs, key=lambda r: keyword_overlap(query, r), reverse=True) + + +def recall_at_1(queries: list[dict], corpus: list[dict], ranker) -> float: + """Fraction of queries whose gold run is ranked first by `ranker`. + + Each query is {"query": str, "gold_id": int}; runs are matched by "id". + """ + if not queries: + return 0.0 + hits = 0 + for q in queries: + ranked = ranker(q["query"], corpus) + if ranked and ranked[0]["id"] == q["gold_id"]: + hits += 1 + return round(hits / len(queries), 3) + + +def embed_corpus(corpus: list[dict], embedder) -> list[dict]: + """Attach a cached embedding to each run (mirrors record-time caching in prod). + + `embedder` is anything with `.encode(text) -> vector`; we embed the same haystack + (transcript + line items) the keyword path searches. + """ + out = [] + for r in corpus: + out.append({**r, "embedding": list(embedder.encode(_haystack(r)))}) + return out + + +def _cosine(a, b) -> float: + import numpy as np + + va, vb = np.asarray(a, dtype=float), np.asarray(b, dtype=float) + na, nb = np.linalg.norm(va), np.linalg.norm(vb) + if na == 0 or nb == 0: + return 0.0 + return float(va @ vb / (na * nb)) + + +def semantic_ranker(query: str, runs: list[dict], embedder) -> list[dict]: + """Rank runs by cosine similarity between the query embedding and each run's + cached embedding. Only the query is embedded at call time (torch out of the hot + path); run embeddings come from embed_corpus / record-time caching (ADR-0003).""" + qv = embedder.encode(query) + return sorted(runs, key=lambda r: _cosine(qv, r.get("embedding", [])), reverse=True) + + +def keyword_recall_at_1(queries: list[dict], corpus: list[dict]) -> float: + """recall@1 using the keyword baseline ranker. + + Note: a query with zero literal overlap leaves the corpus in its original + order, so the first run is a non-match — that miss is the point (it's where + semantic recall should win). + """ + + def ranker(query, runs): + ranked = keyword_ranker(query, runs) + # if nothing overlaps at all, treat it as no result (a guaranteed miss), + # rather than crediting whatever happened to sort first. + if ranked and keyword_overlap(query, ranked[0]) == 0: + return [] + return ranked + + return recall_at_1(queries, corpus, ranker) diff --git a/quillwright/resolver.py b/quillwright/resolver.py index 48b7265ca5db8bd8900af1543780128f2a02a6a7..faef440e0ef628aca90fa0569d8ce3b6a5170c37 100644 --- a/quillwright/resolver.py +++ b/quillwright/resolver.py @@ -32,16 +32,18 @@ class StubModel: # Which concrete model fills each role per Mode. Real backends wired later (ADR-0005). -# Display labels per role (used by the stub backend). +# Display labels per role (used by the stub backend). These are LABELS ONLY — the +# real on-device models are in OLLAMA_TAGS below (the brain is Nemotron, not gpt-oss; +# ADR-0009 superseded the gpt-oss mapping). PRIVATE_STACK = { - "perception": "MiniCPM-V-4.6", - "audio": "whisper-local", - "brain": "gpt-oss-20b", + "perception": "MiniCPM-V", + "audio": "Cohere-Transcribe", + "brain": "Nemotron-3-Nano-4B", } BEST_STACK = { "perception": "Nemotron-3-Nano-Omni", "audio": "Nemotron-3-Nano-Omni", - "brain": "gpt-oss-20b", + "brain": "Nemotron-3-Nano-30B", } # Actual locally-available Ollama tags per role (what we really run on-device). @@ -51,6 +53,114 @@ OLLAMA_TAGS = { "multilingual": "aya", } +# Roles served on Modal (ADR-0005 hosted compute, ADR-0009 Best Stack). Labels are +# informational; each role's modal app pins the real repo id (see backends/modal.py +# ROLE_ENDPOINTS). Perception + audio share the ONE Omni deployment (omnimodal). +MODAL_ROLES = { + "brain": "nemotron-3-nano-30b-a3b", + "perception": "nemotron-3-nano-omni-30b-a3b", + "audio": "nemotron-3-nano-omni-30b-a3b", + "multilingual": "aya-expanse-8b", +} + + +def brain_resolver() -> "ModelResolver": + """The resolver for the agent brain, chosen by env (one source of truth). + + FF_BACKEND=modal -> Best-Stack brain (Nemotron 30B) hosted on Modal (ADR-0009). + otherwise -> Private-Stack brain (Nemotron 4B) via local Ollama. + + The brain is special: FF_BACKEND=modal *means* "brain on Modal", so a missing + FF_MODAL_BRAIN_URL fails LOUD in ModalModel rather than silently downgrading. + Other roles opt in per-URL via modal_resolver_if_configured(). + """ + import os + + if os.environ.get("FF_BACKEND") == "modal": + return ModelResolver(mode="best", backend="modal") + return ModelResolver(mode="private", backend="ollama") + + +def modal_resolver_if_configured(role: str) -> "ModelResolver | None": + """A Best-Stack Modal resolver for `role`, or None when the local path should run. + + Per-role opt-in (mirrors FF_MODAL_PARSE_URL): FF_BACKEND=modal moves ONLY the + brain; perception/audio/multilingual each ride Modal IFF their own URL env is + also set. Callers fall back to their existing local/stub path on None — turning + on the hosted brain never breaks a role whose GPU app isn't deployed. + """ + import os + + if os.environ.get("FF_BACKEND") != "modal": + return None + from quillwright.backends.modal import ROLE_ENDPOINTS + + if role not in ROLE_ENDPOINTS or not os.environ.get(ROLE_ENDPOINTS[role][0]): + return None + return ModelResolver(mode="best", backend="modal") + + +# Human-facing labels for the Modal Best-Stack models (the resolver tags above are +# terse; these read well in the UI badge). +MODAL_LABELS = { + "brain": "Nemotron-3-Nano-30B", + "perception": "Nemotron-Omni-30B", + "audio": "Nemotron-Omni-30B", + "multilingual": "Aya-Expanse-8B", +} + +# Roles shown in the UI badge (audio is omitted — it has no always-on indicator and +# rides the same deployment as perception). +_BADGE_ROLES = ("brain", "perception", "multilingual") + + +def active_models() -> dict: + """Where each Model Role actually resolves right now, for the UI badge + banner. + + Reads the same env the resolvers do (FF_REAL_MODELS / FF_BACKEND / FF_MODAL_*_URL) + so it is one honest source of truth — not a guess. Returns + {"mode": , "roles": {role: