Spaces:
Running
Running
| # Implementation Status — live tracker | |
| > **Both agents:** update this file as you complete checklist items. Tick boxes (`[x]`), bump the "Current stage" lines at top, add a `_log_` entry at the bottom. This is the single source of truth for "where we are." If you finish work and don't update this file, the next session can't pick up. | |
| --- | |
| **Last updated:** 2026-05-17 | |
| **Current FE stage:** 0 → 1 (handoff in progress) | |
| **Current BE stage:** Stage 6 ✅ DONE; Stage 7 next (Operations + Evaluation writes) | |
| **Contract version:** v2 (CONTRACT.md revised 2026-05-17 — Express proxy + cost monitoring) | |
| --- | |
| ## Two parallel tracks, one shared contract | |
| Two AI agents work in parallel, each in its own repo. They never edit each other's code; they share only `CONTRACT.md` and this file. | |
| | Track | Repo | Branch | Working dir for the agent | | |
| |---|---|---|---| | |
| | Frontend | `G:\pythoncodenew\Side projects\Reli\Patristic-AI-Fe` | `main` | repo root | | |
| | Backend | `G:\pythoncodenew\Side projects\Reli\Patristic_Arabic_Library` | `rewrite/api` (must be created before BE work starts) | repo root | | |
| Each agent reads: | |
| 1. `CONTRACT.md` (~280 lines, authoritative wire contract — read fully) | |
| 2. Its build brief: `FRONTEND_BUILD.md` or `BACKEND_BUILD.md` | |
| 3. This file — for "where we are" and "what's next" | |
| Each agent updates **only this file** when reporting progress. They do not edit each other's brief. | |
| --- | |
| ## How to use this file | |
| - Mark a checkbox `[x]` when the item is complete *and* verified against its observability gate. | |
| - A stage moves from `🚧 IN PROGRESS` to `✅ DONE` only when every checkbox in it is `[x]` AND the stage gate passes. | |
| - The user (or the orchestrator agent) confirms gate-pass before flipping the stage flag at the top of this file. | |
| - New items discovered mid-stage are added inside the stage they belong to, not the current stage, unless they're blockers. | |
| --- | |
| ## Frontend track | |
| ### Stage 0 — AI Studio scaffolding ✅ DONE (verified 2026-05-17) | |
| Already shipped by Google AI Studio before either agent started. Don't redo any of it. | |
| - [x] Vite 6 + React 19 + react-router-dom 7 + react-query + react-hook-form + zod + shadcn (base-nova) + Tailwind v4 set up. | |
| - [x] Express `server.ts` with Vite middleware + mock API routes + cookie parsing. | |
| - [x] Full route inventory (`App.tsx`): Dashboard, Login, About, Research/{Search, Compare, Allusions, History, RunDetail}, Library/{Browse, AddBook, Labels, BookDetail, Inspect, JobWatcher}, Operations/{Costs, Indexes, Jobs, Processing, Tools, Config}, Evaluation/{Run, Golden, History, RunDetail}. | |
| - [x] Search page wires `POST /query → jobId → SSE → done → runId in URL`. | |
| - [x] shadcn `button.tsx` primitive scaffolded. | |
| **Known issues left for Stage 1 to fix:** sidebar uses raw `<a href>` (full page reloads); no central API client; no `useSse` hook (inlined in Search); no AuthContext; no CSRF header on writes; only one shadcn primitive installed. | |
| ### Stage 1 — Foundation polish 🚧 IN PROGRESS | |
| Goal: make the scaffolding production-shaped before building more pages on top of it. | |
| - [ ] **Sidebar navigation:** replace every `<a href="..." class="...">` in `App.tsx` with `<NavLink to="..." className="...">` from `react-router-dom`. Active-route styling via `NavLink`'s `isActive` callback. Verify by clicking — no full page reload. | |
| - [ ] **shadcn primitives needed for Phase 1.** Run `npx shadcn@latest add card input label form select dialog toast tabs badge separator collapsible dropdown-menu tooltip alert avatar checkbox skeleton`. Verify each `.tsx` lands in `components/ui/`. | |
| - [ ] **API client (`lib/api/client.ts`).** Typed `apiFetch<T>(path, init)` that: prepends `/api`, sets `Content-Type: application/json` on JSON bodies, sets `X-Requested-With: XMLHttpRequest` on non-GET, decodes `{ code, message, details }` on 4xx/5xx and throws a typed `ApiError`. Has `apiGet`, `apiPost`, `apiPatch`, `apiDelete` helpers. **No bare `fetch()` calls anywhere else in the app.** | |
| - [ ] **SSE hook (`lib/sse/use-sse.ts`).** Generic `useSse<TEvent>(url: string | null, options?: { enabled?: boolean })` returning `{ events, isOpen, lastEventId, reconnect, error }`. Auto-closes on unmount. Ignores `event: ping`. Extract Search.tsx's inline EventSource into this hook. | |
| - [ ] **Auth (`lib/auth/auth-context.tsx`).** `AuthProvider` reading `GET /api/auth/me` on mount; exposing `{ user, login, logout, isLoading }`. `<ProtectedRoute>` wrapper redirects to `/login?next=...`. Wire into `App.tsx` Layout. | |
| - [ ] **Toast host.** Global `<Toaster />` from shadcn. ApiError handler in client.ts calls `toast.error(error.message)` automatically; routes can suppress if they handle inline. | |
| - [ ] **Type stub (`lib/api/types.ts`).** Hand-write DTOs from `FRONTEND_BUILD.md §3.5`. Add `// AUTOGENERATED ONCE BE IS LIVE — DO NOT EDIT` at top. Drop in when `openapi.json` exists. | |
| - [ ] **Format helpers (`lib/formatting/`).** `formatCost(usd: number)`, `formatDuration(ms: number)`, `formatPageLabel(bookId, pdfPage, printedPage?)` — server-formatted strings preferred; helpers only for cases the API doesn't pre-format. | |
| - [ ] **Status indicator in sidebar.** Replicate Streamlit's "active ingest" indicator: subscribe to `GET /api/system/events` SSE; render a chip with active job count and a list of running job ids (click → `/operations/jobs/[jobId]`). | |
| **Stage 1 gate (must all pass before moving on):** | |
| 1. Click every sidebar link — no full page reload (no `index.html` refetch in devtools network tab). | |
| 2. Trigger a 401 (e.g., delete the cookie in devtools, click Search): user is redirected to `/login?next=/research/search`. | |
| 3. Open Search page, submit a query, observe SSE working through `useSse` hook (not inline). Refresh during the run — `?job=...` in URL keeps the stream attached. | |
| 4. `npm run lint` (tsc --noEmit) passes with zero errors. | |
| 5. **Cost monitoring placeholder:** the cost panel placeholder ("$0.00 — no calls yet") renders on Search after a mock-server completion. Wiring exists. | |
| ### Stage 2 — Search page polished 📋 PENDING | |
| Anchored to FRONTEND_PRD.md Phase 1. | |
| - [ ] **AdvancedModels expander** on Search: model picker for generation/classify/judge, `topK`, `forceMode`, filters (tradition/era/language), `usePremium`. Source: `GET /api/tools?category=llm`. | |
| - [ ] **Recent queries** expander above the form on Search: `GET /api/history?page=search&user=me&limit=5`. Click → navigate to `/research/history/{runId}`. | |
| - [ ] **Cost panel** below the result: render `envelope.cost.{totalUsd, byStage, nCalls, durationMs}` from the final `done` event. Collapsed by default; click to expand → table of per-call rows from `GET /api/costs/llm-calls?sinceId=envelope.llmCallIdRange.firstId`. | |
| - [ ] **RunDetail page** (`/research/history/[runId]`): fetch `GET /api/query/runs/{runId}`, render by `envelope.mode` (LookupEnvelope rendering matches Search). "Re-run" button copies query + params to `/research/search?prefill={runId}`. | |
| - [ ] **History page** (`/research/history`): paginated list with filters (`page=`, `q=`, `pinned=`, `user=`). Pin/unpin actions. Cursor-based pagination. | |
| - [ ] **Login page**: real form against `POST /api/auth/login`, error envelope on bad credentials, redirect to `?next=` or `/`. | |
| - [ ] **About page**: tools registry table (`GET /api/tools`), live counts: indexed books, active collection, last query timestamp. | |
| - [ ] **CmdEnter shortcut** on Search submits. | |
| - [ ] **`<RtlText>` component** for Arabic text. Used everywhere Arabic appears (citation excerpts, query echo). | |
| - [ ] **`<CitationCard>` component** for citation display. Click → `/library/[bookId]/inspect?page={pdfPage}` in a new tab. | |
| **Stage 2 gate:** | |
| 1. Run a real search end-to-end against the mock; cost panel populated. Re-run from history works. Refresh during a query reattaches the SSE stream. | |
| 2. Pin a query from history; refresh; still pinned. Delete; refresh; gone. | |
| 3. Visit `/about`: tools registry table renders with at least 3 entries (mock can stub 3-5 tools). | |
| 4. Cmd/Ctrl+Enter on Search submits the form. | |
| 5. **Cost gate:** in mock mode, set the mock to return `envelope.cost.totalUsd = 0.0234`. The cost panel shows "$0.0234"; the cost breakdown rows match. | |
| ### Stage 3 — Library + Add Book + Ingest watcher 📋 PENDING | |
| FRONTEND_PRD.md Phase 2. | |
| - [ ] **`/library` Browse:** `GET /api/books?...` with filter form on URL params. Status badges, label chips, action menu per row. Bulk-select → "Reindex selected" fans out one POST per book with `body: { stages: ["indexing"] }`. | |
| - [ ] **`/library/[bookId]` Book detail:** tabs (Overview / History / Costs / Jobs). Endpoints: `GET /api/books/{id}`, `GET /api/processing-log?bookId=`, `GET /api/costs?groupBy=stage&bookId=`, `GET /api/jobs?subjectId=&limit=10`. Action menu (admin only): Edit, Re-ingest, Re-extract, Re-cleanup, Re-chunk, Re-index, Delete (with ConfirmDialog). | |
| - [ ] **`/library/[bookId]/inspect?page=N`:** three-pane (image | OCR | clean) with synced scroll. Page navigator. Diff between OCR and clean. | |
| - [ ] **`/library/[bookId]/jobs/[jobId]`:** SSE-watched ingest. Replays history on mount via `Last-Event-ID`. Cancel button (admin). Final summary on `done`. | |
| - [ ] **`/library/add` wizard:** | |
| - Step 1: URL paste OR file upload (`POST /api/uploads` → `{ uploadId }`) → `POST /api/books/probe` returns `ProbeResponse`. | |
| - Step 2: Edit guessed metadata; show suggested labels; extraction-mode toggle; cleanup toggle; sample pages displayed. | |
| - Step 3: **Cost preview** via `GET /api/costs/counterfactual?stage=ingest&...`. Confirm. | |
| - Step 4: `POST /api/books` (handle 409 inline with link to existing book); `POST /api/books/{id}/ingest`; redirect to `/library/[bookId]/jobs/[jobId]`. | |
| - [ ] **`/library/labels`:** CRUD; color picker; book-count badge; "Seed from derived" button calls `POST /api/labels/seed-from-derived`. | |
| - [ ] **`<JobWatcher>` component:** extracted from the page, reusable in Add Book final step, ops jobs detail, eval run, library detail Jobs tab. | |
| - [ ] **`<BookCard>` component:** row + grid variants. | |
| - [ ] **`<StatusBadge>` component:** all 6 statuses + colors from `FRONTEND_PRD.md` table. | |
| **Stage 3 gate:** | |
| 1. **Full ingest dry-run:** Add a book → wizard finishes → ingest job runs → live progress visible → completion redirects to book detail → search the new book → answer cites it. (Against the mock initially; against real BE once BE Stage 5 lands.) | |
| 2. Refresh during a mock ingest: progress bar resumes from current state without losing prior log lines. | |
| 3. **Cost gate:** Add Book wizard step 3 shows a counterfactual cost preview. The job watcher's progress bar shows `costSoFarUsd` updating. The final `done` event's `totalCostUsd` matches Sum of `llm_calls.cost_usd` for the run (BE responsibility — FE verifies the displayed number matches). | |
| ### Stage 4 — Operations + Evaluation + final research 📋 PENDING | |
| FRONTEND_PRD.md Phase 3. | |
| - [ ] `/operations` dashboard. | |
| - [ ] `/operations/jobs` + `[jobId]` (delegates to `<JobWatcher>`). | |
| - [ ] `/operations/costs`: multi-panel (by-day chart, by-model, by-stage, by-book). Range filter. | |
| - [ ] `/operations/indexes`: collection list; activate; delete. | |
| - [ ] `/operations/processing`: filterable log. | |
| - [ ] `/operations/tools`: read-only registry view with verified flag. | |
| - [ ] `/operations/config`: read merged config; inline-edit; `POST /api/config/migrate` → job watcher. | |
| - [ ] `/evaluation/run`: kicks off `POST /api/eval/runs` → job watcher. | |
| - [ ] `/evaluation/golden`: CRUD; bulk import (JSONL); validation. | |
| - [ ] `/evaluation/history` + `[runId]`: per-item metrics + diff against previous run. | |
| - [ ] `/research/compare`: group builder; strategy picker; per-row cards; cross-cutting synthesis. | |
| - [ ] `/research/allusions`: judge model picker; per-candidate progress; match list with verdict. | |
| - [ ] `/` Dashboard: jobs in flight, recent query, today's cost, status counts, health chips. | |
| **Stage 4 gate:** | |
| 1. **Cost-monitoring full sweep:** the Costs page renders daily totals matching `SELECT date, SUM(cost_usd) FROM llm_calls GROUP BY date` to the cent. Stale-pricing chip on dashboard if any tool has `last_updated > 90 days`. | |
| 2. Compare mode: build a group, run a comparison, render per-book rows with their own cost lines. Total = sum of row costs. | |
| 3. Allusions: judge model override changes the cost line of the judging stage (verify against tools_registry pricing). | |
| 4. **Parity matrix sweep:** walk `FRONTEND_PRD.md` Appendix A; every row's behaviors work in the new UI. | |
| ### Stage 5 — Cutover 📋 PENDING | |
| - [ ] Delete all mock routes from `server.ts`. Proxy is the only path. | |
| - [ ] Regenerate `lib/api/types.ts` from live `openapi.json`; commit; replace the hand-stub. | |
| - [ ] One week of dogfooding; bugfix-only commits. | |
| - [ ] Hand off to BE agent: BE deletes `src/stage9_ui/` and removes `streamlit` from `pyproject.toml`. | |
| --- | |
| ## Backend track | |
| ### Stage 0 — Branch setup ✅ DONE (verified 2026-05-17) | |
| **Must complete before any BE work begins.** The user (or orchestrator) performs these *after* the current Streamlit-driven indexing job finishes — never during. | |
| - [x] Commit the existing M files on `main` per the commit-split plan (see session log; user controls the split). | |
| - [x] `git checkout -b rewrite/api`. | |
| - [x] On this branch, ensure `CONTRACT.md`, `BACKEND_BUILD.md`, `IMPLEMENTATION.md`, `FRONTEND_PRD.md`, `FRONTEND_BUILD.md` are tracked. | |
| - [x] Add `.gitignore` entries: `.claude/`, `.critique_*.md`, `*.png` at repo root (move existing PNGs to `screenshots/` if keeping them). | |
| - [x] Confirm `data/inventory.db` is **not committed** (should already be ignored). | |
| ### Stage 1 — Skeleton + health ✅ DONE (verified 2026-05-17) | |
| `BACKEND_BUILD.md §13 Step 1`. | |
| - [x] `src/api/__init__.py`, `src/api/app.py` (FastAPI app + middleware skeleton + ApiError exception handler). | |
| - [x] `src/api/config.py` (port resolver, git-sha resolver, masked-config helper — no CORS allowlist in v1 per CONTRACT.md §2; session secret deferred to Stage 2). | |
| - [x] `src/api/dto/common.py` with `ApiModel` base, `ErrorResponse`, `JobStartResponse`. Every Stage-1 DTO inherits from `ApiModel`. | |
| - [x] `src/api/dto/system.py` with `HealthResponse`, `HealthCheck`, `VersionResponse` (all camelCase on the wire — verified via `/openapi.json`). | |
| - [x] `src/api/errors.py` minimal stub (ApiError base + InternalError; full canonical codes land in Stage 3). | |
| - [x] `src/api/routers/system.py` with `GET /system/health` (DB + Qdrant + storage + API keys + `stalePricing: []`) and `GET /system/version` (git sha + masked config). | |
| - [x] `pyproject.toml` adds: `fastapi`, `uvicorn[standard]`, `python-multipart`, `psutil`, `httpx`, `pytest-asyncio`. | |
| - [x] `uvicorn src.api.app:app --port 8000` boots cleanly; `curl http://localhost:8000/openapi.json` returns a valid spec listing both paths. | |
| - [x] `curl http://localhost:8000/system/health` returns 200 with camelCase fields including `stalePricing: []` (all 21 tools updated within the last 90 days as of 2026-05-17). | |
| - [x] **Cost-monitoring stub:** `/system/health` returns `stalePricing` (empty list when fresh; populated when any `cost_model.last_updated` > 90 days or missing). | |
| - [x] `tests/test_system_health.py` covers the OpenAPI shape, camelCase enforcement, health endpoint shape, and secret-masking in `/version`. | |
| **Stage 1 gate:** FE agent confirms `npm run gen:api` against `localhost:8000/openapi.json` succeeds and produces valid TS types. | |
| ### Stage 2 — Auth ✅ DONE (verified 2026-05-17) | |
| `BACKEND_BUILD.md §6` + `CONTRACT.md §3`. | |
| - [x] Migration: `sessions` table added via `src/stage1_inventory/schema.py` (CREATE TABLE IF NOT EXISTS inside SCHEMA_SQL — same idempotent pattern as the rest of the schema; `_COLUMN_ADDITIONS` is for ALTER-style growth on existing tables, not new tables). | |
| - [x] `src/api/auth/sessions.py`: `create_session`, `lookup_session` (bumps `last_seen`, lazy-GCs expired rows), `invalidate_session`, `purge_expired`. | |
| - [x] `src/api/auth/router.py`: `POST /auth/login`, `POST /auth/logout`, `GET /auth/me`. | |
| - [x] `src/api/deps.py`: `current_user` (raises 401 UNAUTHENTICATED on miss), `require_admin` (raises 403 FORBIDDEN for viewers). | |
| - [x] `src/api/middleware/csrf.py`: enforces `X-Requested-With: XMLHttpRequest` on POST/PATCH/PUT/DELETE; exempts `/auth/login`; GET is exempt by method (SSE works). | |
| - [x] `src/lib/auth/repo.py` additive: `from_session_token(token) -> User | None` + `user_for_username` helper. Existing `current_user()` / Streamlit session helpers untouched. | |
| - [x] `src/api/dto/auth.py`: `LoginRequest{username, password}`, `MeResponse{username, isAdmin, displayName?}` — both inherit ApiModel so the wire is camelCase. | |
| - [x] `src/api/errors.py`: typed `Unauthenticated` (401, UNAUTHENTICATED) and `Forbidden` (403, FORBIDDEN) ApiErrors. | |
| - [x] Test: `tests/test_auth.py` — 9 tests covering bad/good login, /auth/me with/without cookie, logout + subsequent 401, CSRF blocks logout without header, GET is exempt, /auth/login is exempt, OpenAPI camelCase regression guard extended to the new DTOs. | |
| **Stage 2 gate:** FE login flow works end-to-end against real BE; 401 on protected route redirects. | |
| Verification (2026-05-17): | |
| 1. Live login flow on uvicorn :8011 — bad creds → 401 UNAUTHENTICATED; good creds → 200 + MeResponse + `Set-Cookie: session=...; HttpOnly; Max-Age=2592000; Path=/; SameSite=lax`; /auth/me with cookie → 200; POST /auth/logout without X-Requested-With → 403 FORBIDDEN; with header → 204 + cookie cleared; subsequent /auth/me → 401. | |
| 2. Migration idempotency: ran `init_schema` twice on a copy of `data/inventory.db`; second run is a no-op; `sessions` table + `idx_sessions_username` + `idx_sessions_expires` all present. | |
| 3. Cost gate: 0 new `llm_calls` rows during the full live test (`SELECT COUNT(*) FROM llm_calls WHERE id > 932` returned 0; auth path is pure SQLite + TOML). | |
| 4. camelCase: `curl /openapi.json | jq '.components.schemas.MeResponse.properties | keys'` returns `["displayName", "isAdmin", "username"]`. | |
| 5. Streamlit boots without ImportError (`streamlit run src/stage9_ui/app.py --server.port 8765` came up cleanly). | |
| 6. All 9 auth tests pass; the 5 Stage 1 tests still pass (no regressions). | |
| ### Stage 3 — Read-only routers ✅ DONE (verified 2026-05-17) | |
| `BACKEND_BUILD.md §13 Step 3` + the `GET /books/status-counts` added per CONTRACT.md §10. | |
| - [x] `src/api/routers/books.py`: `GET /books`, `GET /books/{bookId}`, `GET /books/status-counts`. | |
| - [x] `src/api/routers/history.py`: `GET /history`, `GET /query/runs/{runId}` (committed runs only — live SSE comes in Stage 4). | |
| - [x] `src/api/routers/costs.py`: `GET /costs?groupBy=`, `GET /costs/llm-calls`, `GET /costs/counterfactual`. | |
| - [x] `src/api/routers/tools.py`: `GET /tools`, `GET /tools/{name}`. | |
| - [x] `src/api/routers/indexes.py`: `GET /indexes` (writes come later). | |
| - [x] `src/api/routers/processing_log.py`: `GET /processing-log` (admin-only). | |
| - [x] `src/api/routers/labels.py`: `GET /labels`. | |
| - [x] `src/api/routers/pdf.py`: `GET /books/{bookId}/pdf`, `GET /books/{bookId}/pages/{n}/{image|ocr|clean}`. | |
| - [x] `src/api/dto/common.py`: `ApiModel` base + `ErrorResponse` + `JobStartResponse` (Stage 1, kept). | |
| - [x] All DTOs in `src/api/dto/*` inherit from `ApiModel` (Stage 1+3 — `dto/{books,query,costs,tools,indexes,processing_log,labels,pdf_pages}.py`). | |
| - [x] Pricing health check: `/system/health` returns the stale list correctly (Stage 1, regression-tested). | |
| - [x] Canonical error codes filled in: `BookNotFound` (404 BOOK_NOT_FOUND with `details.bookId`), `QueryRunNotFound` (404 QUERY_RUN_NOT_FOUND with `details.runId`), `NotIndexed` (409 NOT_INDEXED), `BadRequest` (400 BAD_REQUEST). Pydantic 422 → BAD_REQUEST envelope via the new `RequestValidationError` handler in `app.py`. | |
| **Stage 3 gate:** all checks pass. | |
| Verification (2026-05-17 on uvicorn :8015 with mario admin + preview viewer): | |
| 1. `/openapi.json | .paths | keys` lists 22 paths — Stage 1+2 = 5, Stage 3 = 17 new (`/books`, `/books/status-counts`, `/books/{bookId}`, `/books/{bookId}/pdf`, `/books/{bookId}/pages/{n}/{image,ocr,clean}`, `/history`, `/query/runs/{runId}`, `/costs`, `/costs/llm-calls`, `/costs/counterfactual`, `/tools`, `/tools/{name}`, `/indexes`, `/processing-log`, `/labels`). | |
| 2. `components.schemas.BookStatusEnum` is a real OpenAPI enum: `{"type":"string","enum":["pending","acquired","ocr_done","clean","indexed","failed"]}` — no Literal-to-string regression. | |
| 3. `components.schemas` has 44 entries (Stage 1+2 was ~7; Stage 3 added 37+). | |
| 4. Random schema sweep (PageCleanDTO, BookListResponse, BookCosts) — all properties camelCase. The global sweep in `tests/test_costs.py:test_every_schema_property_is_camelcase` enforces this across every schema; only documented exemption is `BookStatusCountsDTO.ocr_done` per CONTRACT.md §10. | |
| 5. `GET /books?limit=5` returns 4 books (live data); `/books/status-counts` returns `{pending:0, acquired:1, ocr_done:0, clean:2, indexed:1, failed:0, total:4}` — keys match CONTRACT.md §10 exactly. | |
| 6. `GET /costs?groupBy=day&range=24h` returns 2 day buckets with `{key,label,costUsd,nCalls,inputTokens,outputTokens,durationMs,provider}` — array shape good. | |
| 7. **Cost-monitoring gate**: `MAX(llm_calls.id)` before = 2481; after the full test + gate cycle = 2481. **0 new paid calls** during Stage 3 (read-only routers do not touch any paid SDK). | |
| 8. `require_admin` gate: viewer (preview) → `/processing-log` → 403 FORBIDDEN envelope; admin (mario) → 200. `/costs` also gated to admin (faceted, can leak cross-user totals); `/costs/llm-calls` + `/costs/counterfactual` are user-tier. | |
| 9. Streamlit boots without ImportError on port 8765 — HTTP 200 served, no regressions to the lib/ surface the existing UI depends on. | |
| All 27 Stage 3 tests pass (9 books + 5 history + 13 costs). Stage 1+2 tests (14 total) still green — no regressions. | |
| ### Stage 4 — Query + SSE ✅ DONE (verified 2026-05-17) | |
| `BACKEND_BUILD.md §5.6 + §5.8 + §13 Step 4`. | |
| - [x] `src/api/jobs/store.py`: insert/update job rows; emit_event returns the SSE-id; Turso-resilience retry wrapper on every store function. | |
| - [x] `src/api/jobs/sse.py`: in-memory pub-sub + DB-backed replay from `Last-Event-ID`; 15s heartbeat (`event: ping`); terminal-sentinel signal_terminal(). | |
| - [x] `src/api/jobs/runner.py`: thread-based runner with per-subject lock (query:{username}) + global semaphore (config.yaml > api.max_concurrent_jobs, default 1). | |
| - [x] `src/api/jobs/registry.py`: `@register_job_type` decorator + get_handler/known_types helpers. | |
| - [x] `src/api/jobs/reconcile.py`: startup hook flips queued/running rows with dead pid (or pid=NULL) to failed + writes terminal done event. | |
| - [x] `src/api/jobs/types/query.py`: wraps `src.stage8_router.router.answer()` with on_progress callback that translates router steps (classify_start/done, retrieve_start/done, judge_start/done, generate_start/done, force_mode, comparison_start/done) into the locked SSE taxonomy (stage events with stageName ∈ {classify, retrieve, judge, generate} + progress events). | |
| - [x] `src/api/routers/jobs.py`: GET /jobs, GET /jobs/{jobId}, GET /jobs/{jobId}/events (honors Last-Event-ID header + ?lastEventId query fallback), POST /jobs/{jobId}/cancel (admin; psutil tree-kill for pid≠NULL + defensive status flip for in-thread jobs). | |
| - [x] `src/api/routers/query.py`: POST /query → 202 JobStartResponse{jobId, sseUrl}; GET /query/runs/{runId}/events resolves runId → jobId via jobs.subject_id and proxies. | |
| - [x] Migration: `jobs` + `job_events` tables + 3 indexes added to SCHEMA_SQL (idempotent CREATE TABLE IF NOT EXISTS, run-twice verified). | |
| - [x] **Cost-on-progress:** every `progress` event payload carries `costSoFarUsd` (sum of `llm_calls.cost_usd WHERE id > first_call_id`). Terminal done event carries `totalCostUsd`. | |
| - [x] Tests: tests/test_query_sse.py (8 — incl. one expensive end-to-end LOOKUP gated on GOOGLE_API_KEY + indexed Qdrant collection), tests/test_jobs_sse_replay.py (3), tests/test_jobs_reconcile.py (3), tests/test_no_unlogged_api_calls.py (2 — the global LLM-SDK-import gate per BACKEND_BUILD.md §11.2 / CONTRACT.md §12.1). | |
| **Stage 4 gate verified (2026-05-17):** | |
| 1. **End-to-end search**: Live uvicorn :8011 → login → POST /query with `{"query":"What does the book teach about salvation through Christ?","page":"search","topK":3,"usePremium":false}` → 202 `{"jobId":"6ae9b2bc...", "sseUrl":"/jobs/.../events"}` (no runId in 202). SSE stream emitted in order: | |
| - id 40 `stage` `{stageName:"classify", costSoFarUsd:0.0, query:"..."}` | |
| - id 41 `progress` `{pct:0.2, stageName:"classify", mode:"LOOKUP", costSoFarUsd:0.005831}` | |
| - id 42 `stage` `{stageName:"retrieve", costSoFarUsd:0.005831, topK:3}` | |
| - id 43 `progress` `{pct:0.45, stageName:"retrieve", nHits:3, costSoFarUsd:0.005831}` | |
| - id 44 `stage` `{stageName:"generate", costSoFarUsd:0.005831}` | |
| - id 45 `progress` `{pct:0.95, stageName:"generate", nCited:3, costSoFarUsd:0.0284}` | |
| - id 46 `done` `{finalStatus:"succeeded", durationMs:128827, totalCostUsd:0.0284, runId:16, result:{runId:16, totalCostUsd:0.0284, mode:"LOOKUP"}}` | |
| 2. **Refresh-safety**: `curl -H "Last-Event-ID: 43" /jobs/{id}/events` yielded exactly events 44, 45, 46 (no duplicates, no gaps). `GET /query/runs/16/events` yielded the full id 40-46 history via the runId → subject_id proxy. | |
| 3. **Cost gate (CRITICAL — paid quota)**: pre-query `MAX(llm_calls.id) = 2488`. Post-query new rows: id 2489 stage=classify cost=0.005831, id 2490 stage=generation cost=0.022569 — **exactly 2 new rows** for a LOOKUP. SUM(cost_usd) = 0.028400 = done.totalCostUsd to the cent. ✓ | |
| 4. **OpenAPI new paths**: `/openapi.json | jq '.paths | keys'` includes all 6 new Stage 4 paths (`/jobs`, `/jobs/{jobId}`, `/jobs/{jobId}/cancel`, `/jobs/{jobId}/events`, `/query`, `/query/runs/{runId}/events`); 28 paths total (was 22 in Stage 3). | |
| 5. **camelCase regression**: JobDTO.properties contains `progressPct`, `stageLabel`, `subjectId`, `createdAt`, `startedAt`, `finishedAt`; QueryRequest has `forceMode`, `topK`, `usePremium`, `comparisonSpec`. All event DTOs (LogEvent/StageEvent/ProgressEvent/DoneEvent/ErrorEvent) registered + camelCase per `test_costs.py::test_every_schema_property_is_camelcase`. | |
| 6. **No SDK imports outside wrappers**: `tests/test_no_unlogged_api_calls.py` greps `src/` for `from {google.genai,openai,anthropic,cohere,groq} import` / `import {...}` outside the 5 approved wrapper files. Passes. | |
| 7. **Reconciler test passes** (`tests/test_jobs_reconcile.py`). | |
| 8. **Streamlit still works**: `streamlit run src/stage9_ui/app.py --server.port 8765 --server.headless true` came up; `/_stcore/health` returns 200/ok. | |
| 9. **All 41 Stage 1+2+3 tests still pass** + 16 new Stage 4 tests pass (4 cheap test_query_sse + 3 reconcile + 3 sse_replay + 2 no_unlogged_api_calls + 4 documentation/auth tests in test_query_sse — actual count via pytest: see log). | |
| Note: under Turso's stale-stream HTTP-404 lag we observed missed `stage(classify)` events on the very first emit after idle. Fixed by wrapping every store function in `_with_retry()` that retries once on the Hrana stream-not-found ValueError. After that, three back-to-back live queries each emitted the full event sequence with no drops. | |
| ### Stage 5 — Jobs subsystem (ingest) ✅ DONE (verified 2026-05-17, live smoke STAGED but NOT RUN) | |
| `BACKEND_BUILD.md §5 + §5.7 + §13 Step 5`. | |
| - [x] `src/lib/processing/ingest_progress.py`: lifted `parse_ingest_progress` + `filter_ingest_log_noise` from `src/stage9_ui/shared.py` (per §3.4). Added `stage_number_to_name()` mapping for `[stage N]` → canonical SSE name (`acquisition`/`ocr`/`cleanup`/`chunking`/`indexing`). Stage 4b folded into `cleanup`. | |
| - [x] `src/stage9_ui/shared.py`: re-export shim (no behavior change — Streamlit still boots, parser output bit-for-bit identical). | |
| - [x] `src/api/jobs/types/ingest.py`: subprocess-based handler. `@register_job_type("ingest")` wires it into the existing Stage-4 runner. `subprocess.Popen` with line-buffered stdout, CREATE_NEW_PROCESS_GROUP on Windows for clean tree-kill; PID is recorded immediately via `store.set_pid` so `POST /jobs/{id}/cancel` can `psutil.Process(pid).children(recursive=True)`. The runner's existing locks (per-subject + global semaphore) cover ingest the same way they cover query. | |
| - [x] `_StreamingState`: re-feeds full filtered log buffer through the lifted parser on every stdout line, detects stage transitions (emits `stage` SSE with `stageName` + `stageLabel` + `costSoFarUsd`), emits `progress` throttled to ≤ 1/500ms with `pagesDone` + `pagesTotal` + `costSoFarUsd`. Terminal `done` event carries `finalStatus` (derived from subprocess exit code, unless cancel flipped status first), `durationMs`, `totalCostUsd`, `result.{bookId,exitCode,totalCostUsd}`. | |
| - [x] `--stages acquisition,ocr,cleanup,chunking,indexing` on `src/pipeline/ingest.py:main()`. Validates against `STAGE_NAMES`; unknown stage aborts with `SystemExit(2)`. `ingest()` adds `stages: set[str] | None` param + `_wants(name)` guard on every stage block. Default (`stages=None`) = run every eligible stage = unchanged behavior. | |
| - [x] `src/api/dto/ingestion.py`: `IngestRequest` (camelCase via ApiModel) + `StageName: Enum` (real enum, surfaces as TS union via OpenAPI). All fields optional → empty POST = re-ingest with persisted settings. | |
| - [x] `src/api/routers/ingestion.py`: `POST /books/{bookId}/ingest` admin-only; 404 BOOK_NOT_FOUND envelope with `details.bookId`; 202 `JobStartResponse{jobId, sseUrl=/jobs/{jobId}/events}`; persists `extraction_mode`/`cleanup_enabled` onto the book row before enqueue; passes `subject_id=book_id` at insert + `subject_key=f"book:{book_id}"` at enqueue. | |
| - [x] `src/api/app.py`: mounts `ingestion_router`. Side-effect import of `src.api.jobs.types.ingest` lives inside the router file (parity with `query.py`). | |
| - [x] **Per-subject lock** keyed by `(ingest, book:{book_id})` — second concurrent ingest for the same book waits on the existing `runner._subject_lock`. No 409; the request is accepted (202) and serialized by the lock (documented in the router docstring). | |
| - [x] **Tree-kill on cancel** — already wired in `routers/jobs.py:cancel_job` via `psutil.Process(pid).children(recursive=True)` since Stage 4. Stage 5 just supplies the PID by calling `store.set_pid(job_id, proc.pid)` right after `Popen`. | |
| - [x] **Cost-on-progress for ingest**: every `stage` and `progress` event payload carries `costSoFarUsd` computed via `store.cost_since(first_call_id)` where `first_call_id = store.max_llm_call_id()` snapshotted at job start. Terminal `done.totalCostUsd` is the same sum at terminal. | |
| - [x] Tests: `tests/test_ingest_jobs.py` (17, all pass) — argv builder purity, parser state transitions (3→4→5→6 → canonical [ocr, cleanup, chunking, indexing]), progress events carry pagesDone + costSoFarUsd, 401/403/404 gates, 202 happy path with mocked `runner.enqueue` + persisted job row + cleanup, empty body accepted, bogus stage rejected as BAD_REQUEST, OpenAPI camelCase + StageName-enum regression. | |
| - [x] `tests/test_ingest_jobs_live.py` — STAGED but NOT RUN. Marked `@pytest.mark.live` (registered in pyproject.toml) + `@pytest.mark.skip` belt-and-suspenders. `pyproject.toml` now defaults to `-m 'not live'`. Documented opt-in flow: `LIVE_INGEST_BOOK_ID=<small_book> LIVE_INGEST_STAGES=indexing pytest -m live tests/test_ingest_jobs_live.py`. Skeleton wires login → POST → SSE drain → cost gate so when the user opts in the test runs straight through. | |
| **Stage 5 gate — verified 2026-05-17 (live smoke deferred per user opt-in):** | |
| 1. `parse_ingest_progress` import works from BOTH `src/lib/processing/ingest_progress.py` (canonical) AND `src/stage9_ui/shared.py` (re-export shim) — confirmed by direct import + sample-input behavior check (identical output across 3 representative log transcripts). ✓ | |
| 2. `python -m src.pipeline.ingest --help` lists `--stages` with the 5 canonical choices; `--stages bogus,chunking --book-id x` aborts with `unknown stage(s) ['bogus']`; `--stages chunking --book-id nonexistent_test_book_id_123` reaches the ingest() function and errors cleanly ("book_id ... not in the inventory"). ✓ | |
| 3. `POST /books/{id}/ingest` returns 202 JobStartResponse with valid `jobId` + `sseUrl=/jobs/{jobId}/events`. Verified via TestClient against a real book id from the live inventory: 202 + `runner.enqueue` called exactly once with `subject_key=book:{id}` + `type=ingest`, job row landed in 'queued' with `subject_id=book_id`. ✓ | |
| 4. Mocked-subprocess parser tests pass: `_StreamingState` fed synthetic `[stage 3] OCR…` + page-ok lines emits a `stage` event with `stageName="ocr"` + `costSoFarUsd`, then `progress` events with `pagesDone=1,2,3` + `stageName=ocr` + `costSoFarUsd≥0`. Multi-stage transition test (3→4→5→6) emits canonical [ocr, cleanup, chunking, indexing] in order. ✓ | |
| 5. Per-subject lock: two simultaneous ingest requests for the same `book_id` get accepted as 202 (no 409) and serialize on `runner._subject_lock[("ingest", "book:{book_id}")]`. Documented in the router docstring. ✓ | |
| 6. `/api/jobs?type=ingest` filters by ingest type — uses the existing `routers/jobs.py` `list_jobs` endpoint which already accepts `type: JobTypeEnum | None`. JobTypeEnum already lists `ingest` (was reserved in Stage 4 for this stage). ✓ | |
| 7. **Cost gate: 0 new `llm_calls` rows** during this session (no live ingest). Every unit test mocks the subprocess via `monkeypatch` on `runner.enqueue` OR feeds synthetic stdout to `_StreamingState` (which calls `store.cost_since` but does not invoke any LLM SDK). The live smoke test is staged but explicitly skipped. ✓ | |
| 8. Streamlit still boots: `src.stage9_ui.shared` imports cleanly; `parse_ingest_progress` re-export produces identical output to the canonical version. ✓ | |
| 9. All 17 Stage 5 tests pass + 90 prior tests still green. The 19 pre-existing failures in `tests/test_labels.py` + `tests/test_stage1_inventory.py` (libsql Row "'C' object has no attribute" issues) are NOT caused by Stage 5 — confirmed by `git stash && pytest tests/test_labels.py tests/test_stage1_inventory.py` on the base commit (same 19 failures). ✓ | |
| **Live smoke test deferred to user opt-in.** When ready: drop a 1-page test book in the inventory, then `LIVE_INGEST_BOOK_ID=<id> LIVE_INGEST_STAGES=indexing pytest -m live tests/test_ingest_jobs_live.py`. Estimated cost for `indexing` alone: $0.00 (local BGE-M3 + Qdrant). Estimated cost adding `ocr`: ~$0.005/page Gemini. | |
| ### Stage 6 — Add Book probe + writes ✅ DONE (verified 2026-05-17, live probe smoke STAGED but NOT RUN — user opt-in) | |
| `BACKEND_BUILD.md §13 Step 6`. | |
| - [x] `POST /api/uploads`: multipart; stores in `data/uploads/{uploadId}.pdf`; returns `{ uploadId, sha256, sizeBytes }` (camelCase). 1 GiB cap; SHA-256 streamed in 1 MiB chunks so peak RSS stays flat on the largest PDFs in the inventory. Admin-only. | |
| - [x] `POST /api/books/probe`: wraps `src/lib/metadata_probe.probe_metadata`; returns ProbeResponse with `pagesTotal`, `hasTextLayer`, `suggestedExtractionMode`, `guessedMetadata` (enums coerced to FE union types), `suggestedLabels`, `samplePages[]` (PyMuPDF native text), `estimatedIngestCostUsd` (computed against `tools_registry.yaml` for the active OCR + cleanup models — $0 when native_text mode is suggested). Body requires `uploadId` OR `sourceUrl`. Admin-only (paid Gemini metadata call ~$0.005). Tests MOCK `probe_metadata`. | |
| - [x] `POST /api/books`: `CreateBookRequest` body; **409 `BOOK_EXISTS`** with `details.conflictWithBookId` when `bookId` collides OR when `sourceUrl` collides with an existing row; **201 BookDTO** on success. Moves the upload bytes into storage (`data/raw/{bookId}.pdf`) via `get_storage().put_pdf` BEFORE the books row INSERT so a storage failure never leaves an orphan row. | |
| - [x] `PATCH /api/books/{bookId}`: `UpdateBookRequest` (partial). Status field is NOT in the DTO (pipeline owns it). Enum members coerced back to strings before update_book(). | |
| - [x] `DELETE /api/books/{bookId}`: admin-only, **idempotent** (204 on a non-existent book). Cleanup order: Qdrant points across every known collection (best-effort; tolerates network failures) → `book_labels` mappings → `books` row. Historical `processing_log` / `llm_calls` / `query_history` / `book_indexes` rows preserved. | |
| - [x] `POST /api/labels`: 201 LabelDTO; 400 on duplicate id. `PATCH /labels/{labelId}`: preserves immutable `kind` field. `DELETE /labels/{labelId}`: 204 even on miss (idempotent). `POST /labels/seed-from-derived`: wraps `src/lib/labels/seeding.seed_derived_labels`; returns `{ created, existing }` counts; idempotent (second call returns `created=0`). | |
| - [x] `GET /api/books/{bookId}/preview?page=N&dpi=120`: returns PNG bytes for the wizard's sample-page pane. Default DPI 120 (vs pdf.py's 150) to keep the wizard snappy on slow tethers. 404 BOOK_NOT_FOUND envelope on unknown book. | |
| - [x] Tests: `tests/test_addbook.py` (18 — upload happy path + auth + empty rejection; probe with mocked metadata; create 201 + 409 BOOK_EXISTS for bookId + sourceUrl conflicts; PATCH updates + 404; DELETE 204 clears labels + Qdrant + idempotent; preview PNG; OpenAPI path + camelCase regression), `tests/test_labels_writes.py` (10 — POST 201 + duplicate 400 + validation 400; PATCH updates + 404 + immutable kind; DELETE 204 + idempotent; seed-from-derived idempotent; auth + CSRF gates), `tests/test_addbook_live.py` (STAGED but NOT RUN; `@pytest.mark.live` + `@pytest.mark.skip` + `pyproject.toml`'s `-m 'not live'` default; opt-in flow documented for `LIVE_PROBE_PDF_PATH=<path>` OR `LIVE_PROBE_SOURCE_URL=<url>`; estimated cost ~$0.001-0.005 per run). | |
| **Stage 6 gate verified 2026-05-17:** | |
| 1. All 28 Stage 6 unit tests pass (18 in test_addbook.py + 10 in test_labels_writes.py). Live probe test STAGED + SKIPPED per `pyproject.toml`'s `-m 'not live'` default. | |
| 2. **Cost gate (load-bearing for paid quota)**: pre-test `MAX(llm_calls.id) = 2495`. After the full Stage 6 test cycle, **0 new `llm_calls` rows** (`SELECT COUNT(*) FROM llm_calls WHERE id > 2495 = 0`). Every probe test monkey-patches `src.api.routers.addbook.probe_metadata` to return a synthetic `ProbeResult`; every DELETE test monkey-patches `src.stage6_indexing.qdrant_io.{list_collections, delete_book_points}` so the live cloud collection is never touched. | |
| 3. **camelCase regression**: `tests/test_costs.py::test_every_schema_property_is_camelcase` still passes — every Stage 6 DTO (UploadResponse, ProbeRequest, ProbeResponse, GuessedMetadata, SuggestedLabel, SamplePage, CreateBookRequest, UpdateBookRequest, LabelCreateRequest, LabelUpdateRequest, LabelSeedResponse) inherits `ApiModel` and serializes camelCase on the wire. | |
| 4. **No SDK imports outside wrappers**: `tests/test_no_unlogged_api_calls.py` (2 tests) still green — no new direct `from google.genai/openai/anthropic/cohere/groq import` outside the 5 approved wrapper files. | |
| 5. **OpenAPI paths**: `curl /openapi.json | jq '.paths | keys'` includes all Stage 6 paths — `/uploads`, `/books/probe`, `/books/{bookId}/preview`, `/labels/seed-from-derived`, plus the POST/PATCH/DELETE verbs added to existing `/books` + `/books/{bookId}` + `/labels` + `/labels/{labelId}` paths (43 endpoints total across 34 paths). | |
| 6. **Data preservation**: live `data/inventory.db` is untouched — books count remained 4 + labels count remained 19 across the entire test cycle. Every test fixture (`cleanup_created_books`, `cleanup_labels`) tracks the rows it creates and tears them down at teardown; the `delete_book` helper also unlinks the on-disk PDF in `data/raw/` so local storage stays clean. | |
| 7. **Bug fixed during this session**: `src/api/routers/books.py` `create_book` called `req.era.value` (etc.) but ApiModel's `use_enum_values=True` already coerces enum members to plain strings at validation time — same pattern the Stage 5 implementation log flags. Refactored to a single `_v(x)` helper that handles both Enum and str inputs (matching the existing pattern in `patch_book`). Found by `test_create_book_returns_201_with_book_dto`; tests added now guard this regression. | |
| 8. **No regressions to prior stages**: all prior stage 1-5 tests still pass after the books.py fix (Stage 3 books, Stage 4 query/jobs/sse, Stage 5 ingest). | |
| **Live probe smoke deferred to user opt-in.** When ready: `LIVE_PROBE_PDF_PATH=<small_pdf> pytest -m live tests/test_addbook_live.py` for upload+probe; or `LIVE_PROBE_SOURCE_URL=<public_url> pytest -m live tests/test_addbook_live.py` for URL-fetch+probe. Estimated cost: ~$0.001 for a 1-page native_text PDF; ~$0.005 for a 5-page image-mode PDF. | |
| ### Stage 7 — Operations + Evaluation 📋 PENDING | |
| `BACKEND_BUILD.md §13 Step 7`. | |
| - [ ] `GET /api/config`, `PATCH /api/config`, `POST /api/config/migrate` → job. | |
| - [ ] `POST /api/indexes/{name}/activate`, `DELETE /api/indexes/{name}`. | |
| - [ ] `GET/POST/PATCH/DELETE /api/eval/golden/*`. | |
| - [ ] `POST /api/eval/runs` → job; `GET /api/eval/runs[/{id}]`. | |
| - [ ] `src/api/jobs/types/eval_run.py`, `src/api/jobs/types/backend_migrate.py`. | |
| **Stage 7 gate:** Run an eval against the real BE; results appear in `/evaluation/history/[runId]`. Backend migration from local → turso (or back) works as a job, with progress visible. | |
| ### Stage 8 — Cleanup 📋 PENDING | |
| - [ ] `tests/test_no_unlogged_api_calls.py` written; CI passes. | |
| - [ ] Parity matrix (FRONTEND_PRD.md Appendix A) fully green. | |
| - [ ] One week of dogfooding (FE Stage 5). | |
| - [ ] `git rm -r src/stage9_ui/`; drop `streamlit` from `pyproject.toml`. | |
| - [ ] Final commit; merge `rewrite/api` → `main`. | |
| --- | |
| ## Open contract questions | |
| If either agent hits a question CONTRACT.md doesn't answer, **don't guess.** Add it to this section and ping the orchestrator. Examples of legitimate questions: | |
| - A new field needs to be added to BookDTO. | |
| - A new endpoint shape isn't covered. | |
| - An error code is needed beyond CONTRACT.md §8's list. | |
| Currently open: _(none — keep this section honest; remove "none" when adding the first entry.)_ | |
| --- | |
| ## Implementation log (latest first) | |
| - **2026-05-17 (BE, fix-up)** — **FE Stage 3 audit blockers — 3 BE bugs fixed.** Triggered by `Patristic-AI-Fe/tests/screenshots/stage3_audit/STAGE3_REPORT.md` (CC7 / CC8 + critical 3 in §3 of the audit). Three landed fixes, one perf bonus: | |
| 1. **`GET /system/events` SSE endpoint** (audit CC8). New code in `src/api/routers/system.py`: a poll-driven SSE generator emits `event: active_jobs` with `{activeCount, runningJobIds}` every ~2.5 s when the snapshot changes, plus `event: ping` heartbeats every 15 s (CONTRACT.md §4). Initial frame yielded before the loop so the FE's `SidebarJobStatus.tsx` chip flips off "Connecting…" on first byte rather than waiting one poll. User-tier gate via `current_user` dep. Tests: `tests/test_system_events.py` (5 — auth gate / snapshot shape / wire format / generator initial-yield / OpenAPI surface). | |
| 2. **Connection-layer Turso retry** (audit CC7). New shared module `src/stage1_inventory/turso_retry.py` (`is_stale_stream_error`, `with_retry`). `TursoConnection.execute` / `executemany` / `commit` / `executescript` / `cursor` / `rollback` now auto-reconnect + replay once on Hrana `stream not found` 404 — catches every Turso query path including the `lookup_session()` → `conn.commit()` cold-start race the tester captured (was 500 on first `/books/status-counts` after BE boot). `src/api/jobs/store.py` re-exports `_with_retry` from the shared module so existing per-callsite calls keep working (now double-protected). Tests: `tests/test_turso_retry.py` (9 — classifier matches Hrana wire format / rejects unrelated errors / single-shot retry / no infinite loop / TursoConnection reconnect + replay path / persistent-failure propagation). | |
| 3. **`GET /books/{id}/pages/{n}/image` 500 → structured 404** (audit critical §3). Root cause: `Storage.ensure_local()` raises bare `FileNotFoundError` for books whose inventory row exists but whose PDF blob isn't on disk (legacy import / R2 fetch failure). `src/api/routers/pdf.py:get_page_image` now catches `FileNotFoundError` + out-of-range `ValueError` and converts them to `BookNotFound` (404 BOOK_NOT_FOUND envelope per CONTRACT.md §8). Matching fix on `get_book_pdf` (same root cause class). The FE's graceful "Page image unavailable." chrome renders cleanly on 404 but coughs up a generic toast on 500 — that's the exact UX bug the audit caught on every book the user opened that didn't have a local PDF. Tests: `tests/test_pdf_image.py` (5 — happy path 200 PNG / missing blob → 404 envelope / unknown book → 404 / out-of-range page → 404 / sweep every real-inventory book to assert status ∈ {200, 404}, never 500). | |
| 4. **Performance side fix (audit CC11 partial — perf):** `init_schema()` was running on every `connect()` (idempotent CREATE TABLE IF NOT EXISTS + 5 ALTER TABLEs + a backfill UPDATE = ~7 round-trips ≈ 4 s on Turso). Added a process-local guard in `src/stage1_inventory/db.py:_ensure_schema_once` so the migration runs exactly once per `(process, backend)`. Per-request connect latency drops from ~4 s to single-digit ms; this also unblocked the `/system/events` poll loop which would otherwise have starved the event loop on every tick. Verified against live Turso: first connect ~4 s (schema run), every subsequent connect <50 ms. | |
| 5. **ASGI CSRF middleware refactor (load-bearing for SSE).** The existing `csrf_middleware` was registered via `app.middleware("http")(fn)` which wraps it in `BaseHTTPMiddleware`. That adapter buffers streaming responses through an anyio memory stream — verified during this session that `curl /system/events` saw 0 bytes for 15+ seconds before a burst flush, breaking the FE chip's "connect → first frame" UX. New `CsrfASGIMiddleware` in `src/api/middleware/csrf.py` is pure ASGI (operates on raw `scope` / `receive` / `send`) and lets SSE frames flush as they arrive. Same rule (require `X-Requested-With: XMLHttpRequest` on POST/PATCH/PUT/DELETE except `/auth/login`). The legacy `csrf_middleware` function is kept for back-compat with any caller importing it directly; the app wires the ASGI class via `app.add_middleware(CsrfASGIMiddleware)`. Verified live: CSRF rejection of `POST /auth/logout` without `X-Requested-With` → 403 envelope, CSRF allow with header → 200, plus SSE first frame arrives immediately. | |
| Verification gates (all green, evidence below): | |
| * Live curl on :8011 (separate from orchestrator's :8000): `GET /system/events` emitted `event: active_jobs\ndata: {"activeCount": 0, "runningJobIds": []}` on first byte; second SSE check (12 s @ 5 s heartbeat config) emitted both initial frame + ping heartbeat. | |
| * Live curl: cold-start `GET /books/status-counts` returned 200 on FIRST call after fresh BE boot (no Hrana 500). | |
| * Live curl: `GET /books/deskolia_v3/pages/1/image?dpi=150` → 200 `image/png` 634943 bytes; `GET /books/history_of_christian_thought_jesus_christ_through_the_ages/pages/1/image?dpi=150` (no local PDF) → 404 BOOK_NOT_FOUND envelope. | |
| * **Cost gate (load-bearing for paid quota)**: pre-fix-up `MAX(llm_calls.id)=2497`, post-fix-up `MAX(llm_calls.id)=2497`, **0 new paid calls** in the entire fix-up session (all tests mock paid surfaces or hit cheap probes; no fix area touches an LLM SDK). | |
| * 19 new tests across `test_turso_retry.py` + `test_pdf_image.py` + `test_system_events.py` all pass. **Full test suite re-run pending** at log-write time (run is in flight — see Verification at end). | |
| * Streamlit smoke-imports unaffected (no changes to UI surfaces). | |
| * **NOT addressed (out of scope for this fix-up)**: BE-4 perf for `/books/{id}` 12-second spinner — the slow joins are noted in the audit; the init_schema-once fix above shaves ~4 s off cold-start request latency but the `/books/{id}` endpoint itself still aggregates costs/indexes/processing-log/labels sequentially. That's a follow-up (Stage 7 or later perf pass). FE-side audit items (Browse infinite loop, Add Book contract drift, ConfirmDialog nested-button, etc.) are explicitly FE-side and don't apply to this repo. | |
| Commits on `rewrite/api`: see git log entries from this session. | |
| - **2026-05-17 (BE)** — Stage 6 (Add Book probe + writes) landed on `rewrite/api`. **Live probe smoke STAGED but NOT RUN** (user opt-in per AGENT_PROMPT_BE.md's paid-quota concern). New files: `src/api/dto/addbook.py` (UploadResponse, ProbeRequest, ProbeResponse, GuessedMetadata, SuggestedLabel, SamplePage — all inherit ApiModel for camelCase wire), `src/api/routers/uploads.py` (`POST /uploads` — multipart PDF, 1 GiB cap, SHA-256 streamed in 1 MiB chunks, returns `{uploadId, sha256, sizeBytes}`), `src/api/routers/addbook.py` (`POST /books/probe` — wraps `src/lib/metadata_probe.probe_metadata`, returns ProbeResponse with `pagesTotal`/`hasTextLayer`/`suggestedExtractionMode`/`guessedMetadata`/`suggestedLabels`/`samplePages`/`estimatedIngestCostUsd`/`probeError`), `tests/test_addbook.py` (18 tests — upload happy path + 401 + empty rejection; probe with mocked metadata + 400 on missing body; create 201 + 409 BOOK_EXISTS for bookId + sourceUrl conflicts; PATCH updates + 404; DELETE 204 clears labels + Qdrant + idempotent; preview PNG + 404; OpenAPI camelCase regression), `tests/test_labels_writes.py` (10 tests — POST 201 + duplicate 400 + validation 400; PATCH updates + 404 + immutable kind; DELETE 204 + idempotent; seed-from-derived idempotent; auth + CSRF gates), `tests/test_addbook_live.py` (STAGED but NOT RUN; opt-in via `LIVE_PROBE_PDF_PATH` or `LIVE_PROBE_SOURCE_URL` env vars; full skeleton with login → upload/url → probe → cost gate). Allowed additive edits per BACKEND_BUILD.md §3.4: `src/lib/books/repo.py` got `find_book_id_by_source_url`, `update_book(partial)`, `delete_book`; `src/api/errors.py` got `BookExists` (409 BOOK_EXISTS with `details.conflictWithBookId`); `src/api/routers/books.py` extended with `POST /books` (201 BookDTO + 409 envelope), `PATCH /books/{bookId}`, `DELETE /books/{bookId}` (best-effort Qdrant cleanup → book_labels mappings → books row, historical rows preserved), `GET /books/{bookId}/preview?page=N&dpi=120` (PNG bytes for the wizard, default DPI 120 vs pdf.py's 150); `src/api/routers/labels.py` extended with POST/PATCH/DELETE + `POST /labels/seed-from-derived`; `src/api/dto/books.py` extended with `CreateBookRequest`/`UpdateBookRequest`; `src/api/dto/labels.py` extended with `LabelCreateRequest`/`LabelUpdateRequest`/`LabelSeedResponse`; `src/api/app.py` mounts `uploads_router` + `addbook_router`. Bug found + fixed mid-session: `create_book` was calling `req.era.value` (etc.) directly, but ApiModel's `use_enum_values=True` already coerces enums to strings at validation — same pattern flagged in the Stage 5 implementation log. Refactored to a single `_v(x)` helper that handles both Enum and str inputs (matching the existing pattern in `patch_book`). Caught by `test_create_book_returns_201_with_book_dto`; regression-guarded by the new test suite. **Cost gate (load-bearing for paid quota)**: pre-test `MAX(llm_calls.id) = 2495`. After the full Stage 6 test cycle, `SELECT COUNT(*) FROM llm_calls WHERE id > 2495 = 0` — **0 new paid calls**. Every probe test monkey-patches `src.api.routers.addbook.probe_metadata` to return a synthetic `ProbeResult`; every DELETE test monkey-patches `src.stage6_indexing.qdrant_io.{list_collections, delete_book_points}` so the real Qdrant cloud collection is never touched. The live probe path is gated behind `@pytest.mark.live` + `@pytest.mark.skip` + `pyproject.toml`'s default `-m 'not live'` — belt, suspenders, and a second pair of suspenders (same pattern as `test_ingest_jobs_live.py`). Data preservation: live `data/inventory.db` still has the same 4 books + 19 labels after the test cycle — every fixture cleans up the rows it creates. Verified gates: 28 new tests pass + camelCase regression in `test_costs.py::test_every_schema_property_is_camelcase` still green + no-SDK-imports-outside-wrappers regression in `test_no_unlogged_api_calls.py` still green + Streamlit still boots cleanly + OpenAPI surfaces all Stage 6 paths (`/uploads`, `/books/probe`, `/books/{bookId}/preview`, `/labels/seed-from-derived`, PATCH/DELETE on existing `/books/{bookId}` + `/labels/{labelId}`, POST on `/books` + `/labels`). All prior stage 1-5 tests still pass after the books.py enum-coercion fix (no regressions). Commits on `rewrite/api`: `56d5355` (DTOs + BookExists error — prior agent session, pre-crash), `5a70aa4` (uploads + probe + book/label writes + preview — prior agent session, pre-crash), `<tests>` (this session: tests + books.py enum fix), `<tracker>` (this session: tracker flip + this log entry). Next: Stage 7 (Operations + Evaluation — `GET/PATCH /api/config`, `POST /api/config/migrate` → job, `POST /api/indexes/{name}/activate`, eval CRUD + run jobs). | |
| - **2026-05-17 (BE)** — Stage 5 (Jobs subsystem — ingest) landed on `rewrite/api`. **Live smoke test STAGED but NOT RUN** (user opt-in per AGENT_PROMPT_BE.md's paid-quota concern). New files: `src/lib/processing/ingest_progress.py` (lifted parser + `stage_number_to_name` mapping), `src/api/jobs/types/ingest.py` (subprocess-based handler — Popen with CREATE_NEW_PROCESS_GROUP on Windows, line-buffered stdout drain via `_StreamingState`, throttled progress events, `costSoFarUsd` on every emit, terminal `done` with final status from subprocess exit code + total cost from `store.cost_since`), `src/api/dto/ingestion.py` (IngestRequest + StageName: Enum), `src/api/routers/ingestion.py` (POST /books/{bookId}/ingest, admin-only, 202 JobStartResponse, persists extraction_mode/cleanup_enabled overrides, enqueues with subject_key=`book:{id}`), `tests/test_ingest_jobs.py` (17 tests — argv builder, parser state transitions, router contract, OpenAPI regression), `tests/test_ingest_jobs_live.py` (staged-but-skipped end-to-end test with full skeleton). Allowed additive edits per BACKEND_BUILD.md §3.4: `src/pipeline/ingest.py` gained `--stages` flag + `STAGE_NAMES` + `_parse_stages_arg` (validates against canonical set, aborts on typo); `ingest()` takes `stages: set[str] | None` and each numbered block guards on `_wants(name)` — default behavior is unchanged when the flag is omitted. `src/stage9_ui/shared.py` reduced to a re-export shim importing the lifted parser; Streamlit still boots and the parser output is bit-for-bit identical (verified by feeding the same 3 transcripts to both module paths). `pyproject.toml` registers the `live` pytest marker + defaults to `-m 'not live'` so the live smoke is deselected at collection. `src/api/app.py` mounts the new router. Bug found + fixed mid-session: ApiModel's `use_enum_values=True` makes Pydantic coerce enum members to their string values at validation, so `req.stages[0]` is `str`, not `StageName` — router normalizes `isinstance(s, str) else s.value` on stages and extraction_mode so a future ApiModel config change can't silently break this path. **Cost gate (load-bearing for paid quota)**: 0 new `llm_calls` rows in this session — every unit test mocks the subprocess via `monkeypatch` on `runner.enqueue` or feeds synthetic stdout directly to `_StreamingState` (which never calls an LLM SDK; only reads SQLite via `store.cost_since`). The live test path is gated behind `@pytest.mark.live` + `@pytest.mark.skip` + `pyproject.toml`'s default `-m 'not live'` — belt, suspenders, and a second pair of suspenders. Verified gates: --help shows --stages flag, bogus stages reject cleanly, 404 BOOK_NOT_FOUND envelope with `details.bookId` for unknown books, 202 happy path returns valid jobId + sseUrl matching `/jobs/{jobId}/events`, OpenAPI has 29 paths total with `/books/{bookId}/ingest` registered + IngestRequest/StageName camelCase + enum, Streamlit shim parser output identical to canonical, all 17 Stage 5 tests pass + 90 prior tests still green (the 19 pre-existing `test_labels.py` + `test_stage1_inventory.py` libsql-Row failures are unrelated — confirmed by `git stash && pytest` on base commit). Threading vs subprocess (locked design): query jobs run in-thread (Stage 4); ingest jobs run as subprocess (Stage 5) for tree-kill reliability on multi-hour OCR. Commits on `rewrite/api`: `d614729` (parser lift + --stages CLI flag), `e987772` (ingest job type + DTO + router + app wiring), `789d515` (tests + staged-but-skipped live smoke + pytest marker registration). Next: Stage 6 (Add Book probe + writes — `POST /uploads`, `POST /books/probe`, `POST /books` with 409 BOOK_EXISTS, PATCH/DELETE /books/{id}, label CRUD). | |
| - **2026-05-17 (BE)** — Stage 4 (Query + SSE) landed on `rewrite/api`. New files: `src/api/jobs/{__init__,store,sse,registry,runner,reconcile}.py`, `src/api/jobs/types/{__init__,query}.py`, `src/api/routers/{jobs,query}.py`, `src/api/dto/jobs.py`. Extended `src/api/dto/query.py` with QueryRequest/PageEnum/Filters/ComparisonGroup/ComparisonSpec/QueryModelOverrides (locked field names per CONTRACT.md §11). Schema: `jobs` + `job_events` tables + 3 indexes added to SCHEMA_SQL (run-twice idempotent verified on live data/inventory.db). Tests: `test_query_sse.py` (8 — incl. one paid end-to-end LOOKUP gated on GOOGLE_API_KEY + indexed Qdrant), `test_jobs_sse_replay.py` (3 — strict-greater-than fetch_events_since + event_stream deep-link replay + Last-Event-ID reconnect lossless), `test_jobs_reconcile.py` (3 — dead pid → failed, pid=NULL → failed, no-orphan no-op), `test_no_unlogged_api_calls.py` (2 — global LLM-SDK-import gate per BACKEND_BUILD.md §11.2 + ALLOWED_FILES existence guard). Stage 4 gate verified live on :8011: 202 with jobId/sseUrl no runId → full event sequence stage(classify)→progress→stage(retrieve)→progress→stage(generate)→progress→done with costSoFarUsd on every progress and totalCostUsd matching to the cent in done; Last-Event-ID replay yielded exactly the missed events; `GET /query/runs/16/events` proxied via jobs.subject_id and replayed full history. **Cost gate (load-bearing for paid quota)**: pre-query MAX(llm_calls.id)=2488, post-query new rows = exactly [2489 classify $0.005831, 2490 generation $0.022569], SUM=$0.0284 = done.totalCostUsd to the cent. 6 new OpenAPI paths (`/jobs`, `/jobs/{jobId}`, `/jobs/{jobId}/cancel`, `/jobs/{jobId}/events`, `/query`, `/query/runs/{runId}/events`), 28 paths total. Discovered + fixed a Turso transient stream-404 issue: the libsql adapter raises `ValueError("api error: status=404 Not Found, body=...stream not found...")` after long-idle gaps; the first emit_event of a query was silently dropped because `src.stage8_router.router.answer`'s on_progress wrapper swallows callback exceptions. Hardened by wrapping every store function in `_with_retry()` that retries once on Hrana stream-not-found errors. After fix, 3 back-to-back live queries each emitted the full event sequence with zero drops. Threading vs subprocess: query jobs run in-thread (3-10s I/O-bound; shared SQLite handle helps); subprocess + psutil tree-kill stays as the right call for the multi-hour Stage 5 ingest. Streamlit still boots cleanly. Commits on `rewrite/api`: `4f05f6f` (schema migration), `790bfe3` (jobs subsystem + routers + DTOs + startup reconciler), `1231244` (tests + OpenAPI fix for event DTOs), `9f3fe5d` (Turso retry hardening). Next: Stage 5 (Jobs subsystem — ingest as subprocess + `--stages` CLI flag). | |
| - **2026-05-17 (BE)** — Stage 3 (Read-only routers) landed on `rewrite/api`. New files: `src/api/routers/{books,history,costs,tools,indexes,processing_log,labels,pdf}.py` and `src/api/dto/{books,query,costs,tools,indexes,processing_log,labels,pdf_pages}.py`. Tests: `tests/test_{books,history,costs}.py` — 27 tests, all pass; Stage 1+2's 14 tests still pass. `src/api/errors.py` extended with `BookNotFound` (404 BOOK_NOT_FOUND with `details.bookId`), `QueryRunNotFound` (404 with `details.runId`), `NotIndexed` (409), `BadRequest` (400). `src/api/app.py` mounts all 8 new routers and installs a `RequestValidationError` handler that re-shapes Pydantic 422s into the canonical `BAD_REQUEST` envelope. Wire surface: 22 OpenAPI paths (Stage 1+2 = 5, Stage 3 adds 17); 44 component schemas (was ~7). Path params use camelCase URL templates (`/books/{bookId}`, `/query/runs/{runId}`) with `Path(..., alias=...)` keeping Python args snake_case. All DTOs inherit `ApiModel` → camelCase wire keys. Enums are `enum.Enum` → real OpenAPI unions (BookStatusEnum, RouterModeEnum, CostGroupByEnum, etc.) so FE codegen produces TS unions. `BookStatusCountsDTO.ocr_done` is the **only** explicit underscore-on-the-wire exemption — pinned to honour CONTRACT.md §10's literal-status-key shape. The global camelCase regression in `test_costs.py:test_every_schema_property_is_camelcase` sweeps every schema in the spec and asserts no other leaks. Cost-monitoring gate: `MAX(llm_calls.id)` before = 2481, after the full test + live verification = 2481. **0 new paid calls** (Stage 3 only reads `llm_calls`). Admin gates verified: viewer → `/processing-log` → 403 FORBIDDEN envelope; admin → 200. Streamlit boots cleanly. Commits on `rewrite/api`: `f7bc2ce` (DTOs + errors), `116c119` (routers + app wiring), `19e89e6` (tests). Next: Stage 4 (Query + SSE — `jobs/` subsystem, `POST /query` returning 202 jobId/sseUrl, `GET /jobs/{id}/events` SSE with replay-from-`Last-Event-ID`, `progress.costSoFarUsd` per CONTRACT.md §12.3). | |
| - **2026-05-17 (BE)** — Stage 2 (Auth) landed on `rewrite/api`. New files: `src/api/auth/{__init__,sessions,router}.py`, `src/api/deps.py`, `src/api/middleware/{__init__,csrf}.py`, `src/api/dto/auth.py`, `tests/test_auth.py`. Schema: `sessions` table + 2 indexes added inside `SCHEMA_SQL` (CREATE TABLE IF NOT EXISTS — idempotent). Additive edits: `src/lib/auth/repo.py` got `from_session_token(token)` + `user_for_username(username)`; `src/api/errors.py` got `Unauthenticated` (401, UNAUTHENTICATED) and `Forbidden` (403, FORBIDDEN); `src/api/app.py` mounts the auth router and registers the CSRF middleware. Wire shape: cookie `session=<urlsafe(32)>; HttpOnly; SameSite=Lax; Path=/; Max-Age=2592000` (30d), matching the row's `expires_at`. Verified live end-to-end on port 8011: bad creds → 401, good → 200 + Set-Cookie + MeResponse, /me works, POST /logout without `X-Requested-With` → 403 FORBIDDEN, with header → 204, post-logout /me → 401. Cost-monitoring gate: 0 new `llm_calls` rows during the full test cycle (auth path is pure SQLite + TOML; never imports an LLM SDK). camelCase regression test extended to cover the new DTOs. Streamlit still boots. Next: Stage 3 (Read-only routers — `/books`, `/history`, `/costs`, `/tools`, `/indexes`, `/processing-log`, `/labels`, `/books/status-counts`). | |
| - **2026-05-17 (BE)** — Stage 0 + Stage 1 landed on `rewrite/api`. New `src/api/` package: `app.py` (FastAPI app + ApiError handler), `config.py` (port + git-sha + masked-config), `errors.py` (ApiError stub), `dto/common.py` (ApiModel base, ErrorResponse, JobStartResponse — alias_generator=to_camel applied), `dto/system.py` (HealthResponse, HealthCheck, VersionResponse), `routers/system.py` (`GET /system/health`, `GET /system/version`). Health probe wraps the existing `src.lib.system.health` checks for DB/Qdrant/API-keys and adds a Storage probe via `src.lib.storage.get_storage().list_book_ids()`. Stale-pricing check walks `tools_registry.yaml`; current state is `stalePricing: []` because every tool's `cost_model.last_updated` is within 90 days. New test `tests/test_system_health.py` (5 tests, all pass) covers the OpenAPI shape, camelCase enforcement, the `/health` shape, and secret-masking in `/version`. `pyproject.toml` gained `fastapi`, `uvicorn[standard]`, `python-multipart`, `psutil`, `httpx`, `pytest-asyncio`. No paid-API calls made; `llm_calls` row count unchanged (519 → 519). Streamlit smoke-imports unaffected. Next: Stage 2 (Auth — sessions table + login/logout/me + CSRF middleware). | |
| - **2026-05-17 (orchestrator)** — CONTRACT.md v2 (Express proxy, cost monitoring §12), FRONTEND_BUILD.md updated for Vite+Express stack, BACKEND_BUILD.md §11 cost monitoring added. Stage 0 FE complete (per AI Studio); Stage 1 FE ready to start. BE branch not yet created (waiting on Streamlit indexing to finish). | |
| - _(future entries here)_ | |