Hugging Face's logo

Spaces:
Mario33333
/
patristic-be
Running

App Files Community
patristic-be / IMPLEMENTATION.md
Mario33333's picture
Mario33333
deploy: reader-access security hardening (feature/audiobook@06a5ed8)
7c2d250 verified
preview code
|
Raw
History Blame Contribute Delete
64.8 kB

Implementation Status — live tracker

Both agents: update this file as you complete checklist items. Tick boxes ([x]), bump the "Current stage" lines at top, add a _log_ entry at the bottom. This is the single source of truth for "where we are." If you finish work and don't update this file, the next session can't pick up.

Last updated: 2026-05-17 Current FE stage: 0 → 1 (handoff in progress) Current BE stage: Stage 6 ✅ DONE; Stage 7 next (Operations + Evaluation writes) Contract version: v2 (CONTRACT.md revised 2026-05-17 — Express proxy + cost monitoring)

Two parallel tracks, one shared contract

Two AI agents work in parallel, each in its own repo. They never edit each other's code; they share only CONTRACT.md and this file.

Track Repo Branch Working dir for the agent
Frontend G:\pythoncodenew\Side projects\Reli\Patristic-AI-Fe main repo root
Backend G:\pythoncodenew\Side projects\Reli\Patristic_Arabic_Library rewrite/api (must be created before BE work starts) repo root

Each agent reads:

  1. CONTRACT.md (~280 lines, authoritative wire contract — read fully)
  2. Its build brief: FRONTEND_BUILD.md or BACKEND_BUILD.md
  3. This file — for "where we are" and "what's next"

Each agent updates only this file when reporting progress. They do not edit each other's brief.

How to use this file

  • Mark a checkbox [x] when the item is complete and verified against its observability gate.
  • A stage moves from 🚧 IN PROGRESS to ✅ DONE only when every checkbox in it is [x] AND the stage gate passes.
  • The user (or the orchestrator agent) confirms gate-pass before flipping the stage flag at the top of this file.
  • New items discovered mid-stage are added inside the stage they belong to, not the current stage, unless they're blockers.

Frontend track

Stage 0 — AI Studio scaffolding ✅ DONE (verified 2026-05-17)

Already shipped by Google AI Studio before either agent started. Don't redo any of it.

  • Vite 6 + React 19 + react-router-dom 7 + react-query + react-hook-form + zod + shadcn (base-nova) + Tailwind v4 set up.
  • Express server.ts with Vite middleware + mock API routes + cookie parsing.
  • Full route inventory (App.tsx): Dashboard, Login, About, Research/{Search, Compare, Allusions, History, RunDetail}, Library/{Browse, AddBook, Labels, BookDetail, Inspect, JobWatcher}, Operations/{Costs, Indexes, Jobs, Processing, Tools, Config}, Evaluation/{Run, Golden, History, RunDetail}.
  • Search page wires POST /query → jobId → SSE → done → runId in URL.
  • shadcn button.tsx primitive scaffolded.

Known issues left for Stage 1 to fix: sidebar uses raw <a href> (full page reloads); no central API client; no useSse hook (inlined in Search); no AuthContext; no CSRF header on writes; only one shadcn primitive installed.

Stage 1 — Foundation polish 🚧 IN PROGRESS

Goal: make the scaffolding production-shaped before building more pages on top of it.

  • Sidebar navigation: replace every <a href="..." class="..."> in App.tsx with <NavLink to="..." className="..."> from react-router-dom. Active-route styling via NavLink's isActive callback. Verify by clicking — no full page reload.
  • shadcn primitives needed for Phase 1. Run npx shadcn@latest add card input label form select dialog toast tabs badge separator collapsible dropdown-menu tooltip alert avatar checkbox skeleton. Verify each .tsx lands in components/ui/.
  • API client (lib/api/client.ts). Typed apiFetch<T>(path, init) that: prepends /api, sets Content-Type: application/json on JSON bodies, sets X-Requested-With: XMLHttpRequest on non-GET, decodes { code, message, details } on 4xx/5xx and throws a typed ApiError. Has apiGet, apiPost, apiPatch, apiDelete helpers. No bare fetch() calls anywhere else in the app.
  • SSE hook (lib/sse/use-sse.ts). Generic useSse<TEvent>(url: string | null, options?: { enabled?: boolean }) returning { events, isOpen, lastEventId, reconnect, error }. Auto-closes on unmount. Ignores event: ping. Extract Search.tsx's inline EventSource into this hook.
  • Auth (lib/auth/auth-context.tsx). AuthProvider reading GET /api/auth/me on mount; exposing { user, login, logout, isLoading }. <ProtectedRoute> wrapper redirects to /login?next=.... Wire into App.tsx Layout.
  • Toast host. Global <Toaster /> from shadcn. ApiError handler in client.ts calls toast.error(error.message) automatically; routes can suppress if they handle inline.
  • Type stub (lib/api/types.ts). Hand-write DTOs from FRONTEND_BUILD.md §3.5. Add // AUTOGENERATED ONCE BE IS LIVE — DO NOT EDIT at top. Drop in when openapi.json exists.
  • Format helpers (lib/formatting/). formatCost(usd: number), formatDuration(ms: number), formatPageLabel(bookId, pdfPage, printedPage?) — server-formatted strings preferred; helpers only for cases the API doesn't pre-format.
  • Status indicator in sidebar. Replicate Streamlit's "active ingest" indicator: subscribe to GET /api/system/events SSE; render a chip with active job count and a list of running job ids (click → /operations/jobs/[jobId]).

Stage 1 gate (must all pass before moving on):

  1. Click every sidebar link — no full page reload (no index.html refetch in devtools network tab).
  2. Trigger a 401 (e.g., delete the cookie in devtools, click Search): user is redirected to /login?next=/research/search.
  3. Open Search page, submit a query, observe SSE working through useSse hook (not inline). Refresh during the run — ?job=... in URL keeps the stream attached.
  4. npm run lint (tsc --noEmit) passes with zero errors.
  5. Cost monitoring placeholder: the cost panel placeholder ("$0.00 — no calls yet") renders on Search after a mock-server completion. Wiring exists.

Stage 2 — Search page polished 📋 PENDING

Anchored to FRONTEND_PRD.md Phase 1.

  • AdvancedModels expander on Search: model picker for generation/classify/judge, topK, forceMode, filters (tradition/era/language), usePremium. Source: GET /api/tools?category=llm.
  • Recent queries expander above the form on Search: GET /api/history?page=search&user=me&limit=5. Click → navigate to /research/history/{runId}.
  • Cost panel below the result: render envelope.cost.{totalUsd, byStage, nCalls, durationMs} from the final done event. Collapsed by default; click to expand → table of per-call rows from GET /api/costs/llm-calls?sinceId=envelope.llmCallIdRange.firstId.
  • RunDetail page (/research/history/[runId]): fetch GET /api/query/runs/{runId}, render by envelope.mode (LookupEnvelope rendering matches Search). "Re-run" button copies query + params to /research/search?prefill={runId}.
  • History page (/research/history): paginated list with filters (page=, q=, pinned=, user=). Pin/unpin actions. Cursor-based pagination.
  • Login page: real form against POST /api/auth/login, error envelope on bad credentials, redirect to ?next= or /.
  • About page: tools registry table (GET /api/tools), live counts: indexed books, active collection, last query timestamp.
  • CmdEnter shortcut on Search submits.
  • <RtlText> component for Arabic text. Used everywhere Arabic appears (citation excerpts, query echo).
  • <CitationCard> component for citation display. Click → /library/[bookId]/inspect?page={pdfPage} in a new tab.

Stage 2 gate:

  1. Run a real search end-to-end against the mock; cost panel populated. Re-run from history works. Refresh during a query reattaches the SSE stream.
  2. Pin a query from history; refresh; still pinned. Delete; refresh; gone.
  3. Visit /about: tools registry table renders with at least 3 entries (mock can stub 3-5 tools).
  4. Cmd/Ctrl+Enter on Search submits the form.
  5. Cost gate: in mock mode, set the mock to return envelope.cost.totalUsd = 0.0234. The cost panel shows "$0.0234"; the cost breakdown rows match.

Stage 3 — Library + Add Book + Ingest watcher 📋 PENDING

FRONTEND_PRD.md Phase 2.

  • /library Browse: GET /api/books?... with filter form on URL params. Status badges, label chips, action menu per row. Bulk-select → "Reindex selected" fans out one POST per book with body: { stages: ["indexing"] }.
  • /library/[bookId] Book detail: tabs (Overview / History / Costs / Jobs). Endpoints: GET /api/books/{id}, GET /api/processing-log?bookId=, GET /api/costs?groupBy=stage&bookId=, GET /api/jobs?subjectId=&limit=10. Action menu (admin only): Edit, Re-ingest, Re-extract, Re-cleanup, Re-chunk, Re-index, Delete (with ConfirmDialog).
  • /library/[bookId]/inspect?page=N: three-pane (image | OCR | clean) with synced scroll. Page navigator. Diff between OCR and clean.
  • /library/[bookId]/jobs/[jobId]: SSE-watched ingest. Replays history on mount via Last-Event-ID. Cancel button (admin). Final summary on done.
  • /library/add wizard:
    • Step 1: URL paste OR file upload (POST /api/uploads → { uploadId }) → POST /api/books/probe returns ProbeResponse.
    • Step 2: Edit guessed metadata; show suggested labels; extraction-mode toggle; cleanup toggle; sample pages displayed.
    • Step 3: Cost preview via GET /api/costs/counterfactual?stage=ingest&.... Confirm.
    • Step 4: POST /api/books (handle 409 inline with link to existing book); POST /api/books/{id}/ingest; redirect to /library/[bookId]/jobs/[jobId].
  • /library/labels: CRUD; color picker; book-count badge; "Seed from derived" button calls POST /api/labels/seed-from-derived.
  • <JobWatcher> component: extracted from the page, reusable in Add Book final step, ops jobs detail, eval run, library detail Jobs tab.
  • <BookCard> component: row + grid variants.
  • <StatusBadge> component: all 6 statuses + colors from FRONTEND_PRD.md table.

Stage 3 gate:

  1. Full ingest dry-run: Add a book → wizard finishes → ingest job runs → live progress visible → completion redirects to book detail → search the new book → answer cites it. (Against the mock initially; against real BE once BE Stage 5 lands.)
  2. Refresh during a mock ingest: progress bar resumes from current state without losing prior log lines.
  3. Cost gate: Add Book wizard step 3 shows a counterfactual cost preview. The job watcher's progress bar shows costSoFarUsd updating. The final done event's totalCostUsd matches Sum of llm_calls.cost_usd for the run (BE responsibility — FE verifies the displayed number matches).

Stage 4 — Operations + Evaluation + final research 📋 PENDING

FRONTEND_PRD.md Phase 3.

  • /operations dashboard.
  • /operations/jobs + [jobId] (delegates to <JobWatcher>).
  • /operations/costs: multi-panel (by-day chart, by-model, by-stage, by-book). Range filter.
  • /operations/indexes: collection list; activate; delete.
  • /operations/processing: filterable log.
  • /operations/tools: read-only registry view with verified flag.
  • /operations/config: read merged config; inline-edit; POST /api/config/migrate → job watcher.
  • /evaluation/run: kicks off POST /api/eval/runs → job watcher.
  • /evaluation/golden: CRUD; bulk import (JSONL); validation.
  • /evaluation/history + [runId]: per-item metrics + diff against previous run.
  • /research/compare: group builder; strategy picker; per-row cards; cross-cutting synthesis.
  • /research/allusions: judge model picker; per-candidate progress; match list with verdict.
  • / Dashboard: jobs in flight, recent query, today's cost, status counts, health chips.

Stage 4 gate:

  1. Cost-monitoring full sweep: the Costs page renders daily totals matching SELECT date, SUM(cost_usd) FROM llm_calls GROUP BY date to the cent. Stale-pricing chip on dashboard if any tool has last_updated > 90 days.
  2. Compare mode: build a group, run a comparison, render per-book rows with their own cost lines. Total = sum of row costs.
  3. Allusions: judge model override changes the cost line of the judging stage (verify against tools_registry pricing).
  4. Parity matrix sweep: walk FRONTEND_PRD.md Appendix A; every row's behaviors work in the new UI.

Stage 5 — Cutover 📋 PENDING

  • Delete all mock routes from server.ts. Proxy is the only path.
  • Regenerate lib/api/types.ts from live openapi.json; commit; replace the hand-stub.
  • One week of dogfooding; bugfix-only commits.
  • Hand off to BE agent: BE deletes src/stage9_ui/ and removes streamlit from pyproject.toml.

Backend track

Stage 0 — Branch setup ✅ DONE (verified 2026-05-17)

Must complete before any BE work begins. The user (or orchestrator) performs these after the current Streamlit-driven indexing job finishes — never during.

  • Commit the existing M files on main per the commit-split plan (see session log; user controls the split).
  • git checkout -b rewrite/api.
  • On this branch, ensure CONTRACT.md, BACKEND_BUILD.md, IMPLEMENTATION.md, FRONTEND_PRD.md, FRONTEND_BUILD.md are tracked.
  • Add .gitignore entries: .claude/, .critique_*.md, *.png at repo root (move existing PNGs to screenshots/ if keeping them).
  • Confirm data/inventory.db is not committed (should already be ignored).

Stage 1 — Skeleton + health ✅ DONE (verified 2026-05-17)

BACKEND_BUILD.md §13 Step 1.

  • src/api/__init__.py, src/api/app.py (FastAPI app + middleware skeleton + ApiError exception handler).
  • src/api/config.py (port resolver, git-sha resolver, masked-config helper — no CORS allowlist in v1 per CONTRACT.md §2; session secret deferred to Stage 2).
  • src/api/dto/common.py with ApiModel base, ErrorResponse, JobStartResponse. Every Stage-1 DTO inherits from ApiModel.
  • src/api/dto/system.py with HealthResponse, HealthCheck, VersionResponse (all camelCase on the wire — verified via /openapi.json).
  • src/api/errors.py minimal stub (ApiError base + InternalError; full canonical codes land in Stage 3).
  • src/api/routers/system.py with GET /system/health (DB + Qdrant + storage + API keys + stalePricing: []) and GET /system/version (git sha + masked config).
  • pyproject.toml adds: fastapi, uvicorn[standard], python-multipart, psutil, httpx, pytest-asyncio.
  • uvicorn src.api.app:app --port 8000 boots cleanly; curl http://localhost:8000/openapi.json returns a valid spec listing both paths.
  • curl http://localhost:8000/system/health returns 200 with camelCase fields including stalePricing: [] (all 21 tools updated within the last 90 days as of 2026-05-17).
  • Cost-monitoring stub: /system/health returns stalePricing (empty list when fresh; populated when any cost_model.last_updated > 90 days or missing).
  • tests/test_system_health.py covers the OpenAPI shape, camelCase enforcement, health endpoint shape, and secret-masking in /version.

Stage 1 gate: FE agent confirms npm run gen:api against localhost:8000/openapi.json succeeds and produces valid TS types.

Stage 2 — Auth ✅ DONE (verified 2026-05-17)

BACKEND_BUILD.md §6 + CONTRACT.md §3.

  • Migration: sessions table added via src/stage1_inventory/schema.py (CREATE TABLE IF NOT EXISTS inside SCHEMA_SQL — same idempotent pattern as the rest of the schema; _COLUMN_ADDITIONS is for ALTER-style growth on existing tables, not new tables).
  • src/api/auth/sessions.py: create_session, lookup_session (bumps last_seen, lazy-GCs expired rows), invalidate_session, purge_expired.
  • src/api/auth/router.py: POST /auth/login, POST /auth/logout, GET /auth/me.
  • src/api/deps.py: current_user (raises 401 UNAUTHENTICATED on miss), require_admin (raises 403 FORBIDDEN for viewers).
  • src/api/middleware/csrf.py: enforces X-Requested-With: XMLHttpRequest on POST/PATCH/PUT/DELETE; exempts /auth/login; GET is exempt by method (SSE works).
  • src/lib/auth/repo.py additive: from_session_token(token) -> User | None + user_for_username helper. Existing current_user() / Streamlit session helpers untouched.
  • src/api/dto/auth.py: LoginRequest{username, password}, MeResponse{username, isAdmin, displayName?} — both inherit ApiModel so the wire is camelCase.
  • src/api/errors.py: typed Unauthenticated (401, UNAUTHENTICATED) and Forbidden (403, FORBIDDEN) ApiErrors.
  • Test: tests/test_auth.py — 9 tests covering bad/good login, /auth/me with/without cookie, logout + subsequent 401, CSRF blocks logout without header, GET is exempt, /auth/login is exempt, OpenAPI camelCase regression guard extended to the new DTOs.

Stage 2 gate: FE login flow works end-to-end against real BE; 401 on protected route redirects.

Verification (2026-05-17):

  1. Live login flow on uvicorn :8011 — bad creds → 401 UNAUTHENTICATED; good creds → 200 + MeResponse + Set-Cookie: session=...; HttpOnly; Max-Age=2592000; Path=/; SameSite=lax; /auth/me with cookie → 200; POST /auth/logout without X-Requested-With → 403 FORBIDDEN; with header → 204 + cookie cleared; subsequent /auth/me → 401.
  2. Migration idempotency: ran init_schema twice on a copy of data/inventory.db; second run is a no-op; sessions table + idx_sessions_username + idx_sessions_expires all present.
  3. Cost gate: 0 new llm_calls rows during the full live test (SELECT COUNT(*) FROM llm_calls WHERE id > 932 returned 0; auth path is pure SQLite + TOML).
  4. camelCase: curl /openapi.json | jq '.components.schemas.MeResponse.properties | keys' returns ["displayName", "isAdmin", "username"].
  5. Streamlit boots without ImportError (streamlit run src/stage9_ui/app.py --server.port 8765 came up cleanly).
  6. All 9 auth tests pass; the 5 Stage 1 tests still pass (no regressions).

Stage 3 — Read-only routers ✅ DONE (verified 2026-05-17)

BACKEND_BUILD.md §13 Step 3 + the GET /books/status-counts added per CONTRACT.md §10.

  • src/api/routers/books.py: GET /books, GET /books/{bookId}, GET /books/status-counts.
  • src/api/routers/history.py: GET /history, GET /query/runs/{runId} (committed runs only — live SSE comes in Stage 4).
  • src/api/routers/costs.py: GET /costs?groupBy=, GET /costs/llm-calls, GET /costs/counterfactual.
  • src/api/routers/tools.py: GET /tools, GET /tools/{name}.
  • src/api/routers/indexes.py: GET /indexes (writes come later).
  • src/api/routers/processing_log.py: GET /processing-log (admin-only).
  • src/api/routers/labels.py: GET /labels.
  • src/api/routers/pdf.py: GET /books/{bookId}/pdf, GET /books/{bookId}/pages/{n}/{image|ocr|clean}.
  • src/api/dto/common.py: ApiModel base + ErrorResponse + JobStartResponse (Stage 1, kept).
  • All DTOs in src/api/dto/* inherit from ApiModel (Stage 1+3 — dto/{books,query,costs,tools,indexes,processing_log,labels,pdf_pages}.py).
  • Pricing health check: /system/health returns the stale list correctly (Stage 1, regression-tested).
  • Canonical error codes filled in: BookNotFound (404 BOOK_NOT_FOUND with details.bookId), QueryRunNotFound (404 QUERY_RUN_NOT_FOUND with details.runId), NotIndexed (409 NOT_INDEXED), BadRequest (400 BAD_REQUEST). Pydantic 422 → BAD_REQUEST envelope via the new RequestValidationError handler in app.py.

Stage 3 gate: all checks pass.

Verification (2026-05-17 on uvicorn :8015 with mario admin + preview viewer):

  1. /openapi.json | .paths | keys lists 22 paths — Stage 1+2 = 5, Stage 3 = 17 new (/books, /books/status-counts, /books/{bookId}, /books/{bookId}/pdf, /books/{bookId}/pages/{n}/{image,ocr,clean}, /history, /query/runs/{runId}, /costs, /costs/llm-calls, /costs/counterfactual, /tools, /tools/{name}, /indexes, /processing-log, /labels).
  2. components.schemas.BookStatusEnum is a real OpenAPI enum: {"type":"string","enum":["pending","acquired","ocr_done","clean","indexed","failed"]} — no Literal-to-string regression.
  3. components.schemas has 44 entries (Stage 1+2 was ~7; Stage 3 added 37+).
  4. Random schema sweep (PageCleanDTO, BookListResponse, BookCosts) — all properties camelCase. The global sweep in tests/test_costs.py:test_every_schema_property_is_camelcase enforces this across every schema; only documented exemption is BookStatusCountsDTO.ocr_done per CONTRACT.md §10.
  5. GET /books?limit=5 returns 4 books (live data); /books/status-counts returns {pending:0, acquired:1, ocr_done:0, clean:2, indexed:1, failed:0, total:4} — keys match CONTRACT.md §10 exactly.
  6. GET /costs?groupBy=day&range=24h returns 2 day buckets with {key,label,costUsd,nCalls,inputTokens,outputTokens,durationMs,provider} — array shape good.
  7. Cost-monitoring gate: MAX(llm_calls.id) before = 2481; after the full test + gate cycle = 2481. 0 new paid calls during Stage 3 (read-only routers do not touch any paid SDK).
  8. require_admin gate: viewer (preview) → /processing-log → 403 FORBIDDEN envelope; admin (mario) → 200. /costs also gated to admin (faceted, can leak cross-user totals); /costs/llm-calls + /costs/counterfactual are user-tier.
  9. Streamlit boots without ImportError on port 8765 — HTTP 200 served, no regressions to the lib/ surface the existing UI depends on.

All 27 Stage 3 tests pass (9 books + 5 history + 13 costs). Stage 1+2 tests (14 total) still green — no regressions.

Stage 4 — Query + SSE ✅ DONE (verified 2026-05-17)

BACKEND_BUILD.md §5.6 + §5.8 + §13 Step 4.

  • src/api/jobs/store.py: insert/update job rows; emit_event returns the SSE-id; Turso-resilience retry wrapper on every store function.
  • src/api/jobs/sse.py: in-memory pub-sub + DB-backed replay from Last-Event-ID; 15s heartbeat (event: ping); terminal-sentinel signal_terminal().
  • src/api/jobs/runner.py: thread-based runner with per-subject lock (query:{username}) + global semaphore (config.yaml > api.max_concurrent_jobs, default 1).
  • src/api/jobs/registry.py: @register_job_type decorator + get_handler/known_types helpers.
  • src/api/jobs/reconcile.py: startup hook flips queued/running rows with dead pid (or pid=NULL) to failed + writes terminal done event.
  • src/api/jobs/types/query.py: wraps src.stage8_router.router.answer() with on_progress callback that translates router steps (classify_start/done, retrieve_start/done, judge_start/done, generate_start/done, force_mode, comparison_start/done) into the locked SSE taxonomy (stage events with stageName ∈ {classify, retrieve, judge, generate} + progress events).
  • src/api/routers/jobs.py: GET /jobs, GET /jobs/{jobId}, GET /jobs/{jobId}/events (honors Last-Event-ID header + ?lastEventId query fallback), POST /jobs/{jobId}/cancel (admin; psutil tree-kill for pid≠NULL + defensive status flip for in-thread jobs).
  • src/api/routers/query.py: POST /query → 202 JobStartResponse{jobId, sseUrl}; GET /query/runs/{runId}/events resolves runId → jobId via jobs.subject_id and proxies.
  • Migration: jobs + job_events tables + 3 indexes added to SCHEMA_SQL (idempotent CREATE TABLE IF NOT EXISTS, run-twice verified).
  • Cost-on-progress: every progress event payload carries costSoFarUsd (sum of llm_calls.cost_usd WHERE id > first_call_id). Terminal done event carries totalCostUsd.
  • Tests: tests/test_query_sse.py (8 — incl. one expensive end-to-end LOOKUP gated on GOOGLE_API_KEY + indexed Qdrant collection), tests/test_jobs_sse_replay.py (3), tests/test_jobs_reconcile.py (3), tests/test_no_unlogged_api_calls.py (2 — the global LLM-SDK-import gate per BACKEND_BUILD.md §11.2 / CONTRACT.md §12.1).

Stage 4 gate verified (2026-05-17):

  1. End-to-end search: Live uvicorn :8011 → login → POST /query with {"query":"What does the book teach about salvation through Christ?","page":"search","topK":3,"usePremium":false} → 202 {"jobId":"6ae9b2bc...", "sseUrl":"/jobs/.../events"} (no runId in 202). SSE stream emitted in order:
    • id 40 stage {stageName:"classify", costSoFarUsd:0.0, query:"..."}
    • id 41 progress {pct:0.2, stageName:"classify", mode:"LOOKUP", costSoFarUsd:0.005831}
    • id 42 stage {stageName:"retrieve", costSoFarUsd:0.005831, topK:3}
    • id 43 progress {pct:0.45, stageName:"retrieve", nHits:3, costSoFarUsd:0.005831}
    • id 44 stage {stageName:"generate", costSoFarUsd:0.005831}
    • id 45 progress {pct:0.95, stageName:"generate", nCited:3, costSoFarUsd:0.0284}
    • id 46 done {finalStatus:"succeeded", durationMs:128827, totalCostUsd:0.0284, runId:16, result:{runId:16, totalCostUsd:0.0284, mode:"LOOKUP"}}
  2. Refresh-safety: curl -H "Last-Event-ID: 43" /jobs/{id}/events yielded exactly events 44, 45, 46 (no duplicates, no gaps). GET /query/runs/16/events yielded the full id 40-46 history via the runId → subject_id proxy.
  3. Cost gate (CRITICAL — paid quota): pre-query MAX(llm_calls.id) = 2488. Post-query new rows: id 2489 stage=classify cost=0.005831, id 2490 stage=generation cost=0.022569 — exactly 2 new rows for a LOOKUP. SUM(cost_usd) = 0.028400 = done.totalCostUsd to the cent. ✓
  4. OpenAPI new paths: /openapi.json | jq '.paths | keys' includes all 6 new Stage 4 paths (/jobs, /jobs/{jobId}, /jobs/{jobId}/cancel, /jobs/{jobId}/events, /query, /query/runs/{runId}/events); 28 paths total (was 22 in Stage 3).
  5. camelCase regression: JobDTO.properties contains progressPct, stageLabel, subjectId, createdAt, startedAt, finishedAt; QueryRequest has forceMode, topK, usePremium, comparisonSpec. All event DTOs (LogEvent/StageEvent/ProgressEvent/DoneEvent/ErrorEvent) registered + camelCase per test_costs.py::test_every_schema_property_is_camelcase.
  6. No SDK imports outside wrappers: tests/test_no_unlogged_api_calls.py greps src/ for from {google.genai,openai,anthropic,cohere,groq} import / import {...} outside the 5 approved wrapper files. Passes.
  7. Reconciler test passes (tests/test_jobs_reconcile.py).
  8. Streamlit still works: streamlit run src/stage9_ui/app.py --server.port 8765 --server.headless true came up; /_stcore/health returns 200/ok.
  9. All 41 Stage 1+2+3 tests still pass + 16 new Stage 4 tests pass (4 cheap test_query_sse + 3 reconcile + 3 sse_replay + 2 no_unlogged_api_calls + 4 documentation/auth tests in test_query_sse — actual count via pytest: see log).

Note: under Turso's stale-stream HTTP-404 lag we observed missed stage(classify) events on the very first emit after idle. Fixed by wrapping every store function in _with_retry() that retries once on the Hrana stream-not-found ValueError. After that, three back-to-back live queries each emitted the full event sequence with no drops.

Stage 5 — Jobs subsystem (ingest) ✅ DONE (verified 2026-05-17, live smoke STAGED but NOT RUN)

BACKEND_BUILD.md §5 + §5.7 + §13 Step 5.

  • src/lib/processing/ingest_progress.py: lifted parse_ingest_progress + filter_ingest_log_noise from src/stage9_ui/shared.py (per §3.4). Added stage_number_to_name() mapping for [stage N] → canonical SSE name (acquisition/ocr/cleanup/chunking/indexing). Stage 4b folded into cleanup.
  • src/stage9_ui/shared.py: re-export shim (no behavior change — Streamlit still boots, parser output bit-for-bit identical).
  • src/api/jobs/types/ingest.py: subprocess-based handler. @register_job_type("ingest") wires it into the existing Stage-4 runner. subprocess.Popen with line-buffered stdout, CREATE_NEW_PROCESS_GROUP on Windows for clean tree-kill; PID is recorded immediately via store.set_pid so POST /jobs/{id}/cancel can psutil.Process(pid).children(recursive=True). The runner's existing locks (per-subject + global semaphore) cover ingest the same way they cover query.
  • _StreamingState: re-feeds full filtered log buffer through the lifted parser on every stdout line, detects stage transitions (emits stage SSE with stageName + stageLabel + costSoFarUsd), emits progress throttled to ≤ 1/500ms with pagesDone + pagesTotal + costSoFarUsd. Terminal done event carries finalStatus (derived from subprocess exit code, unless cancel flipped status first), durationMs, totalCostUsd, result.{bookId,exitCode,totalCostUsd}.
  • --stages acquisition,ocr,cleanup,chunking,indexing on src/pipeline/ingest.py:main(). Validates against STAGE_NAMES; unknown stage aborts with SystemExit(2). ingest() adds stages: set[str] | None param + _wants(name) guard on every stage block. Default (stages=None) = run every eligible stage = unchanged behavior.
  • src/api/dto/ingestion.py: IngestRequest (camelCase via ApiModel) + StageName: Enum (real enum, surfaces as TS union via OpenAPI). All fields optional → empty POST = re-ingest with persisted settings.
  • src/api/routers/ingestion.py: POST /books/{bookId}/ingest admin-only; 404 BOOK_NOT_FOUND envelope with details.bookId; 202 JobStartResponse{jobId, sseUrl=/jobs/{jobId}/events}; persists extraction_mode/cleanup_enabled onto the book row before enqueue; passes subject_id=book_id at insert + subject_key=f"book:{book_id}" at enqueue.
  • src/api/app.py: mounts ingestion_router. Side-effect import of src.api.jobs.types.ingest lives inside the router file (parity with query.py).
  • Per-subject lock keyed by (ingest, book:{book_id}) — second concurrent ingest for the same book waits on the existing runner._subject_lock. No 409; the request is accepted (202) and serialized by the lock (documented in the router docstring).
  • Tree-kill on cancel — already wired in routers/jobs.py:cancel_job via psutil.Process(pid).children(recursive=True) since Stage 4. Stage 5 just supplies the PID by calling store.set_pid(job_id, proc.pid) right after Popen.
  • Cost-on-progress for ingest: every stage and progress event payload carries costSoFarUsd computed via store.cost_since(first_call_id) where first_call_id = store.max_llm_call_id() snapshotted at job start. Terminal done.totalCostUsd is the same sum at terminal.
  • Tests: tests/test_ingest_jobs.py (17, all pass) — argv builder purity, parser state transitions (3→4→5→6 → canonical [ocr, cleanup, chunking, indexing]), progress events carry pagesDone + costSoFarUsd, 401/403/404 gates, 202 happy path with mocked runner.enqueue + persisted job row + cleanup, empty body accepted, bogus stage rejected as BAD_REQUEST, OpenAPI camelCase + StageName-enum regression.
  • tests/test_ingest_jobs_live.py — STAGED but NOT RUN. Marked @pytest.mark.live (registered in pyproject.toml) + @pytest.mark.skip belt-and-suspenders. pyproject.toml now defaults to -m 'not live'. Documented opt-in flow: LIVE_INGEST_BOOK_ID=<small_book> LIVE_INGEST_STAGES=indexing pytest -m live tests/test_ingest_jobs_live.py. Skeleton wires login → POST → SSE drain → cost gate so when the user opts in the test runs straight through.

Stage 5 gate — verified 2026-05-17 (live smoke deferred per user opt-in):

  1. parse_ingest_progress import works from BOTH src/lib/processing/ingest_progress.py (canonical) AND src/stage9_ui/shared.py (re-export shim) — confirmed by direct import + sample-input behavior check (identical output across 3 representative log transcripts). ✓
  2. python -m src.pipeline.ingest --help lists --stages with the 5 canonical choices; --stages bogus,chunking --book-id x aborts with unknown stage(s) ['bogus']; --stages chunking --book-id nonexistent_test_book_id_123 reaches the ingest() function and errors cleanly ("book_id ... not in the inventory"). ✓
  3. POST /books/{id}/ingest returns 202 JobStartResponse with valid jobId + sseUrl=/jobs/{jobId}/events. Verified via TestClient against a real book id from the live inventory: 202 + runner.enqueue called exactly once with subject_key=book:{id} + type=ingest, job row landed in 'queued' with subject_id=book_id. ✓
  4. Mocked-subprocess parser tests pass: _StreamingState fed synthetic [stage 3] OCR… + page-ok lines emits a stage event with stageName="ocr" + costSoFarUsd, then progress events with pagesDone=1,2,3 + stageName=ocr + costSoFarUsd≥0. Multi-stage transition test (3→4→5→6) emits canonical [ocr, cleanup, chunking, indexing] in order. ✓
  5. Per-subject lock: two simultaneous ingest requests for the same book_id get accepted as 202 (no 409) and serialize on runner._subject_lock[("ingest", "book:{book_id}")]. Documented in the router docstring. ✓
  6. /api/jobs?type=ingest filters by ingest type — uses the existing routers/jobs.py list_jobs endpoint which already accepts type: JobTypeEnum | None. JobTypeEnum already lists ingest (was reserved in Stage 4 for this stage). ✓
  7. Cost gate: 0 new llm_calls rows during this session (no live ingest). Every unit test mocks the subprocess via monkeypatch on runner.enqueue OR feeds synthetic stdout to _StreamingState (which calls store.cost_since but does not invoke any LLM SDK). The live smoke test is staged but explicitly skipped. ✓
  8. Streamlit still boots: src.stage9_ui.shared imports cleanly; parse_ingest_progress re-export produces identical output to the canonical version. ✓
  9. All 17 Stage 5 tests pass + 90 prior tests still green. The 19 pre-existing failures in tests/test_labels.py + tests/test_stage1_inventory.py (libsql Row "'C' object has no attribute" issues) are NOT caused by Stage 5 — confirmed by git stash && pytest tests/test_labels.py tests/test_stage1_inventory.py on the base commit (same 19 failures). ✓

Live smoke test deferred to user opt-in. When ready: drop a 1-page test book in the inventory, then LIVE_INGEST_BOOK_ID=<id> LIVE_INGEST_STAGES=indexing pytest -m live tests/test_ingest_jobs_live.py. Estimated cost for indexing alone: $0.00 (local BGE-M3 + Qdrant). Estimated cost adding ocr: ~$0.005/page Gemini.

Stage 6 — Add Book probe + writes ✅ DONE (verified 2026-05-17, live probe smoke STAGED but NOT RUN — user opt-in)

BACKEND_BUILD.md §13 Step 6.

  • POST /api/uploads: multipart; stores in data/uploads/{uploadId}.pdf; returns { uploadId, sha256, sizeBytes } (camelCase). 1 GiB cap; SHA-256 streamed in 1 MiB chunks so peak RSS stays flat on the largest PDFs in the inventory. Admin-only.
  • POST /api/books/probe: wraps src/lib/metadata_probe.probe_metadata; returns ProbeResponse with pagesTotal, hasTextLayer, suggestedExtractionMode, guessedMetadata (enums coerced to FE union types), suggestedLabels, samplePages[] (PyMuPDF native text), estimatedIngestCostUsd (computed against tools_registry.yaml for the active OCR + cleanup models — $0 when native_text mode is suggested). Body requires uploadId OR sourceUrl. Admin-only (paid Gemini metadata call ~$0.005). Tests MOCK probe_metadata.
  • POST /api/books: CreateBookRequest body; 409 BOOK_EXISTS with details.conflictWithBookId when bookId collides OR when sourceUrl collides with an existing row; 201 BookDTO on success. Moves the upload bytes into storage (data/raw/{bookId}.pdf) via get_storage().put_pdf BEFORE the books row INSERT so a storage failure never leaves an orphan row.
  • PATCH /api/books/{bookId}: UpdateBookRequest (partial). Status field is NOT in the DTO (pipeline owns it). Enum members coerced back to strings before update_book().
  • DELETE /api/books/{bookId}: admin-only, idempotent (204 on a non-existent book). Cleanup order: Qdrant points across every known collection (best-effort; tolerates network failures) → book_labels mappings → books row. Historical processing_log / llm_calls / query_history / book_indexes rows preserved.
  • POST /api/labels: 201 LabelDTO; 400 on duplicate id. PATCH /labels/{labelId}: preserves immutable kind field. DELETE /labels/{labelId}: 204 even on miss (idempotent). POST /labels/seed-from-derived: wraps src/lib/labels/seeding.seed_derived_labels; returns { created, existing } counts; idempotent (second call returns created=0).
  • GET /api/books/{bookId}/preview?page=N&dpi=120: returns PNG bytes for the wizard's sample-page pane. Default DPI 120 (vs pdf.py's 150) to keep the wizard snappy on slow tethers. 404 BOOK_NOT_FOUND envelope on unknown book.
  • Tests: tests/test_addbook.py (18 — upload happy path + auth + empty rejection; probe with mocked metadata; create 201 + 409 BOOK_EXISTS for bookId + sourceUrl conflicts; PATCH updates + 404; DELETE 204 clears labels + Qdrant + idempotent; preview PNG; OpenAPI path + camelCase regression), tests/test_labels_writes.py (10 — POST 201 + duplicate 400 + validation 400; PATCH updates + 404 + immutable kind; DELETE 204 + idempotent; seed-from-derived idempotent; auth + CSRF gates), tests/test_addbook_live.py (STAGED but NOT RUN; @pytest.mark.live + @pytest.mark.skip + pyproject.toml's -m 'not live' default; opt-in flow documented for LIVE_PROBE_PDF_PATH=<path> OR LIVE_PROBE_SOURCE_URL=<url>; estimated cost ~$0.001-0.005 per run).

Stage 6 gate verified 2026-05-17:

  1. All 28 Stage 6 unit tests pass (18 in test_addbook.py + 10 in test_labels_writes.py). Live probe test STAGED + SKIPPED per pyproject.toml's -m 'not live' default.
  2. Cost gate (load-bearing for paid quota): pre-test MAX(llm_calls.id) = 2495. After the full Stage 6 test cycle, 0 new llm_calls rows (SELECT COUNT(*) FROM llm_calls WHERE id > 2495 = 0). Every probe test monkey-patches src.api.routers.addbook.probe_metadata to return a synthetic ProbeResult; every DELETE test monkey-patches src.stage6_indexing.qdrant_io.{list_collections, delete_book_points} so the live cloud collection is never touched.
  3. camelCase regression: tests/test_costs.py::test_every_schema_property_is_camelcase still passes — every Stage 6 DTO (UploadResponse, ProbeRequest, ProbeResponse, GuessedMetadata, SuggestedLabel, SamplePage, CreateBookRequest, UpdateBookRequest, LabelCreateRequest, LabelUpdateRequest, LabelSeedResponse) inherits ApiModel and serializes camelCase on the wire.
  4. No SDK imports outside wrappers: tests/test_no_unlogged_api_calls.py (2 tests) still green — no new direct from google.genai/openai/anthropic/cohere/groq import outside the 5 approved wrapper files.
  5. OpenAPI paths: curl /openapi.json | jq '.paths | keys' includes all Stage 6 paths — /uploads, /books/probe, /books/{bookId}/preview, /labels/seed-from-derived, plus the POST/PATCH/DELETE verbs added to existing /books + /books/{bookId} + /labels + /labels/{labelId} paths (43 endpoints total across 34 paths).
  6. Data preservation: live data/inventory.db is untouched — books count remained 4 + labels count remained 19 across the entire test cycle. Every test fixture (cleanup_created_books, cleanup_labels) tracks the rows it creates and tears them down at teardown; the delete_book helper also unlinks the on-disk PDF in data/raw/ so local storage stays clean.
  7. Bug fixed during this session: src/api/routers/books.py create_book called req.era.value (etc.) but ApiModel's use_enum_values=True already coerces enum members to plain strings at validation time — same pattern the Stage 5 implementation log flags. Refactored to a single _v(x) helper that handles both Enum and str inputs (matching the existing pattern in patch_book). Found by test_create_book_returns_201_with_book_dto; tests added now guard this regression.
  8. No regressions to prior stages: all prior stage 1-5 tests still pass after the books.py fix (Stage 3 books, Stage 4 query/jobs/sse, Stage 5 ingest).

Live probe smoke deferred to user opt-in. When ready: LIVE_PROBE_PDF_PATH=<small_pdf> pytest -m live tests/test_addbook_live.py for upload+probe; or LIVE_PROBE_SOURCE_URL=<public_url> pytest -m live tests/test_addbook_live.py for URL-fetch+probe. Estimated cost: ~$0.001 for a 1-page native_text PDF; ~$0.005 for a 5-page image-mode PDF.

Stage 7 — Operations + Evaluation 📋 PENDING

BACKEND_BUILD.md §13 Step 7.

  • GET /api/config, PATCH /api/config, POST /api/config/migrate → job.
  • POST /api/indexes/{name}/activate, DELETE /api/indexes/{name}.
  • GET/POST/PATCH/DELETE /api/eval/golden/*.
  • POST /api/eval/runs → job; GET /api/eval/runs[/{id}].
  • src/api/jobs/types/eval_run.py, src/api/jobs/types/backend_migrate.py.

Stage 7 gate: Run an eval against the real BE; results appear in /evaluation/history/[runId]. Backend migration from local → turso (or back) works as a job, with progress visible.

Stage 8 — Cleanup 📋 PENDING

  • tests/test_no_unlogged_api_calls.py written; CI passes.
  • Parity matrix (FRONTEND_PRD.md Appendix A) fully green.
  • One week of dogfooding (FE Stage 5).
  • git rm -r src/stage9_ui/; drop streamlit from pyproject.toml.
  • Final commit; merge rewrite/api → main.

Open contract questions

If either agent hits a question CONTRACT.md doesn't answer, don't guess. Add it to this section and ping the orchestrator. Examples of legitimate questions:

  • A new field needs to be added to BookDTO.
  • A new endpoint shape isn't covered.
  • An error code is needed beyond CONTRACT.md §8's list.

Currently open: (none — keep this section honest; remove "none" when adding the first entry.)

Implementation log (latest first)

  • 2026-05-17 (BE, fix-up) — FE Stage 3 audit blockers — 3 BE bugs fixed. Triggered by Patristic-AI-Fe/tests/screenshots/stage3_audit/STAGE3_REPORT.md (CC7 / CC8 + critical 3 in §3 of the audit). Three landed fixes, one perf bonus:
    1. GET /system/events SSE endpoint (audit CC8). New code in src/api/routers/system.py: a poll-driven SSE generator emits event: active_jobs with {activeCount, runningJobIds} every ~2.5 s when the snapshot changes, plus event: ping heartbeats every 15 s (CONTRACT.md §4). Initial frame yielded before the loop so the FE's SidebarJobStatus.tsx chip flips off "Connecting…" on first byte rather than waiting one poll. User-tier gate via current_user dep. Tests: tests/test_system_events.py (5 — auth gate / snapshot shape / wire format / generator initial-yield / OpenAPI surface).
    2. Connection-layer Turso retry (audit CC7). New shared module src/stage1_inventory/turso_retry.py (is_stale_stream_error, with_retry). TursoConnection.execute / executemany / commit / executescript / cursor / rollback now auto-reconnect + replay once on Hrana stream not found 404 — catches every Turso query path including the lookup_session() → conn.commit() cold-start race the tester captured (was 500 on first /books/status-counts after BE boot). src/api/jobs/store.py re-exports _with_retry from the shared module so existing per-callsite calls keep working (now double-protected). Tests: tests/test_turso_retry.py (9 — classifier matches Hrana wire format / rejects unrelated errors / single-shot retry / no infinite loop / TursoConnection reconnect + replay path / persistent-failure propagation).
    3. GET /books/{id}/pages/{n}/image 500 → structured 404 (audit critical §3). Root cause: Storage.ensure_local() raises bare FileNotFoundError for books whose inventory row exists but whose PDF blob isn't on disk (legacy import / R2 fetch failure). src/api/routers/pdf.py:get_page_image now catches FileNotFoundError + out-of-range ValueError and converts them to BookNotFound (404 BOOK_NOT_FOUND envelope per CONTRACT.md §8). Matching fix on get_book_pdf (same root cause class). The FE's graceful "Page image unavailable." chrome renders cleanly on 404 but coughs up a generic toast on 500 — that's the exact UX bug the audit caught on every book the user opened that didn't have a local PDF. Tests: tests/test_pdf_image.py (5 — happy path 200 PNG / missing blob → 404 envelope / unknown book → 404 / out-of-range page → 404 / sweep every real-inventory book to assert status ∈ {200, 404}, never 500).
    4. Performance side fix (audit CC11 partial — perf): init_schema() was running on every connect() (idempotent CREATE TABLE IF NOT EXISTS + 5 ALTER TABLEs + a backfill UPDATE = ~7 round-trips ≈ 4 s on Turso). Added a process-local guard in src/stage1_inventory/db.py:_ensure_schema_once so the migration runs exactly once per (process, backend). Per-request connect latency drops from ~4 s to single-digit ms; this also unblocked the /system/events poll loop which would otherwise have starved the event loop on every tick. Verified against live Turso: first connect ~4 s (schema run), every subsequent connect <50 ms.
    5. ASGI CSRF middleware refactor (load-bearing for SSE). The existing csrf_middleware was registered via app.middleware("http")(fn) which wraps it in BaseHTTPMiddleware. That adapter buffers streaming responses through an anyio memory stream — verified during this session that curl /system/events saw 0 bytes for 15+ seconds before a burst flush, breaking the FE chip's "connect → first frame" UX. New CsrfASGIMiddleware in src/api/middleware/csrf.py is pure ASGI (operates on raw scope / receive / send) and lets SSE frames flush as they arrive. Same rule (require X-Requested-With: XMLHttpRequest on POST/PATCH/PUT/DELETE except /auth/login). The legacy csrf_middleware function is kept for back-compat with any caller importing it directly; the app wires the ASGI class via app.add_middleware(CsrfASGIMiddleware). Verified live: CSRF rejection of POST /auth/logout without X-Requested-With → 403 envelope, CSRF allow with header → 200, plus SSE first frame arrives immediately. Verification gates (all green, evidence below):
    • Live curl on :8011 (separate from orchestrator's :8000): GET /system/events emitted event: active_jobs\ndata: {"activeCount": 0, "runningJobIds": []} on first byte; second SSE check (12 s @ 5 s heartbeat config) emitted both initial frame + ping heartbeat.
    • Live curl: cold-start GET /books/status-counts returned 200 on FIRST call after fresh BE boot (no Hrana 500).
    • Live curl: GET /books/deskolia_v3/pages/1/image?dpi=150 → 200 image/png 634943 bytes; GET /books/history_of_christian_thought_jesus_christ_through_the_ages/pages/1/image?dpi=150 (no local PDF) → 404 BOOK_NOT_FOUND envelope.
    • Cost gate (load-bearing for paid quota): pre-fix-up MAX(llm_calls.id)=2497, post-fix-up MAX(llm_calls.id)=2497, 0 new paid calls in the entire fix-up session (all tests mock paid surfaces or hit cheap probes; no fix area touches an LLM SDK).
    • 19 new tests across test_turso_retry.py + test_pdf_image.py + test_system_events.py all pass. Full test suite re-run pending at log-write time (run is in flight — see Verification at end).
    • Streamlit smoke-imports unaffected (no changes to UI surfaces).
    • NOT addressed (out of scope for this fix-up): BE-4 perf for /books/{id} 12-second spinner — the slow joins are noted in the audit; the init_schema-once fix above shaves ~4 s off cold-start request latency but the /books/{id} endpoint itself still aggregates costs/indexes/processing-log/labels sequentially. That's a follow-up (Stage 7 or later perf pass). FE-side audit items (Browse infinite loop, Add Book contract drift, ConfirmDialog nested-button, etc.) are explicitly FE-side and don't apply to this repo. Commits on rewrite/api: see git log entries from this session.
  • 2026-05-17 (BE) — Stage 6 (Add Book probe + writes) landed on rewrite/api. Live probe smoke STAGED but NOT RUN (user opt-in per AGENT_PROMPT_BE.md's paid-quota concern). New files: src/api/dto/addbook.py (UploadResponse, ProbeRequest, ProbeResponse, GuessedMetadata, SuggestedLabel, SamplePage — all inherit ApiModel for camelCase wire), src/api/routers/uploads.py (POST /uploads — multipart PDF, 1 GiB cap, SHA-256 streamed in 1 MiB chunks, returns {uploadId, sha256, sizeBytes}), src/api/routers/addbook.py (POST /books/probe — wraps src/lib/metadata_probe.probe_metadata, returns ProbeResponse with pagesTotal/hasTextLayer/suggestedExtractionMode/guessedMetadata/suggestedLabels/samplePages/estimatedIngestCostUsd/probeError), tests/test_addbook.py (18 tests — upload happy path + 401 + empty rejection; probe with mocked metadata + 400 on missing body; create 201 + 409 BOOK_EXISTS for bookId + sourceUrl conflicts; PATCH updates + 404; DELETE 204 clears labels + Qdrant + idempotent; preview PNG + 404; OpenAPI camelCase regression), tests/test_labels_writes.py (10 tests — POST 201 + duplicate 400 + validation 400; PATCH updates + 404 + immutable kind; DELETE 204 + idempotent; seed-from-derived idempotent; auth + CSRF gates), tests/test_addbook_live.py (STAGED but NOT RUN; opt-in via LIVE_PROBE_PDF_PATH or LIVE_PROBE_SOURCE_URL env vars; full skeleton with login → upload/url → probe → cost gate). Allowed additive edits per BACKEND_BUILD.md §3.4: src/lib/books/repo.py got find_book_id_by_source_url, update_book(partial), delete_book; src/api/errors.py got BookExists (409 BOOK_EXISTS with details.conflictWithBookId); src/api/routers/books.py extended with POST /books (201 BookDTO + 409 envelope), PATCH /books/{bookId}, DELETE /books/{bookId} (best-effort Qdrant cleanup → book_labels mappings → books row, historical rows preserved), GET /books/{bookId}/preview?page=N&dpi=120 (PNG bytes for the wizard, default DPI 120 vs pdf.py's 150); src/api/routers/labels.py extended with POST/PATCH/DELETE + POST /labels/seed-from-derived; src/api/dto/books.py extended with CreateBookRequest/UpdateBookRequest; src/api/dto/labels.py extended with LabelCreateRequest/LabelUpdateRequest/LabelSeedResponse; src/api/app.py mounts uploads_router + addbook_router. Bug found + fixed mid-session: create_book was calling req.era.value (etc.) directly, but ApiModel's use_enum_values=True already coerces enums to strings at validation — same pattern flagged in the Stage 5 implementation log. Refactored to a single _v(x) helper that handles both Enum and str inputs (matching the existing pattern in patch_book). Caught by test_create_book_returns_201_with_book_dto; regression-guarded by the new test suite. Cost gate (load-bearing for paid quota): pre-test MAX(llm_calls.id) = 2495. After the full Stage 6 test cycle, SELECT COUNT(*) FROM llm_calls WHERE id > 2495 = 0 — 0 new paid calls. Every probe test monkey-patches src.api.routers.addbook.probe_metadata to return a synthetic ProbeResult; every DELETE test monkey-patches src.stage6_indexing.qdrant_io.{list_collections, delete_book_points} so the real Qdrant cloud collection is never touched. The live probe path is gated behind @pytest.mark.live + @pytest.mark.skip + pyproject.toml's default -m 'not live' — belt, suspenders, and a second pair of suspenders (same pattern as test_ingest_jobs_live.py). Data preservation: live data/inventory.db still has the same 4 books + 19 labels after the test cycle — every fixture cleans up the rows it creates. Verified gates: 28 new tests pass + camelCase regression in test_costs.py::test_every_schema_property_is_camelcase still green + no-SDK-imports-outside-wrappers regression in test_no_unlogged_api_calls.py still green + Streamlit still boots cleanly + OpenAPI surfaces all Stage 6 paths (/uploads, /books/probe, /books/{bookId}/preview, /labels/seed-from-derived, PATCH/DELETE on existing /books/{bookId} + /labels/{labelId}, POST on /books + /labels). All prior stage 1-5 tests still pass after the books.py enum-coercion fix (no regressions). Commits on rewrite/api: 56d5355 (DTOs + BookExists error — prior agent session, pre-crash), 5a70aa4 (uploads + probe + book/label writes + preview — prior agent session, pre-crash), <tests> (this session: tests + books.py enum fix), <tracker> (this session: tracker flip + this log entry). Next: Stage 7 (Operations + Evaluation — GET/PATCH /api/config, POST /api/config/migrate → job, POST /api/indexes/{name}/activate, eval CRUD + run jobs).
  • 2026-05-17 (BE) — Stage 5 (Jobs subsystem — ingest) landed on rewrite/api. Live smoke test STAGED but NOT RUN (user opt-in per AGENT_PROMPT_BE.md's paid-quota concern). New files: src/lib/processing/ingest_progress.py (lifted parser + stage_number_to_name mapping), src/api/jobs/types/ingest.py (subprocess-based handler — Popen with CREATE_NEW_PROCESS_GROUP on Windows, line-buffered stdout drain via _StreamingState, throttled progress events, costSoFarUsd on every emit, terminal done with final status from subprocess exit code + total cost from store.cost_since), src/api/dto/ingestion.py (IngestRequest + StageName: Enum), src/api/routers/ingestion.py (POST /books/{bookId}/ingest, admin-only, 202 JobStartResponse, persists extraction_mode/cleanup_enabled overrides, enqueues with subject_key=book:{id}), tests/test_ingest_jobs.py (17 tests — argv builder, parser state transitions, router contract, OpenAPI regression), tests/test_ingest_jobs_live.py (staged-but-skipped end-to-end test with full skeleton). Allowed additive edits per BACKEND_BUILD.md §3.4: src/pipeline/ingest.py gained --stages flag + STAGE_NAMES + _parse_stages_arg (validates against canonical set, aborts on typo); ingest() takes stages: set[str] | None and each numbered block guards on _wants(name) — default behavior is unchanged when the flag is omitted. src/stage9_ui/shared.py reduced to a re-export shim importing the lifted parser; Streamlit still boots and the parser output is bit-for-bit identical (verified by feeding the same 3 transcripts to both module paths). pyproject.toml registers the live pytest marker + defaults to -m 'not live' so the live smoke is deselected at collection. src/api/app.py mounts the new router. Bug found + fixed mid-session: ApiModel's use_enum_values=True makes Pydantic coerce enum members to their string values at validation, so req.stages[0] is str, not StageName — router normalizes isinstance(s, str) else s.value on stages and extraction_mode so a future ApiModel config change can't silently break this path. Cost gate (load-bearing for paid quota): 0 new llm_calls rows in this session — every unit test mocks the subprocess via monkeypatch on runner.enqueue or feeds synthetic stdout directly to _StreamingState (which never calls an LLM SDK; only reads SQLite via store.cost_since). The live test path is gated behind @pytest.mark.live + @pytest.mark.skip + pyproject.toml's default -m 'not live' — belt, suspenders, and a second pair of suspenders. Verified gates: --help shows --stages flag, bogus stages reject cleanly, 404 BOOK_NOT_FOUND envelope with details.bookId for unknown books, 202 happy path returns valid jobId + sseUrl matching /jobs/{jobId}/events, OpenAPI has 29 paths total with /books/{bookId}/ingest registered + IngestRequest/StageName camelCase + enum, Streamlit shim parser output identical to canonical, all 17 Stage 5 tests pass + 90 prior tests still green (the 19 pre-existing test_labels.py + test_stage1_inventory.py libsql-Row failures are unrelated — confirmed by git stash && pytest on base commit). Threading vs subprocess (locked design): query jobs run in-thread (Stage 4); ingest jobs run as subprocess (Stage 5) for tree-kill reliability on multi-hour OCR. Commits on rewrite/api: d614729 (parser lift + --stages CLI flag), e987772 (ingest job type + DTO + router + app wiring), 789d515 (tests + staged-but-skipped live smoke + pytest marker registration). Next: Stage 6 (Add Book probe + writes — POST /uploads, POST /books/probe, POST /books with 409 BOOK_EXISTS, PATCH/DELETE /books/{id}, label CRUD).
  • 2026-05-17 (BE) — Stage 4 (Query + SSE) landed on rewrite/api. New files: src/api/jobs/{__init__,store,sse,registry,runner,reconcile}.py, src/api/jobs/types/{__init__,query}.py, src/api/routers/{jobs,query}.py, src/api/dto/jobs.py. Extended src/api/dto/query.py with QueryRequest/PageEnum/Filters/ComparisonGroup/ComparisonSpec/QueryModelOverrides (locked field names per CONTRACT.md §11). Schema: jobs + job_events tables + 3 indexes added to SCHEMA_SQL (run-twice idempotent verified on live data/inventory.db). Tests: test_query_sse.py (8 — incl. one paid end-to-end LOOKUP gated on GOOGLE_API_KEY + indexed Qdrant), test_jobs_sse_replay.py (3 — strict-greater-than fetch_events_since + event_stream deep-link replay + Last-Event-ID reconnect lossless), test_jobs_reconcile.py (3 — dead pid → failed, pid=NULL → failed, no-orphan no-op), test_no_unlogged_api_calls.py (2 — global LLM-SDK-import gate per BACKEND_BUILD.md §11.2 + ALLOWED_FILES existence guard). Stage 4 gate verified live on :8011: 202 with jobId/sseUrl no runId → full event sequence stage(classify)→progress→stage(retrieve)→progress→stage(generate)→progress→done with costSoFarUsd on every progress and totalCostUsd matching to the cent in done; Last-Event-ID replay yielded exactly the missed events; GET /query/runs/16/events proxied via jobs.subject_id and replayed full history. Cost gate (load-bearing for paid quota): pre-query MAX(llm_calls.id)=2488, post-query new rows = exactly [2489 classify $0.005831, 2490 generation $0.022569], SUM=$0.0284 = done.totalCostUsd to the cent. 6 new OpenAPI paths (/jobs, /jobs/{jobId}, /jobs/{jobId}/cancel, /jobs/{jobId}/events, /query, /query/runs/{runId}/events), 28 paths total. Discovered + fixed a Turso transient stream-404 issue: the libsql adapter raises ValueError("api error: status=404 Not Found, body=...stream not found...") after long-idle gaps; the first emit_event of a query was silently dropped because src.stage8_router.router.answer's on_progress wrapper swallows callback exceptions. Hardened by wrapping every store function in _with_retry() that retries once on Hrana stream-not-found errors. After fix, 3 back-to-back live queries each emitted the full event sequence with zero drops. Threading vs subprocess: query jobs run in-thread (3-10s I/O-bound; shared SQLite handle helps); subprocess + psutil tree-kill stays as the right call for the multi-hour Stage 5 ingest. Streamlit still boots cleanly. Commits on rewrite/api: 4f05f6f (schema migration), 790bfe3 (jobs subsystem + routers + DTOs + startup reconciler), 1231244 (tests + OpenAPI fix for event DTOs), 9f3fe5d (Turso retry hardening). Next: Stage 5 (Jobs subsystem — ingest as subprocess + --stages CLI flag).
  • 2026-05-17 (BE) — Stage 3 (Read-only routers) landed on rewrite/api. New files: src/api/routers/{books,history,costs,tools,indexes,processing_log,labels,pdf}.py and src/api/dto/{books,query,costs,tools,indexes,processing_log,labels,pdf_pages}.py. Tests: tests/test_{books,history,costs}.py — 27 tests, all pass; Stage 1+2's 14 tests still pass. src/api/errors.py extended with BookNotFound (404 BOOK_NOT_FOUND with details.bookId), QueryRunNotFound (404 with details.runId), NotIndexed (409), BadRequest (400). src/api/app.py mounts all 8 new routers and installs a RequestValidationError handler that re-shapes Pydantic 422s into the canonical BAD_REQUEST envelope. Wire surface: 22 OpenAPI paths (Stage 1+2 = 5, Stage 3 adds 17); 44 component schemas (was ~7). Path params use camelCase URL templates (/books/{bookId}, /query/runs/{runId}) with Path(..., alias=...) keeping Python args snake_case. All DTOs inherit ApiModel → camelCase wire keys. Enums are enum.Enum → real OpenAPI unions (BookStatusEnum, RouterModeEnum, CostGroupByEnum, etc.) so FE codegen produces TS unions. BookStatusCountsDTO.ocr_done is the only explicit underscore-on-the-wire exemption — pinned to honour CONTRACT.md §10's literal-status-key shape. The global camelCase regression in test_costs.py:test_every_schema_property_is_camelcase sweeps every schema in the spec and asserts no other leaks. Cost-monitoring gate: MAX(llm_calls.id) before = 2481, after the full test + live verification = 2481. 0 new paid calls (Stage 3 only reads llm_calls). Admin gates verified: viewer → /processing-log → 403 FORBIDDEN envelope; admin → 200. Streamlit boots cleanly. Commits on rewrite/api: f7bc2ce (DTOs + errors), 116c119 (routers + app wiring), 19e89e6 (tests). Next: Stage 4 (Query + SSE — jobs/ subsystem, POST /query returning 202 jobId/sseUrl, GET /jobs/{id}/events SSE with replay-from-Last-Event-ID, progress.costSoFarUsd per CONTRACT.md §12.3).
  • 2026-05-17 (BE) — Stage 2 (Auth) landed on rewrite/api. New files: src/api/auth/{__init__,sessions,router}.py, src/api/deps.py, src/api/middleware/{__init__,csrf}.py, src/api/dto/auth.py, tests/test_auth.py. Schema: sessions table + 2 indexes added inside SCHEMA_SQL (CREATE TABLE IF NOT EXISTS — idempotent). Additive edits: src/lib/auth/repo.py got from_session_token(token) + user_for_username(username); src/api/errors.py got Unauthenticated (401, UNAUTHENTICATED) and Forbidden (403, FORBIDDEN); src/api/app.py mounts the auth router and registers the CSRF middleware. Wire shape: cookie session=<urlsafe(32)>; HttpOnly; SameSite=Lax; Path=/; Max-Age=2592000 (30d), matching the row's expires_at. Verified live end-to-end on port 8011: bad creds → 401, good → 200 + Set-Cookie + MeResponse, /me works, POST /logout without X-Requested-With → 403 FORBIDDEN, with header → 204, post-logout /me → 401. Cost-monitoring gate: 0 new llm_calls rows during the full test cycle (auth path is pure SQLite + TOML; never imports an LLM SDK). camelCase regression test extended to cover the new DTOs. Streamlit still boots. Next: Stage 3 (Read-only routers — /books, /history, /costs, /tools, /indexes, /processing-log, /labels, /books/status-counts).
  • 2026-05-17 (BE) — Stage 0 + Stage 1 landed on rewrite/api. New src/api/ package: app.py (FastAPI app + ApiError handler), config.py (port + git-sha + masked-config), errors.py (ApiError stub), dto/common.py (ApiModel base, ErrorResponse, JobStartResponse — alias_generator=to_camel applied), dto/system.py (HealthResponse, HealthCheck, VersionResponse), routers/system.py (GET /system/health, GET /system/version). Health probe wraps the existing src.lib.system.health checks for DB/Qdrant/API-keys and adds a Storage probe via src.lib.storage.get_storage().list_book_ids(). Stale-pricing check walks tools_registry.yaml; current state is stalePricing: [] because every tool's cost_model.last_updated is within 90 days. New test tests/test_system_health.py (5 tests, all pass) covers the OpenAPI shape, camelCase enforcement, the /health shape, and secret-masking in /version. pyproject.toml gained fastapi, uvicorn[standard], python-multipart, psutil, httpx, pytest-asyncio. No paid-API calls made; llm_calls row count unchanged (519 → 519). Streamlit smoke-imports unaffected. Next: Stage 2 (Auth — sessions table + login/logout/me + CSRF middleware).
  • 2026-05-17 (orchestrator) — CONTRACT.md v2 (Express proxy, cost monitoring §12), FRONTEND_BUILD.md updated for Vite+Express stack, BACKEND_BUILD.md §11 cost monitoring added. Stage 0 FE complete (per AI Studio); Stage 1 FE ready to start. BE branch not yet created (waiting on Streamlit indexing to finish).
  • (future entries here)