Docs: overhaul roadmap and config persistence

README.md

@@ -90,7 +90,7 @@ Backend listens on `http://localhost:8000`, Gradio on `http://localhost:7860`.
 
 1. Open the UI URL.
 2. If `APP_PASSWORD` is set, enter it on the login page to unlock the app.
-3. (Optional) Open **Configuration** to choose personas and add per-role prompt additions.
+3. (Optional) Open **Configuration** to edit shared prompts and personas (server-canonical; stored in SQLite at `DB_PATH`).
 4. Click **Start Conversation**.
 5. Wait for the conversation to complete to see analysis results.
    - Note: **Stop Conversation** currently aborts the run and may skip post-conversation analysis.

docs/development.md

@@ -60,7 +60,7 @@ The primary demo UI is served by `frontend/react_gradio_hybrid.py` and includes
 
 When running outside Docker, you typically run the backend and the web UI separately; when running in Docker/HF, the backend is mounted under `/api` inside the same server.
 
-The UI includes a **Configuration** view used to edit persona-specific settings (local-only, stored in the browser). Persona selection for actually running conversations is done in the **AI-to-AI** and **Human-to-AI** panels.
+The UI includes a **Configuration** view used to edit app-wide settings and personas stored in SQLite at `DB_PATH` (shared across users on HF). Persona selection for actually running conversations is done in the **AI-to-AI** and **Human-to-AI** panels.
 
 ## Making Changes Safely
 

docs/persistence.md

@@ -51,7 +51,7 @@ Persistence means:
 1. When a session ends as a **sealed run** (conversation finished and analysis succeeded), we write a **Run record** to durable storage:
    - transcript (messages)
    - analysis outputs (resource agent JSON, evidence catalog, schema versions)
-   - configuration snapshot (LLM backend/model/params, selected personas, prompt additions, etc.)
+   - configuration snapshot (LLM backend/model/params, selected personas, and the effective shared settings used)
    - persona snapshots for historical fidelity (see below)
 2. We expose APIs to:
    - list prior runs (chronological)
@@ -214,7 +214,7 @@ Primary key:
   - `timeout`, `max_retries`, `retry_delay`
   - any generation params used (temperature, max_tokens, top_p, etc.)
 - Mode-specific:
-  - AI↔AI: surveyor/patient persona IDs (and/or names), prompt additions
+  - AI↔AI: surveyor/patient persona IDs (and persona version ids / snapshots)
   - Human↔AI: same + human mode flags
   - Text analysis: `source_name`, optional file metadata (original filename, sha256)
 - App versions:

docs/roadmap.md

@@ -15,93 +15,56 @@ _Last updated: 2026-01-22_
 - Hosted LLM support via OpenRouter (`LLM_BACKEND=openrouter`)
 - Hugging Face Spaces (Docker) deployment
 
-## Near-Term Priorities
+## Status
 
-1. **Evidence Output: remove hard caps**  
-   Remove any fixed limits (e.g., “max 3 evidence snippets”) and let the model return as many evidence references as needed.
+As of 2026-01-22, the v1 feature set is in place:
 
-2. **Evidence Traceability (jump-to-source)** ✅  
+- Evidence output: no hard caps in UI rendering.
+- Evidence Traceability (jump-to-source) ✅  
    Implemented: evidence entries are clickable and jump/scroll to the cited transcript message with a temporary highlight.
 
-3. **Evidence Export (Metadata Download)**  
+- Evidence Export (Metadata Download) ✅  
    Add a “Download” UI action to export transcript + analysis output + provenance metadata (e.g., evidence pointers, prompt/schema versions). Start with JSON (lossless), then add CSV/Excel-friendly formats as needed.
    ✅ Implemented: Download Excel (.xlsx, multi-sheet) as primary; Download JSON as lossless export.
 
-4. **Analysis on pasted/uploaded text**  
+- Analysis on pasted/uploaded text ✅  
    Add a panel to paste text or upload a file, run the same analysis pipeline, render results, and allow download.
    ✅ Implemented: “Upload Text” tab supports paste, text-file upload, and best-effort PDF text extraction; exports work (Excel + JSON).
 
-5. **Modularization / Separation of Concerns (refactor)**  
+- Modularization / Separation of Concerns (refactor) ✅  
    Before adding more major UI modes, refactor to keep the codebase maintainable:
    - Split the growing frontend UI logic (currently concentrated in `frontend/pages/main_page.py`) into smaller, focused modules/components.
    - Split API routers (e.g., `backend/api/*_routes.py`) so endpoints aren’t concentrated in a single catch-all module.
    - Keep behavior unchanged (refactor-only); validate by running Conversation, Upload Text, Configuration, exports, and evidence jump.
    ✅ Verified: refactor-only; UI and API behavior unchanged after smoke testing.
 
-6. **Human ↔ Surveyor chat mode**  
+- Human ↔ Surveyor chat mode ✅  
    Add a panel where a human chats as the patient with the surveyor agent (text input), while keeping the same analysis pipeline at end-of-session.
    ✅ Implemented: “Human-to-AI” tab supports human patient turns; “End session” runs analysis + enables exports.
 
-7. **Persistent storage (HF Spaces `/data`)**  
+- Persistent storage (HF Spaces `/data`) ✅  
    Add a simple storage layer and persist runs (transcript + analysis) and user-created personas so they survive restarts/redeploys.
-   ✅ Implemented (runs): sealed-run persistence to SQLite at `DB_PATH`, `GET /api/runs` + `GET /api/runs/{run_id}`, and server-canonical exports via `/api/runs/{run_id}/export/*`.
-   ⏳ Remaining (personas): persona CRUD + server-side persistence/versioning.
+   ✅ Implemented: sealed-run persistence to SQLite at `DB_PATH`, run history APIs, server-canonical exports, and DB-canonical personas + shared settings.
 
-8. **Run history browser**  
-   Make persisted runs usable from the UI:
-   ✅ Implemented:
+- Run history browser ✅  
    - List sealed runs (sort by recency).
    - Select a run to rehydrate transcript + analysis (read-only replay).
    - Export from the selected run (server-canonical).
    - (Optional later) delete/retain controls and basic metadata (title/notes).
 
-9. **Configuration UX (local-first, non-CRUD)**  
-   Improve “Configuration” as a first-class control surface (even before full CRUD), focusing on clarity and “what applies on the next run”:
-   - **9.1 Surveyor controls v2 (Attributes list)**:
-     - Simplify surveyor controls into an “Attributes” list (keep question bank first-class).
-     - Decouple persona selection for **editing** (Configuration) from persona selection for **running** (AI-to-AI / Human-to-AI panels).
-     - Follow-ups:
-       - Improve Human-to-AI turn-taking clarity (make it obvious who should speak next, especially when AI role = Patient).
-       - Make persona ids backend-oriented and avoid showing raw ids in user-facing dropdowns (prefer friendly display names only).
-   - **9.2 Patient controls v1 (Attributes list)** ✅  
-     Implemented: patient persona attributes list (local-first), compiled into the patient prompt and captured in run snapshots.
-   - **9.3 Analysis Agent controls v1 (Attributes list)** ✅  
-     Implemented: analysis agent attributes list (local-first) + “Apply defaults”, wired into both post-conversation analysis and Upload Text.
-   - **9.4 System controls v1**: (candidate removal) effective runtime config panel may be omitted for PI-facing simplicity.
-   - **9.5 Validation + guardrails**: schema versioning, reset-to-defaults, and explicit “applies next run” UX.
-   - **9.6 Roadmap override (Personas v2 scaffolding)** ✅  
-     Implemented:
-     - Universal type-wide system prompt per persona type (Surveyor, Patient) with ���Apply defaults”.
-     - Default personas converted to attributes-only (no persona-specific system prompts) with a small immutable default set.
-     - Configuration UI adjusted to review defaults read-only; system prompts remain editable.
-   ✅ Implemented (partial, foundational pieces):
-   - Split Configuration into panes (Surveyors / Patients / Analysis Agent / System).
-   - Surveyor knobs wired end-to-end (UI → WS payload → prompt compilation → stored config snapshot).
-   - Surveyor question bank (UI list) wired end-to-end with deterministic “selected question id” tracking.
-
-10. **Server-side persistence + CRUD (Personas + Analysis Templates)**  
- Add create/update/delete flows backed by persistent storage, with versioning and run snapshots:
-   - **10.1 DB-canonical personas (including defaults)**:
-     - Add DB schema/migrations for personas + persona versions.
-     - Seed a small immutable default set (e.g., 3 surveyors + 3 patients) into the DB on startup.
-     - Retire YAML as the runtime source of truth once DB seeding exists.
-     - Universal system prompts (Surveyor + Patient) become DB-backed shared settings.
-   - **10.2 Persona CRUD API**:
-     - Create/update/delete user personas (no description field in v1).
-     - Defaults are immutable (view-only; cannot update/delete).
-     - Updates create new versions; deletes are soft deletes.
-   - **10.3 CRUD UI**:
-     - Create/duplicate/delete personas in Configuration.
-     - Default personas remain read-only; user personas are editable.
-   - **10.4 Run snapshot integrity**:
-     - Sealed runs reference persona_id + persona_version_id and store snapshots to prevent drift.
-   - **10.5 Analysis templates (later)**:
-     - Optional follow-up once persona CRUD is stable.
-
-11. **Evaluation / test coverage chapter (separate)**  
-   Treat evaluation as its own milestone rather than a “basic tests” checkbox:
-   - Integration smoke tests + golden runs for core modes.
-   - Deterministic checks where possible (schema validation, exports, persistence invariants).
+Configuration & personas are now fully server-canonical:
+
+- Surveyor/Patient/Analysis Agent system prompts are shared DB-backed settings (with “Apply defaults” + warning gating in UI).
+- Personas are DB-canonical with CRUD (defaults immutable; user personas editable; updates are versioned).
+- Run snapshots store persona versions and persona snapshots to prevent drift.
+
+## Remaining roadmap item
+
+1. **Analysis templates: editable codebook-driven top-down UI**  
+   Make the analysis “top-down codes” and related UI structure editable/extensible (codebook/categories), while preserving:
+   - strict JSON output contract,
+   - historical run fidelity (snapshots),
+   - exports compatibility.
 
 ## Longer-Term Ideas